I’ve taken and marked a lot of programming tests in the past. I love doing them as it’s always good to see what challenges different programmers have come up with when designing them. Unfortunately, however, most of them aren’t very good. Below is a list of general ideas to help increase the quality of programmer tests.

1. Keep it relevant

Too many of the tests have questions on content that simply isn’t relevant to the job. The whole idea behind a programming test is making sure the candidate is competent enough to fulfill the role. It shouldn’t be asking questions about arbitrary knowledge that you think ‘good programmers’ should have. That’s not to say that you shouldn’t be testing the basics, as your programmers will need to know these. The important thing is that you don’t test certain concepts just for the sake of it.

A good example of this is “friends” in C++. Nearly every programming test I’ve taken has asked a question about these. I’ve been programming for 5 years now, and I’ve only used them twice. Firstly at university and secondly when taking programming tests. I’ve asked a few other senior programmers and none of them have used them either. Why bother asking a question if it’s not relevant to the work they’ll be doing? I’m sure some people somewhere use them, and if that’s important, ask away, however if most programmers on your team have never touched them, why is it important that new programmers know them? Chances are, if the new programmer does use them, no one else on the team will understand them, or have a different solution to solving the same problem.

Of course, it would be ideal to hire programmers that just know everything there is to know about programming. In reality this isn’t going to be the case. It’s therefore important to make sure their knowledge domain includes exactly what you need. Asking irrelevant questions won’t help you find better programmers, it will help you find people who know how to pass programming tests.

2. Don’t ask stupid questions

Why bother asking questions about simple compile errors in code that is written on paper. These days, compilers will pick this up far quicker than a person can. Seeing if a programmer can find these simple mistakes is like asking an English professor to find spelling mistakes rather than fixing your sentence structure. Ask programmers to figure out logic errors in code. Get them to follow code logic and determine problems or simple output.

Also, don’t ask programmers trick questions. A lot of the time with trick questions you’ll spend a long time figuring them out, and once you know them, you’ll know instantly. Trick questions generally test whether someone has been asked that question before rather than whether they are a good programmer.

3. Be careful when asking optimization questions

Compilers and platforms are changing extremely quickly. This is particularly true within the games industry where we have entirely new hardware every few years. For these reasons you must be very careful when asking programmers to optimize code. What used to be a smart optimization even a few years ago can often be the slower option now.

For example some modern CPU’s (particularly on consoles) are very slow at executing branches (eg if statements). The simple question of optimizing a routine by using a lookup table that is cached may not necessarily be the fastest optimization anymore. What’s more important is asking people the general concepts of optimization. If they mention profiling with some explanation of best practices then you’re onto a winner.

If your work isn’t specifically about optimization then I would leave out any specific questions altogether. In games, optimized code is very important, so it’s necessary, but again, ask more high-level concepts rather than giving specific code blocks and asking them to write the fastest code. The best candidate would simply say that without knowing the exact hardware and without profiling they would rather write clean code than what they think would be optimized. Code Complete has an excellent section on optimizing code.

4. Watch your team coding, then write the test

If you’re stuck for questions then a practical hint is to simply sit for a little while and watch or talk to your team about what tasks they are performing. Ask the team what they spend most of their time on. You can then write your test around what they spend most of their time doing.

This will have the added bonus of keeping it relevant and stop you from asking optimization questions if they are not needed. I find that I spend 80% of my job doing fairly similar tasks and capturing what skills are used for these tasks and making sure new employee’s have them will almost guarantee you will find a good candidate. Obviously if you’re looking for a specific role to be filled this won’t work, but when looking for general programmers this will help.

5. KISS (Keep it simple stupid)

I’ve found that too many tests have the problem of being too complex. You know exactly what question they are asking, however they use a large convoluted example to ask the question, rather than simply asking the question itself. Rather than having a large “const” example and having the tester take minutes to figure things out, why not ask them specifically what “const” means.

The questions shouldn’t be complex, the answer should. While you don’t just want textbook answers, it’s better to keep anything complex to an interview. This leads to my final tip.

6. Use the interview for determining thought process

Programming tests are great for weeding out the programmers that just can’t program (and unfortunately there are a lot of them). They aren’t good at finding out the thought processes the programmer has when solving problems. Trying to find this out from a written test just won’t work, leave it for the interview.

I like to ask simple questions where the programmer has to solve a coding problem on a whiteboard. Rather than finding out whether they can write code on paper, give them something that doesn’t use complex code, but that requires a complex design. The main aim of this is to find out the thought processes when solving problems. As a programmer, you’ll know the “right” way to solving the problem and you’ll soon tell if the programmer has no idea what they are doing. You can also be surprised by an interesting solution that you may not have thought of. Either way, don’t be critical of the code they write, but rather of the way they go about solving it. Tell them to say out loud what they are thinking as they solve the problem. Think about how those thought processes may extend to larger problems and whether you want that type of person on your team.

Conclusion

There are a few simple tips for writing a good programming test. I’d love to see more relevant tests that actually find good programmers rather than the cookie cut list of questions that try and find a specific type of programmer (whether good or not).

If you have any questions you’ve particularly liked or disliked in tests in the past please add them as a comment so we can discuss, and perhaps come up with a better question that looks for the same answer.