I am lucky in the sense that I have been given the opportunity to be a ‘Technical Troubleshooter’ for our offshore development team. While I am confident and capable of dealing with most issues, I have come across something that I am not. Based on initial discussions with various team members both on and offshore, a requirement for a ‘repeatable, consistent’ skills assessment has been identified.
In my opinion, the best way to achieve this would be a combination of objective and subjective tests. The former normally being an initial online skills assessment on various subjects, for example General C#, WCF and MVC. The latter being a technical test where the candidate would need to solve various problems and (hopefully) explain the thought processes involved with the solution whilst doing so.
Obviously, the first method is consistent, repeatable and extremely accurate. The second is always going to be subjective and based on the approach, the solution (or possibly not) and other factors. The ‘scoring’ of this is also going to be down to the experience and skills of the assessor and this is where my problem lies;
- The person that is expected to be the assessor initially (me) has no
experience. - The people that will ultimately continue this process for other
people will never remain the same due to project constraints and
internal reasons, this changes the baseline for comparison. - I am not aware of any suitable system that can be classed as
consistent and repeatable for subjective tests with the 2 factors
above, let alone if those did not exist.
So anyway, I have to present a plan that will ultimately generate a skills/gap analysis and it is unlikely that I will be able to use an objective method (budget constraints most likely reason). The only option left is the subjective methods and the issues above.
Does anyone have any suggestions for an approach that may tick all the boxes?
1
A “skills assessment” to me sounds like the classic problem of mixing requirements with solution.
I would backtrack and document the business problem and expected benefit. I don’t know the specifics in your case, but an example might be:
Problem: “40% of the candidates we hire do not meet our needs, which causes our team to lose 200 hours of productivity each month.”
Expected benefit: “Improve candidate selection to reduce our costs to no more than 40 hours per month.”
Estimate: “One FTE month of effort.”
Month 1: Result = -140 hours invested
Month 2: Result = 20 hours saved = Monthly savings (200 – 40) – 140 implementation hours
Month 3: Result = 160 hours saved = Monthly savings (160) + 20 hours accumulated
1 Year: Result = …
2 years: …
3 years: …
Getting to the heart of the problem opens up alternatives that may not be considered otherwise.
The problem with “consistent tests” is that the questions you ask will eventually leak to your candidates, and will no longer be consistent.
You could try solving this by creating many questions and only providing a subset to each candidate, however you would not achieve consistency until a sufficiently large population took your tests.
I suggest you read up on ways to predict IT performance. You are likely to find some surprises, and tests are not required.
With regard to assessing talent, let me point you at Joel Spolsky’s book, “Smart and Gets Things Done”. It is a good, quick read and among other things, stresses that the range of ability between applicants may vary by a factor of 10, 20, or even more.
Having many times interviewed candidates for software development positions, I find candidates are pretty good at differentiating themselves in both subjective and objective tests. Unless you are hiring a great many positions, you might benefit by weighing the cost of an automated test vs. a manually administered test.
I would like to suggest an approach borrowed from software testing that is called operational profile based testing (OPT). In OPT, we identify and operational profile (OP) by observation what features are used most often. To refer to the Spolsky book in which he describes a Yale programming class that had developers write some pretty mundane programs with names like CMDLINE99, MAKE01, or TAR00 to show they could, and measure how fast (i.e. productive) they could be, but the instructor considered his programs to be typical of what professionals had written previously and what his students would write later.
To make an operational profile for the questions or programming problems for your candidates, pair program or just spend an hour or two watching a good performer in a similar role. Sample the activities and code that they do. Discuss problems they are solving, and jot down vocabulary that might differentiate between novice and expert.
If the test is well designed, I suspect success on this kind of test could correlate strongly to fluency in key skills, low need for supervisory assistance, and the ability to complete most portions of the task in a timely fashion.
Hope this helps, thanks for reading this answer.
The best interview question I ever had to do was to do a code review. They gave me the IDE with a project in it and said “we’ve written this but imagine its a junior dev who’s just done this and its your job to review it – show us the bugs, the things you’d do differently”. And so I got to show them I knew about code organisation and best practice as well as the usual trivial bits of programming.
Try that, considering there are no explicit “answers” you can come up with something that is gauging their thought processes as well as a few objective tests (ie a bug or obvious error) that they should find too.
For the second type of test (subjective), you may be able to follow a process such as this. Try and break up each of the topics for which skills are important to you, into groups. So C# can have groups such as:
- String handling
- OO programming
- IO
- Threads
You could do this for other topics as well.
Then for each of these groups try and write some simple statements. For String handling you could write statements like:
- A developer should know how to concatanate strings
- A developer should be able to determine if a String contains a specified substring
- A developer should be able to split a String based on a regex
- A developer should be able to determine if a String matches a regex
Then for each of these you can create some test cases which run against empty functions. Once completed these functions will get the test cases to pass.
This could be your subjective tests which can be run automatically.
Though keep in mind that creating your initial collection of tests will probably take a good deal of time.