Table of Contents
- 1 How Come They Passed the Interview?
- 2 The Blind Method
- 3 Our Experiment
- 4 Step 1. Choose a Test Task
- 5 Step 2. Pick the Candidates
- 6 Step 3. Offering the Test Task
- 7 Step 4. Go!
- 8 Step 5. Evaluating the Results
- 9 Conclusion
- 10 Afterword
One of the software developers started smelling of cheese. I once tried something that a waiter called “the smelliest cheese in France,” so I know what I’m talking about.
After the smell of the developer surpassed the intensity of the smelliest cheese in France, we realized a sudden truth: he stopped showering immediately after the first interview. Ouch! We tried hints, warnings, and threats, only to get him shower once, then continued to perish in his B.O.
Another software developer called in sick frequently; poisoned by rotten salmon bellies, so he said. It was fine, we thought. After all, who doesn’t like rotten salmon bellies?
But after half a dozen cases of food poisoning we started suspecting something wrong with his diet, only to discover the guy had a severe drinking problem.
Long story short, I’ve made tons of hiring mistakes before.
The problems began all unexpectedly. Initially, they all looked like great professionals, and it was only the experience of working together that proved how wrong the first impression was.
How Come They Passed the Interview?
Basically, an interview isn’t good for anything. My experience proves that my judgement fails to recognize just about anything:
- Professional skills — once I hired a designer who spoke little and looked intelligent. Only to learn he didn’t understand what I was saying due to the language gap
- Personality — our driver, when we had one, was complaining all the time. His favorite topics where the rise of vegetable oil prices and anti-smoking laws, though he was creative and seldom complained about the same thing twice
- Well, substance abuse
- Oh yeah, personal hygiene.
I’ve failed to learn anything during the multiple interviews I conducted.
We had to do something. So, is there a better way?
The Blind Method
We adopted a blind method where we distanced ourselves from anything besides the work.
We didn’t invent it. Symphony orchestras have been using it for years. Blind auditions, taken as an industry standard, reformed the way the orchestras hire. And they changed the way the orchestras are — for example, raising the proportion of female members.
Would the blind method work for software development?
Our Experiment
We conducted an experiment. We hired in a new way, not to be distracted by almost anything else:
- Age, gender, nationality, and brand of smartphone — not even unconsciously.
- Communication skills — we don’t care about the communication skills one exercises in a team (we can’t evaluate them in any case at such an early stage). Although it’s a pleasure to talk to interesting people, we won’t hire someone just because his last job was an organist in New Zealand (true story, we had a guy with a resume like that)
- Experience — we used to reject people who misspelled CORBA as COBRA, even if we never used CORBA (and personally I had little experience besides spelling it). Those days are over; hiring is not a spelling contest anymore
- Salary — we accepted any salary they asked for. Fortunately, Melissa Myers never approached us. Should she have passed the test, we would have had to sell our homes
In short, we ignored everything but a single skill: delivering a working product. But which one?
Step 1. Choose a Test Task
As the book Remote suggests, the task should be:
- Big enough for a week or two of work
- Be isolated enough, so the result doesn’t depend on other people
- A feature from scratch. No one likes dealing with legacy code, especially when people are watching.
We asked our applicants to develop Pedro, a tool for generating and sharing images.
The second version of Pedro: with cosmic design, sharing via Twitter API, and improved usability. The first version was quite basic.
I won’t place the job description here, but it was quite complete:
- A short intro
- Axure prototype
- Sketch mockups
- Use case
- Acceptance criteria
- Definition of done
Step 2. Pick the Candidates
I sent this message:
Hi, thanks for writing back!
May I ask you to check our current project and let me know if you would like to participate?
We exchanged a couple of messages and offered a Skype call where we explained our test task with a shared screen.We also made sure the candidate understood it right. It helps to ask something like: “Look, can you give a short summary so I understand that we’re talking about the same thing?”
Step 3. Offering the Test Task
We’ve made an offer:
Hello, thanks for your recent call. It was fun!
We’d like to offer our test task to you.
Goals
- For you, it gives an idea about what we’re going to do together
- For us, it gives an idea about your ability to make a complete product.
- For our customers, they get a cool feature. We’re going to publish the end result, of course.
Would you like to start? If so, would you mind to:
- Describe the task in your own words
- Estimate the completion time
- Split it into subtasks, no more than 3 days each
Everyone accepted.
Salary Doesn’t Matter
We didn’t negotiate salary. Whatever the candidates proposed was fine.
All we cared about was developing our application in a cheap and efficient way.
By saying “cheap,” we mean the price formed by:
-
-
- A candidate’s hourly rate. It was calculated by the salary they asked for
- My time responding questions, charged at a rate of $100/hour.
-
We needed to avoid needless questions but also to see if the candidate has good judgement about which questions were necessary.
We were a little reluctant to charge for answering questions. It’s hard to promote teamwork and charge people for questions. Also, we thought it’s my fault if I didn’t write the test task explicitly.
As the experiment demonstrated, it worked fine.
-
-
- Any expenses they will have. For example, if someone needs help from a friend, fine. HTML coding? QA? We pay any bill.
-
Step 4. Go!
We created a separate Basecamp project for each of the candidates because the project was too small for our usual JIRA. We asked the users to track their time with Hubstaff. It takes screenshots, yes—a couple of people rejected working with it.
I’ve learned that tracking time requires discipline from us, too. We had to track time dedicated to each of the candidates.
If I received a message, I set a timer, read the message, responded to it, stopped the timer. 43 seconds.
We used Minibooks on my iPhone for that, but any timer would work: Toggle is fine, Hubstaff is ok, too, if not linked to JIRA.
Step 5. Evaluating the Results
In short, the results looked like this:
-
-
- Candidate #1 spent 3 days, did nothing, and received $1454.55
- Candidate #2 disappeared
- Candidate #3 fascinated us. He did the whole thing, quickly
-
Candidate #1. 40-dollar job for $1,033.80
I had to write a sad message:
John [name changed to protect the guilty],
Unfortunately we have to stop our partnership.
It’s nothing personal of course; we have to keep track of our finances, and our finances are like this:
We owe you $1,033.80. Another $30 comes from 0.3 hours of our time. For this money, we got… not much.
I understand HTML is not your sharpest skill, and I also know you could have outsourced it to someone else, dealing with API integration meanwhile.
If we complete HTML coding your way, we’ll have to pay 5 times more just for the mockup, resulting in $5,169. That’s more than the whole project estimation from another really expensive candidate.
Also, we’ve got another candidate who is already finished; all we have to do is integrate social networks; they also charge us less.
We appreciate your time and of course we will pay $1,033.80. We do ACH, wire transfers, and PayPal. And of course the commission is on us.
Ivan
Of course, it was huge pay for such work. $1,033.80 + $40 for a wire transfer — that was our fee for rejecting the wrong candidate. Is that much? We don’t think so. A recruiting agency probably would have charged us tenfold.
Developer #2: How to Put a Project on Hold for 22 Days
She was smart and professional. She didn’t speak much, understood everything, confirmed everything. She said “Will do.”
Also, she presented some great recommendations.
Also, she was doing freelance and hated it. She wanted to switch to a full time job. We were that chance.
And then she disappeared.
I called her and she said she had some unfinished freelance work and asked for additional time to deal with it.
And then she disappeared again.
From the beginning of our search for developers until the moment we stopped waiting for her, 22 days passed. If we had her as the only candidate, we’d have put the project on hold for 22 days.
Developer #3: That’s What We’re Talking About
His name was Serge. I sent this message to him:
Whoa! I mean… whoa!
Please pass it through the test cases, then we will test it once again.
He spent 8 minutes and 51 seconds of my time — quite a lot! However he did more than specified in the acceptance criteria. Needless to say, we hired him.
Conclusion
This is how we hired someone with our eyes closed. Judgement should be blind after all, right? I knew nothing about the guy. Where he worked, what kind of expertise he had, if he played golf with the royal family, or taught programming to poor country kids (like one of our team members).
I learned where he lives only when he sent his address for billing.
Afterword
Digging through email, I’ve found a message from my teammate who was also in touch with the 3rd candidate:
Serge confuses me, too direct. The other two are interesting.
I don’t remember what he meant by “direct,” but it’s clear: subjective judgement would have filtered out Serge and left other two.
And that’s how most companies hire! Most, not to say “all but a few.” I hope you are one of those few. I hope you’re with us.
To apply the blind method you don’t have to work in an office. In fact, you don’t even need a home. Perhaps, you’d like to know what gadgets help me manage my design workflow on the road.
If you don’t travel much, but still interested in UX experiments, take a look at one of the most popular of ours – Five Designers Draw One Icon.
About the Author
Ivan Boyko is a founder of Icons8. He got his first job after drawing a banner with CTR of 43%. After years of creating icons, he specializes in rapid prototyping and backlog grooming.
