I'd question how far from this Twitter experiment is the work that the candidate would be doing, personally. If the applications being developed are similar to those using he Twitter API then what you may have discovered is that there is a lot of bad developers out there and it'll take some time to find the few good ones out there. At the same time, if the application isn't like the Twitter API, then you may well be barking up the wrong tree.

Did you ever consider how well some people may not know Twitter's API? Do the candidates know how long they have? I'll concede that there may be a bit of a "Hero" complex that candidates can have as I know I've been there myself a few times. Depending on how long a cycle is from feature request to a release in production, it may be useful to take pieces of that and see what they can do. For example, how well can the clarify the requirements, what kind of testing do they know to do, etc.

Is knowledge of Twitter API libraries a requirement of the job and listed in the job posting? If so, but you still called in candidates like the one who muttered that he never used Twitter, then it's your bad. If it's not part of your job description, then it's not very nice to spring that on someone in an interview. They're going to have "interview nerves" and lots of people also have test anxiety, so they really need to know what is expected going in. I also like Kate's solution very much. Her approach would make me feel less like I'm under a microscope and more like I'm demonstrating an ability to collaborate as well as the specific skill you're asking me to demonstrate.

Some people can't code, or can't code well. But you want to find this out as early and cheaply as possible. A long, complicated programming test is as likely as not to piss off senior candidates who are used to being paid for their time. You need to find the shortest, simplest test that nontheless gives you useful data about whether the candidate can code. I've found that, rather than asking them to code, you can get plenty of insight into their skill level by giving them a few pages of low-quality code and asking them to list what's wrong with it.

I think it is better to do that as a take home test. And here's why. 1) if someone isn't that familiar with twitter it gives them a chance to brush up if they want to. And 2) there is a lot of pressure in an interview already and adding a complicated task much more than reverse the string is hard anyway. While in a real job no one will be looking over their shoulder as they code. Anyway I had one place where you would have meetings and have to do SQL queries on the big projector in front of the whole meeting. Anyway you always made way more mistakes and forgot a lot more things when everyone was watching.

I had one interview where they did give you a take home test. It was actually very simple, sum up the numbers, find the mean and the standard deviation, and then print the list, the mean and standard deviation. The wrinkle was it was in the J language... Anyway that weeded down the candidates and was more realistic as well, because it weeded out the people who didn't care to learn J enough to do the challenge. But anyway I have been on an interview where they give you a simple reverse string task as well in Java. Even this was unrealistic because in a real situation you could go java.sun.com/j2se or java.sun.com/tutorial (okay I think the links changed now that it is Oracle owned... but anyway) and use the API to figure it out. So doing it in an interview is not realistic. Also the 1 hour estimate is crap. Because I think that probably you work with twitter or you wouldn't have thought to do the challenge on twitter. And I'm betting you already used some libraries so you are used to them. Further more you were relaxed and calm. So if it took you one hour, you have to expect it will take a job candidate more.

A take home task seems fine. It's true that some might be able to ask rentacoder or on stackoverflow or something to get it done. But once they come in you can question them on it and quickly find out what they know. Most places I have been give some type of logic test. My current company gives a C test and SQL test, even though they use C# and Java mostly. But there was nothing super obscure on there. You can belt out the C if you know Java or C# because nothing was peculiar to C mostly except the pointers. Anyway other places just give a general logic test or a test that tests basic school skills from high school. I'm not sure how valuable those things are though.

Getting back to your challenge, in my case I know what twitter is. However I haven't used it myself. I get the basic idea. I have no clue what a timeline is. I would have to google around for the features you asked for on twitter in order to understand what I'm looking at. Then google around for libraries to do it. Finally slap a program together. I suspect it would take me more than an hour. Heck quite often it can take an hour just to find/set up a library. But anyway I doubt I would get up and leave. I would probably work until you said time out. But anyway I suspect good progress in an hour would probably be enough. Still the same guy/girl can probably do it much quicker without it being a high pressure interview so it isn't a full test of their programming potential.

But at the same time if you go crazy and send some take home test that is going to take 12 hours, I'm going to wonder if you are trying to bum your way to free work. And I probably would not even bother with it.

I propose you find some obscure task that requires learning something very specific that your company is interested in. So that in order to pass the test they have to learn the bit of knowledge as well as use some programming skills. One of the biggest failings of software developers is a lack of ability/desire to learn new things outside of the direct domain of software development. Heck even within software development often people get in their comfort zone and don't want out of it. When all you have is a hammer everything else looks like a nail....