Picking a winner is tough. Especially as hackathons get bigger, the decisions must be made quickly, and selecting the ‘best’ project is always subjective.

How you design the last chapter of your hackathon is important: it is the last thing your hackers will remember and take away from the event. Do they feel accomplished and successful, or do they feel like they wasted a weekend? A hacker’s sense of belonging in this community is based on what they feel is prized and valued. This is often a reflection of what is prized and valued at the events they go to. Obviously we want them to feel accomplished and like they are part of something bigger so that they continue to go to hackathons, so let’s dive in and look at what makes a finale and the associated judging process successful.

This post will equip you with tools for different styles, methods, and approaches to judging hackathons.

Your Array of Judging Formats

The Demo-athon

The Demo-athon is the tried and true hackathon finale. Every team getting 2-5 minutes on stage to show off what they built was the only way of finishing a hackathon for quite a long time. But it isn’t perfect, and sadly it did not scale well. Let’s dig into why.

Benefits:

• Hackers love to see what their peers have built.Being able to see all of the bizarre, awesome projects that your friends built is great. You can see who you want to work with in the future or what you want to contribute to after the event, and you also get to learn about new, crazy technologies.
• Demos engage your community as a cohesive group.When watching demos, you get to enjoy everybody’s hacks together in one big happy family. You get to see how your friends approached problems that you might have encountered and you discover what is really possible in the scope of a weekend. Everyone laughs, cries, and cringes together and it makes you feel closer as a result.
• Judging is more transparent and simpler to coordinate.A demo-athon only needs one small panel of expert judges to evaluate all of the hacks. In addition, sponsors can see everyone who is eligible for their prizes in one sitting. Hackers only have to worry about having a working demo that one time, and then they can breath a sigh of relief.
• Big Demos Teach Big SkillsDemoing forces you to get comfortable in front of an audience, and learn that it really isn’t that scary. Your best qualities can come out, whether that is your humor or your technical expertise, there will be audience members who resonate with your style.

Drawbacks:

• Holy moly, demos can be long.Raise your hand if you’ve ever conked out because you didn’t sleep all weekend and then some fool asked you to sit and watch 6 hours of hack demos. I have… Once you have more than 40-50 hacks, the time needed to give them all demo slots becomes unreasonable. Some of our biggest events have more than 200 hacks. ‘Nuff said.
• Decisions depend heavily on the pitch.I’m all for hilarious, interactive pitches. But there is a fine balance to be had between enhancing hack demos with funny pitches and turning into a full-on pitch competition. No one wants to hear you pitch Uber for Kittens at a hackathon, sorry.

Pro-tips:

• Have 2 A/V hookups that you can easily toggle between. This makes it possible for your on-deck presenter to get set up before the previous hack is finished and smooth the transition process.
• Have someone in front of the demoer with a visible countdown timer who also gives a signal when they have used up half of their time.
• Make sure whoever is 2nd on deck is ready to get set up and has their hack ready to go. No one should get on stage with their computer in sleep mode…
• If you have time, do dry runs with each of the demoers so they understand how short 2 minutes really is. During this time you can also coach them on tips and tricks like the fact that it is always better to Show, not Tell.
• Give your judges clear criteria on what they will be evaluating for each hack and how they will be picking the top hacks. Sometimes a rubric is helpful to this process.

Parallel Demo-athons

One of the first alternatives to this mass demo-athon debuted at PennApps a few years ago. They realized that hackers did not want to sit through hundreds of demos, and decided to split up their judging into multiple simultaneous demo rooms.

“The worry with multi-tiered judging is this: imagine you spent your entire weekend sleepless. Your implicit reward, prizes or not, is that a hundred or more people will get to see what you did and, prizes or not, applaud your effort. Instead, you go into a tiny closed conference with 4 older-looking folks who listen to you for two minutes and then wait for an email with the list of finalists – a list you’re not on.” – Alexey Komissarouk, Founder of PennApps

Benefits:

• Hackers still get all of the advantages of the Demo-athon, with the scale problem solved.This is pretty self-explanatory. You take demos, and run them in parallel in smaller groups.

Drawbacks:

• The logistics get very complicated, very quickly.When there are multiple demo rooms, each with their own presentation queue, knowing where you have to be and when gets to be pretty confusing, especially if you are trying to juggle watching your friend’s demo with giving your own.
• No single judge sees all of the hacks.When judges get to watch every demo, it is easy to rank them relative to one another. But what if one room has a handful of amazing hacks and another room has no amazing hacks? Judging then becomes more subjective and you run the risk of not really surfacing the best hacks from the event into the final demos because no one has the full picture of how they stack up.

Pro-tips:

• Let folks know which rooms they will be demoing in as far before demo time as possible and have them check in with an organizer in their room to confirm that they are in the right place before demos begin. No one should have to run between rooms because of a mixup.
• Use the A/V and prep tips from the demo-athon section above to make switching between demoers as seamless as possible.
• Publish the full room schedule online so that hackers can find out when their friends are demoing.

Science Fair

Soon after the test of this parallel demo-athon, the organizers of MHacks decided to try something completely different. They decided to have a science fair.. or at least repurposed it from their middle school science teacher. They set up a huge expo hall and gave every team a table to demo at. The top 10 teams bubbled up based on roaming judges and demoed for the entire audience. If you’ve ever been in a room of 1,500 hackers all demoing their mad science projects at once, you know how amazing it feels. If not, I highly recommend you try it some time. The feeling is indescribable and truly makes you think about hackathons in a new light.

Benefits:

• The demos are easy to scale.Since demos are running in parallel, time is no longer an issue. When you have more hacks, you just increase the number of tables. Problem solved!
• Conversations > PitchesDuring the science fair, you have to demo your hack over and over and over. That means that it actually needs to function in a reproducible way. The functional, interactive demo becomes far more important than the pitch, which is great.

Drawbacks:

• The logistics are still pretty complicated.Assigning 1,500 hackers on 250 teams to exhibition tables and getting them set up and ready to demo in a timely manner is really hard.
• Judging takes a lot of time and is hard to coordinate.Though hackers have something to do with their time other than watching lots of hack demos, judging 250 hacks (and physically moving from one to the next in order to do so) takes a lot of time. Aggregating all of those scores to surface the top 10 hacks is really difficult, and requires a lot of coordination between organizers, judges, and hackers. Additionally, no single judge gets to see all of the hacks so surfacing the top projects becomes somewhat subjective.
• Hardware hacks are far more visually appealing than software hacks.We’ve observed that hardware hacks tend to attract much larger crowds (and as a result, more attention from the judges) simply because they are more visually interesting than most software hacks. While this is awesome in a lot of ways, it does tend to unfairly skew the judging towards hardware hacks.
• Hackers don’t get to see other hacks or engage a large audience.Since you have to demo over and over in the science fair, you and your teammates rarely have the opportunity to go browse everyone else’s hacks. This is probably our main gripe with the science fair system because seeing awesome hacks is one of the best parts of going to a hackathon. You also lose the group humor and audience engagement of working with a large audience, which definitely makes people feel closer together and builds community around hackathons.

All of those criticisms aside, the science fair works well. For a hackathon with large numbers of hacks, a better system hasn’t yet been effectively tested that we’ve seen.

Pro-tips:

• Give hackers a large time buffer to get set up at their exhibition tables before judging begins.
• Make it clear to hackers who the judges are and how they will be evaluating the hacks.
• Have a publicly available spreadsheet (or an app such as HackerLeague or ChallengePost) that makes it clear which hacks are at which tables. You would be amazed how complicated judging becomes when it is not clear which hacks are at which numbered tables…
• Provide sponsors with a list of all hacks that are eligible for their prize so that they can easily find and evaluate them in the science fair.
• Consider grouping hacks in some meaningful way to make judging more straightforward based on prize category or type of hack.
• Find judges who are not sponsors or organizers (both of whom are very busy) to evaluate and re-evaluate all of your hacks. The more hacks you have, the more judges you will need. At some events I have seen this number climb to upwards of 30-40 necessary judges.

What Next?

The three judging methodologies we explored earlier each improved on some aspect of the hackathon finale, but there is still a lot of room for growth and experimentation with this vital component of your events.

My recommendation would be to combine the best aspects of the Science Fair with the community feedback frequently seen at Demo-athons.

Eliminate the roaming judges altogether and turn the decision making power over to the hackers themselves. Additionally, force team members to rotate staffing their booth so that every person has a chance to see and vote for a large number of hacks. LAHacks tried this model of restricting each booth to only be staffed by a single team member and it proved effective in forcing people to explore other projects. If each hacker has 3 votes, and you can safely assume that hackers will first explore neighboring hacks to where their team is stationed (at large events where they cannot see every project), I believe you could surface some of the best hacks from an event without having to appoint and coordinate large groups of judges.

This solution is not perfect, nor has it been tested in a meaningful way. But we strongly believe that any successful hackathon finale will incorporate these elements:

1. Let hackers see what their peers have been working on to build community
2. Let sponsors see the projects that are eligible for their prizes
3. Let hackers show off what they built in a meaningful way so that they leave feeling accomplished
4. Encourage hackers to launch early and often and not to fear the demo, despite lack of polish
5. Create a sense of community where what you built is more important than what you won

With those five points at top of mind, I am sure that hackathon organizers will continue to make their event finales even more fulfilling for their hackers.

How to Judge Hacks

We’ve run the gamut of judging variants, now let’s talk about how exactly you pick the very best hack out of a pool of hundreds of amazing hacks.

You Don’t

Sorry for the spoiler, but the truth is that judges rarely, if ever, pick the best hack.

Being the ‘best’ is an entirely subjective assessment that varies from person to person and from event to event.

But Really, How the Heck Do We Pick Who Gets Our Billion Dollar Prize?!

If you have a billion dollar prize, you have already failed. Go read our Prizes post and then come back. We’ll wait…

.

.

.

.

Okay, now that you know better than to offer silly wads of cash at your hackathon, we can talk about some techniques for surfacing high quality projects from a large batch of interesting projects.

1. Focus on awesomeness.Awesomeness is one of the best qualifiers I have seen for what constitutes a great hack and it something that I learned from HackNY, the primordial student hackathon. Everyone’s definition of awesomeness will be different but ultimately it boils down to creativity, depth, and wow factor. You want your winners to be endlessly creative and inventive. They should be building things that no one thought possible in the scope of a weekend. They should be using techniques that are abnormal, often impractical, and weirdly elegant. As subjective as it is, distinguishing between a simple CRUD app and something that is mind-blowingly awesome is fairly easy.
2. Have the right judges.Your judges can make or break your hackathon finale. First off, have a balanced and diverse judging panel. Generally I like to have almost all technical judges (engineers/developers/designers/etc.) but you can certainly have a large amount of skill and demographic diversity on a technical panel. While there are some extremely talented and creative non-technical judges out there, we’ve found that panels of technical experts tend to pick projects that are more technologically interesting regardless of polish, which is the right fit for hackathons.
3. Focus on learning over profit.Sorry to break it to you, but 99% of hackers are going to hackathons to have fun and learn something. It is rare that anyone goes to a hackathon to bootstrap a startup (Startup Weekend/StartupBus are not hackathons), and rarely do those people win hackathons because they spent their time creating a product rather than awesome technology. Occasionally, cool hacks (like Grooply, which became GroupMe) become products over time but rarely do they start out that way. Hacks are built as creative solutions to problems or simply for fun, not to make money.
4. Give beautiful, useful hacks a chance.As much as I like to tout the benefits of ugly and technologically awesome command line hacks, designers are great hackers too. Beautiful hacks, or those with innovative user experiences, should be able to win hackathons if they are functional as well. Hacking design is as valuable as hacking code, but they are best when paired together. Awesome design + awesome code = Awesome hacks.
5. Involve the community.Judges don’t notice the same things that hackers do, regardless of how technical they are. I have seen many hacks that were truly amazing in every capacity but botched their demo or didn’t pitch well who were passed over for top honors. Oftentimes their fellow hackers know how cool their project is and so community voting is able to surface hacks that would not otherwise rise to the top.

With these 5 points in mind, hopefully you can craft judging criteria that help make hackers feel awesome at the end of your event. Distilling these points into a cohesive rubric is difficult and perhaps unnecessary, but explaining to your judges and hackers that Awesomeness is the primary factor goes a long way towards surfacing the best hacks and encouraging hackers to think outside the box.

“When you judge things like business value you begin to restrict creativity. Feasibility is an important factor to be aware of but should not be a decider. The goal, in most cases, should be to foster a culture of innovation at minimum expense. That was the original motivation behind hackathons and one that I believe still makes sense today. Within our realm we have unparalleled creativity and that creativity, like a muscle, must be exercised. Engineers are not unlike artists in this regard. Hackathons are an opportunity to exercise our creative muscles. Don’t constrain that opportunity unnecessarily. Let innovation and creativity be the most powerful motivating factor and we can build amazing things.” – Randall Hunt, Director of Developer Evangelism at MongoDB

Comment below with your own ideas for how to improve hackathon finales, this is a constantly evolving process!

Sign up below to be the first to hear about new excerpts and the release of the full Manual.
[mc4wp_form]