hwwwwooooaaaaahhh!!

neat

YAYAYYAYAYAYAYA. Thanks to Jon Mak for being right!

Grand Prize: World of Goo
Best Browser Game: (I haven't played any of these so I'll leave them out)
Design Innovation: Fez
Visual Art: The Path
Audio: Clean Asia!
Technical Excellence: World of Goo

Big winnar - World of Goo

how the hell does fez get nominated for innovation?
if thats the case, then i would for crush.
rigged, just like the previous igf.

i wonder, is there anybody auditing these awards?

Hahaha, rigged? Audit? hahahahaha

Congratulations to all finalists on my behalf as well.

As for predicting who will win...well, let's see.

Grand Prize: Noitu Love 2 (World of Goo)
Physics games for the win alright, but after actually making it to the finals which many people wouldn't even have thought possible for such an old school platformer, I'm absolutely convinced NL2 might even sweep in for a surprise win as well. Pixel Love!

Best Web Browser Game: Globulos.com (Iron Dukes)
Never been one to spend time on browser games much so my impressions might be way off. I have to admit though I kind of like the idea behind Globulos, what with the literally dozens of different games in one engine.

Design Innovation Award: Fez (World of Goo)
Not that much room for speculation here, eh? I seriously don't expect Kokoromi not to get this one.

Excellence in Visual Art: Clean Asia! (The Path)
The Path sure looks hauntingly stunning. On the other hand, "excellence in visual art" in my book does not just mean the most pretty graphics, but also graphics that are actually conveying the overall feel of a game, fitting perfectly in with it and perhaps most importantly graphics that are ultimately unique. Which spells C-L-E-A-N-A-S-I-A for you.

Excellence in Audio: Audiosurf (Fret Nice)
For me personally the only category which doesn't have a real favorite. All nominees are about equally astounding in their own, quite different way. I went with Audiosurf and Fret Nice because of the impressive way in which they strongly integrate Audio into their gameplay.

Technical Excellence: World of Goo (Goo!)
Would be a crying shame not to have WoG emerge victorious here due to the incredible amount of detail and level of polishment which surpasses that of most recent commercial games in my opinion. Then again, I also extremely dig how Goo! appears to improve on the basic Liquid War (which is the best Linux game ever [also pretty much the only native Linux game you'd ever want to play for more than half an hour, heh]) concept in nearly every imaginable way and wraps it up in one ass-kicking presentation.

Those are my predictions for winner and runner-up. Of course they are only in my humble opinion and probably somewhat biased. But what isn't, really?

I think it's just that the judging process is a bit random (if you look into how it actually works, it's fairly chaotic and subject to luck). One of the judges told me that each game only had one or two judges look at it, and some games up to nine judges, out of the 30+ judges.

And I'm not just saying that cause I didn't get in the finals, I'm relieved I didn't (would save on travel costs and social interaction, which I'm bad at). I've said such for years.

I would defend the voting process - being the Chairman, perhaps, but we make sure that the major games in contention get a sensible amount of voting passes by judges, and that all games get a fair shake.

Obviously, judging 170+ games properly with multiple busy judges is difficult, but I think we do a good and fair job, and we work very hard on it.

I'm sure you try your best, but the problems I see in it are structural.

Here's an analogy. Let's say there's a baking contest with 30 taste testers and 170 different recipes. Each recipe gets tasted by one or two of those 30 taste testers, and scored. The final recipes are then determined by the highest average score.

This means that to get in the finals a recipe *doesn't* have to be better than all of the other recipes, it only has to be better than those of the recipes the taste tester who tasted it has tasted. So the recipes are not really being compared to each other, only to other (randomly selected) recipes.

This favors recipes that were compared to other bad-tasting recipes, and takes away from those recipes that were compared to other good-tasting recipes.

And I do recognize that your judges do work hard on it, but when even one of your judges is (privately) admitting that it's a terrible way to judge games, that's not a good sign.

Paul: This is something I was very cognizant of when I coded this year's judging backend. There isn't an easy solution, though, aside from having all of the judges taste all of the pies. This is an impossibility with the judges' available time.

The one fault in your analogy, though is it presents a worst-case scenario. If you have the following pies, ranked in order of tastiness:

A B C D E F

And judge 1 tastes A-B-C, and judge 2 tastes D-E-F, then sure, you have a problem. Judge 2's most delicious pie is actually worse, objectively, than any of judge 1's.

The first difference is that we have 50 judges with overlapping assignments. Assignments weren't actually random, but rather an interlaced walk from the database. If you have judge 1 on A/B/C, judge 2 on B/C/D, and judge 3 on C/D/E, you start to get an idea of where games lie in a more objective sense.

The second point is that games are far more subjective than pies. Even if all judges scored all games I very highly doubt we would have a different list. There are a handful of people that look at every single game (including Simon), so we are certain that all games receive a fair shake. I was monitoring games with a low amount of scores, due to a judge having a time conflict or technical problems, and made sure to assign more judges to them midway through the judging process.

So, yeah, I just said everything Simon did in much more detail. Please know that we do work very hard at judging and take it very seriously.

Oh, also--I should mention that all judges do play all games in the final round of judging.

I'm glad that was clarified, but if I were an entrant I'd be happier to wait longer for a more in-depth judging process. As an onlooker, I'd still be happier, actually. But if I were a judge, it'd get real tedious real fast. As pointed out, that's a lot of entries to get through - and one could very readily argue that having fewer games per judge means that the games DO get a more thorough testing rather than a 5-minute play.

That system is better than I imagined it, thanks for clarifying. But I agree with haowan that the best way to do this would be to extend the judging period by a few more months. 170 games may be a lot to play, but if the judging period were 6 months long, that'd mean only one per day, which is more than enough time considering how short a lot of them are. I play a few new freeware games per day myself just as a consumer of indie games, and it's not a big load.

There also does seem to be a discrepancy between the claim on the site that each game sees five to ten judges and the (anonymous) judge who told me that each game only saw one to three judges on average. Maybe he was exaggerating and that's inaccurate, but that discrepancy did jump out at me.

I definitely agree with trying to improve the process. I'm a big proponent of constantly trying to improve everything (in any project, not just the IGF).

Reality tends rears its head, though. There are only so many hours judges can commit to this before it intrudes of their day jobs of being kickass journalists, game developers, and what have you.

So we'll do what we can to make it better. I don't want to send the message that "it's good enough now, period". I feel it is quite good, of course, but we'll always see what we can do to make it even better.

(Oh, and also keep in mind a judge may have looked at a game and not scored it because they couldn't get it to run).

It's true that they can only commit so many hours, but I think they should think of that before they agree to judge.

If you're looking for other ways to improve it, one thing that may help is a nomination process, similar to how the Emmys or Oscars and so on work. Instead of having it open to everyone, a game could only get in if it's nominated by two or more of the judges. That may introduce other problems (such as friendship with the judges being too important to winning) but it'd cut down on the large number of entries, and that way each judge could manage to look at a smaller number of nominated games. It's not the method I'd use myself, but it's one option.

What I would do myself to improve it is simply to reduce the number of judges to those who commit to playing (or at least attempting to play, absent hardware problems) every single game, over a period of six months. If that means cutting the judges down to five or seven or something, so be it, since I think it's important that a game be compared to all of its peer rather than only some of them, and I think that's more important a consideration than having a wide variety of judges.

Another big change I'd make if I ran a contest like this is transparency. I'd make every judge's score for every game, and who scored what game, public information. I don't see any real reason to keep it private.

I hope this doesn't come off as me telling you what to do, I don't intend it like that. They're just suggestions and 'what I'd do myself' things.

I appreciate the discussion, actually. Before Steve and I became content directors for the IGF we were participants. We've entered four years of competition and been finalists twice.

When we had the chance to influence the judging process I thought a lot about how best I could improve it using my experience on the other side of the fence.

It would be great to get a core team of judges playing every game, but I think you may be swapping one problem for another there. For instance:

How can you play one game, and score it, and then play another game 6 months later and relate its score to the first?

Currently we allow updates, which would obviously have to cease in a longer judging period. It would be much more unfair to judge Game A the first day and Game B months later if they were both updating.

But then, if each game is judged by a variable number of judges, is its score averaged by the number of judges? or are the scores added?

I'm confident that the IGF committee did the the smart, obvious thing and averaged the scores.

How come Fez is a finalist under Excellence in Visual Art, and The Underside is called a Cave Story ripoff due to the visuals, while both have the same visual style?

Because that's your opinion. And fortunately for us, we are not ruled by it.

Kon-Tiki, the way Fez renders graphics is pretty unique.

Can't say it does or doesn't, all I can tell, is what I've seen from the trailers and screenshots of both games, and both have the same style of graphics on there. What makes it that The Underside is a Cave Story ripoff to you guys, while Fez isn't, purely graphics-wise?

Fez is '3D' in that you can spin the camera angle from what I've seen. The actual graphics themselves though aren't all that impressive in my opinion, even when considering it at first glance resembles the similar simplistic graphic style used in the far prettier Cave Story.