Mon, 16 Oct 2006

LCA 2007 Paper Review Committee.

Its taken me longer than I would have liked, but I'm finally getting down some details of my participation on the paper review committee for LCA 2007.

The review committee included me, Rusty, Mary and about 10 others. As Mary said, reviewing 200 plus submissions for 60 or so speaking slots (ie ignoring the tutorials) was not an easy task. In the end, we threw out of a lot of stuff that would have made great talks for the reasons Mary listed. The area hit the heaviest was of course the kernel related papers and I hope most of the people who got their kernel papers rejected make it along to the kernel mini-conf that will be held in conjunction with LCA.

Just to give people some insight into the process I'll give my view on how things happened. Firstly, I was contacted by the LCA organizing committee in mid August asking if I'd be interested in participating. Of course, I said yes, although there were times I regretted it :-).

The committee got together for the first time in eary September and we were given guidelines on what the organizing committee was looking for. We were then sent away with a password to a web site where we could review the papers and were told to do it at our own pace and to try to avoid influencing one another.

For me, I did my reviewing over about 18 hours of Sunday and Monday of the October long weekend followed by about 3 hours reviewing the tutorials later the same week. For the papers, that was a little less than 5 minutes for each. Some of the paper reviewers did spend more time; one of them spent about 4 full working days which means about 8 minutes per paper. Obviously, anyone wanting to get a paper accepted to LCA should do their utmost to grab the attention of the reviewer in as few words as possible. That said, I know many of the reviewers (myself included) rejected papers because they didn't have enough detail.

On October 7th, the reviewers got together at Palais Pants to decide which papers and tutorials to accept. The majority were able to make it on the day but we did have two members of the committee on a conference call and another on IRC. Rusty ran the proceedings and we tackled the tutorials first as a trial run for the much scarier task of the papers.

For the tutorials, Rusty would call out the proposal number, the rest of us would look up the tutorial on our reviewer's web interface and John Ferlito would read out the mean score from the web reviews. On the basis of this, and a bit of minor haggling, the tutorials were sorted into piles of "yes", "maybe" and "no".

With the experience of the tutorials behind us it was time to tackle the papers. We started out by having John sort them from the most popular to least popular on the basis of the web reviews. We then started at the most popular, accepting papers until we found the first proposal that was only a "maybe". We then worked from the least popular, rejecting papers until we found one that was a "maybe". Finally we worked through the remaining proposals in numerical order, sorting them all into "yes", "maybe" and "no" piles.

The above sorting took about seven hours, not including a lunch break. We then found that we had a few too many kernel related talks and not quite enough sys admin related ones, so some minor adjustments were made.

Being part of the paper review committee was definitely an interesting experience. I really enjoyed it. I was also amazingly impressed with the quality of the submissions. We rejected a bunch of really good papers because we have even better ones. LCA 2007 is going to be fan - bloody - tastic and I'm really looking forward to it.

Posted at: 20:30 | Category: | Permalink