New comment on your post #366 “Class notes on algorithms…”
Author : Sariel Har-Peled
Comment:
I hope you are kidding when you are refereeing to me as a top-notch theoretician. I am not really a theoretician, and I am top notch only if you hold the notch upside down…
Anyway… Oh man. Refereeing is the hardest thing we have to do in our job. In some sense it is the way we can have the most influence on our field. To begin with we have our own bias, then it is a game that involves other people (for example, in NSF refereeing, all proposals are judged to be excellent, or extremely excellent, or splendidly extremely excellent, and ola-la-la-la excellent. If you get only excellent, it means your proposal sucks. [I am stretching reality a bit, but only by a bit.])
Anyway, relating to your question, the first step is to realize that you have to judge proposals and papers in comparisons to the other proposals and papers in the pool. There is no point saying a proposal by say X on this topic would have been much better, since X might be lazy in writing proposals or just not really interested in this topic (although this is a valid consideration for the funding panel chair – he can contact X and ask him to apply). How is this proposal compared with the other proposals? Is this guy going to work with other people that can do the less theoretical part of the job?
In some sense, my feeling is that one has to judge how much value one would get for the money. A proposal can be wonderful, but if the author is clearly unable to deliver (thats why track record is important) what she promises, then it might be completely useless. If on the other hand, the PI can not deliver but her questions are amazingly interesting, then just getting the questions out (and getting strong people to work on them) is worth the money. Ask yourself, in then years from now, if I fund X or fund Y, what (in expectation) would be the higher yield. What is the time frame? In a time frame of 20 years, theoretical ideas are more important, but in a time frame of five years a practical deliverable might be more valuable.
As for myself, I always try to be one level more positive than I feel about such things. In any case, you can put all your considerations in your “referee report”. Thats what the refereeing process is all about…Let the panel/chair/whatever make the final decision. Just make sure that your grading/rank is in sync with the rest of the people on the panel/whatever…
And do not take too seriously the written guidelines. People are probably ignoring them anyway.
Hope this helps…