For them to be actually effective, the questions need to be designed to extract meaningful responses. The old "grade your instructor on a scale of 1 to 10" question is pretty useless without requiring detailed explanations.
At most of the schools where I've taught, most students don't bother to fill out the evaluations unless they're told they are required to, so it's only a minority whose opinions get heard. That puts a huge error factor on top of any results anyway.
Here's a thought: If the process of designing evaluations, and evaluating the responses, isn't done even more thoroughly and carefully than the teaching of the class, can the results possibly be very meaningful ? The effectiveness of an course is a very complex thing to evaluate, and the value of the subject matter itself -- not to mention the manner in which it is conveyed -- is often unclear to the students themselves until years down the road. If you want to get meaningful course evaluations from students, you should wait until they have taken classes for which the course being evaluated was a prerequisite. Only then will they really be in a position to see the value of the material they learned (or didn't).
But that would require too much effort, and worse, too much thought.
= new reply since forum marked as read