Yes, this is a tough problem, isn't it? It's one of the things I liked
best about Robin's idea futures. It seems like a reliable tool for
evaluating people who had good ideas, especially when I don't have the
knowledge or time to look at their credentials and deside if they are
meaningful.
Another method is to pick someone you trust who has the expertise, time,
and inclination to evaluate these people. I knew someone (probably a few
people, come to think of it) who relied on Richard Feynman to evaluate all
or most scientific theories that hit the news. Cold fusion gets announced,
wait until Feynman makes a comment and go with what he says. I'm not sure
who I'd pick today, but you want a combination of knowledge about the
domain and an ability to think rationally about the implications of the
theory. CSICOP is another good resource, and they take the time to address
the more patently absurd ideas.
And then there's basic prejudices. There's a class of idea that always
seems built on flimsy evidence and protected by claims of conspiracy. I
don't spend much time worrying about faces on Mars and aliens at Roswell.
Jerry O'Neil once wrote, "never ascribe to conspiracy what can be
satisfactorily explained by stupidity."
Best,
Drea