2012-06-22

At what likelihood threshold should you cut?

Martin (Strasbourg) and I discussed his project to detect new satellites of M31 in the PAndAS survey. He can construct a likelihood ratio (possibly even a marginalized likelihood ratio) at every position in the M31 imaging, between the best-fit satellite-plus-background model and the best nothing-plus-background model. He can make a two-dimensional map of these likelihood ratios and show a the histogram of them. Looking at this histogram, which has a tail to very large ratios, he asked me, where should I put my cut? That is, at what likelihood ratio does a candidate deserve follow-up? Here's my unsatisfying answer:

To a statistician, the distribution of likelihood ratios is interesting and valuable to study. To an astronomer, it is uninteresting. You don't want to know the distribution of likelihoods, you want to find satellites! The likelihood ratio at which you make your cut depends on your willingness to publish (or really follow up and reject) crap, relative to your desire to get a complete sample. That is, the ROC curve is more interesting than the distribution of likelihoods, but the ROC curve takes a lot of work to generate (since you have to follow up a representative sample of stuff!).

But, fundamentally, where you set the likelihood ratio cut is determined in the end by your best estimate of your expected long-term future discounted free-cash flow. Your LTFDFCF is affected by many things, including the amount of time and effort you will spend on follow-up, the impressiveness of the paper you can publish, the time it takes that paper to be written and posted, the perceived or actual penalties for including non-satellites in your final results, the value of discovery priority, and the opportunity costs of failure to obtain priority, to name a few.

No comments:

Post a Comment