From: Harvey Newstrom (mail@HarveyNewstrom.com)
Date: Thu May 30 2002 - 01:35:19 MDT
On Thursday, May 30, 2002, at 12:58 am, spike66 wrote:
> You collect two kinds of mushrooms, the
> fleebs and the kloongs. 10% of the fleebs will give you
> diarrhea, whereas 20% of the kloongs will.
I don't know how to explain this any clearer. Profiling doesn't work.
Using your own example, consider 10 mushrooms of each type:
Mushroom: Profile: Result: Accuracy:
fleeb-1 good bad WRONG - protection failed
fleeb-2 good good RIGHT - no threat anyway
fleeb-3 good good RIGHT - no threat anyway
fleeb-4 good good RIGHT - no threat anyway
fleeb-5 good good RIGHT - no threat anyway
fleeb-6 good good RIGHT - no threat anyway
fleeb-7 good good RIGHT - no threat anyway
fleeb-8 good good RIGHT - no threat anyway
fleeb-9 good good RIGHT - no threat anyway
fleeb-10 good good RIGHT - no threat anyway
kloong-1 bad bad RIGHT - protection worked
kloong-2 bad bad RIGHT - protection worked
kloong-3 bad good WRONG - good one rejected
kloong-4 bad good WRONG - good one rejected
kloong-5 bad good WRONG - good one rejected
kloong-6 bad good WRONG - good one rejected
kloong-7 bad good WRONG - good one rejected
kloong-8 bad good WRONG - good one rejected
kloong-9 bad good WRONG - good one rejected
kloong-10 bad good WRONG - good one rejected
This system is only right 55% of the time. This is not much better than
flipping a coin.
This system doesn't catch any poisonous fleebs. This group is exempt
from the security precautions.
This system falsely accuses kloongs 80% of the time. This group is
constantly scrutinized, usually unnecessarily.
This system misses 33% of poisonous mushrooms.
This system falsely rejects 47% of edible mushrooms.
Just from a mathematical viewpoint, this system doesn't work. This is
the statistical flaw with profiling. It sounds good to say one type is
"twice as likely" to be bad as another type. But as shown in this
example, using that as a basis for profiling doesn't provide reliable
results. Any system that is only right 55% of the time is a failure.
-- Harvey Newstrom, CISSP <www.HarveyNewstrom.com> Principal Security Consultant <www.Newstaff.com>
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 09:14:30 MST