dwar0123, on 2013-June-28, 10:57, said:
You really don't see how investigating something for bias by only looking at one side is intrinsically biased?
It's like trying to determine if a coin flip is truly 50/50 by only counting the results from when it turns up heads.
Not really a good analogy, since anything you find out about heads tells you everything you need to know about tails as well.
A better analogy would be a die. Suppose it's supposed to be a normal 6-sided die, but you suspect someone has replaced it with a 7-sided die, with two 6s. If you compare the number of 1s and 2s that are thrown, you won't see any problem -- they'll be equal. Only when you look at the number of 6s and see that they're twice as frequent as others will you realize there's something wrong.
Of course, you could just plot the rate of all values, and then the problem with 6 will stick out as well. But that's where the analogy starts losing its ability to mirror the real world. In complex situations, it's hard to analyze all cases because there are too many. You have to narrow down your investigation, usually based on a particular hypothesis you're trying to prove or disprove. Even just figuring out how to categorize everything involves some bias -- are you looking for differences based on organization size (and is that number of employees or revenue), gender of upper management, political affiliation, etc.?
This is simply the nature of analysis of any complicated situation.