...helping clients in complex environments make better decisions, faster.

CALL NOW: 404.898.9096

info@decisionclarityconsulting.com

Posts Tagged ‘Faulty Analysis’

Fairy tales masquerading as evidence

Science bible stories, take 27 by Mark Liberman is a useful discussion (including in the comments) about the tendency of media to take up a topical research paper without regard to the methodological robustness of the study or whether the results are meaningfully true.

As I observed a few years ago, “scientific studies”  have taken over the place that bible stories used to occupy. It’s only fundamentalists like me who worry about whether they’re true. For most people, it’s enough that they can be interpreted to be morally instructive.

[snip]

I’d add a third important factor: by and large, the “wise men” (and now the “wise women”) don’t really care about whether the empirical and theoretical foundations of their opinions are sound . They care about readers, ratings, and reputation — and in some cases about political outcomes or cultural values —  with truth relevant only insofar as it affects those goals.

I think Liberman is correct.  People rarely consider what evidence they need in order to make an argument, instead they go after information that is convenient to get.  At the same time, the market structure for ideas and information is such that there are incentives to produce affirming information to a range of prejudices, regardless of the truth of the matter.  Elsewhere I refer to this as cognitive pollution as it constitutes dirt in the system that tends to occlude rather than clarify.

RELATED:  The culturomic psychology of urbanization by Mark Liberman

To analyze does not necessarily mean to produce useful information

From Reviewing the Movies: Audiences vs. Critics by Catherine Rampell.

It is a fair and interesting question or set of questions.  Do audiences and critics assess movies in different ways?  If so, in what ways do they differ?  Which views, audience or critics, provide a better forecast of future performance?  These questions apply to art, sports, books, etc. There are answers to some of these questions.  The general informed public and specialists do tend to review things differently.  General informed public tend to factor in more context and larger macro considerations than do specialists.  General informed public tend to be better forecasters than are specialists.  Nate Silver covers a lot of this in his The Signal and The Noise: Why So Many Predictions Fail – but Some Don’t
What is notable is that Rampell asks a legitimate question and has an idea on how to answer the question.  Her error is to use information that is available (Rotten Tomatoes Database) rather than information that is needed.  There is a fairly detailed critique of her analysis in the comments.  The article serves as an example of Selection Bias (the distortion of a statistical analysis, resulting from the method of collecting samples) and Information Bias (the tendency to seek information even when it cannot affect action), and possible Mere Exposure Bias (the tendency to express undue liking for things merely because of familiarity with them).

 

Decision Clarity Consulting has no affiliation with the Scheier Group, owner of the DECISION CLARITY®, and the Scheier Group does not sponsor, endorse, or control any material on this site. Decision Clarity Consulting, in accordance with an agreement with Scheier Group, performs no work with non-profit or government entities.