Ari and I covered some basic material with graduate students yesterday, and I ended up having three guidelines for social science analysis in my notes.
- Anecdote is not the singular of data.
- If you found something surprising, you’re probably wrong (but conversely:)
- Don’t believe everything you think.
Feel free to add your own in comments, or to dispute these, or whatever pleases you, as long as it’s nothing to do with the election.
The Edge of the American West 
67 comments
October 30, 2008 at 11:34 am
Josh
Don’t believe everything you think.
“It will be a great day when the schools get all the money they need and the airforce has to hold a bake sale to buy a bomber”?
October 30, 2008 at 11:39 am
eric
In social-science research, Josh, it’s useful to remember that your model might need changing when it confronts a run of uncooperative data. But that’s less catchy.
October 30, 2008 at 11:48 am
Buster
I think my favorite gem is:
Just because you have a technique, it doesn’t mean you have a method.
October 30, 2008 at 12:00 pm
kid bitzer
why is anecdote *not* the singular of data?
1) i knew a guy once who made $35,000. per annum.
2) at that time, the mean annual income for single men in the country was $42,000.
what’s the matter with saying that 2) is made up of several million instances of things like 1)? with saying that 1) is exactly the kind of unit out of which 2) is composed?
why is that anecdote *not* a datum, which when combined with income-data about several million other households, turns into data about national annual incomes?
is it because 1) is based on my mere testimony, and something does not become a ‘given’ (=datum) merely by my assertion?
that can’t be right–then there could be no self-reporting data.
i realize that anecdotes are annoying for many reasons, and that students need to be warned away from attributing to them probative powers that they do not have.
but isn’t there another way to say this than by saying “anecdote is not the singular of data?”
(of course, that is true for trivial linguistic reasons, to wit: “datum”. but that’s presumably not the point either.)
so what am i missing?
October 30, 2008 at 12:01 pm
andrew
Just because you have a madness, doesn’t mean it has a method.
October 30, 2008 at 12:01 pm
Matt L.
My favorite is from my former Hungarian instructor, who also happened to be a sociologist studying nineteenth century immigration.
Q: What is the the Iron Law of Sociology?
A: Some do, some don’t.
October 30, 2008 at 12:04 pm
Vance
To take a famous example, there is no pun, double syntax, or dubiety of feeling, in
but the comparison holds for many reasons; because ruined monastery choirs are places in which to sing, because they involve sitting in a row, because they are made of wood, are carved into knots and so forth, because they used to be surrounded by a sheltering building crystallised out of the likeness of a forest, and coloured with stained glass and painting like flowers and leaves, because they are now abandoned by all but the grey walls coloured like the skies of winter, because the cold and Narcissistic charm suggested by choir-boys suits well with Shakespeare’s feeling for the object of the Sonnets, and for various sociological and historical reasons (the protestant destruction of monasteries; fear of puritanism), which it would be hard now to trace out in their proportions; these reasons, and many more relating the simile to its place in the Sonnet, must all combine to give the line its beauty, and there is a sort of ambiguity in not knowing which of them to hold most clearly in mind.
October 30, 2008 at 12:09 pm
Vance
More a touchstone than a rule of thumb, I suppose.
October 30, 2008 at 12:16 pm
Ahistoricality
on models and methods:
“No good model ever accounted for all the facts, since some data was bound to be misleading, if not plain wrong.” — James Watson, quoted by Francis Crick
on madness:
“They called me mad, and I called them mad, and damn them, they outvoted me.” — Nathaniel Lee (late 17c), cited in R. Porter, A Social History of Madness (1987)
October 30, 2008 at 12:21 pm
dana
kb: because you don’t collect data just by polling your friends, would be one way to put it.
October 30, 2008 at 12:21 pm
politicalfootball
I’ve always been annoyed by people who say, “Life isn’t fair” in response to a complaint of injustice, but I had a high school chemistry teacher who expressed a similar sentiment in a manner I like: “It’s not the ought-ness that matters, it’s the is-ness.”
October 30, 2008 at 12:31 pm
JPool
I would actually take it the other way — that annecdotes are supposed to be illustrative and/or microhistorical in a way that data aren’t.
October 30, 2008 at 12:34 pm
ben
I believe that there is a pun in that line, Vance, and it’s on the two words we now spell “quire” and “choir”, but which were, or in Shakespeare’s more liberal time could be, spelled similarly in the past.
October 30, 2008 at 12:36 pm
ben
Please convey that sentiment to Mr Empson, if you would.
October 30, 2008 at 12:55 pm
Vance
Huh — in the sense of papers folded together. It certainly works in context (“yellow leaves, or none, or few”). A good deal less farfetched than some of his associations.
Or for a real rule: Never put anything in your ear smaller than your elbow.
October 30, 2008 at 1:16 pm
Max Polun
That isn’t really an anecdote, that’s a free floating fact. An anecdote is usually a story of some sort: “I had a cold but I took vitamin C and got better”. That’s not data because it’s not controlled, you don’t know what would have happened otherwise. A medical study (medical, just to keep with the vitamin C example) is not a bunch of stories of people getting taking or not taking something, but is set up in such a way so as to remove people’s inherent biases. In this example the bias is that people will attribute getting better to something they did, when in fact people get better from colds most of the time.
October 30, 2008 at 1:19 pm
Max Polun
And to answer the post’s question (not that I’m in a position to be offering advice to grad students): People tend to get the results they expect to.
October 30, 2008 at 1:41 pm
kid bitzer
dana– polling your friends won’t give you very good data if you were after data about the population at large. but why isn’t a small, lousy, data-set still a data-set? a fuzzy, poor-resolution picture of a star is a picture of a star.
so if i ask a student to find me data on the national average height of 20 y.o. males, and they go out and measure six of their friends, then i would be inclined to say ‘that is a very inadequate data set which is likely to suffer from numerous sampling biases, in addition to the fact that its small size will prevent if from having much statistical power. get me better data.” still, i can’t see in what sense it is *not* data.
and if the data i’m after simply *is* data about my friends (e.g., i want to understand the food-preferences of my friends), then why doesn’t polling my friends give me unsurpassable data?
max–
i can’t see why you deny that my 1) is not an anecdote. looks anecdotal to me. you want more of a story? i’ll tell you his hair color.
about the vitamin c story you say “That’s not data because it’s not controlled”. but here too, there are lots of times when we collect data that is not controlled. if i do polling among voters about whom they plan to vote for, is that not data?
October 30, 2008 at 1:46 pm
Jonathan Rees
People are complicated.
October 30, 2008 at 1:56 pm
Megan
I can’t believe I have to explain again that three times is a rule.
October 30, 2008 at 1:58 pm
andrew
The difference between an anecdote and a data point has to do with how the information is collected, not the quality of the information. Anecdotes are bits of information you’ve happened upon; data is gathered for some reason. So an anecdote could be made into a data point if you were collecting anecdotes into a (probably not very good) data set. Maybe it should be “anecdotes make bad data.”
This is just a story I made up.
October 30, 2008 at 1:59 pm
Mr. Sidetable
Nobody has ever done their “best work.”
October 30, 2008 at 2:06 pm
andrew
A search is only as good as its metadata.
October 30, 2008 at 2:06 pm
kid bitzer
Nobody has ever done their “best work.”
certainly all my best work was done by someone else, before i stole it.
October 30, 2008 at 2:36 pm
drip
My version of Eric’s 3 is: Its OK to talk to yourself, just don’t listen to yourself.
October 30, 2008 at 2:41 pm
TF Smith
A beer in the hand is worth two in the fridge.
October 30, 2008 at 3:20 pm
soup biscuit
In social-science research, Josh, it’s useful to remember that your model might need changing when it confronts a run of uncooperative data. But that’s less catchy.
Catchier: In social-science research, it’s useful to remember that your model might need changing.
October 30, 2008 at 3:24 pm
bill
From Newtonian mechanics: F=mA, and you can’t push a rope.
October 30, 2008 at 4:10 pm
urbino
If you enter the village on the back of a tiger, consider that your stay may be short.
Also, never hit your mother with a shovel.
October 30, 2008 at 4:22 pm
Sam-I-am
Asking someone to be on your committee is like asking someone to marry you: there should be enough of a relationship that they can see the question coming.
Or perhaps that rule of thumb was only necessary at U of Chicago, where as true free-marketeers they allow grad students to sink or swim as they are wont.
October 30, 2008 at 4:46 pm
eric
Catchier: In social-science research, it’s useful to remember that your model might need changing.
In socialist science, model changes you!
Actually, that would probably be funnier if it weren’t true.
October 30, 2008 at 4:51 pm
kathy a.
anecdotes can raise good questions, and invite further explorations.
anecdotes can illustrate larger points. a collection of anecdotes is even better.
but they only stand alone where the question is something like: what is your personal experience with X? and sometimes, that is a good question.
October 30, 2008 at 8:04 pm
bitchphd
never hit your mother with a shovel.
A rule that bears repeating.
October 30, 2008 at 8:11 pm
urbino
That one was for Sundance. Or Butch.
October 31, 2008 at 1:15 am
Brad
kb – a fuzzy, poor-resolution picture of a star is a picture of a star.
actually, most pictures of stars tell you far more about the telescope you used to take a picture of them than the star itself.
Just like many ancedotes tell you far more about the person telling the story than they provide actual data.
October 31, 2008 at 4:50 am
kid bitzer
many reasons to conclude that:
collecting anecdotes is generally a piss poor way to collect data.
and that seems like a good rule to teach your students.
and it avoids suggesting–falsely, as far as i can tell– that anecdotes differ from data in some categorical fashion.
October 31, 2008 at 5:58 am
dana
polling your friends won’t give you very good data if you were after data about the population at large. but why isn’t a small, lousy, data-set still a data-set?
An anecdote isn’t a small, lousy data set. It’s a story that may or may not have some useful data in it. Max’s example is good here: “I took some Vitamin C and I got better” is an anecdote, but we’re not sure what data results from that: something about the efficacy of vitamin C, something about the relatively short duration colds, etc.
October 31, 2008 at 6:46 am
kid bitzer
so that proposes a difference of, what, form? the anecdote contains a variety of data, but they have to be extracted from it?
in that case, would you accept that anecdotes can deliver the singular of data?
October 31, 2008 at 6:48 am
kid bitzer
and max’s example seems to me to show a very good piece of data about the relative therapeutic efficacy of vitamin c and cyanide.
October 31, 2008 at 6:57 am
M. Showperson
kathy a: Were Tom Friedman to add another anecdote, would he get better?
October 31, 2008 at 8:32 am
Cosma
The key to making methodological advice memorable is to take a good idea, over-simplify it, and then exaggerate.
So, yeah, anecdotes have some evidential value. A fairer statement might be “a good data set is a collection of anecdotes which are picked so as to be fairly representative of the larger population of instances.” Doesn’t lodge in the student’s mind so effectively.
October 31, 2008 at 8:33 am
Cosma
In the spirit of “over-simplify, then exaggerate”:
“Regression isn’t causation either.”
October 31, 2008 at 8:35 am
kid bitzer
part of why i’m sounding contrarian here is because i’m worried about a different intellectual vice than the one that is motivating the original rule of thumb.
you’re worried about students who think that anecdotes are just as good as data.
i’m worried about people who think that anecdotes are *better*.
here’s the scenario: you tell someone that studies show that you are more likely to survive a car crash if you are buckled up.
they respond by saying that their uncle bud was coming back from duck-hunting one morning and crashed his pickup truck. he got thrown clear of the truck, and what do you know, it went up in flames. and, they tell you, what happened to uncle bud counts a lot more for them than a bunch of statistics.
i want to be able to say to them: i have a registry of some 20,000 single-vehicle accidents, showing that for every person who survived without a belt, there were five people who died without a belt. whereas the numbers were reversed for the belted.
your uncle bud is in that registry. and there are about 20,000 other people just like him in that registry. and for every anecdote like your story about uncle bud, i can tell you five more about susie, joe, ravi, deon, and bill, each of whom died because they weren’t wearing a belt.
so i want to be able to say that anecdotes and data are commensurable, because i have to deal with people who think that anecdotes have an incommensurably *greater* value than statistics do.
part of breaking the grip that anecdotes enjoy (in virtue of their greater salience, vividness, and corroborative detail), is pointing out that every data point distills an anecdote of exactly that sort.
so that’s the main thing that is driving me to question the common adage that “anecdote is not the singular of data”.
October 31, 2008 at 8:44 am
ben wolfson
Not all things are possible at all times.
October 31, 2008 at 8:50 am
kid bitzer
rules of thumb occasionally fail to apply.
(is that a metarule of thumb, or a rule of metathumb?)
October 31, 2008 at 8:56 am
Vance
Speaking of anecdotes, Packer links to this map, which seems fun (Obama takes Svalbard!) until you notice that BHO gets 88% in the US too.
October 31, 2008 at 9:00 am
kid bitzer
and burkina faso goes unanimously into the mccain camp, 1 vote to zero!
who is that guy in burkina faso? and why does he hate the good?
October 31, 2008 at 9:02 am
Vance
KB, I take your point. Yet we know that (inverting Eric’s formulation) data is not the plural of anecdote — it’s not hard for quacks to accumulate testimonials for snake oil. There’s some further difference, maybe impartial selection, that lends greater significance to “data”….if not, as you say, greater rhetorical effect.
October 31, 2008 at 9:09 am
kid bitzer
sure, but impartial selection cannot be constitutive of what it is to be data, or else “this data set suffers from severe selection biases” would be a contradiction in terms.
you don’t want to ratchet up the criteria on what counts as data to such a point that there’s no bad data.
cause i’d say that one good rule of thumb would be: a lot of data out there isn’t very good data.
October 31, 2008 at 9:19 am
Vance
OK, you may have turned over the last rock there. When we say “anecdote is not the singular of data”, we’re using a standard rhetorical figure to imply that we mean good data — dismissing bad data as “not even data”.
October 31, 2008 at 9:47 am
Vance
For example, someone once told me, “The problem with [some group of] cooks is that they don’t know anything about food.” Meaning, not that these cooks don’t know that butter comes from cows, but that the difference between good and bad food is so clear and so stark that it’s tantamount to the difference between food and not-food.
October 31, 2008 at 10:04 am
joel hanes
Alexander’s Law :
“You cannot simultaneously antagonize and influence”
October 31, 2008 at 10:17 am
kid bitzer
“you may have turned over the last rock there”
that’s certainly an evocative phrase, and i have a feeling that it describes something i will wish i had not done, but other than that i’ll be damned if i know what it means.
October 31, 2008 at 10:23 am
eric
“Regression isn’t causation either.”
Ah, that’s a good one.
October 31, 2008 at 10:29 am
kid bitzer
but in any case, vance, if the non-catchy meaning of ‘anecdote is not the singular of data’ now turns on ‘data’ meaning ‘good data’, then i have no problem with it (except that it buys catchiness at the price of being misleading).
the fully-expanded, non-catchy version turns out to be something like ‘sure, some anecdotes constitute data, but not very good data’. and that’s all i’ve been arguing for.
(did i just turn over the last rock a second time? or the n+1th rock?)
October 31, 2008 at 10:43 am
chris y
“Avoid the Baptist Preacher Fallacy.” Or is that too elementary?
October 31, 2008 at 11:05 am
Vance
Hmm, I just Googled “Baptist Preacher Fallacy” and found another comment by chris y before any explanation. Perhaps it’s the fallacy that a catchily named principle means what one implies it does?
Similarly, Alexander’s Law appears to be “that in individuals with nystagmus, the amplitude of the nystagmus increases when the eye moves in the direction of the fast phase (saccade)”.
I can attest, though, to the wisdom of Michael Jackson, usually given as “The First Rule of Program Optimization: Don’t do it. The Second Rule of Program Optimization (for experts only!): Don’t do it yet.”
October 31, 2008 at 11:17 am
joel hanes
An anecdote is necessarily personal and singular, as all occasions are unique; it carries many sorts of detail and random association that are suppressed when it becomes a datum. No anecdote can ever truly be “typical”.
October 31, 2008 at 11:36 am
Josh
I can attest, though, to the wisdom of Michael Jackson
Does that pre- or post-date Knuth’s Law?
October 31, 2008 at 11:38 am
Vance
Same era; not sure which came first.
October 31, 2008 at 1:55 pm
urbino
Oh, if we’re going for programming, I offer Urbino’s 2 Fundaments for Computer Programmers:
1) Computers is ignorant.
2) If a user can break it, they will.
A full understanding of these is the beginning of wisdom.
More generally, I offer: never bet against a Sicilian when death is on the line.
October 31, 2008 at 2:39 pm
ben wolfson
If the topic is computers, let’s all please refrain from quoting Perlis.
October 31, 2008 at 6:32 pm
joel hanes
Brooks’ Law
Adding programmers to a late software project
makes it later.
November 1, 2008 at 11:42 am
Charlieford
I’ve always held to the maxim, “A historian can’t say the words ‘social science’ with a straight face.”
November 1, 2008 at 7:57 pm
joel hanes
Never get involved in a land war in Asia.
November 2, 2008 at 7:28 pm
Charlieford
And abandon the creeping meatball while you’re at it.
November 4, 2008 at 3:45 am
Wrye
1) Higher Education is a Casino.
2) If you can’t figure out who the sucker at the table is, it’s you.