What's Missing

I believe, but will not attempt to demonstrate here, that the widely used applications on the Internet are great for people looking to encounter information about subjects that they already know they are interested in. Google will generally find us the Wikipedia article for just about any search. This formulation hints at what I think is a big shortcoming in these tools: getting information to a decent depth on any topic is much easier than determining the breadth of topics in which we would want to be interested.

I mean:

• Pandora Radio will be able to tell you that, if you like Beirut, you will probably like Andrew Bird and Devotchka. If you like Andrew Bird, it will tell you that you'd probably like Beirut and Devotchka, and if you like Devotchka it will tell you that you'd probably like Beirut and Andrew Bird. But you won't easily find out, starting from any of these bands, if you like Faye Wong. How can we find out?
  
• There was a lot of discussion on a number of political blogs recently about Epistemic Closure, a term used in this context to describe a conversation that occurs only between individuals who already agree on the topic discussed, and that has the effect of cementing the participants' opinions. This is certainly facilitated by the Internet.
  
• It's been argued that America is self-segregating into demographically and ideologically similar clusters (i.e. geographically constrained units in which within-region variation is much lower than between-region variation).
  

A theme of this blog is the danger associated with too much social learning (too much "received wisdom"), and too little individual learning (including techniques like deriving results from first principles, or from experimentation). Obviously some balance needs to be maintained, and I believe this balance will be different for different subjects, and for different people. One thing I have tried (and am trying) to contribute is a discussion about what this balance should be, under what circumstances. A mathematician probably believes that she needs to understand all the ideas she uses for her own work thoroughly. In my first undergraduate statistics courses, we were taught how to use particular statistical methods, but were taught almost nothing about how or why they work. The mathematician favored individual learning, the statistician (at least at that low level) favored social learning.

In this post, though, I'd like to pose the question: what is the best way to leverage the breadth of communities on the Internet, so that we can discover things in domains in which we haven't shown much interest before? How can we easily discover things that will challenge our beliefs, instead of reinforcing them? If our beliefs are never really challenged, we'll never really know who we are as individuals... Some things are happening in this area already. I have some ideas of my own. I'll try to expand on this later.

Population Variance

Trying to put down why I think other animals are not as cultural as humans, I think it's important to go off on a little tangent about the importance of variation in traits within populations... It'll take me a little while to get to stuff I consider less appreciated (and therefore more interesting), so please bear with me.

Darwin formulated his theory of evolution while ignorant of genetics. Rather than the atomic inheritance model developed by Mendel and others, Darwin believed in blending inheritance - offspring would tend to be about halfway between each of their parents in any trait. I know R. A. Fisher pointed out that it is very difficult to reconcile blending inheritance with evolution (though I don't believe he was the first to make the point), because blending inheritance naturally reduces the variance of a trait in a population from generation to generation. In fact, I believe variance is supposed to be halved in each generation, when each child is thought to be the mathematical average of the parents. The existence of wide variation in nature, when coupled with blending inheritance, implied an impossible amount of adaptively more-or-less neutral mutation.

Hardy and Weinberg showed that this is not the case for Mendelian inheritance - for each generation reproducing by mixture of atomic genes, in a large enough population, with no selective advantage between genetic variants, and with a couple of other constraints, there will be no change in population variance in genes, or in the traits expressed by these genes. This equilibrium is actually a pretty strong force in a large number of species - in other words, the current gene pool has a lot of inertia, and evolutionary forces will generally act very slowly.

Let's take an example: Peter and Rosemary Grant were able to observe evolution occurring in Darwin's finches through two severe ecological changes - in one case, a drought, and in the other, a flood - and watched the distribution of beak sizes in the population change in response to both events - in the one case, the beaks became shorter and wider to crack open the tougher seeds during the drought, and then longer and narrower after the flood to more efficiently access the softer, smaller seeds that then became abundant. To give a sense of what may be happening evolutionarily, I'll construct a deliberately simplistic genetic model of beak shape: say there are four genes, each with two alleles, that determine the beak shape of a finch. For each gene, one of the alleles will cause the beak to be more stout, and the other will cause the beak to be more lean. We'll call the "stout" allele 0, and the "lean" allele 1, and overall beak shape is determined by how many "stout" and (by definition) "lean" alleles the organism possesses. Any individual in this population will then have K stout alleles, and 4-K lean alleles.

I've set this up to deliberately produce a binomial distribution of beak shapes (though there are complications even here - I note these complications in order to ignore them). In our model, we'll say that after the drought individuals with 3 stout alleles and 1 lean allele will have the optimal beak shape; after the flood, individuals with 3 lean alleles and 1 stout allele will have optimal beak morphology. In neither case should any allele be completely eliminated from the population, which means that there will always be the potential to adapt to the flood after a drought, and to the drought after the flood.

This leads directly to what I think is the core of why culture is not nearly so intensely used in non-human animals as it is in humans: cultural evolution includes a lot of blending of received information, blending inheritance reduces population variance, and reduced variance lowers the ability to respond to ecological change. I may unpack this more later...

Culture and the Central Limit Theorem

James Surowiecki famously applied the Central Limit Theorem of statistics to market behavior in his book The Wisdom of Crowds. I have not read the book, however the wikipedia page seems coherent enough (though I disagree with some of it), and anyone with a background in statistics is probably already familiar with the idea. I do, however, want to point out an irony in his approach already apparent from the title of the book: it is difficult for a "wise" crowd to access the wisdom it generates. I think that unpacking this further may begin to account for why there are no other animals as extensively cultural as humans (but does not explain why we are so cultural).

I don't want to get much into the discussion of the statistics, which are probably valid. Rather, I want to talk about the assumptions required for these statistics to work. The major assumption in the CLT is that the random variables in one's sample are independent and identically distributed. In the sorts of scenarios Surowiecki (apparently) describes (like guessing the number of beans in a jar of jelly beans), these assumptions more-or-less holds until the point when the average is taken. I'll try to make my point more clear by a couple of scenarios:

In both scenarios, we'll place a jar of, say, dollar coins in front of a crowd, and the jar gets awarded to the person who guesses closest to the number of coins in the jar. In the first scenario, everyone makes a private guess about this number; in the second, we'll follow a "Price-is-Right" model, and ask participants in order (and never asking more than once) what they think the number is, while allowing later participants to eavesdrop on earlier. As a further assumption, lets say that everyone in the crowd understands and can use the central limit theorem. There are two questions that immediately arise:

1. What is the best strategy for an individual to choose?
   
2. In which of the two scenarios does the average guess of the crowd come closest to the actual number of coins in the jar?
   

In scenario 1, the answer to the first question is obvious: the respondent should make the guess that matches his or her internal idea about the number of coins in the jar. The responses will be independent, and probably have pretty similar distribution centered around the actual contents of the jar, so the CLT ought to be able to characterize the answer to the second question quite well.

In scenario 2, the answer is much less obvious. Each person's answer will depend on both what that person believes the true answer to be, and on the circumstances in which he or she answers the question; If I know I'm the last to answer the question, but my internal idea about the count is widely larger from what all others have answered so far, the best guess to make is *not* what I believe the number is, but rather the guess which seems likely to be closer to the true value than the others, rather than that which will be closest. An epsilon greater than the largest value picked by previous guessers. The upshot is that the best strategy is no longer independent of the actions of other participants, and the CLT can not be applied.

This is likely to be true of any game with eavesdropping. Early guesses may be relatively independent, but later participants who know this will also know that they can make a better guess by taking the mean of those early, independent guesses. The obvious danger is that maybe you've sampled the guess of someone else playing the exact same strategy as you have, without knowing it - now you've increased the number of samples, and decreased the variance within the population of the parameter you've attempted to guess. After several iterations of this strategy, the sample variance is now smaller than the population variance, and the variance of the sampling distribution of the mean rapidly approaches 0. It appears, statistically, like there is a very narrow confidence interval for the population parameter, but it's an illusion - there's no more information added to the system when later samples are describable as functions of earlier samples. The result roughly approximates mobbing behavior, like we see in the market.

Later I'll try to argue why I think this makes culture a generally losing proposition across species...