Pareto Distributions and Social Ills

You might have heard of the Pareto principle, more commonly known as the 80/20 rule. To paraphrase, the 80/20 rule describes systems where 20% of the inputs account for 80% of the outputs. For example:

• 80% of the work is done by 20% of the people.
• 80% of the wealth is owned by 20% of the population.
• 80% of our food comes from 20% of our land.

If you apply the principle recursively (e.g. within those 20% of the people who do 80% of the work, 20% of those people are responsible for 80% of their higher level of output, and so on with any subdivision), you end up with a Pareto distribution with a parameter of log₄(5). That is to say, the distribution is self-similar.

Wealth is usually agreed as being more-or-less Pareto distributed (in fact, it was the study of wealth distribution that lead Pareto to describe this distribution in the first place), though the parameter will be different depending on the level of inequality in the population under study. I think it's interesting to consider the interaction of this distribution with the sociological finding that individual happiness depends more on relative material well-being vs. one's neighbors than on absolute material well-being.

Let's start with the idea that the maximum size of a person's meaningful social network is approximately normally distributed across the population, and is not too dependent on the person's socio-economic-status. Strictly for illustration, we'll choose the mean maximum size of a person's social network as 100. Let's also assume that we tend to socialize with people with similar social status to our own. The question, then, is, under this system, how does the level of inequality that a person is exposed to vary with how much money he makes? And, further, how does his income affect his happiness?


Observing the slopes of these Lorenz curves, it's clear that a person nearer the right of the distribution (in our scenario, a member of the rich) will be exposed to much greater inequality than one further away. He is more likely to know people who have (what he'll consider) unjustifiably more income than himself, which will tend to hurt his happiness. Further, as inequality increases, there will be fewer "rich" who are exposed to more inequality within their own social circle, and who will be less happy with their lot as a result. An interesting corollary is that, in a less equal society, people on the poorer end would be exposed to less inequality within their social network - the perception of inequality is also distributed unequally.

There are a few outcomes that I think result from this dynamic. Speaking in broad generalizations, and ignoring the behavior of specific individuals, I think there will be a tendency for:

• The rich to become ever more individually motivated, as the increase in income from a moderate improvement in position grows exponentially as you become richer.
• A rich person's sense of well-being to be reduced. We try to live within our means, but in an unequal society, the rich will experience much greater volatility in their means than the rest of us do, making it harder to determine what the means are that they are trying to live within. This is stressful.
• The poor to lose motivation, as moderate improvement in relative position will not lead to a proportionate increase in income.

Some of this discussion conflates within-lifetime inequality to across-society inequality. I will try to address that, later.

Intellectually Tractable Code

The task of writing software consists in large part of minimizing the intellectual burden necessary to completely understand a problem and its solution. In fact, I would argue that our intuition about how "clean" a piece of software is is, in essence, a measure of how closely our understanding of a solution matches the "essential" complexity of an ideal solution. A "clean" module will have a low intellectual burden for complete understanding.

As an example, consider two alternative library implementations of a singly-linked-list data structure:

• In implementation A, list operations include semaphore locks, to ensure that the list is always in the appropriate state.
  
• In implementation B, there are no locks - it is assumed that the caller will always take care to ensure that list manipulation / use is appropriately protected for concurrent access.

It is my contention that implementation B is cleaner, for two major reasons:

1. There is no one design decision for the locking semantics that will be universally applicable. For example, there may be state-variable relationships to be maintained that cross the locking boundary for that entry on the list (like "links 1 and 2 must be added and removed from two different lists at the same time"); the type of locking is set in stone at compile-time, though it may be better to be different for different scenarios (e.g. lock using semaphores for this list; lock using critical sections for this other list; no locks necessary at all for this third list). This means that the developer must _still_ reason about locking in order to use this module effectively - only now it isn't as obvious that she needs to do so, or (to another developer) it isn't as obvious if she _has_ reasoned about locking to protect the relationships between her state variables. Run-time use is less intellectually tractable.
   
   
2. It adds unnecessary dependencies for the linked-list module to work. Linked-list maintenance is well-suited to use in a single-threaded environment, but embedding locks within the linked-list maintenance code prevents deploying the module to environments that lack appropriate locking primitives. Unnecessary dependencies can also lead to layering problems - as when the linked-list maintenance depends upon the scheduler's locking primitives, what happens if the scheduler depends upon the linked-list module? What happens if the scheduler depends upon the linked-list module during initialization, prior to the locking primitives becoming available for use? Instantiation is less intellectually tractable.

There exist general guidelines for making code intellectually tractable, which I will try to expand upon in later posts.

Mastery Against Generality (or, "Keep It Concrete, Stupid")

LtU pointed me to Design Principles Behind Smalltalk. I'll quote the first design principle named, because it illustrates what I think is a mistake in reasoning common to those of a mathematical bent:

Personal Mastery: If a system is to serve the creative spirit, it must be entirely comprehensible to a single individual.

The point here is that the human potential manifests itself in individuals. To realize this potential, we must provide a medium that can be mastered by a single individual. Any barrier that exists between the user and some part of the system will eventually be a barrier to creative expression. Any part of the system that cannot be changed or that is not sufficiently general is a likely source of impediment. If one part of the system works differently from all the rest, that part will require additional effort to control. Such an added burden may detract from the final result and will inhibit future endeavors in that area. We can thus infer a general principle of design:

Good Design: A system should be built with a minimum set of unchangeable parts; those parts should be as general as possible; and all parts of the system should be held in a uniform framework.

I strongly sympathize with the point of view outlined here. If one can master simple, general principles, then that reduces the burden for understanding some set of more specific ideas, and can potentially greatly increase the number and scope of the ideas one can understand and use at any given time -- it can improve one's intellect.

That said, it is considerably more difficult to impart understanding of general ideas than of specific ones. If this isn't immediately obvious to you, consider the order in which you learned some mathematical concepts. Take the following problems:

1. If joey has three apples, and gives two away, how many does he now have?
   
2. Solve for x: x = 3 - 2.
   
3. Prove that the addition operation under the set of integers modulo some constant forms a group.
   
4. Give an example of a non-Abelian group.
   
5. What the hell is a left-adjoint functor?

I think you can be expected to gain mastery of each of these problems in the same order in which the problems are listed. Each problem is more abstract than the previous, and each successive problem is, in a sense, simpler and more general than the previous. But each successive problem is also, to my mind, more difficult than the previous: We needed to understand the more specific ideas before we could be expected to generalize. The mechanism for "good design" quoted above can probably be considered to be in some tension with the stated goal of maximizing "personal mastery" of the system.

The implications of this tension are, I think, important. In particular, under the assumption that a more abstract understanding of a problem domain can make the problem more tractable, it's usually in any given author's interest to move "up" the "abstraction ladder", in order to better solve a problem herself. To the extent that this means she happens upon a good solution faster than others, this is all to the good. But to the extent that this means she happens upon much different solutions than others would, then however much more "elegant" her own solution is will be weighed against the cost in comprehension for others she works with.

What's Missing

I believe, but will not attempt to demonstrate here, that the widely used applications on the Internet are great for people looking to encounter information about subjects that they already know they are interested in. Google will generally find us the Wikipedia article for just about any search. This formulation hints at what I think is a big shortcoming in these tools: getting information to a decent depth on any topic is much easier than determining the breadth of topics in which we would want to be interested.

I mean:

• Pandora Radio will be able to tell you that, if you like Beirut, you will probably like Andrew Bird and Devotchka. If you like Andrew Bird, it will tell you that you'd probably like Beirut and Devotchka, and if you like Devotchka it will tell you that you'd probably like Beirut and Andrew Bird. But you won't easily find out, starting from any of these bands, if you like Faye Wong. How can we find out?
  
• There was a lot of discussion on a number of political blogs recently about Epistemic Closure, a term used in this context to describe a conversation that occurs only between individuals who already agree on the topic discussed, and that has the effect of cementing the participants' opinions. This is certainly facilitated by the Internet.
  
• It's been argued that America is self-segregating into demographically and ideologically similar clusters (i.e. geographically constrained units in which within-region variation is much lower than between-region variation).
  

A theme of this blog is the danger associated with too much social learning (too much "received wisdom"), and too little individual learning (including techniques like deriving results from first principles, or from experimentation). Obviously some balance needs to be maintained, and I believe this balance will be different for different subjects, and for different people. One thing I have tried (and am trying) to contribute is a discussion about what this balance should be, under what circumstances. A mathematician probably believes that she needs to understand all the ideas she uses for her own work thoroughly. In my first undergraduate statistics courses, we were taught how to use particular statistical methods, but were taught almost nothing about how or why they work. The mathematician favored individual learning, the statistician (at least at that low level) favored social learning.

In this post, though, I'd like to pose the question: what is the best way to leverage the breadth of communities on the Internet, so that we can discover things in domains in which we haven't shown much interest before? How can we easily discover things that will challenge our beliefs, instead of reinforcing them? If our beliefs are never really challenged, we'll never really know who we are as individuals... Some things are happening in this area already. I have some ideas of my own. I'll try to expand on this later.

Population Variance

Trying to put down why I think other animals are not as cultural as humans, I think it's important to go off on a little tangent about the importance of variation in traits within populations... It'll take me a little while to get to stuff I consider less appreciated (and therefore more interesting), so please bear with me.

Darwin formulated his theory of evolution while ignorant of genetics. Rather than the atomic inheritance model developed by Mendel and others, Darwin believed in blending inheritance - offspring would tend to be about halfway between each of their parents in any trait. I know R. A. Fisher pointed out that it is very difficult to reconcile blending inheritance with evolution (though I don't believe he was the first to make the point), because blending inheritance naturally reduces the variance of a trait in a population from generation to generation. In fact, I believe variance is supposed to be halved in each generation, when each child is thought to be the mathematical average of the parents. The existence of wide variation in nature, when coupled with blending inheritance, implied an impossible amount of adaptively more-or-less neutral mutation.

Hardy and Weinberg showed that this is not the case for Mendelian inheritance - for each generation reproducing by mixture of atomic genes, in a large enough population, with no selective advantage between genetic variants, and with a couple of other constraints, there will be no change in population variance in genes, or in the traits expressed by these genes. This equilibrium is actually a pretty strong force in a large number of species - in other words, the current gene pool has a lot of inertia, and evolutionary forces will generally act very slowly.

Let's take an example: Peter and Rosemary Grant were able to observe evolution occurring in Darwin's finches through two severe ecological changes - in one case, a drought, and in the other, a flood - and watched the distribution of beak sizes in the population change in response to both events - in the one case, the beaks became shorter and wider to crack open the tougher seeds during the drought, and then longer and narrower after the flood to more efficiently access the softer, smaller seeds that then became abundant. To give a sense of what may be happening evolutionarily, I'll construct a deliberately simplistic genetic model of beak shape: say there are four genes, each with two alleles, that determine the beak shape of a finch. For each gene, one of the alleles will cause the beak to be more stout, and the other will cause the beak to be more lean. We'll call the "stout" allele 0, and the "lean" allele 1, and overall beak shape is determined by how many "stout" and (by definition) "lean" alleles the organism possesses. Any individual in this population will then have K stout alleles, and 4-K lean alleles.

I've set this up to deliberately produce a binomial distribution of beak shapes (though there are complications even here - I note these complications in order to ignore them). In our model, we'll say that after the drought individuals with 3 stout alleles and 1 lean allele will have the optimal beak shape; after the flood, individuals with 3 lean alleles and 1 stout allele will have optimal beak morphology. In neither case should any allele be completely eliminated from the population, which means that there will always be the potential to adapt to the flood after a drought, and to the drought after the flood.

This leads directly to what I think is the core of why culture is not nearly so intensely used in non-human animals as it is in humans: cultural evolution includes a lot of blending of received information, blending inheritance reduces population variance, and reduced variance lowers the ability to respond to ecological change. I may unpack this more later...

Culture and the Central Limit Theorem

James Surowiecki famously applied the Central Limit Theorem of statistics to market behavior in his book The Wisdom of Crowds. I have not read the book, however the wikipedia page seems coherent enough (though I disagree with some of it), and anyone with a background in statistics is probably already familiar with the idea. I do, however, want to point out an irony in his approach already apparent from the title of the book: it is difficult for a "wise" crowd to access the wisdom it generates. I think that unpacking this further may begin to account for why there are no other animals as extensively cultural as humans (but does not explain why we are so cultural).

I don't want to get much into the discussion of the statistics, which are probably valid. Rather, I want to talk about the assumptions required for these statistics to work. The major assumption in the CLT is that the random variables in one's sample are independent and identically distributed. In the sorts of scenarios Surowiecki (apparently) describes (like guessing the number of beans in a jar of jelly beans), these assumptions more-or-less holds until the point when the average is taken. I'll try to make my point more clear by a couple of scenarios:

In both scenarios, we'll place a jar of, say, dollar coins in front of a crowd, and the jar gets awarded to the person who guesses closest to the number of coins in the jar. In the first scenario, everyone makes a private guess about this number; in the second, we'll follow a "Price-is-Right" model, and ask participants in order (and never asking more than once) what they think the number is, while allowing later participants to eavesdrop on earlier. As a further assumption, lets say that everyone in the crowd understands and can use the central limit theorem. There are two questions that immediately arise:

1. What is the best strategy for an individual to choose?
   
2. In which of the two scenarios does the average guess of the crowd come closest to the actual number of coins in the jar?
   

In scenario 1, the answer to the first question is obvious: the respondent should make the guess that matches his or her internal idea about the number of coins in the jar. The responses will be independent, and probably have pretty similar distribution centered around the actual contents of the jar, so the CLT ought to be able to characterize the answer to the second question quite well.

In scenario 2, the answer is much less obvious. Each person's answer will depend on both what that person believes the true answer to be, and on the circumstances in which he or she answers the question; If I know I'm the last to answer the question, but my internal idea about the count is widely larger from what all others have answered so far, the best guess to make is *not* what I believe the number is, but rather the guess which seems likely to be closer to the true value than the others, rather than that which will be closest. An epsilon greater than the largest value picked by previous guessers. The upshot is that the best strategy is no longer independent of the actions of other participants, and the CLT can not be applied.

This is likely to be true of any game with eavesdropping. Early guesses may be relatively independent, but later participants who know this will also know that they can make a better guess by taking the mean of those early, independent guesses. The obvious danger is that maybe you've sampled the guess of someone else playing the exact same strategy as you have, without knowing it - now you've increased the number of samples, and decreased the variance within the population of the parameter you've attempted to guess. After several iterations of this strategy, the sample variance is now smaller than the population variance, and the variance of the sampling distribution of the mean rapidly approaches 0. It appears, statistically, like there is a very narrow confidence interval for the population parameter, but it's an illusion - there's no more information added to the system when later samples are describable as functions of earlier samples. The result roughly approximates mobbing behavior, like we see in the market.

Later I'll try to argue why I think this makes culture a generally losing proposition across species...

I'd just like to appropriate this...

I noticed the Atlantic had a new blog by Jeffrey Goldberg, who wrote one of the best articles I've recently read on Israel. I don't know how well the blog will turn out, but I wanted to align myself with this sentiment from his opening post:

This is almost certainly a mistake.

Friends tell me that I will take naturally to blogging because I am in possession of many poorly considered opinions about issues I understand only marginally. I am dubious, however.

Yup.