Broken Symmetry

More on the Long Tail

As Michael Nielsen explains, the value of the exponent in your power law matters a great deal:

If you’re developing a long tail business or collaboration, you need to make sure the exponent in the power law is less than one. The smaller the exponent, the better off you’ll be.

How can you make the exponent as small as possible? In particular, how can you make sure it’s smaller than the magic value of one? To understand the answer to this question, we need to understand what actually determines the value of the exponent. There’s some nice simple mathematical models explaining how power laws emerge, and in particular how the power law exponent emerges. At some point in the future I’d like to come back and discuss those in detail, and what implications they have for site architecture. This post is already long enough, though, so let me make just make three simple comments.

First, focus on developing recommendation and search systems which spread attention out, rather than concentrating it in the short head of what’s already popular. This is difficult to do without sacrificing quality, but there’s some interesting academic work now being done on such recommendation systems – see, for example, some of the work described in this recent blog post by Daniel Lemire.

Second, in collaborative projects, ensure a low barrier to entry for newcomers. One problem Wikipedia faces is a small minority of established Wikipedians who are hostile to new editors. It’s not common, but it is there. This drives newcomers away, and so concentrates edits within the group of established editors, effectively increasing the exponent in the power law.

Third, the essence of these and other similar recommendations is that they are systematic efforts to spread attention and contribution out, not one-off efforts toward developing a long tail of sales or contributions. The problem with one-off efforts is that they do nothing to change the systematic architectural factors which actually determine the exponent in the power law, and it is that exponent which is the critical factor.

If the power law is a description of network clustering, then you could say that the power law is likely to shift from less than 1 to greater than 1 as the strength of preferential attachment increases.

Leverage Causes Fat Tails and Clustered Volatility

From Stefan Thurner, J. Doyne Farmer, John Geanakoplos:

When funds use leverage, price fluctuations become heavy tailed and display clustered volatility, similar to what is observed in real markets. Previous explanations of fat tails and clustered volatility depended on 'irrational behavior', such as trend following. We show that the immediate cause of the increase in extreme risks in our model is the risk control policy of the banks: A prudent bank makes itself locally safer by putting a limit to leverage, so when a fund exceeds its leverage limit, it must partially repay its loan by selling the asset. Unfortunately this sometimes happens to all the funds simultaneously when the price is already falling. The resulting nonlinear feedback amplifies downward price movements. At the extreme this causes crashes, but the effect is seen at every time scale, producing a power law of price disturbances. A standard (supposedly more sophisticated) risk control policy by individual banks makes these extreme fluctuations even worse. Thus it is the very effort to control risk at the local level that creates excessive risk at the aggregate level, which shows up as fat tails and clustered volatility.

Economics that can be Falsified

From K. Bastiaensen, P. Cauwels, D. Sornette, R. Woodard, and W.-X. Zhou.

Amid the current financial crisis, there has been one equity index beating all others: the Shanghai Composite. Our analysis of this main Chinese equity index shows clear signatures of a bubble build up and we go on to predict its most likely crash date: July 17-27, 2009 (20%/80% quantile confidence interval).

Available here. Found via Physics arXiv blog. Can a reader tell me who authors and edits this blog?

Didier Sornette's work on the endogenous and exogenous trends in YouTube video watching was covered in this earlier post.

The argument here rests on the observation that faster-than-exponential growth is not physically sustainable. A similar point was made in an earlier post on this blog regarding venture capital financing and startup growth: The Fundamental Speed Limit on the Growth of a Startup.

UPDATE (August 2, 2009): Make that, economics that has been falsified. The Shanghai Composite blew through the intervaal identified showing few signs of slowing. Conway and Kochen trump Sornette et al.

Jamming transition in air transportation networks

From Lucas Lacasa, Miguel Cea, Massimiliano Zanin:

In this work we present a model of an air transportation traffic system from the complex network modelling viewpoint. In the network, every node corresponds to a given airport, and two nodes are connected by means of flight routes. Each node is weighted according to its load capacity, and links are weighted according to the Euclidean distance that separates each pair of nodes. Local rules describing the behavior of individual nodes in terms of the surrounding flow have been also modelled, and a random network topology has been chosen in a baseline approach. Numerical simulations describing the diffusion of a given number of agents (aircraft) in this network show the onset of a jamming transition that distinguishes an efficient regime with null amount of airport queues and high diffusivity (free phase) and a regime where bottlenecks suddenly take place, leading to a poor aircraft diffusion (congested phase). Fluctuations are maximal around the congestion threshold, suggesting that the transition is critical. We then proceed by exploring the robustness of our results in neutral random topologies by embedding the model in heterogeneous networks. Specifically, we make use of the European air transportation network formed by 858 airports and 11170 flight routes connecting them, which we show to be scale-free. The jamming transition is also observed in this case. These results and methodologies may introduce relevant decision making procedures in order to optimize the air transportation traffic.

The frequency spectrum of the "brownian" states in the middle obey a power law that has been measured in other systems exhibiting flow congestion. Consider the possibility that other behavior modeled as "brownian" is actually the result of self-organized criticality in flow diffusion. Although it's difficult to visualize, it's not entirely unreasonable to consider "brownian" market price signals as flow-limiting nodes in a network of consumption and production. In other words, market price signals are the rate-limiting step in synchronizing flows of cash and goods within an economy.

Very nice, clearly written paper.

Via physics arXiv blog

Robust dynamic classes revealed by measuring the response function of a social system

From Riley Crane and Didier Sornette:

We find that most videos’ dynamics (≈90%) either do not experience much activity or can be described statistically as a Poisson process (verified using a Chi-Squared test). This large set of data is consistent with the endo-subcritical classification. For the remaining 10% (≈500, 000 videos), we find nontrivial herding behavior that accurately obeys the three power-law relations described above. Characteristic examples of endogenous and exogenous dynamics are shown in Fig. 3.

Via Computational Legal Studies blog.

By what mechanism does traffic transition form Poisson to Power-law statistics? Click on the "syncronization" category at on the right to see some speculations. It seems that spatially dispersed clusters are syncing up. In the exogenous case, that's in response to a driving force -- new information. In the endogenous case, the mechanism is more subtle, and may be an example of self-organization. See the Zanette and Kuperman summary especially.

What is the Systems Theory of Economics?

Systems theory is an attempt to put Hayek, Coase, Williamson, North, and Ellickson on more quantitatively rigorous foundations. Systems theory is an umbrella term for physics models of networks and dynamical systems that are used to understand how interconnected flows can end up in stable or unstable local equilibria, spontaneously synchronize, and generally evolve over time in feedback loops.

Here are some of the basic hypotheses about human behavior that might be associated with systems theory:

• Individuals have preferences for goods that can be measured in units of flow (quantity consumed/produced per unit time).
• Individuals seek to optimize the fit between their flow preferences and available flow using available information about past and present available flow.

These together suggest that individuals can be modeled as a nexus of flows of cash and other goods. In turn, the interconnected streams of these flows lead to additional hypotheses about the behavior of the integrated system of flows.

• Feedback loops in flow among individuals can result in more spatially and temporally stable preferences by synchronization of flow.
• A nexus of synchronized flows may be a firm or a market depending on the structure of interconnections. (I.e., the ability to synchronize flow over time determines what structures will persist.)
• Firms that synchronize with higher-bandwidth can operate at lower cost, sell at higher margin, or both.
• Firms with scale-free architecture are the most cost-efficient, but also the most susceptible to catastrophic failure.
• (Perhaps most surprisingly) Under certain circumstances, individual preferences will spontaneously synchronize through market price signals.

There is an evolutionary mechanism at work here in that the nexus of flows that constitutes a firm is in competition with other nexuses of flows for additional flow. Over time, some network structures will survive better. And the success or failure of a particular network will, of course, have an impact on how the preferences of individuals within it will evolve over time. There are deep connections between systems theory and the evolutionary theory of psychologists like Mihaly Csikszentmihalyi and sociologists like Donald T. Campbell. But the kind of data and scale of the models used by systems theory is more traditionally associated with economics.

Much of systems theory has been anticipated by New Institutional Economics, and in some cases sociology and psychology. The work from Haidt and Ehrenreich on hive psychology and collective joy, for example, is consistent within broad contours of the physics models of self-organized synchronization. Robert H. Frank's work on positional goods can be understood in terms of the sustainability of certain integrated flow structures over long times. In general, Thomas Schelling seems to have had these sorts of models in mind in his writing, although he tended to focus on the potentially destructive outcomes after long periods of repeated interactions.

I'm going to try to turn each of these bullet points into a separate post over the next few weeks or months.

Renormalization Group Approach to Oscillator Synchronization

From Oleg Kogan, Jeffrey L. Rogers, M. C. Cross, and G. Refael:

One of the most exciting aspects of modern physics is the investigation of emergent collective structures in many-body problems. Spontaneous synchronization in networks of interacting nonlinear oscillators with different (often randomly distributed) frequencies is one of the best understood collective phenomena in nonequilibrium systems. . . We propose an alternative method to attack the synchronization problem. The method we develop is a real-space renormalization group (RG) designed to work well on systems with strong randomness, where both the intrinsic oscillation frequencies and coupling constants are random variables taken from distributions with long tails.

Available here.

A Poissonian Explanation for Heavy Tails in Email Communication

That's the title of a recent article by Malmgren, Stouffer, Motter, and Amaral (edited by Strogatz) in the Proceedings of the National Academy of Sciences, which was passed on to me by a friend.

Readers of Broken Symmetry may recall from earlier posts on "the Long Tail" here and here that I was and am skeptical that a power-law applies to the consumption or production of anything, including information. This paper provides the first empirical and theoretical work that I've seen so far that proves that "the long tail" is a consequence of (nonhomogeneous) poisson-distributed activities.

In discussing their results, the authors note:

Our results clearly demonstrate that circadian and weekly cycles, when coupled to cascading activity, can accurately describe the heavy tails observed in e-mail communication patterns. The question then is, would rational decision making, together with circadian and weekly cycles, be equally able to describe the statistical patterns observed for e-mail communication? Even if the answer to this question is affirmative, parsimony suggests that rational decision making is not a necessary component of human activity patterns, given our simpler explanation.

I am skeptical that economists in particular will be willing to discard the rational hypothesis in favor of a hypothesis of periodicity anytime soon. But the authors have clearly demonstrated, in my opinion, that a hypothesis of periodicity might have some instrumental value to economists in forecasting consumption and production based on past observation of the same.

The Long Tail Revisited

In an earlier post, I pointed out that the supposed power law of "the long tail" was actually the same exponential decay that you would expect for any demand function.

Nice to see somebody else noticed this too:

That might now start to change, thanks to the article (online at tinyurl.com/3rg5gp), by Anita Elberse, a marketing professor at Harvard's business school who takes the same statistically rigorous approach to entertainment and cultural industries that sabermetricians do to baseball.

Prof. Elberse looked at data for online video rentals and song purchases, and discovered that the patterns by which people shop online are essentially the same as the ones from offline. Not only do hits and blockbusters remain every bit as important online, but the evidence suggests that the Web is actually causing their role to grow, not shrink.

As an aside, the non-power law shape of the demand function for information consumption is obvious with the periodicity of supply and demand in mind.

Does the long tail result from a power law?

No. Wired's Chris Anderson published an article and later book on the "long tail" that characterizes demand for information.

The hypothesis advanced by Anderson and others is that this long tail results from a power law in demand -- i.e., that the slope of the demand curve declines algebraically rather than exponentially, even at high levels of quantity.  Others have noted how this would be a major coup in economics, since the law of demand from economics predicts a faster than algebraic decay.

Let me suggest how this paradox might be resolved.  In fact, the distribution of consumption for information is not governed by a power law.  Rather, the distribution is poissonian in shape.  The poisson distribution can be approximated with power laws when the mean frequency of events is close to zero.  But unlike the power law distribution, the poisson distribution approaches zero at zero frequency.

The effect of these differences is that the cumulative distribution function -- i.e., the aggregate demand, still has a well-defined peak in quantity.  The fat-head is finite.

This mistake is easy to forgive since people confuse local trends with power laws all the time.  I may blog more at some point about how the notion that a power law applies to demand for information is misleading.

UPDATE (7/6/9): Long tails are always only an approximation to power laws. There are many mechanisms that can explain the emergence of a power law. Some power laws don't really signal anything interesting about the underlying dynamics. But some do seem to.