The interesting question, in my mind, is why so many people are so irrational. This is a gold mine for the behavioral economists, and a nice counter-argument for when someone tells you that better information creates more efficient markets.
I long for a Love-ian explanation of why this happens in theoretically rational markets with nearly perfect price information.
Luis's is a good question. I think there are three unrelated but collaborative factors.
First, active investing is not irrational if most folks believe mutual funds are the superior investment vehicle—remember, rational isn't epistemic. ETFs are a relatively-recent invention—they have been available in Europe since only 1999—and have not yet reached ubiquity. Indeed, many folks do not even know what ETFs are, let alone the returns they offer or low costs they charge. Many 401(k) plans, moreover, continue to offer few if any passive funds. Mutual fund fees, such as front-end loads, create incentives for brokers to push expensive funds over cheaper index funds. Thus, given the relative dearth of information about and access to ETFs, the choice of active funds over passive funds is often rational.
Second, not all of the costs of active investing are "wasted" on unachieved excess returns. Some of the costs go toward tax minimization strategies, for example, that a passive fund does not provide. This portion of the cost should not be included in the calculation.
The best for last: I believe the most accurate explanation is that investors, individually and as a group, are generally making the right choices. Recall I closed my previous post with this nugget:
One reason the market and thus passive investors can earn the returns they do is because of active investor's strategies that close arbitrage opportunities, set prices, tighten spreads, and otherwise make the market more efficient.
This suggests that the overall outcome ($100 billion spent chasing excess returns) is net beneficial, but does not comment on why any individual investor rationally puts his or herself in the active camp.
To answer that, let's study investing at the margins. There is both a cost and a return to active investing. Active investors hope that the return is in excess of the market, net costs. This excess is called alpha. Assume there are no active investors, only passive. Then the market would be a cacophony of noise, and any active investing strategy would reap substantial reward. The cost would also be high, both because there would be few suppliers of such service and because the infrastructure for doing so (equity research, realtime operating systems, the city of Greenwich) would not exist, but the alpha would still be significant. Given these outsized returns, capital will flow out of passive and into active investing.
Now let's look at the other extreme and assume all market participants are engaged in active strategies. The outsized returns will be bid down, but the costs would also be much lower as the supply of funds and their managers meets demand and economies of scale kick in, lowering marginal cost. In this all-active world, the typical return would approximate the market's total return. Given the lack of alpha, capital will flow out of active strategies and into index funds.
In either market, put yourself at the margin. Everyone is passive? Achieving alpha is easy, as you just have to beat the noise. Everyone is active? Then just mimic their strategies by tracking the entire market and achieve similar returns without the cost. With each person switching from passive to active, or from active to passive, the marginal utility, along with the efficiency of the market, increases or decreases. Eventually, equities are efficiently priced as folks long the hot stocks and short the stinkers, spreads tighten, and arbitrage opportunities close. At this point, because the market is efficient, it no longer pays to expend excess cost on active investing. Thus the guy at the margin moves into an index fund, simply tracking the whole market and earning the market's return. If enough people choose passive over active strategies, perhaps the next guy will notice that spreads are too wide or some stock is underpriced. If so, he will choose an active strategy and achieve alpha.
And so it goes, until we reach the $100 billion equilibrium we are at today, where the marginal cost of active investing meets its marginal utility. The balance might be imperfect—too many folks free-riding with their passive investments or tilting at windmills in pursuit of alpha—but I bet its pretty damn close. Disagree? Then just move to the other side of the fence—you will gain excess net returns and make the market more efficient.