Random vs. Non-Random Walk

The great debate continues to rage between random walkers and non-random walkers. Two competing books best represent these theories. Originally written by Burton Malkiel in 1973, A Random Walk Down Wall Street has become a classic in investment literature. Malkiel, a Princeton Economist, argues that price movements are largely random and investors cannot outperform the major indices.

Written by Andrew W. Lo and A. Craig MacKinlay in 2001, the appropriately entitled A Non-Random Walk Down Wall Street provides the counter-argument. Lo, an MIT Finance professor and MacKinlay, a Wharton Finance professor, argue that price movements are not all that random and that predictable components do indeed exist. Let the battle begin!

With “random walk”, Malkiel asserts that price movements in securities are unpredictable. Because of this random walk, investors cannot consistently outperform the market as a whole. Applying fundamental analysis or technical analysis to time the market is a waste of time that will simply lead to underperformance. Investors would be better off buying and holding an index fund.

Malkiel offers two popular investment theories that correspond to fundamental analysis and technical analysis. On the fundamental side, the “Firm-Foundation Theory” argues that stocks have an intrinsic value that can be ascertained by discounting future cash flows (earnings). Investors can also use valuation techniques to ascertain the true value of a security or market. Investors decide when to buy or sell based on these valuations.

On the technical side, the “Castle-in-the-Air Theory” assumes that successful investing depends on behavioral finance. Investors must determine the mood of the market - bull or bear. Valuations are not important because a security is only worth what someone is willing to pay for it.

Random walk theory jibes with the semi-strong efficient hypothesis in its assertion that it is impossible to outperform the market on a consistent basis. This theory argues that stock prices are efficient because they reflect all known information (earnings, expectations, dividends). Prices quickly adjust to new information and it is virtually impossible to act on this information. Furthermore, price moves only with the advent of new information and this information is random and unpredictable.

In short, Malkiel attributes any outperformance success to lady luck. If enough people try, some are bound to outperform the market, but most are still likely to underperform.

A Non-Random Walk Down Wall Street is a collection of essays offering empirical evidence that valuable information can be extracted from security prices. Lo and MacKinlay used powerful computers and advanced econometric analysis to test the randomness of security prices. Although this book is a heavy read, the findings should be of interest to technical analysts and chartists. In short, this book documents the presence of predictable components in stock prices.

Just prior to this book, Andrew Lo wrote a paper for the Journal of Finance in 2000: Foundations of Technical Analysis: Computational Algorithms, Statistical Inference, and Empirical Implementation. Harry Mamaysky and Jiang Wang also contributed. The paper's opening remarks say it all:

“Technical analysis, also known as charting, has been part of financial practice for many decades, but this discipline has not received the same level of academic scrutiny and acceptance as more traditional approaches such as fundamental analysis. One of the main obstacles is the highly subjective nature of technical analysis. The presence of geometric shapes in historical price charts is often in the eyes of the beholder. In this paper, we propose a systematic and automatic approach to technical pattern recognition using nonparametric kernel regression and apply this method to a large number of U.S. stocks from 1962 to 1996 to evaluate the effectiveness of technical analysis. By comparing the unconditional empirical distribution of daily stock returns to the conditional distribution conditioned on specific technical indicators, such as head-and-shoulders or double-bottoms, we find that over the 31-year sample period, several technical indicators do provide incremental information and may have some practical value.” This paper can be found at www.nber.org

There is also proof that one of the oldest systems around can outperform the market and reduce risk. Dow Theory seeks to buy when both the Dow Transports and the Dow Industrials record new reaction highs and sell or move into treasuries when both record new reaction lows. The move out of stocks and into treasuries greatly reduces risk because one is not exposed to riskier stocks. There have been a few big bad bear markets over the years and preserving capital is one of the keys to investment success.

Stephen Brown of New York University, William Goetzmann of Yale, and Alok Kumar of the University of Notre Dame published a study on Dow Theory in the Journal of Finance. The Dow theory system was tested against buy-and-hold for the period from 1929 to 1998. Over the 70-year period, the Dow theory system outperformed a buy-and-hold strategy by about 2% per year. In addition, the portfolio carried significantly less risk. If compared using risk-adjusted returns, the margin of outperformance would be even greater. Over the 18 years from 1980 to 1998, the Dow theory system has underperformed the market by about 2.6% per year. However, when adjusted for risk, the Dow theory system significantly outperformed buy-and-hold over this timeframe. Keep in mind that 18 years is not a long time in the history of the market and this period was during one of the greatest bull markets in history (1982 to 2000).

Historic stock returns are not normally distributed. What does this mean? If one were to measure the height of 1000 people and plot the distribution, this distribution would form the classic bell curve. The most recurring height (value) would be in the middle and the remaining heights would be equally distributed on either side. Furthermore, 68.5% of all values would fall within ±1 standard deviation of the mean, 95.4% would fall within ±2 standard deviations and 99.7% would fall within ±3 standard deviations. The solid black line shows a typical bell curve with a normal distribution.

Statisticians have found that a distribution of stock returns forms a curve with “fat tails”. In a normal distribution, 99.7% of all these returns would be within ±3 standard deviations of the mean. This, however, is not the case for stock returns. Instead, the distribution has fat tails, shown in the dotted lines on the graphic above. This means a relatively high number of returns fall outside the normal distribution. Some are lower and some are higher. These abnormal returns provide evidence of extended moves, outsized moves or trends. Note that the image above is just a hypothetical example to illustrate a point.

Anyone who has followed the stock market for any length of time realizes that trends can and will take hold. To be fair, not all stocks trend and trends do not last forever. However, there are enough asset classes, major indices, sectors, industry groups or stocks out there to ensure that something is trending at some point. The challenge, as always, is to find that trend and ride it. The next three charts show some individual stocks with clear signals and trends. Identifying a simple double top and getting out of Citigroup (C) would have avoided a whole lot of pain. The same can be said for Enron, Worldcom and the few other debacles.

ExxonMobil (XOM) was choppy in 2009, down the first half of 2010 and then up sharply from July 2010 to February 2011. Catching this one big trend would have made up for quite a few losses.

Pfizer (PFE) shows an example of three sizable trends emerging over a two year period. The stock was up over 50% in 2009, down about 25% in the first half of 2010 and up around 50% from July 2010 to March 2011.

To be perfectly fair, there are both random and non-random aspects in the financial markets. Stocks sometimes trend and react well to patterns or indicators. Stocks sometimes trade choppy and ignore pattern setups or indicator signals. It is the job of the technical analyst or chartist to separate the wheat from the chaff. Chartists must also be able to adapt to ever-changing conditions.

Andrew Lo notes that beating the market does not come easy, nor is it something that is easy to maintain. Lo likens the pursuit of above-average returns to that of a company trying to maintain its competitive advantage. After introducing a hot new product, a company cannot just sit back and wait for the money to roll in. In order to remain above the competition, management must be flexible and look for ways to continuously improve and innovate; otherwise, the competition will overtake them. Money managers, traders, and investors who find ways to outperform the market must also remain flexible and innovative. Just because a method works today does not mean it will work tomorrow. In an interview with Technical Analysis of Stocks and Commodities, Lo sums it up by stating:

“The more creativity you bring to the investment process, the more rewarding it will be. The only way to maintain ongoing success, however, is to constantly innovate. That's much the same in all endeavors. The only way to continue making money, to continue growing and keeping your profit margins healthy, is to constantly come up with new ideas.”