When I investigate a topic for this blog, I usually have an idea of what I expect to find. As I gather data, usually things fall into place more or less as I expected. But sometime, as in today’s post, as I dig deeper, the fundamental conclusion reverses itself. The deeper understanding results in the opposite of what I expected.
Since the 1970s, the best advice on the stock market was to simply stay in the market at all times, through highs and lows. Stock movements were believed to be random and unpredictable. But a technique developed by Nobel-prize winning economist Robert Shiller seemed to show promise in reducing the noise in stock market returns and providing a reliable way to predictably quantify the return on the market over long periods. I originally intended to simply replicate the work published in Nate Silver’s book “The Signal and the Noise”. But after digging into the numbers, I realized that there were real-world factors not considered in Silver’s analysis. After considering those factors, the essential randomness of stock market returns re-emerged. The Shiller PE may provide some guidance about when it is better or worse to be invested in stocks, but the correlation is weak and returns are not well quantified. I am back to the original advice of Burton Malkiel and Peter Lynch, to stay invested in stocks at all times.
The Shiller PE
The inspiration for this post came from a series of charts in Nate Silver’s remarkable book “The Signal and the Noise”. The charts show a remarkable predictive power for the Shiller PE with respect to future stock market performance. The charts captured my attention, because the power of prediction in the stock market is notoriously difficult and immensely valuable. Nate Silver does not permit reproduction of material from his book, so I gathered data from Robert Shiller’s website and created new charts myself.
The PE, or price/earnings ratio, is the fundamental unit of stock valuation. The current price of a stock is divided by a measure of the annual profits, or earnings of the company. The PE of stock shows whether stocks are relatively cheap or expensive. The PE ratio can also be calculated for a basket of stocks, such as the Standard and Poor’s 500 company index, which represents the largest publically traded companies in the market.
The Shiller PE, formally known as the Cyclically Adjusted Price Earnings Ratio (CAPE), divides the price of a stock by the average of the past ten years of earnings, or annualized ten-year trailing earnings. The use of ten years of earnings removes temporary earnings anomalies and produces a better appraisal of value.
Shiller PE versus Average Market Performance
So, if the Shiller PE is a better appraisal of value, let’s see how well it works in predicting future stock market performance. We will look at data provided on Robert Shiller’s website for stock market performance from 1871 to the present.
The following charts show the market performance of the Standard and Poor 500 Index for varying periods of time, versus the Shiller PE. The scale for each chart is the same to facilitate comparison. Dividends are excluded from the calculation of return. Return is calculated once per year, January-to-January for each period, and all figures are adjusted for inflation. Market return for multiple-year periods is annualized by taking the average of the one-year returns over the period. We will see later how changing the way return is annualized significantly changes the results.
Shiller PE versus One-Year Market Performance
We can see that the Shiller PE is almost useless in predicting the short-term performance of the stock market. The graph of one-year performance versus the Shiller PE looks like a shotgun blast.
Shiller PE versus Average Five Year Market Performance
The graph of Shiller PE against average five year performance is somewhat less scattered than the one-year graph.
Shiller PE versus Average Ten Year Market Performance
The graph of average ten year market performance shows a little orderly tendency in comparison to the Shiller PE.
Shiller PE versus Average Twenty Year Market PerformanceThe graph average twenty year market performance shows a clear inverse relationship to Shiller PE. Eureka! The relationship is strong enough that it could be used for accurate quantitative prediction of average market performance over long periods. When stocks are cheap, as indicated by a Shiller PE in the range of 5 to 15, average market returns for the following 20 year are likely to be about 5 percent to 10 percent per year, after adjustment for inflation. When stocks are expensive, as indicated by a Shiller PE of 15 to 25, the average market return is likely to be less than 5 percent for the next twenty years. When the stocks are very expensive, with a Shiller PE greater than 25, the average stock market return for the next twenty years is likely to be near zero. The current Shiller PE is posted daily on Robert Shiller’s website, and is currently 25.9 (March 18, 2016).
Before we look at caveats to this result, let’s consider why the Shiller PE works as a predictor of long-term stock market performance.
Noise Reduction by Repetition
Repetition is the key to the predictive ability of the Shiller PE.
Repetition is a way to improve the quality of the signal in noisy data. Geophysicists use this principle in making seismic recordings for oil exploration. Geophysicists repeat their measurement of the same subsurface point up to 120 times, using different angles of incidence and reflection. The sum of these records is called a “stack”. Random noise in one seismic record is cancelled by countervailing noise in other records, and the resulting data is much stronger and clearer than unstacked data.
There is a lot of noise in both short-term stock market performance and in annual corporate earnings. Stock prices go up and down according to the mood of the market; prices are influenced by national politics, international crises, demographic changes, and changes in the national economy. Trends in the market affect the psychology of investors, so market momentum is a feedback mechanism. This feedback amplifies any upward or downward movement, producing booms and crashes which are not justified by any change in fundamental information.
Earnings are also subject to noise. A company’s earnings can be severely impacted by short-term events, including recessions, competition, market events, labor disruptions, accidents or legal problems.
The measurement of both market performance and earnings are improved with repetition. The Shiller PE, which takes earnings over a ten-year interval, reduces noise through the repetition of those earnings years. The Shiller PE is a “stack” of ten years of earnings data. Some years may be unusually high or low, but the ten year average will be a better indication of the profitability of the company than any single year. Likewise, taking a long-term view of market performance improves the signal in that data. A five-year stack is better than a one-year stack, and a twenty-year stack is better than a ten-year stack. Longer periods of market performance smooth through the highs and lows, and provide a better picture of the return to the investor over the longer term.
Real Market Return Compared to Average Annual Return
So, what is the problem with the correlation of the Shiller PE and average long-term market return?
There are two problems.
“The most powerful force in the universe is compound interest.” -- Albert Einstein
First, investors don’t receive an average of 20 years of annual returns. Investors receive a real return – the cumulative gain over twenty years. If we take the cumulative gain over a number of years and annualize that gain by dividing by the number of years, we get a different answer than if we simply take the average of a number of single year returns. We can annualize the long-term gain by taking the average of the one-year gains, or by taking the total gain and dividing by the number of years. How we annualize the return makes a difference, and the difference is due to compounding.
For example, $1000 invested at a constant rate of 5% for ten years will increase in value to $1628. If we divide the real gain by the number of years, we have an annualized gain of 6.28%, larger than the average annual gain of 5%. If we change one year to a 35% gain, and another year to -30%, the average gain will still be 5%, but the annualized real gain is now less than 4%. In the real world, the annualized real gain may be greater or less than the average annual return.
I compared the average of 20 years of annual returns with real returns for 20 years, over the period 1871 – 1995. Here is the result.
So let’s compare the predictive power of the Shiller PE with the annualized real gain over twenty years.
And for comparison, here again is the Average 20 year return versus the Shiller PE
Real return shows greater scatter than the average annual return. In particular, returns are often higher at the low end of the Shiller PE range, when stocks are cheap. In this correlation, returns are still generally greater when stocks are cheap and smaller when stocks are expensive, but there is little sense that returns can be quantified predictably.
Real Market Return with Saved Dividends
Of course, market performance, or capital gain, is only one component of the return on owning stocks. Dividends are a significant portion of the total return, even though dividends are now small by historical standards. Dividend yield on the S&P 500 Index has averaged 4.4% since 1871; while the current yield on the Index is a little over 2%.
History of the Dividend Yield on the S&P 500 Index, 1871 – 2016
Suppose that we simply save the dividend yield, and recalculate the return on stocks since 1871. Let’s look at the predictive power of the Shiller PE compared to the real return, including saved dividends.
Five Year Real Return with Saved Dividends, versus the Shiller PE
Ten Year Real Return with Saved Dividends, versus the Shiller PE
Twenty Year Real Return with Saved Dividends, versus the Shiller PE
The graph of twenty year real return with saved dividends shows even more scatter than the plot of real market return. Note that there are no negative returns over the twenty-year time interval (although the 20 year performance of recent markets, marked by high Shiller PE values, is not yet known). There is no particular trend to the data, and a linear regression through the data is essentially flat.
Total Return with ReInvested Dividends
Finally, let’s consider the results of an investment program with reinvested dividends.
Reinvested dividends allow market returns and compounding to contribute and increase the contribution of dividend payments. Not surprisingly, the total return from real market return, plus reinvested dividends and compounding exceeds previous calculations of return.
Comparison of Various Methods of Calculating Return over Twenty Year Intervals
The graph of Shiller PE versus total return including reinvested dividends is again without apparent pattern. All values are positive over twenty year intervals within the period covered by the data, but the Shiller PE provides little predictability about whether return will be high or low.
Twenty-Year Total Return, Including Reinvested Dividends, versus Shiller PE, 1881 – 1996
So here is the result I did not expect. The Shiller PE appears to have little power in predicting real-world investment results for total return, including reinvested dividends.
The Shiller PE appears to have a good ability to predict the average one-year market return over long periods. But the parameter proves to be less useful for the thing that investors really want to know: Is this a good time to invest in the market?”
If we test the ability of the Shiller PE to predict the real market return on stocks, or the total return including dividends, or reinvested dividends, the results are essentially random. Within the history of the United States stock market, twenty-year returns, including dividends, have always been positive (to date), after adjusting for inflation. The Shiller PE has very weak ability to predict the real market return, and even worse ability to predict real returns including dividend payments, whether saved or invested.
From the 1970s, the best advice about stocks was to stay in the market, regardless of whether the market was high or low. Burton Malkiel demonstrated this well in his classic book, “A Random Walk Down Wall Street”, and Peter Lynch agreed, with his adage to “Never Time the Market”.
Stock returns were interpreted as essentially random events, and timing the market was judged to be a fools’ errand. Despite the best efforts of Nobel-prize winning economist Robert Shiller to devise an analytical tool which would allow prediction of the stock market, stock market returns remain essentially random. The best advice seems to be to own stocks for the long term, and to avoid trying to time the market.
An Important Caveat
The dataset used in this study was assembled by Robert Shiller, and covers the history of the Standard and Poor’s 500 Index from 1871 to the present. The conclusions which we can draw from this experience are limited to the envelope of experience represented in that history. From 1871 to the present, there has been essentially no war on United States soil. There has been no revolution, no collapse of the banking system, no devastating national health crisis, and no nationwide natural or environmental disaster. The country has been on an upward trajectory of increasing wealth and well-being for its citizens, and steadily increasing economic productivity and technological innovation. We can see in other countries that such ideal conditions do not necessarily hold, and there is no reason to believe that the United States is immune to trouble beyond the range of what we have experienced in the past. Any prudent financial plan must include contingencies for disaster, of either personal or national scope.
Data for this study was taken from Robert Shiller’s website:
Benjamin Graham, 1949, 1986, 2005, The Intelligent Investor, 640p.
Peter Lynch and John Rothchild, 1989, 2000, One Up on Wall Street, 304p.
Peter Lynch and John Rothchild, 1993 Beating the Street, 336p.
Burton Malkiel, 1973, A Random Walk Down Wall Street, 456p.
Robert Shiller, 2000, 2005, 2015, Irrational Exuberance, 392p.
Nate Silver, 2012, The Signal and the Noise, 534p.
Shiller PE, S&P 500 Index, S&P earnings
FRED Graph Observations
Federal Reserve Economic Data
Economic Research Division
Economic Research Division
Federal Reserve Bank of St. Louis
Inflation, Treasury Bond Yields