Archive for May, 2010

What is that which hath been? it is that which is, and what is that which hath been done? it is that which is done, and there is not an entirely new thing under the sun.

–Ecclesiastes 1:9

Click here to skip the commentary.

One thing I’ve heard from some of the top minds in economics: When modeling financial data, it is critical to use data extending as far back as possible. While occasionally there are good reasons to utilize only a subset of a given series in a particular model, modeling the maximum timespan for which data are available is generally a good idea, especially when analyzing risk. After all, if you don’t look back more than several decades, how can you expect to anticipate a systematic crash that occurs once a century?

Now, nobody is arguing that data from the far-flung past is a panacea. Sometimes it can be downright misleading. There are several things to watch out for when analyzing historical series from far in the past:

1. The quality of data from the distant past varies. Some of it is extremely accurate; some if it is not. In many instances, there is no way to verify its accuracy.
2. In some cases, historians have provided approximations for missing entries in historical data. Usually this is clearly identified, but there is no guarantee. Approximated series may not be appropriate for some types of analysis, and a historian’s choice of approximation scheme may not always match yours.
3. Methodologies used in historic times may vary greatly from modern standards and may even vary unexpectedly within the same dataset. Recently recorded data is often higher quality than data from the distant past.
4. There have been major systematic changes in economic systems. Nations have gone from communist to capitalist, from autarchies to participants in the global economy. Resources have dried up and been discovered. New discoveries have been made, and old inventions have gone out of use. Sometimes, historical data from far in the past simply isn’t relevant in modern times.
5. There have been major systematic changes in financial systems. Metallic standards have come and gone. Central banks have reigned and been dissolved. Financial systems have increased enormously in terms of speed, transparency, and efficiency.
6. There have been major changes in market philosophy. Some of these changes have had an impact on market psychology.
7. There have been major changes in political systems. Wars, treaties, securities regulations–the list goes on.
8. Of course, past performance is not necessarily an indicator of future performance. In particular, when market participants and other entities have the benefit of hindsight, they may act far differently than they did in the past, and anticipating their decisions is not always easy.

Nonetheless, there are striking similarities between modern markets and those of times long ago. Traders still pace the floor of the New York Stock Exchange. Communication systems have improved, but even so, it’s really not that much faster to place a trade by ECN than it was to place one by telephone 30 years ago–or even by telegraph 150 years ago. Sometimes, the revelations held in historical data are surprisingly relevant even in modern times.

• Dr. Robert Schiller (one of the individuals behind the origins of the S&P Case-Schiller Home Price Indices) lists several resources on monthly and quarterly prices of American stocks going as far back as 1871 on his “Online Data” page. He has also listed some historical datasets on real estate prices, bonds, interest, and consumer consumption.
• Wren Research has posted a collection of historical data on U.S. and Australian securities and commodities.
• Yahoo! Finance has made daily price data for the Dow Jones Industrial Average available from since October 1928.
• Among tech stocks, IBM has a long established record of making its financial data in available in electronic format. IBM daily stock price data dating back to 1962 is available on Yahoo! Finance. IBM’s SEC filings in electronic format are available on EDGAR from 1994.
• Compustat offers some finer-grained databases of securities prices than the other sources listed in this post, but it’s not without cost. Their databases may be available for academic use at local libraries.
• MeasuringWorth offers EOD Dow Jones Industrial Average quotes from May 2, 1885 to the very recent past.
• Dr. William Schwert at the University of Rochester offers monthly U.S. stock return data from 1802 to 1925 and daily U.S. stock return data from 1885 to 1962. He would like a small fee for use of his data.
• EH.net has made available a dataset of early U.S. securities prices covering 1790-1860.
• A Yale working-paper by Goetzmann et al, later published in the Journal of Financial Markets, contains tables with some information on stock prices in the NYSE from 1815 to 1925. The complete database can be downloaded for academic use.
• The data published by the Cowles Commission for Research in Economics, “intended to portray the average experience of those investing in this class of security [common stock] in the United States from 1871 to 1938,” is also available from the Yale International Center for Finance historical financial research data sources.
• Proponents of a popular technical analysis technique called candlestick charting claim that the technique originated from the study of 17th century rice prices. 17th century annual rice price data is available from the UC Davis Global Price and Income History Group

more to come