Deep history of stock and bond returns

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Deep history of stock and bond returns

Post by McQ »

[The title refers to data before 1970; for data after 1970 see https://www.bogleheads.org/wiki/Simba%2 ... preadsheet and its support thread: viewtopic.php?f=10&t=2520]

This post introduces a thread designed to provide Bogleheads with a reference point for finding older historical data on the returns earned on stocks, bonds, and where available, other investment assets. There are a quite a few resources out there now, not terribly well known in the Boglehead community it would appear, based on my cursory review of posts to topics current in mid-2021.

Even if you will never analyze historical data yourself, over time you will encounter a stream of claims about historical asset returns. This thread will help you evaluate those claims by providing a better understanding of the underlying data, especially the warts, blemishes, omissions, and other limitations that would never make it into a popular account.

The first few posts will cover domestic data, the next few international data. This is an open thread; please do not hold back or wait if you have questions or comments or additions.
The planned sequence of posts is:
1. US data 1926 to 1970;
2. US data 1871 to 1926;
3. US data 1790 to 1871;
4. International data 1900 to 1970;
5. International data before 1900
[There will be more than one post per period, to keep the length of each tractable.]

I will note the extent to which the data are easily available, and on occasion, offer an evaluation.

In the best of all worlds, this thread will attract posts by others who will point me to datasets of which I am unaware, as I have a long-term commitment to this topic and would love to discover more. These will be added and credited to the contributor.

Stocks for the Long Run

I will often have occasion to refer to Jeremy Siegel’s work in the book by that name. “Siegel” will be my shorthand for referring to the data series in that book, which he spliced together from multiple sources to get a 200+ year history. It was skepticism about some of his findings that provided the motivation for my own historical investigations.

Public availability

Caveat: for best results in terms of accessing historical returns, you will need an account at a University library, and the more elite the university, the better. Many of the sources I will call out exist behind a paywall. If your library subscribes, no problem; if not, you may be out of luck. Fortunately, there are also free data sources which require only an Internet connection and the ability to pop a download into a spreadsheet.

As to paywalls generally: information may want to be free, but entities who expend labor and capital to compile information and structure it into useful datasets generally want to earn a return on their investment. I am sure any Boglehead who invests in stocks will accept the underlying capitalist logic.

The exception is scholars, who will often append a table containing annual returns to the published paper. An electronic copy of the paper will typically be downloadable at your library; if no library access, don’t despair—it is surprising how often you can find a pdf of the paper at scholar.google.com, often posted by the author. Likewise, scholarly depositories, like SSRN.com or NBER.com, may hold an earlier working paper version, downloadable for free, even if the published paper is locked behind a publisher’s paywall.

Expenses

Most historical indexes ignore expenses. In the Panglossian world of historical compilations, stocks are held directly without an expense ratio paid to a mutual fund or other intermediary, there are never any transaction costs, purchases and sales in any quantity can always be made at the midpoint of the bid-ask spread, and dividends can be reinvested in fractional shares without fee. My gosh, it is as if Robinhood had been founded centuries ago.

Please keep this artificiality in mind when comparing pre-1970 returns to the fully burdened returns of actual mutual funds found in the Simba spreadsheet.

Price return, income return, and total return

Most of the sources reviewed here provide three kinds of stock return: 1) price appreciation, ignoring dividends and other kinds of income; 2) income returns, corresponding largely to realized dividend yields; and 3) total return, with all income reinvested. In not a few cases, only #1 and #3 are provided, so that #2 has to be extracted. Use the formula [(1 + TR%) / (1+ price appreciation %)] - 1 to extract the income return as a percent (it is mathematically incorrect to subtract PR from TR).

Warning: once the narrative moves out of official sources into journalistic accounts or bulletin board discussions, these distinctions among types of return may be obscured or lost. Caveat lector.

Follow up warning, which pertains especially to older data: it is much, much easier to compile old price quotes than to observe dividends and other forms of income. Newspapers have been publishing yesterday’s price quotes pretty much since modern stock trading began in Amsterdam about 1600. But hardly anyone before the 1930s thought to go and find the dividends, the stock splits, the rights issues, the merger premia, etc., and then compute total return using successive price quotes as one input among others.

Accordingly, the practice grew up of estimating the dividend yield when it was not observed, and combining this estimate with the observed price change to report total return. That is what Jeremy Siegel had to do up to about 2007 in reporting total return on US stocks before 1871. But how many readers of Siegel realize that Stocks for the Long Run included an estimated rather than observed component?

To compound the felony: I found in my research that prior to the 1920s, all or almost all the total return received on US stocks came from dividends; in the US sustained price appreciation has been a post-war phenomenon. In that case, a failure to observe dividends implies that total return itself is estimated rather than observed; which rather explodes the idea that the reported return rests on historical data.

Nominal versus real returns

Inflation has been highly variable across centuries and countries. The longer the time frame, and the deeper into the past the series goes, the more necessary it becomes to compute inflation-adjusted returns if the results are to be compared to modern results in a meaningful way. The formula again takes the form of [(1 + TR%)/(1 + inflation %)] -1.

And again, the importance of working with real returns often gets lost as one moves away from the official sources into derivative accounts. If you have ever read that “historically US stocks have returned about 10% per year,” be aware that is a nominal reading, in part a creature of the fact that inflation averaged 3% + / - over the period typically measured.

The historical rate of inflation is among the “other assets” which this guide covers, and sources will be noted in a later post.

How far back do the data go?

One can find something like a stock or equity share well before Amsterdam in 1600; it is only necessary to relax the idea that “stock” means a joint stock company traded on an exchange. William Goetzmann at Yale has a book titled Money Changes Everything for those who want to make the deep dive. Likewise, something like a bond has existed for millennia. Schmelzing gives interest rates since the middle ages (see viewtopic.php?f=10&t=352252), and Homer and Sylla, in History of Interest Rates, give rates back to early Mesopotamia.

But if a stricter definition is applied, then here are some rough termini to keep in mind.
1. World stock return data can be taken back to about 1600;
2. The equity premium (stock / bond return) can be measured back to about 1700 internationally;
3. US stock and bond returns can be calculated back to the early 1790s;
4. Inflation can be computed from before the beginning of US and World stock returns;
5. Gold prices can be computed from before the beginning of US and World stock returns;
6. Foreign exchange (for the US) can be computed since at least the beginning of US stock and bond trading.

Other assets that might appear in the post-1970 Simba spreadsheet have much shorter data spans. I was not aware of an index of the total return on real estate prior to 1970 (but see Robert Shiller’s book Irrational Exuberance for a home price index (residential real estate) that goes back to 1890). However memberlist.php?mode=viewprofile&u=144647 found one (see 3rd post in the thread) which will be discussed under other assets. Likewise, commodity indexes get confounded with price indexes beyond a certain point (the price index is often based on the price of selected commodities).

In terms of types of bonds, government bonds are the oldest, with the modern form dating to the late 1600s in England. Curiously, corporate bonds, in quantities sufficient to create a marketplace, appear to have been pioneered in the US, and to have been a bona fide asset class here long before elsewhere. Good indexes for corporate bond returns are available back to the 1850s in the US; in other countries, corporate bond markets may not appear until after WW II. Next, mortgages are of course ancient; but mortgage-backed bonds in the US, and agency bonds, are a creature of the New Deal.

Last, Treasury bills only date to 1929. Short term loans to governments date back millennia; but here I am defining a Treasury bill in theoretical terms as a suitable proxy for the short-term risk-free rate. To fulfill that goal the bill has to be continuously offered in a quantity that allows arbitrarily large amounts to be rolled over indefinitely. That kind of Treasury bill dates to 1929 in the US.

In the literature you will see historical returns on bills dating back much farther; but these returns rest on quotes from individual bankers and merchants, for arbitrarily small amounts of money, offered irregularly, with quotes not always honored, during periods when the government did not happen to be offering short term notes for sale—which was most of the time.
Hence, fixed income returns prior to the modern period are quintessentially long bond returns. In fact, prior to the 1840s most US government bonds did not have a stated maturity.

Reference

This guide originated in research I did, all of which is posted at SSRN.com. A link to the summary paper can be found at this Bogleheads forum started by Simple Gift: viewtopic.php?f=10&t=352394. In turn, that paper references half a dozen working papers, about 1000 pages in all, for those who really want to get down in the weeds, i.e., how did I find the dividends paid on US stocks before 1871, that had been unavailable to Siegel; or how did I determine that the corporate bond return in the SBBI was incorrect; and much more. The Appendix to the summary paper will point you to the particular working paper(s) that might meet your needs.
Last edited by McQ on Mon Oct 25, 2021 11:13 am, edited 2 times in total.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1972: Where to find

Post by McQ »

Historical STOCK returns in the US 1926 to 1972

There are four major sources: a) the Stocks, Bonds, Bills & Inflation yearbook, pioneered in the 1980s by Roger Ibbotson of Yale, b) the Center for Research into Security Prices, based on data first assembled in the mid-1960s by Fisher and Lorie at the University of Chicago; c) the data library maintained by Ken French of Fama and French fame; and d) Global Financial Data, which includes US as well as international data, assembled by Bryan Taylor. At the end of the post I’ll mention a few additional sources of data that might be of interest to US stock investors.

A. SBBI
If you are reading John Bogle or early Bill Bernstein, and see a reference to stock returns, the SBBI volume, updated annually, is probably the source. Any reference you read in the popular press that begins “since 1926 stocks have …” is also probably sourced here. Volumes are currently published by Duff and Phelps; I believe the monthly data are still available online at Morningstar on one of their professional platforms (Morningstar had acquired Ibbotson & Associates at one point). The fee for the Morningstar data is likely to be steep.

D&P charge about $250 per volume, but any older volume purchased at Amazon or elsewhere will of course contain pre-1972 data. Don’t go too far back: the underlying data for older periods has been updated more than once.

The SBBI tracks two stock indexes: the S&P 500 index, labelled “large company stocks,” and a “small company index,” spliced to various DFA mutual funds after 1981. Both indexes have some complexities often overlooked by newcomers to historical stock research.
1. Prior to 1957 it is the S&P 90 that is tracked. These were mostly large stocks, and the index was capitalization-weighted, but membership was by editorial selection within fixed sector proportions. By the 1950s there were over a thousand stocks on the NYSE alone that were not included. An S&P weekly index which initially had about 200 stocks goes back to 1917 but this is not tracked in the SBBI.
2. The small company index was assembled from the smallest stocks on the NYSE, with the roster updated every five years. Since the NYSE listed mostly the largest and strongest firms in the US, prior to 1972 any reported small cap returns represent returns on the “smallest stocks listed on the dominant exchange that hosted virtually all the largest stocks in the US and applied the strictest listing standards.”

B. CRSP
This is the other likely source for any popular account that begins “since 1926 …” CRSP provides the index tracked by the Vanguard Total Market index fund since 2013. It is also the stock market index used by Jeremy Siegel for the post-1926 period. Most people access it through the Wharton Research Data Services (check your library’s database index for either CRSP or Wharton).

Although this is avowedly a capitalization-weighted total market index, there are still complexities. First, it is NYSE-only until 1962. Accordingly, any “total market return” from 1926 – 1962 excludes AMEX stocks and over-the-counter stocks. From 1963 to 1972 over-the-counter stocks continue to be excluded, until the NASDAQ provided them an “exchange.”

These omissions do not sound terribly problematic until you dig a little deeper. Specifically, banks and financial services companies traded mostly over the counter from well before 1926, and only began to be exchange-listed after the advent of the NASDAQ. Accordingly, no bank failure that occurred in the 1930s (there were several :-) will have affected CRSP stock market returns in the slightest. Likewise, most of the smallest stocks that existed in the US prior to 1962 are omitted from CRSP returns, because they would not have qualified for, or even sought, a listing on the NYSE. Therefore, the impact of the Crash of 1929 and the ensuing Depression on the smallest stocks in the US remains unknown.

Next, the CRSP database contains the individual stock prices themselves with their capitalization and their income returns, with some information on sector. For instance, you could calculate the return on ATT stock and other utilities through the Depression, if desired, and compare it to total NYSE market return.

But you should not attempt that backtest before boning up on the Public Utility Holding Company Act of 1935, since the meaning of “utility stock” was not the same before that law as afterwards. I mention this detail as a reminder that the farther back in time you go, the more necessary it is to broaden your lens beyond stock price and dividend data, to the surrounding economy and polity, if you are truly to understand the returns you are calculating.

C. Ken French data library at https://mba.tuck.dartmouth.edu/pages/fa ... brary.html
Fama and French have CRSP and Compustat produce for them a variety of subsets of CRSP data sorted on different variables. All kinds of factors, and not just size, value and growth, are sliced and diced to varying degrees of granularity and available for free download. Because CRSP is the source, all the limitations just described apply to the indexes calculated by Fama and French as well.

D. Global Financial Data (GFD.com)
GFD is the newest of the repositories, and has the same comprehensive database goals as CRSP, while reporting a variety of indexes, as in the SBBI or the Ken French data library. Data are located behind a paywall, requiring library access, but a free trial is available. Bryan Taylor regularly blogs on the public portion of the site about new compilations and analyses of various historical datasets conducted at GFD.

One important distinction relative to CRSP: Bryan has recently compiled prices and returns on stocks trading on the AMEX before 1962.

E. Other stock data
Mutual funds in the US go back quite a bit farther than 1972, with the starting point generally pegged to the 1920s. A database that includes pre-1972 mutual fund returns, purged of survivorship bias, is maintained at the Wharton Research Data site. However, my mid-size, reputable but not quite elite university’s subscription does not include access to that mutual funds data. FYI.

Professor Damodoran of New York University maintains an active depository of asset returns juxtaposed with a variety of derived measures that go far beyond value, growth, size or momentum factors. http://pages.stern.nyu.edu/~adamodar/Ne ... story.html

Daily price of the Dow-Jones averages, along with a historical sketch, can be found at measuringworth.com. Price appreciation can be computed over any desired interval. If you look hard enough on the Web you can also find various attempts to compile dividends on the DJ components in the early decades, but I don’t know of any authoritative account of these dividends.

There must be other post-1926 sources of US stock data—please advise.
Last edited by McQ on Sun Jul 18, 2021 4:07 pm, edited 1 time in total.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
HootingSloth
Posts: 1050
Joined: Mon Jan 28, 2019 2:38 pm

Re: Historical asset returns before 1972: Where to find

Post by HootingSloth »

McQ wrote: Thu Jul 15, 2021 12:24 pm I am not aware of an index of the total return on real estate prior to 1972 (but see Robert Shiller’s book Irrational Exuberance for a home price index (residential real estate) that goes back to 1890).
Many thanks in advance for what you are doing. This seems like it will be a real gold mine of information.

Are you familiar with The Real Return on Everything, 1870 - 2015 by Jorda et al? (https://economics.harvard.edu/files/eco ... s28533.pdf)

It provides what the paper describes as a total return index (imputed rents plus capital gains) on residential real estate for 16 different countries, dating back to between 1871 and 1948, depending on the country.
Global Market Portfolio + modest tilt towards volatility (80/20->60/40 as approach FI) + modest tilt away from exchange rate risk (80% global+20% U.S. stocks; currency-hedge bonds) + tax optimization
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1972: Where to find

Post by McQ »

HootingSloth wrote: Thu Jul 15, 2021 1:10 pm
McQ wrote: Thu Jul 15, 2021 12:24 pm I am not aware of an index of the total return on real estate prior to 1972 (but see Robert Shiller’s book Irrational Exuberance for a home price index (residential real estate) that goes back to 1890).
Many thanks in advance for what you are doing. This seems like it will be a real gold mine of information.

Are you familiar with The Real Return on Everything, 1870 - 2015 by Jorda et al? (https://economics.harvard.edu/files/eco ... s28533.pdf)

It provides what the paper describes as a total return index (imputed rents plus capital gains) on residential real estate for 16 different countries, dating back to between 1871 and 1948, depending on the country.
Fantastic! I was not aware. I'll add it to the "Other Assets" posts coming in a few days, and correct the initial post, with a credit to you.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
siamond
Posts: 6010
Joined: Mon May 28, 2012 5:50 am

Re: Historical asset returns before 1970: Where to find

Post by siamond »

Fascinating initiative! Thanks a lot to the OP for starting this thread and sharing his clearly very extensive knowledge. A few miscellaneous comments:

- The primary data series in the Simba backtesting spreadsheet start no later than 1970. Many of them start earlier via a mix of models, indices and fund returns. Of course, it is a never-ending work in progress, certainly not perfect and restricted by its public nature. It is also important to know that the spreadsheet can be privately customized to add your own raw asset returns, which is especially handy when dealing with historical data sources which are subject to a strong copyright (SBBI or GFD material, as a case in point).

- SBBI small-caps are essentially micro-caps (it is stated in the book if one looks closely enough). This is a very misleading and not terribly useful dataset. As to bonds, it is a somewhat basic computation based on individual bonds, not bond funds returns. We have a loooong thread focusing on modeling bond fund returns, themselves derived from historical research on yields for various maturities. This model is used to generate the data sets we use in Simba before actual indices & bond funds existed.

- As discussed in many threads on this forum, the Fama-French breakdown in factors is strongly misaligned with the way indices define factors, even the simplest ones like size and value. Simba relies on the stock model developed by Tyler (the PortfolioCharts author) to reconstruct numbers which are much more consistent with the way a real-life index works. See here and here.

- The Real Return on Everything work (Jorda and al) is groundbreaking and available for free use, which is refreshing, but one has to look hard at the underlying data sources to assess consistency and meaningfulness. Let's just say that this is rather new work and there is definite room for improvement, even with some basics (e.g. inflation adjustment math is flawed)... Also, the real estate data is about the housing market (e.g. rent), which is VERY different from investing with a REIT fund. I am not aware of proper ways to compute REIT returns before 1972 (no typo, I mean 1972 in this precise case).
MarkRoulo
Posts: 1198
Joined: Mon Jun 22, 2015 10:25 am

Re: Historical asset returns before 1970: Where to find

Post by MarkRoulo »

McQ wrote: Thu Jul 15, 2021 12:24 pm

In the best of all worlds, this thread will attract posts by others who will point me to datasets of which I am unaware, as I have a long-term commitment to this topic and would love to discover more. These will be added and credited to the contributor.
The book "Triumph of the Optimists" has charts and data going back to 1900 for a number of countries besides the US (it has the US too).
User avatar
nisiprius
Advisory Board
Posts: 52216
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Historical asset returns before 1970: Where to find

Post by nisiprius »

It may not be terribly useful because, while the book itself is available at no cost on line, I don't know of a no-cost source for the data in machine-readable (i.e. spreadsheet) form.

But the source of any data that goes back "to 1871" is virtually certain to be Common-Stock Indexes (1939), by the Cowles Commission, which includes data from 1871 through 1938 for dozens of industries--railroads, rayon, theatres and motion pictures--as well as, of course, "All stocks."

Image

Calculated by a team of "volunteer" students at Colorado College using rotary calculators... and it is strange trying to imagine creating a logo like that without Adobe Illustrator.

For more background see our Wiki article on the Cowles Commission.

By the way it bugs the heck of me whenever of articles (hundreds of them!) talk about the return of "the S&P 500" back to 1926 or 1871, but don't get me started on that.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Apologies to Nisiprius and Siamond and to all for the slow pace of the rollout of posts. US bonds post-1926 will be reviewed tomorrow Friday, Cowles data on Sunday, and international data, Dimson and others, not till later next week.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Northern Flicker
Posts: 15363
Joined: Fri Apr 10, 2015 12:29 am

Re: Historical asset returns before 1970: Where to find

Post by Northern Flicker »

McQ wrote: In the Panglossian world of historical compilations, stocks are held directly without an expense ratio paid to a mutual fund or other intermediary, there are never any transaction costs, purchases and sales in any quantity can always be made at the midpoint of the bid-ask spread, and dividends can be reinvested in fractional shares without fee.
I would add a few more caveats:

1. Bid-ask spreads were much wider, especially for stocks other than large caps.

2. Before WW1, and to a lesser extent for a bit after, the US market was undiversified from a sector diversification perspective. If sector risk was undiversifiable, market participants would expect a risk premium to be discounted in to compensate for it, boosting expected return relative to today when sector risk is diversifiable. A similar, but weaker argument can be made for security-specific risk, which would have been more difficult to diversify away than today due to higher transaction cost.

3. Many protections we take for granted today flow from the Securities Exchange Act of 1934 and the Investment Company Act of 1940.

4. The advent of the Federal Reserve and FDIC/NCUA insurance for deposit accounts also reduces equity risk by reducing economic risk.

Before 1934, US stocks were significantly riskier. There were minimal investor protections, and substantial sector risk and security-specific risk either were undiversifiable or less diversifiable. Investors would expect to be compensated for the higher undiversifiable risk.
Last edited by Northern Flicker on Fri Jul 16, 2021 1:39 pm, edited 2 times in total.
AlohaJoe
Posts: 6609
Joined: Mon Nov 26, 2007 1:00 pm
Location: Saigon, Vietnam

Re: Historical asset returns before 1970: Where to find

Post by AlohaJoe »

nisiprius wrote: Thu Jul 15, 2021 8:03 pm It may not be terribly useful because, while the book itself is available at no cost on line, I don't know of a no-cost source for the data in machine-readable (i.e. spreadsheet) form.
I don't want to suggest it is super-easy but a technically-inclined person can use things like https://camelot-py.readthedocs.io/en/master/index.html to extract tables from PDFs like this.
longinvest
Posts: 5682
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical asset returns before 1970: Where to find

Post by longinvest »

siamond wrote: Thu Jul 15, 2021 5:36 pm As to bonds, it is a somewhat basic computation based on individual bonds, not bond funds returns. We have a loooong thread focusing on modeling bond fund returns, themselves derived from historical research on yields for various maturities. This model is used to generate the data sets we use in Simba before actual indices & bond funds existed.
Here's more information about the Bond Fund Simulator thread. The thread discusses a fund model and provides a bond fund simulator spreadsheet which calculates self-correcting synthetic historical returns based on par yields extracted from FRED/NBER, Yield Curves Book, and FRED DGS(X) going back to 1857, 164 years ago.* The spreadsheet is open and available in 3 formats: Google Sheets (online), Microsoft Excel (download), and LibreOffice Calc (original format used for development). The bond fund model is a ladder-like model where, every 6 months, all coupons and the proceeds of the sold rung are reinvested (cost-free) into a new par bond of the longest maturity of the fund. The latest version of the simulator provides the total return, the weighted-average yield to maturity (YTM), and the weighted-average duration of the fund. It also breaks down the annual total return into its three components: the coupon return, the price return, and the reinvestment return. This last component is due to reinvesting coupons and proceeds mid-year.

* Complete references to data sources are provided within the spreadsheet.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Historical BOND returns in the US 1926-1970

There are three major sources and a separate category of sources, no less authoritative nor any less frequently consulted, but subject to crucial limitations when the goal is to calculate bond returns.

First among the major sources is the SBBI, as discussed in the prior post on US stocks. Next is CRSP, also discussed there, and Global Financial Data, also discussed there. Bond data available from these sources are detailed below.

In the separate category, yields are reported. There are mathematical formulae for deriving bond returns from successive yields in a series, requiring just a few simplifying assumptions; but the devil is in the details, as discussed below.

A. SBBI
The SBBI reports three bond indexes and an index of Treasury bill returns. The bond indexes include: 1) the total return on a long government bond, derived from the bond having the maturity closest to 20 years with the fewest complicating factors; 2) total return on an intermediate government bond, defined as the bond with a maturity closest to but not less than five years and fewest complications; and 3) total return on an index of long corporate bonds, first with an average maturity of 20 years and later 25 years.

The government bond indexes are based on selection of a single bond held until another more suitable bond becomes available. Using one bond and frequently switching bonds introduces some extra volatility; but the two government indexes remain suitable for backtesting returns on some other asset relative to safe government debt of a long or intermediate maturity—excepting the intermediate index before 1934.

In those years there weren’t any or very many five-year bonds (the modern Treasury market, with its copious supply of bonds at every maturity, was still far off in the future). Hence, the SBBI takes a yield curve, computes what the yield would have been on a bond with precisely five years remaining, and then backs out the price change that would have been required to create the observed monthly change in that ever-recalculated yield curve. (A yield curve takes observed yields on the few bonds with scattered maturities that are available, and then interpolates what the yield would have been for unobserved maturities, using assumptions about the shape of the curve.) In short: intermediate Treasury returns are estimated rather than observed prior to 1934.

If you are wondering what “fewest complications” means in the preceding paragraphs, you will have to bone up on how different the Treasury market was from its modern post-1970s form with which we are familiar today. See Appendix A to the paper cited below.

Nonetheless, when I constructed a portfolio of government bonds with maturities GT fifteen years, including those with complications, I got very similar results to the SBBI one bond long bond index—a difference within four basis points annualized for 1926-1974. Correlation was not 100% and volatility differed; but returns averaged out.

I found the SBBI corporate bond index to be problematic and misleading; I constructed an improved index from observation of prices, as reported in https://papers.ssrn.com/sol3/papers.cfm ... id=3740190. The problem: in the first twenty years SBBI returns were extracted from a yield index; through the early 1970s returns were extracted from a different index of AA utility yields. The yield-based estimates proved to be faulty when I gathered a large sample of corporate bond prices.

However, after 1974 the SBBI results agree well with the corresponding Lehman / Barclays / Bloomberg index of Aaa and Aa bonds.

The SBBI Treasury bill index is useful in so far as it reveals that even before today’s depressed interest rates, the investor who held Treasury bills for the long-term often earned zero in real terms, or even lost money. The price of perfect safety in the short term is to accept a long-term return perfectly close to zero.

B. CRSP
CRSP provides government bond prices monthly; it is the source for the bond prices used in the SBBI indexes. CRSP is where to go if you wanted to see how government bond prices at a particular maturity responded to, say, Britain leaving gold in Fall of 1931 (response was not uniform).

C. Global Financial Data
GFD has government bond prices and returns, US and International, generally extending farther back than the 1926 CRSP horizon.

Category #2: compilations of yields, not returns

Data on historical yields far exceed in quantity data on bond returns; and as described above, older return series even within the 1926 horizon are often estimated from yields rather than from observed prices.

Several of the key repositories of yield data are best discussed in upcoming posts regarding data before 1926; please don’t call out omissions below without scanning those posts.

A. Sidney Homer
For decades the fundamental source on yields, domestic and international, was the book History of Interest Rates by Sidney Homer, first published in 1963 with the most recent edition co-authored with Richard Sylla in 2005 (Homer passed away in the early 1980s). Long bonds, short term bonds, Federal, municipal, and corporate bonds, high-grade and low-grade bonds, even mortgage rates and agency bonds, are all tabled.

The discussion that begins on p. 433 of the 2005 edition is essential (and sobering) reading for anyone who wants to splice together bond indexes extending far into the past.

B. National Bureau of Economic Research (NBER)
The data page on this site contains many yield series as comma-delimited files with brief explanatory notes, often but not always marching up to or beyond the 1970 horizon. Here is where you go to get the yields needed to calculate the spread between Treasuries and corporate bonds, or AAA bonds versus Baa bonds, etc. Downloads are free.

C. Federal Reserve
In the early 1940s and again about 1970 the Federal Reserve published bulletins with massive amounts of historical data on yields on almost any fixed income category imaginable. The 1943 and 1970 tomes, about 1000 pages each, can be downloaded as pdf files (I note AlohaJoe’s reminder, a few posts up, of what can be done with pdf files). Selected series can be downloaded as spreadsheet-ready files, extending past 1970 up to the present. Some series are at the central Fed site; others at the St. Louis Fed; see links in the post above by longinvest.

D. Moody’s (now Mergent, an FTSE Russell company)
John Moody pioneered bond ratings, with annual volumes published from 1909. If you see a rating as Aaa, that is the Moody’s form; AAA is the Standard & Poor’s form; likewise Baa versus BBB, etc.
Each Moody’s volume from decades ago is a massive tome, 3000 pages or more at the peak, with multiple volumes required to handle corporate and municipal bonds. Every bond that Moody’s could find gets an entry, which includes the coupon, the rating, the amount issued, the call schedule, issue price, details of the collateral, etc. From the late 1920s the corporate volumes also compile various indexes by bond grade at various degrees of granularity, beyond the simple summary indexes reported at NBER or the Federal Reserve.

Moody’s is where you go to identify bonds that were rated less than investment grade, as for instance, to see how these responded under situations of extreme stress, such as the Great Depression of the 1930s, or the mini-Depression following World War I.

Availability: most volumes up to the expiration of copyright (for years GTE 95 years before the past December) can be found on hathitrust.org and perused for free with individual page downloads. To download the entire volume requires a higher level of membership.

The complete series since 1909 has been digitized by Mergent, with a decent viewing and search function. However, this collection exists behind a five-digit paywall, and you will need to have access to an elite research university library to get at it.

Conversely, you might find a pretty good series of physical volumes at a university library or a big city library. Moody’s has been the authoritative source on bond grades for a long, long time, and individual retirees have been consulting Moody’s about their bond holdings for just as long.

The problem with extracting bond investor returns from an index of yields

The example will be Moody’s Baa index, tracking the yields on the lowest rated bonds that still qualify as investment grade. I will return to the point in later posts on earlier periods.

The first problem is tracking the membership of the index (30 bonds across three sectors). Membership is reported in volumes of the Commercial and Financial Chronicle from about 1932 through the 1940s, and might be in Moody’s newsletters, not typically digitized. Without membership, the average maturity and coupon of the index are unknown. I observed maturities and coupons to differ across sectors and over time in a not particularly rule-based way, corresponding in part to the evolution of the US bond market itself.

Fun factoid: today's convention, in which 30 years is the longest maturity likely to be issued, is a creature of the spread of utility bonds in the 1930s and 1940s, which gravitated to that maturity. When railroad bonds dominated the market, as before WW I, 50-year maturities were more common than 30-year, and the 100-year almost as common.

Next, Moody’s would swap out a bond when it was upgraded (an occasional occurrence), downgraded (more common) or called (very common in the late 1930s). The likely negative return on a downgraded bond may not be captured in the change in index yield. The loss on a called bond (these commonly traded above their call prices in the 1930s) also disappears.

Hence, given an assumed coupon and maturity a bond “return” can always be cranked out of the monthly change in the yield index, but without observation of the bond prices, that return verges on fictional, especially during periods of crisis, when the return on high grade versus low-grade bonds might be of greatest interest.

EDIT (added later in the day)

More on yields versus returns

Generally in these posts I will be hostile toward the practice of deriving bond returns from yields, especially for older data, as will emerge in later posts. That said, it is a perfectly acceptable strategy within the scholarly community, and I believe that Siamond, Longinvest and the other BH who labored on the Bond Simulator are as able practitioners of the art as any (see following siamond post and prior longinvest post for links and discussion). I would go further and say that almost any BH concerned with historical bond yields in recent decades should start with the bond simulator rather than go back to the raw data in FRED and elsewhere.

All should keep in mind that the extraction of returns from yields works best when applied to government bonds under conditions where their indentures are homogeneous, trading is liquid, and large numbers of maturities trade with that liquidity (like, the past few decades in the US). Here what economists call the Law of One Price applies: such a government bond cannot really have a yield reading which is out of whack, hence the return extraction, even based on a single bond or an imputed yield curve, is unlikely to be faulty.

However, I don't believe any set of corporate or municipal bonds has ever satisfied those caveats, the less so the older the yield reading.
Last edited by McQ on Fri Jul 16, 2021 4:54 pm, edited 1 time in total.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
siamond
Posts: 6010
Joined: Mon May 28, 2012 5:50 am

Re: Historical asset returns before 1970: Where to find

Post by siamond »

McQ wrote: Fri Jul 16, 2021 12:04 pm Historical BOND returns in the US 1926-1970
[...]
In the separate category, yields are reported. There are mathematical formulae for deriving bond returns from successive yields in a series, requiring just a few simplifying assumptions; but the devil is in the details, as discussed below.
After studying the Homer's book for a while, it became apparent that going back to the primary sources was in order.

This led us to a very useful input for historical yields: the "Historical U.S Treasury Yield Curves" book (latest edition is 1993), from Coleman, Fisher and Ibbotson. This is a printed book, no digital equivalent, but it is fairly easy to scan and OCR. It was THE reference, collating Fed's bulletin information until CRSP took over. We used such data in conjunction with various NBER and FRED data series in the latest update of the bond fund simulator described above. See this post for more details about such data sources.
User avatar
bobcat2
Posts: 6076
Joined: Tue Feb 20, 2007 2:27 pm
Location: just barely Outside the Beltway

Re: Historical asset returns before 1970: Where to find

Post by bobcat2 »

nisiprius wrote: Thu Jul 15, 2021 8:03 pmFor more background see our Wiki article on the Cowles Commission.
From Bogleheads Wiki entry.
The Cowles Commission for Research in Economics was founded in 1932 by economist Alfred Cowles.
Alfred Cowles III was a very interesting man, but he was not an economist despite what Wikipedia says. He was a newspaper reporter and came from a wealthy family. In the mid 1920's he came down with TB and spent about 10 years of treatment by bed rest in a Colorado Springs sanatorium.

During his stay in Colorado Springs he began investigating the stock market as a thorough and creative amateur. He became particularly interested after his family lost a great deal of money in the stock market crash of '29. Sensing he needed the help of pros, in 1932 he asked a mathematician he knew what professionals he should call on for help. The man suggested contacting the Econometric Society, which had been established just two years earlier. Cowles then wrote to Yale economist Irving Fisher, the president of the Econometric Society. The two immediately hit it off since Fisher has been a friend of Cowles' father when they were both undergrads at Yale, and Fisher was a health fanatic who had himself overcome TB. An agreement was reached that Cowles would fund the fledgling Society by establishing the Cowles Commision for Research in Economics and its new journal Econometrica. In return the Commission would focus much of its early research on the stock market.

BTW Cowles is pronounced 'Coles' not 'Cowells'. A fact I learned the hard way when speaking to a receptionist at the Cowles Foundation now located at Yale. :wink:

BobK
In finance risk is defined as uncertainty that is consequential (nontrivial). | The two main methods of dealing with financial risk are the matching of assets to goals & diversifying.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

bobcat2 wrote: Sat Jul 17, 2021 11:58 am
nisiprius wrote: Thu Jul 15, 2021 8:03 pmFor more background see our Wiki article on the Cowles Commission.
From Bogleheads Wiki entry.
The Cowles Commission for Research in Economics was founded in 1932 by economist Alfred Cowles.
Alfred Cowles III was a very interesting man, but he was not an economist despite what Wikipedia says. He was a newspaper reporter and came from a wealthy family. In the mid 1920's he came down with TB and spent about 10 years of treatment by bed rest in a Colorado Springs sanatorium.

During his stay in Colorado Springs he began investigating the stock market as a thorough and creative amateur. He became particularly interested after his family lost a great deal of money in the stock market crash of '29. Sensing he needed the help of pros, in 1932 he asked a mathematician he knew what professionals he should call on for help. The man suggested contacting the Econometric Society, which had been established just two years earlier. Cowles then wrote to Yale economist Irving Fisher, the president of the Econometric Society. The two immediately hit it off since Fisher has been a friend of Cowles' father when they were both undergrads at Yale, and Fisher was a health fanatic who had himself overcome TB. An agreement was reached that Cowles would fund the fledgling Society by establishing the Cowles Commision for Research in Economics and its new journal Econometrica. In return the Commission would focus much of its early research on the stock market.

BTW Cowles is pronounced 'Coles' not 'Cowells'. A fact I learned the hard way when speaking to a receptionist at the Cowles Foundation now located at Yale. :wink:

BobK
Hello bobcat2: I knew Cowles was not trained as an economist (but certainly didn't know how to pronounce the name correctly!). Also knew some other facts from standard sources but not some of the other facts you noted (e.g., TB, Irving Fisher connection, not an investor but an amateur, journalist). You must have read a biography or retrospective, could you link for us?
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Historical OTHER ASSET returns in the US 1926 -1970

Here the key source is measuringworth.com, which presents both an inflation series and a gold price series. Their data essay on the difficulties of estimating inflation decades or centuries ago provides an outstanding tutorial. You can also find foreign exchange rates relative to the British pound. All free.

If you need a monthly CPI series, it can be downloaded from the Bureau of Labor Statistics site, with a 1913 start date: https://www.bls.gov/cpi/data.htm

Global Financial Data also has a miscellany of information, with which I can’t claim to be very familiar, but which will likely include any data that might be out there about assets other than stocks and bonds.

Next, very recently, per the post by HootingSloth near the beginning of the thread, Jorda et al. have developed an index of the returns on residential housing since 1870, in the US and in fifteen other countries. However, please note Siamond’s subsequent post about how this may not be a good proxy for the return on rental real estate, a la REITs. Also, it is not clear to me as yet whether they accounted for property taxes, so don’t run to cash out your equities to buy a bigger home than you need based on their finding that residential real estate outperformed stocks.

EDIT 7-20-2021:
Hmmm, I’m going to leave the italicized portion up as a warning to myself. Further scrutiny of the Jorda et al. paper shows a comparison with REITs, supportive of their measure of real estate returns, and an explicit attempt to address property taxes. Hence, I withdraw the caveats and recommend that Bogleheads should begin to grapple with their finding that real estate returns have been on a par with equity returns over the long term.

A thousand quibbles remain possible, but here I want to emphasize what it means that the paper survived peer review at a journal like QJE, ranked among the top 5 journals in the discipline. A team of 3-4 rather eminent reviewers would have been sent a draft of the paper. They would have written back with a long list of objections and concerns. The authors would have addressed all of these in writing and submitted an improved draft. There would still be some objections, but perhaps none that were universal across reviewers; the authors would then have addressed remaining objections as best as they could in a third draft; after which, in this case, they would have received a conditional acceptance from the editor, indicating exactly what edits had to be made for the fourth draft to be published. Long story short: their real estate returns need not be 100% correct, but are unlikely to be wrong, in any ordinary meaning of that word, or even flawed to any significant extent.

For the Jorda et al. work (it is bigger than one paper), the HootingSloth post has a link to the full 174 page version, downloadable for free. A condensed version was published in the quite prestigious Journal of Quarterly Economics here: https://academic.oup.com/qje/article/13 ... login=true. An accessible summary was published by one of the coauthors here: https://www.nber.org/reporter/2018numbe ... everything

And best of all, all the data can be downloaded for free here: https://www.macrohistory.net/database/ Jorda et al. cover much more than real estate returns, and I’ll revisit their contribution in subsequent posts.

Do you have other asset series to add? Please describe in a post so I can edit this post to acknowledge your contribution.
Last edited by McQ on Tue Jul 20, 2021 1:10 pm, edited 1 time in total.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
Tyler9000
Posts: 740
Joined: Fri Aug 21, 2015 11:57 am

Re: Historical asset returns before 1970: Where to find

Post by Tyler9000 »

McQ wrote: Thu Jul 15, 2021 12:24 pm This post introduces a thread designed to provide Bogleheads with a reference point for finding older historical data on the returns earned on stocks, bonds, and where available, other investment assets.
Just wanted to chime in and thank you for doing this. Good data is indeed hard to find, and your research is greatly appreciated.
User avatar
siamond
Posts: 6010
Joined: Mon May 28, 2012 5:50 am

Re: Historical asset returns before 1970: Where to find

Post by siamond »

McQ wrote: Sat Jul 17, 2021 12:47 pm Historical OTHER ASSET returns in the US 1926 -1970
[...]
Do you have other asset series to add? Please describe in a post so I can edit this post to acknowledge your contribution.
For historical pre-1970 gold prices, we included this one in Simba: https://onlygold.com/gold-prices/histor ... ld-prices/. This being said, while the US$ was pegged to gold, the trajectory of gold wasn't terribly significant. Since 1970, LBMA is the main reference.

Something which has been frustrating me is that I just can't find a proper DJIA data series. The price-only series starts in 1901, but the TR series I have starts in 1963... Note that Bloomberg claims to have TR numbers before 1963, but they are equal to the price-only series... :shock:

In Simba, we also track a few really old (US) mutual funds, the first three started in the 1930s, the two others in the 50s.

Code: Select all

MFS MA Investors	Pioneer R	Fidelity Fund	T. Rowe Small-Cap	ASA Gold & Metals
MITTX			PIORX		FFIDX		OTCFX			ASA
User avatar
siamond
Posts: 6010
Joined: Mon May 28, 2012 5:50 am

Re: Historical asset returns before 1970: Where to find

Post by siamond »

McQ wrote: Sat Jul 17, 2021 12:47 pm Historical OTHER ASSET returns in the US 1926 -1970
[...]
Do you have other asset series to add? Please describe in a post so I can edit this post to acknowledge your contribution.
For historical pre-1970 gold prices, we included this one in Simba: https://onlygold.com/gold-prices/histor ... ld-prices/. This being said, while the US$ was pegged to gold, the trajectory of gold wasn't terribly significant. Since 1970, LBMA is the main reference.

Something which has been frustrating me is that I just can't find a proper DJIA data series. The price-only series starts in 1901, but the TR series I have only starts in 1963... Note that Bloomberg claims to have TR numbers before 1963, but they are identical to the price-only series... :shock:

In Simba, we also track a few really old (US) mutual funds, the first three started in the 1930s, the last two in the 50s. Definitely not passive, but still of historical interest imho.

Code: Select all

MFS MA Investors	Pioneer R	Fidelity Fund	T. Rowe Small-Cap	ASA Gold & Metals
MITTX			PIORX		FFIDX		OTCFX			ASA
User avatar
bobcat2
Posts: 6076
Joined: Tue Feb 20, 2007 2:27 pm
Location: just barely Outside the Beltway

Re: Historical asset returns before 1970: Where to find

Post by bobcat2 »

McQ wrote: Sat Jul 17, 2021 12:40 pm
bobcat2 wrote: Sat Jul 17, 2021 11:58 am
nisiprius wrote: Thu Jul 15, 2021 8:03 pmFor more background see our Wiki article on the Cowles Commission.
From Bogleheads Wiki entry.
The Cowles Commission for Research in Economics was founded in 1932 by economist Alfred Cowles.
Alfred Cowles III was a very interesting man, but he was not an economist despite what Wikipedia says. He was a newspaper reporter and came from a wealthy family. In the mid 1920's he came down with TB and spent about 10 years of treatment by bed rest in a Colorado Springs sanatorium.

During his stay in Colorado Springs he began investigating the stock market as a thorough and creative amateur. He became particularly interested after his family lost a great deal of money in the stock market crash of '29. Sensing he needed the help of pros, in 1932 he asked a mathematician he knew what professionals he should call on for help. The man suggested contacting the Econometric Society, which had been established just two years earlier. Cowles then wrote to Yale economist Irving Fisher, the president of the Econometric Society. The two immediately hit it off since Fisher has been a friend of Cowles' father when they were both undergrads at Yale, and Fisher was a health fanatic who had himself overcome TB. An agreement was reached that Cowles would fund the fledgling Society by establishing the Cowles Commision for Research in Economics and its new journal Econometrica. In return the Commission would focus much of its early research on the stock market.

BTW Cowles is pronounced 'Coles' not 'Cowells'. A fact I learned the hard way when speaking to a receptionist at the Cowles Foundation now located at Yale. :wink:

BobK
Hello bobcat2: I knew Cowles was not trained as an economist (but certainly didn't know how to pronounce the name correctly!). Also knew some other facts from standard sources but not some of the other facts you noted (e.g., TB, Irving Fisher connection, not an investor but an amateur, journalist). You must have read a biography or retrospective, could you link for us?
Most of my information comes from Peter Bernstein's book, Capital Ideas, which is an excellent overview of how financial economics developed.

There is also information about Cowles and the history of the Cowles Foundation at the Cowles Foundation site.
Here are two links from the Foundation -
Alfred C. Cowles 3rd - https://cowles.yale.edu/archives/people/cowles
History of Cowles Foudation - https://cowles.yale.edu/sites/default/f ... cowles.pdf

BobK
In finance risk is defined as uncertainty that is consequential (nontrivial). | The two main methods of dealing with financial risk are the matching of assets to goals & diversifying.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

siamond wrote: Sat Jul 17, 2021 1:26 pm
McQ wrote: Sat Jul 17, 2021 12:47 pm Historical OTHER ASSET returns in the US 1926 -1970
[...]
Do you have other asset series to add? Please describe in a post so I can edit this post to acknowledge your contribution.
Something which has been frustrating me is that I just can't find a proper DJIA data series. The price-only series starts in 1901, but the TR series I have starts in 1963... Note that Bloomberg claims to have TR numbers before 1963, but they are equal to the price-only series... :shock:
There is a reason why your search has been unsuccessful: no official, contemporaneous source tracked the dividends on the components of any DJ average. You can get the list of changing components in annuals like Pierce 1991, The Dow Jones Averages: https://www.google.com/books/edition/Th ... =en&gbpv=0 .

And again, there are amateur efforts out there to supply the dividends, but no official source on the order of Bloomberg or S&P. Given what I encountered in terms of the prevalence of rights issues, fractional stock dividends, assessments, and more in that era, I am not inclined to trust these efforts.

A few other tidbits: although the DJIA dates to 1896, predecessor Dow Jones averages go back to 1885. The 1896 start reflects the fact that before the 1890s, there weren't a dozen industrial stocks to be found on the NYSE; even then, for the first few years, the compiler had to pop in a couple of preferred shares to fill out the index. Railroads dominated.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

siamond wrote: Sat Jul 17, 2021 1:28 pm
McQ wrote: Sat Jul 17, 2021 12:47 pm Historical OTHER ASSET returns in the US 1926 -1970
[...]
Do you have other asset series to add? Please describe in a post so I can edit this post to acknowledge your contribution.
In Simba, we also track a few really old (US) mutual funds, the first three started in the 1930s, the last two in the 50s. Definitely not passive, but still of historical interest imho.

Code: Select all

MFS MA Investors	Pioneer R	Fidelity Fund	T. Rowe Small-Cap	ASA Gold & Metals
MITTX			PIORX		FFIDX		OTCFX			ASA
Regarding older mutual funds, some years ago on a Vanguard subsite I was able to find returns on Wellington back to its origins in the 1920s. But I wasn't able to find it again when I looked awhile back. FYI, it is there somewhere.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
siamond
Posts: 6010
Joined: Mon May 28, 2012 5:50 am

Re: Historical asset returns before 1970: Where to find

Post by siamond »

McQ wrote: Sat Jul 17, 2021 6:07 pmRegarding older mutual funds, some years ago on a Vanguard subsite I was able to find returns on Wellington back to its origins in the 1920s. But I wasn't able to find it again when I looked awhile back. FYI, it is there somewhere.
It's in Simba too, forgot to mention it. Starting from its first full year, 1930. VWENX. Don't know if the historical dividends are reliable, but it's a TR series, available on Morningstar.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Historical STOCK returns in the US 1871 to 1926

By this point you may be wondering, “why is 1926 a hinge date? Did some watershed event occur in US financial markets at that juncture?” The answer is no. Virtually nothing came into existence in 1926 that was not already present in 1925, or 1924, or 1923 … Plus, the same data sources that Fisher and Lorie used to launch the CRSP database were also available for 1925, 1924 …

The short answer: Fisher and Lorie, who had been working back from 1962 to build the CRSP database, were about to run out of money and time on their grant. They made lemonade out of this sour reality by noting that a 1926 start date avoided having to start their stock returns at the depths of the Depression (which would have biased subsequent returns upwards), or at the peak in 1929 (which would have biased subsequent returns downwards). But the fundamental fact remains that they had to wind up data collection well before exhausting the historical record, and this defense of 1926 as a start date is a post-hoc rationalization.

Perhaps more important, at the outset Fisher and Lorie were not aware of the work of Alfred Cowles, which will be featured in this post. Nor do they appear to have been aware that Standard & Poor’s, formed in 1941 from a merger of the Standard Statistics corporation with Poor’s Publishing, had compiled a near-total NYSE market index from the end of 1917. (However, it was a weekly index.)

That is how 1926 got enshrined as The Beginning of Stock Market Time. It is not, as explained in the remainder of this post.
For this period there are four major sources, two of which extensively overlap and will be discussed together.

A. Cowles / Shiller
In the mid-1930s Alfred Cowles, who was neither an academic nor a money manager, but later an editor of Econometrica, a very technical scholarly journal, set out to collect stock return data back to the very beginning, which he deemed to be 1871 (more on that choice below). From the outset he intended to distinguish sectors, beginning with railroads, public utilities, and industrials, later differentiated into dozens of sub-sectors, each with their own index.

Also from the very beginning, Cowles set out to determine the capitalization-weighted, total return that would have been achieved by an investor who chose to own the entire market, defined again as NYSE-listed stocks. In the 1930s the very idea of a market index was still a new concept, and the proper weighting scheme was still a matter of debate; Cowles’ choice of capitalization-weighting had to be defended, and he discusses it at length. Cowles also understood that price quotes plus an assumed dividend yield were not going to be enough to capture total return, and published formulae for adjusting observed price quotes for unobserved ex-dividend dates, rights, stock dividends, and more.

Fun factoid: as late as the 1960s, Fisher and Lorie were able to advocate equal weighting as the best scheme for a market index. Bogleheads who are committed to capitalization-weighting as the One True Way of indexing might enjoy reading their 1964 Journal of Business paper: https://www.jstor.org/stable/2351197

Returning to Cowles, he published his indexes, along with a state-of-the-art critique of indexing methodologies, in 1938; revised it in 1939; and later published updates to 1940. He then turned to other concerns.

Twenty years later, when Fisher and Lorie began their effort, Cowles’ quite massive project had sunk out of sight. Only in the early 1980s, when Robert Shiller recovered Cowles’s work, and began to publish analyses of stock returns since 1871, did Cowles’ contribution come back into view.
Accordingly, whenever you read in a popular account that “since 1871 stocks have …” the source almost has to be Cowles, probably transmitted through Shiller.

For more background on Cowles, see the post by bobcat2 a little bit upthread. If you want a perspective on Cowles’ data and how it compares to other datasets, such as the SBBI and CRSP, see the 2002 paper by Wilson and Jones: https://www.jstor.org/stable/10.1086/33 ... b_contents

Availability and access

Robert Shiller’s web site contains various Cowles indexes that can be downloaded into a spreadsheet: http://www.econ.yale.edu/~shiller/data.htm Elsewhere at Yale the text of Cowles’ book and his tables of returns have also been placed online, e.g., https://cowles.yale.edu/sites/default/f ... -2-all.pdf. With a little copying and pasting you can convert the scanned tables into data, so this is the place to go if you need to look at a broader selection of Cowles’ indexes beyond the summary indexes on Shiller’s web site.
The introductory text of Cowles’ book is well worth the read if you intend to do work in this era. In it he acknowledges that stock price data could have been compiled back to at least 1860, but argues that there were too few non-railroad stocks before 1871 to allow a multi-sector index. When I later went to reconstruct and reexamine Cowles’ index, I had no difficulty pushing the non-railroad entries back to 1866 (see my paper, URL below, especially the Appendices, which unpack Cowles’ dependence on Macaulay, who will be featured in subsequent posts). But I had no need to construct a three-sector index, as Cowles desired. My paper: https://papers.ssrn.com/sol3/papers.cfm ... id=3564496

Cowles also provides an annotated bibliography of other attempts to construct stock indexes for some portion of the 1871 to 1926 period; I won’t be calling these out separately in these posts, as none can hold a candle to Cowles’ effort.

Last, when I undertook to re-examine Cowles, I was reasonably confident that I would find significant survivorship bias, and the kind of over-estimate of returns that I had found in Siegel’s pre-1871 sources (see subsequent posts). However, that proved not to be the case for 1871 to 1897. Despite the survivorship bias that did prove to be present, I found Cowles’ annualized return estimate to be high by only two dozen basis points or so, and even to get there required introducing stocks from outside the NYSE, see below. So I abandoned any further attempt to update later Cowles data, and believe the 1897 to 1926 portion of Cowles’ data series to be by and large an accurate estimate of market returns—on the largest NYSE stocks.

As I reflected on my failed effort to expose significant shortcomings in Cowles’ data, I realized that I had forgotten the statistical power of the mean as sample size grows larger. More particularly, I was concerned with a weighted mean, the capitalization-weighted mean return of the stocks on the NYSE 1871 to 1897, including the ones that Macaulay or Cowles had excluded. When a few weights are disproportionately large, as was true of the stock market then as now, “larger sample” translates to “as more and more of the weights have been captured.” Cowles already had the New York Central RR, the Pennsylvania RR, and virtually every other large railroad; later he would have US Steel and other dominant stocks of the day. In short: even though I found a couple dozen omitted railroads, which were the dogs of their day in terms of performance, their capitalization didn’t amount to enough to budge the needle much from where Cowles had set it.

B. Commercial and Financial Chronicle
This publication, launched in 1865, was the Wall Street Journal, Barron’s and Bloomberg of its day. It provided the price quotes that Cowles used and also the information on dividends, rights, etc. It also provided the quotes that Fisher and Lorie used for the first few years of their database.
Although not many Bogleheads will want to dive into the 19th century and begin to build their own stock index from scratch, this publication is where you would start.

I mention it because a near-complete digitization of its weekly issues from 1865 to 1963 has been posted at the Federal Reserve of St Louis web site, along with a short history of the publication: https://fraser.stlouisfed.org/title/com ... owse=1860s and https://fraser.stlouisfed.org/blog/2017 ... -1865-1928. Many but not all volumes up through the mid-1920s are also on hathitrust.org, where a somewhat more tractable search engine is available.

BTW, by the 1920s the monthly sub-publication called the Quotation and Bank Record carried the price quotes—search for that item, in addition to the CFC, on the Federal Reserve site.

C. Global Financial Data
GFD has expanded price quote collection well beyond the NYSE, in particular, to over-the-counter stocks, which before 1900 included Standard Oil and the major banks among others. Like CRSP post-1926, the GFD database also contains individual stock data pre-1926.

D. Other sources
Dow Jones prices can be obtained from 1885 from measuringworth.com.

By the 1880s the New York Times carried a significant fraction of the same price quotes as the CFC, with coverage expanding as the decades rolled on. And the digital version of the NY Times is very widely held among libraries.

Next, Joseph G. Martin, a Boston stockbroker, compiled prices and dividends for a range of stocks traded in Boston during this era. Peter Rousseau used the Martin data to compile return indexes for banks and manufacturing companies listed in Boston through 1897. Martin’s 1898 book a Century of Finance can be downloaded from books.google.com, and Rousseau’s paper is available at NBER.com: https://www.nber.org/papers/h0117

During this era most of the stocks that traded on the NYSE, and virtually all large stocks, were railroads. From 1868 Poor’s Manual of the Railroads gives copious information on dividends, capitalization, business operations, etc. What isn’t in the CFC or Martin is here in Poor’s; this is where you would go if you wanted to tease out value, growth or size factors prior to 1900. After 1900 railroads became less dominant, and the Moody’s manuals gradually supplanted Poor’s, even though Poor’s branched out to cover industrials and utilities. Although Moody’s connotes bonds to most contemporary investors, the early manuals covered stocks as well as bonds.

Most of the Poor’s manuals, and most of the Moody’s manuals prior to 1925, can be found on hathitrust.org

Last, the otherwise excellent paper by Jorda et al., which will reappear later in these posts, contains no new data for the US after 1871, relying on Shiller / Cowles.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
siamond
Posts: 6010
Joined: Mon May 28, 2012 5:50 am

Re: Historical asset returns before 1970: Where to find

Post by siamond »

^^^^^^
This is a fascinating post, McQ. Call me very impressed by the research you did.
User avatar
bobcat2
Posts: 6076
Joined: Tue Feb 20, 2007 2:27 pm
Location: just barely Outside the Beltway

Re: Historical asset returns before 1970: Where to find

Post by bobcat2 »

Like siamond I too am impressed by the research you have done.

I was particularly impressed by your referencing the 2002 paper by Jack Wilson and Charles Jones which, although not well known, slightly reconstructed the Cowles's data using as a base Cowles's own updated and corrected series.

The Wilson and Jones data are what are used in Triumph of the Optimists for US Stock Market returns from 1900-1925. In other words, what 'Triumph' used before the CRSP series begins in 1926.

In what follows Wilson & Jones briefly describe their research.
The S&P 500 Index price and return data have often been misrepresented, and confusion still continues about their historical coverage. Furthermore, the extensions back in time using Cowles data that have been published in numerous sources have been confusing, based on averaged monthly data rather than month-end prices and often failing to in-corporate Cowles’s own corrections to his data. ...

Cowles made an important contribution to the study of returns over long periods of time by extending the original S&P data backward in time to the beginning of 1871. In subsequent reports Cowles corrected errors in his data, which have not been picked up by most sources reporting this historical series. For example, although Standard & Poor’s reports their historical series based on Cowles’s data, S&P has consistently failed to use the subsequently corrected Cowles's data. The uncorrected S&P series, in turn, has been used by Shiller and by Siegel, as well as others, in studying stock returns over long periods.

We have reconstructed the Cowles data based on his updated and corrected series. We have also attempted to reconstruct the data to deal with time-averaging biases, going beyond Schwert’s attempt to do so. Averaged data have a downward-biased variance and built-in first-order autocorrelation. Extensive analysis and adjustment were required to produce a series that avoids the averaging problem back to February 1885.
BobK
In finance risk is defined as uncertainty that is consequential (nontrivial). | The two main methods of dealing with financial risk are the matching of assets to goals & diversifying.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

[You are both too kind]
One of the most interesting parts of the Wilson & Jones work, to me, was their comparison of Cowles' index to the "large company" index in the SBBI, which too many people insist on calling the S&P 500, to the irritation of Nisiprius and others. As noted upthread, before 1957 the S&P index tabled in the SBBI was the S&P 90.
Cowles had a couple hundred stocks during the period of overlap (1926 to 1940). With most of Cowles' incremental coverage consisting of smaller stocks, theory says that Cowles' index, as a better approximation of the total market, should have outperformed the S&P 90, which contained most of and mostly the largest stocks (=size effect). But Wilson and Jones showed the reverse; and further showed that the CRSP index, an even better proxy for the total market, with hundreds more small stocks than even Cowles, underperformed Cowles, again contrary to the size effect.
Bottom line: prior to 1957, the stock returns in the SBBI, the bible of a generation of historically minded investors, are probably over-stated, benefitting from either a reverse size effect, or a Quality factor tilt. Likewise, Bogleheads need to be very cautious of the size effect; at least, read the paper by Asness et al.: "Size matters if you control your junk" https://www.sciencedirect.com/science/a ... 5X18301326
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Historical US BOND returns 1871 to 1926

Beginning with this post in the series, critical evaluation will begin to supplant neutral pointers to where resources can be found. Put another way, before I came on the scene, with one obscure and long-forgotten exception, no one had ever directly gathered bond returns prior to 1926; they simply consulted the standard sources for bond yields and mathematically extracted an estimated return using simplified assumptions and a standard formula.

That’s correct: Siegel’s assertion that stocks have beaten and must always beat bonds, before 1926 as well as after, did not rest on any observation of bond prices before 1926. None.

To begin, let’s review the major sources for historical bond yields for all or part of the 1871 to 1926 period.
1. Frederick Macaulay, working in the 1930s, computed a yield index for high-grade corporate bonds from 1857 to 1937. This index appears in Homer and Sylla’s History of Interest Rates. Macaulay collected the underlying bond prices himself for the entire period, but after 1879 only reports yields, at the individual bond level and in the form of indexes.
2. For comparison, Macaulay also constructed an index of New England municipal bond yields from 1857 to January 1914 (he dropped this series after the advent of the income tax). He got the municipal bond prices from Joseph G. Martin (previous post), but again, only reports yields, and only on the index, not the individual bonds.
3. David Durand, working in the 1940s, developed an index of yields on prime 30-year corporate bonds from 1900 through 1926 and beyond; this index also appears in Homer and Sylla, converted into a monthly rather than an annual index by an adjustment of Homer’s. Durand used data compiled by W. Braddock Hickman (his books are also at NBER.org). But Hickman’s data are no longer available.

Macaulay’s book, with extensive tables, can be downloaded from NBER.org; likewise, comma delimited files with his index values can be found there. Durand’s initial papers can also be found on NBER.org. The NBER data library also contains other files built from minor indexes (e.g., the S&P index of industrial bonds from 1900).

Now, to the elephant in the room: where are the yields on Treasury bonds, which traded throughout this period? Technically, an equity premium cannot be calculated unless there is a government bond, free of the risk of default, available to compare to stocks.

Answer: beginning with Macaulay, and ratified by Homer, Durand, and other authorities, peculiar conditions prevailing in the Treasury market from 1865 through 1918 made these instruments unsuitable for determining the cost of money, or for estimating fixed income returns. Macaulay (1938, p. 74) puts it this way:
The reader, especially if he is not an American, may wonder why, if we were primarily interested in the highest-grade bonds, we did not use United States government bonds …during most of the period covered by this study, their yields were seriously affected by their circulation privileges. The bonds were intimately tied up with the whole structure of the national banking system. American ‘National Banks’ were allowed to issue ‘National Bank Notes’ based on United States government bonds they had deposited … Consequently the bonds were bought for two reasons: first because of the interest they paid; second because they could be used as collateral for the issuance of currency. The yields were naturally much lower than if the bonds had been valued for their interest payments alone.”

Nonetheless, if you want to go your own way and examine Federal bond yields, Robert Shiller’s web site has an estimate of ten-year Treasury yields in the spreadsheet that holds the Cowles stock returns. History of Interest Rates also has tabled yields for selected Federal bonds. More recently, Hall, Payne and Sargent compiled a massive database of all Federal bonds from the 1790s (and before) through 1926, with terms, amount outstanding by year, and prices. Spreadsheets can be downloaded from github.com: https://people.brandeis.edu/~ghall/pape ... Me_Pub.pdf

Although superseded by the Hall et al. database, the Federal Reserve publications mentioned in the previous BONDS post also carry an index of the yields of long government bonds from January 1919.

Last, if you have gotten back this far in the history, please do take a look at the long data essay on the vicissitudes of determining historical interest rates at measuringworth.com

Critique

Siegel, accepting the authorities that disparaged Treasury bonds in the period, but still requiring a government bond index for his purposes, substituted Macaulay’s New England Municipal index, translating those yields into annual returns using standard assumptions. That is the source of the bond returns in Siegel’s 200-year history for the 1865 to 1920 period. After 1918 there were Liberty bonds from WW I, which although they still had complications, such as partial exemption from income tax, were not terribly unsuitable for measuring fixed income returns on long bonds of the highest quality, and Siegel makes the switch.

What Siegel did not understand, and in fact, what no one seems to have understood, is how Macaulay obtained that New England yield index. First, he got bond prices and terms from Martin. That explains “New England:” by and large, municipal bonds did not trade on the NYSE; nor, prior to 1900, were there very many state or city bonds outside the older Northeastern states. But those state and city bonds that did exist did trade in Boston, albeit off exchange, and Martin would publish circulars showing actual or estimated quotes for a range of issues.

Next, Macaulay did not compute the average yield on the fifteen +/- bonds in the municipal index at any one time (a fact I discovered when I did compute that average from Martin’s prices). Rather, he eyeballed the dispersion in yields, made a subjective judgment about issuer quality, and then set the yield on the index to near the bottom of the yield dispersion for the highest quality issuers. It is these “lowest of the low” yields that Siegel translated into returns.

I don’t believe Siegel understood what Macaulay had done; and Macaulay himself may not have understood why New England yields were as low as they were in the 1870s and 1880s (the same contaminating factors that make Treasuries unsuitable applied there, details in the paper below).
Siegel also did not inquire into Macaulay’s purpose in constructing the New England index. In my judgment based on his text, for Macaulay it was a throwaway foil, a gut check on the main act, which was his railroad bond index. Few contemporary investors understand that prior to WW I, the dominant bond market in the United States was the corporate bond market, i.e., the railroad market, which dwarfed the municipal market in size and the Federal market in trading volume.

But then again, Macaulay’s published railroad yield indexes are not based on portfolio averages either. For these he developed a mathematical procedure that took the dispersion of yields, made some assumptions about issuer quality, and then extracted an estimated yield based on the very highest quality bonds in his set that year (which had already been selected to be the cream of the crop).

When I collected actual bond prices after 1897, I found that both Macaulay and Durand had computed yields that were lower than even the average yield of Aaa bonds as rated by Moody’s.

And now you understand a little better why Siegel can claim that stocks have always beaten bonds--Siegel's bond returns for this period were cranked down way below what a fixed income investor of the era would actually have received from a broad aggregate index of long bonds.

The obscure and forgotten source for bond returns after 1871

Cowles, in a subsidiary table available from the Yale website, obtained the bond prices from Macaulay and provided the information needed to compute annual total return on the Macaulay portfolio (the phrasing is deliberate—Cowles calculated a different quantity, but all the raw materials needed for total return are there). Snowden (1990), unaware of that Cowles spreadsheet, also extracted returns on the Macaulay portfolio from his yield tables using the standard formulae.

To get an index of the total return on bonds from 1871, I assembled an aggregate index from Cowles’ corporate returns, my calculation of returns on Federal bonds through 1887, and my calculation of municipal returns, not confined to New England, but using Martin’s data, through 1897. A gold adjustment has to be made to some bonds from 1862 to 1879; this is described in a later post on pre-1871 bond returns.

After 1897 I collect corporate bond prices for large, long (15 years +), NYSE-listed issues.

Much more than you ever wanted to know about post Civil War bond returns can be found in these two papers:
From 1857: https://papers.ssrn.com/sol3/papers.cfm ... id=3269683
From 1897: https://papers.ssrn.com/sol3/papers.cfm ... id=3633277

A link to my spreadsheet containing annual bond returns can be found about a dozen posts in to the forum that SimpleGift started on the summary paper: viewtopic.php?f=10&t=352394&sid=c91f292 ... 4284fa16b5
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
LadyGeek
Site Admin
Posts: 95696
Joined: Sat Dec 20, 2008 4:34 pm
Location: Philadelphia
Contact:

Re: Historical asset returns before 1970: Where to find

Post by LadyGeek »

This thread is now being tracked in the wiki. See: Historical and expected returns

(Another editor entered the initial info. I'm just documenting the entry here.)
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

McQ wrote: Mon Jul 19, 2021 1:27 pm Nonetheless, if you want to go your own way and examine Federal bond yields, Robert Shiller’s web site has an estimate of ten-year Treasury yields in the spreadsheet...
Thank you, Professor McQuarrie, for this ongoing master class in historical U.S. asset returns. One peculiar observation that arises when looking at Shiller's bond yield data since 1871 is that the yield curve appears to be consistently inverted (negatively sloped) from 1871 to about 1930 (chart below) — when it then switches over to the mostly positively-sloped yield curve we've become accustomed to in modern times.
  • Image
    Data source: Shiller annual series
Was this simply due to persistent expectations of deflation during the 1871-1930 period (where current cash flows would be less valuable than future cash flows), and then to the consistent expectations for future inflation that we've seen since about 1930? Or was something else involved here?

Also, did this difference between lower long-term bond yields and higher short-term interest rates prior to 1930 impact your choice of which bonds to include in your historical data series? It would seem that a negatively-sloped yield curve might further confound the comparison of 19th century bond returns with those of the modern era.
HootingSloth
Posts: 1050
Joined: Mon Jan 28, 2019 2:38 pm

Re: Historical asset returns before 1970: Where to find

Post by HootingSloth »

SimpleGift wrote: Tue Jul 20, 2021 1:43 am
McQ wrote: Mon Jul 19, 2021 1:27 pm Nonetheless, if you want to go your own way and examine Federal bond yields, Robert Shiller’s web site has an estimate of ten-year Treasury yields in the spreadsheet...
Thank you, Professor McQuarrie, for this ongoing master class in historical U.S. asset returns. One peculiar observation that arises when looking at Shiller's bond yield data since 1871 is that the yield curve appears to be consistently inverted (negatively sloped) from 1871 to about 1930 (chart below) — when it then switches over to the mostly positively-sloped yield curve we've become accustomed to in modern times.
  • Image
    Data source: Shiller annual series
Was this simply due to persistent expectations of deflation during the 1871-1930 period (where current cash flows would be less valuable than future cash flows), and then to the consistent expectations for future inflation that we've seen since about 1930? Or was something else involved here?

Also, did this difference between lower long-term bond yields and higher short-term interest rates prior to 1930 impact your choice of which bonds to include in your historical data series? It would seem that a negatively-sloped yield curve might further confound the comparison of 19th century bond returns with those of the modern era.
SimpleGift, the economist John Cochrane offers one explanation for this phenomenon in this blog post.
Global Market Portfolio + modest tilt towards volatility (80/20->60/40 as approach FI) + modest tilt away from exchange rate risk (80% global+20% U.S. stocks; currency-hedge bonds) + tax optimization
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

SimpleGift wrote: Tue Jul 20, 2021 1:43 am
McQ wrote: Mon Jul 19, 2021 1:27 pm Nonetheless, if you want to go your own way and examine Federal bond yields, Robert Shiller’s web site has an estimate of ten-year Treasury yields in the spreadsheet...
Thank you, Professor McQuarrie, for this ongoing master class in historical U.S. asset returns. One peculiar observation that arises when looking at Shiller's bond yield data since 1871 is that the yield curve appears to be consistently inverted (negatively sloped) from 1871 to about 1930 (chart below) — when it then switches over to the mostly positively-sloped yield curve we've become accustomed to in modern times.
  • Image
    Data source: Shiller annual series
Was this simply due to persistent expectations of deflation during the 1871-1930 period (where current cash flows would be less valuable than future cash flows), and then to the consistent expectations for future inflation that we've seen since about 1930? Or was something else involved here?

Also, did this difference between lower long-term bond yields and higher short-term interest rates prior to 1930 impact your choice of which bonds to include in your historical data series? It would seem that a negatively-sloped yield curve might further confound the comparison of 19th century bond returns with those of the modern era.
Interesting, I hadn't spotted that, since I generally ignore bills. Next, I'm better at history than either economics or finance--the link provided by HootingSloth will take you to a reasonable explanation (although one that is rather like some instances of evolutionary psychology, i.e., just so, and post hoc).
Returning to history, Siegel shows bills and bonds performing about the same from 1802 through the 1930s, so the issue is not peculiar to Shiller's data or to the post-1871 deflationary era. This violates finance theory, which expects a term premium, as seen in the SBBI, which has calculated that term premium as a derived series from the beginning. It may be that such a term premium is one more artifact of the mid-20th century, and not the true normal. Alternatively, as Cochrane suggests, the definition of risk in finance theory is mis-specified. Bills are actually quite risky, in the sense that their future cash flows are entirely uncertain, a point made by Sidney Homer long ago. If investors demand to be compensated for risk, then bills should not return much less than bonds, except under unusual circumstances, because bills carry risk too, albeit a different risk than bonds.
[I'll return to your question about my selection of bonds in a later post]
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
nisiprius
Advisory Board
Posts: 52216
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Historical asset returns before 1970: Where to find

Post by nisiprius »

McQ (or anyone), 1) do you have any insights as to when and why the general public pretty much quit paying attention to the Dow Jones Railroad Average, and the Dow Jones Industrial Average began to be taken as the preeminent measure of "the market?"

As best I can remember when I was a kid in the late 1950s, watching the news show where a report had a big easel with pictures of a great big dollar bill and a much smaller dollar bill, to illustrated inflation, the common-mentioned names and numbers were "the Dow Jones Industrial Average" and "S&P's five-hundred stock average." The word "average" rather than "index" was used for the S&P, and I assume that the phrase "S&P's five-hundred stock average" was used to distinguish it from the S&P Composite Index (which was the successor to the Standard Statistics Composite Index).

2) How do you feel about the omission of the Curb Exchange from the Cowles data? At the time it represented about ⅙ of "the stock market," but Cowles omitted it because the curb exchange was rife with fraud and the paper the reported the numbers literally warned readers that they were unreliable. (My own feeling is that it simply means that accurate data from very far in the past is simply and literally unknowable, no matter how much good-faith effort is put into researching it).
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

nisiprius wrote: Tue Jul 20, 2021 12:36 pm McQ (or anyone), 1) do you have any insights as to when and why the general public pretty much quit paying attention to the Dow Jones Railroad Average, and the Dow Jones Industrial Average began to be taken as the preeminent measure of "the market?"

As best I can remember when I was a kid in the late 1950s, watching the news show where a report had a big easel with pictures of a great big dollar bill and a much smaller dollar bill, to illustrated inflation, the common-mentioned names and numbers were "the Dow Jones Industrial Average" and "S&P's five-hundred stock average." The word "average" rather than "index" was used for the S&P, and I assume that the phrase "S&P's five-hundred stock average" was used to distinguish it from the S&P Composite Index (which was the successor to the Standard Statistics Composite Index).

2) How do you feel about the omission of the Curb Exchange from the Cowles data? At the time it represented about ⅙ of "the stock market," but Cowles omitted it because the curb exchange was rife with fraud and the paper the reported the numbers literally warned readers that they were unreliable. (My own feeling is that it simply means that accurate data from very far in the past is simply and literally unknowable, no matter how much good-faith effort is put into researching it).
Here are some possible answers, taking the two questions one by one.
1. Railroads
The forgotten history: there were no industrial stocks to speak of before the 1890s, except textile firms traded in Boston. Railroads were it from the 1840s to the 1890s (banks and insurance firms, which had dominated prior to the 1840s, had gone off-exchange, with shares largely locked up by insiders and heirs). Then, in the first few decades of the 1900s, US steel and other industrials became bigger then the biggest railroads, and began to dominate trading volume. Then the railroads were temporarily nationalized during WW I. By the 1920s, the railroad sector was steadily shrinking as a share of the economy and trading volume. So this is where the DJIA likely began to dominate, especially since it was always daily (S&P didn't have a daily index until 1925).

2. That famous quote from the CFC dates to early in the time span that Cowles aimed to cover. It wasn't really true anymore after 1920 when the Curb moved in doors. By the time Cowles wrote in the late 1930s the NYSE was so dominant that he felt he could ignore the Curb, Boston, over-the-counter stocks, and everything but the NYSE. Turns out, by the 1920s, the Curb had only 5% to 10% of the capitalization of the NYSE. You might enjoy this blog post by Bryan Taylor: https://globalfinancialdata.com/a-new-i ... k-exchange

Bryan found that AMEX stocks slightly underperformed the NYSE from 1921; but given the small size, overall returns might not be lowered by very many basis points (by the 1920s the Curb was like a farm system for the NYSE, to use a baseball analogy: stocks got promoted to the NYSE after performing well on the AMEX). Last, Robert Sobel's book The Curbstone Brokers gives a history from way back in the 1800s. The CFC quote will emerge as rather small minded, by the end of the book, IMHO.

Bottom line: in theory Cowles' omissions could have made his return estimates wildly off. But in fact, when I corrected some of his survivorship bias, I couldn't budge the values much, a testimony to the power of capitalization-weighting, as noted upthread. Cowles had virtually all the big stocks.

3. Last, of course I can't accept your fear that the past is unknowable, because then I would have wasted the last five years :( But by the same token, cognitive dissonance may be blinding me to the truth of what you said.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

McQ wrote: Tue Jul 20, 2021 12:05 pm Returning to history, Siegel shows bills and bonds performing about the same from 1802 through the 1930s, so the issue is not peculiar to Shiller's data or to the post-1871 deflationary era. This violates finance theory, which expects a term premium, as seen in the SBBI, which has calculated that term premium as a derived series from the beginning. It may be that such a term premium is one more artifact of the mid-20th century, and not the true normal.
Thank you for taking a stab at my economics question, even though it's beyond the scope of your historical data gathering.

Certainly since about 1930, the U.S. bond market has fairly consistently demanded a term premium for longer maturity bonds, a large part of which was compensation for future inflation uncertainty. But prior to 1930, it seems longer maturity bonds traded with a term discount — likely as compensation for future deflation uncertainty.

Since, according to the Shiller data, the average term discount for the 1871-1929 period was -1.1% per year, and the average term premium for the 1930-2016 period was +0.8%, it seems a stretch to compare bond returns from these two periods, as the scale of this premium/discount factor is so large. It hardly seems an "apples to apples" comparison — unless I'm missing something.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

SimpleGift wrote: Tue Jul 20, 2021 4:28 pm
McQ wrote: Tue Jul 20, 2021 12:05 pm Returning to history, Siegel shows bills and bonds performing about the same from 1802 through the 1930s, so the issue is not peculiar to Shiller's data or to the post-1871 deflationary era. This violates finance theory, which expects a term premium, as seen in the SBBI, which has calculated that term premium as a derived series from the beginning. It may be that such a term premium is one more artifact of the mid-20th century, and not the true normal.
Thank you for taking a stab at my economics question, even though it's beyond the scope of your historical data gathering.

Certainly since about 1930, the U.S. bond market has fairly consistently demanded a term premium for longer maturity bonds, a large part of which was compensation for future inflation uncertainty. But prior to 1930, it seems longer maturity bonds traded with a term discount — likely as compensation for future deflation uncertainty.

Since, according to the Shiller data, the average term discount for the 1871-1929 period was -1.1% per year, and the average term premium for the 1930-2016 period was +0.8%, it seems a stretch to compare bond returns from these two periods, as the scale of this premium/discount factor is so large. It hardly seems an "apples to apples" comparison — unless I'm missing something.
I had to significantly modify my answer in a later post the day after. Please do not read this post in isolation.

Much depends on the meaning of "compare," I would propose. You'll recall from the summary paper my idea that asset returns move through regimes. Regimes are, if you will, incommensurable; and certainly, one can't extrapolate performance and differential performance from one to the other. You could leverage this concept to claim that "bonds" before 1930 or so do not equal "bonds" after 1930. I'm fine with that, as long as we agree also that "stocks" from before 1900 do not equal "stocks" from after (per the dividend discussion in the paper). And once we go down that road, the time divisions may grow shorter and shorter (are "bonds" from the 1970s, when there were still plenty of Aaa and Aa credits, really the same as "bonds" post 2010, when these credit ratings account for a vanishingly small part of the market?).

From the Siegel standpoint, and he's far from alone, what you are proposing is very troublesome. His goal was a bigger sample that would yield more precise estimates of true stock returns / the equity premium. That project fails if "bonds" from before 1930 aren't the same thing as "bonds" from after 1930; there is no bigger sample to be had.

From my standpoint, your concerns are not troubling at all; I would simply say that you've pinpointed yet another example of regimes, the term premium / deficit regime, or the bond/bill regime. I look at history not as a bigger sample of time-invariant entities, yielding more precise estimates, but as a plenum useful for revealing how different things could be going forward, based on how different things have been in the past.
Last edited by McQ on Wed Jul 21, 2021 11:31 am, edited 1 time in total.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

McQ wrote: Tue Jul 20, 2021 5:54 pm From my standpoint, your concerns are not troubling at all; I would simply say that you've pinpointed yet another example of regimes, the term premium / deficit regime, or the bond/bill regime. I look at history not as a bigger sample of time-invariant entities, yielding more precise estimates, but as a plenum useful for revealing how different things could be going forward, based on how different things have been in the past.
This makes good sense and I appreciate your articulation of this perspective (bold above). It's what makes the study of financial history so interesting, and also of immense value to Boglehead-style, do-it-yourself investors — since we can't be entirely sure that what's "past" is gone forever, never to recur. Forewarned by history is hopefully forearmed for the future.
User avatar
JamesSFO
Posts: 3404
Joined: Thu Apr 26, 2012 10:16 pm

Re: Historical asset returns before 1970: Where to find

Post by JamesSFO »

These are all fantastic resources thanks for everything you are doing to help raise awareness of how to access/understand/etc
HootingSloth
Posts: 1050
Joined: Mon Jan 28, 2019 2:38 pm

Re: Historical asset returns before 1970: Where to find

Post by HootingSloth »

McQ wrote: Tue Jul 20, 2021 5:54 pm From the Siegel standpoint, and he's far from alone, what you are proposing is very troublesome. His goal was a bigger sample that would yield more precise estimates of true stock returns / the equity premium. That project fails if "bonds" from before 1930 aren't the same thing as "bonds" from after 1930; there is no bigger sample to be had.

From my standpoint, your concerns are not troubling at all; I would simply say that you've pinpointed yet another example of regimes, the term premium / deficit regime, or the bond/bill regime. I look at history not as a bigger sample of time-invariant entities, yielding more precise estimates, but as a plenum useful for revealing how different things could be going forward, based on how different things have been in the past.
Thanks for these thoughts. I am reminded of a related sort of view, although one that comes from a quantitative and statistical perspective rather than a qualitative and historical perspective, developed by Mandelbrot over several decades. His basic belief was that markets can be understood as a scale-invariant process, i.e. one which looks just as chaotic and unpredictable when viewed on the scale of decades or centuries as it does when viewed on the scale of seconds, days, or weeks. It contrasts dramatically with the picture Siegel paints of short-term chaos that can be tamed by waiting for the long run to arrive.

Here is one quote I was able to find quickly where he describes his view succinctly, from Scaling in Financial Prices (2000):
The unconventional thinking behind my work on price variation can only emerge gradually through this paper but deserves to be briefly stated here. The natural and usual response is that the data [showing random variation in market prices at different time scales] must be dealt with separately. To the extent that price variation follows any rule, it is, indeed, generally taken for granted that changes over a day, a week, a month, a year, a lifetime or a century follow separate sets of rules. That is, each time increment raises a separate and distinct challenge.

Quite to the contrary, my belief is that the great overall similarity between [the kind of randomness visible in the different time scales] suggests that, in a first approximation, price variation presents very similar features over a century and a lifetime. A similarity of features not only suggests that the existence of underlying rules is not excluded, but that those rules may be the same at all time scales.
In other words, the sudden punctuation of "ordinary randomenss" by dramatic shifts that can be observed in the moment-by-moment movements in the price of a stock are echoed (in a way that Mandelbrot attempts to quantify precisely) by the sudden punctuation of ordinary randomness by the dramatic shifts associated with regime changes at the largest time scales.

I think the practical consequences of these two perspectives are the same: be cautious in relying on patterns, regardless of how long they seem to have persisted; embrace the inherently unpredictable nature of the market at every time scale; and seek to rest your plans on assumptions that are as modest as you are able to given your other constraints.
Global Market Portfolio + modest tilt towards volatility (80/20->60/40 as approach FI) + modest tilt away from exchange rate risk (80% global+20% U.S. stocks; currency-hedge bonds) + tax optimization
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

SimpleGift wrote: Tue Jul 20, 2021 1:43 am One peculiar observation that arises when looking at Shiller's bond yield data since 1871 is that the yield curve appears to be consistently inverted (negatively sloped) from 1871 to about 1930 (chart below) — when it then switches over to the mostly positively-sloped yield curve we've become accustomed to in modern times.
  • Image
    Data source: Shiller annual series
Was this simply due to persistent expectations of deflation during the 1871-1930 period (where current cash flows would be less valuable than future cash flows), and then to the consistent expectations for future inflation that we've seen since about 1930? Or was something else involved here?

Also, did this difference between lower long-term bond yields and higher short-term interest rates prior to 1930 impact your choice of which bonds to include in your historical data series? It would seem that a negatively-sloped yield curve might further confound the comparison of 19th century bond returns with those of the modern era.
Apologies, SimpleGift: in my haste to give you a timely reply, I instead gave you a hasty and misbegotten answer that can only mislead. I failed to do my homework on Shiller’s series.

Long story short: your chart does not show what I interpreted it to show, i.e., an excess of one-year Treasury bill returns over long term Treasury bond returns in the pre-1930 era, which later reversed.

The problem: the data display on Shiller’s web site is not what it appears to be. It does not show 10-year Treasury returns pre-1930. It does not show one year Treasury returns at any point. Hence, the quantities that you subtracted to get the bar chart are not comparable enough for subtraction to be meaningful. It was hasty of me to call it evidence of bill/bond regimes.

To disentangle what Shiller did, it is necessary to open four books side by side:
1. Shiller’s Market Volatility;
2. Shiller’s Irrational Exuberance 3rd edition;
3. Macaulay’s 1938 book
4. Homer’s 1963 1st edition
And then weave back and forth between appendices of the one, footnotes of the other, and table column counts of a third referenced in footnotes of yet another.

Of course, I have all those books on a shelf next to my desk (one of many reasons I am no longer very good at conversing with normal people). Sorting it out was rather a challenge; but by the second glass of wine, I enjoyed an occasional guffaw. These series turn out to be quite the confection.

The long bond

Wait a minute, I thought on my walk that afternoon after writing the post: there were not hardly any 10-year Treasury bonds issued between 1871 and 1930. So what did Shiller actually use? Here, footnote 7, pp. 282-83 of the 3rd edition of Irrational Exuberance, tells the tale. Shiller spliced:

1. 10-year constant maturity series after 1953 from Treasury bulletins;
2. Next, from the references to Homer’s tables in the note: long government bond average (varying maturity) from 1921 to 1953, sourced from the Federal Reserve bulletin;
3. From 1901 to 1920, the High Grade Bond Buyer’s municipal yield (Shiller’s column count in the note is off by one), as given in Homer’s Table 45.
4. From 1871 to 1900, the New England Municipal bond index of Macaulay as given in Homer’s Table 38—the index Macaulay created by eyeballing yields depressed by savings bank regulations. See my BONDS 1871-1926 post for more on this, ah, semi-fictional series, used by Siegel as well.

Short-term rate

Next, where did Shiller get the one-year rate? After 1938, it is the six-month commercial paper rate from Federal Reserve bulletins, bought in January and rolled over at the July rate, the annual rate computed from the two per Shiller’s equation on page 444 of Market Volatility.

And before 1938, it is Macaulay’s construction of rates on at first 60-90 day paper, and then somewhat longer 3-4 month paper, from “the larger department and men’s furnishing stores, jobbers of dry goods, hardware, shoes, groceries, floor coverings, etc., the manufacturers of cotton, silk and woolen goods, clothing, etc.” pp. A349-A350. BTW, anyone wanting to construct a pre-1938 series of short-term interest rates would do well to pore over Macaulay’s Appendix E—but if you use caffeine, be sure to have a copious supply to hand.

In short: the government “long bonds” in Shiller’s data table are first municipals, then Treasuries; and the one-year rate is never a government bill rate, while varying in span from 60 days to six months. Cross-time comparisons are therefore confounded.

Shiller’s “short” and “long” rates are a dog’s breakfast—what could be revealed by subtraction of the rate on a dry goods merchant’s 90-day paper in New York City, from the yield on a small Massachusetts town’s municipal bond that local savings banks were almost forced to hold?

There is a larger lesson here for erstwhile BH with a historical bent. Robert Shiller is a Nobel prize winner and eminent almost beyond imagining. He did nothing wrong in constructing these series—he only followed the conventions of his field. Those conventions require that a short and a long interest rate be available in every row of the data table. To misquote: economists go to data analysis with the sources they can find in the record, not the rates they might ideally want to have. If there is neither Treasury bill nor 10-year Treasury bond available—generally true before 1930—then you assemble some kind of proxy for each, as Shiller did. Whatever journal reviewers will accept—consisting of fellow economists who follow the same conventions—is good to go.

Pity the poor investor, who browses a web site in good faith, and doesn't have the four books listed above to hand, to disentangle the multiple components of a seemingly straightforward series.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
WoodSpinner
Posts: 3505
Joined: Mon Feb 27, 2017 12:15 pm

Re: Historical asset returns before 1970: Where to find

Post by WoodSpinner »

McQ,

Wow, incredible insights into the reality of the data…

WoodSpinner
WoodSpinner
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

McQ wrote: Wed Jul 21, 2021 11:29 am In short: the government “long bonds” in Shiller’s data table are first municipals, then Treasuries; and the one-year rate is never a government bill rate, while varying in span from 60 days to six months. Cross-time comparisons are therefore confounded.
Bravo!!! Thank you for digging into the historical details of Shiller's two bond yield series.

Many of us here on the Bogleheads Forum who have an amateur interest in financial history have just been happy to have some free historical data like Shiller's spreadsheets available to us — without much appreciation for what is "under the hood," especially for the 19th century data.

Your post was a real eye opener, and suggests the limits of indiscriminately relying on published 19th-century asset return series, without a deeper understanding of their actual sources.
User avatar
siamond
Posts: 6010
Joined: Mon May 28, 2012 5:50 am

Re: Historical asset returns before 1970: Where to find

Post by siamond »

McQ wrote: Wed Jul 21, 2021 11:29 amIn short: the government “long bonds” in Shiller’s data table are first municipals, then Treasuries; and the one-year rate is never a government bill rate, while varying in span from 60 days to six months. Cross-time comparisons are therefore confounded. [...]
Yup. We went through the same kind of unraveling in the bond simulator thread. A few salient posts for reference:
link1
link2
link3
link4
link5
link6

And yes, agreed with your overarching conclusion, we are mixing apples and oranges to a certain extent, by lack of consistent instruments over time.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

siamond wrote: Wed Jul 21, 2021 3:35 pm
McQ wrote: Wed Jul 21, 2021 11:29 amIn short: the government “long bonds” in Shiller’s data table are first municipals, then Treasuries; and the one-year rate is never a government bill rate, while varying in span from 60 days to six months. Cross-time comparisons are therefore confounded. [...]
Yup. We went through the same kind of unraveling in the bond simulator thread. A few salient posts for reference:
link1
link2
link3
link4
link5
link6

And yes, agreed with your overarching conclusion, we are mixing apples and oranges to a certain extent, by lack of consistent instruments over time.
Browsing those links, I see that I launched this topic about nine months too late to help you in that effort. FWIW, Macaulay's book, unavailable to you then, can be downloaded from NBER.com chapter by chapter. I did get a copy of the Analytic Record of Yield Spreads (published annually, mine has data through 1969). The corporate series from 1969 are integrated into the SBBI corporate bond index.

My working paper contains a fairly thoroughgoing assault on the Macaulay indexes that appear in Homer and Sylla, with detailed explanations of what is wrong and by extension, why I would avoid the Shiller long bond series. https://papers.ssrn.com/sol3/papers.cfm ... id=3269683
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
LadyGeek
Site Admin
Posts: 95696
Joined: Sat Dec 20, 2008 4:34 pm
Location: Philadelphia
Contact:

Re: Historical asset returns before 1970: Where to find

Post by LadyGeek »

Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

LadyGeek wrote: Wed Jul 21, 2021 6:25 pm ^^^ The link is incorrect. You can find Macaulay's book here:

Some Theoretical Problems Suggested by the Movements of Interest Rates, Bond Yields and Stock Prices in the United States since 1856 | NBER

You can find his other publications here: Frederick R. Macaulay | NBER
LadyGeek is of course correct: it is nber.org, not nber.com. My bad. In addition to the links she gave to Macaulay's texts, if you want to download his data files, first go to https://www.nber.org/research/data/nber ... y-database

That describes multiple sections/topics. If it is interest rates you want, click on that chapter or start here: https://www.nber.org/research/data/nber ... rest-rates. Any series that begins in 1857 likely comes from Macaulay; click on the doc link to confirm. Data frequency is generally monthly.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Historical OTHER ASSET returns 1871 to 1926

Gold prices can be obtained from measuringworth.com, or page A215 of Macaulay, or from books by Mitchell. https://www.google.com/books/edition/Go ... CAAJ?hl=en

Inflation and foreign exchange can be obtained from measuringworth.com.

Short term interest rates can be found at measuringworth.com, often based on Macaulay, in turn sometimes based on Martin. Read Macaulay's Appendix E before using these series.

Jorda et al. have real estate returns from 1870 in the US (and fifteen other countries). See links in the prior OTHER ASSETS post
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

HootingSloth wrote: Tue Jul 20, 2021 7:30 pm
McQ wrote: Tue Jul 20, 2021 5:54 pm From the Siegel standpoint, and he's far from alone, what you are proposing is very troublesome. His goal was a bigger sample that would yield more precise estimates of true stock returns / the equity premium. That project fails if "bonds" from before 1930 aren't the same thing as "bonds" from after 1930; there is no bigger sample to be had.

From my standpoint, your concerns are not troubling at all; I would simply say that you've pinpointed yet another example of regimes, the term premium / deficit regime, or the bond/bill regime. I look at history not as a bigger sample of time-invariant entities, yielding more precise estimates, but as a plenum useful for revealing how different things could be going forward, based on how different things have been in the past.
Thanks for these thoughts. I am reminded of a related sort of view, although one that comes from a quantitative and statistical perspective rather than a qualitative and historical perspective, developed by Mandelbrot over several decades. His basic belief was that markets can be understood as a scale-invariant process, i.e. one which looks just as chaotic and unpredictable when viewed on the scale of decades or centuries as it does when viewed on the scale of seconds, days, or weeks. It contrasts dramatically with the picture Siegel paints of short-term chaos that can be tamed by waiting for the long run to arrive.

Here is one quote I was able to find quickly where he describes his view succinctly, from Scaling in Financial Prices (2000):
The unconventional thinking behind my work on price variation can only emerge gradually through this paper but deserves to be briefly stated here. The natural and usual response is that the data [showing random variation in market prices at different time scales] must be dealt with separately. To the extent that price variation follows any rule, it is, indeed, generally taken for granted that changes over a day, a week, a month, a year, a lifetime or a century follow separate sets of rules. That is, each time increment raises a separate and distinct challenge.

Quite to the contrary, my belief is that the great overall similarity between [the kind of randomness visible in the different time scales] suggests that, in a first approximation, price variation presents very similar features over a century and a lifetime. A similarity of features not only suggests that the existence of underlying rules is not excluded, but that those rules may be the same at all time scales.
In other words, the sudden punctuation of "ordinary randomenss" by dramatic shifts that can be observed in the moment-by-moment movements in the price of a stock are echoed (in a way that Mandelbrot attempts to quantify precisely) by the sudden punctuation of ordinary randomness by the dramatic shifts associated with regime changes at the largest time scales.

I think the practical consequences of these two perspectives are the same: be cautious in relying on patterns, regardless of how long they seem to have persisted; embrace the inherently unpredictable nature of the market at every time scale; and seek to rest your plans on assumptions that are as modest as you are able to given your other constraints.
Apologies for this delayed reply, HootingSloth. Here is a question about Mandelbrot. I’m only vaguely familiar with his work from reading Nassim Taleb (Black Swan etc.)

Drawing on your expertise: would you say that Mandelbrot’s argument re scale invariance is based on roughly the same mathematics as Pastor and Stambaugh? Or do they use different math to reach the same conclusion? Or am I comparing apples and oranges here?

Aside to BH: Pastor and Stambaugh were among the first to reject the comforting conclusion that some investors drew from reading Siegel, which was that stocks were really not that risky if held for a long period. https://onlinelibrary.wiley.com/doi/ful ... 12.01722.x
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
HootingSloth
Posts: 1050
Joined: Mon Jan 28, 2019 2:38 pm

Re: Historical asset returns before 1970: Where to find

Post by HootingSloth »

McQ wrote: Thu Jul 22, 2021 4:31 pm
HootingSloth wrote: Tue Jul 20, 2021 7:30 pm
McQ wrote: Tue Jul 20, 2021 5:54 pm From the Siegel standpoint, and he's far from alone, what you are proposing is very troublesome. His goal was a bigger sample that would yield more precise estimates of true stock returns / the equity premium. That project fails if "bonds" from before 1930 aren't the same thing as "bonds" from after 1930; there is no bigger sample to be had.

From my standpoint, your concerns are not troubling at all; I would simply say that you've pinpointed yet another example of regimes, the term premium / deficit regime, or the bond/bill regime. I look at history not as a bigger sample of time-invariant entities, yielding more precise estimates, but as a plenum useful for revealing how different things could be going forward, based on how different things have been in the past.
Thanks for these thoughts. I am reminded of a related sort of view, although one that comes from a quantitative and statistical perspective rather than a qualitative and historical perspective, developed by Mandelbrot over several decades. His basic belief was that markets can be understood as a scale-invariant process, i.e. one which looks just as chaotic and unpredictable when viewed on the scale of decades or centuries as it does when viewed on the scale of seconds, days, or weeks. It contrasts dramatically with the picture Siegel paints of short-term chaos that can be tamed by waiting for the long run to arrive.

Here is one quote I was able to find quickly where he describes his view succinctly, from Scaling in Financial Prices (2000):
The unconventional thinking behind my work on price variation can only emerge gradually through this paper but deserves to be briefly stated here. The natural and usual response is that the data [showing random variation in market prices at different time scales] must be dealt with separately. To the extent that price variation follows any rule, it is, indeed, generally taken for granted that changes over a day, a week, a month, a year, a lifetime or a century follow separate sets of rules. That is, each time increment raises a separate and distinct challenge.

Quite to the contrary, my belief is that the great overall similarity between [the kind of randomness visible in the different time scales] suggests that, in a first approximation, price variation presents very similar features over a century and a lifetime. A similarity of features not only suggests that the existence of underlying rules is not excluded, but that those rules may be the same at all time scales.
In other words, the sudden punctuation of "ordinary randomenss" by dramatic shifts that can be observed in the moment-by-moment movements in the price of a stock are echoed (in a way that Mandelbrot attempts to quantify precisely) by the sudden punctuation of ordinary randomness by the dramatic shifts associated with regime changes at the largest time scales.

I think the practical consequences of these two perspectives are the same: be cautious in relying on patterns, regardless of how long they seem to have persisted; embrace the inherently unpredictable nature of the market at every time scale; and seek to rest your plans on assumptions that are as modest as you are able to given your other constraints.
Apologies for this delayed reply, HootingSloth. Here is a question about Mandelbrot. I’m only vaguely familiar with his work from reading Nassim Taleb (Black Swan etc.)

Drawing on your expertise: would you say that Mandelbrot’s argument re scale invariance is based on roughly the same mathematics as Pastor and Stambaugh? Or do they use different math to reach the same conclusion? Or am I comparing apples and oranges here?

Aside to BH: Pastor and Stambaugh were among the first to reject the comforting conclusion that some investors drew from reading Siegel, which was that stocks were really not that risky if held for a long period. https://onlinelibrary.wiley.com/doi/ful ... 12.01722.x
I definitely wouldn't call myself an expert in this, but I do have a math background and like to dabble.

I've described the Pastor and Stambaugh piece before as being in the same "genre" as Mandelbrot (and Taleb). They start from a similar place and with a somewhat similar attitude. Namely, instead of asking "Lets assume that the distribution of returns on stocks is drawn from this class of probability distributions X (often normal distributions, but sometimes something a bit more realistic), what can we conclude?", they ask "What can (and can't) the data that we have tell us about what the distribution of returns looks like and where does that lead?"

I think there is also at least some personal link between them as well: Mandelbrot was an important early influence on Eugene Fama, although Fama went on to take his career in a very different direction. Fama, who is at the University of Chicago with Pastor, appears throughout the Pastor and Stambaugh paper and is thanked in the acknowledgments.

When you don't build in strong starting assumptions that the "true" return-generating process of stock market returns takes some narrowly-defined form, you--not surprisingly--end up thinking there is a lot more uncertainty about the future than you do when you make that kind of assumption. And so the papers really clarify that relying on these traditional models and methods means that you are relying a lot on the idea that you already know how stocks like to behave before you even look at the data that shows how they have actually behaved.

Although they share this basic commonality, there are a lot of differences as well. Mandelbrot's roots, as the father of fractals, one of the pioneers of chaos theory, and as an important player in the birth of automated computation and computer graphics, meant that he loved to just expose himself to vast quantities of visual representations of data and let that guide his thinking. His work in mathematical finance, and in math more generally, is very, very different from almost anything else you can find--a lot of very novel intuition, idiosyncratic definitions, etc.

I think it's fair to say, in contrast, that Pastor and Stambaugh's math is a lot more traditional in its definitions, formalism, and methods. The less traditional conclusions that they reach don't come from looking at markets in a totally different way. Instead they come from their insistence on very carefully accounting not just for the uncertainty about what stock prices may do in each moment but for many different types of our uncertainty about what the true generating-process for returns even looks like.
Global Market Portfolio + modest tilt towards volatility (80/20->60/40 as approach FI) + modest tilt away from exchange rate risk (80% global+20% U.S. stocks; currency-hedge bonds) + tax optimization
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Very helpful, HootingSloth--I did want to draw on your math background, and your answer is everything I hoped.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Post Reply