Deep history of stock and bond returns

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Historical US STOCK returns 1793 to 1871

Here again, this post has to include a critique rather than to remain a neutral reference, because it is in this era that I found major problems with Siegel’s sources. However, I was only able to find those problems because of the recent data collection spearheaded by Richard Sylla at New York University, which is now the go-to source for US stock price quotes before 1860.

A. Sylla database
Beginning in the late 1990s the team used grant money from the NSF to send research assistants out to major public and university libraries that had old newspaper archives. The expectation was that stock quotes could be found back to the early 1790s, certainly in New York, given the Buttonwood agreement of 1792. However, the team was somewhat surprised to discover how many stocks were quoted, and how early, and across how many cities. By the Panic of 1837 there were hundreds of stocks with price quotes, across Boston, Philadelphia, and Baltimore as well as New York—and in still more cities.

Best of all: the individual stock data can be downloaded from EH.net in spreadsheet form. But it is very, very raw, with missing observations thickly and unevenly scattered and using no consistent time interval. https://eh.net/database/early-u-s-securities-prices/

Worst of all: no information on dividends was collected, there is no information on any other kind of income, and the price quotes in Philadelphia and Baltimore are not adjusted for changes in par value. You will have to read my paper to understand why par value is important; suffice to say, price appreciation calculated on the raw quotes in the Sylla database, in those markets, will often be misleading.

B. Goetzmann, Ibbotson and Peng (2001)
This team at Yale undertook an independent investigation similar to the Sylla effort, and one that proceeded at about the same time. However, Goetzmann et al. looked only at the New York Stock Exchange, and used a more restricted newspaper sample, with a start date about two decades after the Sylla data. They give price appreciation from 1815 and total return from 1825. They have price quotes past 1871 but rely on Cowles for dividends after that point.

This dataset was swapped in by Siegel after the 3rd edition of his book to cover the 1825 – 1871 period. If you read a popular account that says “since 1824 (1825) US stocks have …” this is almost certainly the source.

Good news: they observed many dividends, a first among historical datasets in this era.

More good news: the spreadsheets can be downloaded from another part of the Yale website, and a discussion of key returns data has appeared as a chapter in the SBBI for the past decade or so. https://som.yale.edu/faculty-research/o ... al-newyork

Not so good news: when I spent hours poring over the spreadsheets, I found not a few failures to clean the data (out-of-range values, etc.). There are also many missing dividends, and quite a few missing stock dividends or rights. Entire years are missing many observations, and the data coverage is very thin at certain points (as few as four stocks at some points in the 1840s, whereas dozens and dozens could be found in the Sylla spreadsheets or even Martin).

Bottom line: it provides a useful supplement to the Sylla data here and there, but stated index returns and dividend yields cannot be taken at face value. The limitation to New York also makes it a misleading source for understanding the Panics of 1819 and 1837, which were much more severe from Philadelphia on south and west.

C. Joseph G. Martin, A Century of Finance. Martin didn’t calculate any indexes of return, but provided annual price ranges and dividends, for stocks and a variety of bonds. Stock data begins in the 1830s and includes share count, bond data begins in the 1790s. Martin’s footnotes are invaluable for pinning down rights issues, stock dividends, changes in par value (not absent in Boston, but not as common as in Philadelphia and Baltimore), and merger premia. The narrative annals are also informative.

D. Smith and Cole, Fluctuations in American Business, published in 1935. This was Siegel’s original source for stock returns from 1802 to 1862, filtered through Schwert (1990), who made some weighting errors when he combined sub-indexes. Whenever you encounter a popular account that reads “since 1802 stocks have …” Smith and Cole’s book is the likely root source, often filtered through Siegel.

Smith and Cole compiled three indexes, each with some overlap: 1) seven banks from 1802 to 1820, in Boston, Philadelphia and New York; 2) six banks and one insurance company in New York from 1815 to 1845; and 3) a varying number of railroad stocks from 1834 to 1862, listed in New York, Philadelphia, Boston, and Baltimore.

These indexes are equal-weighted averages of monthly price relatives. No dividends were collected. I believe they were unaware of par value practices in Philadelphia and Baltimore, leading to some instances of inflated price appreciation in the early years; they also made no adjustment for stock dividends (rights appear to have been uncommon before the 1860s but I found stock dividends from the 1840s).

Next, consider how Smith and Cole decided whether to include a stock in their bank indexes (they collected prices on dozens more than made it into the indexes). First they charted price movement using a crayon on the 1930s equivalent of a transparency. Next, they stacked the transparencies on a light table. Then they winnowed out stocks whose price movements were discrepant from the central tendency, retaining only the small number of stocks that mostly moved synchronously with one another.

Put another way, they had a premodern understanding of statistics. Rather than accepting the mean of all prices as the best linear unbiased estimate, they sought the prototypical stock(s). The reason: they were cycle theorists, hoping to find a measure of stock prices that would be predictive of movements in other economic quantities.

Of course, if a stock plunged to nearly zero, as did the 2nd Bank of the US in the early 1840s, it would be discarded as discrepant. The result was a fairly extreme form of survivorship bias, of which Siegel was unaware (at least initially; because Schwert had compiled the time series he needed, Siegel probably did not pore over Smith and Cole’s appendices or methodology).

E. Macaulay (see 1871 to 1926 BONDS post). Although bond yields were his main concern, Macaulay also compiled a railroad stock index from 1857 through 1936. Filtered through Schwert, this provided early editions of Siegel with a stock index that closed the gap between Smith and Cole’s work and Cowles’ beginning. Macaulay’s index is price only, but capitalization-weighted, with adjustments for stock dividends, rights, merger premia and even assessments (missing from Cowles). It can be downloaded from NBER.org.

Note on assessments. The old practice was for failing railroads in receivership to assess stockholders $6, $10, or more, per $100 share, in order to retain their (diluted) equity and be issued new shares after the reorganization. (Modern bankruptcy law, in which stockholders may lose everything, did not yet exist.) If the assessment is not taken into account, prices on the new shares will seem to have rallied more than in fact occurred.

Survivorship bias: I found that Macaulay tended to omit railroads after they got in trouble, or to not include reorganized roads until the trouble had passed. But because of capitalization-weighting, the bias proved not to be huge, albeit greater before 1871 than after, see the 1871-1926 post upthread.

F. Global Financial Data has both stock indexes and individual stock price information from 1792. A description can be found on their blog. The data are located behind a paywall; your library may subscribe. Blog posts with charts are free and can give a sense of the data.

G. My summary paper draws on all these sources except GFD (I’ve had some input to the GFD data), plus dividends that I gathered. I give a stock index from 1793 that is capitalization-weighted and provides total return with dividends and other adjustments added. I believe it to be the index most free of survivorship bias.

Why start in 1793, and not the 1792 of GFD, or the even earlier instances in the Sylla data? January 1793 was the first January that I could find prices that I trusted, on more than one bank, other than the 1st Bank of the US. Earlier quotes on the 1st BUS were all over the map across cities, trading in half-shares and even three-quarter shares, and also seemingly on a part-paid basis. As a consequence, I had to omit the first bear market panic in US history, which occurred in mid-1792. You can read about it in Robert Sobel’s book, Panic on Wall Street.

Critique

It is only before 1871 that my historical research generated a different pattern of stock returns than Siegel or other existing sources. As noted upthread, my reexamination of Cowles didn’t change much, and after 1897 I rely on existing sources for stock returns. Hence, the new contribution occurs before 1871.

In this section I will briefly explain what Siegel’s pre-1871 sources left out, and why these omissions proved to be important. And again, I intend this language to be precise: in my view, Siegel made no error; rather, he had to rely on the faulty data of others, sources that were already 50 or 100 years old when he first assembled his data three decades ago. Coming on the scene much later, I had the benefit of the new Sylla dataset, and of the digitization revolution.

Omission #1: no dividends or spotty dividends

Smith and Cole collected no data on dividends; Siegel simply made a guess at an average dividend yield, a guess which he changed between the 1992 papers and later book editions, to the consternation of Jason Zweig and other observers. https://jasonzweig.com/does-stock-marke ... 200-years/

Goetzmann et al. did attempt to collect dividends but they didn’t try too hard; for instance, no one on the team consulted Poor’s Manual of the Railroads, or Martin (who had dividends on stocks trading in New York as well as Boston). Put another way, the two senior authors on the paper are as illustrious any financial historians still active; but on the ground, data collection and compilation was conducted by relatively junior assistants who were not established financial historians and did not grasp the range of information sources available—even before the digital revolution.

My second contribution was to exploit newly available digital resources to find the dividends paid (dividend notices were published in what someone of my age would call the classified ads section of the newspapers of the day). Because newly available digitized newspaper archives are machine searchable, without an undue amount of effort I was able to find almost all these dividends (with some supplementation from the Goetzmann et al. database, hereby acknowledged).

Omission #2: no information on capitalization

Smith and Cole used equal weighting. That was not too much of a problem before the 1840s, because all their highly selected large stocks had about the same capitalization. Later, as some railroads ballooned, it might have been a problem; but their post 1825 data is no longer used by Siegel.

Goetzmann et al. used price weighting rather than capitalization weighting. Unfortunately, on the NYSE, most prices were reported as percent of par, so that most prices read as 100 + / -, and price weighting had little effect—in this era.

Using the new digital resources, I was able to find capitalization, same as I found dividends, and this proved important in the 1840s, when one huge stock plunged toward zero. See #4 below.

Omission #3: no stocks outside New York

This omission doesn’t matter much after, say, 1900, when the NYSE had achieved national dominance. But the NYSE (it had a different name before 1870) was not the uber-dominant exchange in 1810, 1820, 1830, 1840, or 1850, that it was to become after the Civil War.

Sources that exclude Philadelphia and Baltimore give an overly rosy view of antebellum stock returns. Stocks on these exchanges were hammered in 1819 and 1837, far worse than what happened in New York (or Boston).

Omission #4: survivorship bias

The big enchilada, the 900 pound gorilla, the Godzilla of omissions, the ultimate in survivorship bias, is the exclusion of the 2nd Bank of the United States from Siegel’s sources. Smith and Cole would have deliberately excluded it because its price behavior was discrepant. Goetzmann et al. inadvertently excluded it because it traded in Philadelphia not New York.

The 2nd BUS accounted for almost 30% of the capitalization of my index just before the Panic of 1837 really turned south—more than Apple etc. as a share of the S&P 500 here in mid-2021. It fell in price by almost 99% and never recovered. Stock returns, how shall I say, do not look as good before the Civil War once the 2nd BUS is included in a capitalization-weighted index.

And the survivorship bias in Siegel’s sources does not stop there. Prior indexes had omitted the canals overwhelmed by railroads, the railroads that never paid a dividend, and sundry other banks that went bust. Or as I sometimes put it, Siegel’s sources had “left out the bad parts.” I included all the larger stocks that I could find, and that is why I get lower stock returns for this era than other scholars.

This working paper has the details of this era: https://papers.ssrn.com/sol3/papers.cfm ... id=3480838
The summary paper has the final data series for this period: https://papers.ssrn.com/sol3/papers.cfm ... id=3805927
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

To quote from your recent paper on new U.S. stock indexes from 1793 to 1871:
Edward McQuarrie wrote:...as a frontier and then emerging market, no bond issue in the 19th century US, Federal or otherwise, can be described as a low-risk credit. Rather, as with emerging market bonds in the 21st century, interest rates should have been high in the early 19th century US.
To further support your point, it's worth noting that since the inception of emerging market bond funds as investable vehicles in the late 1990s, emerging market bond funds have consistently outperformed emerging market stocks ever since:
  • Performance of Emerging Market BONDS vs. Emerging Market STOCKS (in dark red), 1997-2021
    Image
    NOTE: The four active EM bond funds shown have at least a 24-year history, plus consistent Morningstar coverage.
    Source: Morningstar
Though 24 years doesn't mean much, and starting point selection is critical to any performance comparison, it doesn't seem like a stretch at all to believe that U.S. bonds in the early 19th century might naturally be outperforming the U.S. stocks of that era.
Last edited by SimpleGift on Fri Jul 23, 2021 11:21 pm, edited 1 time in total.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

SimpleGift wrote: Fri Jul 23, 2021 1:49 pm To quote from your recent paper on new U.S. stock indexes from 1793 to 1871:
Edward McQuarrie wrote:...as a frontier and then emerging market, no bond issue in the 19th century US, Federal or otherwise, can be described as a low-risk credit. Rather, as with emerging market bonds in the 21st century, interest rates should have been high in the early 19th century US.
To further support your point, it's worth noting that since the inception of emerging market bond funds as investable vehicles in the late 1990s, these emerging market bond funds have consistently outperformed emerging market stocks ever since:
  • Performance of Emerging Market BONDS vs. Emerging Market STOCKS (in dark red), 1997-2021
    Image
    NOTE: The four active EM bond funds shown have at least a 24-year history, plus consistent Morningstar coverage.
    Source: Morningstar
Though 24 years doesn't mean much, and starting point selection is critical to any performance comparison, it doesn't seem like a stretch at all to believe that U.S. bonds in the early 19th century might naturally be outperforming the U.S. stocks of that era.
I'm tickled that you ran those numbers in Morningstar--I had done something similar, using only the oldest of the emerging market bond funds, when the point you quoted first occurred to me; but I could never find a place for the chart in a paper or presentation. Now here it is on Bogleheads, where I can point to it, courtesy of your labor--much obliged!

The general point, which I am still struggling with different ways of expressing, is that finance theory may or may not be correct that investors demand compensation for risk; but to assign "stocks" to the risk category, and "government bonds" to the no / less risk category, is far too facile. Or as Hamlet might have put it, "There are more risks among assets and markets, Professor Siegel, than are dreamt of in your CAPM."
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

McQ wrote: Fri Jul 23, 2021 2:13 pm The general point, which I am still struggling with different ways of expressing, is that finance theory may or may not be correct that investors demand compensation for risk; but to assign "stocks" to the risk category, and "government bonds" to the no / less risk category, is far too facile.
Side-stepping your financial theory question for the moment, I wonder if there aren't a few other factors that might have contributed to the outperformance of bonds over stocks in emerging economies, whether it's the early 19th-century USA or the 21st-century emerging nations:
  • 1. Bond Overperformance. As emerging countries succeed and develop, their sovereign bonds become less risky and less susceptible to default, and thus see falling interest rates and significant price appreciation of their bonds. This has been a major contributor to the outperformance of emerging market bonds in the 20th-21st century, I believe, and may have likewise been a factor in U.S. bonds' outperformance in the early 19th century.

    2. Stock Underperformance. Fast-growing emerging markets tend to fund much of their economic growth with the issuance of new stock shares, often resulting in massive dilution of the earnings-per-share of existing shareholders. See this recent post on China. So while these emerging countries are growing rapidly, their returns to stock shareholders are often much lower than those of less-risky, developed countries with only minor share dilution.
So it may be that bond outperformance over stocks in emerging economies is not a risk question at all (which would challenge financial theory), but is perhaps due to underlying factors that are just intrinsic to emerging-country bond and stock markets.
User avatar
siamond
Posts: 6010
Joined: Mon May 28, 2012 5:50 am

Re: Historical asset returns before 1970: Where to find

Post by siamond »

McQ wrote: Fri Jul 23, 2021 2:13 pmThe general point, which I am still struggling with different ways of expressing, is that finance theory may or may not be correct that investors demand compensation for risk; but to assign "stocks" to the risk category, and "government bonds" to the no / less risk category, is far too facile.
Even without your fascinating addition to historical data, I had reached the same conclusion a long time ago. Stocks and bonds typically display[ed] different types of risks, but the 'bonds are for safety' view is utterly misguided IMHO. :shock:
AlohaJoe
Posts: 6609
Joined: Mon Nov 26, 2007 1:00 pm
Location: Saigon, Vietnam

Re: Historical asset returns before 1970: Where to find

Post by AlohaJoe »

McQ wrote: Fri Jul 23, 2021 2:13 pm The general point, which I am still struggling with different ways of expressing, is that finance theory may or may not be correct that investors demand compensation for risk; but to assign "stocks" to the risk category, and "government bonds" to the no / less risk category, is far too facile. Or as Hamlet might have put it, "There are more risks among assets and markets, Professor Siegel, than are dreamt of in your CAPM."
Thanks for the ongoing historical lessons. I knew some of it, but not all of it, and the additions are fascinating. The whole idea of regime changes in finance fascinates me (and it kinda of low-grade terrifying as an investor banking their retirement on investing).

On risks/safety, I am reminded of the quote ascribed to Einstein about "everything should be made as simple as possible, but not simpler" and I think the prototypical Bogleheads take on bonds is one that goes too far to be "simpler".
dcabler
Posts: 4544
Joined: Wed Feb 19, 2014 10:30 am
Location: TX

Re: Historical asset returns before 1970: Where to find

Post by dcabler »

siamond wrote: Fri Jul 23, 2021 10:10 pm
McQ wrote: Fri Jul 23, 2021 2:13 pmThe general point, which I am still struggling with different ways of expressing, is that finance theory may or may not be correct that investors demand compensation for risk; but to assign "stocks" to the risk category, and "government bonds" to the no / less risk category, is far too facile.
Even without your fascinating addition to historical data, I had reached the same conclusion a long time ago. Stocks and bonds typically display[ed] different types of risks, but the 'bonds are for safety' view is utterly misguided IMHO. :shock:
Similar opinion about bonds - these days I think of bonds as either an asset with low correlation to others or as a source of an income stream which may very well lose to inflation....

Cheers.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

SimpleGift wrote: Fri Jul 23, 2021 3:26 pm
McQ wrote: Fri Jul 23, 2021 2:13 pm The general point, which I am still struggling with different ways of expressing, is that finance theory may or may not be correct that investors demand compensation for risk; but to assign "stocks" to the risk category, and "government bonds" to the no / less risk category, is far too facile.
Side-stepping your financial theory question for the moment, I wonder if there aren't a few other factors that might have contributed to the outperformance of bonds over stocks in emerging economies, whether it's the early 19th-century USA or the 21st-century emerging nations:
  • 1. Bond Overperformance. As emerging countries succeed and develop, their sovereign bonds become less risky and less susceptible to default, and thus see falling interest rates and significant price appreciation of their bonds. This has been a major contributor to the outperformance of emerging market bonds in the 20th-21st century, I believe, and may have likewise been a factor in U.S. bonds' outperformance in the early 19th century.

    2. Stock Underperformance. Fast-growing emerging markets tend to fund much of their economic growth with the issuance of new stock shares, often resulting in massive dilution of the earnings-per-share of existing shareholders. See this recent post on China. So while these emerging countries are growing rapidly, their returns to stock shareholders are often much lower than those of less-risky, developed countries with only minor share dilution.
So it may be that bond outperformance over stocks in emerging economies is not a risk question at all (which would challenge financial theory), but is perhaps due to underlying factors that are just intrinsic to emerging-country bond and stock markets.
Hello SimpleGift: your explanations for why emerging market stocks and bonds behave the way they do seem quite plausible to me. But I’d like to argue that even as they rescue finance theory against the emerging market counter-example, they also undermine it.

Theory is nothing if not general. As soon as we start carving out exceptions (emerging market government bonds aren’t the “bonds” of the theory, but some other thing), the theory weakens. Where do the carve outs stop? Are international stocks the “stocks” covered by theory? Are international bonds in general really “bonds?”

You will be more familiar than I with the countless debates on this board concerning allocations to international stocks and bonds (see this post by Robot Monster with a dozen links: viewtopic.php?p=6132935#p6132935). Every rejection of the cap weighted global portfolio is implicitly a rejection of the reigning finance theory, whether explicitly acknowledged or not. Fine by me; I'm just not that into it.

That theory may turn out not be general at all, but may some day have to be rephrased as “whenever a nation rises to world hegemony, its government bonds will come to approximate the risk free rate, while its equities will enjoy returns to hegemony as well as returns to risk; the conjunction of the two processes predicts a sizable equity premium.” By extension, an equity premium of that magnitude may not generalize to non-hegemons--or to the hegemon itself, once its domination begins to slip.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Historical BOND returns 1793 to 1871

Here the sources for yields mostly overlap with those in the previous BONDS post, and I’ll be brief in describing them here, without links.
First and foremost is the Hall, Payne and Sargent database, which has prices of Federal bonds from the very beginning, with terms and amount outstanding year by year, as well as prices.

Next, History of Interest Rates has prices of Federal bonds and selected municipal bonds, generally sourced from Martin in this period. Siegel uses the yields in the History for Boston or Massachusetts or Federal bonds, and selects whichever yielded least to extract his bond returns.

Fun Factoid: Homer published the first edition in 1963. All US tables numbered 48 and below (data up through 1946) are the same in the 2005 Homer and Sylla edition as in the first edition. Homer was not an academic, by the way; he worked for Salomon Brothers and other investment banks.

The Sylla EH.net database has prices for most Federal bonds, often in different marketplaces (useful for detecting currency effects in the early decades, before there was an established national currency). The Sylla data also has a wealth of municipal bond prices, far more than appeared in History of Interest Rates. And, although very fragmentary before 1850, the Sylla database has some of the very earliest corporate bond quotes I could find (canals around 1830).

Martin has a variety of municipal bonds from the 1850s, and Massachusetts or Boston bonds from 1798 (with gaps).

Smith and Cole have prices for the US 3s monthly from late 1795, now superseded by the Sylla and the Hall, Payne and Sargent collections.

Macaulay has railroad bond yield indexes back to 1857. Use with caution. Before computing any returns or other quantities, study his chart on p. 91. Why is the dashed line different from the others, and what are the implications of the fact that the dashed line does not appear in the much-cited Table 10, whereas the others do?

GFD has a US government bond index back to the early 1790s.

I constructed an aggregate bond index from 1793, all Federal through 1825, then adding municipals, then in 1832 adding corporate bonds. Municipal and corporate bonds carry the index through the interregnum after President Jackson paid off the Federal debt (~1835 to ~1842). The municipal bond portion captures the crisis of the 1840s when some states defaulted on interest payments. The aggregate index continues through 1871 and beyond, see previous BONDS post.

Evaluation

I assumed that Siegel’s selection of the lowest yielding bond, along with the use of only a single bond, would be problematic. But as it turned out, prior to 1857, my new aggregate index based on observed bond prices produced about the same cumulative return as his single-bond, yield-extraction procedure. (This is also the period where even Siegel’s data show bonds returning about the same as stocks over multiple decades.)

The big gap in our respective bond return series begins in 1862. Siegel did not adjust for the gold premium. Specifically, Federal bonds, Massachusetts bonds, and Boston bonds paid interest in gold coin; most others paid interest in greenbacks, which were legal tender. At the peak in 1864, a gold dollar was worth over 2.5X a greenback dollar, even as those greenbacks could be used to purchase more of these gold-paying bonds (because legal tender). It was the later 1870s before the gold premium sunk to de minimus levels. Accordingly, nominal yields on gold-paying bonds were depressed, because prices were high to reflect the value investors placed on being paid in gold, leading Siegel to calculate a misleadingly low total return on these bonds after 1862. And not just Siegel: everybody.

The other point where our series differ is around the Panic of 1837. The Boston or Massachusetts bond that Siegel got from History of Interest Rates, which Homer had gotten from Martin, scarcely traded, according to the Sylla database. Martin’s “quotes” were annual highs and lows, with Siegel’s yield calculated from the midpoint. That midpoint scarcely varied year to year at the depths of the crisis (wonks may wish to consult the Schwert (1990) paper on the “Working effect”). Accordingly, Siegel’s series materially misestimates the volatility of bond returns during that long ago crisis.

The relevant working papers for this era are:
pre-1857: https://papers.ssrn.com/sol3/papers.cfm ... id=3260733
post 1857, the gold premium, and problems with Macaulay: https://papers.ssrn.com/sol3/papers.cfm ... id=3269683
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

McQ wrote: Sat Jul 24, 2021 5:14 pm I constructed an aggregate bond index from 1793, all Federal through 1825, then adding municipals, then in 1832 adding corporate bonds...(snip)...The aggregate index continues through 1871....
One question that's been back of mind when reading through your papers: How many securities were actually involved in the construction of these early 19th-century asset return indexes? No doubt U.S. securities markets were quite thin in the 1790s, and then deepened through the 1800s — but is this quantified anywhere in your papers? Apologies if I've missed it somewhere.

A quick search turned up the chart below in a 1999 paper by Rosseau and Sylla — but I'm sure that with the new research by you and others, many more debt and equity securities have been identified to add to an updated chart. Instead of by regional markets, a graphic showing number of debt vs. equity securities over time might be more informative.
  • Image
Interesting factoid: The paper indicates that by 1803, fully half of all the U.S. securities issued to date were held by European investors. For those in the old countries, the U.S. was apparently the hot emerging (frontier?) market investment of that era.
Last edited by SimpleGift on Sat Jul 24, 2021 11:51 pm, edited 1 time in total.
Alpha4
Posts: 176
Joined: Tue Apr 17, 2012 8:47 pm

Re: Historical asset returns before 1970: Where to find

Post by Alpha4 »

McQ wrote: Sat Jul 24, 2021 5:14 pm Historical BOND returns 1793 to 1871

Here the sources for yields mostly overlap with those in the previous BONDS post, and I’ll be brief in describing them here, without links.
First and foremost is the Hall, Payne and Sargent database, which has prices of Federal bonds from the very beginning, with terms and amount outstanding year by year, as well as prices.

Next, History of Interest Rates has prices of Federal bonds and selected municipal bonds, generally sourced from Martin in this period. Siegel uses the yields in the History for Boston or Massachusetts or Federal bonds, and selects whichever yielded least to extract his bond returns.

Fun Factoid: Homer published the first edition in 1963. All US tables numbered 48 and below (data up through 1946) are the same in the 2005 Homer and Sylla edition as in the first edition. Homer was not an academic, by the way; he worked for Salomon Brothers and other investment banks.

The Sylla EH.net database has prices for most Federal bonds, often in different marketplaces (useful for detecting currency effects in the early decades, before there was an established national currency). The Sylla data also has a wealth of municipal bond prices, far more than appeared in History of Interest Rates. And, although very fragmentary before 1850, the Sylla database has some of the very earliest corporate bond quotes I could find (canals around 1830).

Martin has a variety of municipal bonds from the 1850s, and Massachusetts or Boston bonds from 1798 (with gaps).

Smith and Cole have prices for the US 3s monthly from late 1795, now superseded by the Sylla and the Hall, Payne and Sargent collections.

Macaulay has railroad bond yield indexes back to 1857. Use with caution. Before computing any returns or other quantities, study his chart on p. 91. Why is the dashed line different from the others, and what are the implications of the fact that the dashed line does not appear in the much-cited Table 10, whereas the others do?

GFD has a US government bond index back to the early 1790s.

I constructed an aggregate bond index from 1793, all Federal through 1825, then adding municipals, then in 1832 adding corporate bonds. Municipal and corporate bonds carry the index through the interregnum after President Jackson paid off the Federal debt (~1835 to ~1842). The municipal bond portion captures the crisis of the 1840s when some states defaulted on interest payments. The aggregate index continues through 1871 and beyond, see previous BONDS post.

Evaluation

I assumed that Siegel’s selection of the lowest yielding bond, along with the use of only a single bond, would be problematic. But as it turned out, prior to 1857, my new aggregate index based on observed bond prices produced about the same cumulative return as his single-bond, yield-extraction procedure. (This is also the period where even Siegel’s data show bonds returning about the same as stocks over multiple decades.)

The big gap in our respective bond return series begins in 1862. Siegel did not adjust for the gold premium. Specifically, Federal bonds, Massachusetts bonds, and Boston bonds paid interest in gold coin; most others paid interest in greenbacks, which were legal tender. At the peak in 1864, a gold dollar was worth over 2.5X a greenback dollar, even as those greenbacks could be used to purchase more of these gold-paying bonds (because legal tender). It was the later 1870s before the gold premium sunk to de minimus levels. Accordingly, nominal yields on gold-paying bonds were depressed, because prices were high to reflect the value investors placed on being paid in gold, leading Siegel to calculate a misleadingly low total return on these bonds after 1862. And not just Siegel: everybody.

The other point where our series differ is around the Panic of 1837. The Boston or Massachusetts bond that Siegel got from History of Interest Rates, which Homer had gotten from Martin, scarcely traded, according to the Sylla database. Martin’s “quotes” were annual highs and lows, with Siegel’s yield calculated from the midpoint. That midpoint scarcely varied year to year at the depths of the crisis (wonks may wish to consult the Schwert (1990) paper on the “Working effect”). Accordingly, Siegel’s series materially misestimates the volatility of bond returns during that long ago crisis.

The relevant working papers for this era are:
pre-1857: https://papers.ssrn.com/sol3/papers.cfm ... id=3260733
post 1857, the gold premium, and problems with Macaulay: https://papers.ssrn.com/sol3/papers.cfm ... id=3269683
Dr. McQuarrie,

Thank you for the interesting data on historical corporate and government bond returns. As far as I can tell, though, all the corp bond data was for corporate bond returns for long-term (i.e. 15 years or greater) corporate bonds. Do you have any idea where to get total return data for intermediate-term corporate bonds for the period of, say, 1899 to the early 1970s (ideally monthly or at least quarterly). I have the Durand intermediate-term corporate bond data for 1952-57 (quarterly) and then I have the Ibbotson and Goetzmann intermediate corp bond total return data (albeit only annually) for 1947 to 1978 from their equity risk premium paper but I have no quarterly or monthly data for the period 1958 up to the 1973 inception of the Bloomberg Barclays data set; I also totally lack any data at all (annual, quarterly, monthly, or otherwise) for the period 1900 to 1946. Even GFD has (as far as i know) no data for intermediate-term corporate returns for this period; they do publish yields (albeit not total returns) monthly for S&P AAA-rated 15-year railroad and 15-year industrial bonds from 1937 to the 1990s but one, these are just yields and not total return, two, they have no data for BAA or BBB rated intermediate-term corporate bonds, three, they have no data for pre-1937, and fourth (and most importantly) 15 years is hardly good proxy for intermediate-term; I tend to think of "intermediate-term" as anywhere from 2 or 3 years until maturity all the way up to maybe 11 or 12 years until maturity.

As per your PM there likely were few if any corporate intermediate-term bonds issued before the 1930s but also as per your PM there would've been aged non-callable corp bonds with, say, only 3 or 4 or 6 or 10 years left until maturity that had originally been issued as, say, 30 or 50 year bonds and I'd be very curious to know what the yields and monthly (or at least quarterly) returns on such bonds were; intermediate-term corp data from before 1973 is one of the few areas we are totally lacking much data on at all. Siamond and Longinvest (among others) have done yeoman service in calculating a monthly total return series for long Treasuries, intermediate Treasuries, and short-term Treasuries (see the "Historical bond returns: From Rates to Returns" thread on this very forum); you yourself have also done some very useful work in this area (especially for the pre-1926 timeframe) but a similar monthly return series for intermediate-term corporate bonds does not (AFAIK) exist anywhere.

GFD does publish something called the "Dow Jones Corporate Bond Return Index" (GFD ticker _DJCBTD ) which it has at weekly intervals from Apr 1915 To Dec 1996 and daily from Dec 1996 to present but I have no idea if this is a broad corporate bond index (which would include both long-term, intermediate-term, and short-term corporate bonds and in theory would tend on average overall to be either smack dab in the middle of the intermediate-term range in duration/maturity or at the very least at the long end of the intermediate-term range in duration and maturity) or just another long-term corporate bond index (i.e. with a maturity of at least 15 or 20 years or more) similar to the ones from S&P, Moody's, or Ibbotson/SBBI. On top of this is the fact that even if I did have access to GFD (I unfortunately don't) we can't exactly publish any of its data or redistribute them here for obvious copyright reasons.

Do you know of any sources for monthly or quarterly total return data (or even just yield data) for a broad investment-grade intermediate-term corporate bond index for the periods above (1900-1951 and 1958 to 1972)? I did check the 1942 Durand paper you mentioned in your PM (presumably this was "Basic Yields of Corporate Bonds 1900-1942" ) but alas, it has the same issue as the intermediate-term corporate yield data in Burton Malkiel's "The Term Structure of Interest Rates: Expectations and Behavior Patterns" or for that matter for the corporate bond yield curve data from the 1941-1970 Federal Reserve BMS paper; to wit, it only gives the corporate bond yield curve for one part of the year (the beginning or middle of February of each year); the corporate bond yield curve data taken from Salomon Brothers and Scudder, Stevens, and Clark that was used in the 1983 Federal Reserve Bank of Dallas paper by Ulman and Wood (said paper is titled "Monetary Regimes and the Term Structure of Interest Rates" and extends the data in time from 1965 in Malkiel's book above all the way to 1980) frustratingly has the exact same issue; it only gives the yield curve as it existed in late February or early March of each year.

Is there any existing source of intermediate-term corporate bond return data with any finer/more detailed granularity than annually that you are aware of or have come across in your research/studies?
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

SimpleGift wrote: Sat Jul 24, 2021 7:52 pm
McQ wrote: Sat Jul 24, 2021 5:14 pm I constructed an aggregate bond index from 1793, all Federal through 1825, then adding municipals, then in 1832 adding corporate bonds...(snip)...The aggregate index continues through 1871....
One question that's been back of mind when reading through your papers: How many securities were actually involved in the construction of these early 19th-century asset return indexes? No doubt U.S. securities markets were quite thin in the 1790s, and then deepened through the 1800s — but is this quantified anywhere in your papers? Apologies if I've missed it somewhere.

A quick search turned up the chart below in a 1999 paper by Rosseau and Sylla — but I'm sure that with the new research by you and others, many more debt and equity securities have been identified to add to an updated chart. Instead of by regional markets, a graphic showing number of debt vs. equity securities over time might be more informative.
  • Image
Interesting factoid: The paper indicates that by 1803, fully half of all the U.S. securities issued to date were held by European investors. For those in the old countries, the U.S. was apparently the hot emerging (frontier?) market investment of that era.
Actually, I didn't add to the Sylla count, except for some western railroads after 1850. In fact, I subtracted anything that didn't trade regularly, and then later, anything with too small a capitalization (<$500K, when the median pre-railroads was about $1M, later, pre-war, about $3M); and even later, removed insurance companies (unreliable prices, not reflecting the December 1835 fire in NYC or the later 1845 fire). So for stock counts in my indexes, the Sylla Rousseau chart is a high ceiling.

Nonetheless, I think I have plenty of stocks, especially after 1810 or so, when the 1st BUS was dissolved. Before that point, counts shrink from two dozen back to four at the start in 1793. After that point, I have dozens, and by the 1830s, I have scores. The big move up in counts came in the 1820s, BTW, as clearly visible in the chart.

Bond counts are lower, but still ample I believe:
1. Before Jackson paid off the debt about 1835, I have 3 - 6 Federal bonds
2. From the later 1820s, I have a dozen municipal bonds
3. from the 1840s, a dozen or more corporate bonds.
4. I have all the long federal bonds from 1842 to the 1880s
5. From 1857, I have 1-3 dozen long municipal bonds through 1897
6. The Macaulay long corporate bond index has from 12 to 45 bonds, 20+ by 1871

More to the point, the coverage in terms of total capitalization / face amounts is much better than in terms of counts, sometimes even a census (of long Federal bonds), and generally 80% plus, in terms of "stocks trading regularly on the major exchanges of the day."

If anyone needs the stock or bond detail spreadsheets, with stock by stock and bond by bond tabulations, just email me at the address on the paper.
This now superseded working paper goes into the gory details of working with the Sylla database and has counts and comparison of counts with other studies:
https://papers.ssrn.com/sol3/papers.cfm ... id=3209440
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

McQ wrote: Sun Jul 25, 2021 12:35 pm This now superseded working paper goes into the gory details of working with the Sylla database and has counts and comparison of counts with other studies: https://papers.ssrn.com/sol3/papers.cfm ... id=3209440
Thank you. This linked paper had what was missing in terms of security counts and coverage.

But it also sheds interesting light on your research challenges, which would seem almost insurmountable to someone less dedicated and conscientious. For those not reading this paper, a flavor of what you were dealing with:
Edward McQuarrie wrote:In summary, one can’t take newspaper stock prices in this early era at face value. There’s no substitute for rolling up the sleeves and getting your hands dirty by probing the biographies of the firms behind the prices in the Sylla et al. database, and when indicated, going back to the archived newspapers themselves.
An impressive research effort, indeed.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Alpha4 wrote: Sat Jul 24, 2021 11:34 pm
McQ wrote: Sat Jul 24, 2021 5:14 pm Historical BOND returns 1793 to 1871

Here the sources for yields mostly overlap with those in the previous BONDS post, and I’ll be brief in describing them here, without links.
First and foremost is the Hall, Payne and Sargent database, which has prices of Federal bonds from the very beginning, with terms and amount outstanding year by year, as well as prices.
Dr. McQuarrie,

Thank you for the interesting data on historical corporate and government bond returns. As far as I can tell, though, all the corp bond data was for corporate bond returns for long-term (i.e. 15 years or greater) corporate bonds. Do you have any idea where to get total return data for intermediate-term corporate bonds for the period of, say, 1899 to the early 1970s (ideally monthly or at least quarterly). I have the Durand intermediate-term corporate bond data for 1952-57 (quarterly) and then I have the Ibbotson and Goetzmann intermediate corp bond total return data (albeit only annually) for 1947 to 1978 from their equity risk premium paper but I have no quarterly or monthly data for the period 1958 up to the 1973 inception of the Bloomberg Barclays data set; I also totally lack any data at all (annual, quarterly, monthly, or otherwise) for the period 1900 to 1946. Even GFD has (as far as i know) no data for intermediate-term corporate returns for this period; they do publish yields (albeit not total returns) monthly for S&P AAA-rated 15-year railroad and 15-year industrial bonds from 1937 to the 1990s but one, these are just yields and not total return, two, they have no data for BAA or BBB rated intermediate-term corporate bonds, three, they have no data for pre-1937, and fourth (and most importantly) 15 years is hardly good proxy for intermediate-term; I tend to think of "intermediate-term" as anywhere from 2 or 3 years until maturity all the way up to maybe 11 or 12 years until maturity.

As per your PM there likely were few if any corporate intermediate-term bonds issued before the 1930s but also as per your PM there would've been aged non-callable corp bonds with, say, only 3 or 4 or 6 or 10 years left until maturity that had originally been issued as, say, 30 or 50 year bonds and I'd be very curious to know what the yields and monthly (or at least quarterly) returns on such bonds were; intermediate-term corp data from before 1973 is one of the few areas we are totally lacking much data on at all. Siamond and Longinvest (among others) have done yeoman service in calculating a monthly total return series for long Treasuries, intermediate Treasuries, and short-term Treasuries (see the "Historical bond returns: From Rates to Returns" thread on this very forum); you yourself have also done some very useful work in this area (especially for the pre-1926 timeframe) but a similar monthly return series for intermediate-term corporate bonds does not (AFAIK) exist anywhere.

GFD does publish something called the "Dow Jones Corporate Bond Return Index" (GFD ticker _DJCBTD ) which it has at weekly intervals from Apr 1915 To Dec 1996 and daily from Dec 1996 to present but I have no idea if this is a broad corporate bond index (which would include both long-term, intermediate-term, and short-term corporate bonds and in theory would tend on average overall to be either smack dab in the middle of the intermediate-term range in duration/maturity or at the very least at the long end of the intermediate-term range in duration and maturity) or just another long-term corporate bond index (i.e. with a maturity of at least 15 or 20 years or more) similar to the ones from S&P, Moody's, or Ibbotson/SBBI. On top of this is the fact that even if I did have access to GFD (I unfortunately don't) we can't exactly publish any of its data or redistribute them here for obvious copyright reasons.

Do you know of any sources for monthly or quarterly total return data (or even just yield data) for a broad investment-grade intermediate-term corporate bond index for the periods above (1900-1951 and 1958 to 1972)? I did check the 1942 Durand paper you mentioned in your PM (presumably this was "Basic Yields of Corporate Bonds 1900-1942" ) but alas, it has the same issue as the intermediate-term corporate yield data in Burton Malkiel's "The Term Structure of Interest Rates: Expectations and Behavior Patterns" or for that matter for the corporate bond yield curve data from the 1941-1970 Federal Reserve BMS paper; to wit, it only gives the corporate bond yield curve for one part of the year (the beginning or middle of February of each year); the corporate bond yield curve data taken from Salomon Brothers and Scudder, Stevens, and Clark that was used in the 1983 Federal Reserve Bank of Dallas paper by Ulman and Wood (said paper is titled "Monetary Regimes and the Term Structure of Interest Rates" and extends the data in time from 1965 in Malkiel's book above all the way to 1980) frustratingly has the exact same issue; it only gives the yield curve as it existed in late February or early March of each year.

Is there any existing source of intermediate-term corporate bond return data with any finer/more detailed granularity than annually that you are aware of or have come across in your research/studies?
The short answer is, no I do not know of any index of intermediate corporate bond returns for the periods indicated. And the first part of the longer answer is to explain why it is almost certain that you will not find one. The second, more hopeful part of the answer is instructions on how you or Siamond or Longinvest or other BH could compile the first such intermediate index without undue effort.

[And BTW you are correct, all my indexes are of long bonds. After 1897, I screened for fifteen years or more remaining; before 1897, 7 or more years.] I believe long bonds to be the proper comparison point for stocks.

Why no intermediate indexes?

The question of intermediate bonds allows me to highlight how different older bond markets were relative to more recent markets. In this case, “older” is anything pre-1973 Lehman Brothers/Barclays/Bloomberg indexes, as explained below. Next, I’ll define the “ancient” bond market as the period before the 1840s. Prior to that point, many bonds had no maturity date, making the question of “intermediate” moot. Federal and state and city bonds weren’t perpetuities quite like the British Consols; they typically had a date attached, after which they could be redeemed at the option of the issuer. By the 1850s, more and more bonds did have a stated maturity, and I believe that is as far back as you will be able to take any intermediate index that you choose to construct.

So why are there no existing intermediate indexes? Answer: prior to the 1930s there were no tranches at issuance as we expect to see today (e.g., a 7, 10, and 30 year maturity sold on the same day). A corporation issued a bond at one maturity, almost always longer than 10 years. In fact, a maturity curve can be traced over time (based on Macaulay’s inclusions).
1. In the 1850s, 20 years at issue was typical;
2. After the Civil War, first 30 years and then 40-year maturities came on the scene;
3. By the 1880s, 50-year bonds become more and more common
4. By the 1890s, 100-year bonds proliferate;
5. From 1900 through the mid-1920s, 40- and 50-year maturities were regularly issued within the dominant sector (railroads), becoming the “norm.”
6. Only in the 1930s, first among utilities, as these replaced RR as the dominant bond market, did the familiar 30-year maturity become the normative value for “long.”
7. And the later 1930s are when tranches begin to appear, so that you start to see intermediate bonds-at-time-of-issue. But these still were not common as late as the early 1970s.

As you reminded me, just because all bonds were long bonds at issuance doesn’t mean that seven or five year bonds can’t be found; it only takes 33 years for a 40-year bond to become an intermediate bond, and ten years for a fifteen-year bond to become a five-year bond. And I believe you can find a good number of such, as explained in Part II. Because: prior to the 1920s most bonds were not callable.

To understand why maturities were so long around 1900, and intermediate issues scarce, we need to understand the bond buyer of that era. Today many BH would hesitate to tie up their funds for 50 years in the bonds of a single corporation. But in that older era, Federal bonds were disappearing as the government ran huge surpluses year after year (!). The municipal market was small. Interest rates had been falling for a generation. Older higher coupon bonds had matured. The bond buyer was seeking a steady income that would continue for a long time at the initial coupon rate. They thought they were locking in a rate that might not be available in the future. Bond sellers stretched maturities to accommodate; they thought they were locking in a (low) cost of funds. Corporations were the only sellers in volume, and that is how they came to dominate the bond market, while buyer preference explains why all bonds at issue were long bonds.

Things began to change after 1920, when industrials and utilities began to take a greater share of the bond market. Unlike a railroad, with its roadbed and infrastructure good for centuries, nobody wanted to lend money for 50 years to construct a factory to make a biscuit or a newfangled automobile. Ten year bonds were about all the risk the buyer could stomach in the industrial sector. As the utility industry matured under regulation, fifteen and ultimately 30-year bonds gained a market among buyers.

One further caveat: you mentioned an interest in not just AAA bonds but BBB bonds. That is going to be difficult except for a decade or two after 1930, when the Depression manufactured large numbers of BBB (and worse) bonds out of what had been AAA or AA bonds (aka fallen angels).

From 1909 when ratings began through 1930, virtually all large, regularly traded bonds on the NYSE were rated AAA or AA. No one wanted to lend for 50 years to anything less creditworthy (counts and breakdowns in the W. Braddock Hickman books on NBER.org). After the war-led recovery, BBB bonds again became scarce until the 1960s, and above all until after 1980, when Michael Milken worked his magic, and for the first time, not-so-creditworthy bonds became marketable in volume.

Part II: how to build your own intermediate bond index

[need another day for this how-to piece, but didn’t want to leave your question hanging that long]
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
longinvest
Posts: 5682
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical asset returns before 1970: Where to find

Post by longinvest »

Just to clarify, our bond fund simulator doesn't calculate "index" returns! It simply tries to emulate a ladder-like bond fund which could have possibly (up to a point) been held by an investor. Every 6 months, coupons and the proceeds of the sold rung are reinvested into a new rung. The interesting mathematical property of that fund model is that if the reinvestment yield (longer rate) and the selling yield (shorter rate) are accurate, calculated returns are self-correcting over time, even if estimated yields* for other maturities are inaccurate. Of course, all this is highly dependent on the accuracy of available longer and shorter yields. As we don't take capitalization weightings into account, our model doesn't represent an index. We also make many simplifying assumptions: bonds are bought at par, they're never called, there are no defaults, the investor can buy any amount of a bond (fractional bond), there's no bid-ask spread, there are no transaction costs, etc.

* We use a linear approximation.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

longinvest wrote: Sun Jul 25, 2021 4:33 pm Just to clarify, our bond fund simulator doesn't calculate "index" returns! It simply tries to emulate a ladder-like bond fund which could have possibly (up to a point) been held by an investor. Every 6 months, coupons and the proceeds of the sold rung are reinvested into a new rung. The interesting mathematical property of that fund model is that if the reinvestment yield (longer rate) and the selling yield (shorter rate) are accurate, calculated returns are self-correcting over time, even if estimated yields* for other maturities are inaccurate. Of course, all this is highly dependent on the accuracy of available longer and shorter yields. As we don't take capitalization weightings into account, our model doesn't represent an index. We also make many simplifying assumptions: bonds are bought at par, they're never called, there are no defaults, the investor can buy any amount of a bond (fractional bond), there's no bid-ask spread, there are no transaction costs, etc.

* We use a linear approximation.
Hello longinvest: : indeed, I did not understand the details of what was done in the bond simulator. In my research I have instead taken the historian’s role: to reconstruct the return actually received by a bond buyer who bought the market according to a rule (all large, liquid, maturity > X …) and held until a bond no longer met the rule, or was downgraded out of investment grade, or defaulted.

That said, in my work I made some but not all the same choices as you, parsing your post as follows to compare:
“We also make many simplifying assumptions:
1. bonds are bought at par
• Rarely the case historically; my hypothetical buyer buys in the after market at whatever price, and marks to market each year;

2. they're never called
• not a problem before the 1920s, when most bonds were not callable; big problem 1935 to 1945, when huge numbers of bonds were called, and bonds often traded above the call price, and call schedules became highly complex (if doing history, you’ll need access to the Moody’s manuals to find these elaborate call schedules);

3. there are no defaults,
• Big no no for historians of the bond return actually received; as you can imagine, significant defaults occurred in the 1930s, but also before 1920; bond prices for mark to market plunged after downgrade / default;

4. the investor can buy a bond of any amount (fractional bond),
• Yep, standard historical assumption

5. there are no transaction costs, etc.”
• Likewise, standard assumption.

In short: I have no expertise in simulations, but have learned something about financial market history, which I hope will be helpful to you. See what you think of my "how to" post coming in a little bit.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Part II: how to build your own intermediate bond index

[continuation of reply to alpha4]

Goal: build a monthly or quarterly index of intermediate corporate bond yields and returns; or per longinvest’s caveats, reconstruct what a buy and hold investor would have experienced if they bought all bonds that had less than 10 years maturity and held all such bonds until a bond had less than 36 months remaining.

Materials
1. Macaulay’s list of bonds, his pp. A5 to A177, from NBER.org
2. My bond detail spreadsheet (email me at the address in the front of the SSRN papers)

Source of prices
The Commercial & Financial Chronicle, using its Financial Review up to about 1920, and then the Quotation & Bank Record, both available at FRASER (links up thread). (For 1857 to 1865, use either the American Railroad Journal (hathitrust.org) or the New York Times (your library) for prices). Sudden plunges in price, suggestive of default, can be followed up in the CFC’s Investor Supplement, which will indicate default, reorganization, litigation, or whatever applies.

Coupon payment dates
Generally every six months; payment dates for Macaulay bonds in his description, mine in my spreadsheet.

Procedure
Macaulay and I provide you with the sample frame: the long bonds at issue that will ultimately become intermediate bonds (and are large enough and active enough that you will likely succeed in getting monthly or quarterly quotes). I’d suggest adding a bond to the intermediate portfolio when it reaches nine years and 11 months to maturity, and removing it after the coupon that marks three years remaining. So, for instance, Macaulay’s bond #1, Hudson River RR 7s of 1869 would enter the portfolio in March 1859, when it had 9 years and 11 months to run, and exit after the February 1866 coupon.

Prices will be the midpoint of monthly high and low, same as Cowles for stocks and Macaulay for bonds and stocks.

Alternative Procedure
Whoa, you say: didn’t sign up for that level of archival drudgery? Don’t need to build the portfolio back to the beginning of the corporate bond market in the 1850s? I have a workaround, if you are content with a 1900 start date.

Let’s return to Durand, and your disappointment that his data was only annual. How then was Sidney Homer able to use Durand to produce a table of monthly yields on prime corporate bonds with 30-year maturities (his Table 45, 47 in History of Interest Rates)? Answer: 1. Homer called Durand’s average of monthly high and low prices for the first three months in the year a “February” yield. 2. Then Homer used the monthly fluctuations of Macaulay’s index over the course of the year to map a monthly yield series onto Durand’s annual point values. Example: suppose the Durand 30-year yield in February #1 was 3.10%; in February #2, 3.40%. In between, Macaulay’s index rose through May, fell below the starting point through November, recovered through January, spiked in February. Homer would have varied the 30-year yield monthly in accordance, starting at 3.10 and getting up to 3.40 in a parallel pattern.

You can’t use Macaulay index values because they are for long bonds. Instead, from 1926 you take the monthly fluctuations of the Ibbotson SBBI intermediate government bond as your map, and do the same as Homer.

And before 1926? In 1900, you find the several Macaulay bonds from the 1890s with maturities in the 1903 to 1910 range (#39, #45, #56, etc.). You update the membership as need be. You construct a mini index of their prices/yields monthly from the sources above and then you do for Durand’s 3 – 10 year yields what Homer did for his 30-year yields, fabricating a monthly series onto Durand’s first quarter averages.

Now, beyond this point I cannot help you further. I would mention that carving out intermediate bonds as an asset class is a modern activity. No one before the 1970s would have been able to answer, “Why do that?” Bonds before that time were a means to lock in an income stream for decades. In fact, although I offer the advice above in good faith, I won’t be surprised if you find that shorter maturity bonds fitting the intermediate description didn’t trade often enough to get a monthly index; and bonds with only a few years left may not have traded regularly enough to get even a quarterly index. Before the modern mutual fund era, there was no natural buyer for shortish bonds.

That said, there is nothing standing between you and an intermediate corporate bond index back to 1900, or even 1850, except drudgery and tedium. :happy All the needed materials are freely available on digital platforms. And I, for one, and despite the skeptical sentiments above, will be an eager audience for your findings.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Historical OTHER ASSETS returns 1793 to 1871

Measuringworth.com again has gold prices, foreign exchange, and inflation rates back to the beginning. Books by Warren and Pearson explore the value of gold re-expressed in terms of purchasing power parity: e.g., https://www.google.com/books/edition/Go ... 49DwAAQBAJ. They also have a wholesale price index and prices of various commodities, generally back to the 1700s.

Smith and Cole have a commodities index and some data on public land sales.

This is the final post planned for US historical data; please do comment if you can suggest additions, which I promise to add and acknowledge. Questions about where to find certain types of data, or whether these even exist, are also welcome.

Next post begins to examine the available international data.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Alpha4
Posts: 176
Joined: Tue Apr 17, 2012 8:47 pm

Re: Historical asset returns before 1970: Where to find

Post by Alpha4 »

McQ wrote: Mon Jul 26, 2021 11:47 am Part II: how to build your own intermediate bond index

[continuation of reply to alpha4]

Goal: build a monthly or quarterly index of intermediate corporate bond yields and returns; or per longinvest’s caveats, reconstruct what a buy and hold investor would have experienced if they bought all bonds that had less than 10 years maturity and held all such bonds until a bond had less than 36 months remaining.

Materials
1. Macaulay’s list of bonds, his pp. A5 to A177, from NBER.org
2. My bond detail spreadsheet (email me at the address in the front of the SSRN papers)

Source of prices
The Commercial & Financial Chronicle, using its Financial Review up to about 1920, and then the Quotation & Bank Record, both available at FRASER (links up thread). (For 1857 to 1865, use either the American Railroad Journal (hathitrust.org) or the New York Times (your library) for prices). Sudden plunges in price, suggestive of default, can be followed up in the CFC’s Investor Supplement, which will indicate default, reorganization, litigation, or whatever applies.

Coupon payment dates
Generally every six months; payment dates for Macaulay bonds in his description, mine in my spreadsheet.

Procedure
Macaulay and I provide you with the sample frame: the long bonds at issue that will ultimately become intermediate bonds (and are large enough and active enough that you will likely succeed in getting monthly or quarterly quotes). I’d suggest adding a bond to the intermediate portfolio when it reaches nine years and 11 months to maturity, and removing it after the coupon that marks three years remaining. So, for instance, Macaulay’s bond #1, Hudson River RR 7s of 1869 would enter the portfolio in March 1859, when it had 9 years and 11 months to run, and exit after the February 1866 coupon.

Prices will be the midpoint of monthly high and low, same as Cowles for stocks and Macaulay for bonds and stocks.

Alternative Procedure
Whoa, you say: didn’t sign up for that level of archival drudgery? Don’t need to build the portfolio back to the beginning of the corporate bond market in the 1850s? I have a workaround, if you are content with a 1900 start date.

Let’s return to Durand, and your disappointment that his data was only annual. How then was Sidney Homer able to use Durand to produce a table of monthly yields on prime corporate bonds with 30-year maturities (his Table 45, 47 in History of Interest Rates)? Answer: 1. Homer called Durand’s average of monthly high and low prices for the first three months in the year a “February” yield. 2. Then Homer used the monthly fluctuations of Macaulay’s index over the course of the year to map a monthly yield series onto Durand’s annual point values. Example: suppose the Durand 30-year yield in February #1 was 3.10%; in February #2, 3.40%. In between, Macaulay’s index rose through May, fell below the starting point through November, recovered through January, spiked in February. Homer would have varied the 30-year yield monthly in accordance, starting at 3.10 and getting up to 3.40 in a parallel pattern.

You can’t use Macaulay index values because they are for long bonds. Instead, from 1926 you take the monthly fluctuations of the Ibbotson SBBI intermediate government bond as your map, and do the same as Homer.

And before 1926? In 1900, you find the several Macaulay bonds from the 1890s with maturities in the 1903 to 1910 range (#39, #45, #56, etc.). You update the membership as need be. You construct a mini index of their prices/yields monthly from the sources above and then you do for Durand’s 3 – 10 year yields what Homer did for his 30-year yields, fabricating a monthly series onto Durand’s first quarter averages.

Now, beyond this point I cannot help you further. I would mention that carving out intermediate bonds as an asset class is a modern activity. No one before the 1970s would have been able to answer, “Why do that?” Bonds before that time were a means to lock in an income stream for decades. In fact, although I offer the advice above in good faith, I won’t be surprised if you find that shorter maturity bonds fitting the intermediate description didn’t trade often enough to get a monthly index; and bonds with only a few years left may not have traded regularly enough to get even a quarterly index. Before the modern mutual fund era, there was no natural buyer for shortish bonds.

That said, there is nothing standing between you and an intermediate corporate bond index back to 1900, or even 1850, except drudgery and tedium. :happy All the needed materials are freely available on digital platforms. And I, for one, and despite the skeptical sentiments above, will be an eager audience for your findings.
Dr. McQuarrie,

Again, thank you! This actually sounds like it might be feasible (at least back to 1900 or so) even if it might take a long while (unless we get, say, perhaps fifteen or sixteen Bogleheads together, we each do a separate few years, then have one of the other persons involved also do that same time period as a kind of check on each of our work. Once the two people were in agreement on the data; that data would then become "official" for that given span of a few years for the intermediate corp yields and returns monthly database. Doing it this would reduce the time needed compared to only one or two people doing the whole thing. Many hands make light work).

A couple more questions, if you don't mind:

1. How many bonds would we need to track and get yields/returns on to make up a viable index? Ten? Twelve? Eighteen? Twenty? Twenty-five? Thirty? Fifty? Didn't some of the original Ibbotson, Macaulay, and your own indexes only have say seven or eight bonds or ten bonds at some points in time? If we only had to track fifteen or twenty bonds (of course, some bonds would be added each month at 9 yrs 11 months to maturity and some would be removed each month once they reached 3 years to maturity but overall it would still be a lot easier to do this with fewer bonds than with several dozen) for a given series it would be a lot easier to compile said series than if we had to track, say, forty or fifty of them.

2. Using the Ibbotson SBBI Intermediate Govt data to help with interpolating monthly yields is a reasonable and good idea...but two concerns I have: First, how and where can we get this monthly yield data? Second, what about the few time periods when IT Treasury yields either:

A. fell when corp yields of the same maturity rose,

B. stayed flat or flattish (i.e. didn't rise or fall that much) when corp yields of the same maturity rose,

C. both IT corp yields and IT Treasury yields rose but corp yields rose a good bit or a heck of a lot more both in absolute terms and in terms of amount of rise vs where they started from.

IIRC the first two situations may have happened during the "credit crunch", "bunch of banks failing in a short time frame" and "things in the economy are getting worse and fast" parts of the Great Depression (e.g. various times in late 1929, some of 1930, mid to late 1931, much of the first half of 1932, and late 1932 to March 1933) and the latter (situation C. above) happened from late 1955 to the very first part of early 1957. For that matter, what happens when IT corp yields fall more over a given time period than IT gov't yields (would this not have happened during at least some parts of the mid-1932 to late 1932, April 1933-early 1937, and 1939-1941 economic and/or stock market recovery periods as risk appetites in the markets began to return)? Would one yield series rising while the other was falling (or vice versa) not interfere a bit with trying to map an interpolated monthly yield series from one to the other?

3. Can and should we use the Durand 1952-1957 quarterly corporate yield series (it has 3 yr, 4 yr, 5 yr, 6 yr, 8 yr, 10 yr, 12 yr, 15 yr, and all the way out to 20, 30 and 40 yr yields....these are all given as quarterly yields but are actually monthly average yields for Feb, May, Aug, and Nov of each year) as a sort of checksum on our work for the 1952-57 period? This paper is available free on JSTOR. For that matter, shouldn't we also use that 1983 FRB Dallas paper I mentioned in another post as a checksum of our work for the 1966-1972 period (it only has February average corp bond yields for the yield curve but it at least does have 5, 10, and 15 year yields averages for that one month each year; similarly; the Malkiel book on interest rate term structure has this data for 1900 to 1965 as an average of Jan, Feb, and March yields for 1900-1942, Jan and Feb yields for 1943-1951, and Feb only yields for 1952-1965.

4. The Burton Malkiel work mentioned above; I can scan a copy of the yield curve data if anyone wants it but I think that won't be necessary as it is available free on both NBER and from FRASER--the latter in the BMS 1941-70 data--also mentions--and shows--a GMAC obligation (1 year notes to 25 year bonds) yield curve for Oct 1959, Dec 1959, and IIRC very early 1960. I know GMAC paper traded on the open market since the early 1920s (IIRC circa 1921 or 1922); would it be one of the bonds we would want to use (although admittedly I don't know how long of paper was issued by GMAC each year --and whether it was callable or not--except that they almost always issued some commercial paper and sub-1 year obligations each and every year; not sure about how frequently they issued anything from, say, 2 years to maturity until 20 or 25 years until maturity...at least during the early years i.e. the 1920s and 1930s).

5. I rechecked my list of all the data GFD has and it turns out their AAA industrial monthly 15-year yield series goes all the way back to 1-31-1900; it is only the BBB, A, and AA industrial yield series' that only go back to 1-1-1937; also, their monthly utility 15-year yield series for AA, A, and BAA only go back to 1-1-37 as well (as it happens they do publish an AAA utility yield series all the way back to 1900 but I don't know if it is a long-term series i.e. 20+ years or a 15-year series but I strongly suspect it is the former). They also have railroad bond yield series' (or serieses? Is that the plural for series?) for everything from AAA yields (back to 1857) and then various S&P and Moody's railroad AA, A, BAA, BA, BBB, BB, and B yields back to dates ranging from 1871 to 1937 (some start in 1919 as well) but I again strongly suspect these are for bonds with a minimum of 20+ years remaining to maturity--since that is how the broad Moody's AAA and BAA yield series monthly from 1919 were done--and given that many railroads--the UP and Kansas City Southern among others--issued even forty, fifty and hundred year bonds during the late 1800s or early 1900s; a 20+ year index would likely have an average maturity of over 30 years which isn't intermediate-term or even anything close to it. Still, all things considered, it would be great if we had someone here who could access GFD at least for the 15 year yield datasets; it could act as a checksum of sorts on our longer-term yields (say 9 or 10 or 11 years) to see if we are in the ballpark. I don't mean we should just copy their data (it would be illegal to violate their copyrights and besides....15 years is too long to be an IT bond anyway) but at least it could serve as a helpful "black box" sort of test (like how IBM's BIOS was black-boxed back in the 1980s by various competitors to create PC-compatible clone BIOSes without ever actually having direct access to IBM's source code and thus not violating any copyright or patent laws) to see if any of our longer-term IT corp yields were close enough or whether there were any major "that stands out like a sore thumb and you may want to recheck that" outliers.

6. As I believe I mentioned in another post, Professor Ibbotson (or Dr. Ibbotson if he prefers....I've never met said gentleman so I don't know which he would wish to go by) himself did compile an IT corp bond TR series for 1947-1978 for his and Goetzmann's book titled "The Equity Risk Premium - Essays and Explorations"; said series is given annually in this book for all of these years. While this series would be a bit different from and broader than ours (he classified everything from 1 year left to maturity to 14 years 11 months left to maturity as "intermediate-term" ) it is similar in spirit and concept to what we are trying to do. The reason I mention this is that while this study is referred to as an "unpublished Ibbotson corporate bond study" in several other books, if Professor Ibbotson still has a copy of said detailed study he may be willing to share it...and for all I know it may contain monthly (or at least quarterly) TR data rather than just annual data as in the "The Equity Risk Premium - Essays and Explorations" book (sometimes the underlying data in a study, research paper, or research project is a lot more detailed than what actually gets fully published). This study would've been done in 1981 or 1982 so it would be some 40 years old by now so it's not like we would be asking for fresh, recent, presumably only-available-for-pay-from-Ibboston-Associates data (although if Ibbotson Associates does indeed actually have monthly or quarterly IT corp return data for 1947 to 1978 available for sale I might buy it myself if it isn't unreasonably priced). Anyhow, I don't know how generous and willing Professor Ibbotson is regarding sharing old historical data from decades ago but it can't hurt to ask; the worse that happens is he says no or ignores us (or his data is indeed only annual) and even then we're no worse off than we were before. Even a quarterly IT corp TR data series would act as a valuable checksum for us for the 1947 to 1972 period (we theoretically shouldn't need the 1973-1978 data since we already have the Bloomberg Barclays IT corp indexes past that point). I don't think a (relatively speaking) nobody like me could or should contact Professor Ibbotson but perhaps he might be willing to correspond with either you or siamond or longinvest.

7. Finally, if I recall correctly you said any data other than AAA corp bond yields (for any maturity from 50 or 100 years down to 2 or 3 years) would be difficult or impossible to find before the 1930s; lower rated bonds were few and far between because (quite understandably) very few people were willing to lend to anything less than AAA credits for any reasonably long period of time (especially for industrials rather than railroads or utilities). How then did Moody's compile a BAA long-term (20+ years) corporate yield series from 1-1-1919 onwards if this is the case? Is there any way we can find out some of the bonds they used for this series? If so, maybe some of them would be ten or ten-and-a-half years or less by, say, 1929 or 1930. I really don't want to use only AAA bonds because I could swear I read something on SSRN a while back about how the Ibbotson SBBI LT Corp TR data was actually not entirely accurate and was somewhat distorted in terms of returns vs the real world because it had a higher proportion of AAA-rated bonds than the broad investment-grade LT corp bond market as a whole had and was thus more of a play on interest-rate risk than the broad investment-grade LT corp bond market was (and conversely, LESS of a credit risk play than said actual market was). I do know that there were "junk" bonds that traded in the 1910s and 1920s because I have a copy of the Basile and Rockoff study with monthly below-investment grade bond yields from 1910 through 1955; I believe their index they used to construct their yield series was made up of a mix of the longer end of intermediate-term below-investment grade bonds and the whole range of long-term below-investment grade bonds (i.e. their index used "junk bonds" from ten years to maturity up to forty or fifty years until maturity); presumably if below-investment grade corp bonds traded during this time period then lower rated but still investment grade (i.e. not AAA but still AA, A, BAA, or BBB) corp bonds did trade as well.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

Alpha4 wrote: Tue Jul 27, 2021 11:07 pm
McQ wrote: Mon Jul 26, 2021 11:47 am Part II: how to build your own intermediate bond index

[continuation of reply to alpha4]

Goal: build a monthly or quarterly index of intermediate corporate bond yields and returns; or per longinvest’s caveats, reconstruct what a buy and hold investor would have experienced if they bought all bonds that had less than 10 years maturity and held all such bonds until a bond had less than 36 months remaining.
Dr. McQuarrie,

Again, thank you! This actually sounds like it might be feasible (at least back to 1900 or so) even if it might take a long while (unless we get, say, perhaps fifteen or sixteen Bogleheads together, we each do a separate few years, then have one of the other persons involved also do that same time period as a kind of check on each of our work. Once the two people were in agreement on the data; that data would then become "official" for that given span of a few years for the intermediate corp yields and returns monthly database. Doing it this would reduce the time needed compared to only one or two people doing the whole thing. Many hands make light work).

A couple more questions, if you don't mind:

1. How many bonds would we need to track and get yields/returns on to make up a viable index? Ten? Twelve? Eighteen? Twenty? Twenty-five? Thirty? Fifty? Didn't some of the original Ibbotson, Macaulay, and your own indexes only have say seven or eight bonds or ten bonds at some points in time? If we only had to track fifteen or twenty bonds (of course, some bonds would be added each month at 9 yrs 11 months to maturity and some would be removed each month once they reached 3 years to maturity but overall it would still be a lot easier to do this with fewer bonds than with several dozen) for a given series it would be a lot easier to compile said series than if we had to track, say, forty or fifty of them.

2. Using the Ibbotson SBBI Intermediate Govt data to help with interpolating monthly yields is a reasonable and good idea...but two concerns I have: First, how and where can we get this monthly yield data? Second, what about the few time periods when IT Treasury yields either:

A. fell when corp yields of the same maturity rose,

B. stayed flat or flattish (i.e. didn't rise or fall that much) when corp yields of the same maturity rose,

C. both IT corp yields and IT Treasury yields rose but corp yields rose a good bit or a heck of a lot more both in absolute terms and in terms of amount of rise vs where they started from.

IIRC the first two situations may have happened during the "credit crunch", "bunch of banks failing in a short time frame" and "things in the economy are getting worse and fast" parts of the Great Depression (e.g. various times in late 1929, some of 1930, mid to late 1931, much of the first half of 1932, and late 1932 to March 1933) and the latter (situation C. above) happened from late 1955 to the very first part of early 1957. For that matter, what happens when IT corp yields fall more over a given time period than IT gov't yields (would this not have happened during at least some parts of the mid-1932 to late 1932, April 1933-early 1937, and 1939-1941 economic and/or stock market recovery periods as risk appetites in the markets began to return)? Would one yield series rising while the other was falling (or vice versa) not interfere a bit with trying to map an interpolated monthly yield series from one to the other?

3. Can and should we use the Durand 1952-1957 quarterly corporate yield series (it has 3 yr, 4 yr, 5 yr, 6 yr, 8 yr, 10 yr, 12 yr, 15 yr, and all the way out to 20, 30 and 40 yr yields....these are all given as quarterly yields but are actually monthly average yields for Feb, May, Aug, and Nov of each year) as a sort of checksum on our work for the 1952-57 period? This paper is available free on JSTOR. For that matter, shouldn't we also use that 1983 FRB Dallas paper I mentioned in another post as a checksum of our work for the 1966-1972 period (it only has February average corp bond yields for the yield curve but it at least does have 5, 10, and 15 year yields averages for that one month each year; similarly; the Malkiel book on interest rate term structure has this data for 1900 to 1965 as an average of Jan, Feb, and March yields for 1900-1942, Jan and Feb yields for 1943-1951, and Feb only yields for 1952-1965.

4. The Burton Malkiel work mentioned above; I can scan a copy of the yield curve data if anyone wants it but I think that won't be necessary as it is available free on both NBER and from FRASER--the latter in the BMS 1941-70 data--also mentions--and shows--a GMAC obligation (1 year notes to 25 year bonds) yield curve for Oct 1959, Dec 1959, and IIRC very early 1960. I know GMAC paper traded on the open market since the early 1920s (IIRC circa 1921 or 1922); would it be one of the bonds we would want to use (although admittedly I don't know how long of paper was issued by GMAC each year --and whether it was callable or not--except that they almost always issued some commercial paper and sub-1 year obligations each and every year; not sure about how frequently they issued anything from, say, 2 years to maturity until 20 or 25 years until maturity...at least during the early years i.e. the 1920s and 1930s).

5. I rechecked my list of all the data GFD has and it turns out their AAA industrial monthly 15-year yield series goes all the way back to 1-31-1900; it is only the BBB, A, and AA industrial yield series' that only go back to 1-1-1937; also, their monthly utility 15-year yield series for AA, A, and BAA only go back to 1-1-37 as well (as it happens they do publish an AAA utility yield series all the way back to 1900 but I don't know if it is a long-term series i.e. 20+ years or a 15-year series but I strongly suspect it is the former). They also have railroad bond yield series' (or serieses? Is that the plural for series?) for everything from AAA yields (back to 1857) and then various S&P and Moody's railroad AA, A, BAA, BA, BBB, BB, and B yields back to dates ranging from 1871 to 1937 (some start in 1919 as well) but I again strongly suspect these are for bonds with a minimum of 20+ years remaining to maturity--since that is how the broad Moody's AAA and BAA yield series monthly from 1919 were done--and given that many railroads--the UP and Kansas City Southern among others--issued even forty, fifty and hundred year bonds during the late 1800s or early 1900s; a 20+ year index would likely have an average maturity of over 30 years which isn't intermediate-term or even anything close to it. Still, all things considered, it would be great if we had someone here who could access GFD at least for the 15 year yield datasets; it could act as a checksum of sorts on our longer-term yields (say 9 or 10 or 11 years) to see if we are in the ballpark. I don't mean we should just copy their data (it would be illegal to violate their copyrights and besides....15 years is too long to be an IT bond anyway) but at least it could serve as a helpful "black box" sort of test (like how IBM's BIOS was black-boxed back in the 1980s by various competitors to create PC-compatible clone BIOSes without ever actually having direct access to IBM's source code and thus not violating any copyright or patent laws) to see if any of our longer-term IT corp yields were close enough or whether there were any major "that stands out like a sore thumb and you may want to recheck that" outliers.

6. As I believe I mentioned in another post, Professor Ibbotson (or Dr. Ibbotson if he prefers....I've never met said gentleman so I don't know which he would wish to go by) himself did compile an IT corp bond TR series for 1947-1978 for his and Goetzmann's book titled "The Equity Risk Premium - Essays and Explorations"; said series is given annually in this book for all of these years. While this series would be a bit different from and broader than ours (he classified everything from 1 year left to maturity to 14 years 11 months left to maturity as "intermediate-term" ) it is similar in spirit and concept to what we are trying to do. The reason I mention this is that while this study is referred to as an "unpublished Ibbotson corporate bond study" in several other books, if Professor Ibbotson still has a copy of said detailed study he may be willing to share it...and for all I know it may contain monthly (or at least quarterly) TR data rather than just annual data as in the "The Equity Risk Premium - Essays and Explorations" book (sometimes the underlying data in a study, research paper, or research project is a lot more detailed than what actually gets fully published). This study would've been done in 1981 or 1982 so it would be some 40 years old by now so it's not like we would be asking for fresh, recent, presumably only-available-for-pay-from-Ibboston-Associates data (although if Ibbotson Associates does indeed actually have monthly or quarterly IT corp return data for 1947 to 1978 available for sale I might buy it myself if it isn't unreasonably priced). Anyhow, I don't know how generous and willing Professor Ibbotson is regarding sharing old historical data from decades ago but it can't hurt to ask; the worse that happens is he says no or ignores us (or his data is indeed only annual) and even then we're no worse off than we were before. Even a quarterly IT corp TR data series would act as a valuable checksum for us for the 1947 to 1972 period (we theoretically shouldn't need the 1973-1978 data since we already have the Bloomberg Barclays IT corp indexes past that point). I don't think a (relatively speaking) nobody like me could or should contact Professor Ibbotson but perhaps he might be willing to correspond with either you or siamond or longinvest.

7. Finally, if I recall correctly you said any data other than AAA corp bond yields (for any maturity from 50 or 100 years down to 2 or 3 years) would be difficult or impossible to find before the 1930s; lower rated bonds were few and far between because (quite understandably) very few people were willing to lend to anything less than AAA credits for any reasonably long period of time (especially for industrials rather than railroads or utilities). How then did Moody's compile a BAA long-term (20+ years) corporate yield series from 1-1-1919 onwards if this is the case? Is there any way we can find out some of the bonds they used for this series? If so, maybe some of them would be ten or ten-and-a-half years or less by, say, 1929 or 1930. I really don't want to use only AAA bonds because I could swear I read something on SSRN a while back about how the Ibbotson SBBI LT Corp TR data was actually not entirely accurate and was somewhat distorted in terms of returns vs the real world because it had a higher proportion of AAA-rated bonds than the broad investment-grade LT corp bond market as a whole had and was thus more of a play on interest-rate risk than the broad investment-grade LT corp bond market was (and conversely, LESS of a credit risk play than said actual market was). I do know that there were "junk" bonds that traded in the 1910s and 1920s because I have a copy of the Basile and Rockoff study with monthly below-investment grade bond yields from 1910 through 1955; I believe their index they used to construct their yield series was made up of a mix of the longer end of intermediate-term below-investment grade bonds and the whole range of long-term below-investment grade bonds (i.e. their index used "junk bonds" from ten years to maturity up to forty or fifty years until maturity); presumably if below-investment grade corp bonds traded during this time period then lower rated but still investment grade (i.e. not AAA but still AA, A, BAA, or BBB) corp bonds did trade as well.
All: This reply will be slightly lame because I haven't quite figured out BBC code for interspersing quotes and replies. Bear with me, I'll use Alpha4's numbering. Also, we have exchanged PM, which is why some of his points are not addressed here.

1. How many bonds?
-Short answer: one dozen minimum, not much benefit beyond three dozen. Macaulay/I had 2-6 dozen, but these were more volatile long bonds, intermediate should be fine at small dozens

2. problems with interpolation/source yields
-see Appendix A-13 in SBBI for monthly yields. All your concerns are valid, but only imply that your interpolations will--Horror--be off by a couple of basis points in each monthly value. De minimus, I would say, relative to the countless other sources of error in the data.

3. Malkiel is almost certainly using Durand. Checksums are good; but don't let the perfect be the enemy of the good.

4. Let another BH team do mortgage and agency bonds. [oops--I misread Alpha4, who meant General Motors Acceptance Corporation, not GNMA. Text kept to set up book reference]: Homer and Johannesen, The Price of Money, is the book to buy on Ebay/Alibris. Do something, rather than throw up your hands that you can't do it all, intermediate corporate, agency mortgages, equipment serials, yada yada

5. Most of what you describe as GFD data is probably NBER data--they have lots of yield series. Try their Macrohistory / interest rates subsection.
-if it starts in 1857, it is Macaulay data;
-if it starts in 1900, it is either S&P back-constructed data, or Durand/Hickman data
-if it starts in 1919, it is Moody's data
-confirm: almost all such NBER series are long bond data

6. Roger Ibbotson is still around. If he will respond, he will respond to any good faith email that shows intelligence and knowledge of the area. Won't matter if it is an obscure academic like me or a complete unknown BH. No harm in asking; use all your adult wisdom and keep it brief.

7. Here, you are going to have to buckle down and read the W. Braddock Hickman books on NBER. No raw data, but great perspective. When I reconstructed even the Aaa and Aa Moody's averages from 1932 (membership in CFC issues, see my paper below), I found they had to use tiny bonds, and bonds not traded on the NYSE, to fill out those yield averages.
Bonus question: What was a utility for Moody's 1919? Answer: a street car company

Problems with the SBBI LT corporate index are laid out here: https://papers.ssrn.com/sol3/papers.cfm ... id=3740190
Abstract:

For many years the Stocks, Bonds, Bills & Inflation yearbook has served as the primary source for calibrating historical asset returns. However, uneasiness has grown about its depiction of corporate bond returns prior to the second World War. I document problems with the source data used in the SBBI and replace its flawed dataset with new observations of bond prices from 1926 to 1946 for a sample of several hundred large bonds listed on the NYSE and rated investment-grade. I find that the SBBI overstates corporate bond returns in the 1930s and accordingly, gives an unreliable estimate of the premium received for owning investment grade corporate bonds rather than government bonds during the prewar years. To extend the analysis I collected additional bond price data from 1946 to 1974 and find that the SBBI also overstates corporate bond returns in the 1960s. The problem again stems from a reliance on flawed yield series in place of observing bond prices. I combine the new data with existing data to examine the corporate bond premium from 1909 through 2019. Using ten-year rolling returns, over the past century I find the average premium earned on long maturity corporate bonds to be small, about 15 basis points annualized. For many of the ten-year rolls, the premium was instead a deficit: a bond investor would have done better owning only long government bonds. The small and fitful premium contrasts with the yield spread on investment-grade bonds, which was always positive and substantial throughout the period. Because the premium has been much more variable, the relative size of the yield spread does not seem to be predictive of whether a premium will subsequently be earned and how much. Results are interpreted in terms of the importance of regime change in financial history: sometimes corporate bonds outperform government bonds, sometimes they do not, just as sometimes stocks outperform bonds, and sometimes they do not, contra Siegel (2014). The idea of regime change challenges the notion that a mean computed over a longer rather than a shorter interval contributes any additional predictive power to the study of asset returns over human horizons.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

International STOCK and BOND returns 1900 to 1972

For some time there were two primary sources; and unfortunately, both have been tightly protected behind paywalls. Now at last there is a publicly available data set which can be downloaded without charge. It will be described last.

First, some context. The United States has some of the oldest financial markets. Great Britain is older by a century, and Amsterdam by two centuries (for stocks); but there is not a third entry on that list (France comes closest; the metric is whether a historical record for both stocks and bonds has been reconstructed). Financial markets in the US long predate the emergence of the US as a global power, an emergence that did not really get under way until well after the Civil War, and was not cemented until after World War I.

Conversely, by 1900 there were almost two dozen stock markets for which trading records have been reconstructed, and not a few Western European nations had markets that were decades old by 1900. Accordingly, this was the start date selected by Elroy Dimson, Paul Marsh, and Mike Staunton, in London, for their international data series described below. But as just described, 1900 is an arbitrary cutoff date of convenience, no more a watershed or event threshold than was the 1926 date in the US. As discussed in a subsequent post, Global Financial Data takes each market back as far as the data go, accepting that the World ex-USA index will have a steadily shrinking number of components as one moves back beyond 1900. Likwise, Jorda et al. had no difficulty taking multiple markets back to 1870.

Here are the major sources:

A. Triumph of the Optimists / Credit Suisse Global Investment Returns Yearbook (annual)

In 2002, using data through 2000, at the very peak of the greatest stock market boom the world had ever seen, the Dimson team published Triumph, whose title said it all. Findings were quite consistent with Siegel’s US findings in Stocks for the Long Run.

Then, as many Bogleheads will remember, the dotcom boom crashed. After a brief recovery, the Great Financial Crisis knocked US stocks back down. It was 2013 before US stocks recovered their (real) highs.

Fortunately, the Dimson team kept at it, and began to publish annual updates in yearbook form distributed through Credit Suisse. Accordingly, here in 2021 no serious student of international returns should start with Triumph, now almost two decades out of date. Instead, start with as recent a Credit Suisse yearbook as you can find. Expect to pay retail (address at the end); I did not see any used copies on Amazon today. (However, an abbreviated summary of older yearbooks can be downloaded: search Credit Suisse yearbook 2016, for instance.)

In addition to incorporating the vicissitudes of 2001 – 2009, the yearbooks have improved on Triumph in several ways:
1. Dimson et al. gradually added what might be called the losers—markets that had performed poorly, or even been destroyed, like Russia in 1917. Their current estimates of international asset returns are considerably less triumphant than before.
2. Triumph had included a World return but not a World ex-USA return. The World return had about a 50% weight for the US, obscuring how differently international stocks had performed relative to the US.
3. Tables have been expanded, i.e., the equity premium is called out rather than having to be calculated from other entries.
4. Each year, the authors have added some discussion, bringing the yearbooks more in line with the comprehensive account supplied by the Ibbotson SBBI in the US (i.e., discussing value versus growth and sundry other investment topics).

The key limitation of the yearbooks: tabled returns are provided by decade, and only for decades ending in zero, excepting the last few trailing non-zero years. The dataset available behind the paywall is annual, but Siamond tells me that paywall is tight indeed. In general then, the Credit Suisse yearbooks will be educational, but can’t be tapped for backtesting, unlike the free sources elsewhere in this guide.

Here is a possible workaround, for the obsessive student who won’t take no for an answer (and allowing for the fact that this verbiage may have to be removed by moderators, as I am not an attorney and do not want to get anyone into trouble).

With few exceptions, the individual country returns are all sourced in the work of other scholars, as noted with references by the Dimson team, and those papers will likely have annual series, whose values can be placed in a database. Then you can apply publicly available foreign exchange and US inflation data to approximate the real return for the US investor, similar to that reported in the yearbooks. However, it will be more difficult to produce a properly weighted simulacrum of the World ex-USA index, since the capitalization weights aren’t published.

Alternatively: use the Jorda et al. dataset described below, or access Global Financial Data through your library, if available. Jorda et al. have about two-thirds the country count of Dimson et al., while GFD has almost all.

If you do choose to purchase the Yearbook, you will find, for each country, charts and tables that give stock returns, bond returns, bill returns and the foreign exchange rate relative to the US, along with derived measures like the equity premium or the real return.

As of 2021, copies of the yearbook can be purchased by emailing Patricia Rowham at prowham@london.edu. New editions come out around February of the year.

B. Global Financial Data
GFD aspires to compile international data to at least the same level of granularity as the Dimson team. Data have been collected independently for most of markets in the Credit Suisse yearbooks. Results vary somewhat between the two efforts (Bryan Taylor of GFD has shared annual returns data with me, which I am not able to share in their raw form).
GFD data also live behind a paywall, but if your library subscribes, then you should be able to analyze at will.

C. Jorda et al., The Real Return on Everything
The Jorda team, based in part in Europe, has data from 1870 for the US and fifteen other markets (not all these go back to 1870; countries are incorporated once data on all four of the covered assets are available—i.e., bills, bonds, equities and housing). Like the Dimson team and GFD, the database is still being built, not only by updating with later data, etc., but also by incorporating new countries. A few days before this post, I downloaded the R5 release; it had data through 2015.

A cautionary note: to some extent, the focus of the Jorda et al. effort, and the unique contribution, is the return on housing / real estate; data on bills, bonds and equities are there for comparison. And neither Siamond nor I am fond of all the metrics featured in their paper. But none of that really matters, since the raw data can be downloaded, and the documentation appears very good, so you know what you are getting in each series, and can calculate desired values as needed.

Links, repeated from an earlier post:
The full paper: https://economics.harvard.edu/files/eco ... s28533.pdf
The database, including documentation links: https://www.macrohistory.net/database/

Perspective

International data have come a long way in the past three decades. From the time Siegel wrote and before, there had been widespread concern that US investment returns might be unrepresentative of the range of returns received internationally. After all, since World War I, in geopolitical terms the US has bestrode the world like a colossus. Investors in less fortunate nations, it was suspected, might have recorded less fortunate returns.

Interestingly, the early evidence on international returns seemed to allay these concerns, indicating at first glance that the US was far from alone in recording strong positive returns. To this day, in book editions, Siegel generally feels he can call on international data for support for his thesis.

I do not interpret currently available international data as supportive of Siegel. I would trace an evolution in thinking as follows:
1. Attempts in the 1990s to supplement US data with international data sets were fairly primitive. Survivorship bias was strong, both in some of the historically compiled data sets not yet subject to searching scrutiny, and in the selection of which countries to report.
2. Likewise, as noted in the discussion of Triumph, data were collected during the worldwide boom of the mid- to late-1990s, and were sometimes not taken back very many decades when the market climate had been harsher.
3. Finally, there was a tendency, still the case with the Jorda et al. reported results, to calculate world returns with the US included, rather than World ex-USA returns. Weight for the US was typically 50% or more, thus obscuring any differences between the US and other markets.

It was only about a decade after the publication of Triumph that a more nuanced, and rather darker picture began to emerge. The Japanese market was still way down from its peak, even after two decades; European markets did not enjoy the same recovery from the 2008-09 crisis as the US; and most important, the Credit Suisse yearbooks kept adding more countries, with most of the additions showing worse performance than the US.

By 2020, the Credit Suisse yearbook could report: “A dollar invested in US equities in 1900 resulted in a terminal value of USD 1937 … An equivalent investment in stocks from the rest of the world gave a terminal value of USD 179 … less than a tenth of the US value.”

As long time Bogleheads will know, the question of the proper allocation to international securities has launched a thousand posts. See compilation by Robot Monster here: viewtopic.php?p=6132935#p6132935

The importance of the data sets reviewed in this post is to help Bogleheads go beyond the MSCI data, which generally begin after 1970, in assessing the performance of international assets. Of course, past performance, whether excellent or poor, is no guarantee that future results will continue likewise. Nonetheless, it still seems better to learn about the past rather than to remain ignorant of it.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

McQ wrote: Wed Jul 28, 2021 12:36 pm International data have come a long way in the past three decades.
Dr. Mcquarrie, one data series that hasn't yet been mentioned in this thread is Paul Schmelzing's research on interest rates going back eight centuries to the 1300s. He constructs a global real rate series that is GDP-weighted, splicing together rates for sovereign long-term bond issuers across different countries and centuries, focusing on the "risk free" rate over time.

The most recent version of his paper is here, and an Excel spreadsheet with all of his data is found here.

We recently had a Forum discussion thread on this study: Deep History of Interest Rates, from 1311 to 2018
  • Image
    NOTE: Global real interest rates are GDP-weighted and 7-year rolling averages. Data source: Bank of England
Schmelzing's study is probably not as immediately useful to Boglehead investors as more recent stock and bond returns from the 19th and 20th centuries, but it does give one a long-term historical perspective on today's very low interest rates — and reinforces that, at least on the scale of centuries, interest rates have not shown any tendency to mean revert.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

SimpleGift wrote: Wed Jul 28, 2021 4:06 pm
McQ wrote: Wed Jul 28, 2021 12:36 pm International data have come a long way in the past three decades.
Dr. Mcquarrie, one data series that hasn't yet been mentioned in this thread is Paul Schmelzing's research on interest rates going back eight centuries to the 1300s. He constructs a global real rate series that is GDP-weighted, splicing together rates for sovereign long-term bond issuers across different countries and centuries, focusing on the "risk free" rate over time.

The most recent version of his paper is here, and an Excel spreadsheet with all of his data is found here.

We recently had a Forum discussion thread on this study: Deep History of Interest Rates, from 1311 to 2018
  • Image
    NOTE: Global real interest rates are GDP-weighted and 7-year rolling averages. Data source: Bank of England
Schmelzing's study is probably not as immediately useful to Boglehead investors as more recent stock and bond returns from the 19th and 20th centuries, but it does give one a long-term historical perspective on today's very low interest rates — and reinforces that, at least on the scale of centuries, interest rates have not shown any tendency to mean revert.
Hello SimpleGift: I had a link planned to your forum on Schmelzing for the very next post, but not a bad thing that you beat me to the punch, and put the key chart here in this forum. Speaking of which, this Bogleheads newbie is envious of your apparent ability to drop charts into your posts with the appearance of ease. How do you do it? Save a chart in your Google Docs account, make it public, and paste the link? Save someone else's chart as an image and make it public? Or is there a bit more to it than that? (since I don't see too many other posters doing the same). And if this is all in the FAQ, just point me there.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

McQ wrote: Wed Jul 28, 2021 4:35 pm Speaking of which, this Bogleheads newbie is envious of your apparent ability to drop charts into your posts with the appearance of ease. How do you do it? ...(snip)... And if this is all in the FAQ, just point me there.
Yes, there's an article in the Boglehead's wiki with basic instructions: Posting Images in the Bogleheads Forum

If you have trouble getting the hang of it, please feel free to PM me, to avoid sidetracking this excellent historical returns thread.
AlohaJoe
Posts: 6609
Joined: Mon Nov 26, 2007 1:00 pm
Location: Saigon, Vietnam

Re: Historical asset returns before 1970: Where to find

Post by AlohaJoe »

McQ wrote: Wed Jul 28, 2021 12:36 pm However, it will be more difficult to produce a properly weighted simulacrum of the World ex-USA index, since the capitalization weights aren’t published.
FWIW, I believe the Dimson-Marsh-Staunton research uses GDP-weighting and not market capitalization weighting but it has been a while since I looked so I may be misremembering.

Part of the Jordà-Schularick-Taylor team has also assembled a historical index of market capitalization. See "The Big Bang: Stock Market Capitalization in the Long Run?" by Dmitry Kuvshinov and Kaspar Zimmermann. A few years ago siamond was in contact with one of the authors (Kuvshinov, I think?) and they expressed willingness to make the underlying annual series public after the paper had finished peer review and been published in a journal. But it is still in review & publication hell.... :shock:

The paper: https://dkuvshinov.com/wp-content/uploa ... t.pdf?Nov1
A VOX column by the authors on the paper: https://voxeu.org/article/big-bang-stoc ... n-long-run
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

AlohaJoe wrote: Wed Jul 28, 2021 9:47 pm
McQ wrote: Wed Jul 28, 2021 12:36 pm However, it will be more difficult to produce a properly weighted simulacrum of the World ex-USA index, since the capitalization weights aren’t published.
FWIW, I believe the Dimson-Marsh-Staunton research uses GDP-weighting and not market capitalization weighting but it has been a while since I looked so I may be misremembering.

Part of the Jordà-Schularick-Taylor team has also assembled a historical index of market capitalization. See "The Big Bang: Stock Market Capitalization in the Long Run?" by Dmitry Kuvshinov and Kaspar Zimmermann. A few years ago siamond was in contact with one of the authors (Kuvshinov, I think?) and they expressed willingness to make the underlying annual series public after the paper had finished peer review and been published in a journal. But it is still in review & publication hell.... :shock:

The paper: https://dkuvshinov.com/wp-content/uploa ... t.pdf?Nov1
A VOX column by the authors on the paper: https://voxeu.org/article/big-bang-stoc ... n-long-run
Thanks Aloha Joe--I did not know about the Kuvshinov paper--one more payback for my efforts here. As of the March 2021 release, the Jorda et al. public file still does not have capitalization weights. (I can tell you stories about peer review from my former career--my last prestigious peer reviewed paper required seven (7) iterations and two editorial teams to finally appear in print. Took years, and the material left on the cutting room floor made up much of a (non peer-reviewed) book published later. Sigh. It's not the primary reason I was moved to retire early, but ... )

More importantly here, although the Dimson team formerly used GDP weights, on or about the 2017 yearbook they were able--by their own lights--to claim capitalization weighting. The GFD returns for world ex-USA are also capitalization weighted. Re the Jorda et al. data, a crude weighting scheme ex-USA would be 3 for the UK, 2 for France, Germany, and Japan, and 1 for everything else. Totally wrong in the details, but the result might be off by only basis points from a true capitalization-weighted return.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
AlohaJoe
Posts: 6609
Joined: Mon Nov 26, 2007 1:00 pm
Location: Saigon, Vietnam

Re: Historical asset returns before 1970: Where to find

Post by AlohaJoe »

McQ wrote: Wed Jul 28, 2021 11:44 pm More importantly here, although the Dimson team formerly used GDP weights, on or about the 2017 yearbook they were able--by their own lights--to claim capitalization weighting.
Yeah, looks like I misrembered. (Or mixed it up with the bond returns, which does still use GDP weighting.) You got me curious so I went back through old editions of the Credit Suisse Global Investment Returns Yearbooks to find out when the change happened.

It appears to have been the 2013 edition. In the section on "The world equity premium: survivorship bias" they write:
This year, we have made enhancements to the country weightings, and we have sought to eliminate survivorship bias. In previous years, while our aim was to weight countries in the world equity index by their market capitalizations, the latter were unavailable prior to 1968, so that until then, GDP weights were used instead. This year, thanks to new research and newly discovered archive material, we have been able to estimate market capitalizations for every country since 1900. Since, in aggregate, world equities are held in proportion to their market capitalizations, this allows us to compute a new and more accurate measure of the world index.
As expected, this is a somewhat academic improvement: switching from GDP-weighting to market-cap-weighting only changed the returns by 0.17% per year over the 60+ year period market-cap weights had previously been unavailable.

Unfortunately the Yearbook doesn't really identify what "new research" they're relying on but presumably it is some proprietary GFD stuff :(
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

AlohaJoe wrote: Thu Jul 29, 2021 3:11 am
McQ wrote: Wed Jul 28, 2021 11:44 pm More importantly here, although the Dimson team formerly used GDP weights, on or about the 2017 yearbook they were able--by their own lights--to claim capitalization weighting.
Yeah, looks like I misrembered. (Or mixed it up with the bond returns, which does still use GDP weighting.) You got me curious so I went back through old editions of the Credit Suisse Global Investment Returns Yearbooks to find out when the change happened.

It appears to have been the 2013 edition. In the section on "The world equity premium: survivorship bias" they write:
This year, we have made enhancements to the country weightings, and we have sought to eliminate survivorship bias. In previous years, while our aim was to weight countries in the world equity index by their market capitalizations, the latter were unavailable prior to 1968, so that until then, GDP weights were used instead. This year, thanks to new research and newly discovered archive material, we have been able to estimate market capitalizations for every country since 1900. Since, in aggregate, world equities are held in proportion to their market capitalizations, this allows us to compute a new and more accurate measure of the world index.
As expected, this is a somewhat academic improvement: switching from GDP-weighting to market-cap-weighting only changed the returns by 0.17% per year over the 60+ year period market-cap weights had previously been unavailable.

Unfortunately the Yearbook doesn't really identify what "new research" they're relying on but presumably it is some proprietary GFD stuff :(
Good documentation to have! I was only able to conclude "by 2017." And I had not tracked the impact on returns. Here's a speculation for why the change didn't affect returns that much. First, as is the case for within-country market indexes (e.g., S&P 500), where just as a few very large stocks account for a substantial portion of the index, so a few large markets (US, UK) account for a substantial portion of the world index. And second, market size is probably fairly tightly associated with GDP; and much more tightly associated than is market size with performance. That means a weight vector (GDP) that looked something like this: [10, 8, 5, 3, 2, 1, 1, 0.5, 0.1], and which was very noisy relative to performance, got replaced with another noisy vector (market cap) that looked something like this: [9, 8.5, 6, 2, 3, 1, 1, 0.2, 0.2]. No way multiplication by the second vector is going to change much.

Next, it is interesting to me that Jorda et al. could not find those market capitalization weights; such weights are still not found in their spreadsheet as of this year, some years into their initiative. So, either very proprietary, or perhaps, open to doubt.

Last, I'm guessing that it was a decrease in rate of 0.17%?
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
siamond
Posts: 6010
Joined: Mon May 28, 2012 5:50 am

Re: Historical asset returns before 1970: Where to find

Post by siamond »

AlohaJoe wrote: Wed Jul 28, 2021 9:47 pmPart of the Jordà-Schularick-Taylor team has also assembled a historical index of market capitalization. See "The Big Bang: Stock Market Capitalization in the Long Run?" by Dmitry Kuvshinov and Kaspar Zimmermann. A few years ago siamond was in contact with one of the authors (Kuvshinov, I think?) and they expressed willingness to make the underlying annual series public after the paper had finished peer review and been published in a journal. But it is still in review & publication hell.... :shock:

The paper: https://dkuvshinov.com/wp-content/uploa ... t.pdf?Nov1
A VOX column by the authors on the paper: https://voxeu.org/article/big-bang-stoc ... n-long-run
Oh, they did complete the market cap work (I mean, besides the review process)? I had lost track... Thanks a lot for sharing.

And yes, I was in contact with Dmitry Kuvshinov indeed. He's very approachable by e-mail.
AlohaJoe
Posts: 6609
Joined: Mon Nov 26, 2007 1:00 pm
Location: Saigon, Vietnam

Re: Historical asset returns before 1970: Where to find

Post by AlohaJoe »

McQ wrote: Thu Jul 29, 2021 12:54 pm Last, I'm guessing that it was a decrease in rate of 0.17%?
That is correct. It went from something like (numbers approximate because I'm too lazy to look them up again) 5.3% to 5.15% when switching from GDP weights to market-cap weights.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

AlohaJoe wrote: Thu Jul 29, 2021 7:51 pm
McQ wrote: Thu Jul 29, 2021 12:54 pm Last, I'm guessing that it was a decrease in rate of 0.17%?
That is correct. It went from something like (numbers approximate because I'm too lazy to look them up again) 5.3% to 5.15% when switching from GDP weights to market-cap weights.
Not to worry about approximate. Those values would be "World," not "World ex-USA," I'm guessing? By 2020, World ex-USA was 4.4% ...
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Historical asset returns before 1970: Where to find

Post by McQ »

International STOCK and BOND returns 1600 to 1900

Here the only comprehensive source is Global Financial Data, excepting the Jorda et al. coverage of returns from 1870 to 1900. For capitalization weights since 1870, see the Kuvshinov and Zimmerman paper in the post by AlohaJoe just upthread; and kudos to AlohaJoe for spotting a paper published just last month(!). Kuvshinov and Zimmerman are part of the Jorda et al. network, sharing various datasets.

Amsterdam stock returns make up the GFD World index from about 1600 (no bond returns); British stock and bond returns are added about 1700. Their world ex-USA index begins in 1792. Start dates for the other countries in the GFD and Jorda et al. databases as of mid-2021 are as follows.

-Named countries are In Dimson et al. from 1900
-Left number shows December anchor for when GFD has both stocks and bonds
-Right number shows Jorda et al. December anchor for stocks (if country included)

Australia 1861 1869
Austria 1921 --
Belgium 1831 1869
Canada 1852 --
Denmark 1873 1872
Finland 1913 1895
France 1802 1869
Germany 1836 1869
Ireland 1862 1919
Italy 1924 1869
Japan 1885 1885
Netherlands 1899 1899
New Zealand 1862 --
Norway 1914 1880
South Africa 1860 --
Spain 1880 1899
Sweden 1870 1870
Switzerland 1913 1899
UK 1700 1870
World ex-USA 1792 [world results]
Note: Dimson and Jorda et al. (from 1870) have Portugal as well.

Otherwise, Yale hosts a dataset on London returns prior to 1900 https://som.yale.edu/faculty-research/o ... k-exchange; but see papers by Grossman, who were a key source for Jorda et al. (e.g., https://www.econstor.eu/bitstream/10419 ... 58739X.pdf) In addition, a review of the citations in Jorda et al. show that several of their other sources may extend back beyond 1870.

Paul Schmelzing has a database on interest rates back to the Middle Ages, see links in SimpleGift’s post just upthread, to both the paper, the data, and a discussion thread here at BH.

Perspective

Pre-1900 stock and bond returns outside the US are still coming into focus; this era is not as well understood as the post 1900 era, which has benefitted from what at this point has become decades of scrutiny by the Dimson team. I hazard a guess that there is still a good amount of survivorship bias in the current crop of pre-1900 datasets.

Conclusion

This is the final self-authored post I planned for the thread. I may add vignettes from time to time, or new datasets as I come across them. And of course, I am hoping others will continue to post queries, criticisms, or insights.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
User avatar
LadyGeek
Site Admin
Posts: 95696
Joined: Sat Dec 20, 2008 4:34 pm
Location: Philadelphia
Contact:

Re: Historical asset returns before 1970: Where to find

Post by LadyGeek »

Thank you very much! I've been keeping the wiki updated with your posts. See: Historical and expected returns (Data prior to 1970)
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.
User avatar
SimpleGift
Posts: 4477
Joined: Tue Feb 08, 2011 2:45 pm

Re: Historical asset returns before 1970: Where to find

Post by SimpleGift »

McQ wrote: Fri Jul 30, 2021 1:19 pm Pre-1900 stock and bond returns outside the US are still coming into focus...
Historical financial data can seem a bit dry and remote to modern-day investors, so apologies for something slightly different, but one of the more fascinating studies to come along in recent years traces the stock return history of a single company in France over six centuries, from 1372 to 1946. The study can be accessed via a summary article, the academic paper itself, and a previous Forum discussion.
Goetzmann et al. wrote:In this paper, we use the unique experience of an extraordinarily long-lived company as a laboratory to test the present value relation. This company, called the Honor del Bazacle, was created in Toulouse in 1372 to operate watermills. Shares of the company were owned and regularly traded by individuals and institutions over the course of its history. Following its conversion to a hydroelectric generating company in 1888, the company listed on the Paris Stock Exchange and traded there until nationalization in 1946. We hand collected share price and dividend data over a nearly five-century period during which the company operated, and integrated it with previously collected data up to 1450. The result is an economic record of a corporate enterprise that extends from the Middle Ages to modern times.
  • Bazacle Company share prices and dividends from 1372 to 1946 (in kilograms of silver).
    Image
    NOTE: The left axis is for share prices (in black) the right one is for dividends (in grey).
    Source: Goetzmann et al.
  • Bazacle Company dam and grain mill, Garonne River, France, in 1642.
    Image
    Source: FDMF
For those interested in historical returns or the origins of corporations, the centuries-long history of this one company brings some much needed immediacy to the subject. In fact, if ever in France, one can visit the original site of the Bazacle Company today, and tour the still intact hydro-facility, the museum and waterfront promenade. Financial history tourism at its best!

Image
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Deep history of stock and bond returns

Post by McQ »

Introducing a new total bond index 1900 to 1973

At first, I wasn’t too interested in Alpha4’s request for an intermediate bond index (see our exchange upthread). But later, another project I have under way turned out to require just that: a total bond index with an intermediate duration, similar to the Lehman / Barclays / Bloomberg aggregate index. And I needed that index for the period before 1973 when the Barcap begins.

Long story short, I roughly followed the advice I gave Alpha4 upthread and constructed a predecessor to the Barcap that extends from 1973 back to 1900. Data sources and method of construction are described in this new SSRN paper: https://papers.ssrn.com/sol3/papers.cfm ... id=3947293

The table of annual returns in the Appendix of the paper can be pasted into a spreadsheet if you want to work with the total bond index. It is public use without limit except for attribution.

Here is the title and Abstract: let me know if you have questions or comments, ideally by posting to this thread.

Extending the Barcap Back to 1900: Introducing a New Total Bond Index

The Lehman / Barclay’s / Bloomberg Aggregate Bond Index dates to the 1970s. Historical back tests of investment strategies and outcomes that extend past 1973 have had to use either long bonds or short Treasuries. In terms of capturing returns over a complete market cycle, both long and short bonds are problematic relative to a total bond index, which obtains its yield from across the entire maturity range while maintaining an intermediate duration. This paper assembles eight bond series to simulate returns on a total bond index from 1900 to 1973. The individual series are for government bonds and corporate bonds, and for long, longer intermediate, shorter intermediate, and short maturities. Both the individual series and an equal-weighted composite are presented in downloadable form. Some of the series have been previously compiled from observed prices while others were compiled specifically for this project using multiple sources. A brief interpretation of the behavior of the total bond index relative to its components is offered.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
secondopinion
Posts: 6011
Joined: Wed Dec 02, 2020 12:18 pm

Re: Historical asset returns before 1970: Where to find

Post by secondopinion »

SimpleGift wrote: Fri Jul 30, 2021 4:57 pm
McQ wrote: Fri Jul 30, 2021 1:19 pm Pre-1900 stock and bond returns outside the US are still coming into focus...
Historical financial data can seem a bit dry and remote to modern-day investors, so apologies for something slightly different, but one of the more fascinating studies to come along in recent years traces the stock return history of a single company in France over six centuries, from 1372 to 1946.
Imagine how the investors who bought into this back in 1372 feel about their investment these days. :P

Seriously, it proves that companies can indeed last for a long, long time.
Passive investing: not about making big bucks but making profits. Active investing: not about beating the market but meeting goals. Speculation: not about timing the market but taking profitable risks.
Alpha4
Posts: 176
Joined: Tue Apr 17, 2012 8:47 pm

Re: Deep history of stock and bond returns

Post by Alpha4 »

McQ wrote: Mon Oct 25, 2021 11:18 am Introducing a new total bond index 1900 to 1973

At first, I wasn’t too interested in Alpha4’s request for an intermediate bond index (see our exchange upthread). But later, another project I have under way turned out to require just that: a total bond index with an intermediate duration, similar to the Lehman / Barclays / Bloomberg aggregate index. And I needed that index for the period before 1973 when the Barcap begins.

Long story short, I roughly followed the advice I gave Alpha4 upthread and constructed a predecessor to the Barcap that extends from 1973 back to 1900. Data sources and method of construction are described in this new SSRN paper: https://papers.ssrn.com/sol3/papers.cfm ... id=3947293

The table of annual returns in the Appendix of the paper can be pasted into a spreadsheet if you want to work with the total bond index. It is public use without limit except for attribution.

Here is the title and Abstract: let me know if you have questions or comments, ideally by posting to this thread.

Extending the Barcap Back to 1900: Introducing a New Total Bond Index

The Lehman / Barclay’s / Bloomberg Aggregate Bond Index dates to the 1970s. Historical back tests of investment strategies and outcomes that extend past 1973 have had to use either long bonds or short Treasuries. In terms of capturing returns over a complete market cycle, both long and short bonds are problematic relative to a total bond index, which obtains its yield from across the entire maturity range while maintaining an intermediate duration. This paper assembles eight bond series to simulate returns on a total bond index from 1900 to 1973. The individual series are for government bonds and corporate bonds, and for long, longer intermediate, shorter intermediate, and short maturities. Both the individual series and an equal-weighted composite are presented in downloadable form. Some of the series have been previously compiled from observed prices while others were compiled specifically for this project using multiple sources. A brief interpretation of the behavior of the total bond index relative to its components is offered.
Dr. McQuarrie.

Very interesting.

Your paper on this you linked to on SSRN shows annual returns; four quick questions:

1. It states that these annual returns are "January to January". Does this mean January 1st to January 1st (which I presume and which is basically the same as 12-31 to 12-31.....for instance, Jan 1st of 1960 to Jan 1st of 1961 is essentially the same as Dec 31st of 1959 to Dec 31st of 1960) or does it mean January 31st of one year to January 31st of the next?

2. Have you calculated monthly/quarterly returns as well or only annual? I saw the annual returns data in your paper on SSRN; if you do not have calculated monthly/quarterly return data can you please instead upload an Excel file (on a free hosting site online) showing your calculated monthly yields (and indicate whether they are on the 1st of the month or the end of the month) for the 4, 5, 6, 7, 8, 9, and--if I understand your methodology correctly--10 year corporate bonds (or does your "longer intermediate" corporate series stop at 9 years? )? If we have that then they can be put into something like the Longinvest/Siamond simulated bond fund return calculator and we can then get monthly and/or quarterly total returns for the two intermediate-corporate series.

3. Your "shorter intermediate" corporate series (typically) shows much smaller deviations in return than Ibbotson/Goetzmann's intermediate-term corporate series (from the "Equity Risk Premium - Essays and Explorations" book they published roughly twenty years ago) does; even your "longer intermediate" corporate series (i.e. a simulated ladder of 6, 7, 8, 9, and--I think--10 year corporate bonds) shows typicaly smaller deviations in returns vs their intermediate-term corporate series; for instance, for the years 1956 and 1957 (1956 was a year in which bond yields increased and thus bonds did relatively poorly; 1957 was just the opposite) your "longer intermediate" corporate bond return series shows returns of -1.42% and 4.85% respectively; theirs shows returns of -4.69% and 2.70% respectively. Also, if you look at their data from, say, 1947 to 1953 their returns for this category don't come that close to yours at all.

Whether this is because their "intermediate-term" series was anything from 15 years at max to (I believe IIRC) either 1 year at minimum--which could give a different maturity average weighting--especially if most of the cap weight of the bonds they used to create their series was in the 10 to 15 year range (i.e. the majority of the bonds they chose were 10-15 years until maturity and they only had a few bonds in the "1 year to 9 years and 11 months" range)--or if because some of their bonds may've had credit risk (their book did specify that they excluded all convertible bonds from this series but it didn't say if they used all AAA or if it as well included any AA, A, BAA/BB, or even "defaulted" i.e. "junk" bonds too) and the prices/yields of said bonds may have reflected this in addition to merely responding purely to changes in the interest-rate environment....whereas your corporate bond series's were purely based on yields and yield interpolations.

I also wonder about 1931. Given that your "intermediate-term Treasury" series shows a return of -6.47% and your "long Treasury" series shows a return of -6.72% (FWIW the Simba/Siamond backtest portfolio returns spreadsheet shows a return of -2.11% and -8.74% respectively for these so your returns here on Treasuries appear to be fairly close to what the spreadsheet has--especially given differences in methodology; IIRC that spreadsheet used a 3 year to 9.9 year maturity ladder for IT Treasuries and either a 10-30 year or just a "anything longer than 20 year counts as a long term Treasury" maturity for long Treasuries) for that year I find it a bit implausible that total returns on intermediate corporates would--depending on whether it was the longer intermediate or shorter intermediate series--either be basically flat (-0.02% is basically flat) or be up 1.69%. In 1931 yields on anything with even the remotest amount of credit risk spiked (and prices fell); be it junk and/or defaulted bonds, preferreds, AAA long-term corporate bonds, AA or A or BAA long-term corporate bonds, commercial paper, stock exchange time loans and call loans, etc. I don't know....it's just that in 1973-74 (oil price spike and market crash of '74), 2008-early 2009 (GFC) and even during the brief "COVID crisis" (Feb-Mar 2020) risk asset prices of all kinds--including even prime grade intermediate-term corporate bonds--fell; I just find it somewhat unusual that IT Treasuries would lose enough in price to cause them to post a more than 6% loss while IT corporates were either flat or even were slightly positive when counting reinvested coupons....1931--especially the last half of it--was a year of liquidationist "sell everything first at any price you can get for it and ask questions later" panic and gloom if ever there was one. Any thoughts on this?

4. This last one is not about intermediate corporates at all but about Treasuries. You mentioned not having one-year Treasury series (or indeed any Treasury series) from 1920 or before due to the "banking system circulation privileges distorting yields" issue. The US Treasury did issue (starting in early 1916; I think early April) a one year bond without any circulation privilege in the banking system; it then did another issue like this on either 3-31 or 4-1 of 1917; this was followed by several others throughout 1917 into January of 1919; after this and through 1919 and mid-1920 Treasury issued many "short-term certificates of indebtedness" (short-term Treasuries of anywhere from 1.5 months to 11 or 12 months duration which were used to tide over their funding needs in the time between issues of longer-term Treasuries and/or refundings/reissues of Liberty Loans) and "tax certificates" (short term Treasuries typically either roughly three months, six months, or eight or nine months and that were generally timed to come due on the 15th or 30th of a month when quarterly tax collections were anticipated to come pouring into Treasury's coffers). I don't know if any of these (besides the one-year Treasury note of 1916) lacked circulation privileges but IIRC GFD's 3-month and 1-month T-Bill series starts in late 1917 or early 1918 and uses data from some of the abovementioned Treasury securities so apparently it thinks their yields represented true market yields and were not distorted by circulation privileges (or at least not enough to make a difference). FWIW Robert Shiller also publishes a one-year Treasury yield series back to 1900 (and it may go back to 1871) so you might wish to contact him and see how he got his data for that one.

EDITED TO ADD: 5 and 6 below:

5. Your paper states (on pg 4): "Then judgments initially made in the abstract had to be cross ruffed against historical sources". Cross-ruffed? Unless one is talking about playing bridge then speaking of "cross ruffing" doesn't seem to make much sense. Was this an autocorrect-induced typo and you actually meant "cross referenced"...or were you just using "cross ruffed" ins some kind of metaphorical sense?

6. There may have been no official one-year Treasury Note yield series until 1954 but (going by the monthly issues of the Statement of the US Public Debt from the US Treasury) there was--at least after the mid-1940s--always something available...a Treasury Note, a marketable Treasury Certificate of Indebtedness, or a (issued post mid-1941...this matters because before then Treasury bonds were partially tax-exempt) Treasury Bond that was maturing a year or so from the current date, that could stand in for a one-year Treasury. One would have to look into old issues (or digital copies thereof) of the NY Times, WSJ, or CFC/B&QR to find the prices and yields on these but if one wanted to construct a one-year Treasury yield and/or return series before 1954 it could technically be done (well, at least back to the early to mid-1940s).
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Deep history of stock and bond returns

Post by McQ »

Thanks for engaging with the paper, Alpha4, and for these thoughtful questions. To do them justice I am going to answer in batches, using your numbering.
1. Re January: technically, January 31st to January 31st for the newly collected data. For background, Macaulay and Cowles/Shiller did January high/low midpoint and called it January 15th. Every historian who has ever linked an SBBI or CRSP series (December 31st) to the Cowles/Macaulay series either fudges or throws away data. Thus Jorda et al., strict about a December terminus, do not have a Cowles annual return until the year ending December 1872 (no December 1870 index value, ergo, have to toss the 1871 readings). Siegel links the two series without comment, thus having either an 11.5 month year or 12.5 month year somewhere in the data.

I should add that some of the data collected earlier (long corporate series) is, like Cowles, January high and low midpoint. Likewise, the Durand yields use the midpoint of the high and low prices for January, February and March, presumably dating that price to February 15th.

Last, by the 1940s, as Treasury trading moved off exchange, “prices” were not observed; the midpoint of end of month bid and ask is used as the “price.”

If this is too much imprecision (and it will be, for some purposes), the new total bond index is not going to be useful to you.

2. Re yields: Remember, I am only concerned with returns, and all the values in the tables and charts in the paper are annual returns. For those series where I collected prices anew, I didn’t bother to calculate yields; return is coupon plus price change.

I did use Durand’s yields to calculate price change for the corporate series other than the long series. These are published in his three papers (refs in the SSRN paper), and after 1958, in the 1970 Banking and Monetary Statistics publication of the Fed; after 1970, in Homer and Sylla. But those yields are calculated from the midpoint of six prices in the first quarter (see #1).

If you must have monthly yields (total bond or intermediate bonds) then there will be no alternative to going back to the price sheets (CFC / Quotation & Bank Record, NY Times after 1963) and hand collecting prices. For most of the period most bonds did not trade many days; so you will have to decide if a bid-asked mean from the last day of the month is good enough, or whether you would take every daily price that was there, and average them.

In my opinion, that would be a Herculean effort for very little payoff; but your purposes aren't necessarily the same as mine.

More responses tomorrow.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Deep history of stock and bond returns

Post by McQ »

Alpha4 wrote: Mon Oct 25, 2021 4:46 pm ...

3. Your "shorter intermediate" corporate series (typically) shows much smaller deviations in return than Ibbotson/Goetzmann's intermediate-term corporate series (from the "Equity Risk Premium - Essays and Explorations" book they published roughly twenty years ago) does; even your "longer intermediate" corporate series (i.e. a simulated ladder of 6, 7, 8, 9, and--I think--10 year corporate bonds) shows typicaly smaller deviations in returns vs their intermediate-term corporate series; for instance, for the years 1956 and 1957 (1956 was a year in which bond yields increased and thus bonds did relatively poorly; 1957 was just the opposite) your "longer intermediate" corporate bond return series shows returns of -1.42% and 4.85% respectively; theirs shows returns of -4.69% and 2.70% respectively. Also, if you look at their data from, say, 1947 to 1953 their returns for this category don't come that close to yours at all.

Whether this is because their "intermediate-term" series was anything from 15 years at max to (I believe IIRC) either 1 year at minimum--which could give a different maturity average weighting--especially if most of the cap weight of the bonds they used to create their series was in the 10 to 15 year range (i.e. the majority of the bonds they chose were 10-15 years until maturity and they only had a few bonds in the "1 year to 9 years and 11 months" range)--or if because some of their bonds may've had credit risk (their book did specify that they excluded all convertible bonds from this series but it didn't say if they used all AAA or if it as well included any AA, A, BAA/BB, or even "defaulted" i.e. "junk" bonds too) and the prices/yields of said bonds may have reflected this in addition to merely responding purely to changes in the interest-rate environment....whereas your corporate bond series's were purely based on yields and yield interpolations.

I also wonder about 1931. Given that your "intermediate-term Treasury" series shows a return of -6.47% and your "long Treasury" series shows a return of -6.72% (FWIW the Simba/Siamond backtest portfolio returns spreadsheet shows a return of -2.11% and -8.74% respectively for these so your returns here on Treasuries appear to be fairly close to what the spreadsheet has--especially given differences in methodology; IIRC that spreadsheet used a 3 year to 9.9 year maturity ladder for IT Treasuries and either a 10-30 year or just a "anything longer than 20 year counts as a long term Treasury" maturity for long Treasuries) for that year I find it a bit implausible that total returns on intermediate corporates would--depending on whether it was the longer intermediate or shorter intermediate series--either be basically flat (-0.02% is basically flat) or be up 1.69%. In 1931 yields on anything with even the remotest amount of credit risk spiked (and prices fell); be it junk and/or defaulted bonds, preferreds, AAA long-term corporate bonds, AA or A or BAA long-term corporate bonds, commercial paper, stock exchange time loans and call loans, etc. I don't know....it's just that in 1973-74 (oil price spike and market crash of '74), 2008-early 2009 (GFC) and even during the brief "COVID crisis" (Feb-Mar 2020) risk asset prices of all kinds--including even prime grade intermediate-term corporate bonds--fell; I just find it somewhat unusual that IT Treasuries would lose enough in price to cause them to post a more than 6% loss while IT corporates were either flat or even were slightly positive when counting reinvested coupons....1931--especially the last half of it--was a year of liquidationist "sell everything first at any price you can get for it and ask questions later" panic and gloom if ever there was one. Any thoughts on this?
[Today's response is to your #3:]
There are several possible explanations for the deviation from the prior Ibbotson work.
1. As you note, no indexer today would classify a bond with maturity between ten and fifteen years as “intermediate.”
2. Remember also that a row in my table labeled “1957” would give returns from January 1956 to January 1957; in Ibbotson / SBBI / CRSP, a row with that label would be returns for calendar year 1957.
3. Durand used only the crème de la crème of bonds, and did not even examine yields from any bonds rated less than A;
4. And, my corporate intermediate series are among the weakest in the set, since they are extracted from successive yields by formula, for example, the yield change from the 7-year maturity in 1956 to the 6-year maturity in 1957. The procedure allows for no downgrades or defaults.
5. Hence, the corporate intermediate returns will almost certainly be over-estimated at the depths of the Depression; but their bounce back after 1938 will also be muted. My long corporate series resting on price observations and reflecting downgrades and defaults underperforms long governments at the depths of the Depression and then outperforms them after 1938, thus restoring the corporate bond premium. I’d expect the muted intermediate returns to, likewise, end up in the same place by the mid-1940s as a true, price-observed, downgrades and defaults reflected, series would.

Re 1931

1. Same point, 1931 in my data table has the returns from January 1930 to January 1931. The crisis came in September 1931, after Britain devalued the pound. To your list of assets that plunged, add long Treasuries. Conversely, in January 1932, there was a tremendous rally in corporates just in that one month. Long story short: an annual series like mine is useless for event studies, and year by year comparisons with another compiler not using the end of January are fraught, when prices are as volatile as they were from August 1931 to February 1932.
2. There is a complication regarding longer intermediate Treasuries in those decades; it is discussed in the paper’s method section. Briefly, almost every Treasury bond was callable in X / matures in Y, with the separation typically five to ten years. The convention was to treat the call date as the maturity if the bond traded above par; so in 1935, a bond callable in 1944 / maturing in 1954 is eligible for my ‘longer intermediate’ set if trading above par—as almost every Treasury bond did in those years. On the other hand, if the market didn’t price the call as 100% certain, such “longer intermediate” bonds are more than halfway to “long” bonds.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Deep history of stock and bond returns

Post by McQ »

Alpha4 wrote: Mon Oct 25, 2021 4:46 pm ...

4. This last one is not about intermediate corporates at all but about Treasuries. You mentioned not having one-year Treasury series (or indeed any Treasury series) from 1920 or before due to the "banking system circulation privileges distorting yields" issue. The US Treasury did issue (starting in early 1916; I think early April) a one year bond without any circulation privilege in the banking system; it then did another issue like this on either 3-31 or 4-1 of 1917; this was followed by several others throughout 1917 into January of 1919; after this and through 1919 and mid-1920 Treasury issued many "short-term certificates of indebtedness" (short-term Treasuries of anywhere from 1.5 months to 11 or 12 months duration which were used to tide over their funding needs in the time between issues of longer-term Treasuries and/or refundings/reissues of Liberty Loans) and "tax certificates" (short term Treasuries typically either roughly three months, six months, or eight or nine months and that were generally timed to come due on the 15th or 30th of a month when quarterly tax collections were anticipated to come pouring into Treasury's coffers). I don't know if any of these (besides the one-year Treasury note of 1916) lacked circulation privileges but IIRC GFD's 3-month and 1-month T-Bill series starts in late 1917 or early 1918 and uses data from some of the abovementioned Treasury securities so apparently it thinks their yields represented true market yields and were not distorted by circulation privileges (or at least not enough to make a difference). FWIW Robert Shiller also publishes a one-year Treasury yield series back to 1900 (and it may go back to 1871) so you might wish to contact him and see how he got his data for that one.
I don’t dispute that there were short-term Treasuries available before 1954 (but the 1916 one you found is new learning for me). The publication I reviewed for possible issues to use was the Banking and Monetary Statistics publication (1943 & 1970 Fed Reserve). There was a 9-12 month bills series back to the early 1940s; and before that, shorter bills to the 1919 beginning of that publication's data.

Ultimately, I decided that a bill was not a bond, and that there was no reason to expect splicing four 3-month bills would give the same sort of return as a real one-year bond with two semi-annual coupons remaining. And no matter what, I couldn’t get a consistent series back past 1954.

Hence, the decision to let the one-year corporate yield/bond from Durand anchor the short end prior to 1954.

Keep in mind that the target index, a total bond index like the Aggregate, doesn’t hold bonds past one year. The one year bond in my simulation is there to capture the returns on 1-3 year bonds; yet another argument for avoiding bills.

Now, as to Robert Shiller: you must have missed my exchange with Simple Gift upthread, beginning July 19th. The phrase “one year Treasury” appears in some of Shiller’s work, but careful archaeology using his footnotes across his several books revealed that before 1954, the rate comes from commercial paper of varying lengths up to six months—not Treasuries. Or in the language of his source, Macaulay, commercial paper issued by “the larger department and men’s furnishing stores, jobbers of dry goods, hardware, shoes, groceries, floor coverings, etc., the manufacturers of cotton, silk and woolen goods, clothing, etc..”

I refuse to accept that as a simulacrum of short Treasury bonds.
Alpha4 wrote: Mon Oct 25, 2021 4:46 pm EDITED TO ADD: 5 and 6 below:

5. Your paper states (on pg 4): "Then judgments initially made in the abstract had to be cross ruffed against historical sources". Cross-ruffed? Unless one is talking about playing bridge then speaking of "cross ruffing" doesn't seem to make much sense. Was this an autocorrect-induced typo and you actually meant "cross referenced"...or were you just using "cross ruffed" ins some kind of metaphorical sense?
Sorry, cross-ruffing was in common use in the Harvard case studies I taught, and is used here metaphorically, as a somewhat more vigorous expression.
Alpha4 wrote: Mon Oct 25, 2021 4:46 pm 6. There may have been no official one-year Treasury Note yield series until 1954 but (going by the monthly issues of the Statement of the US Public Debt from the US Treasury) there was--at least after the mid-1940s--always something available...a Treasury Note, a marketable Treasury Certificate of Indebtedness, or a (issued post mid-1941...this matters because before then Treasury bonds were partially tax-exempt) Treasury Bond that was maturing a year or so from the current date, that could stand in for a one-year Treasury. One would have to look into old issues (or digital copies thereof) of the NY Times, WSJ, or CFC/B&QR to find the prices and yields on these but if one wanted to construct a one-year Treasury yield and/or return series before 1954 it could technically be done (well, at least back to the early to mid-1940s).
True, as noted above, there was always available a mishmash of short-term instruments, with different properties and maturities, changing over time. I did not see how this mishmash could strengthen the simulation of a Total Bond index.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
dml130
Posts: 166
Joined: Wed Sep 02, 2020 11:40 am

Re: Deep history of stock and bond returns

Post by dml130 »

Bump.

Apologies if I missed this in the discussion, but was there a determination made of the historical real return of short term (1-3 year) treasuries? In various internet sources, I've seen bills listed at about 0.5%, and intermediate treasuries as about 2% - I suppose that would put short term treasuries somewhere in between, just a guess but maybe at about 1-1.5% real? Is that correct?
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Deep history of stock and bond returns

Post by McQ »

dml130 wrote: Mon Jan 30, 2023 10:32 am Bump.

Apologies if I missed this in the discussion, but was there a determination made of the historical real return of short term (1-3 year) treasuries? In various internet sources, I've seen bills listed at about 0.5%, and intermediate treasuries as about 2% - I suppose that would put short term treasuries somewhere in between, just a guess but maybe at about 1-1.5% real? Is that correct?
Not calculated by me, but you could construct such a series:
1. Go to FRASER and get the Banking and Monetary Statistics compilations, 1941 and 1970
2. Find the pages with short term government notes (multiple series), capture yields, convert yield change to returns
3. Apply the CPI, downloadable from the Bureau of Labor Statistics
Presto, you have your series.

Too much work? Find a copy of the Stocks Bonds Bills Inflation yearbook, take the inflation adjusted T-bill and 5-year series returns, average the two together, and et voila. (average maturity about 2.55 years).
Just remember that there was no regular issue of T-bills until 1929, or 5-year notes until 1932, so there is a fictive element to the first few years.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
dml130
Posts: 166
Joined: Wed Sep 02, 2020 11:40 am

Re: Deep history of stock and bond returns

Post by dml130 »

Thank you for the reply, much appreciated.
AlohaJoe
Posts: 6609
Joined: Mon Nov 26, 2007 1:00 pm
Location: Saigon, Vietnam

Re: Deep history of stock and bond returns

Post by AlohaJoe »

dml130 wrote: Mon Jan 30, 2023 10:32 am Apologies if I missed this in the discussion, but was there a determination made of the historical real return of short term (1-3 year) treasuries? In various internet sources, I've seen bills listed at about 0.5%, and intermediate treasuries as about 2% - I suppose that would put short term treasuries somewhere in between, just a guess but maybe at about 1-1.5% real? Is that correct?
It depends on what you're going to use it for. As a high-level estimate, sure it's fine.

In reality, the yield curve has varied tremendously over time. I think it was basically always negatively sloping until the 1930s. Then the structure of bonds and investing changed and now that would be very unusual outside of a recession. And during the 1950s the whole structure of the bond market was weird because the US Treasury made it that way (in order to pay off the debt from WW2 at super low prices), so spreads were depressed.

Most people who want these kind of estimates want to use them as some kind of estimated forward returns...so does that mean you basically throw out all the data before the 1960s because the investing world was so different? But we're always going through structural changes. It is hard to know which ones are going to be material and which ones won't.
seajay
Posts: 1656
Joined: Sat May 01, 2021 3:26 pm
Contact:

Re: Deep history of stock and bond returns

Post by seajay »

Circumstantially and many investors might simply have opted to hold bonds pre 1930's, and up to that time broadly seen similar total returns whether they held stocks or bonds.

1817 British Sovereign gold coin, one Pound legal tender value, was used as both a domestic and international currency. A Sovereign gold coin contains 0.235 troy ounces of gold. Alongside bank notes, where the one Pound note included the wording "I promise to pay the bearer the sum of One Pound" i.e. a gold Sovereign coin worth of gold. Money was gold, gold was money as were Pound note paper currency that could be exchanged for gold.

1817 to 1931 and Pound/US Dollar exchange rate averaged 4.86, relatively consistently holding around that value across those years.

The British Official Gold Price was 4.25 Pounds/ounce, as it had been from 1717 up to and beyond 1931.

Ounces of gold in a Pound = inverse of Official Gold Price = 1 / 4.24 = 0.235 and hence the Sovereign being 0.235 ounces of gold

US/Britain and others traded with a relatively consistent Pound/Dollar exchange rate and where money was gold (and silver)

Inflation 1817 to 1931 was broadly flat, both in the US and Britain, less than 0.1% annualised. Some volatility, 5.2% standard deviation in yearly changes, median 0%, arithmetic average 0.23%.

Short term interest rates averaged around 3.5%.

In a world where gold is money/currency (Sovereign gold coins in your pocket), and inflation broadly averages 0%, a 3.5% interest rate is like a real rate of return, Lending your money (Sovereigns) to the State in return for interest was in effect the state paying you for it to securely store your gold.

Investors were inclined to just hold bonds. Stocks were for speculators and by the time you factored in the risks generally stock total returns were the same as bonds. Across 1817 to 1931 in the US and both annualised around 5.5%, bonds more progressively so, stocks more zig-zaggy so.

In 1931 that fell apart. The market price of gold deviated (rose) away from the Official Gold Price, which induced many to convert Pound Note paper money into gold such that a end to the conversion of paper money into gold had be called. The US followed that lead and in 1933 compulsory purchased American's gold. Over the following years the US established international agreement to use the Dollar for international trade, and where it agreed to peg the US dollar to gold.

The UK's first fund https://www.fandc.com/ started in the mid 1800's and largely comprised of bond holdings, along with a handful of railway stocks that made up relatively little of the overall portfolio. In the early 1900's however that did transition more into stocks, until by around WW1 it was more heavily weighted into stocks than bonds. Nowadays its a global stock type fund.

Circumstantially I suspect pre 1930's primarily bonds were preferred/held by savers/investors. The Roaring 1920's however I suspect drew many prior bond only savers/investors into stocks. Going back prior to that and I would imagine more a choice between opting for Treasury type bonds or "corporate" type bonds, and where those "corporate" bonds and stock risk/rewards significantly overlapped. Lend to the state for a lower yield but more certainty of return of money, or buy railroad bonds where you may get a great return, or perhaps see a default.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Deep history of stock and bond returns

Post by McQ »

AlohaJoe wrote: Fri Feb 03, 2023 11:04 pm
dml130 wrote: Mon Jan 30, 2023 10:32 am Apologies if I missed this in the discussion, but was there a determination made of the historical real return of short term (1-3 year) treasuries? In various internet sources, I've seen bills listed at about 0.5%, and intermediate treasuries as about 2% - I suppose that would put short term treasuries somewhere in between, just a guess but maybe at about 1-1.5% real? Is that correct?
It depends on what you're going to use it for. As a high-level estimate, sure it's fine.

In reality, the yield curve has varied tremendously over time. I think it was basically always negatively sloping until the 1930s. Then the structure of bonds and investing changed and now that would be very unusual outside of a recession. And during the 1950s the whole structure of the bond market was weird because the US Treasury made it that way (in order to pay off the debt from WW2 at super low prices), so spreads were depressed.

Most people who want these kind of estimates want to use them as some kind of estimated forward returns...so does that mean you basically throw out all the data before the 1960s because the investing world was so different? But we're always going through structural changes. It is hard to know which ones are going to be material and which ones won't.
I wanted to reiterate and expand on the point that AlohaJoe makes here. Durand 1942 (https://www.nber.org/books-and-chapters ... -1900-1942) has pictures of the yield curve for every year from 1900 to 1940. The positively sloped yield curve that we think of as "normal" does not appear until late in the 1920s. Later the government enforced a positive slope on the curve during and after the war; I sometimes think that effort "forced the birth" of the normal yield curve we take for granted.

One reason yield curves from long ago are, to use a technical term, weird, is because short and intermediate bonds were not routinely issued until decades after the war. Most of the "short" and "intermediate" bonds out there were aged long bonds. Traders, it seemed, expected the price to remain the same for a 30-year old bond that was now 23 years old; they didn't have the "ride the yield down the curve" expectation we have today, where bonds at all maturities are issued almost every quarter, at issue rates reflecting that expected positive slope..
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
popoff_1
Posts: 14
Joined: Thu May 16, 2019 1:49 am
Location: Italy

Re: Deep history of stock and bond returns

Post by popoff_1 »

Hello, I have a general question for historical stock data. I am using the data from CRSP, and I am wondering how CRSP have got his data? Does a research team has got all the stock price data and dividends from a newspaper? The financial times is avialable for 125 years back and I think there were printed the daily stock data. I am simply interested in how they got this data.
Thesaints
Posts: 5108
Joined: Tue Jun 20, 2017 12:25 am

Re: Deep history of stock and bond returns

Post by Thesaints »

Going "deep" back in time is not simply an exercise in arithmetic.
"Stocks" in 1890 means something rather different than it does in 2023. Their markets are fundamentally different (just think about who participated to the stock market, and therefore contribute to price formation, 150 years ago and who does now). Also, "shares" represented something different back then and, of course, corporations also were another thing entirely.

Even something (deceptively) simple such as "inflation" is not understood and accounted for by adding CPI multipliers year over year, 150 times. The CPI-U is a triple average: average cost of average consumption of the average urban family. Think about what the average urban family consumed 150 years ago and what it consumes now. In fact, it is a known fact that walking back the CPI year over year begins to require special attention over a few decades and becomes quite meaningless over a century, or even less. Changes in prices give way to changes in lifestyle as one looks further and further back in time.

More data is always better than less data, provided one doesn't get overwhelmed, or otherwise confused, by it. Be very careful when including what effectively amounts to stale data in your assessments.
IMO, very old data is pretty much useless when calculating averages and it is only worth to be considered when calculating variances.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Deep history of stock and bond returns

Post by McQ »

popoff_1 wrote: Mon Feb 06, 2023 12:03 pm Hello, I have a general question for historical stock data. I am using the data from CRSP, and I am wondering how CRSP have got his data? Does a research team has got all the stock price data and dividends from a newspaper? The financial times is avialable for 125 years back and I think there were printed the daily stock data. I am simply interested in how they got this data.
See Fisher and Lorie for a description of the original CRSP data collection: https://www.crsp.org/files/Rates%20of%2 ... Stocks.pdf

See Cowles for the procedures for data collection behind Robert Shiller's data site: https://cowles.yale.edu/research/cfm-31 ... -1871-1937
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Topic Author
McQ
Posts: 1425
Joined: Fri Jun 18, 2021 12:21 am
Location: California

Re: Deep history of stock and bond returns

Post by McQ »

Thesaints wrote: Mon Feb 06, 2023 5:13 pm Going "deep" back in time is not simply an exercise in arithmetic.
"Stocks" in 1890 means something rather different than it does in 2023. Their markets are fundamentally different (just think about who participated to the stock market, and therefore contribute to price formation, 150 years ago and who does now). Also, "shares" represented something different back then and, of course, corporations also were another thing entirely.
Here is a thought experiment on where to draw the "line of irrelevance" as far as historical stock data is concerned.

"Everything before about 2005 is irrelevant when it comes to projecting future stock returns. When Google pioneered the internet business model, and Apple pioneered the smartphone business model, followed by the social media business model of Meta, the returns available from common stocks fundamentally changed. The pitiful returns available from metal-benders and soap manufacturers were no longer the standard. Just look at the returns recorded on the ARK Innovation ETF."

Of course that's a tongue in cheek, devil's advocate account. If you will, explain why the dividing line between relevant history and irrelevant market history is not 2005, but some other date.
Thesaints wrote: Mon Feb 06, 2023 5:13 pm Even something (deceptively) simple such as "inflation" is not understood and accounted for by adding CPI multipliers year over year, 150 times. The CPI-U is a triple average: average cost of average consumption of the average urban family. Think about what the average urban family consumed 150 years ago and what it consumes now. In fact, it is a known fact that walking back the CPI year over year begins to require special attention over a few decades and becomes quite meaningless over a century, or even less. Changes in prices give way to changes in lifestyle as one looks further and further back in time.
You may enjoy the data essay on measuring historical inflation at https://www.measuringworth.com/docs/cpistudyrev.pdf
Thesaints wrote: Mon Feb 06, 2023 5:13 pm More data is always better than less data, provided one doesn't get overwhelmed, or otherwise confused, by it. Be very careful when including what effectively amounts to stale data in your assessments.
IMO, very old data is pretty much useless when calculating averages and it is only worth to be considered when calculating variances.
Please define the difference between "stale" data and "older, longer, more complete" data. I would also appreciate a discussion of why the first moment of the distribution cannot be estimated under circumstances where the second moment can. I would have thought the two were inextricable.
You can take the academic out of the classroom by retirement, but you can't ever take the classroom out of his tone, style, and manner of approach.
Post Reply