Cohort analysis for active investors

Finance_GrowingMoneyWarning!  This week’s article is geeky and technical.

Like The Investor over at the excellent Monevator site, I invest actively via stockpicking.

But most people are probably better off investing on autopilot, dripping money into index trackers each month and getting on with life. As I’ve said before, investing can be made really simple

Cohort analysis for active investors

In investing, I always try to distinguish between what I do (the investment process) and what I get (the investment returns).  Randomness (luck) obscures the links between these in a way that we humans find really difficult to see, let alone understand fully.

The best book on this is Fooled by Randomness by Nassim Taleb, which changed the way I saw the world and which I recommend for investors (even if his writing style isn’t everyone’s cup of tea).

The problem with measuring annual investment returns from a portfolio the conventional way is that it doesn’t tell us anything useful about our investment process and whether we did the right things during that year.  And, as Terry Smith reminds us, a year is the time it takes the earth to revolve around the Sun and not necessarily the right period over which to judge investment performance.

noise and signalWhen assessing investment performance, it’s important to try to tune out market noise (short term price movements) and look for the signal (underlying long term performance trends).

The difference between signal and noise is illustrated here where the form of the sine wave (the signal) is distorted by random “crackle” (the noise), like a bad connection on a phone call to Australia.

The reason that evaluating the results of an investment process is tricky is that the noise drowns out the signal in the short term (in investing, anything under 5 years is the short term in my book).

Imagine an investor called Micky back in 2007.  Specifically, it’s the last day of the year: 31 December 2007. The stockmarket has been in a bull market and gone up by say 20% during the year.  Let’s assume that Mickey’s portfolio has gone up by 21% during that year.

So at first glance 2007 looks like a good year based on the return achieved during the period. But now imagine that on the last day of the year, Mickey does a bunch of crazy things and he:

  • buys shares in a Bolivian gold miner on AIM after a tip heard down the pub
  • buys early stage tech stocks on NASDAQ that he doesn’t understand
  • buys stocks with high dividend yields that are too good to be true
  • buys expensive actively (mis)managed funds
  • daytrades
  • fails to diversify

These are all dumb things to do. But even so, these actions will probably have almost no effect on the 2007 investment performance results as conventionally measured. Remember it’s the last day of the year, so there just isn’t time for the mistakes to show up in performance.

A poor investment process will tend to lead to poor investment returns over time. But in investing it often takes many years for the chickens to come home to roost. Investing is a bit like life: if you keep doing the right things for long enough, you’ll probably get the benefit…eventually.  Just don’t expect the process to feel fair or easy.

The question is then: has 2007 been a good year for Mickey’s investing? The answer is that it’s been a good year for his investment returns and a terrible year for his investment process. We will only have an idea how good or bad his 2007 decisions were when we look back in a few years time and see the returns generated over the years from the shares he bought in 2007.

This is the essence of what I call cohort analysis: looking at the subsequent performance of the shares bought in a given year and comparing them to other “intake” years.  We’d hope that an improving investment process would, over time and on average, lead to improvements in the performance of subsequent cohorts.

The statistical evidence shows that value investing works.  Buying cheaper leads to better returns, all else being equal. But the valuation effect is weak, slow and variable. For example, Vanguard research showed that the CAPE was the best predictor of future market returns but that the R squared was still only 0.43 (where 1 is perfect correlation and zero is no correlation).

When we buy good value assets, we are placing a bet on regression to the mean. The regression can happen quickly or it can happen slowly.  So we need to be prepared to wait. The evidence suggests that, when looking at stock market returns at the index level it can take 7 – 10 years for the value effect to manifest itself fully (although my experience is that my successful individual stock picks usually tend to “come good” within 3 years).

So it may not make sense for value investors to be judging the results of investments made in 2007 until 2014 – 2017.  So, in addition, to measuring my returns the conventional way (i.e. what were the total returns of my portfolio in a given year?) I also use cohort analysis where I track in subsequent years the annualised return generated by the shares that I buy in any given year.

This graph shows (as at 11 April 2016) my annualised returns arranged by year of the original investment over the last 19 years.Returns by year of investment

For example, the return generated by my total portfolio of direct shares during 2011 (as conventionally measured) was 18%.  But the shares that I selected during 2011 generated subsequent annual returns of 30% per year (running up until 11 April 2016).

Of course, returns measured in this way are still subject to randomness, but I think this analysis by “vintage” is a helpful additional way to think about portfolio performance.

Cohort analysis gives us additional information on risk as well as returns. Risk-wise, the years between 1997 and 2002 look like a Wild West show. This reminds me a bit of Gordon Brown’s claims to have abolished boom and bust, which date from the same period.

There were 2 main reasons why my 2002 cohort was so bad. Firstly, I invested in a bunch of value traps: cheap companies with high dividend yields that were cheap for a reason.  These tended to be poor quality companies that ended up slashing their dividends. Secondly, I sold these dogs at the worst time in 2003 as recounted in A Tale of 2 Bear Markets.  Ouch!

I’ve learned from those early years and improved my investment process since then. Looking at the chart, I think I can see 3 factors playing out as time went by:

1) I took less risk by investing in stable (boring) higher quality companies.

This seems to have generated more consistent returns (although it may be that the last 6 years are flattered by chance, style bias and the lack of a bear market).

2) The importance of starting valuations. 

The returns generated from 2006 and 2007 were sub-par.  This was partly my fault.  But it also probably reflects the fact that 2006 and 2007 were “top of the market” years where most shares were expensive.  Similarly, 2011 was a good year partly because there was lots of fear (and hence bargains available) in the market.

2011 was the year of European government debt drama, when the market plunged and it sometimes felt at the time like we were falling back into the 08/09 crisis.  But, with hindsight, we can see that 2011 turned out to be a fantastic year for stock picking, with the 2011 vintage generating subsequent annual returns of 30% per year.

3) Diversification

In the early years I had a limited number of holdings and was not properly diversified.  Over time I learned how to balance and diversify a portfolio.

As well as looking at returns, I think cohort analysis gives us another way to look at risk and volatility.

In textbooks, volatility is defined as the annualised standard deviation of the logarithm of price movements. This is the standard deviation of the continuously compounded return over a year. In plain English, volatility is just prices bobbing up and down. Think about share prices flashing blue and red on a screen.  For the most part, this tells us no more about the underlying performance of companies than lights flashing on a fruit machine.

Price movements are not the same thing as real, economic risk. Firstly, Robert Shiller showed that stock market prices exhibit much more volatility than is justified by changes in the underlying economic fundamentals (such as dividends or earnings) which tend to grow much more steadily.

Secondly, the definition of volatility taught in business schools and universities gives equal weight to share price movements up as well as down.  If volatility equals risk, how come we don’t hear investors complaining about upwards volatility?

Thirdly, we can prove that volatility is often a good thing for savers – see the worked example in January Sales and Bozos circus.

So volatility is not the same thing as real risk.  Value investors think of real economic risk as the permanent impairment of their capital, a loss that can’t be recovered from over a relevant time period.

I think measuring volatility is best done visually by simply looking at the graph above and seeing how the dispersion of annual returns has fallen over time.  No doubt we could use some fancy maths to quantify volatility (e.g. standard deviation, the Sharpe ratio etc etc).  But it’s better to be roughly right than precisely wrong.

Its worth noting however that the results for the more recent years are sensitive to daily price movements.  Because of the way that an internal rate of return (IRR) calculation works, even small price movements over the early period of an investment lead to wild swings in the calculated IRR.

Also, the sample size is too small to be really meaningful. Because trading costs money and is a drag on performance, I try to keep down the number of trades I make. And so, as time has gone by, I have reduced the number of new investments made each year.  In 2015 I only bought 2 new shares (each month I usually reinvest dividend income into buying more of my existing shares and ETFs). So, the daily moves in those 2 shares have a significant impact on the reported IRR for the 2015 cohort.

To illustrate the sensitivity of the results of the 2015 cohort to short term price moves, I have reproduced below the same graph as before but based on prices at 24 May 2016 (rather than 11 April 2016, as above).Cohort analysis 24 5 16

Note that in the space of less than 2 months my 2015 IRR has gone down from about 29% to about 7%, reflecting one share under-performing over that short period.  In contrast, the 1997 IRR is completely unchanged over the period because I don’t still hold any of the shares that I bought in 1997.  All of the 1997 IRR is realised as opposed to unrealised and therefore the results are not affected by share price movements.

But these short term movements are no big deal. Between the 2 dates the overall annualised returns (measured over the last 19 years) for my portfolio rose (to 12.8% per year) as falls in the 2015 cohort results were more than offset by rises in the returns of the other years.

As ever, past returns may not be any guide to the future.  The returns achieved in the period have been pretty good but there are no guarantees in equity investing and these graphs would look very different if measured at the trough of a bear market.


  1. I wonder how your simplicity portfolio would have compared to your active picks over your investing lifetime.

    1. Good question. The active picks significantly outperformed The Simplicity Portfolio over that time. What we can’t know for sure is to what extent this outperformance was due to luck and whether it justified the time and extra effort that I spent on my active investing!

  2. Playing with Fire · · Reply

    Cool post TEA, I’d be interested to know how the returns of individual years compare to the overall return, ie, how quickly can you see whether an investment is likely to do better or worse than the index, and do gains in the past 3/4/5 years have any predictive power over the longer term results.

    I love NNT’s FBR, it was so eye-opening to see how we humans tend to see patterns with very few data points. The chapter on luck vs skill with CEOs and stockbrokers blew me away.

    1. Thanks PWF, although I’m not sure whether cool is the right description(!).Re your question of how quickly I can see whether an investment is going to work out, there is no scientific answer to this, but I do know that my average holding period for the shares currently in my portfolio is 3.5 years.

      Its possible to argue that the market is more efficient at the company level (ie relative pricing between different shares) than it is at the index level (i.e. is the market fairly valued as a whole?). The implication of this might be that 3 – 5 years should be enough for an individual company to get re-rated back to fair value (at least compared to other stocks of comparable quality), whereas it might take the index longer to revert back to fair value. History tells us that the market can stay under or over-priced for the best part of a decade (eg under-priced 1974 – 1984 or over-priced 1998 – 2008).

      And I agree with you on CEOs (and stockbrokers). If anyone wants a copy of Sir Terry Leahy’s (Tesco CEO) book “Management in 10 Words” I’m sure there are plenty of unsold copies lying around at Amazon.

  3. I tried to read Taleb after many recommendations but couldn’t stand his style. And anyway, I’m not convinced that high intelligence – however you measure that – is a virtue. When combined with other character traits, such as arrogance and a lack of personal insight, it’s not an attractive proposition.

    1. SHMD…on the question of whether Taleb is an attractive proposition, I only recommend that investors try his book….not moving in with him 😉

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: