Surprising and inevitable stories

Like me, a couple of people at the Bank of England have also been messing around with text mining and machine learning. In a Working Paper called “Sending firm messages: text mining letters from PRA supervisors to banks and building societies they regulate” 
They look at linguistic patterns in financial language. In their case the corpus they are analysing is letters that they wrote to banks before and after the financial crisis. Their analysis suggests that :

  • The tone of the letters has changed since the financial crisis,
  • The letters to banks closest to failure contain more risk-related language.

The BoE’s technique is sophisticated approach using Random Forrest Classifiers that supports this conclusion with data. A Random Forrest is a collection of decision trees whose predictions are averaged.
Here’s the Bank’s own explanation how it works:
First, for each decision tree we grew, a different random sample from the full set of letters was drawn.
The algorithm then identifies the most important linguistic features that distinguish letters from one another. We looked at 25 linguistic features, including measures of linguistic complexity, sentiment, directiveness, formality and forward-lookingness. Each tree considers only a random sample of the features from the full set of 25.

But the unsurprising conclusion just doesn’t seem worth all that effort. A human reader probably could have told you the same without the sophisticated decision trees and random forests.
Instead, I’m wondering if we can do better than just analysing linguistic style. I’m thinking about how a computer might analyse a whole story.  The key principle of a good story is that the conclusion must be both surprising and inevitable.  Surprising?  Who wants to read a predictable story that we know how it ends before we’ve finished the first page.  Inevitable?  But we don’t enjoy stories that end with a non sequitur.  The ending has to make sense in terms of what came before.

The Northern Rock story

As an example I read dozens of Northern Rock financial statements and trading updates over the years. The former bank actually had a convincing story – about lower costs, allowing it to take more market share with higher profitability following.  Higher profitability allowed the bank to keep growing faster while keeping costs lower, feeding a virtuous circle of more competitive products, higher market share, higher growth and returns.

At the start of 2007, as Northern Rock had successfully been following the same strategy for years, there was a lot of certainty that this could go on indefinitely. The bank could point to its historic numbers to assure investors that the story made sense.

No one wanted to think how the story might end.  The table below shows both the good side, the bad and the ugly side. 

How the story ends
£m 2002 2006 Change CAGR
Total Income 561 976 415 15%
Attributable profits 229 416 187 16%
Return on Equity 20.8% 21.9% Increase
Total Assets 41,875 101,048 59,173 25%
Customer Deposits 15,336 22,631 7,295 10%
  • The good: The top half of the table shows that the bank had consistently grown total income and attributable profits at mid teens rates and with an increase in RoE (profitability).  That all seems very sustainable, keep going Mr Applegarth!
  • The bad: But total assets are growing much faster than both profits and customer deposits.  The shape of the balance sheet is changing.  Although RoE increased, other measures of profitability were falling.  How can RoE be increasing while other measures of profitability are falling?  Only by reducing the “E” part of the RoE equation (ie increasing leverage) proportional to total assets.
  • The ugly: Therefore the third column, headed “change” shows something rather disturbing.  In order for the bank to increase profits by £187m, total assets need to increase by £59bn.  £59bn!  That is a huge amount of £ in absolute terms for a rather puny increase in profits.  Put another way: to grow profits by 80% (4/5) it needed to grow Total Assets by 140% (7/5) –  and by definition that is not scalable forever.

To reiterate: neither the bank’s strategy nor the growth rate changed.  It wasn’t as if management decided to launch new products, or enter new markets.  Instead the strategy broke because it couldn’t last.  Ultimately Northern Rock became too reliant on the short term wholesale funding market perceptions of the risk. Suddenly the frame of reference shifts, and the negative consequences of the strategy become clear. And we think that the unsustainability was so inevitable, we should have seen it all along.
Instead of random forests looking at linguistic style, using machine learning to flag the vulnerability at the core of Northern Rock’s story would be really something. This would mean understanding the relationship between falling marginal returns failing to show up in falling profitability – as measured by RoE which actually increased between 2002 and 2006.

For instance, it’s not obvious to me that the lack of customer deposits was ultimate cause of failure: Provident Financial relied on capital markets more than customer deposits to fund its business.  Yet that lender’s share price ended 2009 at just below 700p, the same level it ended 2006.  It came through the financial crisis well, because the business didn’t rely on short term funding markets. *  Perhaps if Northern Rock had used longer term funding it might have been OK?  I don’t think so, because the analysis above suggests the hidden contradictions within the “virtuous circle” strategy meant ultimately it had to fail.

How to digitise a story

So I’m thinking about how a computer might digitise a story – and analyse the consequences of if X and Y are true, but Z will also be true, how will these colliding variables might lead to a surprising (but inevitable) conclusion.  I think linguistic style is too superficial, and I’ll need to start using Named Entity Recognition The reason is that stories are about relationships between entities.  I’ve been reluctant to do this, because NER often requires manually labelling entities.  
Aside from highlighting hidden risks at a bank like Northern Rock, a narrative analysis using NER could highlight hidden opportunities.
Google was the 18th search engine to arrive on the web – but no one remembers the 17 others. The success has to look highly surprising (most search engines failed) and yet in retrospect inevitable (Google was the best, and the world only needs one search engine).

How would a computer analyse this? Random forests and decision trees are unlikely to be much use. Lots of numbers and data seem the wrong way to go.
Instead, we need to get better about thinking how computers can interact with simple stories. And perhaps a clue lies in the original google investor pitch document that that Sergey Brin and Larry Page made to venture capitalists. According to John Doerr, when the two founders came to his office to pitch their idea for a search engine, their PowerPoint deck had just 17 slides – and only two slides with numbers.  A surprising lack of numbers give that the two maths geeks chose to name their company after a googol  (10100).  Surprising and inevitable. 

Photo by Enis Yavuz on Unsplas

*Ironically Provident Financial ran aground later because they did try to change their business model with a new IT system.

3 Responses

  1. Andrew Latto

    A thought provoking article. I thought the table on Northern Rock was very interesting. People get lots in numbers and it takes skill to pick only the most important number IMO. Also interesting about the Google pitch. People focus on too many number in my view. They therefore don’t see the wood for the trees. I wonder if the Google pitch presentation is available?

    1. Bruce Packard

      Thanks! I can remember reading this at the time. Just re-read it. I agree it would be hard to automate a discussion with management “exploring the limits of their strategy” – but seems far more useful than all the computer power currently trying to profit from price momentum. Or indeed using “random forests” of decisions trees in order to reach obvious conclusions.