Here Is What We Learned Using Text Analysis On Credit Memos

Text Analytics In Credit Underwriting

In a moderate corner of the big data world, lies a discipline called text analytics. Text analysis is the mining of data contained in natural language done to derive bias, sentiment, sentence structure and word usage. If you have seen a “word cloud” you have seen the basic building block of text analysis. While banks generate a lot of numbers, we generate almost as much narrative from such documents like new product approvals, customer surveys, legal agreements and mostly, credit memorandums created to document the underwriting on a loan. We were curious as to what our credit memos said about our industry, and so conducted a sampling of 24 banks, including our own, and ran 55 credit memos through a variety of various text mining engines to see what insight we could gain. The results were insightful.  


Text Analytics


What Is Text Mining


Text analysis can help banks derive valuable business insights from text-based content such as Microsoft Office documents, email, surveys and social media. Unlike numbers, text is wildly unstructured, inconsistent and naturally filled with various syntax, slang, banking-specific vocabulary, vague meanings and, of course, sarcasm. Algorithms then score the words and then reflect back metrics like sentiment, emotion, intensity and relevance. In the last five years, machine learning techniques and statistical modeling frameworks have become commonplace, so these tools are now easy and accessible for the business line person.


Why We Trained The Text Mining Engine On Credit Memos


Whenever a human selects what information to use in a credit memo there is the potential for bias. Of larger consequence, how this information is presented also potentially injects a material bias. It is easy, for example, to talk about a borrower’s financials without pointing out that the last two years have been composed of one-time items such as building sales that are non-repeatable. The question we wanted to ask is how much bias and does this bias impact the approval process?


Are credit memos more positive than negative? Do more positive words increase your chances of getting loan approval? Is speaking positively about the bank’s relationship with the customer more important than the quality of the borrower’s cash flow?  


How We Conducted The Analysis


While we used a variety of models, we ran specific loans on IBM’s Alchemy text mining engine in order to maintain consistency. Every word was scored and then tabulated on a scale between -1 and 1. A score of 0 is neutral, while a score of -1 is negative and 1 is positive. Certain words also carry higher intensity scores as in “love” is more positive than “like.”


Our Findings


Our findings confirm what most bankers intuitively know and that is most credit memos are biased to the positive. The average sentiment score ranged from -7% to 23% with a median of 5%. In other words, for the average credit memo, underwriters use 5% more positive sentiment than neutral and negative sentiment. While we don’t have any quantification of whether the positive sentiment was warranted (since there is a positive selection bias occurring naturally and you only put forward credits that you think you can get approved), this feels like it is in the acceptable bounds.


Further, while we admit to a relatively small sample size, there does seem to be a weak correlation between the strength of positively slanted words and the approval process. It seems that the more positive words that are used, the more likely a memo is to get approved by a correlation of approximately 17%. This is not to say that the positive sentiment is causing approval, only that net positive sentiment and approval can be found together 17% of the time.


Also interesting is that we found that when cash flow and loan-to-value was strong, underwriters seemed to deemphasize these facts and used more neutral words. This was a counterintuitive finding as you would expect the most importance aspects of underwriting should get the main sentiment emphasis. In similar fashion, when debt service coverage and loan-to-value was weak, there were more positive words used to describe management and other qualitative aspects of credit such as the length of the relationship.


What Sentiment Looks Like


While we were generally comforted by the results, there are some banks that have underwriters that inject a strong bias. Some of the banks we studied presented narrative that was overtly positive to the tune of 10% to 23% of the total words carrying positive sentiment. These banks often had narrative that was very supportive of credit and only 2 or 3 negative bullet points.


Below is a heat map of a sample bank that exhibited the above characteristics. Some of the squares have been blocked out because they had borrower or property specific information. Red represents negative sentiment, purple is mixed, gray is neutral and green is positive. The amount of color is equal to the amount of sentiment. As can be seen, very little negative sentiment is raised relative to positive sentiment. 


Text Analytics


You can also see the types of concepts that convey these sentiments. Negative words around annual debt service, rental income and operating expenses all raise red flags. Positive sentiment around new capital investment net worth, statement quality, financial statements and tax returns are equally troubling. This might be a fine loan, but based on just the sentiment and power of the words used, there is reason for close inspection of this loan.


What A Balanced Underwriting Looks Like


Surprisingly, some banks were consistently slightly negative (averaging approximately -1% sentiment score) in their credit memos. This finding was counterintuitive and is likely indicative of a strong credit culture. 


Here we see a word heat map that is a majority of neutral and mixed sentiment. Positive sentiment has a strong intensity but is almost equally offset by negative sentiment. 


Text Analytic Sentiment Analysis


Putting It Into Action


Text analytics can be used in a number of areas within a bank, but have particular applicability when it comes to monitoring the quality of your underwriting. Even without text analysis, by keeping an eye out for bias, sentiment and intensity, banks can better balance their credit memos. We expected to see much more of a bias and were pleased to find that for the average bank, a materially tilted credit memo is not a risk factor.