September 2026
My loss forecast hit the total. It was wrong everywhere underneath.
A backtest of a small-business loan-loss forecast on 1.4 million public SBA loans, and what it says about how lenders check their own numbers.
Follow-up to My AI Suggested a Technique from Cancer Research (May 2026). That piece made the case for survival curves on loan data. This one tests the idea at scale, on public data, and fixes a trap in the simple version.
The background
Every Lender Has to Guess Its Future Losses
When a lender makes a loan, some of those loans won't be paid back. The lender doesn't know which ones, but it has to estimate how much money it will lose across the whole book and set that amount aside. That set-aside is the loss reserve. It drives reported earnings, how much capital the lender needs, and what it charges borrowers.
A reserve is a forecast, and a forecast is only as good as its track record. The honest way to judge one is a backtest: pretend it's an earlier date, make the forecast using only what was known then, and compare it with what actually happened.
I wanted to run that test on real small-business loans. My day job is portfolio analytics for a small-business lending fund, but that data is confidential. The U.S. Small Business Administration publishes every loan in its 7(a) program: about 1.4 million loans since 2000, each with its size, its term and how it ended (paid off, written off, or still open). It's public, and anyone can check my work.
The exercise
Stand in 2020 and Forecast the Next Six Years
- Learn. Take every loan open on 30 June 2014 and measure how many were written off over the next six years, split by loan length, age and size.
- Forecast. Take every loan open on 30 June 2020 (283,358 of them) and apply those rates. Nothing after June 2020 is used.
- Check. Six years later the outcomes are all in. Compare the forecast with what really happened.
This is the question a reserve answers: of the loans on the books today, how much will we lose? Most forecasts are judged on the total. I also checked the pieces.
First, a fix to May's tool
A Paid-Off Loan Can't Default
In May I argued that survival curves beat a single default rate, because they show when loans go bad, not just how many. That still holds. But the simple version of the curve has a trap. It treats a loan that paid off early as if it had simply dropped out of view, so the math keeps assuming it could have defaulted later.
On a book where most loans pay off, that inflates losses badly. On SBA loans with terms of 7 years or less, the simple version overstated 10-year losses by 25 to 33 percentage points for loans made in 2004–2009. The fix is to count a payoff as the loan leaving the book for good (statisticians call it a competing risk). With that fix, the curves match the real share of loans written off to within 0.002 points. Everything below uses the fixed version.
What I found
The Total Was Right by Accident
For short-term loans, the forecast said 22.5% of the money would be written off. The real number was 22.3%. On a summary slide, that looks like a model that works.
Split the same loans by how long they had been on the books, and the forecast misses badly in both directions. Newer loans lost far more than predicted, and older loans far less. The two errors happened to cancel out.
Why were the old loans over-forecast? The forecast learned from 2014. Back then, the short-term loans that had lasted five years were mostly survivors of the 2008 crisis, and many were probably already in trouble but not yet written off. The 2020 book wasn't like that. The timing supports this: in 2014, 37% of those loans' later write-offs came within a year, against 27% for the same group in 2020. It fits the explanation. It doesn't prove it.
A loss reserve can reconcile in total and still be wrong in every segment. Checking only the total would have passed this one.
The second finding
Losses Show Up Years Late
So a young loan book always looks healthier than it is, and a forecast that trusts early results will come in low. Backtested from 2020, the standard method for this (chain-ladder, which scales up the losses seen so far) under-forecast the 2017 short-term loans by 7 points for exactly this reason.
If write-offs are the only thing you watch, you find out about a bad loan years after you made it.
What I'd do on a real book
Three Habits This Points To
- Check the reserve by segment, not just in total. By loan age, product and channel. A total that ties out proves very little.
- Watch payments, not write-offs. Slowing payments show trouble months or years before the write-off does.
- Rebuild history as it looked then. Loan files store today's status. Any backtest has to strip out what wasn't known yet, and prove that it did.
Where this came from
These habits come from running analytics on a small-business lending book at CSL Capital. On that book, payment pacing flags a slowing merchant long before any write-off. The fund's historical reports once showed today's loan status on past months, which made the past look cleaner than it was. I took the lessons, not the numbers: no CSL figures appear here.
Every check in this analysis ships with a deliberately broken version that has to fail, so I know the check can catch something. A second AI model reviewed the method and found issues, which I fixed. One of them was my own mistake: a test that quietly deleted a year of data from the steps after it.
Method and full numbers
The test. Loans open on each date are grouped by loan term (up to 7 years, 7–15, over 15), months on the books (0–12, 13–36, 37–60, 60+) and size (under or over $150K). Each group's six-year write-off rate from the 2014 book is applied to the matching 2020 group.
| Loans open June 2020 | Loans | Forecast | Actual |
|---|---|---|---|
| Term up to 7 years | 101,363 | 22.48% | 22.27% |
| Term 7–15 years | 117,354 | 7.52% | 3.68% |
| Term over 15 years | 64,641 | 7.66% | 1.56% |
| All | 283,358 | 9.12% | 4.45% |
Share of original loan dollars written off within 72 months. Counted by number of loans, the short-term result is 12.28% forecast against 12.26% actual.
Timing of write-offs (loans made 2004–2012): the median was 46 months for terms up to 7 years, 48 months for 7–15 years and 95 months for over 15 years.
Two forecasting methods, backtested from 2020. Chain-ladder came in 4.0 and 7.2 points low on the 2016 and 2017 short-term loans. Bornhuetter-Ferguson, which leans on an expected loss rate instead, came in 2.7–3.4 points high on long-term loans, because its expected rate came from crisis-era loans.
Checks, each with a broken version that must fail:
- Every one of 1.62 million source rows, and every dollar, reconciles by fiscal year.
- Loss curves count a payoff as the loan leaving the book. The simple version overstates 10-year losses by 25–33 points.
- The rebuilt 2020 view rejects any later event. The broken version let through 221,630 of them.
- The 2014 learning window must close before the 2020 test date.
Limits.
- No payment history in the public file, so there are no delinquency stages to track.
- Write-offs are gross. Recoveries aren't in the data.
- Dollars are original loan amounts, not balances.
- History is rebuilt from today's file. Payoffs and write-offs roll back cleanly, but later edits to a loan's amount or term can't.
- SBA loans carry a government guarantee, which changes how lenders behave. I took methods from the CSL book, not numbers.
Tools: Python and DuckDB. Aalen-Johansen estimator with payoff as a competing risk. Code available on request.
Sources
- U.S. Small Business Administration, 7(a) loan-level FOIA files, fiscal 2000 onward, as of 30 June 2026. Loans that were never disbursed are excluded.
- Earlier piece: My AI Suggested a Technique from Cancer Research (May 2026).