Stress tests are scenarios created by financial regulators (in the U.S. the Federal Reserve) to determine how prepared financial institutions are for a sudden market downturn. The most recent tests were conducted this summer, and they are necessary to determine whether banks can make stock buybacks or dividend payments.
As statistician George Box quipped, “all models are wrong but some are useful.” Of course, how useful a model is depends on the model itself. Paul H. Kupiec of the American Enterprise Institute in a paper published by the Journal of Financial Stability questions the accuracy of the Fed’s models by comparing them to the events of the 2008 financial crisis.
Multi-year forecasts of bank performance under stressful economic conditions determine large institution regulatory capital requirements and yet the accuracy of these forecasts is undocumented. I compare the accuracies of alternative stress test model forecasts using the financial crisis as the stress scenario. Models include specifications that mimic the Federal Reserve CLASS model and alternatives that use Lasso, the AIC and an abridged set of explanatory variables. A simple single-equation Lasso model has, by far, the best forecast accuracy. Large differences in model forecast accuracy are undetectable from estimation sample statistics. These findings highlight the need for new methods for validating bank stress test models.
The paper is very mathy, but the punchline is that the Kupiec’s simulation of Fed’s Capital and Loss Assessment under Stress Scenarios (CLASS) model produces errors and shows “that the magnitude of the errors increase with the complexity of the model” (a problem statisticians refer to as “overfitting.”)
Kupiec finds that the main variables responsible for this overfitting are those related to the specific banks being tested, while his model relies more on variables related to broader macroeconomic trends. The variation between Kupiec’s simulation and his simulation of the CLASS model makes the case for greater access to the modeling scenarios or more straightforward capital requirements to reduce systemic risk.
Loading...