Bootstrap Hypothesis Tests in Credit Risk

Bootstrapping in Credit Risk

• Generally underutilized in the credit risk area.

• Bootstrapping finds application across various domains within credit risk modeling.

• Particularly useful for testing metrics where there is little consensus on the standard error of the estimate or when statistical testing procedures are absent.

• Various methods exist for calculating bootstrap confidence intervals and p-values, with the percentile method likely being the most commonly used. Computing the p-value is not always straightforward because bootstrapping does not produce data conforming to the null hypothesis.

• Advantages:

  • does not rely on any distributional assumption
  • flexible application across a wide range of metrics
  • useful for identifying bias
  • easy to implement.

• Disadvantages:

  • computational intensity
  • assumption of independence (although this can be addressed to a certain extent)
  • accuracy concerns for smaller sample sizes.

The following examples demonstrate the percentile method with centered values as: estimate - bootstrapped distribution + null hypothesis value and with the p-value calculated as a percentage of more extreme values than the estimate.

Example 1: Population Stability Index (1-sided test)

Dataset:

##   Bin Base cnt. Base pct. Target cnt. Target pct.  PSI
## 1   1       119      0.22         155        0.35 0.18
## 2   2       130      0.24         139        0.31 0.18
## 3   3        39      0.07          24        0.05 0.18
## 4   4       263      0.48         131        0.29 0.18

Testing Hypothesis:
\[H_{0}: PSI \le 0.15\]

Visualization:

## p-value = 21.26%

Example 2: Herfindahl-Hirschman Index (1-sided test)

Dataset:

##         Rating Grade # obs.   DR   HHI
## 1   01 (-Inf,0.0199)    202 0.01 0.194
## 2 02 [0.0199,0.0263)     54 0.02 0.194
## 3 03 [0.0263,0.0369)     96 0.03 0.194
## 4 04 [0.0369,0.0903)    204 0.06 0.194
## 5   05 [0.0903,0.15)    103 0.11 0.194
## 6    06 [0.15,0.197)     41 0.12 0.194
## 7     07 [0.197,Inf)     50 0.32 0.194

Testing Hypothesis:
\[H_{0}: HHI \ge 0.20\]

Visualization:

## p-value = 23.72%

Example 3: Area Under Curve (2-sided test)

Dataset:

## Bootstrapped AUC summary:

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##  0.7355  0.7784  0.7896  0.7897  0.8008  0.8653

## Development sample AUC 79%.

## Application portfolio AUC 75.2%.

Testing Hypothesis:
\[H_{0}: AUC = 0.79\]

Visualization:

## 2*min(c(left-side p-value, right-side p-value))

## p-value = 2.06%