Up to this point, we have discussed how positions change value, and have used relatively complicated math to calculate the statistics of those changes. This has enabled us to construct estimates of the probability distributions of the future losses and therefore estimate VaR. Now testing is required to tie the results back to reality and give confidence that VaR is a true measure of the risks. This is especially important now that the Basel Committee allows banks to use their own VaR models to assess the amount of regulatory capital that they hold for market risks. The goal of this chapter is to detail the tests that should be carried out on VaR calculators to ensure their validity.
There are three different types of tests that should be carried out on the results generated from a VaR calculator:
• Software-installation test
• Profit-and-Loss (P&L) reconciliation test
• Modeled-probability-distribution back-test
Installation testing is carried out by the risk measurement group after the initial installation of the calculator, and whenever changes are made to the software. The purpose of the test is to ensure that the results coming from the software are in accordance with the equations that the risk management group thinks the software embodies, i.e., to make sure there are no mistakes in the code.
A typical approach is to calculate the VaR for different portfolios, covering all the traded-instrument types, and compare the results with a manual, spreadsheet calculation. Another approach is to set up a series of stress tests, changing a whole series of parameters and ensuring that the results change as expected.
The P&L test is carried out daily by the risk measurement group in conjunction with the finance and operations groups. As part of the VaR calculations, the software should value the current position. Changes in the value should closely match changes in the P&L that are reported in the accounting systems. This ensures that all the positions have been included, and the valuations are reasonably accurate.
This reconciliation is made difficult because discrepancies can arise from many sources, including:
• Positions that were recognized in the accounting system but missed by the VaR calculator.
• The VaR calculator’s inability to capture intraday changes in market values and consequent changes in cash balances. For example, if a stock price rose during the day and the trader sold the stock, the amount of cash received by the accounting system would be more than would be expected by the calculator.
• Different sources of market data being fed to the calculator and accounting systems, e.g., the accounting system valuing the portfolio according to prices at mid-day and the calculator using prices at the end of the day.
• Inaccurate valuation models (e.g., using Black-Scholes for pricing complex options).
Significant discrepancies between the estimated P&L from the VaR calculator and the actual P&L from the accounting system indicate a problem. These problems are most often caused by faulty pricing models in the calculator or the mistake of loading only a few of the bank’s positions into the VaR calculator. This can be caused by a broken data link to some of the position-reporting systems.
If the P&L can be well reconciled to the changes in value predicted by the calculator, we can be reasonably sure that the calculator is valuing the bank’s positions correctly. However, the purpose of a VaR calculator is not only to assess the value, but also to assess the probability distribution of possible value changes. Checking this probability distribution requires a more complex test.
Back-testing requires many days of data and is therefore only carried out monthly or quarterly. The purpose of this test is to make sure that the probability distribution (e.g., the VaR) is consistent with actual losses. This is checked through back-testing. Back-testing compares the loss on any given day with the VaR predicted for that day. Figure 8-1 illustrates VaR and the experienced losses over 100 days. The VaR changes slowly from day to day as positions change and as the market volatility changes. The P&L jumps about depending on the actual trading results each day.
Let us define an exception to mean any day in which the actual P&L fell below the calculated confidence interval for the daily VaR. In 100 trading days, we would expect 1 exception (as on day 73 in the figure). In a year of 250 trading days, we would expect 2 to 3 exceptions.
If it was the case that we always got a representative sample, then we could say that our VaR was a good representation of the actual distribution if we only experience exceptions 1% of the time. If we experience exceptions more or less often, we would conclude that the VaR was not an accurate representation of the distribution of losses.
Unfortunately, there is additional complication because the number of exceptions is in itself a random number. Even if the VaR perfectly represents the probability distribution of losses, sometimes the bank will be lucky and the random market movements will cause fewer losses than usual; sometimes they will be unlucky and suffer many losses. This uncertainty in sampling means that it is difficult to tell whether the experienced number of exceptions is due to a poor model or to bad luck.
Fortunately, there is a framework to calculate the probability of having a given number of exceptions. The exceptions are a binomial variable. Binomial variables are those that can have a value of zero or one. Exceptions are binomial because on any given day there either is or is not an exception. If the VaR calculator is correct, then on each day there is a 1% chance of an exception and a 99% chance of there being no exception.
FIGURE 8-1 A Comparison of VaR and Actual Losses over 100 Days
The number of exceptions over 250 days has a Bernoulli distribution. The Bernoulli distribution describes the probability of having a given number of outcomes that are equal to one if a binomial variable is sampled multiple times.
From the Bernoulli distribution, we can calculate the probability of a given number of exceptions occurring, as shown in Table 8-1. From this table, we can see that if the VaR calculator is correct, there is a 13% chance of having 4 exceptions in 250 trading days, and an 89% chance that there will be 0 to 4 exceptions. We can also see that there is only a 0.01% chance of there being 10 or more exceptions. We can interpret this by saying that it is very unlikely to get 10 or more exceptions if the VaR model is correct i.e., if 10 or more exceptions do occur, it is likely that the model is incorrect.
This principle is used by the Basel Committee to check that a bank’s VaR calculator is performing well. If more than 4 exceptions have occurred in the last 250 trading days, the Capital Accords for market risk require that the bank should hold additional capital to compensate for the possible unreliability of the bank’s calculator. Table 8-2 shows that each number of exceptions puts the calculator into a green, yellow, or red “zone.” Corresponding to each number of exceptions, there is a multiplier by which the amount of market-risk capital must be increased. We investigate capital further in the next chapter.
Back-testing should not only be carried out for the whole portfolio, but also for subportfolios to test if they are being measured accurately. This avoids the possibility of inaccuracies in some subportfolios being undetected because they are masked by the rest of the portfolio.
TABLE 8-1 Probability of Exceptions Experienced in 250 Days if the VaR Model is Correct
TABLE 8-2 Relationship between the Number of Exceptions in Back Testing and the Amount of Regulatory Capital to be Held for Market Risks
This chapter demonstrated how to conduct tests on VaR calculators to ensure their validity and to give us confidence that VaR is a true measurement of the risks. Next, we will describe how to use VaR to calculate the capital needed to withstand market risks.