Long-Range Forecast Skill Maps
The temperature and precipitation anomaly forecasts for each month and season
during the period indicated on the forecast
charts were compared with the observed anomalies
(based on 3 categories: ABOVE, BELOW and NEAR NORMAL). The skill maps show
the percent correct. This score is calculated using a 3
by 3 contingency table. The higher the percentage, the better the forecast
over the verification period. A purely random "chance" forecast would be, on
the average, 33 percent correct. However, the data used to calculate the percent
correct cover only 26 years (Derome et al., 2000; Plante
and Gagnon, 2000; Servranckx et al., 2000). Thus, the threshold to be statistically
significant is 45% (with 10% confidence level) and not 33%. With a longer dataset
of, say, 10000 years the threshold would have been 33%. The fact that the threshold
is 45% ensures that a forecast system is not better than 33% only by pure chance
(true 18 times out of 20, 90% of the time). This means that a percent correct
of 45% or less indicates that the forecast does not show skill. These areas
are identified by the grey color on the skill maps. The percent correct can
then be use to evaluate the confidence in the forecast for the current season (month).
In general, seasonal and monthly forecasts for countries located in the mid-latitudes
(e.g. Canada) show low skill. However, since the skill of the forecasts varies
with the season (month) and the geographical location, some useful information can still
be obtained for many locations in Canada. It has to be note that when a large
ENSO phenomena occur (like the El Nino of 1997-1998) the confidence in the forecast
is much higher in Canadian areas teleconnected with the tropical Pacific oceans
(see El Niņo and
La Niņa web pages
for details). For example during the El nino of 1997-1998, Environment Canada
forecasted above normal temperature for nearly all of Canada and the percent
correct of this forecast was 88%. Due to a number of factors the surface air
temperature forecasts are generally much better than the precipitation forecast
(Plante and Gagnon, 2000; Servranckx et al., 2000).
Locate the area of interest on the map and check if the percent correct there
is equal to or greater than 45%. If it is, ex. 60% , this means that in 60%
of the cases during the period indicated on the chart, the correct category
(ABOVE, BELOW or NEAR NORMAL) was forecasted for the season (month) considered. If the
value is lower than 45% (grey areas), the model is not statistically better
than pure chance and hence the confidence on the forecast is very low.
The skill of the 0-3 month forecasts can be quite different from the 3-12
month forecasts because they are made using very different methods. The 0-3
month forecasts are the result of combining 2 comprehensive dynamical three-dimension
models of the atmosphere (see Derome et al. 2000). A sophisticated statistical
scheme that relates the preceding year global sea surface temperature evolution
to the surface air temperature and precipitation in Canada is used for the 3
to 12 month forecasts. Although the skill generally decrease with the forecast
lead times, sometimes it is not the case because of the differences between
the two approachs.
Derome J., G. Brunet, A. Plante, N. Gagnon, G. J. Boer, F. W. Zwiers, S. J.
Lambert, J. Sheng, and H. Ritchie, 2000: Seasonal Predictions Based on Two Dynamical
Models. Submitted to Atmos. Ocean.
The Green Lane™,How to interpret the skill map for the Environment Canada temperature and
precipitation anomaly forecast ?
How to use the percent correct maps ?
References
Environment Canada's World Wide Web Site.
