Training Analysis Page: Normalized Error

The Modify Field Dialog allows you to analyze and modify the properties of a field. The Training Analysis page allows you to view an analysis of the prediction results versus the desired values.

&  For help with predictions, see Predicting and Modeling Financial Data.

Normalized Error Sub-Page Data

This sub-page displays an analysis of the difference between the desired values and the predicted values. This value is normalized using the difference between the desired values and the last known values. This produces an error value in which 0 indicates no error and 1 indicates that the prediction was no better than simply repeating the last known values.

error = 0 perfect prediction

error < 1 prediction better than last known value

error = 1 prediction of previous value

error > 1 prediction worse than last known value

When predicting values using change or percent change, this number is especially significant. An error of around 1 indicates that the network typically predicted a change or percent change of zero. This is a common occurrence and typically indicates that the neural network does not have adequate information to make a valid prediction. As a result, it ends up predicting the mean of all of the changes, which is typically 0.

The "last known value" for predictions of one sample in advance or more is the current value of the desired output. For models of the current value, it is the previous value of the desired output.

Ä    Note: For predictions of two samples or more in advance, this can result in smaller normalized errors since the change in the desired value is typically much larger. This does not mean that these predictions are better than zero or one samples in advance models, just that the normalization results in different results.

The analysis is performed for the training, cross validation, and accuracy testing sets used for the prediction. Each type of analysis is performed on the entire subset. In addition, each type of analysis is also performed for when the value is predicted to increase or decrease. This is useful for determining if predictions in a particular direction are more accurate than others.

Ä    Note: If a value associated with a predicted direction is listed as "n/a", no predictions were made in that direction.

A normalized error value is calculated for both the actual values and the change or percent change values. Typically, these values are very similar.

Normalized Error Sub-Page Analysis

The analysis presented on this page is based on the normalized error values. It detects common characteristics to look for in the data and is intended only as a starting point for evaluating the model. Some common results include:

·      This might be a good / reasonable / weak predictive model.

This is an analysis of the error in the testing set. If no testing set is used, the cross validation or training set error is used. The following table is used:

error < .80 excellent model

error < .85 very good model

error < .90 good model

error < .95 reasonable model

error >= .95 weak model

The evaluation of models in this way is very abstract. Models that are classified as "excellent" or "good" may be good at mirroring the desired data, but may not predict the values in a way that is useful in the way that was intended. Similarly, models that are classified as "weak" may produce values that are still useful. As stated above, this is intended only as a starting point for evaluating the model.

·      It performed better / worse than simply predicting the last value.

This is an explanation of the rationale for the analysis text. As explained above, an error of 1 is equivalent to predicting the previous value. Therefore, the effectiveness of the model is based on its improvement over simply predicting the last known value.

·      It is specialized on the training data.

This is an analysis of the error in the testing set as compared to the error in the training set. If no testing set is used, the error in the cross validation set is used.

This is significant when determining whether the training has produced a model that is good at making generalizations outside of the training set. If a model over-trains or simply memorizes the training data, it may perform very well at predicting values in the training set; however, it will tend to perform poorly at predicting the values outside of the training set. A good predictive model will perform equally well on data in the training, cross validation, and testing sets.

This message may appear if the samples to weights ratio is not adequate for the type of data being used. A good rule of thumb is to have about ten training samples for each weight in the network. Significantly lower ratios may allow the neural network to simply use the weights to memorize the data, rather than make effective generalizations about its characteristics. See the Prediction Model page for a report of this ratio and help for improving it.

·      It is specialized on upward / downward trends.

This is an analysis of the error across all data sets, comparing the error when upward changes are predicted to the error when downward changes are predicted. If the error is significantly better for predictions in a given direction, the model may be specialized on predicting changes in that direction. This is significant when determining whether directional information from the prediction is useful.

This message may appear when the training set contains data that trends mostly in the reported direction. In this case, the network may determine that the average change is in a given direction and use that average change when it does not have enough information to correctly predict a value. A report of the actual number of changes up and down is on the Overview sub-page.

·      It is only producing upward / downward predictions for this data.

This is an analysis of the error across all data sets. It indicates that a model is predicting changes in only one direction. If the training set does not contain predominantly changes in one direction, this is typically an indication that the model did not have enough relevant information to produce an effective model.

If a problem occurred during the training or calculation phase, the analysis will be replaced with a description of the error. A summary of these error messages is displayed on the help for the Modify Field Dialog: Training Analysis page.

How Did I Get Here?

This is a sub-page of the Modify Field Dialog: Training Analysis page.