Understanding Genetic Optimization
TradingSolutions features flexible genetic algorithms which allow the software to intelligently search through a large number of settings to arrive at the best solution much faster than trying each possibility one at a time. This allows you to have TradingSolutions search for the optimal settings for creating signals and predictions.
For example, let’s say you have an entry/exit system that trades when the values of two moving averages cross each other. If you want to find the optimal values of these moving averages between 1 and 100 bars, you could potentially try up to 10,000 potential combinations of the two moving average lengths (100 x 100) in order to find the optimal solution. Genetic optimization can typically find a very good solution to a problem of this type by trying less than 1000 potential combinations.
What Happens Behind the Scenes
Genetic algorithms start with a small group of potential answers called a population. Each member of the population is called a chromosome, which represents one group of settings – in this case, a pair of moving average lengths. Each setting in a chromosome is called a gene.
Genetic optimization begins by testing each chromosome in the population and assigning a value indicating how good a solution it is, called a fitness. After the entire population is tested, a new population is created based primarily on the chromosomes with the best fitness using methods such as crossover, which combines elements of two chromosomes, and mutation, which modifies genes at random.
Each new population is called a generation. Since all possible settings are not being tried, termination criteria need to be established to determine when to stop. These criteria typically take the form of a maximum amount of time or a maximum number of generations. In the end, the chromosome with the best fitness is kept and used to return the best settings.
& For a more complete overview of how genetic optimization works, see What are Genetic Algorithms?
Optimizing Signals for the Best Results
The goal of technical analysis is to produce effective systems for generating profits in your future trades. In TradingSolutions, these systems help you decide when to trade by providing entry/exit signals. These entry/exit signals can be produced by rule-based entry/exit systems, neural network models of optimal signals, or committees of signals. In each case, genetic optimization can be used to optimize the parameters and settings associated with how the signal is produced so that it is more effective at generating profits.
The fitness of the signal should be a measure of how effective it is at generating profits in future trades. Since future data is not available, the settings are optimized to produce the best results over a set of historical data called the optimization range.
A potential problem with optimizing settings based on a specific range of data is that the values selected by the optimization may be overly specialized on that data. In other words, the final solution may be very good at producing profits on the historical data, but not good at producing profitable trades as new data arrives. Because of this, it is important to take several things into consideration when optimizing a signal.
The first consideration is the selection of a fitness measure. Fitness values are how the genetic algorithm rates how good a particular solution is. Therefore, the best fitness values should indicate that a solution will be profitable in all trading, not just trading in the optimization range.
An example that occurs frequently when optimizing on profit is that the most profitable settings for the optimization range may be due to it locating one or two key trades. Unfortunately, the rest of the time, the system provides only marginal performance. When this system is applied to new data, the marginal performance typically continues since the events that caused the key trades occurred only in the optimization range.
To prevent this from happening, better fitness measures are often those which emphasize the consistency of profitability, rather than profitability itself. Examples of these types of measures are the Sharpe Ratio, the Sortino Ratio, and the Calmar Ratio, all of which balance profitability against a negative aspect of the approach, such as volatility or maximum drawdown.
Ä Note: TradingSolutions automatically defaults to a fitness measure such as the Sharpe Ratio. Other fitness measures can be selected in the Trading Style being used to simulate the trading of the signal being optimized.
Another consideration is whether the market conditions in the optimization range are similar to the market conditions that will affect upcoming trading. In most cases, the market conditions will be similar enough to create effective signals. However, if a stock has been going through a massive growth phase during the optimization range and it is now trading in a relatively flat price range, optimizing over this historical data may be ineffective at producing profitable new signals.
To help prevent this from happening, the optimization range is broken up in to three regions. A training set is data which is used to calculate the fitness for each chromosome. An optional cross validation set is used to determine whether each new generation is producing better general values versus just better results in the training set. Finally, a testing set is set aside for you to test the results of the optimized system on new data.
Why Genetic Optimization Can Take A While
The amount of time taken by a genetic algorithm depends primarily on the amount of time it takes to calculate the fitness. If one group of settings (chromosome) takes 1 second to test, 10 generations of 25 chromosomes will take 250 seconds (~4 minutes). However, if each chromosome takes 10 seconds to test, the overall process will take 10 times longer, or about 40 minutes.
As mentioned above, the amount of time taken by genetic optimization is significantly less than trying every potential setting. However, depending on the complexity of the test, this can still be a large amount of time.
In TradingSolutions, the longest tests occur during the optimization of predicted signals. For each chromosome, the following steps need to take place to arrive at a single fitness value:
1. Calculate the values of any optimized inputs.
2. Iteratively train the neural network for a given number of epochs and passes.
3. Optimize the postprocessing of the predicted signal.
4. Simulate the trading of predicted signal to determine the fitness.
Depending on the amount of data being used to train the prediction and the speed of your computer, steps 2 and 3 can take seconds, tens of seconds, or even minutes. This causes the overall optimization to potentially take several hours.
& For more help with optimization, see Optimizing Signals and Predictions.