The Grail Forum
December 14, 2017, 10:45:46 pm *
Welcome, Guest. Please login or register.

Login with username, password and session length
News: Coming soon!! Oct 2017: Ant Strategy Explorer with powerful Data mining capabilities. Read more at http://nextgentrading.com
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: Genetic / Walk-Forward Optimizer (GGO/GWFO) - FAQ's  (Read 20667 times)
Wouter
Administrator
Hero Member
*****
Posts: 920



View Profile Email
« on: August 20, 2009, 03:23:17 pm »

Frequently Asked Questions

Copyright 2004-2009 Technovest (Pty) Ltd

1 a. What makes GGO/GWFO so unique?
1 b. TS 8.6 now has its own genetic optimizer. Why do I need the GGO?
2. How do I select between the Prepare/evaluate strategy/portfolio/CASB mode from within the GGO SmartEditor?
3. Why do I get zero or no results with GGO?
4 a. How many variables does the GGO allow?
4 b. What is the ideal optimization ranges and increments to be used with GGO?
5 a. Error: Cannot find DDL library file?
5 b. FileAppend error / FastFileAppend failed?
6. What is the longest period of data GGO/GWFO can handle?
7. How does a Genetic Algorithm work?
8. What makes GGO/GWFO different from any other products currently available for TS?
9. What is a Walk forward analysis?
10. Should one deduct commission&slippage before GGO and GWFO? What about position sizing?

11. If I use the GGO with the last 30% excluded for Walk Forward and then run the data thru GWFO,
will the test still be out of sample(Unseen)?
12. Is there any way to restart GGO where it was previously interrupted?
13. Quickstart procedure with GGO/GWFO?
14. Why not fully automating the registration procedure?
15. How do I select robust parameters using GGO/GWFO?
16. What are the 3 different walk-forward methodologies implemented by GGO/GWFO?  
17a. On which inputs should stress testing be used?
17b. What StressTestSize should I be using?
18. Can I use multiple signals/strategies with GGO?  
19. Does the #Trds column in the GGO Optimization Report mean round-turn trades, or single side?
20. I have a problem in viewing GGO equity graphs or performance summaries?

21. Does GGO automatically Backup my strategy?
22. Does norton anti-virus slow down TS 8.x during optimizations?
23. Why may the Underscore or forward slash not be used as part of a symbol or strategyname?
24. May I run both GGO/GWFO simultaneously?
25. What is the unseen gradiant column in GGO and how to interpret it?
26. How is the number of iterations calculated?
27. What is implications of 0% OOS in GGO?
28. Number of records in GGO report is less than number of iterations performed?
29. A few basics about genetic algorithms (GA's)
30. Does GGO include a Sharpe Ratio as a fitness function?

31. Why do I get GGO parameters that are not direct multiples of the step that I selected for optimization?
32. What is the meaning of each column in the trading history files?
33. How to interpret the GWFO cluster analysis?
How much data should I use when doing the re-optimization?
34. GWFO: Where is the "Settings for the next NN days" found?
35. Which is most important? GGO or GWFO?
36. What happens to the data in the GWFO data folder if I run the same optimization with a different fitness number?
37 Is the GGO/GWFO compatible with TS8.x / TS2000i?
38. What regional settings should be used with GGO/GWFO?
39. What is the concept/benefits behind Stress Testing?
40. How does the Weighted fitness function (Fitness #10 in GGO) works?

41. How does the GWFO work?
42. How do I select the best/most robust system/parameters using GWFO?
43. I get an "Array out of bounds error" when using Concurrent genetic walk-forward?
44. How can I verify the WFO result?
45. In GWFO, how do you determine the parameter names for the suggested values to be used for x number of days?
46. When I run GWFO I get all zero parameters for a specific/all runs?
47. What happens to the data in the WFO data folder if I run the same optimization with a different fitness number?
48. What exactly is Walk Forward robustness based on?
49. Can I specify different fitness functions when walk forward testing ?
50. I'm getting a robustness of >100% on one of my systems. Is that possible?

51. What is the difference between a rolling and anchored walk-forward?
52. How does GWFO calculate the in/out-of-sample windows?
53. How can I audit the GWFO OptimizationSummary and WalkForwardSummary?
54. Should look inside bar back testing resolution be turned on, when using GGO/GWFO?
55. When using GGO, I get an error "Exception EInOutError in Module gadll.dll. Read beyond end of file"
56. Why does some traders select parameters as per the GGO optimization report instead of simply using the parameters suggested by the GWFO?
57. What is the difference between the OOS testing during a regular GGO optimization and the OOS testing done by the GWFO?
58. Where can I read more about walk-forward analysis?
59. Is there an efficient way to use GGO to do some Quick Screening of Strategies?
60. How do I set final input values for my GGO optimized strategy, to be used in real-time?
61. Why are there fewer GWFO trading history files than the number of GGO Iterations performed?
62. My concurrent walk-forward produces results with lots of zeros?
63.What factors mostly influence the execution speed of GGO/GWFO and how much can the GGO/GWFO benefit from multithreads/multi-cores?


1a. What makes GGO/GWFO so unique? What is The Grail methodology?
The Grail Genetic Optimizer (GGO) has an important advantage when compared to other genetic optimizers. Where existing genetic optimizers should be carefully monitored during the optimization process to prevent over-optimization, GGO has a much more sophisticated but also more effective approach to prevent curve fitting. This is because the actual selection of final parameters does not take place in GGO but is made with the help of the Walk-forward optimizer (GWFO). GGO merely selects profitable solutions to be properly walk-forward tested at a later stage by GWFO. GWFO enables the user to perform a detailed walk-forward analysis on those solutions selected by GGO, thereby providing much higher statistical evidence/assurance than other methods.
Even though the genetic/walk-forward optimization approach is not as prone to over-optimzation risks as other genetic optimizers, the GGO also introduces an additonal measure to promote stable parameter selection during the initial genetic optimization stage. This is achieved by performing a stress test on parameter combinations to see how performance would have deteriorated if all the parameters were changed a certain percentage (which can be set by the user). While we strongly recommend to use this feature, users should note that the stress test will increase the total time taken to optimize, by a factor of 2-5 depending of the stress level set by the user (the recommended setting is 3).
Running GGO on its own without the stress test or the walk-forward optimizer would be similar to using any other existing genetic optimizer. By adding the stress test would add another dimension to the level of assurance provided by GGO while also using the walk-forward optimizer adds yet another dimension of protection against curve-fitting.
It is totally up to the user to decide which level of protection is sufficient for his specific needs. However, we would strongly advise GGO users, to always use the Genetic optimizer with level 3-5 stress testing, in combination with the walk-forward optimizer. This will provide users with the highest level of assurance without unrealistic waiting periods.

1b. TS 8.6 now has its own genetic optimizer. Why do I need the GGO?
If you are merely looking for a straight forward genetic optimizer on its own, then the TS genetic optimizer is sufficient, but if you are looking for a comprehensive system evaluation methodology including important features like stress testing and walk-forward analysis, then you need the GGO&GWFO.
Refer http://www.thegrailsystem.com/grail-trading-system-faq.htm for more detail, including a flowchart of The Grail methodology.

2. How do I select between the Prepare/evaluate strategy/portfolio and CASB mode from within the GGO SmartEditor?
To make the GGO SmartEditor front-end more user friendly, it has a Mode selector.
The Mode selector (at the top just next to the speedbuttons) allow you to choose between the following 4 modes:
i) Prepare strategy
Used to prepare a strategy for genetic optimization in TradeStation
ii) Evaluate strategy
Used to evaluate a strategy after it has been optimized in TradeStation
iii) Genetic Portfolio Optimizer (optional)
Used to setup/modify a portfolio of systems and to genetically optimize the portfolio
iv) Computer Assisted Strategy Builder (CASB) (optional)

When choosing a specific mode, GGO will only activate the applicable menu options and tabsheets, so that a user can more easily navigate to perform the required task.

3. Why do I get zero or no results with GGO?
I see empty records where all results are zero and no trades were generated along with normal results.

Possible reasons: The most common instance is the following (also explained in GGO Help, Contents, Troubleshooting, Problems frequently experienced):
Why does the NetProfit and number of trades displayed in the GGO optimization report and Print Log/Debug window sometimes differ from the TS Strategy Optimization report?
EasyLanguage uses the LastBarOnChart function to identify the last bar on the chart which is critical for software like GGO to operate.
However, TradeStation has a quirk where if the strategy being used trades on the next bar, then LastBarOnChart will never become true if TS is used in offline mode. (This is because TS is waiting for the begining of the next bar to happen before if will trigger LastBarOnChart to be true.)
The bottom line is that the LastBarOnChart function cannot be trusted to always identify the last bar.
GGO bypass this limitation by using the LastCalcDate function to test for the end of a strategy. As soon as LastCalcDate becomes true, then GGO assumes the end of the chart has been reached.
Since LastCalcDate already becomes true on the first bar of the last day, we are missing out on any trade/s that happens on the last day.
The cumulative profit until the beginning of the last day is printed in the Print Log/Debug window and exported to the GGO optimization report.
As a result, there can be a slight difference between the figures in the GGO optimization report and the TS Strategy Performance/Optimization report. If you compare the total net profit according to the GGO optimization report with the TS trades list, then you will see that it agrees 100% with the cumulative balance as at the beginning of the last day.

(Note that the NetPrft, GrossP, GrossL, #Trds etc. columns of the GGO Optimization report contain values calculated only for the in-sample period.
That would typically be the first 70% of data if you have used the defaults as per the Out-of-sample window setting on the GA Settings tab of the GGO SmartEditor. The 3rd last column, TotPrft, contains the total profit for the whole period where TotPrft = NetPrft + UnseenPrft )

EasyLanguage has another quirk.
If a strategy uses Open,High,Low or Close as a price input
(e.g. Input: Price(C); or Input: Price(H); or Input: Price(L); or Input: Price(O); )
then if you change the input statement to a var statement, then the strategy will not give the same results or will simply generate no trades.
Thus
Input: Price(C); gives the desired result, but
Var: Price(C); does not generate any trades!
While GGO as part of the strategy modification process rename existing inputs as variables, it follows that if a price Input is renamed by GGO, then the strategy might end up generating no trades. A quick solution to this problem is to manually rename the Var: Price(C); statement back to an input statement (after the SmartEditor has been used). This can be done because GGO anyway ignores O,H,L,C inputs while there is no optimization range applicable.

If the variable Trace is True (defined as part of the GGO modified strategy variables), then an user may watch the Print Log/Debug window during optimization.
If the input values listed in the Print Log/Debug window or in the GGO optimization report appears not to be within the ranges as defined by yourself, then you must just re-create the strategy again and carefully select/double check the optimization ranges for the different inputs.
Note that in current versions of GGO, the output to the debug window is switched off by default, thus Trace is False by default. To have output to the Print Log/Debug window, just change the setting of the variable Trace from (False) to (True), in the GGO modified strategy. Even if Trace is switched off, the optimization results will always be saved in an optimization report. To view the report, in GGO SmartEditor:
Set mode to 'Evaluate strategy', then select  File, Open Optimization Report and select the required report. Then you will be able to scroll up and down through the solutions and view equity graphs on the EquityGraph tab.

If a strategy is scaling (trading multiple entries in the same direction, also called pyramiding) and the strategy implements any of the SetStopLoss, SetBreakEven, SetDollarTrailing, SetExitOnClose functions, then the GGO performance report can also differ slightly from the TS performance report. This is because of the fact that the EasyLanguage PositionProfit function combines the P&L of multiple trades within the same bar into one figure. Thus some information is lost. The PositionProfit function is the only EL function available to export trade P&L's for use by external applications, and thus our hands are tied. Note that the cumulative P&L's will always be accurate but the total number of trades and makeup between longs and shorts may differ. We do not believe that these differences have any significant effect on the overall optimization and evaluation process.

Sometimes a user forgets that he has selected only a portion of the file to be used for optimization, and thus the results in the GGO optimization report will only be for that portion e.g. the first 70% of data. (Note that the cutoff period for the very first test is done using an assumed 10000 bars. Since EasyLanguage does not have a function that provides the total number of bars in a chart, it takes at least one completed test to identify the exact number of bars

4a. How many variables does the GGO allow?
GGO V2.00.xx allows 64 inputs
4b. What is the ideal optimization ranges and increments to be used with GGO?
Unlike an exhaustive search, the time a genetic search takes is NOT proportional to the number of inputs, optimization ranges and increments specified. Thus, when working with a genetic optimizer you don't have to be scared to use larger optimization ranges, neither should you hesitate to use smaller increments for each input being optimized.
If uncertain, rather specify a slightly larger optimization range with the smallest increment that makes logical sense. The GGO won't have a problem even if you by accident have specified an optimization range that was too large.

5a. Error: Cannot find DDL library file?
The reason why you get the "Cannot find DDL library file: GADLL.DLL" error, is because you most probably did not fully complete step 1 of the installation instructions in the Help file, Getting Started:
STEP 1: Install and Register GGO
Run the Grail Genetic Optimizer (GGO) installation program that will create a new folder C:\Program Files\TheGrailGGO.
Copy the files GADLL.dll, FastFileAppend.dll and DirUtil.dll which can be found in this folder to the C:\Windows\System folder.
(If your computer does not have a C:\Windows\System folder you may copy the DLL’s to the C:\Program Files\TradeStation 8.x (Build xxxx)\Program folder)

5b. FileAppend error / FastFileAppend failed
Do NOT to use the  'Real time update' EquityGraph feature and close-down the GGO SmartEditor completely while optimizing in TradeStation. The reason for the FileAppend problem is TradeStation trying to access a trading history file simultaneously while GGO is accessing the file.
Where-as GGO has built in error trapping routines to properly handle such instances, it seems that TS does not always handle in/output error checking properly when trying to access a file that is unavailable at the time.
Thus, by closing down the GGO SmartEditor during the optimization, this problem cannot occur anymore.

6. What is the longest period of data GGO/GWFO can handle?
GGO/GWFO can handle as much data as you want or lets rather say as much data TradeStation can provide you with. In fact it is recommended to use as much as possible data when using a product like GGO/GWFO.
GGO modifies your strategy and then you can use the new strategy to genetically optimize it. Thus you can optimize the new strategy on any chart with any bar interval or number of bars.
The same applies for GWFO, because GWFO simply uses the trading history which has been created by GGO during the optimization process.
The time GGO takes to optimize a strategy will always be the time for TS to execute 6000 tests (if you are using the defaults with stress test level=3).
The time GWFO takes to perform the walk-forward analysis depends on the number of trading history files created by GGO.

7. How does a Genetic Algorithm work?
1) GA’s work with a coding of the parameter set and not the parameters themselves. They exploit coding similarities in a very general way. As a result, they are largely unconstrained by the limitations of other methods (e.g. continuity, derivative existence, unimodality etc.)
2) GA’s search from a population of points and not just a single point. In many optimization methods, we move gingerly from a single point in the decision space to the next using some transition rule to determine the next point. This point-to-point method is dangerous because it is a perfect prescription for locating false peaks in multimodal (many-peaked) search spaces. By contrast, GA’s work from a rich database of points simultaneously (a population of strings), climbing many peaks in parallel; thus the probability of finding a false peak is reduced.
3) GA’s use payoff (fitness function) information, not derivatives or other auxiliary knowledge. To perform an effective search for better and better structures, they only require payoff values (fitness function values) associated with individual strings. This characteristic makes a GA a more canonical method than most search schemes.
4) GA’s use probabilistic transition rules, not deterministic rules. GA’s use random choice as a tool to guide their search toward regions of the search space with likely improvement.
Taken together, these four differences, contribute to a genetic algorithm’s robustness and resulting advantage over other more commonly used search techniques.

8. What makes GGO/GWFO different from any other products currently available for TS?
Its capability to be able to do a detailed walk-forward analysis (and not only a single walk-forward on a piece of unseen data). To be able to fully understand the significance of this approach, we refer users to the following book: "Design, testing and Optimization of Trading Systems" , written by Robert Pardo. The approach followed by GWFO is similar to the approach described by mr. Pardo.

What further distinguishes GGO/GWFO from other optimization/testing software is the different levels of protection that its approach provide against over-fitting.
GGO can be used in 3 modes:
1) Use GGO on its own without stress testing
This is similar to most other genetic optimizers. You will simply be optimizing using all of the data with no walk-forward (out-of-sample) testing. Using this method is risky and can easily lead to over-fitting.
2) Use GGO on its own with stress testing
The Stress testing is an efficient measure to promote stable parameter
selection during the genetic optimization stage. This is achieved by performing a stress test on parameter combinations to see how performance would have deteriorated if all the parameters were changed a certain percentage (which can be set by the user).
We strongly recommend to use this feature, although users should note that the stress test will increase the total time taken to optimize, by a factor of 2-5 depending of the stress level set by the user (the recommended setting is 3).
3) Use the Genetic Optimizer (GGO) together with the Walk-forward optimiser (GWFO) as follow:
The user still has the option to decide himself if he wants to switch stress testing on/off.
When GWFO is also used, another useful feature is available. This is to use only a portion of the data for calculating the fitness function during the genetic search process.
We strongly recommend users to use GGO together with GWFO and then use GGO with stress testing enabled AND only select a portion of the data (typically 70%) to be used during optimization. This will provide users with a high level of protection against overfitting.

By adding the stress test we are adding another (2nd) dimension to the level of assurance provided by GGO. By only selecting a portion of the data to be used during
the optimization process we are adding another (3rd) dimension of protection and thereby the user can ensure that a significant amount of data remain totally unseen during the genetic optimization process.
By performing a detailed walk-forward analysis (using GWFO) at the end we add yet another (4th) dimension of protection against over-fitting. By performing a cluster analysis on multiple walk-forward analysis, we add a final (5th) dimension to ensure the selection of most robust parameters.

It is important to note that other genetic optimizers typically use the total period of data for their optimization. If you have a look at their fitness functions, you will see that it uses the standard TS NetProfit,GrossProfit, GrossLoss, NumWinTrades, NumLosTrades functions in the calculation of the fitness values.
These functions are all calculated on ALL the data, so there is no piece of the data that is really "unseen" by the genetic optimization process.
GGO do NOT use the standard TS functions but rather calculates its own net profit, gross profit, number of winning trades etc. figures, based on the portion of data selected by the user.
Thus a portion of the data remains completely "unseen" by the genetic optimizer. The walk-forward optimizer (GWFO) takes it then one step further, by performing a detailed walk-forward analysis, where the systems' overall robustness is verified by simulating that it has seen/ not seen various periods of data where the in and out-of-sample windows are progressively moved forward to form a complete and reliable picture. Also, because our results are based on a much larger sample than testing a single period of unseen data, it offers much greater statistical validity.

Important differences between GGO and some other genetic optimizers that simply use the total period of data during the optimization process
If you have a look at the fitness functions used by other GO's, you will see that it uses the standard TS NetProfit,GrossProfit, GrossLoss, NumWinTrades, NumLosTrades functions in the calculation of the fitness values.
These functions are all calculated on ALL the data, so there is no piece of the data that is truly "unseen" during the genetic optimization process. Thus users never see the true out-of-sample performance of their newly optimized system until they start losing money in real-time!
GGO do not use the standard TS functions but rather calculates its own net profit, gross profit, number of winning trades etc. figures, based on the portion of data selected by the user.

9. What is a Walk forward analysis?
A Walk Forward Analysis (WFA) is the most realistic simulation of the way a trading system is used in real time. It also answers the following important questions:

Will the trading system continue to make money after optimization?
At what rate can I expect the system to make money on unseen data?
What will be the impact of changes in trend, volatility and liquidity on performance in future?
How often should a trading system be re-optimized?
Answering the above questions provides the following benefits:

It verifies the forward-trading ability of the system, i.e. does the system have life after optimization and will it continue to make money in real time? It detects if a system has been overfit or improperly optimized. (Studies have shown that a randomly chosen, poor or overfit system can make money in one or two walk-forward test, but will not make money over a large number of walk-forward tests) A system that makes profit over a large number of walk-forward tests, is most likely to be successful in future.
It reliably measures the rate of post-optimization profit and risk. A WFA produces a statistical profile of multiple in sample optimizations and out-of-sample trading periods. Because it is based on a much larger sample than testing a single period of unseen data, it offers greater statistical validity. It also makes possible a precise comparison and measurement of the rates of out-of-sample versus in-sample trading profit. (A robust system’s future performance should be at levels similar to those achieved during optimization)
A WFA also provides insight into the impact of trend, volatility and liquidity changes on system performance. While these changes can have a very negative impact on trading performance, a robust system will be capable of responding profitably to such changes.
A WFA determines empirically how often a system should be re-optimized for optimal performance.

10. Should one deduct commission&slippage before GGO and GWFO? What about position sizing?
I believe a system must be optimized taking realistic commission and slippage into account so that we can simulate real-time performance as close as possible.
I believe optimization should be done using a fixed trading size (thus without position sizing).
Because position sizing can lead to exponential increase in trading size during certain periods it can easily skew trading results for a specific period


11. If I use the GGO with the last 30% excluded for Walk Forward and then run the data thru GWFO, will the test still be out of sample(Unseen)?
Using the Walk-forward optimizer (GWFO) on its own simulates walking forward successive out-of-sample windows which remain completely unseen. However when using GWFO together with the genetic optimizer (GGO) , GGO genetically selects parameter combinations to be walk-forward tested by GWFO. Thus one could argue that even though GWFO correctly simulates the out-of-sample windows to be unseen, GGO has already "seen" all of the data during the initial selection process. (This is a trap that many other optimizers falls into and their users are not even aware of it. If you have a look at the fitness functions used by some other GO's, you will see that it uses the standard TS NetProfit,GrossProfit, GrossLoss, NumWinTrades, NumLosTrades functions in the calculation of the fitness values.
These functions are all calculated on ALL the data, so there is no piece of the data that is truly "unseen" during the genetic optimization process.
Thus, no matter whatever measures are implemented to prevent over-fitting, users never see the true out-of-sample performance of their newly optimized system until they start trading it in real-time)
GGO do not use the standard TS functions but rather calculates its own net profit, gross profit, number of winning trades etc. figures, based on the portion of data selected by the user.)
For this reason GGO allows you to exclude a significant portion (e.g. 30%) of the data when using GGO together with GWFO.
However, when the walk-forward optimizer (GWFO) is used on its own, this is not necessary because GWFO accurately treats the moving out-of-sample windows to be totally "unseen".

12. Is there any way to restart GGO where it was previously interrupted?
GGO intermittently save all results and will only reset itself and the optimization report when Iteration=1.
Thus if you had to reboot your computer for some reason while optimizing Iteration between 1....6000, you can continue optimizing Iteration again from where it was, e.g. optimize Iteration between 4589....6000 if your computer crashed while TradeStation was busy
with optimization test# 4589. If you had a power failure and you do not know the exact test run# that the computer was busy with, then you can simply use your best guess, e.g. optimize Iteration again between 4500...6000. At worst GGO might have skipped or repeated a few iterations which is nothing serious.

To identify the exact Iteration# that the computer was busy with last,
you can goto the GGO SmartEditor and use File, Open Optimization Report and load the optimization report GGO was busy with. Then Rank the optimization report according to Iteration# and scroll down to the bottom of the report to see what is the last Iteration#.

13. Quickstart procedure with GGO/GWFO?

Refer GGO/GWFO Help, Contents, Getting Started

For GGO....
1. Download,Install and register GGO
2. Import the function ‘GrailWFO.ELD’ into TradeStation
3. Prepare your strategy manually or using the GGO SmartEditor
4. Optimize trading strategy in TradeStation

For GWFO....
1. Install and register GWFO
2. Import the strategy ‘GrailWFO.ELD’ into TradeStation (if not previously)
3. Prepare Tradestation Strategy
4. Specify trading system optimization ranges
5. Optimize trading strategy in TradeStation
6. Perform Walk-Forward Analysis using GWFO
7. Evaluate Walk-forward Test results

14. Why not fully automating the registration procedure?
The answer is simple: It is in the best interest of users NOT to fully automate the registration procedure (thereby requiring an open internet connection).
Why do I say so? The reason is actually a VERY important one that I believe all TS users should know about.
Few users realize that as soon as you permit any piece of software to freely communicate to the outside world, then you basically give that software provider a free pass to do whatever he wants to do on your computer.
While the software is already installed on your computer, it can in its own time snoop around on your computer extracting sensitive information and pass it on to the software developer without being detected.
While this is mostly true of most trading software that uses some kind of real-time data, it does not pose a risk when you are using software from a large and reputable software provider like TradeStation for instance. Why is that?
In a large company like TradeStation with many programmers at different levels, there exist proper separation of duties. Thus, for any fraudulent activity to take place, it would typically need at least three different employees on three different levels of seniority to work together to achieve anything.
Unfortunately the same does not apply to smaller/3rd party software providers which is sometimes a one/two man show. In such a case those persons can basically do whatever they want and nobody else would know.
This risk is particular to any software that in any way manipulate open source code i.e. your own EasyLanguage code. As soon as software has access to you unprotected code, it is the easiest thing in the world to pass on the code to the software provider without the user knowing anything. (While you would only be using your best strategies, the software do not even have to search your hard drive!)

15. How do I select robust parameters using GGO/GWFO?
If you are using GGO together with GWFO then during the genetic optimization you got results that can be viewed using the GGO SmartEditor on the Optimization report tab.
It is important to realize that GGO and GWFO is looking at the same system from different angles and thus, the moment results from the two angles confirm each other, we have more assurance.

First we could identify within GWFO the parameters that appeared to be most robust over different periods as well as the parameters that came out on top during the last walk-forward run. We write down these parameters and then continue our selection in the GGO SmartEditor. You can rank the results on several different fields but will most probably use the Fitness function as ranking criteria.
Now we are looking to see if we can find any of the parameter combinations suggested by GWFO in the top 50 of GGO. (We are saying top 50, but 50 is just an arbitrary figure to indicate that you should not be only looking at the best 5 for example) What is important is the quality of the results taking into account your own specific requirements.
If only the first 20 ranked systems have very good results in terms of the fitness function and other performance statistics that are important to you, then you would only look at the top20. Now you are trying to see if you can
find any of the systems (as suggested by GWFO) also being highly ranked in GGO. If you can find any such system, then you have so much more assurance because now that system has been confirmed by two different approaches.

Traders who strictly follow an approach of re-optimization at fixed intervals would simply use the suggested parameters for the next XX days (on the last line on the OptimizationSummary tab)
Personally I would follow the above selection methods and then from my final shortlist of favourable parameter combinations, select the one that satisfies most of the critieria that you have defined for yourself.


16. What are the 3 different walk-forward methodologies implemented by GGO/GWFO?
1. Brute force walk-forward
Walk-forward using TS brute force optimization. Recommended for strategies with up to 4 parameters.
2. Genetically sampled walk-forward
Use Genetic Optimizer to genetically optimize and sample permutations for further walk-forward testing. Recommended for strategies with 5 or more parameters.
3. Concurrent genetic walk-forward
Use Genetic optimizer to optimize strategy while concurrently walking-forward. Most comprehensive walk-forward method (with realistic waiting period) for strategies with many parameters. Recommended for strategies with 5 or more parameters.

17a. On which inputs should stress testing be used?
Can one generalise that stress test on a "Switch" variable should be avoided.
Yes, variables that serve as "switches", should never be stress tested.

17b. What StressTestSize should I be using?

When doing a Genetic sample walk-forward it is recommended that you select a Stress test size of 3 as to increase the resolution of testing.
Then the number of iterations will typically be  2000 x 3 = 6000

With a concurrent genetic walk-forward, the number of iterations already is increased 10 fold, so selecting a Stress test size of 3 is overkill.
Thus, if you select the Concurrent genetic walk-forward method, just select a Stress test size of 1 or 2.
You will then end up with 20000 or 40000 iterations which is 3 or 6 times more than the resolution with a Genetic Sample Walk-forward.

Remember that the most time efficient method is the Genetic Sample walk-forward.
So we recommend:
1) Either use the Genetic Sample walk-forward method with a Stress test size of 3,4 or 5 (as set on the GA settings tab of the GGO SmartEditor)
   This will result in a resolution of either 6000,8000 or 10000 iterations
OR
2) Use the Concurrent genetic walk-forward method with a Stress test size of 1 or 2 which will result in a resolution of 20000 or 40000 iterations


18. Can I use multiple signals/strategies with GGO?
When using the GGO SmartEditor it is important to know that one can only use single strategies with the SmartEditor.
The RSI LE and RSI SE are two different strategies in their own right and should be combined into one to prevent duplicate definition of variables.
This should be done manually by yourself, before trying to use the GGO SmartEditor

19. Does the #Trds column in the GGO Optimization Report mean round-turn trades, or single side?
GGO uses the TS TotalTrades function to cumulatively calculate the number of closed-out trades as they occur, thus also RT's

20. I have a problem in viewing GGO equity graphs or performance summaries?
Within GGO goto Setup, Environment Settings and make sure that your default folders are setup as follow:
Walk-forward data directory: C:\Program files\TheGrailWFO\Data
Reports directory: C:\Program files\TheGrailGGO\Reports
If your settings are not as above, then use the DirectoryListbox and Default folder selection box to set the right folders. Press Save/OK when finished.
Also note that you may not use the underscore ("_") or forward slash ("/") as part of the symbolname or strategyname. Refer FAQ23.

21. Does GGO automatically Backup my strategy?
Yes, when the Encode button is pressed, the SmartEditor automatically saves both the original and modified (target) strategy as a backup for later use/retrieval.
The backup strategies are saved as text files in the C:\Program files\TheGrailGGO\EasyLanguage folder.
GGO modified strategies are saved under a filename that is the StrategyName (as specified on the Inputs tab) with a .TXT extension for easy retrieval by any text editor.
Original strategies are saved using the StrategyName with an .ORI extension. Original strategies can be re-used in the SmartEditor, by using the ‘Retrieve Original strategy previously saved as StrategyName’ option at the bottom of the Original tab.

22. Does Norton anti-virus slow down TS 8.x during optimizations?
It is possible. A solution is to exclude the TS folder from realtime scanning (and planned scanning if if during TS use).
Disable NAV autoprotect.

23. Why may the Underscore or forward slash not be used as part of a symbol or strategyname?
GGO/GWFO uses the underscore as seperator to distinguish between the symbol, interval, strategyname and fitness function while Windows do not allow the forward slash to be part of a filename.

24. May I run both GGO/GWFO simultaneously?
No, you may not run both GGO and GWFO simultaneously because they might access the same trading history file/s at the same time resulting in an "I/O error 32".

25. What is the unseen gradiant column in GGO and how to interpret it?
This figure represents a ratio of Out-Of-Sample profit (extrapolated over the entire data period), divided by In-Sample profit (extrapolated over the entire data period). It basically tells you what percentage of the profit achieved by the INS result was actually realized in OOS.
UnseenGrad = (OOS Profit/OOS Bars) / (Abs(INS Profit)/INS Bars) * 100
It calculates the gradient of out-of-sample equity compared to the gradient of the in-sample equity curve.

26. How is the number of iterations calculated?
When using a genetic optimizer, the number of tests (e.g. 2000 or 6000) has got nothing to do with the number of parameters or the ranges being optimized, as would be the case with traditional TS optimization.
The number of tests to be used for purposes of GGO will always be calculated as follow:
Tests = MaxGen x Popsize x StressTestsize
(Maxgen and Popsize are both variables defining the genetic search structure. Users only need to know that the default values of MaxGen=40 and Popsize=50 is sufficient for any strategy with up to 100 parameters and thus it is strongly recommended to use the default values)
For the default settings (with stress testing switched on StressTestSize=3) the calculation is 40 x 50 x 3 = 6000 tests.
Thus we will optimize Iteration between 1...6000
This is regardless of how many parameters and the ranges to be optimized.
From the above calculation it can also be seen that it does not make sense to use a genetic optimizer if you want to optimize less than 4 parameters, since you will most probably be able to do a traditional brute force search in less then 6000 tests.

27. What is implications of 0% OOS in GGO?
If I were to set the OOS percentage to 0% for the GGO optimization run (and rely exclusively on GWFO for out of sample testing), what negative effect--if any--would that have on the optimization process itself? Would I only lose the "UnSeenGrad" info? Or is an OOS set greater than 0% necessary to what GGO does during the optimization process?
The temptation to try 0% is in order to make maximum use of my sample data...and have the parameter sets determined based on the most up-to-date data available.
It would have no negative effect on the optimization/evaluation process to set OOS%=0 for GGO, as long as you are using GWFO together with GGO. However, you would lose the UnseenGrad info in the GGO optimization report.
Thus, as you have correctly stated, you would rely exclusively on GWFO for out-of-sample testing.

If one is using GGO in combination with GWFO then I don't see any need to exclude data from GGO?
If you are using GGO in combination with GWFO, then you do not have to exclude data from GGO, because GWFO excludes multiple walk-forward periods, which provides greater statistical evidence.

28. Number of records in GGO report is less than number of iterations performed?
During the genetic optimization process, the genetic algorithm might generate parameter combinations that have already been evaluated (i.e. duplicates).
To save time, GGO employs efficient hashing techniques to identify any duplicates to prevent them from being re-calculated in TS again.

29. A few basics about genetic algorithms (GA's)
A genetic algorithm uses a population of non-overlapping strings to represent the trading strategy being optimized. These non-overlapping strings are called “chromosomes”. The genetic algorithm then uses three operators – reproduction, crossover and mutation – to genetically manipulate the chromosomes and thereby optimizing the trading strategy in an evolutionary manner.
The genetic algorithm will pass several times through the population of chromosomes, each time applying the genetic operators to the entire population. Each pass through a population is referred to as a “generation”.

30. Does GGO include a Sharpe Ratio as a fitness function?
The Perfect profit correlation (ff#6) is a fitness function that falls in the same "category" as the sharpe ratio and should provide similar (or even better) results than the Sharpe ratio


31. Why do I get GGO parameters that are not direct multiples of the step that I selected for optimization?
GGO automatically converts all increment fractions to one of
the following:1 divided by 2,5,10,20,50, etc....

32. What is the meaning of each column in the trading history files?
The columns in the trading history files are:
Date Time TradeProfit TradeDrawdown TradeBars LastBar Days Position

33. How to interpret the GWFO cluster analysis?
- Do I just look at Robustness or do you look at one that is 47% and if everything else is OK, you trade that?
I would recommend that you analyse the cluster analysis and then determine the area (cluster) that looks most promising.
The 50% rule is a guideline and if a system gives 47% RI (which is close to 50%) and everything else looks OK (e.g. it is surrounded by other coordinates all giving close to 50% or higher), then I would still be happy to trade that system. However, one should be very careful when making exceptions. Personally I would not trade a system if it has a RI<40% no matter how good other statistics look.

- On one system the equity curve continues to go up on unseen data and gives me a robustness of >55% with sufficient (over 500) trades in sample. The other equity curve tails off. I would normally not trade the one that tails off, but the walk forward analysis also passed. What should I do?
Personally I also place a high emphasis on the quity curve. As they say: "A picture is worth a thousand words!".
Thus if two systems give the same results on paper, I will always choose the one with the equity curve that looks closest to a straight line going upward!
We must remember that GGO/GWFO forms a complete system evaluation methodolgy only when one looks at the whole picture, thus looking at the system from different angles.

Below is two examples (also in Help,Contents, Optimial re-optimization window) on how to interpret the cluster analysis tables:

EXAMPLE 1

Re-optimized P&L (annualized)

OOS% \ Runs 5 10 15 20 25 30

10 17782.91 16740.29 12553.13 9700.53 9396.77 9281.76
15 16565.13 9624.54 10230.31 7777.74 7351.62 6485.74
20 11310.33 8036.32 9282.90 6389.17 6581.01 7444.95
25 10537.80 8624.54 3847.79 6818.86 6939.32 8815.25
30 7367.65 7496.13 6643.93 3428.18 6402.49 2579.89

Walk-forward Overall result (Pass/Fail)

OOS% \ Runs 5 10 15 20 25 30
10 FAILED PASS PASS PASS PASS PASS
15 PASS PASS PASS PASS PASS PASS
20 PASS PASS PASS PASS PASS PASS
25 PASS PASS PASS PASS PASS PASS
30 PASS PASS PASS PASS PASS PASS

Walk-forward Robustness Index%

OOS% \ Runs 5 10 15 20 25 30
10 52.9% 58.4% 70.0% 70.3% 62.3% 66.3%
15 53.7% 81.7% 70.1% 69.8% 64.3% 66.0%
20 66.7% 70.0% 65.3% 64.8% 63.0% 61.0%
25 72.3% 67.0% 66.5% 59.0% 66.4% 66.6%
30 75.7% 66.0% 61.0% 65.8% 57.7% 50.8%

Re-optimization interval (days)

OOS% \ Runs 5 10 15 20 25 30
10 39 29 23 19 16 14
15 51 35 26 21 18 15
20 61 39 29 23 19 16
25 68 42 30 24 20 16
30 75 44 32 25 20 17

The four tables represent (in the order listed above):

1) maximum continuously Re-optimized P&L (annualized)
2) best Walk-forward Overall result (Pass/Fail)
3) maximum Walk-forward Robustness Index%
4) optimal Re-optimization interval (days)

for the various walk-forward run and out-of-sample tests performed.

First of all we will look at the 1st, 2nd and 3rd table to determine the cluster which produced the best results. A cluster is defined as a specific x,y coordinate (e.g. Walk-forward Runs=10, Out-Of-Sample%=OOS%=20) and the 8 neighbors immediately surrounding that coordinate.

You will see that GWFO automatically calculates and display for you the optimal coordinate on the ClusterAnalysis tab.

For this example, one can easily see that the max. re-optimized P&L is produced in the cluster with center x=10 runs, y=OOS%=15.

However, when looking at table #2 we see that one of the tests failed the walk-forward analysis, thus we will rather look for the best cluster (with the highest average) in terms of re-optimized P&L where all the neighbors also passes the WFA.

The cluster with center x=10 runs and y=OOS%=20 appears to be the best.

This coordinate is indeed also surrounded with very healthy walk-forward robustness index percentages (table#3). Remember that the robustness index is simply the annualized rates of return for the out-of-sample results divided by the in-sample results.

Table #4 then simply tells you what is the re-optimization interval in days for a given coordinate. In our case, we have finally selected x=10,y=20, thus the optimal re-optimization period is 39 days. Note that accross the whole cluster, using any re-optimization period between 26-68 days (with the corresponding number of walk-forward runs) would still have resulted in good results, thus we have indeed a result that is robust.

If you will be using the system in real-time using an approach of constant re-optimization, you will simply re-optimize your system every 39 days. Even if you do not use an approach of constant re-optimization, then the tables are still very useful because they provide additional prove that your system is robust, by generating profits over a wide range walk-forward run and out-of-sample percentage combinations.

EXAMPLE 2

Re-optimized P&L (annualized)

OOS% \ Runs 5 10 15 20 25 30

10 19291.11 18709.31 13809.98 17066.38 15482.08 14174.17
15 16341.30 14297.19 13152.52 12187.08 14114.28 13606.57
20 14208.05 15463.51 11390.87 11544.66 14597.98 13575.22
25 11135.77 12054.39 15473.71 13392.82 11878.76 15208.33
30 11482.46 11563.75 13434.59 10564.72 11821.49 12805.42

Walk-forward Overall result (Pass/Fail)

OOS% \ Runs 5 10 15 20 25 30
10 PASS PASS PASS PASS PASS PASS
15 PASS** PASS PASS PASS PASS PASS
20 PASS PASS PASS PASS PASS PASS
25 PASS PASS PASS PASS PASS PASS
30 PASS PASS PASS PASS PASS PASS

Walk-forward Robustness Index%

OOS% \ Runs 5 10 15 20 25 30
10 96.3% 99.2% 94.4% 82.1% 73.4% 91.9%
15 101.8% 90.3% 81.1% 94.1% 83.2% 74.2%
20 92.5% 91.2% 87.8% 88.4% 78.9% 77.2%
25 97.3% 95.1% 88.1% 86.0% 93.0% 83.2%
30 86.0% 80.1% 89.8% 82.3% 74.6% 74.1%

Re-optimization interval (days)

OOS% \ Runs 5 10 15 20 25 30
10 32 24 19 15 13 12
15 42 28 22 17 15 13
20 50 32 24 19 15 13
25 56 34 25 20 16 14
30 61 36 26 20 16 14

For this example, the max. re-optimized P&L (table#1) is produced in the cluster with center x=10 walk-forward runs, y=OOS%=15.

When looking at table #2 we see that one of the immediate neighbors even passed with distinction, and all the surrounding neighbors have very healthy walk-forward robustness index percentages (table#3).

Thus we will look no further and stick to this coordinate.

Table #4 tells us the corresponding re-optimization period for coordinate (10,15) is 28 days. However, using any re-optimization period between 19-50 days (using the corresponding number of walk-forward runs) would also have resulted in good results, thus we have selected a result that is robust.

If you will be using the system in real-time using an approach of constant re-optimization, you will be re-optimizing this system every 28 days.

How much data should I use when doing the re-optimization?

The amount of data that you will be using when doing the re-optimization will depend on whether you have selected Anchored or rolling window (not anchored) walk-forward optimization.

If you have not selected anchored (thus rolling) walk-forward then you will for instance re-optimize you system every 28 days (the optimal period for example 2 above) with GGO, using only the last 28 day's data. (Note that you must load sufficient data for TS to satisfy its max. bars lookback setting. Thus if the max. bars TS may reference is 50, then you need to load 78 days history)

If you have selected anchored walk-forward then you will re-optimize your system every 28 days with GGO, using ALL the data starting at exactly the same point in time as what you have used for doing the cluster analysis.

34. GWFO: Where is the "Settings for the next NN days" found?
It is found at the bottom of the OptimizationSummary. Just click on the tab and scroll down to the bottom of the page

35. Which is most important? GGO or GWFO?
Do I give more weight to the outcome in WFGO over what I thought I saw in GGO or just ignore GGO and go for WFGO?
The GGO and GWFO looks at the system from different angles. Personally I would give more weight to the outcome of GWFO (provided sufficient GGO iterations were done), but you should also take into account what the GGO report and equity curve tells you, to form a complete picture

36. What happens to the data in the GWFO data folder if I run the same optimization with a different fitness number?

The GGO reports folder has the fitness number in the folder name, but the WFO data folder does not use the fitness number in the folder name.
The GWFO data folder contains the trading history files generated during the genetic optimization.
It does not matter if you optimize the same strategy with different fitness numbers, because all what will happen is that the GWFO Data folder will contain the trading history files of several optimizations using different fitness functions.
The different GGO reports (named according to fitness) will only access the trading history files generated by that specific optimization.

37. Is the GGO/GWFO compatible with TS8.x / TS2000i?
Yes. GGO/GWFO is fully compatible with TS8.x as well as TS2000i

38. What regional settings should be used with GGO/GWFO?
Not only for CASB, but also for GGO/GWFO, the regional settings should preferably be English/USA.
In Windows, goto Regional & Language settings and make sure that the Number setting is 123,456,789.00 and NOT 123.456.789,00

39. What is the concept/benefits behind Stress Testing?
The stress testing feature works as follow:

Let's say the GA generated the following values for a strategy with 3 inputs:

10 20 30

With Stress test level=1 (stress testing off) GGO will only evaluate this combination.

With Stress test level=2 GGO will subtract 1 x Stress test increment (default=10%) to each input and also evaluate the following additional combination:

9 18 27

With Stress test level=3 GGO will both add and subtract 1 x Stress test increment to each input and
evaluate the following additional combinations:

9 18 27
11 22 33

With Stress test level=4 GGO will both add and subtract 1 x Stress test increment to each input and also subtract 2 x stress test increment thus evaluating the following additional combinations:

9 18 27
11 22 33
8 16 24

With Stress test level=5 GGO will both add and subtract 1 and 2 x Stress test increment to each input, thus evaluating the following additional combinations:

11 22 33
9 18 27
8 16 24
12 24 36

and so forth......

To summarize:
Using the default stress test increment of 10%, the stress test values are calculated as follow for different stress test level settings:

Level Range
1 +/- 0%
2 -10%
3 -10% , +10%
4 -20% , -10%, +10%
5 -20% , -10%, +10%, +20%
6 -30% , -20%, -10%, +10%, +20%

Thus you can see that stress testing is not a test that must be passed, but it is the process of "stressing" the system by varying the inputs a certain % in either direction.

40. How does the Weighted fitness function (Fitness #10 in GGO) works?

Fitness #10 gives more control over the selection process based on multiple aspects considered relevant for the desired trading strategy.
It allows the user to assign different weights to different fitness functions.
You can also define cutoff levels beyond which the partial fitness score remains unchanged.

This allows you to create a compound fitness function for instance:

-Look for maximum net profit (Fitness #1)
Values of 10,000$ or more give a max partial score.
Values of 2,000$ or less give 0 partial score.
Assign a weighting factor of 2 for this criterion.

-Try also to maximize the Net profit to Max DrawDown ratio (Fitness #2)
Values of 15 or more give a max partial score.
Values of 5 or less give 0 partial score.
Assign a weighting factor of 1 for this criterion.

-Finally maximize the Net profit per trade (Fitness #9)
Values of 100$ or more give a max partial score.
Values of 15$ or less give 0 partial score.
Assign a weighting factor of 3 for this criterion.

Note that the architecture of the TradeStation/EasyLanguage environment differs
greatly from other trading platforms and makes it unfeasible to do a normalization
only at the end of an optimization. For that reason GGO normalizes the weighted
fitness calculations during the optimization itself. It follows that during the first
100 or so iterations of any new optimization, the maximum and minimum values are insufficiant to do an accurate normalization. To solve this problem, GGO uses a sliding scale to penalize the fitness calculations during the first 200 iterations with between 0-50% depending on the number of the iteration.
Once 200 iterations has passed, the maximums and minimum are accurate enough for a proper normalization. This method is the most efficient way to implement normalization in a TS/EasyLanguage environment and saves a lot of calculation time while still being sufficiently accurate.

What is appropriate lower and upper threshold boundries for each of the fitness functions (fitness #1 to #9) one should consider when using fitness#10?
It is very difficult to provide threshold levels that would be universally applicable.
The range of any fitness function that includes Net Profit can vary substantially from strategy to strategy depending on the amount of data used, the big point value of that instrument etc.etc.
The threshold values for a fitness function like Net Profit on its own is also purely subjective and should be determined by each trader himself depending on his own preferences.

The weighted fitness function is indeed an advanced option and is certainly not recommended for newcomers to GGO.
If you want to properly determine upper/lower threshold levels for a given instrument given a certain period of data, I would suggest that you first
run a very short GGO optimization on the strategy but using the individual fitness functions seperately.
Then closely watch the typical range for each fitness function.
Finally, recreate your GGO strategy using appropriate threshold values for the weighted fitness function.

In general the following threshold values should do a good job in most cases:

Fitness #1: Net profit
Lower threshold = The minimum profit amount you require
Upper threshold = The profit amount you realistically can expect

Fitness #2: Net profit / Max DrawDown
Lower threshold = 5
Upper Threshold = 15

Fitness #9: Net profit / Trade
The threshold values will be determined by the strategy type. For instance, for a scalping strategy a lower threshold of $15 and upper threshold of $100 would be appropriate while a longer term strategy might need higher values e.g. Lower threshold=100, Upper threshold=400.
In general, the lower threshold should be the minimum average trade required, while the upper threshold should be the average trade realistically expected.

If you are uncertain about which threshold values to use, then we recommend to use the weighted fitness function as is with the default lower threshold=0 and
upper threshold=9999999.
Thus your weighted fitness function will effectively impose no threshold values, but will simply be used to combine more than one fitness function into one with a certain weight assigned to each.


41. How does the GWFO work?
Using GWFO consists of the stages:
1) Optimization of your strategy in TradeStation 2) Walk-forward analysis using GWFO standalone
If GWFO is used on its own you will be doing an exhaustive search during stage 1 and then for all realistic purposes you are limit to a maximum of 6 paramters with limited parameter ranges as you have noticed yourself. This is because the optimization in TradeStation in itself is a slow and time consuming process.
Because an exhaustive search is done the number of permutations optimized is increased exponentially very every additional parameter being optimized.
For this reason you should use GWFO together with the Genetic Optimizer (GGO). During stage 1, GGO now performs a genetic search/optimization
(instead of an exhaustive search if GWFO is used on its own) and genetically select permutations/ parameter combinations to be walk-forward tested by GWFO later during stage 2. In this way you can walk-forward test a system over a much wider range for each parameter and also optimize more parameters if you want to.
In short: Start by optimizing your strategy using the GGO (The GGO SmartEditor will automatically prepare your trading strategy).
During stage 1 your GGO modified strategy will create trading history files for each parameter combination searched. Once you are finished with stage 1 (the optimization in TS), you can continue with the GWFO standalone. GWFO will then use the files created during the genetic optimization stage, and walk-forward those files.


FAQ 42 - 59 continues at http://www.thegrailsystem.com/forum/index.php?topic=4.0
« Last Edit: August 07, 2010, 09:49:13 am by Wouter » Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!