Recently, Quandl interviewed a senior quantitative portfolio manager at a large hedge fund. We spoke about how she builds trading strategies–how she transitions from an abstract representation of the market to something concrete with genuine predictive powers.
 
Can you tell us how you design new trading strategies?
It all starts with a hypothesis. I conjecture that there ought to be a relationship between two instruments, or maybe there’s a new instrument in the market that’s gaining popularity, or maybe there’s an unusual macroeconomic factor I’ve discovered that drives micro pricing behavior. So I write down an equation – a model, if you like – that aims to capture this relationship. Typically it’ll be some sort of process equation that shows how the variables evolve over time, with a random (stochastic) component.
The next step is to find a closed-form solution for this model. Sometimes this is easy; sometimes this takes days and weeks of algebra; sometimes there is no closed-form solution and I have to settle for an approximation. I find Mathematica’s symbolic manipulation toolkit very useful in this stage of the process.
Okay, so now I have a model of the market. I need to test if it’s realistic. At this stage, I usually turn to Matlab. I assume some plausible values for various parameters, and run some simulations. Do the simulated outputs look reasonable? Do they reflect, at least conceptually, the actual dynamics of the market?
Assuming the model passes this sanity check, it’s time to move beyond blue-sky exploration or ideation and into formal research.
 
What do you mean by “formal research”? And why is it necessary?
I mean the transition from an abstract, stylized representation of the market, to something that is concrete and unambiguous, with genuine predictive powers.
It’s hard to build truly predictive models. But it’s very easy to fool yourself into thinking you’ve built a predictive model, when in reality you’ve merely over-fitted, or used in-sample testing, or imposed exogenous knowledge in your rules, or what have you. Most “systems” fall apart in the real world for this precise reason.
I don’t want that to happen to my model; I will be risking real money on it. So over the years I’ve built and refined a slow, steady, systematic approach that minimizes the risk of fooling myself. That’s what I call “formal research”.
 
What steps do you include in your formal research process?
Early on, my biggest fear is data contamination. History is a limited resource; once you’ve run out of historical data to test against, you can’t generate any more. I’m paranoid about not exhausting my supply of uncontaminated out-of-sample data.
So I start by dividing my historical data into non-overlapping chunks. I then randomize so that even I don’t know which chunk is which. (This guards against subconscious biases: for instance, being risk-averse when I know my test dataset is 2008, or being risk-seeking in 2009).
I designate one chunk as my calibration set. I usually use Python for calibration: I use their built-in optimization libraries and have written a few of my own. In this particular example, my parameters are constrained and correlated. So I use a 2-step optimization process called the EM algorithm. Optimizers can be sensitive to initial conditions, so I use Monte Carlo to choose a number of starting points in the solution space. All of this is quite easy to do in Python.
The result of this calibration should be a set of “model parameters” – numerical values – that can be combined with actual market observations to predict other market prices.
Once I’ve calibrated the model, I test it out of sample. Are the predictions stable and the residuals mean-reverting? If not, the model doesn’t work; as simple as that. I try various “tricks” to break the model. For instance, I calibrate on monthly data but test on daily data. Or I test US parameters on Canadian market data. If the model truly reflects underlying economic reality, it should be fairly robust to these kinds of attacks. (Economics does not change when you cross borders).
 
So, you strictly separate in-sample and out-of-sample; you blind yourself to date ranges; you use Monte Carlo to avoid starting-point biases; and you try various robustness tricks. What else do you do to ensure that you’re not fooling yourself?
I place a very high premium on parsimony. If my model requires too many parameters or has too many degrees of freedom, it’s just curve-fitting; not a model at all. So I’m constantly trying to remove factors. If the model keeps working (and remains “rich”) with multiple factors removed, then it’s probably a good one.
A second proof of robustness is if the model works well no matter what trading strategy you build on top of it. If you can only make money using a complex non-linear scaling rule with all sorts of edge conditions, then that suggests a lack of robustness.
Finally, there’s no substitute for data. I think of every possible out-of-sample dataset that I can plausibly test the model on: different countries, different instruments, different time frames, different date frequencies. The model has to work on all of them; else you have selection bias in the results.
 
That sounds comprehensive. What happens next?
Armed with a calibrated model, the next step is to build a PL simulation. Mean-reverting residuals might not suffice if the opportunity set is too small to compensate for bid-ask, or if the occasional blowups kill all my profits. So I need to test an actual trading strategy using my model. Here is where I have to exercise the utmost care: it’s all too easy to curve-fit by adding new free variables, or bias the results with subconscious knowledge, or wish away outliers. Simplicity, strict separation of samples, and intellectual honesty are important here.
I use Excel for back-testing. This is a deliberate choice: Excel is not as powerful as Python, and this means there is an upper bound on how complex I can make my trading rules. This is a good thing: a strategy that requires complexity to be profitable is probably not a good strategy in the first place.
Excel also allows me to see my assumptions made explicit; it’s easy to lose track of such things when you’re working in code. It allows me to visualize performance statistics (risk, return, drawdowns, capital efficiency, Sharpe ratio and so on) quickly and clearly. Even if my model “works”, there’s no guarantee that a trading strategy built around the model will be economically viable, so these statistics matter.
Very few trading models make it past all the above steps: blue-sky formulation and sanity checks; historical calibration and out-of-sample performance; trading strategy back-test and profitability. But for the few that do, it’s now time to move into production. This is a whole different ball game.