S&P 90% Accurate Forecasting w/a Trendline!

My journey into forecasting started in 2005 when I came across an article in AI in Finance magazine called “Tahiti or Bust”. It outlined the experiences of an AI Expert who worked with an S&P trader to forecast the market. By combining neural nets and the trader’s expert knowledge, they were able to retire to Tahiti.

After about 2 years of trying to recreate their success, with little success, I discovered a little thing about the S&P and error measures, I found quite a few articles about people who were able to do very wellt rading in the short term. People like Larry Williams, who reportedly made $1 million trading futures in a year, starting with a $10,000 account.

http://www.amazon.ca/Made-Trading-Commodities-Last-Year/dp/0930233107

It was then that I noticed something about the S&P. Which is, if you look at the log of the S&P, which is that if you look at the long-term log value of the S&P, and draw a regression line, that it is 90% accurate. How many of these trading successes and strategies were based upon the long-term rise of the S&P, and how many were based upon other strategies?

Jurik RSX

One of the indicators that I got into a while back after reading Larry Williams Batting .800 was RSI. It is called the Relative Strength Index, and it is an excellent indicator of strength. He has a version called %R that works very well that is measured a bit differently.

Jurik Research (http://www.jurikres.com/) has an excellent version of these types of indicators called RSX. When I was trading forex for a while at night, I used RSX in combination with the JMA, and it helped a lot with with upswings and downswings. The problem with RSI and %R is that they are very jittery and prone to false signals. RSX seemed to be less prone to false signals, and helped out a lot.

One of the experiments that I did because I was tired of trying of re-installing the Jurik Indicators each time I upgraded computers, and because I wanted to try to use indicators with my own custom trading software was trying to reproduce this type of indicator on my own. One of the things that I tried first was a truncated FFT, essentially using a lowpass filter. Woila! Instant success. While not exactly the same as the JMA or RSX, it produced results close enough to use for trading.

Using Williams %R

One of the indicators that I experimented with in my own trading was Williams %R. I got into that indicator after buying the Larry Williams course Batting 800 (80%). It is easier to calculate than RSI, I found, and it is very useful for identifying strength in the market versus weakness. It is an oscillator. Oscillators go up and down, whereas indicators like moving averages tend to follow the market.

Below is a snapshot of Google (GOOG) showing buy signals that were generated from the %R signal. You can choose a value, 70% or 80%, to buy at. Each of these were good trading signals showing a good time to buy.

Links:
http://trader.snowcron.com/williams.htm

Cleaning Data

One of the problems with data that you might run into early is that not all market data is that clean. When I worked at Chase Manhattan Bank in the mid 1990s, I was on a project where we had to 'scrub' the data because we were creating a huge data warehouse of a ton of price data so they could price every single financial instrument that the bank owned. It contained prices for everything from wheat to stocks to bonds to airplanes. I was the GUI project leader, and responsible for making the tools that were used by risk management for updating the prices with a Java based GUI.

The methodology we used was a standard technique, which was to run forward the data via a windowing technique (take, for example, 200 values), and compare the standard deviation of the data with the data values. If the data value was more than a standard deviation away from the rest of the data, it was suspect. If it was several, it was very suspect. This is a standard technique evidently in the industry.

The rest cannot be disclosed because I signed a non-disclosure agreement, but suffice it to say that you want to make sure that when you're forecasting, that you scrub the data and look for outliers.

I especially had this problem when trading Forex, and with some data exchanges later in my trading experience, because there would be huge outliers and run-ups, run-downs especially in times when the exchanges aren't open. For example, Sunday afternoon and early evening Pacific time, Tokyo is just coming online, and there isn't a lot of trading going on. Sometimes during those hours there can be dramatic up and down jumps that show up badly when you start to do calculation. Filtering these out with data scrubbing can help things dramatically -- otherwise your indicators can show wild jumps.

Market Physics Part II

In my last post about market physics, I talked a little about looking at the speed of the market. I used to trade OEX options while I was working at IBM as a contractor for a little while in 1995, when I first got interested in trading. I traded those because it's the S&P 100 instead of 500 (theoretically fewer stocks to track), plus the options were cheaper to trade than the S&P 500.

This turned into a big problem because I'd enter a trade before I'd go to work, and then I'd stress out during the day, and would wonder what my stocks were doing. It helped that at the time the people I was working with were obsessed with the price of IBM stock, almost as much as I was with my options. (Or moreso, because they had more money vested -- I just played with play money, a few thousand extra that I had on hand). It seemed like they were constantly watching the stock price go up and down all day because they mentioned it several times a day. (Most of them still seemed to get the work done). I would call in every couple of hours to see how my options were doing, but I had to quit trading once I made about $3000, because my options were going up and down thousands every couple of hours, and it was too stressful for me to deal with at work while trying to get work done.

Below is a chart that I made off of the talk that I had before. I plotted the delta (change in price from day to day), labeled rd (or raw delta), the speed (or rate of change), plotted as rv (raw velocity). Looking at the acceleration really helps things, I think, because if you look at it, until the acceleration slows to zero and starts going the other direction, and the velocity starts going in the other direction, does the price start moving in the other direction.

One of the things about indicators is that a lot of traders use moving averages. The problem with them is that moving averages are not a leading indicator. A leading indicator is one that tells you before a change happens that it's probably about to happen or will happen. Velocity and acceleration are leading indicators, in that the market has to slow movement in one direction before moving in another direction. The acceleration has to slow before the velocity slows, and the velocity has to slow before the market turns around to move in another direction. (Usually). The big exception to this is sometimes in the morning, the market just starts out at some value that is quite unexpected (especially if there is overnight or weekend news), or when there is some big news item during the day, that affects prices suddenly.

I plotted the other changes as well, and that's when I started wondering about the labels. In Physics, we studied rate of change, velocity, change of position, etc., but I couldn't remember what we called the higher order changes (these are called derivatives in physics). So I looked it up, and the third derivative (change in acceleration) is supposedly called jerk in the U.S. It's called jolt in Britain. I don't know if this is a joke, or what, (http://en.wikipedia.org/wiki/Jerk_(physics)). The higher order ones are called "snap, crackle and pop."???

At any rate, I'm still looking at the higher order derivatives and how they help, but the jolt is definitely a change in acceleration, and you can look at a slowing or speeding up of acceleration to see when acceleration is going to change, and when the speed changes accordingly. When the acceleration slows, then it seems to be a leading indicator that speed is probably going to slow.