Clarification on a timestamp for the candles downloaded from IB

[Kirill Jacobson]

Hello all,

In the past few days I've been downloading historical quotes from Interactive Brokers (1min and 5min candles) and would like to share my observations. First of all, loading data from Interactive Brokers is slow. A limitation of six requests per minute cripples the speed. It takes two minutes to download a month of 1m candles or eight days for a year worth of S&P500 components. I plan to have a look at the Ameritrade API that offers two years of one minute candles and doesn't impose any pacing restrictions. Second, this data feed is not reliable. Every third file (one file per month) had missing candles. It ranged from just few candles per file to a few weeks of missing candles per file (!)

This brings me to %subj% of the post. Looking at the downloaded candles I realized that IB returns time at the beginning of the candle's interval but AQ code treats it as the time at the end of the interval. As a result, a series of 1 minute candles and 5 minute candles downloaded from the same source are not adjusted properly.

Have a look at these three sets of candles (15m, 5m and 1m). The first 15m candle has a time stamp 9:30 and covers three 5m candles (9:30, 9:35 and 9:40). It also covers fifteen 1m candles (from 9:30 till 9:44).

##
## INTC/IB
## timeStamp openPrice highPrice lowPrice closePrice volume highTimeStamp lowTimeStamp
                                                               
9/30/2009 9:30 19.61 19.65 19.48 19.51 42351 9/30/2009 9:30 9/30/2009 9:30 15m
9/30/2009 9:45 19.5 19.51 19.3 19.35 47253 9/30/2009 9:45 9/30/2009 9:45 15m
9/30/2009 10:00 19.34 19.38 19.26 19.33 35938 9/30/2009 10:00 9/30/2009 10:00 15m
                                                               
9/30/2009 9:30 19.61 19.65 19.56 19.57 12042 9/30/2009 9:30 9/30/2009 9:30 5m
9/30/2009 9:35 19.56 19.61 19.48 19.59 12005 9/30/2009 9:35 9/30/2009 9:35 5m
9/30/2009 9:40 19.6 19.62 19.5 19.51 18304 9/30/2009 9:40 9/30/2009 9:40 5m
9/30/2009 9:45 19.5 19.51 19.41 19.41 20203 9/30/2009 9:45 9/30/2009 9:45 5m
9/30/2009 9:50 19.42 19.46 19.36 19.45 11951 9/30/2009 9:50 9/30/2009 9:50 5m
9/30/2009 9:55 19.44 19.45 19.3 19.35 15099 9/30/2009 9:55 9/30/2009 9:55 5m
9/30/2009 10:00 19.34 19.38 19.3 19.32 12881 9/30/2009 10:00 9/30/2009 10:00 5m
                                                               
                                                               
9/30/2009 9:30 19.61 19.65 19.59 19.6 4923 9/30/2009 9:30 9/30/2009 9:30 1m
9/30/2009 9:31 19.6 19.62 19.58 19.61 2548 9/30/2009 9:31 9/30/2009 9:31 1m
9/30/2009 9:32 19.61 19.62 19.59 19.6 1436 9/30/2009 9:32 9/30/2009 9:32 1m
9/30/2009 9:33 19.59 19.59 19.57 19.58 1302 9/30/2009 9:33 9/30/2009 9:33 1m
9/30/2009 9:34 19.58 19.59 19.56 19.57 1833 9/30/2009 9:34 9/30/2009 9:34 1m
9/30/2009 9:35 19.56 19.6 19.55 19.58 3442 9/30/2009 9:35 9/30/2009 9:35 1m
9/30/2009 9:36 19.58 19.61 19.48 19.58 4095 9/30/2009 9:36 9/30/2009 9:36 1m
9/30/2009 9:37 19.58 19.58 19.56 19.57 575 9/30/2009 9:37 9/30/2009 9:37 1m
9/30/2009 9:38 19.56 19.59 19.56 19.58 1484 9/30/2009 9:38 9/30/2009 9:38 1m
9/30/2009 9:39 19.58 19.6 19.57 19.59 2409 9/30/2009 9:39 9/30/2009 9:39 1m
9/30/2009 9:40 19.6 19.62 19.59 19.6 2639 9/30/2009 9:40 9/30/2009 9:40 1m
9/30/2009 9:41 19.6 19.61 19.55 19.59 4213 9/30/2009 9:41 9/30/2009 9:41 1m
9/30/2009 9:42 19.58 19.59 19.51 19.51 4552 9/30/2009 9:42 9/30/2009 9:42 1m
9/30/2009 9:43 19.51 19.55 19.5 19.53 3019 9/30/2009 9:43 9/30/2009 9:43 1m
9/30/2009 9:44 19.54 19.54 19.5 19.51 3881 9/30/2009 9:44 9/30/2009 9:44 1m

9/30/2009 9:45 19.5 19.5 19.41 19.45 6141 9/30/2009 9:45 9/30/2009 9:45 1m
9/30/2009 9:46 19.45 19.51 19.45 19.5 5232 9/30/2009 9:46 9/30/2009 9:46 1m

Is this a bug? Should this be fixed? I have changed the code to add one time frame to each candle. Now it produces correct but strange looking result - a daily candle for 9/30/2009 now has a time stamp 10/1/2009 00:00:00. Subtracting a nanosecond (or a second) makes the result look even worse. What do you think?

Regards,
Kirill

[Mike Kroutikov-2]

I would say that its a bug. Of course, one can add on-the-fly correction when using candles. But I will vote for just fixing it at the adapter level. Yes, I do not like IB's historic candles too - the IB's api is weird and very limited, so that adapter resorts to all kind of tricks to receive the historic data. Its possible that data loss you experience is an adapter bug. Did you try to re-request the same period? Its interesting to see if it will come with same holes or not...

Kirill, could you open a support ticket for this issue, please?

http://activequant.org/projects/aqt-base/issues

Thanks,

-Mike

On Thu, Oct 1, 2009 at 8:47 PM, Kirill Jacobson <activequant.nabble@...> wrote:

Hello all,

In the past few days I've been downloading historical quotes from
Interactive Brokers (1min and 5min candles) and would like to share my
observations. First of all, loading data from Interactive Brokers is slow. A
limitation of six requests per minute cripples the speed. It takes two
minutes to download a month of 1m candles or eight days for a year worth of
S&P500 components. I plan to have a look at the Ameritrade API that offers
two years of one minute candles and doesn't impose any pacing restrictions.
Second, this data feed is not reliable. Every third file (one file per
month) had missing candles. It ranged from just few candles per file to a
few weeks of missing candles per file (!)

This brings me to %subj% of the post. Looking at the downloaded candles I
realized that IB returns time at the beginning of the candle's interval but
AQ code treats it as the time at the end of the interval. As a result, a
series of 1 minute candles and 5 minute candles downloaded from the same
source are not adjusted properly.

Have a look at these three sets of candles (15m, 5m and 1m). The first 15m
candle has a time stamp 9:30 and covers three 5m candles (9:30, 9:35 and
9:40). It also covers fifteen 1m candles (from 9:30 till 9:44).

##
## INTC/IB
## timeStamp     openPrice       highPrice       lowPrice        closePrice      volume
highTimeStamp    lowTimeStamp

9/30/2009 9:30  19.61   19.65   19.48   19.51   42351   9/30/2009 9:30  9/30/2009 9:30
15m
9/30/2009 9:45  19.5    19.51   19.3    19.35   47253   9/30/2009 9:45  9/30/2009 9:45  15m
9/30/2009 10:00 19.34   19.38   19.26   19.33   35938   9/30/2009 10:00 9/30/2009
10:00   15m

9/30/2009 9:30  19.61   19.65   19.56   19.57   12042   9/30/2009 9:30  9/30/2009 9:30
5m
9/30/2009 9:35  19.56   19.61   19.48   19.59   12005   9/30/2009 9:35  9/30/2009 9:35
5m
9/30/2009 9:40  19.6    19.62   19.5    19.51   18304   9/30/2009 9:40  9/30/2009 9:40  5m
9/30/2009 9:45  19.5    19.51   19.41   19.41   20203   9/30/2009 9:45  9/30/2009 9:45  5m
9/30/2009 9:50  19.42   19.46   19.36   19.45   11951   9/30/2009 9:50  9/30/2009 9:50
5m
9/30/2009 9:55  19.44   19.45   19.3    19.35   15099   9/30/2009 9:55  9/30/2009 9:55  5m
9/30/2009 10:00 19.34   19.38   19.3    19.32   12881   9/30/2009 10:00 9/30/2009 10:00
5m


9/30/2009 9:30  19.61   19.65   19.59   19.6    4923    9/30/2009 9:30  9/30/2009 9:30  1m
9/30/2009 9:31  19.6    19.62   19.58   19.61   2548    9/30/2009 9:31  9/30/2009 9:31  1m
9/30/2009 9:32  19.61   19.62   19.59   19.6    1436    9/30/2009 9:32  9/30/2009 9:32  1m
9/30/2009 9:33  19.59   19.59   19.57   19.58   1302    9/30/2009 9:33  9/30/2009 9:33  1m
9/30/2009 9:34  19.58   19.59   19.56   19.57   1833    9/30/2009 9:34  9/30/2009 9:34  1m
9/30/2009 9:35  19.56   19.6    19.55   19.58   3442    9/30/2009 9:35  9/30/2009 9:35  1m
9/30/2009 9:36  19.58   19.61   19.48   19.58   4095    9/30/2009 9:36  9/30/2009 9:36  1m
9/30/2009 9:37  19.58   19.58   19.56   19.57   575     9/30/2009 9:37  9/30/2009 9:37  1m
9/30/2009 9:38  19.56   19.59   19.56   19.58   1484    9/30/2009 9:38  9/30/2009 9:38  1m
9/30/2009 9:39  19.58   19.6    19.57   19.59   2409    9/30/2009 9:39  9/30/2009 9:39  1m
9/30/2009 9:40  19.6    19.62   19.59   19.6    2639    9/30/2009 9:40  9/30/2009 9:40  1m
9/30/2009 9:41  19.6    19.61   19.55   19.59   4213    9/30/2009 9:41  9/30/2009 9:41  1m
9/30/2009 9:42  19.58   19.59   19.51   19.51   4552    9/30/2009 9:42  9/30/2009 9:42  1m
9/30/2009 9:43  19.51   19.55   19.5    19.53   3019    9/30/2009 9:43  9/30/2009 9:43  1m
9/30/2009 9:44  19.54   19.54   19.5    19.51   3881    9/30/2009 9:44  9/30/2009 9:44  1m

9/30/2009 9:45  19.5    19.5    19.41   19.45   6141    9/30/2009 9:45  9/30/2009 9:45  1m
9/30/2009 9:46  19.45   19.51   19.45   19.5    5232    9/30/2009 9:46  9/30/2009 9:46  1m

Is this a bug? Should this be fixed? I have changed the code to add one time
frame to each candle. Now it produces correct but strange looking result - a
daily candle for 9/30/2009 now has a time stamp 10/1/2009 00:00:00.
Subtracting a nanosecond (or a second) makes the result look even worse.
What do you think?

Regards,
Kirill
--
View this message in context: http://www.nabble.com/Clarification-on-a-timestamp-for-the-candles-downloaded-from-IB-tp25708937p25708937.html
Sent from the ActiveQuant mailing list archive at Nabble.com.

_______________________________________________
ccapi mailing list
ccapi@...
http://activestocks.de/cgi-bin/mailman/listinfo/ccapi

_______________________________________________
ccapi mailing list
ccapi@...
http://activestocks.de/cgi-bin/mailman/listinfo/ccapi