Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts MS

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Hi TB n josh,

Lots of queries But its good to have 'thinking heads', with different perspectives and individual domain expertise, to put together for general benefit.

While you guys are looking at things from a C++ / AutoIt coding perspective, with respective levels of individual domain expertise, I can only comment from amibroker's perspective.

Let me first get the basic premise of suggested format control out of the way, before I try to address other queries.

As of ami v4.7, importing data via tickmode will do the following

1. Delete the existing data in Ami from the starting point of the date timestamp of the first data point of the imported data.

i.e to say that if the time now is dd/10:45:30 in a live market and you decided to backfill from dd/09:15:00 via tickmode - Ami will delete Any existing data from (and after) dd/09:15:00 (if present in the database) till the current time.

2. And import the backfill data as a replacement.

Since you guys were getting stuck with slow execution as well as CPU hogging during tick data delete via COM calls this way was suggested, as its blazing fast and leverages the existing ami capability.

Caveat: - In live markets, during backfill in the above example, you will lose a couple of ticks from dd/10:45:30 onwards till the time it takes for the backfill of that individual ticker to complete. While it has negligible impact on price, some amount of actual transacted volume will be lost for that time period. Again, the effect is practically negligible unless a huge amount of volume decides to get transacted in those 1 - 3 seconds.

Basically you will need to couple your existing backfill code (without any coding for existing data delete) with the built in ASCII importer in ami to push data via 'tickmode'.

P.S - For starters, one can test this by importing one day, one min resolution data from a txt/csv file over the existing tick data in ami of that day via the ASCII importer, in static market conditions. Try this only for the existing last day otherwise all data ahead of that day will be deleted if any day other than the last day is chosen.

It worked. :thumb::thumb:
Damn it. In my earlier test, data from VWAP was in Descending Order :annoyed: so the first record was of 15:51:33
When I imported two records of 9:15:33 and 9:20:33, AB deleted all records from 9:15:33 and imported these two. That takes less than 1 second.
So when getting data from VWAP, we have to make sure that it is in chronological order. :annoyed:

Thanks mike for your Persistance.

 

[rmike]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

RTD - Dont maintain O/H/L/C. Instead send with customizable resolution ( second / minute ). AB will manage the Bar creation. This would be nice to have but will make sense only if AB adds volumes of all ticks.

Customized resolution would essentially (and practically) mean any resolution above pure tick level which is manipulated by the utility before parcelling and pushing! Getting into customized resolution is extremely tricky and is a veritable slippery pathway down the proverbial rabbit hole!!! Very difficult to predict all the resulting ramifications, if you don't know what you are doing!

Simpler (and more robust) way is to decide the snapshot duration and let ami aggregate snapshots as per selected granularity.

Sure, there will be (inevitable) loss of some ticks but then one's got to get real! The utility is pulling data from NOW/ NEST and am sure that even the most diehard fan of Omnesys/ Reuters will hesitate to stick his neck out by saying that these (crappy) platforms manage to capture all ticks in the first place.

 

[rmike]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

It worked. :thumb::thumb:
Damn it. In my earlier test, data from VWAP was in Descending Order :annoyed: so the first record was of 15:51:33
When I imported two records of 9:15:33 and 9:20:33, AB deleted all records from 9:15:33 and imported these two. That takes less than 1 second.
Thanks mike for your Persistance.

Good to know buddy! And more power to you both!

Can't resist commenting (just kidding, tongue in cheek) though, that had you read the help file properly, you would have known that in 'tick mode' data has to be in ascending order!

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

That changes the whole perspective.
My backfill utility has to use different format file with $tickmode 1.
TB need not do any changes to his code.It is taking into account each and every tick received from RTD server while creating current array.
@TB - Please correct me if my interpretation is wrong.
Those doing backfill during live trading, may miss couple of ticks at that point of time. That wont make any significant impact. I shall modify my scripts and release soon.

We can have AFL to export data at EOD in 1min format and re-import it in AB using $tickmode 1.
@HappySingh, can you do this?
Edit - I have an AFL that exports intraday data into different files one for each stock.

Alternately, those who started late during day, can do Backfill at EOD from VWAP also.

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Can't resist commenting (just kidding, tongue in cheek) though, that had you read the help file properly, you would have known that in 'tick mode' data has to be ascending order!

Yes, I did, after your last post. That is why I opened the data file and tried again when I saw it was in descending order. :rofl:

 

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Thanks mike, backfill and purge of tick should now be fast. For intraday backfill, i would still prefer to overwrite only missed data but thats for another day

Customized resolution would essentially (and practically) mean any resolution above pure tick level which is manipulated by the utility before parcelling and pushing! Getting into customized resolution is extremely tricky and is a veritable slippery pathway down the proverbial rabbit hole!!! Very difficult to predict all the resulting ramifications, if you don't know what you are doing!

Simpler (and more robust) way is decide the snapshot duration and let ami aggregate snapshots as per selected granularity.

Sure, there will be (inevitable) loss of some ticks but then one's got to get real! The utility is pulling data from NOW/ NEST and am sure that even the most diehard fan of Omnesys/ Reuters will hesitate to stick his neck out by saying that these (crappy) platforms manage to capture all ticks in the first place.

Currently, the tool does manage O/H/L/C for specified resolution. This was needed since we can have more than 1 update per second which would overwrite. ( maybe AB does millisecond timestamp too , i didnt check )
It seems to work fine. The bar extremes matched if you compare it with ZT keeping the chart open.
If you ever find any bug for this in future, please pm me. Also the code is gpl, so anyone can look at it to confirm the bug.

That changes the whole perspective.
My backfill utility has to use different format file with $tickmode 1.
TB need not do any changes to his code.It is taking into account each and every tick received from RTD server while creating current array.
@TB - Please correct me if my interpretation is wrong.
Those doing backfill during live trading, may miss couple of ticks at that point of time. That wont make any significant impact. I shall modify my scripts and release soon.

We can have AFL to export data at EOD in 1min format and re-import it in AB using $tickmode 1.
@HappySingh, can you do this?
Edit - I have an AFL that exports intraday data into different files one for each stock.

Alternately, those who started late during day, can do Backfill at EOD from VWAP also.

Yes ticks will be missed ( maybe for upto a minute ) since VWAP will be updated only till last minute.

To minimize this, we can
1. Try to time it to run just after a minute ends.
2. Make an IPC call to NestRTD asking it to buffer data. Another call to release it. This can be done but we will need code changes in both rtd and in the backfill tool. We will have to find the simplest way to talk between the two processes.

Btw, do you plan to support vwap import from Nest index box?

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Currently, the tool does manage O/H/L/C for specified resolution. This was needed since we can have more than 1 update per second which would overwrite.
Let it be as is.

( maybe AB does millisecond timestamp too , i didnt check )
Yes it does but we have to add millisecs to the timestamp by using Windows API. I think it is absolutely unnecessary unless one wants to automate trading OR wants to put blame for one's trading losses on some external factor.

Yes ticks will be missed ( maybe for upto a minute ) since VWAP will be updated only till last minute.

To minimize this, we can
1. Try to time it to run just after a minute ends.
AB imports ascii file of 300-400 ticks at a blazing speed.

2. Make an IPC call to NestRTD asking it to buffer data. Another call to release it. This can be done but we will need code changes in both rtd and in the backfill tool. We will have to find the simplest way to talk between the two processes.
No need. VWAP Backfill creates an ascii file in temporary folder. Better way is to invoke AFL to export current day data at 1min resolution and append it to the file created by VWAP Backfill. Then import it. We have to find way to invoke AFL from COM. I think there is one.

Btw, do you plan to support vwap import from Nest index box?

Yes. It won't take much time to create separate utility.

 

[rmike]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Currently, the tool does manage O/H/L/C for specified resolution. This was needed since we can have more than 1 update per second which would overwrite. ( maybe AB does millisecond timestamp too , i didnt check )

Yes I saw that while putting the utility through its paces
Ami does have the ability for subsecond resolution but its not practically needed in a utility which is pulling data from a platform which itself is slightly behind the market. Maybe for HFT implementation in a colo server..... In our case, this road is to be trodden only for intellectual gratification.

Whenever we initiate backfill, RT data will be loading in AB, at the same time we could create a temp file and store in it. Once our Backfill ends, this temp file could be loaded into AB immediately so that no tick is lost.

Thanks tomcat, you literally 'front ran' my trade here !!!
When tickmode backfill is done, AB locks the arrays till completion, hence the inevitable tickloss. This effect can be circumvented in 'mixed mode' but that is only between EOD & intraday wherein active overwriting of the OHLC takes place as per the adjusted close. In our particular case, the solution provided by tomcat1978 is absolutely kosher!

There are two ways of backfilling

One - Take on backfilling of each ticker sequentially. This is blistering fast and takes less than a second, resulting in minimal (nil for all practical purposes) tick loss per ticker.

Two - Take on simultaneous backfilling of all tickers by calling multiple asynchronous instances of AB COM object. However I don't know if NOW/ NEST allows multiple instances of the statistics window. Also the simultaneous backfill will actually cause a perceptible time increase (dependent upon the CPU cores and the L1 cache in your rig. The rig RAM has no role to play in this event). Therefore it is likely that the advantage of simultaneous backfill will be frittered away by tick loss for more time duration as ami will lock the arrays of the complete market watch until cessation of the operation.

As per my benchmarking, it'd take 3 - 4 secs (ballpark) for a database of 100 tickers.

So I would rather go with the first option with tomcat's solution implemented

P.S - TB I can see that you are a purist and even a minimal tick loss is irksome hence the implementation of tomcat's solution.

 

[rmike]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

To be very honest, the way I see it, there's not much requirement of further changes. The utility is quite sublime the way it is!!! I interjected when I saw you guys getting 'sidetracked' for the backfill process and didn't want you messing with the perfection of RT data transfer while tinkering with the backfill!!!

For the backfill all one has to do is to couple up Josh's AutoIt with tickmode data transport and Voila!

You guys have been so busy coding up a storm that the full import of what you have done here hasn't yet intruded!!! This utility is the fastest that I've ever tested and I am not only amazed but grateful too that it has been offered so generously! This utility practically brings a retail trader with modest means, a mediocre rig and an average data connection at par with a trader using a commercial datafeed!!! Even commercial datafeeds don't guarantee pure tick transport and here your 'homegrown utility' practically delivers tick by tick data from NOW/ NEST with blistering speed.

This utility will likely be the beginning of the end of the dubious 'fly by night' datafeed operators in the Indian context as it is faster than any of their offerings! The piece-de-resistance is that since data is pulled from the trading platform - all sections (commodities, cash, currencies etc) are 'on the tap'.

Amen

 

[friendforevers0007]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

can we make a application similar to yusi? as we have to run multiple windows for backfilling n data feed..anyways excellent job.. n how to backfill past data from nest as it back fills data of same day.