Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts MS

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

No need. VWAP Backfill creates an ascii file in temporary folder. Better way is to invoke AFL to export current day data at 1min resolution and append it to the file created by VWAP Backfill. Then import it. We have to find way to invoke AFL from COM. I think there is one.

It would be nice if we can do fast export of today's data using AFL from COM.
Right now, i will still stick with the manual backfill process as i need it only for two scrips and i dont want to overwrite all with 1 min bars.
Fast export will mean i can parse it and fill gaps only.

But anyway, i think you only need to fetch the bars within the last minute that is not included in VWAP. This should be fast enough as its max 59 bars per scrip. AFL may be better but quotes from com may also be ok. check.
You can do this just before calling AB import to minimize tick loss.

When tickmode backfill is done, AB locks the arrays till completion, hence the inevitable tickloss.

There will also be data loss for the last minute. If we run say at 09:30:30, VWAP statistics will not include 30 seconds data from 09:30.

There are two ways of backfilling

One - Take on backfilling of each ticker sequentially. This is blistering fast and takes less than a second, resulting in minimal (nil for all practical purposes) tick loss per ticker.

Two - Take on simultaneous backfilling of all tickers by calling multiple asynchronous instances of AB COM object. However I don't know if NOW/ NEST allows multiple instances of the statistics window. Also the simultaneous backfill will actually cause a perceptible time increase (dependent upon the CPU cores and the L1 cache in your rig. The rig RAM has no role to play in this event). Therefore it is likely that the advantage of simultaneous backfill will be frittered away by tick loss for more time duration as ami will lock the arrays of the complete market watch until cessation of the operation.

As per my benchmarking, it'd take 3 - 4 secs (ballpark) for a database of 100 tickers.

So I would rather go with the first option with tomcat's solution implemented

Although i have not tested this, I think josh tool imports using one import call only. We dont expect import to take long so using multiple threads may be overkill anyway.
That will also need AB import to run multithreaded on same db. I dont know if it supports this.

Edit - Blocked AB during backfill import will not cause any tick loss for us. RTD fetch and AB calls are done in two different threads for this reason.
The only loss will be for any ticks written after backfill tools reads data but before calling AB.import().
This can be avoided by syncing the processes or by using same tool for both tasks.

P.S - TB I can see that you are a purist and even a minimal tick loss is irksome hence the implementation of tomcat's solution.

We have data loss in ZT feed itself, so if we can avoid adding more than why not. If RTD and Backfill process is synchronised then there wont be any loss.
Anyway, for now i stick to plain backfill

This utility will likely be the beginning of the end of the dubious 'fly by night' datafeed operators in the Indian context as it is faster than any of their offerings! The piece-de-resistance is that since data is pulled from the trading platform - all sections (commodities, cash, currencies etc) are 'on the tap'.
Amen

Only Nest and NOW are supported which is what i need. Any tool with RTD server can be made to work.

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

@ TracerBullet,

NestRTD is creating a file "quotes.rtd" in the temporary CSV folder for importing into AB. It is opened for writing for writing fresh quotes and closed after writing.
When does it delete the file? Or does it delete previous file while opening?
I did not find any code for deleting it.

 

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

@ TracerBullet,

NestRTD is creating a file "quotes.rtd" in the temporary CSV folder for importing into AB. It is opened for writing for writing fresh quotes and closed after writing.
When does it delete the file? Or does it delete previous file while opening?
I did not find any code for deleting it.

There is no delete. The write is in Worker::writeCsv(). When we open the file, by default its contents get erased by ofstream.

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

There is no delete. The write is in Worker::writeCsv(). When we open the file, by default its contents get erased by ofstream.

I have an idea. :!
1. Open one more file in append mode. Name of file should have time stamp HHmmquotes.rtd. Say 0915quotes.rtd or quotes0915.rtd
2. AB will keep importing from quotes.rtd.
3. After 1 min file name will change to 0916quotes.rtd.
4. Old files will remain as they are. We delete them while closing NestRTD.
5. We start NestRTD live feed.
6. VWAPBackfill will import data upto last minute, somewhere during the current minute. All ticks in memory for current minute lost.
7. VWAP will import datafile for current minute from the one created by NestRTD. All ticks restored.
8. Only problem will arise if RTD server gives data of timestamp earlier than one minute. We filter out such quotes while creating 1 second candles.
9. If required, VWAP Backfill can find out time stamp of latest quote in it and backfill from NestRTD files also accordingly.

This should not require much code change.

This will be useful if there is a disconnect during the day and there is no data feed for more than 1 minute for whatever reason.
For fresh start during the day, Backfill can be done after 1 minute without TICKMODE while NestRTD is running. There will be no data loss.
Option can be given in Backfill to overwrite previous data

 

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

I have an idea. :!
1. Open the file in append mode. Name of file should be appended to time stamp HHmmquotes.rtd. Say 0915quotes.rtd
2. AB will keep importing from it for 1 min in TICKMODE. There may not be more than 200 ticks during 1 minute for 30 scrips. It will be 600 for 100 scrips.
3. After 1 min file name will change to 0916quotes.rtd and AB will import from the new file.
4. Old files will remain as they are. We delete them while closing NestRTD.
5. We start NestRTD live feed.
6. VWAPBackfill will import data upto last minute, somewhere during the current minute. All ticks in memory for current minute lost.
7. AB will import datafile for current minute again(It is importing every second). All ticks restored.
8. Only problem will arise if RTD server gives data of timestamp earlier than one minute. We filter out such quotes while creating 1 second candles.
9. If required, VWAP Backfill can find out time stamp of latest quote in it and backfill NestRTD files also accordingly.

This should not require much code change.

1)This can work once all border cases are taken care of.

2) This will unnecessarily increase cpu usage by constant deletes and writes of same data ( Maybe still not be much ). It will also need code to keep track of minute and splitting up data as minute changes.
I dont want to do this in main branch.

3) A better way will be two synchronize the two processes.
Backfill starts, it asks RTD to pause say using TCP message or any other way, RTD maintains O/H/L/C but does not send to AB , When Backfill is done it calls RTD again and we resume.

4) Even better is to have the same process do RTD and Backfill so that we dont need IPC - but this is for later.

5) Com may also be faster with C++. I could use it to fill only missing data rather than overwrite all. So both options need to be tested in future.

Right now, i have stopped on this as my immediate needs are done and i need to focus on learning to trade. Already have one more code project to handle.
You can create your own branch of it and try to implement. PM me if you get blocked by something.

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

@ TracerBullet,

I had edited my post in the meantime. There wont be any constant deletes and writes of same data.
Also there is no need to keep track of minute since time will be taken from system and no data split required.
Thanks Anyway. I shall try to implement that myself or with someone else's help.

 

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

@ TracerBullet,

I had edited my post in the meantime. There wont be any constant deletes and writes of same data.
Also there is no need to keep track of minute since time will be taken from system and no data split required.
Thanks Anyway. I shall try to implement that myself or with someone else's help.

Before i leave thread for now, suggestion - if new logic doesnt work out, there is another way to speed up your Current code - give an (optional) input of start and end time between which we have mising data. This will speed it up a lot without taking too much effort.

 

[Happy_Singh]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Right now, i have stopped on this as my immediate needs are done and i need to focus on learning to trade.

Just wondering why don't we see similar effort for developing trading systems . . .

Maybe because putting it out there would take away whatever small edge gets built into the trading system . . .


Then again, . . . . it can always be taken off-line


Happy

 

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Just wondering why don't we see similar effort for developing trading systems . . .

Maybe because putting it out there would take away whatever small edge gets built into the trading system . . .


Then again, . . . . it can always be taken off-line


Happy

I would have been sceptical before but seeing how many times these outages happen and somehow sometimes coincide with big moves - maybe there is something going on
or maybe they are just too lazy and buggy and sometimes break.

 

[rmike]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Some thoughts on a third alternative

Re read some of the older posts and had a look at the .ini file controlset of 'NEST RTD' again.

As I understand the control 'BarPeriod=........' provides instruction to the utility for ami updation in milliseconds. Saw that josh uses the value of 60000. This would mean that even though the utility's pulling ticks per second but ami is only updated after one minute (the ticks being collected, in the meanwhile, in the quotes.rtd file).

Therefore there could be one another least invasive (!?) method of implementing backfill. Illustrating by example.

Lets say the trader initiates (automated) backfill at dd:11:55:36

The backfill command is gated in the utility till dd:11:56:00.

In the meanwhile initiating the automated backfill sends a trigger to the utility to set the BarPeriod= 60000 (arbitrary outside parameter limit and can be refined by actual testing) at dd:11:55:59.

Now at time dd:11:56:00

The automated backfill starts

Meanwhile the RT RTD continues (in a disparate threaded process as mentioned by TB) with the ticks being collected in the quotes.rtd file without ami updation.

During backfill the data till dd:11:56:00 is filled by invoking the statistics. The bars will be one minute bars timestamped as dd:hh:mm:00. All previous tick bars in ami database till dd:11:55:59 will be overwritten.

No tick loss in ami database as the RTD stopped updating ami at dd:11:55:59.

Backfill finishes and sends trigger to the utility to reset BarPeriod= 1100.

The RTD updation pushes the collected ticks into ami from that point onwards, populating the database from dd:11:56:00.

If this is workable then (hopefully) it should not be too much of a sweat to figure out minor code changes.

For fresh start during the day, Backfill can be done after 1 minute without TICKMODE while NestRTD is running. There will be no data loss.
Option can be given in Backfill to overwrite previous data

This will cause data corruption. The NOW stats are one minute bars with timestamp of dd:hh:mm:00. During normal backfill, this bar will only replace the tick datapoint which is timestamped the same. All the other tick bars in that minute will remain. That means you will have almost twice the volume record for the same bar. Price OHLC will not be affected during aggregation by ami but volume would distort 2X.