Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts MS
- Thread starter josh1
- Start date
- Status
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M
Nope!
Ordinarily yes, but not with the added format controls.
Yes!
This situation arises not only for your utility but for yusi's utility too wherein the user has the choice to timestamp M1 bars as xx::yy:59. In such a case, a google backfill with M1 timestamp as xx:yy:00 has similar faux multiple ticks addressing the same timeperiod in question.
Have been personally using the quoted method for data purge/ reconciliation
Added the 'overwrite' control as an afterthought to spark/ aid thought process for continuous rollover of contracts for a utility such as yours in adjunct to 'Tickmode' (ami database concepts
But not strictly needed per-se if deleting ticks be the only concern.
Sigh... trying hard to be charitable, so will simply say that for certain people reading may not necessarily equate with understanding
Possibly, the exercise of re-reading may prove to be beneficial.
Regards,
P.S - Pay special attention to the repeated warning.
Nope!
Ordinarily yes, but not with the added format controls.
Yes!
This situation arises not only for your utility but for yusi's utility too wherein the user has the choice to timestamp M1 bars as xx::yy:59. In such a case, a google backfill with M1 timestamp as xx:yy:00 has similar faux multiple ticks addressing the same timeperiod in question.
Have been personally using the quoted method for data purge/ reconciliation
Added the 'overwrite' control as an afterthought to spark/ aid thought process for continuous rollover of contracts for a utility such as yours in adjunct to 'Tickmode' (ami database concepts
But not strictly needed per-se if deleting ticks be the only concern.Sigh... trying hard to be charitable, so will simply say that for certain people reading may not necessarily equate with understanding
Possibly, the exercise of re-reading may prove to be beneficial.
Regards,
P.S - Pay special attention to the repeated warning.
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M
If i understand correctly, you mean to have a single 1 minute bar in AB that is updated continously untill the minute is up. josh wanted it this way but
1) i want bars in seconds to look at 15s bars.
2) Right now code is written such that allowing option for 1min bars will need more code changes than what i want to do now. ( i discussed this some pages back ).
Instead in future, i would try to get the deletes done faster and to delete only overlaping bars and only missing bars.
Warning may not be relevant anymore ( docs not updated ? )
If it does delete old data then its completely useless.
http://www.amibroker.com/devlog/wp-content/uploads/2006/03/ReadMe4781.html
I had tried it with excel once before i started on c++, but was not able to make it work. I initially thought tickmode was to enter sub second bars but it looks like its meant for duplicate bars.
Question - what happens to volume ? Does AB add volume of all of the ticks or only take the last.
Tickmode may allow two things
1. RTD - Dont maintain O/H/L/C. Instead send with customizable resolution ( second / minute ). AB will manage the Bar creation. This would be nice to have but will make sense only if AB adds volumes of all ticks.
2. More interestingly, for backfill will an import of say 10:10:59 delete all second bars in that minute? If yes, then our delete problem is solved and we dont need to remove ourselves. Try to test this.
Code is open. If someone can experiment and prove, ill try to merge / change in free time.
Yes!
This situation arises not only for your utility but for yusi's utility too wherein the user has the choice to timestamp M1 bars as xx::yy:59. In such a case, a google backfill with M1 timestamp as xx:yy:00 has similar faux multiple ticks addressing the same timeperiod in question.
Have been personally using the quoted method for data purge/ reconciliation
This situation arises not only for your utility but for yusi's utility too wherein the user has the choice to timestamp M1 bars as xx::yy:59. In such a case, a google backfill with M1 timestamp as xx:yy:00 has similar faux multiple ticks addressing the same timeperiod in question.
Have been personally using the quoted method for data purge/ reconciliation
1) i want bars in seconds to look at 15s bars.
2) Right now code is written such that allowing option for 1min bars will need more code changes than what i want to do now. ( i discussed this some pages back ).
Instead in future, i would try to get the deletes done faster and to delete only overlaping bars and only missing bars.
Added the 'overwrite' control as an afterthought to spark/ aid thought process for continuous rollover of contracts for a utility such as yours in adjunct to 'Tickmode' (ami database concepts But not strictly needed per-se if deleting ticks be the only concern.
Sigh... trying hard to be charitable, so will simply say that for certain people reading may not necessarily equate with understanding
Possibly, the exercise of re-reading may prove to be beneficial.
Regards,
P.S - Pay special attention to the repeated warning.
But not strictly needed per-se if deleting ticks be the only concern.Sigh... trying hard to be charitable, so will simply say that for certain people reading may not necessarily equate with understanding
Possibly, the exercise of re-reading may prove to be beneficial.
Regards,
P.S - Pay special attention to the repeated warning.
If it does delete old data then its completely useless.
http://www.amibroker.com/devlog/wp-content/uploads/2006/03/ReadMe4781.html
I had tried it with excel once before i started on c++, but was not able to make it work. I initially thought tickmode was to enter sub second bars but it looks like its meant for duplicate bars.
Question - what happens to volume ? Does AB add volume of all of the ticks or only take the last.
Tickmode may allow two things
1. RTD - Dont maintain O/H/L/C. Instead send with customizable resolution ( second / minute ). AB will manage the Bar creation. This would be nice to have but will make sense only if AB adds volumes of all ticks.
2. More interestingly, for backfill will an import of say 10:10:59 delete all second bars in that minute? If yes, then our delete problem is solved and we dont need to remove ourselves. Try to test this.
Code is open. If someone can experiment and prove, ill try to merge / change in free time.
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M
Already Tested with $Overwrite 1 and $Tickmode 1. Documentation is confusing. It replaces old tick of same timestamp and inserts new tick if there is none.
If i understand correctly, you mean to have a single 1 minute bar in AB that is updated continously untill the minute is up. josh wanted it this way but
1) i want bars in seconds to look at 15s bars.
2) Right now code is written such that allowing option for 1min bars will need more code changes than what i want to do now. ( i discussed this some pages back ).
I shuld be able to give a least invasive solution today(market is closed.
I initially thought tickmode was to enter sub second bars but it looks like its meant for duplicate bars.
Correct.
Question - what happens to volume ? Does AB add volume of all of the ticks or only take the last.
AB does not add volumes.
Tickmode may allow two things
1. RTD - Dont maintain O/H/L/C. Instead send with customizable resolution ( second / minute ). AB will manage the Bar creation. This would be nice to have but will make sense only if AB adds volumes of all ticks.
2. More interestingly, for backfill will an import of say 10:10:59 delete all second bars in that minute? If yes, then our delete problem is solved and we dont need to remove ourselves. Try to test this.
Already tested
Code is open. If someone can experiment and prove, ill try to merge / change in free time.
1) i want bars in seconds to look at 15s bars.
2) Right now code is written such that allowing option for 1min bars will need more code changes than what i want to do now. ( i discussed this some pages back ).
I shuld be able to give a least invasive solution today(market is closed.
I initially thought tickmode was to enter sub second bars but it looks like its meant for duplicate bars.
Correct.
Question - what happens to volume ? Does AB add volume of all of the ticks or only take the last.
AB does not add volumes.
Tickmode may allow two things
1. RTD - Dont maintain O/H/L/C. Instead send with customizable resolution ( second / minute ). AB will manage the Bar creation. This would be nice to have but will make sense only if AB adds volumes of all ticks.
2. More interestingly, for backfill will an import of say 10:10:59 delete all second bars in that minute? If yes, then our delete problem is solved and we dont need to remove ourselves. Try to test this.
Already tested
Code is open. If someone can experiment and prove, ill try to merge / change in free time.
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M
rtd client receives data in - SAFEARRAY "*data_sa" from rtd server.
poll() receives it into an array "data" and passes it on to Worker:rocessRTDData().
ProcessRTDData() puts it into Current and also if it is first quote, then Vol_Today into Previous. Current and Previous are fixed arrays. During processing, both arrays are locked so that amibrokerPoller() cannot change them.
If there are multiple quotes for single script, you are not skipping any here. That means all of them will impact Current OHLCV. (You are not sorting the data in ascending order of LTT. Is RTD server sending data chronologically? If yes, Gr8, volume_today will be ascending or same. If not :annoyed:, What will be time/cpu overhead for sorting?
After ProcessRTDData(), array "data" is deleted
Question- Is it possible to add Volume field into Previous array ?
like this -
If I do not want to reset vol_today field in Previous, can I add a function like this?
I appreciate the links given by you but learning all this and then push changes will take months. I prefer to finish it in couple of days with least invasive changes to code.
rtd client receives data in - SAFEARRAY "*data_sa" from rtd server.
poll() receives it into an array "data" and passes it on to Worker:
rocessRTDData().ProcessRTDData() puts it into Current and also if it is first quote, then Vol_Today into Previous. Current and Previous are fixed arrays. During processing, both arrays are locked so that amibrokerPoller() cannot change them.
If there are multiple quotes for single script, you are not skipping any here. That means all of them will impact Current OHLCV. (You are not sorting the data in ascending order of LTT. Is RTD server sending data chronologically? If yes, Gr8, volume_today will be ascending or same. If not :annoyed:, What will be time/cpu overhead for sorting?
After ProcessRTDData(), array "data" is deleted
Question- Is it possible to add Volume field into Previous array ?
like this -
Code:
Worker::ScripState::ScripState() :
ltp(0), vol_today(0), oi(0), bar_high(0), bar_low(std::numeric_limits<double>::infinity()), bar_open(0),vol (0)
{}
Code:
void Worker::ScripState::resetprevious(){
ltp = 0; vol = 0; oi =0; bar_high = 0; bar_low = std::numeric_limits<double>::infinity(); bar_open = 0; last_bar_ltt="";
}
Last edited:
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M
Does it remove all bars in that minute if import is hh::mm ? Did you see any difference between $TICKMODE 0 and $TICKMODE 1 ?
inline below
"I appreciate the links given by you but learning all this and then push changes will take months. I prefer to finish it in couple of days with least invasive changes to code. "
c++ is not very newbie friendly. You will have to get the hang of atleast some basics. Its not just pick and play like these scripting tools. I gave links in case you are interested to read for you free time.
Changing code is simpler, You can try doing minor changes and test/debug + google any time you get stuck.
COM is a minefield in c++. Be careful with any changes in COM stuff
Does it remove all bars in that minute if import is hh::mm ? Did you see any difference between $TICKMODE 0 and $TICKMODE 1 ?
inline below
rtd client receives data in - SAFEARRAY "*data_sa" from rtd server.
Yes this is COM type
poll() receives it into an array "data" and passes it on to Worker:rocessRTDData().
ProcessRTDData() puts it into Current and also if it is first quote, then Vol_Today into Previous. Current and Previous are fixed arrays. During processing, both arrays are locked so that amibrokerPoller() cannot change them.
Locking is needed to avoid concurrent access by 2 threads
If there are multiple quotes for single script, you are not skipping any here. That means all of them will impact Current OHLCV. (You are not sorting the data in ascending order of LTT. Is RTD server sending data chronologically? If yes, Gr8, volume_today will be ascending or same. If not :annoyed:, What will be time/cpu overhead for sorting?
Never recieved multiple quotes from RTD RefreshData(). There are duplicates sometimes(Dont know why) but they always have same data. I had put an abort in code to confirm this while testing
After ProcessRTDData(), array "data" is deleted
Its a COM return data structure. It has to be deleted else will probably get mem leak.
Question- Is it possible to add Volume field into Previous array ?
like this -
You can try anything. Just test and debug to make sure it works. Google "VC++ debugging" if you are not familiar with it ( breakpoints/run/step etc ). Debugging is much better than printing messages.
http://cplus.about.com/od/learningc/ss/vc6compile_6.htm
If I do not want to reset vol_today field in Previous, can I add a function like this?
Previous is never reset directly. We just copy the current bar into previous after picking up in AB thread. You can change this if needed.
Yes this is COM type
poll() receives it into an array "data" and passes it on to Worker:
rocessRTDData().ProcessRTDData() puts it into Current and also if it is first quote, then Vol_Today into Previous. Current and Previous are fixed arrays. During processing, both arrays are locked so that amibrokerPoller() cannot change them.
Locking is needed to avoid concurrent access by 2 threads
If there are multiple quotes for single script, you are not skipping any here. That means all of them will impact Current OHLCV. (You are not sorting the data in ascending order of LTT. Is RTD server sending data chronologically? If yes, Gr8, volume_today will be ascending or same. If not :annoyed:, What will be time/cpu overhead for sorting?
Never recieved multiple quotes from RTD RefreshData(). There are duplicates sometimes(Dont know why) but they always have same data. I had put an abort in code to confirm this while testing
After ProcessRTDData(), array "data" is deleted
Its a COM return data structure. It has to be deleted else will probably get mem leak.
Question- Is it possible to add Volume field into Previous array ?
like this -
Code:
Worker::ScripState::ScripState() :
ltp(0), vol_today(0), oi(0), bar_high(0), bar_low(std::numeric_limits<double>::infinity()), bar_open(0),vol (0)
{}http://cplus.about.com/od/learningc/ss/vc6compile_6.htm
If I do not want to reset vol_today field in Previous, can I add a function like this?
Code:
void Worker::ScripState::resetprevious(){
ltp = 0; vol = 0; oi =0; bar_high = 0; bar_low = std::numeric_limits<double>::infinity(); bar_open = 0; last_bar_ltt="";
}c++ is not very newbie friendly. You will have to get the hang of atleast some basics. Its not just pick and play like these scripting tools. I gave links in case you are interested to read for you free time.
Changing code is simpler, You can try doing minor changes and test/debug + google any time you get stuck.
COM is a minefield in c++. Be careful with any changes in COM stuff
Last edited:
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M
Hi TB n josh,
Lots of queries But its good to have 'thinking heads', with different perspectives and individual domain expertise, to put together for general benefit.
While you guys are looking at things from a C++ / AutoIt coding perspective, with respective levels of individual domain expertise, I can only comment from amibroker's perspective.
Let me first get the basic premise of suggested format control out of the way, before I try to address other queries.
As of ami v4.7, importing data via tickmode will do the following
1. Delete the existing data in Ami from the starting point of the date timestamp of the first data point of the imported data.
i.e to say that if the time now is dd/10:45:30 in a live market and you decided to backfill from dd/09:15:00 via tickmode - Ami will delete Any existing data from (and after) dd/09:15:00 (if present in the database) till the current time.
2. And import the backfill data as a replacement.
Since you guys were getting stuck with slow execution as well as CPU hogging during tick data delete via COM calls this way was suggested, as its blazing fast and leverages the existing ami capability.
Caveat: - In live markets, during backfill in the above example, you will lose a couple of ticks from dd/10:45:30 onwards till the time it takes for the backfill of that individual ticker to complete. While it has negligible impact on price, some amount of actual transacted volume will be lost for that time period. Again, the effect is practically negligible unless a huge amount of volume decides to get transacted in those 1 - 3 seconds.
Basically you will need to couple your existing backfill code (without any coding for existing data delete) with the built in ASCII importer in ami to push data via 'tickmode'.
P.S - For starters, one can test this by importing one day, one min resolution data from a txt/csv file over the existing tick data in ami of that day via the ASCII importer, in static market conditions. Try this only for the existing last day otherwise all data ahead of that day will be deleted if any day other than the last day is chosen.
Hi TB n josh,
Lots of queries
But its good to have 'thinking heads', with different perspectives and individual domain expertise, to put together for general benefit.While you guys are looking at things from a C++ / AutoIt coding perspective, with respective levels of individual domain expertise, I can only comment from amibroker's perspective.
Let me first get the basic premise of suggested format control out of the way, before I try to address other queries.
As of ami v4.7, importing data via tickmode will do the following
1. Delete the existing data in Ami from the starting point of the date timestamp of the first data point of the imported data.
i.e to say that if the time now is dd/10:45:30 in a live market and you decided to backfill from dd/09:15:00 via tickmode - Ami will delete Any existing data from (and after) dd/09:15:00 (if present in the database) till the current time.
2. And import the backfill data as a replacement.
Since you guys were getting stuck with slow execution as well as CPU hogging during tick data delete via COM calls this way was suggested, as its blazing fast and leverages the existing ami capability.
Caveat: - In live markets, during backfill in the above example, you will lose a couple of ticks from dd/10:45:30 onwards till the time it takes for the backfill of that individual ticker to complete. While it has negligible impact on price, some amount of actual transacted volume will be lost for that time period. Again, the effect is practically negligible unless a huge amount of volume decides to get transacted in those 1 - 3 seconds.
Basically you will need to couple your existing backfill code (without any coding for existing data delete) with the built in ASCII importer in ami to push data via 'tickmode'.
P.S - For starters, one can test this by importing one day, one min resolution data from a txt/csv file over the existing tick data in ami of that day via the ASCII importer, in static market conditions. Try this only for the existing last day otherwise all data ahead of that day will be deleted if any day other than the last day is chosen.
Last edited:
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M
If you've tried the above mentioned example, you'll find that it works and is blazing fast. I know because I've been there and done that!
OK! Now as regards queries
Not really! However it is desirable for keeping the database size manageable
The same reason for which you need to carry out regular purge of tick data (and you watch only a couple of tickers)!
There's no need for code modification. At EOD when trading for the day is done, one can overwrite tick data for that day by one min bars. This is possible even by manual import.
The volume (or for that matter, price point) of all ticks resides as raw data/tickpoint in the database. AB aggregates volume and constructs OHLC automatically as per the chosen timeframe granularity e.g 30 secs, 1 Min, 15 Min etc.
How this is aggregated has some quirks. Suggest exploration of preferences in intraday tab. The preferences like start time of interval etc have 'timestamping and aggregation' connotations.
If you've tried the above mentioned example, you'll find that it works and is blazing fast. I know because I've been there and done that!
OK! Now as regards queries
Not really! However it is desirable for keeping the database size manageable
The same reason for which you need to carry out regular purge of tick data (and you watch only a couple of tickers)!
There's no need for code modification. At EOD when trading for the day is done, one can overwrite tick data for that day by one min bars. This is possible even by manual import.
The volume (or for that matter, price point) of all ticks resides as raw data/tickpoint in the database. AB aggregates volume and constructs OHLC automatically as per the chosen timeframe granularity e.g 30 secs, 1 Min, 15 Min etc.
How this is aggregated has some quirks. Suggest exploration of preferences in intraday tab. The preferences like start time of interval etc have 'timestamping and aggregation' connotations.
- Status
Similar threads
-
-
-
-
Any one know how to get free realtime datafeed for metastock(yahoo)
- Started by sanjaychowdary
- Replies: 3
-