Real Time Data Now / Nest Trader to Amibroker, Fcharts

[boarders]

Hi

Interesting developments

You guys rock :thumb:

Writing AFL for manipulating / displaying / aggregating the data present in Amibroker fields in any way is quite simple.

But just wondering if we can maintain the data in 2 separate tickers . . .
for eg. if we have Nifty_F1 with one minute data can we also have a ticker Nifty_T1 in the format Josh has mentioned above.

if we can do that then no need to convert the tick data into OHLCV format using AFL for using existing strategy / charts code / plotting etc.

We can use everything old as it is and now use info from this new ticker for Order Flow Analysis / Foot Print charts etc.

Thanks
Happy

Edit: The data base will be in tick format, but all the existing tickers can be maintain/updated the same 1 minute mode, so back fill for them remains same.

if we make appropriate modifications to rtdman but keeping LTP as close price instead of open price as josh suggested and as raw data (without OHL agregation internal in RTDman), and then modify the other fields input in rtdman, we can achieve what you suggest, otherwise we can also manipulate the config file as suggested in my last post.

 

[josh1]

Hi

Interesting developments

You guys rock :thumb:

Writing AFL for manipulating / displaying / aggregating the data present in Amibroker fields in any way is quite simple.

But just wondering if we can maintain the data in 2 separate tickers . . .
for eg. if we have Nifty_F1 with one minute data can we also have a ticker Nifty_T1 in the format Josh has mentioned above.

if we can do that then no need to convert the tick data into OHLCV format using AFL for using existing strategy / charts code / plotting etc.

We can use everything old as it is and now use info from this new ticker for Order Flow Analysis / Foot Print charts etc.

Thanks
Happy

Edit: The data base will be in tick format, but all the existing tickers can be maintain/updated the same 1 minute mode, so back fill for them remains same.

if we make appropriate modifications to rtdman but keeping LTP as close price instead of open price as josh suggested and as raw data (without OHL agregation internal in RTDman), and then modify the other fields input in rtdman, we can achieve what you suggest, otherwise we can also manipulate the config file as suggested in my last post.

@Boarders,
I suggested Open and Close both as LTP. However, we can have Close as LTP and use Open for something else.

We can have two separate instances of RTDMan, both pulling data from NOW/Nest and both pushing it to the same instance of Amibroker. One pushing it as ticks (Nifty_T1) and another aggregating ticks into 1 sec or 1 min candles (Nifty_F1) as desired and pushing it to Amibroker. We have to see how Ami reacts.
On rethink, We can code RTDMan to do both.

But all this depends upon whether I am able to get tick data from NOW. Otherwise, we may try to reduce time interval up to 150-250 msecs. I will see that in next week.

 

[TracerBullet]

Hi

Interesting developments

You guys rock :thumb:

Writing AFL for manipulating / displaying / aggregating the data present in Amibroker fields in any way is quite simple.

But just wondering if we can maintain the data in 2 separate tickers . . .
for eg. if we have Nifty_F1 with one minute data can we also have a ticker Nifty_T1 in the format Josh has mentioned above.

if we can do that then no need to convert the tick data into OHLCV format using AFL for using existing strategy / charts code / plotting etc.

We can use everything old as it is and now use info from this new ticker for Order Flow Analysis / Foot Print charts etc.

Thanks
Happy

Edit: The data base will be in tick format, but all the existing tickers can be maintain/updated the same 1 minute mode, so back fill for them remains same.

This is a cleaner way of doing it, put Bid/Ask related stuff in separate scrip without loss of info in original scrip.

To support this in RTDMan mods - One possibility will be to have Alias2 in settings. So when alias2 is set and we have B/A topics - push B/A related stuff to alias2 while keeping alias1 untouched.

 

[josh1]

This is a cleaner way of doing it, put Bid/Ask related stuff in separate scrip without loss of info in original scrip.

To support this in RTDMan mods - One possibility will be to have Alias2 in settings. So when alias2 is set and we have B/A topics - push B/A related stuff to alias2 while keeping alias1 untouched.

Why make it too complicated? Just add an "e" to alias. So 'Nifty_F1' will have original stuff and 'Nifty_F1e' will have B|A related stuff in whichever columns we want. Push data to csv in two simultaneous lines.(Minimum change of code)

Edit- See, if you haven't already this

 

[sr114]

But all this depends upon whether I am able to get tick data from NOW. Otherwise, we may try to reduce time interval up to 150-250 msecs.

Josh
better reduce the time interval to 150-200 ms to capture most of the trade, or reduce it more like 100 ms or less that is feasible

better db structure

O = bid price
Hi = bid qty
L = ask price
C= ask qty

vol = ltq or vol traded today

aux1= ltp [aux1 aggregation as last in intraday setup]

in this case we will be able to use the bid/ask parameter and the aux1 aggregation will make our database as only ltp is used to generate the ohlc structure in the tick or second based database.

[see the quote editor to get the idea]

now the question arises how we are going to do the h/L based calculation in the afl . simple - use the aux1 in the hi or lo or hhv or llv to get the result.

rgds

 

[extremist]

Josh
better reduce the time interval to 150-200 ms to capture most of the trade, or reduce it more like 100 ms or less that is feasible

better db structure

O = bid price
Hi = bid qty
L = ask price
C= ask qty

vol = ltq or vol traded today

aux1= ltp [aux1 aggregation as last in intraday setup]

in this case we will be able to use the bid/ask parameter and the aux1 aggregation will make our database as only ltp is used to generate the ohlc structure in the tick or second based database.

[see the quote editor to get the idea]

now the question arises how we are going to do the h/L based calculation in the afl . simple - use the aux1 in the hi or lo or hhv or llv to get the result.

rgds

@ Josh1

tick is just a value.
ideally it does not have OHLC to it.

so like i said if we plot plotohlc(c,c,c,c------); all afls will work properly which r based on H and L values. and no problem will be faced in migrating from tick to other time frames.

I'm big zero in programming except AFL.

so i have one question and suggestion to get better chances of pulling the ticks from now.

Question : How is the logic of pulling values is done?

i hope it based on change in LTP and capturing tht change after some certain period.

Suggestion : in real ticks u may find lots of ticks identical. so ltp remains the same over the period but u may miss the volumes by the method u r applying for pulling.
wht i suggest is keep the current logic and just try with

(change in LTp ) OR (change in total traded volume)

this will decrease the probability of missing ticks.

 

[boarders]

@ Josh1

tick is just a value.
ideally it does not have OHLC to it.

so like i said if we plot plotohlc(c,c,c,c------); all afls will work properly which r based on H and L values. and no problem will be faced in migrating from tick to other time frames.

I'm big zero in programming except AFL.

so i have one question and suggestion to get better chances of pulling the ticks from now.

Question : How is the logic of pulling values is done?

i hope it based on change in LTP and capturing tht change after some certain period.

Suggestion : in real ticks u may find lots of ticks identical. so ltp remains the same over the period but u may miss the volumes by the method u r applying for pulling.
wht i suggest is keep the current logic and just try with

(change in LTp ) OR (change in total traded volume)

this will decrease the probability of missing ticks.

change in LTP is always recorded as LTT thus if we take LTQ as base for volume, then even if LTP remain same with different LTQ the trades will always reflect in LTT. so taking each tick of LTT and recording the LTP/LTQ /bid price/ask price of that moment will give the exact and correct picture.

 

[josh1]

@ Josh1

tick is just a value.
ideally it does not have OHLC to it.

so like i said if we plot plotohlc(c,c,c,c------); all afls will work properly which r based on H and L values. and no problem will be faced in migrating from tick to other time frames.

I'm big zero in programming except AFL.

so i have one question and suggestion to get better chances of pulling the ticks from now.

Question : How is the logic of pulling values is done?

i hope it based on change in LTP and capturing tht change after some certain period.

Suggestion : in real ticks u may find lots of ticks identical. so ltp remains the same over the period but u may miss the volumes by the method u r applying for pulling.
wht i suggest is keep the current logic and just try with

(change in LTp ) OR (change in total traded volume)

this will decrease the probability of missing ticks.

change in LTP is always recorded as LTT thus if we take LTQ as base for volume, then even if LTP remain same with different LTQ the trades will always reflect in LTT. so taking each tick of LTT and recording the LTP/LTQ /bid price/ask price of that moment will give the exact and correct picture.

Currently, I am maintaining two arrays (Current and Previous ) of scrips read from settings.ini for storing and aggregating ticks . Both have a Push Flag in addition to data. Each new tick is first stored in Newdata structure. Flow is as below.

/********************************************************************************************************************
Data from RTD server comes in pairs of Topic_id and field_id. We have topic _id as row numbers of scrips
from settings.ini. Whereas field_ids are columns. We have enumerated these field_id in worker.h as
LTT=0, VOLUME_TODAY=1, OI=2,LTP=3. Fortunately RTD Server sends data of each topic_id together and also in
order of their ENUM. Hence data pairs come sequentially in the order LTT=0, VOLUME_TODAY=1, OI=2,LTP=3
Index does not have any field other than LTP. Equity will not have OI. However every scrip will have LTP.
Therefore LTP is kept last at no.3
Once LTP is received, bar for that scrip is completed and it can be written to current bar.
*********************************************************************************************************************/

1. A Quote is complete when LTP is received. (I don't remember why I did not keep the condition as change of topic_ID. I shall see that next week.)

(LTT is received in the format HH:MM:ss from RTD server. It is truncated to HH:MM if bar period is 60000)

2. Each newdata.LTT is not same as Current LTT and
a. Current.Push =1 then entire Current quote is copied to Previous quote , newdata quote is copied to Current quote and Current Push is set to 1.
b. If Current.Push =0 then Current.ltt and Current.vol_today is copied to Previous
LTP, Vol_today and OI are copied to Current. Volume = Current.vol_today - Previous.vol_today.

3. If newdata.LTT = Current.LTT then LTP, Vol_today and OI are copied to Current. Volume = Current.vol_today - Previous.vol_today. H and L are changed if LTP is higher or lower. and Current Push is set to 1


There is a second thread that does the work of pushing data to csv file and call Amibroker to import it. This is done every second.
1. It will send Previous quote to csv if Previous.Push =1 and set it to 0(zero).
2. Thereafter, it will send Current quote to csv if Current.Push =1 and set it to 0(zero).

I am getting 99.9% accuracy with this logic. When there is high activity on exchange, RTD server hangs for few seconds but thereafter disburses data in bursts. I am getting H and L and Volume correct though a bit late in that case. If I backfill all scrips at EOD, I do not see any difference in charts. Therefore I stopped doing backfill at EOD. (Loveenajyoti also confirmed that)

Last week I changed the code slightly to add LTQ instead of using Volume Traded Today. This I shall test next week and if successful, I shall reduce bar period.

Old code is available from my signature. I did not upload it on git since TB had removed his code from there and I had no energy left after learning C++

 

[TracerBullet]

A Quote is complete when LTP is received. (I don't remember why I did not keep the condition as change of topic_ID. I shall see that next week.)

A possible better way to do stuff after getting ticks is in void Worker :: poll(). This avoids depending on RTD server to send ticks in some order. This function calls processRTDData() in a loop for each field after which all scrips states are correct.

Anyway, this is fyi only - not need to change as long as its working.

Old code is available from my signature. I did not upload it on git since TB had removed his code from there and I had no energy left after learning C++

All code has been reuploaded to github - rtd/backfill/ordering


Also, since you wanted to check it, to do tick updates in RTDMan for AB a very brief and rough outline from my todo is below.

// RTD Callback - Use array and construct bars in AB thread
// tick completes after Worker:oll() processes all fields
// More accurate mins - some data gets shifted now
// Immediate update - use events instead of sleep. Manage minute change
// AB - Send ticks using TICKMODE 1 - No need for OHLC - can send multiple ticks with same Timestamp // Test first
// Will need to maintain all ticks of same minute and send them together. And erase array when LTT changes
// Map of scrip + struct( LTT, Tick array)
// tick export - no duplicates - Maybe have flag for each tick in array

 

[Happy_Singh]

Hi again

For all those of us who are using OFA/FootPrint/MP/Volume based charts for trading decisions,
will the aggregate one minute volume traded at Bid / Ask be useful or you need to capture maximum possible ticks at Ask/Bid.

If the aggregate one minute volume traded at Bid / Ask is sufficient for the volume based analysis then I think
the solution would be simply use Aux1 & Aux2 for Volume Traded at Bid Price / and / Volume Traded at Ask Price.
In this case we have Vol = Aux1 [Bid_Vol (Bullish)] + Aux2 [Ask_Vol (Bearish)]

Alternatively the volume with a higher tick goes to Aux1 and the volume with a lower tick goes to Aux2,
In this case we have Vol = Aux1 [Higher Tick (Bullish)] + Aux2 [Lower Tick (Bearish)] + [Vol at same Price (Noise)]
here we don't store Flat Vol (noise) it will be derived = V - Aux1 - Aux2



Thanks
Happy