Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts MS

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Do you mean its working or that you are fixing it for now.

I am stuck at Time format. Ami does not recognise 1:00 PM (12 hour format). Once I fix that, I will send you the app.

 

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

I am stuck at Time format. Ami does not recognise 1:00 PM (12 hour format). Once I fix that, I will send you the app.

Look at my code. i have a function that converts it - Reader::changeHHFrom12To24.

I guess you are talking of AHK app.
Can you attach the sample data so that i can fix the c++ application too.

 

[sanbadgujar]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Do not bother about Backfill. I am halfway thru it. Will do it soon.

After changing Bar period to 60000, I found that NestRTD sends candle data to Ami only once a minute. I wanted it create 1 min bar internally but send to Ami every second and refresh. The time stamp should be in HH:mm format. For example -
Say first tick collected is at 9:15:00. O, H, L, C will all be equal LTP since it is first tick of the minute. Whatever ticks collected during one minute will modify H or L and C will always be equal to LTP.
The same bar will be sent to Ami each second with time Stamp 9:15. When Ami sees, same time stamp each second, it will keep overwriting earlier candle and refresh.
At 9:16:00 O,H,L,C will again be equal to LTP. Ami will create next candle since time stamp will be 9:16.
If you get time, please do this.
Edit- The advantage is that, there will be 375 quotes in Ami per day. Deleting these before backfill thru COM is faster. If there are 375*60 quotes, in database, it takes much longer time to delete.

Dear josh1 Can you have utilities Get real time NIFTY F&O data in MT4 for real time chart

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Look at my code. i have a function that converts it - Reader::changeHHFrom12To24.

I guess you are talking of AHK app.
Can you attach the sample data so that i can fix the c++ application too.

Here it is.
http://ubuntuone.com/7eeDd3mGB4n2soZvavtBWF

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

1. I didnt understand. Why do you need them in an array. There can 2+ per second. Once i release source you can branch and modify it if needed.

2. Option 2 is for thread A to not maintain O/H/L/C. Instead it collects all ticks in an array for each scrip and thread B constructs O/H/L/C and differentates between 10:00 and 10:01
This is cleaner. It will probably need more cpu time but thats ok.

Now you know why I need them in an array....!

Which thread will send them to Ami ? Thread B or Will there be thread C. May be silly question, but I have not studied threads.
Edit-

This will be the last change i make in this application for next few months

There will be no need to do any more changes except bug fixes. If you are able to do it, I will try to construct GUI around it rather than doing it in Autoit.

In your case, you can use 10:00:00 , 10:00:01, 10:00:02 for 1 second candles. I don't use candles less than 1 min. They create confusion in mind. Even 1 min. is too fast for me.

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Do you mean its working or that you are fixing it for now.

Finished. Testing now.

You can try to do it yourself too. I want people to change and fix code rather than just me. Changes can be merged with git code.

We will see if Mastermind007 of Kelvinhand can help. I am at the moment too junior in C++ to fix code.

3. Are you sure deleting is slow? if i delete from quote editor, its pretty fast.
Also please make sure, you auto backfill tool will delete second bars too even if slow.

Quote Editor is very fast but there is no API for bulk deletion of quotes. Check this Java Script --

// //                                                                   //      
// // File:                DeleteDateRange.js                           //
// // Author:              mastermind007                                //
// // http://www.traderji.com/amibroker/88211-deleting-rt-ticks-2.html#post825413
// // Purpose:             Remove all quotations between                //
// //                      two given Date and Time Stamps               //
// // Language:            JScript (Windows Scripting Host)             //
// // ENJOY :-)                                                         //
// //                                                                   //
   // ////////////////////////////////////////////////////////////////////
   //A D J U S T   N E X T   3   L I N E S   B E F O R E   R U N N I N G
   // ////////////////////////////////////////////////////////////////////
    DataDir = "@@@@@YOURDATA@@@PATH"
    // CAUTION Jan is 00. Feb is 01. Dec is 11.
    var DayDeleteFrom = new Date(2013, 00, 07, 09, 00);
    var DayDeleteUpto = new Date(2013, 00, 07, 23, 30);
   // ///////////////////////////////////////////////////////////////////
   // ///////////////////////////////////////////////////////////////////
   // ///////////////////////////////////////////////////////////////////
    
    var oAB = new ActiveXObject("Broker.Application");
    var fso = new ActiveXObject("Scripting.FileSystemObject");
    var fileX = fso.OpenTextFile( "C:\\DeleteDateRange.Amibroker.log", 2, true );

    oAB.LoadDatabase( DataDir );

    var oStocks = oAB.Stocks;

    var Qty = oStocks.Count;

    var j = 0;
    var k = 0;

    fileX.WriteLine( "Will delete all quotations from all stocks between starting date:" + DayDeleteFrom);
    fileX.WriteLine( "and ending date:" + DayDeleteUpto);
    fileX.WriteLine( "" );
    WScript.Echo("Cleanup Will Start Now !!!. Abort Process if this is not what you want" );

    for( i = 0; i < Qty; i++ )
    {
        k = 0;
        oStock = oStocks( i );
        fileX.Write( i + ". " + oStock.Ticker + "=" );

        while (j < oStock.Quotations.Count)
        {
            tmpDateNum = oStock.Quotations( j ).Date ;
            //fileX.WriteLine( "Date returned by AB is " + tmpDateNum);
            if ((tmpDateNum >= DayDeleteFrom) && (tmpDateNum <= DayDeleteUpto))
            {
                k++;
                oStock.Quotations.Remove(j);
            } else if (tmpDateNum > DayDeleteUpto)
            {
                  break;
            }  else
                j++;
        }
        fileX.WriteLine( "Deleted " + k  + " quotes from " + oStock.Ticker);
    }
    oAB.SaveDatabase( );

    fileX.Close();
    WScript.Echo("Deletion completed :-)" );

 

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Here it is.
http://ubuntuone.com/7eeDd3mGB4n2soZvavtBWF

Thanks. So VWAP data is comma seperated whereas in NEST it is tabs.
In this case we should be able to import it into AB as it is. Maybe only the last comma needs to be remove.

Is the data table in charts also similar?

Now you know why I need them in an array....!

Which thread will send them to Ami ? Thread B or Will there be thread C. May be silly question, but I have not studied threads.
Edit-

There will be no need to do any more changes except bug fixes. If you are able to do it, I will try to construct GUI around it rather than doing it in Autoit.

Thread B ( spawned from main ) reads O/H/L/C and sends to AB.
I have some work piled up now so i am not doing it. Just finalizing fixes to release final build.
But instead, we can make the js script much faster.

Also you can try to export data from AB as 1 min candles and then import it back again to convert DB. I dunno if we can do this from an application
https://groups.yahoo.com/neo/groups/amibroker/conversations/topics/116071

Quote Editor is very fast but there is no API for bulk deletion of quotes. Check this Java Script --

Thanks for this. i can use it as reference to understand AB api for future.
(Example delete overlapping bars in AB for backfill )
On first view, i think can be easily optimized. There are two problems

1. Small issue - I dont know Windows SH. But anyway, i assume that all com calls are repeated as they run. Problem is that there are some unnecessary duplicate com calls
ex oStock.Quotations.Count is called in loop. You can instead do it once per scrip and store its value in a var. Another example - oStock.Ticker called twice for log

2. Big issue - It seems that the quotes recieved from AB is sorted ( since we break if a quote is after DayDeleteUpto in code )
If so, we can get HUGE performance improvement (if quotes to be deleted are much less than no of quotes in DB) by a very simple change of using binary search.
Binary search is very simple - Instead of iterating one by one you look at middle and go left or right recursively.
Look it up in google - its very simple.

Try to fix it. I will do it if needed later.
Right now this script is unusable for me with 1-2 year 1min data for 2 scrips + 1 sec data for less than a month.

Edit - Binary search wont solve performance if we delete all db data. Then it will be same, probably worse. To delete all data, its probably best to try to use some script within AB. In our case we only need to delete todays quote for which binary search is perfect.

Alternately, since we remove data from end, iterate backwards - this should be faster than binary search for our use case as will start from last bar today and Exit as soon as we get last candle of yesterday

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Thanks. So VWAP data is comma seperated whereas in NEST it is tabs.

Sorry... My mistake. It is tabs in both. However, when I copied it into Notepad, Linefeeds did not work.

I do iterate backwards when deleting quotes.

 

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Sorry... My mistake. It is tabs in both. However, when I copied it into Notepad, Linefeeds did not work.

Please attach a sample file for VWAP/Datatable that does not work.
Code is not very flexible right now, so a little difference may break it.
i want to fix it now and release. If its not needed for now, we can do it later.

I do iterate backwards when deleting quotes.

Then cant do any faster. The attached code was not backwards so i asked.
But then it shouldnt be that slow, only todays data that too incomplete.
How long does it take?

Eventually, ill try to port this code to c++ and backfill only missing data to keep 1 second bars. We can even try to call AHK using c++ com interface i think.

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Please attach a sample file for VWAP/Datatable that does not work.
Code is not very flexible right now, so a little difference may break it.
i want to fix it now and release. If its not needed for now, we can do it later.

http://ubuntuone.com/4ii3l7LOZXfleBLjLJrCe6

[/URL][/IMG]

But then it shouldnt be that slow, only todays data that too incomplete.
How long does it take?

20-30 seconds for 375 candles

Eventually, ill try to port this code to c++ and backfill only missing data to keep 1 second bars. We can even try to call AHK using c++ com interface i think.

I am using Autoit which is mother of AHK. Yes it has a DLL to expose its COM interfaces.