Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts MS

Status
Not open for further replies.
X

xsis

Active Member
#1,191
#1,191
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

thks josh! finally got the difference!

btw - i just saw that ur signature has changed. so got curious and read few last pages of this wonderful thread. is there a change in the process/method of getting RT data from various terminals to amibroker. i mean is this a different method? since ur working on sumthing new, it has to be better. but cn u explain how this new method is better than the old one?


josh1 said:
Nest plus is irrelevant. It is additional utilities for Which separate login is required. It is useful for multiple days backfill.
NSENOW is given by NSE. ZT aka Nest trader is its sister given by Zerodha and others.
Click to expand...
 
TracerBullet

TracerBullet

Well-Known Member
#1,192
#1,192
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

josh1 said:
Edit - Can you please change your signature link from the test build to https://github.com/SpiffSpaceman/AmibrokerFeeder/releases/latest
This one will always have the latest build so you wont have to change it. I also forgot to put license and source link in the test build.
Done
Click to expand...
The link redirects to 0.1.2. So what you copied in sig goes to 0.1.2 always. Copy text inside quotes - "https://github.com/SpiffSpaceman/AmibrokerFeeder/releases/latest" exactly

josh1 said:
Configured NestRtd for NSENOW. It is updating everything including options. No issues. May be I did not configure options properly last time.
How can I keep VC++2010 working after 30 days? I may not be able to understand your code in that time
Click to expand...
Great. I think you only need to register once (free). You may need a ms email id, dont remember.
http://msdn.microsoft.com/en-us/library/ms246592(v=vs.90).aspx
 
josh1

josh1

Well-Known Member
#1,193
#1,193
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

TracerBullet said:
Lets hope quotations is the bottleneck. Otherwise we need a way to profile this. ( profile tool shows how much time each function takes cumulatively)

I could not find any tool for autoit using google. Instead we can use timers
http://www.autoitscript.com/wiki/Tutorial_Optimizing

If the bottleneck is in remove of AB then we are out of luck ( Maybe alternative is try to run AFL using a COM call if available )
Click to expand...
Working Script for removing quotes. You will have to install Autoit to run

Code: 
#cs ........Intro
;// //                                                                   //
;// // File:                DelRT.au3                          			//
;// // Author:              Josh1		     	                        //
;// // Purpose:             Remove all quotations between                //
;// //                      two given Date 				               //
;// // Language:            Autoitscript					             //
;// // ENJOY :-)                                                         //
;// //                                                                   //
;   // ////////////////////////////////////////////////////////////////////
#ce

Global $DataDir, $DayDeleteFrom, $DayDeleteUpto

#include <ButtonConstants.au3>
#include <DateTimeConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
#include <GuiDateTimePicker.au3>
#include "DTC.au3"
#include <Date.au3>
#include <Array.au3>
#include <GuiEdit.au3>


#Region ### START Koda GUI section ### Form=
Opt("GUIOnEventMode", 1) ; Change to OnEvent mode
;$Scrip = "NIFTY"

$Form1 = GUICreate("Select Days to delete", 460, 322, 202, 120)
GUISetFont(10, 400, 0, "MS Sans Serif")
GUISetOnEvent($GUI_EVENT_CLOSE, "CLOSEButton")
$Label1 = GUICtrlCreateLabel("From Date", 40, 20, 67, 20)
$Date1 = GUICtrlCreateDate(_NowCalcDate(), 128, 12, 233, 24)
$Label2 = GUICtrlCreateLabel("To Date", 40, 52, 53, 20)
$Date2 = GUICtrlCreateDate(_NowCalcDate(), 128, 52, 233, 24)
$Label3 = GUICtrlCreateLabel("Database Path", 24, 92, 94, 24)
$Input1 = GUICtrlCreateInput(IniRead(@WorkingDir & "\DelRT.ini", "AB", "Datadir", "C:\AmiBroker\"), 128, 92, 297, 24)
$Label4 = GUICtrlCreateLabel("Delete data for", 24, 122, 94, 24)
$Input2 = GUICtrlCreateInput(IniRead(@WorkingDir & "\DelRT.ini", "AB", "Scrip", "NIFTY"), 128, 122, 297, 24)
$Button1 = GUICtrlCreateButton("OK", 168, 162, 105, 25)

$Edit1 = GUICtrlCreateEdit("",128,192,300,120)

GUICtrlSetOnEvent($Date1, "Date1Change")
GUICtrlSetOnEvent($Date2, "Date2Change")
GUICtrlSetOnEvent($Input1, "Input1Change")
GUICtrlSetOnEvent($Input2, "Input2Change")
GUICtrlSetOnEvent($Button1, "DeleteTics")

$DTM_SETFORMAT_ = 0x1032 ; $DTM_SETFORMATW
	$style = "yyyy/MM/dd"
	GUICtrlSendMsg($Date1, $DTM_SETFORMAT_, 0, $style)
	GUICtrlSendMsg($Date2, $DTM_SETFORMAT_, 0, $style)

GUISetState(@SW_SHOW,$Form1)
#EndRegion ### END Koda GUI section ###


While 1

	Sleep (100)
WEnd

Func Date1Change()
	If GUICtrlRead($Date1) <= GUICtrlRead($Date2) Then
;		MsgBox(64,"",$DayDeleteFrom & " - " & GUICtrlRead($Date1))
	Else
		MsgBox(64,"","From Date cannot be greater than To Date")
		GUICtrlSetData($Date1,_NowCalc())
	EndIf
EndFunc

Func Date2Change()
	If GUICtrlRead($Date2) < GUICtrlRead($Date1) Then
		MsgBox(64,"","To date cannot be before From Date")
		GUICtrlSetData($Date2,_NowCalc())
	ElseIf 	GUICtrlRead($Date2) > _NowCalc() Then
		MsgBox(64,"","To date cannot be future Date")
		GUICtrlSetData($Date2,_NowCalc())
	EndIf
EndFunc

Func Input1Change()
	$DataDir = GUICtrlRead($Input1)
EndFunc

Func Input2Change()
	$Scrip = GUICtrlRead($Input2)
EndFunc

Func CLOSEButton()
    Exit
EndFunc   ;==>CLOSEButton

Func DeleteTics()
	$DayDeleteFrom = (_Date_Time_Convert(GUICtrlRead($Date1), "yyyy/MM/dd", "yyyyMMdd")) & "000001"
	$DayDeleteUpto = (_Date_Time_Convert(GUICtrlRead($Date2), "yyyy/MM/dd", "yyyyMMdd")) & "235959"
	$DataDir = GUICtrlRead($Input1)
	$Scrip = GUICtrlRead($Input2)
	IniWrite(@WorkingDir & "\DelRT.ini", "AB", "Datadir", $DataDir)
	IniWrite(@WorkingDir & "\DelRT.ini", "AB", "Scrip", $Scrip)

	$oAB = ObjCreate("Broker.Application")
	$oAB.Visible = 1
;	$oAB.LoadDatabase($DataDir)
	$oStocks = $oAB.Stocks
	$Qty = $oStocks.Count
	$i = 0
    $j = 0
    $k = 0

	For $i = 0 To ($Qty-1)
		$Stock = $oStocks($i)
		$k = 0
		$ScripName = $Stock.Ticker
		$QQ = $Stock.Quotations.Count
		$Quotes = $Stock.Quotations

		if $ScripName = $Scrip Then
			_GUICtrlEdit_AppendText($Edit1,"Total Quotes - "&$ScripName&" -"& $QQ & @CRLF)
			$ts = TimerInit()
			For $j = ($QQ-1) to 0 Step -1
					$Quote = $Quotes($j)
					$tmpDateNum = $Quote.Date

				if (($tmpDateNum >= $DayDeleteFrom) and ($tmpDateNum <= $DayDeleteUpto)) Then
					$k+=1
					$Quotes.Remove($j)
				ElseIf ($tmpDateNum < $DayDeleteFrom) Then
					ExitLoop
				EndIf
			Next
			$oAB.RefreshAll
			$td = TimerDiff($ts)
			_GUICtrlEdit_AppendText($Edit1,"Total" & $k &"- Quotations removed in -" & int($td) & " miliSecs")
		EndIf
   Next
EndFunc
Edit -Actually Database path is redundant. I am not loading it. Taking default/running database.
 
Last edited:
josh1

josh1

Well-Known Member
#1,197
#1,197
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

xsis said:
thks josh! finally got the difference!

btw - i just saw that ur signature has changed. so got curious and read few last pages of this wonderful thread. is there a change in the process/method of getting RT data from various terminals to amibroker. i mean is this a different method?
since ur working on sumthing new, it has to be better. but cn u explain how this new method is better than the old one?
Click to expand...
Yes
Advantages are given by TracerBullet on his site.
 
TracerBullet

TracerBullet

Well-Known Member
#1,198
#1,198
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

josh1 said:
Working Script for removing quotes. You will have to install Autoit to run

Edit -Actually Database path is redundant. I am not loading it. Taking default/running database.
Click to expand...
There are still extra com calls :) but its irrelevant. Quotations is not bottleneck.

I have about 11000 quotes for today.

1. I commented $Quotes.Remove($j) . Time went from 77190 to 51625. i dont think there are extra removes. So even if we magically cut down everything else, it will still take 25 seconds which is too much.

- You will need to have less number of bars.
- Another way to improve here will be to backfill only missing data. Then delete will only remove overlapping bars. But we will still need to query all bars which seems to take 50 seconds for 1 day data per scrip.

2. The most common use case for me is to backfill at start of day. In this case, there is no remove - so backfill should be fast enough in any case.
I need to check if i can do it faster manually for 2 scrips - i only take 5-10 seconds, so should be close :)

3. When it runs, AB is at full cpu usage. So assuming we are calling correctly, the bottleneck is in AB calls - $Quotes($j) / $Quote.Date / Quotes.Remove($j). So, even a c++ port will still be slow.

- You can ask AB (forum?) if there is a better way to do bulk delete. Maybe we can write an AFL scrip to delete data and call it from exe ( if this is possible)
- Gambler said that a yusi application does backfill. Maybe he can review code?
 
josh1

josh1

Well-Known Member
#1,199
#1,199
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

TracerBullet said:
- You can ask AB (forum?) if there is a better way to do bulk delete. Maybe we can write an AFL scrip to delete data and call it from exe ( if this is possible)
- Gambler said that a yusi application does backfill. Maybe he can review code?
Click to expand...
There is no command in AFL to remove quotations. It is possible only thru COM calls and that too by iteration one at a time. That is bottleneck.

Yusi application does backfill. However, his data feed application compiles and sends 1 min. bars to AB. If I backfill at 1 PM, there are approx.225 bars to remove and 375 bars at 3:30 PM. Therefore it is fast.
 
TracerBullet

TracerBullet

Well-Known Member
#1,200
#1,200
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

josh1 said:
There is no command in AFL to remove quotations. It is possible only thru COM calls and that too by iteration one at a time. That is bottleneck.

Yusi application does backfill. However, his data feed application compiles and sends 1 min. bars to AB. If I backfill at 1 PM, there are approx.225 bars to remove and 375 bars at 3:30 PM. Therefore it is fast.
Click to expand...
75/3 = 25. Removing remove function reduced time exactly by 1/3.
We make 3 COM calls per scrip >11000 times.
I think there is a possibility that COM itself is using up time. I tried to profile the exe once, and there was significant time spent on some marshalling stuff. I ignored it, but now i think maybe its trying to convert datatypes.

This gives me hope that we can reduce time with decent c++ implementation.
Ill try to test it later ( not soon ). If there is difference, we can simply call the exe from your autoit script.
 
Status
Not open for further replies.
Thread starter Similar threads Forum Replies Date
vjvijay88 free realtime NIfty data need Data Feeds 0
B Is there any free realtime datafeeder for amibrocker Data Feeds 4
test123 GUi-Based Free RealTime Data NOW NEST 2 AmiBroker, Fcharts Intraday 49
S Any one know how to get free realtime datafeed for metastock(yahoo) Data Feeds 3
S Odin2Ami � Free Odin to Amibroker Realtime Datafeeder Intraday 10

Similar threads