Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts MS

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

@rmike,

This is todays data for Apollo Tyres from Hourly Statistics of NSENOW.

09:14:37 AM	166.70	166.95	166.35	166.40	17677			
09:15:37 AM	166.35	166.70	166.15	166.70	21284			
09:16:37 AM	166.70	166.80	166.55	166.60	6078			
09:17:37 AM	166.60	166.75	166.50	166.75	11375			
09:18:37 AM	166.75	166.85	166.50	166.60	16052			
09:19:37 AM	166.60	166.95	166.55	166.90	6636			
09:20:37 AM	166.90	167.85	166.80	167.25	32865

Apollo Tyres from Nest Trader

09:15:00 AM	166.55	166.95	166.15	166.45	23879	
09:16:00 AM	166.30	166.80	166.25	166.55	18367	
09:17:00 AM	166.70	166.75	166.50	166.50	9161	
09:18:00 AM	166.60	166.75	166.50	166.50	10421	
09:19:00 AM	166.70	166.85	166.50	166.75	12414	
09:20:00 AM	166.65	166.95	166.55	166.95	7671

Nifty Future from NSENOW

09:15:00 AM	6773.10	6775.00	6763.55	6765.10	393000			
09:16:00 AM	6765.10	6767.35	6763.45	6766.15	205850			
09:17:00 AM	6766.15	6768.40	6766.15	6767.20	83800			
09:18:00 AM	6767.15	6771.50	6767.00	6768.10	96250			
09:19:00 AM	6768.10	6768.15	6766.15	6766.55	55000			
09:20:00 AM	6766.35	6769.70	6766.35	6767.90	43150

NIFTYFuture from Nest Trader

09:15:00 AM	6773.10	6775.00	6763.55	6765.10	392800	
09:16:00 AM	6766.20	6766.55	6763.45	6766.15	201650	
09:17:00 AM	6767.00	6768.40	6766.55	6767.20	88200	
09:18:00 AM	6767.00	6771.50	6767.00	6768.10	96250	
09:19:00 AM	6768.05	6768.15	6766.15	6766.55	55000	
09:20:00 AM	6766.35	6769.70	6766.35	6767.90	43150

TataSteel from NSENOW

09:14:37 AM	410.00	411.95	408.00	411.75	61691			
09:15:37 AM	412.00	412.70	411.20	411.45	71847			
09:16:37 AM	411.40	412.00	411.10	411.50	41365			
09:17:37 AM	411.60	411.60	410.65	410.95	27553			
09:18:37 AM	410.75	411.25	410.65	410.70	19273			
09:19:37 AM	410.70	411.20	410.25	410.95	25153			
09:20:37 AM	410.90	411.30	410.80	411.20	11531

TataSteel from NestTrader

09:15:00 AM	410.10	412.70	408.00	412.30	98201	
09:16:00 AM	412.00	412.25	411.10	411.50	51709	
09:17:00 AM	411.65	412.00	410.80	410.80	37708	
09:18:00 AM	410.85	411.25	410.65	411.00	23551	
09:19:00 AM	410.95	411.00	410.25	410.50	17856	
09:20:00 AM	410.85	411.20	410.55	411.00	23117

:lol::lol::lol::lol::lol::lol: Check it for yourself. Are they sacrosanct?
Take data from Google, TT or ODIN user. I bet all will differ.

 

[josh1]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

@rmike

Today, my internet connection conked off at 14:20:14
I resumed after it came alive and NestRTD started data feed at 14:41:25

Now tell me how to maintain volume consistancy with backfill any of the two.
How do I deal with those 14 and 25 seconds. There has to be missing ticks/volume with TickMode or overlap without Tickmode.

Nest Trader is at least consistent with time stamp of hh:mm:00.
And users do backfill and pose questions here why volume does not tally with this and that.

I have stopped replying those questions.

 

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Look around a bit more and you will find that another one is already planning to sell yours. :lol:

you dont say ! who ? pm if needed

Actually kanitkar is a well intentioned guy who developed the utility to help forum members. The fee was only for tech help as he had to hire guys to provide the same. Unfortunately he didn't contend with the general nastiness of the vast majority of 'gimme' mentality guys which abound everywhere (more so in the trading profession

Yes he could be. But he should be upfront about the nature of the product. Maybe i missed something, but i dont think that is a free product. It may still deserve the price he asks.

Nothing wrong in trying to sell what you make. If it was only to recover costs, then you can simply stop selling once those costs are recovered. You dont need 300/500 rs for a simple phone call. Many of the calls would be non-technical by newbies who cant handle a computer. You dont need to hire people for that.
He is an IT PRO a post.
These are pretty simple tools really, they may involve some tedious work but its not complicated in any way.

He says its community product. Maybe i missed it but i did not see multiple programmers contributing.

He says its GPL3. Is it ? Where is my source code link? With GPL 3 - anyone who got a copy of the program, whether free or paid, should get complete source code too.
He can then choose to modify that code. He can redistribute that code too, free ot paid, as long as he complies with GPL3 and publishes the code.

My impression so far is that he is using these statements to generate goodwill. Please correct me if i m wrong and if so i apologize to him in advance.

But since you've brought this up, it is incumbent to sound a gentle warning. Even though your generosity be impeccable, nothing would stop a smart alec from repackaging your app in a plugin (trademarked to boot and distributing it commercially...

Well you have been apprised of the score now

People do bad things because they can. If someone links or compiles my code, legally , he has to release his under gpl3.

I like opensource, i have no desire to make money of this tool and i would like people to contribute if then can. So its gpl3. I use plenty of free tools. This tool is based on open tool from josh. Without it i would not even have had the idea to write this.

Anyway, ill open a new thread in this section so that people can be aware that this tool is free. If more people know, some may choose not to buy if they dont need.

@rmike

Today, my internet connection conked off at 14:20:14
I resumed after it came alive and NestRTD started data feed at 14:41:25

Now tell me how to maintain volume consistancy with backfill any of the two.
How do I deal with those 14 and 25 seconds. There has to be missing ticks/volume with TickMode or overlap without Tickmode.

Nest Trader is at least consistent with time stamp of hh:mm:00.
And users do backfill and pose questions here why volume does not tally with this and that.

I have stopped replying those questions.

josh sir, for now you can even use your tool with COM remove calls.
Just have 2 extra inputs to get from and to time. Most of the bars within that time will be empty so remove will be fast. You will have to go through VWAP data to take only bars within the time.

 

[rmike]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Nest Trader is at least consistent with time stamp of hh:mm:00.
And users do backfill and pose questions here why volume does not tally with this and that.
I have stopped replying those questions.

I can empathize with the anguish.... yours as well as of the questioning users

Back when I started out trading, I was in the same boat with regard to the incomprehensibility of data inconsistencies till the time I took matters in my own hands.....

If you (and the questioning users) are up to a long winded (but well intentioned explanation, then hopefully this may help (a bit).

The most impeccable data exists only in the electronic innards of the exchange.

This data then goes to various broker's servers. Depending upon the physical location (read proximity) of these servers, the data arrives at minutely different times at different broker's end.

The main broker server is called the HO (or Head Office) server.

Here the data is restamped and then despatched to various branch servers.

Why do the broker's server need to restamp the data ??

In fact any in-out transaction is restamped because this provides a frame of reference to the broker for internal quality audit with regard to latency improvement between HO and branch servers, and also for reconstructing the market picture vis-a-vis a client's order in case of a complaint.

When you logically think of it, the broker and his infrastructure is, in fact, an 'Exchange' between you and the actual 'Exchange'. And while you may think that you are dealing with the actual 'Exchange', you are, in fact, virtually dealing with the 'Broker's Exchange' and the actual location, hardware and software setup can have a significant impact upon your trading.

Very few retail traders exist, who trade off a leased line So, you connect to the broker's server via internet. Firstly, you don't know if your platform is connecting to the HO server or the branch server OR even the actual location of these servers (I've had accounts with brokers wherein the platform had user configurable options to connect to either the branch server or the HO server) . Suppose if your platform connects to the branch server which actually is geographically further located than the HO server, then the data reaching you and your neighbour will have different latencies if his platform is connected to the closer HO server.

The inconsistency doesn't end here. There are various ways of net connection. One guy connects by landline broadband, the other by dish broadband, some other by 2g wireless and some other by 3g wireless.

Depending upon the physical cable distance (not the geograhical cable distance, because you'll be surprised by how much the coils in the cable can dramatically alter the actual distance) and/or the tower to tower distance in case of wireless that the data has to travel, the latencies will be minutely different.

What I am underscoring here is that the timestamping of the same data differs minutely depending upon the physical infrastructural wherewithal.

In addition, every broker's server setup is different. Some brokers only offer NOW, some both and some a customized flavour of NEST in addition to NOW. I can't speak with authority regarding the respective setup of each and every broker out there, but as a generalization NOW and NEST run from different servers (there may be one odd exception to this generalization). As a thumb rule, for brokers who don't offer NEST as their trading platform and you as a client choose to subscribe to the free version of NEST (maybe for charting or data pulling reasons) it a sure bet that the data is coming from different servers - hence the timestamping inconsistencies.

You as a retail trader don't have access to tick data, so when you want to backfill, the best that you can hope to get is snapshot data (with the hope that all relevant ticks are captured within that snapshot). Construction of a snapshot has a beginning and an ending partition in terms of time. Hence the timestamp of a datapoint will determine whether it belongs to the previous, the current or the future partition/ time container.

Ergo, therefore data snapshots of the same granularity from different sources will have (slightly/ significantly) different OHLC and volumes. But people who don't know/ understand data transport nuances are always anguished by these inconsistencies.

Where does this, then leave us - the generally disadvantaged retail traders???!!!

To be Contd. - This post is getting too big for comfort

 

[TracerBullet]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Here the data is restamped and then despatched to various branch servers.

Why do the broker's server need to restamp the data ??

In fact any in-out transaction is restamped because this provides a frame of reference to the broker for internal quality audit with regard to latency improvement between HO and branch servers, and also for reconstructing the market picture vis-a-vis a client's order in case of a complaint.

I didnt get why they need to restamp. They can look at latencies using a seperate timestamp. Why tamper with exchange data? It seems wrong to me.
Edit - Also, why would the latency between HO/Branch and my pc matter? Will it be restamped by the client tool? Latency should only cause delay. Bandwidth is hardly used so it should not be dropped unless they dont use tcp.

Also with ZT, it seems they send updates as a single price and not O/H/L/C.
So any ticks missed due to whatever reason is lost. 1 second late data is better than inaccurate for most.

But anyway, everything is an approximation and uncertain in trading - we just have to deal with it.

 

[rmike]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

The problem with a vast majority of hoi polloi retail traders is that they don't really know what to focus on!

Do you need a perfect database??!!
OR
Do you need to make money trading??!!

You need the perfect database IF

You are a (wannabe) HFT/ arb trader who operates on tick/ subsecond timeframes.

You are a (wannabe) Pro system designer who needs this for razor sharp accuracy at subsecond timeframes.

You are a purist volume trader who relies solely on tick volume/ OFA analysis.

If you are the first two of the above then the perfect database is available for purchase at NSE. While being pricey, It would be a worthwhile investment. If you are the third of the above, then you need to understand that theoretical ascribed success (in hindsight) to these methods cannot be replicated under the prevalent conditions unless you buy a seat on the exchange, which will then make you privy to such data.

To be Contd.

 

[rmike]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

My impression so far is that he is using these statements to generate goodwill. Please correct me if i m wrong and if so i apologize to him in advance.

Nothing to correct or to cite as incorrect as I am not privy to specific facts pertaining to his outlay on website maintenance, server charges, office infrastructure n rent, number of staff and their remuneration and overheads like electricity, phone bills etc.
One can draw own inferences depending upon outlook and perspective and there's no point in quibbling over inferential impressions

I didnt get why they need to restamp.

The HO server distributes data to branch servers, restamping provides a frame of reference between the 'Broker's Exchange' and the serviced clients. A broker can't very well go and start investigating NSE server stats in case of a client complaint, which would be the case without any frame of reference.

Minute manipulations of data is a fact, so is rerouting and auctioning of client orders (I wonder to what extent India has caught up with the world in this trend). Nobody with a vested interest will ever give you a straight answer on this subject. In fact the normal lay staff in the broker/ sub broker's office won't even know about these aspects. One just has to live with this and work around it!

why would the latency between HO/Branch and my pc matter? Will it be restamped by the client tool? Latency should only cause delay. Bandwidth is hardly used so it should not be dropped unless they dont use tcp.

Also with ZT, it seems they send updates as a single price and not O/H/L/C.
So any ticks missed due to whatever reason is lost. 1 second late data is better than inaccurate for most.

Meant to highlight impact of infrastructure on latency, the client tool will not restamp data but forgot to mention that it will drop ticks in case of significant varying latency (e.g intermittent sparking in cable connectors of the data transport). With regard to ZT - like I said, diff setup, software, protocol hence diff output.

 

[gambler]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Dear Josh,

I am trying to backfill data with backfill utility from Now on my laptop but it is backfilling data from 12:39:59 and missing data before that. Why its happening?

Thanks!

Gambler & others . . .

He needs feedback . . . on if u r using nest // or now . . . don't forget to mention that


Happy

 

[rmike]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Contd.

As should be amply evident from my preceding (lengthy discourse on the subject, the perfect database is a chimera at the retail trader's level and one can chase after it ad infitum without getting anywhere much.

As a retail trader one needs to focus on making optimal use of the available resources.

One needs to focus on latency and data coherence.

Latency is self explanatory.

Data coherence means that the source, the protocol and the data transport structure remains consistent in the database. For this the source and its timestamping protocol has to be chosen with care.

Why data coherence?! Because all trading is by reference OR rather context.
If the context of the data is consistent even without being 100% accurate, then trading decisions based upon it can be taken with reasonable confidence.

That's why non exchange authorized commercial vendors are a breed to be a bit wary of. One doesn't know his data setup and mining algo, his infrastructure and his data source. One is not apprised if one fine day the data source is changed, there's no accountability for infrastructural lacunae and, in the worst case, the vendor may just fold up his tent and disappear!!! :0

One can opt for an exchange authorized commercial vendor (however sometimes, the approved vendor may not provide your desired segment or have the know how to provide feed for your particular TA platform).

OR, for want of a modicum of control (and economy , a retail trader may like to opt for utilities running on the client's own terminal. One has the option for the data source, upgradation of infrastructure (the rig and the net connection), choice of broker based upon server location proximity, software terminal (and other service/ brokerage related aspects). These utilities won't give absolutely accurate data but if one trades on a timeframe higher than a minute, then these are more than adequate. For backfilling if you find NEST/NOW/SKTT or any other terminal providing consistent timestamped mm:00 data then use that and don't mix sources midway.

Despite cautioning against mixing of data source, I am partial to google and mix it with NOW data (according to my own admission .

The reason is that google buys data directly from NSE for display on its site. NSE is honour (and payment) bound to provide data without any inaccuracies as the accuracy is linked to google's prestige as an information provider. The only stipulation by NSE (and other data provider exchanges) being that google should not display data realtime and that it discontinue the gdata API. But that does not mean that RT data does not reside within google . However the realtime data from google does not have volume information. Volume information is provided in the delayed 1 Min/ 5 Min snapshots. This is an excellent resource for backfill as google has no reason to restamp the received data, it just parcels the data in M1 or M5 containers as per the query. This is as close to accurate data that a retail trader can get.

I would normally recommend a backfill by google data at the end of the day's trading in order to get the optimum in terms of accuracy and coherence. And, of course, to keep the database size manageable.

P.S - Phew!.....I hope at least somebody benefitted from my lengthy diatribe

 

[rmike]

Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

@rmike
Today, my internet connection conked off at 14:20:14
I resumed after it came alive and NestRTD started data feed at 14:41:25
Now tell me how to maintain volume consistancy with backfill any of the two.
How do I deal with those 14 and 25 seconds. There has to be missing ticks/volume with TickMode or overlap without Tickmode.

Backfill in TICKMODE with mm:00 timestamped data till 14:42:00, this will ensure no OHLC or V distortion due to overlap.

While doing this you will lose data from 14:43:00 till 14:43:59.

Take the 14:43:00 timestamped bar from the stats and copy into a text file (either manually or by script). Then run backfill in NORMAL MODE for that single datapoint. The missing data point will insert itself in the requisite vacant timerange in the database.

HTH

Regards,