EOD Download/ Convert from NSE and BSE
- Thread starter pkjha30
- Start date
Hi Dude,
I got EOD data for the past 14 years in this website.But the Nifty data is not available in that.I decided to download nifty EOD using your software.First I tried to download data from 01/01/09 to 16/01/09 the download part did its job well then I tried to use convertion part of your software before that I decided to use Advanced input settings and pressed the button.I could not see any option in that area.Finally I realised without equity data for that particular date your software won't convert the downloaded indices data.One more thing also I noticed while viewing downloaded indices data.Some day indices data don't have volume,turnover,del_vol,del_per,some files don't have del_vol,del_per and few files have all the data.Please help me to solve this problem.I have attached the downloaded data in zip format.
One more thing also I already PM u the problem in using delivery data of equity in metastock.
I got EOD data for the past 14 years in this website.But the Nifty data is not available in that.I decided to download nifty EOD using your software.First I tried to download data from 01/01/09 to 16/01/09 the download part did its job well then I tried to use convertion part of your software before that I decided to use Advanced input settings and pressed the button.I could not see any option in that area.Finally I realised without equity data for that particular date your software won't convert the downloaded indices data.One more thing also I noticed while viewing downloaded indices data.Some day indices data don't have volume,turnover,del_vol,del_per,some files don't have del_vol,del_per and few files have all the data.Please help me to solve this problem.I have attached the downloaded data in zip format.
One more thing also I already PM u the problem in using delivery data of equity in metastock.
Hi Dude,
I got EOD data for the past 14 years in this website.But the Nifty data is not available in that.I decided to download nifty EOD using your software.First I tried to download data from 01/01/09 to 16/01/09 the download part did its job well then I tried to use convertion part of your software before that I decided to use Advanced input settings and pressed the button.I could not see any option in that area.Finally I realised without equity data for that particular date your software won't convert the downloaded indices data.One more thing also I noticed while viewing downloaded indices data.Some day indices data don't have volume,turnover,del_vol,del_per,some files don't have del_vol,del_per and few files have all the data.Please help me to solve this problem.I have attached the downloaded data in zip format.
One more thing also I already PM u the problem in using delivery data of equity in metastock.
Dates ,where there are no data, appear to be holidays or weekends. In indices I have not filtered it probably.
I got EOD data for the past 14 years in this website.But the Nifty data is not available in that.I decided to download nifty EOD using your software.First I tried to download data from 01/01/09 to 16/01/09 the download part did its job well then I tried to use convertion part of your software before that I decided to use Advanced input settings and pressed the button.I could not see any option in that area.Finally I realised without equity data for that particular date your software won't convert the downloaded indices data.One more thing also I noticed while viewing downloaded indices data.Some day indices data don't have volume,turnover,del_vol,del_per,some files don't have del_vol,del_per and few files have all the data.Please help me to solve this problem.I have attached the downloaded data in zip format.
One more thing also I already PM u the problem in using delivery data of equity in metastock.
Dates ,where there are no data, appear to be holidays or weekends. In indices I have not filtered it probably.
As for metastock, I think it may have something to do with headers. You can change firt header line manually to see what is accepted by Meta then that can be inserted into DB to get correct header.
I don't have meta so I can't troubleshoot without feedback.
dates, when no data is present, appears to be weekend or holidays.
pk
Last edited:
After seeing your data, I feel that you need to download delivery data seperately. immediately after eod data.Then Indices data will have delivery volume and percent.Second point is that it is calculated based on EOD delivery data of constituent scrips.Constituents are updated or you might update it. In that case if you are downloading historical data then some older delivery data may not come or may not be correct as Indices constituents keep changing. zero simply means that for that day there is no delivery data for scrips.
As for metastock, I think it may have something to do with headers. You can change firt header line manually to see what is accepted by Meta then that can be inserted into DB to get correct header.
I don't have meta so I can't troubleshoot without feedback.
dates, when no data is present, appears to be weekend or holidays.
pk
As for metastock, I think it may have something to do with headers. You can change firt header line manually to see what is accepted by Meta then that can be inserted into DB to get correct header.
I don't have meta so I can't troubleshoot without feedback.
dates, when no data is present, appears to be weekend or holidays.
pk
Thank you for quick response and spending your valuable time in between your busy schedule.My first query thru PM is related to metastock but the input of that data is the source of the problem(it is generated using your software).I think there may be mismatch in data type (open interest or Delivery Volume).If you find any solution to this problem please let me know about it.
second problem is downloading indices data separately.I already know the basic details of how to use your software.I know holidays or weekends don't have data.Let me elabarate my query I downloaded indices data separately from 01/01/09 to 16/01/09.
03,04,08,10,11 --- Weekend & Holiday
12,13,14,15 --- Don't have volume,turnover,Del_vol,Del_qty
01,02,05,06,07,09 --- Don't have Del_vol,Del_qty
07,16 --- Have all the data
What is the meaning for volume,turnover,del_vol,del_qty in spot market indices(that data too present for only few dates).Because in Equity we can't trade indices they are derived from some calculation (based on 50 scripts in case of nifty).
Atlast I want solution for these two problems.
1)I could not use delivery volume(open interest) data of your software in Meta stock.When I converted the downloaded nse eod data with del_volume column meta stock could not recognize it.
2)I have downloaded metastock format eod data of nse for the past 14 years from some other source.But the problem is there is no indices(Atleast I want nifty) data in that.Is it possible to download indices data separately and convert that data only using your software.When I tried this with your software,first for testing purpose I downloaded two weeks indices data (without equity data) from 01/01/09 to 16/01/09 and tried to conver that data but in your softwares conversion part is not allowing me to select advanced input settings I mean there is no data displayed in input & output sequence and when I pressed convert button it was telling me "no files to convert".
I like to simplify the second question the straight forward question is it possible to convert (your software already downloading indices data separately) nifty spot and other indices details (with ticker,date,open,high,low,close) alone for the past few years(I don't want equity scripts details because I already have these data for 14 years).So that I put these details in Indices directory and use it in metastock.You have already done nice job of classifying the downloaded data in separate directories (Equity,FNO,Indices).But the conversion part is not recognizing indices data without equity scripts data related to that particular period.It would be better if conversion part modified to process indices data separately.
Last edited:
Thanks for detailed reply. There may be issues in sequence of downloading data.I will try to see if I also get same result.
Indices Delivery data is derived data based on scrips' del qty.
As i want to move to Database centric design, some col need to be filled in. So it was also calculated.
As of now the sourcecode is in semi broken stage for correcting/ reducing the size of compiled exe as well as to go for complete db based design. So it is likely to take some time. Till I come with new version, have patience. I will keep in mind your feedback.
pk
Indices Delivery data is derived data based on scrips' del qty.
As i want to move to Database centric design, some col need to be filled in. So it was also calculated.
As of now the sourcecode is in semi broken stage for correcting/ reducing the size of compiled exe as well as to go for complete db based design. So it is likely to take some time. Till I come with new version, have patience. I will keep in mind your feedback.
pk
I tried to download EOD but could not do so.
How to enable pm ?Pls advice.
my email id is [email protected]
How to enable pm ?Pls advice.
my email id is [email protected]
Re: EOD Download/ Convert from NSE , BSE and MCX (NCDEX in future)
great work Pankaj..........
before downloading and installing the downloader I want to ask few thing.
1. whether data will work in fibotrader
2. can we download bulk say 2-3 yrs. data.
Hi Traderji and all the Forum members
I am posting EODDwonloadConvert version 0.0.70 for forum members to
try and give their comments.Feel free to use it and give feedback. No warranties.Code is optimisation under progress.
It needs unzip32.dll which can be obtained from
http://www.mirrorservice.org/sites/ftp.info-zip.org/pub/infozip/WIN32/unz552dN.zip
It need to be placed in \windows\system ditrctory and register it.Rightclick on it will register it else use "regsvr32.exe unzip32.dll" in Run dialog of Start Button. Ignore any error. The version is unzip32.dll 5.5.2.0. . Only dll is available in zipped file in my briefcase.
The programme is compiled using vb6(sp6) and needs runtime files.Normally these are available on almost all systems now but on some older system it may not be available and give errors.You can download runtime files from here if required.
http://www.microsoft.com/downloads/...61-7A9C-43E7-9117-F673077FFB3C&displaylang=en
Its 1.0 MB download.
you need IE 5.5 or above installed.
Dependency Script is attached for your use to verify if required files are present.I have tested it on Win98SE and WINXP system . it should not give much trouble. The zipped file EODDownloadConvert.zip(142.5kb) may be downloaded from my briefcase.
http://briefcase.rediff.com/bn/br/pjhapondy
The programme should be extracted to a seperate folder of your choice. It will create folder structure and required files on first run . Settings related files are in BIN folder. My BIN folder is also placed in the Briefcase in case you may like to download some files such as Favourites.txt,holiday.txt etc. Programme icon is in BIN\images folder.
I have to thank http://www.planetsourcecode.com , www.freevbcode.com
and www.programmersheaven.org and www.info-zip.com for their code
snnipets and dll files. All thanks to respective programmers.My part was limited to certain areas and ideas specific to downloading files from nse and bse website and assemblling it to make a workable solution for members of Traderji.com
I have to thank satya whose dataextractor gave some inspiration for
this and his idea of thread to share software. My code will be freely available once it reaches version 1.0.0 and after your feed back is also incorporated.The programme has its own lifetime and utility.It may stop functioning if there are changes in URL structure of NSE and BSE websites.If newer operating system do not support older programmes. In such a situation it would need to be ported to newer versions.
What it seeks to do:-
Download:
For a given range of dates it will download bhavcopy for nse. For BSE it will download current Bhavcopy which is in zipped format. Itwill unzip the same without user intervention.You have to enter dates from and to in the field and click ADDUrl.
All URLs will be displayed in the box below with save path. As of now use default options it should be ok. For other options if there is any problem please tell me.
On the top there is option for providing download urls if you know it. If it is from BSE or NSE it will be stored in default folders. Other wise it should be available in Others folder.
Convert:
Initially use only default options.
If files are available in default folders then it will be visible in Input Files Box. Defaults are ok i.e. csv and nse or bse. Delimiter is always "," i.e. comma. leave it unchanged. You will see the first two lines of records in Select output options. If first line is header record you can check it . You can leave all other options as it is. On clicking Convert button it will convert all available file in NSE or BSE equities folder One folder will contain date wise EOD data for each date for all stocks, the other folder will contain
individual stock data for all dates available.
Datewise format is symbol,o,h,l,c,v,date
Stockwise format is o,h,l,c,v,date
First line is header line.
Perhaps these files can be used by members for importing into their
programmes. if any error is given I would like to know.
If you have downloaded many files you need to sort these records in the output files. Use sort button for this. It needs to be used for once only. thereafter if you are running your program after 5.30 or whatever time bse/nse post their latest data you dont need to sort.
I understand that bhavcopy is posted after 5.30 pm.
You can exclude any date by editing the holiday.txt file either directly. All dates are posted below respective month. Same pattern needs to be followed. the program will indicate if their is a failure of Download for whatever reason mostly URL is not available i.e. file is not found. It is trapped and failure.log is generated.
You can verify if URL exixts or not by using some other downloader.Error.log file wil keep records of two days errors and may be used for feed back.
Browser:
You can use this to browse your favourit websites wihtout overhead of IE. But you need IE 5.5 or above installed.
Some of the Links will give you direct access to nes/bse html files without going to home page. Loading will be faster. You can add favourites First input is for title and secon input is for URL. If you omit URL title can act as section header.
You can directly edit Favourites in your text editor. Odd lines will be titles and even lines should be URLs.
You can also check if your computer is connected to internet and if NSE website is up and running. Buttons are on Download page.
If this programme works it will be free and once it is optimised it will be given in source code form for maitainance by members if desired.
Future Wishlist
To optimise code.
To incorporate charts and other indicators based on the data collected on daily basis. Chart code and form is included but made invisible as further development is required.
To incorporate multi threaded downloading capability.
To take print( Presently you can print file in one go)
The name given to the executable is only tentative and members are invited to suggest better name. We may vote for name and whichever name gets highest vote will be made as programme name.
thanks are in order for Satya, joymitali,Saint, AmitBE,AMITKR murthymsr and otherboard members for inspiraton.
I dedicate this programme to the enterprising spirit of Traderji.com
and Moderator traderji.
With kind regards
Pankaj
I am posting EODDwonloadConvert version 0.0.70 for forum members to
try and give their comments.Feel free to use it and give feedback. No warranties.Code is optimisation under progress.
It needs unzip32.dll which can be obtained from
http://www.mirrorservice.org/sites/ftp.info-zip.org/pub/infozip/WIN32/unz552dN.zip
It need to be placed in \windows\system ditrctory and register it.Rightclick on it will register it else use "regsvr32.exe unzip32.dll" in Run dialog of Start Button. Ignore any error. The version is unzip32.dll 5.5.2.0. . Only dll is available in zipped file in my briefcase.
The programme is compiled using vb6(sp6) and needs runtime files.Normally these are available on almost all systems now but on some older system it may not be available and give errors.You can download runtime files from here if required.
http://www.microsoft.com/downloads/...61-7A9C-43E7-9117-F673077FFB3C&displaylang=en
Its 1.0 MB download.
you need IE 5.5 or above installed.
Dependency Script is attached for your use to verify if required files are present.I have tested it on Win98SE and WINXP system . it should not give much trouble. The zipped file EODDownloadConvert.zip(142.5kb) may be downloaded from my briefcase.
http://briefcase.rediff.com/bn/br/pjhapondy
The programme should be extracted to a seperate folder of your choice. It will create folder structure and required files on first run . Settings related files are in BIN folder. My BIN folder is also placed in the Briefcase in case you may like to download some files such as Favourites.txt,holiday.txt etc. Programme icon is in BIN\images folder.
I have to thank http://www.planetsourcecode.com , www.freevbcode.com
and www.programmersheaven.org and www.info-zip.com for their code
snnipets and dll files. All thanks to respective programmers.My part was limited to certain areas and ideas specific to downloading files from nse and bse website and assemblling it to make a workable solution for members of Traderji.com
I have to thank satya whose dataextractor gave some inspiration for
this and his idea of thread to share software. My code will be freely available once it reaches version 1.0.0 and after your feed back is also incorporated.The programme has its own lifetime and utility.It may stop functioning if there are changes in URL structure of NSE and BSE websites.If newer operating system do not support older programmes. In such a situation it would need to be ported to newer versions.
What it seeks to do:-
Download:
For a given range of dates it will download bhavcopy for nse. For BSE it will download current Bhavcopy which is in zipped format. Itwill unzip the same without user intervention.You have to enter dates from and to in the field and click ADDUrl.
All URLs will be displayed in the box below with save path. As of now use default options it should be ok. For other options if there is any problem please tell me.
On the top there is option for providing download urls if you know it. If it is from BSE or NSE it will be stored in default folders. Other wise it should be available in Others folder.
Convert:
Initially use only default options.
If files are available in default folders then it will be visible in Input Files Box. Defaults are ok i.e. csv and nse or bse. Delimiter is always "," i.e. comma. leave it unchanged. You will see the first two lines of records in Select output options. If first line is header record you can check it . You can leave all other options as it is. On clicking Convert button it will convert all available file in NSE or BSE equities folder One folder will contain date wise EOD data for each date for all stocks, the other folder will contain
individual stock data for all dates available.
Datewise format is symbol,o,h,l,c,v,date
Stockwise format is o,h,l,c,v,date
First line is header line.
Perhaps these files can be used by members for importing into their
programmes. if any error is given I would like to know.
If you have downloaded many files you need to sort these records in the output files. Use sort button for this. It needs to be used for once only. thereafter if you are running your program after 5.30 or whatever time bse/nse post their latest data you dont need to sort.
I understand that bhavcopy is posted after 5.30 pm.
You can exclude any date by editing the holiday.txt file either directly. All dates are posted below respective month. Same pattern needs to be followed. the program will indicate if their is a failure of Download for whatever reason mostly URL is not available i.e. file is not found. It is trapped and failure.log is generated.
You can verify if URL exixts or not by using some other downloader.Error.log file wil keep records of two days errors and may be used for feed back.
Browser:
You can use this to browse your favourit websites wihtout overhead of IE. But you need IE 5.5 or above installed.
Some of the Links will give you direct access to nes/bse html files without going to home page. Loading will be faster. You can add favourites First input is for title and secon input is for URL. If you omit URL title can act as section header.
You can directly edit Favourites in your text editor. Odd lines will be titles and even lines should be URLs.
You can also check if your computer is connected to internet and if NSE website is up and running. Buttons are on Download page.
If this programme works it will be free and once it is optimised it will be given in source code form for maitainance by members if desired.
Future Wishlist
To optimise code.
To incorporate charts and other indicators based on the data collected on daily basis. Chart code and form is included but made invisible as further development is required.
To incorporate multi threaded downloading capability.
To take print( Presently you can print file in one go)
The name given to the executable is only tentative and members are invited to suggest better name. We may vote for name and whichever name gets highest vote will be made as programme name.
thanks are in order for Satya, joymitali,Saint, AmitBE,AMITKR murthymsr and otherboard members for inspiraton.
I dedicate this programme to the enterprising spirit of Traderji.com
and Moderator traderji.
With kind regards
Pankaj
great work Pankaj..........
before downloading and installing the downloader I want to ask few thing.
1. whether data will work in fibotrader
2. can we download bulk say 2-3 yrs. data.
| Thread starter | Similar threads | Forum | Replies | Date |
|---|---|---|---|---|
| P | DOWNLOAD NSE_FO EOD | Data Feeds | 0 | |
| V | StockD - Another NSE EOD Data Downloader | Software | 112 | |
| R | DQuotes - NSE and BSE and F&O EOD Bhavcopy and Indices downloader | End of Day | 236 | |
| S | Download New Version- NSE Intraday,EOD quote downloader/convertor Metastock/Amibroker | Software | 78 | |
| A | Run-time error while running EOD-Download converter | Software Issues | 1 |
Similar threads
-
-
-
DQuotes - NSE and BSE and F&O EOD Bhavcopy and Indices downloader
- Started by rpc
- Replies: 236
-
Download New Version- NSE Intraday,EOD quote downloader/convertor Metastock/Amibroker
- Started by Stokist
- Replies: 78
-