Market Data Downloader

Getting Started

Market Data Downloader is command line utility for loading market data for stocks, ETFs, indices, and options to CSV on a daily basis.

Market Data Downloader includes the following data providers:

• Yahoo Finance Quotes
• Yahoo Finance Key Statistics
• Yahoo Finance Industries
• Yahoo Finance Historical Prices
• Yahoo Finance Dividends
• Yahoo Finance Options

You may load data for any exchange supported by Yahoo Finance.

The download package includes a preconfigured project for several markets. Please try it to a quick start.

You may automate loading data on schedule and import loaded CSV data to databases or to Excel spreadsheets.

The download package includes ready-to-use format files and SQL codes to create a market data database for Microsoft SQL Server.

 
You may try all the features during a 15-day trial period.

You may start learning on the following topics:

• Using Market Data Downloader
• Command Line
• Application Configuration File
• Preconfigured Project
• Configuring Markets
• Yahoo Finance Data Providers
• Market Data Database
• Frequently Asked Questions

Change History

Version 1.5, November 11, 2016

Improvements:

• Updated YahooFinanceQuotes and YahooFinanceHistoricalPrices URLs.
• New application settings: Timeout, Accept, and UserAgent.

Bug Fixes:

• Fixed exceptions on empty CSV results.

Version 1.4, July 19, 2016

Improvements:

• Updated YahooFinanceKeyStatistics provider.
• Updated YahooFinanceIndustries provider.

Version 1.3, July 13, 2016

Improvements:

• Updated YahooFinanceOptions provider.

Version 1.2, June 27, 2016

New Features:

• Pressing Q stops operations.
• Pressing P pauses operations.
• Downloader shows statistics of the lastest downloads.

Improvements:

• Batch files wait pressing any key before closing windows.
• Options expired in more then 5 years are excluded from download.
• yahoo.Quotes.Name has a new nvarchar(255) type instead of varchar(50).

Bug Fixes:

• Fixed ignoring YahooFinanceHistoricalPricesStartDate.

Version 1.1, March 15, 2016

Bug Fixes:

• Fixed 30-second delay in registered products.
• Fixed exceptions on processing empty files.

Version 1.0, February 24, 2016

New Data Providers:

• Yahoo Finance Key Statistics.
• Yahoo Finance Quotes.
• Yahoo Finance Industries.
• Yahoo Finance Historical Prices.
• Yahoo Finance Dividends.
• Yahoo Finance Options.

Edition Comparison

Market Data Downloader allows loading market data for stocks, ETFs, indices, and options into CSV files.

Feature	Unregistered	Trial	Personal	Enterprise
Data Providers
Yahoo Finance Key Statistics	✓	✓	✓	✓
Yahoo Finance Quotes	✓	✓	✓	✓
Yahoo Finance Industries	✓	✓	✓	✓
Yahoo Finance Historical Prices	✓	✓	✓	✓
Yahoo Finance Dividends	✓	✓	✓	✓
Yahoo Finance Options (US markets only)	✓	✓	✓	✓
Features
Download mode	✓	✓	✓	✓
Reparse mode	✓	✓	✓	✓
Number of tickers to download	10	Unlimited	Unlimited	Unlimited
Minimal delay between requests, milliseconds	30000	500	500	500
Licensing
One-time price per copy*			$99	$299
Commercial use	x	✓	x	✓

You may try all the features during a 15-day trial period. After this period, the application is used in the unregistered mode.

You have to activate the Personal or Enterprise edition after the purchase.

Install and Uninstall

Installing Market Data Downloader

To install Market Data Downloader:

• Download and unzip Market Data Downloader setup package to a local folder.
• Run setup.exe and follow installer instructions.
• Log off and log on (or just restart your computer).

Market Data Downloader requires the following preinstalled Microsoft components:

• NET Framework 4 Client Profile or Microsoft .NET Framework 4.

If components are not installed, you will be asked to download and install them from the Microsoft website.
It may require downloading additionally about 45MB.

Updating Market Data Downloader

To update Market Data Downloader, just install a new version.

Uninstalling Market Data Downloader

To uninstall Market Data Downloader:

• Open Control Panel.
• Open Program and Features.
• Select Market Data Downloader and click the Uninstall button.

Using Portable Version

You may use a portable version instead of the installed one.

Just copy the market-data.exe and market-data.exe.config files to the desired location.

If the application is not working, install the installed version first to install all the required components.

To update the application, just copy the files from a new version package.

Creating Projects

The downloaded package contains the MarketData folder of the preconfigured data project.

The project allows loading data from various exchanges, like US, CA, UK, AU, etc.

If you plan to use the preconfigured market data database, copy the MarketData folder to the C: drive root.
Otherwise, you need to change paths in SQL files.

If you plan to use downloaded CSV data your ways, you may copy the project folder to any location.

System Requirements

Supported Architectures:

• x86
• x64

Supported Operating Systems:

• Windows XP SP3, Vista SP1, 7, 8, 8.1, 10
• Windows Server 2003 SP3, 2008, 2008 R2, 2012, 2012 R2, 2016

.NET Framework 4 Client Profile or higher must be installed.

Trial Version Limitations

You can use Market Data Downloader without any limit during a 15-day trial period.

When the trial period has expired, the Market Data Downloader switches to the unregistered mode, and you have the following options:

1. To purchase a license and register the Market Data Downloader Personal or Enterprise edition.
2. To stop the use and remove Market Data Downloader from a computer.

See also

• Edition Comparison
• End-User License Agreement
• Product Registration

End-User License Agreement

This End-User License Agreement (EULA) is a legal agreement between you (either an individual or a single entity) and Sergey Vaselenko for any Market Data Downloader software, use examples and documentation (Software) that accompany this EULA.

YOU AGREE TO BE BOUND BY THE TERMS OF THIS EULA BY INSTALLING, COPYING, OR OTHERWISE USING THE SOFTWARE.

IF YOU DO NOT AGREE, DO NOT INSTALL, COPY, OR USE THE SOFTWARE.

Your licensing of Software is in accordance with the terms of the EULA in effect at the time of such licensing. By licensing Software, you accept and agree to the EULA in effect at such time.

1. GRANT OF LICENSE: Sergey Vaselenko grants you the following rights provided that you comply with all terms and conditions of this EULA:
   1. Trial Period: You may install, use, access, display and run the Software for purposes of evaluation of the Software for a period of fifteen (15) days from the date of installation of the Software to determine if the Software meets your needs. Upon completion of your evaluation or at the end of the 15-day trial period, whichever occurs first, you must do one of three things: either 1) purchase license for the Market Data Downloader Personal or Enterprise edition in order to continue use of Software; or 2) discontinue use of the Software, and uninstall the Software from any and all machines.
   2. Installation and Use: You may install, use, access, display and run one (1) copy of the Software per license, on a non-exclusive basis and without right of sublicense or concurrent use, on a single computer or machine, such as a workstation.
      The commercial use of the Market Data Downloader Personal edition is not permitted, including the use by professional traders and institutional investors.
   3. Software Transfer: You may transfer the Software to a different internal workstation or user so long as you have purchased a License for each such workstation or user. You may not, however, transfer the Software to a Third Party.
   4. Use of Examples: You may install, access, modify and use Software examples, including Microsoft Excel workbooks, databases, SQL codes, and batch files, for your private or company internal purposes.
2. LIMITATIONS: You may not use, copy, modify, display, rent, lease, loan, transfer, distribute, download, merge, or make any translation or derivative work of the Software, except as expressly provided herein. You may not reverse engineer, decompile, or disassemble the Software, except and only to the extent that such activity is expressly permitted by applicable law notwithstanding this limitation.
3. INTELLECTUAL PROPERTY RIGHTS AND CONFIDENTIALITY: The Software, including methods, processes and/or techniques utilized therein, is owned by, proprietary to and valuable trade secrets of Sergey Vaselenko and is protected by Russian Federation copyright law and international treaties. You agree to take no actions that impair or infringe Sergey Vaselenko's intellectual property rights in the Software. You agree not to remove, efface or obscure any copyright notices, other proprietary markings or confidentiality legends placed upon or contained within the Software.
4. DISCLAIMER OF WARRANTIES: Sergey Vaselenko disclaims all warranties concerning the Software and Services (if any), express, implied, or statutory, including without limitation, any warranties, duties or conditions of merchantability or fitness for a particular purpose, warranties of reliability or availability, of accuracy or completeness of responses, of results, of workmanlike effort, of lack of viruses, and of lack of negligence, all with regard to the Software, and the provision of or failure to provide support or other services, information, software, and related content through the Software or otherwise arising out of the use of the Software. Sergey Vaselenko does not warrant that the Software will operate in combination with other software products selected by you, or that the Software will operate uninterrupted or error-free. Additionally, Sergey Vaselenko and its suppliers provide the Software and Services AS IS AND WITH ALL FAULTS. THERE IS NO WARRANTY OR CONDITION OF TITLE, QUIET ENJOYMENT, QUIET POSSESSION, CORRESPONDENCE TO DESCRIPTION, OR NONINFRINGEMENT WITH REGARD TO THE SOFTWARE.
5. NO LIABILITY: TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT SHALL SERGEY VASELENKO OR ITS SUPPLIERS BE LIABLE FOR ANY SPECIAL, INCIDENTAL, PUNITIVE, INDIRECT, OR CONSEQUENTIAL DAMAGES WHATSOEVER (INCLUDING, BUT NOT LIMITED TO, DAMAGES FOR LOSS OF PROFITS OR CONFIDENTIAL OR OTHER INFORMATION, FOR LOSS OF DATA, FOR BUSINESS INTERRUPTION, FOR PERSONAL INJURY, FOR LOSS OF PRIVACY, FOR FAILURE TO MEET ANY DUTY INCLUDING OF GOOD FAITH OR OF REASONABLE CARE, FOR NEGLIGENCE, AND FOR ANY OTHER PECUNIARY OR OTHER LOSS WHATSOEVER) ARISING OUT OF OR IN ANY WAY RELATED TO THE USE OF OR INABILITY TO USE THE SOFTWARE, THE PROVISION OF OR FAILURE TO PROVIDE SUPPORT OR OTHER SERVICES, INFORMATION, SOFTWARE, AND RELATED CONTENT THROUGH THE SOFTWARE OR OTHERWISE ARISING OUT OF THE USE OF THE SOFTWARE, OR OTHERWISE UNDER OR IN CONNECTION WITH ANY PROVISION OF THIS EULA, EVEN IN THE EVENT OF THE FAULT, TORT (INCLUDING NEGLIGENCE), MISREPRESENTATION, STRICT LIABILITY, BREACH OF CONTRACT OF SERGEY VASELENKO OR ANY SUPPLIER, AND EVEN IF SERGEY VASELENKO OR ANY SUPPLIER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
6. LIMITATION ON REMEDIES: Within the first thirty (30) days after your receipt of the Software, should you encounter and report to Sergey Vaselenko within such time period a reproducible error that causes the Software not to perform in all material respects as set forth in the Software documentation, then Sergey Vaselenko will, at its sole discretion, either: a) resolve the error or malfunction, and modify or replace the Software (if deemed necessary by Sergey Vaselenko); or b) allow you to terminate this EULA with respect to the non-conforming Software and, upon your return of the Software to Sergey Vaselenko, Sergey Vaselenko shall provide you with the lesser of a) the actual damages incurred by you; or b) the amount you paid for the nonconforming Software. The remedies described in this section shall be your sole and exclusive remedies under this EULA.
7. GENERAL PROVISIONS
   1. Reservation of Rights and Ownership: Sergey Vaselenko reserves all rights not expressly granted to you in this EULA. The Software is licensed, not sold.
   2. Consent to Use of Data: You agree that Sergey Vaselenko and its affiliates may collect and use technical information gathered as part of the Software support services provided to you, if any, related to the Software. Sergey Vaselenko may use this information solely to improve Sergey Vaselenko products or to provide customized services or technologies to you and will not disclose this information in a form that personally identifies you.
   3. Links to Third Party Sites: We may link to third party sites through the use of the Software. The third party sites are not under the control of Sergey Vaselenko, and Sergey Vaselenko is not responsible for the contents of any third party sites, any links contained in third party sites, or any changes or updates to third party sites. Sergey Vaselenko is not responsible for webcasting or any other form of transmission received from any third party sites. Sergey Vaselenko is providing these links to third party sites to you only as a convenience, and the inclusion of any link does not imply an endorsement by Sergey Vaselenko of the third party site.
   4. Additional Software/Services: This EULA applies to updates, supplements, add-on components, or Internet-based services components, of the Software that Sergey Vaselenko may provide to you or make available to you after the date you obtain your initial copy of the Software, unless Sergey Vaselenko provides other terms along with the update, supplement, add-on component, or Internet-based services component. Sergey Vaselenko reserves the right to discontinue any Internet-based services provided to you or made available to you through the use of the Software.
   5. Upgrades: To use Software identified as an upgrade, you must first be licensed for the software identified by Sergey Vaselenko as eligible for the upgrade. After upgrading, you may no longer use the software that formed the basis for your upgrade eligibility.
   6. Applicable Law: This EULA is governed by the laws of the Russian Federation. Any legal action or proceeding relating to this EULA shall be instituted in a court of arbitration in the Moscow City, Russian Federation. Sergey Vaselenko and you agree to submit to the jurisdiction of, and agree that venue is proper in, these courts in any such action or proceeding. The prevailing party in any action to enforce this EULA will be entitled to recover its attorney fees and costs in connection with such action.
   7. Waiver: The failure of either party to enforce any of the terms of this EULA shall not be construed as a waiver of future enforcement of that or any other term.
   8. Entire Agreement and Severability: This EULA (including any addendum or amendment to this EULA which is included with the Software) is the entire agreement between you and Sergey Vaselenko relating to the Software and the support services (if any) and it supersedes all prior or contemporaneous oral or written communications, proposals and representations with respect to the Software or any other subject matter covered by this EULA. To the extent the terms of any Sergey Vaselenko policies or programs for support services conflict with the terms of this EULA, the terms of this EULA shall control. If any provision of this EULA is held to be void, invalid, unenforceable or illegal, the other provisions shall continue in full force and effect.
   9. Termination: Without prejudice to any other rights, Sergey Vaselenko may terminate this EULA if you fail to comply with the terms and conditions of this EULA. In such event, you must destroy all copies of the Software and all of its component parts and you will not be entitled to any refund of monies.

Using Market Data Downloader

Market Data Downloader is a command line utility that has two main modes: download and reparse.

In the download mode, the application downloads data, converts data to CSV, and makes zip archives of all the data.

In the reparse mode, the application reparses archived downloaded data and saves the data to CSV.
This mode mainly used to reparse data with a new version of the utility if the previous version does not work properly.

You may create batch files to call the utility with required parameters and configure automatic downloading on schedule.

The download package contains a preconfigured project that allows loading data from several markets and importing data into a Microsoft SQL Server database.

Downloading Market Data

In the download mode the utility has the following parameters:

market-data.exe download <provider> [<symbol file name>] [/config=<configuration file>]

Here is an example of commands for loading data from US exchanges:

market-data.exe download YahooQuotes            symbols\us-quotes.txt       /config=configs\us.config
market-data.exe download YahooKeyStatistics     symbols\us-quotes.txt       /config=configs\us.config
market-data.exe download YahooIndustries        symbols\us-industries.txt   /config=configs\us.config
market-data.exe download YahooOptions           symbols\us-options.txt      /config=configs\us.config
market-data.exe download YahooHistoricalPrices  symbols\us-prices.txt       /config=configs\us.config
market-data.exe download YahooDividends         cache\YahooHistoricalPricesDividendsOrSplits.txt    /config=configs\us.config

You may see that commands use different providers, different symbol list files located at the symbols subfolder, and the same application configuration file located at the configs subfolder.

You may place your tickers to symbol files using a text editor like notepad.exe, or export tickers from your database.

The application configuration file contains important settings like open and close market times, market time zones, used folders, etc.

The commands above produce the following result files:

csv\YahooDividends.csv
csv\YahooHistoricalPrices.csv
csv\YahooIndustries.csv
csv\YahooKeyStatistics.csv
csv\YahooOptions.csv
csv\YahooQuotes.csv

You may import CSV data to a database or to Excel spreadsheets. 

See related topics:

• Command Line
• Application Configuration File
• Preconfigured Project
• Configuring Markets
• Yahoo Finance Data Providers

Preconfigured Project and Market Data Database

The download package includes a preconfigured project.

You may create a Microsoft SQL Server market data database using the included SQL code, and adapt project batch files to your needs or even use them as is.

See related topics:

• Preconfigured Project
• Market Data Database

Reparsing Downloaded Data

In the reparse mode the utility has the following parameters:

market-data.exe reparse  <provider> [<date as yyyymmdd>] [/config=<configuration file>]

Here is an example of commands for reparsing downloaded data from US exchanges for all trading dates:

market-data.exe download YahooQuotes            /config=configs\us.config
market-data.exe download YahooKeyStatistics     /config=configs\us.config
market-data.exe download YahooIndustries        /config=configs\us.config
market-data.exe download YahooOptions           /config=configs\us.config
market-data.exe download YahooHistoricalPrices  /config=configs\us.config
market-data.exe download YahooDividends         /config=configs\us.config

See related topics:

• Command Line
• Application Configuration File

Command Line

Command Line Modes:

market-data.exe download <provider> [<symbol file name>] [/config=<configuration file>]

market-data.exe reparse  <provider> [<date as yyyymmdd>] [/config=<configuration file>]

Examples:

market-data download YahooFinanceOptions symbols.txt

market-data reparse  YahooFinanceOptions 20160224

market-data download YahooFinanceQuotes  symbols\uk-symbols.txt /config=configs\uk.config

market-data reparse  YahooFinanceQuotes  /config=configs\uk.config

Download Mode

In this mode, the application loads and parses data using the specified provider for the specified symbol list.

The symbol file may be omitted if it is specified in the configuration file.

Reparse Mode

In this mode the application reparses loaded data (in the downloads folder) for the specified trade date.

If the trade date is not specified, the application reparsed all the files.

This mode is used to reparse data using a new version of Market Data Downloaded if the previous version does not parse the data correctly.
In the last case, you may continue to download data and reparse the loaded data with a new version later.

Providers

You may use the following data providers:

• YahooFinanceKeyStatistics
• YahooFinanceQuotes
• YahooFinanceIndustries
• YahooFinanceHistoricalPrices
• YahooFinanceDividends
• YahooFinanceOptions

You may use different symbol lists for different providers, different time periods, and different markets.

You may load all the data in one cycle using a batch file. See examples in Preconfigured Project.

/Config Option

This option allows specifying different configuration files with different application settings when the application is used in the same folder.

Exit Codes

You may use the following exit codes in batch files:

Exit Code	Description
0	Success
1	Incomplete command line parameters
2	Wrong command line parameters
3	Exceptions occurred
4	A user pressed Q and stopped operations

Application Configuration File

The application reads settings from the application configuration file.

The default file is market-data.exe.config. Please always keep it with the executable file, market-data.exe.

You may specify the file for the current operation using the config option. For example:

market-data download YahooFinanceQuotes symbols\uk-symbols.txt /config=configs\uk.config

This feature is used for loading different settings for different markets.

The application configuration files may contain the following settings:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="DefaultSymbolFileName" value="" />
    <add key="MarketOpen" value="09:30:00 -05:00" />
    <add key="MarketClose" value="16:00:00 -05:00" />
    <add key="MarketTimeZone" value="-05:00" />
    <add key="NextRequestDelay" value="500" />
    <add key="RetryCycleDelay" value="5000" />
    <add key="RetryCycleCount" value="3" />
    <add key="Timeout" value="30000" />
    <add key="Accept" value="" />
    <add key="UserAgent" value="" />
    <add key="CsvFolder" value="csv" />
    <add key="DownloadFolder" value="downloads" />
    <add key="ZipFolder" value="zip" />
    <add key="CacheFolder" value="cache" />
    <add key="LogFolder" value="logs" />
    <add key="InputCodePage" value="1252" />
    <add key="OutputCodePage" value="1252" />
    <add key="CsvSeparator" value=";" />
    <add key="CsvDateFormat" value="yyyy-MM-dd" />
    <add key="CsvTimeFormat" value="HH:mm:ss" />
    <add key="YahooFinanceHistoricalPricesStartDate" value="2013-12-31" />
    <add key="YahooFinanceQuotesFields" value="snxd1t1ohgl1c1p2kjt8va2m3m4s7ee7e8e9rr6r7r5qr1dyj1b4p6p5f6" />
  </appSettings>
</configuration>

Application Settings

DefaultSymbolFileName

Defines the default symbol file name. If specified, you may omit the file name in the download mode.

MarketOpen

Defines market opening time.
The value is used to calculate trade dates. It is very important.

MarketClose

Defines market closing time.
The value is used to calculate trade dates. It is very important.
Data between closing and opening times are loaded once.

MarketTimeZone

Defines market time zone.
The zone is used to calculate market time of the downloaded data.
It is very important.

NextRequestDelay

Defines delay between data requests in milliseconds.
The minimal value is 500 milliseconds that is used by default.
The unregistered version after the 15-day trial period has 30-second delay.

RetryCycleDelay

Defines delay between cycles of reloading data with errors.
This time should be enough to reconnect your computer to the Internet.

RetryCycleCount

Defines a number of cycles of loading data.
On the first cycle, the application loads data of the current symbol list.
If an error occurred, the symbol is added to a retry list for the next cycle.
The application tries to reload data the specified number of cycles.

Timeout

Defines the web request timeout.

Accept

Defines the web request Accept header.

UserAgent

Defines the web request UserAgent header.

CsvFolder

Defines the folder for parsed CSV data.
The default value is "csv".

DownloadDataFolder

Defines the folder for downloaded data (in HTML or CSV format).
The default value is "downloads".

ZipFolder

Defines the folder for archives of parsed data.
The default value is "zip".

CacheFolder

Defines the folder for cached data. See YahooFinanceHistoricalPrices.
The default value is "cache".

LogFolder

Defines the folder for log files.
The default value is "log".

InputCodePage

Defines a code page for symbol list files.
The default value is the default Windows code page.

OutputCodePage

Defines a code page for output CSV files.
The default value is the default Windows code page.

CsvSeparator

Defines CSV separator.
The default value is a semicolon.

CsvDateFormat

Defines the date format for output CSV files as specified at
https://msdn.microsoft.com/en-us/library/8kb3ddd4.aspx.
The default value is "yyyy-MM-dd".

CsvTimeFormat

Defines the time format for output CSV files as specified at
https://msdn.microsoft.com/en-us/library/8kb3ddd4.aspx.
The default value is "HH:mm:ss".

YahooFinanceHistoricalPricesStartDate

Defines the start date for the YahooFinanceHistoricalPrices and YahooFinanceDividends providers.
Specify the date in the yyyy-MM-dd format. The default value is 2013-12-31.

YahooFinanceQuotesFields

Defines fields loaded by the YahooFinanceQuotes provider.
See details in YahooFinanceQuotes Provider.

Preconfigured Project

Market Data Downloader contains a preconfigured project that allows:

• Loading key statistics, quotes, industries, historical prices, dividends, and options.
• Importing data into a preconfigured Microsoft SQL Server database.

The project allows loading data from exchanges in US, UK, Canada, Europe, Russia, and Australia.

You may use it as is or adapt it to your needs.

Quick Start

1. Copy the MarketData folder from the downloaded package to the root of drive C:
2. Run the C:\MarketData\2-load-csv-us.cmd batch file.
3. Open CSV files in the C:\MarketData\csv folder, for example, in Microsoft Excel.

You may run batch files for other markets like 2-load-csv-au.cmd, 2-load-csv-ca.cmd, etc.

You may edit symbol list files in the symbols folder like us-quotes.txt and run batch files again.
The application will load data for new symbols and reload data for other ones if the market is open.

Next Quick Steps

You may test the following steps if you have local Microsoft SQL Server Express installed.
Note that you may easily install it if you plan to load data into an SQL Server database. See Market Data Database.

1. Start SQL Server Management Studio and open the C:\MarketData\sql\mssql-create-database.sql file.
2. Press F5 to execute the script and to create the Market Data Database.
3. Run the C:\MarketData\0-run-all-us.cmd batch file.
4. Select data from any database table in the yahoo schema, for example, yahoo.KeyStatistics or yahoo.Industries.

That's all. You may add your tickers to symbol files and add batch files to Windows Scheduler to load data automatically.

Project Folders

Folder	Description
csv	Contains output CSV files to load into a database. The files are deleted after a successful import.
cache	Contains outputs of the YahooFinanceHistoricalPrices provider.
downloads	Contains archives of downloaded data. Archives contain data for a single provider and trade date. Keep these files to have the possibility to reparse the loaded data.
zip	Contains archives of output CSV data. You may use these files to import data into a database once again. If you successfully import data and make a backup of your database, you may delete these files.
logs	Contains log files. You may delete this files at any time.
symbols	Contains symbol list files used by batch files for download. Place your tickers into the files using any text editor, or export symbols from your database.
configs	Contains application configuration files for different markets.
formats	Contains format files used by SQL Server to import CSV files.
sql	Contains SQL codes used to import CSV files into SQL Server. Also, the folder contains SQL codes to create and remove Market Data Database.
help	Contains application help files.
languages	Contains application localization files.

Project Files

File	Description
configs\us.config, configs\au.config, etc.	Application configuration files used for different markets. You may edit this files to change settings.
formats\YahooQuotes.fmt formats\YahooOptions.fmt, etc.	The format files are used by SQL Server to import CSV files. You may edit these files if you need to change column names. Reflect such changes to SQL scripts in the SQL folder.
sql\mssql-import-YahooQuotes.sql sql\mssql-import-YahooOptions.sql, etc.	SQL scripts used to import CSV files into SQL Server. The files contain direct paths to CSV and format files like C:\MarketData\csv\YahooQuotes.csv C:\MarketData\formats\YahooQuotes.fmt Edit such paths if you place the MarketData folder in another location.
sql\mssql-create-database.sql sql\mssql-remove-database.sql	SQL scripts to create and remove Market Data Database.
symbols\us-industries.txt symbols\us-options.txt symbols\us-prices.txt symbols\us-quotes.txt, etc.	The files contain symbol list files used by batch files for download. You may place your tickers into the files using any text editor, or export symbols from your database. The files may contain different symbols for different modes. For example, you may load quotes for 2000 weekly monitored stocks and daily option data for 200 most actively traded stocks.
0-run-all-us.cmd 0-run-all-au.cmd, etc.	The batch files execute complete download and import operations. Use these files in Windows Schedule for automatic data download.
1-export-symbols.cmd	This is a dummy file where you may add commands to export tickers from your database to symbol files.
2-load-csv-us.cmd 2-load-csv-au.cmd, etc.	The batch files execute complete download operations. The files contain commands for loading key statistics, quotes, industries, historical prices, dividends, and options. You may edit these files to adapt them to your needs.
3-import-csv.cmd	This file executes complete import operations. It contains calls of the sqlcmd utility to execute SQL scripts from the SQL folder. Edit the sqlcmd command if you have another SQL Server instance or other database credentials.
clean-data.cmd	This file cleans the loaded data and related files. Use it if you want to reset the loaded data completely.
market-data.exe	The application executable file.
market-data.exe.config	The default application configuration file. This file is used if the /config option is not specified. You may change settings to your primary market.
RegisterProduct.exe	The product registration wizard.

Configuring Markets

Market Data Downloader allows loading data for any exchange supported by Yahoo Finance.

The main reason that requires configuring markets is different time zones and trading hours.

By default, the application is configured for US stock exchanges.

If you load data for exchanges in one country only, you may edit default settings.

If you load data for exchanges in several countries, you have two options:

1. Create complete folder copies for different exchanges.
2. Create configuration files for different exchanges.

The second way seems better as you have all the data in the same folder. It is described below.

Copy the application configuration file, market-data.exe.config, to a file for your market and change settings.

Change, at least, the following settings:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    ...
    <add key="MarketOpen" value="09:30:00 -05:00" />
    <add key="MarketClose" value="16:00:00 -05:00" />
    <add key="MarketTimeZone" value="-05:00" />
    <add key="DownloadFolder" value="downloads" />
    <add key="ZipFolder" value="zip" />
    ...
  </appSettings>
</configuration>

For example, you may set the following settings for London Stock Exchange (uk.config):

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    ...
    <add key="MarketOpen" value="08:00:00 +00:00" />
    <add key="MarketClose" value="16:30:00 +00:00" />
    <add key="MarketTimeZone" value="+00:00" />
    <add key="DownloadFolder" value="downloads\uk" />
    <add key="ZipFolder" value="zip\uk" />
    ...
  </appSettings>
</configuration>

Tips: you may find preconfigured settings for several world exchanges in the preconfigured project.

After that, you may run the application using the created configuration file. For example:

market-data download YahooFinanceQuotes symbols\uk-symbols.txt /config=configs\uk.config

market-data reparse  YahooFinanceQuotes /config=configs\uk.config

 

Automating Data Loading

You may automatically load data using Windows Schedule.

First of all, create a batch file that includes all the required commands.
You may find ready-to-use batch files in the preconfigured project.

Then use the following commands to create a scheduled task.

1. Open Control Panel, Administrative Tools, Task Scheduler.

2. Select Task Scheduler Library, right click and open New Folder... Create a new folder, for example, MarketData.

3. Select the created folder, right click and open Create a Basic Task.

4. Type a task name, for example, Load US Market Data. Click Next.

5. Select Weekly and click Next.

6. Specify the start time and required days. Click Next.

7. Select Start a program. Click Next.

8. Click Browse... and select the batch file to run, copy the batch folder and paste it to the Start in field. Click Next.

9. Click Finish.

That's all. You may click on the created task, and check or edit task parameters.

Importing CSV to Databases

Every database platform has specific tools for importing CSV files into databases.

So you may use such tools if you know them.

Market Data Downloader includes the preconfigured project with ready-to-use format files and SQL codes applicable for Microsoft SQL Server.

You may modify these files to adapt them to your databases.

If you need help to import data to databases on another database server, please, contact us.
We can quickly prepare solutions for other database platforms.

Importing CSV to Excel

You may import downloaded CSV data into Microsoft Excel spreadsheets using native Excel features:

• Just open a CSV file,
• or use Data, Get External Data, From Text to import the data.

In the last case, you may refresh the data in the workbook.

 
You may also take a look at the RealTimeToExcel product.
It is specially designed for loading and refreshing market data in Microsoft Excel.

Yahoo Finance Data Providers

Market Data Downloader allows loading data from the Yahoo Finance website and web services.

You may load data for stocks, ETFs, indices, and options.

Data Providers

• Yahoo Finance Quotes
• Yahoo Finance Key Statistics
• Yahoo Finance Industries
• Yahoo Finance Historical Prices
• Yahoo Finance Dividends
• Yahoo Finance Options

Complying with Yahoo! Terms of Services

Yahoo Finance publishes delayed quotes and financial data free of charge for non-commercial use.

See important documents about Yahoo Finance data use:

• Yahoo! Terms of Service at http://info.yahoo.com/legal/us/yahoo/utos/terms/
• Yahoo! APIs Terms of Use at http://info.yahoo.com/legal/us/yahoo/api/api-2140.html

Yahoo Finance Quotes

Usage

Examples:

market-data download YahooFinanceQuotes symbols.txt

market-data download YahooFinanceQuotes symbols\uk-symbols.txt /config=configs\uk.config

market-data reparse  YahooFinanceQuotes 20160224

Example of symbols.txt:

^GSPC
^VIX
SPY
YHOO

More ticker examples: BHP, BLT.L, BHP.AX, ^FCHI, BNP.PA, ^GDAXI, EOAN.DE, ABX.TO.

Use http://finance.yahoo.com/ to find tickers. See also a full list of Yahoo Finance exchanges.

Data Source

Yahoo Finance Website.

Examples:

http://quote.yahoo.com/d/quotes.csv?s=YHOO&f=snxd1t1ohgl1c1p2kjt8va2m3m4s7ee7e8e9rr6r7r5qr1dyj1b4p6p5f6

http://quote.yahoo.com/d/quotes.csv?s=^VIX&f=snxd1t1ohgl1c1p2kjt8va2m3m4s7ee7e8e9rr6r7r5qr1dyj1b4p6p5f6

Output Files

File	Description
csv\YahooQuotes.csv	Parsed data in CSV format. You may delete this file after loading data into databases.
zip\YahooQuotes.zip of YahooQuotes-yyyymmdd.csv	The complete archive of parsed data by trade dates. Keep this file to reload data into databases at any time.
zip\YahooOptions-yyyymmdd.zip of <symbol>.csv	Archives by trade dates of parsed data by symbols. It is for archive purposes only. You may delete these files.
downloads\YahooQuotes-yyyymmdd.zip of <symbol>.csv	Archives by trade dates of downloaded data by symbols. You may delete these files if the data have been parsed correctly. Otherwise, keep these files to reparse them with a new version of Market Data Downloader.

CSV Fields

You may configure loaded data using the YahooFinanceQuotesFields setting in the application configuration file.

Use codes from the Code column to specify the output columns.

Yahoo Finance Key Statistics

Usage

Examples:

market-data download YahooFinanceKeyStatistics symbols.txt

market-data download YahooFinanceKeyStatistics symbols\uk-symbols.txt /config=configs\uk.config

market-data reparse  YahooFinanceKeyStatistics 20160224

Example of symbols.txt:

AAPL
YHOO

More ticker examples: BHP, BLT.L, BHP.AX, BNP.PA, EOAN.DE, ABX.TO.

Yahoo Finance does not publish Key Statistics pages for indices and ETFs. So use only stock tickers.

Use http://finance.yahoo.com/ to find tickers. See also a full list of Yahoo Finance exchanges.

Data Source

Yahoo Finance Website.

Examples:

https://finance.yahoo.com/q/ks?s=AAPL+Key+Statistics

https://finance.yahoo.com/q/ks?s=YHOO+Key+Statistics

Output Files

File	Description
csv\YahooKeyStatistics.csv	Parsed data in CSV format. You may delete this file after loading data into databases.
zip\YahooKeyStatistics.zip of YahooKeyStatistics-yyyymmdd.csv	The complete archive of parsed data by trade dates. Keep this file to reload data into databases at any time.
zip\YahooOptions-yyyymmdd.zip of <symbol>.csv	Archives by trade dates of parsed data by symbols. It is for archive purposes only. You may delete these files.
downloads\YahooKeyStatistics-yyyymmdd.zip of <symbol>.htm	Archives by trade dates of downloaded data by symbols. You may delete these files if the data have been parsed correctly. Otherwise, keep these files to reparse them with a new version of Market Data Downloader.

CSV Fields

See field descriptions at https://help.yahoo.com/kb/finance/SLN2347.html.

$ fields contain values of the underlying fields converted to numbers.

Yahoo Finance Industries

Usage

Examples:

market-data download YahooFinanceIndustries symbols.txt

market-data download YahooFinanceIndustries symbols\uk-symbols.txt /config=configs\uk.config

market-data reparse  YahooFinanceIndustries 20160224

Example of symbols.txt:

AAPL
YHOO

More ticker examples: BHP, BLT.L, BHP.AX, BNP.PA, EOAN.DE, ABX.TO.

Use http://finance.yahoo.com/ to find tickers. See also a full list of Yahoo Finance exchanges.

Data Source

Yahoo Finance Website.

Examples:

http://finance.yahoo.com/q/in?s=YHOO

http://finance.yahoo.com/q/in?s=YHOO

Output Files

File	Description
csv\YahooIndustries.csv	Parsed data in CSV format. You may delete this file after loading data into databases.
zip\YahooIndustries.zip of YahooIndustries-yyyymmdd.csv	The complete archive of parsed data by trade dates. Keep this file to reload data into databases at any time.
zip\YahooOptions-yyyymmdd.zip of <symbol>.csv	Archives by trade dates of parsed data by symbols. It is for archive purposes only. You may delete these files.
downloads\YahooIndustries-yyyymmdd.zip of <symbol>.htm	Archives by trade dates of downloaded data by symbols. You may delete these files if the data have been parsed correctly. Otherwise, keep these files to reparse them with a new version of Market Data Downloader.

CSV Fields

Symbol

Sector

Industry

Yahoo Finance Historical Prices

Usage

Examples:

market-data download YahooFinanceHistoricalPrices symbols.txt

market-data download YahooFinanceHistoricalPrices symbols\uk-symbols.txt /config=configs\uk.config

market-data reparse  YahooFinanceHistoricalPrices 20160224

Example of symbols.txt:

^GSPC
^VIX
SPY
YHOO

More ticker examples: BHP, BLT.L, BHP.AX, ^FCHI, BNP.PA, ^GDAXI, EOAN.DE, ABX.TO.

Use http://finance.yahoo.com/ to find tickers. See also a complete list of Yahoo Finance exchanges.

Important Notes about Dates and Dividends

By default, the start date of the period for the first load is December 31, 2013.

You may configure this start date using the YahooFinanceHistoricalPricesStartDate setting in the application configuration file.

The provider saves data for the last trade dates in the cache\YahooHistoricalPricesLastData.csv file, and uses the latest dates as the start dates for next downloads.

If saved and loaded values of AdjClose are different, the provider reloads data for the entire period and adds the symbol to the cache\YahooHistoricalPricesDividendsOrSplits.txt file.

Accordingly, you may use the cache\YahooHistoricalPricesDividendsOrSplits.txt file as a symbol file for the YahooFinanceDividends mode.

Data Source

Yahoo Finance Website.

Examples:

http://real-chart.finance.yahoo.com/table.csv?s=YHOO&g=d&ignore=.csv

http://real-chart.finance.yahoo.com/table.csv?s=^VIX&g=d&ignore=.csv

Output Files

File	Description
csv\YahooHistoricalPrices.csv	Parsed data in CSV format. You may delete this file after loading data into databases.
zip\YahooHistoricalPrices.zip of YahooHistoricalPrices-yyyymmdd.csv	The complete archive of parsed data by trade dates. Keep this file to reload data into databases at any time.
zip\YahooOptions-yyyymmdd.zip of <symbol>.csv	Archives by trade dates of parsed data by symbols. It is for archive purposes only. You may delete these files.
downloads\YahooHistoricalPrices-yyyymmdd.zip of <symbol>.csv	Archives by trade dates of downloaded data by symbols. You may delete these files if the data have been parsed correctly. Otherwise, keep these files to reparse them with a new version of Market Data Downloader.
cache\YahooHistoricalPricesLastData.csv	The file contains latest trade data records. See comments above. You may delete the file to reload data for entire periods.
cache\YahooHistoricalPricesDividendsOrSplits.txt	The file contains tickers that have different Close and AdjClose values. You may use this file as a symbol file for the YahooFinanceDividends mode. Delete this file after loading dividends.

CSV Fields

The RunningSplitK, OpenK, HighK, LowK and CloseK fields are calculated internally using Close and AdjClose values to have comparable data.

Please check the calculation results before use.

Source Fields	Calculated Fields
Symbol
Date	RunningSplitK
Open	OpenK
High	HighK
Low	LowK
Close	CloseK
AdjClose
Volume

Yahoo Finance Dividends

Usage

Examples:

market-data download YahooFinanceDividends symbols.txt

market-data download YahooFinanceDividends cache\YahooHistoricalPricesDividendsOrSplits.txt

market-data reparse  YahooFinanceHistoricalPrices 20160224

Example of symbols.txt:

YHOO
SPY

More ticker examples: BHP, BLT.L, BHP.AX, BNP.PA, EOAN.DE, ABX.TO.

Use http://finance.yahoo.com/ to find tickers. See also a complete list of Yahoo Finance exchanges.

Important Notes about Dates and Dividends

By default, the start date of the period is December 31, 2013, the same as for YahooFinanceHistoricalPrices.

You may configure this start date using the YahooFinanceHistoricalPricesStartDate setting in the application configuration file.

You may use the cache\YahooHistoricalPricesDividendsOrSplits.txt file as a symbol file.

This file is prepared by the YahooFinanceHistoricalPrices provider for tickers that have different saved and loaded values of AdjClose.

Delete this file after successful loading of dividends.

Data Source

Yahoo Finance Website.

Examples:

http://real-chart.finance.yahoo.com/table.csv?s=YHOO&g=v&ignore=.csv

http://real-chart.finance.yahoo.com/table.csv?s=^VIX&g=v&ignore=.csv

Output Files

File	Description
csv\YahooDividends.csv	Parsed data in CSV format. You may delete this file after loading data into databases.
zip\YahooDividends.zip of YahooDividends-yyyymmdd.csv	The complete archive of parsed data by trade dates. Keep this file to reload data into databases at any time.
zip\YahooOptions-yyyymmdd.zip of <symbol>.csv	Archives by trade dates of parsed data by symbols. It is for archive purposes only. You may delete these files.
downloads\YahooDividends-yyyymmdd.zip of <symbol>.csv	Archives by trade dates of downloaded data by symbols. You may delete these files if the data have been parsed correctly. Otherwise, keep these files to reparse them with a new version of Market Data Downloader.

CSV Fields

Symbol

Date

Dividends

Yahoo Finance Options

Usage

Examples:

market-data download YahooFinanceOptions symbols.txt

market-data reparse  YahooFinanceOptions 20160224

Example of symbols.txt:

YHOO
^VIX

Option data are available only for US exchanges. Use http://finance.yahoo.com/ to find tickers.

Data Source

Yahoo Finance Website.

Examples:

http://finance.yahoo.com/q/op?s=YHOO+Options

http://finance.yahoo.com/q/op?s=^VIX+Options

Output Files

File	Description
csv\YahooOptions.csv	Parsed data in CSV format. You may delete this file after loading data into databases.
zip\YahooOptions.zip of YahooOptions-yyyymmdd.csv	The complete archive of parsed data by trade dates. Keep this file to reload data into databases at any time.
zip\YahooOptions-yyyymmdd.zip of <symbol>-yyyymmdd.csv	Archives by trade dates of parsed data by symbols and expiration dates. It is for archive purposes only. You may delete these files.
downloads\YahooOptions-yyyymmdd.zip of <symbol>-yyyymmdd.htm	Archives by trade dates of downloaded data by symbols and expiration dates. You may delete these files if the data have been parsed correctly. Otherwise, keep these files to reparse them with a new version of Market Data Downloader.

CSV Fields

Symbol

ExpDate

Strike

Type

Contract Name

Last

Bid

Ask

Change

%Change

Volume

Open Interest

Implied Volatility

Market Data Database

The download package contains the preconfigured project to download market data into CSV files.

The package also contains the SQL code, mssql-create-database.sql, to create a Microsoft SQL Server database to import loaded CSV data.

See Data Tables for Yahoo Finance.

Installing Microsoft SQL Server

You may use existing instances of Microsoft SQL Server 2008 or higher.

To use Microsoft SQL Server 2005, change the date and time(0) data types to datetime in the installation script.

You may download, install, and use Microsoft SQL Server Express for free.

It has a limit of a database size in 10 gigabytes that is quite enough.

Use the following link to download Microsoft SQL Server 2014 Express:

https://www.microsoft.com/en-US/download/details.aspx?id=42299

Select and download Express with Tools.

The package contains everything needed to install and configure SQL Server as a database server including the full version of SQL Server 2014 Management Studio.

Installing Market Data Database

Start SQL Server Management Studio and open the MarketData\sql\mssql-create-database.sql file.

Press F5 to execute the script.

Removing Market Data Database

Start SQL Server Management Studio and open the MarketData\sql\mssql-remove-database.sql file.

Press F5 to execute the script.

Data Tables for Yahoo Finance

The download package contains the preconfigured project to download market data.

It includes the SQL code, mssql-create-database.sql, to create an MS SQL database to import loaded CSV data.

You may edit the file to change the table structures described below.

All the tables are created in the schema named 'yahoo'.

Tips: You may leave the default tables as is and load data using the preconfigured project, and then import data from the default tables to your tables.

Tables

yahoo.Dividends

Data provider: Yahoo Finance Dividends

yahoo.HistoricalPrices

Data provider: Yahoo Finance Historical Prices

yahoo.Industries

Data provider: Yahoo Finance Industries

yahoo.KeyStatistics

Data provider: Yahoo Finance Key Statistics

yahoo.Quotes

Data provider: Yahoo Finance Quotes

yahoo.OptionTypes

The table contains two values: C - CALL and P - PUT.

yahoo.OptionContracts

The table contains data about option contracts extracted from option chain data.

Data provider: Yahoo Finance Options

yahoo.Options

Data provider: Yahoo Finance Options

Frequently Asked Questions

What is the purpose of the files in "downloads" folder?

The files can be used for reparsing data if the source page format has been changed.
Usually, you just download a new version of Market Data Downloader and reparse the data.

Can I delete archive files in the downloads folder?

Yes. However, be sure that all the files are parsed to CSV correctly.

Do you release new versions to parse changed web pages correctly?

We make everything to release new versions in this case as soon as possible.

Do you guarantee that the product will work in the future?

No. Data providers may change data formats or prevent using products like Market Data Downloader.
We hope that you will use our product for a long time, and we do everything to address new challenges.

Do I pay monthly or yearly fee?

No.

Where you get the data?

Market Data Downloader loads data from Yahoo Finance web pages and web services.
We do not provide any data.

Is it legal to load data using your software?

Please read Yahoo Terms of Services and answer this question yourself.
Our software communicates with Yahoo websites and web services with required delays.
Do not launch several copies of the product at the same time. Otherwise, you may overhead data provider services and your IP will be banned.

Can you add a data provider for a web page on my request?

Please contact us. Technically, the product can load and parse any web page.

Product Registration