WEB.CSV

WEB.CSV

Getting Started

WEB.CSV allows loading and converting HTML, XML, JSON, CSV, and plain text data to CSV.

It includes universal parsers that you may tune to get the required data in the required format.

The commands are quite simple. For example:

webcsv.exe WebToCSV http://www.nasdaq.com/symbol/aapl/dividend-history aapl.csv

You may separate download and convert commands like this:

webcsv.exe WebToText http://www.nasdaq.com/symbol/aapl/dividend-history aapl.htm

webcsv.exe TextToCSV aapl.htm aapl.csv

Moreover, you may export task files from a database and use the simple command in batch files:

webcsv.exe TextToCSV @tickers.txt

You may add columns to CSV using special functions like this:

webcsv.exe WebToCSV http://finance.yahoo.com/q/hp?s=AAPL aapl.csv /add=Symbol=AAPL;LoadDate=Date()

You may download data from any website including Yahoo Finance, Google Finance, and MSN Money.

See examples in the Examples folder of the downloaded package.

In short, you may save a lot of time using a universal parser utility.

You may start learning about WEB.CSV on the following topics:

Change History

Version 4.8, December 15, 2017

gwebcmd has a new name, WEB.CSV, a new executable, webcsv.exe, and a new website, www.stock-data-solutions.com.

Now investors and traders may get a complete line of data solutions in one place.

Version 4.7, November 3, 2017

Improvements:

  • Added JSONP parser
  • Added PartialHTML parser (HTML inside JSON and JSONP)
  • Improved HTML parser

Bug fixes:

  • Errors with HTTPS requests that support tls12 only
  • Parser errors for quotes in JSON
  • Registration wizard WebException in the offline mode

Version 4.6, August 22, 2017

Breaking changes:

  • Changed .NET platform from .NET Framework 4.0 Client Profile to .NET Framework 4.5.2.
  • Changes in requirements for Windows 7, Windows Server 2008, and Windows Server 2008 R2.
  • Removed support of Windows XP.

Bug Fixes:

  • Issues with registering products.
  • Issues with decoding deflate web data.

Version 4.5, August 1, 2017

New features:

  • /NoBOM option to suppress adding BOM to output

Improvements:

  • Improved JSON parser performance

Version 4.4, July 5, 2017

New features:

  • Register mode

Improvements:

  • Showing columns of historical prices as OHLCV by default
  • Updated registration wizard
  • Updated installer

Bug Fixes:

  • Converting IP-addresses to numbers

Version 4.3, May 31, 2017

gwebcmd has an alternative name, WEB.CSV, used at www.stock-data-solutions.com to be in line with other products: Market.csv, Market.rtd, WEB.RTD, DB.RTD, and RTD.DB.

New features:

  • New functions of the /add option: FileNameDateTime(), FileNameDate(), and FileNameTime().
  • New examples of loading stock quotes, historical prices, intraday prices, option chains, and futures from Yahoo Finance, Google Finance, and MSN Money.
  • Loading multiple option chain pages from Yahoo Finance.
  • Special processing of datetime values of historical prices from MSM Money.
  • Special processing of datetime values of option chains from Yahoo Finance.
  • New /AsIs option to output values without any special processing.
  • New /RowValues option to print all values in rows for XML and JSON documents.

Improvements:

  • HTML parser supports multiple tables in the /RootPath option.
  • Improved JSON parser.

Bug Fixes:

  • Skipped empty values in the columns added using the /add option.
  • Ignored specified date and time formats for data with multiple date and time columns.

Version 4.2, May 11, 2017

Licensing changes

  • A simplified model of the free version limits.
    The free version has a single limit of ten output rows.
    You may remove this limit purchasing gwebcmd subscription starting $25 per year per PC.
    See Edition Comparison.
  • Use of gwebcmd requires registration.
    You may activate the trial version and register the free edition after the trial.
  • gwebcmd Personal has the same features as gwebcmd Enterprise except for the permission of the commercial use.
    See Edition Comparison.

Removed non-core and obsolete features:

  • Downloading data from social networks.
  • Downloading data from protected pages.
  • Downloading data with Internet Explorer.
  • Cleaning HTML tags.
  • Parsing Google web search results.
  • Portable setup package.

Improvements:

  • Rewritten and redesigned documentation.
  • Updated examples.
  • Updated installer.

The new version of gwebcmd is a brilliant tool for core features: downloading and extracting data from financial websites like Yahoo Finance, Google Finance, and MSN Money.

It has a clear structure, powerful core features, and a well-designed documentation.

Version 4.1, April 5, 2017

New features:

  • New parameters to control date and time CSV output formats: /dateFormat=<format> and /timeFormat=<format>.

Improvements:

  • New default formats for dates, times, and date and times: yyyy-MM-dd, HH:mm:ss, and yyyy-MM-dd HH:mm:ss.
    This change allows opening CSV files with default formatting in Excel with any regional settings.
    You may use /dateTimeFormat, /dateFormat and /timeFormat to set the desired format.
  • Improved parsing JSON data.
    The parser supports * as a property name in the rootPath parameters.
    For example, try to load futures data from the Yahoo Finance website using the URL:
    https://query1.finance.yahoo.com/v10/finance/quoteSummary/CK18.CBT?modules=futuresChain;rootpath=quoteSummary.result.futuresChain.futuresChainDetails.*
  • HTML parser converts string values with % to numbers and "n.a." values to empty values.

Bug fixes:

  • Lost of authorization data of HTTP requests.

Version 4.0, January 17, 2017

New features:

  • gwebcmd is available as a subscription.
  • You may upgrade previous versions to gwebcmd 4.0 for free.

Edition Comparison

FeatureFreePersonalEnterprise
Loading web data via HTTP and HTTPs, GET and POST
Loading multiple pages
Converting XML, JSON, CSV, HTML, and plain text10 rows
Task files
Commercial usex

Install and Uninstall

Installing

To install WEB.CSV, unzip the downloaded WEB.CSV package and run setup.exe.

Log off and log on to activate a new PATH environment variable.

Register the application to start a trial period.

Now, you may run the command prompt and type: webcsv.exe

Updating

To update WEB.CSV, unzip the downloaded WEB.CSV package and run setup.exe.

Uninstalling

To uninstall WEB.CSV, open Control Panel, Programs and Features, then select WEB.CSV and click the Uninstall button.

System Requirements

Supported Architectures:

  • x86, x64

Supported Operating Systems:

  • Windows Vista SP2, 7 SP1, 8, 8.1, 10
  • Windows Server 2008 SP2, 2008 R2 SP1, 2012, 2012 R2, 2016

End-User License Agreement

This End-User License Agreement (EULA) is a legal agreement between you (either an individual or a single entity) and Gartle Technology Corporation for any WEB.CSV software, use examples and documentation (Software) that accompany this EULA.

YOU AGREE TO BE BOUND BY THE TERMS OF THIS EULA BY INSTALLING, COPYING, OR OTHERWISE USING THE SOFTWARE.

IF YOU DO NOT AGREE, DO NOT INSTALL, COPY, OR USE THE SOFTWARE.

Your licensing of Software is in accordance with the terms of the EULA in effect at the time of such licensing. By licensing Software, you accept and agree to the EULA in effect at such time.

  1. GRANT OF LICENSE: Gartle Technology Corporation grants you the following rights provided that you comply with all terms and conditions of this EULA:
    1. Trial Period: You may install, use, access, display and run the Software for purposes of evaluation of the Software for a period of fifteen (15) days from the date of installation of the Software to determine if the Software meets your needs. Upon completion of your evaluation or at the end of the 15-day trial period, whichever occurs first, you must do one of four things: either
      1) register the WEB.CSV Free edition in order to continue use of Software;
      2) purchase license for the WEB.CSV Personal or Enterprise edition in order to continue use of Software;
      3) purchase a subscription for the WEB.CSV Personal or Enterprise edition in order to continue use of Software;
      or 4) discontinue use of the Software, and uninstall the Software from any and all machines.
    2. Installation and Use: You may install, use, access, display and run the Software free of charge on a non-exclusive basis and without right of sublicense.
      The free version has several limits. You may purchase and register WEB.CSV Personal or Enterprise edition to remove limits. The commercial use of the WEB.CSV Personal edition is not permitted.
    3. Software Transfer: You may transfer the Software to a different internal workstation or user so long as you have purchased a License for each such workstation or user. You may not, however, transfer the Software to a Third Party.
    4. Use of Examples: You may install, access, modify and use Software examples for your private or company-internal purposes.
  2. LIMITATIONS: You may not use, copy, modify, display, rent, lease, loan, transfer, distribute, download, merge, or make any translation or derivative work of the Software, except as expressly provided herein. You may not reverse engineer, decompile, or disassemble the Software, except and only to the extent that such activity is expressly permitted by applicable law notwithstanding this limitation.
  3. INTELLECTUAL PROPERTY RIGHTS AND CONFIDENTIALITY: The Software, including methods, processes and/or techniques utilized therein, is owned by, proprietary to and valuable trade secrets of Gartle Technology Corporation and is protected by Russian Federation copyright law and international treaties. You agree to take no actions that impair or infringe Gartle Technology Corporation's intellectual property rights in the Software. You agree not to remove, efface or obscure any copyright notices, other proprietary markings or confidentiality legends placed upon or contained within the Software.
  4. DISCLAIMER OF WARRANTIES: Gartle Technology Corporation disclaims all warranties concerning the Software and Services (if any), express, implied, or statutory, including without limitation, any warranties, duties or conditions of merchantability or fitness for a particular purpose, warranties of reliability or availability, of accuracy or completeness of responses, of results, of workmanlike effort, of lack of viruses, and of lack of negligence, all with regard to the Software, and the provision of or failure to provide support or other services, information, software, and related content through the Software or otherwise arising out of the use of the Software. Gartle Technology Corporation does not warrant that the Software will operate in combination with other software products selected by you, or that the Software will operate uninterrupted or error-free. Additionally, Gartle Technology Corporation and its suppliers provide the Software and Services AS IS AND WITH ALL FAULTS. THERE IS NO WARRANTY OR CONDITION OF TITLE, QUIET ENJOYMENT, QUIET POSSESSION, CORRESPONDENCE TO DESCRIPTION, OR NONINFRINGEMENT WITH REGARD TO THE SOFTWARE.
  5. NO LIABILITY: TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT SHALL GARTLE TECHNOLOGY CORPORATION OR ITS SUPPLIERS BE LIABLE FOR ANY SPECIAL, INCIDENTAL, PUNITIVE, INDIRECT, OR CONSEQUENTIAL DAMAGES WHATSOEVER (INCLUDING, BUT NOT LIMITED TO, DAMAGES FOR LOSS OF PROFITS OR CONFIDENTIAL OR OTHER INFORMATION, FOR LOSS OF DATA, FOR BUSINESS INTERRUPTION, FOR PERSONAL INJURY, FOR LOSS OF PRIVACY, FOR FAILURE TO MEET ANY DUTY INCLUDING OF GOOD FAITH OR OF REASONABLE CARE, FOR NEGLIGENCE, AND FOR ANY OTHER PECUNIARY OR OTHER LOSS WHATSOEVER) ARISING OUT OF OR IN ANY WAY RELATED TO THE USE OF OR INABILITY TO USE THE SOFTWARE, THE PROVISION OF OR FAILURE TO PROVIDE SUPPORT OR OTHER SERVICES, INFORMATION, SOFTWARE, AND RELATED CONTENT THROUGH THE SOFTWARE OR OTHERWISE ARISING OUT OF THE USE OF THE SOFTWARE, OR OTHERWISE UNDER OR IN CONNECTION WITH ANY PROVISION OF THIS EULA, EVEN IN THE EVENT OF THE FAULT, TORT (INCLUDING NEGLIGENCE), MISREPRESENTATION, STRICT LIABILITY, BREACH OF CONTRACT OF GARTLE TECHNOLOGY CORPORATION OR ANY SUPPLIER, AND EVEN IF GARTLE TECHNOLOGY CORPORATION OR ANY SUPPLIER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
  6. LIMITATION ON REMEDIES: Within the first thirty (30) days after your receipt of the Software, should you encounter and report to Gartle Technology Corporation within such time period a reproducible error that causes the Software not to perform in all material respects as set forth in the Software documentation, then Gartle Technology Corporation will, at its sole discretion, either: a) resolve the error or malfunction, and modify or replace the Software (if deemed necessary by Gartle Technology Corporation); or b) allow you to terminate this EULA with respect to the non-conforming Software and, upon your return of the Software to Gartle Technology Corporation, Gartle Technology Corporation shall provide you with the lesser of a) the actual damages incurred by you; or b) the amount you paid for the non-conforming Software. The remedies described in this section shall be your sole and exclusive remedies under this EULA.
  7. GENERAL PROVISIONS
    1. Reservation of Rights and Ownership: Gartle Technology Corporation reserves all rights not expressly granted to you in this EULA. The Software is licensed, not sold.
    2. Consent to Use of Data: You agree that Gartle Technology Corporation and its affiliates may collect and use technical information gathered as part of the Software support services provided to you, if any, related to the Software. Gartle Technology Corporation may use this information solely to improve Gartle Technology Corporation products or to provide customized services or technologies to you and will not disclose this information in a form that personally identifies you.
    3. Links to Third Party Sites: We may link to third-party sites through the use of the Software. The third-party sites are not under the control of Gartle Technology Corporation, and Gartle Technology Corporation is not responsible for the contents of any third-party sites, any links contained in third-party sites, or any changes or updates to third-party sites. Gartle Technology Corporation is not responsible for webcasting or any other form of transmission received from any third-party sites. Gartle Technology Corporation is providing these links to third-party sites to you only as a convenience, and the inclusion of any link does not imply an endorsement by Gartle Technology Corporation of the third-party site.
    4. Additional Software/Services: This EULA applies to updates, supplements, add-on components, or Internet-based services components, of the Software that Gartle Technology Corporation may provide to you or make available to you after the date you obtain your initial copy of the Software, unless Gartle Technology Corporation provides other terms along with the update, supplement, add-on component, or Internet-based services component. Gartle Technology Corporation reserves the right to discontinue any Internet-based services provided to you or made available to you through the use of the Software.
    5. Upgrades: To use Software identified as an upgrade, you must first be licensed for the software identified by Gartle Technology Corporation as eligible for the upgrade. After upgrading, you may no longer use the software that formed the basis for your upgrade eligibility.
    6. Applicable Law: This EULA is governed by the laws of the Russian Federation. Any legal action or proceeding relating to this EULA shall be instituted in a court of arbitration in the Moscow City, Russian Federation. Gartle Technology Corporation and you agree to submit to the jurisdiction of, and agree that venue is proper in, these courts in any such action or proceeding. The prevailing party in any action to enforce this EULA will be entitled to recover its attorney fees and costs in connection with such action.
    7. Waiver: The failure of either party to enforce any of the terms of this EULA shall not be construed as a waiver of future enforcement of that or any other term.
    8. Entire Agreement and Severability: This EULA (including any addendum or amendment to this EULA which is included with the Software) is the entire agreement between you and Gartle Technology Corporation relating to the Software and the support services (if any) and it supersedes all prior or contemporaneous oral or written communications, proposals and representations with respect to the Software or any other subject matter covered by this EULA. To the extent the terms of any Gartle Technology Corporation policies or programs for support services conflict with the terms of this EULA, the terms of this EULA shall control. If any provision of this EULA is held to be void, invalid, unenforceable or illegal, the other provisions shall continue in full force and effect.
    9. Termination: Without prejudice to any other rights, Gartle Technology Corporation may terminate this EULA if you fail to comply with the terms and conditions of this EULA. In such event, you must destroy all copies of the Software and all of its component parts and you will not be entitled to any refund of monies.

Trial Version Limitations

You can use WEB.CSV Enterprise during a 15-day trial period for free.

You have to activate the trial.

When the trial period has expired, you have the following options:

  1. To register the WEB.CSV Free edition for free.
  2. To purchase a license and register the WEB.CSV Personal or Enterprise edition.
  3. To purchase a subscription and register the WEB.CSV Personal or Enterprise edition.
  4. To stop the use and remove WEB.CSV from a computer.

See also

Using WEB.CSV

Basics

WEB.CSV solves two tasks:

  1. Downloading data
  2. Converting data to CSV

You may use a single WebToCSV command or two commands: WebToText and TextToCSV.

For example:

webcsv.exe WebToCSV http://www.nasdaq.com/symbol/aapl/dividend-history aapl.csv

webcsv.exe WebToText http://www.nasdaq.com/symbol/aapl/dividend-history aapl.htm

webcsv.exe TextToCSV aapl.htm aapl.csv

WEB.CSV allows converting HTML, XML, JSON, CSV, and plain text files.

Below we discuss specific topics including tuning parsers.

You may find examples of various cases in the downloaded package.

Parsing HTML

WEB.CSV allows extracting data from HTML tables.

You may tune the parser using the /rootPath option with a table number.

For example, you may download dividends from nasdaq.com like

webcsv.exe WebToText http://www.nasdaq.com/symbol/aapl/dividend-history aapl.htm

Then you may get available tables using the command:

webcsv.exe HtmlTables aapl.htm tables.htm

Open tables.htm and find a number of the desired table. Then use the command like this:

webcsv.exe TextToCSV aapl.htm aapl.csv /rootPath=6

or

webcsv.exe WebToCSV http://www.nasdaq.com/symbol/aapl/dividend-history aapl.csv /rootPath=6

Parsing XML

WEB.CSV allows extracting regular data from XML.

For example, we have the following file, test.xml:

<?xml version="1.0" encoding="utf-8"?>
<root>
    <parent>
        <row id="1"><f1>data11</f1><f2>data12</f2></row>
        <row id="2"><f1>data21</f1><f2>data22</f2></row>
        <row id="3"><f1>data31</f1><f2>data32</f2></row>
    </parent>
</root>

If we run the basic command

webcsv.exe TextToCSV test.xml test.csv 

we get the following result in test.csv:

id;"f1";"f2"
1;"data11";"data12"
2;"data21";"data22"
3;"data31";"data32"

WEB.CSV selects the "best" root of the data. This command is equivalent to

webcsv.exe TextToCSV test.xml test.csv /rootPath=root.parent.row

You may change the root path to get data from the specified node. For example:

webcsv.exe TextToCSV test.xml test.csv /rootPath=root.parent

returns the following:

row.id;"row.f1";"row.f2"
1;"data11";"data12"
2;"data21";"data22"
3;"data31";"data32"

You may skip required nodes. For example:

webcsv.exe TextToCSV test.xml test.csv /rootPath=root.parent /skippedNodes=row.f2

returns:

row.id;"row.f1"
1;"data11"
2;"data21"
3;"data31"

Parsing JSON

Parsing JSON is similar to the parsing XML as the both document formats have hierarchical structures.

For example, a file has the content:

{"root":
  {"parent":[
    {"id":1,"f1":"data11","f2":"data12"},
    {"id":2,"f1":"data21","f2":"data22"},
    {"id":3,"f1":"data31","f2":"data32"}
  ]
}

Run the command:

webcsv.exe TextToCSV test.json test.csv

The command returns:

id;"f1";"f2"
1;"data11";"data12"
2;"data21";"data22"
3;"data31";"data32"

Parsing Plain Text

WEB.CSV allows getting data from plain text files if the text contains "visible" columns like these:

id  f1      f2
1   data11  data12
2   data21  data22
3   data31  data32

Just use the standard command:

webcsv.exe TextToCSV test.txt test.csv

Parsing CSV

You may use WEB.CSV to convert CSV files including separators and encoding, and adding additional calculated columns like this:

webcsv.exe TextToCSV test.csv test-65001.csv /outputEncoding=65001 /add=Symbol=AAPL;Date=Date()

See available options here.

Customizing Output CSV

WEB.CSV has useful options to customize the output.

The most useful options are:

/datetimeformat=<format>

/separator=<separator>|Tab

/add=<header=value>[<separator>...]

For example:

webcsv.exe WebToCSV http://finance.yahoo.com/q/hp?s=AAPL aapl.csv /datetimeformat=yyyy-MM-dd /separator=; /add=Symb=AAPL

Batch Files

Here is a simple batch file to get the data for the list of tickers from the tickers.txt file:

@echo off

@for /F %%i in (tickers.txt) do (
    echo %%i
    webcsv.exe WebToCSV http://finance.yahoo.com/q/hp?s=%%i %%i.csv /datetimeformat=yyyy-MM-dd /add=Symb=%%i
    webcsv.exe sleep 500
)

Please use the sleep mode to make a delay between requests to prevent banning your IP by web servers.

You may do the job in an easy way using WEB.CSV task files.

For example, create the task.txt file with following content:

CsvFileName  URL
csv\aapl.csv http://ichart.finance.yahoo.com/table.csv?s=AAPL&ignore=.csv
csv\goog.csv http://ichart.finance.yahoo.com/table.csv?s=GOOG&ignore=.csv

and the batch file can be much simpler:

@echo off

webcsv.exe WebToCSV @task.txt /datetimeformat=yyyy-MM-dd /delay=500

Please use the /delay option instead of the sleep mode to make a delay between requests.

Multiple Pages

WEB.CSV detects several schemes of the next page URLs.

In such cases, you may use the command like this:

webcsv.exe WebToCSV "http://www.google.com/finance/option_chain?q=AAPL&output=json" ^
                  aapl.csv /rootPath=calls,puts /pages=20

Command Line

Command Line Modes

webcsv.exe Help       [chm]
webcsv.exe WebToCSV   <url>|@<task> [[<downloaded file>] <csv file>] [<options>]
webcsv.exe WebToText  <url>|@<task>  [<downloaded file>]             [<options>]
webcsv.exe TextToCSV  <input file|mask> [<csv file|mask]             [<options>]
webcsv.exe HtmlTables <input file|mask> [<htm file|mask]             [<options>]
webcsv.exe DelIfHtml  <input file|mask>
webcsv.exe DelIfEmpty <input file|mask>
webcsv.exe Sleep      <milliseconds>
webcsv.exe Stamp     [<datetime format>]
webcsv.exe Register

Common Options

/inputCodepage=<codepage>
/outputCodepage=<codepage>
/noBOM
/echoUrl
/echoInputFileName
/echoOutputFileName

Query Options

/method=<method>
/accept=<accept>
/userAgent=<useragent>
/timeout=<milliseconds>
/delay=<milliseconds>
/pages=<pages>

Parser Options

/rootPath=<root path>
/collapsedNodes=<node>[;...]
/skippedNodes=<node>[;...]
/keptNodes=<node>[;...]
/ignoredTags=<tag>[;...]
/attibutedColumns=<col>.<attr>[;...]
/inputSeparator=<separator>|Tab
/noSourceHeaders
/asIs
/rowValues

CSV Output Options

/append
/noHeaders
/addRowNum
/dateTimeFormat=<format>
/dateFormat=<format>
/timeFormat=<format>
/[output]Separator=<separator>|Tab
/quoteChar=<char>
/escapeChar=<char>
/add=<header=value>[<sep>...]
/requiredColumns=<name>[;...]

Add Parameter Functions

UtcNow() | UtcDateTime()
UtcDate()
UtcTime()
Now() | DateTime()
Date()
Time()
NyseDateTime()
NyseDate()
NyseTime()
FileDateTime()
FileDateTimeUtc()
FileDateTimeNyse()
FileDate()
FileDateUtc()
FileDateNyse()
FileTime()
FileTimeUtc()
FileTimeNyse()
FileName()
FileNameOnly()
FileNameWithoutExtension()
FileNameWithoutPage()
FilePage()
FileNameDateTime(<file>)
FileNameDate(<file>)
FileNameTime(<file>)

Exit Codes

Exit CodeDescription
0Success
1Incomplete command line parameters
2Wrong command line parameters
3Exception
>200HTTP Status Code

Command Line Modes

WebToCSV

The utility loads and parses the text and outputs CSV data.

If the web data contains a next page URL, the utility loads the specified number of pages.

If an HTML page is not parsed correctly, try to use the HTML page for printing.

Also, you may use the /rootPath option to tune the results.

Examples:

wevcsv.exe WebToCSV http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices aapl.csv

wevcsv.exe WebToCSV @task.txt

WebToText

The utility loads and saves the text.

If the web data contains a next page URL, the utility loads the specified number of pages.

Examples:

wevcsv.exe WebToText http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices aapl.htm

wevcsv.exe WebToText @task.txt

TextToCSV

The utility parses input files and outputs the CSV data.

You may use the /rootPath option to tune the results. 

You may use * in the file mask.

You may join multiple CSV files into one CSV file.

Examples:

wevcsv.exe TextToCSV aapl.htm aapl.csv
wevcsv.exe TextToCSV htm\*.htm csv\*.csv
wevcsv.exe TextToCSV csv\*.csv results.csv
wevcsv.exe TextToCSV @task.txt

HtmlTables

Use this mode to find table numbers for using in the /rootPath option.

Example:

wevcsv.exe htmlTables aapl.htm aapl-tables.htm

wevcsv.exe TextToCSV aapl.htm /rootPath=3

DelIfHtml

Use this mode to delete HTML files if files of other types are expected.

Websites return HTML pages, for example, for the 'Not Found' error.

DelIfEmpty

Use this mode to delete empty files.

Sleep

Use this mode to wait for the specified amount of milliseconds between requests to avoid IP banning by the web server.

You may specify the /delay option instead of the use of the sleep mode.

Example:

wevcsv.exe sleep 500

Stamp

Use this mode in batch files to set environment variables like:

for /F %%i in ('webcsv.exe stamp') do set stamp=%%i

Register

Use this mode to launch the Registration Wizard.

Help

Use this mode to get extended the command line help or to start the CHM help like this:

wevcsv.exe help chm

Command Line Parameters

<URL>

The parameter must be a valid HTTP or HTTPS URL.

<File>

The parameter must be a valid file name.

You may specify absolute or relative paths.

<Mask>

The parameter can be a file name or a file mask.

Input file mask examples:

*.htm
options-*.htm
htm\*.htm
csv\*.csv

Input and output file mask examples:

*.htm         *.csv
options-*.htm *.csv
htm\*.htm     csv\*.csv
csv\*.csv     output\*
csv\*.csv     output\*.*
csv\*.csv     output.csv

@<Task>

Task files can be a plain text or a CSV files and contain command line parameters and options:

ColumnMeaning
URLThe mandatory <URL> parameter
FileNameThe <file> parameter
CsvFileNameThe <csv file> parameter
AddThe /add option
RootPathThe /rootPath option
PagesThe /pages option

Command line parameters have higher priority over task file values.

Plain text task file example:

URL                                                           FileName
https://ichart.finance.yahoo.com/table.csv?s=AAPL&ignore=.csv aapl.csv
https://ichart.finance.yahoo.com/table.csv?s=GOOG&ignore=.csv goog.csv

CSV task file example:

URL;FileName
https://ichart.finance.yahoo.com/table.csv?s=AAPL&ignore=.csv;aapl.csv
https://ichart.finance.yahoo.com/table.csv?s=GOOG&ignore=.csv;goog.csv

Command line example:

webcsv.exe WebToCSV @task.txt

You may create task files from a database using SELECT queries.

Command Line Options

Common Options

/inputCodepage=<codepage>

The option defines the input file code page.

Example:

/inputcodepage=65001

/outputCodepage=<codepage>

The option defines the output file code page.

Example:

/outputcodepage=1250

/noBOM

The option suppresses adding BOM to Unicode output.

Example:

/noBOM

/echoUrl

The option defines printing URLs to the console.

/echoInputFileName

The option defines printing input file names to the console.

/echoOutputFileName

The option defines printing output file names to the console.

Query Options

/pages=<pages>

The option defines the number of loaded pages.

Example:

/pages=5

/accept=<accept>

The option defines the Accept header of web requests.

You may specify this option if the default value is not acceptable.

Example:

/accept=application/json;odata=verbose

/userAgent=<user agent>

The option defines the UserAgent header of web requests.

Example:

"/UserAgent=webcsv 5.0"

/timeout=<milliseconds>

The option defines the timeout of web requests.

The default timeout is 100 seconds (100 000 ms)

Example:

/timeout=10000

/delay=<milliseconds>

The option defines the delay after requests to prevent banning your IP by web servers.

You may also use the sleep mode in batch files.

Example:

/delay=500

Parser Options

/rootPath=<root path>

The option defines the root elements of columns to output like .InvoiceTransmission.Invoice.

For HTML, a root path is a table number obtained in the HtmlTables mode.

Examples:

/rootPath=.InvoiceTransmission.Invoice
/rootPath=calls,puts
/rootPath=1

/collapsedNodes=<node>[;...]

The option defines the XML nodes that include values of children nodes.

Example:

/collapsedNodes=passages

/skippedNodes=<node>[;...]

The option defines suffixes of XML nodes to exclude from the output.

To exclude specific columns, use complete paths with column names separated by dots like parent1.parent2.column.

Example:

/skippedNodes=.type

/ignoredTags=<tag>[;...]

The option defines tags to skip as separate columns. The values are included into parent columns.

The typical scenario is ignoring text highlight tags. The option is applicable only with XML.

Example:

/ignoredTags=hlword

/noSourceHeaders=True

The option suppresses using HTML table headers as column names.

Use this option to import financial statements from the web with static column names.

/asIs

The option suppresses any special processing of the output data.

For example, WEB.CSV converts Yahoo timestamps to datetime values. You may disable this using the /asIs option.

/rowValues

This option applies XML or JSON parsers that print all values in rows.

This option is useful for learning document structures.

CSV Options

/append

The option defines adding data to the output CSV file.

/noHeaders

The option suppresses output of column headers.

/addRowNum

The option adds the row number column to the output.

/dateTimeFormat=<format>, /dateFormat=<format>, /timeFormat=<format>

The option defines the format for datetime values in the CSV output.

See available values at http://msdn.microsoft.com/en-us/library/zdtaw1bw(v=vs.100).aspx

Example:

webcsv.exe WebToCSV http://finance.yahoo.com/q/hp?s=AAPL aapl.csv /datetimeformat=yyyy-MM-dd

Use quotes to specify formats with spaces. For example:

"/datetimeformat=yyyy-MM-dd hh:mm:ss"

/[output]Separator=<separator>|Tab

The option defines the output CSV separator.

The default separator is a semicolon. Use the Tab value to specify the tab.

Example:

webcsv.exe WebToCSV http://finance.yahoo.com/q/hp?s=AAPL aapl.csv /separator=,

/InputSeparator=<separator>|tab

The option defines the input CSV separator.

/quoteChar=<char>

The option defines the character used for quoting string values.

The default value is the double quote. Specify an empty value to disable quoting.

See http://tools.ietf.org/html/rfc4180 about the CSV format.

Examples:

/quoteChar="
/quoteChar=

/escapeChar=<char>

The option defines the character used for quoting quote characters inside string values.

The default value is the double quote. Specify an empty value to disable quoting.

See http://tools.ietf.org/html/rfc4180 about the CSV format.

Examples:

/escapeChar="
/escapeChar=\

/requiredColumns=<name>[<separator>...]

The option defines the required columns for output.

If the columns are absent, the file is not added to the output.

For example, not all companies pay dividends. So, when you load dividends for a list of symbols, you may specify the required columns to skip such companies.

This is the best way to prevent auto-detecting output columns.

Example:

"/requiredColumns=Ex/Eff Date;Type"

/add=<header=value>[<separator>...]

The option defines additional data for the CSV output.

You may use constant values or functions described below.

Examples:

webcsv.exe WebToCSV  http://finance.yahoo.com/q/hp?s=AAPL aapl.csv /add=File=aapl.csv;Symb=AAPL
webcsv.exe TextToCSV aapl.htm *.csv "/add=File=FileName();Symb=FileNameWithoutExtension()"

Functions for /add Options

FunctionMeaning
UtcNow()Current UTC date and time
UtcDateTime()Current UTC date and time
UtcDate()Current UTC date
UtcTime()Current UTC time
Now()Current date and time
DateTime()Current date and time
Date()Current date
Time()Current time
FileDateTime()Input file date and time
FileDateTimeUtc()Input file UTC date and time
FileDate()Input file date
FileDateUtc()Input file UTC date
FileTime()Input file time
FileTimeUtc()Input file UTC time
FileName()Input file name
FileNameOnly()Input file name without extension
FileNameWithoutExtension()Input file name without extension
FileNameWithoutPage()Input file name without extension and page
FilePage()Input file page (2 for "AAPL-02.htm")
FileNameDateTime()Date and time parsed from an input file name *
FileNameDate()Date parsed from an input file name *
FileNameTime()Time parsed from an input file name *

* Supported formats for dates and times in file names:

The value after the first hyphenThe value at the name end
*-yyyyMMddHHmmss*.**yyyyMMddHHmmss.*
*-yyyyMMdd-HHmmss*.**yyyyMMdd-HHmmss.*
*-yyyyMMddTHHmmss*.**yyyyMMddTHHmmss.*
*-yyyy-MM-dd*.**yyyy-MM-dd.*
*-yyyyMMdd*.**yyyyMMdd.*
*-HHmmss*.**HHmmss.*

Product Registration

WEB.CSV requires registration to start a trial and to use it after the trial period.

To register, open Start Menu, All Programs, WEB.CSV, Register Product.

Selecting Edition

You may register the trial version for free; the product code is not required.

To register the WEB.CSV Personal or Enterprise edition fill in the product code received after purchasing.

WEB.CSV Registration - Select the edition

Licensee Data

Please fill in the registration form like this:

WEB.CSV Registration - Fill personal data

The Next button is enabled when all the required fields are filled.

Don't forget to check the required field about the personal data use.

Online Registration

This step allows you to check your email address. If the email is valid, click Next.

You may return to the previous step using the Back button.

WEB.CSV Registration - Check the email

After clicking the Next button, the WEB.CSV connects to the registration server.

If the connection is successful, the final step screen is shown.

If any error occurred during connection, you might try to register the product later or try to register the product by email.

WEB.CSV Registration - The final step

You may set the registration for all users of the computer. This action requires administrator privileges.

Click Finish.

Registration by Email

If you choose the Register by email option on the Licensee Data step, the following screen is displayed:

WEB.CSV Registration - Check and start the registration email

The Next button starts the default email program, creates a registration email, and activates the next step.
Don't forget to send the email.

If starting of the email program has failed, you may create the registration email manually using the shown registration data.
Please carefully copy the To, Subject, and the Message fields.

The registration server sends the reply in a couple of seconds; but you may close the dialog box and open it again, in the same step.

Please copy the activation code from the received registration email and paste it into the Activation code field.

WEB.CSV Registration - Paste the activation code

The Next button is enabled when the activation code is valid.

Click Next to continue.

The wizard checks the registration data and confirms the registration.

WEB.CSV Registration - The final step

You may set the registration for all users of the computer. This action requires administrator privileges.

Click Finish.