Getting WRDS data into MySQL or PostgreSQL

In an earlier post, I talked about getting WRDS data into SQLite. One “advantage” of SQLite is that one doesn’t need to specify data types. But doing so is necessary with MySQL or PostgreSQL. Retaining the basic approach of my earlier post yields the following procedure:

1. Run a SAS program on the WRDS server to create a “schema” file containing the data types of the variables for a table. This is pretty kluge-y and the result of trial and error.

2. Download the SAS data file (dsi.sas7bdat) and the schema file (dsi_schema.csv).

3. Run StatTransfer on the SAS data file. Change to the directory where you placed the files above and then issue
st dsi.sas7bdata dsi.csv and then gzip dsi.csv to save space (and likely speed up the import).

4. Use a Perl script to read the text file created by StatTransfer and the schema and load the data into MySQL or PostgreSQL (and also to create indexes and the like).

Advertisements
This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s