Billy Configuration

Billy has a global configuration object at billy.conf.settings that is used in scraping, import, and serving the API.

All billy scripts attempt to load a billy_settings module which should either be on the import path or in the working directory, this file can contain overrides and custom settings. As of 0.9.2 if no billy_settings module can be located a warning message will be printed to the console.

Default Settings

MONGO_HOST
Host or IP address of MongoDB server. (default: “localhost”)
MONGO_PORT
Port for MongoDB server. (default: “27017”)
MONGO_DATABASE
MongoDB database name. (default: “billy”)
API_BASE_URL
Public URL that the API can be accessed at.
SCRAPER_PATHS
Paths that scraper modules are stored under, will be added to sys.path when attempting to load scrapers.
BILLY_DATA_DIR
Directory where scraped data should be stored. (default: “<cwd>/data”)
BILLY_CACHE_DIR
Directory where scraper cache should be stored. (default: “<cwd>/cache”)
BILLY_ERROR_DIR
Directory where scraper error dumps should be stored. (default: “<cwd>/errors”)
BILLY_MANUAL_DATA_DIR
Directory where manual data files for matched ids/subjects are stored. (default: “<cwd>/manual_data”)
BILLY_SUBJECTS
List of valid subject names
SCRAPELIB_TIMEOUT
Value (in seconds) for url retrieval timeout. (default: 600)
SCRAPELIB_RETRY_ATTEMPTS
Number of retries to make if an unexpected failure occurs when downloading a URL. (default: 3)
SCRAPELIB_RETRY_WAIT_SECONDS
Number of seconds to wait between initial attempt and first retry. (default: 20)

Command-Line Overrides

Most available scripts can override the above default settings with command line switches:

--mongo_host <mongo_host>

Override MONGO_HOST

--mongo_port <mongo_port>

Override MONGO_PORT

--mongo_db <mongo_db>

Override MONGO_DATABASE

-d <data_dir>, --data_dir <data_dir>

Override BILLY_DATA_DIR

--cache_dir <cache_dir>

Override BILLY_CACHE_DIR

--error_dir <error_dir>

Override BILLY_ERROR_DIR

--manual_data_dir <manual_data_dir>

Override BILLY_MANUAL_DATA_DIR

--retries <retries>

Override SCRAPELIB_RETRY_ATTEMPTS

--retry_wait <retry_wait>

Override SCRAPELIB_RETRY_WAIT_SECONDS