Billy Scripts

Scraping

billy-update <STATE>

MODULE

state scraper module name (eg. nc) [required]

--scrape, --import, --report

Which portions of the update process to run, specifying none implies --scrape --import --report

–scrape crawls the specified sites and writes data to disk in JSON format –import imports the JSON format on disk to billy’s MongoDB database –report runs a series of reports on data quality and aggregates

--bills, --legislators, --votes, --committees, --events

include (bill, legislator, vote, committee, event) data in (scrape/import/report) Specifying none is the same as specifying –bills –legislators –votes –committees

--upper, --lower

process upper/lower chamber (if neither is specified will include both)

-s SESSION, --session SESSION

session(s) to scrape, must be present in state’s metadata

-t TERM, --term TERM

term(s) to scrape, must be present in state’s metadata

--strict

fail immediately when encountering validation warnings

-n, --no_cache

do not use cache

--fastmode

operate in “fast mode”, using cached version when possible and removing –rpm induced delays

-r RPM, --rpm RPM

set maximum number of requests per minute (default: 60)

--timeout TIMEOUT

set HTTP timeout in seconds (default: 10s)