Data Sync

Manage all three data source sync pipelines

Database Status

Total Projects

0

Last Synced

—

Manual Sync

webrobots.io
Data Source

Monthly full snapshot · 2016 to present

Downloads the latest webrobots.io dataset (~100 MB compressed, ~2M records) and writes to local SQLite. Sync takes 5–15 minutes — do not close the app.

Auto-sync: checks webrobots daily at 4am and only downloads when a new dataset is available.

KS Live Discovery

kickstarter.com

Real-time new project discovery · fills snapshot gaps

Scrapes the Kickstarter discover page (sorted by newest) to find projects launched after the last webrobots snapshot.

Auto-runs every 2 hours in the background, fetching projects from the last 3 days.

Kicktraq Active Projects

kicktraq.com

Active project supplement · third-party source

Scrapes the Kicktraq active projects list to supplement KS direct scraping coverage.

Auto-runs every 6 hours in the background.

Kicktraq Full Scan (one-time)

full scan

Crawl all category pages to capture all currently active projects

Iterates all ~130 Kicktraq categories, paginates through each, and deduplicates against existing records. Estimated runtime: 15–30 minutes.

⚠️ Note: /archive/ is disabled by Kicktraq. This scan only covers currently active projects (~2800). Historical data relies on webrobots monthly snapshots.

About the Data

  • Source: webrobots.io monthly Kickstarter full snapshots
  • Format: CSV (ZIP-compressed), 20+ fields per project
  • History: March 2016 to present, one snapshot per month
  • Auto-sync: checks for a new dataset on server startup and daily at 4am
  • Database: Local SQLite at data/kickstarter.db
  • KS Live: auto-runs every 2h to discover new projects from Kickstarter discover page
  • Kicktraq: auto-runs every 6h, deduplicates against existing records before inserting