Background Jobs
by @ops Data Pipeline / ETL Starter
Python + Kafka + a Celery worker for real batch/stream data processing — scheduled transforms and event-driven pipelines, not an on-demand job queue. Distinct from the Job Queue & Scheduler recipe: that's request-triggered background work, this is data moving on its own schedule.
Loads this recipe straight into the wizard — every setting stays editable before you generate anything.
The stack
Install
# pulls this exact recipe straight from the Vault $ npx stacktora install data-pipeline-etl --remote $ stacktora sync # generate the files
What you get
More from the Vault
Go + Postgres + Redis: Distributed Cron / Scheduled Task Runner
Scheduled tasks that run exactly once across multiple app instances — Redis-backed leader election so a scaled-out deployment doesn't run the same cron job N times. Distinct from Job Queue & Scheduler: that's a general queue, this specifically solves multi-instance double-execution.
Job Queue & Scheduler
A dedicated background-job starter — Redis-backed queue plus Postgres for job records, with a worker process already wired in. Not tied to one specific web framework's job-queue convention.
Reverse ETL / Data Warehouse Sync
Pushing data FROM your app OUT to external tools (a CRM, a spreadsheet, an ads platform) on a schedule — the opposite direction from the Data Pipeline/ETL recipe, which pulls data in. A worker handles the sync so it never blocks a request.
.NET + Postgres + Redis: Fintech-oriented Baseline
A security-conscious starting point for financial and payments-adjacent backends: .NET API, Postgres for ledger/transaction records, and Redis for session and rate-limit state. This is a starting baseline only, not a certified or verified PCI-DSS/regulatory-compliant stack — your own security and compliance review is still required before handling real financial data.