Long Process Pipelines
Run Anvia pipelines outside the request path with a durable queue.
Use a long process pipeline when the work should not finish inside one HTTP request. The request validates input and creates a job. A worker process runs the Anvia pipeline, persists the result, and reports progress through app-owned status records.
What is BullMQ?
BullMQ is a Node.js queue library built on Redis. A Queue adds jobs, a Worker processes jobs, and QueueEvents can observe queue lifecycle events. BullMQ uses Redis connections so API producers and worker processes can run separately and scale independently.
Install BullMQ in the application that owns the queue and worker:
pnpm add bullmq ioredisThe docs app does not need these dependencies unless it is also running a queue worker.
When to Use It
Use this pattern when:
- a pipeline can run longer than the caller should wait
- work should survive API process restarts
- jobs need retry, backoff, delayed execution, or worker concurrency
- pipeline results should be persisted for later polling or review
- API and worker processes scale independently
Architecture Shape
| Layer | Responsibility |
|---|---|
| API route | authorize caller, validate input, create app record, enqueue job |
| BullMQ queue | store pending jobs, retry policy, backoff, job lifecycle |
| worker process | consume jobs, run the Anvia pipeline, update progress |
| pipeline | own typed stages, agents, extractors, and deterministic transforms |
| app storage | durable status, result, errors, audit metadata |
| observability | connect queue job ids, pipeline stage events, traces, and logs |
Baseline Flow
Keep the API response small. Return a job id and store user-facing status in your own database. BullMQ return values are useful for worker internals and queue inspection, but product status pages should read durable app records.
Pages
- Enqueue Pipeline Jobs shows the API producer boundary.
- Run Pipeline Workers shows the worker and
pipeline.run(...)boundary. - Status, Retries, and Testing covers production behavior and tests.