Turn expensive video jobs into an observable, retryable pipeline.
Create a job from an immutable source
Video conversion can outlast an HTTP request, so create a processing job after upload validation. Include source asset ID, recipe version, desired outputs, and an idempotency key. Workers should write to temporary keys and publish finished outputs only after probing them.
Control concurrency
Transcodes consume CPU, memory, disk, and network. Limit worker concurrency by workload, not simply by queue depth. Large 4K inputs may need a different pool from short clips. Record input duration, output size, processing time, and failure reason so capacity decisions use evidence.
Retry only what can recover
A transient storage failure may justify a retry; a corrupt input or unsupported codec needs a terminal state and a user-facing explanation. Clean up partial outputs and keep the original available for later reprocessing. Test worker crashes at each stage to ensure duplicate jobs do not publish mixed renditions.
A concrete acceptance test
Kill a transcoding worker after it writes one temporary rendition. Restart the job and verify that only complete, probed outputs become visible and no mixed recipe versions appear. Run this test with a realistic asset and record the observed state, output URL, and failure reason.
Implementation check
- Use a source-and-recipe idempotency key.
- Probe output files before publishing URLs.
- Classify corrupt inputs separately from transient failures.
Related guides
Primary reference: Official documentation.
Related field note
Continue with How metadata changes queue backpressure.