Building Event-Driven Python Systems with FastAPI, RabbitMQ and Celery

When building APIs that handle long-running tasks—such as processing documents, coordinating multi-agent AI workflows, or handling massive webhook volume—a synchronous HTTP request cycle quickly becomes a bottleneck.

Event-driven architecture solves this by decoupling the request from the work. This article explores an architecture using FastAPI for the API boundary, RabbitMQ as the message broker, and Celery for distributed task execution.

The Problem with Synchronous Execution

In a traditional synchronous REST API:

1. The client sends a request.
2. The server does the work.
3. The server responds.

If the work takes 45 seconds (e.g., an LLM generating a complex response or a document being embedded), the connection stays open. This leads to timeouts, connection pool exhaustion, and poor user experience.

The Event-Driven Approach

Instead of doing the work immediately, the API simply acknowledges the request and promises to do the work later.

1. FastAPI: The API Boundary

FastAPI provides the entry point. It receives the request, validates the payload using Pydantic, and immediately dispatches a message to the queue.

from fastapi import APIRouter
from models import TaskRequest
from tasks import process_document_task

router = APIRouter()

@router.post("/process-document")
async def process_document(request: TaskRequest):
    # Dispatch task to Celery
    task = process_document_task.delay(request.document_id)
    return {"task_id": task.id, "status": "accepted"}

2. RabbitMQ: The Message Broker

RabbitMQ acts as the durable queue between FastAPI and Celery.

When a task is dispatched, it sits in a RabbitMQ queue until a worker is available. This provides a buffer against traffic spikes. If 1,000 requests arrive at once, the API accepts them immediately, and the queue grows.

Key concepts:

• Acknowledgements (Acks): A worker must acknowledge a message once it successfully completes. If a worker crashes mid-execution, RabbitMQ will not receive an ack and will requeue the message for another worker.
• Dead Letter Exchanges (DLX): If a message fails repeatedly, it can be routed to a DLX for manual inspection rather than clogging the main queue.

3. Celery: The Worker Pool

Celery workers listen to the RabbitMQ queues. They pull messages off the queue and execute the actual Python functions.

from celery import Celery

app = Celery('tasks', broker='amqp://guest@localhost//')

@app.task(bind=True, max_retries=3)
def process_document_task(self, document_id: str):
    try:
        # Perform heavy AI/processing work here
        result = run_heavy_pipeline(document_id)
        return result
    except Exception as exc:
        # Retry with exponential backoff
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

Designing for Failure

Distributed systems will fail. The architecture must account for it:

1. Idempotency: Tasks might be executed more than once (e.g., if a worker crashes right after finishing but before acking). The task must be idempotent—running it twice should have the same effect as running it once.
2. Retries: Celery supports automatic retries with exponential backoff for transient errors (like a rate-limited external API).
3. Observability: With asynchronous systems, tracing a request is crucial. Use correlation IDs passed from the initial FastAPI request through to the Celery task to track logs across the boundary.

Scaling the Workers

The beauty of this architecture is independent scaling. If the API is under heavy read load, you can scale FastAPI horizontally. If there's a backlog of processing tasks, you simply spin up more Celery workers. The broker handles the distribution.

Production Lessons

From deploying this architecture in production environments, several lessons stand out:

• Don't pass huge payloads in messages. Pass an ID and let the worker fetch the data from the database or object storage.
• Set timeouts. Both soft and hard time limits on tasks prevent stalled workers from silently hanging forever.
• Monitor queue depth. The size of the RabbitMQ queue is the primary indicator of whether your worker pool needs to scale.

By decoupling the API from the heavy lifting, you build a resilient, scalable backend capable of supporting complex AI systems and high-throughput workflows.