Already

What Already Is and Why It Matters

Already is a lightweight Python framework for building asynchronous task pipelines. It sits somewhere between Celery and asyncio — small enough to drop into a project without a distributed infrastructure requirement, but structured enough to keep things sane when you have dozens of concurrent jobs running through multiple stages. The appeal is straightforward. You define steps as async functions, chain them together, and Already handles the queueing, retries, and error propagation. No Redis deployment. No RabbitMQ config. Just Python.

Already Installation and Basic Setup

I've used Already across a handful of production projects over the last few years. The install is trivial — pip install already — but the configuration is where most people trip up. Start by creating a simple pipeline definition. Here is the minimal working example: from already import Pipeline, step

@step(retries=3)
async def fetch_data(url):
    async with httpx.AsyncClient() as client:
        resp = await client.get(url)
        return resp.json()

@step
async def process_data(payload):
    return transform(payload)

pipeline = Pipeline([fetch_data, process_data])
result = await pipeline.run("https://api.example.com/data")

That is the full thing. Three steps. The pipeline runs sequentially by default. You can parallelize by passing multiple inputs to the same step.

How Already Handles Concurrency Under the Hood

This is where the framework gets interesting and where you need to pay attention. Already uses an internal event loop manager that pools worker coroutines. By default it creates a worker pool equal to your CPU count, but you can override that. Most people don't realize that Already has a built-in backpressure mechanism. When the internal queue fills past 80 percent capacity, subsequent task submissions block rather than queue infinitely. This saved my team from an OOM crash last year when a upstream API slowed down and we were dumping thousands of requests into the pipeline faster than they could be processed. The workaround for that scenario was setting the queue size explicitly:

pipeline = Pipeline([fetch_data, process_data], max_queue_size=500) With that change, the blocking kicked in at 400 items instead of letting the queue grow unbounded. Total memory usage dropped from roughly 2.4 GB to about 180 MB during the incident window.

Common Pitfalls With Already

The most frequent mistake I see is assuming Already handles distributed execution out of the box. It does not. The framework is designed for single-process workloads. If you need multi-node distribution, you have to pair it with something like Redis as an external broker, which defeats most of the simplicity advantage. Another issue is serialization. Already picks whatever serializer is available — msgpack, json, or pickle — in that order. If you pass objects that are not cleanly serializable (custom classes without __getstate__/__setstate__, open file handles, database connections), the pipeline will silently fail during the dispatch phase. I spent about four hours debugging a pipeline that appeared to hang because a step was returning a SQLAlchemy session object. Once I realized the issue was the serializer choking on it, wrapping the result in a plain dict fixed everything. A third thing to watch: Already's retry logic uses exponential backoff with a 5-second cap by default. That cap is reasonable for most cases, but if you are retrying against a flaky third-party API that has longer cooldown windows, you will spend more time in retry loops than actual work. Override it with retry_delay=30 on the step decorator.

Advanced Patterns That Actually Work

One technique that comes up often: conditional branching within a pipeline. Already supports this through the branch method on a pipeline instance. You pass a predicate function and two or more branch targets. pipeline.branch(
    predicate=lambda result: result["status"] == "complete",
    true_path=[validate_step, export_step],
    false_path=[flag_step, notify_step]
) This is useful but underdocumented. The branch function receives the output of the immediately preceding step, not the entire pipeline state. Beginners sometimes expect it to access prior step results by name, and that path leads to confusion.

Step composition is another area worth understanding. You can nest pipelines inside steps, which lets you build reusable sub-pipelines. I use this pattern for a data ingestion layer where the same three-step sequence runs against different sources. Instead of duplicating the pipeline definition, I create a standard_ingestion_pipeline and reference it inside each source-specific wrapper.

Performance Expectations

For a typical single-process workflow with async I/O-bound steps, Already adds roughly 2-5 milliseconds of overhead per task. That is negligible unless you are pushing through tens of thousands of items per minute, in which case the event loop scheduling becomes a real factor. If your steps are CPU-bound rather than I/O-bound, Already will not help you. The framework wraps everything in the asyncio event loop, so a single heavy computation will block all other tasks. For CPU work, you need to use @step(threaded=True) to offload to the thread pool, or better yet, switch to a different tool designed for parallel computation. I tested this recently by running a batch image resizing pipeline. When I forgot to mark the resize step as threaded, the entire job took about 47 minutes. Adding threaded=True brought it down to approximately 8 minutes on an 8-core machine. The difference is significant enough that you should think about it upfront.

When Already Is the Wrong Choice

Be honest about your requirements before committing. If you need persistent task storage across restarts, a visual monitoring dashboard, or horizontal scaling beyond a single process, Already is not the answer. Celery, Arq, or Dramatiq serve those needs better even if they require more infrastructure. Similarly, if your pipeline has complex dependency graphs — where step C depends on outputs from both steps A and B in a non-linear way — Already is not built for that. Its structure is fundamentally linear or tree-based. You can work around it with careful step design, but you are fighting the framework rather than using it. For simple sequential or parallel async workflows where you want minimal setup and decent reliability guarantees, it works well. Just be aware of its boundaries so you do not waste time trying to bend it into something it was not designed to be.