How one misinterpreted byte took down our Celery worker fleet

We declared an incident one morning after alerts started coming in: Celery workers had stopped processing tasks, queues were building up, and the only error we had to work with was this:

TypeError: unsupported operand type(s) for +: 'float' and 'str'

Not exactly the kind of message that might explain why this was happening.

What were the workers doing ?

These worker pools handle work that directly affects customers: billing runs, notifications, data processing, etc. So when several of them started raising alarms, the pressure was on from the start.

What made the situation particularly frustrating was that the workers weren’t just failing: they were stuck in a loop that had no natural way to resolve itself.

In Celery’s model, each unit of work is a task: a Python callable that travels through RabbitMQ as a message instructing the worker how to run that callable: The arguments to run it with, some metadata about whether it was retried or not, etc. The two terms are meaningfully different: “message” refers to the broker payload, “task” refers to the executable unit on the worker side. That distinction matters here, because the bug lived in the message.

A worker would pick up a message from the broker, deserialize it into a Celery task, then crash deep inside Billiard, Celery’s multiprocessing library, on this line:

if monotonic() >= start + timeout:

The start value was a normal float. The timeout variable, extracted from the message’s timelimit header, contained a comma — just , — when it was supposed to hold a number of seconds. Billiard tried to add them together, blew up, and sent a SIGTERM to the child process:

WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM) Job: 1.')

Because the exception came from Billiard rather than from the task’s own logic, Celery treated it as a worker-level failure, so the message didn’t get acknowledged in the broker and got put back on the queue.

The message being back on the queue meant that another worker would pick it up, try to process it, and crash the same way again.

These messages, each carrying an invalid timelimit value, were cycling endlessly through the pool, and because Celery workers consume from the head of the queue, every legitimate message sitting behind one of them was effectively stuck waiting. The queue lengths climbed, while the workers just kept dying and restarting.

We scaled up the worker pools as a short-term measure to get some of the legitimate work moving again, but that was always going to be temporary because as long as the corrupt messages existed in the queues, any worker was eventually going to pick one up.

Why did this appear now ?

The error wasn’t leading us anywhere obvious, so identifying the exact culprit was no easy task.

Furthermore, the code that sets these timeouts is statically typed, so it couldn’t have possibly been someone enqueueing a message with a comma instead of a float, which led us to believe that this wasn’t something happening from our application’s business logic.

The only context left to correlate this to was that, during the previous day, we’d migrated one of our production environments from classic queues to quorum queues as part of our broader RabbitMQ v4.x upgrade.

However, we had also applied this migration across several other environments many times already, each without an incident, so we had no incentive to treat this one differently, except that the time correlation was telling us the exact opposite.

Part of the migration involved physically transferring messages from classic queues to the new quorum equivalent, and we used aio-pika for this: read from the source queue, optionally transform the ETA (estimated time of arrival) headers, and republish to the destination.

Where did the comma came from ?

The timeout value in Billiard came from Celery’s timelimit message header, which carries a two-element list of [soft_timeout, hard_timeout], and many of our tasks were configured with [None, 300] meaning: no soft limit and a 300-second hard timeout.

After the migration, Celery was reading that header as [None, ',']. Understanding how 300 became , requires a short detour into how RabbitMQ actually puts data on the wire.

AMQP field tables

RabbitMQ uses the Advanced Message Queuing Protocol (AMQP), a binary protocol where messages travel as typed frames.

One of those frame types carries message properties, which are basically metadata like content type, delivery mode, and custom headers.

Celery uses the headers property to attach information to messages: the task name, ETA, time limits, etc.

That headers dictionary is encoded on the wire as what AMQP calls a field table: a binary structure where each entry is a key-value pair.

The format is straightforward:

• 1 byte for the key length (N)
• N key bytes
• 1 byte for the type tag
• K bytes for the actual value, where K depends on the value of the type tag.

The type tag is the important part here because it tells the decoder what kind of value follows and how many bytes to consume for it.

Here’s an example of the byte sequence that represents a header that has amount as key, and 300 as value:

From 300 to ,

The previous animation used 300 as value on purpose, because it plays an important role in the occurring exception.

300 exceeds the signed 8-bit range (-128 .. 127), so aio-pika’s underlying pamqp encoding library promotes it to a signed 16-bit integer, and 300 would then have the following hex value: 0x012C.

So when we’re transferring those messages, pamqp writes three bytes onto the wire:

73 01 2C

73 is the type tag (the ASCII code for s), which in pamqp’s convention means “signed 16-bit integer”. 01 2C are the two bytes encoding 300.

On the receiving end, however, Celery uses py-amqp for decoding.

When py-amqp sees 73, e.g the the character s, it doesn’t interpret it as a signed 16-bit integer. It interprets it as a short string: a length byte followed by that many characters.

So py-amqp read the same three bytes as: s means short string, 01 is the length (one character), 2C is the character content, and ASCII 0x2C is ,.

If we connect the dots, the headers start with a 300 that gets misinterpreted into a ,, and nothing in either library flagged it. The corrupted value decoded cleanly into a valid short string that Celery doesn’t complain about until much later in the worker execution loop: when Billiard finally tried to do arithmetic on it.

Why two AMQP libraries disagree on one byte

You’d naturally expect two mainstream Python implementations of the same protocol to agree on a single-byte type indicator. The reason they don’t is rooted in AMQP’s version history, and it’s documented explicitly in the RabbitMQ errata page:

By the time the AMQP 0-9-1 spec was finalized, RabbitMQ and Qpid had already shipped their own field table extensions, and they’d assigned different meanings to the same byte values. The spec said s means short string but RabbitMQ had already been using s for signed 16-bit integers, and it had chosen to keep its convention:

pamqp deliberately follows the RabbitMQ convention because its v1.5.0 changelog explicitly references RabbitMQ’s errata. py-amqp, on the other hand, follows the 0-9-1 spec. Neither library is doing anything wrong in isolation. The problem only appears when one sits on the encoding side and the other on the decoding side, and nothing tells you that they diverge on this particular tag.

What explains the earlier migrations going cleanly is that the bug only manifests when messages carry a timelimit header with a value that pamqp promotes to a 16-bit integer. In the earlier environments, we were just lucky that they didn’t.

How we responded

The first priority was stopping the crash loop. We couldn’t easily rewrite the messages in place, so we needed RabbitMQ to stop redelivering them indefinitely: We set a delivery limit policy on the affected queues: any message that exceeded 150 delivery attempts would be routed to a dead-letter queue. Once the corrupt messages hit that ceiling, the workers were freed and the queues began draining normally.

With the messages safely sitting in dead-letter queues, we wrote a script using pika (which uses py-amqp under the hood) to inspect and repair them by simply checking the timelimit header for the [None, ','] pattern, replacing it with [None, 300], and redriving those messages back to their original queues. The value was deterministic enough that we could reconstruct it confidently: the comma was always 0x2C, always the result of a 300 being misread as a one-character string.

The real fix was replacing aio-pika with pika for the transfer step. pika encodes 16-bit integers using the U type tag, which is what py-amqp expects on the receiving end. Both sides now agreed on the byte, and the encoding mismatch was gone.

We also added a monitor for new instances of that exception, acting as a signal that some corrupt messages had an ETA set far enough in the future to delay their appearance in the queue, and that the correction script would need to run again.

Takeaways

The thing that made this hard to catch was the silence. Nothing failed at the point of encoding. Nothing failed at the point of decoding. The corrupted value was structurally valid: py-amqp had no reason to complain, because it had decoded a legitimate short string. The crash happened far downstream, in code that had nothing to do with the message transfer.

A few things we’d do differently or emphasize going forward:

When mixing AMQP libraries across a publish/consume boundary, verify they agree on encoding before you rely on them in production. The fact that both libraries implement the same protocol version doesn’t mean they handle every field table tag the same way. The spec has documented conflicts, and the only reliable way to check is with round-trip tests using real message data and not just confirming that each library can encode and decode on its own.

Delivery limits and dead-letter queues belong in your default queue configuration, not your incident runbook. They were the tool that let us break the crash loop without taking queues offline. Without them, pausing consumption entirely would have been the only option. There was no way for us to predict that a queue might carry a corrupt message, and having those safety nets in place + monitoring on dead letter queue length and exceptions is a simple and effective solution to avoid such interruptions.

Silent data corruption is a harder class of failure than a loud one. If py-amqp had thrown a decode error on encountering an unexpected value type, we’d have found the problem within minutes of the first corrupt message hitting a worker. Instead, the corrupted value decoded without complaint, and we spent hours tracing backwards from a crash to find where the data had actually been mangled. The further a bug travels from its source before it surfaces, the harder it is to find.

The spec conflict that caused this is right there in the RabbitMQ errata, clearly documented. It’s not obscure. What’s missing is any signal between the two libraries that they’ve taken different sides of it — and that gap is what turned a one-byte disagreement into a multi-hour incident.

Would you like to help us build a better, greener and fairer future?

We're hiring smart people who are passionate about our mission.

Posted by Amrou Bellalouna Software Engineer on Sep 5, 2026