Exactly-Once vs At-Least-Once Delivery in Data Pipelines
Exactly-once delivery means every record a pipeline reads lands in its destination one time, with nothing lost and nothing duplicated, even when a process crashes mid-write. At-least-once means nothing is lost but a retry can land a record twice, and at-most-once means a record can vanish but never doubles. As of November 2026 the choice turns on one question, what the sender does when an acknowledgment never arrives.
Guarantee | Can lose | Can duplicate | How it happens |
|---|---|---|---|
At-most-once | Yes | No | Commit the position first, then process, never retry |
At-least-once | No | Yes | Process first, then commit, retry on every timeout |
Exactly-once | No | No | At-least-once plus idempotent or transactional writes |
Delivery vs processing, the distinction every vendor leans on
Exactly-once delivery is a claim about the wire, and exactly-once processing is a claim about the effect. Nearly every system that advertises the first is offering the second. The Kafka design docs warn that when systems claim exactly-once delivery “it is important to read the fine print”.
Flink’s fault tolerance guide states the distinction plainly. When the ideal is called exactly once, “this does not mean that every event will be processed exactly once. Instead, it means that every event will affect the state being managed by Flink exactly once.”
Kafka scopes itself the same way. Its docs say transactions give exactly-once delivery when reading, processing, and writing Kafka topics, and that “exactly-once delivery for other destination systems generally requires cooperation with such systems”.
Why is exactly-once delivery impossible on the wire?
Because a sender can never learn with certainty that its last message arrived. The proof is the Two Generals problem in Jim Gray’s 1978 operating systems notes, where two allies must agree on an attack time through messengers who can be captured. Martin Kleppmann’s Cambridge lecture notes put it in one sentence. “It can be proved that they cannot reach certainty by exchanging any finite number of messages.”
A sender that times out has two honest options, retry and risk a duplicate, or stop and risk a loss. Kleppmann’s notes reduce the field to one line, where exactly-once means “retry + idempotence or deduplication”. Idempotence means a repeated write leaves the same state as a single one. “Incrementing a counter is not idempotent, but adding an element to a set is.”
How does Kafka deliver exactly-once?
Kafka layers three mechanisms, and each extends the guarantee one step. The idempotent producer, added in 0.11.0.0, attaches a producer ID and a sequence number to every message so the broker can drop a retried duplicate, per the design docs. KIP-98 limits this to a single producer session.
Transactions extend it across partitions and across the consumer’s own offset commit. A transactional.id ties a producer to a transaction coordinator, and a producer epoch fences any zombie instance that wakes after a restart. The consumer must read with isolation.level=read_committed, which returns only messages up to the last stable offset, per the consumer configs.
Setting | Default | What it does |
|---|---|---|
true | Producer ID and sequence number dedup retries in one session | |
all | Leader waits for every in-sync replica before acknowledging | |
unset | Atomic writes across partitions, zombie fencing by epoch | |
60000 | Coordinator aborts a transaction open longer than this | |
read_uncommitted | read_committed hides aborted and open transactions | |
at_least_once | exactly_once_v2 needs brokers on 2.5 or newer | |
disabled | Connect source transactions, distributed mode only |
The defaults carry a footnote. KIP-679 made idempotence the default in 3.0, but the 3.2 upgrade notes record a bug that left it off in 3.0.0 and 3.1.0 unless set explicitly. Kafka Connect’s exactly-once support for source connectors, added in 3.3.0, depends entirely on the connector.
Checkpoints and two-phase commit in Flink and Spark
Stream processors reach exactly-once by snapshotting state and committing output in step with the snapshot. Flink injects checkpoint barriers into the stream and aligns them at operators with several inputs, and its checkpointing config defaults to EXACTLY_ONCE, with AT_LEAST_ONCE skipping the alignment.
The sink is where end-to-end happens. Flink’s 2018 overview describes the two-phase commit sink, where starting a checkpoint is the pre-commit and the checkpoint-complete callback is the commit, which must be idempotent because Flink commits preemptively on every restore.
Spark’s programming guide states the same recipe, replayable sources plus checkpoints plus idempotent sinks, and its sink table marks only the file sink exactly-once.
What do replication slots and change data capture guarantee?
At-least-once, with the dedup left to the consumer. The Postgres logical decoding docs say a slot emits each change once in normal operation, but its position is persisted only at checkpoint, so a crash can rewind it. The same page assigns the duty. “Logical decoding clients are responsible for avoiding ill effects from handling the same message more than once.”
The client’s tools are the slot’s confirmed_flush_lsn in the pg_replication_slots view and replication origins, which record apply progress crash-safely in the same transaction as the data.
Debezium says it directly on its exactly-once page. “Debezium provides at-least-once delivery guarantees.” A crash can replay events, and the only exactly-once path is Kafka Connect’s source transactions on a distributed cluster running 3.3.0 or later. The same page notes that Kafka’s transaction protocol is still under test, citing Jepsen reports.
MySQL built the dedup into the replica. A GTID is unique across the topology. Once a replica commits one, any later transaction carrying the same GTID is skipped without error, so a source transaction “can be applied no more than once on the replica”.
Where it goes wrong
Five failure patterns account for most duplicate and loss incidents.
The acknowledgment is lost after the write
The write succeeded, the ack did not arrive, and the sender retries. RabbitMQ’s reliability guide describes a confirm that never reaches the producer, NATS JetStream redelivers after the ack wait elapses, and even Pub/Sub’s exactly-once mode admits copies can arrive from publish-side duplicates.
The transaction outlives its timeout
Flink’s Kafka sink docs require the Kafka transaction timeout to exceed the maximum checkpoint duration plus the maximum restart duration, “or data loss may happen when Kafka expires an uncommitted transaction”. The symptom is a ProducerFencedException.
The side effect lives outside the system
An email, a webhook, or an HTTP call cannot join a transaction. Google’s GA announcement for Pub/Sub exactly-once says “side effects are strictly outside the scope of this feature”, and that holds for every system on this page.
The dedup window runs out
Every broker-side dedup is a window, and a retry that lands after it closes is a new message.
System | Dedup key | Window |
|---|---|---|
| 5 minutes | |
| 2 minutes by default | |
Hash of the inserted block | 10,000 blocks or one hour | |
Hash of the inserted block | Off until | |
Pub/Sub message ID | Until the ack deadline, 60 seconds by default |
The sink dedups eventually, not on write
ClickHouse’s ReplacingMergeTree docs say dedup happens only during a background merge at an unknown time, so the engine “does not guarantee the absence of duplicates”. Contrast Postgres ON CONFLICT and BigQuery stream offsets, which reject or merge the duplicate at write time.
How the main systems handle it
Every system below defaults to at-least-once or weaker, and each exactly-once option stops at a stated boundary.
System | Default | Exactly-once option | Boundary |
|---|---|---|---|
At-least-once | Idempotent and transactional producer, Streams exactly_once_v2 | Ends at the Kafka topic | |
EXACTLY_ONCE checkpoints | Two-phase commit sinks | Sink must be transactional or idempotent | |
Exactly-once with idempotent sinks | File sink | Kafka and foreach sinks are at-least-once | |
At-least-once after a crash | Replication origins, two_phase | Client dedups replays | |
At-least-once | Kafka Connect source transactions | Distributed mode, Kafka 3.3.0 or later | |
At-least-once | Exactly-once delivery on pull subscriptions | One region, no side effects | |
Exactly-once processing | Deduplication ID | 5 minute window, 300 TPS per partition | |
At-least-once with acks | None, idempotent consumers advised | Lost confirms duplicate | |
At-least-once | Message ID dedup and double ack | 2 minute window |
What does exactly-once cost?
Throughput and latency, in proportion to how often you commit. The only published vendor figures are Confluent’s 2017 measurements on Kafka 0.11.
Mechanism | Measured cost | Condition | Source |
|---|---|---|---|
Kafka idempotent producer | Negligible | Any | |
Kafka transactions | 3 percent vs ordered at-least-once, 20 percent vs acks=1 | 1 KB messages, 100 ms transactions | |
Kafka Streams exactly-once | 15 to 30 percent, zero at a 30 s commit | 100 ms commit interval | |
Flink barrier alignment | A few milliseconds, with outliers | Operators with several inputs | |
Pub/Sub exactly-once | Significantly higher latency, thousands of messages per second with ordering | Pull subscriptions | |
SQS FIFO | 300 TPS per partition, 3,000 with batching | Standard throughput mode |
The knob is the commit interval. Kafka Streams drops its default commit.interval.ms to 100 under exactly_once_v2, per the Streams configs, and Flink’s concepts page frames the checkpoint interval as overhead traded against records replayed on recovery.
Related guides
The checkpointing explainer covers the resume points that keep replays bounded, and the write modes guide defines the upsert that makes a sink idempotent. The Postgres logical replication guide and the MySQL binary log explainer go deeper on slots and GTIDs, and data integrity verification covers proving nothing was lost or doubled.
Frequently asked questions
What is the difference between at-least-once and exactly-once delivery?
At-least-once guarantees nothing is lost but lets a record arrive more than once, because the sender retries whenever an acknowledgment fails to come back. Exactly-once adds a second rule so each retried record changes the destination only once. In practice that rule is an idempotent write or a transaction, not a different wire protocol.
Is exactly-once delivery actually possible?
Not over an unreliable network, and the Two Generals problem in Jim Gray’s 1978 notes is the proof. A sender can never be certain its last message arrived, so it must retry and risk a duplicate or stop and risk a loss. Systems deliver exactly-once effect instead, where duplicates arrive but are recognized and ignored.
Does Kafka guarantee exactly-once delivery?
Inside Kafka, yes. The idempotent producer drops duplicates from retries, transactions make a read-process-write cycle atomic, and Kafka Streams exposes this as exactly_once_v2. The guarantee ends at the Kafka topic. Writing to a database, a warehouse, or an API needs that system to dedup or join the transaction, which the Kafka docs say outright.
What is the difference between exactly-once delivery and exactly-once processing?
Delivery counts how many times a message crosses the wire, and processing counts how many times its effect is applied. Flink’s docs say its exactly-once mode means every event affects state exactly once, not that every event is processed once. Replays happen during recovery, and the system keeps them out of the result.
How do you handle duplicate messages in a data pipeline?
Give every record a stable key and make the write idempotent, so a repeat overwrites itself. Postgres ON CONFLICT and MERGE, Snowflake MERGE, and BigQuery stream offsets all do this. Without a key, keep a dedup log such as a message ID table or a broker’s duplicate window, sized longer than your longest retry gap.
Does exactly-once increase latency?
Yes, in proportion to how often you commit. Confluent measured a 3 percent producer throughput loss for Kafka transactions against ordered at-least-once, and 15 to 30 percent for Kafka Streams at a 100 millisecond commit interval. Flink’s barrier alignment adds milliseconds, and Google says Pub/Sub exactly-once subscriptions carry significantly higher latency.
When is at-most-once delivery the right choice?
When a lost record costs less than a doubled one and fresh data replaces it soon anyway. Metrics samples, heartbeats, and telemetry overwritten every few seconds fit. Kafka gives it to you by disabling producer retries and committing offsets before processing, and RabbitMQ gives it to any consumer that skips acknowledgments.
Is Postgres logical replication exactly-once?
Only while nothing crashes. The Postgres docs say a logical slot emits each change once in normal operation, but the slot position is persisted only at checkpoint, so after a server crash it can rewind and resend recent changes. The docs place the duty on the client, which must tolerate the same message twice.
More articles
Questions
Answered
FAQ
What does Galaxy do?
What is Filament?
What is enterprise context management?
What does working with Galaxy look like?
How do you handle security and compliance?
Why does Galaxy build in the open?
Company
Copyright © 2026 Galaxy. All rights reserved.
