Microsoft Fabric Structured Streaming: Recover Without Duplicate or Missing Data

Executive summary

A real-time pipeline is not reliable merely because its job status is Running. After a worker failure, maintenance event, credential problem, code deployment, or capacity delay, the harder question is whether the service resumes from the right event, produces each business outcome once, and catches up before the source removes unread data.

Microsoft Fabric provides the foundations for that recovery model. Spark Structured Streaming can record offsets, progress, and state in a durable checkpoint; Delta Lake can coordinate transactional writes; Spark job definitions can supervise production restarts; and Fabric's monitoring experience exposes input rate, process rate, batch duration, input rows, and operation duration. However, the guarantee is end to end only when the source is replayable, each query owns its checkpoint, incompatible changes receive a deliberate migration plan, and every nontransactional effect is idempotent.

For enterprise leaders, the practical response is to govern each production stream as a service with a streaming recovery contract. That contract should define replay boundaries, checkpoint ownership, state compatibility, output guarantees, late-data policy, recovery objectives, security, capacity, and named operational ownership. This article describes how to implement that contract in Microsoft Fabric.

The enterprise problem: green jobs can still produce wrong outcomes

Streaming has moved beyond dashboards. Enterprises use event flows to prioritize cases, detect service failures, update inventory, identify suspicious transactions, and trigger operational interventions. That raises the consequence of a subtle engineering error: a replayed microbatch can create a second ticket, notification, or payment instruction, while a misplaced or reset checkpoint can omit events without making the pipeline visibly fail.

Microsoft documents that a Structured Streaming checkpoint stores query progress, source offsets, and state so a restarted query can resume. Microsoft also cautions that exactly-once processing depends on the source, sink, checkpoint, and query logic; Delta and file sinks support exactly-once writes, while custom and `foreach` writes need idempotent handling because retries can occur. In other words, "exactly once" is not a property that can be assigned to the whole business process simply because Spark is present. It is an outcome that the complete path must earn. See Microsoft's Structured Streaming overview and best-practice guidance .

The problem matters now because Fabric supports both continuously running queries and scheduled incremental processing with an `availableNow` trigger. The same query can move between those operating patterns when its checkpoint remains compatible. That flexibility is valuable, but it also makes recovery state an architectural asset: deployment teams must know when state can be reused, when it must be replaced, and how a new checkpoint will obtain a correct starting position.

Representative scenario: an order stream that is operationally critical

*The following is a representative scenario, not a YuniQ customer case or a report of actual results.*

A multichannel retailer receives order-status events from stores, web checkout, fulfillment centers, and carriers. A Fabric streaming job parses those events, updates a current-order table, feeds an operations dashboard, and asks a downstream service to create an intervention when a high-value order remains unfulfilled beyond its promised window.

The pilot performs well. In production, four failure patterns emerge:

1. A transient sink failure occurs after an intervention is created but before the microbatch is recorded as complete. Spark retries the batch, and the downstream service creates the intervention again.
2. A developer deploys a revised aggregation against the existing checkpoint. The state schema is incompatible, so the stream enters a restart loop.
3. An operator deletes the checkpoint to make the job start. The job runs, but nobody has defined whether it should replay retained events or start at the latest position.
4. During a capacity-constrained period, processing falls behind ingestion. The job remains active, yet the operational view is too old to support the promised service level.

These are not four unrelated defects. They are symptoms of one missing control: the organization has deployed streaming code without defining a recovery contract for the business service.

Root causes and consequences

Recovery state is treated as disposable runtime debris

A checkpoint is part of the deployed query. It binds source progress, query structure, state schema, and output behavior. Microsoft explicitly warns not to share checkpoint directories between queries because doing so can corrupt recovery metadata and cause skipped records, duplicate processing, or failed restarts. Microsoft also advises using a durable lakehouse path rather than local driver storage. A checkpoint therefore needs an owner, access control, retention logic, and a release decision; it is not a cache that operators should casually delete.

Transport delivery is confused with business-effect uniqueness

A Delta write can be transactional while a later API call, email, workflow creation, or multi-destination `foreachBatch` operation is not. Microsoft notes that `foreachBatch` is at-least-once by default and that its writes are not one atomic transaction across all destinations. If a later destination fails, Spark can retry the complete batch. Without a stable event key or destination transaction identifier, the replay can repeat an external effect. See Microsoft's common Structured Streaming patterns .

Stateful logic is allowed to grow without a boundary

Deduplication, windowed aggregation, and stream-to-stream joins retain information between microbatches. If the design lacks watermarks, time-bounded joins, time-to-live rules, or appropriately granular keys, that state can grow continually. Larger state increases memory pressure and recovery work. Fabric supports the RocksDB state store and changelog checkpointing to reduce JVM heap pressure and checkpoint latency, but those mechanisms do not replace an explicit data-retention rule. Microsoft's stateful processing guidance recommends bounding state and testing late, duplicate, and restart scenarios.

Operations monitors process health instead of decision health

A running process can be stale. Input rate, process rate, and batch duration must be interpreted together. When process rate remains below input rate, backlog grows even if the stream has not failed. The resulting consequences include late interventions, inconsistent dashboards, duplicate operational work, manual reconciliation, audit gaps, missed source-retention windows, and avoidable capacity spend.

Why common approaches fall short

"Just enable retries"

Retries are necessary for transient failure, but they amplify deterministic defects. Invalid credentials, incompatible state, or malformed deployment configuration will not improve on the twentieth restart. Microsoft recommends surfacing terminal query exceptions to the parent job, alerting on repeated failures, and requiring intervention for deterministic problems rather than restarting indefinitely.

"Delete the checkpoint and restart"

This restores motion, not correctness. A new checkpoint resets the query's knowledge of processed input. Unless the team explicitly selects and validates a starting offset or replay source, it cannot prove what was duplicated or missed.

"Put every output in one `foreachBatch` function"

That creates shared fate without cross-system atomicity. The first destination may commit before the second fails. On retry, both run again. Every destination must be idempotent, or the design should separate durable ingestion from independently recoverable consumers.

"The notebook ran successfully"

Streaming queries start asynchronously. Microsoft warns that an active query does not by itself keep a triggered notebook or Spark job definition alive. Production code must wait for termination and propagate terminal failures. Fabric recommends Spark job definitions, rather than development notebooks, for continuously running production streams, with a suitable retry policy.

"Scale the cluster until the lag disappears"

More compute can help, but poor partitioning, unbounded state, slow sinks, excessive per-record work, and small-file overhead remain. Scaling without measuring the limiting stage can raise cost while leaving the recovery objective unmet.

The solution: a streaming recovery contract on Microsoft Fabric

The contract should be short enough to operate and precise enough to test. Create one record for every production query, not merely for every application.

Reference architecture

Replayable event source
        |
        v
[Query A: durable ingestion] ---- dedicated checkpoint A
        |
        +----> Bronze Delta: immutable payload + source metadata
        |                |
        |                +----> Quarantine Delta: invalid payloads + reason
        v
[Query B: validate, deduplicate, conform] ---- dedicated checkpoint B
        |
        v
Silver Delta: canonical event and current business state
        |
        +----> Gold metrics / semantic model / operational dashboard
        |
        +----> Action outbox Delta: command + stable idempotency key
                            |
                            v
                 Idempotent action publisher
                            |
                            v
                 External workflow or API
                            |
                            v
                 Receipt / outcome ledger

This design introduces a small amount of storage and latency in exchange for independent recovery domains. Microsoft describes "ingest once to a Bronze Delta table" as an appropriate pattern when downstream streams need durable replay, different schedules, or separate failure domains. A slow external destination no longer blocks source capture, and the raw event remains available for reconciliation and controlled reprocessing.

Land first, then fan out

Use a narrow ingestion query to persist the source payload, source event ID, source partition and offset where available, event time, ingestion time, schema version, and trace identifier. Preserve invalid raw payloads rather than discarding them during parsing. Downstream transformations should read the Bronze Delta table with their own checkpoints.

For downstream changes and deletes, choose the change propagation design deliberately. Microsoft notes that standard Delta streaming reads expect append-only commits; use Delta Change Data Feed when downstream streams must receive updates or deletes.

Give every query one durable checkpoint

Assign a unique, stable checkpoint path for the life of the query. Restrict modification and deletion to the production operating role, because checkpointed state can contain information derived from business records as well as operational metadata. Monitor checkpoint growth and last successful progress.

Do not copy a production checkpoint into development. Test recovery with production-shaped data and an isolated checkpoint. Before reusing state after a release, assess changes to the source, sink, grouping keys, state schema, and stateful operators. An incompatible release should receive a new checkpoint and an explicit starting-point or replay plan.

Make business effects idempotent

Idempotent means that repeating the same command produces the same final business result as processing it once. Derive the key from a stable business event identifier, not processing time or microbatch sequence alone. Store that key with the requested action and the destination receipt.

For Delta-only `foreachBatch` writes, stable transaction identifiers can make a repeated batch safe. When a batch writes to several destinations, remember that the destinations do not commit as one transaction. A durable action-outbox table separates analytical state changes from external side effects. The publisher can then upsert or submit with the same idempotency key until a receipt is recorded.

Bound state and late-data behavior

Define how late is still useful for the business decision. Apply event-time watermarks and time-range join conditions where that rule permits. Set TTL for custom state and select keys that do not create uncontrolled cardinality. Use RocksDB and changelog checkpointing where appropriate to improve state-store behavior, but retain business-led state limits.

Late data should have an explicit route: update the current view, revise only historical measures, enter an exception queue, or be rejected after evidence is retained. This is a policy decision, not a tuning detail.

Supervise production execution

Package continuously running production code as a Spark job definition and configure restart behavior appropriate to the service. After starting a query, call `awaitTermination()`; for multiple queries, use a pattern that detects one query stopping and fails or stops the remaining application cleanly. A terminal streaming exception must not be caught and returned as a successful job.

Treat restart loops as incidents. Separate transient source, sink, network, or infrastructure failure from invalid credentials, incompatible schema, corrupted state, and other deterministic causes. The former may recover automatically; the latter needs diagnosis and controlled change.

Monitor freshness, recovery, and correctness together

Use Fabric's Structured Streaming metrics for technical behavior, then add business-level evidence:

• Decision age: current time minus the event time of the newest fully published business result.
• Backlog: source high-water mark minus the latest committed input position, expressed in records and time.
• Duplicate-effect rate: external actions rejected or suppressed by an existing idempotency key divided by action attempts.
• Reconciliation gap: source events minus accepted, quarantined, and explicitly excluded events for the same control window.
• Recovery time: time from terminal failure to a healthy stream at the agreed freshness threshold.

Input rate, process rate, batch duration, input rows, and operation duration are available in Fabric's Structured Streaming monitoring experience. Alert on their relationship, not isolated values. A useful condition is sustained process rate below input rate combined with increasing decision age.

A phased implementation roadmap

Phase 1: Discover and define the service

Inventory production streams, destinations, checkpoints, job owners, credentials, source-retention periods, and downstream actions. Select one operationally important flow. Establish its recovery contract and classify every sink as transactional, idempotent, or at-least-once.

Exit evidence: an approved contract, source-to-outcome lineage, named owner, baseline decision age, and a list of unprotected side effects.

Phase 2: Build a recoverable thin slice

Create the Bronze Delta landing path, quarantine path, dedicated checkpoints, canonical event key, and action ledger. Run the job as a Spark job definition. Add technical and business telemetry before adding complex transformations.

Exercise failure at awkward points: before a Delta commit, after one destination commits, during a source interruption, during a sink interruption, and after abrupt application termination. Reconcile source, Bronze, Silver, action, and receipt counts.

Exit evidence: repeated recovery tests produce no unexplained loss or duplicate business effect, and the service catches up inside its agreed objective.

Phase 3: Establish release gates

Classify each code change as checkpoint-compatible, uncertain, or incompatible. Use an isolated test checkpoint with production-shaped volume, duplicates, late events, poison records, and restarts. For incompatible changes, document the new checkpoint, starting position, parallel-run window, reconciliation query, cutover, and rollback path.

Spark job definition Git integration can version source and item definitions, but Microsoft currently labels that integration as preview and documents manual considerations for environment and lakehouse references across workspaces. Treat runtime state and stage-specific bindings as separately controlled release assets. See Spark job definition Git integration .

Exit evidence: a release cannot reach production without a state-compatibility decision and a passed restart/reconciliation test.

Phase 4: Scale the operating model

Apply a standard contract template, checkpoint naming convention, owner register, dashboard, and runbook across domains. Separate high-criticality streams from experimental workloads where contention would threaten recovery. Review error budgets, state growth, replay duration, source retention, and cost per accepted event at a regular service review.

Exit evidence: every critical stream has measurable freshness and recovery objectives, tested ownership, and traceable release evidence.

Governance, security, adoption, and cost considerations

Governance and security

Grant least privilege separately for source read, checkpoint write, Bronze/Silver write, Gold read, and action publication. Keep development and production checkpoints, data, and identities separate. Record who can change source positions, delete checkpoints, alter retention, or approve an incompatible release. Apply classification and retention controls to raw, quarantined, and checkpoint-derived state according to the sensitivity of the underlying data.

Retain an audit chain from source event ID through canonical record, command idempotency key, destination receipt, code version, and checkpoint version. For a regulated process, this chain is often more useful than a screenshot showing a successful job run.

Adoption and operating model

Streaming reliability crosses platform, data engineering, source-system, application, security, and business-operations ownership. Assign one service owner who can convene those teams. Train support staff to distinguish replay, retry, restart, and reprocess; these actions are not interchangeable.

Make quarantine an owned queue, not a storage graveyard. A schema owner should classify invalid events, correct producer or consumer logic, and replay recoverable records within an agreed time.

Capacity and cost

Choose the execution pattern from the decision requirement. A continuously running query minimizes scheduling delay; an `availableNow` query processes the current backlog and releases compute, which may be more efficient when periodic freshness is sufficient. Short microbatch intervals reduce latency but tend to create smaller Delta files and more write overhead; longer intervals trade freshness for batching efficiency.

Spark resources are shared by workspaces on a Fabric capacity. Scheduled Spark job definitions can queue when cores are unavailable, while capacity throttling can reject new work. A recovery plan therefore needs reserved headroom or workload isolation sufficient to drain backlog, not merely enough compute for normal volume. Microsoft's Spark concurrency and queueing documentation explains admission, bursting, and queue behavior.

For financial control, measure compute against accepted business events and recovery drills. The Fabric Spark billing guidance distinguishes shared Fabric-capacity consumption from autoscale billing and recommends using the Capacity Metrics app together with Azure Cost Analysis. Validate the chosen model against steady-state load, surge recovery, and the risk of an uncontrolled restart loop.

Success metrics

Set thresholds by service criticality; do not impose one latency target on every stream.

How YuniQ can help

YuniQ's published Microsoft Fabric consulting and implementation services align directly with this recovery problem.

• Fabric Strategy & Readiness covers current-state workload diagnostics, capacity planning, workspace topology, security, governance, operating-model design, and proof-of-concept scoping. Those activities establish the service criticality, ownership, capacity, and recovery objectives before implementation.
• Data Integration & Engineering includes streaming sources, Lakehouse and Data Warehouse implementation, Apache Spark and Delta frameworks, data-quality observability, and exception handling. Those capabilities support the Bronze landing, dedicated recovery paths, quarantine, and idempotent data processing described here.
• OneLake & Data Architecture includes domain workspaces, Bronze/Silver/Gold medallion layers, and open Delta storage patterns. That is the architectural basis for durable replay and independently recoverable downstream consumers.
• Governance, Security & DevOps includes least-privilege role design, lineage, Dev/Test/Prod separation, Git and CI/CD pipelines, automated monitoring, and capacity threshold alerts. Those controls turn checkpoint compatibility, access, release evidence, and capacity health into repeatable operations.
• Analytics, AI & Activation includes near-real-time streaming analytics and operational action alerts. The recovery contract helps those outcomes remain timely without repeating downstream business effects.

YuniQ's delivery framework also describes discovery, architecture, proof with real data, iterative build, automated reconciliation, access testing, dual-run comparison, launch, and ongoing optimization. Applied to a critical stream, that is a practical sequence for proving recovery behavior before expanding the pattern across domains.

Practical next steps

1. Choose one stream that triggers an operational decision, not the easiest technical demo.
2. Trace one source event through every Delta table, metric, command, API call, and human workflow.
3. Complete the streaming recovery contract and identify any destination that cannot suppress a duplicate command.
4. Verify that each query has its own durable checkpoint and that deletion requires production change control.
5. Measure decision age and reconciliation gaps alongside Fabric's processing metrics.
6. Run an abrupt-termination exercise and prove the complete business outcome after restart.
7. Classify the next release for checkpoint compatibility before deploying it.

The executive test is simple: after any allowed failure or deployment, can the organization show which event was last committed, which input remains replayable, which outputs may repeat, how duplicates are suppressed, and when the business decision will be current again? If the answer is documented and tested, streaming has become an enterprise service. If not, a green job is only a process status.