Bulk Batch Replication EAP

Bulk Batch Replication is a batch (polling) sibling of the streaming Realtime Export and Delta Lake Export paths. Instead of forwarding one resource change at a time through a broker channel, a clustered scheduled trigger periodically replicates a whole window of HFJ_RES_VER (resource version) rows in bulk to a configured target repository.

Where the streaming path optimizes for low latency (each change is written as it happens), Bulk Batch Replication optimizes for write throughput: it coalesces many resource changes into large FHIR transaction Bundles and hands each Bundle to a single repository call, so the target receives few large writes rather than many small ones. This suits targets where large, infrequent writes are preferable to a steady stream of single-row operations.

The replication target is any implementation of the HAPI FHIR ca.uhn.fhir.repository.IRepository interface, selected by configuration. Any IRepository works — an analytical data lake (for example, an Iceberg- or Delta-backed repository) is one motivating use case, but the module makes no assumption about what the target does with the Bundles. Out of the box the target defaults to a no-op repository that discards everything, so the module is inert until a real target is configured.

Module EAP

Bulk Batch Replication is configured as its own module type (REALTIME_EXPORT_BULK_BATCH_REPLICATION), independently enabled per deployment. It depends on a persistence module (the source of the HFJ_RES_VER rows) and on the Cluster Manager module (for clustered scheduling).

The persistence module must be a relational one — extraction reads the HFJ_RES_VER table directly, which MongoDB persistence does not have. Wiring the module against a MongoDB persistence module fails the module's startup with a configuration error.

See the Bulk Batch Replication module reference for the module's configuration categories and dependencies.

Architecture EAP

A cluster-singleton scheduled trigger fires on a fixed interval. Each fire starts one gated Batch2 job, so replication runs benefit from Batch2's retry, gating, and observability; the trigger only schedules, it does not do the extraction itself.

Scheduled trigger (every schedule_interval_ms, on one node in the cluster)
        → start one Batch2 job
             ├ fan out: one unit of work per partition (or shard)
             ├ per partition: drain the resource-version rows written since the last run,
             │   in pages; each page → one FHIR transaction Bundle → target repository;
             │   record durable progress (a watermark) after every written page
             └ report: rows written, partitions, recovered errors

Copy

Extraction: resource version history

Each run reads directly from the resource version history (HFJ_RES_VER) using a resumable cursor that is exactly-once at each page boundary: the next run continues precisely where the previous one stopped, never skipping or repeating a row, even when many versions are written in the same instant under concurrent load.

Because extraction is against the version history table (not current state), every version of a resource is replicated, not just its latest state. The lake accumulates full history, which suits analytical, append-oriented tables.

Output: FHIR transaction Bundles

Each page of version rows is assembled into one FHIR transaction Bundle and submitted in a single repository call, so the target batches its writes rather than applying changes one row at a time. Live versions become versioned updates and deleted versions become deletes, so both updates and deletes propagate: for each resource, the target reflects the source's full history, ending at its current state.

Partitioning and MegaScale

The job replicates every partition independently and in parallel (per shard under MegaScale); a failure in one partition is reported without failing the others in the same run.

Narrowing the scope with partition_scope partition_scope restricts replication to specific partition ids; leaving it empty replicates all partitions. It requires a partitioned persistence module — configuring a scope without partitioning enabled fails module startup, since such a scope could never match any rows.

A partial scope (some but not all tenants of a shard) is more expensive to drain than a whole-shard scope, because it filters on a column the database does not index by partition. The cost is bounded and matters mainly during the initial backfill (whose window is the entire table), not in steady state. For a large existing deployment, prefer backfilling a partial scope deliberately — leave the schedule disabled and drive it with $sdh.bulk-batch-replication-start, then enable the schedule once the watermark has caught up. Adding a partition to the scope later re-replicates that partition's history (harmless for idempotent targets, but expect a backfill-sized run). Changing the shard topology The set of shards (and each shard's co-located partitions) is snapshotted when a run starts, and a whole-shard drain records its watermark against that shard's set of co-located partition ids. As a result, changing the MegaScale topology while replication is running is not supported:

• A shard (database) added mid-run is not replicated by that run. The next run picks it up and drains it from the beginning of its history.
• Adding a partition to an existing shard changes that shard's partition-id set, which re-keys its watermark. The next run therefore re-drains that shard's entire history, and the shard's previous watermark Task is left behind unused.

Apply topology changes with replication quiesced: disable the schedule (enabled=false), confirm no run is in flight, make the change, then re-enable. Expect the next run to be backfill-sized for any shard whose partition membership changed — size stale_instance_threshold_ms accordingly, or drive the catch-up manually with $sdh.bulk-batch-replication-start before re-enabling the schedule.

Overlap protection and orphaned instances

Replication runs never pile up: a new run will not start while a prior one is still in flight (this applies to both the schedule and the manual start operation). A run left behind by a crashed or restarted node is automatically treated as orphaned after stale_instance_threshold_ms and cancelled so a fresh run can start — without losing progress, since each written page is durably recorded and the replacement resumes from there.

Because the cluster-wide guard is best-effort, avoid issuing a manual start around a scheduled fire (or disable the schedule while running a manual backfill): two runs briefly overlapping cannot corrupt the target, but they redundantly re-replicate the same rows.

Observability

Because each run executes as a Batch2 job, its progress, status, and history are visible through the standard Batch2 job instance surface. In addition, the module emits the following OpenTelemetry metrics, each tagged with smilecdr.module_id:

Per-partition progress, drain summaries, and final job status are also written to the Realtime Export troubleshooting log.

Manual Trigger Operation EAP

Besides the recurring schedule, a run can be started on demand with the FHIR operation $sdh.bulk-batch-replication-start at the server (system) level. This is useful for ad-hoc backfills and for QA. The operation enforces the same overlap guard as the scheduler — it fails with a precondition error if a prior instance is still in flight.

When more than one bulk-batch-replication module is configured, each is an independent replication target. Identify the one to run with the target parameter, set to that module's id. The parameter may be omitted when exactly one module is configured; if several are configured and target is omitted, the operation fails and lists the configured targets. An unknown target fails with a not-found error.

The enabled setting gates only the recurring scheduled trigger. The module and the manual start operation remain available even when the schedule is disabled, so an operator can keep ad-hoc triggering while leaving the recurring schedule off.

Configuration EAP

All settings are backend-agnostic and apply regardless of the repository target.

module.bulk_batch_replication.config.enabled=false
module.bulk_batch_replication.config.schedule_interval_ms=900000
module.bulk_batch_replication.config.stale_instance_threshold_ms=2700000
module.bulk_batch_replication.config.trailing_lag_slop_ms=300000
module.bulk_batch_replication.config.page_size=500
module.bulk_batch_replication.config.partition_scope=
module.bulk_batch_replication.config.target_repository=ca.cdr.rte.repository.NoOpRepository

Copy

Choosing the duration values

Three settings are durations. Pick each from your deployment's characteristics rather than accepting the defaults blindly.

• schedule_interval_ms — your replication latency floor. A newly written row first appears in the target after at most one interval plus trailing_lag_slop_ms. Shorter intervals give fresher data and smaller per-run windows but run more often; longer intervals give fewer, larger runs. Keep the 15-minute default for analytical targets; drop to a few minutes only if you genuinely need fresher data and the target can absorb more frequent writes. There is no benefit to setting it below how long a typical run takes — runs never overlap (a fire is skipped while a prior run is still going), so an over-short interval just yields back-to-back runs.

• trailing_lag_slop_ms — how far back each run stays clear of "now". It only has to cover the brief moment between a row being stamped and its database transaction committing, plus any clock skew between the application and database servers — seconds in a healthy, time-synchronized deployment. It does not need to cover the duration of your longest transaction. The 5-minute default is deliberately conservative; if you want lower latency and your app and database clocks are synchronized (e.g. NTP), lowering it to 30–60 s is safe. Raise it only if you observe recently-written rows intermittently missing from a run and appearing in the next one — a sign the real gap is larger than configured (for example, unsynchronized clocks).

• stale_instance_threshold_ms — how long a run may look stalled before it is assumed dead and cancelled. Set it above your longest legitimate run, which is almost always the initial backfill rather than a steady-state run. Too low, and a healthy long backfill gets cancelled and restarted; too high, and recovery from a genuinely crashed run is delayed by that much. Size it from an observed or estimated worst-case backfill duration with generous headroom — the 45-minute default suits moderate datasets; increase it for very large initial loads.

Repository target

The target is resolved from target_repository, which accepts either form:

• A fhir-repository: URL, resolved by HAPI's repository loader SPI — for example a filesystem or in-memory target (fhir-repository:exp-kalm-filesystem:/path, fhir-repository:memory:name). Because replication emits every resource version as a separate transaction-Bundle entry, the target must accept a bundle carrying multiple versions of the same resource; lake-style (append/filesystem) loaders do, so those are the appropriate URL targets.
• A fully-qualified class name of a public class implementing ca.uhn.fhir.repository.IRepository, constructable via the standard customer-bean mechanism (a no-argument or autowired constructor; a FhirContext is offered as an available dependency).

The default NoOpRepository accepts and discards every Bundle, which keeps the module inert until a real target repository is configured.

Limitations EAP

• Out of the security perimeter. Like the streaming Realtime Export path, replication egress bypasses the authorization, consent, and audit perimeter. Replicated data is written to the target repository without per-request authorization or consent enforcement.
• $expunge does not propagate. A physical expunge removes rows from HFJ_RES_VER, so there is no version row left for the cursor to observe. Expunged resources are not deleted from the target. This is the same limitation as the streaming path.
• Latency is measured in minutes. Replication latency is at least the schedule interval plus the trailing-lag slop. This suits bulk, analytical targets (a data lake being a typical example) but is not a low-latency replication path — use the streaming Realtime Export path when near-real-time replication is required.
• Full history is replicated. Every resource version is emitted as a separate Bundle entry. Targets that only want current state must collapse history themselves (for example, an upsert-if-newer repository implementation).
• Resources written together can arrive separately. A large set of resources written in one transaction may be replicated in several batches, so the target may briefly hold only some of them. The rest arrives during the same run, or on the next run if a run is interrupted.
• The data is normally complete to a moment before the most recent data. All data before the current highest timestamp will be complete.

MegaScale limitations

• MegaScale topology changes during replication are not supported. Shards (databases) and partitions must not be added while replication is running: a shard added mid-run is skipped by that run, and adding a partition to an existing shard re-keys that shard's watermark, causing the next run to re-drain its full history. Quiesce replication before changing the topology — see Changing the shard topology.
• The target is not complete to a single point in time. Each MegaScale database replicates independently, so different parts of the data reach the target at different times. There is no instant at which the target can be considered complete across all shards.
• The target receives concurrent writes, one stream per MegaScale database. Targets that serialize writes scale poorly as shards are added. If the target is an analytical table format such as Iceberg, partition its tables along the same lines as the CDR partitions, so concurrent writes do not contend for the same table partition.

Delta Lake Export --EAP--

Bulk Batch Replication --EAP--

Realtime Export

Realtime Export Overview Bootstrapping RTE Data Realtime Export Rules Definition Using FHIRPath Delta Lake Export --EAP-- Bulk Batch Replication --EAP--