Oracle Database In-Memory vs. ShannonBase Rapid

This note compares two column-store accelerators: Oracle Database In-Memory (DBIM) and ShannonBase Rapid. The goal is not to rank them. It is to show that they are not the same kind of object, and that this explains why Rapid carries mechanisms that look redundant at first glance: its own per-row WAL, an MVCC version chain, external-abort reconciliation, and table-level quarantine.

Sources

Ref	Source
[O]	Oracle, Oracle Database In-Memory with Oracle Database 19c — Technical Overview, February 2019
[H]	MySQL HeatWave documentation, About Change Propagation (DML does not wait; batched propagation; 200 ms / 64 MB / read-triggered)
[C]	ShannonBase source under storage/rapid_engine/** (file paths cited inline)

0. Summary

1. Rapid’s model = HeatWave’s change-propagation model + Oracle-IM-style column-store structures. It is not a copy of Oracle DBIM: the two make opposite choices on the central question of how a change reaches the column store.
2. Oracle: the column store’s values are never rewritten. A DML only registers row-level staleness (the rowid is recorded in the SMU); the new value lives in the row store alone, and visibility is decided by SCN at scan time. The commit itself does not write to the column store. This is why [O] can state that the column store needs no additional locks or logging.
3. Rapid: the column store’s values are written in place. Propagation is asynchronous: a row image enters the column store as an ACTIVE version (commit_scn == 0), and an observer at the server’s transaction boundary later flips its final state. The column store can therefore hold uncommitted bytes (mechanism in §3.3).
4. That difference forces three costs on Rapid: it must ship its own per-row WAL; it must reconstruct visibility with MVCC version chains; and invalidation can only be contained by table-level quarantine plus reload, where Oracle does row-level staleness plus IMCU-level rebuild.
5. Rapid’s propagation scheduling is HeatWave-style, with three triggers (200 ms / 64 MiB / query demand) — not Oracle’s staleness threshold plus a two-minute trickle repopulate.

1. Positioning and overall architecture

	Oracle DBIM [O]	ShannonBase Rapid [C]
Form	Dual-format inside one instance: buffer cache (rows) + In-Memory Area (columns); a single copy on storage	A MySQL 8.4 secondary engine (in-process plugin ha_rapid); the primary storage remains InnoDB
Data source	Base tables (populated on demand)	Read from InnoDB (SECONDARY_LOAD → ha_rapid::load_table)
Maintenance	The kernel itself keeps the row and column formats consistent	The plugin maintains everything: change propagation, its own WAL, its own recovery
Read/write split	The optimizer routes OLAP to columns and OLTP to rows	Controlled by the three-state use_secondary_engine; with ON the choice is still arbitrated from load (SecondaryEnginePrePrepareHook → Query_arbitrator), and only FORCED skips arbitration (see §9.1)
Memory overhead	Paper states dual-format overhead is under 20%	Not measured on that basis; the column store sits outside the InnoDB buffer pool

Rapid’s handler does not implement write_row / update_row / delete_row (handler/ha_shannon_rapid.h, class ha_rapid). Every write goes to InnoDB; new values can only reach the column store through change propagation.

2. Memory layout and metadata

Item	Oracle [O]	Rapid [C]
Memory region	The SGA’s In-Memory Area, split into a 1 MB pool (columnar data) and a 64 KB pool (object metadata)	MemoryPool + Imcs / Imcu on the process heap; BkgWorkerPool provides background threads
Sizing	INMEMORY_SIZE (default 0, minimum 100 MB); can grow online (≥128 MB per step), cannot shrink online; not managed by AMM	Engine config shannon_rpd_engine_cfg (including pop_buff_sz_max), set at plugin init
Unit	IMCU: the unit of compression, of pruning, and of FastStart write-out	Imcu: fixed capacity, holding StorageIndex + del_mask + null_masks + TransactionJournal + RowDirectory
Column structure	Columns are organised inside each IMCU	One CU per column (secondary fields only), indexed directly by local_row_id
Per-row metadata	Only the stale rowid list in the SMU	RowDirectory::RowEntry (offset/length/checksum plus deleted/has_null/is_overflow flags, sharded locks) + the bit_array_t del_mask
Pruning metadata	Per-IMCU, per-column min/max + dictionary	StorageIndex: numeric min/max, lexicographic string min/max, null statistics; Compress::Dictionary
Compression	Six MEMCOMPRESS levels — DML / QUERY LOW (default) / QUERY HIGH / CAPACITY LOW (OZIP) / CAPACITY HIGH — per column; queries run directly on compressed data	CU_CompressAlgo{NONE, LZ4, ZSTD, ZLIB} + Compress::ENCODING_TYPE + COMPRESS_LEVEL; dictionary encoding; variable-length columns use VarlenDataPool

3. How changes reach the column store (the central difference)

3.1 Oracle: metadata is registered during the DML, values are never rewritten

From [O]:

When a DML statement changes a row in an object that is populated in the IM column store, the corresponding column entries for that row are marked stale in the IMCU and the rowid is added to the metadata in the SMU. The original entries in the IMCU are not immediately replaced in order to provide read consistency and maintain data compression. Any transaction executing against the object in the IM column store, that started before the DML occurred, can still see the original version of the entries in the IMCU. Read consistency in the IM column store is managed via System Change Numbers (SCNs) …

When a query with a newer SCN is executed against the object, it will read all of the entries for the columns in the IMCU except the stale entries. The stale entries will be retrieved from the base table (i.e. the row store).

Three distinct things therefore happen at three distinct times:

Action	When	Note
Register staleness / SMU entry (metadata)	During the DML, before commit	Part of the transaction’s change record
The value inside the IMCU	Never rewritten	It stays as populated until the IMCU is rebuilt
The change becoming visible to readers	At commit (via SCN)	Commit does not write to the column store; it only makes the SCN visible

This mechanism is row-level mixing: one query scans IMCUs and, for stale rows, follows the rowid into the base table (row store) for the current version — the mixing happens per row. Rapid has no such path: its choice is made per statement (see §9.2).

3.2 Rapid: asynchronous in-place writes plus a version chain

The chain (populate/log_dml_notification.{h,cpp}, imcs/cu.cpp, imcs/imcu.cpp):

1. Capture. The primary engine’s DML notifier (NotifyAfterInsert/NotifyAfterUpdate/NotifyAfterDelete) builds a change record on the write path and enqueues it. The contract, verbatim:

   the record leaves this call carrying the real primary InnoDB creator id and commit_scn == 0, i.e. it enters Rapid MVCC as an ACTIVE notification version; the outcome published on primary COMMIT/ROLLBACK is what finalizes it.

2. Asynchronous apply. A per-table worker writes the row image in place into the CUs (new value plus a before-image version chain; mechanism in §3.3). CopyInfoParser::apply_change() handles three races explicitly:

   • ABORTED arrives first → do not manufacture a version at all;
   • COMMITTED arrives first → build it directly as COMMITTED with terminal_scn;
   • neither yet → m_scn = record.m_commit_scn (0 = ACTIVE, until the primary outcome arrives).

3. Finalize. The Trans_observer rapid_transaction_observer dispatches rapid_after_commit / rapid_before_rollback, which publish or undo at the server’s transaction boundary.

3.3 How uncommitted bytes are stored (the MVCC version chain)

“Uncommitted” is not a property of the bytes; it is a property of the version node that sits next to them. Every CU owns a ColumnVersionManager (imcs/cu.h) holding two things:

• a per-row version chain: m_versions, an unordered_map<row_id_t, unique_ptr<Column_Version>> whose head is the newest version and whose prev links walk back in time;
• a per-writer ACTIVE index: m_active_versions, an unordered_map<Transaction::ID, vector<ActiveVersionRef>> recording which nodes are still ACTIVE for that transaction (each node carries its active_idx, making removal O(1)), so commit and rollback cost O(k) rather than a scan.

In addition, m_versioned_rows is an atomic mirror of m_versions.size(): when it is zero a reader skips the hash lookup entirely. That fast path (has_any_version()) is what keeps the per-row, per-column call site cheap on columns that have no update history at all.

Write side: the new value lands in the slot, the old value is pushed onto the chain

CU::update() does both under the same m_data_mutex: it stages the whole new image and publishes it in one step (publish_slot, because readers do not take that mutex), and it pushes the old value as a new head node for that row.

Each version node (Column_Version) stores:

Field	Meaning
txn_id	The real InnoDB writer id; used for read-your-own-writes and InnoDB ReadView queries
scn	0 = ACTIVE (uncommitted); rewritten to the physical commit SCN on finalize
timestamp	When the version was registered
old_value + value_length	The logical before-image: a copy of the old bytes; UNIV_SQL_NULL means “was NULL”. Used for stable serialization and recovery
old_slot	The normalized physical before-image: the old dict-id for dictionary columns, the old VarlenReference for variable-length ones; consumed directly by the vectorized batch path
retained_pool / owns_varlen_ref	Ownership of a pooled variable-length old value: the block cannot be retired while any ReadView might still select it, so ownership moves to the node and is returned when the node dies
prev	The next older version

So the “uncommitted bytes” in the column store are really two things: the new value (already written in place into the CU slot) and the old value (hanging on the chain head). What marks the state as uncommitted is not the bytes but the node’s scn == 0 plus the registration in m_active_versions.

Read side: walk the chain from newest to oldest and pick what the snapshot should see

get_before_image_for_snapshot(row_id, reader_txn_id, reader_scn, reader_trx, table, out) walks from the chain head, decides creator_visible for each version, and stops at the first visible one; each invisible version in between overwrites the output with its before-image, reconstructing the version the snapshot should see:

• reader_txn_id == version->txn_id → its own write, visible (read-your-own-writes);
• otherwise, for SQL readers: visible when txn_id == 0, or when the primary InnoDB ReadView says so via changes_visible(txn_id, table). The code comment spells out the intent: even while Rapid’s asynchronous outcome callback has not yet turned scn == 0 into a real commit SCN, the InnoDB ReadView is authoritative. This is the concrete form of the “ACTIVE-version visibility is resolved by the primary InnoDB ReadView” note quoted in §5;
• for non-SQL callers without a ReadView (internal / recovery time travel): scn == 0 → invisible, otherwise scn <= reader_scn → visible.

Finalize and reclaim

Action	Behaviour
commit	commit_transaction(txn_id, commit_scn) rewrites scn from 0 to commit_scn for every node of that transaction and drops the ACTIVE registration; from then on those versions take part in SCN comparisons
rollback	pop_head(row, expected_txn_id) pops the head, writes old_slot back into the slot and returns variable-length ownership. Ordering is m_data_mutex first, then the version head, matching update(); the comment explains that the reverse order opened a window in which the row had no version entry while the slot still held the uncommitted value, letting a concurrent reader see uncommitted data as committed
purge	purge(min_active_scn) never reclaims ACTIVE versions (scn == 0); only committed versions below min_active_scn — every reader/writer retention fence has moved past them — are unlinked
serialization	snapshot() / VersionEntry export only the logical layer (row_id, txn_id, scn, value_length, value_data); the physical old_slot is rebuilt on load, so allocator-local VarlenReference bytes never hit disk

Contrast with Oracle: there is no version layer at all on that side — the values in the column store do not move, so “uncommitted” never needs to be represented and rollback never needs to undo column-store content. Because Rapid’s new value sits in place, it must maintain this chain and pay for it in memory and ownership terms: a pooled variable-length reference held by a version node delays reclamation of its whole block.

3.4 Side-by-side

	Oracle DBIM	Rapid
Does the column store hold uncommitted values?	No (values live in the row store)	Yes (as ACTIVE versions)
Are values rewritten in place?	No, never	Yes, during the asynchronous apply
What commit does to the column store	Advances the SCN horizon	Flips the state of an already-present version (COMMITTED / undo)
Column store’s own log	Not needed (“[O] no additional locks or logging”)	Needed: per-row log_row_prepare / log_row_commit / wait_durable / mark_applied
Carrier of visibility	SCN + fetch from the row store	before-image version chain (ColumnVersionManager::get_before_image_for_snapshot)
Per-row fallback within one query	Yes: stale rows come from the base table (mixed read)	No: the engine choice is per statement; execution never falls back to the row store (see §9.2)
Relation to in-place rewriting	Values are not rewritten ⇒ row-level fallback and IMCU rebuild are mandatory (closed loop, §3.6)	In-place writes ⇒ no row-level fallback needed, but MVCC version chain + WAL come out of Rapid’s own pocket

3.5 Why Oracle can only register at DML time (a correctness argument)

Suppose staleness were registered only at commit. A query whose snapshot is newer than the change could scan that IMCU in the window between “commit finished” and “registration finished” and read the initial value — a wrong result. Closing that window would require the scan to wait for the registration, i.e. an extra fence.

A change can become visible to others no earlier than commit, so the registration must complete before the change can be visible and must be ordered with respect to SCN publication. That is exactly why [O] can claim the column store needs no extra locks or logging: the registration is nothing more than a piece of metadata that travels with the transaction.

Inference (not stated in the paper): rollback does not have to “withdraw” anything from the column store, but an SMU entry must carry transaction/SCN information; otherwise it cannot answer “is this change visible to this snapshot”. The same fact explains why an older snapshot can still read the IMCU’s original value.

3.6 “Values never rewritten” and “per-row fallback” are one and the same design

They are not two features that can be chosen separately; they are two sides of one coin. The chain runs:

1. The column store keeps only the loaded snapshot — values are never rewritten in place.
2. So the updated value does not exist in the column store: the SMU carries only a rowid, never a new value (the paper: “the rowid is added to the metadata in the SMU”).
3. So a reader that can see the change can only take that rowid and fetch the current version from the base table (the paper: “The stale entries will be retrieved from the base table”) — one query both scans the IMCU and reads that row from the row store.
4. So the fallback ratio grows with the change rate, and scans get slower.
5. So a staleness threshold drives IMCU rebuild and trickle repopulation.
6. And the rebuild itself — re-reading those rows from the base table and overwriting them — is idempotent precisely because the values were never rewritten in place. That closes the loop back to step 1.

Note: this is a consequence of Oracle’s choice, not the only possible design. Keep the new value (or the delta) on the column-store side — a delta store — and no row-store fetch is needed at all. Rapid’s in-place write plus version chain is exactly that class of design (§3.2 / §3.3).

Three links in the chain deserve their own summary:

1. No rewriting ⇒ per-row fallback. The column store only holds the loaded snapshot, so an updated value can only come from the row store. The paper places both facts in the same sentence: “The original entries in the IMCU are not immediately replaced … Any transaction … that started before the DML occurred, can still see the original version”.
2. Per-row fallback ⇒ rebuild is needed to keep the ratio down. More stale rows means slower scans, hence the threshold-triggered IMCU rebuild and trickle repopulate (§6).
3. The rebuild is idempotent precisely because the values were never rewritten: rebuilding means re-reading those rows from the base table and overwriting, with no side effects.

The two reasons the paper itself gives for not replacing entries immediately map onto these two sides: read consistency (older snapshots must still see the original) and maintain data compression (an in-place replacement would break the dictionary/run-length structures).

Rapid, read in reverse: because it writes in place, a reader simply reads the current slot, so it does not need a per-row fallback (and therefore has no such path today, §9.2); instead it pays for an MVCC version chain, rollback undo, and its own WAL (§10 items 1/3/4). One verifiable price: CU::write() / CU::update() must first decompress_locked() the entire CU when it is in compressed form in order to write in place — compression benefit and write frequency are in direct conflict, a conflict Oracle’s “values never rewritten” sidesteps.

That does not mean Rapid should copy Oracle. The column store is in memory, so locating and rewriting a single CU slot is itself cheap (write the slot, push one version node); Oracle’s concern is mainly preserving the compression layout (replacing one dictionary or run-length entry breaks it). These are choices made under different premises, not right versus wrong. §10 closes with optimisation directions that keep the in-place premise.

4. Change-propagation scheduling

4.1 Rapid: the three HeatWave triggers

populate/log_populate.h:88-97 is labelled verbatim “HeatWave-compatible normal change-propagation triggers”:

HeatWave doc [H]	Rapid implementation [C]
Every 200 ms	POP_MAX_WAIT_TIMEOUT = 200 (coordinator batch interval)
Buffer reaches 64 MB	CHANGE_PROPAGATION_BUFFER_TRIGGER_BYTES = 64MiB; crossing it calls RequestAllBufferedTablesFlush()
Changed data read by a subsequent query	request_table_barrier_impl() sets pending_flush + os_event_set (no waiting for the 200 ms timer); the reader then calls wait_table_applied_for(..., QUERY_PROPAGATION_WAIT_SLICE_MS) until the watermark covers this query’s required_change_id
DML does not wait for propagation	When the buffer fills, the commit path is forbidden to wait: mark pending_flush, quarantine the table, mark_stale(ERROR_CLUSTER_OOM), warn, and let the commit proceed. Comment verbatim: “DML on the primary must never be delayed by change propagation”
(Not listed as a trigger in the doc) Lag ceiling	shannon_pop_data_sz > 0.85 × pop_buff_sz_max (default 256MB) → refuse offload (“too much changes need to populate”); ml/query_arbitrator.cpp performs the same check

Implementation detail: each table has its own pop buffer (POP_SHARD_COUNT = 16 shards) and a table-level worker applies changes in batches of BATCH_PROCESS_NUM = 256.

The difference from Oracle is not the threshold values but whether “propagation” exists at all: Oracle’s column store does not participate in transactions, so it has nothing to catch up with. What it has instead is a staleness-ratio threshold → rebuild a single IMCU, plus an IMCO trickle repopulate every two minutes (bounded by INMEMORY_TRICKLE_REPOPULATE_SERVERS_PERCENT, default 1).

5. Read consistency / MVCC

	Oracle [O]	Rapid [C]
Mechanism	SCN: “read consistency … managed via SCNs just as it is without Database In-Memory”	Version chain + reader transaction/SCN: CU::get_visible_cell → ColumnVersionManager::get_before_image_for_snapshot(reader_txn_id, reader_scn, reader_trx, …); chain structure and reclaim in §3.3
Older snapshot reads	The original IMCU value directly (nothing moved it)	The before image
Writer reads its own change	Through the row store	The current CU slot
Visibility of uncommitted writers	Irrelevant (no uncommitted value in the column store)	populate/log_populate.h:157: “ACTIVE-version visibility is resolved by the primary InnoDB ReadView”
Isolation-level limits	None of this kind	READ UNCOMMITTED and SERIALIZABLE refuse offload outright (“Rapid MVCC offload supports READ COMMITTED and REPEATABLE READ”)
Snapshot lifetime	Managed by SCN and undo	Transaction::acquire_snapshot() registers the GC fence before assigning the ReadView; unregister_transaction() performs IMCU rollback outside the lock

Because Rapid’s column store can hold uncommitted versions, it must carry its own “visible or not” decision layer. Oracle never holds them, so that layer does not exist at all.

6. Invalidation detection and repair granularity

	Oracle [O]	Rapid [C]
Granularity of “stale”	Row level: the row’s column entries in the IMCU plus its rowid in the SMU	Table level: propagation watermark lagging / Populate::IsPropagationBroken / RpdMirror::Registry::mark_stale
Repair action	On reaching the staleness threshold, rebuild one IMCU (background worker, online, data always available; concurrent changes are recorded)	A broken/stale table stops being offloaded and falls back to InnoDB, and only a reload (SECONDARY_LOAD) resumes propagation
Scheduling	Threshold (IMCU access frequency + stale row count) plus a two-minute IMCO trickle	The three HeatWave triggers plus the lag ceiling that refuses offload
Worst-case cost	Re-reading one IMCU’s worth of rows	The whole table is unavailable (served by InnoDB) until reload

Why Rapid cannot rebuild at IMCU granularity. Oracle can rebuild locally because it never changed the IMCU’s values: “refill” means re-reading those rows from the base table and overwriting, with no side effects. Rapid’s values are written in place, so there is no “stale but untouched” copy to rebuild; it can only choose between applying in place and invalidating the whole table and reloading it.

7. Persistence and recovery

	Oracle [O]	Rapid [C]
Mechanism	IM FastStart: checkpoints IMCUs into a designated tablespace; on restart, populate from FastStart rather than the base tables	recovery/table_persistence.* plus Rapid’s own WAL and Recovery::rapid_recovery
Consistency check	Compare the SCN at which the IMCU was checkpointed with the most recent modification SCN; three outcomes: use the whole IMCU, populate it with some rows marked invalid, or discard it and go to disk	Per row: log_row_prepare → wait_durable → apply → log_row_commit → mark_applied; a PREPARE without COMMIT is ignored by recovery; an fsync failure is “outcome unknown” and raises recovery_required
Can uncommitted data be persisted?	Not applicable (no uncommitted values in the column store)	No: checkpoint skips IMCUs for which Imcu::has_uncommitted_changes() is true (recovery/table_persistence.cpp)
Enablement	On by default except for NO MEMCOMPRESS, FOR DML, and objects with Join Groups	No equivalent switch; a table whose propagation is broken has its fast-restore proof revoked and is reloaded from the primary engine on restart

8. Scanning and execution

Capability	Oracle [O]	Rapid [C]
Column pruning + predicate pushdown	Only the needed columns are touched; filters run directly on compressed data	Same
min/max pruning	Storage Index, per IMCU per column	StorageIndex (numeric plus lexicographic string bounds)
Dictionary pruning (equality / IN)	Yes	Yes (Compress::Dictionary)
SIMD	Yes: the columnar format is designed to fill vector registers, and the scan itself is vectorised (the paper claims billions of rows per second)	Yes: runtime-dispatched SIMD kernels (AVX2 / SSE4 / NEON) plus a columnar batch-ingestion path (BatchReadable::ReadBatch feeds ColumnChunk directly from the handler, no row-format round trip); the batch path has conditional fallbacks (see §8.1)
Parallel scan	IM dynamic scans (18c): lightweight in-process threads using idle CPU; needs a Resource Manager CPU plan	Parallel_reader_adapter + BkgWorkerPool; batch interface rnd_next_batch / Executor::ColumnChunk
Numeric acceleration	In-Memory Optimized Arithmetic (NUMBER encoded as a fixed-width integer scaled by a common exponent; dual storage, up to +15% space; controlled by INMEMORY_OPTIMIZED_ARITHMETIC)	No equivalent mechanism (regular paths such as Utils::Util::get_field_numeric)
Joins / aggregation	Join Groups (shared dictionaries), In-Memory Aggregation	VectorizedHashJoinIterator (with spill files), plan suitability decided by ML::Query_arbitrator
Expressions / JSON	In-Memory Expressions, virtual columns, automatic detection (ESS), JSON	Determined by Query_arbitrator and optimizer-side support

8.1 What SIMD actually covers

Rapid has SIMD; it is not a scalar-only path. The difference is coverage, not whether the instructions exist.

Item	Fact
Kernel library	utils/SIMD.h + utils/SIMD_Impl.h (all inline; SIMD.cpp is an empty translation unit): bitmap popcount, sum, min, max, count_non_null, filter(predicate), filter_compare_int32/int64 (with CompareOp) — each with _avx2 / _sse4 / _neon / _generic variants
ISA and dispatch	x86 uses <immintrin.h>, ARM uses <arm_neon.h>; detect_simd_support() dispatches at runtime via __builtin_cpu_supports("avx2"); AVX2 kernels use __attribute__((target("avx2"))) as function-level ISA variants (the comment notes that the older form, guarded only by #ifdef __AVX2__, produced baseline builds with no AVX2 kernel — “runtime dispatch in name only”)
Build	The rapid target is compiled with -mavx2; include/rapid_arch_inf.h derives SHANNON_VECTORIZE_SUPPORT and SHANNON_VECTOR_WIDTH (AVX512=512 / AVX2=256 / SSE=128 / NEON=128), falling back to SHANNON_SCALAR_FALLBACK when no vector support is compiled in
Call sites	Batch predicate evaluation (imcs/predicate.cpp, gated by simd_eligible(op), simd_width = 8/4); aggregate reduction (executor/iterators/aggregate_iterator.cpp, counted in m_stats.simd_rows); Kernels::min/max (executor/iterators/iterator.h)

There is also a columnar batch-ingestion path: BatchReadable (iterator.h), deliberately not added to sql/iterators/row_iterator.h — it is a Rapid-owned mixin.

Item	Fact
Interface	BatchReadable::{ ReadBatch(col_chunks, capacity, rows_read), PushbackBatchTail(...), SupportsBatchRead() }
Implementors	VectorizedTableScanIterator (its ReadBatch() calls file->rnd_next_batch() to fill ColumnChunk directly, bypassing the row format, with a lookahead buffer that drains pushed-back rows first), VectorizedFilterIterator, VectorizedHashJoinIterator (which also consumes batches from its build/probe children)
Consumers	VectorizedAggregateIterator (probes with dynamic_cast in Init(); pulls batches with ReadBatch and uses PushbackBatchTail for GROUP BY boundaries), VectorizedHashJoinIterator (build/probe batch pulls in hash_join_iterator.cpp)
Modelled benefit (project’s own note, readme.md §9)	Virtual calls drop from N to ⌈N/B⌉; for pure aggregates the table->field write-back is eliminated entirely

Fallbacks remain (listed in readme.md §7; SupportsBatchRead() can also return false per execution):

• the source does not implement BatchReadable (for example join or subquery nodes that do not);
• m_rollup is true; the aggregation contains non-vectorisable functions (GROUP_CONCAT, JSON_ARRAYAGG); or m_vectorization_enabled is off;
• hash-join inputs whose chunks carry encoded string payloads (SupportsDirectBatchInput() false — the comment reads “Keep those inputs on the scalar path until the batch interface carries decoded data”), or whose child holds record images (SupportsBatchRead() = !m_columns_hold_row_images).

Note that the Problem Statement at the top of executor/iterators/readme.md describes the pre-change state; the same file says Status: Done, which matches the code above. The remaining difference from Oracle is therefore not “is there SIMD / is there columnar ingestion”, but:

1. Oracle’s columnar format is designed to fill vector registers, and the scan itself is vectorised; Rapid’s batches hand column data to ColumnChunk and the SIMD kernels run on the predicate/reduction over that batch;
2. Rapid’s batch path is opt-in with conditional fallbacks, so not every operator or query uses it.

9. Deployment shapes and the offload decision

	Oracle [O]	Rapid [C]
Scale	Single instance / RAC (one IM column store per node; DISTRIBUTE / DUPLICATE; IMCUs never ship between nodes) / PDB / ADG / Exadata flash	Single instance, in-process plugin; partitioned tables handled by ha_rapidpart
Reporting refusals	—	SetSecondaryEngineOffloadFailedReason(); reasons include not loaded, propagation broken (“table has failed DML propagation and must be reloaded”), too many pending changes, unsupported isolation level, unsupported dictionary-encoded projection
Forced path	—	use_secondary_engine = FORCED skips arbitration (SecondaryEnginePrePrepareHook returns true immediately) but does not skip the hard preconditions inside RapidPrepareEstimateQueryCosts

9.1 With use_secondary_engine = ON, the decision is still arbitrated from load

”the server decides whether to offload” is only the entry point for ON; the actual decision is Rapid’s, and its inputs include runtime load:

1. The server first prepares the statement on the primary engine (PRIMARY_TENTATIVELY) and caches the primary plan information (RapidCachePrimaryInfoAtPrimaryTentativelyStep → cache_primary_plan_info);
2. SecondaryEnginePrePrepareHook picks a classifier. The contract: returning true means “go to Rapid”, false means “stay on InnoDB”.

Condition	Classifier	Basis
FORCED	returns true directly	no arbitration
dynamic_offloads = false, or the query is “very fast” (is_very_fast_query)	standard_cost_threshold_classifier	whether the primary estimate exceeds secondary_engine_cost_threshold (cheaper than invoking the model)
otherwise: the populator is not running, or is running with an empty propagation buffer	decision_tree_classifier	an ONNX model, shannon_rapid_classifier.onnx, predicts OLAP vs OLTP; if the model is unavailable, fall back to the primary engine
otherwise: the populator is running and the propagation buffer is non-empty (exactly the lag window)	dynamic_feature_normalization	(1) global lag above SHANNON_MAX_POPULATION_BUFFER_SIZE → stay on InnoDB; (2) any table in this query is still in the pop queue (pop_buff_contains / mark_table_required) → stay on InnoDB

The last row deserves emphasis: “change propagation is in flight” is itself a reason to refuse offload, and it is evaluated per table — the same layer of protection as the three triggers in §4 and the table-level quarantine in §6.

Second gate (prepare phase): PrepareSecondaryEngine → RapidPrepareEstimateQueryCosts (hard preconditions; for non-FORCED it additionally requires the cached primary plan to be available), plus the optimizer’s per-access- path cost hooks (ModifyTableScanCost / ModifyFilterCost and friends can return true; // refuse to offload); RapidOptimize has its own refusal reasons (for example “the change propagation lag is too much”).

The three states in practice: OFF = never offload; ON = decided by layered cost / load / propagation arbitration (the default); FORCED = skip arbitration but still bound by the hard preconditions.

9.2 Which executor runs, and the granularity of fallback

Rapid does not implement its own executor; it reuses MySQL’s execution stack. Rapid has its own optimizer (optimizer/, with rules such as PredicatePushDown, StorageIndexPrune, ProjectionPruning, TopNPushDown, AggregationPushDown, and plan nodes ScanTable, Filter, HashJoin, NestLoopJoin, Sort, Limit, LocalAgg, MySQLNative), and it lowers the result through ToAccessPath() onto the server’s access-path / iterator tree. Every Rapid operator derives from a MySQL base class:

Operator	Base class
VectorizedTableScanIterator	TableRowIterator
VectorizedAggregateIterator / VectorizedHashJoinIterator / VectorizedFilterIterator / VectorizedSortIterator / VectorizedWindowIterator	RowIterator
VectorizedTemptableAggregateIterator	TableRowIterator

BatchReadable is only a Rapid-side mixin, so sql/iterators/ is untouched; when the original MySQL path must be preserved, the MySQLNative node hands original_path back unchanged (temporary and materialised tables, for instance). Partitioned tables are read by MySQL’s own TableScanIterator (ScanTable::ToAccessPath() clears the vectorized flag). For contrast: Oracle’s IM column store row source is kernel-native, and INMEMORY appears directly in the execution plan.

The granularity of fallback is the statement:

• table_offload_blocker() requires every table in lex->query_tables to be loaded, otherwise it refuses the offload;
• when preparation fails, the server restarts the whole statement through PRIMARY_TENTATIVELY → SECONDARY → PRIMARY_ONLY (sql/sql_parse.cc);
• the scan path never asks the primary engine for row data: ha_get_primary_handler() is used only by index_flags() and records_in_range().

So execution never switches engines halfway through a scan: either the whole statement runs on Rapid or the whole statement runs on InnoDB.

10. Engineering consequences of the model

These are not defects; they are the necessary companions of the choice “mutable column store + asynchronous propagation”:

1. Why Rapid needs its own per-row WAL: its column store is itself a writer, so durability cannot be delegated to the row store. Oracle’s “no additional locks or logging” rests on its column store being read-only.
2. Why fsync lands inside the IMCU mutation lock (review item H1): the DML branch requires “persist the WAL before publishing to memory”, and publishing shares the same mutation barrier as the memory write. Oracle has no such position to defend.
3. Why a post-commit failure can only be quarantined, not reported with my_error: the column store’s publish boundary and the server’s transaction boundary are not the same point; once the server has ruled the commit successful, a propagation failure can no longer surface as a statement error (QuarantineFailedNotification / quarantine_failed_transaction).
4. Why external-abort reconciliation exists: in-place writes mean a server-side implicit rollback that goes unnoticed leaves CU content un-undone (reconcile_on_external_abort()).
5. Why checkpoint must skip uncommitted IMCUs: has_uncommitted_changes(), otherwise an ACTIVE version would be persisted.
6. Why propagation latency cannot be reduced by making DML wait: as with HeatWave, DML is explicitly not allowed to be blocked by propagation; the invariant is held by degrading the replica (quarantining the table) instead.

As for optimising away the cost of items 2/3/4, copying Oracle’s “values never rewritten + row-level stale + asynchronous IMCU rebuild” is not the right move:

• Its motivation is largely gone in an in-memory engine. Oracle chose “do not replace immediately” for two reasons — read consistency and maintain data compression (an in-place rewrite breaks the dictionary and run-length structures). Rapid’s column store is in memory, where updating one CU is cheap (locate the slot, write it, push one version node), so there is no need to import row-level staleness and IMCU rebuild just to avoid rewriting a columnar structure.
• Its cost, by contrast, is not small. It requires adding a “fall back to InnoDB per row during a scan” path (§9.2) and turning the in-place write into “two copies + background refill”.
• Do not mistake the existing metadata for it. The TransactionJournal in each IMCU header is a per-row operation/existence delta chain (is_row_visible() / check_visibility_batch() use it to decide whether a row counts as existing for a snapshot, and abort_transaction() also reconciles the physical tombstone bits); the per-CU ColumnVersionManager is the value version chain. Both are “supporting metadata for in-place writes”, not Oracle’s “old value waiting to be rebuilt”, and neither maps onto SMU / IMCU rebuild.

More appropriate directions move cost off the critical path instead of changing the in-place premise:

1. Reduce write amplification: borrow the MEMCOMPRESS FOR DML idea and pick a level that trades compression for DML performance, and avoid the decompress_locked() of a whole CU on write while it is compressed;
2. Move the per-row WAL fsync out of the IMCU mutation barrier (review item H1), for example by committing per batch or per transaction group, while keeping the “WAL first, then publish to memory” ordering intact;
3. Keep propagation and persistence on the HeatWave-style asynchronous model (200 ms / 64 MiB / query demand) without changing the invariants of the DML path.

Appendix A: quick reference

Concept	Oracle	Rapid
Column-store unit	IMCU	Imcu
Column structure inside the unit	Columns within the IMCU	CU
Per-row invalidation data	SMU (stale rowids)	Three layers with distinct jobs: del_mask (physical tombstones), TransactionJournal (operation/existence delta chain), per-CU ColumnVersionManager (value version chain)
Pruning	Storage Index min/max + dictionary	StorageIndex + Dictionary
How change enters	Staleness registered during the DML (values untouched)	Async in-place write (ACTIVE → COMMITTED/undo)
What commit does	Advances the SCN	Flips the version state
Repair granularity	Rebuild one IMCU	Table-level quarantine + reload
Scheduling	Staleness threshold + trickle (2 min)	200 ms / 64 MiB / query demand
Persistence	IM FastStart (three-way SCN check)	table_persistence + per-row WAL
Parallel scan	IM dynamic scans	Parallel_reader_adapter + BkgWorkerPool

Appendix B: sources

• Paper quotations: §3.1, §3.4, §3.6, §4, §5, §6, §7, §8, §9, all from the Technical Overview sections “Populating the In-Memory Column Store”, “In-Memory Compression”, “In-Memory FastStart”, “In-Memory Scans”, “DML and the In-Memory Column Store” (Bulk Data Loads, Transaction Processing, Repopulation), and “The In-Memory Column Store on RAC”.
• HeatWave triggers: §4.1, matching the code comment at populate/log_populate.h:88-97.
• Rapid-side code locations: handler/ha_shannon_rapid.{h,cc}, handler/ha_shannon_rapidpart.{h,cc}, populate/log_populate.{h,cpp}, populate/log_dml_notification.{h,cpp}, imcs/imcu.{h,cpp}, imcs/cu.{h,cpp}, imcs/storage0index.h, imcs/row0row.{h,cpp}, recovery/table_persistence.cpp, trx/transaction.{h,cpp}, ml/query_arbitrator.cpp, include/rapid_const.h, include/rapid_config.h, executor/iterators/, optimizer/, utils/SIMD*.h.