Why the Agent Lives Inside the Kernel

An architecture note on why ShannonBase implements the agent harness inside the database kernel, reusing the SQL authorization model, while the more common industry pattern is an external harness plus branching and sandboxes.

0. In one paragraph

The external pattern — a harness outside the database, plus branching and sandboxing — converts an authorization problem into a copy problem: give the agent a throwaway copy and let it write freely, then verify and merge. That works only if copies are cheap — and cheap copies are a storage-layer capability that Postgres ecosystems provide (copy-on-write branches) and MySQL/InnoDB does not.

So our choice is not a matter of style. Because we cannot bound the risk with a copy, we have to solve it on the authorization side: the agent can never exceed the caller’s privileges, its writes join the caller’s transaction, and its audit lives in the same transaction as the data.

The costs are real and we state them: slow iteration, no streaming interaction, and the LLM’s failure domain folded into the database. Those should be paid for by an external layer — not by asking the kernel to also be a great chat client.

1. Get the coordinate system right

Three things that get conflated, but are not variants of one design:

	What it is	The question it answers
In-kernel harness	The agent loop, tools, approval and memory execute inside the server process; authority comes from the SQL principal	Where does the agent run, and as whom?
External harness	The agent loop runs in the app or a separate process, connecting over a driver or MCP; authority comes from a credential	Same question, with a credential as the identity
Branching + sandbox	Data can be thrown away (CoW branch / preview DB); code execution can be thrown away (sandbox)	How large is the blast radius of a mistake?

”In-kernel” and “external” are the two ends of one axis (where authority lives). “Branching + sandbox” is a different axis entirely, and it composes with either end. Treating it as “a third place to put the agent” is the most common coordinate error in this discussion.

2. Why external harness + branching + sandbox became the default

The usual shape: an external agent (or MCP client) holding a credential, plus a throwaway database copy per session or per pull request.

• The harness side: MCP servers for various databases expose the database as a set of tools to IDE agents and custom harnesses.
• The branching side: copy-on-write branches, preview branches, “one ephemeral database per agent session”.

Three structural reasons it became the default:

1. Branching turns authorization into copying. With a throwaway copy you no longer need an approval state machine, a transaction-ownership model, or reviewer UX — you need diff and merge. That removes the heaviest part of a harness in one move.
2. An external harness buys iteration speed and capability breadth: prompts, models and tools evolve at normal software cadence, and reaching the code repository, the issue tracker or CI is trivial.
3. It happens to be the natural shape of the Postgres ecosystem. CoW branching is a storage capability — one vendor implemented it at the storage layer, another productized it — and the agent just benefits.

One distinction matters here: “branching” means two quite different things in the industry.

Form	What it can verify	Example
Schema-only branch	Migration/DDL rehearsal, no data	PlanetScale deploy requests
Data + storage CoW branch	DDL and DML can both be tried freely, then merged	Neon / Supabase branches

3. The boundaries of the external approach: three things it structurally cannot do

3.1 It cannot write inside the application’s transaction

An external harness must break the work into commit points: the agent produces SQL, the application executes it. That cuts the agent’s decision loop outside the transaction, and the actions it can express are limited to whatever statements the application is willing to run.

3.2 It cannot give you zero credential sprawl plus engine-enforced tenancy

An external design has to rebuild the tenancy model: RLS, a role per user, and a way for the MCP server or agent to carry “who is asking”. But an MCP server typically holds one credential, so “who is asking” degrades from enforced to declared. Downgrading from a superuser with SET ROLE / SET SESSION AUTHORIZATION is worse: it requires the adapter to hold a superuser, so a single injection is a total compromise.

3.3 It cannot make audit transactional

Audit lives on the application side and cannot roll back with the data. If the audit log is the compliance artifact, that is a hard failure.

3.4 Two practical limits

• A branch cannot verify concurrency or the moment of merge. A branch is a single-user copy: no deadlocks, no lock waits, no MVCC contention, no replication lag. But the incidents that agent-authored DDL/DML actually cause in production are locks and latency.
• ”Fix that one wrong row in production” cannot be merged. A schema change diffs into DDL; a batch of row-level repairs diffs into a pile of data differences, and still has to be executed as a statement in production. At that point branching falls back into the in-kernel design’s territory: who approved it, and in which transaction did it run?

4. What ShannonBase does instead

4.1 Implementation

Capability	How
Execution environment	LANGUAGE JAVASCRIPT stored routines (JerryScript, one engine and heap per thread); native helpers sys.exec_sql / fetch_all / send_result_set / engine_heap_bytes
Tool layer	A single registry: one register_tool() declaration drives the argument schema, error text, policy metadata, prompt catalogue and handler — 34 tools; a contract self-check detects drift
Tool protocol	Native tool calling (the provider’s tools channel) plus the JSON-in-text protocol; a native call is converted into the same shape
Authorization	Every routine is SQL SECURITY INVOKER; mysql.agent_policy is the instance baseline and one-directional combinators let a session tighten it, never relax it; destructive DDL and account/code/instance DDL are refused by default
Transactions	Caller-owned vs agent-owned is distinguished; leases plus performance_schema transaction EVENT_ID correlation; a finally safety net that rolls back only the agent’s own transaction
Approval	A state machine inside InnoDB: CAS claims, TTL cancellation, interrupted steps resolved to a terminal indeterminate and never auto-retried; the approved DML and the approval state commit in the same transaction
Context	Token budget derived from the model window, character budget derived from the engine heap; overflow compacts deterministically and continues; large results spill to an artifact store and are paged back
Resource governance	A read ceiling that refuses rather than rewrites (and says results were truncated), statement-level MAX_EXECUTION_TIME, and unattended SELECTs rejected with actionable guidance
Memory	Four layers, isolated by a key derived from SHA2(CURRENT_USER()), not overridable across principals
Termination	12 stop_reason classes; every non-completion ending appends “this answer may be incomplete”, and that note is appended after the leak safety net
Observability	Per-step SQL trace, per-principal usage and quota, approval history and rollback log

The size cost is visible: the agent’s JS closure expands into roughly 2.36 MB of generated SQL, duplicated across four routines; the engine heap is a build-time parameter (2048 KB by default, up from 512 KB).

4.2 What the structure buys

Mapping back to §3:

1. Writes can join the caller’s transaction. The agent’s write and the application’s write commit atomically at the same isolation level; the approval state commits with the DML, closing the “commit succeeded but the state row never landed” crash window.
2. No new credential exists. The agent’s authority ceiling is the caller’s grants, so privilege escalation is architecturally unreachable, and tenant identity comes from the engine instead of being rebuilt outside it.
3. Audit is transactional. Who changed what, when, and whether it can be rolled back are all queryable and roll back with the data.

One more point that matters a great deal for a database vendor: an external agent is the customer’s choice, whereas an in-kernel agent is a product feature we can ship. In multi-tenant SaaS, on-premises deployments and air-gapped environments, customers will not accept a broadly-privileged external process planted into their production.

4.3 What it costs (stated plainly)

• Slow iteration. Changing one line of JS means regenerating 2.36 MB of SQL, rebuilding, and going through the sys-schema upgrade path.
• The engine constraint is hard. Every routine shares one per-thread JS heap, so prompts and budgets must follow the heap. Getting here required making the heap a build-time knob and deriving the fetch/prompt ceilings from it.
• No streaming, no cooperative cancellation. The caller blocks inside CALL for the whole turn; only turn and time budgets bound it.
• The LLM failure domain is folded into the database. Model calls happen on database threads, which forces derived timeouts, bounded retries, and a “stop retrying past the deadline” rule. That is the rent this design pays.
• One tool call per turn (native multi-call is deliberately dropped so it cannot bypass trace or approval), no parallelism, no sub-agents.

5. Side by side

Dimension	In-kernel harness	External harness + branching + sandbox
Authority comes from	The caller’s SQL grants; escalation unreachable	A credential; narrow it yourself
Blast radius	That principal’s privileges + policy gate + approval	That credential’s privileges (often broad)
Transaction participation	✅ commits atomically with the app	❌ only at commit boundaries
Data movement	Data stays put; aggregation pushed down	Every read crosses the wire; context window is the bottleneck
Capability breadth	SQL + registered tools only	Unbounded (MCP, code execution, cross-system)
Multi-tenancy	Enforced by the engine (CURRENT_USER())	Must be rebuilt (RLS + per-user identity)
Audit	Same transaction as the data; rolls back with it	Application-side; cannot roll back with data
Iteration speed	Slow (generate, rebuild, upgrade)	Fast (normal software cadence)
Interaction	Blocking, no streaming	Streaming, interruptible
Failure domain	LLM failures land in the DB	LLM failures only affect the app
Destructive-operation backstop	Policy + approval + transaction	Throwaway copy (diff/merge)
Verification instrument	Transactional dry-run + assertion	Branch / sandbox

6. How we prove these properties hold

Claiming “authority cannot escalate”, “an incomplete answer says so”, or “a read cannot blow up the engine heap” is easy. Making those claims checkable is the hard part. Every safety property in the harness is turned into a scriptable, reproducible failure.

6.1 A scripted model driving the real loop

The only exit towards a model is also the only seam that needs replacing. sys.shannon_agent_loopcheck(case) swaps the model for a queue of prewritten turns (plain text, a text-protocol tool call, a provider-native tool call, a specific finish_reason, a specific error) and then runs the real agent loop.

• Assertions land only on the loop’s own signals: stop_reason, the sequence of tools that actually ran, and whether the answer carried the incompleteness note. Never on generated prose — prose is exactly what a script cannot make realistic.
• 13 cases (9 standard plus 4 named-only manual ones): clean finish, tool-then-answer, native tool call, read-ceiling refusal followed by recovery, repeat detection, error budget, turn ceiling, output cut off at the length limit, unreachable model — plus the paired cases for mysql.agent_policy.
• The scripted model can only be installed by the loopcheck routine, never by a chat option — otherwise “make the agent say anything” would become a session option.

Why this matters: these properties fire on behaviour a real model produces rarely and never on demand — turn exhaustion, consecutive failures, repeated calls, context overflow. Before these cases existed, those paths were argued for in review and implemented carefully, but never triggered end to end.

6.2 Operator policy: paired assertions plus a reverse control

Every assertion in shannon_agent_policy.test is made twice: once with the operator row and once without it.

• Proving a refusal alone means nothing — plenty of things refuse a DROP.
• The difference between the two runs is the evidence that this row caused the refusal.
• And each case ends with an assertion independent of the harness’s own report (SHOW TABLES LIKE 'policy_victim'), so the system under test never certifies itself.

6.3 Contract self-checks: documentation cannot drift from implementation

CALL sys.shannon_agent_selfcheck('tools') checks consistency, not behaviour: that the tool registry and the prompt catalogue still agree, that the policy combinators really only tighten, the read-ceiling allow/deny table, the stop-reason taxonomy (anything other than finish must carry a note), the system-schema gate, and the SQL-mode gate. Adding a tool means changing one declaration; the contract test fails when the two drift.

6.4 Negative controls, and refusing to pin numbers

javascript_sp_heap_exhaustion.test is the negative control for out-of-memory: it asserts the statement fails and returns, that the server is still alive, that the next routine still works, and that all of it is repeatable in one session. It deliberately does not assert the heap size — that is a build-time decision, and pinning it means re-recording every time it moves, which is exactly how a negative control quietly stops being run (the figure is masked out instead).

6.5 What this method does and does not cover

It covers the harness: termination semantics, policy, ceilings, compaction, recovery. It does not measure answer quality — that needs a real model, belongs outside the regression suite, and is deliberately excluded. Harness correctness is the precondition for discussing answer quality at all.

7. Who should use which

Role	Better fit	Why
DBA	Kernel first, external for interaction	The context lives in the instance (real plans, locks, statistics), and applying a change needs real privileges and a real transaction. Heavyweight online DDL, fleet-wide work and cross-system visibility belong outside
On-call / app support / data analyst	Kernel only	They must not hold DDL rights, yet they need read-only diagnosis plus a proposal path. No external design produces that combination structurally
Developer	External (harness + branching + sandbox)	The deliverable is a file (migration/code), the credential is a dev database, and the sandbox already exists (local and CI databases)
Platform / release engineering	External orchestrator + per-instance kernel execution	Fleet work needs idempotency, canaries, rate limiting and resumability. The agent belongs at the two ends (planning and exception diagnosis); the 200 executions in between are deterministic orchestration
Destructive experiments / CI / evaluation	Branches and sandboxes	Turn irreversible into disposable; evaluation needs a clean, reproducible environment

In one line: “operate this one database” belongs to the kernel (DBA, on-call, in-situ analysis); “build against, and orchestrate, many databases” belongs outside (developers, platform, release). The kernel supplies tools and policy; the outside supplies interaction and breadth.

8. The half that is missing

The kernel design is strongest at authorization and execution, weakest at verification and interaction. The engineering design for those lives in an internal companion note. The conclusions:

• DML observability: no branch needed — what is needed is observability before execution: a transactional dry-run, a row-level diff, and an assertion comparing actual impact against the estimate, running on real data, real locks, real isolation and real privileges.
• DDL verification: the practical form of “branching” in the MySQL world is a shadow schema plus a replay of the real workload from performance_schema.events_statements_summary_by_digest, answering “will this change break live queries?” rather than “does it parse?“.
• Interaction and breadth: an MCP entry point so an external harness uses the same policy-constrained tools — one tool description, one execution path, per-caller connections, approval as its own tool.
• Fleets: what is needed is an idempotent, resumable orchestrator, with the agent at the two ends (planning and exception diagnosis); the kernel side needs structured output, parameterised intents, an idempotency change_id, and progress reporting.

9. Conclusion: why the native path

1. Authority. The caller already holds SQL privileges, and only the engine knows the tenant identity. Moving either outside means rebuilding multi-tenancy and turning a credential into a new leak surface. In-kernel is the smallest trust delta.
2. Context. The questions we answer are about this instance: real plans, locks, performance_schema, statistics. Reasoning where the data is beats exporting the data first.
3. Deliverable. We ship the database’s capabilities. An external agent is the customer’s choice; an in-kernel agent is a vendor’s product feature.
4. A storage constraint. MySQL/InnoDB has no cheap data branch. The external pattern became the ecosystem default because the storage layer supplies CoW copies; that premise does not hold here, so the problem must be solved on the authorization side.
5. An identity constraint. External designs make “who is asking” declared. We keep it enforced, because CURRENT_USER() is a fact the engine states and nobody can forge.

This is not a contest between routes; it is a division of labour. We go all the way on authorization, transactions and audit inside the kernel, and we are explicit about what it is bad at — interaction, breadth, experimentation — and close that gap with an MCP entry point that puts external harnesses on the same policy-constrained tools. At that point the in-kernel agent and the external one are not competitors; they are two faces of one product.