System position · 20 July 2026 · 18 min read

Autonomous healthcare migration, without an approval queue disguised as safety.

I kept running into the same architecture. Let the system reason, then make it wait for a person before anything interesting happens. It looks cautious. In practice it turns uncertainty into a queue and calls the queue governance. This is my attempt at a better control plane.

Runtime: autonomousControl: contract-boundKnowledge: retrieved + versionedExpertise: benchmarked skillsHumans: off the critical path

Product thesis

A self-improving migration operating system that can discover, reason about, build, execute, validate, recover, and optimize healthcare data migrations end to end, while proving that each action satisfied its contracts.

Decision reset

Human approval is not the control mechanism.

This was the first thing I had to get straight. A frequent approval dependency creates latency, diffuses accountability, and eventually becomes ceremonial. So the normal execution graph has no person-shaped node in it.

Rejected model

Plan every action, then wait for a generic approval.
Use “human in the loop” as a substitute for understanding.
Let approval clicks absorb semantic and operational risk.
Escalate normal uncertainty instead of resolving it.
Measure safety by the number of gates crossed.

Autonomy-first replacement

Establish a durable operating constitution and risk budget before runtime.
Encode invariants, tolerances, authority, rollback, and evidence in contracts.
Execute without per-step approval while the system stays inside those envelopes.
Use retrieval, simulation, differential tests, and bounded experiments before escalation.
Bring in people only for underdetermined intent, physical access, legal ownership, or a constitutional change.

The distinction matters: accountability still exists. It moves to the governance plane. Runtime authority comes from contracts and evidence, not a button. A customer can still demand manual signoff for a cutover. That is policy, not a load-bearing architectural dependency.

Architecture

Probabilistic intelligence. Deterministic authority.

Agents can form hypotheses, retrieve context, compose skills, generate transformations, and choose experiments. They cannot advance authoritative state or cause effects directly. The deterministic runtime owns that job.

ObserveInventory, telemetry, drift, failures

RetrieveContracts, skills, history, evidence

ReasonHypotheses, plans, uncertainty

SimulateStatic checks, sandbox, counterfactuals

ExecuteTyped effects inside a capability envelope

EvaluateIndependent deterministic acceptance

EvolvePromote, demote, and synthesize skills

Intent plane

migration objectiveoperating constitutionrisk budgetportfolio priorities

Intelligence plane

planner ensemblecontext assemblerskill retrieverskill synthesizercounterfactual simulator

Authority plane

state reducercontract compilerschedulercapability brokerbudget controller

Execution plane

source connectorsDatabricks SDK / RESTJobs and PipelinesUnity Catalogisolated sandboxes

Truth plane

independent evaluatorevidence ledgerlineage reconcileranomaly modelrecovery manager

Autonomous core

Durable state, orchestration, retrieval, skill composition, evaluation, recovery, and learning remain live without a user session.

Customer trust boundary

PHI-bearing execution, detailed evidence, credentials, and source connectivity stay inside the customer-controlled environment.

Independent failure domain

The authoritative controller stays outside the target workspace so it can diagnose, pause, repair, or replace impaired workloads.

Engineering core

State and context are the system.

Not implementation details. Not a transcript. The controller needs different stores for current truth, historical evidence, retrievable knowledge, and evolving capability. Shoving all of that into chat history or one vector database makes recovery and learning feel magical right up until they fail.

Layer	Stores	Invariant	Technology shape
Authoritative state	Migration, plan, run, wave, resource, checkpoint, lease, budget	Strongly consistent, versioned transitions. No model writes directly.	Relational store + deterministic reducer
Event history	Commands, observations, effects, failures, recoveries, model and tool versions	Append-only and reconstructable	Event log + content-addressed artifacts
Working context	Current objective, dependencies, recent observations, retrieved contracts	Built per decision, disposable, and provenance attached	Context assembler with strict token and data budgets
Project memory	Approved mappings, source behavior, exceptions, decisions, runbooks	Tenant-isolated, versioned, and expiration-aware	Structured facts + documents + retrieval index
Skill registry	Capabilities, contracts, evals, permissions, known failure modes	Executable only after certification	Signed packages + benchmark results
Learning corpus	Adjudicated outcomes, failures, counterexamples, synthetic cases	No raw cross-tenant PHI. No unverified production feedback.	Curated replay and adversarial corpus

Context assembly

Start from the objective, current state version, and unresolved uncertainty.
Retrieve only relevant source contracts, skills, prior runs, and failure analogues.
Attach origin, confidence, scope, and freshness to every fact.
Prefer structured facts and evidence over narrative summaries.
Detect contradiction instead of quietly merging memories.

Recovery model

At-least-once dispatch with idempotency and fencing.
Write receipts and read-after-timeout reconciliation.
Checkpointed waves and explicit points of no return.
Automatic rollback only where proven safe. Otherwise isolate and forward-repair.
Controller restart reconstructs truth without replaying a conversation.

Dynamic skill system

A skill is not a prompt.

It is a versioned, testable, permissioned capability with explicit preconditions, effects, evaluation, and recovery. This is how source-specific expertise stays out of the kernel without becoming an ungoverned bag of tools.

skill: migrate.relational.snapshot version: 4.2.1 intent: consistent source snapshot to governed Delta staging inputs: source_contract, target_contract, cutoff preconditions: - source is read-only - consistency primitive is proven - target is isolated and versioned capabilities: - source.describe - source.snapshot.read - databricks.jobs.run - unity_catalog.table.stage effects: staged_dataset, run_receipt, checkpoint evaluator: snapshot-integrity@3 recovery: restore-target-version@2 budgets: rows, bytes, time, cost, retries evidence: manifest, hashes, dispositions, lineage certification: benchmark suite + known-failure corpus

Skill lifecycle

Detect

Capability gap

No certified skill satisfies the current contracts and state.

Compose

Retrieve, adapt, or synthesize

Reuse proven subskills before generating a new implementation.

Prove

Sandbox and adversarial evaluation

Run golden cases, seeded faults, differential checks, and recovery tests.

Release

Autonomous promotion

Promote inside the meta-contract only when every declared benchmark passes.

Operate

Monitor and demote

Compare predicted and observed behavior. Demote automatically on regression.

Contract hierarchy

Operating constitution

Permitted objectives, autonomy limits, data boundaries, risk budget, evidence, and recovery requirements.

global

Source contract

Authority, schema, semantics, consistency, CDC, deletes, versions, defects, access, and volume.

connector

Target contract

Catalog ownership, schema, quality, performance, lineage, retention, and consumer guarantees.

databricks

Semantic contract

Meaning, terminology, identity, temporal rules, amendments, reversals, and accepted transformations.

healthcare

Execution contract

Plan DAG, capabilities, budgets, checkpoints, retry, idempotency, isolation, and rollback.

runtime

Evaluation contract

Ground truth, invariants, tolerances, anomaly handling, evidence, promotion, and demotion thresholds.

truth

Source-agnostic does not mean source-unaware. The kernel speaks capabilities and contracts. Each adapter owns the physical and semantic behavior of its source. Adding X12, FHIR, JDBC, object-store, or MDM connectivity should not require new scheduler or governance logic.

Expert replacement

“Equal to an expert” has to be an evaluation result.

Fluency is cheap evidence. Confidence is worse. If this system is supposed to replace routine, and eventually advanced, data-engineering labor, it needs to beat expert baselines on held-out migrations and seeded failures.

Capability benchmark	Expert baseline	System must demonstrate	Failure signal
Source discovery	Senior engineer inventory and dependency map	Equal or greater asset and flow recall with fewer unsupported assumptions	Hidden source, consumer, PHI path, or source-of-truth ambiguity
Snapshot and CDC	Migration engineer's no-gap design	Exact cutoff proof, deletes, ordering, replay, drift, and restart correctness	Gap, overlap, duplicate logical effect, lost delete, or corrupt checkpoint
Transformation	Reviewed production mapping	Schema and semantic equivalence across golden and counterexample corpora	Silent coercion, unit, status, or time change, or unexplained record loss
Healthcare identity	MDM steward decisions	Better precision and recall with calibrated abstention on ambiguous merge and split cases	Patient or member conflation, or irreversible unsupported merge
Claims reconciliation	Finance and claims control pack	Exact claim, line, billed, allowed, paid, and reversal totals at repeatable cutoffs	Unexplained financial variance
Clinical semantics	Clinical informatics review	Correct status, units, terminology, provenance, amendment, and temporal behavior	Patient-safety-relevant semantic divergence
Recovery	SRE runbook and rehearsal	Faster deterministic diagnosis, checkpoint recovery, rollback, or forward repair	Unknown write outcome, state divergence, or inability to reconstruct
Operational optimization	Platform engineer tuning	Meet correctness and SLA at lower cost and variance without weakening contracts	Regression hidden by aggregate success

Automate fully

Inventory, profiling, code generation, deployment, test construction, reconciliation, diagnosis, retries, recovery selection, evidence, and optimization when contracts are complete.

Resolve through evidence

Unknown source behavior, mappings, drift, performance, and failure semantics through experiments, differential execution, historical retrieval, and bounded probes.

Do not invent intent

If two interpretations are observationally equivalent but mean different things to the business, the system cannot discover the intended one. It needs an authoritative contract or it must carry the ambiguity.

Healthcare capability packs

Provider: HL7 V2, FHIR profiles, clinical terminology, encounters, observations, medication events, documents, imaging references, status, and amendment semantics.
Payer: X12 transaction families, enrollment and eligibility spans, claims and lines, adjudication, denials, replacements, reversals, remittance, and prior authorization.
Identity and MDM: patient, member, subscriber, provider, organization, location, device, and product crosswalks, plus reversible merge and split history.
Platform: Spark and Delta, Lakeflow, Unity Catalog, IAM, networking, cost and performance, lineage, disaster recovery, and declarative deployment.
Assurance: PHI flow mapping, minimum necessary, audit, integrity, retention contracts, exact restore, consumer equivalence, and anomaly detection.

Execution model

Safety from invariants, not waiting.

Production execution is allowed when a plan compiles against current contracts and the runtime can prove bounded authority, independent evaluation, and a valid recovery strategy. Nothing here says “the model felt confident.”

Observe

Read-only discovery, profiling, lineage, telemetry, drift, and hypothesis formation.

Experiment

Autonomous synthetic or isolated tests, skill synthesis, counterfactuals, and adversarial evaluation.

Shadow

Production-data mirroring into isolated targets with no downstream authority.

Operate

Autonomous bounded production waves, expansion, quarantine, rollback, and recovery.

Cut over

Autonomous source-of-truth transition when the operating constitution permits it and every declared condition is current. Customer policy can still require manual signoff.

Evolve

Autonomous skill generation, benchmarking, promotion, and demotion inside an immutable meta-contract. The system cannot quietly rewrite its own constitution.

Pre-effect proof

execute only if: current_state.version == command.expected_state plan.digest == capability.plan_digest capability.scope contains requested_effect capability.not_expired source_contract.read_authority is valid target_contract.allows requested_effect evaluator.version == plan.evaluator_version rollback.proven OR plan.declares_irreversible_boundary evidence_sink.available uncertainty <= contract.max_uncertainty projected_blast_radius <= capability.blast_radius otherwise: retrieve_more_context | simulate | choose_alternate_skill | narrow_scope | quarantine | stop_at_contract_boundary

Automatic scope expansion

Wave size grows only when observed correctness, anomaly rate, recovery freshness, cost, and latency stay inside contract. Expansion is an algorithmic state transition, not a calendar milestone.

Automatic stop and recovery

Unknown state, missing evidence, stale capability, unexplained discrepancy, control-plane partition, or regression triggers bounded retry, isolation, rollback, alternate skill selection, or forward repair.

Non-negotiable: the planner does not grade itself, lower tolerances, broaden permissions, hide evidence, or declare a migration successful. Autonomous does not mean self-authorizing.

Platform split

Databricks is the governed execution plane.

Databricks can host live services and agents. I still prefer an external controller because it spans source systems, accounts, workspaces, failures, and long-lived migration state. The target cannot be its own only lifeboat.

Surface	Use	Do not make it
SDK / REST	Authoritative typed lifecycle control for account and workspace resources	An unbounded generic tool exposed directly to a model
Jobs	Bounded extraction, transformation, validation, repair, and rollback stages	The cross-system migration state machine
Lakeflow Pipelines	Checkpointed ingestion, streaming, CDC, and declarative transformations	The human and physical task coordinator
Unity Catalog	Target access control, governance, lineage, audit, and asset ownership	The only evidence ledger or external-source authority
Apps	Workspace-local API and UI, agent service, or custom MCP server	The sole failure-recovery authority
Model Serving	Private request-driven model or agent inference	Durable workflow state
MCP	Narrow governed retrieval and tool interface for agents	A scheduler, transaction system, or unrestricted mutation API
Notebooks / Git folders	Generated artifacts, investigation, and source-controlled implementation	Production orchestration or memory
Genie	Read-only migration and operations analytics where useful	A core dependency for correctness

Deployment decision

Initial: customer-hosted control plane

This matches a literal private-boundary requirement. The control plane, relay, evidence, secrets, models, and detailed context stay customer-side and outside the target workspace failure domain.

Later: SaaS management + customer relay

This is the preferable commercial topology when outbound signed, PHI-free status and digests are permitted. Detailed execution and credentials remain customer-side.

Terminology worth being fussy about: private connectivity is not an air gap, and failure independence is not network privacy. A disconnected environment needs customer-hosted management or offline signed-envelope exchange.

Build sequence

Prove the autonomy kernel before connector breadth.

The first goal is not a demo that generates a notebook. That is a neat trick, not a control plane. The first goal is a complete loop that survives faults, explains itself through evidence, and improves a skill without an approval queue.

Foundation

Migration IR and deterministic state machine

Define contracts, task, run, and wave transitions, capability envelopes, budgets, evidence schema, connector protocol, evaluator protocol, and recovery semantics.

Vertical slice

One synthetic S3 or file batch to one UC table

Plan, stage, evaluate, restore, and reconstruct the run autonomously. Inject duplicate dispatch, crash-after-write, stale capability, malformed records, missing lineage, and evidence failure.

Skill loop

Generate and promote one missing skill

Detect a capability gap, synthesize a candidate, benchmark it against a fixed corpus, promote it inside the meta-contract, then demote it on an injected regression.

Source depth

Add relational snapshot and CDC

Prove that a new connector changes no scheduler, state, authority, or evidence code. Exercise schema drift, deletes, late events, checkpoint loss, and replay.

Healthcare

Add one domain capability pack

Choose payer claims or provider clinical, not both. Add exact standards, semantics, golden cases, expert baseline, anomaly corpus, and acceptance metrics.

Production

Shadow, bounded waves, autonomous cutover

Expand authority only from observed evidence. Keep the source read-only and authoritative until the cutover contract becomes true. Demonstrate automatic rollback and forward recovery.

First-slice exit criteria

Correctness

100% explicit input disposition.
Zero unexplained count or hash differences.
Every seeded defect detected.
Duplicate dispatch yields one logical effect.
Restore returns the exact initial digest.

Autonomy

No per-step human approval.
Restart resumes from durable state.
Unknown conditions trigger narrowing, testing, quarantine, or recovery.
One skill is synthesized, benchmarked, promoted, and safely demoted.
A reviewer can reconstruct the run without chat history.

What I need next

Turn architecture DNA into contracts and evaluation cases.

I do not want to copy a sample architecture literally. I want to extract its invariants, capabilities, assumptions, failure behavior, and expert decisions. Those become the migration contract and the skill benchmark.

System identity and ownership

Exact UDM or MDM product and version.
Authoritative domains and source-of-truth claims.
Mastering, matching, survivorship, merge, and split rules.
Upstream and downstream system owners.

Representative migration

Current and target architecture.
Data dictionaries and synthetic or de-identified examples.
Mappings, transformations, and rejected-record behavior.
Existing control totals and reconciliation reports.

Movement semantics

Volume, history, partition, and retention.
Snapshot consistency mechanism.
CDC logs, watermarks, ordering, deletes, and backfills.
Late data, amendments, reversals, and schema drift.

Healthcare contract

Provider, payer, or both.
FHIR, HL7, X12, and DICOM implementation-guide versions.
Terminology and code-system versions.
Identity and semantic acceptance evidence.

Databricks target

Cloud, regions, accounts, and workspaces.
UC, metastore, catalog, and external-location layout.
Jobs, Pipelines, SQL, and deployment conventions.
Network, identity, secret, and model-serving boundaries.

Operational truth

SLA, RTO, RPO, quality tolerances, and cost constraints.
Cutover, rollback, and source-retirement behavior.
Runbooks, incidents, and undocumented workarounds.
Cases where senior engineers disagreed and reality resolved it.

The best training signal: completed migrations with source artifacts, produced code, defects, review comments, reconciliation evidence, incidents, and final outcomes. These show expert behavior more honestly than a prose description of expertise.

Evidence base

Primary sources and platform facts.

The architecture is my engineering proposal. The platform and regulatory facts underneath it come from the official material below, reviewed on 20 July 2026.