Before you read this. This is an engineering demo, not a production reference architecture. Every figure below was measured against a live HPE Data Fabric 8.1 cluster rather than estimated, but it is a single lab cluster with both sites on it — the numbers show that the mechanisms work, not what they do at fleet scale. The repository ships the provisioning, the verification and the failure modes, so the claims can be checked rather than taken on trust.

A field team working off a satellite link cannot pull down everything headquarters has. That sounds like a bandwidth problem, and most designs treat it as one — compress harder, sync less often, buy a bigger pipe. It isn’t. It’s a decision problem. The team needs to know what exists so they can choose the few things worth spending the link on.

Satellite Demo is a working model of that pattern, built on HPE Data Fabric. An HQ site ingests imagery, catalogues it, and continuously broadcasts lightweight descriptions to every edge site. The edge browses those descriptions and requests the handful it actually wants. Only then does the imagery itself move. A vision model at the edge can describe what arrived, so an operator gets an answer without opening every file.

The pattern generalises well beyond satellite imagery — disaster response, maritime operations, remote industrial sites, defence — anywhere the link is expensive, intermittent, or both.

The architecture in one idea

Separate the metadata plane from the data plane, and give them different transports and different cadences.

Descriptions are small, and they travel continuously over a replicated stream. Imagery is large, and it travels only when asked for, by volume mirroring the edge triggers itself. Two planes, two mechanisms, two economics.

That separation is what makes the system usable on a bad link. The catalogue stays current for pennies. The expensive transfer happens under human control, at a moment someone has decided is worth it.

One pipeline, four data models

Here is what makes this interesting as a platform story. That single pipeline touches four different kinds of data, and each wants a different interface:

None of that required a second system. No message broker to deploy, no object store to stand up, no separate catalog service, no file-transfer scheduler. One cluster, one namespace, one identity — addressed through whichever standard interface each part of the pipeline naturally speaks.

That is what multi-modal and multi-protocol mean in practice: the platform is not a file system with an object gateway bolted on, or a message queue with a storage tier attached. Files, objects, streams and tables are first-class, they live in the same namespace, and they are governed by the same security model.

The practical consequence is that data does not have to move between systems to become usable by a different consumer. An operator’s imagery lands on a volume via POSIX; the catalogue that indexes it is an Iceberg table on S3; the notification that it exists is a stream event. Same cluster, same volumes, same credentials — no ETL between tiers whose only purpose is to satisfy the next tool’s preferred protocol.

What you would otherwise assemble

It is worth being concrete about the alternative, because “one platform” is easy to say and hard to evaluate. To build this pattern on general-purpose components you would need, at minimum:

Every row in that middle column is a component to deploy, secure, monitor, patch, and capacity-plan — at every site, including the ones with no staff and a satellite link.

And each brings its own disconnection semantics. MirrorMaker’s lag behaviour is not rsync’s retry behaviour, which is not object replication’s queueing behaviour. When the link drops, you are reasoning about three or four independent recovery models at once, and the interactions between them are where the incidents come from.

On Data Fabric, disconnection is handled the same way for everything, because it is the platform’s concern rather than each component’s.

Cutting the link left HQ publishing normally while the edge stayed frozen and the backlog grew to 21 messages; restoring it drained to zero within seconds. Imagery behaved consistently: a requested asset simply sat staged in HQ’s outbound volume until the edge chose to mirror it. Nothing about that is simulated, and none of it is application code.

These are ordinary Data Fabric objects, visible and managed like any other — the edge’s imagery volume is a mirror, and the platform knows it:

What still needs designing

The platform removes the infrastructure problem. It does not remove the design problem, and three decisions mattered more than any other.

Model the link as a state, not an error

The instinct is to treat disconnection as an exception. It isn’t — for an edge deployment it is a normal operating mode, and often a scheduled one. Designing for three explicit states (connected, scheduled, cut) produced a much better system than designing for “connected, with error handling”. The platform supports this directly: pausing and resuming replication is a supported operation, and mirrors are on-demand by nature.

The corollary is to show the link’s real state, read from the cluster, rather than inferring it from whether your own messages happen to be flowing.

Audit what you replicate

HQ’s internal service hand-off runs on its own unreplicated stream. It would have been one line of configuration to put it on the replicated pair — and it would have pushed HQ’s private chatter down the very link the design exists to conserve.

Because replication is so easy to enable, it is equally easy to enable by accident. On a constrained link that is a real and recurring cost. Decide deliberately what crosses.

Name objects by site, not by cluster

Each site owns its own volumes, streams and buckets, distinguished by name rather than by which cluster they happen to live on. The system then behaves identically whether both sites share one cluster or sit on two with a trust relationship. One is not a degraded version of the other, and moving from one to two is a configuration change rather than a redesign.

From two sites to a thousand

This demo is a baseline, not a blueprint. A real deployment has tens or thousands of edge sites, and at that scale the platform features that matter are the ones you never see in a two-site demo. It is worth naming them, because they are the reason this pattern scales without becoming a bespoke integration project per location.

One namespace across many fabrics

Data Fabric’s global namespace aggregates remote and disparate data sources so that multiple fabrics can be viewed and operated as a single logical, local fabric. An application addressing data at another site does not need to know it is remote.

Naming then becomes a fleet convention rather than a per-deployment choice: every cluster in the namespace needs a unique name, and mapr-clusters.conf must carry the same cluster configuration and naming across nodes and clients. The two-site convention in this demo scales only if it is designed as a scheme — region, role, site identifier — before the fleet exists rather than after.

Authorization that spans clusters

For clusters to communicate at all, a secure trust relationship must exist between them. That trust is what permits remote commands, remote replicas and mirrors, and NFS access to another cluster — the exact operations this pattern depends on.

Setting it up is a defined procedure rather than a bespoke integration: configure-crosscluster.sh generates a cross-cluster ticket, copies it to the other cluster’s CLDB node, merges it into that cluster’s maprserverticket and distributes it; manageSSLKeys.sh merges trust stores when a client must reach several clusters. Identity remains the platform’s, not each component’s — which is the point. In an assembled stack, cross-site authorization means federating several independent identity systems and keeping them consistent across sites you cannot reach.

The link itself is managed

Data Fabric treats a constrained link as something to manage rather than saturate.

Mirroring throttles itself. The sending server continuously measures round-trip time and restricts mirror traffic to roughly 30% of available bandwidth, backing off when other traffic needs it. Throttling can be disabled where a maintenance window justifies full speed.

Replication is compressed and can be encrypted. Data Fabric applies network compression — lz4 by default — and traffic between secure clusters can be encrypted on the wire. In this demo the replica record reports exactly that: networkcompression: lz4, networkencryption: false on a lab cluster where encryption was not required.

Mirrors run on schedules with priorities. Schedules carry pre-assigned meanings — critical, important, normal — so a fleet can express that some sites sync hourly and others when someone asks, without writing a scheduler.

Operations and observability at fleet scale

Data Fabric Monitoring collects metrics and logs for nodes, services and jobs, which is what makes a fleet observable centrally rather than site by site. Note that monitoring components are not installed on client or edge nodes — an edge site reports through its own cluster, so the observability design follows the fabric topology.

For genuinely small sites there is HPE Data Fabric Edge: a small-footprint edition running on commodity hardware in three- to five-node configurations, with the full capability set — files, tables and streams, plus snapshots, mirroring, replication and compression. An edge site is a small fabric, not a cut-down client, which is why the same design works at both ends.

Trying it

The demo runs as a single container against your own cluster. Both sites can point at the same cluster — they stay separated by volume, stream and bucket names rather than by geography — so you do not need two clusters to see the pattern work:

git clone https://github.com/erdincka/satellite
cd satellite
docker compose up -d

It starts with no configuration and is pointed at a cluster from the interface, which is the easiest way to try it. The README covers the prerequisites that actually matter — a reachable CLDB, apiserver, S3 and NFS, a client version matching the cluster, and CAP_SYS_ADMIN on the container because imagery is reached over an NFS mount.

The takeaway

The interesting claim isn’t that Data Fabric can move data between sites. Plenty of things move data. It is that a pipeline spanning streams, files, tables and objects — across sites, over a link that comes and goes — can be built on one platform, through standard interfaces, instead of assembled from components that each need their own deployment, security domain, replication mechanism and failure model.

That leaves the application with the one job it should have: deciding what is worth sending, and when. The fabric handles delivery, queueing, resumption and consistency underneath.

The application decides. The fabric delivers.

Before you read this. This is an engineering demo, not a production reference architecture. I used AI extensively while building and documenting it, but I reviewed the implementation and validated every claim below against a running Data Fabric 8.1.0 cluster. The repository ships the verification steps and the failure modes I hit, so the claims can be checked rather than taken on trust.

It deliberately does not try to be a data platform. There is no governance layer, no lineage graph, no catalogue of catalogues, and the fraud model is four lines of arithmetic. The goal is narrower: to show what changes when the storage layer stops being a thing you integrate with and starts being a thing you write to.

Every data platform conversation I have ends up at the same question, and it is never “which engine”. It is how many systems is that, and who runs them?

The honest answer, for a fairly ordinary medallion architecture, is usually uncomfortable. You want streaming, so that is Kafka and ZooKeeper. You want a document store for raw events. You want open table formats, so that is Iceberg, and Iceberg wants a catalogue service, and the catalogue wants a database. You want object storage. You want the whole thing visible as files for the tools that only speak POSIX. You want metrics, so there is a time-series database and a dashboard in front of it. None of these are unreasonable. Together they are a platform team’s entire year.

CatchX is a working demonstration of the other option. It is a fraud detection pipeline — transactions arrive on a stream, customers arrive as a CSV, both move through bronze, silver and gold into a shareable data product with suspected fraud flagged. That part is an ordinary data engineering demo. The part worth an infrastructure architect’s time is that the six steps below touch five different data engines, and the number of services I had to deploy to support them is zero. Streams, document store, NFS and object store are all fabric packages on the cluster I already had. There is no broker, no ZooKeeper, no catalogue service, no metastore database and no time-series store anywhere in this demo.

There is almost no fraud detection in it, which is the point.

I wrote about the same platform from the other direction in Deciding what crosses the link, which takes one workload across two sites and asks what is worth moving. This one stays on a single cluster and asks the opposite question: how many different engines can you serve from it before you start deploying things?

The shape of the thing

Six steps, and every one of them writes to the same cluster.

Five engines. One /mapr mount. No Kafka cluster, no catalogue service, no metastore database, no separate object store to stand up.

A full clean run — reset, provision, and all six steps against a single-node cluster — takes 1 minute 53 seconds. Almost all of it is per-document DocumentDB writes, which is a property of the demo’s chattiness rather than the platform’s.

What “one namespace” actually means

This is the claim that sounds like marketing until you run ls against it.

The app generates its CSVs by opening a POSIX path and writing to it. No upload step, no staging bucket, no client library. Then the same directory, listed over NFS, shows this:

drwxr-xr-x 2 mapr mapr     1 bronze
lr-------- 1 mapr mapr     3 changelog -> mapr::table::3102.97.131518
-rw-r--r-- 1 root root 55713 customers.csv
drwxr-xr-x 2 mapr mapr     0 gold
-rw-r--r-- 1 root root 20480 iceberg.db
lr-------- 1 mapr mapr     3 incoming -> mapr::table::3102.94.131512
drwxr-xr-x 2 mapr mapr     0 silver
-rw-r--r-- 1 root root 18339 transactions.csv

A CSV, a SQLite file, three volumes and two streams, in one directory listing. The streams are not in a broker somewhere with their own retention config and their own ACL model — they are entries in the filesystem, and they are governed by the same volume that governs the CSV next to them.

The Iceberg catalogue is the sharpest example. Iceberg normally needs a catalogue service to answer “where is table X”. Here it is iceberg.db, a SQLite file, sitting in the namespace. Every client that can reach the mount can reach the catalogue, which means there is no catalogue service to deploy, secure, back up or scale. That is not the right answer for a thousand-user estate. It is emphatically the right answer for a demo, and the fact that it is possible is the platform property worth noticing: the namespace is the coordination point, so a lot of coordination infrastructure stops being necessary.

Standard clients, not adapters

The demo’s most persuasive screen is the least dramatic one. Every step has a </> button that shows the function that ran, then follows the call chain down to whatever actually talked to the cluster.

Publishing to a fabric stream is this:

from confluent_kafka import Producer, KafkaException

producer = Producer({"streams.producer.default.stream": stream})
producer.produce(topic, message.encode("utf-8"))

That is the stock confluent_kafka library. Not a fork, not a shim. The entire fabric-specific part is one configuration key naming a path in the namespace instead of a bootstrap server list. Everything downstream of that — consumer groups, offsets, enable.auto.commit, the lot — behaves the way the Kafka documentation says it does.

I care about this more than any throughput number, because it decides what happens when the demo ends. Standard protocols mean the migration question is “change a connection string”, and the skills question is “your team already knows this”. A platform that requires a bespoke client is a platform you have to staff for.

Two table formats in one tier, because the tier does not care

Bronze holds transactions in DocumentDB and customers in Iceberg, at the same time, in the same volume.

That is not cleverness for its own sake. Streaming JSON documents and batch columnar loads have genuinely different access patterns, and a tier that forces one format on both is making you choose against the workload. Here the choice is per-table, and the tier boundary stays where it belongs — a statement about data maturity, not about storage technology.

Gold then uses Delta Lake, which any Delta-aware engine can read straight from the namespace. Three table formats in one pipeline, none of which needed anything installed.

Where the data product becomes shareable

The tier boundary earns its keep in the silver step, and this is the demo’s best moment in front of a governance-minded audience.

The same customer records that were fully visible in bronze — names, dates of birth, locations — come back from silver with birthdate and current_location reading masked, and with ISO 3166-2 subdivision codes added that were not there before. Enrichment and redaction in the same pass, because they are the same decision: what does the next consumer need, and what should it never see?

Gold goes further and drops the direct identifiers entirely, so the flagged-fraud table an analyst consumes carries amounts, categories and timestamps but no account numbers.

Telemetry without a telemetry stack

The record counts in the diagram are not the app remembering what you clicked. Every one of them is read from the cluster: DocumentDB counts by projecting _id, Iceberg and Delta by reading the table, stream depth and consumer lag from stream/topic/info and stream/cursor/list on the cluster’s REST API.

The practical consequence is that step completion is derived from cluster state, not from session state. Reload the page and progress survives. Run half the demo from curl and the UI shows it. Hand the laptop to a colleague mid-presentation and it is still correct. I did not click a single button in the UI before the screenshot above showed all six steps complete — an API run had already done it, and the app simply reported what was true.

There is no Grafana here, no OpenTSDB, no metrics agent. For a demo that needs live numbers, the cluster’s own API was enough, and every service I did not add is a service that cannot fail on stage.

“Provisioned” and “working” are different questions

Now the part that actually cost me a day, because it is the transferable lesson and because a post that only lists capabilities is an advert.

The demo has a reset button. It deleted the four volumes and recreated them, so you could run the whole thing again from a clean cluster. It had presumably worked at some point. When I ran it against this cluster, every DocumentDB write after a reset failed:

insertOrReplace() failed with err code = 19

Nineteen is ENODEV. The code already had a retry loop for this, because a freshly provisioned volume does take a few seconds to serve — so the first theory was that the backoff was too short. It was not. The writes still failed an hour later.

Second theory: the OJAI client caches its connection for the life of the process, and the connection was stale. I restarted the application. Same error.

Third: check whether the table was actually broken, from somewhere else entirely. It was not — dbshell, a fresh process on the cluster node, inserted a document into the same table without complaint.

Which located it. The error was not in my client and not in the table. It was in the Data Access Gateway, the service that speaks OJAI on port 5678:

Inode - syncPut() on '/catchx-demo/bronze/transactions'
        failed with error: No such device (19).

The gateway holds its own MapR client, and that client had resolved the path while the old volume existed. Delete the volume, create a new one at the same mount point, and the gateway keeps serving the handle it already had. Nothing client-side can fix it; only restarting the gateway clears it.

Then the decisive experiment, which took two minutes and should have been the first thing I did: delete and recreate a table at the same path, leaving the volume alone. That works perfectly. The blast radius is volumes, not paths.

So the fix was not a longer retry or a reconnect. It was to stop deleting volumes. The reset now removes the streams, tables and files and leaves the four volumes in place, which is both correct and about twice as fast.

Three things from that are worth carrying elsewhere:

A distributed storage layer is still made of services, and services have caches. “One namespace” is a real and useful abstraction, and it has seams. The seam here was a component I had not thought about all day, holding state I did not know it held.

Retry logic encodes a theory about the failure. The existing backoff assumed “transient, will settle”. That assumption was invisible until a failure that looked identical but was permanent walked into it. A retry loop that cannot distinguish the two will happily hide the second for fifteen seconds and then report the wrong thing.

Ask a different client. The single most useful step in the whole investigation was running dbshell on the cluster. Two minutes, and it cut the search space in half by proving the data layer was fine. I had spent considerably longer than two minutes theorising before I did it.

While I was in there, a related one: the same cached-connection design meant that after any gateway restart, the application never recovered until it was restarted too. That is now fixed as well — the connection is dropped and rebuilt when a channel error is seen — and it is the more common failure in real life, because gateways get restarted for ordinary reasons.

And a cluster-side finding for anyone benchmarking this: check whether your Data Access Gateway is still logging at debug. The default log4j2.xml on this cluster writes every gRPC frame and every document payload to disk. It dominated my write latency, and no amount of application tuning would have touched it.

What it does not show

The honest limits, because the demo is small on purpose.

The one operational claim I will make firmly is the narrow one I actually verified: reset the demo and run it again, repeatedly, and it completes with correct counts at every tier without anything on the cluster being restarted. That was not true when I started.

Worth your time?

If you are choosing a data platform, the interesting question is not whether it can do streaming, documents, tables and files. Most can, eventually, with enough components. It is how many of those components you are agreeing to operate, and whether the code you write against them is code you could write against anything else.

CatchX answers the second question well — stock Kafka, stock OJAI, stock Iceberg and Delta, with the fabric appearing as a path rather than as an SDK. It answers the first question with a number: five engines, one mount, nothing extra installed.

Then it spends a day teaching you that the abstraction has seams, and where one of them is. Which is the more useful lesson of the two.

The code, the verification steps and the failure modes are at github.com/erdincka/catchx. It runs against any Data Fabric 7.x or later cluster; use a lab one, since it creates and deletes its own artefacts.

Before you read this. This is an engineering demo, not a production reference architecture. I used AI extensively while building and documenting it, but I reviewed the implementation and, importantly, validated the security claims against a running Kubernetes cluster. The repository ships the verification commands and the failure modes I hit, so the claims can be checked rather than taken on trust.

It deliberately does not try to solve every part of a production agent platform. There is no elaborate retrieval pipeline, no model evaluation framework, no comprehensive guardrail system and no production-grade observability stack. The goal is narrower: to show how Kubernetes primitives can enforce selected agent capabilities independently of the model.

Sooner or later someone is going to ask your platform team to run AI agents in production, if they haven’t done already. Not a chatbot — agents: things that hold credentials, query databases, call internal APIs, and occasionally write and execute code. And when they do, the question that lands on infrastructure is not “which model” but a much older one: what stops it doing that?

The uncomfortable answer, in most agent frameworks today, is a paragraph of English in a system prompt. You must not access customer records. Do not run code unless asked. That is a request, not a control. It has no audit trail, there is nothing to point a security reviewer at, and it competes for the model’s attention with whatever the user sent and whatever the retrieval step pulled in. Instructions and input do arrive as separate message roles, but the only thing weighing one against the other is the model — which is the component you were trying to constrain in the first place. You can try adding guardrails before and after and during the calls etc, but with the complexity and cost of these extra layers, it’s not always worth it or simply not manageable at scale.

Agentic Meetings is a working demonstration of the alternative, built on Kubernetes Agent Sandbox — a Kubernetes SIG Apps project that adds Sandbox, SandboxClaim, SandboxTemplate and SandboxWarmPool as first-class API objects. The application on top is a multi-agent meeting simulator: a supervisor picks who speaks next, and each participant argues from their organisational role over the company’s documents. That part is an ordinary LLM demo. The part worth an infrastructure architect’s time is underneath — every agent runs in its own gVisor-isolated sandbox, and what each one may do is a Kubernetes authorisation decision.

This piece is about how it is put together and what it is like to operate. There is almost no machine learning in it, which is the point.

The shape of the thing

Three trust boundaries, crossed once per turn:

The backend holds the orchestration graph, the database credential, and the conversation state. Tier A is one gVisor-isolated pod per meeting attendee, drawn from a warm pool, running that persona’s reasoning loop. Tier B is a second, stricter sandbox tier — no network at all — where any Python the model writes actually executes.

The interesting edge is the one from Tier A to Tier B. When an agent decides it needs to run code, its own pod creates a SandboxClaim against the API server using its own ServiceAccount token. The backend is not in that path and cannot broker around the decision. If the persona’s profile carries the right role, the claim succeeds. If it does not, the API server returns 403 — and the agent is told so, in-band, as a tool result.

That last detail is a design choice worth copying. The denial is reported, not raised: the tool returns DENIED_BY_CLUSTER: ..., the agent absorbs it and carries on contributing to the meeting, and the refusal lands in the transcript and the audit matrix as a policy decision rather than a stack trace. A control that crashes the workload gets switched off. A control that degrades it gracefully survives contact with users.

What this is, and is not, trying to protect

The boundary here is deliberately narrow, so it is worth stating before the mechanics.

The model is treated as unreliable. It may be wrong, it may follow instructions buried in something it retrieved, and it may reach for a capability its persona was never granted. The goal is that a specific set of capabilities is enforced below it — by Kubernetes authorization, workload isolation, mounted credentials and network policy — so that whether it cooperates stops being the deciding factor.

Everything below is about the first three rows. The last two are the honest limits, and I come back to them at the end.

Five layers, and the ones that matter

The project describes its enforcement as five layers, and is honest about which are load-bearing:

Read that bottom-up and it is a familiar defence-in-depth story. The top layer is the one every agent framework ships with. It is genuinely useful for steering behaviour — it prevents accidental tool use and sets the default — but it is not a security boundary, because nothing below the model is obliged to honour it. The bottom two are ordinary Kubernetes objects that your cluster already knows how to enforce.

That is the real argument for doing this on Kubernetes rather than inside an agent framework: you are not inventing a permission system. You are reusing the one that already governs everything else you run, expressed in primitives your platform team can already reason about and wire into whatever authorization, policy and audit tooling they have. Kubernetes supplies the primitives and the API audit surface; whether those logs are retained, correlated and actually read is an organisational question, not something the design hands you.

What it’s like to use

You configure it, define who is in the room, and watch a meeting run.

Personas are data, edited in the UI: a role, a tone, a risk level, and a set of granted tools. The grant is the part with teeth — a persona’s tool list resolves to a capability profile, and the profile decides which ServiceAccount its sandbox runs under. Editing a persona is therefore a privilege change, and the app treats it as one: the operator role can do it, the viewer role cannot. That split is not the usual read/write one, and getting it right required noticing that “edit this text field” and “grant this agent the ability to execute code” were the same action.

Start a meeting and the supervisor takes over: it picks a speaker, that persona’s sandbox runs a turn, the utterance streams back over a WebSocket, and the events log records every routing decision beside the transcript. Ask the General Counsel persona to run a calculation and, if the model reaches for the tool, you get the demonstrable moment — a 403 from the API server, surfaced in the audit matrix. Whether it reaches for the tool on any given turn is the model’s decision, and smaller ones are inconsistent about it. The cluster’s answer does not depend on that, which is the next thing to show.

And you do not have to take the UI’s word for it, which is the part that matters if you are the one signing off on this. The same decision is one command away:

Three things there are worth separating. /proc/version returning 4.19.0-gvisor proves the kernel boundary is real rather than a silent fallback to runc. The pod listing shows one ServiceAccount per persona, all under the gvisor runtime class. And kubectl auth can-i answers the actual question — no for the General Counsel, yes for the Finance Director — without involving the model, the prompt or the application at all.

That last property is what makes this auditable. Your security reviewer does not have to trust the app’s audit matrix, or reason about whether a model could be talked out of its instructions. They can ask the API server directly, the same way they would for any other workload on the cluster.

Meetings end with a decision record — notes, agreed actions, identified gaps — rather than just a transcript.

“Installed” and “working” are different questions

This is the transferable lesson, and the project hit it at five separate layers.

A misconfigured RuntimeClass handler does not fail loudly — on several container runtimes, containerd silently falls back to runc. The pod is green, Ready, and completely unisolated. A CNI that ignores NetworkPolicy looks identical to one that enforces it: the policies apply cleanly, kubectl get networkpolicy is reassuring, and nothing is blocked. On kind’s default CNI, a supposedly-restricted persona reached Postgres on five consecutive attempts while the security model claimed the opposite. Gateway API CRDs can be present with no controller watching them — Gateway API is a specification, and installing its CRDs gives you no controller and no dataplane. An internal API router can be written, reviewed, unit-tested and never mounted. And, as above, an egress rule can name a port when you believed it named a destination.

Each of those presents as a healthy deployment. The habit that falls out is provoke the behaviour and observe what happens — not “is the object present”, not “is the pod ready”. The repo ships a make preflight that reports, for the requirements that can be present and inert, whether the thing actually functions. Isolation is verified by reading /proc/version inside a running sandbox, never by a readiness check.

Five decisions an infrastructure architect will care about

1. Keep the orchestration graph in one place

It is tempting to distribute the agent graph across the sandboxes. Don’t. Sandboxes here are turn executors, not graph participants: the backend owns the graph and the state, and hands a sandbox one self-contained turn.

The alternative was tried. Running the reasoning loop remotely while RPC-ing individual tool calls back cost a network round trip per tool call and turned a demo into a distributed-systems project. Distributed checkpointing across untrusted pods is a research problem; it is not a prerequisite for isolating agents.

2. Size the warm pool by concurrent speakers, not attendees

gVisor pods are not fast to start, so both tiers are warm-pooled. Two operational details:

• Acquisition is lazy. A five-person meeting where two attendees never speak should not hold five pods. The claim happens on first selection.
• A sandbox is held per persona, not per turn. Claiming per turn drains a pool of two by the third turn of a four-person meeting, and everyone after that pays a cold start or fails outright.

Budget roughly 8 CPU and 16 GiB allocatable for a full demo, most of it warm pool sitting idle. It runs smaller by reducing warm replicas, at the cost of a several-second pause the first time each persona speaks — worth knowing before someone asks why idle pods are consuming quota.

3. Decide what a sandbox never holds

The rule here is absolute: sandboxes never hold the application database credential. Everything a persona needs arrives through a scoped internal API on the backend, authenticated with the sandbox’s own projected ServiceAccount token — validated by the API server, with identity read from pod labels rather than from a request body the model composed.

Model calls go the same way. Sandboxes never call the inference provider; they reach it through the backend’s /internal/v1/llm proxy, so a persona sandbox holds no provider key and the provider can sit behind a CDN with no stable address.

Which was almost the neatest example in this whole piece of a control that reads as closed and is not. Until recently that policy allowed 0.0.0.0/0 on TCP 443 and 6443, because the apiserver has to stay reachable — block it at the network and an RBAC refusal becomes a connection timeout, which is worse than no control at all. But 443 is the whole HTTPS internet. The rule was a port restriction wearing the costume of a destination one, and the comment beside it cheerfully claimed the sandbox had no path off the cluster.

It does now. The policy is pinned to the apiserver’s own addresses — both the Service ClusterIP and the real endpoints, because whether a NetworkPolicy implementation matches one or the other depends on which side of DNAT it evaluates. A sandbox still reaches the apiserver on the ClusterIP; a connection to 1.1.1.1:443 is now refused.

I mention it rather than quietly fixing it because it is the same failure this post keeps circling: not a control that was missing, but one that was present, plausible, and broader than its own documentation said.

4. Make the boundary legible from outside

Telemetry is optional here and off by default, on the grounds that observability must never be why a system fails to start. When it is on, a single traceparent is propagated over the sandbox RPC and again when a persona claims an exec sandbox, so one meeting turn renders as one trace spanning all three trust boundaries.

That is what makes slow, denied and broken three distinguishable outcomes rather than one indistinguishable pause. A system where those three look identical from the outside is not one you should grant more autonomy to, however good its isolation is.

Two supporting choices: metric labels are low-cardinality by construction — a persona’s profile, never its agent id — and denials are counted separately from errors, because collapsing them would hide the one signal the project exists to surface.

5. Deploys that report success and change nothing

Images are tagged by content digest, not :latest. A mutable tag leaves the Deployment spec unchanged when the image is rebuilt, so helm upgrade finds nothing to roll and the pod keeps serving stale code — a deploy that reports success and changed nothing. Images are also cosign-signed with SLSA provenance, and make verify-images answers “did this come from this repository’s CI”, which is a different question from “did it change”.

Schema changes run as an Alembic Helm pre-upgrade hook rather than as a startup side effect, so a migration is explicit and reviewable. Configuration is environment-only — credentials from Secrets, operator-tunable values in a database table — and attempting to set a credential through the settings API returns a 422.

What it does not claim

Worth stating plainly, because demos in this space usually overclaim.

gVisor is a strong boundary, not a perfect one. The five layers stop specific, enumerated things and the repo documents what each one measures. NetworkPolicy is only a control if your CNI enforces it — on a cluster where it does not, two of the five layers are decorative, and the preflight check says so rather than pretending otherwise.

The backend is the part to be most honest about. It is not assumed trustworthy: a compromised backend still cannot grant a persona more than its sandbox template already provisions, because the runtime intersects what the backend asks for with the capability list mounted into the pod. But it owns the conversation state, the database credential, the orchestration and the model proxy, so compromising it remains a serious event. What this design contains is untrusted agent execution. It does not make the application as a whole compromise-resistant, and it trusts the Kubernetes control plane outright.

Which is the honest version of the tagline. The cluster does not make the model trustworthy, and it has nothing to say about what the model concludes, what it emits, or what it is talked into by something it retrieved. It moves a specific set of capability decisions below the model, into infrastructure that answers the same way whether or not the model cooperates. For those decisions — and only those — the model does not get the final say.

Trying it

You need a cluster that can actually support the controls:

That version floor deserves a footnote, because two things set it and neither is obvious.

ImageVolume is how this demo gets pgvector into Postgres: the extension is mounted from an OCI image, so its version is a value in the cluster manifest instead of a Dockerfile you own and have to rebuild. It went alpha in Kubernetes 1.32 behind a feature gate, beta and on by default in 1.33, and by 1.35 the gate is gone. So on its own it would put the floor at 1.33.

CloudNativePG raises it. Version 1.30 lists Kubernetes 1.34, 1.35 and 1.36 as supported; 1.31 through 1.33 are tested but explicitly not supported. Taking the higher of the two, 1.34+ is the version to build against — and if you read an earlier draft of this post or the repository saying 1.31+, that was wrong, and it is the kind of number worth checking rather than inheriting.

None of this is load-bearing for the architecture. If you would rather bake pgvector into a Postgres image the conventional way, that is a perfectly good route and it relaxes the Kubernetes floor considerably — you would just be changing the deployment, because this repo ships the one mechanism.

Then:

brew install kubectl helm kubeconform cosign uv
cp deploy/cluster/cluster.env.example deploy/cluster/cluster.env   # then edit
make preflight

preflight tells you what is missing and, for the requirements that can be present and inert, whether they actually work. Install what it flags, then:

deploy/cluster/install-prerequisites.sh all
make verify-images && make deploy && make seed
make operator-token

Inference is any OpenAI-compatible endpoint — Ollama on your own machine works for development. There is deliberately no in-cluster model server: serving a model well is a different problem from orchestrating agents, and running both on one cluster makes the demo compete for CPU with the sandboxes it exists to serve.

The repo’s architecture, sandbox security model and lessons learned docs go deeper than this piece does — the last of those is unusually candid about controls that looked right and did nothing.

The takeaway

The agent-isolation problem is not new, and it is not really an AI problem. It is the multi-tenancy problem, arriving through a new door: untrusted code, generated at runtime, that wants credentials and network access.

Kubernetes has spent a decade building answers to that — namespaces, ServiceAccounts, RBAC, NetworkPolicy, runtime classes, admission control — and Agent Sandbox is the work of wiring agent workloads into them rather than inventing a parallel permission system inside a Python framework.

For an infrastructure architect, that reframing is the useful part. You do not need to become an ML engineer to have an opinion about how agents should be run. You need to insist that the enforcement live where you can see it, test it, and show it to an auditor — and then treat “the prompt says not to” as what it is, which is a comment.