Pattern Matrix/Pattern engineering notes

ADPS Pattern Engineering Note · Memory

Agent memory on Kubernetes: storage layers and recovery tests

Files can remain the agent interface; durable backends handle cross-Pod survival, version conflicts, and recovery.

A reader described a concrete deployment failure. Their agent writes timestamped Markdown files under its workspace. The design works on a laptop, but a Kubernetes service may send the user's next request to another Pod. The preference saved by Pod A is not present in Pod B.

Markdown is not the problem. It defines how the body is represented. The storage design determines whether the record survives a process, Pod, or release. A production system should separate the file view presented to the agent from the durable record behind it.

Write, persistence, and workspace projection for agent memory on Kubernetes

Follow one write from end to end

Suppose a user says: “Write future weekly reports in Chinese. Put risks before progress.”

If Pod A only writes the sentence here, the agent cannot honestly confirm durable storage:

/workspace/memory/preferences.md

An emptyDir survives a container crash within the same Pod, but Kubernetes deletes its contents when that Pod is removed. If the next request reaches Pod B, the file is absent and the service has no durable record of the preference.

A confirmable write has at least six steps:

  1. Resolve tenant_id, user_id, and project_id from authenticated identity.
  2. Convert the sentence into a typed candidate record with source and scope.
  3. Check sensitive data, conflicts, write authority, and the current version.
  4. Write the body and metadata to a durable backend.
  5. Commit the new version and its source event.
  6. Confirm the save to the user only after the durable commit succeeds.

The next request loads the active version by identity and task scope, whichever Pod receives it. If the agent expects file tools, the service may project the selected records into an isolated workspace. That path is a working view, not the only copy.

Four objects that are often conflated

Object Question it answers Common implementations
Content representation How do people and agents read or edit the body? Markdown, JSON, typed fields
Runtime workspace Where does this run keep drafts, downloads, and intermediate files? Pod filesystem, emptyDir, isolated sandbox
System of record Where does recovery start, and who may update the record? PostgreSQL, durable KV, object storage, memory service
Retrieval path How are candidates found at scale? SQL, full-text, vector, graph, or entity indexes

“Markdown or database” therefore compares different layers. A database field can contain a Markdown body. Object storage can hold Markdown while a database owns its scope and version. A vector index stores a retrieval representation; a hit should still resolve to an authorized, versioned source record.

A workable memory record

A filename and a body are not enough for tenant isolation, concurrent updates, or retirement:

{
  "memory_id": "mem_01J...",
  "tenant_id": "tenant_acme",
  "subject_id": "user_1842",
  "project_id": "weekly-report",
  "kind": "preference",
  "body_format": "text/markdown",
  "body": "Use Chinese; put risks before progress.",
  "source_ref": "trace://run-8842/message-6",
  "version": 13,
  "valid_from": "2026-09-12T10:30:00Z",
  "supersedes": "mem_01H...",
  "status": "accepted"
}

subject_id identifies whose memory this is. project_id limits where it may be shared. source_ref points to the event that produced it. version and supersedes make updates explicit. status separates candidates, accepted records, and retired records. The body can remain Markdown.

An optimistic version check prevents two Pods from silently overwriting each other:

UPDATE agent_memory
SET body = :body,
    version = version + 1,
    updated_at = now()
WHERE memory_id = :memory_id
  AND version = :expected_version;

If no row changes, the writer reloads the active version and chooses whether to merge, retry, or escalate. A filesystem design needs an equivalent control: a single writer, lock, atomic replacement, or external version ledger.

What each storage class is for

Storage Appropriate content Controls still required
Pod directory / emptyDir Run-local drafts, caches, downloads, rebuildable intermediates quotas, run isolation, cleanup; never the sole cross-Pod copy
Shared persistent volume Bounded shared material that requires a POSIX file interface access mode, directory partitioning, concurrent writes, backup, retention, small-file measurements
Relational database checkpoints, task progress, user preferences, version chains, access metadata large-body separation, indexes, hot/cold tiers, deletion
S3-compatible object storage large documents, attachments, raw session packages, immutable snapshots database references, digest, authority, version, orphan cleanup
Full-text or vector index candidate retrieval across many records source-version binding, permission filters, deletion synchronization, retrieval evaluation
Dedicated memory store memory APIs that already package scope, versioning, retrieval, and lifecycle verify tenant, authority, audit, export, and deletion behavior

If PostgreSQL is already available, it can hold checkpoints, long-term preferences, and task state. Put large attachments in object storage and add a vector index only when semantic retrieval is needed. Capacity, latency, and ownership boundaries should determine the component count.

LangGraph memory distinguishes checkpoint state from cross-thread storage and uses database-backed checkpointers in its production examples. Deep Agents backends preserve file-oriented tools such as read_file and write_file while allowing paths to resolve through a StoreBackend and namespace. In both cases, the agent can work through a file-like interface while durability belongs to the backend.

A shared volume is valid, but ask the next questions

Mounting a PVC does not prove that every Pod can write safely. Kubernetes PersistentVolume access modes distinguish:

  • ReadWriteOnce: read-write on one node; several Pods on that node may still access it.
  • ReadWriteMany: read-write from many nodes when the storage driver supports it.
  • ReadWriteOncePod: restricted to one Pod for supported CSI volumes.

Access modes govern mounting. They do not provide record-level concurrency control after the volume is mounted.

The effect of many Markdown files on startup also needs measurement at the actual boundary:

  • Does startup traverse the whole directory?
  • Does it parse every body or rebuild an index?
  • Does volume setup recursively change file ownership or permissions?
  • Does the application preload all files?
  • What metadata latency does the shared filesystem add for many small files?

Mounting a volume does not load every file into application memory. Scan, parse, permission, and index work usually create the startup cost. Record mount completion, application readiness, index readiness, and first-request completion separately.

Persistence does not replace lifecycle management

Moving files into a database only prevents them from disappearing with a Pod. It does not settle conflicting versions, project retention, stale indexes, checkpoint boundaries, or a mistaken summary promoted to tenant scope.

Retention, merge, retirement, deletion, and index synchronization need background jobs. High-risk memories also need admission and review. A current agent should not turn its own fresh interpretation directly into a durable organization-wide rule.

Recovery tests before release

Failure or operation Expected observation
Save a preference, remove Pod A, route the next request to Pod B B reads the committed version under the same tenant and user identity
Two Pods update one record concurrently One commit wins; the other detects a version conflict
The durable write fails The agent does not confirm success; the candidate remains retryable or clearly failed
Object upload succeeds but database registration fails The object enters a traceable orphan-cleanup path and is not exposed as active memory
The user deletes a memory Body, caches, and indexes become unavailable within the stated period, with a deletion record
A checkpoint resumes before a tool call The system checks the external receipt before repeating an irreversible action
Another tenant supplies the same memory_id Access is denied before body retrieval, with principal and reason recorded
The corpus grows by an order of magnitude Startup, query, assembly, and archive times are measured separately

Relation to ADPS patterns

  • M1 Hierarchical Retention defines organization, project, user, task, and run scopes.
  • M2 RAG retrieves evidence from large collections. A vector index is a retrieval path, not an authoritative state store.
  • M3 Progress Tracking keeps goals, milestones, checkpoints, and resume positions for long-running work.
  • X1 Observability connects write events, versions, retrieval hits, context assembly, and downstream decisions.
  • X3 Security & Identity constrains tenant, user, task, and resource scope.

Skill distribution is a separate engineering decision. Skills need tests, versions, dependencies, and release policy. Adding that subject here would mix “how user memory survives a Pod” with “how capability packages are shipped.”

References

The source question and original Chinese article date to 12 September 2026. This ADPS engineering note was prepared on 13 September 2026.

Suggested citation: ADPS, Agent Memory on Kubernetes: Storage Layers and Recovery, ADPS Pattern Engineering Note · Memory, 13 September 2026.

Pattern Matrix · Pattern engineering notes · CC BY 4.0

Provenance

Source date
First published here

View in the ADPS Chronicle