0040 — Pluggable object stores (GCS, Azure, and the versioning question)¶

Status: IMPLEMENTED (2026-07-25) — P1 (content-addressed blobs + version ledger + media versions/rollback), P2 (portable gocloud.dev/blob adapter: gcs · azure · file · mem), P3 (media migrate, media gc). Remaining follow-up: real-cloud GCS/Azure credential + emulator testing when a first user appears, and running media migrate over the live blog workspaces.

Superseded detail from the original status line: Content-addressed blobs + git-tracked version ledger + media versions/rollback, and the portable gocloud.dev/blob adapter registering gcs · azure · file · mem alongside the native s3 adapter. Live-verified on minio (via the portable adapter) and end-to-end on the file backend — push → version bump → rollback with no cloud and no credentials. Remaining: P3 (media migrate, media gc), real-cloud GCS/Azure credential + emulator testing when a first user appears. Date: 2026-07-25

Resolved (2026-07-25, with Matt): - Direction = Option 3 (§4.3): content-addressed blobs via gocloud.dev/blob (no per-cloud adapters) plus a keryx-owned, git-tracked monotonic version ledger (V1/V2/V3). A sha256 is the identity but not a human version; the store never provides human versions anyway (even S3 version_ids are opaque), so that layer lives in keryx's manifest, decoupled from the backend. - Q2 provider selector → discrete (storage.provider: s3|gcs|azure|file; keryx maps to the gocloud driver, keeps the s3 block shape). - Q3 migration → one-time keryx media migrate per workspace: reads the existing media.lock (already carries each file's sha256), re-keys objects to blobs/<sha256>, seeds every file at V1. Idempotent. (P3.) - Q4 GC → deferred to P3 (keryx media gc); design now, ship later. - Q6 ledger → separate media.log (media.lock stays lean = current pins; media.log = append-only V-history), and version bumps on Commit & push (a version = a distinct committed content state, aligned with git history; an uncommitted re-roll isn't a version yet). - Scope: P1 (CAS core + ledger, over S3) starts now; GCS/Azure = P2, deferred. Relates: 0039 (media object store — the S3 adapter + media.lock model this generalises), 0001 §3.4 (pluggable backends behind config-chosen interfaces).

1. Goal¶

Let a consuming project point keryx's media store at Google Cloud Storage or Azure Blob Storage (not just AWS S3), with as little bespoke code to build/maintain as possible — ideally leaning on a maintained multi-cloud library rather than hand-writing a versioned adapter per cloud.

2. The backend landscape — most "other" stores already work¶

A crucial framing before adding anything: the large majority of S3-alternative object stores speak the S3 API, and keryx already supports them today via the storage.s3.endpoint knob (0039 §3.4). No new code needed for:

The genuinely different APIs — the ones needing real adapter work — are just two: GCS and Azure Blob. (Local filesystem + in-memory are also worth having for dev/self-hosted/tests.) So "support more clouds" is, concretely, "support GCS + Azure without hand-rolling and maintaining two more versioned SDKs."

3. Existing libraries — can we avoid building our own?¶

Evaluated the maintained Go multi-cloud blob abstractions:

The decisive finding: gocloud.dev/blob is exactly the multi-backend CRUD layer we'd want (S3 + GCS + Azure + file + mem behind one interface, well maintained, CGO-free) — but its portable API exposes no object versioning / version IDs. It's a deliberate lowest-common-denominator: Read / Write / List / Delete / Attributes / SignedURL, nothing more. (It also doesn't create buckets — fine, keryx never does; infra does.)

That matters because 0039's media.lock pins the S3 version_id of every file, and media pull fetches that exact version. No portable library exposes that, and per-cloud versioning is three different concepts (S3 versionId, GCS generation, Azure Blob versionId/snapshots). So we can't just drop a library in and keep the current model.

This forces the real decision.

4. The fork: how do we pin versions across backends?¶

Option 1 — Keep the version_id model; hand-write versioned adapters¶

Keep the objectstore.Store interface as-is (Put→VersionID, Get(key, versionID)), and implement GCS (generation) and Azure (versionId) natively, mapping each cloud's versioning concept onto the interface. No library helps — gocloud can't carry versioning, so this is bespoke code + integration tests per cloud, forever. Matches today's architecture; incremental; but it's exactly the "build and maintain our own per-cloud adapters" Matt wants to avoid, ×2 (then ×N for any future cloud).

Option 2 — Content-addressed storage; adopt gocloud.dev/blob (recommended)¶

Stop pinning mutable-key versions and store each blob at an immutable, content-addressed key — the sha256 keryx already computes and already records in media.lock:

<prefix>/blobs/<sha256[:2]>/<sha256>        # immutable; the content IS the version

media.lock already maps vo/01.mp3 → {sha256, size}; the version_id field simply disappears — the hash is the version. Consequences:

• Versioning becomes a non-problem. No version_id, no bucket-versioning requirement, no per-cloud versioning concept to reconcile. Point-in-time restore is unchanged and stronger: each git commit's media.lock names the exact content hashes; media pull fetches those blobs and verifies by construction (the key is the checksum).
• Any blob store works — so we adopt gocloud.dev/blob for CRUD and get S3 + GCS + Azure + filesystem + in-memory for free, maintained upstream. keryx keeps only a thin wrapper (URL/config → blob.Bucket) plus the content-address key logic. This is the "don't build our own" answer.
• Dedup for free — identical takes/covers across reels share one blob.
• Integrity for free — a fetched blob whose bytes don't hash to its key is detectably corrupt.

Costs / new work: - Key-layout migration from 0039's reels/<slug>/… paths to blobs/<hash> (existing media.locks already carry the sha256, so the manifest side needs no new data — only a re-key/re-upload of existing objects, or dual-read during a transition). - Garbage collection replaces S3 noncurrent-version lifecycle: keryx never deletes (R-STO-6), so re-rolled takes leave unreferenced blobs. A new keryx media gc scans the referenced sha256 set (optionally across git history) and deletes unreferenced blobs — opt-in, since storage is cheap. - Doctor check changes — no longer "versioning enabled" (irrelevant); instead "bucket reachable + writable" via the portable API. - Loss of human-browsable keys — blobs/<hash> isn't reels/slug/vo/01.mp3. It's a machine store; the manifest is the human-readable index. Minor.

Option 3 — Content-addressed store + a keryx-owned version ledger (the synthesis, chosen)¶

Option 2's weakness (Matt): a sha256 is the content's identity but not a human version — people want "roll back to V3" or "reselect the earlier variant", not sha256:123abc…. The fix is to notice the object store never gives human versions anyway — even S3's native version_ids are opaque strings with timestamps, not monotonic labels; any "V3" UX would be keryx numbering them itself. So the human version layer belongs in keryx's git-tracked manifest, independent of the backend:

• Storage: content-addressed blobs (blobs/<sha256>) via gocloud.dev/blob — S3/GCS/Azure/file/mem, no per-cloud adapters, no store-side versioning. (As Option 2.)
• Versions: keryx maintains a monotonic, per-file version ledger in git. Each time a promoted file's content changes (a new sha256), keryx appends V<n> → {sha256, size, at, note?}. media.lock records the current version ({v: 3, sha256, size}); a companion history (inline history:[…] or a per-workspace media.log) holds the rest. "Roll back to V3" = read the ledger, point media.lock at V3's sha256, media pull. All diffable, all in git.

Why this is strictly better than leaning on bucket versioning:

• Human-meaningful versions (V1/V2/V3, with an optional note and timestamp) — which no object store gives natively.
• Backend-portable AND backend-independent: the version history lives in git, so it survives switching clouds, and works identically on a store with no versioning (GCS/Azure/file/any gocloud driver).
• Aligns with keryx's existing model: "reselect an old variant" is already first-class via takes → promote; the ledger is just the promoted-state timeline made durable and numbered.
• Keeps every Option 2 win — one maintained library (gocloud), dedup, integrity by construction — and drops the versioning-API dependency no library satisfies.

Chosen: Option 3. Answers both halves of the ask (GCS/Azure + every S3-compatible via one maintained library; no per-cloud code) and preserves the human "V3" rollback UX Matt wants — by putting versions where they belong, in the git manifest, not the store.

5. Design sketch (Option 3)¶

• pkg/objectstore interface simplifies: Put(ctx, key, r, size), Get(ctx, key), Head(ctx, key), Delete(ctx, key) (for GC), Verify(ctx) — no versionID anywhere. Keys are content-addressed; the caller (internal/mediastore) owns the path↔hash mapping and the version ledger.
• The version ledger (git-tracked). media.lock gains a per-file version number; a companion history holds prior versions:

// media.lock — current promoted state (drives `media pull`)
{ "vo/01.mp3": { "v": 3, "sha256": "…", "size": 48123 } }

// media.log — append-only per-file version history (drives rollback UX)
{ "vo/01.mp3": [ {"v":1,"sha256":"…","at":"…"},
                 {"v":2,"sha256":"…","at":"…"},
                 {"v":3,"sha256":"…","at":"…","note":"tighter pace"} ] }

keryx media versions <file> lists them; keryx media rollback <file> v3 repoints media.lock at V3's sha256. Both files are git-committed, so the timeline is diffable and travels with the repo — independent of the store. - One adapter over gocloud.dev/blob (pkg/objectstore/cdk): opens a blob.Bucket from config; the same adapter serves S3, GCS, Azure, file, mem by driver. Registered as storage.provider: s3 | gcs | azure | file (or a single provider taking a gocloud URL — see Q2). - Config shape stays top-level storage: (0039 D5); per-provider blocks:

storage:
  provider: gcs            # s3 | gcs | azure | file
  prefix: blog
  gcs:   { bucket: pbs-prod-reel-media }
  azure: { account: pbsmedia, container: reel-media }
  s3:    { bucket: pbs-prod-reel-media, region: eu-west-2, endpoint: "" }

6. Questions¶

Resolved: - Q1 — the fork → Option 3 (content-addressed + gocloud + git-tracked version ledger). The direction everything hangs on; settled with Matt 2026-07-25. - Q5 — scope → neither GCS nor Azure yet. Spec stays on the shelf; S3 (+ all S3-compatible stores via the endpoint) covers today. Implement when a real GCS/Azure need lands.

Open (revisit at implementation time): - Q2 — provider selector: discrete storage.provider: s3|gcs|azure|file (keryx maps to a gocloud URL) vs exposing gocloud's URL form (storage.url: gs://bucket?...). Lean discrete — hides gocloud, keeps the s3 block shape. - Q3 — migration: the live blog is on 0039's reels/<slug>/… + version_id. Re-key to CAS via an explicit one-time keryx media migrate per workspace, or run both layouts side by side behind the provider selector during transition. - Q4 — GC cadence: ship keryx media gc with the CAS core, or defer (storage is cheap; orphans are harmless until they aren't). - Q6 — ledger shape: inline history in media.lock vs a companion media.log; and whether version bumps are recorded automatically on every promote or only on explicit commit.

7. Phasing (when implemented)¶

1. P1 — CAS + gocloud core + version ledger: simplify the Store interface (drop store-side versioning), add the gocloud adapter, content-address the keys and add the git-tracked version ledger + media versions / media rollback, keep S3 green through gocloud's s3 driver (minio tests).
2. P2 — GCS + Azure: the two new drivers + credential chains (ADC / Managed Identity, CI federated) + emulator-gated integration tests (fake-gcs-server, Azurite) + doctor + docs.
3. P3 — migration + GC: keryx media migrate (0039→0040 re-layout) and keryx media gc (unreferenced-blob reclaim).