0039 — Media object store (S3-backed workspace media, git-tracked authored state)¶

Status: IN PROGRESS — P1 implemented 2026-07-23 (pkg/objectstore seam + S3 adapter with live minio integration tests · media.lock manifest + incremental push AND pinned-version pull in internal/mediastore · keryx media push/media pull --force (pull moved up from P2 — CI needs the hydrate with the sync, not after it) · studio Commit & push sync · doctor check · how-to covering bucket setup, local auth, and GitLab CI OIDC. P2 implemented 2026-07-23: the AUTO-hydrate hooks — reel build (in RenderInto, so the studio's render inherits it), post (before dispatch), and the studio on workspace OPEN (D3: whole workspace, POST …/media/pull, a "fetching media…" line in the bottom bar, preview cache busted when files arrive). All three share mediastore.Hydrate: unconfigured storage no-ops, local edits are never overwritten (no force), and a CONFIGURED store that fails is a hard error on build/post (failing fast beats a confusing missing-media error) but soft on studio open (opening must work offline). Remaining: P3 = blog cutover (bulk push → .gitignore flip §3.6 → media-aware dirty dot). All open questions resolved with Matt 2026-07-23. From Matt's 2026-07-23 direction, following the 0029 §9 note-59 discovery that the blog gitignores reels/* (so Commit & push has been inert). Intent: land fast on the heels of note 59. Resolutions: D1 mp4 pushed to S3 · D2 chat.json stays in git · D3 whole-workspace hydrate on open · D4 media-aware dirty dot as a P3 follow-up · D5 top-level storage: config key (NOT providers.) · *D6** doctor check, no auto-create. Date: 2026-07-23 Relates: 0001 §3.2 (stateless tool — the owning project holds the state), 0012 (studio git seam), 0029 §9 notes 49/59 (explicit Commit & push, the uncommitted-reel indicator), 0031 (spend ledger), R-GLOBAL-9 (content cache).

1. Problem¶

A reel workspace is two very different kinds of state:

• Authored state — storyboard.json, workspace meta, social copy, the posting ledger. Small, text, diffable. This is what review, history, and CI need from git — and today the blog can't have it because reels/* is blanket-ignored.
• Media — VO takes and promoted clips (mp3), card illustrations (png), cover art, music beds, portraits, the rendered reel-<slug>.mp4. Large, binary, regenerated freely (takes are re-rolled; every generation adds files). Tens of MB per reel today, growing with every reel and every re-roll.

Putting the media in git bloats the repo permanently (git never forgets a blob); git-lfs is operational friction Matt explicitly doesn't want; promisor/partial- clone tooling is not there yet. But leaving reels/* ignored means no history for the authored state either, and the studio's Commit & push (and the rail's uncommitted indicator, note 59) do nothing.

2. Direction (Matt, 2026-07-23)¶

• Object store for media, git for authored state. S3 (Matt's default cloud), one bucket with versioning enabled, serving multiple projects via key prefixes.
• Local stays a plain working copy — unversioned files on disk exactly as today; S3 carries the version history, not the filesystem and not git.
• Push on Commit & push — the explicit git action also uploads the workspace's media to its bucket locations.
• Un-ignore reels/ in the blog but not before this lands — flipping the ignore first would push all current media into git, the exact thing this spec avoids. Rendered *.mp4 stays out of git regardless.

3. Design¶

3.1 What lives where¶

3.2 The manifest is the bridge (media.lock)¶

A per-workspace media.lock (JSON, committed to git) records, for every media file at commit time:

{
  "vo/01.mp3": {"sha256": "…", "size": 48123, "version_id": "3sL4…", "etag": "…"},
  "cover.png": {"sha256": "…", "size": 812342, "version_id": "9xQ2…", "etag": "…"}
}

This is what makes "S3 handles versioning" actually usable:

• Point-in-time restore: checking out any git commit + keryx media pull fetches the exact object versions that commit was authored against — not merely "latest". Without it, bucket versioning is write-only history.
• Change detection: push compares local sha256 against the manifest and uploads only what changed — cheap incremental syncs.
• Git-visible media changes: a re-rolled take changes media.lock, so the diff (and the note-59 uncommitted indicator, via the committed manifest) shows that media moved even though git never holds the bytes.
• Integrity: pull verifies sha256 after download.

3.3 Key layout — multiple projects, one bucket¶

s3://<bucket>/<project-prefix>/reels/<slug>/<workspace-relative-path>
e.g. s3://pbs-media/blog/reels/sandboxed-isnt-safe/vo/01.mp3

• <project-prefix> comes from the owning project's config — each consumer project sets its own (blog, keryx-docs, …). No cross-project coupling; the same reusable-tool posture as everything else (0001 §3.2).
• Keys are stable paths; overwrites create S3 versions (the point of bucket versioning). keryx never deletes objects; retiring old versions is a bucket lifecycle policy, not keryx's job.

3.4 The storage seam (pluggable, like every other backend)¶

Per the provider pattern (0001 §3.4): a narrow Go interface, implementation chosen from config at construction, requests provider-neutral.

// pkg/objectstore
type Store interface {
    Put(ctx, key string, r io.Reader, size int64, sha256 string) (PutResult, error) // → VersionID, ETag
    Get(ctx, key, versionID string) (io.ReadCloser, error)   // versionID "" = latest
    Head(ctx, key, versionID string) (Info, error)
}

• First adapter: S3 (pkg/objectstore/s3). SDK: aws-sdk-go-v2, with a configurable endpoint + path-style flag so any S3-compatible store (minio, R2) works — which also gives us env-gated integration tests against a local minio instead of needing AWS in CI.
• Registered through the same config-driven factory machinery as the generation backends, but keyed at a top-level storage: section (D5 resolved — storage is not a generation backend; it gets its own namespace):

storage:
  provider: s3
  prefix: blog          # this project's namespace in the shared bucket
                        # (adapter-neutral: it shapes the KEYS, not the connection)
  s3:
    bucket: pbs-prod-reel-media
    region: eu-west-2
    profile: ""         # optional AWS named profile (SSO/shared config); blank = default chain
    endpoint: ""        # optional — set for minio/R2 (S3-compatible, path-style)

3.5 Flows¶

Push (studio Commit & push, and keryx media push -w <slug>): 1. Hash the workspace's media files; diff against media.lock. 2. Upload changed/new files (parallel, bounded); collect VersionID/ETag. 3. Rewrite media.lock; include it in the git commit (studio: same commit as the board save; the existing scoped AddGlob already picks it up). 4. Surface per-file progress + a summary in the commit note ("3 media files pushed · 2.1 MB").

Pull / hydrate (keryx media pull -w <slug> and on-demand): 1. Read media.lock from the checkout. 2. Fetch any file that is missing locally or whose sha256 differs, at the pinned version_id; verify sha256. 3. Auto-hydrate hooks: reel build, post, and studio workspace-open hydrate missing media before erroring — this is what makes CI from a fresh clone work (the unattended pipeline is keryx's whole reason to exist; a clone with authored state but no media must self-serve).

Local-only edits between commits: untouched — files land on disk exactly as today; S3 is only consulted at push/pull/hydrate points.

3.6 .gitignore recipe for the owning project (the blog cutover)¶

Replace the blanket reels/* ignore with media-shape ignores; authored state + media.lock become trackable:

# reel media lives in the object store (keryx spec 0039) — git holds the
# authored state + media.lock only
reels/*/vo/
reels/*/cards/
reels/*/cover/
reels/*/portrait/
reels/*/music*
reels/*/cover.png
reels/*/reel-*.mp4
reels/*/.cache/

Cutover order (phase 3, only after push/pull work): initial media push for every workspace → flip the ignore → first real commit of authored state.

4. Requirements¶

• R-STO-1 Media pushes are incremental (sha256 vs manifest); unchanged files cost nothing.
• R-STO-2 media.lock pins exact object versions; pull at any git commit reproduces that commit's media byte-for-byte (sha256-verified).
• R-STO-3 One bucket serves many projects via config prefix; keys never collide across projects with distinct prefixes.
• R-STO-4 The config file carries only the bucket name, region, and key prefix — never credentials, which resolve via the AWS default chain.
• R-STO-5 Unconfigured storage degrades to local-only with a visible note — never a hard failure of save or commit.
• R-STO-6 keryx never deletes objects or versions; retention is bucket lifecycle policy.
• R-STO-7 Build/post/studio-open hydrate missing media automatically from the manifest (fresh CI clone works unattended).
• R-STO-8 The storage backend is a provider: adding another store (GCS, azure) is additive (implement + register, no call-site changes).
• R-STO-9 Integration tests run against an S3-compatible local store (minio), env-gated per the GTB pattern (INT_TEST=1).
• R-STO-10 keryx doctor verifies the configured bucket is reachable and has versioning enabled, warning when not (D6); keryx never creates or configures buckets.

5. Open questions — ALL RESOLVED (Matt, 2026-07-23)¶

• D1 — rendered mp4 to S3? → Yes, push it. The postable artefact travels with the workspace; posting from CI uses the exact file the ledger records, no re-render dependency. Ignored in git either way.
• D2 — chat.json → git. Small text, part of the authoring record; revisit only if transcripts balloon.
• D3 — hydrate UX in the studio → whole workspace on open, with a progress line. Simplest correct behaviour; previews/takes/render all just work after open. (media pull remains for CLI/CI.)
• D4 — uncommitted indicator + media → yes, as a P3 follow-up. Phase 1 ships git-visible dirtiness (storyboard/meta/manifest); the DirtySlugs hash-compare against media.lock lands with the blog cutover.
• D5 — config key → top-level storage: (not providers.*). Storage is not a generation backend; own namespace, same factory machinery (§3.4).
• D6 — bucket setup → doctor check, no auto-create. Bucket + versioning are owned infra; keryx doctor verifies reachability and that versioning is enabled, warns otherwise; push errors clearly on a missing bucket.
• D7 — config scope (2026-07-23, follow-up) → project-level for now. The committed .keryx.yaml is the canonical home for the whole storage: block — a fresh CI clone must be self-sufficient (the unattended pipeline reads the repo, not a user profile), and the bucket name is not a secret. This does NOT strain statelessness (§3.2 forbids keryx owning project state; user-scoped tool config is the existing GTB global layer, init/config already enabled). Deferred option: add storage to the 0014 inheritance allowlist (pkg/studio/projectconfig.go) so user-level config can supply bucket DEFAULTS across projects — if adopted, prefix stays project-level (it is the project's identity in the bucket; a shared default would collide keyspaces). Credentials remain the AWS default chain in every scope.

6. Phasing¶

1. P1 — seam + push: pkg/objectstore interface + S3 adapter (+ minio integration tests) · media.lock write/diff · keryx media push · studio Commit & push integration (upload → manifest → commit) · config + docs.
2. P2 — pull + hydrate: keryx media pull · auto-hydrate in build/post/ studio-open · CI how-to (role credentials, pipeline example).
3. P3 — blog cutover: initial bulk push · .gitignore flip (§3.6) · first authored-state commit · note-59 indicator live end-to-end · D4 follow-up.

7. Test plan¶

• Unit: manifest diff/write (afero, table-driven) · key layout (prefix/slug/ path) · degrade-when-unconfigured paths · commit-note reporting.
• Integration (minio, env-gated): put/get/head with real VersionIDs · push→ mutate→push increments only changed keys · pull at a pinned manifest restores bytes (sha256) · multi-prefix isolation.
• e2e (godog): media push/media pull command contracts; build-with-missing- media hydrates then renders (fake store).