Skip to content

Set up the media object store (S3)

How to configure keryx's object store (spec 0039) for a project, authenticate locally and in GitLab CI (OIDC), and sync media with keryx media push/pull.

keryx keeps a workspace's authored state (storyboard, meta, social copy, ledger, media.lock) in the owning project's git repo, and its media (VO, illustrations, covers, music, rendered mp4s) in a versioned S3 bucket. The committed media.lock pins each file's content hash and object version, so any git commit can restore its exact media (media pull), and a fresh CI clone can hydrate itself before rendering or posting.

1. Create the bucket (once)

keryx never creates or configures buckets — that's your infrastructure. The one hard requirement is versioning enabled (the manifest pins version ids):

aws s3api create-bucket --bucket pbs-media --region eu-west-2 \
  --create-bucket-configuration LocationConstraint=eu-west-2
aws s3api put-bucket-versioning --bucket pbs-media \
  --versioning-configuration Status=Enabled

Optional but recommended: a lifecycle policy expiring noncurrent versions after a retention window — keryx never deletes anything, so old take versions accumulate by design and lifecycle rules are the pressure valve.

One bucket serves many projects: each project gets its own prefix namespace.

2. Configure the project

Either edit the owning project's committed .keryx.yaml directly, or use the studio's Settings → Storage tab (which writes these same keys and gives you a Verify button). Project-level is canonical — a fresh CI clone must be self-sufficient; none of this is secret:

storage:
  provider: s3
  prefix: blog          # this project's namespace in the shared bucket
  s3:
    bucket: pbs-prod-reel-media
    region: eu-west-2
    profile: ""         # optional AWS named profile (SSO/shared config); blank = default chain
    endpoint: ""        # only for S3-compatibles (minio, R2) — switches to path-style

profile names an AWS shared-config/SSO profile — it is not a credential (credentials still resolve through the AWS chain). Set it for local convenience; leave it blank in CI so the OIDC web-identity chain is used.

Verify from the CLI or the studio's Storage tab:

keryx doctor            # → [OK] Object store: bucket reachable, versioning enabled

Credentials are never part of keryx config — the AWS SDK's standard credential chain resolves them, exactly as for any AWS tool.

3. Authenticate locally

Anything the AWS default chain supports works untouched. The two common shapes:

  • SSO / named profile (recommended on a workstation) — either export it, or declare it once in config as storage.s3.profile (a profile name, not a credential; the studio's Settings → Storage tab has a field for it):
aws sso login --profile prod-sso
# then either:
AWS_PROFILE=prod-sso keryx media push -w my-post
# or set `storage.s3.profile: prod-sso` in .keryx.yaml and just:
keryx media push -w my-post

An explicit storage.s3.profile overrides the ambient AWS_PROFILE; leave it unset to fall back to the default chain (below). In CI, leave it unset so the OIDC web-identity chain is used.

  • Plain env vars (also how you point at a local minio):
export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… AWS_REGION=eu-west-2
keryx media push -w my-post

4. Authenticate in GitLab CI — OIDC (no long-lived keys)

Use GitLab's ID tokens with AWS AssumeRoleWithWebIdentity, so the pipeline holds no stored AWS secret at all.

AWS side (once): create an OIDC identity provider for https://gitlab.com and a role whose trust policy accepts your project:

{
  "Effect": "Allow",
  "Principal": {"Federated": "arn:aws:iam::<account>:oidc-provider/gitlab.com"},
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {"StringLike": {"gitlab.com:sub": "project_path:phpboyscout/blog:ref_type:branch:ref:main"}}
}

Grant the role s3:PutObject, s3:GetObject, s3:GetObjectVersion, s3:ListBucket, s3:GetBucketVersioning on the bucket/prefix.

Pipeline side:

post-reels:
  id_tokens:
    AWS_ID_TOKEN:
      aud: https://gitlab.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::<account>:role/keryx-media
    AWS_WEB_IDENTITY_TOKEN_FILE: $CI_BUILDS_DIR/aws-web-identity
    AWS_REGION: eu-west-2
  script:
    - echo "$AWS_ID_TOKEN" > "$AWS_WEB_IDENTITY_TOKEN_FILE"
    - keryx media pull -w my-post        # hydrate the fresh clone
    - keryx post all -w my-post

The SDK picks up AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE natively — no keryx flags involved. (Masked AWS_ACCESS_KEY_ID/SECRET CI variables also work if you'd rather defer the OIDC setup.)

5. Day-to-day sync

keryx media push -w my-post     # upload changed media, re-pin media.lock
keryx media pull -w my-post     # hydrate missing media at the pinned versions
keryx media pull -w my-post --force   # ALSO restore locally-modified files
  • Push is incremental by hash — unchanged files cost nothing. The studio's Commit & push runs the same sync automatically, so the refreshed manifest lands in the same commit as the board.
  • Pull verifies by hash and fetches the manifest's pinned versions — check out an old commit and pull to get that commit's exact media.
  • Pull never silently discards local edits — modified files are kept and reported unless --force.
  • Hydrate is automatic where it matters: reel build and post pull any pinned-but-missing media before they run (so the CI pipeline above doesn't strictly need the explicit media pull — it's shown for clarity), and the studio hydrates a workspace on open ("fetching media…" in the bottom bar). Auto-hydrate never overwrites local edits and no-ops when storage is unconfigured.

6. Roll back to an earlier version

Each push that changes a file's content records a new version in the workspace's git-tracked ledger (media.log), so you can go back by number, not by hash:

keryx media versions vo/01.mp3 -w my-post     # v1, v2, v3 … (* = current)
keryx media rollback vo/01.mp3 v2 -w my-post  # re-pin + restore the working copy

Rollback is cheap and non-destructive: storage is content-addressed, so the older blob is still there (a manifest edit + a fetch, never a re-upload), and the history is append-only — v3 remains, and pushing new content appends v4. Commit afterwards to record the rollback.

See also