Greentic · Internal Build Status

Next-Generation Deployment Platform

Rebuilding how Greentic deploys software: an Environment you push bundles to, immutable Revisions, and traffic splitting that's separate from deploying — the pattern every modern cloud platform uses. Here's where we are, in plain words.

Updated 2026-05-25 Plan next-gen-deployment.md Phase 0 complete Phase A complete (13 / 13) Phase B 15 of 16 landed CLI gtc-dev (develop lane)

01At a glance

// what this section is: the 10-second summary — how far along each of the five phases is

The work is split into five phases. Phase 0 (security) and Phase A (foundations) are both complete, and Phase B is nearly there — 15 of its 16 gates have landed as tested building blocks (C2, artifact signing & trust, just landed). What's left is the single observability gate, the wiring that connects those blocks to a live request, the credential flows, and the real cloud deployers.

Phase 0 — Security hotfix stop leaking dev secrets
4 / 4 · done
Dev secrets no longer leak into built bundles; archive extraction is hardened against path-traversal & symlink escapes.
Phase A — Foundations the new object model + CLI
13 / 13 · complete
A1–A10 plus cross-cutting C3 (tool preflight) & C4 (distroless / MUSL / non-root) all landed. The foundation is shipped.
Phase B — Runtime, multi-bundle, traffic splitter make it actually route
15 / 16 · in progress
B0–B12a plus C2 all landed: runtime-config loader, the RevisionDispatcher, per-revision pack activation, the HTTP and static route-scope seams, the runtime-config producer + operator HTTP endpoints, hand-typeable traffic set, Redis-backed pins, drain/evict, the warm/ready health gate, per-customer billing deployments, telemetry stamping, the secrets-backend cutover, and now DSSE artifact signing + a per-environment trust root (§04). The building blocks are built & tested — most still wait on the producer that connects them to a live request. Only C5 (observability fan-out) remains. See §03.
Phase C — Credentials & runtime config two-mode credential flow
0 / 7 · not started
Validate minimum-privilege creds, or bootstrap them from admin and export a rules-pack.
Phase D — Real cloud deployers AWS proving ground, K8s for Zain
0 / 12 · not started
Each cloud ships as a plug-in "env-pack". "Zain ready" = the Kubernetes deployer shipped.
landed in progress planned

02What we're building, in plain words

// what this section is: the problem we're solving and the three ideas the whole rebuild rests on

The problem. Today Greentic can only run one bundle per customer, and deploying a new version overwrites the old one in place. There's no safe way to roll out v2 to 1% of traffic while v1 keeps serving the other 99%. That's the gap we're closing.

The fix. Borrow the pattern from Cloud Run / Kubernetes and split three ideas that used to be tangled together:

Environment — the place you deploy to (local today; later prod-eu, zain-prod). It owns plug-in "env-packs" for secrets, telemetry, sessions, state, and the deployer itself.
Revision — one immutable, frozen version of a deployed bundle. You can have several at once.
Traffic split — a separate dial that says "send 1% here, 99% there". Deploying ≠ shifting traffic.

Why it's built this way. New clouds (AWS, Kubernetes, GCP, Azure, Snap) drop in as plug-ins without rewriting the core — each is described by a string like greentic.deployer.k8s@1.0.0, not a hard-coded list. Add a cloud = publish a pack, not change core code.

03How the system works — today, and where it's going

// what this section is: in plain words — what's built, what's left, what works right now, and what it becomes

The deployment system has two halves. One is the control plane — the part that records what you want: which environment exists, which bundles are deployed, which immutable revisions exist, and how traffic should be split. The other is the data plane — the running server that reads that recorded state, loads each revision's code, and routes live HTTP requests by weight. Today the control plane is real and usable. The data plane's parts are all built and unit-tested, but not yet connected to a live request.

① Control planelive today

The "paperwork" layer. Every gtc-dev op … command writes real, durable, audited state to ~/.greentic/environments/local/ — atomic writes, a per-env lock, an append-only audit log. You can fully describe a multi-revision rollout today.

Create an environment, register a bundle (BundleDeployment), stage an immutable Revision, warm it to ready, record a TrafficSplit, roll it back — all of it works and is verified live on the installed binary.

② Data planebuilt · not yet wired

The "router" layer. Every brain is now written and tested in isolation: the RevisionDispatcher (weighted pick + HMAC cookie + session pin), the Redis/in-memory pin store, the drain/evict coordinator, per-revision pack activation, the HTTP + static route-scope seams, the warm/ready health gate (now with a real DSSE signature check, §04), and per-invocation billing/telemetry stamping.

But nothing connects them to a live request yet. A running greentic-start still boots from a single bundle directory the old way; if it finds a materialized runtime-config it refuses to run and says the wiring lands in later gates. The dispatcher is constructed as None on every live path.

So, concretely — how does it work right now? You run the old, proven single-bundle path to actually serve a flow: gtc-dev start --bundle <bundle> boots one bundle and handles requests, exactly as Greentic always has. Separately, the whole new gtc-dev op surface lets you build up and inspect the new environment/revision/traffic model on disk. The two don't meet yet — the new model is recorded but not yet what serves traffic.

How will it work once Phase B finishes? One environment hosts many deployments at once. You push a bundle (op bundles add), stage v1 and v2 as immutable revisions, warm them, then dial traffic set … v1=99 v2=1. A producer materializes that into a runtime-config.json; greentic-start boots straight from it, activates each revision's packs side-by-side, and the dispatcher sends ~1% of live requests to v2 and ~99% to v1 — with sticky sessions, instant rollback, and per-customer billing tags on every invocation. Deploying a version and shifting traffic to it become two separate, safe acts.

implemented & usable now

What you can do today

  • Bootstrap a local environment in one idempotent command, with 5 default env-packs.
  • Register bundles & stage immutable revisions — full create → stage → warm → ready chain, each step audited.
  • Record traffic splits & roll them back per deployment, with idempotency keys and generation checks.
  • Inspect everythinglist / show / doctor the env, packs, bundles, revisions, traffic.
  • Read the audit trail — who changed what, when, with what authorization decision.
  • Seed & manage a signing trust rootop trust-root bootstrap / list / add / remove (C2, verified live, §04 & §06).
  • Run a real flow the classic waygtc-dev start <bundle> (single bundle, no splitting).
the connective wiring + 2 gates left

What's left before it routes live

  • Wire the dispatcher into ingress — construct a live RevisionDispatcher and resolve a request to its deployment (today both are None stubs). Phase D producer.
  • Boot from runtime-config — activate per-revision packs instead of bailing out (the boot seam exists, it just refuses to run). Phase D producer.
  • Invoke the drain coordinator — a file-watcher (or operator signal) calls the already-built drain/evict coordinator on a lifecycle flip. Phase D producer.
  • C5 — observability fan-out — the last Phase B gate. Every rollout event/log/span carries the full (env, tenant, team, customer, deployment, bundle, revision, pack, generation) tuple under curated cardinality, plus rollout lifecycle events.

C2 (artifact signing & trust root) used to be in this list — it landed 2026-05-25 and is now live-testable. See §04.

Why ship it "dormant" on purpose? Each Phase B gate lands its piece as a tested building block ahead of the producer that will turn it on — the plan calls this "scaffold-ahead-of-producer". It keeps every PR small and reviewable (a router that can be unit-tested deterministically, separate from the HTTP plumbing), and means the day the wiring lands, the hard parts are already proven. The trade-off: the headline "1% to v2" demo isn't runnable end-to-end yet.

04Trust & signing — proving an artifact is what it claims

// what this section is: the newest pillar (C2) — how Greentic now signs deploy artifacts and refuses untrusted ones

A deployment platform pushes code and policy onto customers' machines. Before any of that runs, you have to answer one question: "is this artifact really from us, and has it been tampered with?" C2 — landed 2026-05-25 — is the machinery that answers it. It's the first gate here you can run live on the rebuilt local binary today.

The idea, in plain words. Every important artifact (a revenue-share policy today; .gtbundle and revision manifests next) gets a detached signature made with a private key — using the industry-standard DSSE envelope + Ed25519 + an in-toto provenance statement. To verify, a reader looks up the signer's public key in that environment's trust root and checks the signature against the artifact's content hash. No matching trusted key ⇒ verification fails.

Closed by default. A fresh environment trusts nobody: the trust root starts empty, and an empty trust root means every verification fails closed. You explicitly grant signing rights by seeding a key (op trust-root bootstrap adds the local operator key) or adding one (op trust-root add). The revenue-policy writer never auto-seeds — bootstrap is the one authorized path that opens an env to signing.

Where the line is. Production rejects unsigned or untrusted non-local artifacts; local/dev warns but proceeds, so day-to-day local work isn't blocked. Plain PKCS#8 keys with explicit trust roots and key IDs are the v1 story. Hardware-backed keys (KMS), a transparency log (Rekor), and full supply-chain provenance are deliberately out of scope here — they belong to the separate Trust plan.

Signed & verifiable nowlive

Per-environment trust roottrust-root.json (schema greentic.trust-root.v1), managed by op trust-root {bootstrap,list,add,remove}. Verified live on the rebuilt binary (§06).

Operator key — auto-generated Ed25519 PKCS#8 at ~/.greentic/operator/key.pem (0600, symlink-guarded load, zeroized in memory).

Revenue-share policy (B10) — upgraded from a plain SHA-256 integrity envelope to a real DSSE signature, self-verified right after writing.

Distributor verifier — the old no-op was replaced by real DSSE+Ed25519 verification of the bundle descriptor and every entry in PackList.lock.

Revision health gate (B9) — its SignatureStatus check now does a real bounded-read verify_artifact_dsse against the env trust root, with an empty-digest guard.

Deferred — by designTrust plan / next

.gtbundle / revision signing at build time — the signer exists in greentic-bundle (DSSE + SLSA-provenance predicate); wiring it into the standard build/stage flow is the next step.

KMS-backed keys — v1 uses on-disk PKCS#8; hardware/cloud KMS is Trust-plan work.

Transparency log (Rekor)tlog_entry_id is reserved in the envelope but not yet populated.

Full SLSA provenance & revocation policy — the heavyweight supply-chain story lives in the Trust plan, not here.

Noted non-blocking hardening (follow-ups): cap the trust-root.json read, validate key_id/PEM at read time, multi-subject envelope binding.

Why this matters for the rollout model. Signing is what lets "deploy ≠ shift traffic" stay safe at scale: a revision can't be marked ready (and therefore can't take traffic) unless its signature verifies against the environment's trust root. C2 is the security spine that the traffic splitter leans on once the data plane goes live.

05What's been done — milestone by milestone

// what this section is: every milestone shipped so far — what code actually changed, and why it mattered

Each milestone below shows Changed (what was actually built) and Why (the reason we did it). Phase 0 closed security holes first; Phase A then laid the foundation everything else stands on.

Phase 0 — Security hotfix complete

P0.1Stop secrets leaking into bundlesbundle#112 · setup#109

ChangedBundle builders (both ZIP and SquashFS) now skip dev secret files and plaintext setup answers instead of packing them in.

WhyBundles get uploaded and shared. A secret baked into an artifact is a credential leak — close it before any large refactor begins.

P0.2An automated guard against future leaks

ChangedAdded a doctor secret-leak scanner plus a CI gate that greps the raw archive bytes and fails the build if a secret shape appears.

WhyOne fix isn't enough — a later change could quietly reintroduce the leak. The gate makes that regression impossible to merge.

P0.3Don't break what's already running

ChangedNon-secret config is still written where the current runtime reads it, but that path is now marked deprecated with a one-time warning.

WhyWe're relocating config. Deprecate gradually so existing components keep working until the new config channel exists (Phase C).

P0.4Refuse malicious archives

ChangedBefore extracting a bundle, reject absolute paths, .., symlinks/hardlinks that escape the folder, and duplicate paths.

WhyExtracting an untrusted bundle could otherwise overwrite files anywhere on disk — the classic "zip-slip" attack.

Phase A — Foundations complete · 13 of 13

A1Define the vocabulary oncedeployer#196

ChangedNew crate greentic-deploy-spec holds every new type — Environment, Revision, TrafficSplit, BundleDeployment, Credentials — with validators (e.g. traffic weights must sum to 100%).

WhyThree repos were each inventing their own idea of "environment". One shared definition stops them drifting apart and becomes the single source of truth.

A2Save it safely#197 · #198 · #199

ChangedAn EnvironmentStore with a local-filesystem implementation: atomic writes (temp file + rename), a per-environment lock, and a backup before every change.

WhyThose types are useless if a crash mid-save corrupts them or two operators overwrite each other. This makes the state durable and concurrency-safe.

A3One command line for everything#200 · op#71 · gtc#220

ChangedBuilt the full gtc-dev op surface — 8 nouns (env, env-packs, bundles, revisions, traffic, config, credentials, secrets) × their verbs — and dropped the old hard-coded --provider aws admin subcommands.

WhyOperators need one consistent way to drive the new model, not cloud-specific one-offs scattered across the CLI.

A4Works out of the box#204 · #205 · #206

ChangedFirst gtc-dev setup <bundle> auto-creates the local environment with 5 sensible default env-packs (local-process, dev-store, stdout, in-memory sessions + state) before the bundle wizard runs.

WhyNobody should hand-write JSON to get started. Run one command against a real bundle, get a working local environment.

A4+A first-class op env init verb#216

ChangedNew gtc-dev op env init verb exposes the A4 helper directly — creates the local env if missing, fills in any missing default slots if it already exists, and is idempotent (returns "created" / "healed" / "untouched"). Audit-recorded per A7.

WhyA4's helper was only reachable as a side-effect of setup_or_update (which needs a valid bundle) or run_start (which needs to actually start). From an empty directory there was no clean path. init closes that gap — one verb, no bundle, no hack.

A4bRename "dev" → "local" without breaking anyone#207 · setup#113 · config#43

ChangedA migrate-dev command scans for legacy dev usage and migrates it; if migration isn't safe, it keeps a temporary dev alias that warns and can be hard-disabled with an env var.

WhyOld installs say dev everywhere. We want one canonical name (local) but can't yank it out from under live setups.

A5A revision can't skip steps#208 · #210 · dc#146/147

ChangedA lifecycle state machine (staged → warming → ready → draining → inactive → archived) with a validated transition matrix; you cannot archive a revision that still serves traffic.

WhyRollout state must stay coherent. Routing traffic to a half-warmed revision, or archiving a live one, would silently break deployments.

A6Move old state once, loudly#211

ChangedA one-shot migrate-state that moves the legacy state/deploy/<provider> tree into the new layout and fails loudly if anything is left behind.

WhyReading half-old, half-new state silently would cause subtle, hard-to-trace bugs. Migrate once; refuse to proceed on leftovers.

A7Every change is recorded#213

ChangedAn append-only audit log for every mutating command, plus a local authorization gate; changes to non-local environments fail closed.

WhyYou must be able to answer "who changed this, and when". And until real access control exists, refusing risky operations is safer than allowing them unguarded.

A8Agree the contract before building the remote store#214 · op#74

ChangedWire types + docs for a future remote store: compare-and-swap with ETags, idempotent retries, an integrity (corruption-detection) hash, RBAC decision and backup/restore shapes. No remote server yet — just the contract.

WhyProduction won't use local files. Pinning the contract now means the local path and the eventual cloud path behave identically — no surprises later.

A9Turn a string into running code#215

ChangedAn env-pack registry that maps a descriptor like greentic.deployer.local-process@0.1.0 to a real handler, with built-ins for the 5 local defaults and a plug-in registration hook (EnvPackHandler trait, EnvPackRegistry::register). op env doctor already consults the registry.

WhyThis is the seam every cloud plugs into later. Without it, the descriptor strings are just labels with no code behind them.

A10Wizards know which environment they're forqa#46 · bundle#117 · setup#114 · op#75 · dw#65

Changed5-repo PR train threading env_id through every wizard runtime: qa-lib got the foundation (WizardRunConfig.env_id, SecretsSink trait + NoopSecretsSink, qa-cli --env), then greentic-bundle / greentic-setup / greentic-operator / greentic-dw each threaded it through their wizards. 4 Codex follow-ups closed in-train; /simplify pass shaved (A10) tags out of --help text.

WhyEach environment has its own secrets backend. Until wizards know which env they're answering for, secret routing can't work. Secret persistence reroute itself is deferred to C7 — A10 ships the seam.

C3Fail early, with helpdeployer#217 · #218

ChangedNew tool_check.rs module + gtc op env tool-check verb. Generic primitives (binary-present, version probe) + named catalog for the 9 production tools (terraform, tofu, kubectl, helm, docker, podman, aws, gcloud, az) + typed auth probes (aws-caller-identity, gcloud-auth-list, az-account-show, kubectl-can-i). Each env-pack handler exposes a preflight() seam; built-in local handlers report no external tools.

WhyA clear up-front "install terraform with apt|brew|scoop" saves hours versus a confusing failure halfway through provisioning a cloud. Verify versions, credentials, and reachability — not just binary presence.

C4Tiny, locked-down imagesstart#166 · deployer#219 · op#78 · bundle#119

ChangedAll four host images cut to gcr.io/distroless/static-debian12:nonroot with MUSL-static binaries, USER 65532:65532, strip = true on release. Verified: each <30 MB, ldd reports "not a dynamic executable", image's User field is 65532. Dropped the unsquashfs shell-out + squashfs-tools install in favor of the backhand Rust crate. Codex follow-up added a shebang-helper preflight + ELF-only contract for path extractors.

WhyDistroless = no shell to exploit, no package manager, no root. Standard production hardening for what ships to customers.

Phase B — Runtime & traffic splitter 15 of 16

Phase B is where the foundation starts doing work — multiple deployments in one env, a router that splits traffic by weight, per-revision pack activation. All fourteen build-block gates (B0–B12a) plus C2 (signing & trust) have now landed: the runtime-config loader, the dispatcher, per-revision pack indexing, the HTTP + static route-scope seams, the runtime-config producer + operator HTTP endpoints, hand-typeable traffic set, Redis-backed pins, drain/evict, the warm/ready health gate, per-customer billing deployments, telemetry stamping, the secrets-backend cutover, and DSSE artifact signing. They land as tested building blocks ahead of the producer that connects them to a live request — see §03. Only C5 (observability fan-out) remains.

B0Read the new runtime config from diskstart#167

ChangedNew runtime_config.rs in greentic-start: locates ~/.greentic/environments/<env>/runtime-config.json, deserializes the deploy-spec RuntimeConfig, validates schema + env-match + non-empty revisions + per-block weight_bps ≤ 10000, resolves each block's pack refs through the existing path_safety::normalize_under_root helper, and enforces TrafficSplit invariants per deployment_id (one bundle per deployment, no duplicate revision ids, weights sum to 10,000 bps).

WhyUntil now greentic-start only knew how to boot from a single bundle directory. This is the first read-side support for the new world where one env hosts many deployments × revisions. Codex review caught a symlink-escape via raw is_file() — fixed by canonicalize-then-contain.

B1Decide which revision a request lands onstart#168

ChangedNew revision_dispatcher.rs: per-deployment_id traffic split held in ArcSwap, selection order = trusted header → HMAC cookie → session pin → weighted random over basis points. HMAC-SHA256 cookies bind {env_id, tenant, deployment_id, revision_id, generation, expires_at}; bounded in-memory pin map (MAX_PINS = 16384) with sweep-expired + evict-soonest. apply_traffic_split serialized on a write lock so concurrent operators can't tear state.

WhyThis is the brain of "1% to v2, 99% to v1" — a pure state machine isolated from HTTP plumbing so it can be tested deterministically. 4 Codex findings closed in-PR: write-race, silent bundle rebind, pins outliving generation bumps, unbounded pin map.

B2Pack activation per revisionrunner#345

ChangedActivePacks re-keyed from HashMap<String, …> (tenant only) to HashMap<RuntimeKey, …> where RuntimeKey { tenant, deployment_id, bundle_id, revision_id }. Typed IDs from greentic-deploy-spec. Two read methods (load_pack legacy / load_revision full), three mutators (insert_pack, replace_legacy, replace) all serialized on a write lock; reads stay lock-free via ArcSwap. Codex finding closed: wholesale reload would have evicted revision entries — fixed by partitioning legacy vs revision-keyed state.

WhyOne tenant can host many concurrent revisions of many deployments. The old single-key index can't represent that. The full-key write path (load_revision) ships now and gets its first live caller when boot-from-runtime-config lands.

B3Teach HTTP ingress about revisionsstart#170

ChangedRoute descriptors gained an optional scope: {deployment_id, bundle_id, revision_id} (None = legacy single-bundle, same discipline as B2). A new full-scope matcher, a RevisionDispatcher hook on the ingress state, a fail-closed dispatch_bound_deployment wrapper, and Set-Cookie on the response path for sticky sessions.

WhyThe router has to be able to address one specific (deployment, bundle, revision) before it can split traffic. This is the seam — it's dormant (dispatcher = None, resolve_deployment → None), so the live path is unchanged and overhead is ~zero. Codex hardened two real bugs: a revision-only lookup could cross-route, and a bound deployment must 404/500 rather than silently fall back to the legacy table.

B4Write the runtime-config + expose HTTP endpointsdeployer#221 · op#80

ChangedB4a — a producer that projects an env's validated traffic splits into greentic.runtime-config.v1 (one block per split entry, the only shape that keeps B0's "weights sum to 10,000 bps" invariant), auto-re-materialized inside every traffic mutation and revision transition; writes the file, or deletes it when nothing is live. B4bPOST /deployments/{stage,warm,activate,rollback,complete-drain} on the operator as thin JSON adapters over the deployer verbs.

WhyB4a is the bridge that could feed the dormant boot seam; B4b puts the same verbs over HTTP for remote operators. Codex closed a critical finding: the routes had no caller trust boundary, so a local browser tab could CSRF deployment mutations — fixed with a loopback-peer + same-origin gate that runs before the request body is even read, plus a 512 KiB body cap.

B5Make traffic set hand-typeabledeployer#222

ChangedFirst-class CLI args for gtc op traffic {set,show,rollback}: positional env_id, --deployment <ULID>, variadic <rev>=<weight> entries (accepts N / N% / Nbps), --idempotency-key. The --answers / --schema JSON paths still work — the new args are layered on top.

WhyWriting a JSON file just to dial traffic is clumsy. Now the Phase B acceptance form types directly: traffic set local --deployment D r1=99 r2=1 --idempotency-key K. Codex caught that auto-generating the idempotency key per call broke rollback safety on a retried command — so the key is required on the direct-args path.

B6Sticky sessions that survive across many router replicasstart#171

ChangedExtracted B1's in-process pin map into a RevisionPinStore trait with two backends: in-memory (process-global cap) and Redis (one key per pin, atomic try_pin via a cached Lua script, per-tenant cardinality cap, 50 ms timeout that soft-falls to no-pin). Dispatch became async to allow the Redis round-trip.

WhyIn a horizontally-scaled K8s router, a user's "stick to the revision I first hit" pin has to be shared across replicas — an in-memory map per pod would flap. Redis is that shared store; in-memory stays the default for local. Four Codex findings closed, all about Redis correctness (atomicity, connection multiplexing, bounds, key injectivity).

B7Retire a revision without dropping requestsrunner#346 · start#172

ChangedA two-state drain model on the dispatcher: soft-draining (new picks skip it, but existing pins/cookies still route there so in-flight requests finish) then evicted (removed from routing entirely, so holders re-dispatch to a healthy revision). A RevisionDrainCoordinator sequences mark-draining → wait → evict → close-WS → tear-down, plus ActivePacks::remove_revision on the runner for the surgical teardown.

WhyRolling a revision out of service can't just yank it — that would 500 active sessions. Drain lets them finish, then evicts cleanly. Codex caught a subtle bug: eviction must bump the deployment generation, or generation-scoped pins keep resolving to the dead revision and clients flap across survivors for the full pin TTL.

B8Per-revision routing for non-HTTP ingress tooop#81

ChangedMirrored B3's HTTP route-scope seam onto the static (non-HTTP) route table in greentic-operator. New RevisionScope {deployment_id, bundle_id, revision_id} + optional scope on the static route descriptor (None = legacy). match_request matches legacy routes only; a new match_request_for_revision matches the full triple — both delegate to one predicate-driven match_first so the segment logic can't drift between them.

WhyMessaging connectors and other non-HTTP ingress need the same per-revision addressing the HTTP table got in B3, with identical legacy-fallback discipline. Adding a scope dimension is not the same as rewriting the validators — the producer for it is still Phase D, so this is the match-time shape only.

B9A revision can't take traffic until it's really readydeployer#223 · start#175

ChangedA lifecycle health-gate seam: apply_revision_transition_with_health_gate runs chain-advance → prune-from-splits guard → gate (only when an edge actually advanced) → save. On gate rejection it flips the revision to Failed and persists that, fail-closed. Four check kinds — RouteTable, RuntimeConfig, SignatureStatus, ProviderHealth. greentic-start adds a RevisionHealthGate consumer (runtime-config + signature checks live; route-table + provider-health are Phase-D stubs).

WhyRouting to a half-warmed revision silently breaks deployments. The gate makes "ready" actually mean ready, and a failed warm persists as Failed instead of masquerading as healthy. A CommitMarker fixed a subtle bug where a persist-then-Err path reported "not committed".

B10Per-customer deployments with signed revenue policydeployer#224

ChangedHardened the gtc op bundles surface to the billing contract. customer_id is now the billing principal — required for non-local envs (checked before authz so the error isn't masked), defaulting to local-dev on local only. Every revenue-share mutation writes a new signed, versioned policy doc under billing-policies/<bundle>/<customer>/vN.json{,.sig}, chained backward via previous_version_ref; the version is derived from committed state so partial writes self-heal.

WhyTwo customers can run the same bundle in one env and need different revenue-share terms, rollout history, and rollback state — that's why rollout state belongs to deployment_id, not (env, bundle). A signed, versioned per-customer policy is what downstream invoicing reads. This is also the host table resolve_deployment will bind against.

B11Tag every invocation for billing — without melting metricstelemetry#67 · types#129 · runner#349

ChangedA 3-PR tier-ordered train. TelemetryCtx gained env / customer_id / deployment_id / bundle_id / revision_id (+ with_* builders, marked #[non_exhaustive]); kv() exports all five as gt.* keys; a new metric_attrs() returns a bounded {tenant, env?, bundle_id?} subset. Rollout IDs ride TenantCtx.attributes; the runner stamps them via RolloutIds / stamp_rollout_ids.

WhyEvery invocation must be attributable to a customer / deployment / revision for billing and revenue-share — but metric labels must stay low-cardinality or they melt the time-series DB. So: full IDs on spans & logs, a curated subset on metrics. Codex caught the load-bearing bug — the IDs were never reaching exported telemetry — and fixed the tracing-opentelemetry export wiring.

B12Setup state stops carrying plaintext secretsbundle#121

ChangedPersistedSetupState.secret_values (a BTreeMap<String, Value> of plaintext) → secret_refs of validated secret://<env>/<bundle>/<provider>/<question> newtypes. Secret-marked answers become refs; plaintext is dropped from both the values map and the normalized answers, so the on-disk state holds zero secret material. Schema version bumped 1→2.

WhySetup-state files were carrying plaintext secrets into artifacts. A ref names where a secret lives without embedding it. An xhigh review hardened URI-segment validation (reject empty / slash-bearing segments that silently shift boundaries) and duplicate-question rejection. Note: this secret:// scheme is a distinct address space from the dev store's secrets://.

B12aRuntime readers move off plaintext files entirelystart#179 · deployer#225 · setup#119

ChangedA 3-PR per-crate train migrating every runtime plaintext-secret reader onto SecretsManager, then flipping the producer to stop writing plaintext. Secret-marked answers are dropped from setup-answers.json and written as secrets:// URI refs in config.envelope.cbor; full plaintext flows only to the dev secrets store. Acceptance met: zero plaintext secret values in either file; runtime resolves exclusively via the backend.

WhyB12 closed the producer side; the runtime was still reading secrets from plaintext. Two xhigh rounds fixed two criticals: a block_in_place that panics on the production current-thread deploy runtime (fixed with a dedicated-thread hop — every gtc deploy aws|gcp|azure would have crashed), and an ingress path that base64'd the URI ref instead of fetching the real credential.

C2Sign deploy artifacts; refuse untrusted onesdist-client#152 · bundle#122 · deployer#226 · start#180

ChangedA 4-PR tier-ordered train delivering DSSE (envelope + in-toto Statement) + Ed25519 signing/verification end-to-end, with a per-environment closed-by-default trust root (trust-root.json, schema greentic.trust-root.v1; missing/empty ⇒ verify fails closed). The shared signing module + the real distributor verifier (replacing the dist.rs no-op — now checks the bundle descriptor and every PackList.lock entry); a .gtbundle signer with a SLSA-provenance predicate; per-env trust-root persistence + an auto-generated operator key (~/.greentic/operator/key.pem, 0600, symlink-guarded, zeroized); the gtc op trust-root {bootstrap,list,add,remove} CLI; B10's revenue policy upgraded SHA-256-integrity → real DSSE; and B9's SignatureStatus health check swapped from a placeholder to a real bounded-read verify_artifact_dsse.

WhyA deploy platform pushes code onto customer machines — it must prove artifacts are authentic and untampered before running them. Production rejects unsigned/untrusted non-local artifacts; dev warns. Closes deferrals from B9 (real signature check) and B10 (real policy signing). Codex + xhigh reviews folded in many findings — including an empty-digest guard (an empty expected digest would otherwise bind a signature to no artifact) and trust-loaded-before-reading-the-artifact ordering. KMS / Rekor / full provenance stay in the separate Trust plan. See §04.

Verified 2026-05-25 against live repos and the locally-rebuilt binaries: greentic-deploy-spec tests pass (14/14); B0–B12a plus C2 all merged on develop (C2 = dist-client#152, bundle#122, deployer#226, start#180), plus C3 (#217/#218), C4, A9, A10, and the gtc-dev op routing fix (greentic#225). C2's trust-root flow was run livebootstrap seeded the operator key into trust-root.json and list returned it (§06 ⑥). What "installed" means now: the toolchain version label still reads dev-2026-05-20 (that's the pinned release in ~/.greentic/releases), but the individual -dev binaries have been rebuilt locally: greentic-deployer-dev (2026-05-25, has C2trust-root + B5 direct-args + B10) and greentic-operator-dev (2026-05-22, has through B8/B10). One routing wrinkle: gtc-dev op <noun> dispatches through the operator binary, which predates C2 — so C2's trust-root verb is reachable today via the deployer-direct path (greentic-deployer-dev op trust-root …), not yet via the gtc-dev wrapper. The data plane is still dormant (runtime-config.json is not materialized; boot still needs a <BUNDLE_REF>).

06What you can test right now

// what this section is: commands you can actually run today to see the foundation working

The whole gtc-dev op surface runs against the local environment today — real on-disk state under ~/.greentic/environments/local/, real file locking, a real audit log. With the locally-rebuilt binaries, C2's trust-root flow is now runnable live (⑥). Cloud deploys aren't wired yet (Phase D), so credentials and secrets put/get return an honest "not yet implemented".

✓ Which binary runs what. The gtc-dev wrapper dispatches each op <noun> to a sub-binary. Most nouns (env, env-packs, bundles, revisions, traffic, config) go through greentic-operator-dev (locally rebuilt 2026-05-22). C2's trust-root noun lives in greentic-deployer-dev (rebuilt 2026-05-25) and the wrapper's operator predates it — so reach trust-root via the deployer-direct path: greentic-deployer-dev op trust-root … (⑥). Everything below is verified live on these rebuilt binaries.
⚠ Two ways to drive mutating verbs. Read-only verbs (list / show / doctor) take a positional <env_id>. Every mutating verb accepts either a JSON/YAML payload via --answers <file> (--schema prints the shape) — reproducible, scriptable, auditable, so it stays the canonical form — or, since B5 (now in the rebuilt binary), a hand-typeable form: op traffic set local --deployment <ULID> r1=99 r2=1 --idempotency-key K (both gtc-dev op traffic set and the deployer-direct path expose [ENV_ID] [ENTRIES]… today). The examples below use --answers for clarity and copy-paste safety.

① Bootstrap the local environment

A4 + A4 follow-up (#216) — verified on installed dev binary
# one verb, no bundle, idempotent. Outcomes: created / healed / untouched.
$ gtc-dev op env init
{ "noun": "env", "op": "init",
  "result": { "outcome": "untouched", "pack_count": 5, ... } }

# still want the bundle path? `gtc-dev setup <bundle>` runs the same helper after
# bundle validation passes, then runs the bundle's setup wizard.
$ gtc-dev setup ./my-bundle.gtbundle

② Read-only commands — verified working

inspection
$ gtc-dev op env list
{ "environments": [ { "environment_id":"local", "pack_count":5, ... } ] }
$ gtc-dev op env show local
$ gtc-dev op env doctor local
{ "validate":{"status":"ok"}, "missing_slots":["revocation"], ... }
$ gtc-dev op env-packs list local
deployer   greentic.deployer.local-process@0.1.0
secrets    greentic.secrets.dev-store@0.1.0
telemetry  greentic.telemetry.stdout@0.1.0
sessions   greentic.sessions.in-memory@0.1.0
state      greentic.state.in-memory@0.1.0
$ gtc-dev op bundles list local
$ gtc-dev op revisions list local

③ The full rollout chain — where deployment_id & revision_id come from

The data flow: bundles add creates a BundleDeployment and returns its deployment_id (ULID). revisions stage takes that deployment_id and creates a Revision, returning the revision_id (ULID). revisions warm flips it to ready. traffic set consumes both IDs. You can also recover either ID at any time with bundles list local / revisions list local. Verified live on the installed binary.
step 1 — bundles add → deployment_id
$ cat > /tmp/bundle-add.json <<'EOF'
{ "environment_id": "local",
  "bundle_id":      "fast2flow",
  "route_binding":  { "host": "localhost", "path_prefix": "/" } }
EOF
$ gtc-dev op bundles add --answers /tmp/bundle-add.json
{ "noun":"bundles", "op":"add",
  "result": { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM",
              "bundle_id":     "fast2flow",
              "customer_id":   "local-dev",
              "lifecycle":     "active" } }

# forgot to copy it? list any time — same shape:
$ gtc-dev op bundles list local
{ "result": { "deployments": [ { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM", ... } ] } }
step 2 — revisions stage → revision_id (use the deployment_id from step 1)
$ cat > /tmp/stage.json <<'EOF'
{ "environment_id": "local",
  "deployment_id":  "01KS4MYQ376C5MGJBSAK8TFNAM",
  "bundle_digest":  "sha256:abc123demo" }
EOF
$ gtc-dev op revisions stage --answers /tmp/stage.json
{ "noun":"revisions", "op":"stage",
  "result": { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM",
              "revision_id":   "01KS4N68NZ4AG30J041NA6SC9H",
              "bundle_id":     "fast2flow",
              "lifecycle":     "staged",
              "sequence":      1 } }

# forgot to copy that one too? list shows every revision per env:
$ gtc-dev op revisions list local
{ "result": { "revisions": [ { "revision_id": "01KS4N68NZ4AG30J041NA6SC9H",
                               "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM",
                               "lifecycle": "staged", "sequence": 1, ... } ] } }
step 3 — revisions warm → lifecycle "ready" (uses revision_id from step 2)
$ cat > /tmp/warm.json <<'EOF'
{ "environment_id": "local",
  "revision_id":    "01KS4N68NZ4AG30J041NA6SC9H" }
EOF
$ gtc-dev op revisions warm --answers /tmp/warm.json
{ "result": { "revision_id": "01KS4N68NZ4AG30J041NA6SC9H",
              "lifecycle":   "ready", "sequence": 1 } }

④ Set the 100% traffic split — using both IDs from ③

traffic set — weights are basis-points (0–10000) or percent (0–100); entries sum to 100%
$ cat > /tmp/traffic.json <<'EOF'
{ "environment_id":  "local",
  "deployment_id":   "01KS4MYQ376C5MGJBSAK8TFNAM",
  "idempotency_key": "first-rollout",
  "entries": [
    { "revision_id":    "01KS4N68NZ4AG30J041NA6SC9H",
      "weight_percent": 100 }
  ] }
EOF
$ gtc-dev op traffic set --answers /tmp/traffic.json
{ "noun":"traffic", "op":"set",
  "result": { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM",
              "entries": [ { "revision_id":   "01KS4N68NZ4AG30J041NA6SC9H",
                             "weight_bps":    10000 } ],
              "generation":   0,
              "has_previous": false } }

⑤ Audit log & legacy migrations

audit (A7) & migrations (A4b + A6)
# every mutating command above appended a line here (A7)
$ cat ~/.greentic/environments/local/audit/events.jsonl
{"verb":"init","authorization":{"decision":"allow",...},"result":{"outcome":"ok"}}

# dev → local migration scanner (A4b)
$ gtc-dev op env migrate-dev local --check
{ "clean":true, "findings":[...], "from_env":"dev", "to_env":"local" }

# legacy on-disk state migration (A6)
$ gtc-dev op env migrate-state local --check

⑥ Trust root & signing — C2 verified live 2026-05-25

New & runnable today. C2's trust-root verbs run on the rebuilt greentic-deployer-dev. A fresh env trusts nobody (keys: []); bootstrap seeds the local operator key and the verifier consults it from then on. Output below is real, captured live.
op trust-root — deployer-direct path (the gtc-dev wrapper's operator predates C2)
# closed by default — a brand-new env's trust root is empty:
$ greentic-deployer-dev op trust-root list local
{ "noun":"trust-root", "op":"list", "result":{ "environment_id":"local", "keys":[] } }

# bootstrap seeds the auto-generated operator key (~/.greentic/operator/key.pem, 0600):
$ greentic-deployer-dev op trust-root bootstrap local
{ "noun":"trust-root", "op":"bootstrap",
  "result":{ "environment_id":"local",
             "operator_key_id":"a509047232f6b17346f1c7fd832f9087",
             "operator_public_key_pem":"-----BEGIN PUBLIC KEY-----\n…\n-----END PUBLIC KEY-----\n",
             "trusted_key_count":1 } }

# now the env trusts that key — persisted to trust-root.json (schema greentic.trust-root.v1):
$ greentic-deployer-dev op trust-root list local
{ "result":{ "environment_id":"local",
             "keys":[ { "key_id":"a509047232f6b17346f1c7fd832f9087",
                        "public_key_pem":"-----BEGIN PUBLIC KEY-----\n…" } ] } }

# add / remove a third-party signer's key by (key_id, PEM):
$ greentic-deployer-dev op trust-root add local --public-key-file ./signer.pub.pem
$ greentic-deployer-dev op trust-root remove local <key_id>

⑦ Tool preflight — C3 (from source today)

tool-check (#217 / #218) — via deployer-direct or cargo run
# the rebuilt deployer-dev (2026-05-25) has C3 — runs directly, output captured live:
$ greentic-deployer-dev op env tool-check local
{ "op":"tool-check", "result":{ "environment_id":"local",
   "failed_checks":0, "total_checks":0, "unresolved_bindings":[],
   "bindings":[ {"slot":"deployer","kind":"greentic.deployer.local-process@0.1.0","checks":[]},
                {"slot":"secrets", "kind":"greentic.secrets.dev-store@0.1.0",   "checks":[]},
                {"slot":"telemetry","kind":"greentic.telemetry.stdout@0.1.0",    "checks":[]},
                ... ] } }
# Built-in local handlers report no external tools — clean baseline.
# Cloud env-packs (aws-ecs, k8s, ...) ship in Phase D and will populate per-binding checks.

⑧ Phase B's data plane is built but dormant — here's what that means

Reality check (verified in source on develop, 2026-05-25): even with B0–B12a + C2 all merged, the dispatcher is constructed as None on every live ingress path; resolve_deployment returns None; greentic-start's runtime-config boot seam bail!s instead of serving (the installed binary still requires a <BUNDLE_REF>); ActivePacks::load_revision has no production caller; the drain coordinator is #![allow(dead_code)]. So you still cannot send an HTTP request and watch it split across two revisions — that connective wiring is the Phase D producer. To actually serve a flow today, use the classic single-bundle path.
B0–B12a — every building block passes its own tests; not yet wired together
# 1) Serve a real flow the proven way (no traffic splitting — single bundle):
$ gtc-dev start ./my-bundle.gtbundle

# 2) The new runtime-config boot path (B0) only fires when --bundle AND --config are
#    both absent AND ~/.greentic/environments/<env>/runtime-config.json exists. When it
#    does, the new code refuses to run, by design — pack activation + per-revision
#    routing are built (B2/B3) but the producer that wires them in is Phase D.
#    (Even the rebuilt start binary still requires a positional <BUNDLE_REF> — the
#     bundle-less runtime-config boot is the Phase-D producer's job.)

# 3) The brains are exercised by their own unit tests on develop. Run them to see the
#    dispatcher (B1), pins (B6), drain (B7), health gate (B9) & per-revision packs (B2):
$ cd /home/vampik/greenticai/greentic-start && cargo test revision_dispatcher revision_pin revision_drain
$ cd /home/vampik/greenticai/greentic-runner && cargo test -p greentic-runner-host runtime

⑨ Run the foundation's own test suite

greentic-deploy-spec (A1 + A5 + A8)
$ cd /home/vampik/greenticai/greentic-deployer && cargo test -p greentic-deploy-spec
test result: ok. 14 passed; 0 failed
control plane · live

The full op surface

env / env-packs / bundles / revisions / traffic — create, stage, warm, split, roll back, audit. All real on-disk state, verified live via --answers.

trust & signing · live

C2 trust root

op trust-root bootstrap / list / add / remove on the rebuilt greentic-deployer-dev. Closed-by-default; seeds the operator key; persists trust-root.json. Verified live. See §04.

data plane · dormant

B0–B12a building blocks

Dispatcher, pins, drain, per-revision packs, route seams, health gate, billing + telemetry stamping — all built & tested. Not yet connected to a live request — dispatcher is None, boot bails. See §03.

routing wrinkle

trust-root via deployer-direct

greentic-deployer-dev (05-25) has C2; the gtc-dev op wrapper still routes through the 05-22 operator, so use greentic-deployer-dev op trust-root … until the operator is rebuilt against C2.

stubbed on purpose

credentials · secrets put/get/rotate · env destroy

Return a clear "not yet implemented" — they depend on Phase C/D.

07What's next — the near future

// what this section is: what's left, in the order we'll build it

Phase A is complete and B0–B12a plus C2 (signing & trust) have landed — 15 of 16 Phase B gates. Two things remain to make traffic splitting actually route live requests: the connective wiring that turns the dormant data plane on (the Phase D producer), and the last Phase B gate, C5.

The wiring that turns the data plane live Phase D producer

B0–B12a ship every part; this is the producer + glue that connects them to a real HTTP request. After this, the headline "1% to v2" demo is runnable end-to-end.

Remaining Phase B gate — 1 of 16 left

C2 (signing & trust) just landed — see §04. The next step there is wiring the .gtbundle/revision signer into the standard build/stage flow (the verifier and trust root are already live).

Headline Phase B acceptance test

Stage two revisions of one deployment, traffic set --deployment D r1=99 r2=1, send 1000 HTTP requests, observe 990±20 on r1 and 10±20 on r2 (chi-squared at p=0.95). Two other deployments unaffected. That's "real" traffic splitting.

Phase C

Credentials & runtime config

Two-mode credential flow (validate vs. bootstrap-then-export-rules-pack); a non-secret runtime config channel; env-packs contribute their own wizards.

Phase D

Real cloud deployers

AWS ECS as the first proving ground; Kubernetes is Zain's production target ("Zain ready" = K8s shipped). Then GCP, Azure, Snap/Juju — all as drop-in env-packs.