Greentic · Internal Build Status

Next-Generation Deployment Platform

Rebuilding how Greentic deploys software: an Environment you push bundles to, immutable Revisions, and traffic splitting that's separate from deploying — the pattern every modern cloud platform uses. Here's where we are, in plain words.

Updated 2026-05-21 Plan next-gen-deployment.md Phase 0 complete Phase A complete (13 / 13) Phase B 3 of 16 landed CLI gtc-dev (develop lane)

01At a glance

// what this section is: the 10-second summary — how far along each of the five phases is

The work is split into five phases. Phase 0 (security) is done, and the foundation of Phase A is mostly done. The runtime that actually routes traffic, the credential flows, and the real cloud deployers all come later — and depend on this foundation.

Phase 0 — Security hotfix stop leaking dev secrets
4 / 4 · done
Dev secrets no longer leak into built bundles; archive extraction is hardened against path-traversal & symlink escapes.
Phase A — Foundations the new object model + CLI
13 / 13 · complete
A1–A10 plus cross-cutting C3 (tool preflight) & C4 (distroless / MUSL / non-root) all landed. The foundation is shipped.
Phase B — Runtime, multi-bundle, traffic splitter make it actually route
3 / 16 · in progress
B0 (runtime-config loader) + B1 (RevisionDispatcher) + B2 (per-revision ActivePacks) landed. Dormant until B3 wires HTTP ingress through to them.
Phase C — Credentials & runtime config two-mode credential flow
0 / 7 · not started
Validate minimum-privilege creds, or bootstrap them from admin and export a rules-pack.
Phase D — Real cloud deployers AWS proving ground, K8s for Zain
0 / 12 · not started
Each cloud ships as a plug-in "env-pack". "Zain ready" = the Kubernetes deployer shipped.
landed in progress planned

02What we're building, in plain words

// what this section is: the problem we're solving and the three ideas the whole rebuild rests on

The problem. Today Greentic can only run one bundle per customer, and deploying a new version overwrites the old one in place. There's no safe way to roll out v2 to 1% of traffic while v1 keeps serving the other 99%. That's the gap we're closing.

The fix. Borrow the pattern from Cloud Run / Kubernetes and split three ideas that used to be tangled together:

Environment — the place you deploy to (local today; later prod-eu, zain-prod). It owns plug-in "env-packs" for secrets, telemetry, sessions, state, and the deployer itself.
Revision — one immutable, frozen version of a deployed bundle. You can have several at once.
Traffic split — a separate dial that says "send 1% here, 99% there". Deploying ≠ shifting traffic.

Why it's built this way. New clouds (AWS, Kubernetes, GCP, Azure, Snap) drop in as plug-ins without rewriting the core — each is described by a string like greentic.deployer.k8s@1.0.0, not a hard-coded list. Add a cloud = publish a pack, not change core code.

03What's been done — milestone by milestone

// what this section is: every milestone shipped so far — what code actually changed, and why it mattered

Each milestone below shows Changed (what was actually built) and Why (the reason we did it). Phase 0 closed security holes first; Phase A then laid the foundation everything else stands on.

Phase 0 — Security hotfix complete

P0.1Stop secrets leaking into bundlesbundle#112 · setup#109

ChangedBundle builders (both ZIP and SquashFS) now skip dev secret files and plaintext setup answers instead of packing them in.

WhyBundles get uploaded and shared. A secret baked into an artifact is a credential leak — close it before any large refactor begins.

P0.2An automated guard against future leaks

ChangedAdded a doctor secret-leak scanner plus a CI gate that greps the raw archive bytes and fails the build if a secret shape appears.

WhyOne fix isn't enough — a later change could quietly reintroduce the leak. The gate makes that regression impossible to merge.

P0.3Don't break what's already running

ChangedNon-secret config is still written where the current runtime reads it, but that path is now marked deprecated with a one-time warning.

WhyWe're relocating config. Deprecate gradually so existing components keep working until the new config channel exists (Phase C).

P0.4Refuse malicious archives

ChangedBefore extracting a bundle, reject absolute paths, .., symlinks/hardlinks that escape the folder, and duplicate paths.

WhyExtracting an untrusted bundle could otherwise overwrite files anywhere on disk — the classic "zip-slip" attack.

Phase A — Foundations complete · 13 of 13

A1Define the vocabulary oncedeployer#196

ChangedNew crate greentic-deploy-spec holds every new type — Environment, Revision, TrafficSplit, BundleDeployment, Credentials — with validators (e.g. traffic weights must sum to 100%).

WhyThree repos were each inventing their own idea of "environment". One shared definition stops them drifting apart and becomes the single source of truth.

A2Save it safely#197 · #198 · #199

ChangedAn EnvironmentStore with a local-filesystem implementation: atomic writes (temp file + rename), a per-environment lock, and a backup before every change.

WhyThose types are useless if a crash mid-save corrupts them or two operators overwrite each other. This makes the state durable and concurrency-safe.

A3One command line for everything#200 · op#71 · gtc#220

ChangedBuilt the full gtc-dev op surface — 8 nouns (env, env-packs, bundles, revisions, traffic, config, credentials, secrets) × their verbs — and dropped the old hard-coded --provider aws admin subcommands.

WhyOperators need one consistent way to drive the new model, not cloud-specific one-offs scattered across the CLI.

A4Works out of the box#204 · #205 · #206

ChangedFirst gtc-dev setup <bundle> auto-creates the local environment with 5 sensible default env-packs (local-process, dev-store, stdout, in-memory sessions + state) before the bundle wizard runs.

WhyNobody should hand-write JSON to get started. Run one command against a real bundle, get a working local environment.

A4+A first-class op env init verb#216

ChangedNew gtc-dev op env init verb exposes the A4 helper directly — creates the local env if missing, fills in any missing default slots if it already exists, and is idempotent (returns "created" / "healed" / "untouched"). Audit-recorded per A7.

WhyA4's helper was only reachable as a side-effect of setup_or_update (which needs a valid bundle) or run_start (which needs to actually start). From an empty directory there was no clean path. init closes that gap — one verb, no bundle, no hack.

A4bRename "dev" → "local" without breaking anyone#207 · setup#113 · config#43

ChangedA migrate-dev command scans for legacy dev usage and migrates it; if migration isn't safe, it keeps a temporary dev alias that warns and can be hard-disabled with an env var.

WhyOld installs say dev everywhere. We want one canonical name (local) but can't yank it out from under live setups.

A5A revision can't skip steps#208 · #210 · dc#146/147

ChangedA lifecycle state machine (staged → warming → ready → draining → inactive → archived) with a validated transition matrix; you cannot archive a revision that still serves traffic.

WhyRollout state must stay coherent. Routing traffic to a half-warmed revision, or archiving a live one, would silently break deployments.

A6Move old state once, loudly#211

ChangedA one-shot migrate-state that moves the legacy state/deploy/<provider> tree into the new layout and fails loudly if anything is left behind.

WhyReading half-old, half-new state silently would cause subtle, hard-to-trace bugs. Migrate once; refuse to proceed on leftovers.

A7Every change is recorded#213

ChangedAn append-only audit log for every mutating command, plus a local authorization gate; changes to non-local environments fail closed.

WhyYou must be able to answer "who changed this, and when". And until real access control exists, refusing risky operations is safer than allowing them unguarded.

A8Agree the contract before building the remote store#214 · op#74

ChangedWire types + docs for a future remote store: compare-and-swap with ETags, idempotent retries, an integrity (corruption-detection) hash, RBAC decision and backup/restore shapes. No remote server yet — just the contract.

WhyProduction won't use local files. Pinning the contract now means the local path and the eventual cloud path behave identically — no surprises later.

A9Turn a string into running code#215

ChangedAn env-pack registry that maps a descriptor like greentic.deployer.local-process@0.1.0 to a real handler, with built-ins for the 5 local defaults and a plug-in registration hook (EnvPackHandler trait, EnvPackRegistry::register). op env doctor already consults the registry.

WhyThis is the seam every cloud plugs into later. Without it, the descriptor strings are just labels with no code behind them.

A10Wizards know which environment they're forqa#46 · bundle#117 · setup#114 · op#75 · dw#65

Changed5-repo PR train threading env_id through every wizard runtime: qa-lib got the foundation (WizardRunConfig.env_id, SecretsSink trait + NoopSecretsSink, qa-cli --env), then greentic-bundle / greentic-setup / greentic-operator / greentic-dw each threaded it through their wizards. 4 Codex follow-ups closed in-train; /simplify pass shaved (A10) tags out of --help text.

WhyEach environment has its own secrets backend. Until wizards know which env they're answering for, secret routing can't work. Secret persistence reroute itself is deferred to C7 — A10 ships the seam.

C3Fail early, with helpdeployer#217 · #218

ChangedNew tool_check.rs module + gtc op env tool-check verb. Generic primitives (binary-present, version probe) + named catalog for the 9 production tools (terraform, tofu, kubectl, helm, docker, podman, aws, gcloud, az) + typed auth probes (aws-caller-identity, gcloud-auth-list, az-account-show, kubectl-can-i). Each env-pack handler exposes a preflight() seam; built-in local handlers report no external tools.

WhyA clear up-front "install terraform with apt|brew|scoop" saves hours versus a confusing failure halfway through provisioning a cloud. Verify versions, credentials, and reachability — not just binary presence.

C4Tiny, locked-down imagesstart#166 · deployer#219 · op#78 · bundle#119

ChangedAll four host images cut to gcr.io/distroless/static-debian12:nonroot with MUSL-static binaries, USER 65532:65532, strip = true on release. Verified: each <30 MB, ldd reports "not a dynamic executable", image's User field is 65532. Dropped the unsquashfs shell-out + squashfs-tools install in favor of the backhand Rust crate. Codex follow-up added a shebang-helper preflight + ELF-only contract for path extractors.

WhyDistroless = no shell to exploit, no package manager, no root. Standard production hardening for what ships to customers.

Phase B — Runtime & traffic splitter 3 of 16

Phase B is where the foundation starts doing work — multiple deployments in one env, a router that splits traffic by weight, per-revision pack activation. The first three gates are scaffolding ahead of the producer that wires them together.

B0Read the new runtime config from diskstart#167

ChangedNew runtime_config.rs in greentic-start: locates ~/.greentic/environments/<env>/runtime-config.json, deserializes the deploy-spec RuntimeConfig, validates schema + env-match + non-empty revisions + per-block weight_bps ≤ 10000, resolves each block's pack refs through the existing path_safety::normalize_under_root helper, and enforces TrafficSplit invariants per deployment_id (one bundle per deployment, no duplicate revision ids, weights sum to 10,000 bps).

WhyUntil now greentic-start only knew how to boot from a single bundle directory. This is the first read-side support for the new world where one env hosts many deployments × revisions. Codex review caught a symlink-escape via raw is_file() — fixed by canonicalize-then-contain.

B1Decide which revision a request lands onstart#168

ChangedNew revision_dispatcher.rs: per-deployment_id traffic split held in ArcSwap, selection order = trusted header → HMAC cookie → session pin → weighted random over basis points. HMAC-SHA256 cookies bind {env_id, tenant, deployment_id, revision_id, generation, expires_at}; bounded in-memory pin map (MAX_PINS = 16384) with sweep-expired + evict-soonest. apply_traffic_split serialized on a write lock so concurrent operators can't tear state.

WhyThis is the brain of "1% to v2, 99% to v1" — a pure state machine isolated from HTTP plumbing so it can be tested deterministically. 4 Codex findings closed in-PR: write-race, silent bundle rebind, pins outliving generation bumps, unbounded pin map.

B2Pack activation per revisionrunner#345

ChangedActivePacks re-keyed from HashMap<String, …> (tenant only) to HashMap<RuntimeKey, …> where RuntimeKey { tenant, deployment_id, bundle_id, revision_id }. Typed IDs from greentic-deploy-spec. Two read methods (load_pack legacy / load_revision full), three mutators (insert_pack, replace_legacy, replace) all serialized on a write lock; reads stay lock-free via ArcSwap. Codex finding closed: wholesale reload would have evicted revision entries — fixed by partitioning legacy vs revision-keyed state.

WhyOne tenant can host many concurrent revisions of many deployments. The old single-key index can't represent that. B3 will be the consumer that actually puts revision-keyed entries in.

Verified 2026-05-21 against live repos and the installed binaries: greentic-deploy-spec tests pass; A9 (#215), A4+ (#216), C3 (#217/#218), C4 (#166/#219/#78/#119), A10 train (5 repos), B0 (#167), B1 (#168), B2 (runner#345), and the gtc-dev op routing fix (greentic#225) all merged on develop. Installed binary lag: the local greentic-deployer-dev still predates C3's tool-check verb (#217). greentic-start-dev and greentic-runner-dev predate B0/B1/B2. The new gtc-dev binary (with #225's routing fix) is installed and verified — gtc-dev op env init/list/doctor return clean JSON. Next nightly dev publish will catch the rest up.

04What you can test right now

// what this section is: commands you can actually run today to see the foundation working

The whole gtc-dev op surface runs against the local environment today — real on-disk state under ~/.greentic/environments/local/, real file locking, a real audit log. Cloud deploys aren't wired yet (Phase D), so credentials and secrets put/get return an honest "not yet implemented".

gtc-dev op routing bug fixed (greentic#225). Yesterday's snapshot warned that gtc-dev op stripped its op arg before forwarding — that's now fixed. All commands below use gtc-dev op directly and have been verified live on the installed binary. The deployer-direct path (gtc-dev op …) and the operator-direct path still work too if you want to skip the gtc wrapper.
⚠ How mutating verbs work. Read-only verbs (list / show / doctor) take a positional <env_id>. Every mutating verb (create, add, warm, traffic set, …) takes its entire payload as a JSON file via --answers <file>. There is no inline <rev>=<weight> syntax. Use --schema on any verb to print its JSON shape, then write the file. This is intentional — answer files are reproducible, scriptable, and auditable.

① Bootstrap the local environment

A4 + A4 follow-up (#216) — verified on installed dev binary
# one verb, no bundle, idempotent. Outcomes: created / healed / untouched.
$ gtc-dev op env init
{ "noun": "env", "op": "init",
  "result": { "outcome": "untouched", "pack_count": 5, ... } }

# still want the bundle path? `gtc-dev setup <bundle>` runs the same helper after
# bundle validation passes, then runs the bundle's setup wizard.
$ gtc-dev setup ./my-bundle.gtbundle

② Read-only commands — verified working

inspection
$ gtc-dev op env list
{ "environments": [ { "environment_id":"local", "pack_count":5, ... } ] }
$ gtc-dev op env show local
$ gtc-dev op env doctor local
{ "validate":{"status":"ok"}, "missing_slots":["revocation"], ... }
$ gtc-dev op env-packs list local
deployer   greentic.deployer.local-process@0.1.0
secrets    greentic.secrets.dev-store@0.1.0
telemetry  greentic.telemetry.stdout@0.1.0
sessions   greentic.sessions.in-memory@0.1.0
state      greentic.state.in-memory@0.1.0
$ gtc-dev op bundles list local
$ gtc-dev op revisions list local

③ The full rollout chain — where deployment_id & revision_id come from

The data flow: bundles add creates a BundleDeployment and returns its deployment_id (ULID). revisions stage takes that deployment_id and creates a Revision, returning the revision_id (ULID). revisions warm flips it to ready. traffic set consumes both IDs. You can also recover either ID at any time with bundles list local / revisions list local. Verified live on the installed binary.
step 1 — bundles add → deployment_id
$ cat > /tmp/bundle-add.json <<'EOF'
{ "environment_id": "local",
  "bundle_id":      "fast2flow",
  "route_binding":  { "host": "localhost", "path_prefix": "/" } }
EOF
$ gtc-dev op bundles add --answers /tmp/bundle-add.json
{ "noun":"bundles", "op":"add",
  "result": { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM",
              "bundle_id":     "fast2flow",
              "customer_id":   "local-dev",
              "lifecycle":     "active" } }

# forgot to copy it? list any time — same shape:
$ gtc-dev op bundles list local
{ "result": { "deployments": [ { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM", ... } ] } }
step 2 — revisions stage → revision_id (use the deployment_id from step 1)
$ cat > /tmp/stage.json <<'EOF'
{ "environment_id": "local",
  "deployment_id":  "01KS4MYQ376C5MGJBSAK8TFNAM",
  "bundle_digest":  "sha256:abc123demo" }
EOF
$ gtc-dev op revisions stage --answers /tmp/stage.json
{ "noun":"revisions", "op":"stage",
  "result": { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM",
              "revision_id":   "01KS4N68NZ4AG30J041NA6SC9H",
              "bundle_id":     "fast2flow",
              "lifecycle":     "staged",
              "sequence":      1 } }

# forgot to copy that one too? list shows every revision per env:
$ gtc-dev op revisions list local
{ "result": { "revisions": [ { "revision_id": "01KS4N68NZ4AG30J041NA6SC9H",
                               "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM",
                               "lifecycle": "staged", "sequence": 1, ... } ] } }
step 3 — revisions warm → lifecycle "ready" (uses revision_id from step 2)
$ cat > /tmp/warm.json <<'EOF'
{ "environment_id": "local",
  "revision_id":    "01KS4N68NZ4AG30J041NA6SC9H" }
EOF
$ gtc-dev op revisions warm --answers /tmp/warm.json
{ "result": { "revision_id": "01KS4N68NZ4AG30J041NA6SC9H",
              "lifecycle":   "ready", "sequence": 1 } }

④ Set the 100% traffic split — using both IDs from ③

traffic set — weights are basis-points (0–10000) or percent (0–100); entries sum to 100%
$ cat > /tmp/traffic.json <<'EOF'
{ "environment_id":  "local",
  "deployment_id":   "01KS4MYQ376C5MGJBSAK8TFNAM",
  "idempotency_key": "first-rollout",
  "entries": [
    { "revision_id":    "01KS4N68NZ4AG30J041NA6SC9H",
      "weight_percent": 100 }
  ] }
EOF
$ gtc-dev op traffic set --answers /tmp/traffic.json
{ "noun":"traffic", "op":"set",
  "result": { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM",
              "entries": [ { "revision_id":   "01KS4N68NZ4AG30J041NA6SC9H",
                             "weight_bps":    10000 } ],
              "generation":   0,
              "has_previous": false } }

⑤ Audit log & legacy migrations

audit (A7) & migrations (A4b + A6)
# every mutating command above appended a line here (A7)
$ cat ~/.greentic/environments/local/audit/events.jsonl
{"verb":"init","authorization":{"decision":"allow",...},"result":{"outcome":"ok"}}

# dev → local migration scanner (A4b)
$ gtc-dev op env migrate-dev local --check
{ "clean":true, "findings":[...], "from_env":"dev", "to_env":"local" }

# legacy on-disk state migration (A6)
$ gtc-dev op env migrate-state local --check

⑥ Tool preflight — C3 (from source today)

tool-check (#217 / #218) — installed binary is stale
# The installed greentic-deployer-dev was built before C3 (2026-05-20 11:08 vs.
# #217's 14:53), so `op env tool-check local` returns "unrecognized subcommand".
# Run from source until the next nightly dev publish:
$ cd /home/vampik/greenticai/greentic-deployer
  cargo run --quiet --bin greentic-deployer -- op env tool-check local
{ "op":"tool-check", "result":{ "failed_checks":0, "total_checks":0,
   "per_binding":[ {"slot":"deployer","kind":"...local-process...","outcomes":[]},
                   {"slot":"secrets","kind":"...dev-store...","outcomes":[]},
                   ... ] } }
# Built-in local handlers report no external tools — clean baseline.
# Cloud env-packs (aws-ecs, k8s, ...) ship in Phase D and will populate per-binding outcomes.

⑦ Phase B is landed but dormant — here's what that means

B0 / B1 / B2 — verified, no end-to-end test yet
# greentic-start's new runtime-config boot path (B0) only fires when both --bundle
# AND --config are absent AND ~/.greentic/environments/<env>/runtime-config.json
# exists. No producer writes that file yet — operator materialization is later.
$ greentic-start-dev start
Error: no demo config found; pass --config, --bundle, or create ./demo/demo.yaml
# If the file existed, the error from the new code would be:
# "runtime-config declares N revision(s) ...; booting from a materialized runtime-config
#  is not yet runnable: pack activation lands in B2 and per-revision routing in B3."

# RevisionDispatcher (B1) and ActivePacks/RuntimeKey (B2) ship as scaffolding. They
# get exercised by their own unit tests today; B3 (HTTP route table + ingress) wires
# them into a live request.
$ cd /home/vampik/greenticai/greentic-start && cargo test revision_dispatcher
$ cd /home/vampik/greenticai/greentic-runner && cargo test -p greentic-runner-host runtime

⑧ Run the foundation's own test suite

greentic-deploy-spec (A1 + A5 + A8)
$ cd /home/vampik/greenticai/greentic-deployer && cargo test -p greentic-deploy-spec
test result: ok. 14 passed; 0 failed
fixed · #225

gtc-dev op routing

Yesterday's op-arg-stripping bug is fixed. gtc-dev op <noun> <verb> now forwards cleanly to the operator binary.

stale binary

op env tool-check (C3)

Not in the installed deployer-dev yet — predates #217. Run via cargo run --bin greentic-deployer.

dormant

B0 / B1 / B2

Code is landed. No producer writes runtime-config.json yet, and ingress isn't wired through B1 — both arrive with B3/B4.

stubbed on purpose

credentials · secrets put/get/rotate · env destroy

Return a clear "not yet implemented" — they depend on Phase B/C/D.

05What's next — the near future

// what this section is: what's left, in the order we'll build it

Finish Phase A (4 gates left), then Phase B is the big one — it's where traffic splitting actually routes live requests.

Phase A is done — Phase B is where the payoff lands

B0, B1, B2 are scaffolding. The next four gates wire them together so a live HTTP request actually lands on a chosen revision.

Then the rest of Phase B

Headline Phase B acceptance test

Stage two revisions of one deployment, traffic set --deployment D r1=99 r2=1, send 1000 HTTP requests, observe 990±20 on r1 and 10±20 on r2 (chi-squared at p=0.95). Two other deployments unaffected. That's "real" traffic splitting.

Phase C

Credentials & runtime config

Two-mode credential flow (validate vs. bootstrap-then-export-rules-pack); a non-secret runtime config channel; env-packs contribute their own wizards.

Phase D

Real cloud deployers

AWS ECS as the first proving ground; Kubernetes is Zain's production target ("Zain ready" = K8s shipped). Then GCP, Azure, Snap/Juju — all as drop-in env-packs.