Greentic · Internal Build Status
Rebuilding how Greentic deploys software: an Environment you push bundles to, immutable Revisions, and traffic splitting that's separate from deploying — the pattern every modern cloud platform uses. Here's where we are, in plain words.
// what this section is: the 10-second summary — how far along each of the five phases is
The work is split into five phases. Phase 0 (security) is done, and the foundation of Phase A is mostly done. The runtime that actually routes traffic, the credential flows, and the real cloud deployers all come later — and depend on this foundation.
ActivePacks) landed. Dormant until B3 wires HTTP ingress through to them.// what this section is: the problem we're solving and the three ideas the whole rebuild rests on
The problem. Today Greentic can only run one bundle per customer, and deploying a new version overwrites the old one in place. There's no safe way to roll out v2 to 1% of traffic while v1 keeps serving the other 99%. That's the gap we're closing.
The fix. Borrow the pattern from Cloud Run / Kubernetes and split three ideas that used to be tangled together:
① Environment — the place you deploy to (local today; later prod-eu, zain-prod).
It owns plug-in "env-packs" for secrets, telemetry, sessions, state, and the deployer itself.
② Revision — one immutable, frozen version of a deployed bundle. You can have several at once.
③ Traffic split — a separate dial that says "send 1% here, 99% there". Deploying ≠ shifting traffic.
Why it's built this way. New clouds (AWS, Kubernetes, GCP, Azure, Snap) drop in as plug-ins
without rewriting the core — each is described by a string like greentic.deployer.k8s@1.0.0,
not a hard-coded list. Add a cloud = publish a pack, not change core code.
// what this section is: every milestone shipped so far — what code actually changed, and why it mattered
Each milestone below shows Changed (what was actually built) and Why (the reason we did it). Phase 0 closed security holes first; Phase A then laid the foundation everything else stands on.
ChangedBundle builders (both ZIP and SquashFS) now skip dev secret files and plaintext setup answers instead of packing them in.
WhyBundles get uploaded and shared. A secret baked into an artifact is a credential leak — close it before any large refactor begins.
ChangedAdded a doctor secret-leak scanner plus a CI gate that greps the raw archive bytes and fails the build if a secret shape appears.
WhyOne fix isn't enough — a later change could quietly reintroduce the leak. The gate makes that regression impossible to merge.
ChangedNon-secret config is still written where the current runtime reads it, but that path is now marked deprecated with a one-time warning.
WhyWe're relocating config. Deprecate gradually so existing components keep working until the new config channel exists (Phase C).
ChangedBefore extracting a bundle, reject absolute paths, .., symlinks/hardlinks that escape the folder, and duplicate paths.
WhyExtracting an untrusted bundle could otherwise overwrite files anywhere on disk — the classic "zip-slip" attack.
ChangedNew crate greentic-deploy-spec holds every new type — Environment, Revision, TrafficSplit, BundleDeployment, Credentials — with validators (e.g. traffic weights must sum to 100%).
WhyThree repos were each inventing their own idea of "environment". One shared definition stops them drifting apart and becomes the single source of truth.
ChangedAn EnvironmentStore with a local-filesystem implementation: atomic writes (temp file + rename), a per-environment lock, and a backup before every change.
WhyThose types are useless if a crash mid-save corrupts them or two operators overwrite each other. This makes the state durable and concurrency-safe.
ChangedBuilt the full gtc-dev op surface — 8 nouns (env, env-packs, bundles, revisions, traffic, config, credentials, secrets) × their verbs — and dropped the old hard-coded --provider aws admin subcommands.
WhyOperators need one consistent way to drive the new model, not cloud-specific one-offs scattered across the CLI.
ChangedFirst gtc-dev setup <bundle> auto-creates the local environment with 5 sensible default env-packs (local-process, dev-store, stdout, in-memory sessions + state) before the bundle wizard runs.
WhyNobody should hand-write JSON to get started. Run one command against a real bundle, get a working local environment.
op env init verb#216ChangedNew gtc-dev op env init verb exposes the A4 helper directly — creates the local env if missing, fills in any missing default slots if it already exists, and is idempotent (returns "created" / "healed" / "untouched"). Audit-recorded per A7.
WhyA4's helper was only reachable as a side-effect of setup_or_update (which needs a valid bundle) or run_start (which needs to actually start). From an empty directory there was no clean path. init closes that gap — one verb, no bundle, no hack.
ChangedA migrate-dev command scans for legacy dev usage and migrates it; if migration isn't safe, it keeps a temporary dev alias that warns and can be hard-disabled with an env var.
WhyOld installs say dev everywhere. We want one canonical name (local) but can't yank it out from under live setups.
ChangedA lifecycle state machine (staged → warming → ready → draining → inactive → archived) with a validated transition matrix; you cannot archive a revision that still serves traffic.
WhyRollout state must stay coherent. Routing traffic to a half-warmed revision, or archiving a live one, would silently break deployments.
ChangedA one-shot migrate-state that moves the legacy state/deploy/<provider> tree into the new layout and fails loudly if anything is left behind.
WhyReading half-old, half-new state silently would cause subtle, hard-to-trace bugs. Migrate once; refuse to proceed on leftovers.
ChangedAn append-only audit log for every mutating command, plus a local authorization gate; changes to non-local environments fail closed.
WhyYou must be able to answer "who changed this, and when". And until real access control exists, refusing risky operations is safer than allowing them unguarded.
ChangedWire types + docs for a future remote store: compare-and-swap with ETags, idempotent retries, an integrity (corruption-detection) hash, RBAC decision and backup/restore shapes. No remote server yet — just the contract.
WhyProduction won't use local files. Pinning the contract now means the local path and the eventual cloud path behave identically — no surprises later.
ChangedAn env-pack registry that maps a descriptor like greentic.deployer.local-process@0.1.0 to a real handler, with built-ins for the 5 local defaults and a plug-in registration hook (EnvPackHandler trait, EnvPackRegistry::register). op env doctor already consults the registry.
WhyThis is the seam every cloud plugs into later. Without it, the descriptor strings are just labels with no code behind them.
Changed5-repo PR train threading env_id through every wizard runtime: qa-lib got the foundation (WizardRunConfig.env_id, SecretsSink trait + NoopSecretsSink, qa-cli --env), then greentic-bundle / greentic-setup / greentic-operator / greentic-dw each threaded it through their wizards. 4 Codex follow-ups closed in-train; /simplify pass shaved (A10) tags out of --help text.
WhyEach environment has its own secrets backend. Until wizards know which env they're answering for, secret routing can't work. Secret persistence reroute itself is deferred to C7 — A10 ships the seam.
ChangedNew tool_check.rs module + gtc op env tool-check verb. Generic primitives (binary-present, version probe) + named catalog for the 9 production tools (terraform, tofu, kubectl, helm, docker, podman, aws, gcloud, az) + typed auth probes (aws-caller-identity, gcloud-auth-list, az-account-show, kubectl-can-i). Each env-pack handler exposes a preflight() seam; built-in local handlers report no external tools.
WhyA clear up-front "install terraform with apt|brew|scoop" saves hours versus a confusing failure halfway through provisioning a cloud. Verify versions, credentials, and reachability — not just binary presence.
ChangedAll four host images cut to gcr.io/distroless/static-debian12:nonroot with MUSL-static binaries, USER 65532:65532, strip = true on release. Verified: each <30 MB, ldd reports "not a dynamic executable", image's User field is 65532. Dropped the unsquashfs shell-out + squashfs-tools install in favor of the backhand Rust crate. Codex follow-up added a shebang-helper preflight + ELF-only contract for path extractors.
WhyDistroless = no shell to exploit, no package manager, no root. Standard production hardening for what ships to customers.
Phase B is where the foundation starts doing work — multiple deployments in one env, a router that splits traffic by weight, per-revision pack activation. The first three gates are scaffolding ahead of the producer that wires them together.
ChangedNew runtime_config.rs in greentic-start: locates ~/.greentic/environments/<env>/runtime-config.json, deserializes the deploy-spec RuntimeConfig, validates schema + env-match + non-empty revisions + per-block weight_bps ≤ 10000, resolves each block's pack refs through the existing path_safety::normalize_under_root helper, and enforces TrafficSplit invariants per deployment_id (one bundle per deployment, no duplicate revision ids, weights sum to 10,000 bps).
WhyUntil now greentic-start only knew how to boot from a single bundle directory. This is the first read-side support for the new world where one env hosts many deployments × revisions. Codex review caught a symlink-escape via raw is_file() — fixed by canonicalize-then-contain.
ChangedNew revision_dispatcher.rs: per-deployment_id traffic split held in ArcSwap, selection order = trusted header → HMAC cookie → session pin → weighted random over basis points. HMAC-SHA256 cookies bind {env_id, tenant, deployment_id, revision_id, generation, expires_at}; bounded in-memory pin map (MAX_PINS = 16384) with sweep-expired + evict-soonest. apply_traffic_split serialized on a write lock so concurrent operators can't tear state.
WhyThis is the brain of "1% to v2, 99% to v1" — a pure state machine isolated from HTTP plumbing so it can be tested deterministically. 4 Codex findings closed in-PR: write-race, silent bundle rebind, pins outliving generation bumps, unbounded pin map.
ChangedActivePacks re-keyed from HashMap<String, …> (tenant only) to HashMap<RuntimeKey, …> where RuntimeKey { tenant, deployment_id, bundle_id, revision_id }. Typed IDs from greentic-deploy-spec. Two read methods (load_pack legacy / load_revision full), three mutators (insert_pack, replace_legacy, replace) all serialized on a write lock; reads stay lock-free via ArcSwap. Codex finding closed: wholesale reload would have evicted revision entries — fixed by partitioning legacy vs revision-keyed state.
WhyOne tenant can host many concurrent revisions of many deployments. The old single-key index can't represent that. B3 will be the consumer that actually puts revision-keyed entries in.
greentic-deploy-spec tests pass; A9 (#215), A4+ (#216), C3 (#217/#218), C4 (#166/#219/#78/#119), A10 train (5 repos), B0 (#167), B1 (#168), B2 (runner#345), and the gtc-dev op routing fix (greentic#225) all merged on develop.
Installed binary lag: the local greentic-deployer-dev still predates C3's tool-check verb (#217). greentic-start-dev and greentic-runner-dev predate B0/B1/B2.
The new gtc-dev binary (with #225's routing fix) is installed and verified — gtc-dev op env init/list/doctor return clean JSON. Next nightly dev publish will catch the rest up.// what this section is: commands you can actually run today to see the foundation working
The whole gtc-dev op surface runs against the local environment today — real
on-disk state under ~/.greentic/environments/local/, real file locking, a real audit log.
Cloud deploys aren't wired yet (Phase D), so credentials and secrets put/get
return an honest "not yet implemented".
gtc-dev op routing bug fixed (greentic#225).
Yesterday's snapshot warned that gtc-dev op stripped its op arg before forwarding —
that's now fixed. All commands below use gtc-dev op directly and have been verified live on
the installed binary. The deployer-direct path (gtc-dev op …) and the operator-direct path
still work too if you want to skip the gtc wrapper.list / show / doctor) take a positional <env_id>. Every mutating verb
(create, add, warm, traffic set, …) takes its entire payload as a JSON
file via --answers <file>. There is no inline <rev>=<weight> syntax. Use
--schema on any verb to print its JSON shape, then write the file. This is intentional — answer
files are reproducible, scriptable, and auditable.local environment# one verb, no bundle, idempotent. Outcomes: created / healed / untouched. $ gtc-dev op env init { "noun": "env", "op": "init", "result": { "outcome": "untouched", "pack_count": 5, ... } } # still want the bundle path? `gtc-dev setup <bundle>` runs the same helper after # bundle validation passes, then runs the bundle's setup wizard. $ gtc-dev setup ./my-bundle.gtbundle
$ gtc-dev op env list { "environments": [ { "environment_id":"local", "pack_count":5, ... } ] } $ gtc-dev op env show local $ gtc-dev op env doctor local { "validate":{"status":"ok"}, "missing_slots":["revocation"], ... } $ gtc-dev op env-packs list local deployer greentic.deployer.local-process@0.1.0 secrets greentic.secrets.dev-store@0.1.0 telemetry greentic.telemetry.stdout@0.1.0 sessions greentic.sessions.in-memory@0.1.0 state greentic.state.in-memory@0.1.0 $ gtc-dev op bundles list local $ gtc-dev op revisions list local
deployment_id & revision_id come frombundles add creates a BundleDeployment and returns its
deployment_id (ULID). revisions stage takes that deployment_id and creates a Revision,
returning the revision_id (ULID). revisions warm flips it to ready. traffic set
consumes both IDs. You can also recover either ID at any time with bundles list local /
revisions list local. Verified live on the installed binary.$ cat > /tmp/bundle-add.json <<'EOF' { "environment_id": "local", "bundle_id": "fast2flow", "route_binding": { "host": "localhost", "path_prefix": "/" } } EOF $ gtc-dev op bundles add --answers /tmp/bundle-add.json { "noun":"bundles", "op":"add", "result": { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM", "bundle_id": "fast2flow", "customer_id": "local-dev", "lifecycle": "active" } } # forgot to copy it? list any time — same shape: $ gtc-dev op bundles list local { "result": { "deployments": [ { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM", ... } ] } }
$ cat > /tmp/stage.json <<'EOF' { "environment_id": "local", "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM", "bundle_digest": "sha256:abc123demo" } EOF $ gtc-dev op revisions stage --answers /tmp/stage.json { "noun":"revisions", "op":"stage", "result": { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM", "revision_id": "01KS4N68NZ4AG30J041NA6SC9H", "bundle_id": "fast2flow", "lifecycle": "staged", "sequence": 1 } } # forgot to copy that one too? list shows every revision per env: $ gtc-dev op revisions list local { "result": { "revisions": [ { "revision_id": "01KS4N68NZ4AG30J041NA6SC9H", "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM", "lifecycle": "staged", "sequence": 1, ... } ] } }
$ cat > /tmp/warm.json <<'EOF' { "environment_id": "local", "revision_id": "01KS4N68NZ4AG30J041NA6SC9H" } EOF $ gtc-dev op revisions warm --answers /tmp/warm.json { "result": { "revision_id": "01KS4N68NZ4AG30J041NA6SC9H", "lifecycle": "ready", "sequence": 1 } }
$ cat > /tmp/traffic.json <<'EOF' { "environment_id": "local", "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM", "idempotency_key": "first-rollout", "entries": [ { "revision_id": "01KS4N68NZ4AG30J041NA6SC9H", "weight_percent": 100 } ] } EOF $ gtc-dev op traffic set --answers /tmp/traffic.json { "noun":"traffic", "op":"set", "result": { "deployment_id": "01KS4MYQ376C5MGJBSAK8TFNAM", "entries": [ { "revision_id": "01KS4N68NZ4AG30J041NA6SC9H", "weight_bps": 10000 } ], "generation": 0, "has_previous": false } }
# every mutating command above appended a line here (A7) $ cat ~/.greentic/environments/local/audit/events.jsonl {"verb":"init","authorization":{"decision":"allow",...},"result":{"outcome":"ok"}} # dev → local migration scanner (A4b) $ gtc-dev op env migrate-dev local --check { "clean":true, "findings":[...], "from_env":"dev", "to_env":"local" } # legacy on-disk state migration (A6) $ gtc-dev op env migrate-state local --check
# The installed greentic-deployer-dev was built before C3 (2026-05-20 11:08 vs. # #217's 14:53), so `op env tool-check local` returns "unrecognized subcommand". # Run from source until the next nightly dev publish: $ cd /home/vampik/greenticai/greentic-deployer cargo run --quiet --bin greentic-deployer -- op env tool-check local { "op":"tool-check", "result":{ "failed_checks":0, "total_checks":0, "per_binding":[ {"slot":"deployer","kind":"...local-process...","outcomes":[]}, {"slot":"secrets","kind":"...dev-store...","outcomes":[]}, ... ] } } # Built-in local handlers report no external tools — clean baseline. # Cloud env-packs (aws-ecs, k8s, ...) ship in Phase D and will populate per-binding outcomes.
# greentic-start's new runtime-config boot path (B0) only fires when both --bundle # AND --config are absent AND ~/.greentic/environments/<env>/runtime-config.json # exists. No producer writes that file yet — operator materialization is later. $ greentic-start-dev start Error: no demo config found; pass --config, --bundle, or create ./demo/demo.yaml # If the file existed, the error from the new code would be: # "runtime-config declares N revision(s) ...; booting from a materialized runtime-config # is not yet runnable: pack activation lands in B2 and per-revision routing in B3." # RevisionDispatcher (B1) and ActivePacks/RuntimeKey (B2) ship as scaffolding. They # get exercised by their own unit tests today; B3 (HTTP route table + ingress) wires # them into a live request. $ cd /home/vampik/greenticai/greentic-start && cargo test revision_dispatcher $ cd /home/vampik/greenticai/greentic-runner && cargo test -p greentic-runner-host runtime
$ cd /home/vampik/greenticai/greentic-deployer && cargo test -p greentic-deploy-spec test result: ok. 14 passed; 0 failed
Yesterday's op-arg-stripping bug is fixed. gtc-dev op <noun> <verb> now forwards cleanly to the operator binary.
Not in the installed deployer-dev yet — predates #217. Run via cargo run --bin greentic-deployer.
Code is landed. No producer writes runtime-config.json yet, and ingress isn't wired through B1 — both arrive with B3/B4.
Return a clear "not yet implemented" — they depend on Phase B/C/D.
// what this section is: what's left, in the order we'll build it
Finish Phase A (4 gates left), then Phase B is the big one — it's where traffic splitting actually routes live requests.
B0, B1, B2 are scaffolding. The next four gates wire them together so a live HTTP request actually lands on a chosen revision.
(deployment_id, revision_id, public_path); HttpRouteTable::match_request rewired; RevisionDispatcher picks the revision. This is the one that turns B0/B1/B2 from dormant code into live traffic.POST /deployments/{stage,warm,activate,rollback,complete-drain}, scoped by (env, deployment_id) and gated by the transition matrix.gtc op traffic {set,show,rollback} per deployment Persists TrafficSplit per deployment_id with idempotency key + expected generation. (Today's traffic set writes the spec types; B5 wires it to the dispatcher.)gt:rev_pin:{env}:{deployment_id}:{tenant}; in-memory fallback stays for local.revisions drain semantics Stop new pins, wait drain_seconds, let HTTP finish, close WebSockets with a retryable code, tear down the runtime.Ready.BundleDeployment lifecycle Per-customer BundleDeployment with signed/versioned revenue policy. customer_id required for non-local.customer_id / deployment_id / bundle_id / revision_id for billing & revenue-share.secret_refs; migrate every runtime reader off plaintext files.stage rejects unsigned bundles.(env, tenant, team, customer, deployment, bundle, revision, pack, generation) tuple under curated cardinality.Stage two revisions of one deployment, traffic set --deployment D r1=99 r2=1, send 1000 HTTP requests, observe 990±20 on r1 and 10±20 on r2 (chi-squared at p=0.95). Two other deployments unaffected. That's "real" traffic splitting.
Two-mode credential flow (validate vs. bootstrap-then-export-rules-pack); a non-secret runtime config channel; env-packs contribute their own wizards.
AWS ECS as the first proving ground; Kubernetes is Zain's production target ("Zain ready" = K8s shipped). Then GCP, Azure, Snap/Juju — all as drop-in env-packs.