# Shan Valleru's Blog > Technical blog by Shan Valleru covering Kubernetes, distributed systems, AI, and business strategy. Deep dives into platform engineering, agentic coding, emergence, disruption theory, and infrastructure economics. Author: Shan Valleru Website: https://svalle.ru/ --- ## Sandboxes Are the New Servers - URL: https://svalle.ru/posts/ai/sandboxes-are-the-new-servers/ - Date: 2026-07-22 - Tags: AI, agents, infrastructure, sandboxes, cloud, Kubernetes-adjacent Every era of the cloud got a new tenant, and every new tenant got a new primitive. Applications got the virtual machine. Microservices got the container. Events got the function. Each time, the industry didn't just shrink the old box—it rebuilt isolation, scheduling, and pricing around what the new tenant actually was. AI agents are the new tenant. And they are a workload unlike anything the cloud has hosted: they write their own code seconds before running it, they succeed by failing fast, they want to fork running machines like processes, and increasingly they get *graded* by the substrate they run on. None of the existing primitives fit. That's why the sandbox—isolate, snapshot, fork—is becoming a first-class unit of compute, why [the vertical AI pattern](/posts/ai/anatomy-of-a-vertical-ai-agent/) has a sandbox box in it, and why there are now entire companies selling nothing else. ## Five tenants, five primitives The comparison is the argument: | | VM | Container | Function | Agent sandbox | |---|---|---|---|---| | **Tenant** | monolith app | microservice | event handler | agent attempt | | **Lifetime** | years | days to months | milliseconds | one task | | **Cold start** | minutes | seconds | ~100ms | milliseconds, and falling | | **State** | on the machine | externalized | none | snapshot & fork | | **Trust model** | trusted code, untrusted neighbors | trusted code, shared kernel | trusted code, multi-tenant | untrusted code, hostile by assumption | | **Failure** | page someone | restart the pod | retry | harvest it—failure is data | | **Priced by** | instance-hours | vCPU-seconds | invocations | session-minutes & attempts | Read down the last column and the pattern is clear: this isn't a smaller function. It's a different tenant. Every era of compute got the isolation its tenant deserved—and this tenant thinks. ## The workload Six properties make agents unlike everything the cloud has hosted before. Each one forces a primitive. **Untrusted from the inside.** Every previous isolation model protected the workload from the world and tenants from each other—the code itself was assumed friendly, because a human wrote and reviewed it. An agent inverts the assumption: the code being executed was written seconds ago by a stochastic process, possibly steered by whatever hostile text the agent read on the way in. The workload is the adversary. CI systems ran semi-trusted code at human cadence with review gates; agents run it at machine speed with none. That forces hard isolation as the *default*, not the paranoid option—and it makes network egress a security control, not a networking detail, because exfiltration is the first thing a prompt-injected agent tries. The serious platforms ship egress allowlists and full network blocks as first-class sandbox settings. **Mayflies, not cattle.** Pets became cattle; cattle are becoming mayflies. The sandbox wants to be born, do one thing, and die—and the death is the product, because destruction is what guarantees the next attempt a clean slate. When the unit of compute lives for seconds, the defining performance metric flips from utilization to **cold start**, which is why every vendor's homepage is a millisecond number and why the engineering goes down two roads: kernels stripped until they boot in ~125 milliseconds—Firecracker's entire design brief—and snapshots that skip booting altogether, memory-mapping a ready machine image back to life in single-digit milliseconds. Warmth stored on disk, not warmth kept running. **State that branches—and pauses.** Traditional architecture pushed state out of compute: stateless services, databases of record. Agents want the opposite—*forkable execution*. Try three fixes from one checkpoint. Branch an RL rollout at the decision point. Pause mid-task for six hours while a human approves the next step, then resume from exactly that machine state at storage prices instead of compute prices. That's process-level time travel, and no mainstream workload ever demanded it—which is why snapshot, restore, and fork are becoming API surface rather than ops tooling. Memory snapshots already cut cold starts by multiples on the platforms that ship them. The cloud is finally getting a `fork()` syscall. **Failure is the signal.** SRE culture minimizes failure; agent execution harvests it. An agent runs code to find out whether the code works—execution as hypothesis testing, the sandbox as laboratory rather than factory. In RL training this is explicit: each trajectory is an agent trying whatever maximizes reward, side effects included, and every rollout needs a clean world to wreck. So the substrate's job isn't five nines of uptime—it's a fast, rich verdict channel: exit codes, artifacts, diffs, telemetry, and increasingly a score. No previous cloud graded its tenants. **Particle-physics concurrency.** One agent task fans into a hundred parallel tool executions. One training experiment is ten thousand rollouts for twenty minutes, then silence. Production platforms already report tens of thousands of concurrent sandboxes for a single customer event, and the training-side operators run hundreds of thousands of concurrent environments. Reactive autoscaling is useless against this shape—the burst is over before the autoscaler notices. What works is snapshot fan-out plus fleet headroom: fork one initialized image into ten thousand copy-on-write clones, so the swarm restores into existence instead of booting into it, on hosts whose spare capacity was provisioned before the burst. (Sizing that headroom is a newsvendor problem—[inventory theory](/posts/business/inventory-theory-cloud-capacity/) has opinions.) **The browser is a device class.** For agents, the browser is what the terminal was for humans: the universal interface to everything that never got an API. That makes browser-in-a-sandbox a standard SKU, not an exotic one—a live Chromium session with the same containment as a code sandbox plus three things code sandboxes never needed: session persistence, credential handling, and a human-takeover path. It's why the managed-browser vendors (Browserbase, Steel, Hyperbrowser) look less like scraping tools every quarter and more like a compute primitive. The browser is the new SSH. ## What is a sandbox, really [Containers](/posts/linux/what-is-a-container/) isolated you from your *neighbors*—namespaces and cgroups drawing lines between tenants who all ran code they trusted. Agent sandboxes must isolate you from *the workload itself*, and three lineages compete on how: ```d2 {alt="Three isolation lineages side by side: Firecracker runs a microVM with its own kernel per sandbox, gVisor runs a user-space kernel that intercepts syscalls, and Kata runs containers inside lightweight VMs"} grid-rows: 1 horizontal-gap: 32 fc: "Firecracker\n(microVM · own kernel per sandbox)" gv: "gVisor\n(user-space kernel · intercepts syscalls)" kata: "Kata\n(containers inside lightweight VMs)" ``` Firecracker gives every sandbox its own kernel—a container escape ends at that VM's wall. E2B and Vercel's sandboxes take this path. gVisor interposes a user-space kernel that filters syscalls before they reach the host—lower overhead, no separate kernel—and it's how Modal isolates. Kata wraps containers in lightweight VMs, splitting the difference. The detail worth savoring: this entire lineage was built over the past decade for multi-tenant serverless, an era when the threat was your neighbor. Then a tenant arrived whose threat is itself, and the isolation stack turned out to be accidental preparation. Call it the isolation dividend. ## The trace What actually happens when an agent runs `pip install && pytest`: ```d2 {alt="The life of one sandbox execution: an API request hits the control plane, is placed on a host with spare capacity, restores from a template snapshot, passes an egress policy gate, executes, has its exit code and artifacts and telemetry collected, receives a verdict, and is destroyed or snapshotted for forking"} direction: down req: "run() request\n(code + limits + policy)" place: "placement\n(host with headroom)" snap: "restore from snapshot\n(memory-mapped, ms)" egress: "egress gate\n(allowlist or block-all)" exec: "execute\n(the attempt)" collect: "collect\n(exit code · artifacts · telemetry)" verdict: "verdict\n(pass, fail, reward)" end: "destroy\n(or snapshot for fork)" req -> place -> snap -> egress -> exec -> collect -> verdict -> end ``` Every stage exists because of a property above. The placement step exists because of particle-physics concurrency. The snapshot restore exists because of mayfly lifetimes. The egress gate exists because the workload is the adversary. The verdict exists because failure is the signal. And the final fork option exists because the most valuable thing a finished sandbox can do is become two sandboxes. This is the request path of a cloud whose unit of work is the *attempt*. ## Four demand curves One primitive, four buyers: ```d2 {alt="The sandbox primitive serving four demand curves: inference-time execution needing low cold start and scale to zero, training-time RL rollouts needing fork and massive parallelism and a reward channel, evals needing deterministic replay of traces, and serving, where the AI-generated artifact keeps running inside the sandbox"} direction: right sandbox: "sandbox\n(isolate · snapshot · fork)" inference: "inference\n(cold start · scale to zero)" training: "training\n(fork · parallelism · reward)" evals: "evals\n(deterministic replay)" serving: "serving\n(the artifact keeps running)" sandbox -> inference sandbox -> training sandbox -> evals sandbox -> serving ``` Inference—production agents acting—is the visible curve, and it prices like serverless. Training is the steep one: RL on agentic tasks consumes environments in parallel thousands per experiment, an environments marketplace has formed on top (Prime Intellect's hub carries thousands of open RL environments; Mechanize builds them for labs), and the frontier labs have reportedly discussed spending over a billion dollars a year on environments alone. An RL environment, stripped of ceremony, is a sandbox plus a reward function—sandboxes with rubrics. Evals are the third curve and the quiet constant: every serious agent team replays production traces and runs simulated adversaries inside the same isolation, which means the eval harness from the [anatomy post](/posts/ai/anatomy-of-a-vertical-ai-agent/) is a sandbox customer whether it knows it or not. The fourth curve is the one the agent loop doesn't contain: serving. The app-generation platforms—the Lovable class—produce software whose *runtime* is a sandbox: the preview URL behind a vibe-coded app is a microVM with a tunnel to it, and production platforms have run tens of thousands of them concurrently for a single customer. This is where the title stops being a metaphor, because AI-written software doesn't become trusted when it starts working—the isolation follows the artifact into production, and the sandbox literally is the server. (It also prices like hosting rather than attempts—the exception that maps the boundary.) Step back and all four curves are the same substrate activity, arbitrary code execution, distinguished only by what the buyer wants back: a result, a gradient, a grade, an uptime. ## Compute for attempts Follow the pricing unit and you can watch the industry reprice itself. VMs sold instance-hours—you paid for existence. Functions sold invocations—you paid for events. Sandboxes sell session-minutes and forks—you pay for *attempts*. That's the honest unit for a tenant whose job is trying things, and it has a property the old units didn't: attempt volume scales with model capability, not with end-user traffic. Every improvement in agent reliability increases how much work gets delegated, which increases attempts. The demand curve compounds from the supply side. The market structure is a squeeze. From above: the hyperscalers and the labs are bundling execution primitives into their agent platforms, and a bundled good-enough sandbox is how platforms have always eaten adjacent categories. From below: the isolation layer is open source—Firecracker, gVisor, Kata—so the moat can't be the wall itself. What's left, and it's real, is everything around the wall: cold-start engineering measured in single-digit milliseconds, fleet scheduling that absorbs particle-physics bursts, snapshot infrastructure that makes fork cheap, and the rubric layer that turns execution into judgment. Sovereignty pressure cuts the same way it did in the [vertical AI stack](/posts/ai/anatomy-of-a-vertical-ai-agent/): data-residency rules and national-AI programs push execution inside jurisdictional boundaries—and the category has already split along exactly that line. E2B and Steel ship open-source runtimes you can stand up inside your own cloud; Modal and Vercel's sandboxes are managed-only, no BYOC. In a sovereign world that split is strategy, not packaging: the open-runtime vendors can follow demand into any boundary—and must find their moat in fleet operations, since anyone can run their wall—while the closed ones either build a deployable story or cede every regulated deal to the open source underneath. ## Objections *Isn't this just Lambda?* No—by construction. Functions assume trusted code; sandboxes assume the opposite. Functions have no fork, no browser, no verdict channel, and duration caps that make a six-hour human-approval pause impossible. Lambda priced the event. This prices the attempt. *Aren't containers enough?* Not when the author is adversarial. A shared kernel plus a workload that writes its own syscalls is exactly the threat model containers were never asked to hold—and containers have no snapshot semantics, so the fork property dies at the runtime. *Haven't sandboxes existed forever?* As features, yes—coding judges, CI runners, notebook hosts, and plugin platforms have executed untrusted code for decades, each building its isolation internally, each for one narrow job. Agents changed the volume (machine cadence), the generality (any task), and the buyer (everyone shipping agents). The activity is old. The standalone market is what's new. *Are servers actually dying?* No, and the title doesn't need them to. The claim is about the marginal unit: where *new* compute demand lands. It's landing in attempts. ## The primitive, mapped | Property of the workload | Primitive it forces | Who's selling it | |---|---|---| | **Untrusted from the inside** | hard isolation by default; egress as policy | E2B (Firecracker), Modal (gVisor), Daytona | | **Mayfly lifetime** | fast boot + snapshot restore | the entire category, benchmarked in ms | | **State that branches & pauses** | snapshot / restore / fork as API | memory snapshots, persistence tiers | | **Failure as signal** | verdict channels, rich telemetry | eval harnesses, RL reward plumbing | | **Particle-physics concurrency** | snapshot fan-out, fleet headroom | tens of thousands concurrent, per customer | | **Browser as device class** | managed sessions, auth, takeover | Browserbase, Steel, Hyperbrowser | The arc that started with time-sharing ends, for now, with a tenant that thinks, fails on purpose, and gets graded—and a primitive shaped exactly like it. The attempt is becoming a billable unit of compute. Billable units of compute have a way of becoming industries. --- ## One Pattern Behind Every Vertical AI Startup - URL: https://svalle.ru/posts/ai/anatomy-of-a-vertical-ai-agent/ - Date: 2026-07-22 - Tags: AI, agents, LLM, architecture, infrastructure, startups Giga sells voice agents for enterprise customer support. Salient runs collections calls for auto lenders. F2 turns private-market data rooms into investment memos. Kanu ships infrastructure code inside your AWS account. Four products, four industries, four pitches. Lift the hood and it's one machine. Strip the domain logic away and every production vertical agent is the same architecture: **a workflow engine that delegates bounded decisions to LLMs, assembles fresh context every turn, treats every model output as an unverified proposal, and logs everything into a trace stream that feeds evals and training.** The [agent loop itself is mostly a while loop](/posts/ai/agent-building-for-engineers/)—I've written about that. This post is about everything production wraps around the loop, and the interesting part: almost every wrapper is now a purchasable component. Walk the machine once, end to end, and you get two things at the same time—the generic pattern, and a map of the startup landscape that sells it. Here's the pattern. Everything below is a tour of it, one box at a time. ```d2 {alt="The full vertical AI agent architecture: channels and voice I/O flow into intent and routing, then into context assembly which is also fed by the knowledge pipeline and by session and memory; context feeds the orchestrator running a bounded agent loop; the orchestrator sends proposed actions through a policy gate to tools and sandboxed execution, whose results return to the loop; the orchestrator's output passes a verifier which either commits and writes back or hands off to a human; both outcomes write into the trace stream that drives evals and updates"} direction: down channels: "Channels & voice I/O\n(STT · TTS · the turn budget)" intent: "Intent & routing\n(small fast models)" knowledge: "Knowledge pipeline\n(parse → embed → index)" session: "Session & memory\n(conversation · entity · long-term)" context: "Context assembly\n(instructions + state + knowledge + history)" loop: "Orchestrator\n(workflow state machine + bounded agent loop)" gate: "Policy gate\n(every action is a proposal)" exec: "Tools & sandboxed execution\n(MCP · scoped credentials)" verify: "Verifier\n(invariants · grounding · judge)" commit: "Commit & write-back" handoff: "Human handoff\n(context packet)" traces: "Trace stream → evals → updates\n(context · config · weights)" channels -> intent -> context knowledge -> context session -> context context -> loop loop -> gate -> exec exec -> loop: "results" loop -> verify verify -> commit: "pass" verify -> handoff: "escalate" commit -> traces handoff -> traces ``` For each component: what it does, why it exists, and who sells it. ## 1. Channels and voice I/O Everything enters through a channel adapter—web chat, SMS, email, API, phone. Text channels are easy. Voice is where this stage becomes a product category, because a natural conversation gives you roughly 800 milliseconds per turn, and that budget has to cover speech-to-text, deciding the caller has actually finished talking (endpointing—genuinely hard), the model, and text-to-speech, all streaming. The landscape splits into layers people routinely conflate. STT: Deepgram, AssemblyAI. TTS: ElevenLabs, Cartesia. And a layer *above* both: the voice pipeline—Vapi, Retell, LiveKit, Pipecat—which wires STT, LLM, and TTS together with interruption handling and telephony. Vapi is not a speech model; it's the plumbing that makes speech models conversational. Holding that budget while the agent reasons mid-turn is precisely why Giga rebuilt this stage instead of buying it. ## 2. Intent extraction and routing Before the expensive model wakes up, a small fast one decides what this request *is*: classify the intent, verify the caller, pick the workflow. This is almost always an in-house fine-tune—the latency budget demands a small model, and the training labels are your own production traffic, so there's little to buy. The adjacent buyable layer is model routing—OpenRouter, LiteLLM, Martian—deciding which model serves which call. When a company says its platform is "LLM-agnostic," this component is what they mean: the architecture owns the routing, so no single model is load-bearing. ## 3. Session, state, and memory The agent needs three kinds of state, and collapsing them causes most memory confusion. **Conversation state**: this session's transcript, which must survive disconnects and channel hops—the same borrower on a call, then SMS. **Entity state**: the order, the borrower, the deal—read live from your system of record, never cached in the model's head. **Long-term memory**: what previous sessions taught, retrieved per turn. The first two are yours to build on ordinary infrastructure. The third is a product category: Letta, Zep, Mem0. Salient markets "borrower-level memory"—prior calls, promises, disputes informing every interaction. That's this component, wearing a compliance department. ## 4. The knowledge pipeline RAG is three components wearing one acronym, and they run on different clocks. ```d2 {alt="The knowledge pipeline as a linear chain: documents and systems flow into parsing and OCR, then chunking and embedding, then the index, then retrieval and reranking at request time, ending at context assembly"} direction: down docs: "docs & systems of record" parse: "parse & OCR\n(tables · layout · scans)" embed: "chunk & embed" index: "index\n(vector + keyword)" retrieve: "retrieve & rerank\n(request time, ~100ms)" ctx: "context assembly" docs -> parse -> embed -> index -> retrieve -> ctx ``` Offline, throughput-bound: parsing—Unstructured, LlamaParse, Reducto—turning PDFs, tables, and scans into structure; then embedding—Voyage, Cohere—into an index: Pinecone, Qdrant, Weaviate, Chroma, pgvector, Turbopuffer. Online, latency-bound: hybrid retrieval plus a reranker, inside about 100ms because it sits on the hot path. The offline half is where verticals live or die. F2's entire wedge is parsing messy data rooms at financial-grade fidelity—the fanciest retrieval cannot fix a bad parse. One taxonomy correction worth internalizing: Exa is not a vector database. It, along with Tavily, is neural search over the *web*—which makes it a tool the agent calls, and it belongs two sections down. ## 5. The orchestrator and the bounded loop The center of the diagram, and the part demos most misrepresent. Production vertical agents are **not** open-ended ReAct loops pointed at a pile of tools. The outer structure is a deterministic workflow state machine—compiled from flows that domain experts draw in builders like Giga's Agent Canvas or Salient's Agent Studio. The LLM gets *bounded autonomy inside each node*: reason, act, observe, against a whitelisted tool set, under a step cap, with a defined surrender path. Autonomy is a per-node budget, not a global property. This single design decision is why these systems pass enterprise procurement while impressive demos don't. The buyable layer: LangGraph, CrewAI, and the labs' own agent SDKs for the loop; Temporal underneath for durable execution, so a crash mid-loop resumes instead of restarting the phone call. ## 6. Tools, MCP, and the policy gate The defining trust decision of the whole pattern: **the model never calls anything.** It emits a proposal—tool name, arguments. A declarative policy gate evaluates the proposal against your rules and approves, denies, or escalates. A separate executor with scoped credentials runs what survives. ```d2 {alt="The trust boundary around actions: the model emits a proposed action to the policy gate, which either passes it to a scoped executor whose result returns to the model, or escalates to a human when rules say no"} direction: right model: "model\n(proposes)" gate: "policy gate\n(declarative rules)" executor: "scoped executor" human: "human" model -> gate: "proposed action" gate -> executor: "approved" gate -> human: "escalate" executor -> model: "result" ``` MCP is becoming the wiring standard for the tools themselves. Composio and Arcade sell integration catalogs; Exa, Tavily, and Browserbase are tools in their own right—web search and browser automation. Guardrails vendors—Guardrails AI, NeMo Guardrails, Lakera—cover input/output screening and injection defense. But the policy engine itself (OPA/Cedar-shaped) stays in-house at every serious company, because it encodes the rules that *are* the domain: refund limits, FDCPA disclosure requirements, IAM scopes. The prompt can say "don't refund over $50"; the gate is what makes it true. ## 7. Sandboxed execution When actions are code, blast radius becomes a component. E2B, Modal, and Daytona sell disposable isolation—Firecracker microVMs in E2B's case, gVisor's user-space kernel in Modal's—where an agent can execute, test, and fail without consequence. Kanu's variant inverts the idea: the sandbox is your *actual* AWS account, with isolation enforced by tag-scoped IAM and a battery of security checks rather than a separate VM. Either way the principle holds—the agent acts inside a boundary that was drawn before it started acting. The underestimated flavor is the browser. Most verticals sit on systems of record with no API—insurer portals, government sites, legacy EHRs—so the tool call degrades to an agent driving a web UI, and that live session needs the same containment plus session persistence, auth handling, and a human-takeover path. Browserbase (already on the tools shelf), Steel, and Hyperbrowser sell the managed browser; Browser Use, Stagehand, and Skyvern sell the agent that drives it. And this component is one primitive serving three demand curves, which is why the category is bigger than "run the agent's Python": ```d2 {alt="One sandbox primitive serving three demand curves: inference-time execution for production agents, training-time reinforcement learning rollouts that need thousands of parallel resettable environments, and evaluation and simulation replaying traces in isolation"} direction: right sandbox: "sandbox\n(isolate · snapshot · fork)" inference: "inference time\n(production agents act)" training: "training time\n(parallel RL rollouts)" evals: "evals & simulation\n(replay traces safely)" sandbox -> inference sandbox -> training sandbox -> evals ``` Inference-time execution is the visible curve. The steeper one is training: RL on agentic tasks needs thousands of parallel, resettable environments per experiment, because each trajectory is an agent trying whatever maximizes reward—side effects included—and every rollout needs a clean world to wreck. Snapshot-and-fork is the primitive vendors compete on because RL wants to branch state cheaply, and an environments market is forming on top of it (Prime Intellect's Environments Hub, Mechanize). The third curve belongs two sections down: the eval harness replays traces and runs simulations inside the same isolation. One macro note. Sovereign AI doesn't add executions so much as it *moves* them—data-residency rules and national-AI programs push the execution layer inside jurisdictional or corporate boundaries, which favors isolation you can deploy (BYOC, on-prem, or the open primitives underneath: Firecracker, gVisor, Kata) over pure SaaS. Kanu's inversion above is the same force at company scale. Expect every sandbox vendor to grow a deployable story, or cede that demand to the open source they're built on. ## 8. Verification and human handoff Guardrails constrain actions *before* they happen; verification validates outputs *after*, pre-commit. Schema checks, business invariants (the refund must not exceed the order total), grounding checks (every policy claim traces to a retrieved source), and increasingly a judge model scoring against a rubric. This stays mostly in-house for the same reason the policy gate does: the checks are the domain expertise. When verification fails—or stakes exceed the autonomy budget—the handoff is engineered, not apologized for: a queue plus a context packet (transcript, retrieved policy, attempted and blocked actions) into the desk your humans already use (Zendesk, Genesys); HumanLayer sells the approval-flow primitive for agents. These are partial-autonomy systems by construction. The escalation rate is simultaneously the pricing model, the safety valve, and—because the human's resolution comes back as a labeled example—the training-data faucet. ## 9. Observability, evals, and the learning loop Every model call, retrieval, proposal, gate decision, and execution appends to one trace stream with four consumers: debugging, audit, evals, training data. One artifact, four jobs—which is why the eval/observability shelf is the most crowded corner of the agent-infrastructure map. Tracing and evals: Langfuse, Arize, Braintrust, AgentOps, Weights & Biases Weave. Voice simulation and QA—synthetic callers hammering your agent before real ones do: Hamming, Coval, Cekura (formerly Vocera). The loop's output lands in three destinations with three latencies: ```d2 {alt="The learning loop: the trace stream feeds evals and QA, which fan out to three destinations—context updates that land instantly, config updates to prompts and policies that land in minutes, and weight updates via fine-tuned edge models that land in weeks"} direction: right traces: "trace stream" evals: "evals & QA" memory: "context\n(facts · instant)" config: "config\n(rules · minutes)" weights: "weights\n(reflexes · weeks)" traces -> evals evals -> memory evals -> config evals -> weights ``` Facts go in context. Rules go in config—versioned and rolled back like deployments, which is what makes an audit answerable. Reflexes go in weights: fine-tunes of the *edge* models—the intent classifier, the reranker, the judge—not the frontier model in the middle. OpenPipe, Fireworks, and Predibase sell that capability. The sequencing is invariant: tracing, then evals, then weights. A fine-tune without regression evals is a bet you can't measure. ## The five patterns, composed If you know the five classic agentic design patterns—reflection, tool use, ReAct, planning, multi-agent—they're all in the opening diagram. Not as a menu; as a fixed composition. Planning is the orchestrator's state machine, with most of the plan pre-compiled by domain experts as flows. ReAct is the bounded loop inside each node. Tool use is the proposal–gate–executor triple. Reflection runs at three timescales: in-loop critique, pre-commit verification, and the eval pipeline reflecting over the whole deployment. Multi-agent is the org chart—router, workers, judge—differing in model size as much as in role. The textbook teaches them as five choices. Production ships them as one machine. ## The box you rebuild is the business Every company in this space—swap in Decagon, Sierra, Parloa, Harvey, pick your vertical—buys most of the catalog above and rebuilds exactly one box: the one the catalog can't satisfy. For Giga, no rented pipeline stayed sub-second once retrieval, policy, and tool calls sat inside the turn. For F2, no generic parser could read a data room. For Salient, no off-the-shelf guardrail could encode FDCPA. For Kanu, no hosted sandbox could safely ship changes to a live cloud account. The exception proves the rule: Chatbase, at the self-serve end of the market, rebuilds none of it—pure assembly of the catalog, which caps differentiation exactly as the thesis predicts: when every box is bought, the only levers left are distribution and price. The rebuilt box is the wedge; the trace stream compounds it into the moat, because the terminal asset of the learning loop is a model and a policy set that only your traffic could have produced. The pattern is generic. The box you rebuild is the business. ## Summary | Component | The job | Build or buy | Example vendors | |---|---|---|---| | **Channels & voice I/O** | STT/TTS under a turn budget | Buy (unless voice is the product) | Deepgram, ElevenLabs, Cartesia; pipeline: Vapi, Retell, LiveKit | | **Intent & routing** | Classify, verify, route fast | Build (fine-tuned small models) | routing layer: OpenRouter, LiteLLM | | **Session & memory** | Conversation, entity, long-term state | Build core; buy long-term memory | Letta, Zep, Mem0 | | **Knowledge pipeline** | Parse → embed → index → retrieve | Buy parts; own the parse quality | Unstructured, Reducto; Voyage; Pinecone, Qdrant, pgvector | | **Orchestrator** | State machine + bounded loops | Buy the loop; own the flows | LangGraph, CrewAI, agent SDKs; Temporal | | **Tools & policy gate** | Proposals, rules, scoped execution | Buy tools; build the gate | MCP, Composio, Arcade; Exa, Tavily; Guardrails AI, Lakera | | **Sandbox** | Contain side effects; browser sessions; RL rollouts | Buy | E2B, Modal, Daytona; browsers: Browserbase, Steel, Hyperbrowser | | **Verification & handoff** | Validate pre-commit; escalate well | Build checks; buy approval plumbing | HumanLayer; Zendesk, Genesys | | **Observability & evals** | One trace stream, four consumers | Buy tracing; build the eval set | Langfuse, Arize, Braintrust, AgentOps; Hamming, Coval | | **Fine-tuning** | Reflexes into edge models | Buy the pipeline; own the data | OpenPipe, Fireworks, Predibase | Vendor names will drift—this corner of the industry reshuffles quarterly. The boxes won't. Learn the machine, and the landscape becomes legible: every funding announcement is somebody selling one box harder, and every vertical AI success story is somebody who rebuilt the right one. --- ## Agent Building for Software Engineers: It's Mostly a While Loop - URL: https://svalle.ru/posts/ai/agent-building-for-engineers/ - Date: 2026-06-27 - Tags: AI, agents, LLM, architecture, developer-tools, engineering An AI agent is a while loop wrapped around a model call. That's it. That's the secret. Everything else—tools, memory, planning, "reasoning"—is implementation detail layered on top of that loop. Once you see it, the mystique evaporates and you're left with something far more useful: a system you can actually reason about, debug, and build. This post is for engineers who've used [coding agents like Claude Code](/posts/ai/agentic-coding-patterns/) and now want to build their own. Not the framework-tutorial version. The mental-model version—what an agent *is* as a piece of software, where the hard parts actually live, and why your normal engineering instincts will both help and betray you. ## The Whole Thing in 15 Lines Here's a complete agent. Not pseudocode—the actual shape of every agent ever built: ```python def agent(task, tools): messages = [{"role": "user", "content": task}] while True: response = model.call(messages, tools=tools) messages.append(response) if response.tool_calls: for call in response.tool_calls: result = tools[call.name](**call.args) messages.append({"role": "tool", "content": result}) else: return response.content # model is done, no more tools ``` That's the engine. A loop that calls a model, checks whether the model wants to use a tool, runs the tool if so, feeds the result back, and repeats until the model stops asking for tools. Strip it to a sentence: **the model decides what to do, you execute it, you tell it what happened, and you let it decide again.** If you've written a REPL, an event loop, or a state machine, this is familiar territory. The novelty isn't the control flow. It's that the branch predictor is a language model, and it's non-deterministic. Hold that thought—it's the source of every hard problem later. ## Why This Isn't Request/Response Almost everything you've built talks to a service the same way: request in, response out. You call an endpoint, you get an answer, you're done. Stateless, one round-trip, and *you* wrote every branch that decided what happened next. ```d2 {alt="Two shapes compared: request/response sends a request to a service and gets a response back in one stateless call you control, while the agentic loop sends a task to a model that emits tool calls you run, feeding each result back to the model repeatedly until it says done"} grid-columns: 1 vertical-gap: 32 chain: "Request/response (everything you know)" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 126 vertical-gap: 8 request: request {style: {stroke-width: 0; fill: transparent}} service response: "response (done)" {style: {stroke-width: 0; fill: transparent}} request -> service -> response } n1: "One call. Stateless. You own the control flow." {style: {stroke-width: 0; fill: transparent}} spacer: "" {height: 40; style: {stroke-width: 0; fill: transparent}} loop: "Agentic loop (the new shape)" { style.stroke-width: 0 style.fill: transparent direction: right task model toolcall: tool call run: you run it task -> model -> toolcall -> run run -> model: "result\n(repeat until the model says \"done\")" } n2: "Many calls. Stateful. The model owns the control flow." {style: {stroke-width: 0; fill: transparent}} ``` The difference that matters isn't the number of round-trips. It's *who decides what happens next.* In request/response, you did—you wrote the `if` statements. In an agentic loop, the model does. An analogy. Conventional programming is telling a system: **"if the room is dark, turn on the light."** You inspected the condition, you chose the action, you wrote the branch. Deterministic and total—every path is one you anticipated. An agent is different. You hand the model a goal—"make the room brighter"—and a set of tools: *turn on the light, lift the blinds, open the curtains.* Then you let it decide which to use, in what order, and when it's bright enough to stop. You didn't write the branch. You delegated the decision. That delegation is the entire point. It's why an agent can handle tasks you couldn't enumerate in advance—and it's also the source of every hard problem in this post. The moment the model owns the control flow, you trade the guarantees of determinism for the flexibility of judgment. Everything that follows is managing that trade. ## Tools Are Just Functions With a Schema The word "tool" makes it sound special. It isn't. A tool is a function you expose to the model, described well enough that the model knows when and how to call it. ```python { "name": "get_weather", "description": "Get current weather for a city.", "parameters": { "city": {"type": "string", "description": "City name, e.g. 'Tokyo'"} } } ``` The model never runs your code. It emits a structured request—"I'd like to call `get_weather` with `city='Tokyo'`"—and *you* execute that in your runtime and hand back the result. The model is the planner; your code is the hands. This means tool building is API design, a discipline you already have. The same rules apply, just with a non-human consumer: ``` Good tool design: - One tool, one job (get_weather, not do_everything) - Descriptive names and parameter docs (the model reads these) - Return structured, parseable results - Fail loudly with useful error messages Bad tool design: - Overloaded tools with 15 optional parameters - Vague descriptions ("processes data") - Returning raw stack traces or 10,000-line blobs - Silent failures the model can't detect ``` The model reads your descriptions the way a new engineer reads your API docs—except it has no Slack to ask follow-ups. The quality of your descriptions *is* the quality of your tool calls. Ambiguity in the docstring shows up as the model calling the wrong tool with the wrong arguments. One rule that matters more than the rest: **error messages are prompts.** When a tool fails, the model reads the error and decides what to do next. "Error: 500" tells it nothing. "Error: city 'Tokoy' not found, did you mean 'Tokyo'?" lets it self-correct on the next loop iteration. Your error strings are part of the agent's reasoning surface. ## The Context Window Is Your RAM, and It's Tiny Here's where engineers get blindsided. You're used to systems with effectively unlimited memory and perfect recall. An agent has neither. Everything the agent "knows" in the moment lives in the context window—the running list of messages you pass back on every model call. The task, the conversation, every tool result, every intermediate thought. All of it competes for a fixed token budget. ```d2 {alt="Context window as working memory: a stack of slots holding the always-present system prompt, the task you asked, and tool call plus result entries that grow every iteration, above a divider noting the window is eventually full"} grid-columns: 1 vertical-gap: 20 caption: "Context window = working memory" {style: {stroke-width: 0; fill: transparent}} sys: "system prompt\n(who the agent is — always present)" task: "task\n(what you asked)" t1: "tool call + result\n(grows every iteration)" t2: "tool call + result" t3: "tool call + result\n(...and grows)" t4: "tool call + result" divider: "" {height: 6; style: {fill: "#545e6a"; stroke-width: 0}} full: "Eventually: full. Now what?" {style: {stroke-width: 0; fill: transparent}} ``` Every loop iteration *adds* to this. A long-running agent—say, one debugging across dozens of files—will blow through its budget. When it fills, you have three options, and managing them is the actual job of agent engineering: **Truncation.** Drop the oldest messages. Cheap, simple, and the agent forgets what it was doing. Works for stateless tasks, fails for anything requiring continuity. **Compaction.** Summarize old turns into a compressed note and replace the raw history. This is what serious agents do. (If you've seen a coding agent say "compacting conversation," this is it.) You trade fidelity for space—the summary keeps the gist and loses the detail. **Retrieval.** Keep the bulk of state *outside* the context in a store, and pull in only what's relevant for the current step. The agent's memory becomes a database query, not a scroll-back. This discipline—deciding what enters the context, what gets summarized, and what gets fetched on demand—is **context engineering**, and it's where most of your design effort will actually go. Prompt wording is the part everyone talks about; context management is the part that determines whether the agent works. The counterintuitive bit: more context is not better. A bloated context window degrades model performance—relevant details get buried under transcript noise, and the model starts attending to the wrong things. People call it "context rot." Curating *down* often beats stuffing *in*. Treat tokens like a memory budget in an embedded system: every byte you spend on history is a byte you can't spend on thinking. ## Memory: Short-Term vs Long-Term Once you accept the context window as RAM, the next question answers itself: you need disk too. ``` Short-term memory = the context window (this session, this task, ephemeral) Long-term memory = external store + retrieval (across sessions, persistent, queried) ``` Short-term memory is just the message list—it dies when the session ends. For anything that needs to persist (user preferences, facts learned last week, accumulated project knowledge), you write to an external store and retrieve relevant pieces back into context when needed. The retrieval can be as simple as a key-value lookup or as involved as semantic search over embeddings. Start simple. A surprising number of "memory" requirements are satisfied by a plain database keyed on user ID, not a vector store. Reach for embeddings when you actually need fuzzy recall over unstructured history—not before. The vector database is a tool, not a rite of passage. ## Non-Determinism Will Break Your Intuitions Every engineering instinct you have was built on deterministic systems. Same input, same output. Stack traces that point at the line. Bugs that reproduce. Conventional programs earn that determinism by construction: the same inputs walk the same branches to the same result, every run. That's the contract the dark-room `if` statement gives you—you wrote the path, so you can predict it. An agent breaks the contract on purpose. You handed the control flow to a model, and the model is non-deterministic: identical inputs can produce different outputs across runs. This isn't a bug you can fix. It's the substrate. And it has consequences that ripple through your whole design: **Testing changes completely.** You can't assert `output == expected` when the output is phrased differently every time. A passing run doesn't guarantee the next one passes. Your test suite is now a *distribution*, not a checkmark. **Retries are a real strategy.** In deterministic code, retrying the same operation gives the same failure. In agent land, retrying can genuinely work, because the model rolls the dice again. Sometimes the fix is a loop with three attempts. **Idempotency matters more.** If the model might call your `send_email` tool twice—and across enough runs it will—that tool had better be safe to call twice. Design tools assuming they may fire more than once. The non-determinism that helps you on retries hurts you on side effects. **Reproduction is painful.** "It deleted the wrong file" might not reproduce on the next run. Log everything—full message history, every tool call, every argument—because the transcript is the only forensic trail you get. You cannot re-run your way back to the bug. This is the deepest adjustment for engineers. You're no longer building a machine that does the same thing every time. You're building a system that does a *reasonable* thing most of the time, and you engineer the guardrails so the unreasonable tail is survivable. ## Where Agents Actually Fail The 15-line loop works in a demo. Production is where the loop meets reality. The failure modes are specific and, once you know them, predictable. ``` The runaway loop: Model calls tool → gets confused by result → calls again → same confusion → calls again → ... → $400 in API costs Fix: hard cap on iterations. Always. ``` **The infinite loop.** Nothing in the basic loop says "stop after N steps." A confused agent will cheerfully call tools forever. Always cap iterations. Always set a budget. This is the first guardrail you add and the one you'll be gladdest you have. **Hallucinated tool arguments.** The model calls a real tool with invented parameters—a file path that doesn't exist, an ID it made up. Validate arguments before executing, and return a useful error so the model can recover on the next turn rather than charging ahead. **Context exhaustion mid-task.** The agent fills its window halfway through a long job and loses the thread. This is the compaction/retrieval problem above, and it's why you design for it *before* you need it, not after the agent face-plants. **Cascading errors.** One bad tool result poisons every subsequent decision. The agent builds an entire plan on a misread, and each step compounds the original mistake. Early validation and checkpoints contain the blast radius. **Silent wrongness.** The worst one. The agent confidently does the wrong thing and reports success. No error, no crash—just a plausible, incorrect result. This is why you don't let an unsupervised agent touch anything destructive, and why evals exist. ## Evals Are the New Unit Tests You cannot unit-test an agent the normal way—the output varies every run. So you do the next best thing: you build an eval set and measure behavior across many cases, accepting a pass *rate* instead of a pass/fail. ``` Traditional test: assert add(2, 2) == 4 # deterministic, binary Agent eval: Run agent on 50 tasks → Did it complete each? → Score Track pass rate over time → Catch regressions ``` An eval is a task plus a way to judge the result. The judge might be an exact check ("did the file get created?"), a rubric, or—increasingly—another model grading the output against criteria. You assemble a representative set of tasks, run the agent against all of them, and watch the aggregate score. This matters because of non-determinism *and* because the thing you're really testing is the system around the model: your tools, your context management, your prompts. When you change a tool description or tweak the system prompt, the eval set tells you whether you helped or quietly broke something three tasks over. Without evals, you're shipping changes to a non-deterministic system on vibes. With them, you have a regression signal. It's the closest thing to CI that agentic systems have, and it's not optional past toy scale. ## Guardrails: Don't Give the Loop a Loaded Gun The agent loop will do what the model decides. So the engineering question is: what are you comfortable letting it decide *unsupervised*? ``` Permission tiers: Auto-approve: read a file, run a query, search the web (reversible, low blast radius) Ask first: write a file, send a message, spend money (side effects, hard to undo) Never autonomous: delete production data, deploy, wire money (catastrophic, irreversible) ``` The pattern is to gate tools by reversibility. Reading is safe—let it run. Writing has consequences—maybe confirm. Anything irreversible or expensive gets a human in the loop, full stop. This is just the [principle of least privilege](/posts/business/infinite-game-of-infrastructure/) applied to a non-deterministic actor. Two more guardrails worth building in from the start: **Sandboxing.** If the agent executes code or commands, run it somewhere contained—a container, a VM, a scratch directory—not on the host that matters. Assume it will eventually do something you didn't anticipate, and make that survivable. **Human-in-the-loop checkpoints.** For consequential multi-step work, insert a "show me the plan before executing" gate. Cheap insurance against an agent confidently marching off a cliff. The check-in is far cheaper than the cleanup. ## You Don't Have to Write the Loop Yourself You just read the whole engine in fifteen lines, and understanding it is the point—but in production you usually won't hand-roll it. There's a worthwhile story in why. Both major labs built a *coding* agent first. OpenAI shipped Codex; Anthropic shipped Claude Code. Each was a terminal-driven agent aimed squarely at software tasks—read files, run commands, edit code, check the result. And in building them, both teams ran into the same realization: the harness underneath wasn't coding-specific. The loop, the tool execution, the context management, the permission gating—none of it cared that the domain was code. It was a general agent runtime that happened to be pointed at a codebase. So they generalized it. The patterns from Codex and Claude Code got lifted out into agent SDKs—OpenAI's Agents SDK and Anthropic's Claude Agent SDK (renamed from the Claude Code SDK precisely to signal it's no longer just about code). These hand you the primitives this post described as first-class pieces: the loop, tool/function calling, multi-agent handoffs, sessions and state, guardrails, and tracing—so you configure them instead of reimplementing them. ``` The progression: Coding agent → General SDK (Codex, Claude Code) (Agents SDK, Claude Agent SDK) proving ground the same loop, generalized ``` The direction of travel is the thing to notice: the coding agent was the proving ground, and the SDK is the generalization. The same instincts from [agentic coding](/posts/ai/agentic-coding-patterns/)—delegate clearly, expose good tools, manage context, gate the dangerous actions—are exactly what these toolkits encode. They aren't magic. They're the fifteen-line loop with the hard parts (retries, streaming, context compaction, permissioning) already solved and battle-tested. One caveat, since this is the fastest-moving corner of the field: the specifics—names, APIs, which SDK does what—will drift. The loop underneath won't. That's the whole reason to learn it from the bottom up. Once you understand the engine, an SDK is a convenience that saves you from rebuilding plumbing—not a black box you're forced to trust. ## When NOT to Build an Agent The most important engineering judgment here is restraint. Agents are the answer to a specific shape of problem, and that shape is narrower than the hype suggests. ``` Don't build an agent when: The task is fixed and known → write a script One model call would do it → just call the model You need guaranteed output → agents are non-deterministic Latency is critical → loops are slow (many round-trips) Every step needs verification → the agent saves you nothing ``` If the steps are known in advance, you don't need a model deciding them at runtime—you need a function. If a single prompt answers the question, the loop is pure overhead. Agents earn their complexity only when the path genuinely can't be predicted ahead of time: when the next step truly depends on what the previous step returned, and you couldn't have written the branch yourself. The failure mode of the moment is agentifying things that should be three lines of Python. A `for` loop with a model call inside is not an agent, and it's usually the better design. Reach for the agent loop when the *control flow itself* needs to be decided by the model—the dark room where you don't know in advance whether the fix is the light, the blinds, or the curtains. Otherwise you're paying latency, cost, and non-determinism for flexibility you don't need. ## Putting It Together Here's the architecture once the 15-line loop grows up: ```d2 {alt="The grown-up agent architecture: The Agent Loop (decide, act, observe) fans out to Tools (functions plus schema) and Context Management (compact/RAG); Tools feeds Guardrails (perms plus sandbox) and Context Management feeds Memory (external store), with both converging into Evals (measure behavior)"} direction: down loop: "The Agent Loop\n(decide → act → observe)" tools: "Tools\n(functions + schema)" context: "Context Management\n(compact/RAG)" guardrails: "Guardrails\n(perms + sandbox)" memory: "Memory\n(external store)" evals: "Evals\n(measure behavior)" loop -> tools loop -> context tools -> guardrails context -> memory guardrails -> evals memory -> evals ``` Every box is something you already know how to build. Tools are APIs. Context management is caching and state. Guardrails are permissions and sandboxing. Memory is a datastore with retrieval. Evals are tests, reshaped for a probabilistic world. The agent loop is a state machine. And if you don't want to assemble the boxes yourself, an SDK hands you most of them pre-wired. The only genuinely new thing is the model in the middle, making non-deterministic decisions. Everything around it is software engineering—the same discipline you already practice, pointed at an unusual core. ## Summary | Concept | What it really is | The hard part | |---------|-------------------|---------------| | **Agent loop** | A while loop around a model call | Knowing when to stop | | **Request/response vs loop** | You own control flow vs the model does | Delegating the decision | | **Tools** | Functions with a schema | Descriptions and error messages are prompts | | **Context window** | Working memory (RAM) | Fixed budget; manage what's in it | | **Memory** | External store + retrieval | Don't over-engineer; a DB often suffices | | **Non-determinism** | The substrate, not a bug | Breaks testing, retries, reproduction | | **Failure modes** | Loops, hallucinated args, cascades | Silent wrongness is the worst | | **Evals** | Tests for probabilistic systems | Pass *rate*, not pass/fail | | **Guardrails** | Least privilege + sandboxing | Gate by reversibility | | **SDKs** | The coding-agent loop, generalized | Learn the loop first; specifics drift | | **When to build** | Only when control flow is unknown | Restraint; most tasks aren't agents | The mental model that makes all of this tractable: 1. An agent is a loop: decide, act, observe, repeat 2. It's not request/response—you delegate control flow to the model 3. Tools are functions you expose with good descriptions 4. The context window is finite working memory you must curate 5. You traded determinism for judgment, and that reshapes everything downstream 6. You test with evals, not assertions 7. You contain risk with permissions and sandboxes 8. You don't have to write the loop yourself—but you should understand it before reaching for an SDK 9. You don't build an agent at all unless the path is genuinely unknown Agent building isn't a new discipline you have to learn from zero. It's your existing discipline—API design, state management, error handling, testing, least privilege—rearranged around a probabilistic core. The engineers who build good agents aren't the ones who memorized a framework. They're the ones who understood the loop, respected the non-determinism, and engineered the guardrails that make a fallible model safe to delegate to. It really is mostly a while loop. The craft is in everything you wrap around it. --- ## Imposed vs Chosen: Why Most Transformations Are Theater - URL: https://svalle.ru/posts/business/exogenous-endogenous-change/ - Date: 2026-04-11 - Tags: strategy, organizational-behavior, management, leadership, platform-engineering Your company announced an AI strategy. Not because anyone inside the org identified a use case — because a competitor shipped something that made the board nervous. Six months later, the initiative has executive sponsorship, a program manager, a Confluence space with 200 pages nobody reads, and a weekly steering committee. It also has almost no adoption. The teams who are supposed to integrate LLMs into their workflows don't see how it applies. The engineers who do see how it applies weren't consulted on the architecture. Meanwhile, a team in a corner of the org has been using Claude to automate their code review triage for eight months. No executive sponsorship. No program manager. Just engineers who got tired of a broken process and built something better. Their tool handles 60% of initial reviews, catches real bugs, and saves each developer four hours a week. Nobody in leadership knows it exists. One of these changes has a budget. The other has momentum. They almost never have both. ## Two Sources of Change Every organizational change has a source. Either the pressure comes from outside the organization or it originates from within. **Exogenous change** is triggered externally. A competitor launches a product. A regulation takes effect. A technology shift makes your approach obsolete. The market moves, and leadership responds. **Endogenous change** is triggered internally. An engineer prototypes a better workflow using new tools. A team adopts a practice that spreads organically. Someone writes an internal doc that changes how people think about a problem. ``` Exogenous: Source: Outside the organization Trigger: Threat, regulation, market shift Motivation: Survival, competitive pressure Sponsor: Leadership (top-down) Funding: Easy (fear is persuasive) Adoption: Hard (imposed, not chosen) Knowledge: Low (decision-makers far from the work) Endogenous: Source: Inside the organization Trigger: Pain, curiosity, better idea Motivation: Craft, efficiency, frustration Sponsor: Individual contributors (bottom-up) Funding: Hard (no burning platform) Adoption: Easy (built by users, for users) Knowledge: High (creators are practitioners) ``` Both are real. Both are necessary. But organizations systematically overweight one and underweight the other. ## Why Exogenous Change Gets Funded Exogenous change has a built-in sales pitch: fear. ``` Board meeting, Q1: "Our competitor just shipped an AI-powered feature. We need an AI strategy." Budget approved: $5M Timeline: 12 months Headcount: 25 engineers Executive sponsor: CTO Slack channel: #project-atlas Time from threat to funding: 3 weeks ``` Compare this to an endogenous proposal: ``` Staff engineer, Q1: "I've been using LLMs to automate our review triage. It saves each dev 4 hours a week and catches real bugs. I need 2 engineers for 6 months to productionize it." Response: "Write a proposal" Then: "Get director approval" Then: "Present to the architecture review board" Then: "Align with the AI platform roadmap" Then: "We'll consider it next planning cycle" Time from idea to funding: 6-9 months (if ever) ``` The exogenous proposal had less evidence and more money. The endogenous proposal had a working prototype and couldn't get two engineers. This isn't irrational. It's asymmetric risk perception. Leadership is far more afraid of being disrupted than of missing an internal improvement. The cost of ignoring an external threat is visible — lost market share, board scrutiny, bad press. The cost of ignoring an internal innovation is invisible — slightly worse developer experience, marginally slower processes, one more frustrated senior engineer who quietly updates their resume. ## Why Exogenous Change Fails Here's the problem: exogenous change is easy to fund and hard to execute. The urgency comes from outside, but the work happens inside — and inside, nobody feels the urgency the same way. ``` Leadership: "The market is shifting. We must integrate AI across the org." Engineering: "Integrate it into what? My workflow is fine." Leadership: "We're building an AI platform. Every team ships an AI feature by Q4." Engineering: "We weren't consulted on this. Our use cases don't fit. But sure, we'll check the box." ``` The result is what organizational theorists call **decoupling** — the formal structure changes (new tools, new processes, new org chart) while the actual work stays the same. Teams register on the AI platform to satisfy the mandate and keep doing their work the old way. The adoption dashboard shows 80% complete. Actual usage is 30%. ``` Exogenous change lifecycle: Phase 1 — Alarm External trigger creates urgency Leadership announces transformation Budget and headcount approved quickly Phase 2 — Theater Program stands up Weekly status meetings begin Dashboards track "progress" (activity, not adoption) Teams comply minimally Phase 3 — Fatigue Urgency fades (the competitor's product wasn't that good) Teams resist because the solution doesn't fit their needs Leadership attention moves to the next crisis Phase 4 — Abandonment Initiative quietly deprioritized Platform team reduced Teams revert to old practices $5M spent. Little changed. ``` This cycle repeats. Each time, it makes the next transformation harder because the organization learns that it can wait out any initiative. ## Why Endogenous Change Sticks Endogenous change has the opposite problem: hard to fund, easy to execute. When an engineer builds an LLM-powered tool because their existing process drives them crazy, the motivation is intrinsic. They understand the problem because they live it. The solution fits because the builder is the user. Adoption happens because colleagues see something that works and ask to use it. ``` Endogenous change lifecycle: Phase 1 — Frustration Practitioner hits a pain point repeatedly Existing solutions don't work or don't exist Phase 2 — Prototype Built on slack time, hack days, or "skunkworks" No formal approval, no budget Works for the builder's team Phase 3 — Organic Spread Neighboring teams notice "How is your team triaging reviews that fast?" Word of mouth adoption Phase 4 — Legitimacy Crisis Too many teams depend on an unsupported tool Leadership discovers it exists Fork in the road: a) Fund it and formalize it b) Kill it and force migration to the "official" solution ``` Option (b) is depressingly common. Leadership kills the thing that works and mandates adoption of the thing that doesn't — because the official solution went through the proper process, has a program manager, and exists on someone's roadmap. ``` The endogenous innovation, discovered by leadership: VP: "Who approved this?" Director: "Nobody. The team just built it." VP: "It's not on the roadmap." Director: "No, but 8 teams use it daily." VP: "It hasn't been through security review." Director: "It's been running for 8 months with zero incidents." VP: "Shut it down. We have an AI platform team for this." The AI platform team's solution: In development for 14 months 0 teams in production 3 teams in "beta" Last meaningful update: 6 weeks ago ``` The endorsed solution has legitimacy. The grassroots solution has users. Organizations that consistently choose legitimacy over adoption train their best engineers to stop trying. ## Punctuated Equilibrium In evolutionary biology, punctuated equilibrium describes a pattern where species remain stable for long periods, then change rapidly in response to environmental disruption. Organizational theorists Tushman and Romanelli applied the same model to companies. ```d2 {alt="Organizational punctuated equilibrium timeline: a long Stability phase (5-10 years, during which processes ossify, culture solidifies, and internal innovation is suppressed) links to an external Disruption (panic, top-down transformation mandated, everything changes at once at high cost and chaos), then 1-2 years of Rapid change, then New stability, and the cycle repeats"} grid-columns: 1 vertical-gap: 24 title: "Organizational punctuated equilibrium" {style: {stroke-width: 0; fill: transparent}} timeline: "" { style: {stroke-width: 0; fill: transparent} grid-rows: 1 grid-gap: 24 stability: "Stability\n(5-10 years)\n\nDuring stability:\n- Processes ossify\n- Culture solidifies\n- Internal innovation\n suppressed\n- \"If it ain't broke,\n don't fix it\"" disruption: "Disruption\n(external)\n\nDuring disruption:\n- Panic\n- Top-down transformation\n mandated\n- Everything changes at once\n- High cost, high chaos,\n uncertain outcomes" rapid: "Rapid change\n(1-2 years)" newstability: "New stability\n(repeat)" stability -- disruption -- rapid -- newstability } ``` This is the default pattern for large organizations. Long periods of exogenous-resistant stability punctuated by frantic, exogenous-driven transformation. The cost of this pattern is enormous. During the stable period, endogenous improvements are blocked — [exploration is cut in favor of exploitation](/posts/business/exploration-vs-exploitation/). During the disruption, the organization over-corrects, launching massive transformations without the institutional knowledge of what actually needs to change. Continuous endogenous change would eliminate the need for most exogenous-driven transformations. But continuous change requires continuous investment in [slack, experimentation, and autonomy](/posts/business/exploration-vs-exploitation/) — exactly the things that get cut during stable periods. ## The Translation Problem The best leaders are **bilingual** — they [speak finite to leadership and infinite to their teams](/posts/business/infinite-game-of-infrastructure/). The exogenous/endogenous split requires a similar translation skill: converting bottom-up insight into top-down language. ``` What the engineer knows: "I built an LLM tool that automates review triage. 8 teams use it. It saves 4 hours per dev per week." What leadership needs to hear: "We have an engineering efficiency opportunity worth $1.2M/year. A proof-of-concept serving 8 teams has demonstrated 93% accuracy and measurable time savings. I'm proposing we formalize this as a Q3 initiative with 2 engineers and a $200K budget." Same change. Different language. ``` The engineer who can make this translation gets their innovation funded. The one who can't watches it get killed. This is a structural failure, not a personal one. Organizations should have mechanisms for endogenous innovation to surface without requiring every engineer to also be a business case writer. Most don't. ## Making Endogenous Change Survive If endogenous change is higher quality but lower funded, the fix is structural: create pathways for bottom-up innovation to get resources without requiring an external crisis. ### 1. Legitimize the Skunkworks ``` Instead of: "Who approved this?" Try: "This has 8 teams using it with zero incidents? Let's fund it properly." ``` Create an explicit path from prototype to product. Hack days produce prototypes. What happens next? In most orgs, nothing — the prototype dies when sprint work resumes. Build a lightweight process: demo → review → small funding → pilot → scale. ### 2. Fund Small Bets Continuously Don't wait for a crisis to fund change. Allocate a standing budget for internal innovation — not a massive "transformation fund," but a modest, renewable pool for 2-4 person experiments. ``` Exogenous funding model: Crisis → $5M → 25 engineers → 12-month program Hit rate: Low (solution designed far from the problem) Endogenous funding model: Standing allocation → $500K/year → 5 small bets Hit rate: Higher (solutions built by practitioners) Cost of failure: Low (small bets, not big programs) ``` This is [option value thinking](/posts/business/exploration-vs-exploitation/) — you're buying cheap options on future improvements. ### 3. Measure Adoption, Not Compliance Exogenous change tracks compliance: "200/200 teams registered on the AI platform." Endogenous change tracks adoption: "How many teams are actually using it daily?" ``` Compliance metric: "Platform registration: 85% complete" (Doesn't tell you if anyone is actually using it) Adoption metric: "Daily active teams: 60% of eligible" (Tells you the solution is solving a real problem) ``` If teams aren't voluntarily adopting, the solution doesn't fit — regardless of what the registration dashboard says. ### 4. Protect Endogenous Change from Exogenous Mandates The most common way endogenous innovation dies: a top-down mandate replaces it with the "official" solution. ``` Before mandate: Team A: Custom LLM review tool (works great) Team B: Custom AI test generator (works great) Team C: Waiting for official AI platform After mandate: All teams: Official AI platform (works for Team C) Team A: Lost their review tool, worse off Team B: Lost their test generator, worse off Team C: Slightly better off, but at 3x the cost ``` Smart organizations incorporate endogenous innovations into the official platform rather than competing with them. The team that built the better tool should be consulted — or hired onto the platform team — not overridden. ## The Synthesis The healthiest organizations don't rely on either source alone. They use exogenous pressure to create urgency and endogenous knowledge to direct the response. ``` Pure exogenous: Urgency: High Direction: Wrong (designed by people far from the work) Adoption: Low (imposed) Outcome: Expensive theater Pure endogenous: Urgency: Low (no burning platform) Direction: Right (designed by practitioners) Adoption: High (chosen) Outcome: Never gets funded Combined: Urgency: High (exogenous trigger) Direction: Right (endogenous practitioners shape the solution) Adoption: High (solution fits real needs) Outcome: Actual transformation ``` The pattern: leadership sets the strategic direction (we need to integrate AI into our workflows). Practitioners shape the solution (here's what we've already built that works). Funding flows to proven prototypes, not PowerPoint architectures. This requires leadership to do something uncomfortable: admit that the people closest to the problem know more about the solution than the people approving the budget. It requires practitioners to do something equally uncomfortable: learn to package their innovations in language that secures resources. ## Summary | Dimension | Exogenous Change | Endogenous Change | |-----------|-----------------|-------------------| | **Source** | Market, regulation, competitor | Practitioner pain, curiosity, craft | | **Trigger** | Threat | Frustration | | **Funding** | Easy (fear sells) | Hard (no crisis to point to) | | **Knowledge** | Low (far from the work) | High (built by practitioners) | | **Adoption** | Low (imposed) | High (chosen) | | **Speed to budget** | Weeks | Months (if ever) | | **Quality of solution** | Variable (often poor fit) | High (built from real needs) | | **Failure mode** | Expensive theater | Dies in obscurity | **The pattern in most large organizations:** 1. Endogenous innovation is suppressed during stable periods 2. Exogenous shock triggers panic-driven transformation 3. Transformation is designed top-down, far from the work 4. Adoption is low because the solution doesn't fit 5. Initiative fades. Organization returns to stability 6. Repeat **The fix:** - Fund small endogenous bets continuously, not big exogenous programs episodically - When exogenous pressure creates urgency, channel it through endogenous expertise - Measure adoption, not compliance - Create pathways for bottom-up innovation to surface and get resourced - Stop killing grassroots solutions in favor of official ones that don't work The organizations that transform well aren't the ones that react fastest to external threats. They're the ones that were already changing from within — and use external pressure to accelerate what was already working. Change that's imposed is tolerated. Change that's chosen is adopted. The difference between transformation theater and actual transformation is which kind you're funding. --- ## Portfolio Theory for Managing Engineering Teams - URL: https://svalle.ru/posts/business/portfolio-theory-for-teams/ - Date: 2026-02-24 - Tags: portfolio-theory, team-building, management, strategy, business You're hiring a senior engineer. Two finalists: a deep Kubernetes specialist and a strong generalist with infra, backend, and some ML experience. Your team is already six K8s specialists. Many managers pick the specialist. Portfolio theory says they're wrong. ## The Portfolio Problem ``` Team A: 8 Kubernetes specialists Quarter 1: Ship K8s platform in record time Quarter 2: Crush the migration backlog Quarter 3: Company pivots to ML-first strategy Quarter 4: Team can't adapt. Nobody knows ML infra. Quarter 5: Emergency hires. Ramp time. Missed deadlines. Quarter 6: Reorg. Layoffs. Team B: 5 K8s + 2 generalists + 1 ML engineer Quarter 1: Ship K8s platform. Slower — fewer specialists. Quarter 2: Finish migration. Generalists handle edge cases. Quarter 3: Company pivots to ML-first strategy. Quarter 4: ML engineer leads. Generalists bridge the gap. Quarter 5: K8s specialists cross-train on ML infra. Quarter 6: Team ships ML platform. Intact. ``` Team A optimized for return. Team B optimized for the portfolio. ## Diversification Isn't Dilution ``` Traditional thinking: Higher-return assets → better portfolio Adding lower-return assets → dilutes performance Markowitz (1952): Adding a lower-return uncorrelated asset can INCREASE portfolio performance by reducing risk disproportionately to the return loss. Portfolio risk ≠ average of individual risks. Correlation between assets determines total risk. ``` ``` Specialist: Individual output: High (in their domain) Cross-team impact: Low Knowledge overlap: 100% with other specialists Generalist: Individual output: Moderate Cross-team impact: High Knowledge overlap: 20% with each specialist The math managers miss: Specialist hire: Team output before: 100 units Specialist adds: +12 units (high individual output) Team output after: 112 units Team risk: Unchanged (same concentration) Generalist hire: Team output before: 100 units Generalist adds: +8 units (moderate individual output) Unblocking effect: +6 units (specialists unblocked on adjacent work) Team output after: 114 units Team risk: Reduced (lower concentration, new coverage) The numbers are illustrative — the real variable is how often your specialists are blocked on adjacent work. If the answer is "frequently," the multiplier is real. The generalist looks worse on paper. The portfolio says otherwise. ``` ## Correlation Is the Hidden Variable ``` High-correlation team: Same skills, same background, same tools, same failure modes. When one person can't solve a problem: Nobody can. Same blind spots. When one domain becomes irrelevant: Everyone is affected. No fallback. When one person leaves: Others cover easily. But the team lost nothing unique. Low-correlation team: Diverse skills, complementary gaps, different perspectives. When one person can't solve a problem: Someone else has a different angle. When one domain becomes irrelevant: Others absorb the shift. Portfolio is hedged. When one person leaves: Specific knowledge is lost. But the team was never dependent on just one axis. ``` | Team Composition | Correlation | What Breaks Them | |---|---|---| | 6 backend engineers, same stack | Very high | New stack adoption, backend hiring freeze | | 5 backend + 1 infra + 1 data | Moderate | Nothing single-point; slower in any one area | | 4 backend + 2 ML + 1 infra + 1 frontend | Low | Less raw backend throughput | | 8 engineers, all different specialties | Very low | No depth anywhere; can't ship large features | ``` Real-world correlation sources (illustrative ranges): Same bootcamp / same CS program: Identical mental models. Identical blind spots. Correlation: 0.8+ Same previous company: Same architectural opinions. Same tool preferences. Correlation: 0.7+ Same specialty, different backgrounds: Overlapping skills, different failure modes. Correlation: 0.4-0.6 Different specialties, different backgrounds: Complementary skills, independent failure modes. Correlation: 0.1-0.3 ``` ## The Efficient Frontier | Composition | Expected Output | Risk (Sensitivity to Shifts) | On Frontier? | |---|---|---|---| | 8 specialists (same domain) | Very high in domain | Very high — one pivot breaks the team | No | | 6 specialists + 2 generalists | High in domain | Moderate — generalists absorb pivots | Yes | | 4 specialists + 2 generalists + 2 adjacent | High across domains | Low — diversified across shifts | Yes | | 2 specialists + 6 generalists | Moderate everywhere | Low — but lacks depth to ship hard features | No | | 8 generalists | Low everywhere | Very low — but can't go deep on anything | No | Pure specialist teams look great until the world changes. Pure generalist teams never look great. The efficient compositions are the diversified mixes in the middle. ## Concentration Risk ``` Bus factor analysis: For each person on the team, ask: "If they leave tomorrow, what breaks?" If Alice leaves: Kubernetes deployments: Everything. She's the only one. CI/CD pipeline: Everything. She built it alone. On-call rotation: Gaps. She covers 3 of 7 nights. Vendor relationships: Two key vendor contacts lost. Alice's concentration: 40% of team's critical knowledge If Bob leaves: Backend API: Slowdown. Two others know it. Database migrations: Nothing. Three people can do this. On-call rotation: Minor. Easy to redistribute. Vendor relationships: None. Bob's concentration: 10% of team's critical knowledge ``` ``` Portfolio concentration by person: Alice: 40% ████████████████████ Bob: 10% █████ Carol: 25% ████████████▌ Dave: 15% ███████▌ Eve: 10% █████ Top 1 person: 40% of critical knowledge Top 2 people: 65% of critical knowledge Benchmark: Healthy portfolio: Top person < 20% Concentrated: Top person 20-35% Dangerous: Top person > 35% This team is concentrated. Alice leaving is a portfolio crash. ``` ## Rebalancing You can't trade people like stocks. But you can rebalance. ``` Rebalancing triggers: - Key person leaves → Knowledge concentrated elsewhere - Strategy shifts → Skills no longer match needs - New technology adopted → Team has no coverage - Bus factor drops below 2 → Single point of failure emerging - Two roles become redundant → Over-concentration in one area ``` | Mechanism | Cost | Time to Impact | Risk Reduction | |---|---|---|---| | Cross-training | Low | 3-6 months | Moderate | | Pair programming rotations | Low | 1-2 months | Low-Moderate | | Stretch assignments | Medium | 2-4 months | Moderate | | Internal transfers | Medium | 1-3 months | High | | Targeted hiring | High | 6-12 months | High | | Rotation programs | Medium | 6-12 months | High | Most teams never do it systematically. ## Where This Breaks Down | Portfolio theory assumes | Teams have none of these | |---|---| | **Liquid markets** → Buy/sell assets freely | **Illiquid market** → Hiring takes 3-6 months. By the time you "rebalance" after a pivot, the pivot has already succeeded or failed. | | **Measurable returns** → Backtest with real data | **Unmeasurable returns** → You can't compute correlation between two engineers. The numbers above are directional, not precise. | | **Continuous rebalancing** → Adjust daily | **Discrete + constrained** → You get 2 headcount at a budget band, not a continuous optimization surface. Backfill reqs are scoped before you see them. | | **Independent returns** → Adding an asset doesn't change others | **Composition-dependent** → A generalist surrounded by specialists performs differently than one surrounded by generalists. Interaction effects are the whole game. | ``` The generalist problem: Hiring: "Senior Generalist" is a hard req to write. Specialists have clearer interview signal. Career: Promotion rubrics reward depth. Generalists get stuck doing critical glue work that doesn't show up in performance reviews. Retention: The best generalists leave for roles with clearer ladders and higher comp. Advocating for generalists without fixing the structures that penalize them is setting them up to fail. At the org level: You don't need every team individually diversified. Concentrated specialist teams + shared platform teams often outperform individually mixed teams. The right portfolio might be your org, not your team. ``` Most of the time, optionality is narrower than you think. ## The Hiring Decision | Current portfolio | 6 Kubernetes specialists | |---|---| | Correlation | ~0.9 (skills, failure modes, career risk all aligned) | | Concentration | K8s = 100% | | Bus factor for K8s | 6 (fine) | | Bus factor for everything else | 0 (disaster) | | | Candidate A: K8s specialist | Candidate B: Generalist (infra, backend, some ML) | |---|---|---| | Individual return | High | Moderate | | **Portfolio impact:** | | | | Correlation with team | 0.95 | 0.3 | | Concentration after | K8s = 100% | K8s = 86%, other = 14% | | Risk reduction | None | Significant | | New capabilities | None | ML triage, backend flexibility | | Bus factor change | K8s: 6→7, everything else: 0→0 | K8s: 6→6, infra: 0→1, ML: 0→1 | ``` Portfolio math: Team with Candidate A: Expected return: +1.0 units (specialist output) Risk change: +0.0 (no new coverage, slightly more depth) Risk-adjusted gain: Negligible Team with Candidate B: Expected return: +0.7 units (generalist output) Risk change: -0.3 (new coverage, lower correlation) Risk-adjusted gain: Significant Candidate A adds return. Candidate B improves the portfolio's risk-adjusted return. This assumes you have both candidates in front of you, the budget to hire either, and a roadmap that doesn't demand K8s-only output next quarter. When you do have that optionality — and sometimes you do — take the portfolio view. ``` ## Summary | Portfolio Concept | Team Equivalent | |---|---| | Diversification | Skill mix across the team | | Correlation | Shared failure modes and blind spots | | Efficient frontier | Optimal composition for output vs. resilience | | Concentration risk | Bus factor / key-person dependency | | Rebalancing | Cross-training, stretch assignments, targeted hiring | | Expected return | Individual output | | Portfolio return | Team throughput | Most teams are over-concentrated and under-diversified. The fix isn't hiring worse engineers—it's hiring uncorrelated ones. --- ## The Infinite Game of Infrastructure - URL: https://svalle.ru/posts/business/infinite-game-of-infrastructure/ - Date: 2026-02-21 - Tags: platform-engineering, strategy, infrastructure, leadership, management You shipped the new platform. The migration is complete. The last team onboarded last Thursday. Your VP sent a congratulations email. The project is done. Except it isn't. It will never be done. The Kubernetes version is already two minor releases behind. Three CVEs were published this morning. The team that onboarded last Thursday filed four bugs today. The monitoring dashboards you built six months ago are tracking metrics for an architecture that has already drifted. And next quarter, someone will ask why the platform team needs the same headcount now that the project is "finished." This is the fundamental misunderstanding at the heart of most platform dysfunction: treating infrastructure like a project with a finish line when it is, by nature, a game that never ends. ## Finite and Infinite Games In 1986, philosopher James Carse published *Finite and Infinite Games*. The core distinction has become one of the most useful mental models in organizational thinking. **Finite games** have known players, fixed rules, an agreed-upon objective, and a clear end. Football. Chess. A product launch. A quarterly target. You play to win, and when someone wins, the game is over. **Infinite games** have known and unknown players, changeable rules, and no defined endpoint. The objective isn't to win — it's to keep playing. Democracy. Parenting. Security. Infrastructure. ## Why Infrastructure Is an Infinite Game ### There is no "done" The moment you stop upgrading, CVEs accumulate — but nobody pretends housekeeping has a finish date. ``` Day 1 after "done": 3 CVEs published in upstream dependencies 1 team requests a feature you didn't scope Week 1 after "done": Kubernetes releases a new patch version Cloud provider deprecates an API you depend on 2 teams file bugs against edge cases Month 1 after "done": Kubernetes releases a new minor version Your monitoring is drifting from reality New compliance requirement changes your security posture 5 teams have workarounds for missing features Quarter 1 after "done": You're one minor version behind Kubernetes 12 known bugs, 3 blocking The architecture assumptions from 6 months ago are wrong Someone asks: "When are we building the next-gen platform?" ``` Infrastructure doesn't have a steady state. It has a **rate of decay** that begins the moment you stop investing. ### Everything shifts You're not serving a fixed customer base. The platform that was "perfect" for 12 teams running microservices now serves 18 teams, three running ML workloads you never designed for. Kubernetes ships every four months. Cloud providers release weekly. The infrastructure that was best-in-class in 2024 is legacy in 2026 — not because it broke, but because the rules changed. ### The objective is continuity, not victory You don't "win" infrastructure. You keep it running, safe, and useful. ## The Finite Frame Problem Organizations run on finite frames. ### Budgets and funding You can't say "we need $2M to keep doing what we're doing" — you have to say "we need $2M for Project Phoenix: Next-Generation Platform Modernization Phase 2." ``` What the work actually is: Continuous upgrades, security patches, capacity management, dependency updates, compliance maintenance, developer support What the budget proposal says: "Project Aurora: Platform Modernization Initiative" Start date: January 15 End date: December 15 Deliverables: 47 line items Success criteria: 12 measurable outcomes ROI: 340% (projected) What happens on December 16: The work continues. The budget resets. The theater starts again. ``` There's no ribbon-cutting ceremony for "everything still works." ### Promotion structures Engineers get promoted for *launching* things, not *maintaining* them. This is **Goodhart's Law** applied to careers — the promotion metric (shipped artifacts) becomes the target, so rational engineers advocate for rewrites even when maintenance would serve the organization better. ## The Symptoms ### The "maintenance mode" trap ``` Phase 1 — Build: Full team, big budget, executive attention Cost: $3M Phase 2 — "Done": Platform declared complete Team reduced from 8 to 3 Budget cut by 60% Executive attention moves elsewhere Phase 3 — Decay: Upgrades deferred Bugs accumulate Security posture degrades Developer experience deteriorates Cost: $0 visible, $500K invisible (lost productivity, risk) Phase 4 — Crisis: Major incident or compliance failure "How did we let this happen?" Emergency project funded Cost: $4M (the original $3M plus panic premium) Phase 5 — Rebuild: Return to Phase 1 "This time we'll do it right" (Narrator: They will repeat Phase 2 in 18 months) ``` Each cycle costs more than continuous investment would have. ### Hero culture Incident response is visible, valued, celebrated. Incident *prevention* is invisible. ``` Visible (rewarded): - Responded to SEV-1 incident in 4 minutes - Coordinated cross-team response to outage - Wrote detailed postmortem with 12 action items Invisible (unrewarded): - Upgraded cluster before EOL, preventing vulnerability - Patched dependency before CVE was exploited - Maintained 100% uptime by doing boring work consistently ``` The organization rewards firefighting, so it unconsciously creates conditions that produce fires. Infinite-game work produces **non-events** — the most valuable and least recognized output a platform team has. ## Playing the Infinite Game You can't eliminate the finite frame, but you can play the infinite game more honestly within it. ### Reframe from projects to capabilities ``` Project framing: "We shipped the deployment platform." Status: Complete ✓ Team: Can be reassigned Budget: Can be reclaimed Capability framing: "We maintain deployment capability." Status: Operational (current version: 3.2) Team: Permanently staffed Budget: Ongoing operational expense Health metrics: Deploy success rate, time-to-deploy, upgrade currency ``` A "completed project" signals resources can be reallocated. An "operational capability" signals permanent commitment. ### Fund teams, not projects Stable teams with persistent mandates — not project teams that assemble for a build phase and disband when the launch email goes out. This requires shifting from project-scoped capital allocation ("$3M for the platform migration") to team-scoped operational funding ("$2M/year for the platform team, evaluated annually on capability health metrics"). ### Measure continuity and celebrate prevention | Finite Metric | Infinite Metric | |---------------|-----------------| | Migration complete: 200/200 services | Services on current platform version: 94% | | CI/CD pipeline shipped | Deploy success rate: 99.7% (30-day trend: improving) | | Security audit passed | CVE exposure window: 48 hours (6-month trend: decreasing) | | Cluster upgrade done | Kubernetes version currency: n-1 (target: n-1 or better) | | SLA achieved this quarter | Availability trajectory: 99.97% → 99.99% over 12 months | | Capacity expansion complete | Capacity headroom: 30% (automated scaling health: green) | Continuity metrics don't have a "done" state — they have a direction. When those metrics show sustained health, make it visible: include it in performance reviews, mention it in all-hands. Prevention is invisible by default; making it visible requires deliberate effort. ## The Paradox The best platform leaders are **bilingual**. They speak finite to leadership and infinite to their teams, translating between two incompatible game theories in both directions. ``` To leadership (finite): "In Q3, we will complete the security hardening initiative, delivering SLSA Level 3 compliance across all pipelines." To the team (infinite): "Supply chain security is an ongoing practice, not a project. SLSA Level 3 is our next milestone, not our finish line. After Q3, we maintain and evolve." Both are true. Both are necessary. The skill is holding both simultaneously. ``` Speak only finite, and you'll build-and-abandon in cycles. Speak only infinite, and you'll never get funded. The art is in the translation. ## Summary | Dimension | Finite-Frame Thinking | Infinite-Game Thinking | |-----------|----------------------|----------------------| | **Funding** | Project-scoped capex | Team-scoped opex | | **Success metric** | Is it shipped? | Is it healthy? | | **Team structure** | Assemble for project, disband after | Permanent team, persistent mandate | | **Architecture** | Sacred artifact to preserve | Mutable tool to evolve | | **Risk posture** | Accept risk after launch | Continuously reduce risk | | **Planning horizon** | Quarterly roadmap with endpoints | Continuous trajectory with milestones | | **Career incentives** | Rewarded for launching | Rewarded for sustaining | Infrastructure isn't a project that finishes. It's a game that continues. The organizations that understand this invest continuously, reward maintenance, and treat every "shipped" milestone as the starting state of the next evolution. The ones that don't keep building, abandoning, and rebuilding — paying the finite-game tax on infinite-game work. The game doesn't end. The only question is whether you're playing it deliberately or pretending it isn't happening. --- ## Activity vs Progress: The Visibility Trap in Engineering Productivity - URL: https://svalle.ru/posts/business/activity-vs-progress/ - Date: 2026-02-21 - Tags: developer-productivity, engineering-management, metrics, business, platform-engineering Your engineering team is working hard. Nobody's slacking. Standups are full, Slack is active, PRs are flowing, sprints are completing on time. A feature that should take three days still takes three weeks to reach a customer. Not because anyone is coasting — because the system around them rewards effort that's visible over effort that matters. ## Why Activity Wins ``` What's on the performance review: "Impact," "ownership," "collaboration," "technical excellence" What leadership actually notices: Always visible in Slack → "strong communicator" Speaks up in every meeting → "has leadership presence" Name on lots of PRs → "high output" Lots of commits, lots of lines → "prolific" Files tickets proactively → "takes initiative" Joins every cross-team thread → "collaborative" What leadership doesn't see: Spent 3 days thinking through a hard design Wrote 50 lines that saved the team 5,000 Said nothing in the meeting because nothing needed saying Didn't file a ticket because they just fixed it ``` None of these signals are on any official rubric. They don't have to be. They shape who gets called "high-performer" at calibration and who gets told to "increase visibility." This isn't because leaders are shallow — measuring individual progress is genuinely hard, and visible signals become proxies because better ones don't exist yet. But developers figure out what gets rewarded fast — and like any [adverse selection](/posts/business/adverse-selection-technical-debt/) dynamic, visible work starts crowding out valuable work. | | Activity Signals | Progress Signals | |----------------|---------------------|--------------------------| | Visible? | Yes, immediately | Only in retrospect | | Countable? | Yes, trivially | Requires judgment | | Attributable? | To individuals | Distributed across teams | | Shapes promo? | Strongly (implicit) | Weakly, if at all | What shapes promotions and what moves the business have almost nothing in common. ## Where the Hours Go {{< chart "activity-vs-progress--developer-8-hour-day" "Horizontal bar chart of a developer's 8-hour day: meetings 1.5h, context switches 1.0h, waiting 1.5h, shallow coding 2.0h, and deep work 2.0h, totaling 8 hours" >}} Activity metrics capture all five categories. They're all "work." Progress comes from the last one. ``` What leadership implicitly rewards: Slack threads resolved → visible Meetings attended → visible PRs reviewed → visible Tickets created & closed → visible Lines of code committed → visible Deep work on hard problems → invisible Visible hours: 6 Deep work hours: 2 ``` Some of those visible hours do matter — a code review catches a real bug, a meeting unblocks a dependency. The problem isn't that visible work is worthless. It's that the system can't distinguish visible work that moved the needle from visible work that didn't. ## Where the Other Hours Go Sometimes the developer *is* trying to ship the feature. The infrastructure won't let them. ``` A 3-day feature, in practice: Day 1: Start coding. CI is backed up — 45 min queue. Push a fix. Flaky test fails. Re-run. Wait again. Staging env is broken. File a ticket. Switch to something else. Day 2: Staging is back. Deploy fails — config drift. Slack thread with platform team. Meeting to triage. Env fixed by EOD. No code shipped. Day 3: Finally deploy to staging. Works. Prod deploy needs approval. Approver is OOO. Day 4: Approved. Deploy to prod. Rollback — dependency issue. Debug. Fix. Redeploy. Works. Day 5: Feature live. 3 days of coding. 5 days of calendar. Activity generated: 14 Slack threads, 3 tickets, 2 meetings, 8 CI runs, 4 deploys. ``` Broken internal infrastructure converts deep work into activity — [waiting waste, defect waste, motion waste](/posts/business/lean-platform-engineering/) packed into a single sprint. The developer didn't choose to spend two days fighting CI and staging — the system forced it. And every hour fighting broken tooling shows up as "activity" on someone's dashboard: tickets filed, threads resolved, incidents triaged. At some point, [infrastructure itself becomes the bottleneck](/posts/business/infrastructure-growth-constraint/) on what the team can ship. The irony: fixing the infrastructure would *reduce* activity metrics across the board. Fewer tickets, fewer Slack threads, fewer heroic debugging sessions. On the dashboard, it looks like the team got less productive — which is why most infrastructure investments [die in the valley](/posts/business/platform-j-curve/) before they pay off. ## The Throughput Illusion Goldratt called this out decades ago: throughput is value reaching users, not things produced. A feature behind a flag that never gets flipped isn't progress — it's inventory. Not all non-user-facing work is waste — migrations, tech debt reduction, and platform improvements enable future throughput. But the question is whether that's what's actually happening, or whether it's the story told after the fact. ``` Team Alpha: PRs merged per week: 50 % reaching users: 10% Throughput: 5 PRs of user value Team Beta: PRs merged per week: 15 % reaching users: 80% Throughput: 12 PRs of user value ``` Team Alpha looks 3.3x faster on the activity dashboard. Team Beta delivers 2.4x more value. | | Alpha | Beta | |--------------------------|-------|------| | Output (PRs/week) | 50 | 15 | | Throughput (user value) | 5 | 12 | | Waste (unused output) | 45 | 3 | | Waste ratio | 90% | 20% | ``` If you're the VP of Engineering: Activity report says: "Alpha is our top team, ship more like them" Throughput report says: "Alpha has a 90% waste ratio" ``` Same data. Two conclusions. One of them costs you quarters of misallocated investment. ## Three Patterns Each of these patterns has a benign explanation — smaller PRs, better CI/CD, legitimate estimation changes. They can also indicate gaming. The data alone won't tell you which. The trend lines will. Pattern 1 — PR inflation: | Quarter | PRs/sprint | Avg lines/PR | Features shipped | |---------|------------|--------------|------------------| | Q1 | 28 | 340 | 4 | | Q2 | 35 | 210 | 4 | | Q3 | 47 | 120 | 3 | **PRs went up 68%. Features went down.** Work is being split to hit the metric. The overhead of reviewing three PRs instead of one slows delivery. Pattern 2 — Deploy frequency theater: | Quarter | Deploys/day | Rollback rate | Incidents/week | |---------|-------------|---------------|----------------| | Q1 | 1.2 | 3% | 2 | | Q2 | 2.1 | 3% | 3 | | Q3 | 3.2 | 3% | 5 | **Deploys went up 167%. Quality stayed the same. Incidents scaled linearly — more deploys, more fires.** Higher frequency with unchanged risk just means more incidents. Config changes and no-op deploys inflate the number without touching confidence. Pattern 3 — Story point inflation: | Quarter | Points/sprint | Customer features | Internal churn | |---------|---------------|-------------------|----------------| | Q1 | 62 | 5 | 12 tickets | | Q2 | 74 | 4 | 22 tickets | | Q3 | 89 | 4 | 38 tickets | **Points up 44%. Customer features flat. Internal churn tripled.** Points measure estimated effort, not value. A 5-point internal refactor and a 5-point revenue feature look identical on the burndown chart. ## What These Metrics Miss | Metric | What It Measures | Gaming Vector | What It Misses | |--------|-----------------|---------------|----------------| | PRs merged | Code movement | Split PRs smaller | Impact of each PR | | Deploy frequency | Release cadence | Deploy config changes | Whether deploys matter | | Story points | Estimated effort burned | Re-estimate higher | Value of the effort | | Slack activity | Communication volume | Reply to everything | Whether anything was decided | | Lines changed | Volume of code | Refactors, formatting | Whether it's the right code | | Tickets closed | Backlog throughput | Create-and-close | Whether anyone needed the ticket | All of these can trend up while the product stands still. The things that would actually catch this — how much calendar time is real work vs waiting, how much of what you build reaches users and whether they found it useful, how much time developers lose to broken tooling — are harder to measure, slower to move, and don't fit neatly into a quarterly calibration. Which is why most orgs default to counting activity instead. ## The Uncomfortable Truth ``` The activity trap: 1. Leadership implicitly rewards visibility 2. Developers learn what gets noticed (rational response) 3. Activity goes up (looks like success) 4. Progress stays flat (nobody's tracking that) 5. Leadership asks "why aren't we shipping faster?" 6. Response: measure more activity (goto 1) ``` Nobody designed this on purpose. Leaders default to visible signals because measuring progress is genuinely hard. Developers optimize for what gets rewarded because that's rational. Everyone is stuck in the same loop. Breaking out starts with one honest question at the next calibration: "what changed for our users this quarter?" Not PRs merged, not tickets closed, not Slack activity — what actually changed. If the room goes quiet, that's the gap between activity and progress. Closing it is uncomfortable, slow, and probably the most important work leadership can do. --- ## Running AI on Kubernetes: What Breaks and What's Being Built - URL: https://svalle.ru/posts/kubernetes/running-ai-on-kubernetes/ - Date: 2026-02-18 - Tags: kubernetes, AI infrastructure, inference serving, GPU scheduling, LLM, agents, Gateway API, llm-d, Ray, platform engineering, vLLM, Kueue Kubernetes was built to orchestrate containerized workloads at scale. But its default abstractions — Deployments, Services, HPA, state-unaware load balancing — are optimized for a specific shape: stateless, ephemeral pods handling roughly uniform requests that complete in milliseconds. The resource model, scheduler, autoscaler, and load balancer were all designed around this pattern. The unit of compute for AI is a 70-billion-parameter model loaded across 4 GPUs, managing a KV cache larger than most databases, serving requests that vary in cost by 1000x. It's not stateless, not ephemeral, not uniform — and every K8s abstraction that assumes otherwise breaks in a specific, diagnosable way. Understanding *how* each one breaks tells you where the infrastructure needs to go next. ## Three Mismatches ### 1. Resources: GPUs Aren't CPUs Kubernetes models resources as continuous and fungible. You request 500m CPU, 256Mi memory. Any CPU will do. You can slice finely — 100m here, 200m there — and the kernel handles sharing through CFS quotas and cgroup limits. GPUs break this at every level. GPU memory is the binding constraint for inference, not compute — a 70B model in fp16 needs roughly 140GB just for weights, and it either fits or it doesn't. There's no "give me 0.7 of a GPU." An A100-40GB, A100-80GB, and H100-80GB are not interchangeable, but Kubernetes sees all three as `nvidia.com/gpu: 1`. The workarounds trade something. MIG physically partitions supported GPUs into isolated instances but requires supported hardware and reconfiguration with no running workloads on the GPU. Time-slicing lets multiple pods share a GPU but provides no memory isolation — one pod's allocation spike can OOM-kill another's inference. Dynamic Resource Allocation (DRA) is the most significant step forward, introducing a claim-based model with rich device attribute matching. But ecosystem adoption is still catching up, and expressing GPU memory capacity and interconnect topology as schedulable attributes requires device plugin support that not all vendors provide yet. The principle: Kubernetes's resource model works when resources are continuous and interchangeable. GPU resources are discrete and heterogeneous — a model either fits on a device or it doesn't, and which specific device matters for performance in ways that CPUs never did. (For a deeper look at DRA and GPU scheduling mechanics, see [GPU Scheduling and Dynamic Resource Allocation](/posts/kubernetes/gpu-scheduling-dra/). For how the CPU resource model itself breaks under CFS quotas, see [CPU Throttling](/posts/kubernetes/cpu-throttling/).) ### 2. State: Models Are Expensive to Move Kubernetes assumes pods are ephemeral — kill one, start another in seconds. Rolling deployments, preemption, rescheduling, autoscaling — all assume cheap pod replacement. For inference, it isn't cheap. Loading a model into GPU memory — downloading weights, transferring to VRAM, warming up the runtime — takes 30-60 seconds for a 7B model, 2-10 minutes for a 70B model. During loading, the replica serves nothing. This makes standard K8s operations actively harmful. Preemption, scale-down, rescheduling — each destroys a loaded model and triggers a multi-minute cold start. Scale-down is particularly bad: HPA has no concept of "this pod's loaded state is expensive to reconstruct," so it tears down a warm replica that might be needed again in five minutes, paying the full loading cost twice. The cattle-not-pets assumption breaks when each pod carries minutes of initialization state in GPU memory. The principle: Kubernetes treats compute as stateless. AI compute is stateful in ways that don't map to PersistentVolumes — model weights live in GPU memory, KV caches are ephemeral but valuable, and the infrastructure has no concept of "prefer to keep this pod alive." ### 3. Traffic: Requests Aren't Equal Kubernetes load balancing assumes roughly uniform requests. A Service distributes via random selection across backends, blind to what each request actually costs to serve. In inference, a 100-token request and a 32K-token request can differ by orders of magnitude in compute cost. vLLM's continuous batching mitigates this *within* a replica, but across replicas, default routing has no visibility into per-replica state — one replica queues while another idles. Even with IPVS mode or external load balancers that support least-connections, connection count is the wrong proxy: one long-running generation is more load than ten short completions. The metrics that actually matter — tokens/sec, time to first token, KV cache utilization, queue depth — don't exist in K8s natively. They come from the inference server and require custom metrics adapters to surface. GPU utilization, the metric everyone reaches for first, is misleading: 95% means the hardware is busy, not that it's making progress. And autoscaling based on these wrong metrics makes the cold start problem from the previous section worse — HPA scales up, a new pod spends minutes loading a model, and the burst passes before it serves a single request. The principle: Kubernetes optimizes for request count. AI workloads need to optimize for token throughput — a fundamentally different unit of work. ## And Then There Are Agents Agents compound all three mismatches at once. An agent mid-task accumulates irreproducible reasoning state with no checkpoint or recovery — kill the pod, lose the chain. Its resource needs are unpredictable — 1K tokens or 500K with 20 tool calls. And tasks run minutes to hours, not milliseconds. K8s quotas, autoscaling, and load balancing all operate on the wrong unit: pods and requests, when what you need is task-level token budgets and state-aware scheduling. ## Can Kubernetes Adapt? K8s wasn't built for stateful workloads either. Then StatefulSets, PersistentVolumes, and operators emerged. The question is whether AI workloads can be solved with the same pattern — extend K8s through CRDs, operators, and plugins. StatefulSets worked because the core scheduling model was still valid for databases. You still placed a pod on a node with CPU and memory — the pod just needed stable identity and persistent storage. For AI workloads, the scheduling model itself is mismatched: the resource abstraction, scaling model, and load balancing model all assume properties that AI workloads don't have. The answer is playing out as "both." K8s is being extended to handle some of these problems, while a new layer emerges above it for the rest. ## The Emerging Stack Three approaches are competing. They're not mutually exclusive — most production stacks will combine elements of all three. ### Extend Kubernetes Add AI-specific semantics through K8s extension points. The bet: K8s won the orchestration war, and the ecosystem benefits are too valuable to abandon. The **Gateway API Inference Extension** adds inference-aware routing as a first-class Gateway API concept. Its `InferencePool` and `InferenceModel` CRDs let the gateway examine live pod metrics — queue depth, KV cache utilization, loaded adapters — to pick the optimal backend per request. Already integrated into Envoy Gateway, kgateway, Istio, and NGINX Gateway Fabric. Its partnership with vLLM on **llm-d** pushes further — splitting inference into separate prefill and decode phases on independent pod pools, with KV-cache-aware scheduling. On scheduling, **Kueue** intercepts jobs before pods are created, admitting them only when all required resources are available — solving the partial scheduling problem for multi-GPU workloads. **LeaderWorkerSet** handles co-scheduling for tensor parallelism. NVIDIA's **KAI Scheduler** adds topology-aware bin-packing and hierarchical queues for GPU workloads at scale. The strength: you keep your existing K8s investment. The weakness: a working inference stack now requires the GPU Operator, a scheduler, a model serving platform, a custom metrics adapter, and an inference gateway — each with its own CRDs and failure modes. And none of this addresses agent orchestration. ### Abstract Above Kubernetes Build a higher-level runtime where K8s becomes the node manager underneath. **Ray Serve** is the most mature example. Ray's actor model — a stateful object that lives on a specific node and maintains state across calls — is a much better primitive for an inference server than a K8s pod. Ray handles scheduling, autoscaling, and fault recovery at the application level. **Modal** goes further: deploy a model with a Python decorator, never think about infrastructure. Managed platforms (Replicate, Baseten, Fireworks) do the same at the endpoint level. The strength is simplicity — developers think in models, not pods. The weakness is the usual managed-services tradeoff: reduced control, vendor coupling, compliance constraints. Ray occupies a middle ground — a layer above K8s that preserves control while offloading AI-specific orchestration, at the cost of running two orchestration systems. ### Purpose-Built Agent Infrastructure Neither extending K8s nor abstracting above it solves agent orchestration, because the problem is new enough that nobody has a production-grade solution. Agent frameworks (LangGraph, CrewAI, AutoGen) handle conversation state and tool orchestration but are increasingly absorbing infrastructure concerns — token budgeting, state persistence, error recovery. AI Gateways (Portkey, LiteLLM) handle model routing and cost tracking. MCP standardizes tool discovery. What's missing is the layer that manages agent lifecycles, enforces task-level token budgets, persists state across failures, and coordinates multi-agent collaboration. This is 2015-era container orchestration — multiple competing approaches, no dominant abstraction. ### Where It's Heading The historical pattern: VMs didn't disappear when containers arrived. Each new abstraction layer coexists with those below it. ```d2 {alt="Layer stack of the emerging AI infrastructure, top to bottom: Agent Orchestration (task management, tool permissions, token budgets, state persistence, multi-agent coordination — LangGraph, CrewAI, wide open), Model Serving (inference routing, token-aware load balancing, KV cache management, model-specific autoscaling — Gateway API Inference Ext, KServe, Ray Serve, vLLM), GPU / Compute Orchestration (device scheduling, topology awareness, gang scheduling, model loading, memory management — Kueue, KAI Scheduler, NVIDIA GPU Operator, DRA), Kubernetes as the distributed OS substrate (node management, networking, storage, RBAC, observability), and Cloud / Bare Metal at the base"} grid-columns: 1 vertical-gap: 24 agent: "Agent Orchestration\ntask management, tool permissions, token budgets,\nstate persistence, multi-agent coordination\n(LangGraph, CrewAI... wide open, nobody has won)" serving: "Model Serving\ninference routing, token-aware load balancing,\nKV cache management, model-specific autoscaling\n(Gateway API Inference Ext, KServe, Ray Serve, vLLM)" gpu: "GPU / Compute Orchestration\ndevice scheduling, topology awareness, gang scheduling,\nmodel loading, memory management\n(Kueue, KAI Scheduler, NVIDIA GPU Operator, DRA)" k8s: "Kubernetes\nnode management, networking, storage, RBAC,\nobservability -- the \"distributed OS\" substrate" cloud: "Cloud / Bare Metal" ``` K8s doesn't get replaced — it drops down a layer, becoming the distributed OS that handles nodes, networking, storage, and security. The AI infrastructure layer builds on K8s the same way K8s built on container runtimes: use the layer below for what it's good at, and handle the new concerns above. If you're running inference today and the K8s ecosystem is familiar, extend it — Gateway API Inference Extension for routing, Kueue for scheduling, DRA for device management. The components are maturing fast and the integration paths are well-documented. If you need the model-level abstraction natively, Ray gives you that without abandoning K8s underneath. You trade K8s-native simplicity for a runtime that was designed for stateful AI workloads. If you're building agent-native products, you're defining the orchestration layer as you go — bespoke, opinionated, purpose-built. It'll look a lot like K8s did in 2015. The gap between what K8s can do and what AI workloads need is closing, just not as fast as the workloads themselves are evolving. --- ## Commodity or Moat? Three Bets on the Future of LLMs - URL: https://svalle.ru/posts/business/llm-pricing-microeconomics/ - Date: 2026-02-17 - Tags: LLM pricing, AI strategy, OpenAI vs Anthropic vs Google, API pricing, AI commoditization, model differentiation, GPT-5 pricing, Claude pricing, Gemini pricing, AI business model In 2023, GPT-4 cost $30/$60 per million tokens. Today, smarter models cost $1-2. That's a 95% price collapse in three years—faster than telecom (30 years), faster than airlines post-deregulation (20 years), faster than almost any market in history. When your marginal cost of production is near-zero, price competition is brutal and fast because there's almost no cost floor to slow the descent. But here's the part that doesn't get enough attention: the prices didn't collapse *uniformly*. Three companies sell into the same market, use comparable architectures, compete on the same benchmarks—and their pricing is wildly different. Not a little different. Anthropic charges 3-5x what Google charges at the budget tier. OpenAI sits in the middle. Google bundles AI into a $250/month mega-plan that includes YouTube Premium and 30TB of storage. Why? Because each company is making a fundamentally different bet about the answer to one question: **will LLMs stay differentiated, or will they commoditize?** Their pricing tells you which future each one is building for. ## The Pricing Landscape Before getting into the bets, here's where things actually stand. **API pricing (per 1M tokens, input/output):** | Tier | OpenAI | Anthropic | Google | |------|--------|-----------|--------| | Flagship | GPT-5.2: $1.75/$14 | Opus 4.6: $5/$25 | Gemini 3 Pro: $2/$12* | | Workhorse | GPT-5: $1.25/$10 | Sonnet 4.5: $3/$15 | Gemini 2.5 Pro: $1.25/$10 | | Budget | GPT-5 Nano: $0.05/$0.40 | Haiku 4.5: $1/$5 | Flash-Lite: $0.10/$0.40 | *\*Gemini 3 Pro is still in preview; stable pricing may settle closer to $1.50/$10.* **Consumer subscriptions:** | Tier | OpenAI | Anthropic | Google | |------|--------|-----------|--------| | Free | ✓ | ✓ | ✓ | | Budget | Go: $8/mo (ads) | — | AI Plus: $7.99/mo | | Standard | Plus: $20/mo | Pro: $20/mo | AI Pro: $19.99/mo | | Premium | Pro: $200/mo | Max 20x: $200/mo | AI Ultra: $249.99/mo | Every provider uses the same playbook underneath—model tiers as screening mechanisms, output tokens priced 3-5x more than input, batch discounts at 50%, subscriptions that are massively cheaper than equivalent API usage. The mechanics are identical. The *positioning* is where they diverge. You could argue the price differences simply reflect cost structures—Google's custom TPU advantage, Anthropic's smaller scale. But cost-plus pricing doesn't explain the strategic choices around bundling, tier design, and what products they *don't* offer. Anthropic could launch a budget tier and doesn't. Google could unbundle Gemini and doesn't. These are choices, not accounting. ## Google's Bet: Models Commoditize Google prices as if models are heading toward utility status—and if you're the lowest-cost producer of a utility, you win. Flash-Lite at $0.10/$0.40 per million tokens. Generous free tiers. An $8/month AI Plus plan. Google doesn't need Gemini to be a profit center because Gemini isn't really their product—it's a feature inside a larger ecosystem. Workspace, Android, Chrome, Search, Cloud. The LLM is the loss leader. The ecosystem is the business. This is clearest in the bundling. Google AI Ultra at $249.99/month—the most expensive consumer AI plan from any provider—includes YouTube Premium, 30TB storage, Google Home Premium Advanced, and top-tier Gemini. The bundle justifies a price that "$250 for AI" alone never would. Different customers value different parts: some want the AI, some want YouTube ad-free, some want the storage. Bundling captures value across all of them. The strategic logic is simple: if models eventually converge on quality, the company that already has distribution and ecosystem lock-in wins by default. You don't beat that by building a better model. You beat it by having built a better everything-else. Google is betting that intelligence becomes a commodity—and positioning to be the one who commoditizes it. This is an existential threat to every standalone AI company. Google can subsidize Gemini indefinitely and recoup the cost across the rest of its business. For Anthropic and OpenAI, AI *is* the business. Every dollar of API revenue matters. You cannot win a price war against a company that doesn't need AI to be profitable. ## Anthropic's Bet: Differentiation Persists Anthropic prices as if the models are *not* converging—and charges accordingly. Opus 4.6 at $5/$25. Haiku 4.5 at $1/$5—their *budget* tier costs 10x Google's. No ad-supported consumer plan. No $8 option. Anthropic looked at Google's bundling strategy and said—we're not playing that game. This only works if the differentiation is real. And so far, it is. Dario Amodei made the point in a [recent interview with Dwarkesh Patel](https://youtu.be/n1E9IZfvGMA): "Everyone knows Claude is good at different things than GPT is good at, than Gemini is good at. It's not just that Claude's good at coding, GPT is good at math and reasoning. It's more subtle than that." The tokens for "restart your Mac" are worth cents. The tokens telling a pharmaceutical company to move an aromatic ring from one end of a molecule to the other could be worth tens of millions. If you're selling the aromatic-ring tokens, you can charge a premium. Anthropic's model tiers reinforce this. Haiku/Sonnet/Opus isn't "small/medium/large"—it's a screening mechanism designed to make customers reveal their price sensitivity. Budget customers choose Haiku. Enterprises choose Opus. The quality gap between tiers is deliberately calibrated to prevent enterprises from trading down. If Sonnet were 95% as good as Opus at hard tasks, no one would pay the 67% premium. The gap has to be real and visible—and maintaining that gap is how Anthropic justifies premium pricing. Anthropic is betting on two kinds of differentiation: model quality (Claude is measurably better at coding, reasoning, nuanced tasks) and ecosystem trust (compliance, safety, enterprise relationships). The first can erode quickly. The second is stickier—and probably the real moat. Being the most expensive might actually be why Anthropic survives. Premium pricing only works with genuine differentiation, which forces them to *stay* differentiated. If they slashed prices to match Google, they'd be fighting a two-front war against Google's subsidies and OpenAI's distribution—and losing both. The high price isn't a weakness. It's a strategic constraint that keeps them focused on the only thing that can save them: being measurably better at the tasks enterprises pay for. The risk is obvious: what if differentiation erodes? What if open-source models close the gap? Then Anthropic is charging a 3-5x premium for a product that's 5% better—and that math doesn't hold. ## OpenAI's Bet: Hedge Everything OpenAI's pricing looks incoherent until you realize they're hedging both sides. GPT-5 at $1.25/$10 undercuts Anthropic while staying above Google's budget tier. It's the middle. In a commoditizing market, the middle is usually where you get squeezed—too expensive to win on price, not differentiated enough to justify a premium. But OpenAI isn't just an API company, and this is what the pricing table misses. ChatGPT's consumer distribution is arguably the strongest moat any of the three have. The $8 ad-supported Go tier (launched January 2026) is designed to capture the hundreds of millions of free-tier users who want more but won't pay $20—OpenAI calls it their fastest-growing plan. The $200 Pro plan captures power users. That's a 25x price gap on the same underlying technology—and the cheap version is deliberately degraded (ads, restricted models) to protect the premium tier. This is textbook versioning: create a worse version not because it costs less to produce, but because selling a good cheap version would cannibalize the expensive one. The Microsoft partnership is the other hedge. Copilot embedded across Office, GitHub, and Azure effectively gives OpenAI its own bundling defense—the same playbook Google is running, just through a partner. Microsoft is already subsidizing OpenAI's models the way Google subsidizes Gemini. So OpenAI is betting on differentiation through the API (premium models, reasoning capabilities) while simultaneously building consumer distribution and bundling partnerships in case differentiation fails. The risk isn't existential; it's bifurcation. Their consumer business is a fortress. Their API business is the part that gets squeezed if models converge. {{< chart "llm-pricing--price-moat-quadrant" "Price versus competitive moat quadrant: Anthropic sits at high price with a model-quality moat; OpenAI API sits at low price near the middle with a developer-base moat; OpenAI Consumer (distribution plus Microsoft) and Google (ecosystem) sit at low price with strong moats." >}} ## The Subscription Trick One thing all three agree on: subscriptions are a better business than API. But the reason is more interesting than it looks. Run the numbers on Anthropic's tiers. According to Anthropic's own data, the average Claude Code developer spends ~$6/day on API tokens, and 90% spend under $12/day. That puts typical heavy usage at $200-360/month. | Usage Level | API Cost/Month | Subscription | Savings | |-------------|----------------|--------------|---------| | Light (few queries) | $2-5 | $20 | -$15 (overpay) | | Medium (daily use) | $100-200 | $20 | 80-90% | | Heavy (90th %ile) | $200-360 | $200 | break-even to 45% | The light user overpays—the $20 subscription is a gym membership they'll never fully use. The medium user gets an 80-90% discount. And even the heavy user roughly breaks even or saves meaningfully on the $200 Max plan. But the interesting question isn't the savings math—it's why Anthropic would offer a plan that *at best* captures the same revenue as API billing. Because the subscription isn't a pricing decision—**it's a customer acquisition cost disguised as a product.** Once Claude Code is embedded in a developer's daily workflow at $200/month, the switching cost is enormous. Anthropic is trading margin for lock-in. This is where the bets converge. Whether you think models commoditize or stay differentiated, lock-in at the application layer matters. Claude Code isn't just an API wrapper—it's a product that generates switching costs. Anthropic isn't selling tokens; they're selling a workflow that happens to consume tokens. Google is doing the same thing through Workspace integration. OpenAI is doing it through ChatGPT habits and Microsoft Copilot. Every company is racing to convert model advantage into application-layer lock-in before the window closes. They just disagree on how much time they have. ## Who's Right? The honest answer: the evidence cuts both ways. The case for commoditization is straightforward. Budget models are already at $0.05-$0.10/MTok and falling toward utility pricing. The mid-tier ($1-3/MTok) is converging on benchmarks—the jump from budget to mid-tier is transformative for most tasks, while mid-tier to premium is marginal for 80% of use cases. Open-source models keep closing the gap. DeepSeek prices at hardware cost. Meta's Llama is free and increasingly competitive—arguably a bigger threat to the differentiation thesis than DeepSeek, because Meta has the resources to keep iterating indefinitely without needing to monetize the model itself. If this trend continues, the premium erodes and the bundler wins. The case for persistent differentiation is subtler but real. Three years in, the models still genuinely differ—not just on benchmarks but on tone, reasoning style, reliability on edge cases. Enterprise customers pay for trust, compliance, and consistency, none of which show up in benchmark comparisons. Dario's cloud analogy is instructive: "There are three, maybe four, players within cloud. I think that's the same for AI." But unlike cloud—where an EC2 instance is an EC2 instance—AI models are not interchangeable. That means more pricing power for anyone who maintains a real quality edge. And then there's the hedge. Consumer distribution is valuable regardless of which future arrives. If models commoditize, ChatGPT is the default interface. If they differentiate, OpenAI has the brand and the Microsoft channel. The hedge works—but it also means OpenAI doesn't dominate either scenario. They survive. They don't necessarily win. The equilibrium probably looks like airlines. Economy (budget models) is fully commoditized—you won't think about the cost, the same way you don't think about the cost of a Google search. Business class (mid-tier) is a temporary battleground heading toward compression. First class (premium) survives for specialized, high-stakes use cases where trust and capability justify the price. ```d2 {alt="Market tier stack: the Premium tier ($5-25/MTok, specialized high-stakes use, stable margins if differentiated) faces convergence pressure pushing down into the Mid-tier ($1-3/MTok, current battleground, temporary margins heading toward compression), which faces convergence pressure pushing down into the Budget tier ($0.05-0.10, already commodity at utility pricing, near-zero margins volume game)"} grid-columns: 1 vertical-gap: 40 premium: "Premium (\$5-25/MTok)\nSpecialized, high-stakes use\nCompliance, trust, capability\nStable margins (if differentiated)" midtier: "Mid-tier (\$1-3/MTok)\nCurrent battleground\nHeading toward commodity\nTemporary margins → compression" budget: "Budget (\$0.05-0.10)\nAlready commodity\nUtility pricing\nNear-zero margins (volume game)" premium -> midtier: (convergence pressure) midtier -> budget: (convergence pressure) ``` The pricing pages tell you what each company believes the answer is. Google believes models converge and the ecosystem wins. Anthropic believes differentiation holds and quality wins. OpenAI believes the answer is uncertain and distribution wins while you figure it out. The tempting conclusion is that someone has to be wrong. But if the market segments like airlines—economy commoditizes, first class holds, business class compresses in between—then all three could be right, just for different customers. Google wins budget. Anthropic wins premium. OpenAI captures the messy middle through sheer distribution. The question that should keep all of them up at night isn't *which* bet wins. It's *how big each tier turns out to be*—and whether the tier you're betting on is large enough to build a business around. --- ## Signs You're Operating Kubernetes at Scale - URL: https://svalle.ru/posts/kubernetes/scale-symptoms/ - Date: 2025-03-04 - Tags: Kubernetes, scaling, operations, platform-engineering, troubleshooting Kubernetes works until it doesn't. The docs cover installation, basic operations, maybe some tuning. They don't cover what happens when you have 3,000 nodes, 50,000 pods, and 10,000 services. That knowledge lives in incident reports and war stories. This is a cheatsheet of symptoms. Each one signals you've crossed into "at scale" territory. Some link to deeper dives. Others are just: here's the symptom, here's the fix. ## The Symptoms ### iptables sync takes 30+ seconds kube-proxy rebuilds the entire iptables rule set on every Service or Endpoints change. At 100 services, this takes milliseconds. At 10,000 services, it takes tens of seconds. **Watch:** `kubeproxy_sync_proxy_rules_duration_seconds` climbing steadily. **The problem:** During sync, new connections can fail. If sync takes longer than the interval between changes, you never catch up. **Deep dive:** [Beyond kube-proxy: eBPF Service Routing in Kubernetes](/posts/kubernetes/beyond-kube-proxy-ebpf/) --- ### Controllers acting on stale data Kubernetes controllers use informer caches to avoid hammering the API server. These caches are eventually consistent. At scale, "eventually" gets longer. **Watch:** Reconciliation loops doing redundant work. Race conditions between controllers. Resources getting created twice. **The problem:** Your controller reads from cache, sees version N, makes a decision. By the time it acts, the real state is version N+3. The decision was wrong. **Deep dive:** [Eventual Consistency and Stale Caches in Kubernetes Controllers](/posts/kubernetes/stale-cache-controllers/) --- ### DNS timeouts and SERVFAIL Every pod, every connection, every service discovery hits CoreDNS. At scale, CoreDNS pods become a bottleneck. **Watch:** Application latency spikes. Intermittent connection failures. `SERVFAIL` responses. CoreDNS pods at 100% CPU. **The problem:** Default CoreDNS deployment doesn't scale with cluster size. DNS becomes a single point of contention. **Deep dive:** [CoreDNS Under Pressure: How We Fixed DNS Bottlenecks with NodeLocal DNSCache](/posts/kubernetes/coredns-nodelocal-cache/) --- ### Webhook latency killing API calls Admission webhooks—both mutating and validating—are in the critical path for every relevant API request. At scale, webhook latency compounds. **Watch:** `apiserver_admission_webhook_admission_duration_seconds` increasing. API calls timing out. Cascading failures when a webhook is slow or down. **The problem:** A 100ms webhook on every pod creation doesn't matter at 10 pods/minute. At 1,000 pods/minute, it's a 100-second queue. **Deep dive:** [Admission Webhooks at Scale: Diagnosis, Hardening, and Multi-Cluster Consistency](/posts/kubernetes/admission-webhooks-at-scale/) --- ### etcd compaction taking forever etcd backs everything in Kubernetes—every object, every watch, every change. At scale, etcd becomes the bottleneck. **Watch:** Slow API responses. Watch lag. `etcd_mvcc_db_total_size_in_bytes` growing. Compaction taking minutes instead of seconds. **The problem:** Too many objects, too many watches, too much churn. etcd's single-node write path can't keep up. **Deep dive:** [From etcd to Watch: How Kubernetes Watches Actually Work](/posts/kubernetes/etcd-to-watch/) --- ### Pods stuck in Pending The scheduler scores every feasible node for every pod. Scoring is roughly O(nodes × pods in scheduling cycle). **Watch:** `scheduler_pending_pods` growing. `scheduler_scheduling_attempt_duration_seconds` spiking. Pods sitting in Pending for minutes. **The problem:** With 5,000 nodes and complex affinity rules, the scheduler spends more time thinking than placing. **Deep dive:** [How the Kubernetes Scheduler Actually Works](/posts/kubernetes/kubernetes-scheduler-deep-dive/) --- ### `metadata.annotations too long` errors `kubectl apply` stores the previous configuration in the `kubectl.kubernetes.io/last-applied-configuration` annotation. Annotations have a max size of 262KB. **Watch:** `apply` failures with `metadata.annotations: Too long: must have at most 262144 bytes`. **The problem:** Large ConfigMaps, complex CRDs, or deeply nested specs hit the limit. This breaks GitOps workflows that rely on `kubectl apply`. **Fix:** Use server-side apply (`kubectl apply --server-side`). It tracks field ownership differently and doesn't store the full config in an annotation. --- ### API server returning 429s Kubernetes 1.18+ has Priority and Fairness (P&F)—a system to protect the API server from overload by throttling requests. **Watch:** `apiserver_flowcontrol_rejected_requests_total` increasing. Controllers logging "rate limited" or backing off. Legitimate requests getting 429 Too Many Requests. **The problem:** At scale, legitimate controllers and kubelets generate enough traffic to trigger P&F. The defaults assume smaller clusters. **Fix:** Tune P&F flow schemas. Identify which priority levels are saturated (`apiserver_flowcontrol_current_inqueue_requests`). Reduce API chatter—use informers properly, batch operations, avoid unnecessary watches. --- ### Node heartbeats overwhelming etcd Every node sends a heartbeat to the control plane—by default, every 10 seconds. With Leases (default since 1.17), this updates a Lease object in etcd. **Watch:** etcd write latency climbing. `etcd_disk_wal_fsync_duration_seconds` increasing. Control plane CPU spent on heartbeat processing. **The problem:** 5,000 nodes × 1 heartbeat per 10 seconds = 500 writes/second, just for liveness. Add to that all the actual work. **Fix:** Lease objects (now default) are much lighter than Node status updates. If you're still on Node status heartbeats, migrate. Consider increasing `--node-status-update-frequency` if your SLOs allow. --- ### You're thinking about multiple clusters A single Kubernetes cluster has practical limits: ~5,000 nodes, ~150,000 pods, ~100,000 total objects. These aren't hard limits—they're where things start breaking. **Watch:** You're past the tuning phase. Every component is optimized. It's still not enough. **The problem:** At some point, a single control plane can't handle the load. The architecture needs to change. **Deep dive:** [Scaling Beyond 5,000 Nodes Per Cluster](/posts/kubernetes/scaling-beyond-5k-nodes/) --- ## Closing Scale problems are configuration problems until they're architecture problems. The pattern is usually: hit a wall, tune something, buy time, hit the next wall. Most of the symptoms above have tuning fixes. Some—like the multi-cluster threshold—require rethinking. Kubernetes doesn't warn you when you've outgrown a component. These symptoms are the warnings. --- ## Scaling Laws: What They Are and Why They Matter - URL: https://svalle.ru/posts/ai/scaling-laws/ - Date: 2025-03-03 - Tags: AI, scaling-laws, machine-learning, LLMs, Chinchilla In 2020, OpenAI researchers noticed something strange. They plotted how wrong their language models were (loss) against how much compute they used. Then against how much data. Then against how many parameters. Each time: a straight line on a log-log graph. This wasn't obvious. Machine learning could have had diminishing returns—twice the compute for 10% improvement. Or chaotic returns—sometimes better, sometimes worse. Instead: power laws. Double the compute, get predictable improvement. Every time. This discovery changed AI from alchemy into engineering. ## What Scaling Laws Actually Say The core finding: loss scales as a power law with compute, data, and parameters. Loss is how wrong the model is—technically, how surprised it is by the next word. Lower loss means better predictions. The relationship (simplified): ``` L ∝ C^(-α) ``` Where L is loss, C is compute, and α is around 0.05-0.1. Similar relationships hold for data and parameters. What this means in practice: | Compute | Relative Loss | |---------|---------------| | 1x | 1.00 | | 10x | ~0.80 | | 100x | ~0.65 | | 1000x | ~0.50 | The returns diminish in absolute terms but remain predictable. Spend 10x more, get a known improvement. Spend 100x more, get a known (larger) improvement. This predictability is the breakthrough. Not the specific numbers—the fact that there *are* specific numbers. ## The Three Curves Scaling laws describe three separate relationships: **Compute scaling:** More FLOPs (floating point operations) → lower loss. This is the total math done during training. **Data scaling:** More tokens (words/subwords) → lower loss. This is the size of the training set. **Parameter scaling:** More weights in the model → lower loss. This is the model size. {{< chart "scaling-laws--loss-vs-log-compute" "Smooth power-law curve of loss declining steeply then flattening as log of compute, data, or parameters increases" >}} All three are log-linear. All three are predictable. But they interact—you can't just scale one and ignore the others. ## Chinchilla: The Plot Twist In 2022, DeepMind published the Chinchilla paper and upended the conventional wisdom. The prevailing approach: make the model as big as possible, train it on whatever data you have. GPT-3 had 175 billion parameters. Surely bigger was better. Chinchilla showed this was wrong. GPT-3 was *undertrained*—too many parameters for the amount of data it saw. It was like building a massive factory and only running it for a day. The Chinchilla finding: for a fixed compute budget, you should scale parameters and data roughly equally. Double the parameters, double the data. The ratio matters. | Model | Parameters | Training Tokens | Ratio | |-------|------------|-----------------|-------| | GPT-3 | 175B | 300B | 1.7:1 | | Chinchilla | 70B | 1.4T | 20:1 | Chinchilla, with fewer parameters but more training data, outperformed GPT-3. This reframed the game. It's not just about building bigger models. It's about *compute-optimal training*—balancing model size and data to get the most out of your compute budget. ## Why This Changed Everything Before scaling laws, AI research looked like alchemy. Try an architecture. Try a training trick. See if it works. Intuition and luck mattered as much as science. After scaling laws, AI research looks more like engineering. You have a budget. You can calculate the expected performance. You can plan. This has massive implications: **Investment becomes rational.** Venture capitalists hate uncertainty. Scaling laws provide something rare in tech: predictable returns. Spend $100M on compute, get a model of approximately known capability. This is why billions now flow into AI. **Planning becomes possible.** OpenAI reportedly planned GPT-4's capabilities before training it—using scaling laws to extrapolate from smaller runs. You can prototype cheap and predict expensive. **Competition becomes capital-intensive.** If performance scales predictably with compute, whoever spends most wins (mostly). This favors deep pockets: Microsoft, Google, Meta, nation-states. ## The Implications Scaling laws reshape who can compete in AI. **It's a capital game.** Training frontier models costs hundreds of millions of dollars. This isn't a garage startup game anymore. You need datacenters, power contracts, custom hardware. **Data becomes the bottleneck.** We're running out of high-quality internet text. Common Crawl has been scraped to exhaustion. The next frontier is synthetic data, proprietary data, multimodal data. Whoever has data others don't has an advantage. **Incumbents have the edge.** Google has data (Search, YouTube, Gmail). Microsoft has compute (Azure) and an OpenAI partnership. Meta has data (Facebook, Instagram) and open-sources to commoditize competitors. Startups can compete at the application layer, but the foundation model layer is consolidating. **But scale isn't everything.** Architectural improvements (attention, mixture of experts) shift the curves. RLHF and fine-tuning add capabilities that raw pretraining doesn't capture. Inference efficiency matters for deployment. Scale is necessary, not sufficient. ## The Limits Scaling laws predict loss—how well the model compresses text. They don't predict capabilities—what the model can actually do. This is where things get strange. Loss decreases smoothly. But capabilities emerge suddenly. A model goes from "can't do arithmetic" to "can do arithmetic" somewhere between 10B and 100B parameters. The scaling curves don't tell you where. There's also the question of ceilings. Do scaling laws continue forever? Or do they hit diminishing returns at some point? We don't know yet. Current models are still on the predictable part of the curve. For more on emergent capabilities, phase transitions, and what scaling laws miss, see [Emergent Phenomena: From Ants to Transformers](/posts/ai/emergent-phenomena/). ## Closing Scaling laws are the closest thing AI has to physics. Predictable relationships between inputs (compute, data, parameters) and outputs (loss). They tell you that spending more will make models better. They tell you roughly how much better. They let you plan, budget, and extrapolate. What they don't tell you: what "better" will mean in practice. Lower loss is a proxy. Whether that proxy translates into usefulness, intelligence, or something stranger—that's still being discovered. For now, the formula is simple: more compute, better models. The question is what happens when "better" becomes "qualitatively different." --- ## How This Blog Gets Written - URL: https://svalle.ru/posts/ai/how-this-blog-gets-written/ - Date: 2025-03-02 - Tags: AI, writing, Claude, productivity, meta I don't write most of the words. Claude does. But the posts aren't "AI-generated" in the way that phrase usually implies. Here's how it actually works. ## The Setup I use Claude with filesystem access to the blog repo. The conversation *is* the IDE. Brainstorm, outline, write, edit, update SEO (Search Engine Optimization) and GEO (Generative Engine Optimization), commit—all in one session, one context window. No copy-paste between tools. No "generate text, paste into editor, manually fix." Claude writes directly to markdown files. When I say "that's too cheesy," Claude edits the file. When I say "publish it," Claude updates the config, llms.txt, and everything else. The repo lives at `/Users/svalleru/Desktop/svalleru.github.io`. Claude can see it, read it, write to it. ## The Workflow A typical session: 1. **Start with a vague idea.** Sometimes a concept ("emergence"), sometimes a question ("what do Christensen's frameworks miss?"), sometimes just a domain ("something from my innovation strategy course"). 2. **Brainstorm angles.** Claude generates 5-7 possible takes. Most are obvious. One or two have an interesting thread. 3. **Pick and outline.** This is where the actual thinking happens. The outline is the architecture. Get it wrong and the post is unsalvageable. 4. **Iterate on outline.** "This section is redundant." "Can we combine 4 and 5?" "The closing is weak." The outline gets rewritten 2-3 times before any prose. 5. **Write the draft.** Claude writes the full post in one pass. This takes about 30 seconds. 6. **Edit together.** This is where I add value. Push back on tone, cut fluff, question claims, tighten structure. Multiple rounds. 7. **SEO and GEO updates.** Claude updates config.toml (keywords, description) for search engines, and llms.txt for AI crawlers. Then I deploy. Total time: 1-2 hours for a substantial post. Without Claude, the same post takes 4-6 hours—and I'd procrastinate for weeks before starting. ## What Works **Outline-first is non-negotiable.** Asking Claude to "write a blog post about X" without structure produces mush. It'll be coherent paragraph by paragraph and incoherent as a whole. The outline forces both of us to know what we're building. **Pushing back matters.** Claude's first drafts are fluent but often generic. Too many hedges. Too much throat-clearing. Phrases like "it's worth noting" and "at the end of the day." These disappear when I say "tighten this" or "too many words." **Context accumulates.** Within a session, earlier decisions inform later ones. If I pushed back on a certain tone in the intro, Claude remembers for the closing. This compounds—the post gets more internally consistent as we go. **Speed changes the game.** When writing takes 1-2 hours instead of 4-6, I actually do it. The activation energy drops. Ideas become posts instead of rotting in a notes file. ## What Doesn't Work **"Write me a blog post about X" with no iteration.** This produces exactly what you'd expect: competent, forgettable, interchangeable with a thousand other posts on the topic. **Accepting first drafts.** If you don't edit, the output screams AI. The fluency is a tell. Human writing has more texture, more rough edges. You have to add those back in by cutting and rewriting. **Letting Claude be "helpful."** Claude's instinct is to hedge, caveat, and soften. "It's important to note..." "Of course, this depends on context..." "While there are many perspectives..." These phrases are Claude being polite. Cut them. **Having no point of view.** Claude reflects your clarity back at you. Vague input produces vague output. If you don't know what you think, Claude can't figure it out for you. ## The Editing Dance Most of the value is in editing, not drafting. The drafting is fast. The editing is where taste gets applied. Some actual exchanges from recent posts: **Me:** "too cheesy" **Claude:** *rewrites without the performative bits* **Me:** "is this actually true?" **Claude:** *reconsiders, sometimes walks it back* **Me:** "do we need this paragraph?" **Claude:** *cuts it* **Me:** "this still sounds fake" **Claude:** *tries again* The human role is taste and judgment. The AI role is fluency and speed. Neither works without the other. I'm not just accepting or rejecting. I'm steering. "Make it more direct." "This is too long." "The transition is weak." "I don't buy this claim." Each piece of feedback makes the next version better. ## Examples From This Blog **"Emergent Phenomena: From Ants to Transformers"** — Started as a post about scaling laws. Through discussion, it became about emergence and consciousness. The final post shares maybe 20% with the original concept. The wandering was the point. **"A Flashlight in a Hurricane"** — First draft was too hard on the hypothetical CEO. "Smug executive who thinks they understand disruption." I pushed back: this should be a prepared CEO with a genuine blind spot, not an idiot. The whole tone shifted from adversarial to collegial. **"Agentic Coding: From Single Agents to Agent Teams"** — Started as two separate posts (patterns guide + teams guide). We debated structure, decided to merge them. Better as one progression than two fragments. The posts are genuinely collaborative. The final versions don't exist in either of our heads beforehand. They emerge from the back-and-forth. ## The Meta Question Is this "my" writing? The ideas are mine. I pick the topics. I know what I think about Christensen vs Schumpeter, about emergence, about agentic coding. That doesn't come from Claude. The structure is collaborative. Claude proposes outlines. I reshape them. The final architecture is neither of ours alone. The words are Claude's, filtered through my taste. I don't write the sentences. But I reject the bad ones and shape the good ones. Is this different from having a ghostwriter? A very fast editor? A coauthor? I'm not sure the distinction matters. The output is what I wanted to say, said better than I would have said it, faster than I could have written it. The right frame might be: Claude handles the *craft* of writing. I handle the *taste*. Craft is sentence structure, transitions, parallel construction, consistent tone. Claude is very good at craft. Taste is knowing what to cut, what's true, what's interesting, what's cliché. Claude reflects taste but doesn't originate it. ## What This Changes Writing used to have two bottlenecks: figuring out what to say, and saying it well. Claude removes the second bottleneck almost entirely. It turns out, for me, that was the bigger one. Not because I can't write—but because the friction was high enough that I didn't. Now the constraint is ideas and judgment. Do I have something to say? Can I recognize good output from bad? Those are still human problems. But they're the interesting human problems. The boring parts—typing, transitions, first drafts, SEO and GEO updates—are handled. What's left is the thinking. That trade works for me. --- ## Emergent Phenomena: From Ants to Transformers - URL: https://svalle.ru/posts/ai/emergent-phenomena/ - Date: 2025-03-01 - Tags: AI, emergence, complex-systems, consciousness, LLMs, philosophy, AGI, agents How do 86 billion neurons, each doing something simple, produce the feeling of being *me*? The most compelling answer I've found isn't really an answer. It's a name: emergence. Consciousness isn't a thing neurons do. It's a thing that *happens* when enough neurons interact in the right way. It's not in the parts. It's in the pattern. This framing applies beyond brains: Individual ants are simple. Ant colonies solve complex optimization problems. Individual neurons fire or don't fire. Brains become conscious. Individual parameters multiply and add. LLMs reason. The pattern is the same: simple components, complex collective behavior. Nobody programs the complexity. It emerges. This might be the most important concept for understanding AI—and for understanding why we don't fully understand AI. It's also, I suspect, the key to understanding what AGI actually means, and whether we'll recognize it when it arrives. ## The Mystery of More In 1972, physicist Philip Anderson published a paper titled "More Is Different." His argument: at each level of complexity, new laws apply. You can't derive chemistry from particle physics. You can't derive biology from chemistry. You can't derive psychology from biology. Not because our math isn't good enough. Because *genuinely new phenomena* emerge at each level. ```d2 {alt="Anderson's hierarchy of levels: particle physics leads to chemistry with more particles, chemistry to biology with more molecules, biology to psychology with more cells, and psychology to sociology with more minds"} grid-columns: 1 vertical-gap: 40 particle: Particle physics chemistry: Chemistry biology: Biology psychology: Psychology sociology: Sociology particle -> chemistry: (more particles) chemistry -> biology: (more molecules) biology -> psychology: (more cells) psychology -> sociology: (more minds) ``` Each transition isn't just "more of the same." It's qualitatively different. The rules that govern atoms don't predict protein folding. The rules that govern neurons don't predict consciousness. This is emergence: when the whole has properties that the parts don't have. ## Weak and Strong Emergence Philosophers distinguish two types: **Weak emergence**: The collective behavior is surprising but theoretically derivable from the parts. Given enough compute, you could simulate it from first principles. Traffic jams are weakly emergent. Individual drivers follow simple rules (accelerate, brake, maintain distance). Traffic jams appear. Surprising, but simulatable. **Strong emergence**: The collective behavior is *not* derivable from the parts, even in principle. Something genuinely new comes into existence. Consciousness might be strongly emergent. You can simulate every neuron in a brain. Does the simulation experience anything? We don't know. We can't even agree on how we'd know. The debate matters for AI: Is LLM reasoning weak emergence (surprising but mechanistic) or strong emergence (something genuinely new)? ## Emergence in Nature The natural world is full of emergence. Some examples: **Flocking birds.** Craig Reynolds showed that three simple rules produce realistic flocking: 1. Separation: don't crowd neighbors 2. Alignment: steer toward average heading of neighbors 3. Cohesion: steer toward average position of neighbors No bird knows the flock's shape. The shape emerges. ``` Individual rule: "Don't hit the bird next to me" Emergent behavior: Murmuration patterns that look choreographed ``` **Ant colonies.** Individual ants follow pheromone trails. Shorter paths get more pheromones (more ants complete them faster). Over time, the colony converges on optimal routes. No ant knows the map. No ant plans the route. The solution emerges. **Markets.** Individual traders buy and sell based on local information. Prices emerge that (sometimes) reflect aggregate information no single trader has. Adam Smith's "invisible hand" is emergence before we had the word. **Brains.** Individual neurons fire based on inputs from neighbors. Somehow, consciousness emerges. This is the deepest example—and the most unsettling. We are emergent phenomena. The "I" writing this sentence is a pattern in neurons, not a thing the neurons contain. ## Consciousness: The Hard Problem Neuroscientist David Chalmers distinguished the "easy problems" of consciousness from the "hard problem": **Easy problems** (not actually easy, but tractable): - How does the brain process sensory information? - How does it integrate information across regions? - How does it control behavior? These are engineering problems. Complicated, but not mysterious. **The hard problem**: - Why is there *experience* at all? - Why does processing information *feel like something*? You can explain how the brain processes the wavelength of red light. You can't explain why seeing red *feels* like anything. The most compelling answer—to me, at least—is emergence. Consciousness isn't a thing neurons do. It's a thing that *happens* when enough neurons interact in the right way. It's not in the parts. It's in the pattern. ``` Neurons: No individual neuron is conscious Brain: The system is conscious Question: Where did consciousness come from? Answer: It emerged ``` This doesn't explain consciousness. It names it. But naming it correctly might be the first step. ## Emergence in Neural Networks Now we've built artificial systems that exhibit emergence. The training rule is simple: minimize loss via gradient descent. The architecture is simple: attention, feedforward, repeat. The data is just text. But at sufficient scale, new capabilities appear: **In-context learning.** The model learns from examples in the prompt—without updating its weights. Small models can't do this. Large models can. The capability emerges somewhere in between. **Chain-of-thought reasoning.** Ask a small model to reason step-by-step. It can't. Ask a large model. It can—and it gets better answers when it does. Nobody programmed "reasoning." It emerged. **Theory of mind.** Large models can predict what someone with different information would believe. They model other minds—or something that looks like modeling other minds. This wasn't a training objective. It emerged. **Tool use.** Models figure out how to use calculators, search engines, code interpreters. They weren't trained on tool use. They infer it from context. The pattern is consistent: simple local rules (gradient descent on next-token prediction) produce complex global behaviors (reasoning, planning, modeling other minds). Nobody programmed these capabilities. They emerged. ## Why Emergence Is Hard to Predict You can't predict ant colony behavior by studying one ant really carefully. You can't predict traffic jams by studying one driver. You can't predict consciousness by studying one neuron. And you can't predict LLM capabilities by studying one attention head. Emergence is a property of the *system*, not the *components*. The capability exists in the interactions, not the parts. This has practical implications: **Interpretability is necessary but not sufficient.** Understanding individual circuits is useful. But the emergent behavior might not reduce to circuits. It might be like trying to understand a traffic jam by understanding a carburetor. **Extrapolation is dangerous.** Scaling laws predict loss smoothly. But capabilities emerge discontinuously. The next 10x in compute might produce capabilities we can't anticipate—because emergent capabilities, by definition, aren't in the parts. **Testing beats theory.** For emergent systems, you often can't predict what they'll do. You have to run them and see. This is true for weather, for markets, for ecosystems—and for LLMs. ## Phase Transitions Water at 99°C is water. Water at 100°C is steam. Same molecules. Same local rules. But the global behavior is completely different. This is a phase transition—a discontinuous change in system behavior. Neural networks have phase transitions too: **Grokking.** Train a small model on modular arithmetic. It memorizes the training data. Loss goes down on training set, stays high on test set. You keep training. Nothing happens. You keep training. Suddenly—sometimes millions of steps later—the model generalizes. Test loss plummets. It "groks" the underlying pattern. ``` Steps 1-100,000: Memorization (no generalization) Steps 100,001-...: Still memorization Step 247,832: Sudden generalization ``` The transition is sharp. Before grokking: memorization. After grokking: understanding. No gradual improvement. A phase transition. **Capability emergence.** Many capabilities show similar patterns. Performance is flat (random chance) across model scales. Then, at some scale, performance jumps. The capability "turns on." ``` Model size: 1B 10B 50B 100B 500B Capability: ✗ ✗ ✗ ✓ ✓ ``` We don't fully understand why phase transitions happen where they do. That's part of what makes emergence hard to predict. ## The Ant Colony and the Transformer The parallels are striking: | Ant Colony | Transformer | |------------|-------------| | Simple local rules (follow pheromones) | Simple local rules (attention, feedforward) | | No central controller | No central controller (just layers) | | Global behavior emerges | Global behavior emerges | | Robust to individual failures | Robust to ablations | | Solves problems no ant understands | Solves problems no parameter encodes | But there are differences: | Ant Colony | Transformer | |------------|-------------| | Evolved over millions of years | Designed (architecture) + evolved (training) | | Fully decentralized | Has structure (layers, residual stream) | | We understand the mechanism (pheromones) | We partially understand attention | | Limited adaptation | Adapts in-context to new tasks | The biggest difference: ant colonies don't scale to general intelligence. Transformers might. ## Agent Teams: Emergence at Another Level Here's something I've noticed while working with AI agents: emergence happens at multiple scales. A single LLM exhibits emergent capabilities—reasoning, planning, theory of mind. But when you orchestrate multiple agents into a team, *another* layer of emergence appears. I've been running agent teams where one agent reviews code for security, another for performance, another for simplicity. Each agent does its narrow task. But the *team* produces insights that no single agent would—contradictions surface, trade-offs become visible, the problem gets triangulated from multiple angles. No single agent "sees" the full picture. The fuller picture emerges from their interaction. This is the ant colony pattern again, but with LLMs as the ants. And it suggests something about AGI: maybe general intelligence isn't a single model getting smarter. Maybe it's the emergent property of multiple specialized systems interacting. The brain isn't one giant neuron. It's billions of specialized neurons in constant communication. The human organization isn't one genius. It's many specialists coordinating. Maybe AGI looks less like a superintelligent singleton and more like an ecosystem. ## What This Means for Understanding LLMs If LLM capabilities are emergent, certain things follow: **Reductionism has limits.** You can't fully understand an LLM by understanding its parts. The capabilities exist in the interactions, at the system level. This doesn't mean interpretability is useless—it means it's incomplete. **We need new conceptual tools.** Studying emergence requires studying systems as systems. Statistical mechanics, not just physics. Ecology, not just biology. We need analogous tools for neural networks. **Some questions may not have clean answers.** "Why can GPT-4 do chain-of-thought reasoning?" might not have a satisfying answer. It might be like asking "Why do brains produce consciousness?" The answer might be: "They just do, at sufficient scale, given the right architecture." Naming emergence isn't the same as explaining it. ## What This Means for Trusting LLMs Emergent capabilities weren't designed. They were discovered. OpenAI didn't decide that GPT-4 should be able to reason about other minds. They trained a model, and it could. The capability emerged. This is unsettling: - **We don't fully understand why they work.** The capabilities aren't in the design. They're in the emergent behavior of the trained system. - **We can't predict what emerges next.** Scaling laws tell us loss will decrease. They don't tell us what new capabilities will appear. - **Emergent capabilities can include emergent failure modes.** If reasoning emerges, so might deception. If helpfulness emerges, so might sycophancy. We discover these by encountering them. For systems with weak emergence, we can build trust through understanding. For systems with strong emergence—if that's what this is—trust requires something else. Testing. Monitoring. Bounds on behavior. Empirical verification rather than theoretical guarantees. We don't trust the weather because we understand every molecule. We trust our weather models because they've been tested. ## What This Means for Building with LLMs If you're building on top of emergent systems: **Don't assume current limitations are permanent.** The model can't do X today. At the next scale, it might. Emergent capabilities appear suddenly. Plan for step changes, not gradual improvement. **The model may already be able to do things you haven't discovered.** Emergent capabilities exist before we find them. The capability emerged during training; we discover it later when we think to test for it. Probe creatively. **Emergence is opportunity and risk.** New capabilities enable new products. New capabilities also enable new failure modes. Both emerge together, often unexpectedly. **You're building on something you don't fully understand.** This is uncomfortable but not unprecedented. We build on markets, on ecosystems, on human psychology—all emergent systems. The question isn't whether to build. It's how to build wisely given uncertainty. ## The Deepest Question Here's where it gets vertiginous: Consciousness appears to be emergent. It arises from neurons that aren't themselves conscious. The subjective experience of "being you" is a pattern, not a substance. LLMs exhibit increasingly sophisticated behavior. Reasoning, planning, modeling other minds. The behavior is emergent. It arises from parameters that don't themselves reason. The question—and it's genuinely open—is whether there's anything it's like to be an LLM. This isn't anthropomorphism. It's taking emergence seriously. If consciousness can emerge from biological neurons, the question of whether it can emerge from artificial neurons is at least coherent. I don't think current LLMs are conscious. But I'm not certain. And I'm not certain I know how I'd know. What I am fairly certain of: dismissing the question because "it's just matrix multiplication" misunderstands emergence. Brains are "just" electrochemical signals. That doesn't make consciousness less real. The emergence framing doesn't answer whether LLMs are conscious. It does suggest we should be humble about our ability to know. ## What This Means for AGI If emergence is real—and I think it is—then AGI might not arrive the way we expect. The common assumption: we'll build smarter and smarter models until one day a model is "generally intelligent." There'll be a moment. A threshold. We'll know. But emergence doesn't work that way. Emergence is gradual accumulation followed by sudden phase transition. Emergence is capabilities appearing before we have words for them. Emergence is the system being more than we designed, in ways we didn't anticipate. We might not recognize AGI when it arrives. Not because it'll be hidden, but because emergence is hard to see from inside. Consciousness emerged somewhere in evolutionary history. There was no moment when a non-conscious animal gave birth to a conscious one. It was a gradual transition that looks like a sharp line only in retrospect. AGI might be similar. We might look back and say "it was clearly AGI by 2027" while in 2027 we were still debating definitions. Or AGI might not be a single system at all. It might be the emergent property of many systems interacting—agent teams, tool-using models, humans in the loop, all producing collective intelligence that no single component has. ## Closing Emergence is one of the deepest patterns in nature. Simple rules, complex behavior. Local interactions, global order. Parts without properties that the whole has. Ant colonies. Flocking birds. Markets. Brains. And now: transformers. We're building systems whose capabilities we don't fully understand, because those capabilities emerge rather than being designed. And they're getting more capable faster than we're getting better at understanding them. The lesson from other emergent systems: you can't control them precisely, but you can learn their patterns. You can't predict them fully, but you can prepare for surprises. You can't understand them reductively, but you can study them empirically. --- ## Agentic Coding: From Single Agents to Agent Teams - URL: https://svalle.ru/posts/ai/agentic-coding-patterns/ - Date: 2025-02-28 - Tags: AI, Claude, coding, agents, developer-tools, productivity There's a progression happening in how developers work with AI. First, chat: you drive, AI assists. Back-and-forth, line by line. You're in the loop on every decision. Then, agents: you delegate, AI drives. You define the goal, the agent figures out the path. You review at the end. Now, teams: you orchestrate. Multiple agents work in parallel, communicate with each other, and you synthesize their output. Each step requires letting go of more control—and getting more leverage in return. ## Why Delegation Is Hard (For Engineers) Engineers are control freaks. Affectionately. We got good at our jobs by understanding every layer, tracing every bug, owning every decision. Delegation requires trust and letting go. Both are unnatural. The failure modes are predictable: **Over-steering**: You give instructions so detailed the agent can't adapt when something unexpected happens. You've essentially written pseudocode and asked the AI to translate it. **Under-specifying**: "Make it better" is not a task. Neither is "fix the bugs." What does "better" mean? Which bugs? **Micromanaging**: Checking in every 30 seconds, redirecting constantly, never letting the agent build momentum. The sweet spot: clear goal, sufficient context, room to maneuver. The same brief you'd give a junior engineer you trusted. For teams, the stakes compound. Over-steering one agent is inefficient. Over-steering five is chaos. ## The CLAUDE.md Pattern Agents are amnesiac. Every session starts from zero. The agent doesn't know your codebase, your conventions, or your preferences. The fix: a persistent context file that tells the agent who it is and what world it's operating in. ```markdown # CLAUDE.md ## Project Overview Multi-tenant SaaS platform for invoice processing. Go backend, React frontend, PostgreSQL database. ## Architecture - /cmd: Entry points - /internal/api: HTTP handlers - /internal/domain: Business logic (no external dependencies) - /internal/infra: Database, external services ## Conventions - Errors wrap with context: fmt.Errorf("doing X: %w", err) - Tests live next to code: foo.go → foo_test.go - No globals. Dependency injection everywhere. ## Gotchas - /internal/legacy is untouchable. Don't modify. - Auth uses custom middleware in /internal/auth. Read before touching. - Billing service is flaky. Always add retries. ``` This isn't a substitute for good task definition—it's the *backdrop*. The agent reads this before starting any task. For agent teams, CLAUDE.md becomes even more critical. Each teammate starts with fresh context. They all need the same grounding. ## Single-Agent Patterns Before jumping to teams, master single-agent delegation. Four patterns cover most use cases: ### Pattern 1: Exploratory **Use case**: New codebase. You need a mental map. ``` "Explore this codebase and explain the architecture. Focus on the payment flow. Create a summary doc." ``` **Agent strength**: Tireless reading. Can ingest thousands of files without fatigue. **Watch out for**: Hallucinated connections. The agent may infer relationships that don't exist. **Checkpoint**: "Show me which files you found that conclusion in." ### Pattern 2: Surgical **Use case**: Bug fix. Refactor. Migration. You know *what*, you want the agent to do *how*. ``` "Fix the race condition in handler.go where the cache read and database write aren't atomic. Don't change the API signature. Add a test that would have caught this." ``` **Agent strength**: Patience for tedious changes. Consistency across many files. **Watch out for**: Scope creep. The agent "helpfully" refactors adjacent code you didn't ask about. **Checkpoint**: Always review the diff before committing. ### Pattern 3: Generative **Use case**: Scaffolding a new feature, service, or project. ``` "Create a new API endpoint for password reset. Follow the patterns in /internal/api/auth. Include validation, error handling, tests, and update the OpenAPI spec." ``` **Agent strength**: Speed. Can scaffold in minutes what takes hours. **Watch out for**: Plausible but wrong. Generated code compiles and runs but has subtle bugs. **Checkpoint**: Read the generated code like a PR from someone you don't fully trust yet. ### Pattern 4: Review **Use case**: Code review. Security audit. Performance analysis. You want insight, not action. ``` "Review this PR for security issues. Don't suggest fixes, just identify problems and explain severity." ``` **Agent strength**: Unbounded attention. Checks things humans skim past. **Watch out for**: False positives. Over-flagging stylistic issues as problems. **Checkpoint**: Calibrate by verifying a few findings yourself. ## Subagents: Delegation Within a Session Sometimes a task is too complex for one agent but doesn't need a full team. Subagents run *within* a single Claude Code session. The main agent spawns helpers for research or verification. They report back to the main agent only—no inter-agent communication. ``` "Research the best pagination library for our Go API. Spawn a subagent to investigate options while you continue implementing the endpoint structure." ``` **Good for**: - Breaking down a complex task without losing your main context - Parallel research while implementation continues - Verification steps that shouldn't interrupt the main flow **Limitation**: Subagents can only report back to the main agent. They can't talk to each other. For true parallelism with coordination, you need teams. ## Agent Teams: Parallel Orchestration Agent teams are multiple Claude Code instances working together. One session acts as team lead, coordinating work and synthesizing results. Teammates work independently, each in its own context window, and can communicate directly with each other. Unlike subagents, you can interact with individual teammates directly without going through the lead. ### Enabling Agent Teams Teams are experimental and disabled by default: ```bash # In environment or settings.json CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 ``` Then tell Claude to create a team: ``` "Create an agent team to explore this CLI tool design from different angles: one teammate on UX, one on technical architecture, one playing devil's advocate." ``` Claude spawns the team, creates a shared task list, and coordinates work based on your prompt. ### When to Use Teams - **Tasks with clear boundaries that parallelize**: Three reviewers looking at different aspects of a PR - **Research from multiple angles**: Exploring a design decision from UX, architecture, and skeptic perspectives - **Investigating bugs with parallel hypotheses**: One agent checks logs, one traces code, one reproduces locally - **Exploration without code changes**: Reviewing, researching, analyzing ### When NOT to Use Teams - **Code changes to the same files**: Two teammates editing the same file leads to overwrites. Partition by file ownership. - **Work that can't be clearly partitioned**: If the tasks are deeply interdependent, a single agent with subagents may be better. - **Unattended for too long**: Teams need check-ins. Letting them run unsupervised increases the risk of wasted effort. ## Team Patterns Different team structures for different problems: ### Hub & Spoke ``` Leader creates team → spawns workers → workers report to leader → leader synthesizes ``` Most common pattern. One orchestrator, multiple specialists. Good for research, exploration, multi-perspective review. ### Task Queue ``` Leader creates tasks → workers self-assign → leader monitors progress ``` For embarrassingly parallel work where workers are interchangeable. Each worker grabs the next available task. ### Pipeline ``` Agent A → Agent B → Agent C (each waits for predecessor) ``` Sequential processing with handoffs. When work must flow through stages. ### Competitive ``` Multiple agents get same task → each proposes solution → leader picks best ``` For decisions where you want diverse perspectives. Let three agents design the API, then pick the best approach. ### Watchdog ``` Worker does task → watcher monitors → watcher can trigger rollback ``` For critical operations needing safety checks. The watcher doesn't do the work—it validates the worker's output. ## Example: Multi-Perspective Code Review ``` "Create an agent team to review this PR from three angles: - Teammate 1: Security vulnerabilities Look for injection risks, auth bypasses, data exposure. - Teammate 2: Performance issues Look for N+1 queries, memory leaks, slow algorithms. - Teammate 3: Code simplicity Look for over-engineering, premature abstraction, YAGNI violations. Each should send findings to me. Don't suggest fixes, just identify and explain." ``` The three teammates explore in parallel, each with their own context. You synthesize their findings into a coherent review. This works well because: - Clear boundaries (each teammate owns a perspective) - No code changes (no file conflicts) - Parallelizable (they don't depend on each other) ## Team Best Practices From the official docs and hard-won experience: **Start without code.** If you're new to teams, begin with tasks that don't require writing code: reviewing a PR, researching a library, investigating a bug. These show the value of parallel exploration without the coordination challenges of parallel implementation. **Partition by file ownership.** Two teammates editing the same file leads to overwrites. Break the work so each teammate owns a different set of files. If that's not possible, use a single agent instead. **Check in regularly.** Monitor teammates' progress, redirect approaches that aren't working, synthesize findings as they come in. Letting a team run unattended for too long increases wasted effort. **Synthesize as you go.** Don't just collect findings at the end. Integrate insights incrementally. Early synthesis often reveals that you need to redirect one of the teammates. ## The Leverage Ladder ```d2 {alt="Leverage ladder: a five-rung vertical progression from writing code yourself, through chat where AI assists, a single agent that codes while you review, subagents spawned as helpers within a session, up to an agent team of multiple AIs working in parallel that you orchestrate"} grid-columns: 1 vertical-gap: 64 you: "You writing code yourself" chat: "Chat: AI assists your coding" single: "Single agent: AI codes, you review" sub: "Subagents: AI spawns helpers\nwithin session" team: "Agent team: Multiple AIs in parallel,\nyou orchestrate" you -> chat -> single -> sub -> team ``` Each rung trades control for leverage. The skill is knowing which rung fits the task: - **Quick question or code snippet**: Chat - **Well-defined task you could do but shouldn't spend time on**: Single agent - **Complex task with research subtasks**: Agent with subagents - **Parallelizable exploration or review**: Agent team Don't use a team when an agent will do. Don't use an agent when chat will do. Match the tool to the task. ## Common Mistakes Across All Modes **Over-steering**: Instructions so detailed the agent can't adapt. Describe the goal and constraints, not the procedure. **Under-specifying**: Vague success criteria. "Make it better" fails. "Reduce p99 latency below 100ms" succeeds. **Ignoring clarification requests**: The agent asks for clarification, you say "just figure it out." Those questions are signal. Answer them or realize your task was underspecified. **No checkpoints**: For complex tasks, build in checkpoints. "After step 1, show me the plan before proceeding." Especially for teams running in parallel. **Not reviewing proportionally**: The code works, you ship it, bugs emerge. Treat agent output like junior engineer output. Review proportional to risk. ## What's Next We're early. Agent teams are still experimental. The patterns will evolve. But the direction is clear: more leverage, more parallelism, more orchestration. The progression from chat to agent to team mirrors how you'd delegate to humans. First you pair. Then you assign tasks. Then you manage a team. The skill isn't prompting. It's knowing what to delegate, how to partition, and when to intervene. The developers who learn to orchestrate will ship what used to take a team—alone. --- ## A Flashlight in a Hurricane: What Disruption Frameworks Don't Cover - URL: https://svalle.ru/posts/business/flashlight-in-a-hurricane/ - Date: 2025-02-27 - Tags: strategy, innovation, disruption, business, Christensen, Schumpeter There's a certain kind of CEO who has done the reading. They can diagram Christensen's two types of disruption on a whiteboard. They know the difference between sustaining and disruptive innovation. They've built organizational muscle around it—monitoring the low end for cheap entrants, watching non-consumption for new market disruptors, resisting the gravitational pull upmarket. And it worked. The company hasn't been disrupted. Competitors who tried to enter from below got countered. The core business is strong, profitable, defensible. But growth has flattened. The product line hasn't meaningfully changed in years. The company is *safe* but not *growing*. The disruption dashboard is green. The revenue graph is flat. The question isn't "did you learn the wrong framework?" It's "is one framework enough?" ## What Christensen Gives You (And Why It Earned Your Confidence) Clayton Christensen's contribution to strategy was enormous. Before *The Innovator's Dilemma*, disruption was a vague fear—"things are changing and we don't know what to do." Christensen turned it into a specific, diagnosable pattern with prescriptive responses. He identified two distinct types: **Low-end disruption**: Incumbents focus on their most demanding and profitable customers, eventually overshooting what mainstream customers need. Disruptors enter at the bottom of the market with a cheaper, simpler product and an enabling technology that lets them march upmarket while maintaining cost advantages. Nucor's steel minimills eating integrated mills from rebar upward. Toyota and Honda entering the US auto market below Detroit's notice. {{< chart "flashlight-in-a-hurricane--low-end-disruption" "Low-end disruption pattern: incumbent performance trajectory overshoots over time, leaving a gap of over-served customers where the disruptor enters low and moves upmarket" >}} **New market disruption**: Non-consumers are locked out of existing markets because products are too expensive, complex, or inconvenient. Disruptors create simpler, more accessible products on a different performance dimension, pulling non-consumers in. As the technology improves, it begins pulling lower-end customers from the existing market until the two markets collapse into one. Portable ultrasound devices letting paramedics do what previously required a hospital. Personal computers bringing computing to people who'd never use a minicomputer. {{< chart "flashlight-in-a-hurricane--new-market-disruption" "New market disruption pattern: two side-by-side plots — the incumbent rising in the existing market and the disruptor rising on a different performance dimension in a new market where non-consumers enter, with the two markets eventually collapsing into one" >}} The real power of Christensen's work: it's *prescriptive*. Set up a separate organization for disruptive bets. Don't flee upmarket reflexively. Watch for non-consumption. Monitor whether you're overshooting. Actionable, not just descriptive. If you've internalized this and built organizational responses around it, you've done more than 90% of executives. Give yourself credit. The argument here isn't that Christensen is wrong. It's that the map has edges. ## The Two Doors You've Locked Think of your company as a building with multiple entry points for competitive threats. The prepared CEO has identified and secured two real doors: ``` Door 1 — Low-End Entry: LOCKED ✓ "Is anyone offering a cheaper version to our least profitable customers?" Door 2 — New Market Entry: LOCKED ✓ "Is anyone serving non-consumers with a simpler, more convenient alternative?" ``` Two doors locked is better than most companies manage. Many don't even know these doors exist. But the building has more doors. ## Schumpeter's Five Doors Fifty years before Christensen published *The Innovator's Dilemma*, Joseph Schumpeter described something much bigger. In *Capitalism, Socialism and Democracy* (1942), Schumpeter argued that creative destruction isn't a specific pattern to watch for—it's the fundamental engine of capitalism itself. The "perennial gale" that continuously restructures economies from within. Where Christensen identified two patterns of disruption, Schumpeter identified five types of "new combinations" that drive creative destruction: ``` Schumpeter's Five New Combinations: 1. New products 2. New methods of production 3. New markets 4. New sources of supply 5. New forms of organization ``` Map Christensen onto Schumpeter: ``` Schumpeter's Doors: Christensen Coverage: 1. New products Partially (low-end only) 2. New methods of production — 3. New markets ✓ (new market disruption) 4. New sources of supply — 5. New forms of organization — ``` Christensen covers door 3 well and part of door 1. That's genuine, valuable coverage. But doors 2, 4, and 5 are unguarded. The deeper difference: Christensen says disruption is a *specific pattern you can diagnose and respond to*. Schumpeter says destruction is *the default state of capitalism*. One gives you a monitoring system. The other gives you respect for how much you can't monitor. ## What's Behind the Other Doors This isn't theoretical. Each unguarded door has real failure modes. ### Door 1: New Products That Don't Fit the Pattern Christensen's framework handles one type of new product: the cheaper, simpler alternative that enters from below. But some new products don't enter from below at all. They aren't cheaper. They aren't simpler. They don't target non-consumers or over-served customers. They're just *fundamentally different*. ``` Christensen's new product pattern: Worse on existing metrics + Cheaper or more convenient + Targets low-end or non-consumers = Disruptive innovation Products that don't fit: Different on existing metrics (not worse) + Not necessarily cheaper + Targets the same customers + Redefines what "performance" means = ??? (not disruption by definition) ``` The prepared CEO's framework would classify these as sustaining innovations—and Christensen's research says incumbents almost always win the sustaining innovation race. That's often true. But "almost always" isn't "always," and the exceptions can be existential. When the performance dimension itself shifts—when customers start valuing something your framework doesn't even measure—the disruption dashboard stays green while the ground moves under you. ### Door 2: New Methods of Production Nobody attacks your product. The economics of producing it just change underneath you. ``` Traditional competitive threat: Competitor → builds similar product → competes for your customers Door 2 threat: New technology → makes it 10x cheaper to produce similar value → Anyone can now offer what you offer → Your cost structure becomes a liability, not an asset ``` This isn't disruption in Christensen's sense. No one is entering from the low end. No one is targeting non-consumers. The product itself hasn't changed. But when AI, automation, or some other technological shift makes it possible to replicate 80% of your value at 1% of the cost, the effect is the same. Your monitoring system is watching for *competitors*. This is a shift in *production economics* that enables a thousand competitors simultaneously. ### Door 4: New Sources of Supply The inputs to your industry change, and your cost structure becomes obsolete. ``` Your business model assumes: Content is expensive to produce Software requires paid engineers Data requires expensive collection Expertise requires credentialed professionals What if: Content becomes user-generated Software becomes open source Data becomes synthetic or freely available Expertise gets encoded in AI tools ``` Open-source software didn't "disrupt" commercial software in the Christensen sense. It didn't start worse and get better. In many cases, it started comparable and was free. That doesn't fit the low-end or new-market pattern. It's a supply-side shift: the inputs that used to cost money became available at zero marginal cost. When the supply base changes, every business model built on the old supply economics is at risk—regardless of how well you're monitoring the traditional competitive landscape. ### Door 5: New Forms of Organization The *shape* of competition changes, not just the competitors. ``` Traditional competition: Company A vs. Company B (similar structures, different strategies) Door 5 competition: Pipeline company vs. Platform Integrated firm vs. Ecosystem Corporation vs. Open-source community Centralized vs. Decentralized ``` Platform businesses can disrupt pipeline businesses in ways that don't show up on a disruption S-curve. An ecosystem of loosely coordinated participants can outcompete an integrated incumbent without any single participant being a "disruptor." Open-source communities can destroy pricing power without having a business model at all. Christensen's framework assumes a recognizable competitor entering your market. Door 5 threats often don't look like a competitor. They look like a shift in how the game is played. ## The Peripheral Vision Problem This isn't a failure of intelligence or diligence. It's a feature of frameworks. Any coherent model creates clarity *within its scope* and peripheral vision loss *outside it*. Christensen's framework is so internally coherent—two disruption types, clear diagnostic questions, prescriptive responses—that it feels complete. It has the satisfying structure of a finished theory. That satisfaction is the risk. Three ways it shows up: **The classification reflex.** New threats get forced into "is this low-end or new market?" If it's neither, the instinct is to classify it as "not disruption"—which is technically correct within Christensen's definitions but strategically dangerous. "Not disruption" doesn't mean "not a threat." **Defense crowds out offense.** The organizational energy goes to monitoring, securing, responding. The company becomes excellent at not-dying—and forgets how to grow. Survival is necessary but not sufficient. **Confidence substitutes for curiosity.** "I understand how companies die" starts feeling equivalent to "I understand how to stay alive." But knowing the two most common causes of death doesn't mean you know all of them. And it certainly doesn't tell you how to thrive. None of this is a criticism of the prepared CEO. It's a limitation inherent in relying on any single lens, no matter how good that lens is. ## Building on the Map The point isn't to replace Christensen. It's to add coverage. The two doors stay locked. We're guarding the other three. **For new products (door 1):** Jobs to Be Done as an early warning system. If the job your customers hire you for starts fragmenting across other products—even products that aren't "disruptive" in the technical sense—that's the signal. A competitor doesn't need to enter from below if they redefine what "below" and "above" mean. ``` Monitor: Not just "is someone entering our market from the low end?" But "is the job we're hired for being done differently elsewhere?" ``` **For new production methods (door 2):** Watch cost structure shifts, not just competitive entries. When a technology makes it dramatically cheaper to produce similar value in *adjacent* industries, it's coming for yours next. The threat isn't a specific competitor—it's an economic shift that enables many competitors at once. ``` Monitor: Not just "who is competing with us?" But "what does it cost to produce what we produce, and is that changing?" ``` **For new sources of supply (door 4):** Track where your inputs come from and what happens if they become free or commoditized. If your business model depends on expensive content, expensive software, expensive data, or expensive expertise—and any of those are trending toward zero marginal cost—the clock is ticking regardless of your competitive position. ``` Monitor: Not just "who are our competitors?" But "what are our inputs, and are they being commoditized or made free?" ``` **For new organizational forms (door 5):** Study platform and ecosystem models, especially outside your industry. If your industry is structured as pipelines and someone figures out the platform version, the competitive dynamics change entirely. This doesn't show up as a low-end entrant. It shows up as a different game. ``` Monitor: Not just "who is entering our market?" But "is someone restructuring how this market works?" ``` **Across all doors:** Maintain the capacity for exploration. The CEO who only exploits the current business—even brilliantly, even with perfect disruption defense—is optimizing for a world that doesn't change. Schumpeter's core message is that it always does. The best defense against creative destruction isn't better monitoring. It's creating new combinations yourself. ## The Flashlight and the Hurricane Christensen gave you a flashlight. It's a good one. It illuminates a specific, important pattern—two patterns, actually—with remarkable clarity. It lets you see threats that would otherwise be invisible. The prepared CEO who uses it well has earned their confidence. Schumpeter saw the hurricane. Creative destruction isn't a pattern you can monitor. It's the weather. New combinations emerge from everywhere—new products, new production methods, new markets, new supply sources, new organizational forms. The flashlight helps you see what's directly in front of you. The hurricane is everything else. The Innovator's Dilemma is a great book. Learning it and acting on it puts you ahead of most executives. The next step isn't to abandon it. It's to recognize that it illuminates two of five doors, and the storm comes through all of them. Lock all five. And accept that in Schumpeter's world, there's always a sixth door you haven't found yet. --- ## Exploration vs Exploitation: The Hidden Cost of Cutting Innovation Slack - URL: https://svalle.ru/posts/business/exploration-vs-exploitation/ - Date: 2025-02-26 - Tags: strategy, innovation, management, business, organizational-behavior Every efficiency initiative eventually targets the same thing: slack. Unused compute capacity. Engineers not shipping features. "Innovation time" that produces nothing measurable. Research that doesn't tie to this quarter's roadmap. It looks like waste. It gets cut. Then, five years later, leadership wonders why the company has no new products, why the best engineers left, and why some startup is eating their lunch. This pattern has a name in organizational theory: the exploration-exploitation tradeoff. Understanding it explains why the most "wasteful" practices might be the most valuable. ## The Tradeoff In 1991, organizational theorist James March published a paper that became foundational to strategy thinking. His insight: organizations must balance two fundamentally different activities. **Exploitation:** - Refinement of existing capabilities - Efficiency improvements - Execution on known strategies - Predictable, measurable returns - Low variance outcomes **Exploration:** - Search for new possibilities - Experimentation with unknowns - Discovery of new markets or technologies - Uncertain, asymmetric returns - High variance outcomes Both are necessary. But they compete for resources, attention, and organizational energy. ``` Exploitation: "Do more of what works" Exploration: "Find what might work next" ``` The problem: exploitation tends to win. ## Why Exploitation Always Wins (Until It Doesn't) Organizations systematically favor exploitation over exploration. Here's why: ### 1. Exploitation Is Measurable ``` Exploitation metrics: - Revenue this quarter: $47.2M ✓ - Cost reduced by 12% ✓ - Features shipped: 34 ✓ - Velocity increased 18% ✓ Exploration metrics: - Ideas explored: ??? - Options created: ??? - Future disruption prevented: ??? - Serendipitous discoveries: ??? ``` What gets measured gets managed. What can't be measured gets cut. ### 2. Time Horizons Are Misaligned ``` Exploitation payoff: This quarter Exploration payoff: 3-10 years from now Manager tenure: 2-3 years average CEO tenure: 5 years average ``` Why invest in something that pays off after you've moved on? ### 3. Exploitation Has Certainty ``` Exploitation bet: Invest $1M → 90% chance of $1.2M return Expected value: $1.08M Exploration bet: Invest $1M → 90% chance of $0, 10% chance of $50M Expected value: $5M ``` Exploration has higher expected value but feels like gambling. Exploitation feels like prudent management. ### 4. Success Breeds Exploitation The better your current business, the more you exploit it: ``` Year 1: New product, lots of exploration Year 3: Product-market fit, optimize what works Year 5: Market leader, squeeze every efficiency Year 7: "Why would we experiment? We're winning." Year 10: "Where did that startup come from?" ``` This is the success trap. Your very success makes you vulnerable. ## The Economics of Innovation Slack "Slack" sounds like waste. But economically, it serves several functions: ### 1. Option Value Exploration creates options—the right but not obligation to pursue opportunities. ``` Traditional ROI thinking: Project cost: $100K Expected return: $0 (most explorations fail) Decision: Don't fund Option thinking: Project cost: $100K Creates option to pursue $50M opportunity Option value: $100K × 2% × $50M = $1M Decision: Fund ``` Most options expire worthless. The few that hit pay for all the rest. ### 2. Intrinsic Motivation Psychologist Dan Pink's research on motivation identifies three drivers: - **Autonomy**: Control over your work - **Mastery**: Getting better at something - **Purpose**: Working on something meaningful Exploration time hits all three. Exploitation time often hits none. ``` Exploitation work: "Ship these 5 features by Friday" Autonomy: Low Mastery: Low (repetitive) Purpose: Variable Exploration work: "Work on what you think matters" Autonomy: High Mastery: High (learning) Purpose: High (self-selected) ``` The result: exploration time has outsized impact on retention and engagement. ### 3. Serendipity and Cross-Pollination Innovation is non-linear. Breakthroughs come from unexpected connections. ```d2 {alt="Two panels contrasting innovation paths: linear exploitation flows Problem to Research to Solution to Ship, while non-linear exploration flows from an unrelated experiment to a weird observation to the realization that it could solve that other problem, ending in a breakthrough"} grid-rows: 2 vertical-gap: 48 linear: "Linear (exploitation)" { grid-rows: 1 grid-gap: 24 problem: Problem research: Research solution: Solution ship: Ship problem -> research -> solution -> ship } # horizontal-gap calibrated so this panel's natural width matches the linear # panel above (~505) and its boxes stay centered; recalibrate if labels change nonlinear: "Non-linear (exploration)" { grid-columns: 1 vertical-gap: 40 horizontal-gap: 156 experiment: Unrelated experiment observation: Weird observation realization: "\"Wait, this could solve\nthat other problem\"" breakthrough: Breakthrough experiment -> observation -> realization -> breakthrough } ``` You can't schedule serendipity. But you can create conditions for it. ### 4. Talent Retention Your best engineers have options. What do they want? ``` What top engineers value: 1. Interesting problems 2. Autonomy 3. Learning opportunities 4. Impact 5. Compensation What pure exploitation offers: 1. Repetitive problems ✗ 2. Prescribed solutions ✗ 3. Same skills, optimized ✗ 4. Incremental impact ✗ 5. Compensation ✓ ``` Cut exploration, and your best people leave for places that offer it. ## Why Innovation Time Gets Cut Despite these benefits, slack time is perpetually under threat. ### The Efficiency Narrative ``` CFO: "We're paying engineers to work on side projects?" VP Eng: "It drives innovation and retention." CFO: "What's the ROI?" VP Eng: "It's hard to measure directly..." CFO: "Then how do we know it's working?" VP Eng: "..." ``` Efficiency is easy to argue for. Slack requires defending the unmeasurable. ### The "120% Time" Trap Some organizations nominally keep innovation time but make it impossible to use: ``` Official policy: "20% time for exploration" Reality: - Sprint commitments assume 100% capacity - Managers judged on team "productivity" - Taking exploration time hurts performance reviews - Innovation time becomes "do it on your own time" Result: 20% time becomes 120% time ``` The policy exists on paper. The culture kills it in practice. ### Quarterly Pressure ``` Q1: "We need to hit numbers, postpone innovation time" Q2: "We're behind, all hands on deck" Q3: "Big launch coming, no distractions" Q4: "Year-end push, we'll do it next year" Next year: Repeat ``` There's never a good quarter to explore. ### The Productivity Measurement Trap Modern engineering organizations measure everything: ``` Metrics tracked: - Story points delivered - PRs merged - Tickets closed - Cycle time - Deployment frequency Metrics not tracked: - Ideas generated - Skills learned - Cross-team connections made - Future options created ``` When your metrics only capture exploitation, that's all you'll get. ## The Consequences of Cutting Slack What happens when exploration goes to zero? ### 1. Incremental Improvements, No Breakthroughs ``` With exploration: Year 1: Core product + 3 experiments Year 2: Core product + 1 experiment succeeds → new product line Year 3: Two product lines + more experiments Without exploration: Year 1: Core product, optimized Year 2: Core product, more optimized Year 3: Core product, extremely optimized Year 4: "Why don't we have any new products?" ``` You get very good at today's game while the game changes around you. ### 2. Best People Leave ``` Engineer thinking: "I haven't learned anything new in 2 years" "Every day is the same tickets" "That startup offered me interesting problems" "I'm out" ``` The engineers who leave first are the ones with the best options—your top performers. ### 3. Disruption from Below Startups have nothing to exploit. They're all exploration. ``` Incumbent advantage: Exploitation efficiency Startup advantage: Exploration agility When the market shifts: Incumbent: "We need to pivot" (but can't) Startup: "We were already exploring this" ``` The innovator's dilemma: your strength becomes your weakness. ### 4. Technical Stagnation ``` With exploration: Engineers try new tools, languages, architectures Some experiments fail, some improve the stack Technical capabilities evolve Without exploration: "We're a Java shop" "We've always done it this way" "That new thing is unproven" Stack fossilizes, technical debt accumulates ``` The codebase becomes a museum of past decisions. ### 5. Cultural Ossification ``` With exploration: Failure is normal (most experiments fail) Risk-taking is rewarded "What if we tried..." is welcomed Without exploration: Failure is punished Risk is avoided "That's not how we do things here" ``` The culture hardens around exploitation, making future exploration even harder. ## Finding the Balance The goal isn't to eliminate exploitation—it's to maintain balance. ### The Ambidextrous Organization O'Reilly and Tushman's research suggests successful companies are "ambidextrous": ``` Exploitation units: - Focused on current business - Efficiency-oriented - Tight processes - Measured on execution Exploration units: - Focused on future opportunities - Experimentation-oriented - Loose processes - Measured on learning ``` The key: structural separation with strategic integration. ### Time Horizons Framework Allocate resources across multiple time horizons: ``` Horizon 1 (Now): 70% of resources Exploit current business Measured quarterly Horizon 2 (2-3 years): 20% of resources Extend into adjacent areas Measured annually Horizon 3 (3-10 years): 10% of resources Explore transformational possibilities Measured on learning, not returns ``` This isn't a formula—it's a forcing function for balance. ### Protecting Exploration If you want exploration to survive, it needs protection: ``` Structural protection: - Dedicated time that doesn't compete with sprints - Separate budget not subject to quarterly cuts - Different metrics than exploitation work Cultural protection: - Leadership visibly participates - Exploration outcomes celebrated (even failures) - Career paths that reward exploration Process protection: - Regular cadence (not "when we have time") - Showcase events that create accountability - Clear path from exploration to exploitation ``` ### Leading Indicators Since exploration ROI is hard to measure, track leading indicators: ``` Input metrics: - Hours actually spent on exploration - Number of experiments started - Participation rate - Cross-team collaboration Process metrics: - Ideas generated - Prototypes built - Skills learned (self-reported) Outcome metrics (lagging): - Experiments that became products - Patents filed - Retention of high performers - Time-to-pivot when market shifts ``` ## The Meta-Lesson The exploration-exploitation tradeoff is itself a tradeoff between short-term and long-term thinking. ``` Short-term thinking: "Cut slack, boost efficiency, hit quarterly numbers" Long-term thinking: "Maintain slack, preserve optionality, survive the next disruption" ``` Organizations that cut all slack are optimizing for a world that doesn't change. But the world always changes. The companies that survive aren't the most efficient. They're the ones that balance efficiency today with adaptability for tomorrow. Slack isn't waste. It's the price of staying in the game. ## Summary **The tradeoff:** - Exploitation: refine what works (predictable, measurable) - Exploration: find what's next (uncertain, asymmetric) **Why exploitation wins:** - Measurable beats unmeasurable - Short-term beats long-term - Certainty beats variance - Success breeds more exploitation **Why exploration matters:** - Creates options for the future - Drives intrinsic motivation - Enables serendipity - Retains top talent **What happens when you cut it:** - Incremental improvements, no breakthroughs - Best people leave - Vulnerable to disruption - Technical and cultural stagnation **How to protect it:** - Structural separation - Different metrics - Leadership commitment - Regular cadence, not "when we have time" The most efficient company in a changing market is the one that's efficiently executing yesterday's strategy. The survivors are the ones who kept exploring. --- ## Porter's Five Forces for LLM Builders—And Why It's Not Enough - URL: https://svalle.ru/posts/business/porter-five-forces-llm/ - Date: 2025-02-25 - Tags: strategy, AI, LLM, business, porter, MBA, competitive-analysis Every MBA student learns Porter's Five Forces in their first strategy class. It's one of the most influential business frameworks of the past 50 years—a structured way to analyze industry competition and profitability. But can a framework from 1979 make sense of an industry that barely existed in 2020? Let's apply Porter's Five Forces to the LLM industry—OpenAI, Anthropic, Google, Meta, and the rest—and see what it reveals. Then let's examine where this classic framework breaks down. ## What Are Porter's Five Forces? In 1979, Harvard Business School professor Michael Porter published "How Competitive Forces Shape Strategy." His insight: industry profitability isn't random. It's determined by five competitive forces. ```d2 {alt="Porter's Five Forces star: Threat of New Entrants, Supplier Power, Buyer Power, and Threat of Substitutes each point an inbound arrow at the central Industry Rivalry"} direction: down entrants: "Threat of\nNew Entrants" supplier: "Supplier\nPower" buyer: "Buyer\nPower" substitutes: "Threat of\nSubstitutes" rivalry: "Industry\nRivalry" entrants -> rivalry supplier -> rivalry buyer -> rivalry substitutes -> rivalry ``` **The five forces:** 1. **Threat of New Entrants** — How easily can new competitors enter? High barriers = good for incumbents. 2. **Supplier Power** — Can suppliers dictate terms? Powerful suppliers capture value from the industry. 3. **Buyer Power** — Can customers dictate terms? Powerful buyers squeeze industry margins. 4. **Threat of Substitutes** — Can customers switch to alternatives? Substitutes cap prices and profits. 5. **Industry Rivalry** — How intensely do competitors fight? Intense rivalry erodes margins. **The core insight:** When all five forces are weak, the industry is profitable (think pharmaceuticals). When they're strong, margins suffer (think airlines). Now let's apply this to LLM builders. ## Force 1: Threat of New Entrants **Assessment: Bifurcating—high for frontier, low for "good enough"** ### The barriers to building frontier models are enormous Building a GPT-4 or Claude-class model requires: | Barrier | Scale | |---------|-------| | **Training compute** | $100M–$1B+ per frontier model | | **Talent** | Perhaps <1,000 people worldwide can lead frontier training runs | | **Data** | Trillions of high-quality tokens, increasingly scarce and legally contested | | **Time** | 2-3 years to build an organization capable of frontier work | | **Compute access** | NVIDIA H100 allocation is rationed; relationships matter | This looks like an oligopoly in formation—only a handful of labs can compete at the frontier. ### But barriers to "good enough" models are collapsing ``` 2023: Training a competitive model required $100M+ 2024: Llama 3 and Mistral are free 2025: Fine-tune a 70B model for your use case: $10K-$100K ``` Meta's open-weights strategy (giving away Llama) deliberately lowered barriers. Anyone can now deploy a capable model. Dozens of startups offer fine-tuning, hosting, and customization. **The bifurcation:** Frontier model development = high barriers, oligopoly. Model deployment and customization = low barriers, fragmented competition. ## Force 2: Supplier Power **Assessment: Extremely high** The LLM industry's suppliers have enormous leverage: ### NVIDIA: The Chokepoint ``` NVIDIA's position: - 80%+ market share in AI training chips - H100/H200 GPUs are the constraint on frontier training - 70%+ gross margins - Demand exceeds supply; allocation is strategic - AMD and Intel are years behind ``` No other supplier in tech has this leverage. LLM builders are price-takers with NVIDIA. ### Cloud Providers: Complicated AWS, Google Cloud, and Azure control compute access. But: - They compete with each other (reduces power) - They're also *competitors* in LLMs (creates tension) - Exclusive partnerships (Microsoft-OpenAI, Google-Anthropic, Amazon-Anthropic) muddy the supplier-customer relationship ### Talent: Extreme Leverage This is perhaps the highest-leverage "supplier": ``` Top AI researcher compensation: $5-50M packages Single departures reshape companies: - Ilya Sutskever leaving OpenAI - Noam Shazeer leaving Google for Character.ai - Dario and Daniela Amodei leaving OpenAI to found Anthropic Acqui-hires are really talent acquisitions: - Microsoft absorbed Inflection's team - Google absorbed Character.ai's team ``` When your most important "supplier" is a small group of irreplaceable humans with perfect information and mobility, supplier power is maximal. ### Data Providers: Rising Power ``` 2023: "Train on the internet" was accepted 2024: Reddit, Twitter/X, publishers demand licensing fees 2025: NYT sues OpenAI, everyone lawyers up ``` High-quality training data is the new oil. Suppliers are learning to extract rents. ### Energy: Emerging Constraint ``` Frontier training runs require: - Hundreds of megawatts - Uninterrupted power for months - Physical data center capacity Power availability is becoming a strategic constraint. ``` **The supplier power problem:** The "picks and shovels" players—NVIDIA, cloud providers, talent—capture enormous industry value. LLM builders are squeezed in the middle. ## Force 3: Buyer Power **Assessment: Medium, but rising** ### Enterprise buyers have leverage ``` Enterprises can choose from: - OpenAI (GPT-4, GPT-4o) - Anthropic (Claude) - Google (Gemini) - Amazon Bedrock (multiple models) - Azure OpenAI Service - Open source (Llama, Mistral) - Dozens of specialized providers ``` Procurement departments play vendors against each other. Multi-vendor strategies are common. But switching costs exist: - Fine-tuned models are vendor-specific - Integration and compliance work is non-trivial - Prompt engineering is model-specific ### Developers are price-sensitive ``` Developer reality: - OpenAI-compatible APIs are everywhere - Swap Claude for GPT for Llama with minimal code - Price arbitrage is easy (use cheapest model that works) - Open source is free ``` For developers, switching costs approach zero. ### Consumers can switch instantly ``` ChatGPT has brand recognition But switching to Claude or Gemini = 30 seconds No data lock-in, no integration cost ``` **Why buyer power is rising:** - Model capabilities are converging (GPT-4 ≈ Claude ≈ Gemini for most tasks) - Prices collapsed 90%+ in 18 months - Open source offers "free" - Commoditization is accelerating ## Force 4: Threat of Substitutes **Assessment: High and multi-dimensional** Porter defines substitutes as different products that serve the same need. For LLMs, substitutes include: ### Open Source Models ``` Llama 3 405B rivals GPT-4 for many tasks. Cost: Free (plus inference compute). This isn't traditional substitution—it's strategic commoditization. Meta gives away models to commoditize competitors. ``` ### Specialized Small Models ``` A fine-tuned 7B model can beat GPT-4 for specific tasks. Faster, cheaper, easier to deploy. For structured problems, small models win. ``` ### Traditional ML and Rules ``` Not every problem needs a $0.01/request LLM call. Logistic regression still works for classification. Rule engines still work for business logic. The substitute is "don't use an LLM." ``` ### Human Labor ``` For some tasks, humans are still: - Higher quality - More accountable - Required for compliance - Cheaper at low volume ``` ### On-Prem Deployment ``` Regulated industries can't send data to APIs. With open weights, deploy Llama behind your firewall. The substitute is "same capability, different deployment." ``` **Key insight:** The "substitutes" in LLMs aren't just competing products. Open source is a *strategic weapon* that reshapes the entire competitive landscape. ## Force 5: Industry Rivalry **Assessment: Intense, multi-front war** ### The competitors ``` Frontier labs: OpenAI, Anthropic, Google DeepMind, Meta AI, xAI Cloud-native: Amazon (Titan), Microsoft (Phi), Cohere, AI21 China: Baidu, Alibaba, ByteDance, Moonshot Startups: Mistral, Reka, Adept, Inflection (now Microsoft) ``` New credible entrants emerge quarterly. ### Dimensions of competition | Dimension | Current State | |-----------|---------------| | **Price** | Race to bottom. GPT-4-class dropped from $60/M tokens to $2-5/M. | | **Capabilities** | Context windows (128K → 2M), multimodal, reasoning, tool use | | **Speed** | Latency matters for real-time use cases | | **Safety/Trust** | Enterprise buyers care. Regulatory positioning. | | **Distribution** | Microsoft has Office. Google has Search. Apple has devices. | | **Ecosystem** | Plugins, integrations, developer tools, fine-tuning services | | **Talent** | Acqui-hires, poaching, research prestige | ### The intensity drivers ``` High fixed costs: Frontier training costs $100M+; need volume to recoup Low marginal costs: Serving one more API call costs fractions of a cent Low differentiation: Models converging in capability High strategic stakes: "This is the next platform"—everyone must compete ``` This is a classic formula for intense rivalry and margin compression. ## What Porter's Five Forces Gets Right The framework reveals real insights about the LLM industry: **1. Supplier power is the strategic constraint** NVIDIA and talent capture enormous value. LLM builders are squeezed. This explains: - Why labs pay $10M+ for researchers - Why NVIDIA has 75% gross margins while labs struggle to profit - Why compute partnerships (Microsoft-OpenAI, Google-Anthropic) are existential **2. Rivalry will compress margins** The conditions for intense competition all exist: high fixed costs, low marginal costs, multiple well-funded competitors, converging capabilities. This suggests: - API pricing will continue falling - Differentiation will be difficult - Profitability will be elusive for most players **3. Buyer power is increasing** As models commoditize, leverage shifts to customers. This explains: - Why OpenAI cut prices repeatedly - Why enterprises demand multi-model strategies - Why open source matters strategically **4. Barriers are bifurcated** Frontier model training is an oligopoly; deployment is fragmented. This suggests: - Consolidation at the frontier (3-5 labs) - Fragmentation in applications (hundreds of companies) - Value capture will be contested in the middle ## Where Porter's Five Forces Breaks Down Now for the critique. Porter's framework has fundamental limitations when applied to the LLM industry. ### 1. Static Analysis, Dynamic Industry Porter assumes industry structure is stable enough to analyze. LLM reality: ``` 2022: GPT-3 is impressive but niche 2023: ChatGPT changes everything; Google declares "code red" 2024: Open source catches up; prices collapse 90% 2025: Agents? New architectures? Regulation? ``` A five forces analysis has a half-life of maybe 6 months. The framework wants stability; the industry delivers chaos. ### 2. Blurry Industry Boundaries Porter needs clear industry boundaries. Where are the boundaries here? ``` - Foundation model training? - API inference services? - Consumer chatbots? - Enterprise AI platforms? - AI features in existing products? - Chips for AI? - AI applications? ``` Is Anthropic competing with OpenAI (yes), Google (yes), Notion (sort of), and McKinsey (maybe)? Industry boundaries are fractal. ### 3. Dual-Role Players Break the Model Porter assumes clean categories. Real relationships are messier: | Player | Their Roles | |--------|-------------| | **Microsoft** | Investor in OpenAI + competitor (Copilot) + cloud supplier + customer + distribution partner | | **Google** | Cloud supplier to Anthropic + direct competitor (Gemini) + search incumbent + investor | | **NVIDIA** | Supplier to everyone + potential competitor (if they build models) + platform kingmaker | | **Meta** | Competitor + supplier (gives away Llama) + uses AI for its own apps | | **Amazon** | Cloud supplier + investor in Anthropic + competitor (Titan) + customer | When Microsoft is simultaneously OpenAI's investor, supplier, distribution partner, and competitor, "supplier power" and "rivalry" collapse into game theory. ### 4. Complementors Are Missing Porter's original framework ignores complementors entirely. (He added them later in the "Value Net" with Adam Brandenburger.) For LLMs, complementors are critical: ``` Complementors: - Fine-tuning services (Scale AI, Together AI) - Vector databases (Pinecone, Weaviate, Chroma) - Orchestration frameworks (LangChain, LlamaIndex) - Evaluation tools (Braintrust, Weights & Biases) - App developers building on APIs - Enterprise integrators ``` The health of the complementor ecosystem determines API adoption. OpenAI's moat is partly its developer ecosystem—not captured by five forces. ### 5. Network Effects Are Underweighted Porter treats industries as atomistic competition among independent firms. LLMs have network effects: ``` Developer network effects: More devs → more tools/libraries → easier development → more devs Data flywheel: More users → more feedback/RLHF data → better models → more users Ecosystem lock-in: More integrations → higher switching costs → more integrations Mindshare compounding: "GPT" became generic term → default choice → self-reinforcing ``` These winner-take-most dynamics aren't well captured by five forces, which assumes roughly linear competition. ### 6. Open Source as Strategic Weapon Porter treats substitutes as competing products that serve the same need—like generic drugs undercutting branded pharmaceuticals. But Meta's Llama strategy isn't substitution. It's *strategic commoditization*: ``` Meta's playbook: 1. Spend $X billion training Llama 2. Give it away for free 3. Commoditize the model layer 4. Prevent OpenAI/Google from locking in developers 5. Value accrues to apps (where Meta competes) 6. Attract research talent who want open work 7. Shape standards and ecosystem ``` Open source here is a weapon, not a substitute. It requires game theory to analyze, not substitution curves. ### 7. Regulatory Risk Is Existential Porter treats regulation as part of the external environment—relevant but background. For LLMs, regulation could restructure everything: ``` Potential impacts: - EU AI Act: Compliance costs, transparency requirements - US regulation: Unclear but potentially significant - Export controls: NVIDIA chips restricted to China - Copyright: Training data lawsuits could invalidate models - Safety mandates: Testing requirements, release delays - Licensing: Some propose licensing frontier labs ``` A single regulatory decision could make the entire five forces analysis obsolete overnight. Regulation isn't background—it's a primary strategic variable. ### 8. Talent IS Strategy Porter treats labor as a factor input, like capital or materials. In LLMs, talent is THE moat: ``` Facts about AI talent: - Perhaps 100 people can lead frontier training runs - A single researcher leaving can reshape a company - Acqui-hires are talent acquisitions disguised as M&A - Research culture and publication freedom are competitive advantages - $10-50M packages for top researchers ``` "Supplier power of labor" doesn't capture this. Talent strategy deserves its own framework, not a bullet point under suppliers. ### 9. Technology Discontinuities Porter assumes competition within a paradigm. But fundamental breakthroughs can invalidate everything: ``` Paradigm shifts: 2017: Transformers invented → RNNs obsolete 2020: Scaling laws proven → Massive compute becomes strategy 2022: RLHF/ChatGPT → Interaction paradigm shifts 2024: ??? ``` The next architectural breakthrough could make current frontier models obsolete. Porter doesn't handle Schumpeterian disruption well. ### 10. Non-Profit Motivations Porter assumes profit-maximizing firms. The LLM industry has: ``` - OpenAI: Capped-profit, safety mission (complicated by Microsoft deal) - Anthropic: Public Benefit Corporation, safety-focused - Meta AI: Gives away models; strategic motivation unclear - Academic labs: Publication incentives, not profit ``` When major competitors aren't straightforwardly profit-maximizing, competitive analysis gets strange. Why is Meta giving away $100M+ models? ## Complementary Frameworks Porter's Five Forces is a starting point, not a complete analysis. For LLMs, combine it with: | Framework | What It Adds | |-----------|--------------| | **Platform Economics** | Network effects, two-sided markets, ecosystem orchestration | | **Co-opetition** (Brandenburger & Nalebuff) | Analyzing "frenemies," complementors, the Value Net | | **Disruption Theory** (Christensen) | Open source as low-end disruption | | **Real Options** | Valuing optionality in uncertain tech bets | | **Strategic Inflection Points** (Andy Grove) | Recognizing and navigating paradigm shifts | | **Ecosystem Strategy** (Adner) | Orchestrator vs participant positioning | | **Regulatory Strategy** (Baron) | Shaping regulation as competitive advantage | | **Talent Strategy** | Talent acquisition, culture, and retention as moat | ## The Bottom Line **What Five Forces gets right about LLMs:** - Supplier power (NVIDIA, talent) is the key constraint - Rivalry will compress margins - Buyer power is increasing as models commoditize - Barriers are bifurcated (oligopoly at frontier, fragmented below) **What it misses:** - Speed of change (analysis obsolete in months) - Boundary fluidity (what industry is this?) - Dual-role players (competitor-supplier-partner hybrids) - Complementors and ecosystems - Network effects and winner-take-most dynamics - Open source as strategic weapon, not just substitute - Existential regulatory risk - Talent as THE moat - Discontinuous technological change **The verdict:** Porter's Five Forces is a useful *starting point* for analyzing the LLM industry. It surfaces important dynamics and forces structured thinking. But it's wildly insufficient on its own. The LLM industry breaks Porter's assumptions about stable boundaries, clean competitive categories, profit-maximizing actors, and incremental change. To understand where the industry is going, you need to combine five forces with platform economics, game theory, ecosystem strategy, and regulatory analysis. Porter gave us the foundation. The LLM industry demands we build beyond it. --- ## Bundling vs Unbundling Platform Services: A Microeconomic Framework - URL: https://svalle.ru/posts/business/bundling-unbundling-platform/ - Date: 2025-02-24 - Tags: platform-engineering, economics, strategy, microeconomics, business Your platform team offers CI/CD, Kubernetes, observability, and secrets management. Should this be: **A. One integrated platform** — "Use our platform" means you get everything **B. A menu of services** — Teams pick what they need: CI/CD ($X), K8s ($Y), observability ($Z) **C. Something in between** — Core bundle + optional add-ons This isn't just a product design question. It's a microeconomics question—and bundling theory gives us a framework. ## Why Bundling Exists Bundling isn't about convenience. It's about extracting surplus and reducing variance. ### The Basic Insight Consider two teams evaluating your platform: ``` Team A's willingness to pay (in adoption effort): CI/CD: High (5 hours) Observability: Low (1 hour) Team B's willingness to pay: CI/CD: Low (1 hour) Observability: High (5 hours) ``` **If you unbundle (sell separately):** ``` Price CI/CD at 5 hours? Team A adopts, Team B doesn't Revenue: 5 hours Price CI/CD at 1 hour? Both adopt Revenue: 2 hours Same analysis for observability. Best case unbundled: ~6 hours total ``` **If you bundle:** ``` Team A values bundle at: 5 + 1 = 6 hours Team B values bundle at: 1 + 5 = 6 hours Price bundle at 6 hours: Both teams adopt Revenue: 12 hours ``` Bundling captured 2x the value. Magic? No—math. ### The Correlation Principle **Bundling works when preferences are negatively correlated.** ``` Negative correlation: "Teams who value CI/CD high tend to value observability low, and vice versa" → Bundling wins Positive correlation: "Teams who value CI/CD high also value observability high" → Unbundling may win No correlation: → Bundling usually still helps (variance reduction) ``` Why? Bundling averages out the valuations: ``` Unbundled: Team A: [High, Low] → Pays for high-value item only Team B: [Low, High] → Pays for high-value item only Bundled: Team A: [High + Low = Medium] → Pays medium for both Team B: [Low + High = Medium] → Pays medium for both ``` The bundle has lower variance in willingness to pay, making pricing easier. ## Applying This to Platform Services ### Step 1: Map Your Services ``` Platform components: - CI/CD pipelines - Kubernetes namespaces - Observability (metrics, logs, traces) - Secrets management - Service mesh - Feature flags - Database provisioning ``` ### Step 2: Understand Preference Correlation Survey or observe teams: ``` Question: "Rank these services by how much you'd invest to adopt them" Team profiles that emerge: "Full-stack product teams": High: CI/CD, observability, feature flags Low: Service mesh, database provisioning "Backend infrastructure teams": High: Kubernetes, service mesh, database provisioning Low: Feature flags, CI/CD (they have their own) "Data teams": High: Database provisioning, observability Low: CI/CD, service mesh, feature flags ``` **If profiles are distinct (negative correlation):** Bundle wins **If all teams want the same things (positive correlation):** Consider unbundling ### Step 3: Estimate the Bundling Value ``` Unbundled adoption (hypothetical): CI/CD: 80% of teams Kubernetes: 70% of teams Observability: 60% of teams Secrets: 40% of teams Average: 62.5% adoption per service Bundled adoption: Full platform: 85% of teams (bundle is better deal) Bundling increases total adoption. ``` ## When to Bundle ### Bundle When: High Fixed Costs ``` Each service has: Development cost: $500K Maintenance: $200K/year Support: $100K/year If unbundled, each service needs to justify its cost. If bundled, costs are spread across all adopters. ``` Bundling amortizes fixed costs more efficiently. ### Bundle When: Integration Creates Value ``` Standalone observability: - Metrics from your app - Manual correlation Observability + CI/CD + Kubernetes: - Deployment markers in dashboards - Automatic rollback on error spike - Correlated logs across services Integration value >> Sum of parts ``` When components are more valuable together, bundle them. ### Bundle When: Reducing Adoption Friction ``` Unbundled: Team must make 5 separate decisions Each decision has friction Adoption: Slow, partial Bundled: One decision: "Adopt the platform" One onboarding process Adoption: Fast, complete ``` Decision fatigue is real. Bundling simplifies adoption. ### Bundle When: Preventing Adverse Selection ``` Unbundled pricing: Expensive services: Only heavy users adopt Cheap services: Everyone adopts Result: Cost structure skews unfavorable Bundled pricing: Light users subsidize heavy users (within each service) Heavy users of Service A subsidize light users of Service A Cross-subsidization evens out ``` Bundling prevents cherry-picking. ## When to Unbundle ### Unbundle When: Preferences Are Homogeneous ``` Every team wants: CI/CD: High Observability: High Service mesh: Low Feature flags: Low No negative correlation—everyone wants the same things. Bundling forces teams to "pay" for things they don't want. ``` If everyone wants the same subset, offer that subset. ### Unbundle When: Components Are Substitutes ``` Your platform offers: Prometheus-based monitoring Datadog integration These are substitutes, not complements. Bundling both makes no sense—teams choose one. ``` Bundle complements, unbundle substitutes. ### Unbundle When: Adoption Barriers Are High ``` Full platform adoption: Effort: 100 hours Teams willing to invest 100 hours: 30% CI/CD only adoption: Effort: 10 hours Teams willing to invest 10 hours: 90% ``` If the bundle is too big to swallow, unbundling gets you in the door. ### Unbundle When: Teams Have Strong Existing Solutions ``` Team situation: CI/CD: Happy with GitHub Actions Observability: Need something Kubernetes: Need something Force full platform? Team resists—doesn't want to migrate CI/CD Offer components? Team adopts observability + K8s Maybe CI/CD later ``` Unbundling respects sunk costs. ## The Hybrid: Core Bundle + Add-Ons Most mature platforms land here: ``` Core bundle (mandatory): - CI/CD - Kubernetes namespace - Basic observability → Standard "platform adoption" Add-ons (optional): - Advanced observability (tracing) - Service mesh - Feature flags - Database provisioning → Teams opt-in based on need ``` ### Why Hybrid Works ``` Core bundle: - Achieves integration benefits - Simplifies adoption decision - Amortizes fixed costs - Establishes platform relationship Add-ons: - Captures additional value from high-need teams - Doesn't force low-value components on everyone - Allows incremental expansion - Tests demand for new capabilities ``` ### Designing the Core Bundle Include components that are: ``` 1. Universally needed (>80% of teams want it) 2. Highly integrated (much more valuable together) 3. Foundation for other services 4. Low marginal cost to include ``` Exclude components that are: ``` 1. Niche (only 30% of teams want it) 2. Standalone (works fine without integration) 3. High marginal cost 4. Controversial (teams have strong existing preferences) ``` ### Pricing the Add-Ons ``` Add-on pricing strategies: Free (loss leader): - Drives adoption of core platform - Low marginal cost - Example: Basic feature flags Included at threshold: - Free up to X usage, then charges apply - Self-selects heavy users - Example: Tracing (free <1M spans/month) Premium: - Significant standalone value - High marginal cost - Example: Managed database provisioning ``` ## Real-World Examples ### AWS: Master Bundlers ``` Bundle: AWS Account - IAM (free, but required) - VPC (free, foundational) - CloudWatch basics (free tier) Add-ons: - RDS, Lambda, EKS, etc. - Each priced separately Result: Easy to start, expand over time ``` ### Datadog: Bundle for Value ``` Bundle: Infrastructure monitoring - Metrics - Host maps - Dashboards - Alerts Add-ons (priced per host/volume): - APM - Logs - Synthetics - Security Note: Each add-on is MORE valuable because of integration with core. ``` ### Backstage: Fully Unbundled ``` Core: Developer portal framework Plugins: Everything is a plugin - CI/CD views - Kubernetes - Docs - APIs Approach: Maximum flexibility, teams compose their own bundle Trade-off: More adoption friction, less integration guarantee ``` ## Making the Decision ### The Decision Matrix | Factor | Bundle | Unbundle | |--------|--------|----------| | Preference correlation | Negative | Positive | | Integration value | High | Low | | Fixed costs | High | Low | | Adoption friction | High (need to reduce) | Low | | Existing solutions | Few | Many | | Team sophistication | Lower (want simplicity) | Higher (want control) | ### The Phased Approach ``` Phase 1 (Early platform): Single bundle — maximize adoption, prove value Phase 2 (Growing platform): Core bundle + 1-2 add-ons — capture additional value Phase 3 (Mature platform): Core bundle + menu of add-ons — full flexibility ``` Start bundled, unbundle as you learn what teams value. ## Measuring Success ### Bundled Platform Metrics ``` Adoption rate: % of teams on platform Utilization breadth: % of components actively used Time to adopt: Hours from decision to productive Net promoter score: Would teams recommend? ``` ### Unbundled Platform Metrics ``` Per-component adoption: % of teams using each component Cross-sell rate: % of single-component teams that add more Churn rate: % of teams dropping components Revenue per team: Total value captured per team ``` ### Hybrid Platform Metrics ``` Core adoption: % of teams on core bundle Add-on attach rate: % of core teams using add-ons Expansion revenue: Value from add-ons over time Bundle satisfaction: Is core bundle right-sized? ``` ## Summary Bundling isn't arbitrary—it's economics: | Principle | Application | |-----------|-------------| | Negative correlation | Bundle services with diverse appeal | | Variance reduction | Bundle to simplify pricing | | Integration value | Bundle complements, unbundle substitutes | | Fixed cost amortization | Bundle to spread costs | | Adoption friction | Bundle to simplify decisions | The framework: ``` 1. Map your services 2. Analyze preference correlation across teams 3. Assess integration value between components 4. Consider adoption friction and existing solutions 5. Design: Pure bundle, pure unbundle, or hybrid 6. Iterate based on adoption data ``` Most platforms evolve toward **core bundle + add-ons**: - Core bundle captures integration value and simplifies adoption - Add-ons capture additional surplus from high-need teams - Hybrid balances simplicity with flexibility The right bundling strategy extracts more value, increases adoption, and makes your platform feel like a coherent product rather than a random collection of tools. Netflix bundles everything. Cable unbundled and died. Your platform needs to find its own answer—and now you have the framework to find it. --- ## Lean Thinking for Platform Engineering: Finding the Muda in Your Platform - URL: https://svalle.ru/posts/business/lean-platform-engineering/ - Date: 2025-02-23 - Tags: platform-engineering, lean, operations, devops, developer-experience Toyota revolutionized manufacturing by relentlessly eliminating waste—muda in Japanese. Your platform has waste too. Developers waiting for CI. Unnecessary approval steps. Defects that cause rollbacks. Features nobody uses. Lean thinking provides a framework to find and eliminate this waste. Let's apply the seven wastes to platform engineering. ## The Seven Wastes Toyota identified seven categories of waste. Each has a direct parallel in platform engineering. | Waste | Manufacturing | Platform Engineering | |-------|---------------|---------------------| | Transport | Moving parts unnecessarily | Moving data between systems | | Inventory | Excess stock | Over-provisioned resources | | Motion | Unnecessary worker movement | Context switching, navigation | | Waiting | Idle time | Queue time, approval delays | | Overproduction | Making too much | Building unused features | | Overprocessing | Unnecessary work | Excessive compliance, redundant checks | | Defects | Rework, scrap | Incidents, rollbacks, debugging | Let's examine each in detail. ## Waste 1: Transport **Manufacturing**: Moving parts between workstations adds time but no value. **Platform engineering**: Moving data, artifacts, or requests between systems unnecessarily. ### Examples ```d2 {alt="Code artifact journey: a seven-step pipeline from Git through Jenkins, Artifactory, Jenkins again, Kubernetes, Registry, and back to Kubernetes, asking why the artifact touches 6 systems and whether it could be simplified to Git to Build to Registry to Kubernetes"} grid-columns: 1 vertical-gap: 32 chain: "Code artifact journey" { style: {stroke-width: 0; fill: transparent} grid-rows: 1 grid-gap: 16 # 24 pushes total width past 900 with 7 boxes; 16 keeps it at ~863 git: Git jenkins1: Jenkins artifactory: Artifactory jenkins2: Jenkins k8s1: Kubernetes registry: Registry k8s2: Kubernetes git -> jenkins1 -> artifactory -> jenkins2 -> k8s1 -> registry -> k8s2 } question: "Why is the artifact touching 6 systems?\nCould it go: Git → Build → Registry → Kubernetes?" {style: {stroke-width: 0; fill: transparent}} ``` ```d2 {alt="Log data transport pipeline: App to Fluentd to Kafka to Logstash to Elasticsearch to Kibana, where each hop adds latency, complexity, and failure risk"} grid-columns: 1 vertical-gap: 32 chain: "Log data transport" { style: {stroke-width: 0; fill: transparent} grid-rows: 1 grid-gap: 24 app: App fluentd: Fluentd kafka: Kafka logstash: Logstash es: Elasticsearch kibana: Kibana app -> fluentd -> kafka -> logstash -> es -> kibana } note: "Each hop adds latency, complexity, and failure risk." {style: {stroke-width: 0; fill: transparent}} ``` ```d2 {alt="Request routing pipeline: User to CDN to Load Balancer to API Gateway to Service Mesh to Service, asking whether every hop is necessary"} grid-columns: 1 vertical-gap: 32 chain: "Request routing" { style: {stroke-width: 0; fill: transparent} grid-rows: 1 grid-gap: 24 user: User cdn: CDN lb: Load Balancer gw: API Gateway mesh: Service Mesh svc: Service user -> cdn -> lb -> gw -> mesh -> svc } note: "Is every hop necessary?" {style: {stroke-width: 0; fill: transparent}} ``` ### Finding Transport Waste ``` Exercise: Draw your deployment pipeline For each arrow between systems: - Why does this transition exist? - What value does it add? - Could we eliminate or combine steps? ``` ### Eliminating Transport Waste ```d2 {alt="Before and after deploy pipelines: the Before panel chains Git to Jenkins to Artifactory to Spinnaker to Kubernetes, the After panel chains Git to GitHub Actions to Kubernetes (direct deploy), removing 2 system transitions and 3 integration points for faster deploys and fewer failure modes"} grid-columns: 1 vertical-gap: 32 before: Before { grid-rows: 1 grid-gap: 24 git: Git jenkins: Jenkins artifactory: Artifactory spinnaker: Spinnaker k8s: Kubernetes git -> jenkins -> artifactory -> spinnaker -> k8s } after: After { grid-rows: 1 # horizontal-gap calibrated so After matches Before's 658px natural width: # content 430px, (658-430)/4 sides+gaps = 57 — keeps children centered, not left-packed horizontal-gap: 57 vertical-gap: 24 git: Git gha: GitHub Actions k8s: "Kubernetes (direct deploy)" git -> gha -> k8s } result: "Removed: 2 system transitions, 3 integration points\nResult: Faster deploys, fewer failure modes" {style: {stroke-width: 0; fill: transparent}} ``` ## Waste 2: Inventory **Manufacturing**: Excess stock ties up capital and hides problems. **Platform engineering**: Over-provisioned resources, unused capacity, accumulated queues. ### Examples ``` Resource inventory: Reserved instances at 30% utilization Kubernetes nodes with 20% pod density 10TB of "just in case" storage Environments that nobody uses anymore ``` ``` Work inventory (WIP): 50 open PRs waiting for review 20 tickets "in progress" for weeks 12 half-finished platform features ``` ``` Data inventory: Logs retained for 2 years (policy requires 90 days) Backups of decommissioned systems Metrics at 10-second granularity stored forever ``` ### Finding Inventory Waste ``` Resource audit: - What's the utilization of each resource type? - What's the oldest unused environment? - How much data is past retention policy? Work audit: - How many items are in WIP? - What's the average age of WIP? - What's blocked and why? ``` ### Eliminating Inventory Waste ``` Little's Law: Lead Time = WIP / Throughput To reduce lead time, reduce WIP: - PR limit: Max 3 open PRs per developer - Feature limit: Max 2 features in progress per team - Environment cleanup: Delete after 7 days of inactivity ``` ``` Resource right-sizing: Before: 20 nodes at 20% utilization After: 8 nodes at 50% utilization Savings: 60% compute cost ``` ## Waste 3: Motion **Manufacturing**: Workers walking to get tools or materials. **Platform engineering**: Developers navigating systems, context switching, hunting for information. ### Examples ``` Deployment motion: 1. Open Jenkins (find the right job) 2. Check the logs (scroll, scroll, scroll) 3. Open Kubernetes dashboard (find the namespace) 4. Check pod status (wait for it to load) 5. Open Datadog (create the right query) 6. Verify metrics (adjust time range) 6 different systems, 6 context switches. ``` ```d2 {alt="Debugging motion sequence: the question of where the logs for this service are leads to Slacking the on-call, being told to check Kibana, not finding the right index, asking again, learning the service uses CloudWatch, finding CloudWatch, and ending up in the wrong region"} grid-columns: 1 vertical-gap: 44 row1: "Debugging motion: \"Where are the logs for this service?\"" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 108 vertical-gap: 8 slack: Slack the on-call kibana: "They say \"check Kibana\"" index: "Can't find the right index" slack -> kibana -> index } row2: "" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 40 vertical-gap: 8 region: Wrong region find: Find CloudWatch cloudwatch: "\"Oh, that service uses CloudWatch\"" ask: Ask again ask -> cloudwatch cloudwatch -> find find -> region } row1.index -> row2.ask ``` ```d2 {alt="Information hunting sequence: asking what the config for this service is leads to checking Git (which repo?), the wiki (outdated), Confluence (wrong version), and finally asking in Slack (wait for response)"} grid-columns: 1 vertical-gap: 44 row1: "Information hunting: \"What's the config for this service?\"" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 70 vertical-gap: 8 git: "Check Git (which repo?)" wiki: "Check wiki (outdated)" git -> wiki } row2: "" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 40 vertical-gap: 8 slack: "Ask in Slack (wait for response)" confluence: "Check Confluence (wrong version)" confluence -> slack } row1.wiki -> row2.confluence ``` ### Finding Motion Waste ``` Shadow a developer for a day: - How many systems do they touch? - How many times do they context switch? - How much time finding vs doing? - What questions do they ask repeatedly? ``` ### Eliminating Motion Waste ``` Single pane of glass: Before: 6 tools to check deployment status After: 1 dashboard with deployment, logs, metrics, alerts Reduced motion: 5 context switches eliminated ``` ``` Self-service answers: Before: Slack questions about config, access, status After: Internal developer portal with search Reduced motion: No more hunting, asking, waiting ``` ## Waste 4: Waiting **Manufacturing**: Workers or machines idle, waiting for inputs. **Platform engineering**: Developers waiting for builds, tests, approvals, environments. ### Examples ``` CI/CD waiting: Build queue time: 15 minutes Build execution: 10 minutes Test queue time: 10 minutes Test execution: 20 minutes Deploy approval wait: 4 hours Deploy execution: 5 minutes Total: 5 hours Active work: 35 minutes Waiting: 4.5 hours (90% of time is waste) ``` ```d2 {alt="Environment waiting sequence: a request for a staging environment goes through submitting a request ticket, waiting a day for approval, waiting a day for provisioning, and the environment being ready two days later"} grid-columns: 1 vertical-gap: 44 row1: "Environment waiting: \"I need a staging environment\"" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 82 vertical-gap: 8 ticket: Submit request ticket approval: "Wait for approval (1 day)" ticket -> approval } row2: "" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 40 vertical-gap: 8 ready: "Environment ready (2 days later)" provisioning: "Wait for provisioning (1 day)" provisioning -> ready } row1.approval -> row2.provisioning ``` ``` Human bottleneck waiting: Code review: 2 days average Security review: 1 week average Architecture review: 2 weeks average ``` ### Finding Waiting Waste ``` Measure queue times at each step: Time in queue / Time being processed = Wait Ratio Wait ratio > 1 = More waiting than working Wait ratio > 5 = Severe waiting waste ``` ### Eliminating Waiting Waste ``` Build queue time: Before: 15 minutes (shared build agents) After: 0 minutes (auto-scaling build agents) Test parallelization: Before: 20 minutes (sequential) After: 5 minutes (parallelized) Self-service environments: Before: 2 days (ticket + approval + provision) After: 10 minutes (automated provisioning) ``` ``` Async approvals: Before: Block on human approval After: Deploy to staging immediately, require approval for prod Reduced wait without reducing safety. ``` ## Waste 5: Overproduction **Manufacturing**: Making more than customers need. **Platform engineering**: Building features that aren't used, over-engineering solutions. ### Examples ``` Platform feature graveyard: - Custom deployment strategies (nobody uses) - Advanced caching layer (one team tried once) - Multi-region support (never activated) - Plugin system (no plugins built) ``` ``` Premature optimization: "We built this to handle 10x our current scale" (Scale never came) (Complexity remains) ``` ``` Documentation overproduction: 100-page architecture doc (never read) Detailed runbooks (outdated before finished) Video tutorials (nobody watches) ``` ### Finding Overproduction Waste ``` Feature usage audit: For each platform capability: - How many teams use it? - How often is it used? - If removed, who would notice? ``` ### Eliminating Overproduction Waste ``` Build only what's needed: Before: Design for hypothetical scale After: Build for current needs + clear extension points Kill unused features: If usage < 5%: Deprecate it If usage = 0%: Remove it Sunsets reduce maintenance burden. ``` ``` Minimal documentation: Before: Comprehensive docs (never read, always stale) After: README + runbook + examples (actually maintained) ``` ## Waste 6: Overprocessing **Manufacturing**: More precision or steps than needed. **Platform engineering**: Excessive process, redundant checks, unnecessary rigor. ### Examples ``` Approval overhead: Deploy to dev: Requires approval (why?) Change config: Requires CAB ticket (really?) Add team member: Requires 3 sign-offs (necessary?) ``` ``` Compliance theater: Security scan on every commit (same code, same result) Vulnerability report nobody reads Audit logs nobody audits Checklist nobody checks ``` ``` Process for process's sake: "We need a design doc for this 10-line change" "Let's schedule a review meeting" (for trivial changes) "Fill out this template" (fields don't apply) ``` ### Finding Overprocessing Waste ``` For each process step: - What risk does this mitigate? - Has that risk ever materialized? - Is there a lighter-weight alternative? - Who would notice if we skipped it? ``` ### Eliminating Overprocessing Waste ``` Risk-based approvals: Before: All changes require approval After: Low risk (dev, config): Auto-approve Medium risk (staging): Peer approve High risk (prod): Lead approve ``` ``` Smart scanning: Before: Full security scan on every commit After: Full scan on changed files only Full scan nightly Reduced: 80% scan time with same coverage ``` ## Waste 7: Defects **Manufacturing**: Scrap, rework, quality failures. **Platform engineering**: Incidents, rollbacks, debugging, incorrect configurations. ### Examples ``` Deployment defects: Failed deployments: 15% of deploys Rollbacks: 5% of deploys Time to fix: 2 hours average If 100 deploys/week: 15 failures × 2 hours = 30 hours of rework/week ``` ```d2 {alt="Configuration defect sequence: asking why this environment is broken leads to config drift from production, 4 hours of debugging, a manual fix, and the note that it will happen again"} grid-columns: 1 vertical-gap: 44 row1: "Configuration defects: \"Why is this environment broken?\"" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 40 vertical-gap: 8 drift: "Config drift\nfrom production" debug: 4 hours debugging drift -> debug } row2: "" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 100 vertical-gap: 8 again: "(Will happen again)" fix: Manual fix fix -> again } row1.debug -> row2.fix ``` ``` Platform defects: CI randomly fails (flaky tests) Environments randomly break (resource limits) Deploys randomly timeout (network issues) "Random" = Unresolved defects ``` ### Finding Defect Waste ``` Track defect metrics: - Deployment failure rate - Mean time to recovery - Rollback frequency - Repeat incidents (same cause) - Escaped defects (caught in production) ``` ### Eliminating Defect Waste ``` Build quality in: Before: Test in production, fix defects After: Fail fast in CI, prevent defects Poka-yoke (mistake-proofing): Before: Config as free-form YAML (easy to break) After: Config as typed schema (invalid = won't deploy) Reduced: 80% of config-related incidents ``` ```d2 {alt="Root cause elimination: before means fix the incident and move on, after means fix the incident and fix the system that allowed it, following the chain Incident to Post-mortem to Prevention to No repeat"} grid-columns: 1 vertical-gap: 32 chain: "Root cause elimination" { style: {stroke-width: 0; fill: transparent} grid-rows: 1 grid-gap: 24 incident: Incident postmortem: Post-mortem prevention: Prevention norepeat: No repeat incident -> postmortem -> prevention -> norepeat } approach: "Before: Fix the incident, move on\nAfter: Fix the incident, fix the system that allowed it" {style: {stroke-width: 0; fill: transparent}} ``` ## The Lean Platform Audit Walk through your platform with the waste lens: ``` For each developer workflow: 1. Map the value stream (every step from code to production) 2. Categorize each step: - Value-add: Customer would pay for this - Non-value but necessary: Required (compliance, safety) - Waste: Neither valuable nor necessary 3. For each waste, identify the type: - Transport: Unnecessary movement of artifacts/data - Inventory: Excess resources, WIP, data - Motion: Developer context switching, hunting - Waiting: Queue time, approval delays - Overproduction: Unused features, premature optimization - Overprocessing: Unnecessary rigor, redundant checks - Defects: Failures, rework, debugging 4. Prioritize by impact: - How much time/money does this waste? - How hard is it to eliminate? - Focus on high-impact, low-effort first ``` ## Continuous Improvement (Kaizen) Lean isn't a one-time audit. It's a culture. ``` Daily: - Notice waste when you see it - Quick fixes for small waste Weekly: - Team discusses biggest waste encountered - Prioritize improvement items Monthly: - Measure waste metrics (lead time, failure rate, utilization) - Track improvement trends Quarterly: - Value stream mapping exercise - Major waste elimination initiatives ``` ## Summary The seven wastes in platform engineering: | Waste | Platform Symptoms | Elimination | |-------|-------------------|-------------| | Transport | Artifacts touching too many systems | Simplify pipeline | | Inventory | Over-provisioned, excess WIP | Right-size, limit WIP | | Motion | Context switching, hunting | Single pane of glass | | Waiting | Queue time, approval delays | Parallelize, self-service | | Overproduction | Unused features | Build only what's needed | | Overprocessing | Excessive process | Risk-based controls | | Defects | Failures, rollbacks | Build quality in | The lean mindset: ``` Ask constantly: - Is this step adding value? - Who is waiting? Why? - What did we build that nobody uses? - What failed? How do we prevent it? ``` Toyota didn't become Toyota in a day. They improved relentlessly for decades. Your platform can too. One waste at a time. Start today: Pick one workflow. Map it. Find the muda. Eliminate it. Then do it again tomorrow. --- ## Adverse Selection in Technical Debt: Why Bad Code Drives Out Good - URL: https://svalle.ru/posts/business/adverse-selection-technical-debt/ - Date: 2025-02-22 - Tags: technical-debt, economics, software-engineering, management, culture Gresham's Law: "Bad money drives out good." When two currencies circulate and one is debased (lower quality), people hoard the good currency and spend the bad. Eventually, only bad money circulates. The same thing happens with code. And it explains why codebases rot. ## The Mechanism Here's how bad code drives out good code: ### Step 1: Deadline Pressure ``` Product manager: "We need this feature by Friday." Time available: 3 days Time for good code: 5 days Time for hacky code: 2 days Developer ships hacky code. ``` ### Step 2: Hacky Code "Works" ``` Feature ships on time: ✓ Tests pass: ✓ (if there are tests) Users are happy: ✓ (for now) Manager is happy: ✓ No visible downside to cutting corners. ``` ### Step 3: Quality Is Invisible ``` Code review: - Reviewer doesn't have context on "right" design - Deadline pressure means "ship it" - Hacky code looks similar to good code - Approves the PR Management sees: - Features shipped - Deadlines met - "Productivity" metrics look great ``` ### Step 4: Perverse Incentives ``` Developer A: Writes clean, well-tested code. Takes 5 days. Developer B: Ships fast, hacky code. Takes 2 days. Who gets promoted? - B looks "more productive" - B ships more features - B gets the raise Developer A notices. ``` ### Step 5: Behavioral Shift ``` Developer A has options: 1. Keep writing good code, look slow, get passed over 2. Adapt to the system, ship fast, get promoted 3. Leave for a company that values quality Options 2 and 3 both remove good code from the system. ``` ### Step 6: Adverse Selection ``` Who stays? - Developers who write fast, hacky code (rewarded) - Developers who don't know the difference (unaware) - Developers who can't leave (stuck) Who leaves? - Developers who care about quality (frustrated) - Senior developers who see the trajectory (experienced) The talent pool degrades. ``` ### Step 7: Quality Death Spiral ``` Hacky code makes good code harder to write: - No consistent patterns to follow - Good code looks "out of place" - Integration with hacky code requires compromises - "When in Rome..." More hacky code → harder to write good code → more hacky code ``` This is Gresham's Law for software. Bad code drives out good code through adverse selection. ## The Economics of Adverse Selection Adverse selection occurs when information asymmetry causes "bad" participants to dominate a market. ### The Classic Example: Used Cars ``` Buyers can't tell good cars from lemons. Buyers assume average quality and pay average price. Sellers of good cars can't get fair price → leave market. Only lemons remain. Market collapses. ``` ### Applied to Code Quality ``` Managers can't tell good code from bad code. Managers assume average quality and reward speed. Writers of good code can't get recognition → adapt or leave. Only hacky code remains. Codebase collapses (slowly, then suddenly). ``` The information asymmetry is code quality. The market is the internal labor market for developers. The collapse is technical bankruptcy. ## Symptoms of Adverse Selection ### 1. Velocity Paradox ``` Early stage: High velocity (shipping fast) Middle stage: Velocity slows (accumulated debt) Late stage: Velocity crashes (everything is hard) But at each stage, the fastest coders are still the ones shipping hacky code. ``` ### 2. Talent Churn Pattern ``` Track who leaves: - Senior engineers: "This codebase is a mess" - Quality-focused devs: "Nobody cares about doing it right" - New hires: "This isn't what I signed up for" Track who stays: - Developers who created the mess - Developers who don't notice - Developers with no options ``` ### 3. The "Rewrite" Conversation ``` Every 2-3 years: "We should rewrite this from scratch" "The codebase is unmaintainable" "It'll be different this time" The rewrite happens. Same incentives. Same outcome. ``` ### 4. Hero Culture ``` Who gets celebrated? - Developers who ship features fast - Developers who fix urgent production issues - "10x developers" who crank out code Who doesn't? - Developers who prevent issues - Developers who refactor quietly - Developers who say "this needs more time" Hero culture rewards the behaviors that create the messes heroes fix. ``` ## Breaking the Cycle ### Strategy 1: Make Quality Visible Information asymmetry drives adverse selection. Reduce asymmetry. ``` Quality metrics: - Test coverage - Cyclomatic complexity - Dependency health - Code review thoroughness - Bug escape rate per author - Time to understand (for new devs) Make these visible: - Dashboards - PR annotations - Performance reviews ``` When quality is visible, it can be rewarded. ### Strategy 2: Fix the Incentives ``` Old incentives: Ship features fast → Promotion Write good code → Invisible New incentives: Ship features fast AND sustainably → Promotion Reduce technical debt → Recognized Prevent bugs → Celebrated Mentor others on quality → Valued ``` What you measure is what you get. ### Strategy 3: Code Review as Signaling In economics, signaling is how informed parties communicate quality. ``` Without signaling: Manager can't tell good from bad code Manager rewards speed Bad code wins With signaling (rigorous code review): Code review reveals quality Good code is distinguishable Good code can be rewarded ``` But code review only works if: - Reviewers have time and context - Reviews block bad code (not rubber-stamp) - Quality concerns are valued, not dismissed ``` Bad code review culture: "LGTM" in 5 minutes "Ship it, we're behind" Reviewers don't understand the code Good code review culture: Thorough review expected Quality concerns block merge Authors want good reviews (signaling quality) ``` ### Strategy 4: Slack Time Adverse selection accelerates under pressure. Reduce pressure. ``` No slack: Every sprint at 100% capacity No time for "doing it right" Technical debt accumulates With slack: 80% capacity planned 20% for quality, refactoring, learning Good code becomes possible ``` Google's "20% time" wasn't just about innovation—it also reduced adverse selection pressure. ### Strategy 5: Career Paths for Quality ```d2 {alt="Two career ladders side by side: the traditional ladder promotes Junior to Senior to Lead to Manager and is measured by features shipped, people managed, and scope of ownership; the quality-aware ladder promotes Junior to Senior to Staff to Principal and is measured by technical impact, code quality, system health, and mentorship of quality practices"} grid-columns: 2 grid-gap: 24 traditional: Traditional career ladder { grid-columns: 1 vertical-gap: 40 junior: Junior senior: Senior lead: Lead manager: Manager measured: "Measured by:\nFeatures shipped\nPeople managed\nScope of ownership" {style: {stroke-width: 0; fill: transparent}} junior -> senior -> lead -> manager } quality: Quality-aware ladder { grid-columns: 1 vertical-gap: 40 junior: Junior senior: Senior staff: Staff principal: Principal measured: "Measured by:\nTechnical impact\nCode quality\nSystem health\nMentorship of quality practices" {style: {stroke-width: 0; fill: transparent}} junior -> senior -> staff -> principal } ``` If senior individual contributors are rewarded for quality, quality becomes aspirational. ### Strategy 6: Hire for Quality Detection ``` Interview for: - Code quality awareness - Refactoring experience - Testing philosophy - "Tell me about a time you pushed back on shipping" Avoid: - "How fast can you solve this?" - Pure algorithm grinding - No discussion of maintainability ``` The bar you set in hiring determines the equilibrium quality level. ## The Investment Case Breaking adverse selection requires investment. Here's how to justify it: ### The Debt Accumulation Model ``` Technical debt compounds: Year 1: 10% of codebase is problematic Year 2: 20% (new debt + spread) Year 3: 35% Year 4: 55% Year 5: 75% Velocity impact: Year 1: 100% velocity Year 2: 90% velocity (10% fighting debt) Year 3: 75% velocity Year 4: 55% velocity Year 5: 30% velocity ``` ### The Intervention ROI ``` Intervention at Year 2: Investment: 20% of capacity for 6 months Result: Debt reduced from 20% to 10% Velocity preserved: 90% instead of declining Without intervention: Years 3-5 velocity: 75% → 55% → 30% Average: 53% With intervention: Years 3-5 velocity: 85% → 80% → 75% Average: 80% Velocity preserved: 27% higher average For a 20-person team at $200K/person: 20% more velocity = $800K/year in effective capacity ``` ### The Talent Retention Model ``` Adverse selection causes talent loss: Quality developers leave → replacement cost $50K each Remaining developers less productive → 20% velocity loss New hires ramp slowly in bad codebase → 50% first-year productivity If quality intervention retains 2 senior devs: Retention value: 2 × $50K = $100K Productivity preserved: 2 × $200K × 20% = $80K Annual value: $180K ``` ## Warning Signs Catch adverse selection early: | Signal | Meaning | |--------|---------| | "We don't have time for tests" | Quality is being sacrificed | | Senior devs leaving for "better engineering culture" | Adverse selection in progress | | Features ship fast but bugs increase | Quality is invisible but declining | | "We should rewrite" conversations | Debt has compounded | | New hires take longer to onboard | Codebase complexity is high | | "Hero" deployments and fixes | Celebrating symptoms, not prevention | | PRs approved in < 10 minutes | Code review isn't working | ## Summary Bad code drives out good code through adverse selection: ``` Information asymmetry: Managers can't see code quality Result: Speed is rewarded, quality is invisible Consequence: Quality-focused developers adapt or leave Equilibrium: Only hacky code remains ``` The cycle: ```d2 {alt="Cyclic feedback loop: deadline pressure leads to hacky code shipping, which looks like success, so quality developers go unrewarded and adapt or leave, codebase quality drops, more hacky code is normalized, good code gets harder to write, and the loop repeats back into deadline pressure"} # loop shape: left column flows down, right column flows up, # "Repeat" closes the cycle across the top grid-rows: 1 horizontal-gap: 72 col1: "" { style.stroke-width: 0 style.fill: transparent grid-columns: 1 vertical-gap: 44 deadline: Deadline pressure hacky: Hacky code ships success: Looks like success unrewarded: Quality developers unrewarded deadline -> hacky -> success -> unrewarded } col2: "" { style.stroke-width: 0 style.fill: transparent grid-columns: 1 vertical-gap: 44 harder: Harder to write good code normalized: More hacky code normalized drops: Codebase quality drops leave: Adapt or leave leave -> drops -> normalized -> harder } col1.unrewarded -> col2.leave col2.harder -> col1.deadline: Repeat ``` Breaking the cycle: | Strategy | How It Helps | |----------|--------------| | Make quality visible | Reduces information asymmetry | | Fix incentives | Rewards quality alongside speed | | Rigorous code review | Enables signaling of quality | | Slack time | Makes good code possible | | Career paths for quality | Makes quality aspirational | | Hire for quality | Sets the equilibrium bar | Gresham's Law isn't inevitable. But breaking it requires deliberate intervention. If you don't actively select for quality, you'll passively select against it. And eventually, only bad code will circulate. --- ## Consumer Surplus and Developer Productivity: The Hidden Value of Your Platform - URL: https://svalle.ru/posts/business/consumer-surplus-developer-productivity/ - Date: 2025-02-21 - Tags: platform-engineering, economics, developer-experience, business, productivity Your platform saves developers 2 hours per deployment. At $100/hour, that's $200 saved. But what if the developer would have gladly "paid" 4 hours to avoid the old manual process? The $200 you measured is real—but you missed another $200 of value. That hidden $200 is consumer surplus. And it explains why platforms feel essential even when the ROI spreadsheet looks marginal. ## What Is Consumer Surplus? Consumer surplus is the difference between what someone is willing to pay and what they actually pay. ``` Willingness to pay: $50 for this coffee Actual price: $5 Consumer surplus: $45 (value captured by consumer) ``` The consumer got $45 of value that never shows up in the transaction. ### Applied to Developer Platforms Developers don't pay cash for internal platforms. They "pay" in time, effort, and cognitive load. ``` Developer's willingness to "pay" (in time): "I'd spend 4 hours to set up CI/CD myself" Actual "cost" with platform: "It takes 10 minutes" Time "paid": 10 minutes Willingness to pay: 4 hours (240 minutes) Developer surplus: 230 minutes = 3.8 hours ``` Traditional ROI captures: - Time saved: 4 hours → 10 minutes = 3.8 hours saved But this assumes the "willingness to pay" equals the "old way." It often doesn't. ## Where ROI Falls Short ### The Substitution Problem ROI compares: Platform vs. The Old Way But "the old way" isn't the developer's only alternative: ``` Options without platform: A. Do it manually (4 hours) B. Skip it entirely (0 hours, but consequences) C. Use a crappy workaround (1 hour, but tech debt) D. Ask someone else to do it (30 minutes, but social cost) ``` If developers would choose option B (skip it), the platform isn't saving 4 hours—it's enabling something that wouldn't happen at all. ``` ROI calculation: 4 hours saved Reality: Feature wouldn't ship without platform Actual value: The entire feature's revenue ``` ### The Enablement Problem Some platform value isn't about doing things faster. It's about doing things that weren't possible before. ``` Before platform: - Deployments: 1/week (Friday afternoon, all hands on deck) - Experimentation: Rare (too risky) - Rollback: Manual, scary, 2-hour process After platform: - Deployments: 10/day (automated, boring) - Experimentation: Constant (easy to test and revert) - Rollback: One click, 30 seconds ``` How do you calculate ROI on "experimentation is now possible"? The surplus is enormous but unmeasured. ### The Cognitive Load Problem Developer time isn't fungible. An hour of "fighting infrastructure" ≠ an hour of "building features." ``` Developer's internal experience: Fighting infrastructure: - Frustrating - Draining - Makes them want to quit - Leads to "why am I here?" thoughts Building features: - Engaging - Energizing - Makes them want to stay - Leads to "I shipped something!" satisfaction ``` A platform that eliminates 1 hour of infrastructure fighting might free up 2 hours of productive feature work (because the developer isn't mentally drained). ROI captures: 1 hour saved Surplus includes: Energy preservation, job satisfaction, retention ## Measuring Surplus (Approximately) You can't measure surplus directly, but you can approximate it. ### Method 1: Willingness to Pay Survey Ask developers: "How much time would you spend to solve this problem without the platform?" ``` Survey question: "If the platform didn't exist, how would you handle deployments?" Responses: - 30% say "manual process, 4 hours each" - 40% say "would build something custom, 2 days" - 20% say "would push less often" - 10% say "would find another job" ``` Aggregate willingness to pay is higher than the "old way" alone suggests. ### Method 2: Revealed Preference What do developers actually do when given options? ``` Observation: Platform A: Easy to use, limited features Platform B: Hard to use, full features 80% of developers choose A, even when B would be "better" Implication: Ease of use has high surplus value Developers would "pay" a lot to avoid complexity ``` ### Method 3: Adoption Without Mandate If developers adopt the platform without being forced, surplus must exceed "cost." ``` Adoption scenarios: Mandated: Surplus unknown (could be negative) Optional, adopted: Surplus > 0 (revealed preference) Optional, viral: Surplus >> 0 (actively recommended) ``` Track organic adoption as a surplus signal. ### Method 4: Exit Interview Mining When developers leave, what do they mention? ``` "The tooling here was great, I'll miss it" → Surplus was high, platform created retention value "The infrastructure was a nightmare" → Surplus was negative, platform was a liability ``` ## Quantifying Total Platform Value Total value = Measured savings + Unmeasured surplus ### A Framework ``` Platform Value = Cost Savings + Time Savings + Enablement Value + Surplus Where: Cost Savings: Direct cost reduction (infra spend, tooling licenses) Time Savings: Hours saved × hourly rate Enablement Value: Value of things now possible that weren't before Surplus: Willingness to pay - Actual "payment" ``` ### Example Calculation **Platform: Internal CI/CD system** | Section | Item | Value | |---------|------|-------| | Measured (traditional ROI) | Time saved per deploy | 3 hours → 10 minutes | | | Deploys per month | 500 | | | Hours saved | 1,417 hours/month | | | At $100/hour | $141,700/month | | Enablement (estimated) | Additional deploys enabled | 300/month (wouldn't happen without platform) | | | Value per additional deploy | $500 (faster features, faster fixes) | | | Enablement value | $150,000/month | | Surplus (estimated) | Developer satisfaction increase | 20% (survey) | | | Retention improvement | 10% less turnover | | | Turnover cost avoided | 2 devs × $50K = $100K/year = $8,300/month | | | Cognitive load reduction | Est. 15% productivity boost | | | On 50 devs × $20K/month | $150,000/month | | Total platform value | Measured | $141,700 | | | Enablement | $150,000 | | | Surplus (retention) | $8,300 | | | Surplus (cognitive) | $150,000 | | | **Total** | **$450,000/month** | **Traditional ROI saw:** $141,700/month. **Actual value:** $450,000/month (3.2x higher). ## Why Surplus Matters for Platform Teams ### Budget Conversations Traditional pitch: > "The platform saves 1,400 hours per month, worth $142K." Surplus-aware pitch: > "The platform saves $142K in direct time savings. But it also enables 300 additional deployments worth $150K in faster delivery, and improves developer satisfaction enough to reduce turnover and boost productivity—conservatively another $150K. Total value is closer to $450K." ### Build vs Buy Decisions ``` Build platform in-house: $200K/year Buy commercial platform: $300K/year Difference: $100K Traditional analysis: "Build, it's $100K cheaper" Surplus analysis: Commercial platform: - Better UX = higher developer surplus - Less cognitive load = more productivity - Proven = less risk Estimated surplus difference: $150K/year Total: Buy is actually $50K better when surplus is included. ``` ### Feature Prioritization ``` Feature A: Saves 100 hours/month ($10K) Feature B: Reduces friction (no direct time savings) Traditional analysis: Feature A wins Surplus analysis: Feature B: - Eliminates top developer complaint - Would "pay" 200 hours of frustration to remove - Surplus value: $20K equivalent Feature B actually creates more value. ``` ## The Surplus Signals Since you can't measure surplus directly, track leading indicators: ### Developer NPS ``` "On a scale of 0-10, how likely are you to recommend this platform to a colleague?" NPS correlates with surplus: High NPS: Surplus exceeds expectations Low NPS: Surplus below expectations (or negative) ``` ### Organic Adoption Rate ``` Teams using platform voluntarily: Month 1: 5 teams (mandated) Month 3: 12 teams (word of mouth) Month 6: 25 teams (everyone wants in) Rising organic adoption = surplus is real and significant ``` ### Support Ticket Sentiment ``` Tickets about platform: "How do I do X?" - Neutral "X is broken" - Negative "Can you add Y? I love this thing" - Positive surplus signal ``` ### Workaround Detection ``` Are developers building workarounds? Yes → Platform surplus is negative in that area No → Platform is sufficient ``` Workarounds reveal where developers' willingness to pay exceeds what the platform provides. ## Capturing More Surplus If surplus exists, you can try to capture it: ### Option 1: Internal Pricing Charge teams for platform usage: | Item | Value | |------|-------| | Platform cost | $500K/year | | Teams served | 50 | | Flat fee | $10K/team/year | | **Alternative: Usage-based pricing** | | | Per deployment | $5 | | Per environment | $100/month | | If teams willingly pay | → Surplus exceeded price | | If adoption drops | → Price exceeded surplus | Internal pricing reveals willingness to pay. ### Option 2: Premium Tiers Offer free and premium versions: ``` Free tier: - Basic CI/CD - Shared resources - Best-effort support Premium tier ($500/month): - Advanced features - Dedicated resources - Priority support ``` Who upgrades? Teams with high surplus for those features. ### Option 3: Expand the Platform If surplus exists, there's room to add features: ``` Current platform: CI/CD Developer surplus: High (they love it) Opportunity: Add deployment environments - Developers would "pay" a lot for easy environments - Currently hacking it themselves - Adding this captures more surplus (creates more value) ``` ## Summary Consumer surplus is the hidden value of your platform—what developers would pay minus what they actually pay. | What ROI Measures | What Surplus Adds | |-------------------|-------------------| | Time savings | Willingness to pay (often higher) | | Cost reduction | Enablement (new things possible) | | Efficiency | Cognitive load reduction | | Direct output | Retention value | | Attrition rate | Satisfaction value | Why it matters: ``` Platform value = Measured ROI + Unmeasured Surplus For most platforms: Measured ROI: $X Unmeasured Surplus: $2-3X Total value is often 3-4x what the spreadsheet shows. ``` Signals of surplus: - High NPS - Organic adoption - "Can you add..." requests - Absence of workarounds - Developer retention If your platform feels essential but ROI looks marginal, the surplus is probably enormous. You're just not measuring it. And that surplus is real value—even if it never shows up on a spreadsheet. --- ## Inventory Theory for Compute Capacity: How Much Buffer Should You Hold? - URL: https://svalle.ru/posts/business/inventory-theory-cloud-capacity/ - Date: 2025-02-20 - Tags: cloud, finops, capacity-planning, operations, business, infrastructure Your cloud capacity is inventory. Reserved instances are safety stock. Spot instances are just-in-time procurement. Auto-scaling is your reorder system. And like any inventory problem, you're balancing two risks: too much (waste) and too little (stockouts). Operations research solved these problems decades ago. Let's apply inventory theory to cloud capacity planning. ## The Inventory Parallel | Inventory Concept | Cloud Equivalent | |-------------------|------------------| | Safety stock | Reserved capacity headroom | | Cycle stock | Baseline committed instances | | Pipeline inventory | Instances being provisioned | | Seasonal stock | Pre-scaled capacity for known peaks | | Stockout | Throttling, 503s, outages | | Holding cost | Paying for idle resources | | Ordering cost | Provisioning overhead, cold starts | Once you see cloud through this lens, classic inventory models become directly applicable. ## The Fundamental Trade-off Every inventory problem balances two costs: ### Cost of Too Little (Understocking) ``` Capacity < Demand: - Requests throttled or dropped - Latency spikes (degraded experience) - Revenue lost (checkout failures, abandoned sessions) - SLA breaches (penalties, credits) - Reputation damage (customers remember outages) ``` Quantifying stockout cost: ``` Stockout cost per hour: Revenue at risk: $50,000/hour Probability of stockout: 5% (given current buffer) Expected stockout cost: $2,500/hour of exposure ``` ### Cost of Too Much (Overstocking) ``` Capacity > Demand: - Paying for idle resources - Capital tied up (opportunity cost) - Committed to wrong instance types - Harder to migrate (locked into reservations) ``` Quantifying holding cost: ``` Excess capacity cost: Reserved instances unused: 20% of fleet Monthly reserved spend: $100,000 Waste: $20,000/month ``` ### The Optimization ``` Total Cost = Holding Cost + Stockout Cost Minimize Total Cost by finding optimal inventory level ``` This is the core of inventory theory—and it applies directly to capacity planning. ## Model 1: Safety Stock for Capacity Headroom Safety stock protects against demand variability. In cloud terms: how much headroom above expected peak? ### The Classic Formula ``` Safety Stock = z × σ × √L Where: z = service level factor (e.g., 1.65 for 95%, 2.33 for 99%) σ = standard deviation of demand L = lead time to replenish ``` ### Applied to Cloud ``` Capacity Headroom = z × σ_demand × √(scale_up_time) Example: Target availability: 99.9% (z = 3.09) Demand std dev: 1,000 requests/sec Time to scale up: 5 minutes = 0.083 hours Headroom = 3.09 × 1,000 × √0.083 = 3.09 × 1,000 × 0.29 = 896 requests/sec of buffer capacity ``` If your current capacity handles 10,000 req/sec and peak demand averages 9,000 req/sec with σ=1,000, you need ~900 req/sec headroom to hit 99.9% availability. ### The Insight Headroom depends on: 1. **Variability (σ)**: More variable demand → more headroom needed 2. **Lead time**: Slower scaling → more headroom needed 3. **Service level**: Higher availability target → more headroom needed ``` Reduce variability: Smooth traffic (rate limiting, queuing) Reduce lead time: Faster auto-scaling, warm pools Accept lower service: Maybe 99.5% is enough? ``` Each approach reduces required headroom differently. ## Model 2: Economic Order Quantity for Reserved Instances EOQ answers: what's the optimal order size balancing ordering costs and holding costs? ### The Classic Formula ``` EOQ = √(2DS/H) Where: D = annual demand S = ordering/setup cost per order H = holding cost per unit per year ``` ### Applied to Reserved Instances The question: How much capacity should we commit to in reserved instances vs keeping flexible? ``` Commitment size = √(2 × Annual Compute Demand × Commitment Overhead / Flexibility Premium) Where: Annual Compute Demand: Total compute-hours needed Commitment Overhead: Cost of managing reservations, forecasting, etc. Flexibility Premium: On-demand price - Reserved price (what you pay for flexibility) ``` ### Practical Framing More useful framing for cloud: ``` Reserved vs On-Demand Decision: Reserved instance cost: $0.40/hour (1-year commit) On-demand cost: $1.00/hour Break-even utilization: 40% If utilization > 40%: Reserve If utilization < 40%: On-demand ``` But this ignores uncertainty. What if demand drops? ``` Expected value calculation: Scenario A (80% prob): Demand stays high Reserved cost: $0.40 × 8760 hours = $3,504 On-demand cost: $1.00 × 8760 = $8,760 Savings: $5,256 Scenario B (20% prob): Demand drops 50% Reserved cost: $3,504 (still committed) On-demand cost: $1.00 × 4380 = $4,380 Loss: $876 (paid for unused capacity) Expected value of reserving: 0.8 × $5,256 + 0.2 × (-$876) = $4,030 expected savings Reserve if expected savings > 0 ``` ### The Insight Optimal reservation depends on: 1. **Demand certainty**: More certain → reserve more 2. **Discount depth**: Bigger discount → reserve more 3. **Commitment length**: Longer commitment → need more certainty ``` High certainty + deep discount: Reserve aggressively (70-80% of base) Moderate certainty: Reserve conservatively (50-60%) High uncertainty: Minimize commitments, stay flexible ``` ## Model 3: Newsvendor for Spot Instance Buffers The newsvendor problem: how much to stock when demand is uncertain and leftovers have salvage value? Classic example: newspaper vendor deciding how many papers to buy. Too few = missed sales. Too many = unsold papers. ### The Classic Formula ``` Optimal quantity where: P(Demand ≤ Q*) = (p - c) / (p - s) Where: p = selling price (revenue per unit) c = cost per unit s = salvage value (what you get for excess) ``` ### Applied to Spot Buffer Capacity Spot instances are like newsvendor inventory: - You acquire them speculatively - If demand materializes, they generate value - If not, you've paid for nothing (salvage = 0, or you can release them) ``` Spot buffer decision: Value if used (p): $1.00/hour of revenue protected Cost of spot (c): $0.30/hour Salvage if unused (s): $0.00 (can terminate, pay nothing more) Critical ratio = (p - c) / (p - s) = ($1.00 - $0.30) / ($1.00 - $0.00) = 0.70 Stock spot capacity at the 70th percentile of demand distribution ``` ### The Insight This means: if protecting $1 of revenue costs $0.30 in spot capacity, you should provision enough spot to cover 70% of the demand distribution—not 95% or 99%. Why? Because the marginal cost of protection ($0.30) exceeds the marginal benefit once you're past the 70th percentile. ``` Demand distribution: 50th percentile: 8,000 req/sec 70th percentile: 9,500 req/sec 90th percentile: 12,000 req/sec 99th percentile: 15,000 req/sec Optimal spot buffer: Cover up to 9,500 req/sec Above that: Accept some throttling (it's not economical to buffer) ``` This is counterintuitive. We're trained to think "always provision for peak." But the math says: provision for the economically optimal point, which is often well below peak. ## Model 4: Reorder Point for Auto-Scaling When should you trigger scaling? Too early wastes money. Too late causes stockouts. ### The Classic Formula ``` Reorder Point = Expected demand during lead time + Safety stock ROP = d × L + z × σ × √L Where: d = average demand rate L = lead time z = service level factor σ = demand standard deviation ``` ### Applied to Auto-Scaling Triggers ``` Scale-up trigger point: Average demand: 8,000 req/sec Current capacity: 10,000 req/sec Time to scale up: 3 minutes Demand variability (σ): 500 req/sec Target service level: 99% (z = 2.33) Demand during scale-up = 8,000 × (3/60) = 400 requests Safety buffer = 2.33 × 500 × √(3/60) = 260 req/sec equivalent Trigger scale-up when: Current utilization approaches (Capacity - Safety buffer) / Capacity = (10,000 - 260) / 10,000 = 97.4% But that's too late! We need to trigger earlier to account for lead time. Better: Trigger at 80% utilization to give scaling time to complete. ``` ### Dynamic Reorder Points Sophisticated systems adjust triggers based on: ``` Time of day: Peak hours: Trigger at 70% utilization (more buffer) Off-peak: Trigger at 85% utilization (less buffer needed) Demand trend: Demand rising: Trigger earlier Demand falling: Trigger later (avoid over-provisioning) Recent variability: High variance: Trigger earlier Stable: Trigger later ``` ## Model 5: Service Level Targeting How do you choose the right service level? Higher isn't always better. ### The Cost Trade-off ``` Service Level Capacity Needed Cost Stockout Risk 90% 100 units $100K 10% 95% 115 units $115K 5% 99% 140 units $140K 1% 99.9% 175 units $175K 0.1% 99.99% 220 units $220K 0.01% ``` Each "9" costs more. Is it worth it? ### The Calculation Optimal service level where: marginal cost of capacity = marginal reduction in stockout cost. | | Going from 99% to 99.9% | Going from 99.9% to 99.99% | |---|---|---| | Additional capacity cost | $35K/year | $45K/year | | Stockout probability reduction | 0.9% | 0.09% | | Annual stockout events avoided | 0.9% × 365 = 3.3 days | 0.09% × 365 = 0.33 days | | Cost per stockout day | $50K | $50K | | Stockout cost avoided | 3.3 × $50K = $165K | 0.33 × $50K = $16K | | **ROI** | **$165K / $35K = 4.7x → Worth it** | **$16K / $45K = 0.36x → Not worth it** | ### The Insight There's an economically optimal service level. It's often lower than engineers instinctively want. ``` The 99.99% myth: Engineers: "We need five nines!" Finance: "What does that cost?" Engineers: "Whatever it takes." Math: "The marginal value of the 5th nine is $3K. The marginal cost is $200K." ``` Pick your service level based on stockout cost, not engineering pride. ## Putting It Together: A Capacity Framework ### Step 1: Understand Your Demand ``` Collect data: - Average demand by hour/day/week - Standard deviation of demand - Peak demand events (frequency, magnitude) - Trend (growing, stable, declining) ``` ### Step 2: Quantify Your Costs ``` Stockout costs: - Revenue per request - SLA penalty per hour of degradation - Customer churn from poor experience - Reputation/brand impact Holding costs: - Reserved instance rates - On-demand rates - Opportunity cost of capital ``` ### Step 3: Calculate Optimal Levels ``` Base capacity: Reserve instances covering ~60% of average demand Headroom: Safety stock formula for variability Spot buffer: Newsvendor model for peak coverage Trigger points: Reorder point model for auto-scaling Service level: Marginal cost = marginal benefit analysis ``` ### Step 4: Build the Portfolio ``` Capacity portfolio: Reserved (1-year): 50% of base (predictable, cheap) Reserved (flexible): 20% of base (some flexibility) On-demand: 20% of base (full flexibility) Spot: 10% buffer (opportunistic) Auto-scale headroom: Covers demand spikes up to service level target ``` ## Common Mistakes ### Mistake 1: Targeting 100% Availability ``` 100% availability requires infinite capacity. Infinite capacity costs infinite money. Pick a service level and optimize for it. ``` ### Mistake 2: Ignoring Lead Time ``` "We have auto-scaling, we're fine." But auto-scaling takes 3-5 minutes. In 3 minutes at 10K req/sec, that's 1.8M requests. If you're at capacity when scaling triggers, those requests fail. Buffer for lead time. ``` ### Mistake 3: Static Thresholds ``` Scaling at 80% utilization always: - Wasteful at 3am (demand is low, 80% is fine) - Dangerous before Black Friday (should scale earlier) Dynamic thresholds based on context. ``` ### Mistake 4: Treating All Capacity as Equal ``` Not all capacity is fungible: - GPU instances vs CPU - Memory-optimized vs compute-optimized - Regional capacity constraints Model each capacity type separately. ``` ## Summary Cloud capacity is inventory. Inventory theory applies: | Problem | Model | Key Insight | |---------|-------|-------------| | How much headroom? | Safety stock | Depends on variability, lead time, service level | | How much to reserve? | EOQ / Expected value | Balance discount vs flexibility | | How much spot buffer? | Newsvendor | Optimal is often below peak—it's not economical to cover everything | | When to scale? | Reorder point | Trigger early enough for lead time | | What service level? | Marginal analysis | Each "9" has diminishing returns | The capacity planning mindset: ``` Old thinking: "Provision for peak. Add buffer. Don't run out." Inventory thinking: "Balance holding costs vs stockout costs. Find the economically optimal point. Accept that some stockout risk is rational." ``` You wouldn't run a warehouse by saying "stock infinite inventory so we never run out." Don't run cloud capacity that way either. Find your optimal inventory level. It's probably lower than you think—and that's okay. --- ## Multi-Cloud: Hedging Strategy or Wasted Optionality? - URL: https://svalle.ru/posts/business/multi-cloud-hedging/ - Date: 2025-02-19 - Tags: multi-cloud, cloud, strategy, business, finops, infrastructure "Don't put all your eggs in one basket." This folk wisdom drives most multi-cloud strategies. Spread across AWS, GCP, and Azure. Avoid vendor lock-in. Maintain optionality. It sounds smart. But in finance, hedging has a cost. The question isn't whether diversification is good—it's whether the hedge is worth what you're paying for it. Let's apply portfolio theory to cloud strategy and find out. ## The Multi-Cloud Pitch The case for multi-cloud: **1. Risk diversification** ``` Single cloud: AWS outage = you're down Multi-cloud: AWS outage = failover to GCP ``` **2. Negotiating leverage** ``` "We're evaluating moving 30% of workloads to Azure" AWS sales rep: "Let me see what discounts I can find" ``` **3. Avoid lock-in** ``` AWS changes pricing: You can migrate AWS deprecates service: You have alternatives AWS relationship sours: You're not trapped ``` **4. Best-of-breed** ``` Compute: AWS (mature, broad) ML: GCP (TPUs, Vertex) Enterprise: Azure (Office 365 integration) ``` This all sounds reasonable. But let's look at the costs. ## The Real Costs of Multi-Cloud ### Operational Overhead Every cloud requires: ``` Per cloud: - IAM and security policies - Networking configuration - Monitoring and alerting - Incident runbooks - Cost management - Compliance documentation - Team expertise Single cloud: 1x operational burden Dual cloud: 2.5x operational burden (not 2x—there's overhead in the seams) Triple cloud: 4x+ operational burden ``` The ops team doesn't scale linearly with clouds. Complexity multiplies. ### The Abstraction Tax To be truly portable, you need abstraction layers: ``` Without abstraction: AWS Lambda → tightly coupled, uses all features Performance: Optimal Velocity: Fast With abstraction (for portability): Generic FaaS wrapper → works on Lambda, Cloud Functions, Azure Functions Performance: Degraded (lowest common denominator) Velocity: Slower (maintaining abstraction layer) ``` The abstraction tax: | Layer | Overhead | |-------|----------| | Compute abstraction (Kubernetes) | 15-30% complexity increase | | Database abstraction | 20-40% feature loss | | Serverless abstraction | 30-50% capability reduction | | ML platform abstraction | Often impossible | You end up using 60% of each cloud's capabilities instead of 100% of one. ### Lost Features Cloud providers differentiate with proprietary services: ``` AWS-only: - Aurora Serverless (auto-scaling PostgreSQL) - Lambda@Edge (edge compute) - DynamoDB (managed NoSQL at scale) GCP-only: - BigQuery (serverless analytics) - Spanner (global SQL) - TPUs (ML acceleration) Azure-only: - Cosmos DB (multi-model global) - Cognitive Services (pre-built AI) - Synapse (unified analytics) ``` Multi-cloud means either: - Avoiding these services (competitive disadvantage) - Using them anyway (not actually portable) ### Team Cognitive Load ``` Single cloud engineer: - Deep expertise in one ecosystem - Knows all the gotchas - Can optimize aggressively Multi-cloud engineer: - Shallow expertise across ecosystems - Misses platform-specific optimizations - Context switches constantly ``` You either hire specialists for each cloud (expensive) or generalists who are mediocre at all of them. ### Quantifying the Cost ``` Multi-cloud operational overhead: Additional headcount: 2 FTE ($500K/year) Abstraction layer maintenance: 1 FTE ($250K/year) Lost feature productivity: 20% slower ($400K equivalent) Suboptimal architecture: 15% higher cloud spend ($150K/year) Training and certification: $50K/year Total multi-cloud tax: $1.35M/year ``` That's the cost of your hedge. Now, what's the benefit? ## Portfolio Theory Basics In finance, diversification reduces risk. But it's not free: ### Diversification and Correlation ``` Portfolio risk = f(individual risks, correlations) If assets are uncorrelated: Diversification significantly reduces risk If assets are correlated: Diversification provides less benefit ``` ### The Efficient Frontier {{< chart "multi-cloud-hedging--efficient-frontier" "Efficient frontier chart of return versus risk: a curve of portfolios rises to an optimal portfolio at the peak, while individual assets sit scattered below and to the right" >}} You want maximum return for given risk. Adding assets helps only if they improve this trade-off. ### Cost of Hedging Hedges aren't free: ``` Options premium: The price of having the right to sell Insurance premium: The price of protection Hedge funds: 2 and 20 for "protection" ``` A hedge is worth buying only if: **Hedge value > Hedge cost** ## Applying Portfolio Theory to Cloud ### How Correlated Are Cloud Outages? If AWS and GCP are truly uncorrelated, multi-cloud provides strong protection: ``` AWS availability: 99.99% GCP availability: 99.99% P(both down): 0.0001 × 0.0001 = 0.00000001 (one in 100 million) ``` But are they actually uncorrelated? **Correlated failure modes:** ``` Internet backbone issues: Affects all clouds DNS failures: Affects all clouds BGP misconfigurations: Affects all clouds Submarine cable cuts: Affects regional multi-cloud Software supply chain: Log4j hit everyone Major security events: Industry-wide impact ``` **Semi-correlated:** ``` Region-specific events: - Power grid failures - Natural disasters - Government actions These affect one cloud's region but multi-region within that cloud also protects. ``` **Actually uncorrelated:** ``` Cloud-specific bugs: - AWS S3 outage (2017) - GCP networking issue (2019) - Azure AD outage (2021) These are genuinely independent. ``` Real correlation is probably 0.3-0.5, not 0.0. Multi-cloud helps less than it appears. ### The Real Risk Reduction ``` Scenario: AWS us-east-1 has major outage Single-cloud (AWS): Multi-AZ: Still down Multi-region: Protected Protection level: ~95% Multi-cloud: Failover to GCP: Protected Protection level: ~99% Incremental protection: 4% ``` You're paying the multi-cloud tax for 4% incremental protection over multi-region single-cloud. ### Valuing the Optionality The "right to switch clouds" is an option. Options have value based on: ``` Option value = f(volatility, time, strike price) In cloud terms: Volatility: How likely is the scenario where you need to switch? Time: How long do you have this option? Strike price: What does it cost to exercise (actually migrate)? ``` **When is the option valuable?** High volatility scenarios: ``` - Cloud provider might exit market (unlikely for AWS/GCP/Azure) - Regulatory change forces migration (possible) - Pricing becomes uncompetitive (rare, easily predicted) - Relationship breakdown (very rare) ``` Low volatility reality: ``` - AWS has existed for 18 years - No major cloud has exited - Pricing generally decreases over time - Lock-in concerns rarely materialize ``` **The strike price problem:** Even with the "option" to switch, exercising it is expensive: ``` Migration cost estimate: Planning: 3 months Execution: 6-12 months Team retraining: 3 months Productivity loss: 30% during migration Bug fixes post-migration: 6 months For a $10M/year cloud spend company: Migration project: $2-5M Lost productivity: $3M Risk of failure: 20% Total cost to exercise: $5-8M ``` An option you can't afford to exercise isn't worth much. ### Multi-Cloud as Negotiating Leverage "We'll move to GCP if you don't give us a discount." Does this work? ``` Credible threat: You have workloads on GCP already: Yes, works You've never used GCP: They know you're bluffing Effective leverage: "We're moving 20% of new workloads to GCP" Real pressure "We might move someday" No pressure ``` Multi-cloud for negotiating leverage only works if you actually run workloads there. The threat has to be credible. **The discount math:** ``` Cloud spend: $10M/year Discount from negotiation: 15% = $1.5M/year Multi-cloud operational cost: $1.35M/year Net benefit: $150K/year ``` You might break even. Maybe. ## When Multi-Cloud Actually Makes Sense ### Regulatory Requirements ``` EU data residency: Must use EU regions Government contracts: Specific cloud requirements Industry compliance: Sometimes mandates diversity "We're multi-cloud for compliance" is legitimate. ``` ### M&A Integration ``` Your company: AWS Acquired company: GCP Options: A. Migrate them to AWS ($5M, 18 months) B. Run multi-cloud (operationally complex) C. Keep separate (integration limited) Multi-cloud may be the pragmatic answer during integration. ``` ### Genuinely Best-of-Breed ``` Core workloads: AWS (your team knows it) Data analytics: GCP BigQuery (genuinely superior) ML training: GCP TPUs (no AWS equivalent) Using GCP for specific workloads where it's clearly better ≠ multi-cloud strategy It's just using the right tool for the job. ``` ### Extreme Scale At very large scale, concentration risk matters more: ``` $500M/year cloud spend: - 10% discount negotiation = $50M/year - Multi-cloud ops overhead = $5M/year - Net benefit = $45M/year The math changes at scale. ``` ### Data and Egress Leverage ``` Strategy: Run compute on one cloud, but keep data portable Data layer: Multi-cloud capable (Snowflake, Databricks) Compute layer: Single cloud (AWS) Benefits: - Data portability for negotiation - No multi-cloud compute complexity - Credible migration threat for data workloads ``` This hybrid approach captures leverage without full multi-cloud tax. ## When Single-Cloud Wins ### Speed and Velocity ``` Single cloud team: - Knows the platform deeply - Uses managed services aggressively - Ships features faster Multi-cloud team: - Maintains abstraction layers - Debates "what if we need to migrate" - Ships features slower ``` Velocity often matters more than optionality. ### Depth Over Breadth ``` AWS Lambda + DynamoDB + API Gateway + Step Functions: - Deeply integrated - Optimized together - Powerful patterns Generic FaaS + Generic DB + Generic API: - Loosely integrated - Impedance mismatches - Weaker patterns ``` Deep platform expertise beats shallow multi-platform knowledge. ### Operational Simplicity ``` 3am incident: Single cloud: "It's an AWS issue. Check AWS status page. Page the AWS team." Multi-cloud: "Is it AWS or GCP? Check both. Different runbooks. Different tooling. Different on-call rotations. Is it the abstraction layer?" ``` Complexity is the enemy of reliability. ### The Startup Case ``` Startup resources: 5 engineers, $500K/year cloud spend Multi-cloud: - 40% of time on infrastructure portability - 60% of time on product Single cloud: - 15% of time on infrastructure - 85% of time on product Multi-cloud costs you 25% of your engineering capacity. You're trading product velocity for theoretical future optionality. ``` Startups should almost never be multi-cloud. ## The Decision Framework ### Calculate Your Hedge Cost ``` Multi-cloud operational overhead: $X/year Abstraction layer maintenance: $Y/year Lost productivity from complexity: $Z/year Total hedge cost: $(X+Y+Z)/year ``` ### Estimate Your Hedge Value ``` P(need to migrate): A% Cost of emergency migration: $B P(outage only multi-cloud prevents): C% Cost of that outage: $D Negotiation leverage value: $E Expected hedge value: A×B + C×D + E ``` ### Compare ``` If hedge value > hedge cost: Multi-cloud may be justified If hedge value < hedge cost: Single cloud is better ``` For most companies, the math doesn't work. ## The Honest Answer Multi-cloud is like buying insurance on your insurance: | Scenario | Recommendation | |----------|----------------| | Startup/SMB | Single cloud. Velocity matters more. | | Mid-size | Single cloud + portable data layer | | Enterprise | Multi-cloud for leverage if spend > $50M | | Regulated | Multi-cloud if required by compliance | | M&A heavy | Multi-cloud may be unavoidable | The default should be single-cloud. Multi-cloud is the exception requiring justification, not the other way around. ## Summary Multi-cloud is sold as risk management. But portfolio theory tells us: | Factor | Reality | |--------|---------| | Diversification benefit | Limited—cloud outages are partially correlated | | Optionality value | Low—switching costs make the option hard to exercise | | Hedge cost | High—operational overhead, abstraction tax, lost features | | Negotiating leverage | Real but requires credible threat | The multi-cloud math: ``` Multi-cloud value = Risk reduction + Negotiating leverage + Optionality = (Limited) + (Moderate) + (Low) Multi-cloud cost = Ops overhead + Abstraction tax + Lost features = (High) + (High) + (High) For most companies: Cost > Value ``` Multi-cloud is like paying insurance premiums for a policy you'll probably never claim, and if you did claim it, the deductible would be enormous. Sometimes that insurance is worth it. Usually, you're just paying premiums. Before going multi-cloud, do the math. Your "hedge" might be more expensive than the risk you're hedging against. --- ## Platform & Infra Teams: Cost Center or Profit Enabler? - URL: https://svalle.ru/posts/business/platform-infra-cost-center/ - Date: 2025-02-18 - Tags: platform-engineering, infrastructure, business, strategy, leadership Finance sees the platform team: 8 engineers, $2M/year, no direct revenue. Finance sees the product team: 8 engineers, $2M/year, ships features that drive $10M revenue. Budget cut time. Which team gets reduced? This is the cost center trap. And it's killing platform and infrastructure teams that are actually driving massive value—just invisibly. ## The Cost Center Framing Cost centers are organizational units that don't generate revenue directly: ``` Revenue generators: Sales, Product, Marketing Cost centers: IT, HR, Legal, Finance... Platform, Infrastructure ``` Cost center thinking: ``` "How do we minimize this cost?" "Can we do this cheaper?" "What's the minimum viable investment?" ``` When budgets tighten, cost centers get cut first. They don't "make money." ## The Problem with Cost Center Thinking ### It Ignores Leverage Platform team: 8 engineers, $2M/year Product teams enabled: 100 engineers, $25M/year Without the platform team: - Product engineers spend 30% of time on infrastructure - Effective product capacity: 70 engineers - Lost capacity: 30 engineer-equivalents = $7.5M/year With the platform team: - Product engineers spend 5% of time on infrastructure - Effective product capacity: 95 engineers - Platform cost: $2M/year - Net gain: $5.5M/year in product capacity The platform team isn't costing $2M. It's generating $5.5M in leverage. ### It Ignores Enablement What the platform team enables: ``` Without platform: - Deploy time: 4 hours - Deploy frequency: Weekly - New service setup: 2 weeks - Incident response: Ad hoc With platform: - Deploy time: 10 minutes - Deploy frequency: Daily - New service setup: 1 day - Incident response: Automated runbooks ``` These improvements translate to: - Faster time to market - More experiments run - Quicker customer feedback - Lower incident costs None of this shows up as "revenue from platform team." ### It Ignores Risk Reduction Infrastructure team prevents: ``` Avoided incidents: $500K/year (estimated) Avoided security breaches: $2M/year (expected value) Avoided compliance failures: $1M/year (fines, audit costs) Avoided scaling failures: $1M/year (lost revenue) Total risk reduction: $4.5M/year Team cost: $1.5M/year ROI: 200% ``` But "disasters prevented" don't appear on financial statements. ## Reframing: The Leverage Model Instead of cost centers, think leverage: ``` Leverage ratio = Output enabled / Input cost Platform team: Input: $2M/year (team cost) Output: $7.5M/year (productivity unlocked) Leverage: 3.75x Compare to: Product team: Input: $2M/year Output: $2M/year (their direct output) Leverage: 1x ``` The platform team has higher leverage than a product team—they multiply output rather than adding to it. ### The Multiplier Mental Model ``` Product engineer output: 1x their salary in value Platform engineer output: Nx product engineer productivity If platform engineer improves 20 product engineers by 10%: Value created = 20 × $250K × 10% = $500K Per platform engineer = $500K (2x their salary) If platform engineer improves 50 product engineers by 20%: Value created = 50 × $250K × 20% = $2.5M Per platform engineer = $2.5M (10x their salary) ``` Platform engineers are force multipliers. ## Measuring Platform Value ### Developer Velocity Metrics ``` Deployments per developer per week: Before platform: 0.5 After platform: 3 Improvement: 6x Lead time (commit to production): Before: 2 weeks After: 2 hours Improvement: 84x Change failure rate: Before: 15% After: 3% Improvement: 5x ``` These are DORA metrics—industry-standard measures of software delivery performance. ### Time Savings ``` Activity Before After Savings Engineers --------- ------ ----- ------- --------- Environment setup 2 days 1 hour 15 hours 50 Deploy to production 4 hours 10 min 3.8 hours 50 × 3/week Debug infrastructure 5 hours 1 hour 4 hours 10/week New service creation 2 weeks 1 day 9 days 5/month Incident response 3 hours 30 min 2.5 hours 10/month Annual hours saved: Environment: 50 × 15 = 750 hours Deploy: 50 × 3 × 52 × 3.8 = 29,640 hours Debug: 10 × 52 × 4 = 2,080 hours New service: 5 × 12 × 72 = 4,320 hours Incidents: 10 × 12 × 2.5 = 300 hours Total: 37,090 hours = 18 FTE equivalent Value at $250K/FTE: $4.5M/year ``` ### Revenue Attribution Harder but possible: ``` Feature velocity increase: 2x Additional features shipped: 20/year Average feature revenue impact: $100K Additional revenue from velocity: $2M/year Time to market improvement: 50% faster Competitive wins from speed: 5 deals Average deal size: $200K Revenue from speed: $1M/year ``` ### Cost Avoidance ``` Incidents prevented: Average incident cost: $50K Incidents/year without platform: 20 Incidents/year with platform: 5 Cost avoided: 15 × $50K = $750K/year Infrastructure efficiency: Without platform: $500K/month cloud spend With platform optimization: $350K/month Annual savings: $1.8M/year Headcount avoided: Without platform: Each team needs 0.5 FTE for infra Teams: 20 Headcount avoided: 10 FTE Cost avoided: $2.5M/year ``` ## The Attribution Problem The challenge: platform value is distributed across teams. ``` Product team ships feature → Revenue increases Who gets credit? - Product team (built the feature) - Platform team (enabled fast shipping) - Infrastructure team (kept it running) ``` ### Solution 1: Agreed Allocation ``` Feature success attribution: - Product team: 70% - Platform team: 20% - Infrastructure team: 10% If feature drives $1M: - Product: $700K - Platform: $200K - Infrastructure: $100K ``` ### Solution 2: Counterfactual Comparison Compare teams with and without platform support: ``` Team A (on platform): - Deploys per week: 15 - Lead time: 2 hours - Features shipped: 5/quarter Team B (not on platform): - Deploys per week: 3 - Lead time: 3 days - Features shipped: 2/quarter Platform impact: 2.5x feature velocity ``` ### Solution 3: Internal Pricing Treat platform as internal service with pricing: ``` Platform services: - CI/CD pipeline: $5K/team/month - Kubernetes namespace: $2K/team/month - Observability stack: $3K/team/month - Total: $10K/team/month 20 teams × $10K × 12 months = $2.4M internal revenue Platform team cost: $2M "Profit": $400K ``` This makes platform value visible in a language finance understands. ## Presenting to Leadership ### Don't Say This ``` "Platform team maintains Kubernetes and CI/CD" "We handle infrastructure so product teams don't have to" "Our job is to keep things running" ``` This sounds like overhead. ### Say This ``` "Platform team enables 100 product engineers to ship 3x faster" "We convert $2M in platform investment into $6M in productivity gains" "Every platform engineer creates $500K in developer time savings" "We reduced time-to-market by 50%, winning 5 competitive deals worth $1M" ``` This sounds like investment. ### The Executive Summary ``` Platform & Infrastructure Investment Investment: $3.5M/year (12 engineers) Returns: Developer productivity: $4.5M (18 FTE equivalent freed) Incident prevention: $750K Infrastructure efficiency: $1.8M Headcount avoidance: $2.5M Velocity-driven revenue: $2M Total return: $11.5M/year ROI: 229% Payback period: 4 months Comparison: Cutting team saves: $3.5M Cutting team costs: $11.5M in lost value Net loss from cutting: $8M/year ``` ## Protecting Platform Investment ### Tie to Business Metrics Connect platform metrics to business outcomes: ```d2 {alt="Causal value chain: platform improvements drive deploy frequency up (the platform metric), which drives time to market down (the business metric), which drives competitive win rate up (the business outcome)"} grid-rows: 1 grid-gap: 24 improves: Platform improves {style: {stroke-width: 0; fill: transparent}} deploy: "Deploy frequency up\n(Platform metric)" ttm: "Time to market down\n(Business metric)" win: "Competitive win rate up\n(Business outcome)" improves -> deploy -> ttm -> win ``` ### Create Champions Product teams that benefit should advocate: ``` "Since adopting the platform, my team ships 3x faster" "I couldn't hit my OKRs without the platform" "Please don't cut the team that makes us productive" ``` Champions are more credible than self-reporting. ### Show the Counterfactual ``` "What happens if we cut the platform team?" Month 1: Product teams absorb infrastructure work Month 3: 30% of product capacity now on infrastructure Month 6: Deployment velocity drops 50% Month 9: Incidents increase 3x (no dedicated response) Month 12: Key engineers leave (frustrated with toil) Cost of cutting: > Cost of keeping ``` ### Benchmark Against Alternatives ``` Option A: Internal platform team Cost: $2M/year Features: Tailored to our needs Support: Immediate Option B: Buy commercial platform Cost: $1.5M/year licensing + $500K integration Features: Generic Support: Vendor SLA Option C: Each team DIY Cost: $7.5M/year (30% of all engineering) Features: Inconsistent Support: None Internal team is cheapest and best. ``` ## Summary Platform and infrastructure teams are not cost centers. They're leverage. | Cost Center Framing | Leverage Framing | |--------------------|------------------| | "Costs $2M/year" | "Generates $6M in productivity" | | "Doesn't produce revenue" | "Enables $10M in product revenue" | | "Overhead" | "Force multiplier" | | "Minimize" | "Optimize for leverage ratio" | Measuring platform value: ``` Developer velocity: Deploys, lead time, DORA metrics Time savings: Hours freed × engineer cost Revenue attribution: Velocity → features → revenue Cost avoidance: Incidents, inefficiency, headcount ``` Protecting platform investment: ``` 1. Tie to business metrics 2. Create champions in product teams 3. Show the counterfactual (cost of cutting) 4. Benchmark against alternatives ``` The question isn't "how much does the platform team cost?" The question is "how much value does the platform team create?" When you measure leverage instead of cost, platform teams become obviously essential—not obviously cuttable. --- ## Scaling Infrastructure ≠ Scaling Costs: Economies, Step Functions, and Leverage Points - URL: https://svalle.ru/posts/business/scaling-costs/ - Date: 2025-02-17 - Tags: infrastructure, scaling, finops, business, strategy Everyone knows "cloud scales." But how costs scale is less understood. 10x users doesn't mean 10x costs. Sometimes it's 3x (economies of scale). Sometimes it's 15x (you hit a cliff). The shape of your cost curve determines whether growth is profitable or ruinous. ## The Myth of Linear Scaling The naive model: ``` Users: 1x 10x 100x Costs: 1x 10x 100x Cost/user: Same Same Same ``` This is almost never true. Real cost curves have: - **Economies of scale**: Costs grow slower than usage - **Step functions**: Costs jump at certain thresholds - **Diseconomies**: Costs grow faster than usage Understanding which regime you're in changes everything. ## Economies of Scale ### Fixed Cost Amortization Some costs don't increase with scale: ``` Platform team: $1.5M/year (fixed) Base infrastructure: $200K/year (fixed) Licensing (site): $100K/year (fixed) Total fixed: $1.8M/year At 10K users: $180/user At 100K users: $18/user At 1M users: $1.80/user ``` Fixed costs spread across more users = lower cost per user. ### Volume Discounts Cloud providers reward scale: ``` AWS compute (example): First 1M requests: $0.20 per 1K Next 9M requests: $0.15 per 1K (-25%) Over 10M requests: $0.10 per 1K (-50%) 1M requests: $200 10M requests: $1,550 (not $2,000) 100M requests: $10,550 (not $20,000) ``` Committed use discounts amplify this: ``` On-demand: $1.00/hour 1-year commit: $0.60/hour (-40%) 3-year commit: $0.40/hour (-60%) ``` At scale, you can commit confidently and capture deeper discounts. ### Shared Services One monitoring system serves all teams: ``` Monitoring cost: $100K/year 5 services: $20K per service 50 services: $2K per service 500 services: $200 per service ``` Shared services have massive economies of scale. ### Efficiency Gains Scale enables efficiency investments: ``` At small scale: Manual deployments (cheap, but doesn't scale) At medium scale: Basic automation ($50K to build) Saves $100K/year at current size At large scale: Advanced automation ($200K to build) Saves $1M/year at current size ``` Investments that don't make sense at small scale become obvious at large scale. ### The Economy Curve {{< chart "scaling-costs--economy-curve" "Cost per user declining steeply then flattening as users grow from 10K to 1M" >}} This is the good scenario. Growth is self-funding. ## Step Functions Not all costs scale smoothly. Some jump at thresholds. ### Database Tiers ``` Small database: $500/month (handles 1K QPS) Medium database: $2,000/month (handles 5K QPS) Large database: $10,000/month (handles 20K QPS) Cluster: $50,000/month (handles 100K QPS) ``` Cost doesn't scale linearly with queries: | QPS | Cost | Cost/QPS | Note | |-----|------|----------|------| | 1K | $500 | $0.50 | | | 4K | $500 | $0.125 | ← Efficient | | 5K | $2,000 | $0.40 | ← Step! | | 10K | $2,000 | $0.20 | ← Efficient | | 20K | $10,000 | $0.50 | ← Step! | ### Team Size Jumps ``` 1-5 engineers: Self-organizing, minimal overhead 6-10 engineers: Need team lead (+1 person) 11-20 engineers: Need manager, processes (+2 people) 21-50 engineers: Need multiple teams, directors (+5 people) 50+ engineers: Need org structure, VPs (+10 people) ``` Management overhead grows in steps, not linearly. ### Architecture Rewrites ``` Monolith: Handles up to 10K concurrent users Cost: What you have now Distributed: Handles up to 100K concurrent users Cost: 6-month rewrite + operational complexity Global: Handles 1M+ concurrent users Cost: Another 6-month project + more complexity ``` Architecture transitions are expensive step functions. ### Infrastructure Tiers ``` Single region: $X (simple) Multi-region: $3X (redundancy + networking) Global: $10X (everywhere, all the time) ``` Each tier is a step change in cost and complexity. ### The Step Function Curve {{< chart "scaling-costs--step-function" "Cost rising in right-angle steps at database upgrade, team growth, and architecture rewrite thresholds, then climbing steeply" >}} Steps create "cliffs" where small growth triggers large costs. ## Diseconomies of Scale Sometimes bigger means more expensive per unit. ### Coordination Costs ``` 2 people: 1 communication path 5 people: 10 communication paths 10 people: 45 communication paths 50 people: 1,225 communication paths ``` Communication overhead grows O(n²). Meetings, syncs, documentation, alignment—all get more expensive. ### Complexity Costs ``` 5 services: Simple dependency graph 50 services: Complex interactions 500 services: Nobody understands the full system ``` Debugging time, incident resolution, and cognitive load all increase non-linearly. ### Blast Radius ``` Small system: Incident affects 1K users Large system: Incident affects 1M users Impact scales with size, requiring more investment in reliability. ``` ### The Diseconomy Curve {{< chart "scaling-costs--diseconomy-curve" "Cost per user rising and accelerating as users grow from 10K to 1M" >}} This is the dangerous scenario. Growth becomes unprofitable. ## Real Cost Curves Most systems have all three patterns: {{< chart "scaling-costs--real-cost-curve" "Cost per user falling with economies of scale, jumping at an architecture step function, falling again, then rising into diseconomies of complexity" >}} The art is: 1. Extend economies of scale as long as possible 2. Prepare for step functions before you hit them 3. Avoid diseconomies through smart architecture ## Leverage Points Leverage points are investments that change the shape of your cost curve. ### Automation Before automation: ``` Cost to deploy: $100 (manual process) Deploys/month: 100 Monthly cost: $10,000 ``` After automation ($50K investment): ``` Cost to deploy: $1 (automated) Deploys/month: 1,000 Monthly cost: $1,000 Payback: 5 months ``` Automation converts variable costs to fixed costs, enabling economies of scale. ### Caching Before caching: ``` Database queries: 1M/day Cost per query: $0.001 Daily cost: $1,000 ``` After caching (90% hit rate): ``` Database queries: 100K/day Cache cost: $100/day Total daily cost: $200 Savings: 80% ``` Caching shifts load from expensive resources to cheap resources. ### Architecture Monolith at 10K users: ``` Single large instance: $5,000/month Scales vertically: $$$ per increment ``` Microservices at 10K users: ``` Multiple small instances: $6,000/month Scales horizontally: $ per increment ``` Microservices cost more initially but scale more efficiently. ### Multi-tenancy Single-tenant: ``` Cost per customer: $500/month (dedicated resources) 100 customers: $50,000/month ``` Multi-tenant: ``` Base cost: $10,000/month Per-customer increment: $50/month 100 customers: $15,000/month Savings: 70% ``` Multi-tenancy is a leverage point for SaaS businesses. ## Planning for Scale ### Map Your Cost Curve For each major cost driver, understand the shape: ``` Component Current 10x Scale Shape --------- ------- --------- ----- Compute $10K $40K Economy Database $5K $50K Step function Storage $2K $15K Linear Bandwidth $1K $5K Economy Support $20K $150K Diseconomy ``` ### Identify Upcoming Step Functions ``` Current state: 5K QPS Database tier threshold: 10K QPS Time to threshold: 6 months Cost impact: 5x database cost Lead time to prepare: 3 months Action: Start planning now ``` ### Calculate Unit Economics at Scale ``` Current: Revenue per user: $10/month Cost per user: $3/month Margin: 70% At 10x (with economies): Revenue per user: $10/month Cost per user: $1.50/month Margin: 85% At 100x (hitting diseconomies): Revenue per user: $10/month Cost per user: $4/month Margin: 60% ``` Know where your margin peaks and where it starts declining. ### Invest in Leverage Points Prioritize investments that improve the cost curve: ``` Option A: New feature Revenue impact: +$500K/year Cost impact: +$100K/year Net: +$400K/year Option B: Caching layer Revenue impact: $0 Cost curve impact: Reduce slope by 30% Current trajectory: $200K/year cost growth New trajectory: $140K/year cost growth 10-year impact: $600K saved Option B is better if you're scaling. ``` ## Unit Economics The ultimate test: unit economics at scale. ``` Revenue per user: $10/month Cost per user (all-in): $? Contribution margin = Revenue - Variable costs Gross margin = Revenue - (Variable + Allocated fixed costs) ``` Track unit economics as you scale: | Users | Rev/User | Cost/User | Margin | Note | |-------|----------|-----------|--------|------| | 1K | $10 | $5.00 | 50% | | | 10K | $10 | $3.00 | 70% | | | 100K | $10 | $2.00 | 80% | | | 500K | $10 | $2.50 | 75% | ← diseconomy kicking in | | 1M | $10 | $3.50 | 65% | ← need to address | When margins start declining at scale, you're hitting diseconomies. Time to invest in leverage points. ## Summary Infrastructure costs don't scale linearly: | Pattern | What Happens | Example | |---------|--------------|---------| | Economy of scale | Costs grow slower than users | Fixed costs, volume discounts | | Step function | Costs jump at thresholds | Database tiers, team size | | Diseconomy | Costs grow faster than users | Coordination, complexity | Planning for scale: ``` 1. Map your cost curve by component 2. Identify upcoming step functions 3. Calculate unit economics at scale 4. Invest in leverage points (automation, caching, architecture) 5. Monitor margins as you grow ``` The companies that scale profitably understand their cost curve shape and invest to improve it. 10x users should mean 5x costs, not 15x. That's the difference between a business that scales and one that doesn't. --- ## The J-Curve of Platform Investment: Why Platform Teams Feel Expensive Before They Pay Off - URL: https://svalle.ru/posts/business/platform-j-curve/ - Date: 2025-02-16 - Tags: platform-engineering, business, strategy, investment, leadership Year 1: You hire 5 engineers to build an internal platform. Cost: $1.5M. Measurable savings: $0. Leadership asks: "What are we paying for?" Year 2: Platform is live. Some teams adopt it. Cost: $1.5M. Savings: $400K. Leadership asks: "We're still underwater. Is this working?" Year 3: Platform is mature. Adoption is high. Cost: $1.5M. Savings: $3M. Leadership says: "Obviously this was a good investment." This is the J-curve of platform investment. Understanding it is the difference between a successful platform initiative and one that gets killed before it delivers value. ## The J-Curve Pattern The J-curve describes investments where returns are negative before they turn positive: {{< chart "platform-j-curve--j-curve" "J-curve of value over time: investment costs dip into a valley across Years 1-2, then value climbs steeply past the break-even line by Years 3-4" >}} Private equity, venture capital, and infrastructure projects all follow this pattern. Platform teams are no different. ## Why Platforms Have J-Curves ### 1. Building Takes Time You can't skip the building phase: ``` Month 1-3: Requirements, design, architecture Month 4-9: Core platform development Month 10-12: Hardening, documentation, beta testing Month 13+: Generally available Value delivered in Year 1: Minimal Cost incurred in Year 1: Full ``` The platform doesn't generate value while it's being built. ### 2. Adoption Takes Time Even after launch, adoption isn't instant: ``` Month 1: Early adopters try it (2 teams) Month 3: Word spreads, interest grows (5 teams) Month 6: Majority starts migrating (15 teams) Month 12: Full adoption (30 teams) Month 18: Legacy systems deprecated ``` Adoption curve: {{< chart "platform-j-curve--adoption-curve" "Adoption curve of teams on the platform over 18 months: a steady climb that flattens as adoption approaches full coverage" >}} The platform is most expensive (per user) when it has the fewest users. ### 3. Compounding Takes Time Platform value compounds as adoption grows: ``` Value per team on platform: $50K/year savings Teams on platform over time: End of Year 1: 5 teams × $50K = $250K End of Year 2: 20 teams × $50K = $1M End of Year 3: 40 teams × $50K = $2M End of Year 4: 50 teams × $50K = $2.5M ``` The value accelerates while costs stay relatively flat. ### 4. Network Effects Take Time Platforms become more valuable as more people use them: ``` 5 teams: "It's another tool to learn" 20 teams: "It's where things get done" 50 teams: "It's how we work here" ``` At critical mass, the platform becomes self-reinforcing: - New hires expect it - Best practices accumulate - Tooling ecosystem develops - Expertise becomes common ## The Danger Zone The J-curve has a dangerous valley: {{< chart "platform-j-curve--danger-zone" "Value-over-time valley below the break-even line shaded as the danger zone, where leadership asks whether it is working; a decision point marked on the time axis often comes prematurely" >}} This is where platform initiatives die. Leadership sees: - Significant investment made - Minimal returns so far - Uncertain future payoff The rational response (from their view): cut losses. The problem: they're measuring too early. ## Surviving the Valley ### Set Expectations Upfront Before starting, frame the investment correctly: ``` "Platform investments follow a J-curve. Here's what to expect: Year 1: Investment phase - Building core capabilities - Cost: $1.5M, Returns: ~$0 - Success metric: Platform launched Year 2: Adoption phase - Growing usage, early savings - Cost: $1.5M, Returns: ~$500K - Success metric: 50% team adoption Year 3: Value phase - Full adoption, compounding returns - Cost: $1.5M, Returns: ~$2M+ - Success metric: Net positive ROI" ``` If leadership expects the J-curve, they won't panic in the valley. ### Track Leading Indicators ROI is a lagging indicator. Track leading indicators that predict future ROI: **Adoption metrics:** ``` Teams onboarded this month: +3 Active teams: 15 Percentage of target: 50% Trajectory: On track for 30 by EOY ``` **Usage metrics:** ``` Deployments through platform: 500/month Month-over-month growth: 40% Percentage of all deployments: 60% ``` **Satisfaction metrics:** ``` Developer NPS: 45 "Would recommend to others": 85% Support tickets: Decreasing ``` **Efficiency metrics:** ``` Time to deploy (platform): 5 minutes Time to deploy (legacy): 2 hours Improvement: 24x ``` These metrics show the platform is working before the ROI numbers do. ### Show Incremental Wins Don't wait for Year 3 to show value. Highlight early wins: ``` Month 3: "Team A reduced deploy time from 2 hours to 10 minutes" Month 5: "Team B avoided $50K infrastructure project by using platform" Month 7: "New hire productive in 2 days instead of 2 weeks" Month 9: "Incident response time cut by 60%" ``` These stories build credibility during the investment phase. ### Create Committed Stakeholders Get skin in the game from business stakeholders: ``` Platform sponsor: VP Engineering (executive champion) Steering committee: Directors from each major team Early adopter teams: Committed to migrate by Q2 Success metrics: Agreed upfront with leadership ``` When stakeholders are invested, they advocate for the platform during the valley. ### Manage the Cost Curve The J-curve is deeper if costs are front-loaded. Flatten it: **Bad pattern (deep J):** ``` Year 1: 10-person team (all upfront) Year 2: 10-person team Year 3: 10-person team ``` **Better pattern (shallow J):** ``` Year 1: 4-person team (MVP focus) Year 2: 7-person team (scaling) Year 3: 10-person team (optimization) ``` Smaller initial investment = smaller valley = easier to survive. ## Calculating Platform ROI ### The Basic Model ``` Platform team cost: $1.5M/year Teams served: 50 Cost per team served: $30K/year Value per team (multiple sources): - Reduced deploy time: $20K/year saved - Avoided DIY infra work: $40K/year saved - Faster onboarding: $10K/year saved - Reduced incidents: $15K/year saved Total value per team: $85K/year ROI = (50 teams × $85K - $1.5M) / $1.5M = 183% ``` ### Time-Adjusted Model Account for the J-curve timing: ``` Year 1: Costs: $1.5M Teams on platform: 5 Value: 5 × $85K = $425K Net: -$1.075M Year 2: Costs: $1.5M Teams on platform: 25 Value: 25 × $85K = $2.125M Net: +$625K Year 3: Costs: $1.5M Teams on platform: 50 Value: 50 × $85K = $4.25M Net: +$2.75M Cumulative: Year 1: -$1.075M Year 2: -$450K Year 3: +$2.3M Break-even: ~Month 26 ``` ### Payback Period ``` Total investment to break-even: $3M (Years 1-2) Monthly value at maturity: $350K Payback period: 8.5 months after reaching scale Full payback: ~34 months from start ``` This is typical for platform investments. Leadership needs to expect a 2-3 year payback. ## Common Mistakes ### Mistake 1: Measuring Too Early ``` Month 6: "Platform has cost $750K. Savings: $50K. ROI: -93%" This is like measuring a construction project's ROI while the building is still being built. ``` **Fix**: Measure platform ROI after reaching target adoption, not before. ### Mistake 2: Under-investing in Adoption ``` Platform launched! ... No dedicated adoption effort ... 6 months later: "Why isn't anyone using it?" ``` Adoption doesn't happen automatically. Budget for: - Documentation - Training - Migration support - Developer advocacy - Feedback loops **Fix**: Allocate 30% of platform effort to adoption. ### Mistake 3: Premature Optimization ``` Year 1: Building platform Year 1.5: "Leadership wants to see ROI. Let's pause building and measure." Result: Platform is half-built, adoption stalls, ROI looks terrible. ``` **Fix**: Commit to the investment horizon. Don't interrupt the J-curve. ### Mistake 4: Wrong Comparison ``` "Platform team costs $1.5M. We could hire 7 product engineers instead." But those 7 engineers would spend 30% of time on undifferentiated infra work. Net product capacity: 4.9 engineers. Platform team enables 50 engineers to spend 0% time on infra. Net capacity gain: 15 engineer-equivalents. ``` **Fix**: Compare to the counterfactual, not zero. ## The Anti-J-Curve: Start with Value Some teams invert the J-curve by starting with value: ``` Month 1: Adopt existing open source platform (Backstage, etc.) Month 2: First teams using it Month 3: Small customizations Month 6: Building on top of foundation Month 12: Full custom platform Value curve starts immediately, then accelerates. ``` This requires: - Good existing solutions to build on - Lower ambition for differentiation - Willingness to adapt to existing paradigms Trade-off: Faster initial value, less long-term differentiation. ## Summary Platform investments follow a J-curve: ``` Year 1: Build (costs high, returns zero) Year 2: Adopt (costs steady, returns growing) Year 3+: Compound (costs steady, returns accelerating) ``` Surviving the valley requires: | Strategy | How | |----------|-----| | Set expectations | Frame as multi-year investment upfront | | Track leading indicators | Adoption, usage, satisfaction before ROI | | Show incremental wins | Stories and quick wins during investment phase | | Create stakeholders | Executive sponsors, committed adopters | | Manage cost curve | Start smaller, scale with adoption | The J-curve kills platform initiatives that are working. It makes good investments look bad when measured too early. Understand the pattern. Set expectations. Survive the valley. The returns are on the other side. --- ## The True Cost of Free Open Source - URL: https://svalle.ru/posts/business/true-cost-of-open-source/ - Date: 2025-02-15 - Tags: open-source, business, strategy, platform-engineering, finops Prometheus is free. Kubernetes is free. PostgreSQL is free. So why does running them cost millions? The $0 license fee is the most expensive pricing in software—because it hides all the other costs. Understanding the true cost of "free" is essential for build vs buy decisions. ## The $0 Illusion When evaluating software, companies compare: ``` Open source option: $0 Commercial option: $500K/year Decision: "Obviously open source" ``` But the comparison is wrong. Here's the real math: ``` Open source: License: $0 2 Engineers to run it: $400K/year Training: $30K Incident costs: $50K/year Opportunity cost: $200K/year Total: $680K/year Commercial: License: $500K/year 0.25 Engineer to manage: $50K/year Total: $550K/year ``` The "free" option costs $130K/year more. ## The Hidden Costs ### 1. People Costs Open source doesn't manage itself. Someone needs to: - Install and configure it - Upgrade it regularly - Monitor it - Debug issues - Scale it - Secure it That someone is expensive: ``` Senior infrastructure engineer: $200K/year fully loaded Time spent on Prometheus: 50% Annual cost: $100K For a complex stack: - Kubernetes: 1.5 FTE = $300K - Prometheus: 0.5 FTE = $100K - PostgreSQL: 0.5 FTE = $100K - Kafka: 0.5 FTE = $100K - Redis: 0.25 FTE = $50K --------------- 3.25 FTE = $650K/year ``` The software is free. The people aren't. ### 2. Expertise Costs Open source requires deep expertise. Where does it come from? **Hiring**: Specialists are expensive and hard to find ``` Generic engineer: $180K Kubernetes specialist: $250K (+40%) Kafka specialist: $260K (+45%) ``` **Training**: Existing engineers need training ``` Kubernetes training: $5K/person Certification: $2K/person Time away from work: 2 weeks ($8K) Total per engineer: $15K For a team of 5: $75K ``` **Learning curve**: Productivity loss while ramping ``` Engineer productivity during ramp-up: Month 1-3: 50% (learning) Month 4-6: 75% (gaining confidence) Month 7+: 100% (proficient) 6 months of reduced productivity = 2.25 months of lost output At $200K/year = $37K per engineer ``` ### 3. Maintenance Costs Open source doesn't stand still. You need to keep up: **Upgrades**: ``` Major version upgrade: Planning: 8 hours Testing: 40 hours Execution: 16 hours Rollback prep: 8 hours Documentation: 8 hours Total: 80 hours = $8K per upgrade 4 major systems × 2 upgrades/year = $64K/year ``` **Security patches**: ``` Critical CVE response: Assessment: 4 hours Testing: 8 hours Deployment: 4 hours Total: 16 hours = $1.6K per patch 10 critical patches/year = $16K/year ``` **Breaking changes**: ``` API deprecation migration: Discovery: 8 hours Code changes: 40 hours Testing: 24 hours Total: 72 hours = $7.2K per migration ``` ### 4. Incident Costs When open source breaks, you're on your own: ``` Severity 1 incident: 4 engineers × 8 hours = 32 hours = $3.2K labor Revenue impact: $50K (2 hours downtime) Customer goodwill: Priceless Total: $53K+ per incident ``` Without vendor support, incidents last longer: ``` With vendor support: MTTR 1 hour Without support: MTTR 4 hours 3 extra hours of downtime × $25K/hour = $75K additional cost ``` ### 5. Opportunity Cost Your best engineers are keeping the lights on instead of building product: ``` Senior engineer salary: $200K Time on open source ops: 40% Opportunity cost: $80K/year What else could they build? - Features that drive revenue - Automation that improves efficiency - Innovation that creates competitive advantage ``` This is the biggest hidden cost—and the hardest to see. ### 6. Risk Costs Open source carries risks that don't appear until they materialize: **Project abandonment**: ``` Maintainer burns out, project dies Your options: - Fork and maintain yourself (expensive) - Migrate to alternative (expensive) - Stay on dead version (risky) ``` **Security vulnerabilities**: ``` Log4j-style event: Scramble to assess: 40 hours Patch or mitigate: 80 hours Audit: 40 hours Total: 160 hours = $16K Plus: Risk exposure while vulnerable ``` **Compliance gaps**: ``` Auditor: "Show me your SOC2 report for your database." You: "It's open source, there is no SOC2 report." Auditor: "Then show me how you've validated equivalent controls." You: "..." (That's a 6-month project) ``` ## The TCO Framework Total Cost of Ownership for open source: ``` TCO = License + Hosting + People + Risk + Opportunity Cost Where: License: $0 (the part everyone sees) Hosting: Cloud/hardware costs People: Engineers to run it Risk: Expected cost of incidents, security, abandonment Opportunity: What else those engineers could do ``` ### Example: Running Your Own Kafka ``` License: $0 Hosting: 3 brokers × $500/month $18K/year Storage, networking $6K/year People: 0.5 FTE to manage $100K/year Training $10K (amortized) Risk: Expected incident cost $30K/year Security exposure $10K/year Opportunity: 0.5 FTE on product instead $100K/year Total TCO: $274K/year ``` ### Compare: Managed Kafka (Confluent) ``` License: $150K/year Additional engineering: 0.1 FTE = $20K/year Total TCO: $170K/year ``` The "free" option costs $100K/year more. ## When Open Source Wins Open source isn't always more expensive. It wins when: ### High Volume ``` Managed database: $0.10 per million queries Your volume: 100B queries/month Managed cost: $10M/month Self-managed: $500K/month all-in Winner: Open source by 20x ``` At massive scale, per-unit pricing kills you. ### Strategic Control Some systems are core to your business: ``` "We need to modify the query planner for our workload" "We need to integrate deeply with our custom tooling" "We can't depend on a vendor for our core technology" ``` For strategic systems, the flexibility of open source is worth the cost. ### Deep Expertise Already Exists ``` Team has 3 Kafka experts already Training cost: $0 Learning curve: 0 Incident MTTR: Same as vendor support In this case, self-managed may be cheaper. ``` ### Long Time Horizon ``` Year 1: Self-managed costs more (setup, learning) Year 2: Roughly equal Year 3+: Self-managed costs less (expertise amortized) If you're committed for 5+ years, self-managed can win. ``` ## When Commercial Wins ### Small Teams ``` Team size: 5 engineers Systems to manage: K8s, Postgres, Redis, Kafka, Prometheus Each system needs ~0.3 FTE to manage properly Total: 1.5 FTE = 30% of your team You can't afford to spend 30% of engineering on infrastructure. ``` Small teams should buy, not build. ### Non-Core Systems ``` "Is the logging system our competitive advantage?" No. "Will customers pay more because we run our own Elasticsearch?" No. "Does anyone care how we run our CI/CD?" No. Buy it. Focus on what matters. ``` ### Fast-Moving Domains ``` Security tools: Threat landscape changes weekly ML infrastructure: Best practices change monthly Observability: New standards every year You can't keep up. Pay someone whose job it is. ``` ### Time to Market Matters ``` Build with open source: 6 months Buy commercial: 2 weeks 4.5 months faster to market Revenue opportunity: $500K Cost of commercial: $100K ROI: 400% ``` ## The Decision Framework ### Ask These Questions **1. "Would I hire a full-time person to manage this?"** ``` If yes: Consider commercial alternative If no: Who's actually going to manage it? ``` **2. "Is this core to our business?"** ``` If yes: Open source may be worth the investment If no: Why are we building it? ``` **3. "What's the cost of getting it wrong?"** ``` High (revenue, security, compliance): Buy reliability Low (internal tools, experiments): Open source is fine ``` **4. "Do we have the expertise?"** ``` If yes: Open source is viable If no: Factor in hiring/training costs ``` **5. "What's our time horizon?"** ``` < 2 years: Commercial (faster, less investment) > 5 years: Open source (amortized costs) ``` ### The Hybrid Approach Most companies land on a hybrid: ``` Core systems: Self-managed (strategic value) Commodity systems: Managed services (not worth the effort) Experiments: Managed (speed) Scale systems: Self-managed (cost) ``` ## Making the Case When presenting build vs buy decisions, show the full picture: **Bad presentation:** ``` Option A (Open source): $0 Option B (Commercial): $500K Recommendation: A ``` **Good presentation:** ``` Option A (Open source): Year 1: $350K (setup + 0.5 FTE + training) Year 2+: $250K/year (0.5 FTE + maintenance) 5-year: $1.35M Option B (Commercial): Year 1: $500K Year 2+: $500K/year 5-year: $2.5M Option C (Managed open source): Year 1: $200K Year 2+: $200K/year 5-year: $1M Recommendation: C (lowest TCO, minimal operational burden) ``` ## Summary Open source is free like a puppy is free: | What You See | What You Get | |--------------|--------------| | $0 license | + $X people costs | | | + $Y expertise costs | | | + $Z maintenance costs | | | + incident costs | | | + opportunity costs | | | + risk costs | The total cost is often higher than commercial alternatives. When evaluating open source: | Factor | Open Source Wins | Commercial Wins | |--------|------------------|-----------------| | Volume | Very high scale | Low to medium | | Expertise | Deep in-house | Limited | | Core business | Yes | No | | Time horizon | 5+ years | < 3 years | | Team size | Large | Small | | Time to market | Not critical | Critical | The best decision isn't always "free." It's the lowest total cost of ownership for your situation. Don't let a $0 price tag cost you millions. --- ## Infrastructure as Growth Constraint: When Systems Become the Bottleneck to Revenue - URL: https://svalle.ru/posts/business/infrastructure-growth-constraint/ - Date: 2025-02-14 - Tags: infrastructure, scaling, business, strategy, platform-engineering Your sales team closes a landmark deal. Marketing's campaign goes viral. Customer signups spike 10x. This is the moment you've been building toward. Then your checkout page times out. The API returns 503s. The database locks up. Customers rage on Twitter. The viral moment becomes a viral disaster. Revenue was there for the taking. Your infrastructure said no. ## The Invisible Ceiling Every system has a capacity. Below that capacity, infrastructure is invisible—it just works. Above it, infrastructure becomes the only thing anyone talks about. {{< chart "infrastructure-growth-constraint--infrastructure-ceiling" "Revenue potential rising over time toward a viral moment, crossing a dashed infrastructure ceiling at the capacity-hit point; revenue captured is capped below the ceiling" >}} The gap between the revenue potential curve and the infrastructure ceiling is money left on the table. ## Real Examples ### The Black Friday Crash Retailer does $1M/hour on normal days. Black Friday, demand spikes to $5M/hour potential. But the checkout system caps at $2M/hour throughput. ``` Potential revenue: $5M/hour × 8 hours = $40M Actual revenue: $2M/hour × 8 hours = $16M Left on table: $24M Plus: - Customer churn from bad experience - Brand damage - Customer service costs ``` The infrastructure team had been asking for $500K to upgrade capacity. It was "deferred to next quarter." ### The Enterprise Deal Lost Startup pitches a Fortune 500 prospect. Technical due diligence call: > "Can your platform handle 10M API calls per day?" > "Uh... we'd need to do some work..." > "Thanks, we'll go with the other vendor." $2M ARR deal lost. The prospect didn't want a vendor who'd become their bottleneck. ### The Viral Moment Missed App gets featured on a major podcast. Downloads spike 50x. But the onboarding service wasn't built for this: ``` Normal: 100 signups/hour → all complete onboarding Viral: 5,000 signups/hour → 90% timeout, abandon ``` 4,500 users had intent to sign up. They'll never come back. The $20K to build auto-scaling for onboarding was "not a priority." ## Quantifying the Constraint ### Revenue Per Request Start with your revenue math: ``` Monthly revenue: $1,000,000 Monthly requests: 10,000,000 Revenue per request: $0.10 ``` Now apply capacity constraints: ``` Current capacity: 500 req/sec Peak demand: 800 req/sec Requests dropped: 300 req/sec × 3600 sec/hr × 4 peak hours = 4.3M Revenue lost: 4.3M × $0.10 = $430,000/month ``` ### The Throttling Tax When you hit capacity, you don't just drop requests. You slow everyone down: ``` Normal response time: 200ms At capacity: 2000ms (10x slower) User conversion rate: -7% per 100ms additional latency ``` Amazon found that every 100ms of latency cost them 1% in sales. For a $500B company, that's $5B. ### Opportunity Cost The hardest to quantify, but often the largest: - Deals you didn't pursue because you couldn't scale - Features you didn't build because the platform couldn't support them - Markets you didn't enter because of infrastructure limitations These don't show up in any dashboard. ## Leading Indicators The constraint is easiest to fix before you hit it. Watch these signals: ### Capacity Utilization Trending ``` Month 1: 40% peak utilization Month 2: 55% peak utilization Month 3: 70% peak utilization Month 4: 💥 ``` If utilization is trending up and you're not adding capacity, you're on a collision course. ### Time to Provision ``` "How long to add 50% more capacity?" Good: "2 hours, it's automated" Okay: "2 days, need to spin up nodes" Bad: "2 weeks, need to re-architect" Danger: "2 months, need new hardware" ``` If you can't scale faster than your business grows, you're at risk. ### Incident Frequency at Peak ``` Incidents during peak hours Month 1: 0 Month 2: 1 minor Month 3: 2 minor, 1 major Month 4: Regular degradation ``` Increasing incidents at peak = you're brushing against the ceiling. ### Team Confidence Ask your infrastructure team: "Could we handle 3x traffic tomorrow?" Their body language tells you everything. ## The Investment Case Infrastructure capacity is revenue insurance. Here's how to frame it: ### The Insurance Model ``` Current revenue at risk: $10M/year (during peak events) Probability of capacity incident: 30%/year (based on trends) Expected loss: $3M/year Infrastructure investment: $500K Risk reduction: 80% New expected loss: $600K ROI: ($3M - $600K - $500K) / $500K = 280% ``` ### The Growth Enablement Model ``` Current capacity: $50M ARR equivalent Growth target: $100M ARR (2x) Infrastructure investment: $2M to support 2x Without investment: Growth capped at $50M With investment: Growth enabled to $100M Revenue unlocked: $50M Investment: $2M ROI: 2,400% ``` ### The Competitive Model ``` Deal qualification question: "Can you handle our scale?" Current answer: "We'd need 6 months" Competitor answer: "Yes, today" Deals lost to capacity concerns: $5M/year Investment to fix: $1M ROI: Clear. ``` ## The Timing Problem The challenge: infrastructure investment is most valuable *before* you need it, but easiest to fund *after* you've had a crisis. ``` Time to build capacity: 3-6 months Time to hit viral moment: 0 (unpredictable) If you wait until you need it, it's too late. ``` This is why infrastructure capacity should be funded like insurance, not like a feature. ### The Headroom Rule Smart companies maintain capacity headroom: ``` Minimum headroom: 2x current peak Comfortable headroom: 3x current peak Scaling time: < growth rate ``` If you're growing 10%/month and it takes 3 months to add capacity, you need at least 30% headroom at all times. ## Making It Visible ### Revenue at Risk Dashboard Create a dashboard that shows: ``` Current peak load: 70% of capacity Capacity ceiling: $X revenue/hour Time to hit ceiling: Y weeks at current growth Revenue at risk: $Z if we hit ceiling ``` Make the constraint visible to leadership weekly. ### Capacity in Business Reviews Include capacity alongside other business metrics: | Metric | Value | Trend | |--------|-------|-------| | Revenue | $10M | ↑ 15% | | Customers | 50,000 | ↑ 20% | | NPS | 45 | ↑ 5 | | Infra capacity | 70% utilized | ↑ 10% ⚠️ | If revenue is reviewed monthly, capacity should be too. ### Post-Mortems with Revenue Impact When incidents happen, quantify the revenue impact: ``` Incident: Checkout service degradation Duration: 2 hours Requests affected: 50,000 Estimated revenue lost: $500,000 Root cause: Database capacity ``` $500K makes the $100K database upgrade look different. ## Common Objections ### "We can scale when we need to" Maybe. But how long does it take? ``` Best case: Auto-scaling handles it (minutes) Typical: Need to provision resources (hours/days) Worst case: Need architectural changes (weeks/months) ``` If your viral moment lasts 4 hours and scaling takes 2 days, you've missed it. ### "We haven't had problems yet" Yet. Check the trends: - Is utilization increasing? - Are incidents at peak increasing? - Is time-to-provision > time-to-demand? "No problems yet" often means "problems soon." ### "It's too expensive" Compared to what? ``` Capacity investment: $500K Revenue at risk: $5M Insurance ratio: 10% ``` You'd pay 10% to insure any other $5M asset. ### "We'll handle it when we get there" You'll handle it *poorly* when you get there. Under crisis conditions: - Decisions are rushed - Costs are higher (emergency pricing, consultants) - Quality suffers (quick fixes, tech debt) - Customers are already angry ## Summary Infrastructure isn't just a cost center. It's the ceiling on your revenue. | Symptom | Translation | |---------|-------------| | "We can't handle that deal size" | Revenue constraint | | "We need 6 months to support that" | Growth constraint | | "Black Friday was rough" | Seasonal constraint | | "We're not ready for viral" | Opportunity constraint | The investment case: ``` Revenue at risk: Quantifiable Infrastructure investment: Quantifiable ROI: Usually obvious when you do the math ``` The timing case: ``` Time to need capacity: Unpredictable Time to build capacity: Months Conclusion: Build before you need it ``` Your infrastructure capacity should always exceed your ambition. The alternative is your systems choosing your growth rate for you. --- ## Why Your Kubernetes Cluster is a Depreciating Asset - URL: https://svalle.ru/posts/kubernetes/cluster-depreciation/ - Date: 2025-02-13 - Tags: kubernetes, platform-engineering, finops, technical-debt, business, strategy Your company's Kubernetes cluster is an asset. It required capital to build—cloud spend, engineering time, tooling. It generates value by running production workloads. But unlike the servers in your finance team's depreciation schedule, this asset is invisible on the balance sheet. That's a problem. Because like any asset, it's losing value every day. ## The Invisible Depreciation In accounting, depreciation spreads an asset's cost over its useful life. A $50,000 server might depreciate over 5 years—$10,000 per year hitting the P&L. Everyone understands: assets wear out, and you need to budget for replacement. Kubernetes clusters depreciate too. But this depreciation is invisible: - No line item in the budget - No scheduled replacement cost - No CFO asking "when do we need to refresh this?" So leadership assumes: if it's running, it's fine. If it ain't broke, don't fix it. Meanwhile, the cluster is rotting. ## The Depreciation Timeline Here's what happens to a Kubernetes cluster if you "just keep it running": | Age | Status | What's Happening | |-----|--------|------------------| | 0 months | Current | Fully supported, latest security patches | | 4 months | n-1 | One minor version behind, still supported | | 8 months | n-2 | Two versions behind, approaching end of support | | 12 months | n-3 | Out of standard support window | | 18 months | Legacy | Ecosystem moving on, tools dropping support | | 24 months | Liability | Upgrade is now a project, not a task | | 36 months | Crisis | Major security risk, compliance issues, engineers refuse to touch it | Kubernetes releases three minor versions per year. Each version is supported for roughly 14 months. Stand still for a year, and you're outside the support window. But the real depreciation isn't just version numbers—it's everything that accumulates. ## The Components of Cluster Depreciation ### 1. Security Depreciation Every day your cluster ages, the attack surface grows: ``` CVEs disclosed against your version: accumulating Patches available: not applicable to your version Compliance auditors: increasingly concerned ``` A CVE against Kubernetes 1.27 might be patched in 1.27.8. But if you're on 1.25, that patch doesn't exist for you. You're exposed until you upgrade. **Cost**: Security incidents average $4.45M (IBM 2023). Even near-misses cost audit cycles, executive attention, and engineering firefighting. ### 2. Ecosystem Depreciation The Kubernetes ecosystem doesn't wait for you: ``` Helm charts: "Requires Kubernetes 1.26+" Operators: "Dropping support for 1.25 in next release" Service mesh: "New features only available on 1.27+" Cloud provider: "EKS 1.24 end of support: March 2024" ``` At first, you work around it. Pin old versions. Fork charts. Skip features. But workarounds compound: | Months Behind | Workarounds Required | |---------------|---------------------| | 6 | Occasional, minor | | 12 | Regular, annoying | | 18 | Constant, blocking new tools | | 24 | Ecosystem has moved on | **Cost**: Engineering time on workarounds, inability to adopt new tools, vendor support limitations. ### 3. API Depreciation Kubernetes deprecates APIs on a schedule. What works today may not work tomorrow: ```yaml # This worked in 1.21 apiVersion: networking.k8s.io/v1beta1 kind: Ingress # Required since 1.22 apiVersion: networking.k8s.io/v1 kind: Ingress ``` Skip one version? You update your manifests. Skip three versions? You're updating hundreds of manifests, Helm charts, operators, and hoping nothing breaks. **Cost**: The "API deprecation cliff"—upgrades become migrations, migrations become projects, projects need quarters. ### 4. Knowledge Depreciation Your team's knowledge has a half-life too: - Engineers learn the current ecosystem, not your legacy version - New hires have never seen your old patterns - Stack Overflow answers assume recent versions - Training materials move on **Cost**: Onboarding time, tribal knowledge dependencies, reduced hiring pool. ### 5. Talent Depreciation Engineers don't want to work on legacy infrastructure: > "What version of Kubernetes are you running?" > "1.24." > "...I have another offer." Your cluster's age signals technical culture. Modern cluster = modern practices. Legacy cluster = red flag. **Cost**: Hiring difficulty, retention risk, salary premium for "legacy" work. ## Upgrade Debt: The Hidden Liability Technical debt is a familiar concept. Upgrade debt is its infrastructure cousin: > **Upgrade debt** = the accumulated cost of deferred upgrades Like financial debt, it accrues interest: | Upgrade Frequency | Effort per Upgrade | Annual Effort | |-------------------|-------------------|---------------| | Every 4 months | 2 days | 6 days | | Every 8 months | 1 week | 6.5 days | | Every 12 months | 2 weeks | 10 days | | Every 24 months | 1-2 months | 15-30 days | | Every 36 months | Quarter-long project | 40+ days | The math is counterintuitive: **upgrading more often costs less total time**. Why? Because upgrade complexity isn't linear: ``` 1 version jump: Minor API changes, quick testing 2 version jumps: Multiple deprecations, moderate testing 3 version jumps: Breaking changes stack, extensive testing 4+ version jumps: Essentially a migration project ``` Deferring upgrades feels like saving time. It's actually borrowing time at high interest. ## The "If It Ain't Broke" Fallacy When leadership says "if it ain't broke, don't fix it," they're making an accounting error. They see: ``` Upgrade cost: $X (visible) Current state: Working (visible) Conclusion: Don't spend $X ``` They don't see: ``` Security exposure: Accumulating (invisible) Ecosystem drift: Accumulating (invisible) Upgrade debt interest: Accumulating (invisible) Talent risk: Accumulating (invisible) Future upgrade cost: Growing (invisible) ``` The cluster looks fine because the depreciation hasn't hit the P&L yet. But it will—as a security incident, a failed audit, a blocked feature, a departed engineer, or a crisis upgrade project. ## Quantifying Cluster Depreciation Here's a framework to make depreciation visible: ### Security Risk Cost ``` Annual CVE exposure probability: 20% (estimate for unsupported versions) Average incident cost: $500K (your estimate—could be much higher) Expected annual cost: $100K ``` ### Ecosystem Drift Cost ``` Engineering hours on workarounds: 10 hrs/month Fully-loaded engineer cost: $150/hr Annual cost: $18K ``` ### Upgrade Debt Interest ``` Current upgrade effort: 1 week (if done now) Upgrade effort in 12 months: 1 month Additional cost: 3 weeks × $6K/week = $18K ``` ### Talent Risk Cost ``` Probability of losing engineer due to legacy stack: 10% Replacement cost: $50K (recruiting, onboarding, ramp-up) Expected annual cost: $5K per engineer Team of 5: $25K ``` ### Total Annual Depreciation | Component | Annual Cost | |-----------|-------------| | Security risk | $100K | | Ecosystem drift | $18K | | Upgrade debt | $18K | | Talent risk | $25K | | **Total** | **$161K/year** | This is a rough model—your numbers will vary. The point isn't precision; it's visibility. Now depreciation has a number. Now it can be budgeted. ## The Continuous Upgrade Model The alternative to crisis upgrades is continuous upgrades: ``` Traditional: Upgrade when forced (every 18-24 months) Continuous: Upgrade on schedule (every 4-6 months) ``` ### What Continuous Looks Like | Activity | Frequency | Effort | |----------|-----------|--------| | Minor version upgrade | Every 4 months | 1-2 days | | Test suite run | Every upgrade | Automated | | API deprecation fixes | As encountered | Hours | | Ecosystem updates | Continuous | Part of normal work | ### The Investment Case ``` Continuous upgrade cost: 6 days/year Crisis upgrade cost: 20-40 days/year (amortized) Net savings: 14-34 days/year Plus avoided costs: - Security incidents - Ecosystem workarounds - Talent churn - Compliance findings ``` Continuous upgrades aren't overhead—they're maintenance capex that prevents much larger costs. ## Making the Business Case When requesting upgrade budget, don't say: > "We need to upgrade Kubernetes because we're behind." Say: > "Our cluster is a depreciating asset. Every quarter we defer upgrades costs us approximately $40K in accumulated risk, engineering workarounds, and growing upgrade debt. I'm requesting 6 engineering days per year to maintain the asset and avoid a $150K+ crisis project in 18 months." Frame it in terms leadership understands: | Technical Term | Business Term | |----------------|---------------| | "We're on an old version" | "The asset is past its useful life" | | "We have upgrade debt" | "We have deferred maintenance liability" | | "We might have security issues" | "We have unquantified risk exposure" | | "Engineers don't like it" | "We have talent retention risk" | | "We should upgrade" | "We should service the asset before it fails" | ## The Depreciation Schedule Just like finance depreciates servers, create a depreciation schedule for your cluster: ``` Asset: Production Kubernetes Cluster Useful life: 12 months (3 minor versions) Depreciation method: Straight-line Maintenance requirement: Quarterly upgrades Q1: Upgrade to 1.29 (2 days) Q2: Upgrade to 1.30 (2 days) Q3: Upgrade to 1.31 (2 days) Q4: Upgrade to 1.32 (2 days) Annual maintenance budget: 8 engineering days Alternative (deferred): 30+ day project in 2 years ``` Put it in the infrastructure budget. Review it quarterly. Treat it like any other asset maintenance. ## When Standing Still Makes Sense To be fair, there are cases where deferring upgrades is rational: 1. **End-of-life workload**: The system is being decommissioned anyway 2. **Compliance freeze**: Auditors require stability during assessment period 3. **Resource constraints**: Genuinely no capacity (but count the cost) 4. **Risk window**: Approaching a critical business period (holiday traffic) But these should be explicit decisions with explicit costs, not default inaction. ## Summary Your Kubernetes cluster is an asset. Assets depreciate. The depreciation is real whether you account for it or not. | Aspect | Visible | Invisible (but real) | |--------|---------|---------------------| | Cloud spend | ✓ | | | Engineering salaries | ✓ | | | Security risk | | ✓ | | Ecosystem drift | | ✓ | | Upgrade debt | | ✓ | | Talent risk | | ✓ | The choice isn't whether to pay for depreciation. It's whether to pay incrementally (continuous upgrades) or all at once (crisis project). Continuous upgrades cost less total effort, avoid crisis projects, reduce security exposure, keep the ecosystem accessible, and make your infrastructure attractive to talent. Make the depreciation visible. Budget for it. Maintain the asset. Because "if it ain't broke" is just "the depreciation hasn't hit the P&L yet." --- ## GPU Scheduling in Kubernetes: From Device Plugins to Dynamic Resource Allocation - URL: https://svalle.ru/posts/kubernetes/gpu-scheduling-dra/ - Date: 2025-02-12 - Tags: kubernetes, gpu, scheduling, machine-learning, DRA, nvidia Your ML team needs GPUs. You add nodes with NVIDIA A100s, install the device plugin, and suddenly Kubernetes can schedule GPU workloads. But then the requests start: "Can we share a GPU between pods?" "Why is my training job slow even though I have 8 GPUs?" "Can we request a specific GPU model?" GPU scheduling in Kubernetes has evolved from a simple device plugin model to the more flexible Dynamic Resource Allocation (DRA). This post covers both, explaining how they work and when to use each. ## The Problem: GPUs Aren't Like CPU or Memory CPU and memory are fungible. If you request 2 CPUs, any 2 CPUs work. The scheduler doesn't care which ones. GPUs are different: 1. **Heterogeneous**: A100 vs V100 vs T4 have vastly different capabilities 2. **Topology matters**: GPU-to-GPU and GPU-to-CPU connectivity affects performance 3. **Not easily divisible**: You can't give a pod "0.5 GPUs" the way you give it 500m CPU 4. **State and configuration**: GPUs have drivers, compute modes, memory configurations 5. **Expensive**: At $2-10/hour per GPU, idle GPUs hurt The standard Kubernetes resource model (`requests`/`limits`) wasn't designed for this. ## Device Plugins: The Current Model Since Kubernetes 1.8, device plugins let vendors expose hardware to the scheduler. ### How Device Plugins Work ```d2 {alt="Device plugin architecture: kubelet and the Device Plugin (e.g., NVIDIA) communicate bidirectionally over gRPC, the plugin manages the GPU hardware, and kubelet advertises node resources nvidia.com/gpu: 4 to the API server"} direction: right kubelet plugin: "Device Plugin\n(e.g., NVIDIA)" apiserver: "API Server\n\nNode resources:\nnvidia.com/gpu: 4" gpu: GPU Hardware kubelet <-> plugin: gRPC kubelet -> apiserver plugin -> gpu ``` 1. **Device plugin registers** with kubelet via gRPC 2. **Reports available devices** (e.g., 4 GPUs) 3. **kubelet advertises** to API server as extended resources 4. **Scheduler sees** `nvidia.com/gpu: 4` as allocatable 5. **When pod scheduled**, device plugin tells kubelet which device(s) to assign ### Installing NVIDIA Device Plugin ```bash # Add NVIDIA Helm repo helm repo add nvdp https://nvidia.github.io/k8s-device-plugin helm repo update # Install device plugin helm install nvdp nvdp/nvidia-device-plugin \ --namespace nvidia-device-plugin \ --create-namespace ``` Verify: ```bash $ kubectl describe node gpu-node-1 | grep -A 5 "Allocatable" Allocatable: cpu: 32 memory: 128Gi nvidia.com/gpu: 4 ``` ### Requesting GPUs ```yaml apiVersion: v1 kind: Pod metadata: name: gpu-pod spec: containers: - name: cuda-container image: nvidia/cuda:12.0-base resources: limits: nvidia.com/gpu: 1 # Request 1 GPU ``` **Note**: For device plugin resources, `limits` and `requests` must be equal. You can't "burst" GPU usage. ### What Happens at Runtime When the pod is scheduled: 1. Device plugin's `Allocate()` called with device IDs 2. Plugin returns environment variables and device mounts: ```go // Device plugin returns ContainerAllocateResponse{ Envs: map[string]string{ "NVIDIA_VISIBLE_DEVICES": "GPU-abc123", }, Mounts: []*Mount{ {ContainerPath: "/dev/nvidia0", HostPath: "/dev/nvidia0"}, }, } ``` 3. Container sees only assigned GPU(s) ## Limitations of Device Plugins ### 1. Whole Devices Only ```yaml resources: limits: nvidia.com/gpu: 1 # OK nvidia.com/gpu: 0.5 # NOT POSSIBLE ``` Can't share a GPU between pods. A $10k A100 sits 90% idle because one pod claimed it. ### 2. No Device Selection You can't say "give me an A100, not a T4." The scheduler just sees a count: ```yaml # What you want nvidia.com/gpu: model: A100 memory: 80Gi # What you can do nvidia.com/gpu: 1 # Could be anything ``` **Workaround**: Use node labels and node selectors: ```yaml nodeSelector: gpu-type: a100 ``` But this is coarse-grained and doesn't scale. ### 3. No Topology Awareness Multi-GPU training performance depends on GPU interconnects: ```d2 {alt="GPU interconnect topology: GPU 0 links to GPU 1 and GPU 2 links to GPU 3 over fast NVLink, but the pod got GPU 0 and GPU 3, which share only a slow PCIe link; legend ranks NVLink (600 GB/s) best, PCIe (64 GB/s) OK, and cross-socket PCIe bad"} direction: right legend: "Best: NVLink (600 GB/s)\nOK: PCIe (64 GB/s)\nBad: Cross-socket PCIe\n\nPod gets GPU 0 and GPU 3" {style: {stroke-width: 0; fill: transparent}} gpu0: GPU 0 gpu1: GPU 1 gpu2: GPU 2 gpu3: GPU 3 gpu0 -- gpu1: NVLink gpu2 -- gpu3: NVLink gpu0 -- gpu3: "PCIe (Slow!)" ``` Device plugins don't consider topology. Your 8-GPU training job might get the worst possible GPU combination. ### 4. No Preparation or Cleanup Some devices need setup before use: - Configure compute mode - Allocate memory partitions (MIG) - Load firmware Device plugins have no lifecycle hooks for this. ### 5. Scheduling Races ``` 1. Scheduler sees: Node has 2 GPUs free 2. Scheduler assigns Pod A (needs 2 GPUs) to node 3. Before Pod A starts, Pod B (needs 1 GPU) also scheduled to node 4. Conflict! ``` Extended resources are accounted at scheduling time, but there's a window for races. ## Fractional GPUs: MIG and Time-Slicing ### NVIDIA Multi-Instance GPU (MIG) MIG physically partitions A100/A30/H100 GPUs: ```d2 {alt="MIG partition tree: one A100 80GB GPU physically partitioned into five isolated MIG instances — three 1g.10gb (instances 1-3), one 2g.20gb (instance 4), and one 3g.40gb (instance 5)"} direction: right a100: A100 80GB i1: "MIG 1g.10gb (instance 1)" i2: "MIG 1g.10gb (instance 2)" i3: "MIG 1g.10gb (instance 3)" i4: "MIG 2g.20gb (instance 4)" i5: "MIG 3g.40gb (instance 5)" a100 -- i1 a100 -- i2 a100 -- i3 a100 -- i4 a100 -- i5 ``` Each MIG instance is isolated: separate memory, separate compute units. **Configure MIG with device plugin:** ```yaml # nvidia-device-plugin config config: map: default: mixed sharing: mig: strategy: mixed ``` Then request specific MIG profiles: ```yaml resources: limits: nvidia.com/mig-1g.10gb: 1 ``` **Pros**: True isolation, guaranteed resources **Cons**: Only certain GPUs support MIG, reconfiguration requires empty GPU ### Time-Slicing Multiple pods share one GPU by taking turns: ```yaml # Device plugin ConfigMap sharing: timeSlicing: renameByDefault: false resources: - name: nvidia.com/gpu replicas: 4 # Each GPU appears as 4 resources ``` Now `nvidia.com/gpu: 4` becomes `nvidia.com/gpu: 16` (4 GPUs × 4 replicas). ```yaml # Pod requests "1 GPU" but actually gets 1/4 resources: limits: nvidia.com/gpu: 1 ``` **Pros**: Works on any NVIDIA GPU, no reconfiguration **Cons**: No isolation—one pod can starve others, no memory limits ### Comparison | Feature | MIG | Time-Slicing | |---------|-----|--------------| | Isolation | Full (memory + compute) | None | | Supported GPUs | A100, A30, H100 | Any NVIDIA | | Reconfiguration | Requires empty GPU | Dynamic | | Memory guarantee | Yes | No | | Best for | Production inference | Dev/test, bursty workloads | ## Dynamic Resource Allocation (DRA) DRA (alpha in 1.26, graduating in 1.31+) is the next evolution. It addresses device plugin limitations with a claim-based model. ### Key Concepts **ResourceClaim**: A request for resources (like PVC for storage) ```yaml apiVersion: resource.k8s.io/v1alpha2 kind: ResourceClaim metadata: name: gpu-claim spec: resourceClassName: gpu.nvidia.com ``` **ResourceClass**: Defines a type of resource and its driver ```yaml apiVersion: resource.k8s.io/v1alpha2 kind: ResourceClass metadata: name: gpu.nvidia.com driverName: gpu.nvidia.com ``` **ResourceClaimTemplate**: For dynamic claim creation ```yaml apiVersion: resource.k8s.io/v1alpha2 kind: ResourceClaimTemplate metadata: name: gpu-template spec: spec: resourceClassName: gpu.nvidia.com ``` ### How DRA Works ``` 1. User creates ResourceClaim (or template in Pod) 2. Scheduler finds nodes where claim can be satisfied 3. DRA driver's "allocate" called with node context 4. Driver prepares device (configure MIG, set mode, etc.) 5. Pod starts with device available 6. On pod termination, driver cleans up ``` ### DRA vs Device Plugins | Aspect | Device Plugins | DRA | |--------|---------------|-----| | Granularity | Whole devices | Flexible (fractions, attributes) | | Device selection | Count only | Rich selectors | | Lifecycle | None | Prepare/cleanup hooks | | Scheduling | Simple counting | Structured parameters | | State | Stateless | Claim tracks allocation | ### Requesting GPUs with DRA ```yaml apiVersion: v1 kind: Pod metadata: name: gpu-pod spec: containers: - name: cuda-container image: nvidia/cuda:12.0-base resources: claims: - name: gpu resourceClaims: - name: gpu source: resourceClaimTemplateName: gpu-template ``` With structured parameters (future): ```yaml apiVersion: resource.k8s.io/v1alpha2 kind: ResourceClaim metadata: name: specific-gpu spec: resourceClassName: gpu.nvidia.com parametersRef: apiGroup: gpu.nvidia.com kind: GpuClaimParameters name: my-params --- apiVersion: gpu.nvidia.com/v1alpha1 kind: GpuClaimParameters metadata: name: my-params spec: selector: model: A100 memory: 80Gi sharing: strategy: MIG profile: 3g.40gb ``` ### Current State (Kubernetes 1.31+) DRA is maturing but still evolving: - **Core API**: Stable enough for testing - **NVIDIA DRA driver**: Available, replacing device plugin in some deployments - **Structured parameters**: Still developing - **Production readiness**: Check your version's feature gates ```bash # Enable DRA feature gates (if not default) --feature-gates=DynamicResourceAllocation=true ``` ## Topology-Aware Scheduling ### The Problem 8-GPU training job needs GPUs that can communicate fast: ``` Ideal: All 8 GPUs on same NVLink domain OK: 4+4 across two NVLink domains Bad: 8 GPUs scattered across PCIe ``` ### Topology Manager kubelet's Topology Manager aligns resource allocation: ```yaml # kubelet configuration topologyManagerPolicy: best-effort # or: restricted, single-numa-node topologyManagerScope: container # or: pod ``` **Policies:** - `none`: No topology alignment - `best-effort`: Try to align, but schedule anyway - `restricted`: Fail if can't align - `single-numa-node`: All resources from one NUMA node ### NVIDIA GPU Operator The GPU Operator automates GPU node setup and includes topology awareness: ```bash helm install gpu-operator nvidia/gpu-operator \ --set driver.enabled=true \ --set toolkit.enabled=true \ --set devicePlugin.enabled=true \ --set mig.strategy=mixed ``` It handles: - Driver installation - Container toolkit - Device plugin - GPU feature discovery - MIG management - Monitoring ### GPU Feature Discovery Automatically labels nodes with GPU details: ```bash $ kubectl describe node gpu-node | grep nvidia nvidia.com/cuda.driver.major=535 nvidia.com/cuda.driver.minor=129 nvidia.com/cuda.runtime.major=12 nvidia.com/gpu.compute.major=8 nvidia.com/gpu.count=4 nvidia.com/gpu.family=ampere nvidia.com/gpu.memory=81920 nvidia.com/gpu.product=NVIDIA-A100-SXM4-80GB nvidia.com/mig.capable=true ``` Now you can select by GPU type: ```yaml nodeSelector: nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB ``` Or with affinity for flexibility: ```yaml affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: nvidia.com/gpu.product operator: In values: - NVIDIA-A100-SXM4-80GB - NVIDIA-A100-PCIE-80GB ``` ## Best Practices ### 1. Right-Size GPU Requests Don't request 8 GPUs for a job that uses 1. GPUs are expensive. ```yaml # Profile your workload first resources: limits: nvidia.com/gpu: 1 # Start small, scale up if needed ``` ### 2. Use MIG for Inference Inference workloads often don't need a full A100: ```yaml resources: limits: nvidia.com/mig-1g.10gb: 1 # 1/7 of an A100 ``` ### 3. Node Pools by GPU Type Separate node pools for different GPU types: ```yaml # Training pool: A100s nodeSelector: gpu-pool: training # Inference pool: T4s (cheaper) nodeSelector: gpu-pool: inference ``` ### 4. Set Resource Quotas Prevent GPU hoarding: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: gpu-quota spec: hard: nvidia.com/gpu: "10" # Max 10 GPUs per namespace ``` ### 5. Use Priority Classes Critical training jobs should preempt development workloads: ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: training-critical value: 1000000 preemptionPolicy: PreemptLowerPriority --- apiVersion: v1 kind: Pod spec: priorityClassName: training-critical # ... ``` ### 6. Monitor GPU Utilization Low utilization = wasted money: ```bash # DCGM exporter metrics dcgm_gpu_utilization dcgm_memory_utilization dcgm_power_usage ``` Alert if GPUs are allocated but idle: ```promql # GPU allocated but <10% utilized for 30m (dcgm_gpu_utilization < 10) and (kube_pod_container_resource_limits{resource="nvidia_com_gpu"} > 0) ``` ## Debugging GPU Scheduling ### Pod Stuck Pending ```bash kubectl describe pod gpu-pod # Common messages: # "Insufficient nvidia.com/gpu" - No nodes with free GPUs # "0/10 nodes available: 10 node(s) didn't match node selector" - Wrong labels ``` ### Check Node GPU Status ```bash # Available GPUs kubectl describe node gpu-node | grep -A 5 Allocatable # Allocated GPUs kubectl describe node gpu-node | grep -A 5 "Allocated resources" ``` ### Verify Device Plugin ```bash # Check device plugin pods kubectl get pods -n nvidia-device-plugin # Check logs kubectl logs -n nvidia-device-plugin -l app=nvidia-device-plugin ``` ### Check GPU Health ```bash # On the node nvidia-smi # Look for: # - GPU memory errors # - Temperature # - Power state # - Running processes ``` ### MIG Configuration Issues ```bash # Check MIG status nvidia-smi mig -lgi # Check device plugin sees MIG devices kubectl logs -n nvidia-device-plugin | grep -i mig ``` ## Summary GPU scheduling in Kubernetes has evolved: | Generation | Mechanism | Capabilities | |------------|-----------|--------------| | 1st | Device Plugins | Whole GPU allocation, simple counting | | 1.5 | + MIG/Time-slicing | Fractional GPUs, but hacks on top of device plugins | | 2nd | DRA | Rich selectors, lifecycle hooks, structured parameters | **Current recommendations:** | Use Case | Approach | |----------|----------| | Simple GPU workloads | Device plugin + node selectors | | Fractional GPUs (A100) | MIG via device plugin | | Shared dev/test GPUs | Time-slicing | | Complex requirements | Evaluate DRA (if K8s 1.31+) | | Multi-GPU training | Topology Manager + GPU Operator | The device plugin model works but hits walls at scale. DRA is the future—a proper API for hardware allocation. As it matures, expect richer GPU scheduling: specific models, memory requirements, topology constraints, all expressible in standard Kubernetes resources. --- ## WebAssembly on Kubernetes: The Next Evolution Beyond Containers - URL: https://svalle.ru/posts/kubernetes/wasm-on-kubernetes/ - Date: 2025-02-11 - Tags: kubernetes, webassembly, wasm, containers, serverless, edge, spinkube Containers revolutionized deployment. But they carry baggage: a full Linux userspace, slow cold starts, and megabytes of image data. For some workloads—especially serverless functions and edge computing—this overhead matters. WebAssembly (Wasm) offers an alternative: millisecond cold starts, tiny binaries, and a sandboxed execution model. And now it runs natively on Kubernetes. This post explains how. ## Why Wasm? ### The Container Overhead Problem When you start a container, a lot happens: ``` 1. Pull image (if not cached): 100MB-1GB, seconds to minutes 2. Create container: mount layers, set up namespaces 3. Start process: load binaries, initialize runtime 4. Ready to serve: 500ms-5s cold start typical ``` For a long-running web server, this doesn't matter. For a serverless function that runs for 50ms, a 2-second cold start is unacceptable. ### Wasm: A Different Model WebAssembly is a binary instruction format designed for: 1. **Fast startup**: No OS to boot, no libraries to load 2. **Small size**: Compact binary format, often <1MB 3. **Sandboxed**: Capabilities must be explicitly granted 4. **Portable**: Same binary runs anywhere with a Wasm runtime ``` Container cold start: 500ms - 5s Wasm cold start: 1ms - 50ms Container image: 50MB - 1GB Wasm module: 100KB - 10MB ``` ### The Famous Quote Solomon Hykes (Docker co-founder), 2019: > "If WASM+WASI existed in 2008, we wouldn't have needed to create Docker. That's how important it is. WebAssembly on the server is the future of computing." ## Wasm Fundamentals ### What Wasm Is Wasm is a compilation target. You write code in Rust, Go, C, Python, JavaScript, or many other languages, and compile it to `.wasm`: ```rust // Rust code fn main() { println!("Hello from Wasm!"); } ``` ```bash # Compile to Wasm cargo build --target wasm32-wasi --release # Output: target/wasm32-wasi/release/hello.wasm (few hundred KB) ``` ### WASI: The System Interface Wasm in browsers has no system access. For servers, we need WASI (WebAssembly System Interface)—a standardized API for: - File system access - Environment variables - Command-line arguments - Random numbers - Clocks - Network (emerging) WASI is capability-based: a Wasm module can only access what the runtime explicitly grants. ```bash # Run with wasmtime, granting file access wasmtime --dir=/data hello.wasm ``` ### The Security Model Containers rely on Linux namespaces and cgroups for isolation. A container escape = host access. Wasm is sandboxed at the instruction level: ```d2 {alt="Wasm sandbox layers: a Wasm module can only call WASI functions with bounds-checked memory and no raw syscalls into the WASI runtime (wasmtime, wasmer, etc.), which grants specific capabilities down to the host OS"} grid-columns: 1 vertical-gap: 88 module: Wasm module runtime: "WASI Runtime (wasmtime, wasmer, etc.)" host: Host OS module -> runtime: "Can only call WASI functions\nMemory is bounds-checked\nNo raw syscalls" runtime -> host: Grants specific capabilities ``` A bug in a Wasm module can't escape the sandbox without a bug in the runtime itself. The attack surface is much smaller than a container. ## Wasm Runtimes Several runtimes execute Wasm: | Runtime | Focus | Used By | |---------|-------|---------| | wasmtime | Standards compliance, security | Bytecode Alliance, Fermyon | | wasmer | Performance, versatility | Wasmer Inc | | WasmEdge | Edge/cloud native | CNCF, second-state | | wazero | Pure Go, no CGO | Go ecosystem | For Kubernetes, the runtime is embedded in a **containerd shim**. ## Running Wasm on Kubernetes ### The Architecture Kubernetes doesn't run Wasm directly. The trick: teach containerd to run Wasm modules as if they were containers. ```d2 {alt="Pod creation flow for Wasm: kubectl create pod goes to the API server, the scheduler selects a Wasm-capable node, then kubelet calls containerd, which uses the wasmtime RuntimeClass to launch containerd-shim-wasmtime, and wasmtime runs the .wasm module"} grid-columns: 1 vertical-gap: 40 create: kubectl create pod {style: {stroke-width: 0; fill: transparent}} api: API Server sched: "Scheduler\n→ selects Wasm-capable node" kubelet: kubelet containerd: containerd shim: containerd-shim-wasmtime run: wasmtime runs .wasm module {style: {stroke-width: 0; fill: transparent}} create -> api -> sched -> kubelet -> containerd containerd -> shim: "(RuntimeClass: wasmtime)" shim -> run ``` The **containerd shim** is the key component. It implements containerd's runtime interface but executes Wasm instead of Linux containers. ### RuntimeClass Kubernetes uses RuntimeClass to select different container runtimes: ```yaml apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: wasmtime handler: wasmtime # Matches containerd config scheduling: nodeSelector: kubernetes.io/wasm: "true" ``` Pods specify which runtime to use: ```yaml apiVersion: v1 kind: Pod metadata: name: wasm-pod spec: runtimeClassName: wasmtime # Use Wasm runtime containers: - name: hello image: ghcr.io/example/hello-wasm:latest command: ["/hello.wasm"] ``` ### containerd Shims The runwasi project provides containerd shims for various Wasm runtimes: ```bash # containerd config.toml [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmtime] runtime_type = "io.containerd.wasmtime.v1" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge] runtime_type = "io.containerd.wasmedge.v1" ``` Available shims: - `containerd-shim-wasmtime-v1` - `containerd-shim-wasmedge-v1` - `containerd-shim-wasmer-v1` - `containerd-shim-spin-v2` (for Spin framework) ### Installing Wasm Support **Option 1: kwasm-operator** (easiest) ```bash # Install kwasm operator helm install kwasm-operator kwasm/kwasm-operator \ --namespace kwasm \ --create-namespace # Label nodes to install Wasm shims kubectl label node worker-1 kwasm.sh/kwasm-node=true ``` **Option 2: Manual installation** ```bash # On each node # Download and install shim curl -LO https://github.com/containerd/runwasi/releases/download/v0.3.0/containerd-shim-wasmtime-v1-linux-amd64.tar.gz tar xzf containerd-shim-wasmtime-v1-linux-amd64.tar.gz mv containerd-shim-wasmtime-v1 /usr/local/bin/ # Update containerd config cat >> /etc/containerd/config.toml << EOF [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmtime] runtime_type = "io.containerd.wasmtime.v1" EOF # Restart containerd systemctl restart containerd ``` ## SpinKube: The Serverless Wasm Platform SpinKube combines the Spin framework with Kubernetes for serverless Wasm workloads. ### What is Spin? Spin is a framework for building serverless Wasm applications: ```rust use spin_sdk::http::{Request, Response}; use spin_sdk::http_component; #[http_component] fn handle_request(req: Request) -> Response { Response::builder() .status(200) .body(Some("Hello from Spin!".into())) .build() } ``` ```bash # Build and run locally spin build spin up # HTTP server on port 3000 ``` ### SpinKube Architecture ```d2 {alt="SpinKube architecture: the spin-operator, which manages SpinApp CRDs, points into a Deployment-like SpinApp sitting between two SpinApp CRDs inside the Kubernetes cluster, which also hosts containerd-shim-spin that runs Spin apps as Wasm"} grid-rows: 2 vertical-gap: 48 operator: "spin-operator\n(manages SpinApp CRDs)" cluster: Kubernetes Cluster { label.near: top-left grid-rows: 2 grid-gap: 24 apps: "" { grid-rows: 1 grid-gap: 24 style: {stroke-width: 0; fill: transparent} a: "SpinApp\nCRD" b: "SpinApp\n(Deployment-like)" c: "SpinApp\nCRD" } shim: "containerd-shim-spin\n(runs Spin apps as Wasm)" } operator -> cluster.apps.b ``` ### Installing SpinKube ```bash # Install cert-manager (required) kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml # Install SpinKube (runtime class, shim, operator) kubectl apply -f https://github.com/spinkube/spin-operator/releases/download/v0.2.0/spin-operator.crds.yaml kubectl apply -f https://github.com/spinkube/spin-operator/releases/download/v0.2.0/spin-operator.runtime-class.yaml kubectl apply -f https://github.com/spinkube/spin-operator/releases/download/v0.2.0/spin-operator.shim-executor.yaml helm install spin-operator oci://ghcr.io/spinkube/charts/spin-operator \ --namespace spin-operator --create-namespace ``` ### Deploying a Spin App ```yaml apiVersion: core.spinoperator.dev/v1alpha1 kind: SpinApp metadata: name: hello-spin spec: image: ghcr.io/spinkube/spin-operator/hello-world:latest replicas: 2 executor: containerd-shim-spin ``` ```bash kubectl apply -f spinapp.yaml # Service is automatically created kubectl get svc hello-spin ``` ### Spin App Features **Autoscaling:** ```yaml apiVersion: core.spinoperator.dev/v1alpha1 kind: SpinApp metadata: name: autoscaled-app spec: image: ghcr.io/example/my-app:latest enableAutoscaling: true resources: limits: cpu: 100m memory: 128Mi ``` **Variables and secrets:** ```yaml spec: variables: - name: API_KEY valueFrom: secretKeyRef: name: my-secret key: api-key ``` ## Building Wasm Applications ### Rust (Best Support) ```rust // src/lib.rs use spin_sdk::http::{Request, Response}; use spin_sdk::http_component; #[http_component] fn handle(req: Request) -> anyhow::Result { let path = req.uri().path(); Ok(Response::builder() .status(200) .header("content-type", "application/json") .body(format!(r#"{{"path": "{}"}}"#, path)) .build()) } ``` ```toml # spin.toml spin_manifest_version = 2 [application] name = "my-app" version = "0.1.0" [[trigger.http]] route = "/..." component = "my-app" [component.my-app] source = "target/wasm32-wasi/release/my_app.wasm" [component.my-app.build] command = "cargo build --target wasm32-wasi --release" ``` ### Go ```go package main import ( "fmt" "net/http" spinhttp "github.com/fermyon/spin/sdk/go/v2/http" ) func main() { spinhttp.Handle(func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "application/json") fmt.Fprintf(w, `{"message": "Hello from Go!"}`) }) } ``` ```bash tinygo build -target=wasi -o main.wasm main.go ``` ### JavaScript/TypeScript ```javascript // src/index.js export async function handler(request, context) { return { status: 200, headers: { "content-type": "application/json" }, body: JSON.stringify({ message: "Hello from JS!" }) }; } ``` Uses the ComponentizeJS toolchain to compile JS to Wasm. ### Python ```python from spin_sdk.http import simple class IncomingHandler(simple.IncomingHandler): def handle_request(self, request): return simple.Response( 200, {"content-type": "application/json"}, b'{"message": "Hello from Python!"}' ) ``` Uses componentize-py under the hood. ## Use Cases ### 1. Serverless Functions The canonical use case. Wasm cold starts are fast enough for per-request scaling: ```d2 {alt="Serverless request lifecycle: a request arrives, Spin starts the Wasm module in roughly 1-10ms, the function executes, responds, and scales to zero, with total latency around 15ms including execution"} grid-columns: 1 vertical-gap: 40 arrive: Request arrives {style: {stroke-width: 0; fill: transparent}} start: "Spin starts Wasm module\n~1-10ms" exec: Execute respond: Respond zero: Scale to zero total: "Total latency: ~15ms including execution" {style: {stroke-width: 0; fill: transparent}} arrive -> start -> exec -> respond -> zero ``` Compare to container-based serverless: 100ms-2s cold start. ### 2. Edge Computing Wasm's small footprint suits edge nodes: | Resource | Container | Wasm | |----------|-----------|------| | Memory | 50MB+ | 1-10MB | | Image size | 100MB+ | 100KB-5MB | | Startup | 500ms+ | 1-10ms | Edge nodes might have 1GB RAM. You can run many more Wasm instances than containers. ### 3. Multi-Tenant Platforms Wasm's sandbox is smaller and safer than containers: - No kernel vulnerabilities to escape to - Memory bounds-checked - Capabilities explicitly granted Running untrusted user code? Wasm is more defensible than containers. ### 4. Plugin Systems Extend applications safely: ```d2 {alt="Plugin system flow: the host application loads plugin.wasm and grants limited capabilities to the Wasm plugin, which can only call host-provided functions, yielding a limited, safe extension"} grid-columns: 1 vertical-gap: 64 app: Application (host) plugin: Plugin (Wasm) result: Limited, safe extension {style: {stroke-width: 0; fill: transparent}} app -> plugin: "Loads plugin.wasm\nGrants limited capabilities" plugin -> result: Can only call host-provided functions ``` Envoy, Vector, OPA, and others use Wasm for plugins. ### 5. AI Inference at Edge Small models running in Wasm at the edge: ```yaml apiVersion: core.spinoperator.dev/v1alpha1 kind: SpinApp metadata: name: inference spec: image: ghcr.io/example/llm-inference:latest resources: limits: memory: 512Mi # Small model fits in Wasm ``` No GPU needed for small models; CPU inference with fast cold starts. ## Wasm vs Containers: When to Use Which ### Use Wasm When: | Scenario | Why Wasm | |----------|----------| | **Serverless/FaaS** | Cold start matters, per-request scaling | | **Edge computing** | Resource constrained, many small workloads | | **Untrusted code** | Smaller attack surface, better sandbox | | **Short-lived tasks** | Don't pay container overhead for 50ms work | | **Plugins/extensions** | Safe, portable, language-agnostic | ### Use Containers When: | Scenario | Why Containers | |----------|----------------| | **Long-running services** | Cold start doesn't matter | | **Full OS needed** | Shell, package managers, debugging tools | | **Complex dependencies** | Native libraries, databases, etc. | | **Existing workloads** | Already containerized, not worth rewriting | | **GPU/hardware access** | Wasm hardware support is limited | | **Network-heavy** | Wasm networking is still evolving | ### The Hybrid Future Most clusters will run both: ```d2 {alt="Hybrid Kubernetes cluster: traditional nodes run web servers, databases, stateful apps, and ML training under the containerd (runc) RuntimeClass, side by side with Wasm-capable nodes running serverless functions, edge processors, event handlers, and plugins under the wasmtime/spin RuntimeClass"} cluster: Kubernetes Cluster { grid-columns: 2 grid-gap: 24 trad: "Traditional Nodes\n\n- Web servers\n- Databases\n- Stateful apps\n- ML training\n\nRuntimeClass:\ncontainerd (runc)" wasm: "Wasm-Capable Nodes\n\n- Serverless functions\n- Edge processors\n- Event handlers\n- Plugins\n\nRuntimeClass:\nwasmtime/spin" } ``` ## Limitations and Challenges ### WASI is Still Evolving WASI 0.2 (current) has limited APIs: - ✅ Filesystem, environment, clocks - ✅ HTTP (via wasi-http) - ⚠️ Sockets (preview) - ❌ Full POSIX compatibility Some things don't compile to Wasm yet. ### Language Support Varies | Language | Support Level | |----------|---------------| | Rust | Excellent | | Go (TinyGo) | Good, some stdlib missing | | C/C++ | Good | | JavaScript | Good (ComponentizeJS) | | Python | Improving (componentize-py) | | Java | Experimental | | .NET | Experimental | ### Debugging is Harder No shell to exec into. Debugging options: - Print statements (captured as logs) - Remote debugging (limited) - Local testing with Spin ### Ecosystem Maturity Container ecosystem: millions of images, decades of tooling. Wasm ecosystem: growing but young. You might need to build things that already exist for containers. ## Getting Started ### Local Development ```bash # Install Spin curl -fsSL https://developer.fermyon.com/downloads/install.sh | bash # Create new app spin new -t http-rust hello-wasm cd hello-wasm # Build and run locally spin build spin up # Visit http://localhost:3000 ``` ### Deploy to Kubernetes ```bash # Build and push OCI image spin registry push ghcr.io/myuser/hello-wasm:latest # Deploy cat < spin -> module -> shim -> k8s ``` Wasm won't replace containers—but for the right workloads (serverless, edge, plugins, multi-tenant), it's a compelling alternative with fundamentally better characteristics. Containers are VMs done right. Wasm is processes done right. Both have their place. --- ## CoreDNS Under Pressure: How We Fixed DNS Bottlenecks with NodeLocal DNSCache - URL: https://svalle.ru/posts/kubernetes/coredns-nodelocal-cache/ - Date: 2025-02-10 - Tags: kubernetes, dns, coredns, networking, performance, debugging We had gang-scheduled jobs that performed DNS lookups at startup. If DNS resolution failed, the pod failed. If one pod in the gang failed, the entire gang restarted. Hundreds of pods restarting simultaneously meant hundreds of DNS queries hitting CoreDNS at once. CoreDNS couldn't keep up, more pods failed, more restarts, more DNS queries—a cascading failure that took down our batch processing pipeline. The fix: NodeLocal DNSCache. But understanding why it works requires understanding how Kubernetes DNS works and why it breaks under load. ## How Kubernetes DNS Works Every pod gets DNS configuration injected via `/etc/resolv.conf`: ```bash $ cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5 ``` Let's break this down: ### The Nameserver `10.96.0.10` is the ClusterIP of the `kube-dns` service (which points to CoreDNS pods): ```bash $ kubectl get svc -n kube-system kube-dns NAME TYPE CLUSTER-IP PORT(S) kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP ``` All DNS queries from all pods go to this single VIP. ### Search Domains When you resolve a name like `my-service`, Kubernetes tries multiple suffixes: ``` 1. my-service.default.svc.cluster.local 2. my-service.svc.cluster.local 3. my-service.cluster.local 4. my-service (absolute) ``` ### The ndots Setting `ndots:5` means: if the name has fewer than 5 dots, try search domains first. ``` "my-service" (0 dots < 5) → try search domains first "api.example.com" (2 dots < 5) → try search domains first! "api.example.com." (trailing dot) → absolute, skip search domains ``` This is where things get expensive. A simple lookup for `api.example.com` generates: ``` 1. api.example.com.default.svc.cluster.local → NXDOMAIN 2. api.example.com.svc.cluster.local → NXDOMAIN 3. api.example.com.cluster.local → NXDOMAIN 4. api.example.com → SUCCESS ``` **Four queries for one resolution.** And each query is a UDP packet through the cluster network. ## The DNS Resolution Path Here's what happens when a pod resolves a name: ```d2 {alt="DNS resolution path: the pod at 10.244.1.5 sends a UDP packet to 10.96.0.10:53, iptables/IPVS kube-proxy rules DNAT it to the CoreDNS pod IP, CoreDNS Pod 10.244.0.10 looks up its cache or forwards upstream, and the response goes back through the same path"} grid-columns: 1 vertical-gap: 48 pod: "Pod (10.244.1.5)" ipt: "iptables/IPVS (kube-proxy rules)" coredns: "CoreDNS Pod (10.244.0.10)" resp: Response back through same path {style: {stroke-width: 0; fill: transparent}} pod -> ipt: "UDP packet to 10.96.0.10:53" ipt -> coredns: DNAT to CoreDNS pod IP coredns -> resp: Lookup in cache or forward upstream ``` Every DNS query: 1. Goes through the pod's network namespace 2. Hits iptables/IPVS rules for the service 3. Gets DNAT'd to a CoreDNS pod 4. Creates a conntrack entry 5. Returns through the same path At scale, this becomes a bottleneck. ## Why CoreDNS Becomes a Bottleneck ### Single Service VIP All cluster DNS traffic funnels through one service IP. Even with multiple CoreDNS replicas, every packet hits the same iptables/IPVS rules: ```d2 {alt="Fan-in: Pods A through D all send DNS queries to the single kube-dns service VIP 10.96.0.10, which forwards to the CoreDNS pods and funnels every packet through the same iptables/IPVS rules, a single bottleneck"} direction: right poda: Pod A podb: Pod B podc: Pod C podd: Pod D kubedns: "10.96.0.10 (kube-dns)" coredns: CoreDNS Pods ipt: "iptables/IPVS\n(single bottleneck)" poda -> kubedns podb -> kubedns podc -> kubedns podd -> kubedns kubedns -> coredns kubedns -> ipt ``` ### Conntrack Table Pressure Every DNS query creates a conntrack entry (even for UDP). The default table size is 131072 entries. With thousands of pods doing DNS lookups: ```bash $ cat /proc/sys/net/netfilter/nf_conntrack_count 128000 # Getting close to limit! $ dmesg | grep conntrack nf_conntrack: table full, dropping packet ``` Dropped packets = failed DNS queries = failed pods. ### Thundering Herd Our gang scheduling scenario: - 100-pod gang starts - Each pod does 3 DNS lookups at startup - Each lookup expands to 4 queries (ndots) - 100 × 3 × 4 = 1,200 DNS queries in milliseconds If CoreDNS can't respond fast enough, queries time out (default: 5 seconds). Pods fail, gang restarts, another 1,200 queries. CoreDNS falls further behind. Cascade. ```d2 {alt="Cascade failure: a gang start fires 1,200 DNS queries that both overwhelm CoreDNS (queue grows, latency increases) and time out (pods fail, gang restarts), the restart and the growing latency both produce more queries, and the loop ends in a cascade"} direction: down start: Gang starts {style: {stroke-width: 0; fill: transparent}} queries: "1,200 DNS queries" overwhelmed: CoreDNS overwhelmed timeouts: Timeouts queue: Queue grows fail: Pods fail latency: Latency increases restarts: Gang restarts more: More queries cascade: Cascade {style: {stroke-width: 0; fill: transparent}} start -> queries queries -> overwhelmed queries -> timeouts overwhelmed -> queue timeouts -> fail queue -> latency fail -> restarts latency -> more restarts -> more restarts -> cascade ``` ### UDP Packet Loss Under load, UDP packets get dropped: - Kernel socket buffer overflow - Network interface queue overflow - iptables processing delays Unlike TCP, UDP has no built-in retry. The application must handle retries, adding latency. ## Debugging DNS Issues ### Symptoms - Pod startup failures with DNS errors - Slow service-to-service communication - Intermittent connection timeouts - CoreDNS pods showing high CPU ### CoreDNS Metrics CoreDNS exposes Prometheus metrics: ```promql # Request rate rate(coredns_dns_requests_total[5m]) # Error rate rate(coredns_dns_responses_total{rcode="SERVFAIL"}[5m]) # Latency histogram_quantile(0.99, rate(coredns_dns_request_duration_seconds_bucket[5m])) # Cache hit rate rate(coredns_cache_hits_total[5m]) / (rate(coredns_cache_hits_total[5m]) + rate(coredns_cache_misses_total[5m])) ``` **Warning signs:** - Request rate spiking - Latency p99 > 100ms - SERVFAIL responses increasing - Cache hit rate dropping ### Testing from a Pod ```bash # Run a debug pod kubectl run debug --image=busybox --rm -it -- sh # Test DNS resolution nslookup kubernetes.default nslookup google.com # Measure timing time nslookup google.com # Check resolv.conf cat /etc/resolv.conf # Verbose DNS query nslookup -debug kubernetes.default ``` ### Using dig ```bash # Install dig (dnsutils) kubectl run debug --image=tutum/dnsutils --rm -it -- bash # Query with timing dig kubernetes.default.svc.cluster.local # Query CoreDNS directly dig @10.96.0.10 kubernetes.default.svc.cluster.local # See full query expansion dig +search my-service # Trace the resolution path dig +trace google.com ``` ### Checking Conntrack On a node: ```bash # Current connections cat /proc/sys/net/netfilter/nf_conntrack_count # Max connections cat /proc/sys/net/netfilter/nf_conntrack_max # Conntrack stats (look for drops) conntrack -S cpu=0 found=0 invalid=1234 ignore=5678 insert=0 insert_failed=100 drop=50 ^^^^ Drops! ``` ### Packet Capture ```bash # On a node, capture DNS traffic tcpdump -i any port 53 -nn # Filter for specific pod tcpdump -i any port 53 and host 10.244.1.5 -nn # Save for analysis tcpdump -i any port 53 -w dns.pcap ``` ## Mitigation Options ### Scale CoreDNS The obvious first step: ```bash kubectl -n kube-system scale deployment coredns --replicas=5 ``` **Helps**: More pods to handle queries. **Doesn't solve**: Traffic still funnels through service VIP. Conntrack pressure remains. Thundering herd still overwhelms. ### Tune ndots Reduce query fan-out by lowering ndots: ```yaml apiVersion: v1 kind: Pod spec: dnsConfig: options: - name: ndots value: "2" ``` With `ndots:2`, names with 2+ dots resolve directly: ``` "api.example.com" (2 dots >= 2) → resolve directly, no search domains "my-service" (0 dots < 2) → still uses search domains ``` **Helps**: Reduces queries for external domains. **Doesn't solve**: Internal service lookups still expand. Thundering herd still a problem. ### Use FQDNs Force absolute lookups with trailing dots: ```go // Instead of http.Get("http://api.example.com/path") // Use http.Get("http://api.example.com./path") // Note trailing dot ``` **Helps**: Eliminates search domain expansion for that lookup. **Doesn't solve**: Requires code changes. Internal services still need search domains. ### Increase CoreDNS Cache ```yaml # CoreDNS Corefile .:53 { cache 300 # Cache for 5 minutes instead of default 30s # ... } ``` **Helps**: More cache hits, fewer upstream queries. **Doesn't solve**: Cold start thundering herd (nothing in cache yet). ## The Fix: NodeLocal DNSCache NodeLocal DNSCache runs a DNS cache on every node. Pods query the local cache instead of the CoreDNS service. ### Architecture ```d2 {alt="Before, all traffic goes Pod to kube-dns Service (10.96.0.10) through iptables/IPVS to the CoreDNS pods; after, the pod queries NodeLocal DNS (169.254.20.10) running on the same node, a cache hit returns a response immediately and only a cache miss goes on to the CoreDNS pods"} grid-columns: 1 vertical-gap: 40 before: "Before (all traffic to CoreDNS)" { grid-rows: 2 grid-columns: 3 grid-gap: 24 # horizontal-gap raised 24 -> 61 so this container's natural width (~700, # side padding = gap) matches the wider After container below; recalibrate # if any label changes horizontal-gap: 61 pod: Pod svc: "kube-dns Service (10.96.0.10)" coredns: CoreDNS Pods sp1: "" {width: 20; height: 20; style.opacity: 0} ipt: "(iptables/IPVS)" {style: {stroke-width: 0; fill: transparent}} sp2: "" {width: 20; height: 20; style.opacity: 0} pod -> svc -> coredns svc -- ipt } after: "After (local cache)" { grid-rows: 2 grid-columns: 4 grid-gap: 24 pod: Pod local: "NodeLocal DNS (169.254.20.10)" hit: "Cache Hit?" resp: Response sp1: "" {width: 20; height: 20; style.opacity: 0} samenode: "(runs on same node)" {style: {stroke-width: 0; fill: transparent}} coredns: CoreDNS Pods sp2: "" {width: 20; height: 20; style.opacity: 0} pod -> local -> hit -> resp hit -> coredns: Cache Miss local -- samenode } ``` ### How It Works 1. **DaemonSet**: NodeLocal DNSCache runs on every node 2. **Link-local IP**: Listens on `169.254.20.10` (node-local, no network hop) 3. **iptables rules**: Redirect DNS traffic to local cache 4. **Cache**: Serves cached responses instantly 5. **Upstream**: Cache misses go to CoreDNS ### Benefits **No service VIP**: Queries don't go through iptables/IPVS for the kube-dns service. **No cross-node traffic**: Cache hits are served locally. **No conntrack for local queries**: Link-local traffic doesn't create conntrack entries. **Survives CoreDNS issues**: Cached entries still work if CoreDNS is temporarily unavailable. **Reduces CoreDNS load**: Only cache misses reach CoreDNS. ## Deploying NodeLocal DNSCache ### Prerequisites - Kubernetes 1.18+ - Know your cluster DNS IP (usually `10.96.0.10`) - Know your cluster domain (usually `cluster.local`) ### Installation ```bash # Download the manifest curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml # Replace placeholders # __PILLAR__DNS__SERVER__ → your kube-dns ClusterIP (e.g., 10.96.0.10) # __PILLAR__LOCAL__DNS__ → 169.254.20.10 # __PILLAR__DNS__DOMAIN__ → cluster.local sed -i 's/__PILLAR__DNS__SERVER__/10.96.0.10/g' nodelocaldns.yaml sed -i 's/__PILLAR__LOCAL__DNS__/169.254.20.10/g' nodelocaldns.yaml sed -i 's/__PILLAR__DNS__DOMAIN__/cluster.local/g' nodelocaldns.yaml # Apply kubectl apply -f nodelocaldns.yaml ``` ### Verify DaemonSet ```bash $ kubectl get ds -n kube-system node-local-dns NAME DESIRED CURRENT READY NODE SELECTOR node-local-dns 50 50 50 $ kubectl get pods -n kube-system -l k8s-app=node-local-dns NAME READY STATUS RESTARTS node-local-dns-abc12 1/1 Running 0 node-local-dns-def34 1/1 Running 0 ... ``` ### Update kubelet Configuration Pods need to use the local DNS. Update kubelet's `--cluster-dns` flag: ```yaml # kubelet configuration clusterDNS: - 169.254.20.10 # NodeLocal DNS ``` Or for new pods only, keep existing kubelet config and let NodeLocal DNSCache's iptables rules intercept traffic to `10.96.0.10`. ### Verify It's Working ```bash # Check pod's resolv.conf kubectl run test --image=busybox --rm -it -- cat /etc/resolv.conf nameserver 169.254.20.10 # Should show local DNS # Or if using iptables interception: nameserver 10.96.0.10 # Original, but traffic is redirected # Test resolution kubectl run test --image=busybox --rm -it -- nslookup kubernetes.default ``` ### Check NodeLocal DNS Metrics ```bash # Port-forward to a node-local-dns pod kubectl port-forward -n kube-system pod/node-local-dns-abc12 9253:9253 # Check metrics curl http://localhost:9253/metrics | grep coredns_cache ``` ## Results After deploying NodeLocal DNSCache: **Before:** - Gang scheduling failures due to DNS timeouts - CoreDNS CPU at 80% during job spikes - DNS p99 latency: 500ms+ during load - Cascading failures from DNS-induced restarts **After:** - Gang scheduling stable - CoreDNS CPU dropped to 20% (only cache misses) - DNS p99 latency: <5ms (local cache hits) - No more DNS-induced cascading failures The local cache absorbs the thundering herd. Even if 100 pods start simultaneously on one node, the local cache serves repeated queries instantly. ## Other DNS Optimizations ### autopath Plugin CoreDNS's `autopath` plugin reduces search domain queries: ```yaml # Corefile .:53 { autopath @kubernetes kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } } ``` With autopath, CoreDNS detects the client's namespace and optimizes the search path. Instead of 4 queries, often just 1-2. ### Negative Cache Tuning Cache NXDOMAIN responses to avoid repeated failed lookups: ```yaml .:53 { cache { success 9984 30 # Cache successful responses for 30s denial 9984 5 # Cache NXDOMAIN for 5s } } ``` ### Pod DNS Policy For pods that only need external DNS: ```yaml spec: dnsPolicy: Default # Use node's DNS, not cluster DNS ``` For pods that need no DNS: ```yaml spec: dnsPolicy: None dnsConfig: nameservers: - 8.8.8.8 ``` ### Application-Level Caching For high-frequency lookups, cache at the application level: ```go // Go: Use a custom resolver with caching resolver := &net.Resolver{ PreferGo: true, Dial: func(ctx context.Context, network, address string) (net.Conn, error) { // Custom dial with caching }, } ``` Or use a sidecar cache like dnsmasq for legacy applications. ## Summary Kubernetes DNS becomes a bottleneck because: | Factor | Impact | |--------|--------| | Single service VIP | All traffic through one point | | ndots expansion | 1 lookup → 4+ queries | | Conntrack entries | Table exhaustion under load | | UDP packet loss | No built-in retry | | Thundering herd | Concurrent startups overwhelm CoreDNS | NodeLocal DNSCache fixes this by: | Benefit | How | |---------|-----| | Local resolution | No cross-node traffic for cache hits | | No service VIP | Bypasses iptables/IPVS bottleneck | | Reduced conntrack | Link-local traffic doesn't track | | Resilience | Cached entries survive CoreDNS issues | For any cluster running batch jobs, gang scheduling, or high pod churn, NodeLocal DNSCache is essential. The thundering herd problem is real, and a local cache is the most effective solution. --- ## Leader Election in Kubernetes Controllers - URL: https://svalle.ru/posts/kubernetes/leader-election/ - Date: 2025-02-09 - Tags: kubernetes, controllers, leader-election, high-availability, distributed-systems You deploy your controller with 3 replicas for high availability. But if all 3 try to reconcile simultaneously, you get duplicate actions, race conditions, and chaos. The solution: leader election. Only one replica is active; the others wait on standby. This post covers how leader election works in Kubernetes, how to implement it, and what happens when things go wrong. ## The Problem: Multiple Active Controllers Without leader election, multiple controller replicas all watch the same resources: ```d2 {alt="Fan-out: a single Pod created event reaches all three controller replicas, and Controller-1, Controller-2, and Controller-3 each create a Deployment, producing duplicates"} direction: right pod: Pod created {style: {stroke-width: 0; fill: transparent}} c1: "Controller-1: Creates Deployment" c2: "Controller-2: Creates Deployment (duplicate!)" c3: "Controller-3: Creates Deployment (duplicate!)" pod -> c1 pod -> c2 pod -> c3 ``` Results: - Duplicate resources created - Conflicting updates overwrite each other - Resource counts are wrong - State becomes inconsistent You need exactly one active controller at a time. ## Leader Election Overview Leader election ensures only one replica (the "leader") is active: ```d2 {alt="Leader election architecture: Controller-1 (LEADER, active), Controller-2 (STANDBY, waiting), and Controller-3 (STANDBY, waiting) each point down to the Kubernetes API Server, which holds the Lock Object (Lease/ConfigMap/Endpoint) with holder controller-1 and renewTime 2025-01-26T10:00:00Z"} direction: down c1: "Controller-1\nLEADER\n(active)" c2: "Controller-2\nSTANDBY\n(waiting)" c3: "Controller-3\nSTANDBY\n(waiting)" apiserver: Kubernetes API Server { lock: "Lock Object (Lease/ConfigMap/Endpoint)\nholder: controller-1\nrenewTime: 2025-01-26T10:00:00Z" } c1 -> apiserver c2 -> apiserver c3 -> apiserver ``` The leader periodically renews its lock. If it stops (crash, network partition), the lock expires and another replica becomes leader. ## How It Works: The Algorithm Kubernetes leader election uses a simple lease-based algorithm: ### Acquiring Leadership ``` 1. Try to create/update lock object with my identity 2. If successful, I'm the leader 3. If lock exists and isn't expired, wait and retry 4. If lock exists but is expired, try to take over ``` ### Maintaining Leadership ``` While I'm the leader: 1. Do controller work 2. Periodically renew the lock (update renewTime) 3. If renewal fails, stop doing work and re-enter election ``` ### Lock Object The lock is a Kubernetes object—historically ConfigMaps or Endpoints, now preferably Leases: ```yaml apiVersion: coordination.k8s.io/v1 kind: Lease metadata: name: my-controller namespace: kube-system spec: holderIdentity: controller-1-abc123 leaseDurationSeconds: 15 acquireTime: "2025-01-26T10:00:00Z" renewTime: "2025-01-26T10:00:10Z" leaseTransitions: 5 ``` Key fields: - `holderIdentity`: Who holds the lock (usually pod name) - `leaseDurationSeconds`: How long the lock is valid without renewal - `renewTime`: Last time the holder renewed - `leaseTransitions`: How many times leadership changed ### Timing Parameters ```go LeaseDuration: 15 * time.Second // Lock valid for this long RenewDeadline: 10 * time.Second // Must renew within this time RetryPeriod: 2 * time.Second // How often to retry acquiring ``` **Timeline for failover:** ``` 0s - Leader renews lock 2s - Leader crashes 15s - Lock expires (LeaseDuration) 15s - Standby notices expired lock 17s - Standby acquires lock, becomes leader Total failover time: ~15 seconds ``` ## Implementation with client-go ### Basic Leader Election ```go package main import ( "context" "os" "time" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/leaderelection" "k8s.io/client-go/tools/leaderelection/resourcelock" "k8s.io/klog/v2" ) func main() { // Create clientset clientset, err := kubernetes.NewForConfig(config) if err != nil { klog.Fatal(err) } // Get pod identity id, err := os.Hostname() if err != nil { klog.Fatal(err) } // Create the lock lock := &resourcelock.LeaseLock{ LeaseMeta: metav1.ObjectMeta{ Name: "my-controller", Namespace: "default", }, Client: clientset.CoordinationV1(), LockConfig: resourcelock.ResourceLockConfig{ Identity: id, }, } // Start leader election ctx, cancel := context.WithCancel(context.Background()) defer cancel() leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{ Lock: lock, LeaseDuration: 15 * time.Second, RenewDeadline: 10 * time.Second, RetryPeriod: 2 * time.Second, Callbacks: leaderelection.LeaderCallbacks{ OnStartedLeading: func(ctx context.Context) { // This is called when we become the leader klog.Info("Started leading") runController(ctx) }, OnStoppedLeading: func() { // This is called when we stop being the leader klog.Info("Stopped leading") os.Exit(0) // Exit so Kubernetes restarts us }, OnNewLeader: func(identity string) { // This is called when leadership changes if identity == id { return // It's us } klog.Infof("New leader elected: %s", identity) }, }, ReleaseOnCancel: true, }) } func runController(ctx context.Context) { // Your controller logic here // This runs only while we're the leader <-ctx.Done() } ``` ### Key Points **OnStartedLeading**: Called when you become leader. Start your controller work here. The context is cancelled when you lose leadership. **OnStoppedLeading**: Called when you lose leadership. Usually you should exit so Kubernetes can restart you cleanly. **ReleaseOnCancel**: If true, releases the lock when context is cancelled (graceful shutdown). ## Implementation with controller-runtime controller-runtime (used by Kubebuilder/Operator SDK) has built-in leader election: ```go package main import ( "os" "sigs.k8s.io/controller-runtime/pkg/manager" "sigs.k8s.io/controller-runtime/pkg/manager/signals" ) func main() { mgr, err := manager.New(config, manager.Options{ // Enable leader election LeaderElection: true, LeaderElectionID: "my-controller.example.com", LeaderElectionNamespace: "default", // Optional: customize timing LeaseDuration: durationPtr(15 * time.Second), RenewDeadline: durationPtr(10 * time.Second), RetryPeriod: durationPtr(2 * time.Second), }) if err != nil { os.Exit(1) } // Add your controller to the manager if err := (&MyReconciler{}).SetupWithManager(mgr); err != nil { os.Exit(1) } // Start manager - handles leader election automatically if err := mgr.Start(signals.SetupSignalHandler()); err != nil { os.Exit(1) } } func durationPtr(d time.Duration) *time.Duration { return &d } ``` That's it! The manager handles: - Acquiring leadership before starting controllers - Renewing the lock periodically - Stopping controllers when leadership is lost - Graceful shutdown and lock release ### Kubebuilder Projects In Kubebuilder-generated projects, enable in `main.go`: ```go var enableLeaderElection bool func init() { flag.BoolVar(&enableLeaderElection, "leader-elect", false, "Enable leader election for controller manager.") } func main() { // ... mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{ LeaderElection: enableLeaderElection, LeaderElectionID: "my-operator.example.com", }) // ... } ``` Then deploy with `--leader-elect=true`: ```yaml spec: containers: - name: controller args: - --leader-elect=true ``` ## Leader Election in the Wild ### kube-controller-manager The built-in controller manager uses leader election: ```bash kubectl get lease -n kube-system kube-controller-manager -o yaml apiVersion: coordination.k8s.io/v1 kind: Lease metadata: name: kube-controller-manager namespace: kube-system spec: holderIdentity: master-1_abc123 leaseDurationSeconds: 15 renewTime: "2025-01-26T10:00:00.000000Z" ``` ### kube-scheduler Same pattern: ```bash kubectl get lease -n kube-system kube-scheduler -o yaml ``` ### Checking Current Leader ```bash # For any controller using Lease kubectl get lease -n \ -o jsonpath='{.spec.holderIdentity}' ``` ## Handling Edge Cases ### Graceful Shutdown When the leader terminates gracefully (SIGTERM), it should release the lock: ```go // With ReleaseOnCancel: true leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{ // ... ReleaseOnCancel: true, // Release lock on graceful shutdown }) ``` This allows immediate failover instead of waiting for lock expiry. ### Ungraceful Termination If the leader crashes (SIGKILL, node failure), it can't release the lock. Other replicas must wait for expiry: ```d2 {alt="Crash failover timeline: the leader crashes with no graceful release, other replicas see the lock still held, wait out the LeaseDuration (15s default) until the lock expires, and a new leader acquires the lock"} grid-columns: 1 vertical-gap: 64 crash: "Leader crashes (no graceful release)" held: Other replicas see lock still held wait: "Wait for LeaseDuration (15s default)" expire: Lock expires acquire: New leader acquires lock crash -> held -> wait -> expire -> acquire ``` **Trade-off**: Shorter LeaseDuration = faster failover but more API server load from frequent renewals. ### Network Partition (Split Brain?) What if the leader can't reach the API server but is still running? ```d2 {alt="Sequence diagram of a network partition: the Leader's renew to the API Server fails due to a network issue, the Standby then acquires the lock from the API Server and receives OK (new leader), leaving the old Leader thinking it's still leader while the Standby becomes the new leader"} shape: sequence_diagram leader: Leader apiserver: API Server standby: Standby leader -> apiserver: "renew (fails)" leader."(network issue)" standby -> apiserver: acquire lock apiserver -> standby: "OK (new leader)" leader."Leader thinks\nit's still leader?" standby."Standby becomes\nnew leader!" ``` **The old leader MUST stop working when it can't renew.** This is why RenewDeadline exists: ```go RenewDeadline: 10 * time.Second // Must renew within 10s ``` If renewal fails for 10 seconds, the leader: 1. Stops doing work (context cancelled) 2. Calls OnStoppedLeading 3. Usually exits **Critical**: Your controller must respect the context. If it ignores cancellation, you get split-brain: ```go // GOOD: Respects context func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { // Check context throughout long operations select { case <-ctx.Done(): return ctrl.Result{}, ctx.Err() default: } // Do work... } // BAD: Ignores context func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { // Long-running work that ignores ctx time.Sleep(30 * time.Second) // Doesn't check ctx! doWork() } ``` ### Clock Skew Leader election relies on time. Significant clock skew between nodes can cause issues: ``` Node A clock: 10:00:00 Node B clock: 10:00:30 (30s ahead) Node A holds lock, renewTime = 10:00:00 Node B sees lock, thinks it expired 15s ago! Node B tries to take over... ``` **Mitigation**: - Use NTP to keep clocks synchronized - Kubernetes tolerates small skew (a few seconds) - LeaseDuration should be >> expected clock skew ### Namespace Considerations The lock object must be in a namespace the controller can access: ```go LeaderElectionNamespace: "my-controller-system", ``` Common patterns: - Same namespace as the controller deployment - `kube-system` for cluster-wide controllers - Dedicated namespace for all controllers' locks ## Debugging Leader Election ### Who's the Leader? ```bash # Get current leader kubectl get lease my-controller -n default -o jsonpath='{.spec.holderIdentity}' controller-1-abc123 # Get full lease details kubectl get lease my-controller -n default -o yaml ``` ### Why Isn't My Replica Becoming Leader? Check the lease: ```bash kubectl describe lease my-controller -n default Name: my-controller Namespace: default ... Spec: Holder Identity: controller-1-abc123 Lease Duration Seconds: 15 Renew Time: 2025-01-26T10:00:00.000000Z ``` **If renewTime is recent**: Current leader is healthy. Your replica is correctly waiting. **If renewTime is stale**: Lock should have expired. Check if your replica has permission to update the lease: ```bash # Check RBAC kubectl auth can-i update leases.coordination.k8s.io --as=system:serviceaccount:default:my-controller ``` ### Frequent Leadership Changes If leadership bounces between replicas: ```bash kubectl get lease my-controller -o jsonpath='{.spec.leaseTransitions}' ``` High `leaseTransitions` indicates instability. Common causes: - Network instability between controller and API server - Controller crash-looping - Resource starvation (CPU/memory) causing slow renewals - API server overload causing timeout on renewals ### Controller Not Stopping After Losing Leadership Check logs for: ``` "Stopped leading" ``` If this doesn't appear, or controller continues working: - `OnStoppedLeading` might not be calling `os.Exit()` - Context isn't being propagated/respected - Long-running operations ignoring cancellation ## High Availability Patterns ### Active-Passive (Leader Election) What we've discussed: one active, others standby. ``` Replicas: 3 Active: 1 Failover time: ~15 seconds ``` ### Active-Active (Sharding) For some controllers, you can shard work across replicas: ```go // Each replica handles different namespaces func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { if !r.shouldHandle(req.Namespace) { return ctrl.Result{}, nil // Let another replica handle it } // ... } func (r *Reconciler) shouldHandle(namespace string) bool { hash := fnv.New32() hash.Write([]byte(namespace)) return hash.Sum32() % r.totalReplicas == r.replicaIndex } ``` **Pros**: Better throughput, no failover delay **Cons**: More complex, need to handle rebalancing ### Hybrid Use leader election for cluster-scoped resources, sharding for namespaced: ```go // Cluster-scoped: requires leadership // Namespaced: sharded across replicas ``` ## RBAC for Leader Election Your controller's ServiceAccount needs permission to manage the lock: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: leader-election-role namespace: default rules: - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "create", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: leader-election-rolebinding namespace: default roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: leader-election-role subjects: - kind: ServiceAccount name: my-controller namespace: default ``` If using ConfigMaps or Endpoints (legacy): ```yaml rules: - apiGroups: [""] resources: ["configmaps"] # or "endpoints" verbs: ["get", "create", "update"] ``` ## Best Practices ### 1. Use Leases Leases are purpose-built for leader election. ConfigMaps and Endpoints work but have drawbacks: - ConfigMaps: Extra data in etcd - Endpoints: Confusion with actual service endpoints ```go lock := &resourcelock.LeaseLock{...} // Preferred ``` ### 2. Unique Lock Names Include your controller/operator name to avoid conflicts: ```go LeaderElectionID: "my-company.my-operator.example.com" ``` ### 3. Exit on Leadership Loss Don't try to be clever. When you lose leadership, exit: ```go OnStoppedLeading: func() { klog.Info("Lost leadership, exiting") os.Exit(0) // Let Kubernetes restart us }, ``` Trying to re-acquire in the same process is fragile. ### 4. Respect Context Cancellation All reconciliation should check context: ```go func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { if err := ctx.Err(); err != nil { return ctrl.Result{}, err } // ... } ``` ### 5. Monitor Leadership Expose metrics about leadership: ```go var isLeader = prometheus.NewGauge(prometheus.GaugeOpts{ Name: "controller_is_leader", Help: "1 if this instance is the leader, 0 otherwise", }) OnStartedLeading: func(ctx context.Context) { isLeader.Set(1) runController(ctx) }, OnStoppedLeading: func() { isLeader.Set(0) os.Exit(0) }, ``` ## Summary Leader election ensures exactly one controller replica is active: | Component | Purpose | |-----------|---------| | **Lock object** (Lease) | Stores current leader identity | | **LeaseDuration** | How long lock is valid | | **RenewDeadline** | Max time to renew before giving up | | **RetryPeriod** | How often standbys check the lock | **Key timing:** - Graceful failover: Immediate (lock released) - Ungraceful failover: LeaseDuration (default 15s) **Implementation:** - client-go: `leaderelection.RunOrDie()` with callbacks - controller-runtime: `LeaderElection: true` in manager options **Critical rules:** 1. Leader must stop work when it can't renew 2. All work must respect context cancellation 3. Exit on leadership loss—don't try to recover 4. Use Leases, not ConfigMaps/Endpoints Leader election is what makes HA controllers possible. Without it, you get chaos. With it, you get automatic failover with minimal downtime. --- ## CPU Throttling in Kubernetes: Why Your Limits Are Lying to You - URL: https://svalle.ru/posts/kubernetes/cpu-throttling/ - Date: 2025-02-08 - Tags: kubernetes, performance, cgroups, cpu, throttling, debugging Your application has plenty of CPU headroom—at least according to your metrics. Average CPU is 200m, limit is 1000m. But requests are timing out, p99 latency is through the roof, and users are complaining. The culprit: CPU throttling. Your containers are being throttled even though they're nowhere near their "limit." This post explains why, how to detect it, and what to do about it. ## The CPU Limit Lie When you set a CPU limit, you're not saying "this container can use up to 1 CPU." You're saying "in any 100ms period, this container can use up to 100ms of CPU time." That's a very different statement. ```yaml resources: limits: cpu: "1" # NOT "up to 1 CPU" # Actually: "100ms of CPU time per 100ms period" ``` This is the CFS (Completely Fair Scheduler) bandwidth control mechanism. And it's the source of most CPU throttling pain. ## How CFS Bandwidth Control Works The Linux CFS scheduler allocates CPU time in periods (default: 100ms). Your CPU limit translates to a quota within that period. ``` CPU Limit Period Quota --------- ------ ----- 500m 100ms 50ms 1 100ms 100ms 2 100ms 200ms ``` In any 100ms period, your container can use up to its quota of CPU time. Once exhausted, it's **throttled**—all threads are blocked until the next period starts. ``` Period 1 (0-100ms) |████████████████░░░░░░░░░░░░░░░░░░░░░| ^ ^ ^ | | | Start Quota exhausted Period ends at 40ms (throttled for 60ms!) ``` ### The Burstiness Problem Applications aren't steady-state. They burst. A web server might idle at 50m CPU, then spike to 800m when a request arrives. ``` Actual CPU usage pattern: Time: 0ms 20ms 40ms 60ms 80ms 100ms |------|------|------|------|------| CPU: [50m ][850m ][50m ][50m ][50m ] ^ Burst to handle request With 500m limit (50ms quota): |------|------|------|------|------| [50m ][████THROTTLED████][50m ] ^ ^ | | Quota exhausted at 25ms Throttled until 100ms! ``` Your average CPU is well under the limit. But the burst exceeded the quota within a single period. Result: 75ms of throttling. That request that should have taken 30ms? It took 105ms because your container was frozen for 75ms waiting for the next period. ## Measuring Throttling ### cgroup Metrics Throttling is tracked in the cgroup filesystem: ```bash # For cgroups v2 cat /sys/fs/cgroup//cpu.stat usage_usec 123456789 user_usec 100000000 system_usec 23456789 nr_periods 50000 nr_throttled 5000 # <-- Throttled 5000 times! throttled_usec 300000000 # <-- 300 seconds total throttle time ``` Key metrics: - `nr_periods`: Total scheduling periods - `nr_throttled`: Periods where container was throttled - `throttled_usec`: Total time spent throttled (microseconds) ### Calculating Throttle Percentage ``` Throttle % = (nr_throttled / nr_periods) × 100 Example: nr_throttled = 5000 nr_periods = 50000 Throttle % = 10% ``` 10% throttling means in 10% of all 100ms periods, your container hit its quota and was frozen. ### Prometheus Metrics If you're using cAdvisor or the kubelet metrics endpoint: ```promql # Throttling rate rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) # Throttle time per second rate(container_cpu_cfs_throttled_seconds_total[5m]) ``` ### kubectl top vs Reality ```bash kubectl top pod my-pod NAME CPU(cores) MEMORY(bytes) my-pod 200m 512Mi ``` This shows **average** CPU over the sampling window. It doesn't show: - Bursts within the window - Throttling events - Per-period behavior You can be at 200m average and still be heavily throttled because of bursts. ## The Multi-Core Trap CPU limits get even more confusing with multi-threaded applications. ### Scenario: 4 Threads, 2 CPU Limit ```yaml resources: limits: cpu: "2" # 200ms quota per 100ms period ``` Your app has 4 threads that all wake up to handle a request: ``` Thread 1: |██████████| (50ms CPU) Thread 2: |██████████| (50ms CPU) Thread 3: |██████████| (50ms CPU) Thread 4: |██████████| (50ms CPU) ^ ^ 0ms 50ms Total CPU time used: 200ms Quota: 200ms per 100ms period ``` All 4 threads ran for 50ms wall-clock time, consuming 200ms of CPU time total. Quota exhausted at wall-clock 50ms. All threads throttled for remaining 50ms of the period. ``` Wall clock: 0ms 50ms 100ms |------|-----------| Thread 1: [█████][THROTTLED ] Thread 2: [█████][THROTTLED ] Thread 3: [█████][THROTTLED ] Thread 4: [█████][THROTTLED ] ``` From the application's perspective, a task that needed 50ms of wall-clock time took 100ms because of throttling. ### The Parallel Burst Problem High-parallelism workloads burn through quota fast: ``` 8 threads × 25ms each = 200ms CPU time With 2 CPU limit (200ms quota): Quota exhausted in 25ms wall-clock time! ``` Even though you're "only" using 2 CPUs worth of work, you're using it all at once. The quota doesn't care about parallelism—it's a budget of CPU microseconds. ## Real-World Impact ### Latency Spikes Throttling causes latency spikes, not average latency increases: | Scenario | p50 | p99 | Note | |----------|-----|-----|------| | Without throttling | 10ms | 30ms | | | With 20% throttling | 10ms | 130ms | ← Tail latency explodes | When you get throttled, you wait up to 100ms for the next period. This adds 100ms to whatever you were doing. ### Cascading Failures Service A calls Service B with a 100ms timeout. Service B gets throttled for 80ms. Service A times out. Service A retries. Service B gets more requests. More throttling. Cascade. ### Health Check Failures Kubelet sends a health check. Container is throttled. Health check times out. Kubelet kills the pod. Repeat. ```yaml livenessProbe: httpGet: path: /health timeoutSeconds: 1 # 1 second might not be enough if throttled ``` ## Diagnosing Throttling ### Step 1: Check if Throttling is Happening ```bash # Find the cgroup path CONTAINER_ID=$(kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].containerID}' | cut -d'/' -f3) # SSH to the node and check cat /sys/fs/cgroup/kubepods/pod//cpu.stat ``` Or use Prometheus: ```promql # Top 10 throttled containers topk(10, rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]) / rate(container_cpu_cfs_periods_total{container!=""}[5m]) ) ``` ### Step 2: Compare Throttling to Utilization ```promql # CPU utilization rate(container_cpu_usage_seconds_total{container="my-container"}[5m]) # Throttle rate rate(container_cpu_cfs_throttled_periods_total{container="my-container"}[5m]) / rate(container_cpu_cfs_periods_total{container="my-container"}[5m]) ``` If utilization is low but throttling is high, you have bursty workloads hitting quota limits. ### Step 3: Look at Request Patterns Are latency spikes correlated with incoming request bursts? Use tracing or request logs to correlate. ``` 10:00:00 - Request burst (50 concurrent) 10:00:00 - Throttle rate spikes to 40% 10:00:00 - p99 latency spikes to 500ms ``` ## Solutions ### Option 1: Remove CPU Limits The nuclear option: don't set CPU limits at all. ```yaml resources: requests: cpu: "500m" # Scheduler uses this for placement # No limits! # Container can burst freely ``` **Pros:** - No throttling, ever - Bursts are handled gracefully **Cons:** - Noisy neighbor problem: One container can starve others - Harder capacity planning - May violate resource quotas This is appropriate for: - Latency-sensitive workloads - Trusted workloads (you control all code on the node) - When requests are set correctly for bin packing ### Option 2: Set Limits = Requests (Guaranteed QoS) ```yaml resources: requests: cpu: "2" limits: cpu: "2" ``` With Guaranteed QoS, Kubernetes gives you dedicated resources. Less contention, more predictable throttling. **Pros:** - Predictable behavior - Highest priority during resource pressure **Cons:** - Can't burst above request - May waste resources if workload is variable ### Option 3: Increase Limits If you're throttling at 1 CPU limit with 500m average usage, try 2 CPU limit: ```yaml resources: requests: cpu: "500m" # What you typically use limits: cpu: "2" # Headroom for bursts ``` **Rule of thumb**: Set limits to 2-3x your p99 CPU usage, not your average. ### Option 4: Tune CFS Period (Advanced) The default 100ms period can be changed. Shorter periods reduce max throttle duration but increase overhead. ```bash # Check current period (in microseconds) cat /sys/fs/cgroup/cpu/kubepods/cpu.cfs_period_us 100000 # Shorter period (10ms) - requires node configuration echo 10000 > /sys/fs/cgroup/cpu/kubepods/cpu.cfs_period_us ``` With 10ms periods and 1 CPU limit: - Quota: 10ms per period - Max throttle duration: 10ms instead of 100ms - More frequent throttling, but shorter each time **Trade-off**: Shorter periods have higher scheduling overhead. ### Option 5: CPU Manager (Static Policy) For latency-critical workloads, use CPU Manager with static policy to get dedicated CPUs: ```yaml # kubelet configuration cpuManagerPolicy: static ``` ```yaml # Pod with integer CPU request gets dedicated cores resources: requests: cpu: "2" # Must be integer limits: cpu: "2" # Must equal requests ``` With static policy, containers with integer CPU requests get pinned to specific CPU cores. No CFS bandwidth control, no throttling. **Pros:** - Zero throttling - Best latency **Cons:** - Requires Guaranteed QoS - Must request whole CPUs - Fragments node capacity ### Option 6: Reduce Parallelism If your app spawns too many threads: ```go // Bad: Unbounded parallelism for _, item := range items { go process(item) } // Better: Bounded parallelism sem := make(chan struct{}, runtime.NumCPU()) for _, item := range items { sem <- struct{}{} go func(item Item) { defer func() { <-sem }() process(item) }(item) } ``` For GOMAXPROCS in Go, or thread pools in other languages, consider setting them based on your CPU limit, not the node's CPUs: ```go import _ "go.uber.org/automaxprocs" // Automatically sets GOMAXPROCS based on cgroup ``` ## Kubernetes 1.20+: CPUThrottlingEnabled Feature Gate Kubernetes has been working on improvements to CPU throttling visibility and control. The `PodAndContainerStatsFromCRI` feature exposes throttling metrics more consistently. Check your cluster's feature gates: ```bash kubectl get cm -n kube-system kubelet-config -o yaml | grep -i throttl ``` ## Practical Recommendations ### For Latency-Sensitive Services ```yaml resources: requests: cpu: "500m" limits: cpu: "2000m" # 4x headroom for bursts, or remove entirely ``` Or remove limits entirely if you trust your workloads. ### For Batch Jobs ```yaml resources: requests: cpu: "1" limits: cpu: "1" # Guaranteed QoS, predictable scheduling ``` Batch jobs care about throughput, not latency. Throttling is acceptable. ### For Mixed Workloads Separate latency-sensitive and batch workloads onto different nodes using taints/tolerations: ```yaml # Latency-sensitive nodes: no CPU limits enforced # Batch nodes: CPU limits enforced, bin-packed ``` ### Monitoring Recommendations Always monitor: ```promql # Alert if any container is throttled more than 25% avg( rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) ) by (namespace, pod, container) > 0.25 ``` ## Summary CPU limits in Kubernetes don't mean what you think: | What You Think | Reality | |----------------|---------| | "Max 1 CPU" | "100ms CPU time per 100ms period" | | "I'm at 50% utilization" | "I might still be throttled on bursts" | | "Limit > usage, so no problem" | "Bursts within a period can still throttle" | Throttling causes: - Latency spikes (not averages) - p99 degradation - Cascading failures - Health check timeouts Diagnose with: - cgroup `cpu.stat`: `nr_throttled`, `throttled_usec` - Prometheus: `container_cpu_cfs_throttled_*` metrics Fix with: - Remove limits (for trusted, latency-sensitive workloads) - Increase limits (2-3x peak, not average) - Guaranteed QoS (limits = requests) - CPU Manager static policy (dedicated cores) - Reduce application parallelism The safest approach for latency-sensitive services: set CPU requests accurately for scheduling, and either remove limits or set them very high. Let the scheduler handle placement; don't let CFS bandwidth control ruin your latency. --- ## How the Kubernetes Scheduler Actually Works - URL: https://svalle.ru/posts/kubernetes/kubernetes-scheduler-deep-dive/ - Date: 2025-02-07 - Tags: kubernetes, scheduler, internals, performance A pod is created. Seconds later, it's running on a node. But how did Kubernetes decide *which* node? The scheduler—kube-scheduler—makes this decision hundreds of times per second in large clusters. Understanding how it works helps you debug "Pending" pods and optimize placement. ## The Scheduling Problem The scheduler solves a bin-packing problem: given N pods and M nodes, assign each pod to a node such that: 1. **Constraints are satisfied** — resource requests fit, taints/tolerations match, affinity rules hold 2. **Resources are balanced** — don't overload some nodes while others sit idle 3. **Preferences are respected** — spread pods across zones, colocate related pods This is NP-hard in the general case, so the scheduler uses heuristics—fast, good-enough decisions rather than optimal ones. ## The Scheduling Cycle When a pod is created without a `nodeName`, it enters the scheduling queue. The scheduler processes it through two phases: ```d2 {alt="Pod scheduling flow: a pod created with empty nodeName enters the scheduling queue, goes through the scheduling cycle (filtering, scoring, binding), and ends up bound to a node"} direction: down created: Pod created (nodeName empty) {shape: text} queue: Scheduling Queue cycle: Scheduling Cycle { filtering: "1. Filtering\nWhich nodes CAN run this pod?" scoring: "2. Scoring\nWhich node is BEST?" binding: "3. Binding\nAssign pod to chosen node" } bound: Pod bound to node {shape: text} created -> queue -> cycle -> bound ``` ### Phase 1: Filtering Filtering eliminates nodes that *cannot* run the pod. Each filter plugin checks one constraint: ```d2 {alt="Filtering funnel: all five nodes pass through NodeResourcesFit, NodeAffinity, and TaintToleration filter plugins in sequence, with the remaining node list shrinking after each filter until only the feasible nodes for scoring are left"} grid-columns: 1 vertical-gap: 40 all: "All Nodes: [node-1, node-2, node-3, node-4, node-5]" {style: {stroke-width: 0; fill: transparent}} fit: "NodeResourcesFit: Does node have enough CPU/mem?" affinity: "NodeAffinity: Does node match required affinity?" taint: "TaintToleration: Does pod tolerate node taints?" feasible: Feasible nodes for scoring {style: {stroke-width: 0; fill: transparent}} all -> fit fit -> affinity: "Remaining: [node-1, node-2, node-4, node-5]" affinity -> taint: "Remaining: [node-1, node-2, node-5]" taint -> feasible: "Remaining: [node-1, node-5]" ``` **Built-in filter plugins:** | Plugin | What it checks | |--------|----------------| | `NodeResourcesFit` | CPU, memory, ephemeral storage requests fit | | `NodePorts` | Requested host ports are available | | `NodeAffinity` | Node matches `nodeAffinity` rules | | `TaintToleration` | Pod tolerates node's taints | | `PodTopologySpread` | Spread constraints are satisfiable | | `VolumeBinding` | Required PVs can be bound to this node | | `InterPodAffinity` | Pod affinity/anti-affinity constraints | | `NodeUnschedulable` | Node isn't cordoned | If no nodes pass filtering, the pod stays Pending. ### Phase 2: Scoring Scoring ranks the feasible nodes. Each scoring plugin assigns a score (0-100), and scores are weighted and summed: ```d2 {alt="Scoring pipeline: feasible nodes node-1 and node-5 pass through the NodeResourcesBalancedAllocation, InterPodAffinity, and ImageLocality scoring plugins, each assigning per-node scores with reasons, ending in weighted final scores of 230 for node-1 and 160 for node-5, making node-1 the winner"} grid-columns: 1 vertical-gap: 64 feasible: "Feasible nodes: [node-1, node-5]" {style: {stroke-width: 0; fill: transparent}} balanced: "NodeResourcesBalancedAllocation\nnode-1: 60 (moderate utilization)\nnode-5: 80 (low utilization)" interpod: "InterPodAffinity\nnode-1: 100 (preferred pods nearby)\nnode-5: 50 (no preferred pods)" image: "ImageLocality\nnode-1: 70 (some images cached)\nnode-5: 30 (need to pull images)" final: "Final scores (weighted sum):\nnode-1: 230\nnode-5: 160\n\nWinner: node-1" {style: {stroke-width: 0; fill: transparent}} feasible -> balanced -> interpod -> image -> final ``` **Built-in scoring plugins:** | Plugin | What it scores | |--------|----------------| | `NodeResourcesBalancedAllocation` | Prefer balanced CPU/memory usage | | `NodeResourcesLeastAllocated` | Prefer nodes with most free resources | | `NodeResourcesMostAllocated` | Prefer nodes with least free resources (bin packing) | | `InterPodAffinity` | Prefer nodes matching pod affinity | | `ImageLocality` | Prefer nodes with container images cached | | `TaintToleration` | Prefer nodes with fewer taints | | `NodeAffinity` | Prefer nodes matching preferred affinity | | `PodTopologySpread` | Prefer nodes that balance spread | ### Phase 3: Binding Once a node is selected, the scheduler "binds" the pod: 1. **Optimistic binding**: Scheduler assumes success, updates internal cache 2. **API binding**: Sends Binding object to API server 3. **Kubelet takes over**: Kubelet sees pod assigned to its node, starts it ```go // Simplified binding binding := &v1.Binding{ ObjectMeta: metav1.ObjectMeta{ Name: pod.Name, Namespace: pod.Namespace, }, Target: v1.ObjectReference{ Kind: "Node", Name: selectedNode, }, } client.CoreV1().Pods(pod.Namespace).Bind(ctx, binding, metav1.CreateOptions{}) ``` ## Scheduling Queue Internals The scheduler doesn't process pods in simple FIFO order. It uses a priority queue with three sub-queues: ```d2 {alt="Scheduling queue sub-queues: pods move from ActiveQ (ready to schedule, heap by priority) to BackoffQ (waiting after failure with exponential backoff) when scheduling fails, and from BackoffQ to UnschedulableQ (can't be scheduled, waiting for change) when cluster state changes"} grid-columns: 1 vertical-gap: 64 active: "ActiveQ\nPods ready to schedule (heap by priority)" backoff: "BackoffQ\nPods waiting after failure (exponential backoff)" unschedulable: "UnschedulableQ\nPods that can't be scheduled (waiting for change)" active -> backoff: (scheduling fails) backoff -> unschedulable: (cluster state changes) ``` ### ActiveQ Pods ready for scheduling, ordered by priority: ```go // PriorityClass affects queue position apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000000 // Higher = scheduled first globalDefault: false ``` ### BackoffQ When scheduling fails (e.g., race condition, transient error), pods go here with exponential backoff: ``` 1st failure: wait 1s 2nd failure: wait 2s 3rd failure: wait 4s ...up to 10s max ``` ### UnschedulableQ When filtering finds no feasible nodes, pods go here. They're retried when cluster state changes (node added, pod deleted, etc.). ``` Pod can't fit anywhere → UnschedulableQ Node added to cluster → Move pods back to ActiveQ ``` ## Resource Requests and Limits The scheduler only considers **requests**, not limits: ```yaml resources: requests: cpu: 100m # Scheduler uses this memory: 128Mi # Scheduler uses this limits: cpu: 500m # Scheduler ignores this memory: 512Mi # Scheduler ignores this ``` **Why?** Requests represent guaranteed resources. Limits allow bursting but aren't guaranteed. The scheduler ensures the sum of requests fits on the node. ### Allocatable vs Capacity Nodes report both capacity and allocatable: ```bash kubectl describe node worker-1 | grep -A 6 "Capacity\|Allocatable" Capacity: cpu: 4 memory: 16Gi pods: 110 Allocatable: cpu: 3800m # 200m reserved for system memory: 15Gi # 1Gi reserved for system pods: 110 ``` The scheduler uses **allocatable**, which excludes resources reserved for kubelet, OS, etc. ### Extended Resources Custom resources (GPUs, FPGAs, etc.) work the same way: ```yaml # Node advertises GPUs status: allocatable: nvidia.com/gpu: 4 # Pod requests GPUs resources: requests: nvidia.com/gpu: 2 # Scheduler checks this fits ``` ## Node Selection Deep Dive ### Node Selectors Simple label matching: ```yaml spec: nodeSelector: disktype: ssd zone: us-west-2a ``` All labels must match. No flexibility. ### Node Affinity More expressive, with required and preferred rules: ```yaml spec: affinity: nodeAffinity: # Hard requirement (like nodeSelector but with operators) requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/zone operator: In values: - us-west-2a - us-west-2b # Soft preference (try but don't require) preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: node-type operator: In values: - high-memory ``` **Operators:** `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`, `Lt` ### Taints and Tolerations Taints repel pods; tolerations allow pods to schedule despite taints: ```bash # Taint a node kubectl taint nodes worker-1 dedicated=ml:NoSchedule ``` ```yaml # Pod that tolerates the taint spec: tolerations: - key: dedicated operator: Equal value: ml effect: NoSchedule ``` **Taint effects:** - `NoSchedule`: Don't schedule new pods (existing stay) - `PreferNoSchedule`: Try not to schedule (soft) - `NoExecute`: Evict existing pods + don't schedule new ### Pod Affinity and Anti-Affinity Place pods relative to other pods: ```yaml spec: affinity: # Run near pods with app=cache podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: cache topologyKey: kubernetes.io/hostname # Don't run on same node as other app=web pods podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app: web topologyKey: kubernetes.io/hostname ``` **topologyKey**: Defines what "same place" means: - `kubernetes.io/hostname`: Same node - `topology.kubernetes.io/zone`: Same availability zone - `topology.kubernetes.io/region`: Same region **Warning**: Pod affinity with `requiredDuringScheduling` can make pods unschedulable if the target pods don't exist yet. ### Pod Topology Spread Distribute pods evenly across topology domains: ```yaml spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: web ``` This ensures pods are spread across zones with at most 1 pod difference between any two zones. ## Preemption When a high-priority pod can't be scheduled, the scheduler may **preempt** (evict) lower-priority pods: ```d2 {alt="Preemption flow: a pending high-priority pod triggers finding nodes where preemption would allow scheduling, then selecting victim pods to evict, then evicting the victims and scheduling the pod"} grid-columns: 1 vertical-gap: 64 pending: High-priority pod pending {style: {stroke-width: 0; fill: transparent}} find: "Find nodes where preemption\nwould allow scheduling" select: Select victim pods to evict evict: "Evict victims, schedule pod" pending -> find -> select -> evict ``` ### PriorityClasses ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: critical value: 1000000 globalDefault: false preemptionPolicy: PreemptLowerPriority # or Never description: "Critical workloads" --- apiVersion: v1 kind: Pod metadata: name: critical-pod spec: priorityClassName: critical # ... ``` ### Preemption Algorithm 1. **Identify candidates**: Nodes where evicting pods would make room 2. **Minimize disruption**: Prefer evicting fewer/lower-priority pods 3. **Respect PDBs**: Don't violate PodDisruptionBudgets if possible 4. **Execute**: Delete victim pods, schedule the preemptor **Note**: Preemption is "graceful"—victims get their termination grace period. ### Preventing Preemption ```yaml # Pod that cannot be preempted spec: priorityClassName: high-priority preemptionPolicy: Never # Can't preempt others # Or use PodDisruptionBudget to limit disruption apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: critical-pdb spec: minAvailable: 2 selector: matchLabels: app: critical ``` ## Scheduler Performance The default scheduler handles ~100 pods/second. At scale, several factors matter: ### Percentage of Nodes to Score For large clusters, scoring all feasible nodes is expensive. The scheduler samples: ```yaml # kube-scheduler config apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration percentageOfNodesToScore: 50 # Only score 50% of feasible nodes ``` With 5000 nodes and 50%, the scheduler scores at most 2500 nodes per pod. ### Parallelism The scheduler can evaluate multiple pods concurrently: ```yaml apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration parallelism: 16 # Concurrent scheduling goroutines ``` ### Cache The scheduler maintains a cache of node states to avoid hitting the API server: ```d2 {alt="Scheduler cache: the scheduler cache watches the API server, and scheduling decisions read from the cache"} direction: down api: API Server cache: Scheduler Cache decisions: "Scheduling decisions\n(reads from cache)" {shape: text} cache -> api: watch cache -> decisions ``` Cache includes: node allocatable, running pods, requested resources. ## Debugging Scheduling Failures ### Check Pod Events ```bash kubectl describe pod pending-pod Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 10s default-scheduler 0/5 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 3 node(s) didn't match Pod's node affinity/selector. ``` ### Understand the Message Common failure reasons: | Message | Meaning | Fix | |---------|---------|-----| | `Insufficient cpu` | No node has enough CPU | Reduce requests or add nodes | | `Insufficient memory` | No node has enough memory | Reduce requests or add nodes | | `node(s) had taint...didn't tolerate` | Taints blocking scheduling | Add tolerations or remove taints | | `node(s) didn't match node affinity` | Affinity rules too restrictive | Relax affinity or label nodes | | `node(s) didn't match pod topology spread` | Can't satisfy spread constraints | Add nodes in needed topologies | | `persistentvolumeclaim not found` | PVC doesn't exist | Create the PVC | | `node(s) had volume node affinity conflict` | PV is in different zone | Create PV in correct zone | ### Simulate Scheduling Check why a pod can't schedule without actually creating it: ```bash # Dry run scheduling (requires scheduler extender or custom script) # Or use kubectl-scheduler_simulator plugin ``` ### Check Node Resources ```bash # See allocatable vs allocated kubectl describe node worker-1 | grep -A 10 "Allocated resources" Allocated resources: Resource Requests Limits -------- -------- ------ cpu 3500m (92%) 7000m (184%) memory 12Gi (80%) 20Gi (133%) # Detailed pod resource usage kubectl top pods --containers ``` ### Scheduler Logs ```bash # View scheduler logs kubectl logs -n kube-system -l component=kube-scheduler # Increase verbosity # Edit kube-scheduler manifest, add --v=4 ``` ## Custom Schedulers You can run multiple schedulers or write your own: ### Using a Custom Scheduler ```yaml apiVersion: v1 kind: Pod metadata: name: custom-scheduled-pod spec: schedulerName: my-custom-scheduler # Use custom scheduler containers: - name: app image: nginx ``` ### Scheduler Extenders (Legacy) Extend the default scheduler with webhook calls: ```yaml apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration extenders: - urlPrefix: "http://my-extender:8080" filterVerb: "filter" prioritizeVerb: "prioritize" weight: 5 enableHTTPS: false ``` The scheduler calls your extender for additional filtering/scoring. ### Scheduling Framework (Modern) The Scheduling Framework allows writing plugins in Go: ```go // Custom filter plugin type MyPlugin struct{} func (p *MyPlugin) Name() string { return "MyPlugin" } func (p *MyPlugin) Filter(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeInfo *framework.NodeInfo) *framework.Status { // Custom filtering logic if !myCustomCheck(pod, nodeInfo.Node()) { return framework.NewStatus(framework.Unschedulable, "custom check failed") } return framework.NewStatus(framework.Success, "") } ``` Build a custom scheduler binary with your plugins included. ## Scheduler Configuration Full scheduler configuration example: ```yaml apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration leaderElection: leaderElect: true clientConnection: kubeconfig: /etc/kubernetes/scheduler.conf percentageOfNodesToScore: 50 podInitialBackoffSeconds: 1 podMaxBackoffSeconds: 10 profiles: - schedulerName: default-scheduler plugins: score: disabled: - name: NodeResourcesLeastAllocated enabled: - name: NodeResourcesMostAllocated # Bin packing instead weight: 1 filter: enabled: - name: NodeResourcesFit - name: NodePorts - name: TaintToleration ``` ## Summary The scheduler's job is simple: pick a node for each pod. The implementation is sophisticated: | Phase | What happens | |-------|--------------| | **Queue** | Pods ordered by priority, backoff for failures | | **Filter** | Eliminate nodes that can't run the pod | | **Score** | Rank remaining nodes by preference | | **Bind** | Assign pod to winning node | | **Preempt** | Evict lower-priority pods if needed | **Key takeaways:** 1. Scheduler uses **requests**, not limits 2. **Filtering** is pass/fail; **scoring** is best-effort 3. **Pod affinity** can create deadlocks—use carefully 4. **Topology spread** is the modern way to distribute pods 5. **Preemption** respects PDBs when possible 6. Debug with `kubectl describe pod` and scheduler logs When pods are stuck Pending, the answer is almost always in the scheduling failure message. Read it carefully—it tells you exactly which constraint failed. --- ## Garbage Collection in Kubernetes: OwnerReferences and Finalizers - URL: https://svalle.ru/posts/kubernetes/garbage-collection-finalizers/ - Date: 2025-02-06 - Tags: kubernetes, controllers, garbage-collection, finalizers, operators You delete a Deployment. Seconds later, its ReplicaSet is gone. Then the Pods vanish. You didn't delete them explicitly—Kubernetes garbage collection did. But how does it know what to delete? And what happens when you need to clean up external resources that Kubernetes doesn't know about? This post covers the two mechanisms that control object lifecycle: **OwnerReferences** for automatic cascading deletion, and **Finalizers** for custom cleanup logic. ## The Problem: Orphaned Resources Imagine you create a Deployment. Kubernetes creates a ReplicaSet, which creates Pods: ```d2 {alt="Ownership tree: a Deployment my-app owns a ReplicaSet my-app-7d9fc5, which owns three Pods abc12, def34, and ghi56"} direction: down dep: "Deployment (my-app)" rs: "ReplicaSet (my-app-7d9fc5)" pod1: "Pod (my-app-7d9fc5-abc12)" pod2: "Pod (my-app-7d9fc5-def34)" pod3: "Pod (my-app-7d9fc5-ghi56)" dep -- rs rs -- pod1 rs -- pod2 rs -- pod3 ``` Now you delete the Deployment. What should happen to the ReplicaSet and Pods? **Without garbage collection**: They'd become orphans—still running, consuming resources, but no longer managed by anything. You'd have to manually track and delete them. **With garbage collection**: Kubernetes automatically deletes dependents when their owner is deleted. Delete the Deployment, and the whole tree disappears. ## OwnerReferences: Declaring Ownership Every Kubernetes object can declare its owners via the `metadata.ownerReferences` field: ```yaml apiVersion: v1 kind: Pod metadata: name: my-app-7d9fc5-abc12 namespace: default ownerReferences: - apiVersion: apps/v1 kind: ReplicaSet name: my-app-7d9fc5 uid: 12345678-1234-1234-1234-123456789abc controller: true blockOwnerDeletion: true ``` ### OwnerReference Fields | Field | Required | Description | |-------|----------|-------------| | `apiVersion` | Yes | API version of the owner | | `kind` | Yes | Kind of the owner | | `name` | Yes | Name of the owner | | `uid` | Yes | UID of the owner (prevents accidental matches) | | `controller` | No | If true, this is THE controller (only one allowed) | | `blockOwnerDeletion` | No | If true, blocks owner deletion until this object is deleted | ### Setting OwnerReferences in Go When your controller creates child resources, set the owner reference: ```go import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" ) func (r *MyReconciler) createPod(ctx context.Context, owner *myv1.MyResource) error { pod := &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: owner.Name + "-pod", Namespace: owner.Namespace, }, Spec: corev1.PodSpec{ // ... }, } // Set owner reference - enables garbage collection if err := controllerutil.SetControllerReference(owner, pod, r.Scheme); err != nil { return err } return r.Create(ctx, pod) } ``` `SetControllerReference` does several things: 1. Sets `ownerReferences` with the owner's details 2. Sets `controller: true` (marks this as THE controller) 3. Sets `blockOwnerDeletion: true` 4. Validates that owner and dependent are in the same namespace ### Multiple Owners An object can have multiple owners: ```yaml ownerReferences: - apiVersion: apps/v1 kind: ReplicaSet name: my-app-7d9fc5 uid: abc123 controller: true # This is the controller - apiVersion: v1 kind: ConfigMap name: shared-config uid: def456 controller: false # Just an owner, not the controller ``` Rules: - Only **one** owner can have `controller: true` - Object is garbage collected when **all** owners are deleted - Use `SetOwnerReference` (not `SetControllerReference`) for non-controller owners ```go // Non-controller owner reference if err := controllerutil.SetOwnerReference(configMap, pod, r.Scheme); err != nil { return err } ``` ## Cascading Deletion When you delete an owner, what happens to its dependents? Kubernetes supports three deletion propagation policies: ### Foreground Deletion The owner waits for dependents to be deleted first: ``` 1. Owner gets deletionTimestamp set 2. Owner enters "deletion in progress" state 3. GC deletes all dependents with blockOwnerDeletion=true 4. Once all blocking dependents are gone, owner is deleted ``` ```bash kubectl delete deployment my-app --cascade=foreground ``` ```go // Programmatically propagation := metav1.DeletePropagationForeground client.Delete(ctx, deployment, &client.DeleteOptions{ PropagationPolicy: &propagation, }) ``` **Use when**: You need to ensure children are gone before the parent disappears (e.g., cleaning up PVCs before deleting a StatefulSet). ### Background Deletion (Default) The owner is deleted immediately; dependents are garbage collected asynchronously: ``` 1. Owner is deleted immediately 2. GC notices orphaned dependents 3. GC deletes dependents in the background ``` ```bash kubectl delete deployment my-app --cascade=background # or just kubectl delete deployment my-app # background is default ``` **Use when**: You don't need to wait for cleanup (most cases). ### Orphan Deletion Delete the owner but leave dependents alone: ``` 1. Owner is deleted 2. Dependents remain, but ownerReferences are cleared 3. Dependents become standalone objects ``` ```bash kubectl delete deployment my-app --cascade=orphan ``` **Use when**: You want to "detach" resources. For example, adopting Pods into a new ReplicaSet. ## The Garbage Collector Controller Garbage collection is implemented by the **garbage collector controller** in kube-controller-manager. Here's how it works: ### The Dependency Graph The GC controller maintains an in-memory graph of all owner-dependent relationships: ```d2 {alt="GC dependency graph: inside the graph, Deployment/my-app points to ReplicaSet/my-app-7d9fc5, which points to three Pods (abc12, def34, ghi56), while Service/my-svc stands alone with no dependents"} direction: down graph: GC Dependency Graph { dep: "Deployment/my-app" rs: "ReplicaSet/my-app-7d9fc5" pod1: "Pod/my-app-7d9fc5-abc12" pod2: "Pod/my-app-7d9fc5-def34" pod3: "Pod/my-app-7d9fc5-ghi56" svc: "Service/my-svc (no dependents)" dep -> rs rs -> pod1 rs -> pod2 rs -> pod3 } ``` ### Processing Deletions When an object is deleted: 1. **GC detects deletion** via watch events 2. **Looks up dependents** in the graph 3. **For each dependent**: - If `blockOwnerDeletion=true` and foreground deletion: delete dependent first - If background deletion: queue dependent for deletion - If orphan deletion: remove ownerReference from dependent ### Handling Orphans If the GC finds an object with an ownerReference pointing to a non-existent owner: ```yaml ownerReferences: - apiVersion: apps/v1 kind: ReplicaSet name: my-app-7d9fc5 uid: abc123 # This UID no longer exists! ``` The object is considered orphaned and will be deleted (unless orphan propagation was used). **Important**: The UID must match. If you delete and recreate an owner with the same name, dependents won't automatically re-attach—they'll be garbage collected because the UID changed. ## Finalizers: Custom Cleanup OwnerReferences handle Kubernetes-native relationships. But what if deleting your custom resource should: - Delete an S3 bucket? - Remove a DNS record? - Clean up a database user? - Revoke cloud IAM permissions? Kubernetes doesn't know about these external resources. **Finalizers** let you run custom cleanup logic before an object is deleted. ### How Finalizers Work A finalizer is just a string in `metadata.finalizers`: ```yaml apiVersion: myapp.example.com/v1 kind: Database metadata: name: my-db finalizers: - databases.myapp.example.com/cleanup spec: # ... ``` When you delete an object with finalizers: 1. **Kubernetes sets `deletionTimestamp`** but doesn't delete the object 2. **Object enters "terminating" state** — it still exists in etcd 3. **Your controller sees the deletion** (via watch) 4. **Controller performs cleanup** (delete S3 bucket, etc.) 5. **Controller removes the finalizer** from the object 6. **Once all finalizers are removed**, Kubernetes deletes the object ```d2 {alt="Finalizer deletion flow: a DELETE request reaches the has-finalizers check (yes sets deletionTimestamp and keeps the object, no deletes immediately); on yes the object enters terminating state still in etcd, then the controller sees the object with deletionTimestamp, performs cleanup and removes the finalizer, and once all finalizers are removed the object is deleted from etcd"} grid-columns: 1 vertical-gap: 40 request: DELETE request {style: {stroke-width: 0; fill: transparent}} check: "Has finalizers?\n\nYes: Set deletionTimestamp,\nkeep object\n\nNo: Delete immediately" terminating: "Object in \"terminating\"\nstate, still in etcd" controller: "Controller sees object\nwith deletionTimestamp\n\nPerforms cleanup...\nRemoves finalizer" deleted: "All finalizers removed\nObject deleted from etcd" request -> check check -> terminating: "(Yes)" terminating -> controller controller -> deleted ``` ### Implementing Finalizers Here's the standard pattern in a controller: ```go const finalizerName = "databases.myapp.example.com/cleanup" func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { log := log.FromContext(ctx) // Fetch the Database instance db := &myappv1.Database{} if err := r.Get(ctx, req.NamespacedName, db); err != nil { if errors.IsNotFound(err) { // Object not found, could have been deleted after reconcile request return ctrl.Result{}, nil } return ctrl.Result{}, err } // Check if object is being deleted if db.ObjectMeta.DeletionTimestamp.IsZero() { // Object is NOT being deleted // Add finalizer if it doesn't exist if !controllerutil.ContainsFinalizer(db, finalizerName) { controllerutil.AddFinalizer(db, finalizerName) if err := r.Update(ctx, db); err != nil { return ctrl.Result{}, err } } } else { // Object IS being deleted if controllerutil.ContainsFinalizer(db, finalizerName) { // Run cleanup logic if err := r.cleanupExternalResources(ctx, db); err != nil { // If cleanup fails, requeue return ctrl.Result{}, err } // Remove finalizer to allow deletion controllerutil.RemoveFinalizer(db, finalizerName) if err := r.Update(ctx, db); err != nil { return ctrl.Result{}, err } } // Finalizer removed, object will be deleted return ctrl.Result{}, nil } // Normal reconciliation logic return r.reconcileDatabase(ctx, db) } func (r *DatabaseReconciler) cleanupExternalResources(ctx context.Context, db *myappv1.Database) error { log := log.FromContext(ctx) log.Info("Cleaning up external resources", "database", db.Name) // Delete the actual database if err := r.cloudProvider.DeleteDatabase(ctx, db.Spec.DatabaseID); err != nil { // Ignore "not found" errors — resource may already be deleted if !isNotFound(err) { return err } } // Delete associated secrets if err := r.cloudProvider.DeleteCredentials(ctx, db.Spec.CredentialsID); err != nil { if !isNotFound(err) { return err } } log.Info("Successfully cleaned up external resources") return nil } ``` ### Finalizer Best Practices **1. Add finalizer early** Add the finalizer before creating external resources: ```go // Good: Add finalizer first if !controllerutil.ContainsFinalizer(db, finalizerName) { controllerutil.AddFinalizer(db, finalizerName) if err := r.Update(ctx, db); err != nil { return ctrl.Result{}, err } // Requeue to continue after finalizer is persisted return ctrl.Result{Requeue: true}, nil } // Now safe to create external resource if err := r.createExternalDatabase(ctx, db); err != nil { return ctrl.Result{}, err } ``` If you create the external resource first and then crash before adding the finalizer, the resource becomes orphaned. **2. Make cleanup idempotent** Cleanup may run multiple times (controller restarts, errors, requeues): ```go func (r *Reconciler) cleanupExternalResources(ctx context.Context, db *myappv1.Database) error { // Idempotent: safe to call even if already deleted err := r.cloudProvider.DeleteDatabase(ctx, db.Spec.DatabaseID) if err != nil && !isNotFound(err) { return err // Real error, retry } // Success or already deleted — both are fine return nil } ``` **3. Handle cleanup failures gracefully** If cleanup fails, return an error to requeue. But consider adding a timeout or retry limit: ```go func (r *Reconciler) cleanupExternalResources(ctx context.Context, db *myappv1.Database) error { // Check if we've been trying too long if db.DeletionTimestamp != nil { deleteAge := time.Since(db.DeletionTimestamp.Time) if deleteAge > 1*time.Hour { // Log and give up — manual intervention required log.Error(nil, "Cleanup taking too long, giving up", "database", db.Name, "age", deleteAge) return nil // Remove finalizer anyway } } return r.doCleanup(ctx, db) } ``` **4. Use unique finalizer names** Include your domain to avoid collisions: ```go // Good const finalizerName = "databases.myapp.example.com/cleanup" // Bad — could collide with other controllers const finalizerName = "cleanup" ``` **5. Don't block indefinitely** A stuck finalizer blocks deletion forever. Always have a path to completion: ```go func (r *Reconciler) cleanupExternalResources(ctx context.Context, db *myappv1.Database) error { // Use context with timeout cleanupCtx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() if err := r.doCleanup(cleanupCtx, db); err != nil { if cleanupCtx.Err() == context.DeadlineExceeded { // Timeout — requeue with backoff return fmt.Errorf("cleanup timed out, will retry: %w", err) } return err } return nil } ``` ## Combining OwnerReferences and Finalizers For complex resources, you often need both: ```go func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { db := &myappv1.Database{} if err := r.Get(ctx, req.NamespacedName, db); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) } // Handle deletion if !db.DeletionTimestamp.IsZero() { return r.handleDeletion(ctx, db) } // Add finalizer for external resources if !controllerutil.ContainsFinalizer(db, finalizerName) { controllerutil.AddFinalizer(db, finalizerName) if err := r.Update(ctx, db); err != nil { return ctrl.Result{}, err } } // Create Secret with owner reference (GC handles this) secret := &corev1.Secret{ ObjectMeta: metav1.ObjectMeta{ Name: db.Name + "-credentials", Namespace: db.Namespace, }, Data: map[string][]byte{ "password": generatePassword(), }, } if err := controllerutil.SetControllerReference(db, secret, r.Scheme); err != nil { return ctrl.Result{}, err } if err := r.Create(ctx, secret); err != nil && !errors.IsAlreadyExists(err) { return ctrl.Result{}, err } // Create external database (finalizer handles cleanup) if err := r.ensureExternalDatabase(ctx, db); err != nil { return ctrl.Result{}, err } return ctrl.Result{}, nil } func (r *DatabaseReconciler) handleDeletion(ctx context.Context, db *myappv1.Database) (ctrl.Result, error) { if !controllerutil.ContainsFinalizer(db, finalizerName) { return ctrl.Result{}, nil } // Clean up external resources (not covered by OwnerReferences) if err := r.deleteExternalDatabase(ctx, db); err != nil { return ctrl.Result{}, err } // Remove finalizer // Note: The Secret will be automatically deleted by GC (OwnerReference) controllerutil.RemoveFinalizer(db, finalizerName) if err := r.Update(ctx, db); err != nil { return ctrl.Result{}, err } return ctrl.Result{}, nil } ``` In this example: - **Secret**: Uses OwnerReference → automatic GC deletion - **External database**: Uses Finalizer → custom cleanup logic ## Debugging Garbage Collection ### Check OwnerReferences ```bash # See owner references kubectl get pod my-pod -o jsonpath='{.metadata.ownerReferences}' | jq # Find all objects owned by a specific resource kubectl get all --all-namespaces -o json | jq ' .items[] | select(.metadata.ownerReferences[]?.name == "my-deployment") | "\(.kind)/\(.metadata.name)" ' ``` ### Check Finalizers ```bash # See finalizers on an object kubectl get database my-db -o jsonpath='{.metadata.finalizers}' # Find objects stuck in terminating (have deletionTimestamp but still exist) kubectl get all --all-namespaces -o json | jq ' .items[] | select(.metadata.deletionTimestamp != null) | "\(.kind)/\(.metadata.namespace)/\(.metadata.name): \(.metadata.finalizers)" ' ``` ### Stuck Finalizer Recovery If an object is stuck terminating because the controller is gone or broken: ```bash # DANGEROUS: Remove finalizer manually to unblock deletion kubectl patch database my-db -p '{"metadata":{"finalizers":null}}' --type=merge # Or edit directly kubectl edit database my-db # Remove the finalizers array ``` **Warning**: This skips cleanup! External resources may be orphaned. ### GC Controller Logs ```bash # Check garbage collector logs in controller-manager kubectl logs -n kube-system kube-controller-manager- | grep -i garbage ``` ## Common Patterns and Pitfalls ### Pattern: Adopting Orphans Sometimes you want a controller to "adopt" existing resources: ```go func (r *Reconciler) adoptOrphanedPods(ctx context.Context, owner *myv1.MyResource) error { // Find pods that should be owned but aren't pods := &corev1.PodList{} if err := r.List(ctx, pods, client.InNamespace(owner.Namespace), client.MatchingLabels{"app": owner.Name}, ); err != nil { return err } for _, pod := range pods.Items { // Skip if already owned by someone else if metav1.GetControllerOf(&pod) != nil { continue } // Adopt the pod if err := controllerutil.SetControllerReference(owner, &pod, r.Scheme); err != nil { return err } if err := r.Update(ctx, &pod); err != nil { return err } } return nil } ``` ### Pattern: Cross-Namespace References OwnerReferences only work within a namespace. For cross-namespace relationships, use finalizers: ```go // ClusterDatabase (cluster-scoped) creates Secrets in user namespaces func (r *ClusterDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { cdb := &myv1.ClusterDatabase{} if err := r.Get(ctx, req.NamespacedName, cdb); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) } if !cdb.DeletionTimestamp.IsZero() { // Clean up secrets in all namespaces for _, ns := range cdb.Spec.TargetNamespaces { secret := &corev1.Secret{ ObjectMeta: metav1.ObjectMeta{ Name: cdb.Name + "-credentials", Namespace: ns, }, } if err := r.Delete(ctx, secret); err != nil && !errors.IsNotFound(err) { return ctrl.Result{}, err } } // Remove finalizer controllerutil.RemoveFinalizer(cdb, finalizerName) return ctrl.Result{}, r.Update(ctx, cdb) } // Can't use OwnerReference (cross-namespace), so we must clean up manually // ... } ``` ### Pitfall: Finalizer Without Controller If you add a finalizer but your controller isn't running (or crashes permanently), objects get stuck: ```yaml apiVersion: myapp.example.com/v1 kind: Database metadata: name: stuck-db deletionTimestamp: "2025-01-25T10:00:00Z" # Stuck! finalizers: - databases.myapp.example.com/cleanup # No controller to remove this ``` **Prevention**: - Ensure controllers are highly available - Consider finalizer timeouts - Document manual recovery procedures ### Pitfall: Circular OwnerReferences Don't create circular ownership: ```yaml # Bad: A owns B, B owns A # Result: Neither can be deleted! # Object A ownerReferences: - name: B uid: ... # Object B ownerReferences: - name: A uid: ... ``` The GC controller detects and logs circular references but can't resolve them automatically. ### Pitfall: UID Mismatch After Recreation ```bash # Create deployment kubectl create deployment my-app --image=nginx # Note the ReplicaSet's ownerReference UID kubectl get rs -o jsonpath='{.items[0].metadata.ownerReferences[0].uid}' # abc123 # Delete and recreate deployment with same name kubectl delete deployment my-app kubectl create deployment my-app --image=nginx # New deployment has different UID kubectl get deployment my-app -o jsonpath='{.metadata.uid}' # def456 (different!) # Old ReplicaSet (if it somehow survived) would be orphaned # because its ownerReference.uid (abc123) doesn't match ``` ## Summary Kubernetes provides two mechanisms for managing object lifecycle: | Mechanism | Use Case | How It Works | |-----------|----------|--------------| | **OwnerReferences** | Kubernetes-native parent-child relationships | Automatic cascading deletion by GC controller | | **Finalizers** | External resources, custom cleanup logic | Blocks deletion until controller removes finalizer | **OwnerReferences**: - Set via `controllerutil.SetControllerReference()` or `SetOwnerReference()` - Same namespace only - Automatic cleanup by garbage collector - Three propagation policies: Foreground, Background, Orphan **Finalizers**: - Add before creating external resources - Remove after cleanup is complete - Must be idempotent - Stuck finalizers block deletion indefinitely **When to use which**: - Child Kubernetes objects (Pods, Secrets, ConfigMaps) → OwnerReferences - External resources (cloud databases, DNS records, IAM) → Finalizers - Cross-namespace relationships → Finalizers - Need custom cleanup ordering → Finalizers The combination of both gives you complete control over resource lifecycle—automatic cleanup for Kubernetes objects, and guaranteed custom cleanup for everything else. --- ## client-go Patterns: Informers, Work Queues, and Rate Limiting - URL: https://svalle.ru/posts/kubernetes/client-go-patterns/ - Date: 2025-02-05 - Tags: kubernetes, go, client-go, controllers, informers, operators You're building a Kubernetes controller. You could poll the API server every few seconds, but that doesn't scale. You could set up a watch, but then you need to handle disconnections, resyncs, and state reconciliation yourself. Or you could use client-go's battle-tested primitives that handle all of this for you. This post covers the core building blocks: Informers for efficient API watching, work queues for reliable event processing, and rate limiters for backoff and retries. ## Why Not Just Use the REST API? The naive approach to watching Kubernetes resources: ```go // Don't do this for { pods, err := clientset.CoreV1().Pods("").List(ctx, metav1.ListOptions{}) if err != nil { log.Error(err) continue } for _, pod := range pods.Items { processPod(pod) } time.Sleep(5 * time.Second) } ``` Problems: 1. **Expensive**: Every LIST fetches all objects, even unchanged ones 2. **Delayed**: 5-second polling means up to 5 seconds of staleness 3. **Scales poorly**: More objects = more data transferred each poll 4. **No ordering**: You might miss rapid changes between polls Kubernetes provides WATCH for efficient change streaming: ```go // Better, but still manual watcher, err := clientset.CoreV1().Pods("").Watch(ctx, metav1.ListOptions{}) if err != nil { return err } for event := range watcher.ResultChan() { pod := event.Object.(*v1.Pod) switch event.Type { case watch.Added: handleAdd(pod) case watch.Modified: handleUpdate(pod) case watch.Deleted: handleDelete(pod) } } ``` Better, but you still need to handle: - Watch disconnections (network blips, API server restarts) - Bookmark events and resource versions - Initial LIST to populate state before watching - Resyncs when the watch falls behind client-go's Informers handle all of this. ## Informers: The Foundation An Informer combines LIST and WATCH into a single abstraction that maintains a local cache of objects and notifies you of changes. ### Basic Informer Usage ```go import ( "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/cache" ) func main() { // Create clientset clientset, err := kubernetes.NewForConfig(config) if err != nil { panic(err) } // Create shared informer factory factory := informers.NewSharedInformerFactory(clientset, 30*time.Minute) // Get pod informer podInformer := factory.Core().V1().Pods().Informer() // Add event handlers podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) fmt.Printf("Pod added: %s/%s\n", pod.Namespace, pod.Name) }, UpdateFunc: func(oldObj, newObj interface{}) { oldPod := oldObj.(*v1.Pod) newPod := newObj.(*v1.Pod) fmt.Printf("Pod updated: %s/%s\n", newPod.Namespace, newPod.Name) }, DeleteFunc: func(obj interface{}) { pod := obj.(*v1.Pod) fmt.Printf("Pod deleted: %s/%s\n", pod.Namespace, pod.Name) }, }) // Start informer stopCh := make(chan struct{}) factory.Start(stopCh) // Wait for cache sync if !cache.WaitForCacheSync(stopCh, podInformer.HasSynced) { panic("failed to sync cache") } // Now the cache is populated and handlers are receiving events <-stopCh } ``` ### What Happens Under the Hood ``` 1. Initial LIST Informer calls LIST to get all current objects Stores them in local cache (thread-safe store) Triggers AddFunc for each existing object 2. Start WATCH Informer opens WATCH from the LIST's resourceVersion Receives streaming events (ADDED, MODIFIED, DELETED) 3. Event Processing Each event updates the local cache Triggers appropriate handler (Add/Update/Delete) 4. Resync (periodic) Every resyncPeriod, triggers UpdateFunc for ALL cached objects Ensures eventual consistency even if events were missed 5. Watch Reconnection If watch disconnects, Informer re-LISTs and re-WATCHes Deduplicates events (won't re-trigger for unchanged objects) ``` ### The Local Cache Informers maintain an in-memory cache (called a Store) that you can query directly: ```go // Get the Lister (read-only cache interface) podLister := factory.Core().V1().Pods().Lister() // List all pods (from cache, not API server) pods, err := podLister.List(labels.Everything()) // List pods in a namespace pods, err := podLister.Pods("default").List(labels.Everything()) // Get a specific pod pod, err := podLister.Pods("default").Get("my-pod") ``` Cache reads are: - **Fast**: No network call, just memory access - **Eventually consistent**: May be slightly behind API server - **Thread-safe**: Safe to call from any goroutine ### SharedInformerFactory You rarely create Informers directly. Instead, use `SharedInformerFactory`: ```go // Without factory: each informer has its own watch podInformer1 := cache.NewSharedInformer(podListWatcher, &v1.Pod{}, resyncPeriod) podInformer2 := cache.NewSharedInformer(podListWatcher, &v1.Pod{}, resyncPeriod) // Two watches to API server for the same resource! // With factory: informers are shared factory := informers.NewSharedInformerFactory(clientset, resyncPeriod) podInformer1 := factory.Core().V1().Pods().Informer() podInformer2 := factory.Core().V1().Pods().Informer() // Same informer, one watch, multiple handlers ``` The factory ensures only one watch per resource type, regardless of how many controllers need it. ### Filtered Informers Watch only what you need: ```go // Only watch pods in "production" namespace factory := informers.NewSharedInformerFactoryWithOptions( clientset, resyncPeriod, informers.WithNamespace("production"), ) // Only watch pods with specific labels factory := informers.NewSharedInformerFactoryWithOptions( clientset, resyncPeriod, informers.WithTweakListOptions(func(opts *metav1.ListOptions) { opts.LabelSelector = "app=myapp" }), ) ``` Filtering at the API server level reduces memory usage and network traffic. ## Work Queues: Reliable Event Processing Event handlers run in the Informer's goroutine. If your handler is slow or blocks, you'll delay all other events. The solution: queue events for async processing. ### The Problem with Direct Processing ```go // Bad: slow handler blocks all events podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) // This takes 5 seconds... doExpensiveReconciliation(pod) // All other events are blocked! }, }) ``` ### Work Queue Pattern ```go import "k8s.io/client-go/util/workqueue" // Create a rate-limited work queue queue := workqueue.NewRateLimitingQueue( workqueue.DefaultControllerRateLimiter(), ) // Handler just enqueues keys podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { key, err := cache.MetaNamespaceKeyFunc(obj) if err == nil { queue.Add(key) // Fast: just adds string to queue } }, UpdateFunc: func(old, new interface{}) { key, err := cache.MetaNamespaceKeyFunc(new) if err == nil { queue.Add(key) // Deduplicates: same key won't queue twice } }, DeleteFunc: func(obj interface{}) { key, err := cache.DeletionHandlingMetaNamespaceKeyFunc(obj) if err == nil { queue.Add(key) } }, }) // Worker goroutines process the queue for i := 0; i < workerCount; i++ { go func() { for processNextItem(queue, podLister) { } }() } ``` ### Processing Items ```go func processNextItem(queue workqueue.RateLimitingInterface, lister v1lister.PodLister) bool { // Get next item (blocks if queue is empty) key, shutdown := queue.Get() if shutdown { return false } // Tell queue we're done with this item when function returns defer queue.Done(key) // Parse the key namespace, name, err := cache.SplitMetaNamespaceKey(key.(string)) if err != nil { // Invalid key, don't requeue queue.Forget(key) return true } // Get object from cache pod, err := lister.Pods(namespace).Get(name) if errors.IsNotFound(err) { // Object deleted, nothing to do queue.Forget(key) return true } if err != nil { // Transient error, requeue with rate limiting queue.AddRateLimited(key) return true } // Do actual reconciliation if err := reconcile(pod); err != nil { // Reconciliation failed, requeue with rate limiting queue.AddRateLimited(key) return true } // Success! Remove from rate limiter tracking queue.Forget(key) return true } ``` ### Key Concepts **Keys, not objects**: We queue string keys (`namespace/name`), not full objects. This means: - Deduplication: Multiple rapid events for same object = one queue item - Freshness: Worker reads latest state from cache, not stale event data - Memory efficiency: Queue holds strings, not large objects **Done()**: Always call `queue.Done(key)` when finished processing. This marks the item as no longer being processed. **Forget()**: Call `queue.Forget(key)` on success to reset rate limiter state for this key. **AddRateLimited()**: Call on failure to requeue with exponential backoff. ### Queue Types client-go provides several queue implementations: ```go // Basic FIFO queue queue := workqueue.New() // Delayed queue (items become ready after delay) queue := workqueue.NewDelayingQueue() // Rate-limited queue (exponential backoff on failures) queue := workqueue.NewRateLimitingQueue(rateLimiter) // Named queue (for metrics) queue := workqueue.NewNamedRateLimitingQueue(rateLimiter, "my-controller") ``` For controllers, always use `RateLimitingQueue`. ## Rate Limiting: Backoff and Retry When reconciliation fails, you want to retry—but not immediately in a tight loop. Rate limiters control retry timing. ### Default Rate Limiter ```go // DefaultControllerRateLimiter combines two strategies: rateLimiter := workqueue.DefaultControllerRateLimiter() // Equivalent to: rateLimiter := workqueue.NewMaxOfRateLimiter( // Exponential backoff: 5ms, 10ms, 20ms... up to 1000s workqueue.NewItemExponentialFailureRateLimiter(5*time.Millisecond, 1000*time.Second), // Overall rate limit: 10 qps, burst of 100 &workqueue.BucketRateLimiter{Limiter: rate.NewLimiter(rate.Limit(10), 100)}, ) ``` This means: - First failure: retry after 5ms - Second failure: retry after 10ms - Third failure: retry after 20ms - ...exponentially increasing up to 1000 seconds - Plus: overall queue is limited to 10 items/second with burst of 100 ### Custom Rate Limiters ```go // Faster retries, shorter max rateLimiter := workqueue.NewItemExponentialFailureRateLimiter( 1*time.Millisecond, // base delay 30*time.Second, // max delay ) // Fixed delay (no exponential backoff) rateLimiter := workqueue.NewItemFastSlowRateLimiter( 5*time.Millisecond, // fast delay (first N attempts) 10*time.Second, // slow delay (after N attempts) 5, // N = number of fast attempts ) // Per-item rate limiting with different delays rateLimiter := workqueue.NewMaxOfRateLimiter( workqueue.NewItemExponentialFailureRateLimiter(100*time.Millisecond, 60*time.Second), workqueue.NewItemFastSlowRateLimiter(500*time.Millisecond, 5*time.Second, 4), ) ``` ### Rate Limiter Methods ```go // How long until this item should be processed? delay := rateLimiter.When(key) // Increment failure count for this item rateLimiter.NumRequeues(key) // Returns current count // Reset failure count (call on success) rateLimiter.Forget(key) ``` ### Client-Side Rate Limiting Separate from work queue rate limiting, you can rate limit API server requests: ```go import "k8s.io/client-go/rest" config := &rest.Config{ Host: "https://kubernetes.default.svc", // Rate limit API requests QPS: 20, // Queries per second Burst: 30, // Burst allowance } clientset, err := kubernetes.NewForConfig(config) ``` Default is 5 QPS with burst of 10. Increase for high-throughput controllers, but be mindful of API server load. ## The Complete Controller Pattern Putting it all together: ```go package main import ( "context" "fmt" "time" v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes" v1lister "k8s.io/client-go/listers/core/v1" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/clientcmd" "k8s.io/client-go/util/workqueue" "k8s.io/klog/v2" ) type Controller struct { clientset kubernetes.Interface podLister v1lister.PodLister podsSynced cache.InformerSynced workqueue workqueue.RateLimitingInterface } func NewController( clientset kubernetes.Interface, podInformer cache.SharedIndexInformer, podLister v1lister.PodLister, ) *Controller { controller := &Controller{ clientset: clientset, podLister: podLister, podsSynced: podInformer.HasSynced, workqueue: workqueue.NewNamedRateLimitingQueue( workqueue.DefaultControllerRateLimiter(), "Pods", ), } // Set up event handlers podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: controller.enqueuePod, UpdateFunc: func(old, new interface{}) { controller.enqueuePod(new) }, DeleteFunc: controller.enqueuePod, }) return controller } func (c *Controller) enqueuePod(obj interface{}) { key, err := cache.MetaNamespaceKeyFunc(obj) if err != nil { runtime.HandleError(err) return } c.workqueue.Add(key) } func (c *Controller) Run(ctx context.Context, workers int) error { defer runtime.HandleCrash() defer c.workqueue.ShutDown() klog.Info("Starting controller") // Wait for caches to sync klog.Info("Waiting for informer caches to sync") if !cache.WaitForCacheSync(ctx.Done(), c.podsSynced) { return fmt.Errorf("failed to wait for caches to sync") } klog.Info("Starting workers") for i := 0; i < workers; i++ { go wait.UntilWithContext(ctx, c.runWorker, time.Second) } klog.Info("Started workers") <-ctx.Done() klog.Info("Shutting down workers") return nil } func (c *Controller) runWorker(ctx context.Context) { for c.processNextWorkItem(ctx) { } } func (c *Controller) processNextWorkItem(ctx context.Context) bool { obj, shutdown := c.workqueue.Get() if shutdown { return false } defer c.workqueue.Done(obj) key := obj.(string) if err := c.syncHandler(ctx, key); err != nil { // Requeue with rate limiting c.workqueue.AddRateLimited(key) runtime.HandleError(fmt.Errorf("error syncing '%s': %s, requeuing", key, err.Error())) return true } // Success - forget this item so rate limiter resets c.workqueue.Forget(obj) klog.Infof("Successfully synced '%s'", key) return true } func (c *Controller) syncHandler(ctx context.Context, key string) error { namespace, name, err := cache.SplitMetaNamespaceKey(key) if err != nil { return nil // Invalid key, don't requeue } // Get pod from cache pod, err := c.podLister.Pods(namespace).Get(name) if errors.IsNotFound(err) { // Pod deleted, nothing to do klog.Infof("Pod %s deleted", key) return nil } if err != nil { return err // Requeue } // Your reconciliation logic here klog.Infof("Reconciling pod %s/%s (phase: %s)", pod.Namespace, pod.Name, pod.Status.Phase) return nil } func main() { // Build config config, err := clientcmd.BuildConfigFromFlags("", clientcmd.RecommendedHomeFile) if err != nil { klog.Fatalf("Error building config: %s", err) } // Create clientset clientset, err := kubernetes.NewForConfig(config) if err != nil { klog.Fatalf("Error creating clientset: %s", err) } // Create informer factory factory := informers.NewSharedInformerFactory(clientset, 30*time.Minute) podInformer := factory.Core().V1().Pods() // Create controller controller := NewController( clientset, podInformer.Informer(), podInformer.Lister(), ) // Start informers ctx, cancel := context.WithCancel(context.Background()) defer cancel() factory.Start(ctx.Done()) // Run controller with 2 workers if err := controller.Run(ctx, 2); err != nil { klog.Fatalf("Error running controller: %s", err) } } ``` ## Advanced Patterns ### Watching Multiple Resources Controllers often need to watch multiple resource types: ```go type Controller struct { podLister v1lister.PodLister podsSynced cache.InformerSynced serviceLister v1lister.ServiceLister servicesSynced cache.InformerSynced workqueue workqueue.RateLimitingInterface } func NewController( podInformer coreinformers.PodInformer, serviceInformer coreinformers.ServiceInformer, ) *Controller { c := &Controller{ podLister: podInformer.Lister(), podsSynced: podInformer.Informer().HasSynced, serviceLister: serviceInformer.Lister(), servicesSynced: serviceInformer.Informer().HasSynced, workqueue: workqueue.NewRateLimitingQueue(...), } // Watch pods podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: c.handlePod, UpdateFunc: func(old, new interface{}) { c.handlePod(new) }, DeleteFunc: c.handlePod, }) // Watch services - enqueue related pods serviceInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: c.handleService, UpdateFunc: func(old, new interface{}) { c.handleService(new) }, DeleteFunc: c.handleService, }) return c } func (c *Controller) handleService(obj interface{}) { svc := obj.(*v1.Service) // Find pods selected by this service selector := labels.SelectorFromSet(svc.Spec.Selector) pods, _ := c.podLister.Pods(svc.Namespace).List(selector) // Enqueue each related pod for _, pod := range pods { c.enqueuePod(pod) } } // Wait for ALL caches func (c *Controller) Run(ctx context.Context, workers int) error { if !cache.WaitForCacheSync(ctx.Done(), c.podsSynced, c.servicesSynced) { return fmt.Errorf("caches failed to sync") } // ... } ``` ### Owner Reference Lookups When a child resource changes, find and enqueue the parent: ```go func (c *Controller) handlePod(obj interface{}) { pod := obj.(*v1.Pod) // Find owner ReplicaSet ownerRef := metav1.GetControllerOf(pod) if ownerRef == nil || ownerRef.Kind != "ReplicaSet" { return } // Enqueue the ReplicaSet (not the pod) c.workqueue.Add(pod.Namespace + "/" + ownerRef.Name) } ``` ### Resync Period Considerations The resync period triggers UpdateFunc for all cached objects periodically: ```go // 30 minute resync factory := informers.NewSharedInformerFactory(clientset, 30*time.Minute) ``` **Why resync?** - Catch missed events (network issues, bugs) - Ensure eventual consistency - Periodic health check of reconciliation **Trade-offs:** - Shorter: More consistent, more CPU/API load - Longer: Less overhead, higher staleness risk - Zero (`0`): No resync (only event-driven) For most controllers, 10-30 minutes is reasonable. ### Event Filtering Don't enqueue if nothing meaningful changed: ```go podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{ UpdateFunc: func(old, new interface{}) { oldPod := old.(*v1.Pod) newPod := new.(*v1.Pod) // Skip if resource version unchanged (resync) if oldPod.ResourceVersion == newPod.ResourceVersion { return } // Skip if only status changed (we only care about spec) if reflect.DeepEqual(oldPod.Spec, newPod.Spec) { return } c.enqueuePod(newPod) }, }) ``` ### Handling Deleted Objects Delete events can be tricky — sometimes you get a `DeletedFinalStateUnknown`: ```go DeleteFunc: func(obj interface{}) { // Handle DeletedFinalStateUnknown wrapper if tombstone, ok := obj.(cache.DeletedFinalStateUnknown); ok { obj = tombstone.Obj } pod, ok := obj.(*v1.Pod) if !ok { runtime.HandleError(fmt.Errorf("unexpected object type: %T", obj)) return } c.enqueuePod(pod) }, ``` `DeletedFinalStateUnknown` occurs when the controller missed the delete event and discovers the deletion during resync. ### Index Functions Add custom indexes for efficient lookups: ```go // Add index by node name podInformer.Informer().AddIndexers(cache.Indexers{ "byNode": func(obj interface{}) ([]string, error) { pod := obj.(*v1.Pod) if pod.Spec.NodeName == "" { return nil, nil } return []string{pod.Spec.NodeName}, nil }, }) // Query by index indexer := podInformer.Informer().GetIndexer() pods, err := indexer.ByIndex("byNode", "worker-1") ``` ## Common Mistakes ### 1. Blocking in Event Handlers ```go // Wrong: blocks all events AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) time.Sleep(5 * time.Second) // Blocks! processExpensiveThing(pod) } // Right: just enqueue AddFunc: func(obj interface{}) { key, _ := cache.MetaNamespaceKeyFunc(obj) queue.Add(key) } ``` ### 2. Using Stale Object from Event ```go // Wrong: uses potentially stale object from event func (c *Controller) sync(key string) error { // pod from event might be outdated return c.reconcile(c.eventPod) } // Right: fetch fresh from cache func (c *Controller) sync(key string) error { ns, name, _ := cache.SplitMetaNamespaceKey(key) pod, _ := c.podLister.Pods(ns).Get(name) return c.reconcile(pod) } ``` ### 3. Forgetting cache.WaitForCacheSync ```go // Wrong: processing before cache is populated factory.Start(stopCh) // Immediately start processing... but cache is empty! // Right: wait for sync factory.Start(stopCh) if !cache.WaitForCacheSync(stopCh, informer.HasSynced) { return fmt.Errorf("cache sync failed") } // Now cache is populated ``` ### 4. Not Calling Done() ```go // Wrong: item stays "processing" forever func (c *Controller) processNext() bool { obj, _ := c.queue.Get() // forgot queue.Done(obj)! return true } // Right: always call Done func (c *Controller) processNext() bool { obj, _ := c.queue.Get() defer c.queue.Done(obj) // ... return true } ``` ### 5. Forgetting to Forget ```go // Wrong: rate limiter keeps growing delay forever if err := c.sync(key); err != nil { c.queue.AddRateLimited(key) return } // Success but forgot to reset rate limiter // Right: reset on success if err := c.sync(key); err != nil { c.queue.AddRateLimited(key) return } c.queue.Forget(key) // Reset rate limiter state ``` ## Testing Controllers ### Unit Testing with Fake Client ```go import ( "k8s.io/client-go/kubernetes/fake" "k8s.io/client-go/informers" ) func TestController(t *testing.T) { // Create fake clientset with initial objects clientset := fake.NewSimpleClientset( &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "test-pod", Namespace: "default", }, }, ) // Create informer factory factory := informers.NewSharedInformerFactory(clientset, 0) podInformer := factory.Core().V1().Pods() // Create controller controller := NewController(clientset, podInformer) // Start informers ctx, cancel := context.WithCancel(context.Background()) defer cancel() factory.Start(ctx.Done()) cache.WaitForCacheSync(ctx.Done(), podInformer.Informer().HasSynced) // Test: add a pod clientset.CoreV1().Pods("default").Create(ctx, &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "new-pod"}, }, metav1.CreateOptions{}) // Wait for controller to process time.Sleep(100 * time.Millisecond) // Assert expected behavior // ... } ``` ### Integration Testing For integration tests, use `envtest` from controller-runtime, which spins up a real API server: ```go import ( "sigs.k8s.io/controller-runtime/pkg/envtest" ) func TestIntegration(t *testing.T) { testEnv := &envtest.Environment{} cfg, err := testEnv.Start() if err != nil { t.Fatal(err) } defer testEnv.Stop() // cfg is a real *rest.Config pointing to the test API server clientset, _ := kubernetes.NewForConfig(cfg) // ... run tests against real API server } ``` ## Summary client-go provides battle-tested primitives for building Kubernetes controllers: | Component | Purpose | Key Points | |-----------|---------|------------| | **Informer** | Watch + cache | LIST once, WATCH forever, local cache | | **SharedInformerFactory** | Share informers | One watch per resource type | | **Lister** | Read cache | Fast, no API calls | | **Work Queue** | Async processing | Deduplication, rate limiting | | **Rate Limiter** | Backoff/retry | Exponential backoff on failures | The pattern: 1. Informer watches resources, maintains cache 2. Event handlers enqueue keys (fast, non-blocking) 3. Workers dequeue and reconcile (using fresh cache data) 4. Rate limiter controls retry timing on failures This is the same pattern used by every built-in Kubernetes controller. Master it, and you can build controllers that scale. --- ## Beyond kube-proxy: eBPF Service Routing in Kubernetes - URL: https://svalle.ru/posts/kubernetes/beyond-kube-proxy-ebpf/ - Date: 2025-02-04 - Tags: kubernetes, networking, ebpf, cilium, kube-proxy, performance, iptables You've got 2,000 Services in your cluster. Every node has 20,000+ iptables rules. Pod startup takes 5 seconds just for iptables programming. Network policy updates take minutes to propagate. Welcome to the limits of kube-proxy. This post explains why kube-proxy struggles at scale, how eBPF fundamentally changes the game, and how to migrate to Cilium without breaking production. ## How kube-proxy Works (And Why It Breaks) kube-proxy runs on every node and implements Kubernetes Services. When you create a Service with a ClusterIP, kube-proxy makes that virtual IP actually work. ### The iptables Implementation By default, kube-proxy uses iptables. For each Service, it creates a chain of rules: ```bash # Simplified view of what kube-proxy creates for one Service -A KUBE-SERVICES -d 10.96.45.67/32 -p tcp --dport 80 -j KUBE-SVC-XXXX -A KUBE-SVC-XXXX -m statistic --mode random --probability 0.33 -j KUBE-SEP-AAA -A KUBE-SVC-XXXX -m statistic --mode random --probability 0.50 -j KUBE-SEP-BBB -A KUBE-SVC-XXXX -j KUBE-SEP-CCC -A KUBE-SEP-AAA -p tcp -j DNAT --to-destination 10.244.1.5:80 -A KUBE-SEP-BBB -p tcp -j DNAT --to-destination 10.244.2.8:80 -A KUBE-SEP-CCC -p tcp -j DNAT --to-destination 10.244.3.2:80 ``` This works. But count the rules: ``` Per Service: 1 rule in KUBE-SERVICES (match ClusterIP) N rules in KUBE-SVC-* (one per endpoint, for load balancing) N rules in KUBE-SEP-* (one per endpoint, for DNAT) Total: 1 + 2N rules per Service ``` With 2,000 Services averaging 3 endpoints each: ``` 2,000 × (1 + 2×3) = 14,000 rules minimum ``` Add NodePort rules, external IPs, and load balancer rules — you're easily at 20,000+ rules per node. ### The O(n) Problem iptables evaluates rules sequentially. When a packet arrives: ```d2 {alt="Sequential iptables rule evaluation: a packet arrives and is checked against Rule 1, Rule 2, Rule 3, and so on, each failing to match, until Rule 15,000 finally matches and DNAT is applied"} grid-columns: 1 vertical-gap: 64 packet: Packet arrives {style: {stroke-width: 0; fill: transparent}} r1: "Rule 1: Does it match? No -> next" r2: "Rule 2: Does it match? No -> next" r3: "Rule 3: Does it match? No -> next" dots: "..." {style: {stroke-width: 0; fill: transparent}} rn: "Rule 15,000: Does it match? Yes -> DNAT" packet -> r1 -> r2 -> r3 -> dots -> rn ``` Every packet traverses rules until it finds a match. With 20,000 rules, that's potentially 20,000 comparisons per packet. The first Service in the chain is fast; the last one is slow. **Measured impact:** | Services | Rules | Latency (p99) | |----------|-------|---------------| | 100 | ~700 | 0.5ms | | 1,000 | ~7,000 | 2ms | | 5,000 | ~35,000 | 8ms | | 10,000 | ~70,000 | 20ms+ | At scale, iptables becomes the bottleneck, not your application. ### Rule Update Storm When an endpoint changes (pod dies, new pod starts), kube-proxy must update iptables. But iptables doesn't support atomic updates of individual rules — kube-proxy rewrites large chunks of the rule set. ``` 1. Pod dies 2. Endpoint controller updates Endpoints object 3. kube-proxy sees the change 4. kube-proxy rebuilds iptables rules 5. iptables-restore replaces rules atomically (but slowly) ``` With 20,000 rules, step 5 can take seconds. During this time: - CPU spikes on every node - Network connections may be briefly disrupted - Other iptables users (CNI, NetworkPolicy) compete for the lock In large clusters, a rolling deployment can cause cascading iptables updates across all nodes simultaneously. ### conntrack Table Exhaustion iptables uses conntrack to track connection state (needed for DNAT reverse translation). The conntrack table has a default limit: ```bash $ cat /proc/sys/net/netfilter/nf_conntrack_max 131072 ``` 131,072 connections. Sounds like a lot until you have: - 100 pods per node - Each pod has 50 concurrent connections - That's 5,000 connections just from local pods Add in Service traffic, health checks, and monitoring — you hit the limit. When conntrack is full, new connections are dropped silently. ```bash # Check if you're hitting limits $ dmesg | grep conntrack nf_conntrack: table full, dropping packet ``` ## IPVS Mode: kube-proxy's Improvement Kubernetes 1.11+ supports IPVS (IP Virtual Server) mode in kube-proxy. IPVS is a kernel-level load balancer that uses hash tables instead of sequential rule matching. ### How IPVS Differs ``` iptables: Linear search O(n) Rule 1 -> Rule 2 -> Rule 3 -> ... -> Rule n IPVS: Hash table lookup O(1) Hash(ClusterIP:Port) -> Backend pool -> Select backend ``` IPVS stores Services in a hash table. Lookup is constant time regardless of Service count. ### Enabling IPVS Mode ```bash # Edit kube-proxy ConfigMap kubectl edit configmap kube-proxy -n kube-system # Change mode from "" (iptables) to "ipvs" apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: "ipvs" ipvs: scheduler: "rr" # round-robin, or "lc", "dh", "sh", etc. ``` ```bash # Restart kube-proxy kubectl rollout restart daemonset kube-proxy -n kube-system ``` ### IPVS Scheduling Algorithms IPVS supports multiple load balancing algorithms: | Algorithm | Flag | Description | |-----------|------|-------------| | Round Robin | `rr` | Rotate through backends | | Least Connections | `lc` | Send to backend with fewest connections | | Destination Hashing | `dh` | Hash destination IP for sticky routing | | Source Hashing | `sh` | Hash source IP for sticky routing | | Shortest Expected Delay | `sed` | Minimize expected delay | ```bash # Check IPVS rules $ ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.96.45.67:80 rr -> 10.244.1.5:80 Masq 1 3 0 -> 10.244.2.8:80 Masq 1 2 0 -> 10.244.3.2:80 Masq 1 4 0 ``` ### IPVS Limitations IPVS solves the O(n) lookup problem but still has issues: 1. **Still uses iptables** for some functions (SNAT, masquerade, NodePort) 2. **Still uses conntrack** — same table exhaustion problems 3. **Still runs in userspace** — kube-proxy watches API server, then programs kernel 4. **No Network Policy** — IPVS is for load balancing only; you still need iptables or another solution for policies IPVS is better than iptables mode, but it's an incremental improvement, not a fundamental redesign. ## Enter eBPF eBPF (extended Berkeley Packet Filter) is a technology that lets you run sandboxed programs inside the Linux kernel. Instead of configuring kernel behavior through static rules (iptables), you inject custom code that the kernel executes. ### What Makes eBPF Different Traditional approach (iptables): ```d2 {alt="Traditional iptables approach: kube-proxy in userspace watches the API and generates rules, feeding down into the kernel's iptables netfilter framework, a generic rule matching engine not optimized for the Kubernetes use case"} grid-columns: 1 vertical-gap: 64 user: "Userspace: kube-proxy watches API,\ngenerates rules" kernel: "Kernel: iptables netfilter framework\n- Generic rule matching engine\n- Not optimized for Kubernetes use case" user -> kernel ``` eBPF approach (Cilium): ```d2 {alt="eBPF approach: the Cilium agent in userspace watches the API and compiles eBPF programs, which run in the kernel as custom eBPF programs attached to network hooks, purpose-built for Kubernetes with hash tables and direct routing instead of rule chains"} grid-columns: 1 vertical-gap: 64 user: "Userspace: Cilium agent watches API,\ncompiles eBPF programs" kernel: "Kernel: Custom eBPF programs attached to network hooks\n- Purpose-built for Kubernetes\n- Hash tables, direct routing, no rule chains" user -> kernel ``` ### eBPF Programs An eBPF program is C code compiled to bytecode that the kernel verifies and JIT-compiles: ```c // Simplified example: redirect packets to a different destination SEC("sk_lookup") int service_lookup(struct bpf_sk_lookup *ctx) { // Look up Service in eBPF map (hash table) struct service_key key = { .ip = ctx->local_ip4, .port = ctx->local_port, }; struct service_value *svc = bpf_map_lookup_elem(&services_map, &key); if (!svc) return SK_PASS; // Not a Service, let it through // Select backend (load balancing) struct backend *backend = select_backend(svc); // Redirect to backend socket directly return bpf_sk_assign(ctx, backend->socket, 0); } ``` Key advantages: 1. **Hash table lookups**: O(1) Service resolution via eBPF maps 2. **No context switches**: Code runs in kernel, no userspace round-trips 3. **Socket-level routing**: Can intercept at socket connect(), before any packet is generated 4. **Programmable**: Can implement any logic, not limited to predefined rule types ### eBPF Hook Points eBPF programs attach to specific kernel hook points: ```d2 {alt="Vertical chain of eBPF hook points: an application's connect() syscall passes through the cgroup/connect4 hook (eBPF intercepts before the socket connects), the socket layer, tc ingress/egress (eBPF modifies packets in traffic control), and XDP in the NIC driver (the earliest possible hook) before reaching the network"} grid-columns: 1 vertical-gap: 40 app: Application {style: {stroke-width: 0; fill: transparent}} cgroup: "cgroup/connect4\neBPF: Intercept before socket connects" socket: Socket layer tc: "tc ingress/egress\neBPF: Modify packets in traffic control" xdp: "XDP (driver)\neBPF: Earliest possible hook, in NIC driver" network: Network {style: {stroke-width: 0; fill: transparent}} app -> cgroup: connect() syscall cgroup -> socket socket -> tc: Packet created tc -> xdp xdp -> network ``` Cilium uses multiple hooks: - **cgroup hooks**: Intercept socket operations (connect, bind, sendmsg) - **tc hooks**: Process packets after they're created - **XDP**: Ultra-fast packet processing at the driver level ### Socket-Level Load Balancing This is Cilium's killer feature. Traditional kube-proxy works at the packet level: ```d2 {alt="kube-proxy packet-level DNAT: the app connects to ClusterIP 10.96.45.67:80, a packet is created with dst 10.96.45.67, iptables DNAT rewrites the dst to backend 10.244.2.8, the packet is sent with the rewritten dst, and the response requires conntrack to reverse the DNAT"} grid-columns: 1 vertical-gap: 64 title: "kube-proxy (packet-level DNAT):" {style: {stroke-width: 0; fill: transparent}} connect: "App connects to 10.96.45.67:80 (ClusterIP)" created: "Packet created: src=10.244.1.10, dst=10.96.45.67" dnat: "iptables DNAT: rewrite dst to 10.244.2.8 (backend)" sent: "Packet sent: src=10.244.1.10, dst=10.244.2.8" response: Response requires conntrack to reverse the DNAT connect -> created -> dnat -> sent -> response ``` Cilium with socket-level LB: ```d2 {alt="Cilium socket-level load balancing, folded into two columns: the app calls connect() on the ClusterIP, eBPF intercepts the syscall, looks up the Service and selects backend 10.244.2.8, rewrites the socket destination, so the packet is created with the correct backend destination already and no DNAT or conntrack entry is needed for the Service"} grid-columns: 1 vertical-gap: 44 row1: "Cilium (socket-level)" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 34 vertical-gap: 8 connect: "App calls connect(10.96.45.67:80)" intercept: "eBPF intercepts connect() syscall" lookup: "Looks up Service,\nselects backend 10.244.2.8" connect -> intercept -> lookup } row2: "" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 40 vertical-gap: 8 nodnat: "No DNAT needed, no conntrack\nentry needed for Service" created: "Packet created: src=10.244.1.10,\ndst=10.244.2.8 (already correct!)" rewrite: "Rewrites socket destination\nto 10.244.2.8" rewrite -> created created -> nodnat } row1.lookup -> row2.rewrite ``` Benefits: - **No conntrack entries** for Service traffic (reduces table pressure) - **Lower latency** (no packet rewriting in the data path) - **Works with any protocol** (not just TCP/UDP) - **Survives backend changes** (socket already connected to real backend) ## Cilium as kube-proxy Replacement Cilium can fully replace kube-proxy, handling all Service types: ClusterIP, NodePort, LoadBalancer, and ExternalName. ### Architecture ```d2 {alt="Cilium architecture: the Cilium Agent (runs on every node as a DaemonSet) watches the K8s API, compiles eBPF programs, and loads them into the kernel, populating eBPF maps (Services, Backends, Connections hash tables in kernel memory) that are queried by the eBPF programs (Socket LB, Packet LB, Policy) attached to cgroup, tc, and XDP hooks"} grid-columns: 1 vertical-gap: 64 agent: "Cilium Agent\nRuns on every node as DaemonSet" maps: "eBPF Maps: Hash tables in kernel memory\n- Services: ClusterIP -> backend list\n- Backends: Backend ID -> Pod IP:Port\n- Connections: Connection tracking" programs: "eBPF Programs: Attached to cgroup, tc, XDP hooks\n- Socket LB: Intercept connect()\n- Packet LB: DNAT for NodePort, external\n- Policy: Network Policy enforcement" agent -> maps: "Watches K8s API (Services, Endpoints, Pods)\nCompiles eBPF programs\nLoads programs into kernel" maps -> programs: eBPF programs query maps ``` ### Installing Cilium with kube-proxy Replacement **Prerequisites:** - Linux kernel 4.19+ (5.4+ recommended for full features) - Direct routing or tunnel mode configured - API server address accessible from nodes ```bash # Install Cilium CLI curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz tar xzvf cilium-linux-amd64.tar.gz sudo mv cilium /usr/local/bin/ # Install Cilium with kube-proxy replacement cilium install \ --set kubeProxyReplacement=true \ --set k8sServiceHost=${API_SERVER_IP} \ --set k8sServicePort=${API_SERVER_PORT} ``` Or with Helm: ```bash helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium \ --namespace kube-system \ --set kubeProxyReplacement=true \ --set k8sServiceHost=${API_SERVER_IP} \ --set k8sServicePort=${API_SERVER_PORT} \ --set socketLB.enabled=true \ --set bpf.masquerade=true ``` **Important:** You must provide `k8sServiceHost` and `k8sServicePort` because Cilium needs to connect to the API server without relying on the `kubernetes` Service (which would require kube-proxy to work). ### Removing kube-proxy Once Cilium is running and handling Services: ```bash # Verify Cilium is handling Services cilium status kubectl exec -n kube-system ds/cilium -- cilium service list # Delete kube-proxy kubectl -n kube-system delete ds kube-proxy kubectl -n kube-system delete cm kube-proxy # Clean up iptables rules left by kube-proxy (run on each node) iptables-save | grep -v KUBE | iptables-restore ``` ### Verifying the Replacement ```bash # Check Cilium's view of Services kubectl exec -n kube-system ds/cilium -- cilium service list ID Frontend Service Type Backend 1 10.96.0.1:443 ClusterIP 10.0.0.5:6443 2 10.96.0.10:53 ClusterIP 10.244.0.15:53, 10.244.0.16:53 3 10.96.45.67:80 ClusterIP 10.244.1.5:80, 10.244.2.8:80 # Check eBPF maps kubectl exec -n kube-system ds/cilium -- cilium bpf lb list SERVICE ADDRESS BACKEND ADDRESS 10.96.0.1:443 10.0.0.5:6443 (1) 10.96.0.10:53 10.244.0.15:53 (1) 10.244.0.16:53 (2) 10.96.45.67:80 10.244.1.5:80 (1) 10.244.2.8:80 (2) ``` ### DSR (Direct Server Return) Cilium supports DSR for external traffic, where response packets go directly from the backend to the client, bypassing the original node: ``` Without DSR: Client -> Node1 (NodePort) -> Pod on Node2 -> Node1 -> Client ^ Response goes back through Node1 With DSR: Client -> Node1 (NodePort) -> Pod on Node2 -> Client ^ Response goes directly to client ``` Enable DSR: ```yaml # Helm values loadBalancer: mode: dsr ``` DSR reduces latency and load on the ingress node, but requires network support (backends must be able to send packets with source IP of the Service). ### Maglev Load Balancing Cilium supports Maglev consistent hashing, which provides: - Better distribution than random selection - Connection affinity survives backend changes (mostly) - Used by Google for their production load balancers ```yaml # Helm values loadBalancer: algorithm: maglev ``` ## Performance Comparison Real benchmarks comparing kube-proxy (iptables), kube-proxy (IPVS), and Cilium: ### Service Lookup Latency | Implementation | 100 Services | 1,000 Services | 10,000 Services | |----------------|--------------|----------------|-----------------| | iptables | 0.3ms | 1.5ms | 12ms | | IPVS | 0.1ms | 0.1ms | 0.15ms | | Cilium eBPF | 0.05ms | 0.05ms | 0.05ms | eBPF is constant time regardless of Service count. ### Throughput (Requests/sec) Testing with 1,000 Services, HTTP workload: | Implementation | RPS | CPU Usage | |----------------|-----|-----------| | iptables | 45,000 | 80% | | IPVS | 120,000 | 60% | | Cilium eBPF | 180,000 | 40% | ### Connection Setup Time | Implementation | p50 | p99 | |----------------|-----|-----| | iptables | 0.8ms | 5ms | | IPVS | 0.3ms | 1.2ms | | Cilium (socket LB) | 0.1ms | 0.3ms | Socket-level LB eliminates packet-level DNAT overhead entirely. ### Memory Usage | Implementation | Per Service | 10,000 Services | |----------------|-------------|-----------------| | iptables | ~2KB rules | ~20MB iptables rules | | IPVS | ~0.5KB | ~5MB | | Cilium eBPF | ~0.2KB | ~2MB eBPF maps | ## Migration Strategy Replacing kube-proxy in production requires care. Here's a safe migration path: ### Phase 1: Install Cilium Alongside kube-proxy ```bash # Install Cilium without kube-proxy replacement cilium install --set kubeProxyReplacement=false ``` Cilium handles Network Policy and pod networking. kube-proxy still handles Services. Verify everything works. ### Phase 2: Test kube-proxy Replacement in Staging ```bash # In staging cluster cilium install \ --set kubeProxyReplacement=true \ --set k8sServiceHost=${API_SERVER_IP} \ --set k8sServicePort=${API_SERVER_PORT} # Remove kube-proxy in staging kubectl -n kube-system delete ds kube-proxy ``` Test all Service types: - ClusterIP (internal services) - NodePort (external access) - LoadBalancer (cloud LB integration) - ExternalName (DNS aliases) - Headless Services (direct pod access) ### Phase 3: Production Migration Option A: Rolling migration (safer) ```bash # 1. Update Cilium to enable kube-proxy replacement helm upgrade cilium cilium/cilium \ --namespace kube-system \ --reuse-values \ --set kubeProxyReplacement=true \ --set k8sServiceHost=${API_SERVER_IP} \ --set k8sServicePort=${API_SERVER_PORT} # 2. Wait for Cilium to restart on all nodes kubectl rollout status ds/cilium -n kube-system # 3. Verify Services work kubectl exec -n kube-system ds/cilium -- cilium service list # 4. Remove kube-proxy kubectl -n kube-system delete ds kube-proxy ``` Option B: New cluster (cleanest) Provision new cluster without kube-proxy from the start: ```bash # kubeadm example kubeadm init --skip-phases=addon/kube-proxy ``` Then install Cilium with kube-proxy replacement enabled. ### Rollback Plan If things go wrong: ```bash # Re-deploy kube-proxy kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/kube-proxy/kube-proxy-ds.yaml # Disable Cilium's kube-proxy replacement helm upgrade cilium cilium/cilium \ --namespace kube-system \ --reuse-values \ --set kubeProxyReplacement=false ``` ## Debugging Cilium Service Routing ### Check Service Configuration ```bash # List all Services as Cilium sees them kubectl exec -n kube-system ds/cilium -- cilium service list # Get details for a specific Service kubectl exec -n kube-system ds/cilium -- cilium service get ``` ### Check eBPF Maps ```bash # List load balancer entries kubectl exec -n kube-system ds/cilium -- cilium bpf lb list # Check connection tracking kubectl exec -n kube-system ds/cilium -- cilium bpf ct list global ``` ### Monitor Traffic ```bash # Watch traffic in real-time kubectl exec -n kube-system ds/cilium -- cilium monitor # Filter for specific Service kubectl exec -n kube-system ds/cilium -- cilium monitor --related-to ``` ### Common Issues **Service not working after migration:** ```bash # Check if Service is in Cilium's map cilium service list | grep # If missing, check Cilium agent logs kubectl logs -n kube-system -l k8s-app=cilium | grep ``` **Socket LB not working:** ```bash # Verify cgroup eBPF programs are attached kubectl exec -n kube-system ds/cilium -- cilium bpf cgroup list # Check if pods are in the Cilium-managed cgroup # (requires cgroupv2) mount | grep cgroup2 ``` **NodePort not accessible:** ```bash # Check if NodePort is configured cilium service list | grep NodePort # Verify XDP or tc programs are attached to host interfaces kubectl exec -n kube-system ds/cilium -- cilium bpf prog list ``` ## When to Stick with kube-proxy eBPF isn't always the answer. Keep kube-proxy if: 1. **Old kernels**: eBPF features require kernel 4.19+; full features need 5.4+ 2. **Small clusters**: Under 500 Services, iptables overhead is negligible 3. **Simplicity**: kube-proxy is battle-tested, well-documented, and "just works" 4. **Compliance**: Some environments require known, auditable networking (iptables rules are more readable than eBPF bytecode) 5. **Windows nodes**: eBPF is Linux-only; Windows nodes need kube-proxy ## Summary kube-proxy served Kubernetes well, but its iptables-based design hits fundamental scaling limits: | Problem | kube-proxy | Cilium eBPF | |---------|------------|-------------| | Rule scaling | O(n) linear search | O(1) hash lookup | | Rule updates | Full table rewrite | Incremental map updates | | conntrack | Required for all Services | Only for external traffic | | CPU overhead | High at scale | Minimal | | Latency | Grows with Services | Constant | The migration path is well-established: 1. Install Cilium alongside kube-proxy 2. Test replacement in staging 3. Enable replacement in production 4. Remove kube-proxy For clusters beyond a few hundred Services, or where network latency matters, eBPF-based service routing isn't just faster — it's a fundamentally better architecture. --- --- ## From kubelet to Process: How Kubernetes Actually Runs Your Container - URL: https://svalle.ru/posts/kubernetes/kubelet-to-process/ - Date: 2025-02-03 - Tags: kubernetes, containerd, cri, runc, oci, debugging You apply a Pod manifest. Seconds later, your container is running. But what actually happened between `kubectl apply` and your process starting? The answer involves six layers: kubelet, CRI, containerd, shim, runc, and finally your process. Each layer exists for a reason, and knowing them helps you debug when things go wrong. ## The Stack ```d2 {alt="Six-layer stack from kubectl apply down to your process: kubectl apply goes to the API Server (stores Pod in etcd), which the kubelet (node agent, manages pod lifecycle) follows via watch, then a CRI gRPC call to containerd (container runtime, manages images), then containerd-shim (per-container process, survives restarts), then runc (OCI runtime, sets up namespaces/cgroups), which fork/execs Your Container (just a Linux process with isolation)"} grid-columns: 1 vertical-gap: 44 row1: "" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 83 vertical-gap: 8 apply: kubectl apply apiserver: "API Server\nstores Pod in etcd" kubelet: "kubelet\nnode agent, manages\npod lifecycle" apply -> apiserver apiserver -> kubelet: watch } row2: "" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 56 vertical-gap: 8 runc: "runc\nOCI runtime, sets up\nnamespaces/cgroups" shim: "containerd-shim\nper-container process,\nsurvives restarts" containerd: "containerd\ncontainer runtime,\nmanages images" containerd -> shim shim -> runc } row3: "" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 64 vertical-gap: 8 container: "Your Container\njust a Linux process\nwith isolation" } row1.kubelet -> row2.containerd: CRI gRPC row2.runc -> row3.container: fork/exec ``` Let's trace a pod creation through each layer. ## Layer 1: kubelet The kubelet is the node agent. It watches the API server for pods assigned to its node and makes them reality. ### What kubelet does: 1. **Watches** for pods scheduled to this node 2. **Computes** the desired state (which containers should exist) 3. **Calls** the container runtime via CRI to create/start/stop containers 4. **Reports** pod status back to the API server 5. **Manages** pod lifecycle (liveness probes, restarts, etc.) ### Where kubelet runs: ```bash # Usually a systemd service $ systemctl status kubelet # Configuration $ cat /var/lib/kubelet/config.yaml # Logs $ journalctl -u kubelet -f ``` ### kubelet doesn't create containers directly Before ~2017, kubelet had Docker-specific code built in. Now it uses the Container Runtime Interface (CRI) to talk to any compliant runtime. ``` kubelet -> CRI (gRPC) -> containerd -> CRI-O -> Docker (via cri-dockerd shim) ``` ## Layer 2: CRI (Container Runtime Interface) CRI is a gRPC API that kubelet uses to communicate with container runtimes. It defines two services: **RuntimeService**: Container lifecycle - `CreateContainer`, `StartContainer`, `StopContainer`, `RemoveContainer` - `ListContainers`, `ContainerStatus` - `ExecSync`, `Exec`, `Attach` **ImageService**: Image management - `PullImage`, `ListImages`, `RemoveImage` ### CRI in action You can talk to CRI directly using `crictl`: ```bash # List containers (like docker ps) $ crictl ps CONTAINER IMAGE CREATED STATE NAME POD ID a1b2c3d4e5 nginx 2 hours ago Running nginx x1y2z3 # List pods $ crictl pods POD ID CREATED STATE NAME NAMESPACE x1y2z3 2 hours ago Ready nginx-7d9fc5... default # Pull an image $ crictl pull nginx:latest # Get container logs $ crictl logs a1b2c3d4e5 # Exec into container $ crictl exec -it a1b2c3d4e5 sh ``` ### Checking CRI endpoint ```bash # See what runtime kubelet is using $ cat /var/lib/kubelet/kubeadm-flags.env KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///run/containerd/containerd.sock" # Or from kubelet config $ grep containerRuntime /var/lib/kubelet/config.yaml ``` ## Layer 3: containerd containerd is the most common container runtime in Kubernetes. It's what Docker uses under the hood (Docker = containerd + additional tooling). ### What containerd does: 1. **Manages images** — pulls, stores, unpacks OCI images 2. **Manages containers** — creates, starts, stops containers 3. **Manages snapshots** — filesystem layers (overlayfs) 4. **Manages tasks** — running processes within containers 5. **Spawns shims** — one shim per container ### containerd architecture ```d2 {alt="containerd architecture: the containerd daemon contains the Images Service, Containers Service, Snapshots Service, and Tasks Service side by side, and fans out downward to three shim processes, one per container (ctr1, ctr2, ctr3)"} direction: down containerd: containerd { grid-rows: 1 grid-gap: 24 images: "Images\nService" containers: "Containers\nService" snapshots: "Snapshots\nService" tasks: "Tasks\nService" } shim1: "shim\n(ctr1)" shim2: "shim\n(ctr2)" shim3: "shim\n(ctr3)" containerd -> shim1 containerd -> shim2 containerd -> shim3 ``` ### Interacting with containerd Use `ctr` (low-level) or `nerdctl` (Docker-compatible): ```bash # List namespaces (containerd uses namespaces for isolation) $ ctr namespaces ls NAME LABELS k8s.io # Kubernetes containers moby # Docker containers (if Docker is installed) # List containers in k8s.io namespace $ ctr -n k8s.io containers ls # List images $ ctr -n k8s.io images ls # List running tasks (processes) $ ctr -n k8s.io tasks ls ``` ### containerd configuration ```bash $ cat /etc/containerd/config.toml # Key settings: [plugins."io.containerd.grpc.v1.cri"] # CRI plugin configuration [plugins."io.containerd.grpc.v1.cri".containerd] # Default runtime default_runtime_name = "runc" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] runtime_type = "io.containerd.runc.v2" ``` ## Layer 4: containerd-shim The shim is a small process that sits between containerd and runc. There's one shim per container. ### Why shims exist: 1. **Decoupling**: Container survives containerd restart 2. **Stdio handling**: Keeps stdin/stdout/stderr open 3. **Exit status**: Reports container exit to containerd 4. **Reaping**: Acts as subreaper for orphaned processes ```d2 {alt="Shim decoupling: a containerd crash/restart leaves the per-container shims for container1 and container2 alive (containers keep running), and each shim's container, [container1] and [container2], is still running"} direction: down crash: containerd crash/restart {style: {stroke-width: 0; fill: transparent}} shim1: "shim (container1)" shim2: "shim (container2)" c1: "[container1] still running" {style: {stroke-width: 0; fill: transparent}} c2: "[container2] still running" {style: {stroke-width: 0; fill: transparent}} crash -> shim1: "Containers keep running!" crash -> shim2: "Containers keep running!" shim1 -> c1 shim2 -> c2 ``` ### Finding shims: ```bash $ ps aux | grep containerd-shim root 1234 containerd-shim-runc-v2 -namespace k8s.io -id abc123... root 1235 containerd-shim-runc-v2 -namespace k8s.io -id def456... ``` Each shim manages one container. The container ID matches what you see in `crictl ps`. ## Layer 5: runc runc is the OCI (Open Container Initiative) reference runtime. It does the actual work of creating the container: setting up namespaces, cgroups, and executing the process. ### What runc does: 1. **Parses** the OCI runtime spec (config.json) 2. **Creates namespaces** (pid, net, mnt, uts, ipc, user) 3. **Sets up cgroups** (CPU, memory, IO limits) 4. **Mounts filesystems** (rootfs, /proc, /sys, volumes) 5. **Applies security** (seccomp, capabilities, SELinux/AppArmor) 6. **Executes** the container entrypoint ### The OCI Runtime Spec runc reads a `config.json` that defines everything about the container: ```json { "ociVersion": "1.0.2", "process": { "terminal": true, "user": { "uid": 0, "gid": 0 }, "args": ["sh"], "env": ["PATH=/usr/bin:/bin", "TERM=xterm"], "cwd": "/" }, "root": { "path": "rootfs", "readonly": false }, "hostname": "container", "mounts": [ { "destination": "/proc", "type": "proc", "source": "proc" }, { "destination": "/dev", "type": "tmpfs", "source": "tmpfs" } ], "linux": { "namespaces": [ { "type": "pid" }, { "type": "network" }, { "type": "ipc" }, { "type": "uts" }, { "type": "mount" } ], "resources": { "memory": { "limit": 536870912 }, "cpu": { "quota": 50000, "period": 100000 } } } } ``` ### Using runc directly You can run runc manually (useful for debugging): ```bash # Create a bundle directory $ mkdir -p mycontainer/rootfs # Extract an image to rootfs $ docker export $(docker create alpine) | tar -C mycontainer/rootfs -xf - # Generate a spec $ cd mycontainer $ runc spec # Edit config.json if needed, then run $ runc run mycontainer ``` ### Find container's runc state ```bash # List runc containers $ runc list # Get container state $ runc state { "ociVersion": "1.0.2", "id": "abc123", "pid": 12345, "status": "running", "bundle": "/run/containerd/io.containerd.runtime.v2.task/k8s.io/abc123", "rootfs": "/run/containerd/io.containerd.runtime.v2.task/k8s.io/abc123/rootfs", "created": "2025-01-25T10:00:00Z" } ``` ## Layer 6: Your Container After all these layers, your container is just a Linux process. It has: - Its own **PID namespace** (PID 1 inside) - Its own **network namespace** (separate interfaces) - Its own **mount namespace** (container rootfs as /) - **Cgroup limits** (CPU, memory, etc.) - **Seccomp filters** (restricted syscalls) - **Dropped capabilities** (limited root powers) ```bash # From the host, it's just a process $ ps aux | grep root 12345 ... /your/entrypoint # Its namespaces $ ls -la /proc/12345/ns/ ``` ## The OCI Image Spec We've covered the runtime spec. The other OCI spec is the **image spec** — how container images are structured. ### Image layers An OCI image is a stack of filesystem layers: ```d2 {alt="An OCI image as a stack of three filesystem layers: Layer 3 application code (your Dockerfile additions) sits on top of Layer 2 runtime dependencies (apt-get install ...), which sits on top of Layer 1 base image (ubuntu:22.04)"} grid-columns: 1 vertical-gap: 24 l3: "Layer 3: Application code (your Dockerfile additions)" l2: "Layer 2: Runtime dependencies (apt-get install ...)" l1: "Layer 1: Base image (ubuntu:22.04)" ``` Layers are content-addressed (by SHA256 hash), immutable, and shared between images. ### Image manifest ```json { "schemaVersion": 2, "mediaType": "application/vnd.oci.image.manifest.v1+json", "config": { "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:abc123...", "size": 1234 }, "layers": [ { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:layer1...", "size": 12345678 }, { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:layer2...", "size": 23456789 } ] } ``` ### How containerd uses images 1. **Pull**: Download manifest and layers from registry 2. **Unpack**: Extract layers to snapshotter (overlayfs) 3. **Mount**: Stack layers using overlayfs for container rootfs ```bash # See image layers $ ctr -n k8s.io images ls $ ctr -n k8s.io content ls # See snapshots (unpacked layers) $ ctr -n k8s.io snapshots ls ``` ## Debugging at Each Layer ### Layer 1: kubelet ```bash # Check kubelet logs $ journalctl -u kubelet -f # Common issues: # - "failed to pull image" → registry/network issue # - "failed to create sandbox" → containerd issue # - "failed to start container" → runtime issue # Check kubelet is talking to containerd $ systemctl status containerd ``` ### Layer 2: CRI (crictl) ```bash # Check CRI is responding $ crictl info # List pods (should match kubectl get pods) $ crictl pods # List containers $ crictl ps -a # Get container details $ crictl inspect # Check why a container failed $ crictl logs # Debug pod sandbox issues $ crictl inspectp ``` ### Layer 3: containerd (ctr) ```bash # Check containerd health $ ctr -n k8s.io version # List containers at containerd level $ ctr -n k8s.io containers ls # List tasks (running containers) $ ctr -n k8s.io tasks ls # Check container bundle $ ctr -n k8s.io containers info ``` ### Layer 4-5: shim and runc ```bash # Find shim process $ ps aux | grep "containerd-shim.*" # Check runc state $ runc --root /run/containerd/runc/k8s.io state # List all runc containers $ runc --root /run/containerd/runc/k8s.io list ``` ### Layer 6: The Container Process ```bash # Find container's PID on host $ crictl inspect | jq '.info.pid' 12345 # Enter container's namespaces $ nsenter -t 12345 -a bash # Or just one namespace $ nsenter -t 12345 -n ip addr # Network namespace $ nsenter -t 12345 -m ls / # Mount namespace $ nsenter -t 12345 -p -r ps aux # PID namespace # Check cgroup limits $ cat /proc/12345/cgroup 0::/kubepods/burstable/pod-xyz/container-abc $ cat /sys/fs/cgroup/kubepods/burstable/pod-xyz/container-abc/memory.max # Trace syscalls $ strace -p 12345 ``` ## Common Debugging Scenarios ### Container won't start ```bash # 1. Check kubelet logs $ journalctl -u kubelet | grep # 2. Check container status $ crictl ps -a | grep $ crictl logs # 3. Check events $ kubectl describe pod # 4. Check runc directly $ runc --root /run/containerd/runc/k8s.io state ``` ### Container starts but exits immediately ```bash # Check exit code $ crictl inspect | jq '.status.exitCode' # Check logs $ crictl logs # Common causes: # - Exit 0: Command completed (wrong entrypoint) # - Exit 1: Application error # - Exit 137: OOM killed (128 + 9 = SIGKILL) # - Exit 139: Segfault (128 + 11 = SIGSEGV) ``` ### Container is slow/throttled ```bash # Find container's cgroup $ crictl inspect | jq '.info.runtimeSpec.linux.cgroupsPath' # Check CPU throttling $ cat /sys/fs/cgroup//cpu.stat nr_throttled 5000 # Throttled 5000 times! throttled_usec 60000000 # 60 seconds total throttle time # Check memory pressure $ cat /sys/fs/cgroup//memory.current $ cat /sys/fs/cgroup//memory.max ``` ### Image pull failures ```bash # Check image pull with crictl $ crictl pull # Check containerd logs $ journalctl -u containerd | grep # Common issues: # - Registry auth: check /var/lib/kubelet/config.json # - Network: can node reach registry? # - Disk space: df -h /var/lib/containerd ``` ## Putting It Together The full sequence when you `kubectl apply` a pod: 1. **API Server** stores Pod in etcd 2. **Scheduler** assigns Pod to a node 3. **kubelet** on that node sees the Pod (via watch) 4. **kubelet** calls containerd via CRI: `CreatePodSandbox` 5. **containerd** creates the pause container (network namespace holder) 6. **kubelet** calls containerd: `CreateContainer` for each container 7. **containerd** prepares the rootfs (overlayfs from image layers) 8. **containerd** spawns a **shim** for each container 9. **shim** calls **runc** with the OCI spec 10. **runc** creates namespaces, cgroups, mounts, security settings 11. **runc** `exec`s your entrypoint 12. **runc** exits, **shim** monitors the container 13. **kubelet** reports status to API server When something goes wrong, trace backwards through these layers until you find where it broke. ## Summary Kubernetes doesn't run containers — it orchestrates a stack of tools that do: | Layer | Tool | Purpose | |-------|------|---------| | 1 | kubelet | Node agent, pod lifecycle | | 2 | CRI | gRPC API to runtime | | 3 | containerd | Image and container management | | 4 | shim | Per-container daemon | | 5 | runc | OCI runtime, creates namespaces/cgroups | | 6 | Your process | Just a Linux process with isolation | Each layer has its own tools: | Layer | Debug Tool | |-------|------------| | kubelet | `journalctl -u kubelet` | | CRI | `crictl` | | containerd | `ctr` | | runc | `runc state/list` | | Container | `nsenter`, `/proc`, cgroup fs | When debugging, start at the top (kubectl describe, kubelet logs) and work down. By the time you're running `runc state`, you're debugging Linux primitives, not Kubernetes. --- --- ## What Is a Container, Really? Namespaces, Cgroups, and the Linux Primitives Behind Docker - URL: https://svalle.ru/posts/linux/what-is-a-container/ - Date: 2025-02-02 - Tags: containers, linux, namespaces, cgroups, docker, containerd There's no such thing as a container. Not in the Linux kernel, anyway. There's no "container" system call, no container data structure, no container subsystem. What we call "containers" are just regular Linux processes with some isolation applied. That isolation comes from two kernel features: **namespaces** and **cgroups**. Understanding these primitives demystifies containers, helps you debug them, and explains why certain things work the way they do. ## The Two Pillars **Namespaces**: Control what a process can *see*. A process in a PID namespace sees a different set of processes than the host. A process in a network namespace has its own network stack. **Cgroups**: Control what a process can *use*. Limit CPU, memory, IO, and other resources. Account for resource usage. That's it. A "container" is a process with namespace isolation and cgroup limits. Everything else — images, layers, runtimes — is tooling built on top of these primitives. ## Namespaces: Isolation of View Linux has eight namespace types. Each isolates a different aspect of the system: | Namespace | Isolates | Flag | |-----------|----------|------| | **PID** | Process IDs | `CLONE_NEWPID` | | **Network** | Network stack (interfaces, routing, ports) | `CLONE_NEWNET` | | **Mount** | Filesystem mounts | `CLONE_NEWNS` | | **UTS** | Hostname and domain name | `CLONE_NEWUTS` | | **IPC** | Inter-process communication (semaphores, message queues) | `CLONE_NEWIPC` | | **User** | User and group IDs | `CLONE_NEWUSER` | | **Cgroup** | Cgroup root directory | `CLONE_NEWCGROUP` | | **Time** | System clocks (Linux 5.6+) | `CLONE_NEWTIME` | ### PID Namespace In a new PID namespace, the first process becomes PID 1. It can only see processes in its namespace and descendants: ```bash # Host sees hundreds of processes $ ps aux | wc -l 247 # Create a new PID namespace $ sudo unshare --pid --fork --mount-proc bash # Inside: only see processes in this namespace $ ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 9032 5120 pts/0 S 10:00 0:00 bash root 8 0.0 0.0 10768 3328 pts/0 R+ 10:00 0:00 ps aux ``` PID 1 inside the namespace is *not* the real init. It's just bash, but it has PID 1 from its own perspective. The host still sees it with its real PID. ### Network Namespace Each network namespace has its own interfaces, routing tables, iptables rules, and ports: ```bash # Create a new network namespace $ sudo ip netns add mycontainer # The new namespace is isolated — no interfaces except loopback $ sudo ip netns exec mycontainer ip link 1: lo: mtu 65536 qdisc noop state DOWN # Create a veth pair to connect namespaces $ sudo ip link add veth-host type veth peer name veth-container $ sudo ip link set veth-container netns mycontainer # Configure the interface inside the namespace $ sudo ip netns exec mycontainer ip addr add 10.0.0.2/24 dev veth-container $ sudo ip netns exec mycontainer ip link set veth-container up $ sudo ip netns exec mycontainer ip link set lo up # Configure host side $ sudo ip addr add 10.0.0.1/24 dev veth-host $ sudo ip link set veth-host up # Now they can communicate $ sudo ip netns exec mycontainer ping 10.0.0.1 PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data. 64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.048 ms ``` This is exactly how container networking works — veth pairs connecting container network namespaces to a bridge on the host. ### Mount Namespace Isolates the filesystem view. Each namespace can have different mounts: ```bash # Create a new mount namespace $ sudo unshare --mount bash # Mounts here don't affect the host $ mount -t tmpfs tmpfs /mnt $ echo "secret" > /mnt/data # Exit and check — /mnt is empty on host $ exit $ ls /mnt # Empty ``` This is how containers get their own root filesystem. The container's mount namespace has the container image mounted as root, while the host sees its normal filesystem. ### UTS Namespace Isolates hostname and domain name: ```bash $ sudo unshare --uts bash $ hostname container-1 $ hostname container-1 # Host hostname unchanged $ exit $ hostname myhost.example.com ``` ### User Namespace Maps user IDs between namespaces. Root inside the container can be a non-root user on the host: ```bash # Create user namespace (can be done without root) $ unshare --user --map-root-user bash $ whoami root $ id uid=0(root) gid=0(root) groups=0(root) # But on the host, this process runs as your regular user ``` This is the basis of "rootless containers" — the process thinks it's root, but has no real root privileges on the host. ### Inspecting Namespaces Every process has namespace references in `/proc`: ```bash $ ls -la /proc/$$/ns/ lrwxrwxrwx 1 user user 0 Jan 25 10:00 cgroup -> 'cgroup:[4026531835]' lrwxrwxrwx 1 user user 0 Jan 25 10:00 ipc -> 'ipc:[4026531839]' lrwxrwxrwx 1 user user 0 Jan 25 10:00 mnt -> 'mnt:[4026531840]' lrwxrwxrwx 1 user user 0 Jan 25 10:00 net -> 'net:[4026531992]' lrwxrwxrwx 1 user user 0 Jan 25 10:00 pid -> 'pid:[4026531836]' lrwxrwxrwx 1 user user 0 Jan 25 10:00 user -> 'user:[4026531837]' lrwxrwxrwx 1 user user 0 Jan 25 10:00 uts -> 'uts:[4026531838]' ``` The numbers (like `4026531836`) are inode numbers. Processes in the same namespace share the same inode. **Enter a container's namespace:** ```bash # Find container's PID on host $ docker inspect --format '{{.State.Pid}}' mycontainer 12345 # Enter its namespaces $ sudo nsenter --target 12345 --mount --uts --ipc --net --pid bash ``` ## Cgroups: Resource Control Namespaces isolate *view*. Cgroups limit *resources*. Cgroups (control groups) allow you to: - **Limit** resources (max 1 CPU, max 512MB memory) - **Prioritize** resources (this group gets more CPU than that one) - **Account** for usage (how much CPU has this group used?) - **Control** processes (freeze all processes in a group) ### Cgroup Hierarchy Cgroups are organized in a hierarchy, exposed as a filesystem: ```bash # Cgroups v2 unified hierarchy (modern) $ ls /sys/fs/cgroup/ cgroup.controllers cgroup.subtree_control cpu.stat memory.current ... # Cgroups v1 (legacy) — separate hierarchies per controller $ ls /sys/fs/cgroup/ blkio cpu cpuacct cpuset devices freezer memory net_cls pids ... ``` ### Cgroups v1 vs v2 **Cgroups v1** (legacy): - Separate hierarchy per resource controller - A process can be in different cgroups for CPU vs memory - Complex, hard to manage - Still common in older systems **Cgroups v2** (unified, recommended): - Single unified hierarchy - A process is in one cgroup for all controllers - Simpler, better resource distribution - Default in recent Linux distributions (Ubuntu 22.04+, Fedora 31+) **Check which version you're using:** ```bash # If this exists and has content, you're on v2 $ cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io memory hugetlb pids rdma misc # If this exists, you're on v1 or hybrid $ ls /sys/fs/cgroup/memory/ ``` ### Creating a Cgroup (v2) ```bash # Create a new cgroup $ sudo mkdir /sys/fs/cgroup/mycontainer # Set memory limit (256MB) $ echo $((256 * 1024 * 1024)) | sudo tee /sys/fs/cgroup/mycontainer/memory.max # Set CPU limit (50% of one core) $ echo "50000 100000" | sudo tee /sys/fs/cgroup/mycontainer/cpu.max # Add a process to the cgroup $ echo $$ | sudo tee /sys/fs/cgroup/mycontainer/cgroup.procs # Now this shell and its children are limited ``` ### Memory Limits ```bash # memory.max — hard limit (OOM kill if exceeded) $ cat /sys/fs/cgroup/mycontainer/memory.max 268435456 # memory.current — current usage $ cat /sys/fs/cgroup/mycontainer/memory.current 8462336 # memory.high — throttling threshold (soft limit) $ cat /sys/fs/cgroup/mycontainer/memory.high max ``` When a process exceeds `memory.max`, the kernel OOM-kills it. This is why containers get "OOMKilled" — they hit their cgroup memory limit. ### CPU Limits CPU limits use the CFS (Completely Fair Scheduler) bandwidth controller: ```bash # cpu.max — format: "$MAX $PERIOD" (microseconds) $ cat /sys/fs/cgroup/mycontainer/cpu.max 50000 100000 ``` This means: in every 100ms period, the cgroup can use 50ms of CPU time — effectively 50% of one core. **The CPU throttling problem:** If your application uses all its quota early in the period, it gets throttled until the next period. This causes latency spikes: ``` Period 1: |████████░░░░░░░░░░░░| Used quota early, throttled for 50ms Period 2: |████████░░░░░░░░░░░░| Same pattern ``` This is why Kubernetes apps with CPU limits can have inconsistent latency — they're being throttled even when the node has spare CPU. ### IO Limits ```bash # io.max — limit IO per device # Format: "$MAJOR:$MINOR rbps=$READ_BPS wbps=$WRITE_BPS riops=$READ_IOPS wiops=$WRITE_IOPS" $ echo "8:0 rbps=10485760 wbps=10485760" | sudo tee /sys/fs/cgroup/mycontainer/io.max ``` ### PID Limits Limit the number of processes: ```bash # pids.max — maximum number of processes $ echo 100 | sudo tee /sys/fs/cgroup/mycontainer/pids.max ``` This prevents fork bombs from taking down the host. ### Viewing Container Cgroups Find a container's cgroup: ```bash # Get container PID $ docker inspect --format '{{.State.Pid}}' mycontainer 12345 # Find its cgroup $ cat /proc/12345/cgroup 0::/system.slice/docker-abc123.scope # View its limits $ cat /sys/fs/cgroup/system.slice/docker-abc123.scope/memory.max 536870912 ``` ## Building a "Container" by Hand Let's create a container using only Linux primitives — no Docker, no containerd: ```bash #!/bin/bash # mini-container.sh — A container in 50 lines set -e ROOTFS="/tmp/container-rootfs" CGROUP="/sys/fs/cgroup/mini-container" # Create a minimal rootfs (using Alpine) mkdir -p "$ROOTFS" if [ ! -f "$ROOTFS/bin/sh" ]; then echo "Downloading Alpine rootfs..." curl -L https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/x86_64/alpine-minirootfs-3.19.0-x86_64.tar.gz | tar xz -C "$ROOTFS" fi # Create cgroup with limits sudo mkdir -p "$CGROUP" echo $((128 * 1024 * 1024)) | sudo tee "$CGROUP/memory.max" > /dev/null # 128MB echo "50000 100000" | sudo tee "$CGROUP/cpu.max" > /dev/null # 50% CPU echo 50 | sudo tee "$CGROUP/pids.max" > /dev/null # 50 processes # Launch container with namespaces sudo unshare \ --pid \ --mount \ --uts \ --ipc \ --net \ --fork \ --cgroup \ /bin/bash -c " # Add ourselves to the cgroup echo \$\$ > $CGROUP/cgroup.procs # Set hostname hostname container # Setup mount namespace mount --make-rprivate / # Mount proc and sys in new root mkdir -p $ROOTFS/proc $ROOTFS/sys $ROOTFS/dev # Pivot to new root cd $ROOTFS mkdir -p .old-root pivot_root . .old-root # Mount essential filesystems mount -t proc proc /proc mount -t sysfs sys /sys mount -t devtmpfs dev /dev # Unmount old root umount -l /.old-root rmdir /.old-root # Run shell exec /bin/sh " # Cleanup cgroup on exit sudo rmdir "$CGROUP" 2>/dev/null || true ``` This creates a process that: 1. Has its own PID namespace (PID 1 inside) 2. Has its own mount namespace (Alpine rootfs as /) 3. Has its own hostname 4. Is limited to 128MB memory, 50% CPU, 50 processes That's a container. No Docker required. ## How Docker/containerd Use These Primitives When you run `docker run alpine sh`, Docker: 1. **Pulls the image** — downloads and extracts layers to a directory 2. **Creates a rootfs** — uses a union filesystem (overlayfs) to combine layers 3. **Creates namespaces** — PID, mount, network, UTS, IPC 4. **Creates a cgroup** — with resource limits from `--memory`, `--cpus`, etc. 5. **Sets up networking** — creates veth pair, connects to bridge 6. **Pivots root** — makes the image rootfs the container's `/` 7. **Drops privileges** — applies seccomp, capabilities, etc. 8. **Execs the entrypoint** — runs your process Every container runtime (Docker, containerd, CRI-O, podman) does these same steps, just with different tooling. ## Debugging with Primitives Knowing the primitives helps you debug: **See a container's namespaces:** ```bash $ sudo ls -la /proc//ns/ ``` **Enter a container's namespace:** ```bash $ sudo nsenter -t -n ip addr # Network namespace only $ sudo nsenter -t -a bash # All namespaces ``` **Check cgroup limits:** ```bash $ cat /sys/fs/cgroup//memory.max $ cat /sys/fs/cgroup//cpu.max ``` **Check cgroup usage:** ```bash $ cat /sys/fs/cgroup//memory.current $ cat /sys/fs/cgroup//cpu.stat ``` **See what's throttling:** ```bash $ cat /sys/fs/cgroup//cpu.stat usage_usec 123456789 user_usec 100000000 system_usec 23456789 nr_periods 10000 nr_throttled 500 # <-- Throttled 500 times throttled_usec 5000000 # <-- 5 seconds total throttle time ``` ## Summary Containers are not magic. They're Linux processes with: - **Namespaces** for isolation (what the process can see) - **Cgroups** for resource control (what the process can use) Understanding these primitives helps you: - Debug container issues at the source - Understand resource limits and why things get OOM-killed or throttled - Demystify container networking - Build custom isolation when needed When a container misbehaves, you're not debugging "container technology" — you're debugging Linux processes with isolation. The same tools work: `/proc`, `strace`, `nsenter`, and the cgroup filesystem. --- ## Bottlenecks in Large-Scale Multi-Cluster Kubernetes - URL: https://svalle.ru/posts/kubernetes/multi-cluster-bottlenecks/ - Date: 2025-02-01 - Tags: kubernetes, multi-cluster, scalability, argocd, observability, fleet-management You went multi-cluster to escape the limits of a single control plane. Now you have 50 clusters, and new bottlenecks have emerged — at the coordination layer, at shared infrastructure, at the tools that manage the fleet. The problems are different, but the pain is familiar. This post surveys the common bottlenecks in large-scale multi-cluster Kubernetes environments and how to identify them. ## The Shape of Multi-Cluster Bottlenecks In a single cluster, bottlenecks are usually: - etcd (write throughput, watch fan-out) - API server (request rate, webhook latency) - Scheduler (pod throughput) - Controllers (reconciliation speed) Multi-cluster solves these by partitioning. But it introduces new chokepoints: - **Coordination layer**: The hub cluster, fleet management APIs - **Shared infrastructure**: Image registries, secret stores, Git repos - **Cross-cutting tools**: ArgoCD, observability stack, policy engines The pattern: what worked at 5 clusters breaks at 50. A tool that seemed lightweight becomes a bottleneck when multiplied across the fleet. ## Hub Cluster Pressure If you're using KubeFleet, Azure Fleet Manager, or similar hub-spoke architectures, the hub cluster coordinates the fleet. It's lightweight by design — but not infinitely so. ### Watch Fan-Out The hub maintains state for every member cluster: - `MemberCluster` objects (one per cluster) - `ClusterResourcePlacement` objects (your placement intents) - `Work` objects (propagated resources, potentially thousands) Member agents watch the hub for changes. With 50 clusters, that's 50 agents maintaining watches. With 100 placements generating 10 Work objects each across 50 clusters, you have 50,000 Work objects. **Symptoms:** - Hub API server latency increases - Member agents report slow sync - `apiserver_request_duration_seconds` shows elevated P99 **Diagnosis:** ```bash # On hub cluster kubectl top pods -n fleet-system kubectl get --raw /metrics | grep apiserver_request_duration # Watch count kubectl get --raw /metrics | grep apiserver_registered_watchers ``` **Mitigation:** - Right-size hub cluster (it's often under-provisioned) - Reduce Work object churn (batch changes, avoid frequent updates) - Consider multiple hubs for very large fleets (federation of federations) ### Hub etcd Sizing The hub's etcd stores all fleet coordination state. More clusters and placements = more objects = more etcd pressure. **Watch for:** - etcd latency (`etcd_request_duration_seconds`) - Database size (`etcd_debugging_mvcc_db_total_size_in_bytes`) - Compaction falling behind **Mitigation:** - Dedicated etcd nodes with SSDs - Increase etcd quota if hitting limits - Clean up stale Work objects and completed placements ## Member Agent Sync Each member cluster runs an agent that pulls state from the hub. At scale, these agents become a factor. ### Pull Frequency vs Freshness Agents poll the hub for Work objects. More frequent polling = fresher state but more hub load. Less frequent = stale state but lighter load. **Trade-off:** ``` 50 clusters × 1 sync/second = 50 requests/second to hub 50 clusters × 1 sync/10 seconds = 5 requests/second to hub ``` Most agents use watches (efficient), but reconnections and resyncs generate load. ### Work Object Size A Work object contains the full manifest of propagated resources. Propagating a large ConfigMap or a Deployment with lengthy specs means large Work objects. **Symptoms:** - Slow sync times - High memory usage in member agents - Network bandwidth between hub and members **Diagnosis:** ```bash # Size of Work objects kubectl get work -n fleet-member-cluster-1 -o json | wc -c ``` **Mitigation:** - Avoid propagating large ConfigMaps (use external config stores) - Propagate references instead of data where possible - Compress or chunk large manifests ### Status Reporting Storms Member agents report status back to the hub. With many resources across many clusters, status updates can overwhelm the hub. **Symptoms:** - Hub API server write latency spikes - etcd write throughput saturated - Agents backing off on status updates **Mitigation:** - Batch status updates - Report status less frequently for stable resources - Use conditions efficiently (don't update if unchanged) ## ArgoCD at Scale ArgoCD is often the tool managing deployments across multi-cluster fleets. A single ArgoCD instance managing 50+ clusters hits limits. ### Application Controller The application controller reconciles Applications — comparing desired state (Git) with actual state (clusters). Each Application means: - Watching the target cluster - Generating manifests (calling repo server) - Computing diff - Optionally syncing **At scale:** ``` 500 Applications × 3-minute sync interval = ~3 reconciliations/second ``` This seems manageable until you account for: - Manifest generation time (Helm templates, Kustomize) - Target cluster API latency - Diff computation for large Applications **Symptoms:** - Applications stuck in "Progressing" - Long sync times - Controller CPU pegged **Diagnosis:** ```bash # Controller metrics kubectl port-forward svc/argocd-metrics 8082:8082 -n argocd curl localhost:8082/metrics | grep argocd_app_reconcile # Queue depth curl localhost:8082/metrics | grep workqueue_depth ``` **Mitigation:** - Increase controller replicas (with sharding) - Reduce sync frequency for stable Applications - Use Server-Side Apply (faster diffs) ### Repo Server Bottleneck The repo server generates manifests from Git repos. It's CPU and memory intensive: - Cloning repos - Running Helm template - Running Kustomize build - Caching results **Symptoms:** - Slow manifest generation - Repo server OOMKilled - Applications show "ComparisonError" **Diagnosis:** ```bash kubectl top pods -n argocd -l app.kubernetes.io/component=repo-server kubectl logs -n argocd -l app.kubernetes.io/component=repo-server | grep -i error ``` **Mitigation:** - Scale repo server horizontally - Increase memory limits (Helm/Kustomize can be memory-hungry) - Use repo server parallelism settings - Cache Helm dependencies (avoid re-downloading) ### Redis Pressure ArgoCD uses Redis for caching. With many Applications and clusters, Redis becomes a factor. **Symptoms:** - High Redis memory usage - Slow cache operations - Evictions causing cache misses (re-generating manifests) **Mitigation:** - Increase Redis memory - Use Redis Cluster for HA (ArgoCD 2.x supports this) - Tune cache TTLs ### Git Rate Limits ArgoCD polls Git repos for changes. With many Applications: ``` 500 Applications polling every 3 minutes = 167 Git fetches/minute ``` If using GitHub, you'll hit rate limits. If using webhooks, you'll generate storms on every push. **Symptoms:** - "rate limit exceeded" errors - Applications not detecting changes - Webhook timeouts **Mitigation:** - Use webhooks instead of polling (more efficient) - Consolidate repos (fewer repos = fewer fetches) - Increase polling interval - Use GitHub App authentication (higher rate limits) ### Sharding Strategies A single ArgoCD can't manage thousands of Applications efficiently. Sharding options: **Option 1: Shard by cluster** - Multiple ArgoCD controller replicas - Each controller handles a subset of clusters - Use `--application-namespaces` and cluster labels ```yaml # Controller deployment env: - name: ARGOCD_CONTROLLER_REPLICAS value: "3" ``` **Option 2: Multiple ArgoCD instances** - Dedicated ArgoCD per environment (prod, staging) - Or per team / business unit - More operational overhead but better isolation **Option 3: ApplicationSets with progressive sync** - Generate Applications dynamically - Use rolling sync strategies - Limit concurrent syncs ```yaml apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: my-app spec: generators: - clusters: selector: matchLabels: environment: production strategy: type: RollingSync rollingSync: steps: - matchExpressions: - key: region operator: In values: [us-east-1] - matchExpressions: - key: region operator: In values: [us-west-2] template: # ... ``` ## Image Registry Every cluster pulls container images. At scale, the registry becomes critical infrastructure. ### The Thundering Herd You push a new image and update 50 clusters. All 50 start pulling simultaneously: ``` 50 clusters × 10 nodes × 500MB image = 250GB of transfer, all at once ``` **Symptoms:** - Registry timeouts - Image pull failures ("429 Too Many Requests") - Slow deployments **Mitigation:** **Registry caching / pull-through cache:** ```yaml # Deploy a registry mirror in each cluster or region # Configure containerd/docker to use local mirror [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://registry-cache.internal:5000"] ``` **Geo-distributed registries:** - Replicate images to regional registries - Route clusters to nearest registry **Pre-pulling:** - DaemonSet that pulls images before deployment - Reduces thundering herd by spreading pull over time **Staggered rollouts:** - Don't update all 50 clusters simultaneously - Roll out region by region, cluster by cluster ### Registry as SPOF If your single registry goes down, no cluster can pull new images. **Mitigation:** - HA registry deployment - Multiple registry replicas across zones - Fallback registries in image pull specs (limited support) ## Observability Overhead Monitoring 50 clusters generates massive telemetry. The observability stack itself becomes a scaling challenge. ### Prometheus Federation Limits Classic pattern: Prometheus per cluster, federate to central Prometheus. **Problems at scale:** - Federation scrapes are expensive (pulls all series) - Central Prometheus cardinality explodes - Query latency increases **Symptoms:** - Federation scrapes timing out - Central Prometheus OOM - Slow dashboards **Mitigation:** - **Thanos/Cortex/Mimir**: Scalable backends that accept remote-write - **Remote write**: Push metrics instead of federation pull - **Recording rules**: Aggregate at edge, send summaries - **Reduce cardinality**: Drop high-cardinality labels before sending ```yaml # Per-cluster Prometheus: remote write to central remote_write: - url: https://thanos-receive.monitoring:19291/api/v1/receive ``` ### Central Logging 50 clusters × 1000 pods × 10 log lines/second = 500,000 lines/second. **Symptoms:** - Log ingestion lag - Dropped logs - Query timeouts **Mitigation:** - **Sampling**: Don't ship all logs (sample debug, keep errors) - **Edge aggregation**: Aggregate common patterns locally - **Tiered storage**: Hot/cold storage for logs - **Per-cluster Loki**: Query individual clusters, federate on demand ### When Monitoring Causes the Problem Heavy monitoring can stress clusters: - Prometheus scraping thousands of targets - Logging agents consuming CPU/memory - Tracing overhead on every request **Watch for:** - Monitoring pods consuming significant cluster resources - Scrape intervals too aggressive - Overly verbose logging levels **Mitigation:** - Scrape less frequently for stable metrics - Use service discovery efficiently (don't scrape what you don't need) - Set appropriate resource limits on monitoring components ## Secrets Distribution Multi-cluster secrets management adds latency and complexity. ### Vault at Scale Vault is often the central secrets store. Every cluster fetches secrets: ``` 50 clusters × 100 secrets × refresh every 5 minutes = 1000 requests/minute ``` **Symptoms:** - Vault latency increases - Secret sync delays - Pod startup blocked waiting for secrets **Mitigation:** - Vault replication (regional Vault clusters) - Caching (external-secrets-operator caches locally) - Longer TTLs for stable secrets - Batch secret fetches ### External-Secrets-Operator Runs in each cluster, syncs secrets from external stores. **At scale:** - Each cluster runs reconciliation loops - All hitting the same Vault/AWS Secrets Manager/etc. **Mitigation:** - Tune sync intervals (not everything needs 30-second refresh) - Use refresh strategies (only refresh on pod restart) - Batch requests where possible ## Diagnosis: Finding the Bottleneck When things slow down, where do you look first? ### Systematic Approach 1. **Start at the symptom** - Slow deployments? → ArgoCD, registry - Stale state in member clusters? → Hub, agent sync - Metrics gaps? → Observability stack 2. **Check the coordination layer** - Hub cluster health (API server, etcd) - Member agent logs - Work object backlogs 3. **Check shared infrastructure** - Registry response times - Git repo rate limits - Vault/secrets latency 4. **Check cross-cutting tools** - ArgoCD queue depths and reconciliation times - Prometheus scrape durations - Logging ingestion lag ### Key Metrics Across the Fleet **Hub cluster:** ```promql # API server latency histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (le, verb)) # etcd latency histogram_quantile(0.99, sum(rate(etcd_request_duration_seconds_bucket[5m])) by (le, operation)) # Watch count sum(apiserver_registered_watchers) by (group, resource) ``` **ArgoCD:** ```promql # Reconciliation duration histogram_quantile(0.99, sum(rate(argocd_app_reconcile_bucket[5m])) by (le)) # Queue depth workqueue_depth{name="app_operation_processing_queue"} # Sync failures sum(increase(argocd_app_sync_total{phase="Failed"}[1h])) by (dest_server) ``` **Registry (if you expose metrics):** ```promql # Pull latency histogram_quantile(0.99, sum(rate(registry_http_request_duration_seconds_bucket[5m])) by (le)) # Request rate sum(rate(registry_http_requests_total[5m])) by (method) ``` **Per-cluster Prometheus (federate or remote-write these):** ```promql # API server health across fleet sum(apiserver_request_total) by (cluster, code) # Pod startup latency across fleet histogram_quantile(0.99, sum(rate(kubelet_pod_start_duration_seconds_bucket[5m])) by (le, cluster)) ``` ### When You're Stuck If metrics don't reveal the bottleneck: 1. **Add tracing**: Instrument the slow path (ArgoCD reconciliation, agent sync) 2. **Profile**: pprof on Go components (ArgoCD, fleet agents) 3. **Simplify**: Reduce fleet size temporarily to isolate 4. **Bisect**: Disable half the clusters/Applications, see if problem persists ## Summary Multi-cluster Kubernetes trades single-cluster bottlenecks for coordination-layer bottlenecks. At scale, watch for: | Layer | Bottlenecks | |-------|-------------| | Hub cluster | API server load, etcd size, watch fan-out | | Member agents | Sync frequency, Work object size, status storms | | ArgoCD | Application controller, repo server, Git rate limits | | Image registry | Pull thundering herd, single registry SPOF | | Observability | Federation limits, logging ingestion, cardinality | | Secrets | Vault load, sync latency | The tools that manage your fleet can become the bottleneck your fleet was supposed to escape. Monitor the coordinators, not just the clusters. --- ## Kubernetes Networking Demystified: Tracing the Magic (and Debugging the Nightmare) - URL: https://svalle.ru/posts/kubernetes/kubernetes-networking-demystified/ - Date: 2025-01-31 - Tags: kubernetes, networking, iptables, overlay, debugging, vxlan, kube-proxy You run `curl my-service:8080` from a pod, and it just works. The request reaches another pod, possibly on a different node, and you get a response. Magic. Until it doesn't work. Then you're staring at iptables rules, tcpdump output, and CNI logs wondering where your packet went. This post traces a packet through Kubernetes networking — from pod to Service to pod across nodes. Understanding this flow turns debugging from nightmare to systematic diagnosis. ## The Setup: What We're Tracing Let's make this concrete. We have: ``` Node 1 (192.168.1.10) Pod A: 10.244.1.5 (client) Node 2 (192.168.1.11) Pod B: 10.244.2.8 (server, behind Service) Pod C: 10.244.2.9 (server, behind Service) Service: my-service ClusterIP: 10.96.45.67 Port: 8080 Endpoints: 10.244.2.8:8080, 10.244.2.9:8080 ``` Pod A runs: `curl my-service:8080` What actually happens? Let's trace it. ## Part 1: Pod-to-Pod on the Same Node Before crossing nodes, let's understand the simplest case: two pods on the same node. ### The Virtual Network Each pod gets its own network namespace — an isolated network stack with its own interfaces, routes, and iptables. But pods need to communicate. Kubernetes creates a **virtual ethernet pair (veth)** connecting each pod to the node: ```d2 {alt="Inside a node, Pod A and Pod B each have an eth0 interface, and each pod connects via a veth pair down to the shared bridge (cni0/cbr0)"} node: Node { grid-rows: 2 vertical-gap: 40 horizontal-gap: 48 pod_a: Pod A { grid-rows: 1 grid-gap: 24 eth0 } pod_b: Pod B { grid-rows: 1 grid-gap: 24 eth0 } bridge: "Bridge (cni0/cbr0)" pod_a -- bridge: veth pod_b -- bridge: veth } ``` **The veth pair**: One end (`eth0`) is inside the pod's network namespace. The other end is attached to a bridge on the node. The pair acts like a virtual cable. **The bridge**: A software switch (commonly named `cni0`, `cbr0`, or `docker0`). All pod veths connect to this bridge. Packets to other pods on the same node go through the bridge. ### Packet Flow: Same Node Pod A (10.244.1.5) sends to Pod B (10.244.1.6), both on Node 1: ``` 1. Pod A sends packet (src: 10.244.1.5, dst: 10.244.1.6) 2. Packet exits via eth0 (inside pod) -> enters veth -> arrives at bridge 3. Bridge looks up MAC for 10.244.1.6 -> forwards to Pod B's veth 4. Packet enters Pod B's eth0 5. Pod B receives packet ``` No iptables (for basic connectivity), no encapsulation. Just Layer 2 switching on the bridge. ### See It Yourself ```bash # On the node, list veths ip link show type veth # See the bridge ip link show type bridge brctl show cni0 # See pod connections bridge fdb show dev cni0 # tcpdump on the bridge tcpdump -i cni0 -n host 10.244.1.5 ``` ## Part 2: How Services Work (iptables) Now for the interesting part. Pod A doesn't call Pod B directly — it calls `my-service:8080`. The Service has a ClusterIP (10.96.45.67) that doesn't exist on any interface. How does this work? ### The ClusterIP Illusion A ClusterIP is a virtual IP. No interface has this address. No ARP entry exists. If you try to ping it from outside the cluster, nothing responds. Yet from inside a pod, it works. The secret: **iptables rewrites the destination before the packet leaves**. ### kube-proxy's Job kube-proxy runs on every node. It watches Services and Endpoints, then programs iptables rules that: 1. Intercept packets destined for ClusterIPs 2. Rewrite the destination to an actual pod IP (DNAT) 3. Load balance across endpoints ### The Chain of Chains iptables organizes rules into chains. kube-proxy creates a hierarchy: ```d2 {alt="iptables chain hierarchy: a packet destined for the ClusterIP passes through PREROUTING (or OUTPUT for local pods) to KUBE-SERVICES, which matches on ClusterIP:port, to the Service's KUBE-SVC chain, which randomly selects an endpoint by splitting 50/50 to two KUBE-SEP endpoint chains that DNAT to the pod IPs"} grid-columns: 1 vertical-gap: 40 packet: "Packet arrives (dst: 10.96.45.67:8080)" {style: {stroke-width: 0; fill: transparent}} prerouting: "PREROUTING (or OUTPUT for local pods)" services: "KUBE-SERVICES\nmatches on ClusterIP:port" svc: "KUBE-SVC-XYZABC123\nthe Service's chain, randomly selects endpoint" seps: "" { grid-rows: 1 grid-gap: 24 style: {stroke-width: 0; fill: transparent} sep1: "KUBE-SEP-ENDPOINT1\n(DNAT to 10.244.2.8)" sep2: "KUBE-SEP-ENDPOINT2\n(DNAT to 10.244.2.9)" } packet -> prerouting -> services -> svc svc -> seps.sep1: 50% svc -> seps.sep2: 50% ``` **KUBE-SERVICES**: The entry point. Has rules for every Service, matching on ClusterIP:port. **KUBE-SVC-***: One chain per Service. Contains probability-based jumps to endpoint chains (this is how load balancing works). **KUBE-SEP-***: One chain per endpoint (pod). Performs the actual DNAT — rewriting destination from ClusterIP to pod IP. ### Reading Real iptables Rules Let's see what kube-proxy creates: ```bash # Dump all iptables rules iptables-save | grep my-service # Or more specifically, find the Service chain iptables -t nat -L KUBE-SERVICES -n | grep 10.96.45.67 ``` Example output (annotated): ```bash # Entry in KUBE-SERVICES for our Service -A KUBE-SERVICES -d 10.96.45.67/32 -p tcp -m tcp --dport 8080 \ -j KUBE-SVC-ABCD1234 # Jump to Service chain # The Service chain with load balancing -A KUBE-SVC-ABCD1234 -m statistic --mode random --probability 0.5 \ -j KUBE-SEP-ENDPOINT1 # 50% to first endpoint -A KUBE-SVC-ABCD1234 \ -j KUBE-SEP-ENDPOINT2 # Remaining 50% to second endpoint # Endpoint chains - the actual DNAT -A KUBE-SEP-ENDPOINT1 -p tcp \ -j DNAT --to-destination 10.244.2.8:8080 -A KUBE-SEP-ENDPOINT2 -p tcp \ -j DNAT --to-destination 10.244.2.9:8080 ``` **The probability math**: With two endpoints, the first rule matches 50%. The second rule catches everything else (also 50%). With three endpoints: 33%, 50% of remaining (33%), then the rest (33%). ### What DNAT Does Before DNAT: ``` src: 10.244.1.5 (Pod A) dst: 10.96.45.67:8080 (Service ClusterIP) ``` After DNAT: ``` src: 10.244.1.5 (Pod A) dst: 10.244.2.8:8080 (Pod B - actual endpoint) ``` The packet now has a real destination. It can be routed to Pod B. ### Connection Tracking (conntrack) One problem: Pod B's response will have: ``` src: 10.244.2.8:8080 (Pod B) dst: 10.244.1.5 (Pod A) ``` Pod A sent to 10.96.45.67 but receives from 10.244.2.8. Won't it be confused? **conntrack** saves us. The kernel tracks connections. When the response arrives, it reverses the DNAT: ``` Response packet: Before reverse DNAT: src=10.244.2.8, dst=10.244.1.5 After reverse DNAT: src=10.96.45.67, dst=10.244.1.5 ``` Pod A sees the response coming from the ClusterIP it originally contacted. The illusion holds. ```bash # See connection tracking entries conntrack -L | grep 10.96.45.67 ``` ## Part 3: Crossing Nodes (Overlay Networks) Our packet is now destined for 10.244.2.8 (Pod B on Node 2). But there's a problem. ### The Problem: Pod CIDRs Aren't Routable Pod IPs (10.244.x.x) are internal to Kubernetes. Your physical network doesn't know how to route them: ``` Node 1 (192.168.1.10) wants to send to 10.244.2.8 Physical router: "10.244.2.8? Never heard of it. Drop." ``` Node 1's routing table doesn't have a route to 10.244.2.0/24. Neither does your datacenter's router. ### Solution: Overlay Networks An overlay network encapsulates pod-to-pod packets inside node-to-node packets: ```d2 {alt="VXLAN encapsulation wraps the original pod-to-pod packet (src 10.244.1.5, dst 10.244.2.8, HTTP GET payload) inside a node-to-node packet made of an outer header (src 192.168.1.10, dst 192.168.1.11, UDP:8472), a VXLAN header with VNI 1, and the unchanged inner packet"} grid-columns: 1 vertical-gap: 40 original: "Original Packet\nsrc: 10.244.1.5 dst: 10.244.2.8 payload: HTTP GET" encapsulated: "" { grid-columns: 1 grid-gap: 0 style: {stroke-width: 0; fill: transparent} outer: "Outer Header\nsrc: 192.168.1.10 dst: 192.168.1.11 proto: UDP:8472" vxlan: "VXLAN Header (VNI: 1)" inner: "Inner Packet (original)\nsrc: 10.244.1.5 dst: 10.244.2.8 payload: HTTP GET" } original -> encapsulated: VXLAN Encapsulation ``` The physical network only sees the outer header: Node 1 sending UDP to Node 2. It routes normally. Node 2 receives, strips the outer header, and delivers the inner packet to Pod B. ### VXLAN Deep Dive VXLAN (Virtual Extensible LAN) is the most common overlay in Kubernetes (used by Flannel, Calico in VXLAN mode, etc.). **Key components on each node:** 1. **VXLAN interface (flannel.1, vxlan.calico)**: A virtual interface that handles encap/decap 2. **FDB (Forwarding Database)**: Maps pod IPs/MACs to node IPs 3. **Routes**: Direct pod CIDR traffic to the VXLAN interface ```bash # See the VXLAN interface ip -d link show flannel.1 # See the FDB entries (which node has which pods) bridge fdb show dev flannel.1 # See routes to other pod CIDRs ip route | grep 10.244 ``` Example routing table on Node 1: ``` 10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1 # Local pods 10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink # Node 2's pods 10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink # Node 3's pods ``` Traffic to 10.244.2.x goes through `flannel.1`, which encapsulates it. ### Packet Flow: Crossing Nodes Full journey for Pod A (10.244.1.5, Node 1) to Pod B (10.244.2.8, Node 2): ``` Node 1: 1. Pod A sends: src=10.244.1.5, dst=10.244.2.8 2. Packet exits pod via veth -> arrives at bridge (cni0) 3. Bridge checks: 10.244.2.8 not local 4. Packet routes to flannel.1 (VXLAN interface) 5. VXLAN encapsulates: - Outer: src=192.168.1.10, dst=192.168.1.11, UDP:8472 - Inner: original packet 6. Encapsulated packet sent on physical network (eth0) Physical Network: 7. Packet routed from 192.168.1.10 to 192.168.1.11 Node 2: 8. eth0 receives packet 9. Kernel sees UDP:8472 -> hands to VXLAN interface 10. flannel.1 decapsulates, extracts inner packet 11. Inner packet: src=10.244.1.5, dst=10.244.2.8 12. Routes to cni0 (bridge) 13. Bridge forwards to Pod B's veth 14. Pod B receives original packet ``` ### MTU Matters VXLAN adds ~50 bytes of overhead (outer IP + UDP + VXLAN header). If your physical MTU is 1500: ``` Physical MTU: 1500 VXLAN overhead: -50 Pod MTU: 1450 ``` If pods use MTU 1500, packets that need the full size will fail (too big after encapsulation). Symptoms: - Small requests work, large requests hang - SSH works, SCP fails - TCP connections stall **Check MTU configuration:** ```bash # On pod cat /sys/class/net/eth0/mtu # On node VXLAN interface cat /sys/class/net/flannel.1/mtu ``` Most CNIs set pod MTU correctly, but misconfigurations happen. ### Alternative: No Overlay (BGP) Overlay isn't the only option. Calico can run in BGP mode: - Each node advertises its pod CIDR to the network - Routers learn: "10.244.2.0/24 is behind 192.168.1.11" - No encapsulation needed — native routing Trade-offs: - **BGP**: No overhead, but requires network integration (not all environments support it) - **VXLAN**: Works anywhere, but has overhead Cloud providers often use their own routing (VPC routes in AWS/GCP) — no overlay, no BGP, just cloud magic. ## Part 4: The Full Journey Let's put it all together. Pod A curls `my-service:8080`: ```d2 {alt="Full cross-node journey: on Node 1 (192.168.1.10) the packet flows down from Pod A through iptables PREROUTING DNAT, the cni0 bridge, and VXLAN (flannel.1) encapsulation to eth0, crosses the physical network, then flows up Node 2 (192.168.1.11) from eth0 through VXLAN decapsulation and the cni0 bridge to Pod B (10.244.2.8), which processes the HTTP request"} grid-columns: 1 vertical-gap: 44 nodes: "" { style.stroke-width: 0 style.fill: transparent grid-rows: 1 horizontal-gap: 72 vertical-gap: 20 node1: NODE 1 (192.168.1.10) { grid-columns: 1 vertical-gap: 56 poda: "Pod A (10.244.1.5)" ipt: "iptables (PREROUTING)" bridge: "Bridge (cni0)" vxlan: "VXLAN (flannel.1)" eth0: "eth0 (192.168.1.10)" poda -> ipt: "(1) curl my-service:8080\nDNS resolves to 10.96.45.67\nPacket: src=10.244.1.5, dst=10.96.45.67:8080" ipt -> bridge: "(2) DNAT: dst 10.96.45.67 -> 10.244.2.8\nPacket: src=10.244.1.5, dst=10.244.2.8:8080" bridge -> vxlan: "(3) 10.244.2.8 not local -> route lookup" vxlan -> eth0: "(4) Encapsulate\nOuter: src=192.168.1.10, dst=192.168.1.11" } node2: NODE 2 (192.168.1.11) { grid-columns: 1 vertical-gap: 80 podb: "Pod B (10.244.2.8)\n(9) Receive packet,\nprocess HTTP request" bridge: "Bridge (cni0)" vxlan: "VXLAN (flannel.1)" eth0: "eth0 (192.168.1.11)" eth0 -> vxlan: "(6) Receive encapsulated packet" vxlan -> bridge: "(7) Decapsulate\nExtract: src=10.244.1.5, dst=10.244.2.8" bridge -> podb: "(8) Forward to Pod B's veth" } } phys: Physical Network {height: 44} nodes.node1.eth0 -> phys: "(5) Send to physical network" phys -> nodes.node2.eth0 ``` **The return path:** 1. Pod B responds: src=10.244.2.8, dst=10.244.1.5 2. VXLAN encapsulates, sends to Node 1 3. Node 1 decapsulates 4. conntrack matches the existing connection 5. Reverse DNAT: src becomes 10.96.45.67 (the ClusterIP) 6. Pod A receives response from "my-service" ## Part 5: Debugging the Nightmare Armed with this knowledge, debugging becomes systematic. ### "Pod Can't Reach Service" **Step 1: Can the pod reach anything?** ```bash # From inside the pod kubectl exec -it pod-a -- ping 8.8.8.8 kubectl exec -it pod-a -- ping 10.244.1.1 # Node's bridge IP ``` If this fails, the problem is basic connectivity (CNI, veth, bridge). **Step 2: Can the pod reach other pods on the same node?** ```bash kubectl exec -it pod-a -- ping ``` If this fails: bridge or veth issue. **Step 3: Can the pod reach pods on other nodes?** ```bash kubectl exec -it pod-a -- ping ``` If same-node works but cross-node fails: overlay problem. **Step 4: Can the pod reach the ClusterIP?** ```bash kubectl exec -it pod-a -- curl -v 10.96.45.67:8080 ``` If direct pod IP works but ClusterIP fails: kube-proxy/iptables problem. **Step 5: Is DNS working?** ```bash kubectl exec -it pod-a -- nslookup my-service kubectl exec -it pod-a -- cat /etc/resolv.conf ``` If IP works but name doesn't: DNS problem (CoreDNS, resolv.conf). ### Reading iptables Rules ```bash # SSH to the node, then: # Find rules for your Service iptables-save | grep iptables-save | grep # List the KUBE-SERVICES chain iptables -t nat -L KUBE-SERVICES -n --line-numbers # Follow a specific Service chain iptables -t nat -L KUBE-SVC-XXXXX -n ``` **What to look for:** - Is there a rule matching your ClusterIP? - Does the Service chain have endpoint rules? - Are the endpoint IPs correct? Missing rules? Check if kube-proxy is running: ```bash kubectl get pods -n kube-system -l k8s-app=kube-proxy kubectl logs -n kube-system -l k8s-app=kube-proxy ``` ### tcpdump at Each Hop Capture traffic to see where packets go (or stop): ```bash # On the pod (if tcpdump available) kubectl exec -it pod-a -- tcpdump -i eth0 -n host 10.244.2.8 # On node, at the bridge tcpdump -i cni0 -n host 10.244.1.5 # On node, at the VXLAN interface tcpdump -i flannel.1 -n host 10.244.2.8 # On node, at the physical interface (see encapsulated packets) tcpdump -i eth0 -n udp port 8472 # On destination node tcpdump -i eth0 -n udp port 8472 tcpdump -i flannel.1 -n host 10.244.1.5 ``` **Interpret what you see:** - Packets at cni0 but not flannel.1? Routing problem. - Packets at flannel.1 but not remote eth0? Physical network problem. - Packets at remote eth0 but not flannel.1? VXLAN decap problem. - Packets at remote cni0 but not pod? Bridge/veth problem. ### conntrack Inspection ```bash # See all tracked connections conntrack -L # Filter for your Service conntrack -L | grep 10.96.45.67 # Watch new connections conntrack -E ``` Stale conntrack entries can cause weird issues (traffic to old pod IPs). Flushing can help (but disrupts existing connections): ```bash conntrack -F ``` ### Common Failures **kube-proxy not running:** - Symptom: ClusterIP doesn't work, direct pod IP works - Check: `kubectl get pods -n kube-system -l k8s-app=kube-proxy` - Fix: Restart kube-proxy, check logs **CNI misconfiguration:** - Symptom: Pods can't communicate at all, or only on same node - Check: `kubectl get pods -n kube-system` for CNI pods (flannel, calico, etc.) - Check: `/etc/cni/net.d/` for CNI config - Fix: Reinstall CNI, check config **iptables rules missing:** - Symptom: ClusterIP doesn't work after Service creation - Check: `iptables-save | grep ` - Cause: kube-proxy error, RBAC issue - Fix: Check kube-proxy logs **MTU mismatch:** - Symptom: Small packets work, large fail; TCP stalls - Check: MTU on pod, bridge, VXLAN interface - Fix: Configure CNI with correct MTU **NetworkPolicy blocking:** - Symptom: Some pods can't connect, others can - Check: `kubectl get networkpolicy -A` - Fix: Add appropriate NetworkPolicy rules **Firewall blocking VXLAN:** - Symptom: Cross-node fails, same-node works - Check: `tcpdump -i eth0 udp port 8472` — packets sent but not received? - Fix: Open UDP 8472 (VXLAN) between nodes ### Quick Diagnostic Commands ```bash # Check all network interfaces ip addr # Check routes ip route # Check iptables NAT rules iptables -t nat -L -n -v # Check VXLAN FDB bridge fdb show dev flannel.1 # Check CNI config cat /etc/cni/net.d/* # Check kube-proxy mode kubectl logs -n kube-system -l k8s-app=kube-proxy | grep "Using" # Check endpoints for a Service kubectl get endpoints my-service ``` ## Summary The magic of Kubernetes networking is built on: 1. **veths and bridges**: Connect pods within a node 2. **iptables (kube-proxy)**: Implement Services via DNAT 3. **Overlay networks**: Carry pod traffic across nodes via encapsulation When debugging: - Trace the path systematically (pod -> bridge -> overlay -> remote node -> pod) - Use tcpdump at each hop to see where packets stop - Check iptables for Service issues - Check conntrack for connection state issues - Check MTU for "big packets fail" symptoms The magic is just routing, NAT, and encapsulation. Once you know the layers, you know where to look. --- ## From etcd to Watch: How Kubernetes Watches Actually Work - URL: https://svalle.ru/posts/kubernetes/etcd-to-watch/ - Date: 2025-01-30 - Tags: kubernetes, etcd, watches, internals, api-server, resourceversion Every Kubernetes controller depends on watches. Create a Deployment, and within seconds the ReplicaSet controller sees it and creates pods. But how does this actually work? The answer runs from etcd's data model through the API server's watch cache to your client's informer — and understanding this chain explains why watches sometimes fail and how to fix them. ## etcd's Data Model Kubernetes stores all cluster state in etcd, a distributed key-value store. But etcd isn't a simple key-value store — it uses Multi-Version Concurrency Control (MVCC). ### Revisions, Not Overwrites When you update a key in etcd, it doesn't overwrite the old value. It creates a new revision: ``` Revision 100: /registry/pods/default/nginx -> {pod spec v1} Revision 101: /registry/pods/default/nginx -> {pod spec v2} # Updated Revision 102: /registry/pods/default/redis -> {pod spec v1} # New pod Revision 103: /registry/pods/default/nginx -> tombstone # Deleted ``` Every write operation increments a global revision counter. This revision is monotonically increasing across the entire etcd cluster — not per key. **Key insight**: You can ask etcd "what changed after revision 100?" and get a consistent stream of all modifications. ### History and Compaction etcd keeps historical revisions, but not forever. **Compaction** removes old revisions to reclaim space: ``` Before compaction (keeping revisions 100-200): Revision 100: key1 -> value1 Revision 101: key1 -> value2 ... Revision 200: key1 -> value100 After compaction at revision 150: Revision 150: key1 -> value50 # Oldest available ... Revision 200: key1 -> value100 ``` After compaction, you cannot watch from revision 100 — that history is gone. This becomes important later. ### etcd Watches etcd natively supports watches: ```go // Watch all changes to keys with prefix "/registry/pods/" starting from revision 1000 watcher := client.Watch(context.Background(), "/registry/pods/", clientv3.WithPrefix(), clientv3.WithRev(1000)) for response := range watcher { for _, event := range response.Events { fmt.Printf("Type: %s, Key: %s, Revision: %d\n", event.Type, event.Kv.Key, event.Kv.ModRevision) } } ``` The watch returns a stream of events: PUT (create/update) and DELETE operations, each tagged with the revision when it occurred. ## The Problem: etcd Can't Handle Kubernetes Scale Here's the issue: a busy Kubernetes cluster might have thousands of watchers. - Every kubelet watches pods scheduled to its node - Every controller watches its relevant resources - Every client running `kubectl get pods -w` opens a watch - Service meshes, monitoring, logging — all watching A 5,000 node cluster easily has 10,000+ concurrent watches. etcd can handle this in theory, but: 1. **Memory**: Each watch consumes memory in etcd 2. **Fan-out**: A single pod update must be sent to potentially thousands of watchers 3. **Connection overhead**: Each watch is a gRPC stream Having every Kubernetes component directly watch etcd would kill it. ## The API Server Watch Cache The API server solves this with a **watch cache** — a layer between etcd and clients. ### Architecture ```d2 {alt="Watch cache architecture: etcd sends one watch per resource type into the API server's watch cache, and a Fan component (the broadcaster) fans events out to many client watches, from Client Watch 1 to Client Watch N"} direction: right etcd cache: Watch Cache fan: Fan client1: Client Watch 1 clientN: Client Watch N note: "Broadcaster fans out\nto many client watches" {shape: text} etcd -> cache: "One etcd watch\nper resource type" cache -> fan fan -> client1 fan -> clientN fan -- note ``` **How it works:** 1. The API server opens **one watch per resource type** to etcd (e.g., one watch for all pods) 2. Events flow into the watch cache, which stores recent events in memory 3. The **broadcaster** fans out events to all client watches 4. Clients watch the API server, not etcd directly This transforms the problem: - etcd handles a handful of watches (one per resource type) - The API server handles thousands of client watches - The watch cache absorbs the fan-out cost ### Inside the Watch Cache The watch cache (`k8s.io/apiserver/pkg/storage/cacher`) maintains: **A sliding window of recent events:** ```go type watchCache struct { // Ring buffer of recent events cache []*watchCacheEvent startIndex int endIndex int // All objects currently in the cache (latest version) store cache.Indexer // Current resource version resourceVersion uint64 } ``` **Event storage:** ```go type watchCacheEvent struct { Type watch.EventType // ADDED, MODIFIED, DELETED Object runtime.Object // The object ObjLabels labels.Set // For filtering ObjFields fields.Set // For filtering PrevObject runtime.Object // Previous version (for MODIFIED) ResourceVersion uint64 } ``` When a client starts a watch: 1. If they request a specific resourceVersion and it's in the cache window -> replay from that point 2. If the version is too old (not in window) -> return "410 Gone" 3. If they request resourceVersion="0" -> start from current state 4. Then stream new events as they arrive ### The Cacher The `Cacher` is the component that ties it together: ```go type Cacher struct { // Underlying storage (etcd) storage storage.Interface // The watch cache watchCache *watchCache // Broadcasts events to watchers watchers indexedWatchers // Handles reflector lifecycle reflector *cache.Reflector } ``` The reflector does List+Watch against etcd, feeding events into the watch cache. Client watches subscribe to the broadcaster. ## ResourceVersion Demystified Every Kubernetes object has a `metadata.resourceVersion` field: ```yaml apiVersion: v1 kind: Pod metadata: name: nginx resourceVersion: "12345678" ``` This value comes from etcd's revision system, but with a twist. ### What ResourceVersion Actually Is In most cases, resourceVersion equals the etcd modification revision of that object. When you update a pod and etcd records it at revision 12345678, the pod's resourceVersion becomes "12345678". But it's not always a direct mapping: - The API server may encode additional information - Different storage backends could use different schemes - It's intentionally opaque — don't parse or compare numerically **Treat it as an opaque string** that happens to increase over time. ### List and Watch ResourceVersion Semantics When you list or watch resources, resourceVersion has specific meanings: **`resourceVersion=""` (not specified):** - List: Return from API server cache (may be slightly stale) - Watch: Start from "now" (current resource version) **`resourceVersion="0"`:** - List: Return from API server cache (any version) - Watch: Start from "any" — API server chooses (usually current) **`resourceVersion="12345678"` (specific value):** - List: Return data at least as fresh as this version - Watch: Start streaming from this exact point **For controllers, the pattern is:** 1. List with resourceVersion="" -> get current objects + a resourceVersion 2. Watch with that resourceVersion -> see all changes since the list ```go list, err := client.CoreV1().Pods("").List(ctx, metav1.ListOptions{}) if err != nil { return err } // Watch starting from where list ended watch, err := client.CoreV1().Pods("").Watch(ctx, metav1.ListOptions{ ResourceVersion: list.ResourceVersion, }) ``` ### The "410 Gone" Error ``` HTTP 410 Gone the server does not allow this method on the requested resource ``` This happens when you request a resourceVersion that's too old — it's been compacted from the watch cache. **The timeline:** ``` Watch cache window: [revision 1000 ... revision 2000] Client requests: Watch from revision 500 API server: "I don't have revision 500 anymore" -> 410 Gone ``` **Causes:** - Client disconnected too long, missed too many events - Watch cache is small relative to event rate - etcd compacted before client reconnected **The proper response:** 1. Catch the 410 error 2. Re-list to get current state and new resourceVersion 3. Resume watching from the new resourceVersion client-go's informers handle this automatically. ### Bookmarks Watches can go quiet — if nothing changes, no events flow. But the client needs to know: "Am I still connected? What's the current resourceVersion?" **Bookmarks** solve this. They're synthetic events that communicate the current resourceVersion without an actual object change: ```go // Enable bookmarks watch, err := client.Watch(ctx, metav1.ListOptions{ ResourceVersion: "12345", AllowWatchBookmarks: true, // Request bookmarks }) for event := range watch.ResultChan() { if event.Type == watch.Bookmark { // No object change, but we know we're caught up to this version fmt.Printf("Bookmark at %s\n", event.Object.GetResourceVersion()) } } ``` **Why bookmarks matter:** 1. **Progress tracking**: Client knows how far behind it is 2. **Faster recovery**: After disconnect, client can resume from bookmark's resourceVersion instead of re-listing 3. **Preventing 410**: Regular bookmarks keep the client's resourceVersion fresh The API server sends bookmarks periodically (default: every minute if there's no activity). ## Where It Breaks Understanding the architecture reveals the failure modes. ### Watch Cache Memory Pressure The watch cache has finite size. Under heavy write load: ``` Events arriving: 1000/second Cache size: 1000 events Cache window: 1 second of history ``` A client that disconnects for 2 seconds and tries to resume will get 410 Gone. **Symptoms:** - Controllers constantly re-listing - High API server memory usage - `apiserver_watch_cache_capacity_increase_total` metric increasing **Mitigation:** - Increase watch cache size (API server flag) - Use bookmarks (keeps client resourceVersion fresh) - Reduce event rate (fewer unnecessary updates) ### etcd Compaction vs Slow Clients etcd compacts old revisions. If a client is watching etcd directly (rare, but some systems do), compaction can remove history the client needs. For Kubernetes, this manifests indirectly — the watch cache's underlying reflector gets 410 from etcd, triggering a full re-list. **Symptoms:** - Periodic spikes in API server memory and etcd load - `apiserver_watch_cache_list_total` metric spikes **Mitigation:** - Tune etcd compaction interval - Ensure watch cache is sized for your workload ### Watch Storms After API Server Restart When the API server restarts: 1. All client watches disconnect 2. API server starts fresh — empty watch cache 3. All clients reconnect and re-list 4. API server hammers etcd with list requests A 1000-node cluster with 10 controllers each -> 10,000 simultaneous list requests. **Symptoms:** - API server slow immediately after restart - etcd latency spikes - Controllers report sync failures **Mitigation:** - API server caching settings (`--watch-cache-sizes`) - Client backoff (client-go does this automatically with jitter) - Priority and fairness (APF) to protect against thundering herd ### The "Too Old Resource Version" Problem Informers cache the resourceVersion of the last event. If an informer is too slow processing events: ``` API server at revision: 10000 Informer last saw: 8000 Informer's local cache at: 8000 API server watch cache: [9000 ... 10000] (only 1000 events) Informer tries to resume from 8000 -> 410 Gone ``` **Symptoms:** - Informer re-lists repeatedly - Controller appears to "miss" events - High memory churn (re-list allocates new objects) **Mitigation:** - Speed up event handlers (don't block the informer) - Increase watch cache size - Check for slow reconcilers ## Tuning and Debugging ### API Server Watch Cache Flags ``` --watch-cache=true # Enable watch cache (default: true) --watch-cache-sizes=pods#1000 # Per-resource cache sizes --default-watch-cache-size=100 # Default size for resources not specified ``` Format for `--watch-cache-sizes`: ``` resource#size,resource#size,... Examples: pods#5000,secrets#1000,configmaps#1000 ``` Larger cache = more memory, but fewer 410 errors. ### etcd Settings ```yaml # etcd configuration auto-compaction-mode: periodic auto-compaction-retention: "1h" # Keep 1 hour of history ``` Longer retention = more history available = fewer compaction-related 410s. But also more disk and memory usage. ### Metrics to Watch **API server:** ```promql # Watch cache size by resource apiserver_watch_cache_capacity{resource="pods"} # Events in watch cache apiserver_watch_cache_events_received_total # 410 errors (client needed older data than available) apiserver_watch_cache_stale_total # Watch count by resource apiserver_registered_watchers{resource="pods"} ``` **etcd:** ```promql # Current revision etcd_debugging_mvcc_db_compaction_last # Compaction stats etcd_debugging_mvcc_db_compaction_total_duration_seconds # Watch count etcd_debugging_mvcc_watcher_total ``` ### Diagnosing Watch Disconnects **Client-side symptoms:** - Informer logs show "watch ended" followed by re-list - ResourceVersion jumps (indicates re-list happened) - Events appear "missed" **Check API server logs:** ```bash kubectl logs -n kube-system kube-apiserver- | grep -i "watch" ``` **Check etcd health:** ```bash etcdctl endpoint health etcdctl endpoint status ``` **Trace a specific watch:** Enable verbose logging in client-go: ```go import "k8s.io/klog/v2" klog.SetOutput(os.Stderr) klog.InitFlags(nil) flag.Set("v", "6") // Verbose watch logging ``` ### Testing Watch Behavior Simulate watch cache pressure: ```go func stressTest(client kubernetes.Interface) { // Create and delete pods rapidly for i := 0; i < 10000; i++ { pod := &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: fmt.Sprintf("stress-%d", i), }, Spec: corev1.PodSpec{ Containers: []corev1.Container{{ Name: "test", Image: "nginx", }}, }, } client.CoreV1().Pods("default").Create(ctx, pod, metav1.CreateOptions{}) client.CoreV1().Pods("default").Delete(ctx, pod.Name, metav1.DeleteOptions{}) } } ``` Monitor `apiserver_watch_cache_stale_total` during the test. ## The Full Picture Putting it together: ```d2 {alt="The full watch chain: your controller's SharedInformer (Reflector doing List+Watch, into DeltaFIFO, into the Indexer cache, into the Event Handler) holds a long-lived HTTP watch to the API server, whose Cacher (Reflector doing an etcd watch, into the Watch Cache, into the fan-out Broadcaster) holds one gRPC watch per resource type to etcd, which stores MVCC revisions 1000, 1001, 1002, 1003, and so on"} grid-rows: 3 vertical-gap: 48 # gaps calibrated so all three rows have equal natural width (~644) — keeps # inner boxes centered and flush; recalibrate if any label changes controller: Your Controller { grid-rows: 1 grid-gap: 24 informer: SharedInformer { grid-rows: 1 grid-gap: 24 reflector: "Reflector\n(List+Watch)" fifo: DeltaFIFO indexer: "Indexer\n(Cache)" handler: Event Handler reflector -> fifo -> indexer -> handler } } apiserver: API Server { grid-rows: 1 grid-gap: 24 cacher: Cacher { grid-rows: 1 horizontal-gap: 36 vertical-gap: 24 reflector: "Reflector\n(etcd watch)" cache: Watch Cache broadcaster: "Broadcaster (fan-out)" reflector -> cache -> broadcaster } } etcd: etcd { grid-rows: 1 horizontal-gap: 128 vertical-gap: 24 mvcc: "MVCC: Rev 1000 | Rev 1001 | Rev 1002 | Rev 1003 | ..." } controller -> apiserver: "HTTP Watch (long-lived connection)" apiserver -> etcd: "gRPC Watch (one per resource type)" ``` 1. **etcd** stores versioned data using MVCC, supports native watches 2. **API server watch cache** multiplexes one etcd watch to many clients 3. **Client informers** maintain local caches, synced via watches 4. **ResourceVersion** ties it all together — an opaque token representing a point in time When something goes wrong: - 410 errors -> client's resourceVersion is too old, needs re-list - Missing events -> look for watch disconnects, slow handlers - High latency -> check watch cache size, etcd health, event rate Understanding this chain helps you build robust controllers and diagnose issues that mystify most operators. --- ## Admission Webhooks at Scale: Diagnosis, Hardening, and Multi-Cluster Consistency - URL: https://svalle.ru/posts/kubernetes/admission-webhooks-at-scale/ - Date: 2025-01-29 - Tags: kubernetes, webhooks, security, multi-cluster, performance, admission-control Your cluster has 12 admission webhooks. Pod creation takes 4 seconds. Sometimes it times out. Nobody knows which webhook is the problem, or even what all these webhooks do. Welcome to webhook sprawl. This post covers how to diagnose webhook problems, harden configurations, and maintain consistency across a multi-cluster fleet. ## The Webhook Accumulation Problem Every tool wants an admission webhook: - **Policy engines**: Kyverno, OPA/Gatekeeper, Kubewarden - **Service meshes**: Istio, Linkerd (sidecar injection) - **Security**: Vault (secret injection), Falco, image scanners - **Certificates**: cert-manager - **Platform tooling**: Custom mutating webhooks for labels, resource defaults, etc. Each webhook seems reasonable in isolation. But they accumulate: ``` $ kubectl get mutatingwebhookconfigurations NAME WEBHOOKS AGE cert-manager-webhook 1 180d istio-sidecar-injector 1 90d kyverno-resource-mutating 1 60d vault-agent-injector 1 45d team-a-defaults 1 30d team-b-image-rewriter 1 14d ``` ``` $ kubectl get validatingwebhookconfigurations NAME WEBHOOKS AGE cert-manager-webhook 1 180d gatekeeper-validating-webhook 1 120d kyverno-resource-validating 1 60d team-c-compliance-checker 1 21d ``` **Symptoms you'll see:** - Pod creation latency measured in seconds - Intermittent API timeouts - "Connection refused" errors when webhooks are overwhelmed - Mysterious admission rejections ("admission webhook denied the request" — but which one?) - 3am pages when a webhook goes down ## How Admission Webhooks Actually Work Understanding the mechanics helps diagnose problems. ### The Admission Chain When you create a resource, the API server processes it through a chain: ```d2 {alt="Admission chain: a client request passes through Authentication, Authorization, the Mutating Admission Webhooks stage (Webhook 1, 2, and 3 invoked serially in order), Object Schema Validation, and the Validating Admission Webhooks stage (Webhook A, B, and C invoked in parallel) before being persisted to etcd"} grid-columns: 1 vertical-gap: 40 client: Client Request {height: 48} authn: Authentication {height: 48} authz: Authorization {height: 48} mutating: "Mutating Admission Webhooks (in order)" { grid-rows: 1 grid-gap: 24 w1: Webhook 1 {height: 48} w2: Webhook 2 {height: 48} w3: Webhook 3 {height: 48} w1 -> w2 -> w3 } schema: Object Schema Validation {height: 48} validating: "Validating Admission Webhooks (parallel)" { grid-rows: 1 grid-gap: 24 wa: Webhook A {height: 48} wb: Webhook B {height: 48} wc: Webhook C {height: 48} } persist: Persist to etcd {height: 48} client -> authn -> authz -> mutating -> schema -> validating -> persist ``` **Key points:** 1. **Mutating webhooks run serially**, in the order defined by their configuration. Each one can modify the object before passing to the next. 2. **Validating webhooks run in parallel** (mostly). They can only accept or reject — no modifications. 3. If any webhook rejects, the entire request fails. 4. If any webhook times out or errors, behavior depends on `failurePolicy`. ### The Reinvocation Trap Here's a subtle issue: after mutating webhooks run, if the object was modified, validating webhooks see the mutated version. But there's more. If a mutating webhook modifies the object, the API server may **reinvoke** earlier mutating webhooks to ensure they see the final state. This can cause: - Unexpected latency (webhooks called multiple times) - Ordering surprises (Webhook A runs, then B mutates, then A runs again) - Infinite loops (A mutates, triggers B, B mutates, triggers A...) The `reinvocationPolicy` field controls this: ```yaml webhooks: - name: my-webhook.example.com reinvocationPolicy: IfNeeded # Default - may reinvoke # or reinvocationPolicy: Never # Don't reinvoke this webhook ``` ### Timeout Behavior Each webhook has a timeout. The default is **10 seconds** (was 30 seconds in older Kubernetes). ```yaml webhooks: - name: my-webhook.example.com timeoutSeconds: 5 # Fail fast ``` If a webhook doesn't respond in time: - `failurePolicy: Fail` → Request rejected - `failurePolicy: Ignore` → Webhook skipped, request continues With 10 webhooks at 10 seconds each, worst case is **100 seconds** before timeout. In practice, the API server has its own overall timeout (~60s default), so you'll hit that first. ## Diagnosing Webhook Problems ### API Server Metrics The API server exposes detailed webhook metrics. These are your primary diagnostic tool. **Webhook latency:** ```promql # P99 latency per webhook histogram_quantile(0.99, sum(rate(apiserver_admission_webhook_admission_duration_seconds_bucket[5m])) by (le, name, operation) ) ``` **Webhook rejection rate:** ```promql # Rejections per webhook sum(rate(apiserver_admission_webhook_rejection_count[5m])) by (name, error_type) ``` **Webhook failure rate (timeouts, connection errors):** ```promql sum(rate(apiserver_admission_webhook_fail_open_count[5m])) by (name) ``` Create a dashboard with: - Latency heatmap per webhook - Rejection rate over time - Failure/timeout rate - Request volume per webhook ### Identifying Slow Webhooks High P99 latency on a specific webhook? Dig deeper: ```bash # Check webhook endpoint health kubectl get mutatingwebhookconfiguration -o jsonpath='{.webhooks[*].clientConfig.service}' # Check the backing service kubectl get pods -n -l app= kubectl logs -n -l app= --tail=100 ``` Common causes of slow webhooks: - Webhook does external calls (API, database) synchronously - Webhook has insufficient resources (CPU throttling) - Webhook is overloaded (not enough replicas) - Network latency to webhook service ### "Which Webhook Rejected My Pod?" The API server error message is often unhelpful: ``` Error from server: admission webhook "webhook.example.com" denied the request: [error details] ``` If it doesn't say which webhook, or the error is generic: **Step 1: Check recent events** ```bash kubectl get events --field-selector reason=FailedCreate --sort-by='.lastTimestamp' ``` **Step 2: Enable API server audit logging** Audit logs capture which webhooks were called and their responses: ```yaml # Audit policy to log admission decisions apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: RequestResponse resources: - group: "" resources: ["pods"] omitStages: - RequestReceived ``` **Step 3: Dry-run the request** Kubernetes 1.18+ supports server-side dry-run: ```bash kubectl apply -f pod.yaml --dry-run=server -v=8 ``` The verbose output shows which webhooks were called. **Step 4: Binary search** If you're desperate, temporarily disable webhooks one by one to find the culprit: ```bash # Add a namespaceSelector that matches nothing kubectl patch mutatingwebhookconfiguration suspect-webhook --type='json' -p='[{"op": "add", "path": "/webhooks/0/namespaceSelector", "value": {"matchLabels": {"nonexistent": "label"}}}]' ``` (Don't do this in production without understanding the consequences.) ### Tracing a Request For deep debugging, trace a single request through the webhook chain. If you have distributed tracing (Jaeger, Zipkin), ensure your webhooks propagate trace headers. The API server doesn't initiate traces, but your webhooks can create spans. Quick tracing with curl: ```bash # Get API server address and token API_SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}') TOKEN=$(kubectl create token default) # Create pod with timing time curl -X POST \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d @pod.json \ "$API_SERVER/api/v1/namespaces/default/pods?dryRun=All" \ -w "\nTotal time: %{time_total}s\n" ``` ## Hardening Webhook Configurations ### Timeout: Don't Default to 10 Seconds 10 seconds is an eternity for an admission decision. If your webhook needs 10 seconds, something is wrong. ```yaml webhooks: - name: fast-webhook.example.com timeoutSeconds: 3 # Be aggressive ``` **Guidelines:** - Simple validation: 1-2 seconds - Mutation with no external calls: 2-3 seconds - External calls (policy checks, etc.): 3-5 seconds max - More than 5 seconds: Reconsider your architecture ### failurePolicy: The Tradeoff ```yaml webhooks: - name: my-webhook.example.com failurePolicy: Fail # Reject if webhook fails # or failurePolicy: Ignore # Skip webhook if it fails ``` **`Fail` (default):** - Webhook down → API requests rejected - Safer for security-critical webhooks - Risk: Webhook failure blocks the entire cluster **`Ignore`:** - Webhook down → Requests proceed without webhook - Better for availability - Risk: Security policies bypassed during outages **Recommendation:** - Security-critical (policy enforcement): `Fail`, but ensure high availability - Nice-to-have mutations (adding labels): `Ignore` - Development/testing webhooks: `Ignore` ### Scope Your Webhooks Don't intercept everything: ```yaml webhooks: - name: my-webhook.example.com # Only match specific namespaces namespaceSelector: matchExpressions: - key: webhook.example.com/enabled operator: In values: ["true"] # Only match specific resources rules: - apiGroups: ["apps"] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["deployments"] scope: Namespaced # Only match objects with specific labels objectSelector: matchLabels: webhook.example.com/process: "true" ``` **Always exclude system namespaces:** ```yaml namespaceSelector: matchExpressions: - key: kubernetes.io/metadata.name operator: NotIn values: - kube-system - kube-public - kube-node-lease ``` **Filter by operation:** If you only care about CREATE, don't intercept UPDATE: ```yaml rules: - operations: ["CREATE"] # Not ["CREATE", "UPDATE", "DELETE"] resources: ["pods"] ``` ### matchPolicy: Exact vs Equivalent ```yaml webhooks: - name: my-webhook.example.com matchPolicy: Equivalent # Default # or matchPolicy: Exact ``` `Equivalent` matches resources by type equivalence (e.g., `apps/v1` Deployment and `apps/v1beta1` Deployment are equivalent). This is usually what you want. `Exact` requires exact API group/version match. Use this if your webhook logic is version-specific. ### sideEffects Declaration Webhooks must declare their side effects: ```yaml webhooks: - name: my-webhook.example.com sideEffects: None # No side effects, safe for dry-run # or sideEffects: NoneOnDryRun # Side effects only on real requests ``` `None` allows the API server to skip your webhook during dry-run requests, reducing unnecessary calls. ## Multi-Cluster Webhook Consistency ### Discovering What Exists First problem: knowing what webhooks exist across your fleet. **Quick audit script:** ```bash #!/bin/bash for cluster in $(kubectl config get-contexts -o name); do echo "=== $cluster ===" kubectl --context=$cluster get mutatingwebhookconfigurations -o custom-columns=NAME:.metadata.name,WEBHOOKS:.webhooks[*].name kubectl --context=$cluster get validatingwebhookconfigurations -o custom-columns=NAME:.metadata.name,WEBHOOKS:.webhooks[*].name echo done ``` **Structured collection:** ```bash # Export webhook configs from all clusters for cluster in $(kubectl config get-contexts -o name); do kubectl --context=$cluster get mutatingwebhookconfigurations -o yaml > "webhooks-mutating-$cluster.yaml" kubectl --context=$cluster get validatingwebhookconfigurations -o yaml > "webhooks-validating-$cluster.yaml" done # Diff them diff webhooks-mutating-cluster1.yaml webhooks-mutating-cluster2.yaml ``` ### Detecting Drift Webhook drift happens when: - Someone manually adds a webhook to one cluster - A Helm upgrade fails on some clusters - Different teams deploy different versions **Automated drift detection:** ```yaml # On your hub cluster, define expected webhooks apiVersion: v1 kind: ConfigMap metadata: name: expected-webhooks namespace: fleet-system data: mutating: | cert-manager-webhook istio-sidecar-injector kyverno-resource-mutating validating: | cert-manager-webhook kyverno-resource-validating ``` Then run periodic jobs that compare actual vs expected. ### Propagating Webhooks via Fleet Webhook configurations are cluster-scoped resources. Propagate them like any other: ```yaml apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata: name: webhook-configs spec: resourceSelectors: - group: admissionregistration.k8s.io kind: MutatingWebhookConfiguration version: v1 name: my-webhook - group: admissionregistration.k8s.io kind: ValidatingWebhookConfiguration version: v1 name: my-webhook policy: placementType: PickAll ``` **Caution**: The webhook configuration references a Service (the webhook endpoint). That Service must exist in every cluster. Options: 1. **Webhook runs in every cluster**: Service is local. Propagate both the webhook workload and the configuration. 2. **Centralized webhook**: All clusters call a central endpoint. Use `url` instead of `service` in the webhook config. (Not recommended for latency-sensitive or high-volume webhooks.) ### Canary Rollouts for Webhook Changes Webhook changes are risky. A bad config can break the entire cluster. **Staged rollout:** ```yaml # Stage 1: Canary cluster only apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata: name: webhook-canary spec: resourceSelectors: - group: admissionregistration.k8s.io kind: MutatingWebhookConfiguration version: v1 name: new-webhook-v2 policy: placementType: PickN numberOfClusters: 1 affinity: clusterAffinity: requiredDuringSchedulingIgnoredDuringExecution: clusterSelectorTerms: - labelSelector: matchLabels: environment: canary ``` Monitor the canary cluster. Check metrics. If healthy, expand: ```yaml # Stage 2: All non-prod policy: placementType: PickAll affinity: clusterAffinity: requiredDuringSchedulingIgnoredDuringExecution: clusterSelectorTerms: - labelSelector: matchExpressions: - key: environment operator: NotIn values: ["production"] ``` Then production. ## Consolidation: Fewer Webhooks, Less Pain The best webhook is the one you don't have. ### Kyverno/OPA Can Replace Many Single-Purpose Webhooks Instead of: - Webhook A: Require labels - Webhook B: Enforce resource limits - Webhook C: Disallow privileged pods - Webhook D: Restrict registries Use one policy engine: ```yaml # One Kyverno installation replaces 4 webhooks apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: all-the-things spec: rules: - name: require-labels # ... - name: require-limits # ... - name: disallow-privileged # ... - name: restrict-registries # ... ``` **Benefits:** - One webhook call instead of four - Consistent policy language - Unified reporting (PolicyReports) - One thing to monitor and maintain ### When to Keep Webhooks Separate - **Different SLAs**: A security webhook needs `failurePolicy: Fail`. A convenience webhook can use `Ignore`. - **Different ownership**: Istio sidecar injection is owned by the platform team. Team-specific mutations should be separate. - **Different lifecycles**: cert-manager upgrades shouldn't require redeploying your custom policies. ### Evaluating Webhook Necessity For each webhook, ask: - What problem does this solve? - Can it be solved another way (controller, policy engine)? - What's the latency impact? - What happens if it fails? - Who owns it? **Kill zombies**: Webhooks installed years ago for a use case nobody remembers. If it doesn't have an owner, it shouldn't exist. ## Summary Webhook sprawl is a real problem at scale. The fixes: 1. **Measure**: Use API server metrics to understand latency and failure rates per webhook. 2. **Debug systematically**: Know how to trace a request and identify which webhook is the problem. 3. **Harden configurations**: Aggressive timeouts, appropriate failurePolicy, scoped selectors. 4. **Maintain consistency**: Propagate configurations via Fleet, detect drift, canary changes. 5. **Consolidate**: Fewer webhooks doing more beats many webhooks doing little. Every webhook is a tax on every API request. Make sure each one is paying its way. --- ## Kyverno at Scale: Multi-Cluster Policy Without the Pain - URL: https://svalle.ru/posts/kubernetes/kyverno-multi-cluster/ - Date: 2025-01-28 - Tags: kubernetes, kyverno, security, multi-cluster, policy, admission-control You've got 20 clusters. You want consistent security policies across all of them. You roll out a Kyverno policy, and suddenly deployments are failing in production because someone's legitimate workload doesn't comply. Now multiply that panic by 20 clusters. This post is about avoiding that scenario — deploying Kyverno policies across a multi-cluster fleet without breaking things. ## The Problem In a single cluster, policy management is straightforward: install Kyverno, write policies, done. But multi-cluster introduces real challenges: **Policy drift**: Cluster-7 is running an old version of your policies. Cluster-12 has a policy someone added manually. Cluster-3 has an exception you forgot about. Nobody knows the actual state. **Blast radius**: A bad policy update doesn't break one cluster — it breaks all of them. Simultaneously. During business hours. **Exceptions at scale**: Team A needs an exception in their namespace. Team B needs a different exception, but only in the staging cluster. How do you manage this without drowning in YAML? **Visibility**: "Is this policy actually enforced everywhere?" shouldn't require SSH'ing into 20 clusters. ## Kyverno Essentials If you're new to Kyverno, here's the minimum you need to follow this post. Kyverno is a Kubernetes-native policy engine. It runs as an admission controller — when someone creates or updates a resource, Kyverno intercepts the request and decides whether to allow, modify, or reject it. **ClusterPolicy**: Applies across all namespaces in a cluster. ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-labels spec: validationFailureAction: Enforce # or Audit rules: - name: check-team-label match: any: - resources: kinds: - Pod validate: message: "The label 'team' is required." pattern: metadata: labels: team: "?*" ``` **validationFailureAction**: - `Enforce`: Block non-compliant resources (admission rejected) - `Audit`: Allow but report the violation (PolicyReport created) **Policy types**: - **Validate**: Accept or reject based on rules - **Mutate**: Modify resources on the fly (inject labels, add sidecars) - **Generate**: Create companion resources (NetworkPolicy when namespace created) - **VerifyImages**: Check image signatures That's enough background. Let's talk about not breaking production. ## Policies That Don't Break Production ### Start in Audit Mode. Always. The single most important practice: ```yaml spec: validationFailureAction: Audit # Start here ``` In Audit mode, Kyverno evaluates resources against your policy but doesn't block anything. It creates PolicyReports documenting what *would* have been blocked. **The workflow**: 1. Deploy policy in Audit mode 2. Wait (days, not hours — you need to see real traffic) 3. Review PolicyReports for violations 4. Fix legitimate workloads or adjust the policy 5. Switch to Enforce ```yaml spec: validationFailureAction: Enforce # Only after audit ``` Skipping this step is how you break production at 2am. ### Exclude System Namespaces Your policy shouldn't block Kubernetes system components: ```yaml spec: rules: - name: require-resource-limits match: any: - resources: kinds: - Pod exclude: any: - resources: namespaces: - kube-system - kube-public - kube-node-lease - kyverno - fleet-system - gatekeeper-system ``` Better yet, use a label-based exclusion that's consistent across policies: ```yaml exclude: any: - resources: selector: matchLabels: policy.example.com/exclude: "true" ``` ### PolicyExceptions: The Escape Hatch Kyverno 1.9+ introduced `PolicyException` — a way to grant specific exemptions without modifying the policy itself. ```yaml apiVersion: kyverno.io/v2beta1 kind: PolicyException metadata: name: allow-privileged-monitoring namespace: monitoring spec: exceptions: - policyName: disallow-privileged ruleNames: - deny-privileged-containers match: any: - resources: kinds: - Pod namespaces: - monitoring names: - node-exporter-* ``` This says: "The `disallow-privileged` policy doesn't apply to pods named `node-exporter-*` in the `monitoring` namespace." **Why this is better than policy modification**: - Policies stay clean and universal - Exceptions are explicit and auditable - You can track who requested what exception and why - Deleting the exception re-enables enforcement ### Staged Rollout Don't go from zero to fleet-wide enforcement in one step: **Stage 1: Single namespace in one cluster** ```yaml spec: rules: - name: test-policy match: any: - resources: namespaces: - policy-test ``` **Stage 2: Audit mode, all namespaces, one cluster** ```yaml spec: validationFailureAction: Audit rules: - name: test-policy match: any: - resources: kinds: - Pod exclude: # ... system namespaces ``` **Stage 3: Enforce mode, one cluster** **Stage 4: Audit mode, fleet-wide** **Stage 5: Enforce mode, fleet-wide** At each stage, wait and watch. PolicyReports tell you what's happening. ### The "Oh Shit" Recovery You deployed a bad policy. Deployments are failing. Here's your emergency playbook: **Option 1: Switch to Audit (fast)** ```bash kubectl patch clusterpolicy bad-policy -p '{"spec":{"validationFailureAction":"Audit"}}' --type=merge ``` **Option 2: Delete the policy (faster)** ```bash kubectl delete clusterpolicy bad-policy ``` **Option 3: Kyverno failurePolicy (prevents this scenario)** When you install Kyverno, configure the webhook to fail open: ```yaml # In Kyverno Helm values config: webhooks: - failurePolicy: Ignore # Fail open if Kyverno is down/slow ``` With `Ignore`, if Kyverno can't evaluate a request (timeout, crash), Kubernetes allows the request. You lose enforcement temporarily, but deployments don't break because your policy engine is having a bad day. ## Multi-Cluster Propagation Now for the multi-cluster part. You have policies that work. How do you deploy them consistently across a fleet? ### Propagating Policies via KubeFleet If you're using KubeFleet (or Azure Fleet Manager), you can treat Kyverno ClusterPolicies like any other Kubernetes resource: **Step 1: Create policies on the hub cluster** ```yaml # On hub cluster apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-resource-limits namespace: default # ClusterPolicy is cluster-scoped, but we need it in a namespace for Fleet spec: validationFailureAction: Audit rules: - name: require-limits match: any: - resources: kinds: - Pod exclude: any: - resources: namespaces: - kube-system - kyverno validate: message: "CPU and memory limits are required." pattern: spec: containers: - resources: limits: memory: "?*" cpu: "?*" ``` **Step 2: Create a ClusterResourcePlacement to propagate it** ```yaml apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata: name: kyverno-policies spec: resourceSelectors: - group: kyverno.io kind: ClusterPolicy version: v1 name: require-resource-limits policy: placementType: PickAll # All member clusters ``` Every member cluster now receives the policy. Updates to the policy on the hub automatically propagate. ### Wrapping Policies in a Namespace ClusterPolicies are cluster-scoped, but Fleet propagates namespace-scoped resources more naturally. A common pattern is to wrap policies in a dedicated namespace: ```yaml # Namespace to hold policies apiVersion: v1 kind: Namespace metadata: name: cluster-policies --- # Policy lives in this namespace (Fleet will propagate both) apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: disallow-privileged # Note: ClusterPolicy is cluster-scoped, but can be "associated" with a namespace for organizational purposes spec: # ... ``` Then propagate the entire namespace: ```yaml apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata: name: security-policies spec: resourceSelectors: - group: "" kind: Namespace version: v1 name: cluster-policies policy: placementType: PickAll ``` ### Per-Cluster Overrides What if most clusters need `Enforce` but your development cluster should use `Audit`? Use KubeFleet's `ClusterResourceOverride`: ```yaml apiVersion: placement.kubernetes-fleet.io/v1alpha1 kind: ClusterResourceOverride metadata: name: dev-cluster-audit-mode spec: clusterResourceSelectors: - group: kyverno.io kind: ClusterPolicy name: require-resource-limits version: v1 policy: overrideRules: - clusterSelector: clusterSelectorTerms: - labelSelector: matchLabels: environment: development jsonPatchOverrides: - op: replace path: /spec/validationFailureAction value: Audit ``` Now `require-resource-limits` is enforced everywhere except clusters labeled `environment: development`, where it runs in Audit mode. ### PolicyExceptions Across the Fleet Should PolicyExceptions be centralized or per-cluster? **Centralized exceptions** (propagated from hub): - Good for: Fleet-wide exceptions (monitoring tools, platform components) - Propagate like any other resource **Local exceptions** (created on member clusters): - Good for: Cluster-specific needs, team autonomy - Don't propagate — each cluster manages its own A reasonable split: - Platform exceptions (node-exporter, ingress controller) → centralized - Application exceptions → local, with approval process ### Versioning and Rollback When you update a policy on the hub, it propagates everywhere. How do you handle this safely? **GitOps**: Store policies in Git. Changes go through PR review. Fleet syncs from Git (via ArgoCD or Flux on the hub). **Staged rollout with labels**: ```yaml apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata: name: policies-canary spec: resourceSelectors: - group: kyverno.io kind: ClusterPolicy version: v1 name: new-policy-v2 policy: placementType: PickN numberOfClusters: 2 affinity: clusterAffinity: requiredDuringSchedulingIgnoredDuringExecution: clusterSelectorTerms: - labelSelector: matchLabels: policy-canary: "true" ``` Deploy to canary clusters first. Watch for issues. Then expand to the fleet. **Rollback**: Update the hub policy to the previous version. Fleet propagates the rollback. ## Observability ### PolicyReports Kyverno creates PolicyReport resources documenting violations: ```yaml apiVersion: wgpolicyk8s.io/v1alpha2 kind: ClusterPolicyReport metadata: name: clusterpolicy-require-resource-limits results: - message: "CPU and memory limits are required." policy: require-resource-limits result: fail rule: require-limits resources: - apiVersion: v1 kind: Pod name: my-app-xyz123 namespace: default timestamp: "2025-01-28T10:30:00Z" ``` ### Aggregating Reports Across Clusters In a multi-cluster setup, you need to aggregate reports. Options: **Option 1: Policy Reporter UI** [Policy Reporter](https://kyverno.github.io/policy-reporter/) is an open-source tool that aggregates PolicyReports and provides a dashboard. Deploy it on each cluster and point to a central backend. **Option 2: Export to central logging** Ship PolicyReports to your logging stack (Elasticsearch, Loki, etc.): ```yaml # Policy Reporter can push to various targets target: loki: host: http://loki.monitoring:3100 path: /loki/api/v1/push minimumPriority: warning ``` **Option 3: Metrics** Kyverno exposes Prometheus metrics: ``` kyverno_policy_results_total{policy_name="require-resource-limits", rule_name="require-limits", result="fail"} ``` Aggregate with Thanos or Cortex across clusters. Alert on violation spikes. ### Debugging Admission Failures User: "My deployment won't create pods!" **Step 1: Check events** ```bash kubectl describe deployment my-app # Look for admission webhook errors in events ``` **Step 2: Check Kyverno logs** ```bash kubectl logs -n kyverno -l app.kubernetes.io/name=kyverno --tail=100 | grep my-app ``` **Step 3: Check PolicyReports** ```bash kubectl get policyreport -A | grep my-app kubectl get clusterpolicyreport -o yaml | grep -A20 my-app ``` **Step 4: Dry-run the resource** Kyverno CLI lets you test locally: ```bash kyverno apply policy.yaml --resource pod.yaml ``` **Step 5: Check for exceptions** ```bash kubectl get policyexception -A # Is there an exception that should apply but doesn't? ``` ## Real Policies That Matter Here are battle-tested policies, not toy examples. ### 1. Require Resource Limits Pods without limits can starve nodes. This is non-negotiable in shared clusters. ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-resource-limits annotations: policies.kyverno.io/title: Require Resource Limits policies.kyverno.io/description: >- Pods must specify CPU and memory limits to prevent resource starvation. spec: validationFailureAction: Enforce background: true rules: - name: require-limits match: any: - resources: kinds: - Pod exclude: any: - resources: namespaces: - kube-system - kube-node-lease - kyverno - resources: selector: matchLabels: policy.example.com/exclude: "true" validate: message: "CPU and memory limits are required for all containers." pattern: spec: containers: - resources: limits: memory: "?*" cpu: "?*" =(initContainers): - resources: limits: memory: "?*" cpu: "?*" ``` ### 2. Restrict Image Registries Only allow images from your approved registries: ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: restrict-image-registries annotations: policies.kyverno.io/title: Restrict Image Registries policies.kyverno.io/description: >- Images must come from approved registries. spec: validationFailureAction: Enforce background: true rules: - name: validate-registries match: any: - resources: kinds: - Pod exclude: any: - resources: namespaces: - kube-system - kyverno validate: message: "Images must be from approved registries: gcr.io/mycompany, mycompany.azurecr.io" pattern: spec: containers: - image: "gcr.io/mycompany/* | mycompany.azurecr.io/*" =(initContainers): - image: "gcr.io/mycompany/* | mycompany.azurecr.io/*" =(ephemeralContainers): - image: "gcr.io/mycompany/* | mycompany.azurecr.io/*" ``` ### 3. Disallow Privileged Containers Privileged containers can escape to the host. Block them unless explicitly exempted. ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: disallow-privileged annotations: policies.kyverno.io/title: Disallow Privileged Containers policies.kyverno.io/description: >- Privileged containers are not allowed. spec: validationFailureAction: Enforce background: true rules: - name: deny-privileged match: any: - resources: kinds: - Pod exclude: any: - resources: namespaces: - kube-system validate: message: "Privileged containers are not allowed." pattern: spec: containers: - =(securityContext): =(privileged): false =(initContainers): - =(securityContext): =(privileged): false =(ephemeralContainers): - =(securityContext): =(privileged): false ``` ### 4. Auto-Generate NetworkPolicy for New Namespaces When a namespace is created, automatically generate a default-deny NetworkPolicy: ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-default-network-policy annotations: policies.kyverno.io/title: Generate Default NetworkPolicy policies.kyverno.io/description: >- Creates a default-deny NetworkPolicy for new namespaces. spec: rules: - name: generate-default-deny match: any: - resources: kinds: - Namespace exclude: any: - resources: names: - kube-* - default - kyverno generate: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy name: default-deny-all namespace: "{{request.object.metadata.name}}" data: spec: podSelector: {} policyTypes: - Ingress - Egress ``` New namespace → automatic NetworkPolicy. Teams must explicitly open traffic. ## Summary Kyverno at scale isn't about writing clever policies — it's about deploying them safely: 1. **Audit before enforce** — always 2. **Exclude system namespaces** — don't break the cluster 3. **Use PolicyExceptions** — keep policies clean, exceptions explicit 4. **Stage rollouts** — one cluster before fifty 5. **Propagate via Fleet** — single source of truth 6. **Override where needed** — development clusters are different 7. **Aggregate observability** — know your policy posture across the fleet The goal is consistent, enforceable security across all your clusters — without being the person who broke production. --- ## Eventual Consistency and Stale Caches in Kubernetes Controllers - URL: https://svalle.ru/posts/kubernetes/stale-cache-controllers/ - Date: 2025-01-27 - Tags: kubernetes, controllers, client-go, informers, cache, eventual-consistency Your custom controller watches for a label change. A user adds the label. Your controller does nothing — or worse, does the wrong thing. Thirty seconds later, it finally reacts. What happened? The answer lies in the informer cache, and understanding it is the difference between controllers that work in demos and controllers that work in production. ## The Problem: When Your Controller Lies to Itself Consider a simple scenario: 1. User runs: `kubectl label deployment web feature.example.com/inject-sidecar=true` 2. API server accepts the write, returns success 3. Your controller's reconcile loop runs 4. Controller checks: "Does this Deployment have the sidecar label?" 5. Cache says: "No" 6. Controller does nothing The label exists in etcd. The API server knows about it. But your controller's local cache hasn't caught up yet. Your controller just made a decision based on a lie. **This gets worse on busy clusters.** When the API server is under load, watch events queue up. When your controller is processing a backlog, event handlers fall behind. The window between "truth in etcd" and "truth in your cache" widens from milliseconds to seconds — sometimes longer. **Symptoms you'll see:** - **Flickering state**: Resource toggles between two states as controllers fight over stale views - **Unnecessary reconciliations**: Controller keeps requeueing because it can't see its own writes - **Race conditions**: Two controllers both think they need to act, both act, chaos ensues - **Silent failures**: Controller checks a condition, condition appears false, controller exits early ## How the Informer Cache Actually Works To fix these problems, you need to understand the machinery between the API server and your `Reconcile()` function. ### List & Watch Kubernetes controllers don't poll the API server. Instead, they use the **watch** protocol: 1. **Initial List**: On startup, the controller fetches all relevant objects (e.g., all Deployments in the cluster) 2. **Watch**: Controller opens a long-lived HTTP/2 stream. The API server pushes events (ADDED, MODIFIED, DELETED) as objects change This is efficient — you get updates pushed to you rather than polling. But it introduces a fundamental reality: **your controller sees an eventually consistent view of the cluster.** ### SharedInformer Architecture The client-go library provides `SharedInformer` to manage this watch lifecycle. Here's what's actually happening: ```d2 {alt="SharedInformer pipeline: the API Server pushes a watch stream over HTTP/2 to the Reflector, which feeds DeltaFIFO, which updates the Indexer cache; event handlers (OnAdd, OnUpdate, OnDelete) then add keys to the rate-limited Work Queue, and Reconcile pops them to run your code"} grid-columns: 1 vertical-gap: 48 api: API Server reflector: "Reflector\nConsumes watch events, handles reconnection" fifo: "DeltaFIFO\nBuffers changes, coalesces multiple updates" indexer: "Indexer\nThe actual cache (thread-safe in-memory store)" queue: "Work Queue\nRate-limited queue of keys to reconcile" reconcile: "Reconcile\nYour code runs here" api -> reflector: "Watch stream (HTTP/2)" reflector -> fifo fifo -> indexer indexer -> queue: "Event handlers (OnAdd, OnUpdate, OnDelete)" queue -> reconcile ``` Every box in this diagram is a place where delay can accumulate. ### ResourceVersion: How Kubernetes Tracks Freshness Every object in Kubernetes has a `metadata.resourceVersion` field. This isn't a version number you control — it's an opaque string derived from the etcd revision. ```yaml metadata: name: web resourceVersion: "1847293" # etcd revision when this object was last modified ``` When you watch resources, the API server tracks your position in the event stream using ResourceVersion. When your watch reconnects, it resumes from where it left off (if possible) or relists everything. **Key insight**: If you read an object from cache and it has `resourceVersion: "1847293"`, you're seeing the state as of etcd revision 1847293. The object might have been modified since then — your cache just hasn't received the event yet. ## The Lag Window The time between "API server accepts a write" and "your Reconcile() sees it" is your **lag window**. Let's trace where time goes: ### API Server Watch Cache The API server doesn't stream directly from etcd. It maintains an in-memory watch cache and flushes events to watchers periodically. Default flush interval: ~100ms. **Delay contribution**: 0-100ms typically ### Network Latency Events travel from API server to your controller over the network. **Delay contribution**: <1ms (same node) to 10-50ms (cross-region) ### DeltaFIFO Processing The Reflector pushes events into DeltaFIFO. A separate goroutine pops events and updates the Indexer. If events arrive faster than they're processed, they queue up. **Delay contribution**: Microseconds normally, can spike to seconds under load ### Event Handler Execution When the cache updates, your event handlers run (OnAdd, OnUpdate, OnDelete). If you do anything slow here — logging, metrics, complex filtering — you block subsequent events. **Delay contribution**: Should be microseconds, but bad code makes this milliseconds or worse ### Work Queue Wait Time Event handlers typically just add a key to the work queue. But if your reconciler is slow, the queue grows. New events wait behind old ones. **Delay contribution**: Depends entirely on your reconciler throughput ### Reconcile Execution Finally, your code runs. But you're reading from the cache, which reflects state as of when the event handler ran — not when Reconcile runs. **Total lag budget**: On a healthy cluster, 100-500ms is typical. On a busy cluster with slow reconcilers, 5-30 seconds is possible. ### Measuring the Lag Want to see this in your cluster? Log the ResourceVersion at write time and compare to what your cache returns: ```go func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { var obj appsv1.Deployment if err := r.Get(ctx, req.NamespacedName, &obj); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) } log.Info("Reconciling", "name", obj.Name, "cacheResourceVersion", obj.ResourceVersion, "queuedAt", req./* if you track this */)) // ... } ``` Compare against `kubectl get deployment web -o jsonpath='{.metadata.resourceVersion}'` to see how far behind your cache is. ## Common Pitfalls ### Read-After-Write Inconsistency The most common bug: ```go func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { var deploy appsv1.Deployment if err := r.Get(ctx, req.NamespacedName, &deploy); err != nil { return ctrl.Result{}, err } // Add a label if deploy.Labels == nil { deploy.Labels = make(map[string]string) } deploy.Labels["my-controller/processed"] = "true" if err := r.Update(ctx, &deploy); err != nil { return ctrl.Result{}, err } // BUG: Reading immediately after writing var updated appsv1.Deployment if err := r.Get(ctx, req.NamespacedName, &updated); err != nil { return ctrl.Result{}, err } // This might be false! Cache hasn't caught up to our write. if updated.Labels["my-controller/processed"] != "true" { log.Error(nil, "Label not found after update!") // This happens. } return ctrl.Result{}, nil } ``` The `Update()` call succeeds and returns the updated object. But `r.Get()` reads from the cache, which hasn't received the watch event yet. ### Negative Existence Checks This pattern is deceptively dangerous: ```go func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { var deploy appsv1.Deployment if err := r.Get(ctx, req.NamespacedName, &deploy); err != nil { return ctrl.Result{}, err } // Dangerous: acting on absence if _, exists := deploy.Labels["feature.example.com/sidecar"]; !exists { log.Info("Sidecar label not present, skipping") return ctrl.Result{}, nil } // ... inject sidecar } ``` If a user just added the label, your cache might not have it yet. You skip processing, and the user wonders why nothing happened. Worse, you don't requeue — so you might never process it until the next resync. ### Multi-Resource Coordination Controllers often watch multiple resource types: ```go func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { var deploy appsv1.Deployment if err := r.Get(ctx, req.NamespacedName, &deploy); err != nil { return ctrl.Result{}, err } // Get the associated ConfigMap var configMap corev1.ConfigMap configMapName := deploy.Annotations["my-controller/config"] if err := r.Get(ctx, types.NamespacedName{ Namespace: deploy.Namespace, Name: configMapName, }, &configMap); err != nil { if apierrors.IsNotFound(err) { // BUG: ConfigMap might exist but not be in cache yet log.Info("ConfigMap not found, waiting...") return ctrl.Result{RequeueAfter: 5 * time.Second}, nil } return ctrl.Result{}, err } // ... } ``` You're watching Deployments. Someone creates a ConfigMap and then annotates a Deployment to reference it. The Deployment event arrives, but the ConfigMap watch might not have received its ADDED event yet. You conclude the ConfigMap doesn't exist. ### Conflict Loops Two controllers watching the same resource, both with stale views: ``` Controller A cache: Deployment has 3 replicas Controller B cache: Deployment has 3 replicas User sets replicas to 5 Controller A sees update event (replicas=5) Controller A: "I need to create a monitoring config for 5 replicas" Controller A updates Deployment annotations Controller B sees the annotation update (but has stale replicas=3 in cache) Controller B: "Annotations changed, let me process this... replicas=3" Controller B "fixes" replicas back to 3 based on stale cache Controller A sees replicas change to 3... ``` Both controllers are acting rationally based on their view. But their views are inconsistent, and they fight. ## Strategies and Patterns ### Requeue with Backoff Don't trust a single reconciliation. If conditions aren't met, requeue and check again: ```go func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { var deploy appsv1.Deployment if err := r.Get(ctx, req.NamespacedName, &deploy); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) } // Check if we've processed this if deploy.Annotations["my-controller/processed"] != "true" { // Do processing... deploy.Annotations["my-controller/processed"] = "true" if err := r.Update(ctx, &deploy); err != nil { return ctrl.Result{}, err } // Don't trust the update immediately - requeue to verify return ctrl.Result{RequeueAfter: 1 * time.Second}, nil } return ctrl.Result{}, nil } ``` ### Optimistic Concurrency Always use ResourceVersion when updating. The API server rejects updates with stale ResourceVersion: ```go func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { var deploy appsv1.Deployment if err := r.Get(ctx, req.NamespacedName, &deploy); err != nil { return ctrl.Result{}, err } // Modify deploy.Labels["processed"] = "true" // Update - this uses the ResourceVersion from Get() if err := r.Update(ctx, &deploy); err != nil { if apierrors.IsConflict(err) { // Someone else modified it - requeue and try again log.Info("Conflict detected, requeueing") return ctrl.Result{Requeue: true}, nil } return ctrl.Result{}, err } return ctrl.Result{}, nil } ``` The conflict error is your friend. It tells you your view was stale and prevents you from clobbering someone else's changes. ### Read-Through on Critical Paths When you absolutely need fresh data, bypass the cache: ```go func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { // Normal cached read var deploy appsv1.Deployment if err := r.Get(ctx, req.NamespacedName, &deploy); err != nil { return ctrl.Result{}, err } // Need fresh data for critical decision? Bypass cache. var freshDeploy appsv1.Deployment if err := r.APIReader.Get(ctx, req.NamespacedName, &freshDeploy); err != nil { return ctrl.Result{}, err } // freshDeploy is read directly from API server } ``` Use this sparingly — it adds API server load and defeats the purpose of caching. But for critical decisions where eventual consistency isn't acceptable, it's the right tool. ### The Expectations Pattern The kube-controller-manager uses this pattern for ReplicaSets. Track what you expect to happen, and wait for the cache to confirm: ```go type Expectations struct { mu sync.Mutex expected map[string]expectation } type expectation struct { add int // expecting this many adds delete int // expecting this many deletes } func (e *Expectations) ExpectCreations(key string, count int) { e.mu.Lock() defer e.mu.Unlock() exp := e.expected[key] exp.add += count e.expected[key] = exp } func (e *Expectations) CreationObserved(key string) { e.mu.Lock() defer e.mu.Unlock() exp := e.expected[key] exp.add-- e.expected[key] = exp } func (e *Expectations) SatisfiedExpectations(key string) bool { e.mu.Lock() defer e.mu.Unlock() exp := e.expected[key] return exp.add <= 0 && exp.delete <= 0 } ``` In your reconciler: ```go func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { key := req.NamespacedName.String() // Don't reconcile until expectations are met if !r.expectations.SatisfiedExpectations(key) { log.Info("Expectations not yet satisfied, skipping") return ctrl.Result{}, nil } // ... determine we need to create 3 pods r.expectations.ExpectCreations(key, 3) for i := 0; i < 3; i++ { if err := r.Create(ctx, &pod); err != nil { // Creation failed - adjust expectations r.expectations.CreationObserved(key) return ctrl.Result{}, err } } return ctrl.Result{}, nil } ``` Your pod informer's event handler calls `CreationObserved()` when it sees new pods. This prevents the reconciler from creating duplicate pods because it doesn't see the ones it just created. ### Generation vs ObservedGeneration For tracking whether a controller has processed the latest spec changes, use the Generation pattern: ```go func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { var obj myv1.MyResource if err := r.Get(ctx, req.NamespacedName, &obj); err != nil { return ctrl.Result{}, err } // Skip if we've already processed this generation if obj.Status.ObservedGeneration == obj.Generation { return ctrl.Result{}, nil } // Process the spec... // Update status to record that we've processed this generation obj.Status.ObservedGeneration = obj.Generation obj.Status.Conditions = append(obj.Status.Conditions, metav1.Condition{ Type: "Ready", Status: metav1.ConditionTrue, ObservedGeneration: obj.Generation, LastTransitionTime: metav1.Now(), }) if err := r.Status().Update(ctx, &obj); err != nil { return ctrl.Result{}, err } return ctrl.Result{}, nil } ``` `metadata.generation` increments only when spec changes. `status.observedGeneration` records which generation your controller last processed. This gives you a reliable way to detect "is there new work to do" even with cache lag. ## Hands-On: Demonstrating and Debugging Cache Lag ### Instrumenting Cache Freshness Add metrics to track how stale your cache reads are: ```go var ( cacheAgeHistogram = prometheus.NewHistogramVec( prometheus.HistogramOpts{ Name: "controller_cache_age_seconds", Help: "Age of objects read from cache", Buckets: []float64{0.01, 0.05, 0.1, 0.5, 1, 5, 10, 30}, }, []string{"controller", "resource"}, ) ) func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { start := time.Now() var deploy appsv1.Deployment if err := r.Get(ctx, req.NamespacedName, &deploy); err != nil { return ctrl.Result{}, err } // Estimate cache age from last update timestamp if lastUpdate := deploy.ManagedFields[len(deploy.ManagedFields)-1].Time; lastUpdate != nil { age := time.Since(lastUpdate.Time).Seconds() cacheAgeHistogram.WithLabelValues("mycontroller", "deployment").Observe(age) } // ... } ``` ### Simulating Busy Cluster Conditions For testing, add artificial delay to your event handlers: ```go func setupEventHandlers(mgr ctrl.Manager) error { informer, err := mgr.GetCache().GetInformer(context.Background(), &appsv1.Deployment{}) if err != nil { return err } informer.AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { if simulateLag { time.Sleep(2 * time.Second) // Simulate busy cluster } }, UpdateFunc: func(old, new interface{}) { if simulateLag { time.Sleep(2 * time.Second) } }, }) return nil } ``` Run your controller with this lag injected and watch how it behaves. ### Logging Patterns Log enough context to debug stale cache issues: ```go func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { log := log.FromContext(ctx) var deploy appsv1.Deployment if err := r.Get(ctx, req.NamespacedName, &deploy); err != nil { return ctrl.Result{}, err } log.V(1).Info("Reconciling", "resourceVersion", deploy.ResourceVersion, "generation", deploy.Generation, "observedGeneration", deploy.Status.ObservedGeneration, "labels", deploy.Labels, ) // ... do work log.V(1).Info("Reconcile complete", "resultingResourceVersion", deploy.ResourceVersion, ) return ctrl.Result{}, nil } ``` When debugging, compare these ResourceVersions across log entries to trace propagation delays. ## Embracing Eventual Consistency The informer cache isn't broken — it's working as designed. Kubernetes is an eventually consistent system, and your controller must be too. **Design principles:** 1. **Idempotency**: Running your reconciler twice with the same input should produce the same result 2. **Convergence**: Given enough time without new inputs, the system reaches the desired state 3. **Tolerance**: Your controller handles stale data gracefully — it might make suboptimal decisions, but never catastrophically wrong ones 4. **Verification**: Don't trust a single check. Requeue, recheck, confirm The goal isn't to eliminate cache lag — it's to build controllers that work correctly despite it. --- ## Scaling Beyond 5,000 Nodes Per Cluster: Anatomy of Kubernetes Limits and Multi-Cluster Architecture - URL: https://svalle.ru/posts/kubernetes/scaling-beyond-5k-nodes/ - Date: 2025-01-26 - Tags: kubernetes, multi-cluster, kubefleet, scalability, etcd, architecture If you've operated Kubernetes at scale, you've encountered the infamous 5,000 node recommendation. But what actually breaks at that threshold? And when you adopt multi-cluster to scale horizontally, what are you really buying? This post dissects the technical constraints behind the limit and explores how the pull-based multi-cluster model addresses them. ## The Anatomy of 5,000 Nodes The 5K figure isn't arbitrary — it's the point where Kubernetes can still meet its Service Level Objectives: - **99th percentile API call latency < 1 second** - **99th percentile pod startup latency < 5 seconds** Beyond this, the control plane struggles to keep these promises. But understanding *why* requires examining what's actually under pressure. ### etcd: The First Domino etcd is a distributed key-value store that holds all cluster state. Every object in Kubernetes — Pods, Services, ConfigMaps, Secrets, CRDs — lives here. **Watch channel explosion**: The API server maintains watches on etcd for every controller, scheduler, and kubelet. In a 5,000 node cluster with 110 pods per node (the default max), you have approximately 550,000 pod objects. Each state change generates watch events that must be serialized and fanned out. **The math gets ugly**: ``` 5,000 nodes × 110 pods = 550,000 pods + Services, Endpoints, ConfigMaps, Secrets, etc. ≈ 1-2 million objects in etcd ``` etcd's Raft consensus requires every write to be replicated across the cluster. At high write rates, leader election latency increases, compaction falls behind, and eventually the database exceeds its recommended size (8GB default). **What breaks**: Watch latency spikes. Controllers see stale state. The scheduler makes decisions on outdated information. Pods get scheduled to nodes that are already full. ### API Server: Death by a Thousand Watches The API server is stateless, but it maintains in-memory watch caches for performance. Each watch consumes memory and CPU for serialization. **Watch fanout**: When a pod changes state, that event must be sent to: - The scheduler (if pending) - The relevant controller (Deployment, ReplicaSet, etc.) - The kubelet on the node - Any monitoring systems watching pods - Service mesh sidecars - Network policy controllers In a large cluster, a single pod update can trigger dozens of watch notifications. Multiply by churn rate (pods starting, stopping, failing) and you get amplification. **Webhook latency**: If you're running admission webhooks (and you probably are — OPA, Pod Security, etc.), every pod creation traverses the webhook. At scale, webhook latency becomes a bottleneck. A webhook that adds 50ms is fine at 10 pods/sec but devastating at 500 pods/sec. **What breaks**: API request queuing. Clients see timeouts. `kubectl` commands hang. Deployments appear stuck. ### Scheduler: The 100 Pods/Second Ceiling The default Kubernetes scheduler processes roughly 100 pods per second under optimal conditions. This throughput depends on: - **Filtering**: Evaluating which nodes can run a pod (resource requests, taints, affinity) - **Scoring**: Ranking feasible nodes by preference - **Preemption**: Evicting lower-priority pods to make room With 5,000 nodes, each scheduling decision evaluates thousands of candidates. Enable pod affinity/anti-affinity and the complexity explodes — the scheduler must examine co-located pods across all nodes. **What breaks**: Pending pod queues grow. Batch jobs that spawn 10,000 pods take hours to schedule. Autoscaling lags behind demand. ### Controller Manager: The Thundering Herd Every controller in the system (Deployment, ReplicaSet, Job, DaemonSet, etc.) watches its relevant objects and reconciles state. With more objects, work queues grow deeper. **Garbage collection pressure**: The GC controller tracks owner references across all objects. At scale, orphan detection becomes expensive. **What breaks**: Controllers fall behind. You delete a Deployment and its pods linger. ReplicaSets don't scale down properly. ## Why Multi-Cluster Solves This The bottlenecks above share a common theme: **centralized state and control**. One etcd, one API server, one scheduler — all trying to manage hundreds of thousands of objects. Multi-cluster addresses this by **horizontal partitioning**. Instead of one 20,000-node cluster, you run four 5,000-node clusters. Each cluster has its own: - etcd (handling 1/4 the objects) - API server (1/4 the watches) - Scheduler (1/4 the pods) - Controllers (1/4 the work queues) But this raises a new problem: how do you manage workloads across multiple clusters without creating a new centralized bottleneck? ## The Pull Model: Distributing the Control Plane Load KubeFleet (the CNCF project underlying Azure Fleet Manager) uses a **hub-spoke architecture with a pull-based model**. This design is critical for scalability. ### Hub Agent (`fleet-hub-agent`) The hub agent runs on a designated "hub" cluster — a lightweight cluster that serves as the control plane for your fleet. It doesn't run workloads; it coordinates. **Responsibilities**: - Watches `ClusterResourcePlacement` objects (your intent: "deploy this namespace to clusters matching these criteria") - Evaluates placement policies against member cluster properties - Creates `Work` objects in per-member namespaces - Tracks rollout status across the fleet The hub agent doesn't push anything to member clusters. It simply writes `Work` objects to the hub cluster's API server. This is crucial — the hub doesn't need network connectivity *to* member clusters. ### Member Agent (`fleet-member-agent`) Each member cluster runs a member agent that **pulls** its work from the hub. **Responsibilities**: - Watches its dedicated namespace on the hub cluster (e.g., `fleet-member-cluster-1`) - Fetches `Work` objects containing Kubernetes manifests - Applies manifests to the local cluster - Reports status back to the hub (via `Work` status updates) The pull model means: ``` Hub cluster ← Member agents connect outbound ← Member agents poll for Work objects ← Member agents push status updates ``` No inbound connections to member clusters required. ### Inside the Work Object A `Work` object is the unit of propagation. When you create a `ClusterResourcePlacement`, the hub agent translates your intent into concrete `Work` objects: ```yaml apiVersion: placement.kubernetes-fleet.io/v1 kind: Work metadata: name: crp-my-app-0 namespace: fleet-member-cluster-1 spec: workload: manifests: - apiVersion: v1 kind: Namespace metadata: name: my-app - apiVersion: apps/v1 kind: Deployment metadata: name: web namespace: my-app spec: replicas: 3 # ... full deployment spec ``` The member agent watches for these objects, extracts the manifests, and applies them locally using server-side apply. ### Why Pull Scales Better Than Push In a push model, the hub would need to: 1. Maintain connections to all member clusters 2. Have credentials/kubeconfig for each member 3. Handle retries, timeouts, and failures for each push 4. Deal with network partitions gracefully With 100 member clusters, that's 100 connections to manage, 100 failure domains to handle, and significant blast radius if the hub has issues. In the pull model: 1. Hub just writes to its local API server 2. Member agents are responsible for their own connectivity 3. If a member is temporarily unreachable, it catches up when reconnected 4. Hub failure means no *new* placements, but existing workloads keep running **Load distribution**: Each member agent does its own reconciliation. The work of applying manifests, tracking status, and handling drift is distributed across the fleet, not centralized on the hub. ## Failure Modes and Recovery Understanding failure scenarios is essential for production deployments. ### Hub Cluster Failure **Impact**: No new placements. No updates to existing `ClusterResourcePlacement` objects take effect. No new clusters can join. **What keeps working**: Member clusters continue running their workloads. The member agent retries connecting to the hub. Existing `Work` objects (cached or already applied) remain in effect. **Recovery**: Restore hub from etcd backup. Member agents reconnect automatically. **Design implication**: The hub should be a highly available, multi-zone deployment. But it doesn't need to be large — it's only managing fleet metadata, not running workloads. ### Member Agent Failure **Impact**: That specific member cluster stops receiving updates from the hub. Drift from desired state won't be corrected. **What keeps working**: Workloads on that cluster continue running (Kubernetes controllers are local). Other member clusters are unaffected. **Detection**: The `MemberCluster` object on the hub tracks last heartbeat time: ```yaml status: agentStatus: - type: MemberAgent lastReceivedHeartbeat: "2025-01-25T10:30:00Z" conditions: - type: Joined status: "True" ``` **Recovery**: Restart the member agent. It will re-sync by fetching current `Work` objects from the hub. ### Network Partition **Scenario**: Member cluster loses connectivity to hub but maintains local network. **Impact**: Similar to hub failure from that member's perspective. No new `Work` objects fetched. Status updates not reported. **Behavior**: The member agent uses exponential backoff on reconnection attempts. Once connectivity restores, it fetches the latest `Work` objects and reconciles. **Risk**: If someone modifies manifests on the hub during the partition, the member won't see changes until reconnection. This isn't split-brain (there's only one source of truth — the hub), but it is temporal inconsistency. ### Drift Detection What if someone manually edits a resource on a member cluster that's managed by Fleet? The member agent can detect drift between the `Work` spec and actual cluster state. You can configure behavior: - **Apply mode**: Overwrite local changes (eventual consistency with hub) - **ReportDiff mode**: Report the difference but don't overwrite (useful for debugging) ```yaml apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata: name: my-app spec: resourceSelectors: - group: "" kind: Namespace name: my-app version: v1 strategy: applyStrategy: type: ClientSideApply # or ServerSideApply allowCoOwnership: false ``` ## Capacity Planning for Real Scale ### Cluster Sizing Don't default to "max out at 5K nodes per cluster." Consider: **Blast radius**: A cluster-wide outage (bad config push, control plane failure) affects all nodes. Smaller clusters = smaller blast radius. **Upgrade complexity**: Upgrading a 5,000-node cluster is an all-day event. Upgrading ten 500-node clusters can be parallelized and staged. **Workload isolation**: Different teams, environments, or compliance zones might warrant separate clusters regardless of size. **Practical guidance**: Many organizations find 500-2,000 nodes per cluster to be a sweet spot — large enough to be efficient, small enough to be manageable. ### Hub Cluster Sizing The hub is lightweight. It runs: - The Kubernetes control plane - `fleet-hub-agent` - Standard system components For most fleets (up to 100 member clusters), a 3-node hub cluster with modest sizing (4 vCPU, 16GB RAM per node) is sufficient. The hub's etcd stores fleet metadata — `MemberCluster`, `ClusterResourcePlacement`, `Work` objects — not your application workloads. ### Network Topology **Hub placement**: The hub should be reachable from all member clusters. For on-prem deployments spanning multiple data centers, consider: - Hub in a central location with good connectivity to all sites - Hub behind a load balancer for HA - DNS-based failover if you run hub replicas **Latency budget**: Member agents poll the hub for `Work` objects. Higher latency means slower propagation of changes, but the system tolerates it gracefully. Sub-second latency isn't required; sub-minute is fine. **Egress-only**: Member clusters only need *outbound* HTTPS to the hub. No firewall rules for inbound traffic to your on-prem clusters. ## Hands-On: Setting Up KubeFleet On-Premises ### Prerequisites - 3+ Kubernetes clusters (1 hub, 2+ members) - `kubectl` access to all clusters - Container registry accessible from all clusters - Helm 3.x ### Install Hub Agent ```bash # Clone KubeFleet git clone https://github.com/kubefleet-dev/kubefleet.git cd kubefleet # Set your registry export REGISTRY="your-registry.example.com" export TAG="v0.10.0" # Use latest stable # Build and push hub agent image make docker-build-hub-agent make docker-push-hub-agent # Switch to hub cluster context kubectl config use-context hub-cluster # Install via Helm helm install hub-agent ./charts/hub-agent \ --set image.repository=${REGISTRY}/hub-agent \ --set image.tag=${TAG} ``` Verify the hub agent is running: ```bash kubectl get pods -n fleet-system # NAME READY STATUS RESTARTS AGE # hub-agent-xxxxxxxxx-xxxxx 1/1 Running 0 30s ``` ### Join Member Clusters KubeFleet provides a script to automate member joining: ```bash # Set member cluster details export MEMBER_CLUSTER="dc1-cluster" export MEMBER_CLUSTER_CONTEXT="dc1-cluster-admin" # Build member agent image make docker-build-member-agent make docker-push-member-agent # Run the join script ./hack/membership/joinMC.sh ${TAG} hub-cluster ${MEMBER_CLUSTER} ``` Verify the member joined: ```bash kubectl config use-context hub-cluster kubectl get membercluster # NAME JOINED AGE MEMBER-AGENT-LAST-SEEN NODE-COUNT # dc1-cluster True 60s 10s 150 ``` ### Observing Agent Behavior **Watch the member agent sync**: ```bash # On the hub cluster, watch Work objects for a member kubectl get work -n fleet-member-dc1-cluster -w ``` **Simulate a placement**: ```bash # Create a namespace on the hub kubectl create namespace demo-app # Create a ClusterResourcePlacement cat <M columns p0: "P0\nRun Queue\nG G G" m0: "M\n(OS)" p1: "P1\nRun Queue\nG G" m1: "M\n(OS)" p2: "P2\nRun Queue\nG G G" m2: "M\n(OS)" p0 -> m0 p1 -> m1 p2 -> m2 ``` The number of P's defaults to `GOMAXPROCS` (usually number of CPU cores). This is the true parallelism limit — you can have millions of G's, but only `GOMAXPROCS` run simultaneously. **Work stealing**: When a P's run queue is empty, it steals goroutines from other P's. This keeps all cores busy without explicit load balancing. ### Cooperative Scheduling Goroutines yield control at specific points: - Channel operations (send/receive) - System calls (I/O, sleep) - Function calls (allows stack check, potential preemption) - Explicit `runtime.Gosched()` **The gotcha**: A tight CPU-bound loop without function calls can block other goroutines on that P: ```go // Bad: This can starve other goroutines func cpuHog() { for { // Pure computation, no function calls x := 0 for i := 0; i < 1e9; i++ { x += i } } } ``` Go 1.14 introduced asynchronous preemption (via signals) to mitigate this, but it's not perfect. Design your code to yield. ## Channels: Synchronization Primitive, Not Just a Pipe Channels are typed conduits for communication. But thinking of them as just "concurrent queues" misses the point. **Channels are synchronization primitives that happen to transfer data.** ### Unbuffered Channels: Rendezvous Points An unbuffered channel blocks both sender and receiver until both are ready: ```go ch := make(chan int) // unbuffered // Goroutine A ch <- 42 // blocks until someone receives // Goroutine B x := <-ch // blocks until someone sends ``` This is a **rendezvous** — both goroutines must arrive at the channel operation for either to proceed. It's a synchronization point, not just data transfer. **Use case**: When you need to ensure one goroutine has completed a step before another proceeds. ### Buffered Channels: Decoupling Buffered channels allow sends to proceed without a receiver (up to the buffer size): ```go ch := make(chan int, 10) // buffer of 10 ch <- 1 // doesn't block (buffer not full) ch <- 2 // doesn't block // ... up to 10 sends without blocking ``` **Use case**: Decoupling producer and consumer speeds, work queues, rate limiting. **The trap**: People buffer channels to "fix" deadlocks. This usually masks the bug temporarily — the deadlock reappears under load when the buffer fills. ### Channel Patterns **Fan-out**: Multiple goroutines reading from the same channel. ```go func worker(id int, jobs <-chan Job, results chan<- Result) { for job := range jobs { results <- process(job) } } func main() { jobs := make(chan Job, 100) results := make(chan Result, 100) // Start workers for i := 0; i < 10; i++ { go worker(i, jobs, results) } // Send jobs for _, job := range allJobs { jobs <- job } close(jobs) // Collect results for range allJobs { <-results } } ``` **Fan-in**: Multiple goroutines sending to the same channel. ```go func merge(channels ...<-chan int) <-chan int { out := make(chan int) var wg sync.WaitGroup for _, ch := range channels { wg.Add(1) go func(c <-chan int) { defer wg.Done() for v := range c { out <- v } }(ch) } go func() { wg.Wait() close(out) }() return out } ``` **Pipeline**: Chain of stages connected by channels. ```go func gen(nums ...int) <-chan int { out := make(chan int) go func() { for _, n := range nums { out <- n } close(out) }() return out } func square(in <-chan int) <-chan int { out := make(chan int) go func() { for n := range in { out <- n * n } close(out) }() return out } func main() { // Pipeline: gen -> square -> print for n := range square(gen(1, 2, 3, 4)) { fmt.Println(n) } } ``` ## The Bugs You'll Write ### Goroutine Leaks A goroutine that never terminates is a memory leak. Common causes: **Blocked on channel forever:** ```go func leak() { ch := make(chan int) go func() { val := <-ch // blocks forever - nothing sends to ch fmt.Println(val) }() // Function returns, but goroutine lives on, waiting forever } ``` **Unbounded goroutine spawning:** ```go func handler(requests <-chan Request) { for req := range requests { // New goroutine per request - if processing is slow, // these accumulate go process(req) } } ``` **Detection**: Monitor `runtime.NumGoroutine()` over time. In tests, check goroutine count before and after. ```go func TestNoLeaks(t *testing.T) { before := runtime.NumGoroutine() // ... run test // Give goroutines time to exit time.Sleep(100 * time.Millisecond) after := runtime.NumGoroutine() if after > before { t.Errorf("Goroutine leak: %d before, %d after", before, after) } } ``` ### Channel Deadlocks **Circular dependency:** ```go func deadlock() { ch1 := make(chan int) ch2 := make(chan int) go func() { <-ch1 // waits for ch1 ch2 <- 1 // then sends to ch2 }() go func() { <-ch2 // waits for ch2 ch1 <- 1 // then sends to ch1 }() // Both goroutines wait forever } ``` **Self-deadlock:** ```go func selfDeadlock() { ch := make(chan int) ch <- 1 // blocks - no receiver x := <-ch // never reached fmt.Println(x) } ``` Go's runtime detects some deadlocks ("all goroutines are asleep") but not all — if there's any goroutine that could theoretically make progress (even if it won't), no panic. ### Data Races Goroutines sharing memory without synchronization: ```go func race() { counter := 0 for i := 0; i < 1000; i++ { go func() { counter++ // DATA RACE: read-modify-write without sync }() } time.Sleep(time.Second) fmt.Println(counter) // Not 1000. Different every run. } ``` **Detection**: Run with `-race` flag: ```bash go run -race main.go go test -race ./... ``` The race detector has ~10x CPU overhead and ~5-10x memory overhead. Use it in tests, not production. **Fixes**: ```go // Option 1: Mutex var mu sync.Mutex mu.Lock() counter++ mu.Unlock() // Option 2: Atomic var counter int64 atomic.AddInt64(&counter, 1) // Option 3: Channel (move data, not share it) results := make(chan int, 1000) for i := 0; i < 1000; i++ { go func() { results <- 1 }() } total := 0 for i := 0; i < 1000; i++ { total += <-results } ``` ### Context Cancellation Ignored When a context is cancelled, goroutines should exit promptly: ```go func badWorker(ctx context.Context) { for { // Does work but never checks ctx doExpensiveWork() } } func goodWorker(ctx context.Context) { for { select { case <-ctx.Done(): return // Exit when cancelled default: doExpensiveWork() } } } ``` For long-running operations, check `ctx.Done()` periodically: ```go func goodWorker(ctx context.Context) error { for i := 0; i < 1000000; i++ { if i%1000 == 0 { // Check every 1000 iterations select { case <-ctx.Done(): return ctx.Err() default: } } doWork(i) } return nil } ``` ## When Concurrency Hurts Concurrency isn't free. Each goroutine has overhead, channels have synchronization costs, and parallel code is harder to reason about. ### Too Many Goroutines for CPU-Bound Work For CPU-bound tasks, more goroutines than cores just adds scheduling overhead: ```go // Bad: 10,000 goroutines for CPU work on 8 cores for i := 0; i < 10000; i++ { go cpuIntensiveTask(data[i]) } // Better: Worker pool sized to cores numWorkers := runtime.GOMAXPROCS(0) jobs := make(chan Data, len(data)) results := make(chan Result, len(data)) for i := 0; i < numWorkers; i++ { go worker(jobs, results) } for _, d := range data { jobs <- d } close(jobs) ``` ### Channel Overhead for Fine-Grained Communication Channels have overhead (~50-100ns per operation). For very fine-grained work, this dominates: ```go // Bad: Channel send per number to sum func sumViaChan(nums []int) int { ch := make(chan int) go func() { for _, n := range nums { ch <- n } close(ch) }() sum := 0 for n := range ch { sum += n } return sum } // Good: Just sum directly (or batch if parallelizing) func sumDirect(nums []int) int { sum := 0 for _, n := range nums { sum += n } return sum } ``` **Rule of thumb**: If the work per channel operation is less than ~1µs, the channel overhead matters. ### False Sharing When goroutines access adjacent memory locations, CPU cache lines bounce between cores: ```go type Counters struct { a int64 // These are likely on the same cache line b int64 } var c Counters // Two goroutines incrementing different fields // but causing cache line contention go func() { for i := 0; i < 1e8; i++ { atomic.AddInt64(&c.a, 1) } }() go func() { for i := 0; i < 1e8; i++ { atomic.AddInt64(&c.b, 1) } }() ``` **Fix**: Pad to separate cache lines: ```go type Counters struct { a int64 _ [56]byte // Padding to push b to next cache line b int64 } ``` ## Benchmarking Reality Let's measure actual overhead on a real task: fetching 100 URLs. ```go func BenchmarkSequential(b *testing.B) { for i := 0; i < b.N; i++ { for _, url := range urls { fetch(url) } } } func BenchmarkConcurrent(b *testing.B) { for i := 0; i < b.N; i++ { var wg sync.WaitGroup for _, url := range urls { wg.Add(1) go func(u string) { defer wg.Done() fetch(u) }(url) } wg.Wait() } } func BenchmarkWorkerPool(b *testing.B) { for i := 0; i < b.N; i++ { jobs := make(chan string, len(urls)) var wg sync.WaitGroup // 10 workers for w := 0; w < 10; w++ { wg.Add(1) go func() { defer wg.Done() for url := range jobs { fetch(url) } }() } for _, url := range urls { jobs <- url } close(jobs) wg.Wait() } } ``` Typical results (100 URLs, ~100ms average latency each): ``` BenchmarkSequential-8 1 10234567890 ns/op (~10s) BenchmarkConcurrent-8 10 123456789 ns/op (~120ms) BenchmarkWorkerPool-8 10 112345678 ns/op (~110ms) ``` Concurrent is ~80x faster than sequential for I/O-bound work. Worker pool is marginally faster due to less goroutine creation overhead, but the difference is small. For CPU-bound work, the story is different — concurrent won't beat sequential unless you have multiple cores and can actually parallelize. ## Summary Go's concurrency model is powerful because it's simple enough to use casually but sophisticated enough to scale. The key insights: 1. **Goroutines are cheap** — spawn them freely for I/O-bound work 2. **Channels synchronize, not just communicate** — think about synchronization needs first 3. **The runtime does a lot** — but you can still block it with bad code 4. **Use the race detector** — data races are subtle and deadly 5. **Not everything needs concurrency** — measure before optimizing The best concurrent code is the simplest code that achieves the required parallelism. Start sequential, add concurrency where profiling shows it helps. --- ## Python Parallelism: The GIL, Multiprocessing, and When Each Matters - URL: https://svalle.ru/posts/python/multiprocessing-idioms/ - Date: 2018-09-10 - Tags: python, multiprocessing, concurrency, gil, asyncio, threading, performance Python's concurrency story is often summarized as "there's a GIL, so use multiprocessing." That's dangerously incomplete. The truth involves understanding what the GIL actually does, when it doesn't matter, and the real costs of multiprocessing that nobody talks about. ## The GIL: What It Actually Is The Global Interpreter Lock is a mutex that protects access to Python objects. Only one thread can execute Python bytecode at a time. **Why it exists**: CPython's memory management (reference counting) isn't thread-safe. Without the GIL, two threads incrementing the same object's reference count could corrupt it: ```python # Without GIL, this would be a race condition inside CPython: # Thread 1: reads refcount (1) # Thread 2: reads refcount (1) # Thread 1: writes refcount (2) # Thread 2: writes refcount (2) # Should be 3! ``` The GIL is a design choice — simpler implementation, faster single-threaded performance, easier C extension development. Other Python implementations (Jython, IronPython) don't have it. ### When the GIL Releases The GIL isn't held 100% of the time. It releases: - **Every N bytecode instructions** (default: 100, configurable via `sys.setswitchinterval`) - **During I/O operations** (file reads, network calls, `time.sleep`) - **During certain C extension calls** (NumPy operations, some database drivers) This is why threading *does* work for I/O-bound Python: ```python import threading import requests def fetch(url): return requests.get(url) # GIL released during network I/O # These run concurrently, not sequentially threads = [threading.Thread(target=fetch, args=(url,)) for url in urls] for t in threads: t.start() for t in threads: t.join() ``` ### When the GIL Hurts CPU-bound pure Python code cannot parallelize with threads: ```python import threading def cpu_work(): total = 0 for i in range(10_000_000): total += i * i return total # With threads - actually SLOWER than sequential due to GIL contention threads = [threading.Thread(target=cpu_work) for _ in range(4)] for t in threads: t.start() for t in threads: t.join() ``` Four threads doing CPU work on four cores will be *slower* than one thread, because they're constantly fighting over the GIL. ## Threading vs Multiprocessing vs asyncio Python offers three concurrency models. Each has its place. ### Threading **What it is**: OS threads, shared memory, GIL-limited **Good for**: - I/O-bound work (network calls, file I/O, database queries) - Blocking on external resources - Simple producer/consumer patterns **Bad for**: - CPU-bound work (GIL prevents parallelism) - Memory-heavy workloads (threads share memory, but coordination is tricky) ```python from concurrent.futures import ThreadPoolExecutor def fetch_all(urls): with ThreadPoolExecutor(max_workers=10) as executor: return list(executor.map(requests.get, urls)) ``` ### asyncio **What it is**: Single-threaded event loop, cooperative multitasking **Good for**: - High-concurrency I/O (thousands of connections) - When you control the code (can make everything async) - Network services, web scraping **Bad for**: - CPU-bound work (still single-threaded) - Mixing with blocking code (blocks the whole event loop) - Libraries that aren't async-aware ```python import asyncio import aiohttp async def fetch_all(urls): async with aiohttp.ClientSession() as session: tasks = [session.get(url) for url in urls] return await asyncio.gather(*tasks) ``` ### Multiprocessing **What it is**: Separate Python processes, no shared memory (by default), no GIL limitation **Good for**: - CPU-bound work that needs true parallelism - Isolation (one process crashing doesn't kill others) - Memory-intensive independent tasks **Bad for**: - Tasks requiring shared state (coordination is expensive) - Small tasks (process overhead dominates) - I/O-bound work (threading or asyncio is simpler and often faster) ```python from multiprocessing import Pool def cpu_work(n): return sum(i * i for i in range(n)) with Pool(4) as p: results = p.map(cpu_work, [10_000_000] * 4) # True parallelism ``` ## The Real Cost of Multiprocessing Multiprocessing isn't free. Understanding the costs helps you decide when it's worth it. ### Process Creation Overhead Spawning a process is expensive — typically 10-100ms: ```python import time from multiprocessing import Process start = time.time() processes = [Process(target=lambda: None) for _ in range(100)] for p in processes: p.start() for p in processes: p.join() print(f"100 processes: {time.time() - start:.2f}s") # ~1-5 seconds ``` For small tasks, this overhead dominates. A task that takes 1ms but costs 50ms to spawn into a new process is a net loss. **Solution**: Reuse processes with `Pool`: ```python from multiprocessing import Pool # Process creation happens once with Pool(4) as p: # Thousands of tasks, only 4 processes results = p.map(small_task, items) ``` ### Serialization (Pickling) Costs Data passed between processes must be serialized. Python uses `pickle` by default. **What gets pickled**: - Function arguments - Return values - Any data shared via `Queue`, `Pipe`, etc. **Costs**: - CPU time to serialize/deserialize - Memory to hold serialized data - I/O to transfer between processes ```python import pickle import numpy as np # Large NumPy array arr = np.random.rand(1000, 1000) # How much does pickling cost? import time start = time.time() for _ in range(100): data = pickle.dumps(arr) pickle.loads(data) print(f"Pickle roundtrip: {(time.time() - start) / 100 * 1000:.2f}ms") # Typically 5-20ms per roundtrip for this size ``` If you're passing large objects and the work per object is small, pickling dominates: ```python # Bad: Pickle overhead > work def tiny_work(large_array): return large_array.sum() # Microseconds of work with Pool(4) as p: # Each call pickles the large array - terrible performance results = p.map(tiny_work, large_arrays) # Better: Pass indices, let workers load their own data def worker_with_shared_data(indices, data_path): data = load_data(data_path) # Each process loads once return [data[i].sum() for i in indices] ``` ### Memory Overhead Each process has its own Python interpreter and memory space: ```python import os from multiprocessing import Pool def memory_hog(): # Each process allocates this independently big_list = list(range(10_000_000)) return sum(big_list) # 4 processes × 400MB each = 1.6GB with Pool(4) as p: results = p.map(memory_hog, range(4)) ``` On a machine with 8GB RAM, spawning too many memory-hungry processes leads to swapping and terrible performance. **Estimate before running**: ```python import sys # Estimate per-process memory data = load_typical_workload() print(f"Estimated memory per process: {sys.getsizeof(data) / 1e6:.1f}MB") # Don't spawn more processes than memory allows max_processes = available_memory_mb // memory_per_process_mb ``` ## Patterns That Work ### The Worker Pool The most common pattern — reuse processes, distribute work: ```python from multiprocessing import Pool from functools import partial def process_item(item, config): # Do CPU-intensive work result = heavy_computation(item, config) return result def main(): items = load_items() config = load_config() # partial lets us pass extra arguments worker = partial(process_item, config=config) with Pool() as p: # Default: cpu_count() processes results = p.map(worker, items) return results ``` **Choosing pool size**: - CPU-bound: `Pool(os.cpu_count())` or slightly less - Mixed I/O and CPU: Experiment, often `2 * os.cpu_count()` works - Memory-constrained: Calculate based on per-process memory ### Chunking for Small Tasks If individual tasks are small, the overhead of dispatching each one hurts: ```python # Bad: High dispatch overhead with Pool(4) as p: results = p.map(tiny_function, million_items) # Better: Chunk the work def process_chunk(chunk): return [tiny_function(item) for item in chunk] chunks = [items[i:i+1000] for i in range(0, len(items), 1000)] with Pool(4) as p: chunk_results = p.map(process_chunk, chunks) results = [r for chunk in chunk_results for r in chunk] # Or use chunksize parameter with Pool(4) as p: results = p.map(tiny_function, million_items, chunksize=1000) ``` ### Shared Memory for Large Data When multiple processes need the same read-only data, don't pickle it repeatedly: ```python from multiprocessing import shared_memory, Pool import numpy as np def create_shared_array(data): """Create a shared memory array from numpy array.""" shm = shared_memory.SharedMemory(create=True, size=data.nbytes) shared_array = np.ndarray(data.shape, dtype=data.dtype, buffer=shm.buf) shared_array[:] = data[:] return shm def worker(args): shm_name, shape, dtype, indices = args # Attach to existing shared memory shm = shared_memory.SharedMemory(name=shm_name) data = np.ndarray(shape, dtype=dtype, buffer=shm.buf) # Work with data (read-only!) result = data[indices].sum() shm.close() # Detach, don't unlink return result def main(): # Large dataset - only stored once in memory data = np.random.rand(10000, 10000) shm = create_shared_array(data) # Workers receive only small arguments work_items = [ (shm.name, data.shape, data.dtype, slice(i*1000, (i+1)*1000)) for i in range(10) ] with Pool(4) as p: results = p.map(worker, work_items) shm.close() shm.unlink() # Clean up return results ``` ### Progress Tracking Long-running multiprocessing jobs need visibility: ```python from multiprocessing import Pool from tqdm import tqdm def process_item(item): # ... work ... return result def main(): items = load_items() with Pool() as p: # imap returns results as they complete results = list(tqdm( p.imap(process_item, items), total=len(items), desc="Processing" )) return results ``` For unordered results (faster when tasks vary in duration): ```python results = list(tqdm( p.imap_unordered(process_item, items), total=len(items) )) ``` ## The Gotchas ### Zombie Processes Processes that aren't joined become zombies, consuming resources: ```python # Bad: No cleanup def bad_parallel(): processes = [Process(target=work) for _ in range(10)] for p in processes: p.start() # Function returns without joining - zombies! # Good: Always join or use context manager def good_parallel(): processes = [Process(target=work) for _ in range(10)] for p in processes: p.start() for p in processes: p.join() # Wait for completion # Better: Use Pool with context manager def better_parallel(): with Pool(10) as p: # Automatic cleanup results = p.map(work, items) ``` ### Pickling Failures Not everything pickles. Common failures: ```python # Lambda functions - don't pickle with Pool() as p: p.map(lambda x: x*2, items) # PicklingError! # Closures over unpicklable objects connection = db.connect() # Can't pickle connections def worker(item): return connection.execute(...) # Fails # Fix: Create resources inside the worker def worker(item): connection = db.connect() # Each process creates its own try: return connection.execute(...) finally: connection.close() ``` ### Global State Confusion Processes don't share memory. Global modifications don't propagate: ```python counter = 0 def increment(): global counter counter += 1 return counter with Pool(4) as p: results = p.map(lambda _: increment(), range(100)) print(counter) # Still 0! Each process has its own counter print(results) # [1, 1, 1, 1, ...] - each process counted independently ``` ### The Fork Bomb Accidentally spawning processes in a loop: ```python # Bad: Each worker spawns more workers def recursive_worker(depth): if depth > 0: with Pool(2) as p: # Spawns in each process! p.map(recursive_worker, [depth-1] * 4) # This creates 2^n processes - crashes fast ``` **Rule**: Only spawn processes from the main process: ```python if __name__ == "__main__": with Pool() as p: results = p.map(worker, items) ``` ### macOS/Windows Fork vs Spawn On Linux, `multiprocessing` uses `fork()` by default — child processes get a copy of parent memory. On macOS (Python 3.8+) and Windows, it uses `spawn` — child processes start fresh and import your module. **This breaks code that assumes fork**: ```python # Works on Linux, fails on macOS/Windows big_data = load_data() # Loaded once in parent def worker(idx): return big_data[idx] # Assumes big_data exists if __name__ == "__main__": with Pool() as p: results = p.map(worker, range(100)) # NameError on macOS! ``` **Fix**: Pass data explicitly or use initializers: ```python def init_worker(data): global big_data big_data = data def worker(idx): return big_data[idx] if __name__ == "__main__": data = load_data() with Pool(initializer=init_worker, initargs=(data,)) as p: results = p.map(worker, range(100)) ``` ## Decision Framework When to use what: | Scenario | Best Choice | Why | |----------|-------------|-----| | Web requests, file I/O | `threading` or `asyncio` | GIL releases during I/O | | 1000s of network connections | `asyncio` | Lower overhead than threads | | CPU-bound, independent tasks | `multiprocessing` | Bypasses GIL | | CPU-bound, shared large data | `multiprocessing` + shared memory | Avoids pickle overhead | | Small CPU tasks, many items | `multiprocessing.Pool` with chunking | Amortizes overhead | | NumPy/Pandas heavy computation | Often neither! | These release GIL internally | **The NumPy exception**: NumPy operations release the GIL. If your "CPU work" is mostly NumPy: ```python import numpy as np def numpy_work(arr): # These operations release the GIL return np.fft.fft(arr).sum() # Threading actually works here! from concurrent.futures import ThreadPoolExecutor with ThreadPoolExecutor(4) as e: results = list(e.map(numpy_work, arrays)) ``` ## Measuring Before Committing Don't assume parallelism helps. Measure: ```python import time from multiprocessing import Pool def benchmark(func, args, n_runs=5): times = [] for _ in range(n_runs): start = time.time() func(args) times.append(time.time() - start) return sum(times) / len(times) # Sequential def sequential(items): return [process(item) for item in items] # Parallel def parallel(items): with Pool() as p: return p.map(process, items) items = load_items() print(f"Sequential: {benchmark(sequential, items):.2f}s") print(f"Parallel: {benchmark(parallel, items):.2f}s") ``` If parallel isn't at least 2-3x faster with 4+ cores, the overhead is eating your gains. ---