Three layers, top to bottom: applications (the AI features users see), platform (model registry, serving, observability, governance), substrate (K8s, GPUs, storage). Decisions as Code runs through every layer. Centralize the decisions. Project them everywhere.
Decisions as Code specifies the standard. OPA Gatekeeper enforces it. Constraint templates as the policy definition layer. Required tags, model version pinning, namespace quotas, network policies for AI workloads. The 'specify, then verify' loop, finally complete.
Every platform starts with traceability as a goal. Most lose it by month six. The predictable failure modes, log-format drift, ID-namespace collisions, the 'we'll add structured logging later' debt, the asymmetric incentive to write logs but never read them. What survives, and why.
Argo Workflows for the long-running, branching, fan-out AI ops pipelines that don't fit a CI runner. RAG indexing jobs, fine-tuning runs, eval suite execution. WorkflowTemplates as the Decisions as Code surface, same pipeline shape, different inputs per project.
Most teams treat HIPAA as a feature you bolt on at the end. The teams who've actually shipped under it know it shapes the data model, the audit log, the auth model, and the deletion semantics, before you draw a single arrow on the diagram.
The GPU-as-a-service market consolidated through 2025. The crowded field of neoclouds is smaller than it was; the surviving providers are more differentiated. Worth a survey of who's still around and what they actually cost.
The NVIDIA device plugin, node taints and tolerations, the multi-tenancy reality of one GPU per pod vs MIG vs vGPU. The scheduling primitives that work at homelab scale on a single 1080 Ti and the ones that change shape entirely at fleet scale.
The default for AI in 2025 is hosted. The hosted default makes sense for some workloads and is structurally wrong for others. Worth being explicit about which workloads belong on-prem and why the inertia is hard to break.
Latency, token throughput, cost-per-request, queue depth, drift. Prometheus for the metrics, Loki for the prompts and responses with PII discipline, Tempo for the tracing across agent calls. The dashboard JSON itself as a Decisions as Code surface, projected per-environment.
KServe, vLLM, Triton, BentoML, four different answers to the same question. Each layer trades operational complexity for serving features. Worth being concrete about which layer is right for which workload, tested on a single GTX 1080 Ti.
Qdrant vs Weaviate vs Milvus on K8s. The foundation question for retrieval. StatefulSets, persistent volumes, replication, the operational reality. RAG indexing patterns at homelab scale on engine-01, and the decisions that change shape at fleet scale.
Helm values.yaml is where Decisions as Code lives in Kubernetes. Centralized business decisions, schema-validated, composed via library charts, projected into every workload. The approach is the contribution. The tool changed.
