airflow
The log level that only looked configurable
The Stackable Airflow operator generates Airflow’s logging config from your CRD. You set a level, it renders the Python. An open issue asked for one specific thin...
cat article →
$
whoami
Notes from the platform layer
Kubernetes, AWS, ArgoCD, Karpenter, and the Grafana stack, written up right after I figure something out the hard way.
$ ls ./writing
argo-cd
The cache was right there, just not reachable
Argo CD’s notification controller had a performance regression in v3.4.x: with a lot of applications and triggers, it was making a large number of API calls per r...
cat article →
alloy
Letting pyroscope.write decide what to do with a 429 merged
An Alloy issue asked for something small and reasonable: a way to tell pyroscope.write not to retry when the backend returns HTTP 429. By default it retries rate-...
cat article →
argo-cd
Argo CD resolved my Helm chart as a git repo
Someone reported an Argo CD regression that only shows up in a fairly specific shape of Application: a multi-source app that combines an untyped Helm chart source...
cat article →
tempo
The label that wrapped merged
Someone configured the Tempo data source to show resource.k8s.cluster_name and resource.k8s.namespace as static search fields in Explore, and the labels overflowe...
cat article →
istio
The CNI pod was waiting for a file only it could write
A node reboots. Every pod that tries to come back up fails to get a network:
cat article →
istio
Istio built the validation context, then refused to authorize it
A Gateway API listener using OPTIONAL_MUTUAL TLS termination was resetting every connection. The same listener configured as MUTUAL, with the identical secret in ...
cat article →
istio
A CIDR and a list of /32s should be the same thing, but one broke the sidecar
Someone reported a ServiceEntry that broke their sidecar in a way that made no sense at first. Write the addresses one way and the pod starts. Write the same addr...
cat article →
argo-cd
Argo CD could not lock the ref, so it gave up merged
A maintainer filed an Argo CD bug that caught my eye because it lived right at the seam between a distributed system and a plain old git command. The source hydra...
cat article →
alloy
A secret got printed as a Go struct into my scrape target
Someone reported an Alloy config where they read a value out of a file and used it as a scrape target address. Instead of the address, the target came out as {tru...
cat article →
external-dns
ExternalDNS created the AAAA record, then forgot it existed merged
ExternalDNS is a reconcile loop. Every minute it reads the records that exist at your DNS provider, reads the records your Kubernetes resources say should exist, ...
cat article →
mimir
When zone-aware Mimir pointed its ingress at a service that was never created
Someone hit a sharp edge in the Mimir Helm chart: turn on zone-aware replication for the alertmanager and the ingress starts returning errors, because it routes t...
cat article →
loki
Running loki-canary to catch the logs Loki silently drops merged
The failure mode that scares me most with a logging stack is not the loud one. A crashed ingester pages you. The quiet one — Loki accepting writes, returning 200s...
cat article →
karpenter
Karpenter provisions, kube-scheduler binds: why your pod landed somewhere surprising
A pod is pending. Karpenter launches a node for it. The node comes up, the pod schedules, and then a minute later that node is gone again, empty, consolidated awa...
cat article →
mimir
What it takes to integration-test a Mimir compactor (and why I trust the test)
I added an integration test to Grafana Mimir’s compactor recently, and the interesting part was not the test itself — it was how much the project’s test harness f...
cat article →
alloy
A timestamp was quietly busting my Go build cache in Alloy merged
Every time I rebuilt Grafana Alloy locally, it relinked the whole binary even when I had changed nothing. For a project this size that is a real chunk of wall-clo...
cat article →
loki
Loki retention and object store lifecycle policies: the rule that quietly corrupts your bucket merged
If you run Grafana Loki on object storage, there is a sentence in the retention docs that is easy to read past and expensive to get wrong:
cat article →