I started by trying to prove the comment was wrong. It looked wrong. The CRD has a loggers map, the docs show an airflow.task entry in it, and setting that entry does change what you see in the Airflow UI. So the feature seemed to already exist and nobody had noticed.

Handlers filter, loggers gate

It does work, but not as the same thing. Python’s logging has two thresholds on the path from a log call to an output, and they are not interchangeable.

logger = logging.getLogger('airflow.task')   # threshold 1: the logger
handler = logger.handlers[0]                 # threshold 2: the handler

A record is dropped at the logger before any handler is consulted. If it survives, each handler applies its own level independently. So the logger is a gate on the whole path, and a handler level filters one destination.

That distinction is the entire issue. Airflow’s task handler is what feeds the log view in the web UI. The other handlers write to files, which is what the Vector agent ships off the node. If you raise the airflow.task logger to suppress noise in the UI, you have also stopped those records from ever reaching the files. The thing people actually wanted, from a second issue on the same topic, was DEBUG in the files and in the log aggregator while the UI stays readable at INFO. A logger level cannot express that. Only a handler level can.

So the comment in the source was right and my read of it was wrong. Good thing to establish before writing a line of code.

Building the obvious thing

The CRD shape looked like the blocker. Levels come from a struct with named fields:

pub struct AutomaticContainerLogConfig {
    pub loggers: BTreeMap<String, LoggerConfig>,
    pub console: Option<AppenderConfig>,
    pub file: Option<AppenderConfig>,
}

console and file are fields. loggers is a map, so any key you invent works without changing the struct, which is exactly why the airflow.task workaround needed nothing upstream. But a task field next to console and file is a struct change, and that struct lives in a shared crate. I checked: thirteen operators consume it.

I built it anyway. Added the field, added it to the hand-written merge implementation, updated twenty-five struct literals the compiler pointed me at, regenerated the CRDs, wrote the tests, opened the upstream PR, and wired the operator side to it. It worked. I verified it end to end by rendering the generated Python and running it through dictConfig, because a test asserting on generated source does not prove the level takes effect at runtime.

The reply that made it smaller

A maintainer answered roughly: I would rather not add a field to every product’s CRD when only Airflow uses it. Then he offered an alternative, and noted that reading the docs he would have expected airflow.task in loggers to already be exactly that knob.

Which is the point I had walked straight past. If the level has to come from somewhere, and loggers already accepts any key, and the docs already tell people to put airflow.task there, then the value is already in the CRD. Nothing needs adding. What needed changing was what the operator does with it: send it to the handler instead of to the logger.

fn task_log_level(log_config: &AutomaticContainerLogConfig) -> LogLevel {
    log_config
        .loggers
        .get(TASK_LOGGER)
        .map(|logger| logger.level)
        .unwrap_or(LogLevel::INFO)
}

fn task_logger_level(log_config: &AutomaticContainerLogConfig) -> LogLevel {
    cmp::min(LogLevel::INFO, task_log_level(log_config))
}

The second function is the part worth explaining. The handler gets whatever you asked for. The logger gets the lower of INFO and your level, never the higher. Asking for DEBUG has to open the logger up too, or it would drop the records before the handler could pass them. Asking for ERROR must not raise the logger, because the console and file handlers still want everything from INFO up. Clamping in one direction is what lets one knob mean the right thing in both.

He also offered a second option: just reuse the file level for the task handler, as a close-enough approximation. I pushed back on that one. It collapses the exact distinction the issue is about, and it would silently start showing DEBUG in the UI to anyone who has file set to DEBUG today. Agreeing with the reviewer on the good idea does not mean agreeing with all of them.

What the rewrite cost

The CRD version touched two repositories, needed a release of the shared crate before it could merge, changed a struct thirteen operators depend on, and added 88 keys to the generated CRD schema. The rewrite touches three files in one repository and changes no schema at all. Same feature.

There is one honest wart, and I put it in the PR rather than in a footnote. This reinterprets a key that already does something. Today, raising airflow.task to ERROR suppresses records everywhere; afterwards it only quietens the UI. That is the improvement, and it is also a behaviour change, and someone relying on the old side effect to cut log volume would notice.

The takeaway

I spent most of the effort on the version that got thrown away, and the reason is not that the design was wrong. It is that I accepted the framing in my own head, that a handler level needs a handler field, and never asked whether the value could come from somewhere that already existed. The maintainer did not know Airflow’s logging internals better than I did by then. He just had not adopted my constraint.

Worth noticing too that he arrived at the answer by reading the documentation and saying what he expected to be true. When a reviewer tells you what they assumed the feature already did, that is not confusion to correct. It is a fairly direct hint about where the feature belongs. The change is in stackabletech/airflow-operator#829.

unable to resolve git revision : failed to list refs: repository not found

The chart repository is not a git repo, so of course listing git refs against it fails. The real question is why Argo CD was trying to list git refs against a Helm chart URL at all. It had all the information it needed to know this was a Helm source.

The guard that used to short-circuit

UpdateRevisionForPaths in the reposerver decides, per source, whether it needs to resolve a git revision for the manifest-generate-paths optimization. A non-git source with no ref sources has nothing to resolve, so the function returns early:

if repo.Type != "git" && len(request.RefSources) == 0 {
    return &apiclient.UpdateRevisionForPathsResponse{}, nil
}

And the git work itself is gated the same way:

if repo.Type == "git" {
    // resolve the git revision...
}

For an untyped Helm chart source with ref sources, this is exactly the seam where things go wrong.

Normalize turned “unset” into “git”

The regression came in with one added line. Someone put a repo.Normalize() call at the top of the function to tidy up the request:

repo = repo.Normalize()

Normalize is helpful in most places, but it does one thing that matters here: when the repository type is unset, it coerces it to the default, which is git. An untyped Helm chart source (no explicit type, chart repo not registered as a repository secret) has an empty type. After Normalize, that empty type is git.

Now walk the two guards again. The source has ref sources, so the early return does not fire. Its type is now git, so it enters the git branch and tries to resolve a git revision against the Helm chart URL. repository not found. Before Normalize was added, the empty type was not git, the early return fired, and the function did the right thing by doing nothing.

The bug is not in Normalize. It is in treating “unset” as “definitely git” at a decision point where unset could just as easily mean Helm or OCI.

The fix: decide git-ness before normalizing

Instead of letting Normalize guess, I capture whether the source is really git before it runs, and fall back to the application source type when the repository type is genuinely unset:

isGitSource := repo.Type == "git"
if repo.Type == "" {
    isGitSource = request.ApplicationSource == nil ||
        (!request.ApplicationSource.IsHelm() && !request.ApplicationSource.IsOCI())
}

Then both guards key off isGitSource instead of repo.Type. An untyped source is only treated as git when the application source is not Helm and not OCI. A genuine git repo that happens to use Helm value overrides (a git source with Helm: set but no Chart:) stays on the git path, because IsHelm keys on the chart, not on the presence of Helm value config. That distinction is the whole reason the fix is safe.

Proving it before trusting it

The test is the part I never skip. I added a case to TestUpdateRevisionForPaths for an untyped Helm chart source carrying ref sources, and wired the git client mock so that any git resolution returns failed to list refs: repository not found. That means the test only passes if the git path is never entered. On master the new case fails with the exact error from the issue; with the fix it passes and does zero external cache work. A test that reproduces the reported error message before the fix, and goes green after, is the one I trust.

An aside on trusting autofixes

While iterating on this PR I accepted a Copilot autofix suggestion in the GitHub web UI, and it committed a dead-assignment lint error (SA4006) straight into the branch, plus a bit of working-tree contamination that leaked unrelated notification files into the diff. Accepting a web autofix commits it as-is, with no local gofmt, lint, or build in front of it. I caught both and cleaned them up, but the lesson stuck: an autofix is a suggestion, not a verified change. Run it through the same gate as anything you write by hand.

The takeaway

Normalize-style helpers that fill in defaults are convenient right up to the moment a downstream decision depends on the difference between “unset” and “the default value.” Coercing empty-to-git erased exactly the signal the guard needed. When a default is applied, ask whether anything later treats the defaulted value as if the user had chosen it, because that is where the surprises live. The change is in argoproj/argo-cd#28904.

The behavior was hardcoded

The retry decision lives in shouldRetry, and 429 was baked in alongside 408:

func shouldRetry(err error) bool {
    if errors.Is(err, context.DeadlineExceeded) {
        return true
    }

    var writeErr *PyroscopeWriteError
    if errors.As(err, &writeErr) {
        status := writeErr.StatusCode
        if status == http.StatusTooManyRequests || status == http.StatusRequestTimeout {
            return true
        }
        return status >= http.StatusInternalServerError
    }
    // ...
}

There is nothing wrong with the logic. It just was not a choice you could make from config.

Following the precedent instead of inventing one

Because two sibling components already solved this, the interesting part is not the mechanism, it is matching their shape so the option feels native to Alloy. I added the field with the same name, same type, and the same default those components use:

RetryOnHTTP429 bool `alloy:"retry_on_http_429,attr,optional"`

Default true, set in GetDefaultEndpointOptions, so every existing config keeps behaving exactly as it does today. Then shouldRetry takes the flag and consults it only for 429:

func shouldRetry(err error, retryOnHTTP429 bool) bool {
    // ...
    if status == http.StatusTooManyRequests {
        return retryOnHTTP429
    }
    if status == http.StatusRequestTimeout {
        return true
    }
    return status >= http.StatusInternalServerError
}

Splitting the combined 429 || 408 check into two is the whole change in spirit: 429 becomes configurable, while 408 and 5xx stay unconditionally retried, because those are not what the user asked to control. Both call sites, the push path and the ingest path, pass ec.options.RetryOnHTTP429 through.

Proving it before trusting it

I tested this at two levels. shouldRetry is pure, so I tested it directly: a 429 is retried with the flag on and dropped with it off, while 408, 503, and 400 return the same answer regardless of the flag. That last part matters as much as the feature itself, since it proves the new knob only touches 429 and leaves the other status codes alone.

Then, because a unit test of a predicate does not prove the flag is actually wired into the retry loop, I added two suite tests that stand up a server always returning 429 and count the requests. With retry_on_http_429 disabled the profile is sent exactly once. With it enabled the request is retried up to MaxBackoffRetries times. Asserting on the observed request count is what convinces me the flag reaches the loop and not just the predicate.

What review caught

The reviewer, agreeing with an automated review comment, pointed out something I had missed: pyroscope.write does not only talk HTTP. The push v1 path speaks Connect, and there a rate limit arrives as connect.CodeResourceExhausted rather than an HTTP 429. My first version left that path always retrying, so the new flag was only half honoured depending on which API the profile went out through. Same rate limit, same user intent, two code paths, and I had only found one. The fix was to gate the Connect code the same way, correct the docs to say plainly that 408 is always retried regardless of the flag, and add a test that drives fanOutClient.Append through a real Connect test server returning CodeResourceExhausted, because a unit test on the predicate would never have caught the gap. Worth remembering next time I add a knob to a component: find every transport the option is supposed to apply to before claiming it is done.

The takeaway

When a project already has an established convention for a thing, the best contribution is often the least original one: find the precedent, match its name, its default, and its behavior, and make the new option indistinguishable from the ones that came before. A user who already knows retry_on_http_429 from loki.write should not have to learn anything new to use it on pyroscope.write. Consistency across siblings is a feature. The change is in grafana/alloy#6763, now merged.

What made this one interesting is that the fix was not “add a cache.” The cache already existed. It just could not be reached from the code that needed it.

Two ways to fetch the same AppProject

The notification controller already runs a synced AppProject informer. There is even a helper, getAppProj, that reads projects straight out of that informer’s indexer. No API call, just a local cache lookup on an already-synced store.

But the template variables are built in a different package, util/notification/settings, by getAppProjectForTemplate. That function had no access to the informer, so it did the only thing it could:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

appProjectObj, err := argocdService.GetAppProject(ctx, projectName, namespace)

A live Get, on every trigger evaluation. The fast path existed a package away; the settings package simply had no handle to it.

Threading the cache in without forcing it on everyone

The fix is to give the settings package an optional way to read from a cache, and let the controller supply one backed by its existing informer. I added a small function type:

// AppProjectGetter retrieves an AppProject from a local cache, avoiding a live
// API call on every trigger evaluation. It returns a nil object without an
// error when the AppProject is not present in the cache.
type AppProjectGetter func(namespace, name string) (*unstructured.Unstructured, error)

getAppProjectForTemplate prefers it when present, and only falls back to the live lookup when it is nil:

if appProjectGetter != nil {
    appProjectObj, err := appProjectGetter(namespace, projectName)
    if err != nil {
        log.WithFields(logFields).Warnf("Failed to get AppProject for notification template: %v", err)
        return nil
    }
    if appProjectObj == nil {
        return nil
    }
    return appProjectObj.Object
}

// ... otherwise the previous live Get, unchanged.

The notification controller builds a getter that reads the informer’s indexer:

func newAppProjectGetter(appProjInformer cache.SharedIndexInformer) settings.AppProjectGetter {
    return func(namespace, name string) (*unstructured.Unstructured, error) {
        projObj, exists, err := appProjInformer.GetIndexer().GetByKey(fmt.Sprintf("%s/%s", namespace, name))
        // ... return the unstructured project, or nil if missing
    }
}

Callers that have no local cache, like the API server and the CLI, pass nil and keep the exact behavior they had before. So the controller, the hot path, gets the cache; everyone else is untouched.

The honest note about the signature

GetFactorySettings is an exported function, and this change adds an AppProjectGetter parameter to it. I updated every in-tree caller, but if some out-of-tree code calls that function, this is a breaking change. I flagged that plainly in the PR body rather than hiding it. It seems like a reasonable cost for a real performance fix, but it is the maintainers’ call, and they should get to make it with the fact in front of them, not discover it later.

Proving it before trusting it

The test I care most about here does not assert a return value, it asserts an absence. TestGetAppProjectForTemplateUsesCache provides a getter and a mock Service with no GetAppProject expectation configured at all. If the code ever fell back to the live lookup, the mock would fail the test. So the test proves the cache path is taken and the API is never touched, which is the whole point of the fix. A second test covers a cache miss returning nil without falling back. I confirmed both by reverting the cache branch and watching them fail.

The takeaway

Sometimes the fix for “this does too much I/O” is not a new cache or a new client, it is a wire. The right data was already sitting in memory, synced and ready; it just was not plumbed to the caller that needed it. When you see a hot path doing live lookups, check whether something nearby already has the answer cached before you reach for a bigger hammer. The change is in argoproj/argo-cd#28905.

Where the label comes from

Static search fields do not store a label. They store a tag, and the label is computed from the tag every time the editor renders. That computation lives in filterTitle:

export const filterTitle = (f: TraceqlFilter, lp: TempoLanguageProvider) => {
  // Special case for the intrinsic "name"
  if (f.tag === 'name') {
    return 'Span Name';
  }
  // Special case for the resource service name
  if (f.tag === 'service.name' && f.scope === TraceqlSearchScope.Resource) {
    return 'Service Name';
  }
  return startCase(filterScopedTag(f, lp));
};

The last line is the general case, and it is where the length comes from. filterScopedTag prepends the scope, so the tag becomes resource.k8s.cluster.name, and then lodash startCase runs over it. I had a guess about what that produces and I was wrong, so I ran it:

startCase('resource.k8s.cluster.name')  // "Resource K 8 S Cluster Name"

startCase splits on the dots and on the digit boundary inside k8s, capitalizes each piece, and joins with spaces. So a perfectly normal Kubernetes attribute renders as Resource K 8 S Cluster Name. That string is long, and it is also slightly mangled — there is no label you could choose for a k8s attribute that startCase would not stretch out.

Where the length hurts

That title is handed to an InlineField with a fixed label width:

<InlineField label={label} labelWidth={28} grow tooltip={tooltip}>

labelWidth={28} is a fixed column. A short label fits; Resource K 8 S Cluster Name does not, so it wraps, and once it wraps it pushes into the row below. The form has no idea the label is auto-generated and possibly too long — it just lays out what it is given.

So there are two real problems stacked on top of each other: the label is derived rather than chosen, and the column it goes into cannot flex. The issue asked to fix the first one.

The fix: let the field carry a name

The change is to give a static field an optional label and prefer it when it is set. One field on the filter model:

export interface TraceqlFilter {
  // ...
  /**
   * A custom label to display for the filter in the search editor.
   * When set, it overrides the auto-generated title derived from the tag.
   */
  label?: string;
}

One early return at the top of filterTitle, before any of the special cases:

if (f.label) {
  return f.label;
}

And an input on the data source configuration page where you define these static fields, so you can actually type the name in. That input is gated behind a showLabel prop threaded through the components, so it shows up where you configure the fields and not in the query editor itself. The query editor needs no changes at all: it already renders through filterTitle, so the moment a label exists, the editor picks it up.

The whole thing is additive. A field with no label behaves exactly as before — same startCase, same special cases for name and service.name. Nothing in the stored query changes. You opt in per field, or you ignore it and keep the old behavior.

On the other fix

There is an earlier comment on the issue suggesting the real fix is to drop the fixed label width, or let the row grow so wrapping does not overflow. That is a good idea and I said so in the PR. It is a CSS fix to the second problem — the column that cannot flex — and it would help every label, including the auto-generated ones, without anyone having to type anything. It is complementary, not competing.

I went with the rename because it is what the issue asked for, and because a later commenter confirmed a custom label would solve their case. A flexible column makes a long name fit; a custom label lets you decide the name should not be long in the first place. Resource K 8 S Cluster Name becoming Cluster is a better outcome than Resource K 8 S Cluster Name fitting more gracefully. Both can ship; neither blocks the other.

Checking the derivation, not just the override

The tests cover the new path — a label wins, and it wins even over the name special case — but the one I cared most about is the fallback, because that is the assertion I had wrong in my head. The test pins filterTitle with no label to Resource K 8 S Cluster Name, the actual startCase output, not the cleaned-up version I assumed. If someone changes how titles are derived, that test fails and tells them the visible label changed. Writing down the ugly real string is more useful than writing down the pretty wrong one.

The change is in grafana/grafana-tempo-datasource#205, now merged.

# works: eight explicit hosts
addresses:
- 10.80.68.0/32
- 10.80.68.1/32
# ... through .7/32

# fails: one CIDR for the same eight addresses
addresses:
- 10.80.68.0/29

With the /29, the sidecar logs a listener it cannot accept:

error adding listener: 'virtualOutbound' has duplicate address '0.0.0.0:15001' as existing listener
Listener rejected
Envoy proxy is NOT ready: lds updates: 0 successful, 1 rejected

One rejected listener fails the whole LDS update, so the proxy never becomes ready and the pod is stuck. The clue is in the address: 0.0.0.0:15001. Port 15001 is istio’s own outbound capture port, the virtualOutbound listener. Somehow this ServiceEntry was generating a second listener on top of it.

Why the two forms diverge

When istio builds an outbound listener for a service, it has to decide what address to bind. If the service has a concrete IP, it binds that IP. If the address is a CIDR, you cannot bind a range, so it falls back to the wildcard 0.0.0.0 and adds a filter chain match for the CIDR instead:

if !strings.Contains(svcListenAddress, "/") {
    listenerOpts.bind.binds = append([]string{svcListenAddress}, svcExtraListenAddresses...)
} else {
    // Address is a CIDR. Fall back to 0.0.0.0 and filter chain match
    listenerOpts.bind.binds = actualWildcards
    listenerOpts.cidr = ...
}

That is the whole divergence. Eight /32 entries are concrete addresses, so each binds 10.80.68.x:15001, none of which collide with anything. One /29 is a CIDR, so it binds 0.0.0.0:15001, which is precisely where virtualOutbound already lives. Same eight IPs, completely different listener address, and only one of them steps on a reserved port.

There is a guard, and the CIDR walked around it

Istio is not naive about service ports clashing with its own. There is a check, conflictWithReservedListener, that knows about 15001 and the other reserved ports, and the listener-building loop already skips services that conflict. So why did this one slip through? Look at how the check bails out early:

if bind != "" {
    if bind != wildcard {
        return false
    }
} else if !protocol.IsHTTP() {
    // bind unspecified and not HTTP: assume it will bind its own address
    return false
}

The check runs before the CIDR fallback has happened. At that point the bind is still empty and the protocol is TLS, not HTTP, so it takes the early return false: no conflict, this service will bind its own address. That assumption is true for a normal service. It is false for a CIDR service, which is about to be rewritten to the wildcard a few lines later. The guard made its decision based on a bind address that the code then changed out from under it.

Fix it where the decision actually becomes true

The conflict only becomes real at the moment the listener falls back to the wildcard, so that is where I put the re-check. Right after the CIDR branch sets the wildcard bind:

listenerOpts.bind.binds = actualWildcards
listenerOpts.cidr = append([]string{svcListenAddress}, svcExtraListenAddresses...)
// Now that we are binding to the wildcard address, this listener can
// collide with a reserved listener ... re-check here and skip the port
// if it conflicts.
wildcard := actualWildcards[0]
if canbind, knownlistener := lb.node.CanBindToPort(...); !canbind {
    ...
    return
}

Now a CIDR ServiceEntry on a reserved port is skipped, exactly like a concrete-address service on that port already was. It reuses the same CanBindToPort the rest of the loop uses, so the behaviour matches rather than inventing a second notion of what conflicts.

Proving it before trusting it

Before writing the fix I wrote the reproduction, because I wanted to see the bad listener with my own eyes, not just trust the report. The test harness has a ValidateListeners helper that already flags duplicate addresses, so a throwaway test that built a CIDR ServiceEntry on port 15001 was enough: it produced 0.0.0.0_15001 and failed, the bug, in a unit test, in under two seconds.

Then I turned that into a proper test with three cases: a CIDR service on 15001 and on 15006 must produce no listener, and a CIDR service on an ordinary port must still get its wildcard listener. That last case matters as much as the first two, because the easy wrong fix is to drop CIDR wildcard listeners too eagerly and quietly break the normal case. I reverted the fix and watched the two reserved-port cases fail while the ordinary-port case kept passing, then put it back and all three passed.

(One small note to my future self: I first wrote that throwaway test with a shell heredoc, and the colorized shell snuck an escape byte into the .go file, which the compiler rejected as an illegal character. Write Go from an editor, not a heredoc.)

The takeaway

When a value gets rewritten partway through a function, any check that ran before the rewrite was answering a question about a state that no longer exists. The conflict guard here was correct for the bind it saw and wrong for the bind the code produced. The fix was not to make the early check smarter about the future, but to ask the question again at the point where the answer actually settles. And the reproduction-first habit paid for itself twice: it confirmed the bug was real before I touched anything, and it caught the over-eager version of the fix that would have broken ordinary CIDR services. The change is in istio/istio#60722.

warn ads proxy <pod>.<ns> attempted to access unauthorized certificates <cred>-cacert: cross namespace secret reference requires ReferenceGrant

The cert secret holds tls.crt, tls.key, and ca.crt. Istio splits that into two SDS resources: the cert itself, and a -cacert resource for the CA bundle that validates client certificates. The handshake needs both. istioctl pc secret showed the -cacert resource stuck in WARMING, never delivered, because SDS said the proxy was not allowed to have it.

Two places decide, and they disagreed

The thing that makes this bug tidy is that the listener is built from one set of rules and authorized by another, and the two had drifted apart.

When istiod builds the TLS context for the listener, it wires the -cacert validation context for both mutual modes. That code is explicit about it:

// If tls mode is MUTUAL/OPTIONAL_MUTUAL, create SDS config for gateway/sidecar to fetch certificate validation context
if tlsOpts.Mode == networking.ServerTLSSettings_MUTUAL || tlsOpts.Mode == networking.ServerTLSSettings_OPTIONAL_MUTUAL {

So the listener that gets pushed to Envoy references <cred>-cacert. Envoy asks SDS for it. SDS checks the request against the set of credentials the proxy is allowed to see, VerifiedCertificateReferences, and that set is built somewhere else entirely, in mergeGateways:

verifiedCertificateReferences.Insert(rn)
if s.GetTls().GetMode() == networking.ServerTLSSettings_MUTUAL {
    verifiedCertificateReferences.Insert(rn + credentials.SdsCaSuffix)
}

The plain cert reference is always added. The -cacert suffix is added only for MUTUAL. For OPTIONAL_MUTUAL the listener asks for a credential that the authorization step never put on the allow-list. One half of the code believed in OPTIONAL_MUTUAL, the other half had never heard of it.

That also explains why a ReferenceGrant did not help. There are two branches in mergeGateways, same-namespace and explicitly-allowed-by-policy, and both gated the -cacert insert on MUTUAL. No grant could open a door that neither branch was willing to add.

The fix is to make both sides agree

The authorization step should authorize the same modes the listener-build step builds for. So both inserts now match the expression already used when constructing the context:

if s.GetTls().GetMode() == networking.ServerTLSSettings_MUTUAL ||
    s.GetTls().GetMode() == networking.ServerTLSSettings_OPTIONAL_MUTUAL {
    verifiedCertificateReferences.Insert(rn + credentials.SdsCaSuffix)
}

This is not a new idea in the codebase, which is what gave me confidence it would be accepted. The exact same bug existed for MUTUAL once, where the validation context was built but the reference was not authorized, and it was fixed the same way. OPTIONAL_MUTUAL is just the mode that got left behind when that earlier fix went in. The change is the analog of the one already merged, applied to the mode nobody circled back to.

Proving it before trusting it

The merge logic has a table-driven test that counts how many references end up authorized. The existing mutual-cred-in-allowed-ns case expects two: the cert and its -cacert. I added an optional-mutual-cred-in-allowed-ns case that is identical except for the mode, and it expects two as well.

Then the step I never skip: I reverted the one-line fix and ran the test. The new case failed with Expected: 2 Got: 1, which is the bug stated in numbers, the cert authorized and the CA bundle not. Put the fix back and it reads two. A test you have not watched fail is decoration, not a guard.

The takeaway

When one part of a system decides what to build and a different part decides what to permit, every new case has to be taught to both, and a list-of-known-modes is exactly the kind of place where one side gets updated and the other does not. The smell here was specific: the same enum, OPTIONAL_MUTUAL, named in the context-building code and absent two functions away in the authorization code. Grepping for the places a mode is mentioned, and asking why one site has it and another does not, is often faster than reading the data flow end to end. The change is in istio/istio#60720.

failed to setup network for sandbox ...: ... istio-cni-kubeconfig: no such file or directory

The file that is missing, istio-cni-kubeconfig, is written by the istio-cni node agent. So the agent needs to start, write the file, and then everything else can use it. The catch is that the agent is also a pod, and bringing up a pod runs the CNI plugin, which wants to read that same kubeconfig. The thing that creates the file is blocked behind the file it creates. Classic deadlock, and a reboot is the moment it bites because nothing has written the file yet.

Someone already saw this coming

What is interesting is that the code already had the escape hatch. The CNI plugin’s CmdAdd does a preemptive check: before it builds a Kubernetes client, it asks “is the pod I am being called for actually my own node agent?” If so, it returns early and never touches the kubeconfig. The comment spells out the reasoning exactly:

// Preemptively check if the pod is a CNI pod.
// It is possible that the kubeconfig is not available if it hasn't been written yet
// by the CNI pod ... to avoid a deadlock on the kubeconfig when the k8s client is
// unnecessary to process the CNI add event for the CNI pod itself.

That is the deadlock, named and defended against. The detection is a small helper that checks whether the pod name has the istio-cni-node- prefix and lives in the agent’s own namespace. If it does, skip the client entirely and let the pod through.

The guard only ran in one mode

Here is the line that turned a fixed bug back into an open one:

if conf.AmbientEnabled {
    k8sArgs := K8sArgs{}
    ...
    if isCNIPod(conf, &k8sArgs) {
        return pluginResponse(conf)
    }
}

The whole preemptive check sits inside if conf.AmbientEnabled. So the deadlock guard only fires for clusters running ambient mode. A plain sidecar-mode cluster never runs it, and right after this block the code does what it always does:

client, err := newK8sClient(*conf)

newK8sClient runs unconditionally, in every mode. It is the call that blocks on the missing kubeconfig. So the guard protects against a deadlock that can happen in both modes, but only in the one mode it happens to be nested under. The reasoning in the comment is not specific to ambient at all, it is about the agent pod needing a client it is not yet able to build. The condition around it was narrower than the problem it described.

The fix is to stop nesting it

The check has nothing to do with ambient mode, so it should not be gated by it. I lifted it out:

k8sArgs := K8sArgs{}
if err := types.LoadArgs(args.Args, &k8sArgs); err != nil {
    return fmt.Errorf("failed to load args: %v", err)
}
if isCNIPod(conf, &k8sArgs) {
    return pluginResponse(conf)
}

Now the agent pod is recognised and skipped before the client is built, regardless of mode. The conditional was load-bearing for ambient only by accident; the behaviour it guards belongs to both.

Proving it before trusting it, on Linux

The CNI package is full of Linux network-namespace code, so the tests do not even compile on my Mac. I ran them in a golang:1.26 container instead, which is worth knowing if you ever touch this corner of istio: a green run locally on macOS is not available to you here, you have to get to Linux somehow.

The test calls CmdAdd for a pod named like the node agent, in both ambient and sidecar mode, and asserts it returns without error. The mock kubeconfig does not exist, so if the code reaches newK8sClient the test fails. I ran it both ways. With the fix, both modes pass. With the guard still nested under AmbientEnabled, the ambient subtest passes and the sidecar subtest fails, reaching the client and erroring out, which is the deadlock reproduced in a unit test. The two-mode split in the result is the bug, drawn precisely.

The takeaway

A fix that is correct but scoped too narrowly is its own kind of bug, and it is harder to spot than a missing fix because the defensive code is right there, commented, clearly intended. The tell was the mismatch between the comment and the condition: the comment justified the check in mode-neutral terms while the if wrapped it in one mode. When a guard’s rationale is broader than the branch it lives in, the branch is usually wrong, not the rationale. The change is in istio/istio#60721.

The setup that breaks

The config reads a file and feeds its content into a discovery.relabel target:

local.file "service_ip" {
  filename  = "/etc/alloy/service_ip"
  is_secret = true
}

discovery.relabel "backend" {
  targets = [{
    __address__ = local.file.service_ip.content,
  }]
}

You expect __address__ to be the file’s contents. You get {true 10.0.0.1:9090}. Flip is_secret to false and it becomes {false 10.0.0.1:9090}. That true/false leading the value is the tell.

Why a string turns into a struct

local.file does not export its content as a plain string. It exports an alloytypes.OptionalSecret, which is a small struct:

type OptionalSecret struct {
	IsSecret bool
	Value    string
}

That is so Alloy can carry “this might be sensitive” alongside the value and redact it when rendering config. It is a capsule type, not a string.

Targets are built in discovery’s Target.ConvertFrom. When it walks the map of target values, it handles strings and then falls back to formatting anything else:

case v.IsString():
	strValue = v.Text()
case v.Reflect().CanInterface():
	strValue = fmt.Sprintf("%v", v.Reflect().Interface())

An OptionalSecret is not a string, so it hits the second branch, and %v on a struct gives you exactly {true 10.0.0.1:9090}. The boolean is IsSecret, the rest is Value. The code was never wrong about the value — it just printed the whole box instead of what was inside it. And convert.nonsensitive() worked around it precisely because it turned the capsule back into a plain string before it reached this branch.

The fix: unwrap the box

The two capsule types that wrap a string here are OptionalSecret and Secret. The fix is to pull the underlying string out of them before the %v fallback:

case v.Reflect().CanInterface():
	switch raw := v.Reflect().Interface().(type) {
	case alloytypes.Secret:
		strValue = string(raw)
	case alloytypes.OptionalSecret:
		strValue = raw.Value
	default:
		strValue = fmt.Sprintf("%v", raw)
	}

I thought about doing this generically — ask any capsule to convert itself to a string — but OptionalSecret deliberately refuses that conversion when it actually holds a secret, so a generic path would either drop the value or fall back to the struct again. Naming the two types keeps the behavior explicit, which is what you want in code that builds scrape targets.

One thing worth being honest about: when the value really is a secret, this now puts the real string into the target label. That is not a new leak. The old code already put that same string into the label — just wrapped in {true ...}. If you put a secret in a target, it ends up in the target either way; this change only stops mangling it. Putting a secret in a target address is arguably the wrong move, but that is a separate conversation from “the value should not be a struct literal.”

Proving it before trusting it

The part I never skip: I wrote the test, then reverted the fix to watch it fail. With the fix gone the test produces {true 10.0.0.1:9090} — the exact string from the bug report — and with the fix back it produces 10.0.0.1:9090. A test that has never been seen to fail is not a regression guard, it is decoration. Three cases cover a non-secret optional, a secret optional, and a plain Secret, all run through Alloy’s real evaluator so it exercises the same path the config does.

The takeaway

fmt.Sprintf("%v", x) is a fine last resort and a bad default. The moment x can be a struct that wraps the thing you actually wanted, %v will happily hand you the wrapper. When a value passes through a type system that boxes things — secrets, optionals, capsules — the conversion back out is a real step, not something to leave to default formatting. The change is in grafana/alloy#6605.

 ! [remote rejected]     refs/notes/hydrator.metadata -> refs/notes/hydrator.metadata (cannot lock ref 'refs/notes/hydrator.metadata': is at 4771346... but expected 30b2c90...)
error: failed to push some refs to '...'

To reproduce it you run two apps pointing at different destinations, handled by different application controller shards. Two shards, both hydrating, both pushing git notes to the same ref. That is a race, and git tells you so. The interesting part is not that the race happens, it is what the code did when it lost.

Notes pushes already had a retry

The hydrator stores metadata as git notes under refs/notes/hydrator.metadata, and AddAndPushNote in the git client already knows these pushes can collide. It runs the push inside a retry loop: fetch the latest notes, add the note locally, push, and if the push fails for a reason that looks like a concurrent update, fetch and try again with exponential backoff.

So someone had already thought about the race. The question was why this particular failure was not being retried.

The retry only recognised four phrasings

The loop decides whether to retry by string-matching the error:

errStr := err.Error()
isRetryable := strings.Contains(errStr, "fetch first") ||
    strings.Contains(errStr, "reference already exists") ||
    strings.Contains(errStr, "incorrect old value") ||
    strings.Contains(errStr, "failed to update ref")

if !isRetryable {
    return struct{}{}, backoff.Permanent(...)
}

Four substrings, each a real way git can report a ref that moved under you. But the message from the bug report does not contain any of them. The concurrent push held the server-side lock, so git said cannot lock ref, which is a fifth phrasing nobody had added. The check returned false, the error got wrapped in backoff.Permanent, and the retry loop it was sitting inside refused to run. A failure the loop was built to absorb fell straight through it.

This is the trap with classifying errors by substring: the list is only as complete as the failures you have seen. Add a server that phrases the collision differently and the safety net has a hole in it, in exactly the spot it was meant to cover.

The fix is one more phrasing, and not three

I pulled the check into a small helper and added cannot lock ref to the set:

func isRetryableNotePushError(errStr string) bool {
    return strings.Contains(errStr, "fetch first") ||
        strings.Contains(errStr, "reference already exists") ||
        strings.Contains(errStr, "incorrect old value") ||
        strings.Contains(errStr, "failed to update ref") ||
        strings.Contains(errStr, "cannot lock ref")
}

The tempting move is to also catch remote rejected and failed to push some refs, because both show up in that same error block. I left them out on purpose. Those two are generic wrappers: git prints them for protected branches, pre-receive hook denials, and other things that will never succeed no matter how many times you retry. Retrying on them would burn the whole backoff window on pushes that are permanently dead, and worse, it would hide the real reason behind a timeout. cannot lock ref is the specific signal that another push is holding the lock right now, which is the one case where trying again is the correct thing to do. Matching the precision of the four entries already there mattered more than catching every line in the message.

Proving it before trusting it

The classifier is pure string logic, so it is cheap to test directly and there is no excuse not to. I wrote a table-driven test with all four existing signals, a non-retryable auth error that must stay false, and the real cannot lock ref block copied from the issue. Then I did the step I never skip: I removed the new line and ran the test. The cannot lock ref case failed, returning false where it should return true, which is the bug exactly as reported. Put the line back and it passes. A test you have never watched fail is not guarding anything.

The takeaway

Retry logic that classifies failures by matching error strings is really a list of the failures someone happened to know about. It looks complete until a different git version or a different server phrases the same race a new way, and then the net has a hole precisely where you needed it. When you add an entry, it is worth asking the opposite question too: which lines look retryable but are actually permanent, so you do not paper over real errors while fixing the transient one. The change is in argoproj/argo-cd#28469, now merged.