Fix oc process namespace bug and PRGate LokiStack test flakiness

[IshwarKanse]

Summary

Two related fixes for PRGate test failures found while validating openshift/release#85177 on a live cluster:

1. oc process calls missing explicit namespace

processTemplate() invokes oc process server-side against the namespaced Template API. Without an explicit -n, the namespace falls back to the client's ambient kubeconfig context. In the PRGate CI job (e2e-prgate), that ambient namespace doesn't exist on the claimed test cluster, causing:

error processing file: failed to process template with the provided parameters

This was the root cause of the majority of PRGate test failures seen in the rehearsal runs (9/11 in cluster-logging-operator, 3/5 in loki-operator).

• Added explicit -n to the 6 call sites that were missing it: clusterlogforwarder.create(), clusterlogforwarder.update(), logFileMetricExporter.create(), eventRouter.deploy(), SubscriptionObjects.SubscribeOperator()'s namespace-creation branch, and the UIPlugin template processing in logging_operators.go. Matches the pattern already used elsewhere in utils.go (applyFromTemplate, OperatorGroup/Subscription, minio, lokiStack).
• For the two sites creating cluster-scoped resources (Namespace, UIPlugin) where there's no natural target namespace, -n default routes the call to an existing namespace.

2. Cross-test resource collisions and a CA bundle race in LokiStack tests

While validating fix #1 by running the full PRGate suites repeatedly on a live cluster, several LokiStack tests intermittently failed with errors unrelated to the namespace bug:

• loki.go — "Reconcile and re-create objects on accidental user deletes" and "Define limits and overrides per tenant" listed pods/configmaps by component or created-by labels only. Since LokiStack tests share the openshift-logging namespace, and a deleted LokiStack's owned resources are garbage-collected asynchronously, these selectors could match resources belonging to a different, concurrently-running LokiStack test — one test even waited for a pod owned by a completely different test to disappear. Scoped every affected list/get by app.kubernetes.io/instance=<lokistack-name>, the convention already used elsewhere in this codebase. Also switched the configmap-count check from a single point-in-time List to a poll, since the controller can take a few seconds after LokiStack becomes Ready to finish creating them.
• loki_utils.go — createSecretFromGateway() extracted the service-ca.crt key from a *-gateway-ca-bundle configmap immediately after it appeared, before the service-ca operator had actually populated that key. Added a poll for the key itself before extracting.

Verification

Reproduced and fixed on a live OCP 4.22 cluster with cluster-logging-operator, loki-operator, and opentelemetry-product installed:

• Fix Add operator-specific PR-gate regression suites #1: with a kubeconfig context namespace pointed at a nonexistent namespace (mimicking the CI failure condition), the pre-fix code reproduces the exact failure; the fix resolves it. With the original, unmodified kubeconfig (ambient namespace default), both pre- and post-fix code pass — confirming this bug only manifests when the ambient namespace doesn't exist, as happens in CI.
• Fix Clear GOFLAGS and use -mod=mod to fix CI build without vendor directory #2: reran the cluster-logging-operator and loki-operator PRGate suites repeatedly, plus isolated reruns of every distinct failure signature. The loki-operator suite went from 1/7 to consistently passing as each real bug was found and fixed; a small number of remaining failures in full-suite runs were confirmed via isolated reruns to be resource-contention artifacts of the small, freshly-provisioned validation cluster (cold image pulls, service-ca lag under load) — not defects, and not touched here per "don't change test intent."

An independent code and security review (command injection, secret handling, resource leaks, error handling) found no blocking issues.

Test plan

• make build succeeds
• Reproduced both failure classes locally and confirmed the fixes resolve them
• Reran full PRGate suites and isolated reruns of every distinct failure signature
• Re-rehearse the PRGate jobs on Add optional PR-gate e2e jobs for cluster-logging-operator and loki main branches, with QE agent triage openshift/release#85177 once this merges and the image promotes

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jcantrill for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment