Kubernetes troubleshooting often requires engineers to correlate pod events, container logs, resource metrics, deployment configurations, and application dependencies before identifying the root cause of an incident. While traditional automation can execute predefined recovery steps, Agentic AI introduces a more adaptive approach: investigating failures, testing hypotheses, gathering evidence, and recommending context-aware actions.
This blog explores how to build an AI-powered Kubernetes troubleshooting agent using Google Cloud, Gemini, and Google Kubernetes Engine (GKE). The agent follows a structured investigation workflow, collects operational evidence through controlled tools, identifies probable root causes, and proposes remediation actions with supporting evidence.
The central focus is safe autonomy. Instead of granting an AI agent unrestricted cluster access, the architecture incorporates least-privilege IAM and Kubernetes RBAC, policy-driven action boundaries, human approval gates for sensitive operations, and comprehensive audit logging. Read-only diagnostics are the default, while changes to workloads or cluster configurations require explicit authorization and validation.
Through practical scenarios such as **CrashLoopBackOff**, **ImagePullBackOff**, and failed Kubernetes deployments, the blog demonstrates how organizations can move from manual incident investigation toward AI-assisted, auditable, and controlled autonomous operations.
The goal is not to replace SRE engineers, but to help them resolve incidents faster while retaining control over production environments.
The core problem: autonomy versus blast radius
Every agent design trades speed for risk, and the trade is set by what the agent can do, not by how smart the model is. Three failure modes show up repeatedly in Kubernetes:
• Confident misdiagnosis. The model reads a transient error as the root cause and proposes a fix for the wrong problem.
• Over-broad action. A reasonable fix such as restarting a pod becomes a bad one when applied to every pod in a namespace.
• Prompt injection through data. Logs, events and annotations are untrusted text. A crafted log line can try to steer the agent into running a command.
You cannot prompt your way out of these. A system prompt that says "never do anything destructive" is a request, not a control. The controls have to live outside the model: in identity, permissions, policy and a human approval step. The rule of thumb for this post is simple: the model proposes, the platform disposes.
Reference architecture on Google Cloud
Alerts reach the agent, which investigates through read-only telemetry. Every action passes through the executor, which checks scope and risk class and waits for a human before it uses the write identity. The agent can run on GKE itself or on Cloud Run, and the same pattern works with Google's Agent Development Kit, a Gemini function-calling loop, or an MCP server that exposes Kubernetes tools
Autonomy tiers: earn trust one level at a time
Designing the human-in-the-loop approval flow
An approval step only protects you if the human can make a real decision in about thirty seconds. A bare "Approve? y/n" trains people to click yes. Design the request so the reviewer sees what they need:
- The exact action. Show the literal command or API call, with namespace, resource name and arguments, not a paraphrase.
- The evidence. Link the log lines, events and metrics that led the agent here, so the reviewer can check the reasoning.
- The blast radius. State how many pods, services or users the action can affect.
- The rollback. Say how to undo it, and whether the agent can do so automatically.
- An expiry. An approval applies to one action on one object for a short window, say 15 minutes. It is never a standing permission. Route the request to where responders already are. Chat tools, a ticket, or a pager acknowledgement all work. The key property is that the approval is recorded with the approver's identity and bound to the specific action, so the executor can verify it before running anything.
Two habits keep the loop healthy. First, require a different person than the one who triggered the agent for high-risk actions, the same four-eyes principle you use for production changes. Second, measure approval fatigue: if reviewers approve more than about 95% of requests in under five seconds, the gate has become decoration and the tier should be reviewed
Guardrails that do not depend on the model
The approval gate is one layer. Stack others underneath it so a bug in any one layer is not an incident.
• Separate identities for read and write. Run two Kubernetes ServiceAccounts: a read-only one for tier 0 and a narrowly scoped one used only by the executor after approval. The model never holds the write credential.
• Workload Identity Federation for GKE. Bind those ServiceAccounts to Google Cloud service accounts so the agent reaches Cloud Logging, Monitoring and Vertex AI without long-lived keys. Grant only roles such as log viewer and monitoring viewer to the read identity.
• Least-privilege RBAC. Use namespaced Roles, not ClusterRoles, and list verbs explicitly. Allow delete on pods, never on namespaces, persistentvolumeclaims or secrets.
• Admission policy. Use Policy Controller (Gatekeeper) or a ValidatingAdmissionPolicy to reject risky changes even if RBAC allows them, for example scaling a production deployment to zero or removing resource limits.
• Redact before the model sees data. Strip secrets, tokens and personal data from logs and manifests. Treat everything retrieved from the cluster as untrusted input, since log text can carry injected instructions.
• Rate and scope limits. Cap actions per incident and per hour, and restrict the agent to named namespaces and clusters.
• Audit everything. Cloud Audit Logs and Kubernetes audit logs record what the executor did. Also log the agent's prompts, tool calls and the approval record so an incident review can replay the reasoning.
A useful test: if the model were fully compromised, what is the worst thing the executor could do? Design until the answer is acceptable.
Code walkthrough: tools, risk classes and the approval gate
The pattern below is framework-agnostic.
It works with Google's Agent Development Kit, a plain Gemini function-calling loop on Vertex AI, or an MCP server that exposes Kubernetes tools. Check the current ADK documentation for its built-in tool confirmation features before writing your own gate.
The key idea is that the model never calls kubectl directly. It calls tools, and each tool declares a risk class that the platform enforces.
from dataclasses import dataclass
from enum import Enum
import time, uuid
class Risk(Enum):
READ = 0 # tier 0: runs immediately
MUTATE = 1 # tier 2: needs human approval
FORBIDDEN = 2 # never exposed to the model
@dataclass
class ToolSpec:
name: str
risk: Risk
fn: callable
REGISTRY: dict[str, ToolSpec] = {}
def tool(risk: Risk):
def wrap(fn):
REGISTRY[fn.__name__] = ToolSpec(fn.__name__, risk, fn)
return fn
return wrap
@tool(Risk.READ)
def get_pod_logs(namespace: str, pod: str, tail: int = 200) -> str:
# uses the READ-ONLY service account
return k8s_read.logs(namespace, pod, tail=min(tail, 500))
@tool(Risk.READ)
def describe_pod(namespace: str, pod: str) -> str:
return k8s_read.describe(namespace, pod)
@tool(Risk.MUTATE)
def rollout_restart(namespace: str, deployment: str) -> str:
# uses the WRITE service account, only reachable via the executor
return k8s_write.rollout_restart(namespace, deployment)
The executor sits between the model and the tools. Read tools run straight through. Mutating tools create a pending approval and stop.
ALLOWED_NAMESPACES = {"checkout", "payments"}
APPROVAL_TTL_SECONDS = 900
class Executor:
def __init__(self, approvals):
self.approvals = approvals # backed by Firestore or Cloud SQL
def call(self, name: str, args: dict, incident_id: str):
spec = REGISTRY.get(name)
if spec is None or spec.risk is Risk.FORBIDDEN:
raise PermissionError(f"tool not allowed: {name}")
if args.get("namespace") not in ALLOWED_NAMESPACES:
raise PermissionError("namespace out of scope")
if spec.risk is Risk.READ:
return spec.fn(**args)
# MUTATE: request approval, do not run
req = self.approvals.create(
id=str(uuid.uuid4()),
incident=incident_id,
tool=name,
args=args,
expires=time.time() + APPROVAL_TTL_SECONDS,
)
notify_oncall(req) # chat card with command, evidence, rollback
return {"status": "pending_approval", "request_id": req.id}
def execute_approved(self, request_id: str, approver: str):
req = self.approvals.get(request_id)
if req.expired() or req.used:
raise PermissionError("approval expired or already used")
if approver == req.requested_by:
raise PermissionError("requester cannot self-approve")
req.mark_used(approver) # single use
audit_log(req, approver) # who, what, when
return REGISTRY[req.tool].fn(**req.args)
Three details matter here. The args are validated and stored at request time, so the action that runs is exactly the action the human saw. The approval is single-use and expires. And the write credential is only loaded inside execute_approved, which the model cannot call.
An incident end to end: CrashLoopBackOff in checkout
Here is how the flow plays out for a realistic case. The scenario is illustrative.
- Trigger. A Cloud Monitoring alert fires for the checkout-api deployment and opens an incident. The alert payload goes to the agent through Pub/Sub.
- Observe (tier 0). The agent lists pods, sees three of four in CrashLoopBackOff, and pulls recent logs and events. The logs show a failed database connection, and the events show a ConfigMap changed twenty minutes earlier.
- Correlate. The agent compares the new ConfigMap with the previous revision from the rollout history and finds a changed database host value that no longer resolves.
- Recommend (tier 1). It posts a summary with evidence: the failing log lines, the config diff, and the timing. It does not guess beyond that evidence.
- Request approval (tier 2). The best fix is a rollback of the last revision. The agent calls the rollback tool, and the executor sends a chat card showing the exact command, the single deployment affected, and the undo path.
- Approve and execute. The on-call engineer checks the config diff, taps approve, and the executor runs the rollback using the write identity.
- Verify and close. The agent watches pod readiness and error rate for ten minutes, posts the result, and attaches the full trace to the incident for the postmortem. The human spent under a minute and still made the call. The agent did the slow part: reading logs, diffing config and checking recovery.
Pitfalls and how to evaluate the agent
Most failures come from the plumbing around the model, not the model itself.
• Overly broad tools. A generic run_kubectl(command) tool defeats every control above. Expose narrow, typed tools instead.
• Approval on summaries, not actions. If the card shows the agent's description rather than the real command, the human approves something other than what runs.
• Stale context. Cluster state changes during an incident. Re-read the object just before executing, and fail the approval if it has drifted.
• No kill switch. Provide one flag that drops the agent to tier 0 across every cluster.
• Skipping staging. Replay past incidents and run fault injection (bad image tag, missing ConfigMap, memory limits too low) in a test cluster before production.
Conclusion and rollout checklist
Safe autonomy is an engineering problem with a known shape: give the model narrow tools, keep credentials and policy outside it, put a well-designed human decision in front of every change, and widen its freedom only as the data justifies. Done well, the agent handles the tedious investigation and the human keeps the judgment.
A practical rollout order:
