- .ai/ instruction set (root, behavior, skills, constraints) - apply.sh workflow documentation (apply.md) - architecture documentation (docs/architecture.md) - .github/copilot-instructions.md for VS Code auto-load - .gitignore (tmp/, .ai-instructions.conf)"
5.1 KiB
5.1 KiB
Container Image Limitations
🐳 CRITICAL: Production Containers Use Minimal Images
Most production containers do NOT include common debugging tools
Why:
- Security: Minimal attack surface
- Size: Smaller images = faster pulls
- Best practices: Containers should do one thing only
❌ Tools NOT Available in Pods
Network Tools:
- ❌
curl- Not in Prometheus, Grafana, Loki, Tempo, MinIO - ❌
wget- Not in most containers - ❌
ping- Not available - ❌
netstat- Not available - ❌
telnet- Not available - ❌
nc(netcat) - Not available
Shells:
- ❌
bash- Many containers only have/bin/sh(ash/dash) - ⚠️
/bin/sh- Usually available but limited (no arrays, fewer features)
Editors:
- ❌
vim- Not available - ❌
nano- Not available - ❌
vi- Not available
Utilities:
- ❌
jq- Not available - ❌
yq- Not available - ❌
less/more- Often not available - ❌
grep- Sometimes available, sometimes not
✅ What IS Available
Usually Present:
- ✅
/bin/sh(basic shell - ash/dash) - ✅
cat- Read files - ✅
echo- Print text - ✅
ls- List files - ✅
pwd- Current directory - ✅
env- Environment variables
Application-Specific:
- ✅ Prometheus:
promtool(Prometheus CLI) - ✅ Grafana:
grafana-cli(Grafana CLI) - ✅ MinIO:
mc(MinIO Client)
🚫 DON'T Suggest These
❌ Wrong:
# This will fail - curl not available
kubectl exec -n monitoring pod-name -- curl http://service:8080/health
# This will fail - wget not available
kubectl exec -n monitoring pod-name -- wget http://example.com/file
# This will fail - no ping
kubectl exec -n monitoring pod-name -- ping google.com
# This might fail - bash might not exist
kubectl exec -n monitoring pod-name -- bash -c "echo test"
✅ DO Suggest These
Option 1: Port-Forward + Local Tools (BEST)
# User creates port-forward:
kubectl port-forward -n monitoring svc/prometheus 9090:80
# AI uses local curl:
curl http://localhost:9090/api/v1/query?query=up
Advantages:
- ✅ All local tools available (curl, jq, etc.)
- ✅ More powerful than container shell
- ✅ Better for complex queries
- ✅ Can save results locally
Option 2: Check Logs
# Instead of exec curl to health endpoint:
kubectl logs -n monitoring deployment/prometheus-server --tail=50
# Check for startup messages, errors
Option 3: Use Kubernetes API
# Check service endpoints
kubectl get endpoints -n monitoring prometheus
# Check pod status
kubectl get pods -n monitoring -l app=prometheus
# Check events
kubectl get events -n monitoring --sort-by='.lastTimestamp'
Option 4: Application-Specific CLI (If Available)
# MinIO Client (mc is available in minio containers)
kubectl exec -n monitoring deployment/minio -- mc admin info local
# Promtool (available in Prometheus containers)
kubectl exec -n monitoring deployment/prometheus-server -- promtool check config /etc/prometheus/prometheus.yml
📋 Decision Matrix
| Need | ❌ DON'T | ✅ DO |
|---|---|---|
| Test HTTP endpoint | kubectl exec ... curl |
Port-forward + local curl |
| Check connectivity | kubectl exec ... ping |
Check pod logs, endpoints |
| Download file | kubectl exec ... wget |
kubectl cp or port-forward + curl |
| Parse JSON | kubectl exec ... jq |
Port-forward + local jq |
| Check logs | kubectl exec ... cat log |
kubectl logs |
| Test DNS | kubectl exec ... nslookup |
Check service/endpoint resources |
💡 Best Practices
When debugging pods:
-
First check logs:
kubectl logs -n monitoring pod-name kubectl logs -n monitoring pod-name --previous # Previous crash -
Check pod events:
kubectl describe pod -n monitoring pod-name kubectl get events -n monitoring -
Check service endpoints:
kubectl get svc -n monitoring kubectl get endpoints -n monitoring service-name -
If need HTTP testing:
# User port-forwards kubectl port-forward -n monitoring svc/service-name 8080:80 # AI tests locally curl http://localhost:8080/health curl http://localhost:8080/metrics -
Only use exec when:
- Checking config files inside container
- Using application-specific CLI tools
- No other option available
🎯 Communication Pattern
Wrong:
AI: "Run this to check health endpoint:"
kubectl exec -n monitoring pod-name -- curl http://localhost:8080/health
Right:
AI: "Let's check the health endpoint. Please run:"
kubectl port-forward -n monitoring svc/service-name 8080:80
AI: "Now I'll test it with curl:"
[AI runs: curl http://localhost:8080/health]
📝 Remember
- Minimal containers = fewer tools
- Port-forward pattern = local tools available
- Logs first = most issues visible in logs
- Kubernetes API = rich information without exec
- Application CLI = use when available
Last Updated: 2026-01-19 Purpose: Guide proper debugging without assuming container tools exist