- .ai/ instruction set (root, behavior, skills, constraints) - apply.sh workflow documentation (apply.md) - architecture documentation (docs/architecture.md) - .github/copilot-instructions.md for VS Code auto-load - .gitignore (tmp/, .ai-instructions.conf)"
212 lines
5.1 KiB
Markdown
212 lines
5.1 KiB
Markdown
# Container Image Limitations
|
|
|
|
## 🐳 CRITICAL: Production Containers Use Minimal Images
|
|
|
|
**Most production containers do NOT include common debugging tools**
|
|
|
|
### Why:
|
|
- **Security:** Minimal attack surface
|
|
- **Size:** Smaller images = faster pulls
|
|
- **Best practices:** Containers should do one thing only
|
|
|
|
---
|
|
|
|
## ❌ Tools NOT Available in Pods
|
|
|
|
### Network Tools:
|
|
- ❌ `curl` - Not in Prometheus, Grafana, Loki, Tempo, MinIO
|
|
- ❌ `wget` - Not in most containers
|
|
- ❌ `ping` - Not available
|
|
- ❌ `netstat` - Not available
|
|
- ❌ `telnet` - Not available
|
|
- ❌ `nc` (netcat) - Not available
|
|
|
|
### Shells:
|
|
- ❌ `bash` - Many containers only have `/bin/sh` (ash/dash)
|
|
- ⚠️ `/bin/sh` - Usually available but limited (no arrays, fewer features)
|
|
|
|
### Editors:
|
|
- ❌ `vim` - Not available
|
|
- ❌ `nano` - Not available
|
|
- ❌ `vi` - Not available
|
|
|
|
### Utilities:
|
|
- ❌ `jq` - Not available
|
|
- ❌ `yq` - Not available
|
|
- ❌ `less/more` - Often not available
|
|
- ❌ `grep` - Sometimes available, sometimes not
|
|
|
|
---
|
|
|
|
## ✅ What IS Available
|
|
|
|
### Usually Present:
|
|
- ✅ `/bin/sh` (basic shell - ash/dash)
|
|
- ✅ `cat` - Read files
|
|
- ✅ `echo` - Print text
|
|
- ✅ `ls` - List files
|
|
- ✅ `pwd` - Current directory
|
|
- ✅ `env` - Environment variables
|
|
|
|
### Application-Specific:
|
|
- ✅ **Prometheus:** `promtool` (Prometheus CLI)
|
|
- ✅ **Grafana:** `grafana-cli` (Grafana CLI)
|
|
- ✅ **MinIO:** `mc` (MinIO Client)
|
|
|
|
---
|
|
|
|
## 🚫 DON'T Suggest These
|
|
|
|
### ❌ Wrong:
|
|
```bash
|
|
# This will fail - curl not available
|
|
kubectl exec -n monitoring pod-name -- curl http://service:8080/health
|
|
|
|
# This will fail - wget not available
|
|
kubectl exec -n monitoring pod-name -- wget http://example.com/file
|
|
|
|
# This will fail - no ping
|
|
kubectl exec -n monitoring pod-name -- ping google.com
|
|
|
|
# This might fail - bash might not exist
|
|
kubectl exec -n monitoring pod-name -- bash -c "echo test"
|
|
```
|
|
|
|
---
|
|
|
|
## ✅ DO Suggest These
|
|
|
|
### Option 1: Port-Forward + Local Tools (BEST)
|
|
|
|
```bash
|
|
# User creates port-forward:
|
|
kubectl port-forward -n monitoring svc/prometheus 9090:80
|
|
|
|
# AI uses local curl:
|
|
curl http://localhost:9090/api/v1/query?query=up
|
|
```
|
|
|
|
**Advantages:**
|
|
- ✅ All local tools available (curl, jq, etc.)
|
|
- ✅ More powerful than container shell
|
|
- ✅ Better for complex queries
|
|
- ✅ Can save results locally
|
|
|
|
### Option 2: Check Logs
|
|
|
|
```bash
|
|
# Instead of exec curl to health endpoint:
|
|
kubectl logs -n monitoring deployment/prometheus-server --tail=50
|
|
|
|
# Check for startup messages, errors
|
|
```
|
|
|
|
### Option 3: Use Kubernetes API
|
|
|
|
```bash
|
|
# Check service endpoints
|
|
kubectl get endpoints -n monitoring prometheus
|
|
|
|
# Check pod status
|
|
kubectl get pods -n monitoring -l app=prometheus
|
|
|
|
# Check events
|
|
kubectl get events -n monitoring --sort-by='.lastTimestamp'
|
|
```
|
|
|
|
### Option 4: Application-Specific CLI (If Available)
|
|
|
|
```bash
|
|
# MinIO Client (mc is available in minio containers)
|
|
kubectl exec -n monitoring deployment/minio -- mc admin info local
|
|
|
|
# Promtool (available in Prometheus containers)
|
|
kubectl exec -n monitoring deployment/prometheus-server -- promtool check config /etc/prometheus/prometheus.yml
|
|
```
|
|
|
|
---
|
|
|
|
## 📋 Decision Matrix
|
|
|
|
| Need | ❌ DON'T | ✅ DO |
|
|
|------|---------|------|
|
|
| Test HTTP endpoint | `kubectl exec ... curl` | Port-forward + local curl |
|
|
| Check connectivity | `kubectl exec ... ping` | Check pod logs, endpoints |
|
|
| Download file | `kubectl exec ... wget` | kubectl cp or port-forward + curl |
|
|
| Parse JSON | `kubectl exec ... jq` | Port-forward + local jq |
|
|
| Check logs | `kubectl exec ... cat log` | kubectl logs |
|
|
| Test DNS | `kubectl exec ... nslookup` | Check service/endpoint resources |
|
|
|
|
---
|
|
|
|
## 💡 Best Practices
|
|
|
|
### When debugging pods:
|
|
|
|
1. **First check logs:**
|
|
```bash
|
|
kubectl logs -n monitoring pod-name
|
|
kubectl logs -n monitoring pod-name --previous # Previous crash
|
|
```
|
|
|
|
2. **Check pod events:**
|
|
```bash
|
|
kubectl describe pod -n monitoring pod-name
|
|
kubectl get events -n monitoring
|
|
```
|
|
|
|
3. **Check service endpoints:**
|
|
```bash
|
|
kubectl get svc -n monitoring
|
|
kubectl get endpoints -n monitoring service-name
|
|
```
|
|
|
|
4. **If need HTTP testing:**
|
|
```bash
|
|
# User port-forwards
|
|
kubectl port-forward -n monitoring svc/service-name 8080:80
|
|
|
|
# AI tests locally
|
|
curl http://localhost:8080/health
|
|
curl http://localhost:8080/metrics
|
|
```
|
|
|
|
5. **Only use exec when:**
|
|
- Checking config files inside container
|
|
- Using application-specific CLI tools
|
|
- No other option available
|
|
|
|
---
|
|
|
|
## 🎯 Communication Pattern
|
|
|
|
**Wrong:**
|
|
```
|
|
AI: "Run this to check health endpoint:"
|
|
kubectl exec -n monitoring pod-name -- curl http://localhost:8080/health
|
|
```
|
|
|
|
**Right:**
|
|
```
|
|
AI: "Let's check the health endpoint. Please run:"
|
|
kubectl port-forward -n monitoring svc/service-name 8080:80
|
|
|
|
AI: "Now I'll test it with curl:"
|
|
[AI runs: curl http://localhost:8080/health]
|
|
```
|
|
|
|
---
|
|
|
|
## 📝 Remember
|
|
|
|
- **Minimal containers** = fewer tools
|
|
- **Port-forward pattern** = local tools available
|
|
- **Logs first** = most issues visible in logs
|
|
- **Kubernetes API** = rich information without exec
|
|
- **Application CLI** = use when available
|
|
|
|
---
|
|
|
|
**Last Updated:** 2026-01-19
|
|
**Purpose:** Guide proper debugging without assuming container tools exist
|