# Container Image Limitations

## 🐳 CRITICAL: Production Containers Use Minimal Images

**Most production containers do NOT include common debugging tools**

### Why:
- **Security:** Minimal attack surface
- **Size:** Smaller images = faster pulls
- **Best practices:** Containers should do one thing only

---

## ❌ Tools NOT Available in Pods

### Network Tools:
- ❌ `curl` - Not in Prometheus, Grafana, Loki, Tempo, MinIO
- ❌ `wget` - Not in most containers
- ❌ `ping` - Not available
- ❌ `netstat` - Not available
- ❌ `telnet` - Not available
- ❌ `nc` (netcat) - Not available

### Shells:
- ❌ `bash` - Many containers only have `/bin/sh` (ash/dash)
- ⚠️ `/bin/sh` - Usually available but limited (no arrays, fewer features)

### Editors:
- ❌ `vim` - Not available
- ❌ `nano` - Not available
- ❌ `vi` - Not available

### Utilities:
- ❌ `jq` - Not available
- ❌ `yq` - Not available
- ❌ `less/more` - Often not available
- ❌ `grep` - Sometimes available, sometimes not

---

## ✅ What IS Available

### Usually Present:
- ✅ `/bin/sh` (basic shell - ash/dash)
- ✅ `cat` - Read files
- ✅ `echo` - Print text
- ✅ `ls` - List files
- ✅ `pwd` - Current directory
- ✅ `env` - Environment variables

### Application-Specific:
- ✅ **Prometheus:** `promtool` (Prometheus CLI)
- ✅ **Grafana:** `grafana-cli` (Grafana CLI)
- ✅ **MinIO:** `mc` (MinIO Client)

---

## 🚫 DON'T Suggest These

### ❌ Wrong:
```bash
# This will fail - curl not available
kubectl exec -n monitoring pod-name -- curl http://service:8080/health

# This will fail - wget not available
kubectl exec -n monitoring pod-name -- wget http://example.com/file

# This will fail - no ping
kubectl exec -n monitoring pod-name -- ping google.com

# This might fail - bash might not exist
kubectl exec -n monitoring pod-name -- bash -c "echo test"
```

---

## ✅ DO Suggest These

### Option 1: Port-Forward + Local Tools (BEST)

```bash
# User creates port-forward:
kubectl port-forward -n monitoring svc/prometheus 9090:80

# AI uses local curl:
curl http://localhost:9090/api/v1/query?query=up
```

**Advantages:**
- ✅ All local tools available (curl, jq, etc.)
- ✅ More powerful than container shell
- ✅ Better for complex queries
- ✅ Can save results locally

### Option 2: Check Logs

```bash
# Instead of exec curl to health endpoint:
kubectl logs -n monitoring deployment/prometheus-server --tail=50

# Check for startup messages, errors
```

### Option 3: Use Kubernetes API

```bash
# Check service endpoints
kubectl get endpoints -n monitoring prometheus

# Check pod status
kubectl get pods -n monitoring -l app=prometheus

# Check events
kubectl get events -n monitoring --sort-by='.lastTimestamp'
```

### Option 4: Application-Specific CLI (If Available)

```bash
# MinIO Client (mc is available in minio containers)
kubectl exec -n monitoring deployment/minio -- mc admin info local

# Promtool (available in Prometheus containers)
kubectl exec -n monitoring deployment/prometheus-server -- promtool check config /etc/prometheus/prometheus.yml
```

---

## 📋 Decision Matrix

| Need | ❌ DON'T | ✅ DO |
|------|---------|------|
| Test HTTP endpoint | `kubectl exec ... curl` | Port-forward + local curl |
| Check connectivity | `kubectl exec ... ping` | Check pod logs, endpoints |
| Download file | `kubectl exec ... wget` | kubectl cp or port-forward + curl |
| Parse JSON | `kubectl exec ... jq` | Port-forward + local jq |
| Check logs | `kubectl exec ... cat log` | kubectl logs |
| Test DNS | `kubectl exec ... nslookup` | Check service/endpoint resources |

---

## 💡 Best Practices

### When debugging pods:

1. **First check logs:**
   ```bash
   kubectl logs -n monitoring pod-name
   kubectl logs -n monitoring pod-name --previous  # Previous crash
   ```

2. **Check pod events:**
   ```bash
   kubectl describe pod -n monitoring pod-name
   kubectl get events -n monitoring
   ```

3. **Check service endpoints:**
   ```bash
   kubectl get svc -n monitoring
   kubectl get endpoints -n monitoring service-name
   ```

4. **If need HTTP testing:**
   ```bash
   # User port-forwards
   kubectl port-forward -n monitoring svc/service-name 8080:80
   
   # AI tests locally
   curl http://localhost:8080/health
   curl http://localhost:8080/metrics
   ```

5. **Only use exec when:**
   - Checking config files inside container
   - Using application-specific CLI tools
   - No other option available

---

## 🎯 Communication Pattern

**Wrong:**
```
AI: "Run this to check health endpoint:"
kubectl exec -n monitoring pod-name -- curl http://localhost:8080/health
```

**Right:**
```
AI: "Let's check the health endpoint. Please run:"
kubectl port-forward -n monitoring svc/service-name 8080:80

AI: "Now I'll test it with curl:"
[AI runs: curl http://localhost:8080/health]
```

---

## 📝 Remember

- **Minimal containers** = fewer tools
- **Port-forward pattern** = local tools available
- **Logs first** = most issues visible in logs
- **Kubernetes API** = rich information without exec
- **Application CLI** = use when available

---

**Last Updated:** 2026-01-19
**Purpose:** Guide proper debugging without assuming container tools exist