Cloudflare ล่ม 12 ก.ย. 2025: React useEffect Bug → Tenant Service 5xx Kubernetes

Cloudflare ล่ม 12 ก.ย. 2025: React useEffect Bug → Tenant Service API 5xx ทั่วโลก
Cloudflare outage จาก React useEffect hook bug ใน Dashboard เรียก Tenant Service API ซ้ำ → Kubernetes pods overload Global rate limit + Argo Rollouts fix

Root Cause: React useEffect Bug

🐛 Dashboard component:
useEffect(() => {
  fetchTenantAPI();  // ❌ Runs on EVERY render
}, []);  // Dependency array ignored
✅ Fixed version:
useEffect(() => {
  let mounted = true;
  fetchTenantAPI().then(data => {
    if (mounted) setTenant(data);
  });
  return () => { mounted = false; };
}, []);

Thundering Herd: 10K+ users hit API simultaneously

Timeline Cloudflare Outage

09:15 UTC: Dashboard deploy (buggy useEffect)
09:22 UTC: Tenant Service CPU 1200%
09:28 UTC: API 5xx errors spike
09:45 UTC: Global rate limit (100 req/min)
10:15 UTC: Scale pods 2x → Still failing
10:30 UTC: Rollback dashboard → Secondary outage
11:12 UTC: Staggered API calls (random delay)
12:05 UTC: Full recovery

Duration: 2 ชม. 50 นาที

Impact: Services & Metrics

DownDetector: 850K reports peak

Technical Deep Dive: Kubernetes Overload

Kubernetes Metrics:
- Pods: 50 → 200 (auto-scaling failed)
- CPU: 1200% sustained
- Memory: 95% cluster capacity
- API calls/sec: 50K → 500K spike

Rate Limiting Fix:

// Global rate limiter
const rateLimit = new RateLimiter({
  points: 100,  // requests
  duration: 60, // seconds
});

Cloudflare Response & Mitigation

🔧 09:45: Global rate limit 100/min
⚙️ 10:15: Horizontal Pod Autoscaler 2x
🔄 10:30: Argo Rollout rollback
⏱️ 11:12: Random jitter (100-500ms delay)
📊 11:45: Custom metrics (new vs retry)

Argo Rollouts Config:

strategy:
  canary:
    steps:
    - setWeight: 20
    - pause: {duration: 300}

Lessons Learned: Production Best Practices

✅ useEffect cleanup + abort controller
✅ Rate limiting ALL public APIs
✅ Staggered deploys (Argo CD)
✅ Circuit breakers (Istio)
✅ Thundering herd protection

Post-Mortem Action Items

Code: The Bug + Production Fix

Buggy Dashboard (React 18):

useEffect(() => {
  fetch('/api/tenant').then(setData);  // Runs infinitely
}, [userId]);  // Missing deps

Fixed Production:

const abortController = useRef();
useEffect(() => {
  abortController.current = new AbortController();
  fetch('/api/tenant', { signal: abortController.current.signal })
    .then(setData)
    .catch(err => err.name === 'AbortError' || handleError(err));
  return () => abortController.current?.abort();
}, []);