Observability: from tail -f to Grafana
SSH and tail -f worked when we had one box. Then one box became a fleet, and feelings stopped being an acceptable substitute for metrics.
For years, our debugging story was ssh into the box and tail -f the log file. It worked when we had one box. Then one box became a load-balanced fleet plus queue workers plus microservices, and one morning I realized I could not answer a simple question: is the platform slow right now, or does it just feel slow?
Feelings are not metrics
The turning point was a result-day incident where support said "everything is down," the servers said CPU was fine, and the truth — a saturated database connection pool — took forty minutes to find. Forty minutes of guessing, on the one day parents care most. That week we committed to proper observability.
The stack, in adoption order
- Structured logs first. JSON logs with request IDs, shipped off the boxes. Being able to trace one parent's request across services was worth the whole project.
- Metrics second. Prometheus scraping everything, Grafana dashboards for the four golden signals — latency, traffic, errors, saturation. Our "result day" dashboard is now the most-viewed page in the company after the product itself.
- Alerts last, and reluctantly. Every alert must be actionable at 3 AM or it doesn't exist. We deleted half our alerts in the first quarter — the noisy ones were training people to ignore the real ones.
A dashboard nobody looks at is decoration. An alert everybody ignores is worse than none.
What surprised me
Observability changed our conversations more than our uptime. Arguments that used to run on opinion — "the new reports module is slowing everything down" — now end in thirty seconds with a graph. That cultural shift, engineers reaching for data before theories, turned out to be the real return on investment.