Loki Logging, Alerts & Node.js Metrics

Centralized logging with Loki

Loki stores and indexes logs for search in Grafana — without SSHing into every server.

App → log file → Promtail → Loki → Grafana Explore

Why it matters

Multiple servers, containers, and restarts make local log files painful. Centralize once, query everywhere.

Add Loki as a Grafana datasource (http://loki:3100). Use Explore to filter streams (for example {job="varlogs"} depending on your Promtail config).

Production alerting

Alerting = automatic notification when production crosses a threshold.

Prometheus metrics → Grafana alert rule → SMTP / Slack → Engineer

Channels: Email, Slack, Discord, Telegram, PagerDuty, Teams.

SMTP email (lab outline)

1. Create a Gmail app password (or other SMTP credentials)
2. Configure Grafana SMTP (GF_SMTP_* env vars or grafana.ini)
3. Restart Grafana
4. Create a Contact point (email)
5. Create an alert rule (for example high CPU from Node Exporter)
6. Optionally force a condition to verify notification

Prometheus metrics
        ↓
Grafana alert rule
        ↓
SMTP email
        ↓
Inbox

Node.js metrics with prom-client

Expose Prometheus metrics from the app:

npm install prom-client

const client = require("prom-client");
const register = new client.Registry();
client.collectDefaultMetrics({ register });
 
const httpRequests = new client.Counter({
  name: "http_requests_total",
  help: "Total HTTP requests",
  labelNames: ["method", "route", "status_code"],
  registers: [register],
});
 
// after each response:
httpRequests.inc({ method, route, status_code: String(status) });
 
app.get("/metrics", async (_req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});

Scrape from Prometheus:

- job_name: "nodejs"
  static_configs:
    - targets: ["<APP_IP>:3000"]
  metrics_path: /metrics

Verify: curl http://localhost:3000/metrics

Day 1 wrap-up

Fundamentals & architecture
        ↓
Docker stack (Prometheus / Grafana / Loki)
        ↓
App host (Exporter + Nginx + PM2)
        ↓
PromQL + Grafana dashboards
        ↓
Logs + alerts + Node.js /metrics