Monitoring (Prometheus) #

The gateway service exposes metrics in the Prometheus format at GET /metrics. Use them to build dashboards of load, response time and error rate, and to set up alerts.

Metrics are collected only by gateway. All API requests pass through it, so the response time includes the work of the address, phone and name services. The address, phone and name services do not expose metrics themselves.

Enabling metrics #

Metrics are enabled by the [metrics] section of gateway-config.toml (see Install ApiDQ application):

[metrics]
Enabled = true # Prometheus metrics: GET /metrics

If the section is missing or Enabled = false, /metrics responds with 404 and no metrics are collected.

Restart gateway after changing the configuration:

cd ~/apidq
docker compose restart gateway

Check that metrics are served:

curl -s http://127.0.0.1:8080/metrics | grep '^http_'

The http_requests_total and http_request_duration_seconds lines appear after the first API request.

Metrics #

Label values:

• path - request path in lower case, without query string, e.g. /api/v1/clean/address;
• method - HTTP method: GET, POST;
• status of http_requests_total - HTTP response code: 200, 400, 500, 503 etc.;
• status of http_cache_requests_total - hit (response taken from the cache) or miss (response computed anew).

The http_request_duration_seconds histogram uses the default Prometheus buckets: 5 ms, 10 ms, 25 ms, 50 ms, 100 ms, 250 ms, 500 ms, 1 s, 2.5 s, 5 s, 10 s. For every path + method pair it exposes the _bucket, _sum and _count series.

http_cache_requests_total applies only to address standardization /api/v1/clean/address and appears when response caching is enabled in the address service. Caching is disabled in the default standalone configuration, so this metric is absent.

All requests to gateway are counted, including the service endpoints /health and /metrics and requests to non-existent paths (with code 404). Exclude the service endpoints in your queries - see the examples below.

/metrics also exposes the standard Go process metrics of gateway: go_* (goroutines, memory, garbage collection) and process_* (CPU, memory, open files).

Counters are kept in memory and reset when the gateway container restarts. The Prometheus rate() and increase() functions handle such resets automatically.

Prometheus configuration #

Add a scrape job for all ApiDQ servers to prometheus.yml. If there are several servers, list each of them in targets:

scrape_configs:
  - job_name: apidq
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets:
          - 192.168.0.20:8080 # ApiDQ server ip and gateway port

Reload the Prometheus configuration and open Status → Targets in its web interface: the apidq job must be UP.

If Prometheus runs in Docker on the same server as ApiDQ, set targets to the Docker network gateway address - 172.17.0.1:8080. The address 127.0.0.1 inside a container points to the container itself, not to the server.

Restricting access to metrics #

/metrics requires no authorization and is available to anyone who can reach the gateway port. If the gateway port is reachable from an external network, close /metrics for everyone except the Prometheus server - for example, on the load balancer or in the firewall.

Example for nginx proxying requests to gateway:

location ~* ^/metrics {
    allow 192.168.0.30; # Prometheus server ip
    deny all;
    proxy_pass http://127.0.0.1:8080;
}

gateway converts the request path to lower case, so /METRICS serves metrics too. Use case-insensitive matching in access rules, like ~* in the example above.

Restricting access also protects against series growth: every unique path, including non-existent ones, creates a new http_requests_total series, and an external scan of the server can produce thousands of them.

PromQL examples #

Requests per second by API method:

sum by (path) (rate(http_requests_total{job="apidq", path!~"/health|/metrics"}[5m]))

Share of 5xx responses:

sum(rate(http_requests_total{job="apidq", status=~"5..", path!~"/health|/metrics"}[5m]))
/
sum(rate(http_requests_total{job="apidq", path!~"/health|/metrics"}[5m]))

95th percentile of response time by API method:

histogram_quantile(0.95,
  sum by (path, le) (rate(http_request_duration_seconds_bucket{job="apidq", path!~"/health|/metrics"}[5m]))
)

Share of address standardization responses served from the cache:

sum(rate(http_cache_requests_total{job="apidq", status="hit"}[5m]))
/
sum(rate(http_cache_requests_total{job="apidq"}[5m]))

Check ApiDQ server readiness separately, by the GET /health response code (200 - all services are ready, 503 - not ready), e.g. with blackbox_exporter. The up metric only shows that Prometheus could fetch /metrics from gateway - the address service may still be not ready to accept requests.