Understanding Probe Behavior
This page describes how the collector resolves the settings for each probe from the override hierarchy, when configuration changes take effect, and how often each kind of probe collects data. Managing Probes describes how to change probe settings in the Workbench, with SQL, and through the REST API.
Hierarchical Probe Overrides
Probe settings can be customized at multiple levels of the server hierarchy. The override system uses the following precedence order:
- Server overrides apply to a specific connection.
- Cluster overrides apply to all servers in a cluster.
- Group overrides apply to all clusters in a group.
- Global default settings apply when no override exists.
- Hardcoded default values apply if no database configuration exists.
When a monitored connection first needs a probe, the Collector creates a
server-level configuration for the connection by copying the values from the
first level of the hierarchy that has a configuration. Until a user overrides
it, that copy follows its parent: on each configuration reload, the Collector
updates it to match any change made in Probe Defaults or to a cluster or group
override. A row whose user_modified column is TRUE is an explicit override,
and the Collector leaves it alone; the PROBE CONFIGURATION tab and the
PUT /api/v1/probe-configs/{id} endpoint both set this column when they change
a server, cluster, or group setting. To return a server to the inherited
settings, reset its override on the server's PROBE CONFIGURATION tab.
Automatic Configuration Reload
The Collector automatically reloads probe configurations from the database
every 5 minutes. Changes to collection_interval_seconds or is_enabled take
effect within 5 minutes without requiring a restart, at any level of the
hierarchy. Each reload first brings every server-level configuration without
an override in step with its parent, and then applies the result: the
Collector stops a disabled probe at its next scheduled run, starts it again
when the probe is re-enabled, and restarts a probe's schedule when its interval
changes. A change to retention_days reaches the server-level configurations
at the same reload and takes effect on the next garbage collection run, within
24 hours, as Adjusting Retention describes.
Default Collection Intervals
Different probe types have different default collection intervals based on how frequently their data changes:
- Fast probes run every 30 to 60 seconds and cover replication and activity data.
- Normal probes run every 300 seconds (5 minutes) and cover most statistics.
- Slow probes run every 600 seconds (10 minutes) and cover checkpointer and WAL data.
- Very slow probes run every 900 seconds (15 minutes) and cover I/O statistics.
- Change-tracked probes run every 3600 seconds (1 hour) and only store data when changes are detected.
Collection Frequency Guidelines
Balance collection frequency against the following trade-offs:
- More frequent collection provides more current data.
- More frequent collection generates more queries on monitored servers.
- More frequent collection produces more data points and increases storage usage.
The following guidelines apply to common probe types:
- Fast-changing data such as replication lag should use 30 to 60 second intervals.
- Moderate data such as table statistics should use 300 second (5 minute) intervals.
- Slow-changing data such as archiver statistics should use 600 second (10 minute) or longer intervals.
Intervals and Alerting
Keep each collection interval well inside the windows that the alert rules read. Each alerting metric considers only the samples collected in the last few minutes, commonly 5 or 15 minutes depending on the probe. An interval raised beyond that window leaves the rule query with nothing to read most of the time; rules on that probe stop firing, and an alert that is already active stays active, because the alerter does not report a condition resolved on data it cannot see. As a rule of thumb, an interval of a third of the window or less is safe; the shipped defaults follow this rule.
The high_max_connections rule is the exception. Its
pg_settings.max_connections metric comes from the change-tracked
pg_settings probe, and the rule reads each server's newest snapshot whatever
its age, so the pg_settings interval does not limit it.