Skip to content

Anomaly Detection Properties

Use the properties in the anomaly section of the alerter configuration file to turn anomaly detection on or off, set the statistical limits that mark a metric value as unusual, and control how the alerter compares and classifies the resulting candidates before raising an alert. The alerter evaluates each candidate in three tiers; each tier passes on only the candidates that it does not resolve:

  • tier 1 compares each metric with its baseline and flags values whose z-score exceeds the configured sensitivity.
  • tier 2 compares each candidate with the embeddings of past anomalies to recognize patterns that the alerter has already seen.
  • tier 3 asks an LLM to classify each remaining candidate and decides whether to raise an alert.

Tiers 2 and 3 use the providers that the llm section selects; LLM Provider Properties describes those settings. The Configuration File Properties page describes the other sections of the file and the locations the alerter searches for it.

The following table describes the top-level anomaly option:

Option Type Default Description
enabled boolean true Enable anomaly detection

Tier 1: Statistical Analysis

The anomaly.tier1 section configures z-score-based statistical detection. The following table describes the tier 1 options:

Option Type Default Description
enabled boolean true Enable tier 1
default_sensitivity float 3.0 Z-score threshold
evaluation_interval_seconds integer 60 Evaluation interval
clear_count integer 3 Consecutive in-band samples before an anomaly alert clears

The anomaly.tier1.clear_count option sets how many consecutive samples of the metric behind an active anomaly alert must score back inside the sensitivity band before the alerter clears the alert. Only a newly collected sample counts toward the total; tier 1 usually evaluates more often than the probe collects, and an evaluation that re-scores a sample already counted does not add to the count. The value must be at least 1; the alerter refuses to start, or to apply a reload, with a smaller value. For a probe that collects every 300 seconds, the default of 3 clears an alert about ten minutes after the first in-band sample. A SIGHUP reload applies a new value from the next evaluation.

Tier 1: Variance Floor, Warmup, and Z-Score Cap

Three additional blocks under anomaly.tier1 prevent the detector from firing on baselines that have not yet stabilised. These blocks are most relevant on young datastores; on a datastore with one or two days of metric history, a baseline's stored standard deviation can collapse far below the metric's natural variation, producing z-scores in the thousands. The variance floor and warmup gate suppress this failure mode, and the z-score cap acts as defence in depth.

The anomaly.tier1.max_z_score option clamps the absolute z-score symmetrically around zero before the sensitivity comparison. The default is 100.0; any genuine outlier sits well below this value, and the cap simply prevents a runaway divisor from generating multi-thousand-sigma scores. Setting max_z_score: 0 disables the cap.

Option Type Default Description
max_z_score float 100.0 Z-score clamp; 0 disables

The anomaly.tier1.variance_floor block enforces a minimum divisor on the z-score calculation. The effective standard deviation is the larger of the raw stored value and a hybrid floor; the floor itself is the larger of relative_pct times the absolute baseline mean and absolute_floor. The relative term dominates for non-zero metrics, while the absolute term acts as a safety net when the mean approaches zero. Setting both relative_pct: 0 and absolute_floor: 0 disables the floor entirely; the detector then falls back to the existing stddev == 0 guard.

Option Type Default Description
relative_pct float 0.05 Floor as fraction of abs(mean)
absolute_floor float 0.001 Absolute minimum stddev

The anomaly.tier1.warmup block suppresses detection for baselines that have not accumulated enough samples or enough wall-clock observation time. Each period_type (all, hourly, and daily) has its own pair of thresholds, and a baseline is considered warm only when both are met. The all baseline defaults require roughly one day of operation at the 60 second collection interval; the hourly and daily defaults require enough span for each time bucket to have been observed multiple times. Setting both min_samples: 0 and min_span_hours: 0 for a given period_type disables warmup suppression for that type.

Option Type Default Description
all.min_samples integer 100 Minimum sample count
all.min_span_hours integer 24 Minimum span in hours
hourly.min_samples integer 5 Minimum sample count
hourly.min_span_hours integer 120 Minimum span in hours
daily.min_samples integer 3 Minimum sample count
daily.min_span_hours integer 336 Minimum span in hours

In the following example, the anomaly.tier1 section tightens the variance floor and extends the all warmup window:

anomaly:
  tier1:
    max_z_score: 100.0
    variance_floor:
      relative_pct: 0.10
      absolute_floor: 0.005
    warmup:
      all:
        min_samples: 200
        min_span_hours: 48
      hourly:
        min_samples: 5
        min_span_hours: 120
      daily:
        min_samples: 3
        min_span_hours: 336

Warmup suppressions are recorded at debug log level only; the detector does not write a candidate row or an alert when it skips a cold baseline. The detector logs a fall-through from a cold hourly or daily baseline row to a warm, less specific row at the same level. To inspect recent suppressions, enable debug logging in the alerter and run sudo journalctl -u pgedge-ai-dba-alerter --since 10m. The log line names the connection, metric, period type, and sample count, which is enough to confirm whether a missing alert reflects warmup suppression or a genuinely quiet metric.

Tier 1: Per-Metric Floors

The anomaly.tier1.metric_floors map sets two floors for each metric, so that a value that is unusual for a quiet baseline but harmless in absolute terms does not raise an alert. Each key is a metric name, and each entry accepts two options:

Option Type Description
min_value float Value below which the metric is never anomalous; 0 disables
min_stddev float Minimum z-score divisor in the metric's units; 0 disables

A value under min_value raises no anomaly candidate, and the value counts as in band when the alerter decides whether to clear an open anomaly alert. The min_stddev option sets a lower bound on the divisor of the z-score, so a small change on a baseline that has never moved scores a small z-score. The alerter divides by the largest of min_stddev, the baseline's own standard deviation, relative_pct times the absolute baseline mean (5% by default), and absolute_floor, so min_stddev takes effect only when it exceeds the other three values. When the baseline has no spread of its own, the alerter caps the alert severity at warning; threshold rules remain the way to raise a critical alert on an absolute value.

The alerter has built-in floors for its standard metrics, for example min_value: 50 and min_stddev: 5 for CPU usage percent, and min_value: 10 and min_stddev: 5 for the session count; the anomaly detection guide lists them all. An entry overrides a built-in floor field by field, so an entry that sets only min_value keeps the built-in min_stddev. The alerter refuses a negative or non-finite value, or an empty metric name, and logs a warning at startup and on each reload for an entry naming a metric that anomaly detection does not score, such as a mistyped name.

In the following example, the metric_floors map raises the CPU value floor and disables both floors for the session count:

anomaly:
  tier1:
    metric_floors:
      pg_sys_cpu_usage_info.processor_time_percent:
        min_value: 70
      pg_stat_activity.count:
        min_value: 0
        min_stddev: 0

Tier 2: Embedding Similarity

The anomaly.tier2 section configures pgvector-based similarity search for pattern matching.

Option Type Default Description
enabled boolean true Enable tier 2
suppression_threshold float 0.85 Suppression threshold
similarity_threshold float 0.3 Similarity threshold

Tier 3: LLM Classification

The anomaly.tier3 section configures LLM-based classification for complex anomalies.

The following table describes the tier 3 options:

Option Type Default Description
enabled boolean true Enable tier 3
timeout_seconds integer 30 Timeout for each LLM API call
decision_suppress_threshold number 0.5 Probability that a candidate is not a real issue at which the decision model suppresses it; must be greater than 0 and less than 1 while a decision provider is configured

Tier 3 classifies each candidate with the reasoning model unless llm.decision_provider selects a decision model. With a decision provider, tier 3 asks the decision model first and uses the reasoning model only as a fallback; the Decision Model section describes the behavior. Tier 3 runs while tier 3 is enabled and either a reasoning or a decision provider initialized at startup.

The decision model returns the probability that an anomaly is a real issue requiring attention. Tier 3 suppresses the candidate when one minus that probability is at least decision_suppress_threshold, and raises the alert otherwise. The default of 0.5 suppresses a candidate that the model judges more likely than not to be a false positive; a higher value suppresses fewer candidates. The alerter checks the value only when a decision provider is configured, and ignores the option otherwise. Each provider calibrates its probabilities differently, so tune the threshold for the provider in use rather than carrying a value across providers.

Each model call has its own timeout_seconds limit. A candidate whose decision model call fails and then falls back to the reasoning model can therefore take up to twice the timeout.

Re-evaluation (reevaluation)

The anomaly.reevaluation section configures the worker that periodically asks the reasoning model whether an acknowledged anomaly alert still applies. When the model decides that the alert no longer applies, the alerter clears the alert and sends a clear notification; otherwise the alert stays acknowledged until the next pass.

The following table describes the re-evaluation options:

Option Type Default Description
enabled boolean true Enable the re-evaluation worker
interval_seconds integer 300 Seconds between passes, and the minimum time before the worker re-evaluates the same alert again
timeout_seconds integer 30 Timeout for each LLM call; a value of 0 or less uses 30 seconds
max_per_cycle integer 10 Maximum number of acknowledged alerts that one pass re-evaluates

The worker runs only when anomaly.enabled and enabled are both true at startup and a reasoning provider initialized; the decision model is never used for re-evaluation. A SIGHUP reload applies changed values on the next pass, but does not start a worker that was not running at startup.

Provider Health (provider_health)

The anomaly.provider_health section controls the system alert that the alerter raises when calls to the embedding, reasoning, or decision provider keep failing. While a provider fails, anomaly detection runs in a degraded mode whose effect depends on the provider that fails:

  • while the embedding provider fails, tier 2 compares no candidate with past anomalies and passes each candidate to tier 3, where a working reasoning or decision model still classifies the candidate.
  • while the reasoning provider fails, tier 3 raises each candidate that the decision model does not classify without LLM classification.
  • while the decision provider fails, tier 3 classifies each candidate with the reasoning model where one is configured, and otherwise raises the candidate without classification.
  • while the reasoning provider fails, each re-evaluation call fails and the re-evaluation worker leaves the acknowledged anomaly alert unchanged; the worker does not check provider health, so a failing embedding or decision provider does not stop re-evaluation.

The following table describes the provider health options:

Option Type Default Description
failure_threshold integer 3 Consecutive failed calls that raise the alert; must be at least 1
failure_rate number 0.3 Share of the last failure_rate_window calls that raises the alert when they fail; must be greater than 0 and at most 1
failure_rate_window integer 20 Number of recent calls the failure rate covers; must be between 1 and 1000. A small window can set a limit of one failed call, as failure_rate: 0.3 with a window of 3 does, so the first failure raises the alert before failure_threshold is reached

The alerter counts consecutive failures separately for tier 2 embeddings, tier 3 classification, tier 3 decisions, and re-evaluation, and for each provider. When one count reaches the threshold, the alerter raises a single warning alert for that tier and provider, and updates the alert's failure count as further calls fail. The next successful call to that provider resets the count and clears the alert, unless the failure rate described below keeps the alert open; tier 3 classification and re-evaluation share the reasoning provider, so a successful call from either clears the alerts of both. The decision provider serves tier 3 decisions alone, so no reasoning call clears its alert; a successful decision call clears it, as does a successful startup check of the decision provider, described below. After clearing an alert, the alerter does not raise the alert again for five minutes, so that a provider that fails intermittently cannot send a burst of notifications. A call canceled before it completes counts as neither a failure nor a success, although a call that times out counts as a failure. A tier 3 or re-evaluation call also counts as a failure when the provider answers with a response that holds no decision the alerter can parse; a decision call counts as a failure when the model refuses to answer, gives no answer, or gives a probability outside the range 0 to 1. The alerter records the failure whatever the alerter then does with the candidate or alert, which may be to act on a keyword found in the response or to take the tier's default decision.

When anomaly detection is enabled, the alerter also checks the providers of the enabled tiers at startup. A failed check raises the alert for its tier immediately, whatever the threshold or the five-minute wait; a failed reasoning check raises the tier 3 alert. A successful reasoning check clears both the tier 3 and the re-evaluation alerts. When tier 3 is enabled, the alerter also checks the decision provider by listing its models, which costs no tokens; a successful listing counts as a successful decision call, so the listing can clear the decision alert left open by a previous run. At startup, the alerter also clears any open provider alert for a tier or provider that is no longer configured or enabled. See System Alerts for how the alert appears.

The failure rate catches a provider that fails a steady share of its calls without failing consecutively, such as a provider that rejects every other request. The alerter keeps the outcomes of the last failure_rate_window calls for each tier and provider, and sets a limit of failure_rate times the window, rounded up; the defaults give a limit of 6 failed calls out of 20. The alerter raises the alert when the failed calls among those outcomes reach the limit, even before the window is full, and the five-minute wait still applies. A raised alert stays open while the failed calls remain at or above the limit, and clears on the first successful call that leaves fewer failed calls than the limit. A run of consecutive failures that reached the threshold counts as a single failed call when the provider recovers, provided that single failed call leaves the provider below the limit; an outage therefore clears on the first successful call, as the consecutive count alone would. Setting failure_rate to 1 leaves the consecutive count as effectively the only rule that raises the alert, provided failure_rate_window is at least failure_threshold.

In the following example, the alerter raises the alert after five consecutive failures, or when 10 of the last 40 calls have failed:

anomaly:
  provider_health:
    failure_threshold: 5
    failure_rate: 0.25
    failure_rate_window: 40