Alerter Configuration

The pgEdge AI DBA Workbench Alerter supports configuration through YAML files and command-line flags.

Configuration Precedence

The alerter applies configuration settings in the following order; later sources override earlier ones:

1. Default values built into the application.
2. Configuration file settings (YAML format).
3. Command-line flag overrides.

The alerter overrides a configuration file setting only when you actually pass the corresponding flag; a flag you omit never overrides the file, even though the flag has a built-in default. A flag you do pass always wins, including when the value you give it happens to equal that flag's default.

Configuration File

The alerter searches for its configuration file in the following order:

1. The path specified via the -config flag.
2. The per-user config directory at ~/.config/pgedge/ai-dba-alerter.yaml on Linux (honouring $XDG_CONFIG_HOME), ~/Library/Application Support/pgedge/ai-dba-alerter.yaml on macOS, and %AppData%\pgedge\ai-dba-alerter.yaml on Windows.
3. /etc/pgedge/ai-dba-alerter.yaml (system-wide).

If -config is set and the file is missing, the alerter exits with an error. If -config is not set and none of the default locations contain a configuration file, the alerter uses built-in defaults silently. The alerter no longer searches the binary directory or the current working directory. A SIGHUP signal re-runs discovery on each reload, so a configuration file installed at a default location after startup is picked up on the next signal.

A complete example configuration file is available at ai-dba-alerter.yaml in the project repository.

Command-Line Flags

The alerter accepts the following command-line flags:

In the following example, the alerter starts with debug logging and a custom configuration file:

./ai-dba-alerter -debug \
    -config /etc/ai-workbench/alerter.yaml

In the following example, the alerter connects to a specific database without a configuration file:

./ai-dba-alerter \
    -db-host db.example.com \
    -db-name ai_workbench \
    -db-user alerter \
    -db-password secret

Configuration File Reference

The configuration file uses YAML format. The following sections describe all available options.

Datastore Connection (datastore)

The datastore section configures the connection to the AI DBA Workbench PostgreSQL datastore.

The sslmode option accepts the following values:

• disable disables SSL encryption.
• allow attempts non-SSL first and falls back to SSL.
• prefer attempts SSL first and falls back to non-SSL.
• require requires SSL without certificate verification.
• verify-ca requires SSL and verifies the server certificate.
• verify-full requires SSL and verifies the certificate and hostname.

In the following example, the datastore section configures a secure connection with certificate verification:

datastore:
  host: db.example.com
  database: ai_workbench
  username: ai_workbench
  password_file: /etc/ai-workbench/password.txt
  port: 5432
  sslmode: verify-full
  sslcert: /etc/ai-workbench/client-cert.pem
  sslkey: /etc/ai-workbench/client-key.pem
  sslrootcert: /etc/ai-workbench/ca-cert.pem

Connection Pool (pool)

The pool section configures the database connection pool.

Threshold Evaluation (threshold)

The threshold section configures threshold-based alert evaluation.

The trigger_count and clear_count options stop an alert from being raised and cleared repeatedly when a metric hovers near its threshold. The alerter raises an alert only after the threshold has been breached on trigger_count consecutive metric samples, and clears an active alert only after the metric has been back within the threshold on clear_count consecutive samples. A sample is one collection by the collector rather than one evaluation cycle; the evaluator and the alert cleaner read the same sample several times between collections, but each sample counts once. A sample that breaches the threshold starts the clear count again, and one that does not breach it starts the trigger count again.

Setting both options to 1 restores the earlier behaviour of raising and clearing on a single sample. The counts are held in memory, so restarting the alerter starts them again from zero; at worst a restart delays a raise or a clear by a few samples. The counts do not apply to the metric_staleness and probe_unavailable rules or to connection error alerts, which do not judge metric samples.

Anomaly Detection (anomaly)

The anomaly section configures the tiered anomaly detection system.

Tier 1: Statistical Analysis

The anomaly.tier1 section configures z-score-based statistical detection.

The anomaly.tier1.clear_count option sets how many consecutive samples of the metric behind an active anomaly alert must score back inside the sensitivity band before the alerter clears the alert. Only a newly collected sample counts; tier 1 usually evaluates more often than the probe collects, and an evaluation that re-scores a sample already counted does not add to the count. The value must be at least 1; the alerter refuses to start, or to apply a reload, with a smaller value. For a probe that collects every 300 seconds, the default of 3 clears an alert about ten minutes after the first in-band sample. A SIGHUP reload applies a new value from the next evaluation.

Tier 1: Variance Floor, Warmup, and Z-Score Cap

Three additional blocks under anomaly.tier1 prevent the detector from firing on baselines that have not yet stabilised. These blocks are most relevant on young datastores; on a datastore with one or two days of metric history, a baseline's stored standard deviation can collapse far below the metric's natural variation, producing z-scores in the thousands. The variance floor and warmup gate suppress this failure mode, and the z-score cap acts as defence in depth.

The anomaly.tier1.max_z_score option clamps the absolute z-score symmetrically around zero before the sensitivity comparison. The default is 100.0; any genuine outlier sits well below this value, and the cap simply prevents a runaway divisor from generating multi-thousand-sigma scores. Setting max_z_score: 0 disables the cap.

The anomaly.tier1.variance_floor block enforces a minimum divisor on the z-score calculation. The effective standard deviation is the larger of the raw stored value and a hybrid floor; the floor itself is the larger of relative_pct times the absolute baseline mean and absolute_floor. The relative term dominates for non-zero metrics, while the absolute term acts as a safety net when the mean approaches zero. Setting both relative_pct: 0 and absolute_floor: 0 disables the floor entirely; the detector then falls back to the existing stddev == 0 guard.

The anomaly.tier1.warmup block suppresses detection for baselines that have not accumulated enough samples or enough wall-clock observation time. Each period_type (all, hourly, and daily) has its own pair of thresholds, and a baseline is considered warm only when both are met. The all baseline defaults require roughly one day of operation at the 60 second collection interval; the hourly and daily defaults require enough span for each time bucket to have been observed multiple times. Setting both min_samples: 0 and min_span_hours: 0 for a given period_type disables warmup suppression for that type.

In the following example, the anomaly.tier1 section tightens the variance floor and extends the all warmup window:

anomaly:
  tier1:
    max_z_score: 100.0
    variance_floor:
      relative_pct: 0.10
      absolute_floor: 0.005
    warmup:
      all:
        min_samples: 200
        min_span_hours: 48
      hourly:
        min_samples: 5
        min_span_hours: 120
      daily:
        min_samples: 3
        min_span_hours: 336

Warmup suppressions are recorded at debug log level only; the detector does not write a candidate row or an alert when it skips a cold baseline, and a fall-through from a cold hourly or daily row to a warm, less specific one is logged at the same level. On the long-term test host, enable debug logging in the alerter and inspect recent suppressions with sudo journalctl -u ai-workbench-alerter.service --since 10m. The log line names the connection, metric, period type, and sample count, which is enough to confirm whether a missing alert reflects warmup suppression or a genuinely quiet metric.

Tier 2: Embedding Similarity

The anomaly.tier2 section configures pgvector-based similarity search for pattern matching.

Tier 3: LLM Classification

The anomaly.tier3 section configures LLM-based classification for complex anomalies.

Provider Health (provider_health)

The anomaly.provider_health section controls the system alert that the alerter raises when calls to the embedding or reasoning provider keep failing. Whilst a provider fails, anomaly detection runs in a degraded mode: tier 2 compares no candidate with past anomalies, tier 3 raises each candidate that reaches the tier without LLM classification, and the re-evaluation worker leaves acknowledged anomaly alerts unchanged.

The following table describes the provider health options:

The alerter counts consecutive failures separately for tier 2 embeddings, tier 3 classification and re-evaluation, and for each provider. When one count reaches the threshold, the alerter raises a single warning alert for that tier and provider, and updates the alert's failure count as further calls fail. The next successful call to that provider resets the count and clears the alert, unless the failure rate described below keeps the alert open; tier 3 classification and re-evaluation share the reasoning provider, so a successful call from either clears the alerts of both. After clearing an alert, the alerter does not raise it again for five minutes, so that a provider that fails intermittently cannot send a burst of notifications. A call cancelled before it completes counts as neither a failure nor a success, although a call that times out counts as a failure. A tier 3 or re-evaluation call also counts as a failure when the provider answers with a response that holds no decision the alerter can parse. The failure is recorded whatever the alerter then does with the candidate or alert, which may be to act on a keyword found in the response or to take the tier's default decision.

When anomaly detection is enabled, the alerter also checks the providers of the enabled tiers at startup. A failed check raises the alert for its tier immediately, whatever the threshold or the five-minute wait; a failed reasoning check raises the tier 3 alert. A successful reasoning check clears both the tier 3 and the re-evaluation alerts. At startup the alerter also clears any open provider alert for a tier or provider that is no longer configured or enabled. See System Alerts for how the alert appears.

The failure rate catches a provider that fails a steady share of its calls without failing consecutively, such as a provider that rejects every other request. The alerter keeps the outcomes of the last failure_rate_window calls for each tier and provider, and sets a limit of failure_rate times the window, rounded up; the defaults give a limit of 6 failed calls out of 20. The alerter raises the alert when the failed calls among those outcomes reach the limit, even before the window is full, and the five-minute wait still applies. A raised alert stays open whilst the failed calls remain at or above the limit, and clears on the first successful call that leaves fewer failed calls than the limit. A run of consecutive failures that reached the threshold counts as a single failed call once the provider recovers, provided that single failed call leaves the provider below the limit; an outage therefore clears on the first successful call, as the consecutive count alone would. Setting failure_rate to 1 leaves the consecutive count as effectively the only rule that raises the alert, provided failure_rate_window is at least failure_threshold.

In the following example, the alerter raises the alert after five consecutive failures, or once 10 of the last 40 calls have failed:

anomaly:
  provider_health:
    failure_threshold: 5
    failure_rate: 0.25
    failure_rate_window: 40

Baseline Calculation (baselines)

The baselines section configures baseline metric calculation for anomaly detection.

The lookback also caps how old a baseline's earliest sample can be, because each refresh reads only samples inside the window and rewrites earliest_sample_at from them. A warmup tier whose min_span_hours exceeds lookback_days multiplied by 24 can therefore never become warm, and the alerter logs a warning on each baseline cycle naming the tier. The default of 15 days covers the daily tier's 336 hour span with a day to spare and gives every weekday at least two occurrences in the window.

Correlation (correlation)

The correlation section configures alert correlation across metrics.

LLM Providers (llm)

The llm section configures LLM providers for tier 3 anomaly detection and embedding generation.

Each provider section names its own embedding model, and the alerter accepts any model name the chosen provider recognises rather than checking the name against a fixed list. The defaults in the tables below apply when the configuration leaves the model empty, and an OpenAI-protocol-compatible local model server can supply embeddings under whatever model name that server uses.

The configured embedding model must not produce vectors with more than 4000 dimensions. The alerter stores anomaly embeddings as halfvec(4000) and zero-pads each embedding to 4000 dimensions; 4000 is pgvector's HNSW index limit for the halfvec type. The alerter does not check the width of a model at startup: when the model turns out to be wider than 4000 dimensions, the alerter never truncates the vector, but instead logs an error naming the vector width and the limit for each anomaly candidate it processes, stores no embedding for the candidate, and passes the candidate straight to tier 3. Watch the alerter log after changing the embedding model, because the error first appears when the first anomaly candidate reaches tier 2, not when the alerter starts. Storing embeddings as halfvec requires pgvector 0.7.0 or newer; upgrading an existing installation against an older pgvector causes the embedding-column migration to fail.

Output Token Budget (max_tokens)

The max_tokens option caps the number of output tokens a reasoning model may generate for a single call. The budget applies to the following reasoning work:

• The tier 3 anomaly classification applies the budget to each candidate that the alerter examines.
• The re-evaluation of acknowledged alerts applies the budget to each alert that the alerter revisits.

The default value is 4096 tokens; a value less than or equal to zero falls back to the default. Earlier releases capped both calls at 500 tokens and offered no way to change the limit.

A reasoning model counts its internal thinking tokens against this same budget. A budget that suits a conventional model can therefore leave no room for the verdict itself. The model then returns a response that contains no text, which the alerter treats as a tier 3 failure. A failed tier 3 classification fails safe and raises the alert; the alerter records the cause in the candidate's tier 3 error field. Raise max_tokens when these failures appear; alternatively, select a model that emits less reasoning.

In the following example, the llm section raises the budget for a reasoning model that a local llama.cpp server hosts:

llm:
  reasoning_provider: openai
  max_tokens: 8192
  openai:
    base_url: http://localhost:8080/v1
    reasoning_model: qwen3-8b

Ollama Configuration

The llm.ollama section configures the local Ollama provider.

OpenAI Configuration

The llm.openai section configures the OpenAI provider.

The openai provider works with any server that implements the OpenAI-compatible API. Set base_url to point at a local inference server and set embedding_model to the model that server provides. The API key is optional when using a custom base URL.

The following local inference servers are compatible:

• Docker Model Runner uses http://localhost:12434/engines/llama.cpp/v1 as the default endpoint.
• llama.cpp uses http://localhost:8080/v1 as the default endpoint.
• LM Studio uses http://localhost:1234/v1 as the default endpoint.
• EXO uses http://localhost:52415/v1 as the default endpoint.

In the following example, the llm.openai section configures a local llama.cpp server:

llm:
  reasoning_provider: openai
  openai:
    base_url: http://localhost:8080/v1
    reasoning_model: my-local-model

A local server that hosts a reasoning model usually needs a larger output budget than the default. For details, see Output Token Budget.

Anthropic Configuration

The llm.anthropic section configures the Anthropic provider.

Gemini Configuration

The llm.gemini section configures the Google Gemini provider. Model availability varies by Gemini API key tier; run ListModels to verify which models a given key can access.

When Gemini is the embedding provider, the embedding_model must match the model that the KB Builder used to produce the knowledgebase; the KB Builder uses gemini-embedding-001.

Voyage Configuration

The llm.voyage section configures the Voyage provider for embeddings.

Notifications (notifications)

The notifications section configures the notification delivery system for sending alerts through external channels.

The secret_file option specifies a file containing the same plain text secret used by the server component. The alerter uses this secret to decrypt notification channel credentials that the server encrypted. The alerter and the server must reference the same secret file.

In the following example, the notifications section enables delivery with custom retry settings:

notifications:
  enabled: true
  secret_file: /etc/ai-workbench/ai-dba-server.secret
  process_interval_seconds: 30
  max_retry_attempts: 5
  retry_backoff_minutes: [5, 15, 30]
  http_timeout_seconds: 60

API Key Management

Store API keys for LLM providers in files with restricted permissions. The alerter reads API keys from the paths specified in the api_key_file options.

In the following example, the commands create an API key file with secure permissions:

echo "sk-your-api-key-here" \
    > /etc/ai-workbench/openai-api-key.txt
chmod 600 /etc/ai-workbench/openai-api-key.txt

The corresponding configuration references the key file:

llm:
  embedding_provider: openai
  reasoning_provider: openai
  openai:
    api_key_file: /etc/ai-workbench/openai-api-key.txt
    embedding_model: text-embedding-3-small
    reasoning_model: gpt-4o

Signal Handling

The alerter responds to Unix signals for operational control:

• SIGINT and SIGTERM trigger a graceful shutdown.
• SIGHUP reloads the configuration file without restarting the process.

In the following example, the kill command reloads the configuration:

kill -HUP $(pidof ai-dba-alerter)