Changelog

All notable changes to pgEdge Vectorizer will be documented in this file.

The format is based on Keep a Changelog.

[Unreleased]

[1.1-beta2] - 2026-08-18

Fixed

• Fixed the extension failing to compile under compilers that default to C23 (GCC 15 and later), such as ubuntu:resolute's toolchain. pgedge_vectorizer.h declared pgedge_vectorizer_launcher_main() and pgedge_vectorizer_worker_main() as extern PGDLLEXPORT PGEDGE_NORETURN void ..., and on PostgreSQL 18 and later PGEDGE_NORETURN expands to PostgreSQL's pg_noreturn macro, which under C23 is itself [[noreturn]]. A C23 attribute-specifier-sequence must appear at the very start of a declaration, not between extern and the return type, so the declarations failed to parse with "expected identifier or '(' before 'void'" wherever the build compiled in C23 mode; under the older C11 _Noreturn function-specifier expansion the same ordering was merely unconventional, not an error, which is why this only surfaced now. Both declarations now put PGEDGE_NORETURN first, matching the ordering PostgreSQL's own core headers use for pg_noreturn (#65).

[1.1-beta1] - 2026-08-18

Security

• Restricted pgedge_vectorizer.provider, api_key_file, api_url and extra_headers to superusers (PGC_SUSET); they were settable by any session (#30). api_key_file names a file the backend opens as the server's operating system user, and api_url and extra_headers decide where its contents are sent and what accompanies them, so together they allowed a user who could reach the embedding functions to read any file the postgres user could read and have it delivered to a host of their choosing in an Authorization header. Changing provider alone was enough to present a superuser-configured key to a different vendor's endpoint. This is a behaviour change: an application relying on setting any of the four per session must now either run as a superuser, have the parameter delegated with GRANT SET ON PARAMETER (PostgreSQL 15 and later), or set it in postgresql.conf.
• Hardened API key file loading in OpenAI and Voyage providers (#17)
  • Reject API key files larger than 4096 bytes (MAX_API_KEY_FILE_SIZE) before reading, preventing unbounded memory allocation (CWE-20 / CWE-120)
  • Replaced the stat()-then-fopen() pattern with open(O_RDONLY|O_CLOEXEC)
  • fstat() + fdopen() to close a TOCTOU race where the file could be swapped between check and open
  • Added an S_ISREG() check to reject non-regular files (devices, FIFOs, directories) and a runtime byte counter that aborts the read if the file grows past the limit mid-read
• Stopped trusting the HOME environment variable when expanding ~ in API key paths; expand_tilde() now uses getpwuid(geteuid()) so an attacker-controlled environment cannot redirect the path (CWE-807)
• Replaced strncpy + manual null-termination in the background worker with strlcpy, guaranteeing null termination of the database name buffer (CWE-120)
• Pinned all GitHub Actions in the CI workflow to full commit SHAs rather than mutable @v4 tags, so a repointed tag in an upstream action cannot silently change what runs in CI (CWE-1357)
• Replaced strtok() with strtok_r() when parsing pgedge_vectorizer.databases in the background worker; strtok() keeps its parsing position in a single process-wide static, which any other strtok() caller reached from the same loop would corrupt (CWE-676)
• The OpenAI, Voyage and Gemini providers built their Authorization header into a fixed 512 byte buffer, leaving roughly 489 bytes for the key itself, whilst the API key file it can be loaded from is accepted up to 4096 bytes; a key longer than the buffer was silently truncated, producing an authentication failure indistinguishable from a wrong credential. JWT-style bearer tokens for OpenAI-compatible gateways routinely exceed a thousand characters, so this was reachable in practice, not theoretical. The header is now built with psprintf(), which cannot truncate at any length. A related fixed-size buffer parsing a provider's numeric literals is also removed: provider_parse_float_array() kept only the leading 32 characters of an overlong literal and scored on whatever atof() made of the fragment rather than reporting a bad response (#31).

Added

• Added pgedge_vectorizer.worker_service_quantum, which bounds how long a worker services one database before yielding its slot when there are more configured databases than workers. It is ignored when every configured database can have its own worker, in which case workers stay resident.
• pgedge_vectorizer.refresh_triggers() recreates the DELETE and TRUNCATE cleanup triggers for every registered vectorizer, returning the number recreated. Upgrading repairs existing tables automatically, so this is only needed if a trigger has been dropped by hand, or on an installation whose vectorized tables were created under a build predating those triggers.
• pgedge_vectorizer.vectorizers now records the document identifier column and its type, so cleanup triggers can be recreated without re-detecting it.
• BM25 now caches a chunk table's corpus size and mean document length per backend rather than deriving them on every call (#53). Both come from an aggregate that cannot use an index, so it was a full scan of the chunk table for every search and for every queue item the worker processed; a batch of ten scanned the same table ten times under a snapshot in which the answer could not have changed. Two settings bound how stale a cached reading may become, and whichever is reached first triggers a re-read:
  • pgedge_vectorizer.corpus_stats_cache_ttl (default 60 seconds) bounds it in time. Setting it to 0 disables the cache entirely, so the figures are read afresh on every call as they were before.
  • pgedge_vectorizer.corpus_stats_cache_max_uses_pct (default 5) bounds it in proportion to the corpus, which is how staleness actually harms ranking: a thousand chunks added to a million barely move the weights, while the same thousand added to two hundred change them several fold. Each use of a cached reading counts as one chunk that may have been added since — a proxy rather than a measurement, so searches spend the budget too. Setting it to 0 bounds by time alone.

A cached reading is also dropped and read again, whichever of the two bounds it is still within, when a term's document frequency comes back above the cached corpus size: a term cannot appear in more documents than exist, so that is proof the cached size is stale rather than merely old. An operator watching for the scan should expect it from this as well as from the two settings. If the fresh reading still contradicts the data then the statistics are inconsistent rather than stale, and re-reading cannot fix that, so the largest observed document frequency is adopted as the corpus size for the rest of that entry's life; the weights stay sane, and nothing rescans to no effect until the entry expires and the true size is read again.

The figures feed a ranking heuristic rather than an account that has to balance, and during ingest they are a moving target in any case, so holding one briefly costs a little precision in a number that was never precise. They are not maintained incrementally: chunk rows are inserted and deleted from many places across the SQL — chunking, content updates, recreate_chunks(), row deletion, truncation and disabling — and counters left wrong in any of them would skew ranking silently and permanently, whereas a cache that expires cannot drift for longer than its bounds and recovers by itself. - A background worker that exits within a second of being spawned is now treated as having failed to start, and its database is held off with a backoff that doubles from 5 seconds to a 5 minute cap, clearing once a worker for that database outlives the threshold. Without it, a configured database that does not exist, or that cannot be connected to (datallowconn = false), sent the launcher straight back round to spawn a replacement as fast as it could fork: measured at roughly 240 forks a second, sustained indefinitely, with nothing short of a restart to stop it (#48).

Changed

• pgedge_vectorizer.num_workers now sets the maximum number of concurrent workers rather than a fixed pool size, and can be changed with a reload instead of requiring a restart. Configurations that raised it purely to obtain coverage of every configured database no longer need to do so. Note that this is a change in meaning: a value chosen to guarantee coverage now caps concurrency instead.
• Background workers are now spawned per database by a launcher process rather than being statically registered at startup, so the set of serviced databases follows pgedge_vectorizer.databases as it changes, without a restart.
• pgedge_vectorizer.max_retries is now actually read (#26). It was declared and documented but never used anywhere; every queued embedding got its retry limit from queue.max_attempts's hardcoded DEFAULT 3 instead, regardless of the configured value. The GUC now sets max_attempts at every point a chunk is queued: on initial chunking, on content updates, and on recreate_chunks() and reprocess_chunks().
• BM25 IDF statistics tables no longer store total_docs or idf_weight. Both derive from the corpus size, which is a single value shared by every term, so materialising them per row meant that any change to the corpus invalidated every row at once and deleting a source row had to rewrite the entire vocabulary to keep them true. That rewrite took a lock on every term, so deletes of unrelated rows serialised against each other, and it ran even for statements that removed nothing. A delete now updates only the terms belonging to the rows it removed. The weights are computed when the statistics are read, so ranking is equivalent wherever a stored weight was current, and corrected wherever it had gone stale — which is the situation this change exists to fix.
• HTTP requests to embedding providers now reuse one curl handle per backend instead of opening a fresh connection for every request, and grow the response buffer geometrically rather than reallocating to the exact size on every write callback. Measured against a local stub over loopback, 200 requests dropped from 200 TCP connections to 1 and wall-clock time fell 31%; against a real provider each avoided connection also avoids a TLS handshake, typically one to two further round trips per batch (#28).
• Documented that changing pgedge_vectorizer.model to a model with a different embedding dimension breaks existing chunk tables, since the dimension is baked into the chunk table's embedding vector(N) column when it is created. The worker already detects this safely and marks the batch failed with an actionable Dimension mismatch warning, but recreate_chunks() cannot repair it: it deletes and requeues rows without touching the column type, so every requeued row fails identically. configuration.md, best_practices.md and troubleshooting.md now cover the constraint and both recovery routes: restoring the previous model, or rebuilding via disable_vectorization(..., drop_chunk_table => TRUE) followed by enable_vectorization() (#27).

Removed

• Removed the unused pgedge_vectorizer.auto_chunk GUC (#26). It was documented as disabling automatic chunking but was never checked anywhere; setting it to false had no effect. Making it do something real would mean gating trigger creation in enable_vectorization(), a larger behavioural change than removing a no-op setting, and is being tracked separately if wanted. SET and ALTER SYSTEM SET still accept pgedge_vectorizer.auto_chunk, since PostgreSQL permits any two-part name as a placeholder for an extension GUC it does not recognise, but reading it back with SHOW or current_setting() without having set it first in that session now fails with unrecognized configuration parameter, rather than quietly returning a value nothing acts on.

Fixed

• A BM25 term whose document frequency exceeded the corpus size is no longer dropped from the sparse vector altogether. The IDF is ln((N + 1) / (df + 0.5)), which goes negative once df passes N, and only scores above zero are kept, so the term vanished rather than merely being underweighted. df can outrun N whenever the two disagree: the corpus size may be a cached reading whilst every document frequency is read fresh, and a decrement that failed part way through a delete leaves the stored frequency too high indefinitely. The corpus size is now taken as at least the largest document frequency being weighted against it, which changes nothing in the ordinary case where N >= df, and a common term is scored at close to zero, which is what a term appearing in every document should carry. Because the worker writes what it computes into sparse_embedding, the dropped term was persisted rather than recomputed on the next search.
• Fixed BM25 length normalisation being driven by an incorrect average document length, which suppressed the sparse half of hybrid search. AVG() over the integer token_count column returns numeric, whose Datum is a pointer, and the result was read as a float8 — so bm25_avg_doc_len() returned a denormal (around 1e-315) rather than the real mean, and passed the non-negative guard because a denormal is positive. Dividing document length by that value made every term's frequency component collapse toward zero, so sparse scores were effectively flat and contributed almost nothing to the fused ranking. Present since hybrid search was introduced.
• Fixed databases beyond pgedge_vectorizer.num_workers never being processed (#23). Workers were assigned to databases by worker_id % db_count, and because worker_id only ranged over 0 to num_workers-1, any database at a later position in the list was silently never serviced: its queue accumulated entries that were never handled, with nothing logged to indicate it. With the default num_workers = 2 and five databases configured, three of them were affected.
• Deleting rows from a vectorized source table, or truncating it, no longer leaves orphaned chunks, embeddings, queue entries and BM25 document frequencies behind (#24). Vectorization previously installed an AFTER INSERT OR UPDATE trigger only, so removing source data left every piece of derived data in place: vector and hybrid search returned hits pointing at rows that no longer existed, the queue spent embedding API calls on chunks for deleted rows, and the IDF weighting drifted for the whole corpus, distorting relevance ranking for every query rather than only those touching deleted rows.

Vectorization now installs three triggers per column, adding an AFTER DELETE and an AFTER TRUNCATE trigger. The DELETE trigger is statement-level and uses a transition table, so a bulk delete stays set-based rather than doing per-row work. Upgrading also repairs tables that were already vectorized, which otherwise would have kept leaking silently. - Fixed enable_vectorization() and recreate_chunks() failing with type of parameter N does not match that when preparing the plan when called for a table with one primary key type and then, in the same session, a table with a different primary key type (#39). Both functions loop over existing source rows into a RECORD variable and pass one of its fields to a dynamic query; PL/pgSQL fixes that field's parameter type the first time the statement runs, and the type mismatch broke any second vectorized table whose primary key differed. The failure aborted partway through, after the chunk table and trigger had already been created, leaving the vectorizer for that column half set up. - Querying via bm25_query_vector() from inside a statement that owns relations, such as CREATE TABLE AS, could crash the backend with relcache reference ... is not owned by resource owner TopTransaction, because bm25_load_idf_stats() and bm25_update_idf_stats() restored the memory context and resource owner on their error path but not on the ordinary return from the IDF subtransaction. Loading the IDF statistics also parented its lookup hash on TopMemoryContext rather than the caller's context, so any error reached before the hash was destroyed leaked roughly 8 KB for the life of the session; the worker's per-chunk failure handler took this path on every retry of a failing item. Both are fixed by scoping the hash to the caller's context and restoring the saved context and resource owner on every path out of the subtransaction, not only the error path (#47). - Fixed the worker claiming queue items ordered by attempts DESC, created_at, which put items that had already failed the most at the head of every batch. A provider outage left a backlog of once-failed items jumping ahead of newly queued work on every poll until each exhausted max_attempts, so freshly queued embeddings could wait behind a backlog of doomed retries. The claim is now ordered by created_at alone; next_retry_at already spaces retries out, so age is the only ordering the queue needs (#49). - BM25's tokenizer and IDF tables both key terms on BM25_MAX_TERM_LEN (128) bytes, and two distinct terms sharing that much of a prefix silently merged into one dynahash entry: a single token carrying their combined frequency, and a query for either term matching the other. Terms that do not fit the key are now skipped at tokenization instead, on both the indexing and query paths, so nothing is matched by a truncation the other side does not produce (#50). - A queue item that failed deterministically was reclaimed and retried on every poll forever: the failure was recorded by an UPDATE issued from inside the transaction the failure had already aborted, so it never ran, attempts never advanced past 0, next_retry_at was never set and max_attempts was never reached. That in turn defeated the queue's per-item backoff, its one-at-a-time handling of a suspect item, and its eventual retirement of one that cannot succeed, and it billed every item sharing the batch for embeddings fetched and discarded on every cycle. The failure is now noted before the aborting error is raised and recorded in a fresh transaction afterwards, matched against the row's state as this worker last saw it so a row another worker has since claimed or completed is not disturbed. Capturing the message safely took two further corrections: copying it out of ErrorContext needed a context switch first, since the copy otherwise landed in the context FlushErrorState() was about to reset and could crash the backend outright; and clipping an over-length message into its 1024 byte buffer needed to stop on a character boundary, since a byte-based strlcpy() could leave a partial multi-byte character in queue.error_message, which then made length(), substring() and any client-side decoding of that row raise.

Failures that cannot be attributed to any single item, such as a misconfigured or unreachable provider, are not charged to a row for the same reason as above, and previously spun in exactly the same way: the same batch was reclaimed and failed identically on every poll, filling the log at whatever rate pgedge_vectorizer.worker_poll_interval allowed. This case now backs off too, 5 seconds doubling to a 5 minute cap, cleared by any batch that gets through or by a reload (#51, #52). - Fixed chunk boundaries landing inside a multi-byte character when pgedge_vectorizer.strip_non_ascii is disabled. get_char_offset_for_tokens() counted UTF-8 lead bytes and returned an offset one byte past the last one, so callers cut the source text there and stored the resulting fragment; a fragment ending mid-character then made length() and similar functions raise on the affected chunk, and enable_vectorization() could abort partway through processing existing rows. The offset is now clipped to the last character that fits entirely via pg_mbcharcliplen(), reading the server's actual encoding rather than assuming UTF-8. This affects token_based, markdown and hybrid chunking alike, all of which reach the boundary through the same function; the default strip_non_ascii = on was not affected, since it removes every non-ASCII byte before chunking (#64).

[1.0] - 2026-03-13

Added

• Support for any single-column primary key type in vectorized tables (#11)
  • Auto-detect PK column name and type from pg_index instead of hardcoding BIGINT
  • Chunk table source_id column now matches the source table's PK type (UUID, TEXT, VARCHAR(n), etc.)
  • New source_pk parameter on enable_vectorization() for explicit column selection
  • Composite primary key tables supported by specifying source_pk explicitly

Fixed

• Fixed stale embeddings and orphaned queue entries on content update (#12)
  • Queue entries are now cleaned up before deleting chunks in the vectorization trigger
  • Stale high-index chunks are cleaned up when re-enabling vectorization
  • Worker warns when a chunk is deleted by a concurrent source update
• Fixed queue processing not starting until SIGHUP after CREATE EXTENSION (#10)
  • Workers now use exponential backoff (5s, 10s, 20s, ... up to 5 min) when checking for extension installation, instead of a fixed 5-minute sleep
  • Extension is discovered within seconds of running CREATE EXTENSION, no SIGHUP needed
  • Improved log messages with actionable hints on first check failure

[1.0-beta2] - 2026-01-13

Added

• Hybrid chunking strategy (hybrid) inspired by Docling's approach
  • Parses markdown structure (headings, code blocks, lists, blockquotes, tables)
  • Preserves heading context hierarchy in each chunk for better RAG retrieval
  • Two-pass refinement: splits oversized chunks, merges undersized consecutive chunks with same context
  • Significantly improves retrieval accuracy for structured documents
• Markdown chunking strategy (markdown) - structure-aware without refinement passes
  • Simpler and faster alternative to hybrid
  • Good balance of structure awareness and performance
• Automatic fallback detection for hybrid and markdown strategies
  • Detects if content is likely markdown based on syntax patterns
  • Falls back to token_based chunking for plain text to avoid overhead
  • Ensures optimal strategy is always used regardless of content type

Fixed

• Fixed potential buffer over-read vulnerabilities in markdown detection
• Fixed infinite recursion in markdown/hybrid fallback when content is plain text

[1.0-beta1] - 2025-12-15

Changed

• Promoted to beta status after extensive testing and bug fixes

Fixed

• Fixed table name reference in vectorization code

[1.0-alpha5] - 2025-12-12

Fixed

• Fixed token-based chunking producing corrupted chunks when overlap > 0 (chunks would start mid-word like "ntence." instead of proper word boundaries)
• Fixed potential negative index access in find_good_break_point() function

[1.0-alpha4] - 2025-12-08

Fixed

• Fixed uninitialized dimension variable in generate_embedding() that caused spurious "Dimension mismatch" errors with random dimension values

[1.0-alpha3] - 2025-12-03

Added

• Added a garbage collector to automatically delete old queue entries based on the age defined in the pgedge_vectorizer.auto_cleanup_hours GUC.
• generate_embedding() function for generating embeddings from query text directly in SQL

[1.0-alpha2] - 2025-12-02

Added

• PostgreSQL 18 support

Changed

• Updated pgvector dependency to v0.8.1 for PostgreSQL 18 compatibility

[1.0-alpha1] - 2025-11-21

Added

• Initial release of pgEdge Vectorizer
• Automatic text chunking with configurable strategies (token_based, semantic, markdown)
• Background worker processing for asynchronous embedding generation
• Support for multiple embedding providers:
  • OpenAI (text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002)
  • Voyage AI (voyage-2, voyage-large-2, voyage-code-2)
  • Ollama (nomic-embed-text, mxbai-embed-large, all-minilm)
• Multi-column vectorization support
• Queue management with monitoring views (queue_status, failed_items, pending_count)
• Maintenance functions:
  • enable_vectorization() - Enable automatic vectorization for a table column
  • disable_vectorization() - Disable vectorization
  • chunk_text() - Manual text chunking
  • retry_failed() - Retry failed queue items
  • clear_completed() - Remove completed items from queue
  • reprocess_chunks() - Queue existing chunks for reprocessing
  • recreate_chunks() - Complete rebuild of chunks from source
  • show_config() - Display configuration settings
• Configurable chunking parameters (chunk_size, chunk_overlap)
• Automatic retry with exponential backoff
• Batch processing for efficient API usage
• Non-ASCII character stripping option
• Comprehensive test suite with pg_regress