Skip to content

Troubleshooting

Common issues and solutions when using the pgEdge Document Loader.

Connection Issues

Cannot Connect to Database

Error:

Error: failed to connect to database: failed to ping database:
connection refused

Solutions:

  1. Verify PostgreSQL is running:
pg_isready -h localhost -p 5432
  1. Check host and port settings:
pgedge-docloader --db-host localhost --db-port 5432 ...
  1. Verify network connectivity:
telnet localhost 5432
  1. Check PostgreSQL is accepting connections (in postgresql.conf):
listen_addresses = '*'

Authentication Failed

Error:

Error: failed to connect to database: pq: password authentication
failed for user "myuser"

Solutions:

  1. Verify password is correct

  2. Set password using environment variable:

export PGPASSWORD=your_password

  1. Check pg_hba.conf allows the connection method

  2. Verify user exists:

SELECT usename FROM pg_user WHERE usename = 'myuser';

SSL/TLS Issues

Error:

Error: failed to connect to database: pq: SSL is not enabled on the
server

Solutions:

  1. Change SSL mode to disable if SSL is not needed:
pgedge-docloader --db-sslmode disable ...
  1. Or enable SSL in PostgreSQL (postgresql.conf):
ssl = on
  1. Verify certificate paths are correct:
ls -la /path/to/cert.pem /path/to/key.pem

File Processing Issues

No Files Processed

Error:

Processing files from: ./docs
No documents to process.

Solutions:

  1. Verify source path exists:
ls -la ./docs
  1. Check glob pattern syntax:
pgedge-docloader --source "./docs/*.md" ...

  1. Ensure files have supported extensions (.html, .htm, .md, .rst)

  2. List supported formats:

pgedge-docloader formats

Unsupported File Type

Error (single file):

Error: unsupported file type: document.txt

Solution:

Convert the file to a supported format or use a different file.

Message (directory):

Skipping unsupported file: readme.txt

Solution:

This is informational. Unsupported files in directories/globs are skipped automatically.

Database Issues

Table Does Not Exist

Error:

Error: failed to insert documents: pq: relation "documents" does not
exist

Solutions:

  1. Create the table first (see Database Setup)

  2. Verify table name is correct:

\dt
  1. Check schema if using non-public schema:
--db-table myschema.documents

Column Does Not Exist

Error:

Error: failed to insert documents: pq: column "content" of relation
"documents" does not exist

Solutions:

  1. Verify column mapping matches table structure:
\d documents
  1. Update column mapping:
--col-doc-content actual_column_name

Permission Denied

Error:

Error: failed to insert documents: pq: permission denied for table
documents

Solutions:

  1. Grant necessary permissions:
GRANT SELECT, INSERT, UPDATE ON documents TO myuser;
GRANT USAGE, SELECT ON SEQUENCE documents_id_seq TO myuser;
  1. Verify user has permissions:
SELECT grantee, privilege_type
FROM information_schema.role_table_grants
WHERE table_name = 'documents';

Duplicate Key Violation

Error:

Error: failed to insert documents: pq: duplicate key value violates
unique constraint "documents_filename_key"

Solutions:

  1. Use update mode to update existing rows:
pgedge-docloader --update ...
  1. Or delete existing rows:
DELETE FROM documents WHERE filename = 'duplicate.md';
  1. Or remove UNIQUE constraint if not needed

Type Mismatch

Error:

Error: failed to insert documents: pq: column "source" is of type bytea
but expression is of type text

Solutions:

  1. Verify column types match expectations (see Database Setup)

  2. For source content, use BYTEA type:

ALTER TABLE documents
ALTER COLUMN source TYPE bytea
USING source::bytea;

Configuration Issues

Config File Not Found

Error:

Error: failed to load configuration: failed to read config file: open
config.yml: no such file or directory

Solutions:

  1. Verify config file path:
ls -la config.yml
  1. Use absolute path:
pgedge-docloader --config /full/path/to/config.yml

Invalid YAML Syntax

Error:

Error: failed to load configuration: failed to read config file: yaml:
line 5: could not find expected ':'

Solutions:

  1. Validate YAML syntax:
yamllint config.yml

  1. Check for correct indentation (use spaces, not tabs)

  2. Ensure colons are followed by spaces

Missing Required Options

Error:

Error: failed to load configuration: source path is required

Solutions:

  1. Provide required options via config file or command line

  2. Required options:

  3. source
  4. db-host
  5. db-name
  6. db-user
  7. db-table
  8. At least one column mapping

Performance Issues

Slow Processing

If processing is slow:

  1. Database connection: Use connection pooling (already enabled)

  2. Network latency: Use local database or faster network connection

  3. Indexes: Ensure appropriate indexes exist (see Database Setup)

High Memory Usage

For large documents:

  1. Process files in smaller batches

  2. Use glob patterns to process subsets:

pgedge-docloader --source "./docs/section1/*.md" ...
pgedge-docloader --source "./docs/section2/*.md" ...

Getting Help

If you encounter issues not covered here:

  1. Check the Usage guide

  2. Verify your Configuration

  3. Review Database Setup

  4. Run with verbose output (if available in future versions)

  5. Report issues at: https://github.com/pgedge/pgedge-docloader/issues

Common Debugging Steps

  1. Test database connection:
psql -h localhost -U myuser -d mydb
  1. Test with minimal config:
pgedge-docloader \
  --source testdata/sample.md \
  --db-host localhost \
  --db-name test \
  --db-user postgres \
  --db-table test_docs \
  --col-doc-content content \
  --col-file-name filename
  1. Check PostgreSQL logs:
tail -f /var/log/postgresql/postgresql-*.log
  1. Verify table structure:
\d+ documents
  1. Test manual insert:
INSERT INTO documents (content, filename)
VALUES ('test', 'test.md');

Next Steps