Skip to content
This documentation is sourced from a third-party project and is not maintained by pgEdge.

Tokenizer

Tokenizer configuration is a critical aspect of effective text processing, significantly impacting the performance and accuracy. Here are some key considerations and options to help you choose the right tokenizer for your use case.