Relative Content

Tag Archive for tokenizeopenai-apihuggingface-tokenizers

Can I increase tiktoken throughput?

Hello I’m trying to speed up processing when using tiktoken. Is by default a limitation set regarding the processing of documents using tiktoken or can i somehow change thread settings? Would appreciate if someone knows.