python3-tiktoken

fast BPE tokeniser for use with OpenAI's models
  https://github.com/openai/tiktoken
  0
  no reviews


A fast Byte pair encoding tokeniser for use with OpenAI's models.