sentencepiece

Unsupervised text tokenizer and detokenizer
  https://github.com/google/sentencepiece
  0
  no reviews



SentencePiece is an unsupervised text tokenizer/detokenizer mainly designed for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training.