Byte Pair Encoding Text Tokenization R package tokenizers.bpe version 0.1.6 CRAN Task View: NaturalLanguageProcessing (https://CRAN.R-project.org/view=NaturalLanguageProcessing) Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library <https://github.com/VKCOM/YouTokenToMe> which is an implementation of fast Byte Pair Encoding (BPE) <https://aclanthology.org/P16-1162/>. Software Depends: R (>= 2.10) Imports: Rcpp (>= 0.11.5) LinkingTo: Rcpp Jan Wijffels <jwijffels@bnosac.be> Comprehensive R Archive Network (CRAN) Jan Wijffels [aut, cre, cph] (R wrapper), BNOSAC [cph] (R wrapper), VK.com [cph], Gregory Popovitch [ctb, cph] (Files at src/parallel_hashmap (Apache License, Version 2.0), The Abseil Authors [ctb, cph] (Files at src/parallel_hashmap (Apache License, Version 2.0), Ivan Belonogov [ctb, cph] (Files at src/youtokentome (MIT License)) MPL-2.0 2026-06-13 application/tgz https://CRAN.R-project.org/package=tokenizers.bpe doi:10.32614/CRAN.package.tokenizers.bpe