Fast Text Tokenization R package tok version 0.2.3 Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts. Software Depends: R (>= 4.2.0) Imports: R6, cli Suggests: rmarkdown, testthat (>= 3.0.0), hfhub (>= 0.1.1), withr Tomasz Kalinowski <tomasz@posit.co> Comprehensive R Archive Network (CRAN) Tomasz Kalinowski [ctb, cre], Daniel Falbel [aut], Regouby Christophe [ctb], Posit [cph] MIT + file LICENSE (https://CRAN.R-project.org/package=tok/LICENSE) 2026-06-21 application/tgz https://CRAN.R-project.org/package=tok doi:10.32614/CRAN.package.tok