Fast, Consistent Tokenization of Natural Language Text R package tokenizers version 0.3.0 CRAN Task View: NaturalLanguageProcessing (https://CRAN.R-project.org/view=NaturalLanguageProcessing) Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'. Software Depends: R (>= 3.1.3) Imports: stringi (>= 1.0.1), Rcpp (>= 0.12.3), SnowballC (>= 0.5.1) LinkingTo: Rcpp Suggests: covr, knitr, rmarkdown, stopwords (>= 0.9.0), testthat Lincoln Mullen <lincoln@lincolnmullen.com> Comprehensive R Archive Network (CRAN) Lincoln Mullen [aut, cre] (ORCID: <https://orcid.org/0000-0001-5103-6917>), Os Keyes [ctb] (ORCID: <https://orcid.org/0000-0001-5196-609X>), Dmitriy Selivanov [ctb], Jeffrey Arnold [ctb] (ORCID: <https://orcid.org/0000-0001-9953-3904>), Kenneth Benoit [ctb] (ORCID: <https://orcid.org/0000-0002-0797-564X>) MIT + file LICENSE (https://CRAN.R-project.org/package=tokenizers/LICENSE) 2022-12-22 application/tgz https://CRAN.R-project.org/package=tokenizers doi:10.32614/CRAN.package.tokenizers