A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker R package robotstxt version 0.7.15 CRAN Task View: WebTechnologies (https://CRAN.R-project.org/view=WebTechnologies) Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain. Software Depends: R (>= 3.0.0) Imports: stringr (>= 1.0.0), httr (>= 1.0.0), spiderbar (>= 0.2.0), future.apply (>= 1.0.0), magrittr, utils Suggests: knitr, rmarkdown, dplyr, testthat, covr, curl Pedro Baltazar <pedrobtz@gmail.com> Comprehensive R Archive Network (CRAN) Pedro Baltazar [aut, cre], Peter Meissner [aut], Kun Ren [aut, cph] (Author and copyright holder of list_merge.R.), Oliver Keys [ctb] (original release code review), Rich Fitz John [ctb] (original release code review) MIT + file LICENSE (https://CRAN.R-project.org/package=robotstxt/LICENSE) 2024-08-29 application/tgz https://CRAN.R-project.org/package=robotstxt doi:10.32614/CRAN.package.robotstxt