Parse and Test Robots Exclusion Protocol Files and Rules R package spiderbar version 0.2.5 CRAN Task View: WebTechnologies (https://CRAN.R-project.org/view=WebTechnologies) The 'Robots Exclusion Protocol' <https://www.robotstxt.org/orig.html> documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' <https://github.com/seomoz/rep-cpp> C++ library for processing these 'robots.txt' files. Software Depends: R (>= 3.2.0) Imports: Rcpp LinkingTo: Rcpp Suggests: covr, robotstxt, tinytest Bob Rudis <bob@rud.is> Comprehensive R Archive Network (CRAN) Bob Rudis (bob@rud.is) [aut, cre], SEOmoz, Inc [aut] MIT + file LICENSE (https://CRAN.R-project.org/package=spiderbar/LICENSE) 2023-02-11 application/tgz https://CRAN.R-project.org/package=spiderbar doi:10.32614/CRAN.package.spiderbar