Security

Combating web crawling bots

Attempting to implement throttling and ip restriction for reducing web crawling by bots who ignore or do not comply with robots.txt.

There is much discussion of this issue in the Geo4Lib and other related slack channels and other community dicussions.

We’re attempting to implement Bot Challenge Page from the Samvera community.

We’ve set up a Cloudflare Turnstile under my account, but we might want to reconsider its ownership once it’s in production. Maybe set up a digilib account?