↳ In-reply-to » reviewing logs this morning and found i have been spammed hard by bots not respecting the robots.txt file. only noticed it because the OpenAI bot was hitting me with a lot of nonsensical requests. here is the list from last month:

@lyse@lyse.isobeef.org yeah, i have the following as well:

User-agent: *
Disallow: /

now i some middleware that looks at the header, and if they are polite enough to include “bot” in the user agent, they politely get a 404 response.