↳
In-reply-to
»
@lyse Found it and fixed it! 🎉 The crawler's discovery spider was fetching every feed a second time, without any conditional headers (plus a couple of other politeness bugs: redirected feed URLs never stored their cache validators, and there was no floor between re-fetches). Now every feed is fetched at most once per crawl, always with If-Modified-Since / If-None-Match, and never more than once per 15m no matter what. Just deployed — please keep an eye on your access logs and let me know if you still see anything impolite from the crawler 🙏
⤋ Read More
@lyse@lyse.isobeef.org Good! 👍