Hey folks 👋 Today I announce the re-release of the Twtxt Search Engine now live and running and actively re-crawling and re-indexing. 🎉 Please report bugs or any useability issues to me! 🙏 #Twtxt #Search
@prologic@twtxt.net That’s cool to hear!
Looking at my access logs, something seems to be off, though:
2026-07-04T04:11:26+02:00 200
2026-07-04T05:16:09+02:00 304
2026-07-04T06:33:34+02:00 304
2026-07-04T06:37:46+02:00 304 # just four minutes since last crawl
2026-07-04T06:41:55+02:00 304 # another four minutes
2026-07-04T07:11:33+02:00 304
2026-07-04T07:11:40+02:00 200 # no conditional request just seven seconds later
2026-07-04T07:43:25+02:00 304
2026-07-04T07:43:33+02:00 200 # just eight seconds since last crawl
2026-07-04T09:18:06+02:00 304
2026-07-04T09:22:53+02:00 304 # just four minutes since last crawl
2026-07-04T09:27:03+02:00 304 # another four minutes
@lyse@lyse.isobeef.org Thanks! I’ll look into that! Could be a bug in the crawler.
@lyse@lyse.isobeef.org I believe this is fixed now 🤞
@lyse@lyse.isobeef.org Found it and fixed it! 🎉 The crawler’s discovery spider was fetching every feed a second time, without any conditional headers (plus a couple of other politeness bugs: redirected feed URLs never stored their cache validators, and there was no floor between re-fetches). Now every feed is fetched at most once per crawl, always with If-Modified-Since / If-None-Match, and never more than once per 15m no matter what. Just deployed — please keep an eye on your access logs and let me know if you still see anything impolite from the crawler 🙏
@prologic@twtxt.net Excellent, thank you for this!
@itsericwoodward@itsericwoodward.com Ahh you’re welcome bud! 👌