my websites have become more of an api for ai bots than websites for real humans…. robots.txt gets completely ignored