Whenever I think about updating my robots.txt file, I can't help but feel a little like Robert Neville hanging garlic on his front door. Is it enough to deter the vampires? Who knows, but why risk it?
Also, like my favorite Richard Matheson Novella, I wonder if, during some unknown spot in our digital timeline, we became the things that bump in the night. We are now the alien element amongst the ones and zeros; it is we who are Legend.
Silly, I know. It doesn't help that theories like The Dead Internet seem to have a point if not a bombastic tone. And I'm not trying to bestow sentience on robots here. Not after Timnet Gebru's Stochastic Parrot won Word of the Year.
What I'm saying, I guess, is we don't really have a say like we think we do. Not in regards to what web crawlers can crawl or scrapers can scrape. Until now, we've relied on ancient robot.txt folklore to protect us. But vampires don't need an invitation before they walk through our front doors. Garlic is, at best, a deterrent.
Perhaps we need a robot.txt file with a little legislative tooth— or a wooden spike; sorry, I'm getting lost in this analogy. There must be consequences for ignoring our do not crawl/do not index/do not inject tracking code wishes. And not just for personal websites either. For everything, including and especially our social media profiles.
The good news is, internet bots haven't learned the horrors of screaming our name in the night— Nevvvviillllle! Whew.