Just a reminder that robots.txt is largely useless as a mechanism to stopping or controlling bots from hitting your website.
You don't have to take my word for it - Slack themselves straight up admits they don't honour it because it was too annoying to deal with.
https://api.slack.com/robots
=> More informations about this toot | View the thread | More toots from ecn@mastodon.social
text/geminiThis content has been proxied by September (3851b).