=> Re: "My intermittent capsule outages are being caused by what..." | In: u/jsreed5
Thatβs the way to do it! Also can you publish which crawler it is - what IP it is from? Maybe the creator will see it hereβ¦
=> π requiem
2024-05-13 Β· 9 months ago
=> π jsreed5 [OP] Β· 2024-05-13 at 14:24:
Good point! The crawler's IP address is 104.207.150.107.
=> π requiem Β· 2024-05-13 at 16:29:
Reverse DNS resolves to celery.eu.org; over HTTP it says 'unplanned maintenance', copy-pasting the IP into the browser redirects you to a rickroll. TBH I would just keep the domain in your blacklist for now.
=> π jsreed5
My intermittent capsule outages are being caused by what appears to be a very aggressive crawler. The capsule's robots.txt file tells bots not to index my CGI scripts, but this crawler is ignoring the file and sending multiple requests per second against my scripts, which overloads the server and causes it to crash. I've temporarily solved the problem by blocking the crawler entirely; I'll look for a more permanent solution.
=> π¬ 3 comments Β· 3 likes Β· 2024-05-13 Β· 9 months ago This content has been proxied by September (3851b).Proxy Information
text/gemini; charset=utf-8