a1studmuffin@aussie.zonetoTechnology@lemmy.world•I just wanted to take a moment to enjoy how clean the web can beEnglish
1·
1 month agorobots.txt is the perfect summary of the web era. A plain text file that politely asked web crawlers not to do certain things. Such an innocent time.
Smells like something IDF unit 8200 might have been involved with.