http://fileformats.archiveteam.org/index.php?action=history&feed=atom&title=Robots_Exclusion_Standard 2026-04-13T03:34:46Z Revision history for this page on the wiki MediaWiki 1.19.2 http://fileformats.archiveteam.org/index.php?title=Robots_Exclusion_Standard&diff=32536&oldid=prev 2019-07-02T02:42:19Z

<p></p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr valign='top'> <td colspan='2' style="background-color: white; color:black;">← Older revision</td> <td colspan='2' style="background-color: white; color:black;">Revision as of 02:42, 2 July 2019</td> </tr><tr><td colspan="2" class="diff-lineno">Line 3:</td> <td colspan="2" class="diff-lineno">Line 3:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>|subcat=Web</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>|subcat=Web</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>|extensions={{ext|txt}}</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>|extensions={{ext|txt}}</div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">|wikidata={{wikidata|Q80776}}</ins></div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>}}</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>}}</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td></tr> </table>

Dan Tobias http://fileformats.archiveteam.org/index.php?title=Robots_Exclusion_Standard&diff=32535&oldid=prev 2019-07-02T02:41:09Z

<p></p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr valign='top'> <td colspan='2' style="background-color: white; color:black;">← Older revision</td> <td colspan='2' style="background-color: white; color:black;">Revision as of 02:41, 2 July 2019</td> </tr><tr><td colspan="2" class="diff-lineno">Line 29:</td> <td colspan="2" class="diff-lineno">Line 29:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>** And they also have a [http://www.google.com/killer-robots.txt killer-robots.txt]!</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>** And they also have a [http://www.google.com/killer-robots.txt killer-robots.txt]!</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.ibm.com/robots.txt IBM's robots.txt]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.ibm.com/robots.txt IBM's robots.txt]</div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">== Utilities ==</ins></div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">* [https://github.com/google/robotstxt Google's robots.txt parser, now open-source]</ins></div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">* [https://github.com/randomstring/ParseRobotsTXT Perl robots.txt parser]</ins></div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Specific search engine / robot policies ==</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Specific search engine / robot policies ==</div></td></tr> <tr><td colspan="2" class="diff-lineno">Line 40:</td> <td colspan="2" class="diff-lineno">Line 44:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.beussery.com/blog/index.php/2014/06/robots-txt-disallow-20/ ROBOTS.TXT DISALLOW: 20 Years of Mistakes To Avoid]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.beussery.com/blog/index.php/2014/06/robots-txt-disallow-20/ ROBOTS.TXT DISALLOW: 20 Years of Mistakes To Avoid]</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://archiveteam.org/index.php?title=Robots.txt Robots.txt] article from the Archive Team website</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://archiveteam.org/index.php?title=Robots.txt Robots.txt] article from the Archive Team website</div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">* [https://opensource.googleblog.com/2019/07/googles-robotstxt-parser-is-now-open.html Google's robots.txt Parser is Now Open Source]</ins></div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>[[Category:File formats with a distinctive filename]]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>[[Category:File formats with a distinctive filename]]</div></td></tr> </table>

Dan Tobias http://fileformats.archiveteam.org/index.php?title=Robots_Exclusion_Standard&diff=25587&oldid=prev 2016-05-04T15:42:39Z

<p>Category:File formats with a distinctive filename</p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr valign='top'> <td colspan='2' style="background-color: white; color:black;">← Older revision</td> <td colspan='2' style="background-color: white; color:black;">Revision as of 15:42, 4 May 2016</td> </tr><tr><td colspan="2" class="diff-lineno">Line 40:</td> <td colspan="2" class="diff-lineno">Line 40:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.beussery.com/blog/index.php/2014/06/robots-txt-disallow-20/ ROBOTS.TXT DISALLOW: 20 Years of Mistakes To Avoid]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.beussery.com/blog/index.php/2014/06/robots-txt-disallow-20/ ROBOTS.TXT DISALLOW: 20 Years of Mistakes To Avoid]</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://archiveteam.org/index.php?title=Robots.txt Robots.txt] article from the Archive Team website</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://archiveteam.org/index.php?title=Robots.txt Robots.txt] article from the Archive Team website</div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">[[Category:File formats with a distinctive filename]]</ins></div></td></tr> </table>

Jsummers http://fileformats.archiveteam.org/index.php?title=Robots_Exclusion_Standard&diff=23203&oldid=prev 2015-08-25T23:00:55Z

<p>‎<span dir="auto"><span class="autocomment">Other links and references</span></span></p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr valign='top'> <td colspan='2' style="background-color: white; color:black;">← Older revision</td> <td colspan='2' style="background-color: white; color:black;">Revision as of 23:00, 25 August 2015</td> </tr><tr><td colspan="2" class="diff-lineno">Line 39:</td> <td colspan="2" class="diff-lineno">Line 39:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.mcanerin.com/EN/search-engine/robots-txt.asp Robots.txt generator/tutorial]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.mcanerin.com/EN/search-engine/robots-txt.asp Robots.txt generator/tutorial]</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.beussery.com/blog/index.php/2014/06/robots-txt-disallow-20/ ROBOTS.TXT DISALLOW: 20 Years of Mistakes To Avoid]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.beussery.com/blog/index.php/2014/06/robots-txt-disallow-20/ ROBOTS.TXT DISALLOW: 20 Years of Mistakes To Avoid]</div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">* [http://archiveteam.org/index.php?title=Robots.txt Robots.txt] article from the Archive Team website</ins></div></td></tr> </table>

Jsummers http://fileformats.archiveteam.org/index.php?title=Robots_Exclusion_Standard&diff=21396&oldid=prev 2015-03-30T16:03:51Z

<p></p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr valign='top'> <td colspan='2' style="background-color: white; color:black;">← Older revision</td> <td colspan='2' style="background-color: white; color:black;">Revision as of 16:03, 30 March 2015</td> </tr><tr><td colspan="2" class="diff-lineno">Line 23:</td> <td colspan="2" class="diff-lineno">Line 23:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Standards ==</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Standards ==</div></td></tr> <tr><td class='diff-marker'>−</td><td style="background: #ffa; color:black; font-size: smaller;"><div>* [http://www.robotstxt.org/<del class="diffchange diffchange-inline">wc/norobots.html Standards document</del>] (actually a non-binding consensus, not a formal standard)</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div>* [http://www.robotstxt.org/ <ins class="diffchange diffchange-inline">&quot;Official&quot; site</ins>] (actually a non-binding consensus, not a formal standard)</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Sample Files ==</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Sample Files ==</div></td></tr> </table>

Dan Tobias http://fileformats.archiveteam.org/index.php?title=Robots_Exclusion_Standard&diff=18686&oldid=prev 2014-08-18T15:50:24Z

<p>‎<span dir="auto"><span class="autocomment">Sample Files</span></span></p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr valign='top'> <td colspan='2' style="background-color: white; color:black;">← Older revision</td> <td colspan='2' style="background-color: white; color:black;">Revision as of 15:50, 18 August 2014</td> </tr><tr><td colspan="2" class="diff-lineno">Line 27:</td> <td colspan="2" class="diff-lineno">Line 27:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Sample Files ==</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Sample Files ==</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.google.com/robots.txt Google's robots.txt]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.google.com/robots.txt Google's robots.txt]</div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">** And they also have a [http://www.google.com/killer-robots.txt killer-robots.txt]!</ins></div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.ibm.com/robots.txt IBM's robots.txt]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.ibm.com/robots.txt IBM's robots.txt]</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td></tr> </table>

Dan Tobias http://fileformats.archiveteam.org/index.php?title=Robots_Exclusion_Standard&diff=18076&oldid=prev 2014-07-01T03:36:16Z

<p>‎<span dir="auto"><span class="autocomment">Other links and references</span></span></p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr valign='top'> <td colspan='2' style="background-color: white; color:black;">← Older revision</td> <td colspan='2' style="background-color: white; color:black;">Revision as of 03:36, 1 July 2014</td> </tr><tr><td colspan="2" class="diff-lineno">Line 37:</td> <td colspan="2" class="diff-lineno">Line 37:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://en.wikipedia.org/wiki/Robots_exclusion_standard Robots Exclusion Standard (Wikipedia)]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://en.wikipedia.org/wiki/Robots_exclusion_standard Robots Exclusion Standard (Wikipedia)]</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.mcanerin.com/EN/search-engine/robots-txt.asp Robots.txt generator/tutorial]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.mcanerin.com/EN/search-engine/robots-txt.asp Robots.txt generator/tutorial]</div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">* [http://www.beussery.com/blog/index.php/2014/06/robots-txt-disallow-20/ ROBOTS.TXT DISALLOW: 20 Years of Mistakes To Avoid]</ins></div></td></tr> </table>

Dan Tobias http://fileformats.archiveteam.org/index.php?title=Robots_Exclusion_Standard&diff=16721&oldid=prev 2014-03-24T12:53:33Z

<p>‎<span dir="auto"><span class="autocomment">Other links and references: </span> Link is dead; not in Internet Archive either.</span></p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr valign='top'> <td colspan='2' style="background-color: white; color:black;">← Older revision</td> <td colspan='2' style="background-color: white; color:black;">Revision as of 12:53, 24 March 2014</td> </tr><tr><td colspan="2" class="diff-lineno">Line 37:</td> <td colspan="2" class="diff-lineno">Line 37:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://en.wikipedia.org/wiki/Robots_exclusion_standard Robots Exclusion Standard (Wikipedia)]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://en.wikipedia.org/wiki/Robots_exclusion_standard Robots Exclusion Standard (Wikipedia)]</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.mcanerin.com/EN/search-engine/robots-txt.asp Robots.txt generator/tutorial]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.mcanerin.com/EN/search-engine/robots-txt.asp Robots.txt generator/tutorial]</div></td></tr> <tr><td class='diff-marker'>−</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del style="color: red; font-weight: bold; text-decoration: none;">* [https://twitter.com/AKKuhn/status/432604323600539649/photo/1 If we outlaw wget, only criminals will wget (picture)]</del></div></td><td colspan="2">&#160;</td></tr> </table>

Dan Tobias http://fileformats.archiveteam.org/index.php?title=Robots_Exclusion_Standard&diff=16344&oldid=prev 2014-02-23T13:45:11Z

<p></p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr valign='top'> <td colspan='2' style="background-color: white; color:black;">← Older revision</td> <td colspan='2' style="background-color: white; color:black;">Revision as of 13:45, 23 February 2014</td> </tr><tr><td colspan="2" class="diff-lineno">Line 21:</td> <td colspan="2" class="diff-lineno">Line 21:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>There are some meta tags like &quot;noindex&quot; and &quot;nofollow&quot; that can be used in [[HTML]] for related effects.</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>There are some meta tags like &quot;noindex&quot; and &quot;nofollow&quot; that can be used in [[HTML]] for related effects.</div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">== Standards ==</ins></div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins style="color: red; font-weight: bold; text-decoration: none;">* [http://www.robotstxt.org/wc/norobots.html Standards document] (actually a non-binding consensus, not a formal standard)</ins></div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Sample Files ==</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Sample Files ==</div></td></tr> <tr><td colspan="2" class="diff-lineno">Line 33:</td> <td colspan="2" class="diff-lineno">Line 36:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Other links and references ==</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>== Other links and references ==</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://en.wikipedia.org/wiki/Robots_exclusion_standard Robots Exclusion Standard (Wikipedia)]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://en.wikipedia.org/wiki/Robots_exclusion_standard Robots Exclusion Standard (Wikipedia)]</div></td></tr> <tr><td class='diff-marker'>−</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del style="color: red; font-weight: bold; text-decoration: none;">* [http://www.robotstxt.org/wc/norobots.html Standards document] (actually a non-binding consensus, not a formal standard)</del></div></td><td colspan="2">&#160;</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.mcanerin.com/EN/search-engine/robots-txt.asp Robots.txt generator/tutorial]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.mcanerin.com/EN/search-engine/robots-txt.asp Robots.txt generator/tutorial]</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [https://twitter.com/AKKuhn/status/432604323600539649/photo/1 If we outlaw wget, only criminals will wget (picture)]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [https://twitter.com/AKKuhn/status/432604323600539649/photo/1 If we outlaw wget, only criminals will wget (picture)]</div></td></tr> </table>

Dan Tobias http://fileformats.archiveteam.org/index.php?title=Robots_Exclusion_Standard&diff=16343&oldid=prev 2014-02-23T13:34:06Z

<p>‎<span dir="auto"><span class="autocomment">References</span></span></p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr valign='top'> <td colspan='2' style="background-color: white; color:black;">← Older revision</td> <td colspan='2' style="background-color: white; color:black;">Revision as of 13:34, 23 February 2014</td> </tr><tr><td colspan="2" class="diff-lineno">Line 26:</td> <td colspan="2" class="diff-lineno">Line 26:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.ibm.com/robots.txt IBM's robots.txt]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.ibm.com/robots.txt IBM's robots.txt]</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"></td></tr> <tr><td class='diff-marker'>−</td><td style="background: #ffa; color:black; font-size: smaller;"><div>== <del class="diffchange diffchange-inline">References </del>==</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div>== <ins class="diffchange diffchange-inline">Specific search engine / robot policies ==</ins></div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline">* [https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt Google / Googlebot]</ins></div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline">* [https://help.yahoo.com/kb/search/slurp-crawling-page-sln22600.html?impressions=true Yahoo / Slurp]</ins></div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline">* [http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbot-s-question.aspx Bing / Bingbot]</ins></div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div>&#160;</div></td></tr> <tr><td colspan="2">&#160;</td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline">== Other links and references </ins>==</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://en.wikipedia.org/wiki/Robots_exclusion_standard Robots Exclusion Standard (Wikipedia)]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://en.wikipedia.org/wiki/Robots_exclusion_standard Robots Exclusion Standard (Wikipedia)]</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.robotstxt.org/wc/norobots.html Standards document] (actually a non-binding consensus, not a formal standard)</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.robotstxt.org/wc/norobots.html Standards document] (actually a non-binding consensus, not a formal standard)</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.mcanerin.com/EN/search-engine/robots-txt.asp Robots.txt generator/tutorial]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [http://www.mcanerin.com/EN/search-engine/robots-txt.asp Robots.txt generator/tutorial]</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [https://twitter.com/AKKuhn/status/432604323600539649/photo/1 If we outlaw wget, only criminals will wget (picture)]</div></td><td class='diff-marker'>&#160;</td><td style="background: #eee; color:black; font-size: smaller;"><div>* [https://twitter.com/AKKuhn/status/432604323600539649/photo/1 If we outlaw wget, only criminals will wget (picture)]</div></td></tr> </table>

Dan Tobias