RSS
 

Obama’s New Robots.txt

20 Jan

Change is afoot…

Within a few moments of President Obama being sworn in, a new whitehouse.gov was revealed.  The new site looks great, and promises greater transparency about the actions of our President and government.

Along those lines, check out its robots.txt file, in entirety:

User-agent: *
Disallow: /includes/

Compare this to the robot.txt from Bush’s era, and I think you’ve got a lovely little microcosm; one that points to a hopeful, and open future.

 
26 Comments

Posted in general

 
  • http://thenextweb.com/2009/01/20/comparing-robotstxt-files-from-obamas-new-site-and-bushs-old-site/ Comparing Robots.txt Files From Obama’s New Site and Bush’s Old Site « TheNextWeb.com

    [...] Codeulate. I hope you like that post! The Next Web Blog covers start-up news from all over the world (not [...]

  • Bif Powell

    Show the robots.txt from Bush’s first day and you might have something. Otherwise; “baaaaaaaaaaaaaaaaa”

  • Anonymous

    Give it four years. You’ll likely see /economy, /iraq and /oh-my-god-we\’re-turning-into-France in there.

  • conrad

    The old robots.txt was simply blocking text-only versions of pages. The full versions of the pages were available, not blocked and were fully indexed by google. Run a search and you’ll see.

  • Lukifer

    > /oh-my-god-we\’re-turning-into-France

    …that’s a feature, not a bug.

  • http://joel.barciausk.as Joel Barciauskas

    You have too much time on your hands, man.

  • http://gtcaz.com Geoff

    Disallow: /privacy.html

  • Jacob Johnson

    The Barrakessiah may not disallow webpages, but he’ll disallow your right to own so-called *assault* weapons.

    That is treason.

  • http://www.seriouslywtf.net/2009/01/20/obamas-robotstxt/ seriously. wtf.

    [...] siracusa: http://codeulate.com/?p=24 [3:54pm] tachikoma: Title: Obamas New Robots.txt [3:55pm] jonathan: can you explain the bush one to [...]

  • http://www.mitgr81.com/?p=308 Daily List – MiT Gr8 1

    [...] New White House website (with improved robot exclusion list). [...]

  • Anonymoose

    Bif Powell +1. Also,

    @Jacob Johnson: “The Barrakessiah may not disallow webpages, but he’ll disallow your right to own so-called *assault* weapons.” “That is treason.”

    No. Criticizing the President’s policies is basically not even frowned upon anymore, let alone considered treasonous.

  • http://feedblog.org/2009/01/20/a-change-in-robotstxt/ A Change in Robots.txt « Kevin Burton’s NEW FeedBlog

    [...] January 20, 2009 in Uncategorized Change is in the air. Including the robots.txt at whitehouse.gov: [...]

  • http://mcdevzone.com/2009/01/20/new-openness-at-whitehousegov/ New openness at whitehouse.gov | /dev/random

    [...] the old site, the new white house site is now open to search engines. Codeulate analyzes the robots.txt file. The current robots.txt consists of two lines, allowing search engines [...]

  • Andrew

    I am *deeply* unnerved by the presence of a folder called /photos/text/. Ascii art? maybe?!

  • http://l33tminion.livejournal.com L33tminion

    Evidently, the “text” directories contained versions of various documents for printing, and the webmasters wanted those excluded to avoid them being double-counted by search engines (although Google is already pretty good at filtering such duplicates). So this isn’t really a sign of greater transparency, probably. On the other hand, it is probably a sign of superior web design.

  • http://j.snyder.name/2009/01/20/obamas-new-robotstxt Obama’s New Robots.txt | J’s blog

    [...] Obama’s New Robots.txt. addthis_url = ‘http%3A%2F%2Fj.snyder.name%2F2009%2F01%2F20%2Fobamas-new-robotstxt’; addthis_title = ‘Obama%26%238217%3Bs+New+Robots.txt’; addthis_pub = ”; [...]

  • http://www.jasonhill.com/2009/01/links-for-2009-01-20-2/ links for 2009-01-20

    [...] Obama’s New Robots.txt : Codeulate. (tags: politics robots.txt obama bush) [...]

  • http://blog.delaranja.com/obamas-inauguration-around-the-web/ Obama’s inauguration around the Web : blog.delaranja.com

    [...] last a a question about Obama’s New Robots.txt and the new White House website which has a blog (and an RSS [...]

  • http://186-kps.com/blog/2009/01/21/whitehousegov-obama-v-bush-v-robots/ The whitehouse.gov website and the robots | 186 k per second

    [...] Yes even Robots have become a topic of partisan pot shots! [...]

  • http://queerandloathing.com/2009/01/21/whitehousegovs-open-door-policy-even-for-spiders/ whitehouse.gov’s open door policy. Even for spiders. | queer and loathing

    [...] While doing research on robot.txt files, which sit on a web site and tell visiting search engine spiders what pages and directories to skip when they index your pages, I found this article at codeulate.com regarding the new whitehouse.gov’s robot.txt file. [...]

  • http://trublion.wordpress.com/2009/01/21/liens-du-jour-193/ Liens du jour « The Trublion’s blog

    [...] Obama’s New Robots.txt codeulate.com/?p=24 [...]

  • http://www.taggle.org/2009/01/2028-geekeries-28/ Geekeries | taggle.org

    [...] Le robots.txt du nouveau site de la Maison Blanche (1 ligne) versus l’ancien (nnn lignes) [...]

  • http://humorlessbitch.com zo

    @Jacob Johnson …. those “so-called” assault weapons. We feel for you. We really do. (What would Freud say about men with such fears. Not hard to guess.)

  • Yoshi

    It seems the BBC has picked up on this and got it completely wrong: http://www.jamiedigi.com/2009/01/bbc-gets-it-completely-wrong-about-whitehousegov-robotstxt/

  • http://omegageek.wordpress.com/2009/01/26/molehills-mountains-robots/ Molehills, Mountains, Robots. « Geekspeak

    [...] Obama’s New Robots.txt Change is afoot… [...]

  • http://www.zeldman.com/2009/02/10/saved-from-the-deep/ Jeffrey Zeldman Presents : Saved from the deep

    [...] Obama’s New Robots.txt : Codeulate. [...]