User-agent: Slurp Disallow: /ref/*.html Disallow: /jsparts/* Disallow: /diffx/* Disallow: /editx/* Disallow: /rss/*