projects / project/drupal.git / blame
| Commit | Line | Data |
|---|---|---|
| a9080f70 | 1 | # |
| 5b455a2e DB |
2 | # robots.txt |
| 3 | # | |
| a8584f1b | 4 | # This file is to prevent the crawling and indexing of certain parts |
| 937f9099 DB |
5 | # of your site by web crawlers and spiders run by sites like Yahoo! |
| 6 | # and Google. By telling these "robots" where not to go on your site, | |
| 5b455a2e DB |
7 | # you save bandwidth and server resources. |
| 8 | # | |
| a8584f1b DB |
9 | # This file will be ignored unless it is at the root of your host: |
| 10 | # Used: http://example.com/robots.txt | |
| 11 | # Ignored: http://example.com/site/robots.txt | |
| 12 | # | |
| 5b455a2e DB |
13 | # For more information about the robots.txt standard, see: |
| 14 | # http://www.robotstxt.org/wc/robots.html | |
| 15 | # | |
| a8584f1b | 16 | # For syntax checking, see: |
| 5b455a2e DB |
17 | # http://www.sxw.org.uk/computing/robots/check.html |
| 18 | ||
| 19 | User-agent: * | |
| 20 | Crawl-delay: 10 | |
| 21 | # Directories | |
| 5b455a2e DB |
22 | Disallow: /includes/ |
| 23 | Disallow: /misc/ | |
| 24 | Disallow: /modules/ | |
| 84ea54a1 DB |
25 | Disallow: /profiles/ |
| 26 | Disallow: /scripts/ | |
| 5b455a2e | 27 | Disallow: /themes/ |
| 5b455a2e | 28 | # Files |
| 84ea54a1 | 29 | Disallow: /CHANGELOG.txt |
| 5b455a2e | 30 | Disallow: /cron.php |
| 5b455a2e DB |
31 | Disallow: /INSTALL.mysql.txt |
| 32 | Disallow: /INSTALL.pgsql.txt | |
| 39f4393f | 33 | Disallow: /INSTALL.sqlite.txt |
| 84ea54a1 DB |
34 | Disallow: /install.php |
| 35 | Disallow: /INSTALL.txt | |
| 5b455a2e | 36 | Disallow: /LICENSE.txt |
| 84ea54a1 DB |
37 | Disallow: /MAINTAINERS.txt |
| 38 | Disallow: /update.php | |
| 5b455a2e | 39 | Disallow: /UPGRADE.txt |
| 84ea54a1 | 40 | Disallow: /xmlrpc.php |
| 5b455a2e DB |
41 | # Paths (clean URLs) |
| 42 | Disallow: /admin/ | |
| 5b455a2e | 43 | Disallow: /comment/reply/ |
| 6bf6ee29 | 44 | Disallow: /filter/tips/ |
| 5b455a2e DB |
45 | Disallow: /node/add/ |
| 46 | Disallow: /search/ | |
| 47 | Disallow: /user/register/ | |
| 48 | Disallow: /user/password/ | |
| f4163900 | 49 | Disallow: /user/login/ |
| bc719d0c | 50 | Disallow: /user/logout/ |
| 5b455a2e DB |
51 | # Paths (no clean URLs) |
| 52 | Disallow: /?q=admin/ | |
| 5b455a2e | 53 | Disallow: /?q=comment/reply/ |
| 6bf6ee29 | 54 | Disallow: /?q=filter/tips/ |
| 5b455a2e DB |
55 | Disallow: /?q=node/add/ |
| 56 | Disallow: /?q=search/ | |
| 57 | Disallow: /?q=user/password/ | |
| 58 | Disallow: /?q=user/register/ | |
| f4163900 | 59 | Disallow: /?q=user/login/ |
| bc719d0c | 60 | Disallow: /?q=user/logout/ |