Der user-agent ist der Spider der Suchmaschine
das ist natürlich brachialer unsinn, jeder browser identifiziert sich ebenso darüber.
das obige beispiel verbietet
alle zugriffe von ausserhalb der seite auf die genannten verzeichnisse
um die robots daran zu hindern, zb den inhalt des adminbereichs zu indizieren, dann nenn sie beim namen oder verbiete, wie im obigen fall einfach global (mit dem stern) den zugriff von ausserhalb
so sieht meine aktuelle aus:
[php:1:4999dfb297]User-agent: Alexibot
Disallow: /
User-agent: *alexa*
Disallow: /
User-agent: asterias
Disallow: /
User-agent: BackDoorBot
Disallow: /
User-agent: Baidu
Disallow: /
User-agent: Black.Hole
Disallow: /
User-agent: BlowFish
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: *Gigabot*
Disallow: /
User-agent: Googlebot-Image
Disallow: /
User-agent: Harvest
Disallow: /
User-agent: hloader
Disallow: /
User-agent: httplib
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Kenjin.Spider
Disallow: /
User-agent: Keyword.Density
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: LinkScan/8.1a.Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: Mata.Hari
Disallow: /
User-agent: Mediapartners-Google*
Disallow: /
User-agent: Microsoft.URL
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: Mister.PiX
Disallow: /
User-agent: moget
Disallow: /
User-agent: Mozilla/3.Mozilla/2.01
Disallow: /
User-agent: Mozilla.*NEWT
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: Offline.Explorer
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: ProPowerBot/2.14
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: psbot
Disallow: /
User-agent: QueryN.Metasearch
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: RMA
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: spanner
Disallow: /
User-agent: suzuran
Disallow: /
User-agent: Szukacz/1.4
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: Telesoft
Disallow: /
User-agent: The.Intraformant
Disallow: /
User-agent: TheNomad
Disallow: /
User-agent: TightTwatBot
Disallow: /
User-agent: Titan
Disallow: /
User-agent: toCrawl/UrlDispatcher
Disallow: /
User-agent: True_Robot
Disallow: /
User-agent: turingos
Disallow: /
User-agent: URLy.Warning
Disallow: /
User-agent: VCI
Disallow: /
User-agent: WebAuto
Disallow: /
User-agent: WebBandit
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebEMailExtrac.*
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: Web.Image.Collector
Disallow: /
User-agent: WebmasterWorldForumBot
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: Website.Quester
Disallow: /
User-agent: Webster.Pro
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebZip
Disallow: /
User-agent: [Ww]eb[Bb]andit
Disallow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: Xenu's
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: *
Disallow: /admin.php
Disallow: /error.php
Disallow: /admin/
Disallow: /arcade/
Disallow: /blocks/
Disallow: /cache/
Disallow: /ef/
Disallow: /error/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /pub/
Disallow: /stats/
Disallow: /themes/
Disallow: /uploads/
Disallow: /webalizer/
[/php:1:4999dfb297]