# This says to apply these settings to ALL search engine spiders/crawlers User-agent: * # These settings will keep spiders from indexing your unwanted pages Disallow: /WTS Disallow: /CustomTags Disallow: /Includes Disallow: /Collections Disallow: /Images User-agent: Googlebot-Image Disallow: /Images