Allow "ALL" in the site User-agent: * Disallow: # Deny all other spiders #User-agent: * #Disallow: /