User-agent: * Disallow: /dev/ Go here to make robot file, go to crawler access in Google webmaster tools