#robots text file is used to control the access to web #site files and directories which should not be indexed # User-agent: * # #Block spidering of specific top level directories # #Updated 11-AUG-06 #Added "/admin/" # Disallow:/admin/ Disallow:/comps/ Disallow:/emailTemplate/ Disallow:/includes/