The robots.txt file is used by website owners to give instructions about their site to web robots. The web robot will read the robots.txt file before visiting a site, to check whether any directories should be excluded from the visit.
In the example file below, the 'User-agent' is specifying ALL robots, and 'Disallow' is instructing them not to visit the /customers directory.
Customer Self Service allows for easy maintenance of the robots.txt file by site Administrators. Simply follow the procedure below.
Step-by-step guide
To View and Update the robots.txt file:
- Login as an Administrator.
- Navigate to Settings → robots.txt
- Make the required changes
- Click the Save button.
This feature isn't available on older versions
For versions older than 4.42.29, you'll need to request the robots.txt file is updated manually by staff using the steps provided below:
- Login as an Administrator.
- Navigate to Settings → SEO Maintenance (/zSearchEngineOptimisationMaintenance.aspx).
- Click the Robots.txt tab.
For further information on robots.txt, see http://www.robotstxt.org/robotstxt.html.
Related help
