PDA

View Full Version : Robots.txt



thesheild
08-02-2018, 04:40 PM
i have prepare robots.txt file in which i want to allow all search engines to crawl all pages of my site except three pages. my file looks like following:

User-Agent: *
Disallow:/indexA.html

User-Agent: *
Disallow:/indexA.php

User-Agent: *
Disallow:/index-login.php

Do This file is right or do i have done some mistakes?

Angemon
08-02-2018, 08:48 PM
i have prepare robots.txt file in which i want to allow all search engines to crawl all pages of my site except three pages. my file looks like following:

User-Agent: *
Disallow:/indexA.html

User-Agent: *
Disallow:/indexA.php

User-Agent: *
Disallow:/index-login.php

Do This file is right or do i have done some mistakes?

First why don't you want search engine to index those pages? You do know that anyone can see your robots.txt and know exactly what pages you want to hide