Noindex nofollow of equal sites
I have two identical sites, same content, design, everything(one is homologation the other production). Are in different domains, only thing I change, is in Robots that one is indexed index follow, and the other is noindex nofollow, I would like to know if I will have duplicate content problem with these sites?
1 answers
Doing an update of the answer.
There is another way to do blocking that would be straight through the HTTP response header using the X-Robots-Tag
Instead of a metatag or robots.txt, you can also return a X-Robots-Tag:noindex header in response to a page request. Here is an example of an HTTP response with a X-Robots-Tag which instructs crawlers not to index a page:
HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow
(…)
Here is more information about X-Robots-Tag: https://developers.google.com/search/reference/robots_meta_tag
Another tip about <meta> tags is that you can block specific Bots by them, such as:
<meta name="googlebot" content="NOINDEX, NOFOLLOW">
<meta name="MSNBot" content="NOINDEX, NOFOLLOW">
Blocking by robots.txt of the site that you do not want to be indexed:
User-agent: *
Disallow: /
In User-agent: * the * means that this section applies to all robots.
And the Disallow: / tells the robot that it should not visit any pages on the site.
Something else, the Noindex Nofollow must be inserted into a meta tag <meta> not inside the robots.txt! The correct one should be:
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
In any case the tag <meta> can be ignored by some search engines, as well as the robots.txt mainly malware-robots. And the nofollow I only blocked the links of the page you are on, if there is any link to your site on some other page that does not also have the nofollow the bot can find your site by that link, having the robots or not.