A robots.txt file just isn't an acceptable or effective method of blocking delicate or confidential materials. It only instructs properly-behaved crawlers that the pages are certainly not for them, but it does not reduce your server from delivering These pages to some browser that requests them. A single rationale is that search engines like yahoo … Read More