Disallow file extension in robots.txt

To disable search engine crawling of specific extension files (txt, csv, etc) use:

User-agent: *
Disallow: /*.txt$
Allow: /
Click to rate this post!
[Total: 0 Average: 0]

You May Also Like

About the Author: deniskoronets

Leave a Reply

Your email address will not be published. Required fields are marked *