Hellp with a Pattern in Robots.txt
I am using a CMS that is generating all sorts of urls related to same page. So I want to block all variations of the the url but sill let search engines crawl the url without parameters, you think this will work?
As the pattern contains "?" I believe /member/profile_*.html will still be crawled
I want /member/profile_*.html to be crawled but none of the other variations with the parameters.
Please advise, thank you!
From my reading of info at robotstxt.org, it seems that while you can wildcard robots with a * you cannot do the same with files, nor can you match a pattern or parameters. You can disallow entire folders and specifically named files.
Read more at: http://www.robotstxt.org/robotstxt.html
you cannot have lines like "User-agent: *bot*", "Disallow: /tmp/*" or "Disallow: *.gif".
Thanks for your reply but the above url does explain much about file paths, read this post from Google, especially the section "URL matching based on path values", it explains the use "*" and "$" to match file paths.
Originally Posted by Kevin2
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)