View Single Post
  #1  
Old January 13th, 2014, 03:40 PM
proneone proneone is offline
 
Join Date: Jan 2014
Posts: 33
Default blocking robots from crawling buy links?

i was told that it was relatively easy to block search engines from crawling my affiliate links with datafeedr, given the semi-static link structure it uses.

does anyone have an example robots.txt file i can implement to accomplish this?

our buy now link structure looks like:

http://domain.com/store/product/buy/...24671/product/

thanks!