Datafeedr Support Forums  

Go Back   Datafeedr Support Forums > Current Version (V3) Forum > Questions
Home Register FAQ Members List Calendar Search Today's Posts Mark Forums Read

Important - The support forums are now in "read-only" mode as we transition to an alternative help desk solution. Feel free to contact us here with any of your questions or search the forums for an existing solution.
 
 
Thread Tools Search this Thread Display Modes
  #1  
Old January 13th, 2014, 03:40 PM
proneone proneone is offline
 
Join Date: Jan 2014
Posts: 33
Default blocking robots from crawling buy links?

i was told that it was relatively easy to block search engines from crawling my affiliate links with datafeedr, given the semi-static link structure it uses.

does anyone have an example robots.txt file i can implement to accomplish this?

our buy now link structure looks like:

http://domain.com/store/product/buy/...24671/product/

thanks!
  #2  
Old January 13th, 2014, 03:42 PM
Eric's Avatar
Eric Eric is offline
Datafeedr Team
 
Join Date: Feb 2008
Posts: 16,902
Default

Hi,

Adding this to your robots.txt file should do it:

Code:
User-agent: *
Disallow: /store/product/buy
Eric
  #3  
Old January 13th, 2014, 03:43 PM
Eric's Avatar
Eric Eric is offline
Datafeedr Team
 
Join Date: Feb 2008
Posts: 16,902
Default

But make sure to validate it:
https://www.google.ca/search?q=valid...sm=91&ie=UTF-8
  #4  
Old January 14th, 2014, 08:48 AM
proneone proneone is offline
 
Join Date: Jan 2014
Posts: 33
Default

thanks eric!

i'll get this implemented today and confirm.
  #5  
Old January 17th, 2014, 10:10 AM
proneone proneone is offline
 
Join Date: Jan 2014
Posts: 33
Default

ok here are the results i get when validating the robots.txt

User-agent Access? Reason
* yes no Disallow directive found

This is the default access policy for any robot without a matching User-agent field.
Rule records

1 User-agent: *
2 Disallow: /store/product/buy/


that doesn't look right.. correct?

thanks!
  #6  
Old January 17th, 2014, 10:13 AM
proneone proneone is offline
 
Join Date: Jan 2014
Posts: 33
Default

nevermind - fixed it.. i needed a blank line after the disallow

thanks!
  #7  
Old January 17th, 2014, 10:13 AM
Eric's Avatar
Eric Eric is offline
Datafeedr Team
 
Join Date: Feb 2008
Posts: 16,902
Default

It looks fine to me...
  #8  
Old January 24th, 2014, 07:05 PM
hosfield1010 hosfield1010 is offline
 
Join Date: Feb 2013
Posts: 47
Default

Hi may I ask why one would do this? Is this something necessary for SEO?

Thx.
  #9  
Old January 25th, 2014, 02:32 PM
gpsugar1 gpsugar1 is offline
 
Join Date: Nov 2013
Posts: 85
Default

So by doing all the above, This ensures web crawlers are blocked from discovering ALL affiliate links right? Are there other measures to take as well?
 


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
blocking items chunkyjewellery Questions 6 August 24th, 2011 11:01 AM
Blocking products macfink Feature Requests 0 August 13th, 2011 12:54 PM


All times are GMT -5. The time now is 01:16 PM.


Powered by vBulletin® Version 3.6.8
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.