Datafeedr Support Forums  

Go Back   Datafeedr Support Forums > Current Version (V3) Forum > Problems
Home Register FAQ Members List Calendar Search Today's Posts Mark Forums Read

Important - The support forums are now in "read-only" mode as we transition to an alternative help desk solution. Feel free to contact us here with any of your questions or search the forums for an existing solution.
 
 
Thread Tools Search this Thread Display Modes
  #1  
Old May 8th, 2013, 09:13 AM
adglomera adglomera is offline
 
Join Date: Feb 2013
Posts: 12
Default Sitemap blocked by robots.txt

I'm trying to add the Datafeedr sitemap to Webmaster Tools but keep getiing the following error: URL restricted by robots.txt



Robots.txt is generated by Wordpress and does not contain anything that would block Google.
http://www.reiswegwijzer.be/robots.txt

I'm running Wordpress 3.5.1 and have no plugins installed apart from Datafeedr.
  #2  
Old May 8th, 2013, 09:33 AM
Eric's Avatar
Eric Eric is offline
Datafeedr Team
 
Join Date: Feb 2008
Posts: 16,902
Default

Hi,

What are your Privacy Settings here: WordPress Admin Area > Settings > Reading Settings > Search Engine Visibility

Is the Discourage search engines from indexing this site checkbox checked?

Eric
  #3  
Old May 8th, 2013, 09:47 AM
adglomera adglomera is offline
 
Join Date: Feb 2013
Posts: 12
Default

Hi Eric, no that box is unchecked.
  #4  
Old May 8th, 2013, 09:59 AM
Eric's Avatar
Eric Eric is offline
Datafeedr Team
 
Join Date: Feb 2008
Posts: 16,902
Default

Hi,

When did you submit your sitemap to Google?

Eric
  #5  
Old May 8th, 2013, 10:07 AM
adglomera adglomera is offline
 
Join Date: Feb 2013
Posts: 12
Default

Today was the first time. I also tried changing the URL Settings for the sitemap in the factory but that didn't help.
  #6  
Old May 8th, 2013, 10:10 AM
Eric's Avatar
Eric Eric is offline
Datafeedr Team
 
Join Date: Feb 2008
Posts: 16,902
Default

Hi,

Can you try installing this plugin and testing the sitemap that comes with that plugin?

Eric
  #7  
Old May 8th, 2013, 10:17 AM
adglomera adglomera is offline
 
Join Date: Feb 2013
Posts: 12
Default

I get the same result in Webmaster Tools: URL restricted by robots.txt.

I also recreated the robots.txt manually in the root directory but still no luck.
  #8  
Old May 8th, 2013, 10:24 AM
adglomera adglomera is offline
 
Join Date: Feb 2013
Posts: 12
Default

Google says:
The page could not be crawled at this time because it is blocked by the most recent robots.txt file Googlebot downloaded. Note that if you recently updated the robots.txt file, it may take up to two days before it's refreshed.

I guess it will just need some time then.
  #9  
Old May 8th, 2013, 10:25 AM
Eric's Avatar
Eric Eric is offline
Datafeedr Team
 
Join Date: Feb 2008
Posts: 16,902
Default

Hi,

OK, good to know. Now we know that this is not a Datafeedr-specific issue.

Here are some interesting threads discussing the very same issue:

http://www.metasmash.net/google-webm...by-robots-txt/
http://wordpress.org/support/topic/p...ols?replies=13

Eric
  #10  
Old May 8th, 2013, 10:29 AM
Eric's Avatar
Eric Eric is offline
Datafeedr Team
 
Join Date: Feb 2008
Posts: 16,902
Default

Quote:
Originally Posted by adglomera View Post
Google says:
The page could not be crawled at this time because it is blocked by the most recent robots.txt file Googlebot downloaded. Note that if you recently updated the robots.txt file, it may take up to two days before it's refreshed.

I guess it will just need some time then.
Thanks for sharing! That's good info to know. Check out the solution in the link above.
 


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
User is blocked jontek2 Problems 2 May 5th, 2013 09:45 AM
Automating - am I being blocked? estuary Questions 5 June 13th, 2011 06:52 AM


All times are GMT -5. The time now is 12:18 PM.


Powered by vBulletin® Version 3.6.8
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.