Increase in Soft 404 Errors

[1800HART]

Should I be worried?

I've been getting a lot of emails from Webmaster Tools for all my stores lately.

Webmaster Tools sent you the following important messages about sites in your account. To keep your site healthy, we recommend regularly reviewing these messages and addressing any critical issues.

http://rabbitlvr.com/: Increase in soft 404 errors

Your server is redirecting requests for a non-existent page, instead of returning a 404 response code. This creates a poor experience for searchers and search engines.

I've used RabbitLvr but, I've been getting email warnings on all my datafeedr domain/subdomain store/blogs and repeatedly periodically. I really should fix this I figure!

Basically, the crawl errors are such like these URLS:
http://rabbitlvr.com/store/toys/tuff...00/?price=150-
http://rabbitlvr.com/store/toys/tuff...0900658061500/
http://rabbitlvr.com/store/calendars...&price=100-150
http://ratlvr.com/store/leashes/supe...2201544184285/
http://ratlvr.com/store/calendars/ge...1500625507782/

etc

As you may notice, a good majority of these links crawled end up with the price=etc although that is not the standard as there are a lot of links of products that, I guess once the store is updated doesn't appear anymore.

My theme doesn't have a 404 page, and just redirects to the main URL from the "404 Redirection" plugin I use.

Can you think of an easy way to stop the WMT messages and errors? And, if it is needed by adding a 404 page or better wordpress plugin I can use?

I tried to "blend" the default wordpress twentytwelve 404.php to work with my site, but I keep getting white screen of death. My theme is a child theme on Genesis framework. When I used to create 404 pages, I just used to copy/modify a page.php file and rename it 404.php and modify .. but, this theme doesn't even have a page.php theme. I tried to modify the home.php file and the archive.php to be the 404.php page, but I also get the white screen or severe delays.

This is what my archive.php looks like

<?php get_header(); ?>


	<div id="home">
						
		<div id="home-top">
			<div class="home-top-left">
				<?php if (!dynamic_sidebar('Home Top Left')) : ?>
					<div class="widget">
						<h4><?php _e("Home Top Left", 'genesis'); ?></h4>
						<div class="wrap">

Welcome to the <a href="<?php echo home_url( '/' ) ?>" title="<?php echo esc_attr( get_bloginfo( 'name', 'display' ) ); ?>" rel="home">
					<?php bloginfo( 'name' ); ?> ...

							<p><?php _e("This is a widgeted area which is called Home Top Left. It is using the Genesis - Featured Posts widget to display what you see on the Sleek child theme demo site. To get started, log into your WordPress dashboard, and then go to the Appearance > Widgets screen. There you can drag the Genesis - Featured Posts widget into the Home Top Left widget area on the right hand side. To get the image to display, simply upload an image through the media uploader on the edit page screen and publish your page. The Featured Posts widget will know to display the post image as long as you select that option in the widget interface.", 'genesis'); ?></p>
						</div><!-- end .wrap -->
					</div><!-- end .widget -->
				<?php endif; ?>
			</div><!-- end .home-top-left -->
			<div class="home-top-right">
				<?php if (!dynamic_sidebar('Home Top Right')) : ?>
					<div class="widget">
						<h4><?php _e("Home Top Right", 'genesis'); ?></h4>
						<div class="wrap">
							<p><?php _e("This is a widgeted area which is called Featured Bottom Right. It is using the Genesis - Featured Posts widget to display what you see on the Sleek child theme demo site. To get started, log into your WordPress dashboard, and then go to the Appearance > Widgets screen. There you can drag the Genesis - Featured Posts widget into the Home Top Right widget area on the right hand side. To get the image to display, simply upload an image through the media uploader on the edit page screen and publish your page. The Featured Posts widget will know to display the post image as long as you select that option in the widget interface.", 'genesis'); ?></p>
						</div><!-- end .wrap -->
					</div><!-- end .widget -->
				<?php endif; ?>
			</div><!-- end .home-top-right -->
		</div><!-- end #home-top -->	
		
		<div id="home-bottom">
			<div class="wrap">
    			<div class="home-bottom-1">

				<div class="widget">
				<div class="widget-wrap">

				<h4>You are here are the bottom of the page!</h4>
				<div id="site-info">
				<a href="<?php echo home_url( '/' ) ?>" title="<?php echo esc_attr( get_bloginfo( 'name', 'display' ) ); ?>" rel="home">
					<?php bloginfo( 'name' ); ?>
				</a>

 | <a href="http://HARTmarket.com/shops/" target="_blank">List of Shops</a> |
<a href="http://HARTmarket.com/privacy-policy/" target="_blank">Privacy Policy</a> | 
<a href="<?php echo get_settings('home'); ?>/auction-deals/"><img src="http://HARTmarket.com/pics/logoEbay_x45.gif" title="Check out the latest Auction Deals on eBay!"></a>
<hr/>
<center>
<h2>SUBSCRIBE TO OUR NEWSLETTER</h2>
<script type="text/javascript" src="http://forms.aweber.com/form/55/903802055.js"></script>
</center>
		
			</div><!-- #site-info -->

	</div>
            			</div>




   				</div><!-- end .home-bottom-1 -->

    			<div class="home-bottom-2">
        			<?php if ( !function_exists('dynamic_sidebar') || !dynamic_sidebar('Home Bottom #2') ) : ?>
						<div class="widget">
						<div class="widget-wrap">
<h4>Shop @ HART Market Policy</h4>
This Shop is a real store filled with real products from real vendors! We will earn small commissions from purchases that you may make with these third party vendors. It is advisable to familiarize yourself with the different terms and conditions of the vendor prior to purchase. Thank-You!
<hr/>
<center>
<img src="http://hartmarket.com/pics/HARTmarket-minimall.gif" width="280">
</center>
            				</div>
            			</div>
	    			<?php endif; ?> 
    			</div><!-- end .home-bottom-2 -->
    			<div class="home-bottom-3">

					<div class="widget">
					<div class="widget-wrap">
            					<h4>The Usual Stuff</h4>
<center>
<a href="http://whos.amung.us/show/xooajaej"><img src="http://whos.amung.us/widget/xooajaej.png" alt="web tracker" width="81" height="29" border="0" /></a>
<br/>

<!-- Site Meter -->
<script type="text/javascript" src="http://s50.sitemeter.com/js/counter.js?site=s50HARTmarket">
</script>
<noscript>
<a href="http://s50.sitemeter.com/stats.asp?site=s50HARTmarket" target="_top">
<img src="http://s50.sitemeter.com/meter.asp?site=s50HARTmarket" alt="Site Meter" border="0"/></a>
</noscript>
<!-- Copyright (c)2006 Site Meter -->
<br/>
<img src="http://www.HART-Empire-Network.com/pics/www_pic-a01.gif">
<img src="http://www.HART-Empire-Network.com/pics/www_pic-a02.gif">
<img src="http://www.HART-Empire-Network.com/pics/www_pic-a03.gif">
<img src="http://www.HART-Empire-Network.com/pics/www_pic-a04.gif">
<img src="http://www.HART-Empire-Network.com/pics/www_pic-a05.gif">
<br/>
<!-- Site Meter XHTML Strict 1.0 -->
<script type="text/javascript">var site="s22HART-Empire-Network-All"</script>
<script type="text/javascript" src="http://s22.sitemeter.com/js/counter.js?site=s22HART-Empire-Network-All">
</script>
<noscript><div>
<a href="http://s22.sitemeter.com/stats.asp?site=s22HART-Empire-Network-All">
<img src="http://s22.sitemeter.com/meter.asp?site=s22HART-Empire-Network-All" alt="Site Meter" /></a>
</div></noscript>
<!-- Copyright (c)2005 Site Meter -->
<br/>
<SCRIPT type='text/javascript' language='JavaScript' src='http://xslt.alexa.com/site_stats/js/t/a?url=HARTmarket.com'></SCRIPT>
<br/>
<iframe frameborder='0' scrolling='no' marginheight='0' marginwidth='0' height='43' width='88' src='http://widget.quantcast.com/p-bbZ0j5-fIT7D-/2'></iframe>
<br/>
<script type="text/javascript" language="javascript" src="http://twittercounter.com/embed/?username=HARTmarket&style=blue"></script>

</center>


            				</div>
            			</div>
	    			
    			</div><!-- end .home-bottom-3 -->
			</div><!-- end .wrap -->
		</div><!-- end #home-bottom -->
		

<center>
<a href="http://birthday.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/birthday-hartmarket.jpg" title="[The BirthdayShop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="http://christmas.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/christmas-hartmarket.jpg" title="[The Christmas Shop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="http://halloween.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/halloween-hartmarket.jpg" title="[The Halloween Shop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="http://furniture.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/furniture-hartmarket.jpg" title="[The Furniture Shop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="http://treadmill.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/hartmarket.jpg" title="[The Treadmill Shop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="http://lingerie.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/lingerie-hartmarket.jpg" title="[The Lingerie Shop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="http://medieval.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/medieval-hartmarket.jpg" title="[The Medieval Shop @ HART Market]" width="55" alt="Shop @ HART Markets" /></a>
<a href="http://petcare.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/petcare-hartmarket.jpg" title="[The Pet Care Shop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="http://software.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/software-hartmarket.jpg" title="[The Software Shop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="http://tools.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/tools-hartmarket.jpg" title="[The Tools Shop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="http://vision.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/vision-hartmarket.jpg" title="[The Vision Shop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="http://wedding.HARTmarket.com"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/wedding-hartmarket.jpg" title="[The Wedding Shop @ HART Market]" width="55"  alt="Shop @ HART Markets" /></a>
<a href="<?php if (function_exists('dfrad')) { echo dfrad('randomsites'); } ?>"><img style="float:left;margin:0px 10px 10px 0px;" src="http://HARTmarket.com/pics/mini_hart.gif" title="[Click Here For A Random Shop @ HART Market]" width="55"  alt="Click Here For A Random Shop @ HART Market" /></a>
Surf a Random Shop @ HART Market Store<br/>
</center>


	</div><!-- end #home -->

<?php get_footer(); ?>

Thank in advance /
HART

[Eric]

Hi,

I think a 404.php page would be very helpful and I'm surprised Studio Press themes don't ship with one. I would ask the SP guys how to implement a 404 page. That should take care of the issue.

However, if you want to display some specific store information on your 404 pages, you can create a new custom view in the Factory ( The Factory > Your Store > VIEWS > CUSTOM VIEW > Create a new view). Then name the view Missing Product. Then add whatever modules you want to the page. Now when product URLs result in 404s, this new Custom view named Missing Product will be displayed.

Eric

[1800HART]

Thanks Eric..

It seems that Genesis has a built in 404 error message and not a 404.php page per say, so I have deactivated the redirection plugin, and implemented your option. It's a shame I can't combine the two into one ... missing links from store doesn't show up on the 404 error, and invalid urls that are non-store doesn't pick up the custom view page error message. I will think of something suitable and copy over to all my stores, but here is what I did so far...

e.g. if not a store product - the default Genesis 404 page (without a 404.php page)
http://rabbitlvr.com/tuffut-baby-rab...00/?price=150-

One of the missing links mentioned from Google's crawl bot - Datafeedr Missing Product view
http://rabbitlvr.com/store/toys/tuff...00/?price=150-

Thanks!
HART

Originally Posted by Eric

Hi,

I think a 404.php page would be very helpful and I'm surprised Studio Press themes don't ship with one. I would ask the SP guys how to implement a 404 page. That should take care of the issue.

However, if you want to display some specific store information on your 404 pages, you can create a new custom view in the Factory ( The Factory > Your Store > VIEWS > CUSTOM VIEW > Create a new view). Then name the view Missing Product. Then add whatever modules you want to the page. Now when product URLs result in 404s, this new Custom view named Missing Product will be displayed.

Eric

[Eric]

Hi Hart,

Unfortunately, I don't think they can be combined.

Eric

[1800HART]

Hi Eric.

Can you advise?

I copied the front code from the store where I list all the merchants in the error page ... inside that "Missing Product" custom view widget ... but, when the error page comes up the link is not taking place and keeps the error'ed URL ..

EG - this is an example of a missing product results - which doesn't work
http://rabbitlvr.com/store/toys/tuff...chant_id=22198 (where it looks like http://rabbitlvr.com/store/toys/tuff...rchant_id=1291)

EG - this is how those links should work
http://rabbitlvr.com (where it looks like http://rabbitlvr.com/?merchant_id=1291)

Is there a way to fix that?

Thx

[Eric]

Hi,

If you just want to display the same information as on your store's front page on your store's 404 page, just add a Text module to your Missing Product view and add the following code to that Text module:

[DFR:home]

Eric

[1800HART]

Originally Posted by Eric

Hi,

If you just want to display the same information as on your store's front page on your store's 404 page, just add a Text module to your Missing Product view and add the following code to that Text module:

[DFR:home]

Eric

Hi Eric..

This did not work. shop=38133

Granted. This replicates the home page, but everything in the 404 URL is added after the URL before any links. Try it out with examples above in earlier discussions on http://rabbitlvr.com

http://rabbitlvr.com/store/toys/tuff...rchant_id=1291

all merchant codes looks like that - the "?merchant_id=1291" is appended after the current 404 url. So, when you click it - it also goes to 404. It never works.

[Eric]

Hi,

I'm a little confused as to what you want. The product pages in your store are now displaying custom information.

Here's an example of a non-store 404 page page on your site: http://rabbitlvr.com/qwer/zvxcz/?asdf=1234

Nothing in the URL changes if it's a 404. A 404 is served but the extra information is displayed. Isn't this what you want?

Eric

[1800HART]

Originally Posted by Eric

Hi,

I'm a little confused as to what you want. The product pages in your store are now displaying custom information.

Here's an example of a non-store 404 page page on your site: http://rabbitlvr.com/qwer/zvxcz/?asdf=1234

Nothing in the URL changes if it's a 404. A 404 is served but the extra information is displayed. Isn't this what you want?

Eric

Picture is worth a thousand words ....

Your example is an example of a blog URL that doesn't exist. Genesis theme resorts to a 404 error and gives out a list of my archives, pages, recent posts, etc.

What Datafeedr has been doing when products are removed from the store is kicking out soft 404 crawl errors with google because I was redirecting it to root. We went through above and I had the same problem all all my sites - where I created that "Missing Product" view with some text and code. On RabbitLvr store I just tried your [DFR:Home] in the text widget and nothing else.

Here's one of those thousands of links in my google WMT crawl error list and what happens as a direct result of your Missing Product code:
http://rabbitlvr.com/store/toys/tuff...0900658061500/

See image: https://www.diigo.com/item/image/1d8vg/sdqm?size=o

// This is an afterthought - but I think it related too //

When I go back into my WMT account and review all the thousands of crawl errors .. there were only THREE types of store errors - ones like the URL's listed above

http://rabbitlvr.com/store/toys/tuff...0900658061500/
> the actual missing product url

http://rabbitlvr.com/store/toys/tuff...00/?price=150-
>> /?price= category missing

http://rabbitlvr.com/store/calendars...&price=100-150
>> similar to my problem .. /?merchant_id= appended after an invalid missing product url

Here's a screenshot from another WMT store crawl error list

https://www.diigo.com/item/image/1d8vg/yr7d?size=o

Hope that now makes it more clearer, if not more sense
HART

[Eric]

Hi

Originally Posted by 1800HART

See image: https://www.diigo.com/item/image/1d8vg/sdqm?size=o

OK, to solve this issue you need to add some code to your dfr_get_merchant_logos() function. Find this line:

PHP Code:

$html .= '<a href="?merchant_id='.$v->merchant_id.'">'; 


And put in your store's front page URL such as this:

PHP Code:

$html .= '<a href="http://rabbitlvr.com/?merchant_id='.$v->merchant_id.'">'; 


The other pages I would recommend setting up canonical links for your product pages at least (and possibly your category pages).

Here's how to add a canonical link to your product detail pages:
http://www.datafeedr.com/forums/show...4284#post24284

Here's how to add a canonical link to your category pages (if needed):
http://www.datafeedr.com/forums/show...8685#post38685

That should tell the search engines that they should index the canonical version of the page, not a page with anything in the query string.

Eric

[1800HART]

Thanks ... will try but, ... Where do I find dfr_get_merchant_logos() function?
HART

Originally Posted by Eric

Hi
OK, to solve this issue you need to add some code to your dfr_get_merchant_logos() function. Find this line:

PHP Code:

$html .= '<a href="?merchant_id='.$v->merchant_id.'">'; 


And put in your store's front page URL such as this:

PHP Code:

$html .= '<a href="http://rabbitlvr.com/?merchant_id='.$v->merchant_id.'">'; 


The other pages I would recommend setting up canonical links for your product pages at least (and possibly your category pages).

Here's how to add a canonical link to your product detail pages:
http://www.datafeedr.com/forums/show...4284#post24284

Here's how to add a canonical link to your category pages (if needed):
http://www.datafeedr.com/forums/show...8685#post38685

That should tell the search engines that they should index the canonical version of the page, not a page with anything in the query string.

Eric

[Eric]

Hi,

You must have added that function to your theme's functions.php file....

Eric

[1800HART]

Originally Posted by Eric

Hi,

You must have added that function to your theme's functions.php file....

Eric

got it! Just one more thing ...

I'm using one template for all my stores ... and can't use the.

PHP Code:

$html .= '<a href="http://rabbitlvr.com/?merchant_id='.$v->merchant_id.'">'; 


What's the code to replace rabbitlvr.com to just root (or sub)domain i.e. HOME page ?
Thx

[Eric]

PHP Code:

$html .= '<a href="'.site_url().'/?merchant_id='.$v->merchant_id.'">'; 


[1800HART]

Originally Posted by Eric

PHP Code:

$html .= '<a href="'.site_url().'/?merchant_id='.$v->merchant_id.'">'; 


Thank-you kindly!

That was an easy fix, since it fixes every site at once with the same theme

[1800HART]

This is frustrating.... I am still receiving daily WMT error warning messages ... I can't figure out what's wrong or how to fix this and make it stop. This is an example of the http://TurtleLvr.com site - just one of many with the same theme (http://HARTmarket.com/shops)

http://turtlelvr.com/: Increase in not found errors

Google detected a significant increase in the number of URLs that return a 404 (Page Not Found) error. Investigating these errors and fixing them where appropriate ensures that Google can successfully crawl your site's pages.

Recommended action
Check the Crawl Errors page in Webmaster Tools.
Check that pages on your site don't link to non-existent pages. (If another site lists a broken link to your site, Google may list that URL but you may not be able to fix the error.)
Learn more in our Help Center.

This is what that Crawl Errors link page looks like
https://www.diigo.com/item/image/1d8vg/otng

Here's an example of the first link on that image, that does not exist with the code/page I've created..
http://turtlelvr.com/store/leashes/s...3700865177673/


Can anybody offer suggestions why WMT keeps sending me warnings?

[Eric]

Hi,

I guess that seems normal to me. As products disappear from your store (as they will) your 404 pages will increase.

How does Google say to handle these errors?

Eric

[1800HART]

In each email I received each day .. besides the link to the list of crawl errors, they refer to this page:
http://support.google.com/webmasters...answer=2409439

Dealing with Not Found errors

Generally, 404 errors don’t impact your site’s ranking in Google, and you can safely ignore them. Typically, they are caused by typos, misconfigurations (for example, for links that are automatically generated by a content management system) or by Google’s increased efforts to recognize and crawl links in embedded content such as JavaScript.

To see where invalid links are coming from, click the URL. In the error dialog, click the Linked from these pages tab.

If the links are coming from your own site, fix or delete them.

etc/etc/etc

When to return a 404 status code

If you’re getting rid of that content entirely and don’t have anything on your site that would fill the same user need, then the old URL should return a 404 or 410. Currently Google treats 410s (Gone) the same as 404s (Not found).

Returning a code other than 404 or 410 for a non-existent page (or redirecting users to another page, such as the homepage, instead of returning a 404) can be problematic. Such pages are called soft 404s, and can be confusing to both users and search engines.

Originally Posted by Eric

Hi,

I guess that seems normal to me. As products disappear from your store (as they will) your 404 pages will increase.

How does Google say to handle these errors?

Eric

[Eric]

Hi,

I guess the answer I would say (but you might not want to hear) is to not worry about them.

Even Google says:

Generally, 404 errors don’t impact your site’s ranking in Google, and you can safely ignore them.

Is there a reason you are worried about these errors?

Eric

[1800HART]

Originally Posted by Eric

Hi,

I guess the answer I would say (but you might not want to hear) is to not worry about them.

Even Google says:

Is there a reason you are worried about these errors?

Eric

I guess I was/am a little worried, because I've taken the time to create a ?hard? 404 / error page and then had the sites recrawled, and then cleared the crawl errors, and then more of the same or new or different links keeps popping up - or rather, more emails about these 'soft 404' error warnings.

Is it possible - that some sites are just down or not accessible while other data feeds are available? I have my automatic upload/refresh set to 3 days in the factory. If all sites are available except one site, then every day google will crawl my site all products from that one site will be missing? Then, on day 3 it should come back and maybe it's picking up other feeds that are missing products or being worked on.

I wonder - if I change the auto-update interval in the Basic Store Settings in the factory to 1 day instead of 3 days ... do you think it will make a difference? If so, I'll go change it - otherwise, I'll go with Google Help Doc and not worry too much about it - and move on.