Frequent Search Engine Crawler Activity

I have noticed in my statistics now for three days, that Google does not spider my site www.aris-titanium.com It still gets referals from Google. So the site is searchable in Google.
Baruch Avraham LED #2133

Baruch, a search for your site shows Google has indexed 156 pages and you turn up ranked relatively well for many of your important search phrases (#14 for "Titanium Jewelry"). Once those pages are indexed and ranked by the engines, there is very little reason for the spiders to return and re-index them unless you are routinely changing them or adding new products.

"Has anyone else seen anything like this? What should I do about it."

Why do anyting about it unless you have new content you want crawled? This is entirely normal for a mature (domain reserved December 3, 2001 according to Whois records) and stable site without frequent updates. What would be the point? With billions of pages online, the search engines need to conserve resources. Repeatedly crawling millions of mature sites that rarely change would be a waste of crawler time when they could be discovering new and fresh content added daily. Take a look at your site record at the web archive project.

The WayBack Machine at archive.org shows only 8 changes in the past two years for your site (look for asterisks beside the dates on the list at the link above). This "last changed" number is not to be considered highly accurate as the web archive project doesn't crawl that often either. It does note any changes to your pages on its' infrequent visits.

But the point is that unless you have new content which changes often, there is very little reason for search engine crawlers to visit your site more often than you make changes. This is one reason that I counsel clients to consistently add new content in the form of blogs about their products, articles discussing history, unique facts, and any interesting stuff about your industry, designs, materials, etc. I always suggest that they NEVER stop adding new material and NEVER consider their site finished.

Do a search for "Blogging Chocolate Purses" and you'll find an article I wrote about a client site. If you are interested in seeing frequent crawler activity and want to know how to easily blog about your products or services, that article is very helpful.

If you simply enjoy seeing the crawler and don't want to add content, try changing a sentence on your home page, watch for the crawler in your logs, then change another sentence and watch for the crawler to return, change another sentence, etc. If you want the crawlers to visit more pages, simply make that sentence you change link to one of your interior pages and you'll see the crawler visit that interior page soon after.

Once you do that and build up the frequency of those changes, you'll see the crawlers return much more frequently. If your site changes as often as the crawler visits, it will return more often.

Clearly though, it makes no sense to have frequent crawler visits unless you add textual content or new products. Frequent updates can lead to better ranking and varied, interesting background material on your industry leads to more keyword combinations ranking well for your site.

I'm adding this post to my blog and I'll bet you see the Google crawler visit when it sees my link to your site. If you have changed anything on the home page, you can bet that the crawler will return within a few days to see if it changed again. If you don't want to start your own blog, try getting others to blog about you and you'll also see crawler activity increase.

When it comes to crawler activity, change is good. But you may as well make the changes substantive and actually serve your visitors with valuable and informative information rather than change for the sake of change.