Yahoo Directory Most Valuable Text Link

Rock Your Rank With a Dynamite Text Link Yahoo Directory Explodes Rankings

Copyright © August 23, 2005 by Mike Banks Valentine

Last week a client called me excitedly exclaiming that their Google PageRank had jumped a notch and their targeted keyword term now ranked #23 (up from #45) for their competitive search phrase. I asked the client if he'd been notified by Yahoo that his site was now included in the index after we had submitted it three weeks ago. "Yes," he said, "but why are you changing the subject?"

"I'm not changing the subject. Inclusion in Yahoo Directory is the most likely reason for the jump in both your search position and your PageRank. Remember when you doubted the value of inclusion in the Yahoo Directory and I pushed for submission anyway? Now you know why I insisted."

That seemingly expensive Yahoo Directory listing has one little known benefit to your website. It is the most important and valuable text link you could ever purchase. That one link from one source will do your ranking more good than any other single link (except possibly the Open Directory).

Many webmasters look at potential traffic referred from the Yahoo Directory as the determining factor for submissions, when that is not the best reason for inclusion - It's the link value that matters above all else in this case.

I've had several SEO clients see a leap in ranking for targeted search terms a week or so after that Yahoo Directory link goes live for them. Many clients have argued with me about the value of that Yahoo Directory text link. But at $299, the yearly fee is cheaper than many of those commercial text link ads sites and does far more for your ranking in search engines OTHER than Yahoo.

Why? ItÕs purely the value of that link. Search engines know that only sites of a certain quality level will submit and get accepted into Yahoo Directory. They know that serious businesses will pay that yearly fee, while marginal or hobby sites will not pay that $299 every year. Surely there is some level of value assigned in search algorithms to inclusion in the Yahoo Directory.

It's a little known technique for gains in ranking which is based purely on empirical observation over time. But the result of inclusion in the Yahoo Directory is the same for every client, every time - their PageRank ratchets up and targeted search terms suddenly take a big jump just a week or two after inclusion. The same is true of inclusion in the free Open Directory Project at http://DMOZ.org .

I've seen client sites jump from positions on page three at MSN search to top 5 positions on page one of the MSN SERP's following inclusion in the Yahoo Directory. Now we are submitting this same client to the Open Directory for the 5th time in as many months hoping for elusive editors to add the site, leading to another jump in search position and PageRank if and when they get around to adding the site to the DMOZ database.

You do know that Google uses that Open Directory listing in their own directory, don't you? It's worth submitting and resubmitting until they finally include your site. It really is worth the trouble to keep trying, no matter how long they ignore your submissions.

One caveat always applies to Directory submissions! They must be done with great care applied to keyword phrases used in the site description. That single line of text you submit in the "Site Description" text box on the submission page will strongly affect your keyword phrase ranking in OTHER search engines for a very long time.

Take care in crafting a keyword rich and effective description for your site. I always request that clients either have me submit for them or use text I've written for them in that description. If you do it badly, it will be re-written by an editor who cares more for categorization than keyword rankings. Be Careful!

Before you run off begging for reciprocal links from slick webmasters or purchasing text links of dubious value from text link outfits, submit to Yahoo Directory and pay the $299 (or $24.92 monthly) for the most undervalued text link available. Then swallow your pride and re-submit to the Open Directory until they finally include your site.

Rock your rank with dynamite text links! Yahoo Directory and the Open Directory Project.

Mike Banks Valentine operates http://Publish101.com Free Web Content Distribution for Article Marketers and Provides content aggregation, press release optimization and custom web content for Search Engine Positioning http://www.seoptimism.com/SEO_Contact.htm

robots.txt File Important to Search Engine Crawlers!

robots.txt File Important to Search Engine Crawlers!

The robots.txt is a web standard required by all web crawlers/robots to tell them what files and directories to stay OUT of on your site. Not all crawlers/bots follow the exclusion standard and will continue crawling your site anyway. I like to call them "Bad Bots." We block them by IP exclusion which is another story entirely. ;-)

To see the proper format look below. That file should be at the root of the domain because that is where the crawlers expect it to be, not in some secondary directory.

Below is the proper format for a robots.txt file ----->

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /group/

User-agent: aipbot
Disallow: /

User-agent: BecomeBot
Disallow: /

User-agent: msnbot
Crawl-delay: 10

User-agent: psbot
Disallow: /

User-agent: Teoma
Crawl-delay: 10

User-agent: Slurp
Crawl-delay: 10

--------> End of robots.txt file

The above is what we currently use at Publish101.com Web Content Distributor site, just launched in March of 2005. We did an extensive case study and published a series of articles on crawler behavior and indexing delays known as the Google Sandbox. That Google Sandbox Case Study is highly instructive on many levels for webmasters everywhere and, although I'm biased ;-) I highly recommend that you consider reading it in full. All articles are linked from the one above so you can read the entire series.

One thing we didn't expect to glean from the research involved in indexing delays was the importance of robots.txt files to quick and efficient crawling by the spiders from the major search engines and the number of heavy crawls from bots that will do no earthly good to the site owner, yet crawl most sites extensively and heavily, straining servers to the breaking point with requests for pages coming as fast as 7 pages per second.

The "User Agent: MSNbot" is from MSN, Googlebot is obvious, Slurp is from Yahoo and Teoma is from AskJeeves. The others listed are "Bad" bots that crawl very fast and to nobody's benefit but their own, so we ask them to stay out entirely. The * asterisk is a wild card that means "All" crawlers/spiders/bots should stay out of that group of files or directories listed.

The bots that are instructed to "Disalow: /" means they should stay out entirely and those with "Crawl-delay: 10" are those that crawled our site too quickly and caused it to bog down and overuse the server resources. Google crawls more slowly than the others and doesn't require that instruction. Crawl-delay instruction is only needed on very large sites with hundreds or thousands of pages.

Those we requested Crawl-delay from were requesting as many as 7 pages per second and so we asked them to slow down. The number you see is seconds and you can change it to suit your server capacity and based on their crawling rate. You can discover how fast they are crawling by looking at your raw server logs - which show pages requested by precise times to within a hundredth of a second - available from your web host (or ask your web or IT person).

This file is saved as a plain text document and ALWAYS with the name "robots.txt" in the root of your domain. Your server logs can be found in that same root directory if you have server access, you can usually download compressed files by calendar day right off your server. You'll need a utility that can expand .gz files to open and read those plain text files.

To see the contents of any robots.txt file just type robots.txt after any domain name. If they have that file up, you will see it displayed in your browser. Click on the link below to see that file for Amazon.com

Amazon.com robots.txt

You can see the contents of any website robots.txt file that way.

We discovered in our launch of the new site that Google and Yahoo will crawl the site whether or not you use a robots.txt file, but MSN seems to require it before they will begin crawling at all. All of the search engine robots seem to request the file on a semi-regular basis to verify it hasn't changed.

Sometimes when you DO change it, they will stop crawling for a week or so and repeatedly ask for that robots.txt file during that time without crawling any additional pages. (Perhaps they had a list of pages to visit that included the directory or files you have instructed them to stay out of and must adjust their crawling schedule to eliminate those files from their list.)

Most webmasters instruct the bots to stay out of "image" directories and the "cgi-bin" directory as well as any directories containing private or proprietary files intended only for users of an intranet or password protected sections of your site.

True Paid Inclusion Programs are a “Thing of the Past” . . . Or Are They?

True Paid Inclusion Programs are a “Thing of the Past” . . . Or Are They?

By Robin Nobles

Most search engine optimizers get rather nostalgic when they remember the “good old days” of paid inclusion programs. Do you remember when we could submit a Web page for a one-time fee and get it spidered and indexed within 24 to 48 hours? The page would then be respidered every 48 hours for an entire year.

We could work on building our on-page and off-page factors while confidently knowing that the paid inclusion spider would be back to recrawl our page within that time frame. We didn’t have to pay any pay-per-click fees. For new pages or Web sites, or for pages that we had problems getting in the indexes, paid inclusion programs were ideal. We never minded spending the money at all.

These days, however, with the exception of a few directory submissions, true paid inclusion programs are a thing of the past . . . or are they?

Some Paid Inclusion Programs have Per-Click Fees Attached as Well

The paid inclusion programs of the past certainly didn’t come with pay-per-click fees attached, but some of them do now. Yes, the visibility can be impressive, but the programs come with definite disadvantages:

* You not only pay to submit the URL, you also pay a per-click fee, depending on what industry you’re in. So before you rush into these programs, it would be to your advantage to see how much you would pay to bid on your keywords through straight pay- per-click programs.

* It’s important to remember that with these paid inclusion/PPC programs, you’re simply “paying” to be “included.” You’re not “paying” for your “rankings.” Therefore, you’re in the rankings along with all of the free search engine listings, yet you’ve paid to be there and you’re paying per click through.

* Though I have known a few people who have stopped the paid inclusion listing (and the per-click fee) and have remained in the listings, most people totally drop out of the rankings. So if you get where you want to be, be prepared to pay to stay there.

A True Paid Inclusion Program – With Listings Seen Over 120 Million Times a Month!

The Big Three aren’t the only engines on the block. At 120 million searches a month being conducted across 50 search engines and directories and through ABCSearch.com, ExactSeek.com’s Featured Listings are certainly an excellent deal.

The cost? $36 per URL for the first 5 URLs for a year, or $12 per URL for the first 5 URLs for 3 months. More good news: no pay- per-click fees and no bidding for keywords.

What search engines are covered?

ExactSeek.com, Aesop.com, OnSeek.com, MaxPromo.com, Best- SearchEngine.com, SitesOnDisplay.com. and MetaWebSearch.com. View the entire listing at http://www.exactseek.com/featured_listings.html.

These search engines and directories have formed the Independent Search Engine & Directory Network (ISEDN.org http://www.isedn.org/) for the purpose of providing Web site owners with affordable and effective search engine advertising.

In fact, the Featured Listings program not only gives you “paid inclusion,” but it also gives you “guaranteed top 10 exposure.” I asked Mel Strocen, CEO of Jayde Online to explain.

“The sale of any keyword term (word or phrase) is restricted to 30 buyers. So, for example, a phrase like ‘online games’ can only be purchased 30 times. This is to prevent over sale of a term. Since the maximum number of paid listing ad boxes that can appear on a SERP is 10 and 30 is the maximum that can be sold for any keyword term, we rotate ad boxes within the SERP and between SERPs for every search on a given term.

“If an advertiser buys a keyword term that has been previously purchased 10 or fewer times, then that advertiser's paid listing ad box will appear on the first page of search results every time the purchased keyword term is searched. If the keyword term has been purchased between 11 and 20 times, then the advertiser's PI ad box will appear on the first page of search results one of every 2 searches performed for that keyword term. And, if the keyword term has been purchased between 21 and 30 times, then the advertiser's PI ad box will appear on the first page of search results one of every 3 searches performed.

“This rotational system gives everyone who has purchased a keyword term equal exposure.”

According to Strocen, a number of engines and directories are poised to join the ISEDN, and they’re targeting for a membership of 100 engines and directories by year-end. This is serious visibility. Your listings will be added within 6-8 hours to all of the above search engines. Just like the good old days!

It’s important to note that paid listing results are served from a separate database. Having a paid listing does not mean you also have a regular, free listing. To get in their regular, free results, Webmasters still need to submit their URLs at http://www.exactseek.com/add.html.

For more information about their paid inclusion program, visit http://www.exactseek.com.

Paid Inclusion as an Alternative to PPC

With the rising costs of pay per clicks, when you can find a paid inclusion model as attractive as ExactSeek.com’s, it’s time to give it a try.

For example, at the time of writing this article, the top three bids for “life insurance quotes” at Yahoo!’s Sponsored Search were going for close to $6 a click through. If you have one of the top three pay-per-click ads, and if six people click on your ad, you’ll spend $36 a day.

You could spend the same amount of money one time at ExactSeek.com and get your URL seen over 120 million times a month for a solid year.

Isn’t it worth a try?

In Conclusion

Thank goodness, the days of true paid inclusion programs aren’t over. ExactSeek has a very attractive program with excellent visibility . . . just waiting to be tested. Remember: in the search engine game, testing is the best way to see if something works. Now, get busy!

Robin Nobles teaches 2-, 3-, and 5-day hands-on search engine marketing http://www.searchengineworkshops.com workshops in locations across the globe as well as online SEO training http://www.onlinewebtraining.com courses. They have recently launched localized SEO training http://www.searchengineacademy.com centers through Search Engine Academy.

Released From Google Sandbox, Searching Playground

The Google Sandbox Effect has been discussed at length in our case study of a new website first crawled in May by Googlebot. We can now further the case study with indexing comparisons and discuss interesting Googlebot crawler behavior after release, at the 75 day mark, of the study website from that very confining Sandbox.

This case study is not for the faint of heart - those just launching a new web business on a new domain name with hopes of instant indexing and immediate traffic may find their website very lonely for two and a half months - if it is in a competitive market segment. You may as well plan to stay in the Google Sandbox for at least 45 days on average. If some early release stories are to be believed, search phrases nobody wants to play with are taken pity on by Google and sent home for early release.

Those non-competitive or obscure search phrases seem to be seen as good, quiet little children, playing by themselves in Sandbox playground and are sent home early on good behavior. Googlebot probably sees good behavior as playing well with others, like a good little baby domain and NOT being competitive as some young domains can be. Throwing sand in other childrens' faces and insisting on having your site indexed, throwing sand out of the Sandbox with your bright plastic toy shovel and bucket.

Now that the site discussed in this study is out of the Sandbox, it still lingers on the playground, unable to escape the community park and leave for the business world to play with the big boys in the outside world. It does indeed take time to grow up and be the model citizen in this new search playground. Though on the first full day after this first week of being released from the sandbox, the site has gotten 68 visitors referred by searches done at Google, the first referred search traffic coming into the site. MSN has sent 8 visitors, Yahoo has sent 6, 4 came from AOL searches, 2 from Netscape and 1 from Dogpile.

The indexing behavior of Yahoo and MSN has been nothing short of bizarre with numbers of indexed pages increasing rapidly over the first two months to reflect 6,941 pages indexed until 8 weeks into this study and we outlined previously how numbers changed as you click through results pages first upward, then downward to about half the total of highest numbers listed along the top of the results pages.

It appears that Yahoo and MSN are playing on the 'slippery slide' in this playground, climbing to the top of the ladder of results at about 10 week mark showing 8,210 and 6,941 pages respectively indexed, then sliding down again to 3,510 for Yahoo and 373 for MSN, as of this writing two weeks later on August 6. Still, Yahoo will show you only 1,000 (100 pages) of those results and MSN will show you only 250 results, or 25 pages, no matter how many they claim to index. MSNbot is crawling the site faster and more consistently than any of the engines, yet shows by far fewer pages indexed than the others.

One of the interesting comparisons between Google and MSN in our Sandbox study is that Google will show you most of what they claim to have indexed after you click that link at the bottom of the first page showing only 3 or 4 results when you use the "site:Publish101.com" query operator then go to the bottom of the page and click the link under the line reading, "In order to show you the most relevant results, we have omitted some entries very similar to the 3 already displayed. If you like, you can repeat the search with the omitted results included."

Go ahead and click that link, then you'll be presented with the claimed total of indexed pages. That number has very steadily increased since Sandbox release after 75 days from first crawling of this Sandbox study site. The timing and numbers of indexed pages at Google goes upward, and ONLY upward with VERY distinct patterns noted from raw log files. Crawling schedules seem to have been established for this site by Google and indexing changes occur on a very regular schedule.

The first observation of Sandbox release was at noon on Thursday July 28, seventy-five days from first crawling by Googlebot when a search turned up 379 pages indexed with a "site:Publish101.com" query. That number increased later the same evening to 3,660 pages at a search done around the dinner hour Pacific time. Oddly, the next day, Friday July 29, the number took a slight hop upward to 3,700 pages and on the following Monday, showed 3,770 pages indexed.

That schedule and pattern have repeated on the second week of Sandbox release when a "site:Publish101.com" query produced 5,660 results from from Google for the site on Thursday August 4 at just after noon and then nearly doubled at around the dinner hour to 10,700 pages on that same query. A final check just now on Saturday shows it at 12,100 pages indexed by Google. It should be pointed out to those who wonder about the total number of pages that this is a dynamic site with a very large archive of articles that increases daily as new submissions are contributed by member authors at the site.

Those articles are added through a content management system on a daily basis by an editor who reviews submissions and processes them for approvals or rejections. Those approved are made live from the home page nightly. We've started doing this on the crawler's schedules as we've noted very regular visits by Yahoo's Slurp crawler to the site home page just once daily at around 5pm each evening and Googlebot visiting the home page only once, at near 11pm nightly, so we've instituted a midnight activation of each day's new article submissions on the home page of the site so that none of the new pages are missed by those crawlers. MSNbot seems to hit the home page multiple times through the day, so timing is less important for MSN.

Crawler activity has been heated, with Yahoo crawling the least and the slowest, barely seeming to attempt any updates and the total of indexed pages has not changed for over three weeks since it peaked at 8,210 pages indexed and then dropped to it's current level of 3,510. As previously stated, Slurp seems to be unhindered by any form of consistency in indexing or crawling behavior. MSNbot has crawled extensively and fairly regularly for weeks, but that odd indexing behavior is a serious flaw in their utility as a search tool.

It should be mentioned here that AskJeeves had been noted to crawl the site extensively early in this case study and displayed a very regular and consistent crawl, but stopped abruptly three weeks ago on july 13, after hitting most of the pages then available on the site. Teoma, their spider, has been absent ever since and they have not indexed this domain at all since first crawling on May 23, over 10 weeks ago. Clearly, Teoma appears to have the longest Sandbox of all the search engines.

Much has been learned in this Sandbox case study about crawler behavior, indexing delays, robots.txt requirements and index updates at each of the top three search engines. Where that knowledge leads will, of course, change as algorithms and crawling schedules are adjusted by MSN, Yahoo and Google. But valuable information has been shared that may help other webmasters to better understand each of the factors that determine the success of any website.

We'll outline the findings in a follow-up article at the 3 month mark and explore search referrals gained as Google adds more pages and rankings fluctuations begin to level. Meanwhile, we'd like to encourage others to review traffic logs to monitor crawler behavior on other new domains to verify findings and disclose indexing behavior and timing for new domains.

http://Publish101.com

Google Delivers 4 Times the Buyers Of MSN

This article at WebProNews falls into the "Lies, Damned Lies and Statistics" trap by suggesting that MSN delivers far fewer buyers than does Google. That being the case, since Google delivers 65% of the search engine referred traffic to the vast majority of web sites and MSN only 15%, their statistically higher percentage of buyers is far less significant to the numbers of visitors delivered.

Let's turn 100% into 100 people, OK? Given 48% of 15 people (being generous with estimates of MSN sending 15% of traffic) that is 7 people ready to buy. Google on the other hand delivers 65 people (percent) and 42% of them ready to buy something - THAT IS 27 people who came from Google "Ready to buy" things versus 7 people from MSN "Ready to buy" things. MSN may have higher percentages, but FAR lower numbers to draw that percentage FROM.

This shows clearly that Google sends 4 times the number of buyers as MSN does. That is if there really are many ecommerce sites that receive 15% of their referred search engine traffic from MSN. Most studies show that number as being much lower in actuality.

This percentage game is an absurd numbers game ComScore MediaMetrics plays to make MSN & Yahoo look good every quarter as we're discussing financial predictions and it makes me crazy. Every news organization dutifully reports the percentages and perpetuates the lie that Yahoo and MSN are doing better than Google. It is a complete fabrication.

Let's rate AskJeeves Search against Google. Since AskJeeves sends 1 visitor for every 5000 that Google sends, would it matter if 100 percent of AskJeeves searchers ALWAYS purchased something? No.

If only 1 percent of Google's 5000 visitors purchase something, then they still beat AskJeeves by 50 times, even though their "intention to buy" statistic is (in this absurd example I've given here) only 1 in a hundred. It matters far more that Google sends 500 times the traffic as AskJeeves than "Intention to Buy" matters. How do those guys figure out so many ways to look at irrelevant data?

Review my previous article on this topic at:

Search Engine Journal

Mike Banks Valentine
Publish101.com