We work very hard to keep our system simple without compromising on the quality of results. This is an ongoing effort, and a worthy one. We make about ten ranking changes every week and simplicity is a big consideration in launching every change. Our engineers understand exactly why a page was ranked the way it was for a given query. This simple understandable system has allowed us innovate quickly, and it shows. The “keep it simple” philosophy has served us well.

Their philosophy, says Singhal, breaks down to essentially three fairly obvious points:

1. Best locally relevant results served globally.
2. Keep it simple.
3. No manual intervention.

Brilliant.  Sounds essentially like Ron Paul’s foreign policy, except for this one little problem:

The second reason we have a principle against manually adjusting our results is that often a broken query is just a symptom of a potential improvement to be made to our ranking algorithm. Improving the underlying algorithm not only improves that one query, it improves an entire class of queries, and often for all languages. I should add, however, that there are clear written policies for websites recommended by Google, and we do take action on sites that are in violation of our policies [emphasis added] or for a small number of other reasons (e.g. legal requirements, child porn, viruses/malware, etc).

Ah, etc., formally known as the good old et cetera, ancient Rome’s own version of the Yadda, Yadda, a meaningless catch-all to let your audience know that you know that there’s more relevant details than you can explicitly state for whatever reason.  We all do it, and I don’t fault Amit for doing it here.  I think he’s speaking from the depths of his beFellowed heart about Google’s best intentions.  However, I think the message is pretty clear if you squint a bit to see it.  If you can’t see it, maybe you can hear it.  Take it Freddy…

Weeeee wiiiiillllll… Weeeee wiiiiillllll… EDIT YOU!

*boom, boom, clap*

Amit’s blog posts, and any of his other for that matter, are certainly ones to keep an eye on.  As he said:

Stay tuned for my followup post, where I will discuss in detail the technologies behind our ranking and show examples of several state-of-the-art ranking techniques in action.

Judging by some of the other numerous and scholarly titles he’s put out, it’s a safe bet his upcoming Google blog posts will be at least as informative as Rand Fishkin’s excellent analysis of the Google patent application or even fellow Fellow Jeff Dean’s detailed speech on Google’s data center architecture at the I/O Conference.  We’re watching you, bro… ever-fascinated with just what you’ll do for/to us next.

And remember kids, practice safe serving.  Always use a unique IP.

(Shout out to Matt Cutts for the heads-up on this via Twitter.)

I doubted that we could trust that redirects on sitemaps would necessarily be followed by every service which supports them. I asked Google’s Matt Cutts about it, and he offered the following suggestions:

Suggestion #1: Write a script in your favorite server-side scripting language which itself requests the master sitemap URL directly and returns it as output.

Example sitemap.php script (PHP 4+):

In PHP, readfile() not only reads the specified file or URL, it automatically outputs it in one step. So if you put this script in each server’s document root, then all you have to do is provide the various Webmaster consoles with http://yoursite.com/sitemap.php for a sitemap URL and it will take care of the rest rather elegantly.

Suggestion #2: Use cross-submit sitemap files.

This is a very useful solution to a lot of problems, though it only solves the problem for Google, and requires creating a site in the Google Webmaster Tools console
for each possible subdomain (or cluster node, I suppose), which will each need their own verification files placed in the server’s document root.

If you’re talking about a lot of sites (and thus a lot of unique verification files), one possible solution may be to wildcard redirect Google verification file requests so literally any google*.html will return a status 200, which is all the Google Sitemaps verification looks for. Sound like a cool idea? Think a little harder…

Imagine you are an unscrupulous practitioner of corporate espionage who wants to access a competitor’s Google Webmaster Tools console for a variety of creative reasons ranging from competitive intelligence gathering (such as their true Google inlink count) to outright sabotage through de-listing a few key URL’s for which they outrank you. Note: Please do not do this and if you do, it wasn’t my fault!

All you would have to do is look for people who have followed the advice of Reaper-X linked above and wildcard-spoofed their own Google verification URL’s, which is easly accomplished by trying a few random Google Webmaster Tools verification URLs on their domain and, once you find one that never returns 404 (always 200), you have found your mark. Then it’s a simple matter of setting their site up in your Webmaster console, immediately verifying ownership, and you’re in like Flynn.

I suppose if you really think you need to implement this hack for your own site, you could reduce (but not eliminate) your risk of this sort of attack by adding a rewrite condition which first checks the User-Agent for Googlebot, and perhaps Googlebot IP addresses as well, but any marginally savvy attacker could write an automated detection script which could properly spoof these HTTP headers, so at best this solution still only provides security through obscurity, and that ain’t no kinda security a’tall.

Search Ranking Acquisition:

Anonymizer Web Harvesting Proxy Service

Yahoo Search API

Windows Live Search API

Google SOAP Search API tombstone and function reference for legacy API key holders

Keyword Search Volume Estimations:

KeywordDiscovery API

WordTracker API

WordsFinder Traffic Estimator API

Google Trends API coming soon

Microsoft adCenter Keyword Research API beta signup email

Search Term Extraction:

Yahoo Term Extraction API and PHP tutorial

WordsFinder Keyword Extractor API

ClickTracks Pro 6+ Reporting API

Average Click-Thru Rate (CTR) of Search Engine Result Pages (SERP’s):

Cornell University Eye-Tracking Study

EyeTools heat map study of Google SERP

CTR analysis of leaked AOL search database

That is, if your web server even has Apache mod_rewrite installed, which some hosts don’t and won’t, so if you have one of those hosts my best advice is to run far, far away. Oh, and make sure your new host doesn’t also delete your log files after 5 days, which a surprising number of bad web hosting services do. Imagine if a stock broker permanently deleted all stock performance data after a week and never looked back! What NERVE!

If you’re hosting on IIS and have have administrative access to the server, do not despair that Redmond has let you down once again, for third-party developers have come yet again to save the day for anyone foolhardy enough to run a server off such a toy operating system. You can install either ISAPI Rewrite or IISRewrite to mimick the behavior of Apache mod_rewrite on a Windows IIS server. This is a great option if legacy code, butt-headed bosses or just plain lack of qualified human resources preclude a platform switch to a real server OS such as BSD or Linux.

If your host won’t offer mod_rewrite or one of those IIS plugins, and moving hosts is not an option, or for a quick-fix Macgyver solution, you can perform 301 redirects by sending the appropriate HTTP headers using your server-parsed scripting language of choice, such as PHP, ASP, ASP.NET or ColdFusion. It looks the same to search bots, but may be a bit slower than a solution that operates from within the web server software itself.

It is with all these inherent difficulties in mind that I wrote this comprehensive 301 redirect howto page to try to shed some light on this poorly-understood and tremendously-important methodology of protecting your search traffic from your changing IA and enticing the bots to crawl friendlier URL’s. It includes instructions for Apache .htaccess (mod_rewrite), IIS (w/admin access), PHP, ASP and ColdFusion.

To create a custom result, click on “Custom Search Results” in the admin menu, then “Create New Custom Result”. Enter the title and URL you wish to appear, and the search terms (separated by commas if you like) which should trigger the display of this custom result. If you have an image you would like to show next to this result, enter the image URL here as well. Hit “Save Changes”. Test your custom result by performing a search, then clicking on your custom result link to register a click in the database.As your site search function gets used, you should periodically check the “Searches with zero results” report to review searches being performed on your site which are not returning standard search results from the index, as these are obviously prime candidates for a customized result.From the admin menu, click on “Statistics” then “Searches with zero results” to see a list of searches which did not return any results.To periodically review the effectiveness of your custom search results, click on “Statistics” from the admin menu then click either “Recent Custom Result Clicks” or “Custom Result Clicks by Keyword”.

As it is still technically in beta and I have not had the time to devote to my ideas for improving the project, I have finally set up a public repository at SourceForge where you can download the beta, report bugs and make feature requests. When i have more time I will add the full source tree to the CVS server for proper management of open-source code submissions. In the meantime, go ahead and download the release and start playing with it right away, and let me know what you think!