Search engine optimization is a hot topic these days for webmasters and content writers alike. SEO means designing your web site so as to get the best possible response from search engines. This doesn’t just mean showing up in a lot of searches; they have to be the right searches, or your visitors will leave your site in disgust as soon as they realize they’re in the wrong place. Without knowing the basics of how search engines operate, you’ll be handicapped in your quest for optimization.

Search engines use a type of software program called a spider, or crawler, to visit web pages. Each web page is “crawled” and then indexed, meaning added to the search engine’s list of active web pages. When a spider visits a web page it will follow every possible link to get every page on the site indexed.

Because of the sheer volume of material on the web, crawling each and every page is extremely resource-intense, even though spiders can crawl pages at an average rate of 100 pages a second. Different search engines use different indexing strategies to best manage their resources. For example, Lycos’ spiders look at specific parts of a web page such as headings and links, and crawl every word in the first twenty lines of each page. Google’s spiders skip over “a,” “an,” and “the” and other articles of speech, but index every other word on every page. AltaVista crawls every single word, including the basic articles of speech.

Once search engines have collected this data, they build an index – a list of all the words and URLs which the spiders have gathered across the Web. This index doesn’t just contain a list of words, however; search engines use a number of tricks and techniques to ensure that the most relevant pages for your search appear at the top of the list, while the least relevant will be clustered at the bottom.

As an example, try firing up Google and enter a search for “jade elephant.” Even this fairly obscure search phrase returned over 200,000 results, and obviously not even the most dedicated jade elephant-lover is going to wade through hundreds of thousands of web pages.

That’s why Google and other search engines try to present the best pages for your search at the beginning of the list. To accomplish this, search engines use “weighting” techniques to rank some pages above others for different search terms. The specific weighting techniques for each search engine vary and are kept secret by the search engine programmers, to try to keep webmasters from rigging their pages so as to take advantage of the weighting methods. SEO experts observe searches and keyword results to deduce how the spiders rank pages, and search engine programmers constantly change their ranking methods to keep the playing field as even as possible.

Most search engines will give more weight to a word or phrase that’s repeated several times in a specific page. They also tend to pay attention to words in important parts of the page, such as text marked with the heading tags and links. A page’s meta tags are also evaluated closely, particularly the title tag.

Now that you’ve had a quick glimpse of how search engines work, you have a leg up on designing your own pages to be as attractive as possible for those little spiders!