A web author's guide to search engines

Part 2 - How to make search engines work for you

There are three parts to this:

• Promote your web pages.
• Improve your ranking in search results - help people find you.
• Improve the presentation of your search listing - make people want to look at your pages.

You should also be aware of:

• The last word.

Promote your web pages

Yes, search engines build their databases automatically, but it's well worth spending some time intervening personally to help them find you.

• Robots work by following links from page to page, so if you ask other web authors to link to your pages the robots should find you more quickly (and if a lot of other pages link to yours most search engines will rank your pages higher).
• Don't worry too much about informing all the search engines about your pages: the University website is well-indexed by the major search engines, so as long as there are links to your pages from elsewhere on the University site your pages will be found.
• Visit search engines and check to see if your pages are listed. If not, you could let them know your pages exist: most have an 'Add URL' or 'Submit Pages' form. Some 'services' and programs are available that will submit details of your website to several search engines simultaneously. Be wary of these, as many search engines will block such submissions. There are few shortcuts: you will almost certainly achieve better results by submitting sites individually.
• You will probably receive e-mails that start off by saying things like 'I visited your website and noticed that you are not listed in most of the major search engines and directories...' These are scams - they even receive them at Google!
• Most search engines have some form of paid placement scheme which will either offer advertising space on relevant search results pages or may even offer some kind of boosted placement in search results. These can be effective for some commercial ventures but would appear to have limited use in higher education, as yet.

[top of page]

Improving your ranking in search results

This is all about making it easy to identify the subject of your web page through keywords and other criteria.

Robots harvest keywords from your web pages; search engines catalogue these keywords in their databases; users type keywords into search engines.

Search engine robots have various methods of harvesting keywords from web pages, but some general principles apply. If you place your keywords carefully they are more likely to be gathered, and so your pages are more likely to be ranked highly when users search for those terms. Important places for keywords include: page title, various areas of the visible area of the page itself, and for local use in the keywords meta tag.

Keywords in page titles

When robots visit your pages they follow certain rules in order to identify the important words and terms that might describe the pages' content. One rule they tend to follow is that words used in a page's title are considered important.

This is because the title should tell you what the page is about.

So if you give your main page the title 'Home Page', you're not helping the robots to classify your website. If, however, you call it something like 'Entomology: undergraduate and postgraduate degrees' you will be using four keywords in one of the most helpful ways possible.

Note:  The page title normally appears in the bar at the very top of a browser window. In many web editing packages you can change the title by editing a page's properties (in SharePoint Designer or FrontPage, right-click anywhere on the page and select 'Page Properties...'). In HTML, the title appears (along with any meta tags) in the HEAD section of a page, as follows:

Page titles are also particularly important in the presentation of search results, discussed below.

Keywords as part of your web page

Search robots look at the content of web pages and catalogue your text, ignoring words like 'and', 'it' and 'the' (and certain others: see How not to make search engines work for you...). Your page's subsequent ranking in searches will be influenced by both the number of times a search term appears and where it appears on the page:

• Keywords appearing at the beginning of a page are usually considered more important than anything that follows. So try to use the 'inverted pyramid' technique used in newspapers: the headline and first paragraph should immediately answer the questions 'who, what, where, when and why?' Indeed, this is good practice in any case: if your page doesn't quickly convince users that they've come to the right place they're likely to hit the 'Back' button...
• Some search engines are influenced by the concentration of keywords: a keyword repeated twice in a hundred-word page has more influence than if the same word is used twenty times on a ten thousand word page. Beware of keyword spamming, though (see How not to make search engines work for you...).
• Robots often consider words appearing in headings as particularly important. So if you're one of the many web authors who uses <B> and <FONT> tags to mark up the headings within a page, you're missing an opportunity to influence robots. Use <H1>, <H2> and so on, instead, and make sure you use important words in these headings. In this guide, for example, instead of merely labelling the first main section 'Introduction', I've used 'Introduction: how do search engines work?'
• If you use frames, your main page might appear to be virtually content-free (other than the FRAMESET instructions). A search robot might interpret that as a content-free website... The simple rule is: don't use frames. Many search engines have serious problems with frames-based sites, resulting in poor indexing or even a complete failure to list such sites; and frames-based sites contravene the University's accessibility guidelines.
• Alternative text for images: the main reason for adding "alt" text captions to images is to improve accessibility of your web pages, but if you make sure that the text is informative many search engines will also index your alt tags, making them a good place to include appropriate keywords.

Keywords in meta tags

Meta tags contain information about your web page. These tags appear in the HEAD section of a page and so are hidden from users (although they can easily be seen by looking at the page's HTML source - a very good way to learn from the example of others). One of these was introduced specifically so that web authors could specify keywords for their pages. This was very useful for a time, but then people started to use misleading keywords, repeating keywords over and over to load a rating for that term, and so on.

While the keyword meta tag has become redundant for most purposes, it still has uses in a controlled environment. In particular, our University search page recognises these tags, so they remain useful for influencing University-based searches. The best advice is not to bother with the keywords meta tag unless you have a problem page whose local search performance you want to influence - it can be useful for including common mis-spellings for example: accomodation, harrasment, etc. Don't, however, go to the trouble of removing these tags from existing pages - it's not worth it.

Note: the results of occasional tests carried out by WaLT suggest that, despite claims to the contrary, Google and other search engines do indeed still appear to notice the words in the keyword meta tag. So while this meta tag is far less significant than it was in the late 1990s, it's still worth putting synonyms and mis-spelling in there.

For a fuller exploration of meta tags, see Adding metadata to your pages.

Keywords in filenames

Some search engines look for keywords in file and directory names, and will recognise word-separators such as hyphens and underscores. So a URL containing 'fruitandvegetables/applesandpears.htm' would only contain the keywords 'fruitandveg' and 'applesandpears'. Renaming that file and directory to 'fruit_and_vegetables/apples_and_pears.htm' would yield the keywords 'fruit', 'vegetables', 'apples' and 'pears'.

Keywords in link text

Elsewhere in this guide we stress the importance of asking other web authors to link to your pages. Links from other pages are one of the best ways to boost your search ranking, particularly if the linking pages belong to sites that are, themselves, highly-ranked.

It's not only the link that is important for this, though: any keywords associated with that link will play a part in your page's ranking. So, for example, a link from an external site phrased as follows...

link to another site

...yields very little in the way of harvestable information about the page it links to (ie your page). If that external site linked to your page like this...

link to a site about studying entomology at undergraduate and postgraduate levels

...tells a search robot exactly what this page links to, and those keywords will be given high importance in the ranking process.

The easiest way to achieve this is to suggest the phrasing when you ask the web author to provide a link. There's nothing like making it as easy as possible for them to link to you!

Keyword placement: conclusion

So, in short, make make effective use of keywords in all the right places, but particularly in the page title, main page heading and sub-headings, and at the start of your page.

Choice of keywords

• How many? Robots will ignore enormously long lists of keywords in meta tags. Choose a dozen or so, and try to include variations of important terms: 'promote, promotion' for example.
• Only use words relevant to your page - you have to tell the robots what the page is about, after all. But try, also, to think of distinctive (but still relevant) keywords that set you apart from similar pages.
• Wordle will generate a word cloud if you copy all the text from your web page and paste it in, for a quick visual guide to the dominant keywords on your page. Addme's keyword density tool provides a keyword count. Either of these will give you a quick guide to which words might be indexed from your page. You can also use these tools to compare the keyword lists of two competing pages.
• Use phrases as well as single words. For example, you might use 'search engine' as a key phrase for this page. Similarly, in your titles, headings and body text try to use your keywords close together and in combination: the example used above ('Entomology: undergraduate and postgraduate degrees') should be more effective than simply 'Entomology'.
• Some older search engines are sensitive to case and punctuation. For example, some search engines would regard  'entomology' and 'ENTOMOLOGY' differently; sticking to lower case is the most inclusive option, so if you use upper case titles you're limiting your potential audience.
• How do you choose keywords? Use a thesaurus, brainstorm with colleagues, ask your potential audience what they might search for, look at the meta tags and use Wordle for pages that rank highly on the kind of search you think should turn up your pages.
• Beware of over-doing things: even if your robot-baiting is effective, your page might become cumbersome for users, with over-long, wordy headings and so on; and things like excessive repetition might result in your pages being ignored altogether (see How not to make search engines work for you...).

  [top of page]

Other criteria

The methods used to identify pages' subject matter have moved on since the days when "search engine optimisation" meant adding a few keywords in a meta tag. Simply measuring keyword density is easy to influence so, as discussed in How not to make search engines work for you..., most search engines have taken measures to ignore inappropriate loading of pages with keywords. Other methods, using "off-the-page criteria", greatly improve the accuracy of automated indexing:

• Link popularity: search engines count the number of links pointing to your pages and use this to help rank their search results - the theory being that if a lot of sites have chosen to link to a page it must, to some extent, indicate good quality information: each link counts as a "vote" for the site it links to. Google say that link popularity is the single biggest factor in determining whether a site is actually indexed or not.
• Link weighting: many search engines, led by Google, will also regard some links as more equal than others, weighting results so that links from "important" sites have more influence than links from "minor" sites.
• Context of links: as noted above, some search engines will analyse the text of (and near) links that lead to your pages. For example, Entomology research at Essex is very likely to point to a page about entomology research, and so the target page's ranking for those keywords will be boosted.
• Click-through monitoring: some search engines, including MSN and Lycos, watch the links followed for a particular search - over time, high-ranking pages that don't attract clicks will be down-graded, while lower-ranking pages that prove popular will be promoted.
• Paid placement: some search engines allow advertisers to pay for a high ranking against specified keywords, or to buy advertising space alongside search results.
• And...? ...in their efforts to (a) prevent keyword spamming and (b) gain competitive advantage over other search services, the search engines tend to be very secretive about their indexing processes. Who knows what other techniques they might use to index our sites? Google say that there are over a hundred factors used in their ranking of web pages.

[top of page]

Improving the presentation of your search listing

The final part of the recipe for improving your search engine performance involves improving the presentation of your search listing: it's all very well getting your page into the top ten sites listed on searches for a particular term - all you have to do now is persuade people to follow the link to your page.

Titles

As we've seen, your page title should include keywords wherever possible. The title should also be meaningful when listed out of context on a search results page. If your page title is something like 'Home Page' that's exactly how it will be listed in the search results. If the page has a relevant and descriptive title (something like 'Entomology: undergraduate and postgraduate degrees') you are far more likely to reach your target audience.

Summaries

If you look at how search results are presented, as well as the page title there is often a brief summary of the web page. Correct use of the 'Description' meta tag allows you to dictate the contents of this summary for some search engines.

Search engines that do not make use of meta tags may compile the summary description from the first text they encounter on a page - another good reason to pay particular attention to the first few lines of text on your page. Another approach is to quote text surrounding where the search term actually appears in the page, so if possible you should try to write text in such a way that it makes sense out of context (eg avoid too many abbreviations which won't make sense when quoted in isolation) - but this is a lot easier said than done!

So if you follow these guidelines, you should get a listing that looks something like:

Instead of:

[top of page]

The last word

All web authors should pay attention to site promotion: if you provide information on the WWW you generally want people to find it. Search engines use a variety of automated processes to catalogue websites: you can influence how your pages are processed by appropriate use of meta tags, keyword placement, page titling, etc. This is particularly relevant here at the University of Essex, to ensure that your pages are correctly indexed by our search engine.

But the personal touch is increasingly important. You should promote your site by asking people to link to it, including its URL in publicity materials, personally submitting URLs to search engines and directories where your site is not already listed, and so on.

Above all, you should do your best to produce good, clear pages that are easy to index and understand. Google's advice is probably best of all: make web pages for users, not search engines.

[...Previous | Next...]