A web author's guide to search engines
Part 2 -
How to make search engines work for you
There are three parts to this:
You should also be aware of:
Promote your web pages
Yes, search engines build their databases automatically, but it's well worth
spending some time intervening personally to help them find you.
- Robots work by following links from page to page, so if
you ask other web authors to link to your pages the robots should find you
more quickly (and if a lot of other pages link to yours most search engines
will rank your pages higher).
- Don't worry too much about informing all the search engines about your
pages: the University website is well-indexed by the major search engines,
so as long as there are links to your pages from elsewhere on the University
site your pages will be found.
- Visit search engines and check to see if your pages are listed. If
not, you could let them know your pages exist: most have an 'Add URL' or
'Submit Pages' form. Some 'services' and programs are available that will
submit details of your website to several search engines simultaneously. Be
wary of these, as many search engines will block such submissions. There are
few shortcuts: you will almost certainly achieve better results by
submitting sites individually.
- You will probably receive e-mails that start off by saying things like 'I
visited your website and noticed that you are not listed in most of the
major search engines and directories...' These are scams - they even
receive them at Google!
- Most search engines have some form of paid placement scheme which
will either offer advertising space on relevant search results pages or may
even offer some kind of boosted placement in search results. These can be
effective for some commercial ventures but would appear to have limited use
in higher education, as yet.
[top of page]
Improving your ranking in search results
This is all about making it easy to identify the subject of your web page
through keywords and other criteria.
Robots harvest keywords from your web pages; search engines catalogue these
keywords in their databases; users type keywords into search engines.
Search engine robots have various methods of harvesting keywords from web
pages, but some general principles apply. If you place your keywords carefully
they are more likely to be gathered, and so your pages are more likely to be
ranked highly when users search for those terms. Important places for keywords
include: page title, various areas of the visible area of the page
itself, and for local use in the keywords meta tag.
Keywords in page titles
When robots visit your pages they follow certain rules in order to identify
the important words and terms that might describe the pages' content. One rule
they tend to follow is that words used in a page's title are considered
This is because the title should tell you what the page is
So if you give your main page the title 'Home Page', you're not helping the
robots to classify your website. If, however, you call it something like
'Entomology: undergraduate and postgraduate degrees' you will be using four
keywords in one of the most helpful ways possible.
Note: The page title normally appears in the bar at the
very top of a browser window. In many web editing packages you can change the
title by editing a page's properties (in
SharePoint Designer or FrontPage, right-click
anywhere on the page and select 'Page Properties...'). In HTML, the title
appears (along with any meta tags) in the HEAD section of a page, as follows:
<title>Entomology: undergraduate and postgraduate degrees</title>
Page titles are also particularly important in the presentation of search
results, discussed below.
Keywords as part of your web page
Search robots look at the content of web pages and catalogue
your text, ignoring words like 'and', 'it' and 'the' (and certain others: see
How not to make search engines work for you...). Your page's
subsequent ranking in searches will be influenced by both the number of times a
search term appears and where it appears on the page:
- Keywords appearing at the beginning of a page are usually
considered more important than anything that follows. So try to use the
'inverted pyramid' technique used in newspapers: the headline and first
paragraph should immediately answer the questions 'who, what, where, when
and why?' Indeed, this is good practice in any case: if your page doesn't
quickly convince users that they've come to the right place they're likely
to hit the 'Back' button...
- Some search engines are influenced by the concentration of
keywords: a keyword repeated twice in a hundred-word page has more
influence than if the same word is used twenty times on a ten thousand word
page. Beware of keyword spamming, though (see How not to make search engines work for
- Robots often consider words appearing in headings as
particularly important. So if you're one of the many web authors who uses
<B> and <FONT>
tags to mark up the headings within a page, you're missing an opportunity to
influence robots. Use <H1>, <H2> and so on, instead, and
make sure you use important words in these headings. In this guide, for
example, instead of merely labelling the first main section 'Introduction',
I've used 'Introduction: how do search engines work?'
- If you use frames, your main page might appear to be
virtually content-free (other than the FRAMESET instructions). A search
robot might interpret that as a content-free website... The simple rule is:
don't use frames. Many search engines have serious problems with
frames-based sites, resulting in poor indexing or even a complete failure to
list such sites; and frames-based sites contravene
the University's accessibility guidelines.
- Alternative text for images: the main reason for adding "alt" text
captions to images is to improve accessibility of your web pages, but if you
make sure that the text is informative many search engines will also index
your alt tags, making them a good place to include appropriate keywords.
Keywords in meta tags
Meta tags contain information about your web page. These
tags appear in the HEAD section of a page and so are hidden from users (although
they can easily be seen by looking at the page's HTML source - a very good way
to learn from the example of others). One of these was introduced specifically
so that web authors could specify keywords for their pages. This was very useful
for a time, but then people started to use misleading keywords, repeating
keywords over and over to load a rating for that term, and so on.
While the keyword meta tag has become redundant for most purposes, it
still has uses in a controlled environment. In particular, our University search page recognises these
tags, so they remain useful for influencing University-based searches. The best
advice is not to bother with the keywords meta tag unless you have a problem
page whose local search performance you want to influence - it can be useful for
including common mis-spellings for example: accomodation, harrasment,
etc. Don't, however, go to the trouble of removing these tags from existing
pages - it's not worth it.
Note: the results of occasional tests carried out by WaLT suggest
that, despite claims to the contrary, Google and other search engines do indeed
still appear to notice the words in the keyword meta tag. So while this meta tag
is far less significant than it was in the late 1990s, it's still worth putting
synonyms and mis-spelling in there.
For a fuller exploration of meta tags, see
Adding metadata to your pages.
Keywords in filenames
Some search engines look for keywords in file and directory names, and will
recognise word-separators such as hyphens and underscores. So a URL containing 'fruitandvegetables/applesandpears.htm'
would only contain the keywords 'fruitandveg' and 'applesandpears'. Renaming
that file and directory to 'fruit_and_vegetables/apples_and_pears.htm' would
yield the keywords 'fruit', 'vegetables', 'apples' and 'pears'.
Keywords in link text
Elsewhere in this guide we stress the importance of asking other web
authors to link to your pages. Links from other pages are one of the
best ways to boost your search ranking, particularly if the linking
pages belong to sites that are, themselves, highly-ranked.
It's not only the link that is important for this, though: any
keywords associated with that link will play a part in your page's
ranking. So, for example, a link from an external site phrased as
link to another site
...yields very little in the way of harvestable information about the
page it links to (ie your page). If that external site linked to your
page like this...
link to a site about studying
entomology at undergraduate and postgraduate levels
...tells a search robot exactly what this page links to, and those
keywords will be given high importance in the ranking process.
The easiest way to achieve this is to suggest the phrasing when you ask
the web author to provide a link. There's nothing like making it as easy as
possible for them to link to you!
Keyword placement: conclusion
So, in short, make make effective use of keywords in all the right places,
but particularly in the page title, main page heading and sub-headings, and at
the start of your page.
Choice of keywords
- How many? Robots will ignore enormously long lists of
keywords in meta tags. Choose a dozen or so, and try to include variations
of important terms: 'promote, promotion' for example.
- Only use words relevant to your page - you have to tell the robots what
the page is about, after all. But try, also, to think of distinctive
(but still relevant) keywords that set you apart from similar pages.
- Wordle will generate a word
cloud if you copy all the text from your web page and paste it in, for a
quick visual guide to the dominant keywords on your page.
Addme's keyword density
tool provides a keyword count. Either of these will give you a quick
guide to which words might be indexed from your page. You
can also use these tools to compare the keyword lists of two competing
- Use phrases as well as single words. For example, you
might use 'search engine' as a key phrase for this page. Similarly, in your
titles, headings and body text try to use your keywords close together and
in combination: the example used above ('Entomology: undergraduate and
postgraduate degrees') should be more effective than simply 'Entomology'.
- Some older search engines are sensitive to case and punctuation.
For example, some search engines would regard 'entomology' and 'ENTOMOLOGY'
differently; sticking to lower case is the most inclusive option, so if you
use upper case titles you're limiting your potential audience.
- How do you choose keywords? Use a thesaurus, brainstorm
with colleagues, ask your potential audience what they might search for,
look at the meta tags and use Wordle
for pages that rank highly on the kind of search you think should turn up
- Beware of over-doing things: even if your robot-baiting
is effective, your page might become cumbersome for users, with over-long,
wordy headings and so on; and things like excessive repetition might result
in your pages being ignored altogether (see How not to make search engines work for
[top of page]
The methods used to identify pages' subject matter have moved on since the
days when "search engine optimisation" meant adding a few keywords in a meta
tag. Simply measuring keyword density is easy to influence so, as discussed in How not to make search engines work for
you..., most search engines have taken measures to ignore inappropriate
loading of pages with keywords. Other methods, using "off-the-page criteria",
greatly improve the accuracy of automated indexing:
- Link popularity: search engines count the number of
links pointing to your pages and use this to help rank their search
results - the theory being that if a lot of sites have chosen to link to
a page it must, to some extent, indicate good quality information: each
link counts as a "vote" for the site it links to. Google say that link popularity is the
single biggest factor in determining whether a site is actually indexed
- Link weighting: many search engines, led by
Google, will also regard some links as
more equal than others, weighting results so that links from "important"
sites have more influence than links from "minor" sites.
- Context of links: as noted above, some
search engines will analyse the text of (and near) links that lead to
your pages. For example, Entomology research at Essex
is very likely to point to a page about entomology research, and so the
target page's ranking for those keywords will be boosted.
- Click-through monitoring: some search engines, including
MSN and Lycos, watch the links followed for a
particular search - over time, high-ranking pages that don't attract
clicks will be down-graded, while lower-ranking pages that prove popular
will be promoted.
- Paid placement: some search engines allow advertisers to pay
for a high ranking against specified keywords, or to buy advertising
space alongside search results.
- And...? ...in their efforts to (a) prevent
keyword spamming and (b) gain competitive advantage over other
search services, the search engines tend to be very secretive about
their indexing processes. Who knows what other techniques they might use
to index our sites? Google say that there are over a hundred factors
used in their ranking of web pages.
[top of page]
Improving the presentation of your search listing
The final part of the recipe for improving your search engine performance
involves improving the presentation of your search listing: it's all very well
getting your page into the top ten sites listed on searches for a particular
term - all you have to do now is persuade people to follow the link to your
As we've seen, your page title should include keywords wherever possible. The
title should also be meaningful when listed out of context on a search results
page. If your page title is something like 'Home Page' that's exactly how it
will be listed in the search results. If the page has a relevant and
(something like 'Entomology: undergraduate and postgraduate degrees') you are
far more likely to reach your target audience.
If you look at how search results are presented, as well as the page title
there is often a brief summary of the web page. Correct use of
the 'Description' meta tag allows you to dictate the contents of this summary
for some search engines.
Search engines that do not make use of meta tags may compile the summary
description from the first text they encounter on a page - another good reason
to pay particular attention to the first few lines of text on your page. Another
approach is to quote text surrounding where the search term actually appears in
the page, so if possible you should try to write text in such a way that it
makes sense out of context (eg avoid too many abbreviations which won't make
sense when quoted in isolation) - but this is a lot easier said than done!
So if you follow these guidelines, you should get a listing that looks
Entomology: undergraduate and postgraduate degrees
An online guide to courses and research in the Department of Entomology at the
University of Clacton.
Welcome to our home page! Welcome Academic Staff Administrative Staff Courses
News and Events Contact Us Our aim is to provide a
[top of page]
The last word
All web authors should pay attention to site promotion: if you provide
information on the WWW you generally want people to find it. Search engines use
a variety of automated processes to catalogue websites: you can influence how
your pages are processed by appropriate use of meta tags, keyword placement,
page titling, etc. This is particularly relevant here at the University of
Essex, to ensure that your pages are correctly indexed by our search engine.
But the personal touch is increasingly important. You should
promote your site by asking people to link to it, including its URL in publicity
materials, personally submitting URLs to search engines and directories where
your site is not already listed, and so on.
Above all, you should do your best to produce good, clear pages that are easy
to index and understand. Google's advice is probably best of all: make web
pages for users, not search engines.