What Are Search Engine Spiders And Bots?

When reading or learning about Search Engine Optimisation (SEO), a lot is made of search engine spiders, also referred to as search engine bots. At their most basic level, a search engine spider is a software application that is deployed by search engines and some other sites in order to crawl the web. Search engine spiders follow links they find on web pages in order to discover yet more pages belonging to more sites.

Spiders are the work horse of the search engines and it is these scripts that develop indexes and provide a list of pages used to produce search engine results. Due ot its importance, one of the first stages in search engine optimisation is ensuring that the spiders have a means to locate your pages and then to navigate around every relevant page of your site.

Presenting your pages to the search engine spiders ensures that you get indexed quickly. While it's possible to request that the search engines direct a spider to your site, it's much more effective to get a link from an existing website that is regularly indexed. By doing so it's possible to have the pages of your site indexed in a matter of days.

Once a site is indexed, the site owner needs to ensure that bots are able to follow internal links from one page to another. Text links are the preferred method of navigational options because search engines can struggle to properly follow and subsequently index pages. However, it is conceivable that a website design can include a single text link from the home page to a text based sitemap.