Indexed

Every webpage on the Internet is Indexed by search engines. A Web Index is a huge Internet database created and maintained by individual search engines (such as Yahoo!, Google, or MSN). Large search engines such as Google generally maintain Web Indexes of over 3 billion web pages. Each page is indexed separately by the major search engines, rather than indexing the whole website together.

Web indexing is a process that involves sending spiders (bots) crawling through the text (the written content). The search engine parses (analyzes) the text (which may include picking out keywords, and analyzing the grammatical structure of the content to determine if it's just a list of keywords or presumably helpful information that will make sense to human visitors). The search engine classifies the webpage by subject(s). Search engines make a copy of the webpage (creating a cache), and then place the web pages into a huge index of all the web pages on the Internet, so the webpage can be easily found and retrieved.

Individual search engines have different limits as to how much information they will index per webpage. This is an important thing to keep in mind for search engine optimization purposes. The question is: will search engines catch important keywords at the end of a long page of text? Do we have to include all of the important keywords near the top of the page? A safe answer is… sometimes. Traditionally, search engines paid more attention to the first and last of the text on a web page. However, following this advice may not be helpful if you're dealing with a page that's so large (say, 1500 KB) that the last portion of it won't be indexed.

While it was once believed that search engine spiders stopped crawling at 100 KB per page, this is often no longer true. Since 2006, most of the major search engines will generally index a much larger portion of text per page (sometimes up to 1000 KB per page). How much more? In general, Google, Yahoo!, and MSN will stop indexing somewhere between 250 KB and 1000 KB per page.

However, search engine giant Google is still on the wimpier side when it comes to per page indexing power (in an effort to keep the Web index as small and manageable as possible, and make it as fast as possible to get search results). Although Google's main search bot shows up larger sized pages, Google's Fetch googlebot will only show you 100 KB per page (leaving off the final portions in some cases).  So, for search engine optimization reasons, it is still a safer bet to include key information early on the page if you're dealing with a very text-heavy page and not trust that the Index will catch the end of the page.