"But... But... Why Can't I Get All My Web Pages Indexed By The Search Engines?"

Search engine optimization is about paying attention to the basics. From making sure that you vary your anchor text in your inbound links to ensuring that your keyword density is between 2 and 5 percent, it is all about getting the details right. The only other thing required than that is time.

As part of search engine optimization, there is one factor that is often either missed, or not done properly.

I'm talking about maintaining a good site layout.

There are several very real, money-valuable benefits to having a good site layout.

Proper layout techniques, such as having a site map and executing a proper, planned linking strategy throughout your website will not only get your pages indexed easily (but not quicker), but in some cases proper linking will squeeze out every last sliver of ‘votes' towards your important pages. I'll talk more about site maps and linking strategies later on.

A good site layout is all about converting your visitors into customers. By making an easy to- use, uncluttered and user-centric layout, you are increasing the chances of leading your customer to into making the ‘critical move', whether it is signing up to your newsletter, filling a survey or buying your product.

User satisfaction should be central aim when designing a site layout. Put yourself in the shoes of your visitors, and decide what you want from your website. It is a subtle shift in perception, but it will help you decide whether you really need all those extra menu options on the left or if the design could be simplified by placing those extra links at the bottom of the page; out of immediate view, thus reducing clutter and confusion but within reach if the user needs extra information.

A good site layout will improve the image of your website. Don't just think about search engine rankings – keep your users as your first priority and ensure that your visitors do not go away without being impressed by the clarity and simplicity of your design. Word-of-mouth marketing (either through natural linking or plain blog and forum activity) is a power marketing tool that is largely based on how user-friendly and helpful your website actually is.

A site map has been widely proclaimed as the basic linking tool for site-wide search engine optimization, and with good reason. It presents your website's content – the linking structure of your website – on one single page for search engines and users alike.

Accessibility

Your site map is something like a table of contents for your website. While not the first resort for users looking for information on websites, today's increasingly-aware user audience will more and more turn to a site map if they cannot find something on your website through the traditional menu structure, or if they need to get somewhere really quickly.

Search Engines

A site map, properly mapped and linked from your home page, is the search engine's guide to the depth and breadth of your website. When a search engine spider finally decides that your site in interesting (read important) enough to be indexed further, it will start by exploring links on your home page. Through the site map, it gains immediate one-link access to your complete website, and this greatly speeds up the indexing of all your pages.

Even when the spider does not engage in a deep crawl, a two-level initial crawl is not uncommon, and that will invariably give the spider the opportunity to see all the pages.

For examples of good site maps, check out the following site maps:

• http://www.google.com/sitemap.html – Google Site Map
• http://pages.ebay.com/sitemap.html – eBay Site Map
• http://www.apple.com/find/sitemap.html – Apple Site Map

To ensure that your website is optimally indexed, there are some specific linking strategies that you need to follow. It is NOT as difficult as you might expect. At the very basic level, there are two things you must take care of.

Template-based web design

Design templates before you start designing your website. Using templates to add new pages to your website will not only bring in consistency, but also allow you to standardize the optimal pattern of in-site linking.

This might sound terribly complicated unless you base all pages on a template. With a well-designed template the process is simplified to just updating the placeholder hyperlinks. Then create sub-templates for categories of pages (main category pages, subcategory pages, etc.) to further ease your burden of reconfiguring each page manually.

Site Structure

A template-based design should, apart from speeding up the design process, focus on optimizing your site-wide linking. This will not only with indexing, but also help in increasing SERPS placement due to extra inbound links for your important pages. Base your site structure on solid, site-wide linking strategies like these:

• Each page should link back to the home page.
• Each page should further link back to its main category page.
• Each category page should provide clear links to any sub categories.
• If possible, each page should have the main menu structure – so as to give maximum link exposure to the most important pages of your website.
• Each page should further link to those important pages on your website that do not have any clear category (privacy policy, Help section, user guide, search page, members section, etc.).

If your template is properly designed and as mentioned earlier you specialize your template into sub-templates, your site structure will become more defined and manageable, and your linking strategy will help in both improving the search engine indexing and increasing your rankings.

Reality Check

Don't spend more time than necessary on site structure and optimizing your linking process. The important thing here is to automate as much as possible, and to plan thoroughly. However, simply arranging your site with a basic linking strategy and a detailed site map is enough for your indexing optimization. As for search engine rankings, use your linking strategy to help you get that extra edge, but don't depend on it – inbound links from other websites are much more valuable.

Cascading Style Sheets, or CSS, have rapidly become main-stream for their ability to separate style and formatting from the content. There is a wealth of very useful material on CSS on the Internet – for now I'll just tell you how they can help your site design.

Separating Style from Content

The aim of Style Sheets was to aggregate the style elements common to the whole site into one, single, easily accessible location. The results are nothing short of spectacular; if used properly, you can change the whole design of a website (even those with thousands of pages) by altering just one page. CSS is also a great way to standardize your site design.

Of course, like all design tools, CSS is much better used when it is part of your website design from the very beginning. You can use separate style sheets for content pages, category pages and your main page, or use the same style sheet for all your pages. Whatever division you end up choosing, the advantages of CSS more than justify making the effort to understand how it works.

For dynamic, database driven websites, there are two types of problems that hinder indexing of their site pages. Let's look at both.

Non-HTML Pages

For quite some time, having non-html pages in your website meant that spiders could not properly index your website. That does not hold true anymore, with all major search engines being able to index pages of all extensions – it doesn't matter if your page is .htm, .html, .asp, .aspx, .php or even a file (.pdf, .doc, etc.), a search engine can easily include it in its index.

As for which get indexed quicker, there is no specific evidence suggesting that .asp pages don't get indexed as quickly as .html pages, or vice versa. What matters more is that you have links from other websites pointing to yours so that you can get picked up by search engine spiders as quickly as possible.

Dynamic Database-Driven Pages

If you have a database driven website that involves pages that return results from search queries (e.g. product pages), these cannot be indexed properly by the search engines due to their dynamic content. Essentially, the problem is this:

A search engine spider indexes a page by acquiring its URL, and then parsing the page's content (i.e. the code behind the page – or what we would see if we opened it in a text file). With dynamic, generated-on-the-fly pages, the spider's request to view a page is invalid, since page requests of dynamic pages must be accompanied by some information.

For example, if you have a help section that dynamically acquires help pages from the database, it will probably generate a URL similar to:

http://www.yourwebsite.com/help/info.php?sec=1&q=5

In this case, the page “info.php” will process the request (user-request for a help page) only when provided values for the required variables – “sec” and “q” in this case. Since the search engine does not know this, it cannot provide the values and thus cannot index this page.

Fixing dynamic pages

There is a cure for this problem, and it involves some programming – the code itself is beyond the scope of this article, but one possible answer, if your website is hosted on an Apache Linux server, is to use the ‘mod-rewrite' module and rename all dynamic pages into something short and understandable, such as

http://www.yourwebsite.com/help/help_page.php

For more information, just search for “mod-rewrite” in Google or the search engine of your choice.

Your site layout is full of small details that need to be taken care off. The best advice in this matter is to begin planning right from the beginning – everything from linking strategy to using CSS, ensure that your site design is mapped out. Otherwise, you will spend more time revising the old design rather than promoting your website and adding new content to it.

That wraps up this lesson. I hope all of you will really begin to take your site layout seriously, if you're not already, as the layout of your website plays a MAJOR role in your search engine rankings.

Stay tuned for the next lesson, as I'm going to tell you about a powerful resource that has enabled me to make over $150 extra income every single day... and it's all on complete auto-pilot. I don't do anything!

All the best,

Brad Callen
Professional SEO
SEO Elite: SEO Software