Back to Posts List

Diagnosing web site architecture issues – SMX West 2009 session summary

Feb 27, 2009   |   Web Design and Promotion
Add to Bookmarks
SMX West 2009 session summary

Vanessa Fox, Search Engine Land
Micheal Gray, Atlas Web Service
Daniel Schulman, Google Inc

How do you diagnose your website issues? Is it related to indexing, ranking or crawling? The session provides a good checklist for diagnosing your website search issues and architecture problems.

Important steps for Diagnosing search issues

  1. Identify problems – There are the questions to ask while trying to identify the problem with the website
    – Is it a page rank dropped or a traffic drop or both? If it is page rank drop, its not necessarily an issue. A very common issue heard of is ‘I used to rank at Number 5, but now I can not find myself any where?
    – Is there a drop in the number of pages indexed? Are fewer pages getting indexed? Is the entire site de-indexed?
    – Is the website not getting any traffic?
    – Is the content is SERP not appealing?
  2. Root Cause analysis – Get the data that helps narrow out the problem. There are lots of tools out there. A few resourceful tools are listed here
    Use the Google webmaster Tools – Look at the XML Sitemap stats. in Google Webmaster Tools, which shows pages submitted vs total indexed URLS. This helps figure out if it’s a structural issue with specific pages.
  3. Make improvements – Once the problem is narrowed down, deploy the right changes to your website and use the appropriate tools to doublecheck if problem has been eliminated. A quick case study on ‘The now network’ The site does not show up anywhere in the SERPS, the main reason being that it has a lot of redirect issues. i.e the main website is a 301 redirect to the different website passing on the traffic to a different domain.. Once you know the main problem, you can troubleshoot the problem easily.

Different Sitemaps to consider

HTML Sitemap

• This is a regular page of its own on the website
• Links to all the main sections of the website
• Link from every page – Best SEO practice

XML sitemap – This is an inventory of URL’s.
If pages are being created dynamically, it is a good idea to generate sitemap dynamically by writing code that updates sitemap.xml once a day.

Conceptual sitemaps – They are overall great for planning and reorganizing. It shows order of navigation in order of importance. Several tools available are Visio,, whiteboards etc.

Actual Site Maps – This is how webcrawler see your website.. There are several tools which can help you generate actual site maps such as The other tool recommended was powermapper – This tool scans your website and shows what the bots see. Goal is to realy make sure conceptual sitemap is close to the actual sitemaps.

Domain Canonicalization – At any point, it is advisable to only use one and only one URL structure. www or without www.

Best Practices for Canonicalization

  • Set the Preferred domain in google webmaster. Just tell google which one to use.
  • Do a 301 redirect in .htaccess (use conditional rewrite code)
  • Set default page index.html, index.asp. index.php etc
  • Trailing slashes, query parameters etc can all give canonical issues . Use conditional rewrites to do a 301 redirect back to the homepage
  • Protocol issues – http vs https: especially with secure pages. If you use relative links in your path, it loads https version of the site. Solution is to always use absolute links or path reference.
  • Capitalization issues – always treat url’s with lowercase

Diagnostics for Canonical URL issues –

  • Use Google analytics to troubleshoot. Export the pages out, sort it in excel and analyze issues
  • Duplicate title tags – Use the content analysis tool. Possibility is pages are facing canonical penalty if title tags are same.

Getting new pages indexed

  • Get links on the home page, if possible
  • Create a site map of just new items in xml, submit that to the search engines.
  • If it’s a blog, use a blog ping, social media websites.
  • Track Crawl times regularly

Resourceful tools that help in troubleshooting
Firefox Web developer Toolbar –

It is crucial to identify issue your site is facing and carefully analyze indexing problem, site map problem, duplicate content and domain issue..

Related Articles
1440 Multiversity Provides 60,000 Meals to 25+ Shelters Across Santa Cruz County and Grows During Pandemic by Following Its True North
Holiday Shopping with COVID Highlights Digital Transformation Opportunities with Forecast 51% Increase in Online Purchases