Diagnosing web site architecture issues – SMX West 2009 session summary
How do you diagnose your website issues? Is it related to indexing, ranking or crawling? The session provides a good checklist for diagnosing your website search issues and architecture problems.
Important steps for Diagnosing search issues
- Identify problems – There are the questions to ask while trying to identify the problem with the website
– Is it a page rank dropped or a traffic drop or both? If it is page rank drop, its not necessarily an issue. A very common issue heard of is ‘I used to rank at Number 5, but now I can not find myself any where?
– Is there a drop in the number of pages indexed? Are fewer pages getting indexed? Is the entire site de-indexed?
– Is the website not getting any traffic?
– Is the content is SERP not appealing?
- Root Cause analysis – Get the data that helps narrow out the problem. There are lots of tools out there. A few resourceful tools are listed here http://janeandrobot.com/
Use the Google webmaster Tools – Look at the XML Sitemap stats. in Google Webmaster Tools, which shows pages submitted vs total indexed URLS. This helps figure out if it’s a structural issue with specific pages.
- Make improvements – Once the problem is narrowed down, deploy the right changes to your website and use the appropriate tools to doublecheck if problem has been eliminated. A quick case study on ‘The now network’ http://www.google.com/search?q=the%20now%20network. The site does not show up anywhere in the SERPS, the main reason being that it has a lot of redirect issues. i.e the main website is a 301 redirect to the different website passing on the traffic to a different domain.. Once you know the main problem, you can troubleshoot the problem easily.
Different Sitemaps to consider
• This is a regular page of its own on the website
• Links to all the main sections of the website
• Link from every page – Best SEO practice
XML sitemap – This is an inventory of URL’s. http://xml-sitemaps.org/
If pages are being created dynamically, it is a good idea to generate sitemap dynamically by writing code that updates sitemap.xml once a day.
Conceptual sitemaps – They are overall great for planning and reorganizing. It shows order of navigation in order of importance. Several tools available are Visio, writemaps.com, whiteboards etc.
Actual Site Maps – This is how webcrawler see your website.. There are several tools which can help you generate actual site maps such as http://www.microsystools.com/products/sitemap-generator/. The other tool recommended was powermapper – http://www.powermapper.com. This tool scans your website and shows what the bots see. Goal is to realy make sure conceptual sitemap is close to the actual sitemaps.
Domain Canonicalization – At any point, it is advisable to only use one and only one URL structure. www or without www.
Best Practices for Canonicalization
- Set the Preferred domain in google webmaster. Just tell google which one to use.
- Do a 301 redirect in .htaccess (use conditional rewrite code)
- Set default page index.html, index.asp. index.php etc
- Trailing slashes, query parameters etc can all give canonical issues . Use conditional rewrites to do a 301 redirect back to the homepage
- Protocol issues – http vs https: especially with secure pages. If you use relative links in your path, it loads https version of the site. Solution is to always use absolute links or path reference.
- Capitalization issues – always treat url’s with lowercase
Diagnostics for Canonical URL issues –
- Use Google analytics to troubleshoot. Export the pages out, sort it in excel and analyze issues
- Duplicate title tags – Use the content analysis tool. Possibility is pages are facing canonical penalty if title tags are same.
Getting new pages indexed
- Get links on the home page, if possible
- Create a site map of just new items in xml, submit that to the search engines.
- If it’s a blog, use a blog ping, social media websites.
- Track Crawl times regularly
Resourceful tools that help in troubleshooting
Firefox Web developer Toolbar – https://addons.mozilla.org/en-US/firefox/addon/60
It is crucial to identify issue your site is facing and carefully analyze indexing problem, site map problem, duplicate content and domain issue..