While the fastest way for manual link checking is the LinkChecker Firefox Plugin, it is not so easy to check our whole site for 404’s. Same thing with validation. For fast manual checking of a single page I would recommend to install the Web Developer Toolbar in Firefox and simply press Shift + Ctrl + H or Shift + Ctrl + A. But how to validate all pages or our site? And how doing it offline for performance reasons?

My approach is to mirror the whole web site with wget, look at the wget log for dead links and use the WDG Offline Validator to validate the mirrored HTML pages.

Mirror with wget

I assume, you use Linux or a Unix like system. So wget wouldn’t be new for you or you will be able to get it for your system. It’s a pretty basic but powerful tool.

First you should create a new directory where wget could download all your pages. On your console you can execute this command line to get your whole site:

1. wget --mirror --keep-session-cookies -o wget.log

I use the --mirror switch to simply fetch all. The --keep-session-cookies switch is useful is your site is dynamically created as this blog for example. -o wget.log says, it should put the output into this file. Be sure your server would hold against the stress!

Once wget finishes, you could use less or your favorite editor to search inside the wget.log for 404’s and the string error. This is all what you will need for link checking.

Validation with the WDG Offline Validator

I searched a while for a usable offline validator. The W3C one is a CGI script which needs a running Apache and you have to do a HTTP post in order to check your local HTML file. It is basically the same thing as the public W3C Validator. The next one I did not choose is a Windows application called A Real Validator. The disadvantages are that it costs money and has a little bit dated GUI which does not allow to filter for only invalid pages. So you have to scroll though hundreds of valid pages to find your invalide ones.

So at the end I use the WDG Offline Validator. You can get it from this site but the best thing is that Debian and Ubuntu have it available in there package repositories. So you can just type:

1. sudo apt-get install wdg-html-validator

(Be sure you have the universe repository in our list.)

To validate all the HTML pages wget downloaded, just type:

1. find . -name "*.html" -exec validate -w {} \; > validation.log

This command finds all yout HTML files in your current directory, executes the validate command on everyone and outputs the results in the validation.log file. While this runs, you can look at the validation.log with tail or you can view it afterwards in whatever editor you like best.

So that is basically all what you need to check your whole site for dead links and valid HTML.

On 456 Berea Street I read the following on the Accessibility Page:

This site is built on valid HTML 4.01 Strict for structure and CSS for presentation.

A modern web browser like Firefox, Safari or Opera is needed to make the most out of this site, but thanks to the separation of content and presentation it should be accessible to any browsing device, including Internet Explorer.

Take this as the funny part of this post. Now I’m going into the silly HTML vs. XHTML discussion.

HTML 4.01 Strict?

So the real cracks use HTML 4.01 Strict today even if they used XHTML in the past. The key point choosing XHTML is senseless today is that you have to deliver (X)HTML pages as text/html because the famous Internet Explorer doesn’t understand application/xhtml+xml. But if you deliver as text/html all browsers interpret your nice XHTML as tag soup anyway. Thats why some of the people caring about this stuff switched back to HTML.

After playing some time with Wordpress and its Themes/Plugins I realized that it is not simple to switch to HTML 4.01 Strict. The whole Wordpress world uses XHTML 1.0 and every single peace outputs /> instead of > on empty tags. So without rewriting nearly all Wordpress code, it would not be possible to generate real valid HTML.

The other fact that pushes me towards XHTML is a article of Christoph Schneegans which reads XHTML oder HTML?. He says that a HTML Validator wouldn’t complain about valid SGML shortcuts which can cause rendering errors in browsers. To cite him – this markup is perfectly valid HTML:

1. <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
2. <>
3. <title//
4. <p ltr<span></span</p>
5. </>

and it is equivalent to:

1. <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
2. <html>
3. <head>
4. <title></title>
5. <body>
6. <p dir="ltr"><span></span></p>
7. </body>
8. </html&gt

To summarize: I will use XHTML 1.0 Strict delivered as “text/html” for now and maybe (X)HTML5 later on in some years.