Make some of the failures into recomendations
A couple of the tests seem a little esoteric and based on wishlists rather than best practices, eg. 'Redirecting 404 traffic to Search Page'. The 404 response code is semantically informative to both humans and I would say it's possibly better practice to preserve this.
Also Captcha on forms is should be weighed up against UX considerations and is appropriate for some sites and not for others, we apply captcha alternatives such as honeypot on many sites, which massively reduce bot submissions without impacting on user experience. One of the reasons given for this test in the report is to prevent brute force attacks on login pages, but there is already brute-force protection in the form of the flood mechanism and password stretching (see http://joncave.co.uk/2011/01/password-storage-in-drupal-and-wordpress/)
That said, You have built a fantastic tool and many thanks for making it publicly available.
Thank you for the feedback. The framework currently supports tests as either pass or fail. Perhaps we could update the Description for such tests to pass on such information.
Regarding the Search404 module option – the module does give 404 response back when the URL is not found and at the same time show the search results for the terms in the URL in the same page. It is not a redirection.
I have to agree with the point you mention about captcha though. We will see how we can address the recommendation option as a generic mechanism and update the description for the captcha tests in the meanwhile.
Thanks once again for the feedback and the appreciation.