TargetUrl should allow the usage of Basic Auth credentials.

Add a rule to completely validate the SSL certificate of a webpage

As I was reviewing code for another issue, I began thinking about a sitewide SEO prelaunch check. I would imagine it being run in one of two ways:
php ./bin/kickoff.php seocheck mysite.com/sitemap.xml
or
php ./bin/kickoff.php sitemap mysite.com

Basically, it would read the URLs from sitemap.xml and then run the 'seocheck' on each page resulting in a single report that could be used to fix SEO issues throughout the site.

Perhaps there could be some sort of paging in the case of too many URLs:
php ./bin/kickoff.php sitemap mysite.com 0-19
or
php ./bin/kickoff.php sitemap mysite.com link:0-19
to run through the first 20 links.

Does this sound feasible or even something you want right now? Do you think it would require a great amount of rework?

New rule to validate according to
https://dev.twitter.com/cards/markup

Required, to keep german lawyers unemployed

Add a rule to test whether either a valid x-ua-compatible HTTP-header or meta tag exists.

There should be a new command, which can be called to perform basic security checks on a given website.
Example:

php ./bin/kickoff.php securitycheck http://www.google.de

There should be a new command, which performs basic performance settings checks.
Example:

php ./bin/kickoff.php performancecheck --image https://www.google.de/images/nav_logo242.png --styles https://www.google.de/whatever.css http://www.google.com 

As stated in #21 by hanzi, the parser currently uses UTF-8 as default, but it should be read from the HTTP-headers instead and fall back to ISO-8859-1 .
There might be some need of exception handling, if the content cannot be read properly.

See: https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1

The

target:
    path:

element should be either a normal string (as it is right now), or an array of strings, which applies the section with all its rules and setings to all listed url paths.

Add a CSP-Rule.
There is no need for a meta-CSP-tag check. Those checks are far more done via HTTP Header (if at all).

There might be a 2 different rules:

• Check for existence of CSP
• Check if the current page meets the CSP requirements. This should be possible, since the body and the headers are fetched in the same request and thus, the body could be checked according to the CSP header.

There should be a new command, to perform basic seo checks on a given website
Example:

php ./bin/kickoff.php seocheck http://www.google.com

The readme.md file needs improvement by a native english speaker or at least somebody, who is more fluent in english than me :-)

Add <meta name="generator" content=...> not Present Rule

Before I dive into the implementation, I wanted to run this by you guys first. What do you think?
(Sorry for the long text. Below is an example if you just want to see the result.)

The Problem

While trying to assemble a config file for one of my projects, I ran into a few inconsistencies and things that feel unnecessarily complicated. Here are my thoughts, in no particular order.

1. In a section, you cannot use a target node to overwrite the one in defaults. Instead, you have to call it config here even though you’re overriding target with it.
2. The name of the uri node is misleading. It contains the path, not a URI.
   URI: https://www.example.com/foo/bar?baz=qux (more specifically a URL, but whatever)
   Path: /foo/bar
3. Defining scheme, host and port separately seems rather useless. Not a big problem, but for any use case that I can imagine it should be enough to define a base (https://www.example.com) and a path (/foo/bar).
4. Since we already have to manually define names for our rules, there is actually no reason for them to HaveAWeirdNotationLikeThis. Without any infrastructural changes whatsoever, YAML would already support more human-readable names.
5. The syntax for setting rule options may be easy to parse, but is harder to write and to read. For example, HttpRequestTime: [["maxTransferTime", 1500]] looks a bit weird. Also, it mentions the same thing (download time) twice and uses different words (request time and transfer time.)

My Proposal

Issues 1 and 2 are easily solved: Rename config to target and uri to path. Each section is always deep-merged with the defaults section so everything behaves as you would expect.

Issue 3 is a matter of opinion. If we choose to do this, my vote goes towards having base and path that simply get concatenated. So if you define { base: http://www.example.com/foo, path: /bar }, the resulting URL would be http://www.example.com/foo/bar.
path defaults to /.
base would be a required setting for now, but later we could even add the option of setting it via command line (bin/kickoff.php check foo.yml http://www.example.com.)

So much for the easy part.

Rule Sets

This references #5, #6 and #7.

The idea is to define groups of rules that can be applied in bulk, and that contain some sensible defaults.

Now the issues are about creating dedicated commands for those rule sets, but I don’t see any reason why we shouldn’t offer them in the config as well. So next to the rules node I would introduce a rule sets node that contains an array of strings:

rule sets:
    - seo
    - security

rules:
    - …

This would require looking up the rule sets seo and security and simply merging them into the rules node internally.

Natural Language Rule Names

Instead of HttpRequestTime: [["maxTransferTime", 1500]] you could just say downloads faster than: 1500ms.

downloads faster than would be the name of the rule and 1500ms its option.

I was considering the following rules:

1. Rule names are case insensitive.
2. Punctuation is ignored, so contains an <h1> tag is interpreted as contains an h1 tag. Of course, this is limited by YAML itself. Can’t use colons or quotes too freely.
3. A rule can optionally have a parameter, like shown above. It does not have to: exists would be a rule as well.
4. An option can either be a scalar value or an array or a hash. It’s the rule’s job to properly parse that.
5. Names and options should be chosen so that the resulting config is a reasonably readable list of English sentences without too much weird syntax. Again: limitations imposed by YAML.

See below for a full example of what I mean.

Allow For A Simpler Version?

Another idea is that if neither defaults nor sections nodes exist, we just treat the config as if it were part of a sections block.

But I’m not sure whether that is actually helpful or just adding more fragmentation.

Or, we could drop the sections node entirely and just treat defaults as a special kind of section.

Full Example

defaults:
    target:
        host: https://www.google.com
        path: /
        headers:
            X-Foo: bar

    rules:
        - exists
        - is gzipped


sections:
    main page:
        target: { path: / }

        rule sets:
            - seo
            - security
            - performance

        rules:
            - downloads faster than: 1000ms
            - contains: "I’m Feeling Lucky"
            # The rules “exists” and “is gzipped” are included as well because
            # they are defined in the "defaults" section.


    non-existant page:
        target: { path: /404 }

        rules:
            - is missing


    all rules:
        target:
            host: http://www.example.com
            path: /foo/bar
            headers:
                X-Test: yes
                # disables the X-Foo header that has been set in the defaults
                X-Foo: ~

        rules:
            - enables xss protection
            - does not expose server software:
                tolerate server without version: true
            - downloads faster than: 1500ms
            - has expires header in the far future
            - contains etag
            - exists
            - is gzipped
            - disables content sniffing
            - is missing
            - responds 304 to if modified since
            - prevents framing
            - restricts framing to origin
            - sets httponly flag for cookie: session_id
            - sets secure flag for cookie: session_id
            - enables hsts:
                includes subdomains: true
            - title tag length is good:
                min: 10
                max: 50
            - meta description length is good
            - open graph properties are present
            - twitter properties are present
            - canonical link is present
            - has hreflang tag
            - has apple touch icons
            - does not have generator meta tag
            - contains: "foo bar"
            - contains an h1 tag
            - contains exactly one h1 tag

There should be a new main area:
Rulesets

Each rule set should consist of a name and one or more rule assigned rules. Example:

Rulesets:
    - security:
            - ValidSslCertificate
            - HttpHeaderXSSProtectionSecure
            - HttpHeaderExposeLanguage

Those rulesets can be referenced in each section ( additionally to normal rules ). Example:

Sections:
    website:
         rulesets:
             - security
             - seo
         rules:
            - HttpHeaderHasEtag
            - HttpHeaderResourceFound

The console error messages need some cosmetics.
It should basically still look similar, but there need to be some empty lines where appropriate and the section:Rulename hint looks somehow weird

new rule for <link rel="apple-touch-icon" sizes="60x60" href=...

https://developer.apple.com/library/content/documentation/AppleApplications/Reference/SafariWebContent/ConfiguringWebApplications/ConfiguringWebApplications.html