E.g. the "core purpose" check on the non-js / non-css spec. For a specific page, a project could implement a specific automated check for this, rather than falling back on the manual one.

Need to create some kind of extension point so projects can override the default manual check.

It would be helpful to have a summary report when all checks have been completed. What format this should take is open for discussion, but perhaps a simple table mapping pass/fail of tests against pages checked might do for starters.

Ref #44.

e.g.

3 pages checked, 0 errors found, 1 standard skipped

@joshrp had the idea to write out a HTML report per page / URL, with the filename generated from a hash of the page URL and the time. We'd need to:

• allow customisation of the path so Jenkins can save the reports as artefacts.
• make sure to print the page URL in the report so we do know what it's for
• maybe also create an 'overview' report in HTML that lets you like to the individual pages?

So it will work for a JS linter

@joshrp suggested this at yesterday's hack day. The reason is that there will be whole categories of page that should have specific checks disabled.

One idea I have in mind is something like this:

BBC::A11y.configure do
  skip_scenario "W3C", for_pages: /list/
end

That would add the skip rule for any page whose URL matches the regepx /list/.

WDYT @joshrp?

URLS: http://news.bbc.co.uk, http://www.bbc.co.uk/news/politics, http://bbc.co.uk/news/uk
These errors are all from ORB

  - Field has no label or title attribute: /html/body/div/header/div/div[1]/div[3]/form/div/input[1]
  - Field has no label or title attribute: /html/body/div/header/div/div[1]/div[3]/form/div/input[3]
  - Field has no label or title attribute: /html/body/div/header/div/div[1]/div[3]/form/div/input[4]
  - Form has no submit button: /html/body/div/header/div/div[1]/div[3]/form

URLS: http://news.bbc.co.uk, http://www.bbc.co.uk/news/politics, http://bbc.co.uk/news/uk
This is a 1px x 1px graphic either from stats tracking or from advertising

  - Image has no alt attribute (src="http://b.scorecardresearch.com/p?c1=2&c2=17986528&cv=2.0&cj=1")

URLS: http://bbc.co.uk/mundo, http://bbc.co.uk/cymrufyw
These errors are all from ORB

  - Field has no label or title attribute: /html/body/div/header/div/div[1]/div[3]/form/div/input[1]
  - Field has no label or title attribute: /html/body/div/header/div/div[1]/div[3]/form/div/input[2]
  - Field has no label or title attribute: /html/body/div/header/div/div[1]/div[3]/form/div/input[4]
  - Field has no label or title attribute: /html/body/div/header/div/div[1]/div[3]/form/div/input[5]
  - Form has no submit button: /html/body/div/header/div/div[1]/div[3]/form

We can skip standards, but it would be handy to exclude areas of the page as well, so that we can leave those standards enabled for the rest of the page.

From #12:

URL: http://m.live.bbc.co.uk/news ( I have used 'm.live' instead of 'm.' as it forces the mobile site to load for the tests. 'm.' redirects to 'www' on desktop browser widths)
This looks more like a genuine accessibility issue on the page. @IanPouncey could you please confirm.
(On the page in 2nd column the Most - Read tab - 'most-popular__header__tabs--read open' has tabindex="0").

Correctly use tabindex attributes

Scenario: Check all tabindex values ✗
expected [#<Capybara::Element tag="a">].empty? to return true, got false
/Library/Ruby/Gems/2.0.0/gems/rspec-expectations-3.2.0/lib/rspec/expectations/fail_with.rb:29:in fail_with' /Library/Ruby/Gems/2.0.0/gems/rspec-expectations-3.2.0/lib/rspec/expectations/handler.rb:40:inhandle_failure'
/Library/Ruby/Gems/2.0.0/gems/rspec-expectations-3.2.0/lib/rspec/expectations/handler.rb:50:in block in handle_matcher' /Library/Ruby/Gems/2.0.0/gems/rspec-expectations-3.2.0/lib/rspec/expectations/handler.rb:27:inwith_matcher'
/Library/Ruby/Gems/2.0.0/gems/rspec-expectations-3.2.0/lib/rspec/expectations/handler.rb:48:in handle_matcher' /Library/Ruby/Gems/2.0.0/gems/rspec-expectations-3.2.0/lib/rspec/expectations/expectation_target.rb:54:into'
/Library/Ruby/Gems/2.0.0/gems/bbc-a11y-0.0.4/lib/bbc/a11y/cucumber_support/page.rb:62:in must_not_have_any_positive_tabindex_values' /Library/Ruby/Gems/2.0.0/gems/bbc-a11y-0.0.4/features/step_definitions/page_steps.rb:45:in/^there should be no elements with a tabindex attribte of 0 or greater$/'

.a11y.rb is considered a system file, and therefore hidden by default on Mac. It would be nice to change this.

Split off from #11.

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/14_colour-meaning.md

Split off from #11.

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/19_form-interactions.md

Two reasons:

1. we have the potential to run in the browser
2. we could lean on tools like http://pa11y.org/ (based on https://github.com/squizlabs/HTML_CodeSniffer)

The error message when the W3C validator fails is too messy. Needs to just show a summary of the number of errors, and where to go to get more information.

Split off from #11.

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/18_focus-styles.md

Split off from #11.

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/17_control-styles.md

Right now it only respects the configuration file

@IanPouncey has reported:

I've checked out the latest bbc-accessibility-standards-checker, I'm currently experiencing a problem with it though. The tests run, but fails on connecting to the W3C:

When I submit the page to the W3C Markup Validation Service # features/step_definitions/steps.rb:1
     Operation timed out - connect(2) for "validator.w3.org" port 80 (Errno::ETIMEDOUT)
     ./features/support/world.rb:19:in `validate'
     ./features/step_definitions/steps.rb:2:in `/^I submit the page to the W3C Markup Validation Service$/'
     features/02_validation.feature:35:in `When I submit the page to the W3C Markup Validation Service'

I'm able to curl www.bbc.co.uk from the same terminal session, but I'm going to try it at home off the Reith network to make sure that's not the issue.

It would be helpful to have the examples documented, explaining what they show and how a user could replicate them.

if you run a11y bbc.co.uk you get a very confusing error message.

In implementing the checks for Form Labels (#27) I had a question about the title attribute rule. In the examples under "Fail" we see:

<input type="text" name="name" title="Name" />
<input type="text" name="email" title="Email" />

Yet these both have a title attribute. I'm wondering why they're supposed to fail? Is the rule that the title attribute is only OK if there's only one field in the form?

Split off from #11

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/11_visible-on-focus.md

At the moment there is a chicken-egg problem when creating a .a11y.rb file, as there is nowhere explaining the different things that can be configured within one.

Good first pass, and will let it run in any browser.

The process exit status should reflect whether all the tests passed in order for CI servers to understand it.

At the moment you can't run it in the same browser because of a clash on the _ function.

After following your readme instructions we're having issues running these accessibility tests.

The tests seem to run after setting up the Gemfile, installing dependencies and creating the .a11y.rb config file, but each test isn't able to complete the "I visit the page" step, resulting in a test run output like:

Correctly use headings
----------------------

  - Check headings... ?

      When I visit the page
      Then there must be exactly one h1 element
      And each heading must be followed by content or a heading of one level deeper (h2-h6)

    Undefined step: "I visit the page"


Correctly use `title` attributes
--------------------------------

  - Check the elements with title attributes... ?

      When I visit the page
      Then there must be no elements with a title attribute whose content is repeated within the element
      And any form fields with associated labels do not have a title attribute

    Undefined step: "I visit the page"


Correctly use `tabindex` attributes
-----------------------------------

  - Check all tabindex values... ?

      When I visit the page
      Then there should be no elements with a tabindex attribte of 0 or greater

    Undefined step: "I visit the page"

Can you shed any light on this? A couple members of our team have tried this with the same results.

The console output is coming together, but what it lacks is a summary at the bottom showing:

• a list of all the pages tested, with a summary of pass / fail / pending / skipped
• an overall number for passed / failed / pending skipped

This could be done by making the webdriver configurable, or just switching over to phantom full stop, or both.

From #12

URL: http://www.bbc.co.uk/mundo & http://m.live.bbc.co.uk/news

1. • Scenario: Check headings ✗

Unable to find xpath "/html/head/meta/meta/meta/meta/meta/meta/link/meta/link/link/link/link/link/meta/meta/meta/meta/meta/meta/meta/meta/link/link/link/link/body/div/div[3]/div/div/section/div/div/h2"
/Users/sarafs02/.gem/ruby/2.0.0/gems/capybara-2.4.4/lib/capybara/node/finders.rb:41:in block in find' /Users/sarafs02/.gem/ruby/2.0.0/gems/capybara-2.4.4/lib/capybara/node/base.rb:84:insynchronize'
/Users/sarafs02/.gem/ruby/2.0.0/gems/capybara-2.4.4/lib/capybara/node/finders.rb:30:in find' /Users/sarafs02/.gem/ruby/2.0.0/gems/capybara-2.4.4/lib/capybara/session.rb:676:inblock (2 levels) in class:Session'
/Library/Ruby/Gems/2.0.0/gems/bbc-a11y-0.0.4/lib/bbc/a11y/cucumber_support/heading_hierarchy.rb:53:in block in heading_elements' /Library/Ruby/Gems/2.0.0/gems/bbc-a11y-0.0.4/lib/bbc/a11y/cucumber_support/heading_hierarchy.rb:52:inmap'
/Library/Ruby/Gems/2.0.0/gems/bbc-a11y-0.0.4/lib/bbc/a11y/cucumber_support/heading_hierarchy.rb:52:in heading_elements' /Library/Ruby/Gems/2.0.0/gems/bbc-a11y-0.0.4/lib/bbc/a11y/cucumber_support/heading_hierarchy.rb:45:inheadings'
/Library/Ruby/Gems/2.0.0/gems/bbc-a11y-0.0.4/lib/bbc/a11y/cucumber_support/heading_hierarchy.rb:30:in to_s' /Library/Ruby/Gems/2.0.0/gems/bbc-a11y-0.0.4/features/step_definitions/page_steps.rb:32:in/^each heading must be followed by content or a heading of one level deeper (h2-h6)$/'

If you use skip_scenario "WIP" it no longer works. You need to use skip_scenario /WIP/ instead.

Split off from #11.

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/22_resize-zoom.md

I had a little epiphany the other day as I was working on the unit tests for the form labels rule

I think we've been using the testing tools the wrong way around. I'll try and explain.

The business rules that we need to use examples to illustrate, are the standards themselves. So what we need are features with several scenarios that give examples of good and bad content. We can use these features to test the tool itself using Cucumber, but the tool does not necessarily need to then use Cucumber to actually test people's pages. In fact RSpec would be a much better fit, I think.

So for example, rather than having a scenario like this:

  Scenario: Validate use of labels
    When I visit the page
    Then the following form elements must have a title attribute or associated label tag:
      | select                     |
      | textarea                   |
      | input[type=checkbox]       |
      | input[type=color]          |
      | input[type=date]           |
      | input[type=datetime]       |
      | input[type=datetime-local] |
      | input[type=email]          |
      | input[type=file]           |
      | input[type=month]          |
      | input[type=number]         |
      | input[type=password]       |
      | input[type=radio]          |
      | input[type=range]          |
      | input[type=search]         |
      | input[type=tel]            |
      | input[type=text]           |
      | input[type=time]           |
      | input[type=url]            |
      | input[type=week]           |

We could have scenarios like this instead:

Scenario: Single-field form label associated via `for` attribute
  Given a page with this html in the body:
    """
    <label for="search">Search the BBC</label>
    <input type="text" id="search" name="q" />
    """
  When the page is checked for compliance with the accessibility standards
  Then the form labels check should pass

Scenario: Field with no label or title attribute
  Given a page with this html in the body:
    """
    <input type="text" name="q" value="Search the BBC" />
    """
  When the page is checked for compliance with the accessibility standards
  Then the form labels check should fail with:
    """
    'input' tag at /html/body/input has no title attribute or associated label
    """

Split off from #11.

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/16_table.md

Split off from #11.

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/20_image-alt.md

Rather than passing just a URL on the command-line, it would be more useful to pass a path to a configuration file.

Things to configure:

• page url
• scenarios to mask / mute / ignore
• page width
• webdriver?

Could do this with a json / yaml file, or maybe a Ruby script calling a configuration DSL. Ruby script would give more options for working with extension points.

Let's play around with how we can create an interactive UI for a11y running in a browser.

We could use this to:

• run checks on an individual page
• run checks on several pages
• allow you to pull out a JSON config that you can use for your CI build
• diagnose issues by drilling down to individual elements on the page that are causing problems.

It might be a standard chrome plugin-in, or a developer tools plug-in, or a combination of both.

Split off from #11

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/13_colour-contrast.md

@IanPouncey can you do this for me when you get a chance? Once I have these as raw material I can start going through them and turning them into Gherkin.

As originally reported by @sarafs in #11.

1. features/02_validation.feature View
   Scenario: Submit to W3C validation Service
   See http://validator.w3.org
   When I submit the page to the W3C Markup Validation Service
   Then there should be no errors
   Note: Any site invariably always fails for the W3C validator

2. Heading hierarchy tests are failing -
   Then(/^each heading must be followed by content or a heading of one level deeper (h2-h6)$/) do
   puts "Heading hierarchy:"
   puts page.heading_hierarchy.to_s
   page.must_have_correct_heading_hierarchy

Note: from this spec heading_hierarchy_spec.rb the HeadingHierarchy module returns an invalid Xpath error.

3. def must_have_title_that_contains_h1_text
   expect(browser.title).to include(browser.find('h1').text)
   Note: Fails due to a miss-match in Page Title and h1 text

4. Scenario: View the page with JavaScript and CSS disabled
   When I view the page with JavaScript and CSS disabled
   Then all core content is available and functional
   Note: Manual test

We might use selenium, or phantom as the bridge from Ruby to JS.

Keep it thin, push the responsibility for reading config etc into JavaScript.

We can abandon the Ruby checks and linter.

Split off from #11.

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/15_focusable-controls.md

I need to clarify some of the rules around valid header hierarchies.

It is every OK for a h1 to appear after a h2, h3, h4 etc?

e.g. here's the header hierarchy from http://bbc.co.uk:

  h2
  h2
  h2
  h2
    h3
    h3
  h2
    h3
      h4
    h3
      h4
    h3
      h4
    h3
    h3
      h4
    h3
    h3
      h4
      h4
    h3
    h3
      h4
    h3
    h3
    h3
      h4
    h3
      h4
    h3
      h4
    h3
      h4
    h3
    h3
      h4
    h3
      h4
    h3
      h4
    h3
      h4
    h3
      h4
    h3
      h4
    h3
      h4
    h3
    h3
      h4
    h3
    h3
      h4
    h3
      h4
    h3
      h4
    h3
    h3
      h4
    h3
    h3
h1
  h2
    h3
    h3

Right now, my tests pass because there aren't any jumps e.g. h1..h3 but I wonder if what we have here is OK?

@IanPouncey what do you say?

Also, I'm not sure what you meant by this note

To do:

• Form labels
• Visible on focus
• Colour contrast
• Colour meaning
• Focusable controls
• Table
• Control styles
• Focus styles
• Form interactions
• Image alt
• Min font sizes
• Resize zoon

If the team have extra accessibility tests they'd like to run, it would be great if there were a way for them to configure a11y to pull in features from another location, and have those form part of the overall spec to be run.

Split off from #11.

See https://github.com/cucumber-ltd/bbc-a11y/blob/master/standards/21_min-font-sizes.md

e.g.

bundle exec ./bin/a11y http://bbc.co.uk/iplayer -- features/08_title_attribute.feature:60
...
Then there must be no elements with a title attribute whose content is repeated within the element # features/step_definitions/page_steps.rb:38
      expected `[#<Capybara::Element tag="a">, #<Capybara::Element tag="a">, #<Capybara::Element tag="a">, #<Capybara::Element tag="a">, #<Capybara::Element tag="a">, #<Capybara::Element tag="a">].empty?` to return true, got false (RSpec::Expectations::ExpectationNotMetError)
      ./lib/bbc/a11y/page.rb:51:in `must_have_no_elements_with_title_attribute_content_repeated_within'
      ./features/step_definitions/page_steps.rb:39:in `/^there must be no elements with a title attribute whose content is repeated within the element$/'
      features/08_title_attribute.feature:70:in `Then there must be no elements with a title attribute whose content is repeated within the element'

All you get from this is that you know there's a problem, but no help to identify where the problem is within the document.

Ideally, we'd highlight the bad nodes in the HTML doc and take a screenshot. We could at least do with better helping the user to identify which ones are bad, maybe by showing the HTML source.

Need further research / experimentation.