1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
|
---
breadcrumbs:
- - /developers
- For Developers
- - /developers/accessibility
- Accessibility for Chromium Developers
- - /developers/accessibility/testing
- Accessibility Testing
page_name: axe-core
title: Axe-Core Testing
---
TL;DR
Writing an accessibility (a11y) test with axe-core will create 1 test for each
rule and test created. This will prevent timeouts.
**A11y tests DO NOT run in DEBUG builds** (also to prevent timeouts), this
shouldn't be an issue since we're only testing web UI with axe-core.
Simple Test
// Copyright 2019 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// Polymer BrowserTest fixture and aXe-core accessibility audit.
GEN_INCLUDE(\[
'//chrome/test/data/webui/a11y/accessibility_test.js',
'//chrome/test/data/webui/polymer_browser_test_base.js',
\]);
TestFixture = class extends PolymerTest {
/\*\* @override \*/
get browsePreload() {
return 'chrome://settings/'; // URL of page to test.
}
};
AccessibilityTest.define('TestFixture', {
// Must be unique within the test fixture and cannot have spaces.
name: 'TestSuiteName',
// Optional. Configuration for axe-core. Can be used to disable a test.
axeOptions: {},
// Optional. Filter on failures. Use this for individual false positives.
violationFilter: {},
// Optional. Any setup required for all tests. This will run before each one.
setup: function() {
},
tests: {
'Test case description': function() {
// Set up an individual test case. You can open up a dialog or click
// buttons to get the test to the state that should be tested for a11y.
// This will run once per axe-core a11y rule.
// Can return a promise for async setup.
}
},
});
Examples
* [MD
Settings](https://cs.chromium.org/chromium/src/chrome/test/data/webui/settings/a11y/)
* [MD
Extensions](https://cs.chromium.org/chromium/src/chrome/test/data/webui/extensions/a11y/)
How it Works
Accessibility audits can take a while to run, so we broke each audit out into
its own test. This would be tedious to do by hand so the
AccessibilityTest.define function will create them based on some params. This
calls TEST_F for each test that's being created and uses a test fixture like all
web UI tests. In the past, a11y audit was tacked onto the end of all web UI
tests but we want to avoid timeouts since the number of tests in axe-core is
much larger than ADT, so a11y tests now have to be created manually-ish.
Each test inside the tests data member will be run as its own Mocha test.
Example:
AccessibilityTest.define('TestFixture', {
name: 'TestSuiteName',
tests: {
'Test1': function() {
// No special setup
},
'Test2': function() {
// Show the dialog before testing for a11y.
let dialog = document.querySelector('#dialog');
dialog.showModal();
},
},
});
That code snippet will create the following tests:
> TestFixture.TestSuiteName_accesskeys
> TestFixture.TestSuiteName_area_alt
> TestFixture.TestSuiteName_aria_allowed_attr
> TestFixture.TestSuiteName_aria_hidden_body
> TestFixture.TestSuiteName_aria_required_attr
> TestFixture.TestSuiteName_aria_required_children
> TestFixture.TestSuiteName_aria_required_parent
> TestFixture.TestSuiteName_aria_roles
> TestFixture.TestSuiteName_aria_valid_attr
> TestFixture.TestSuiteName_aria_valid_attr_value
> TestFixture.TestSuiteName_audio_caption
> TestFixture.TestSuiteName_blink
> TestFixture.TestSuiteName_button_name
> TestFixture.TestSuiteName_bypass
> TestFixture.TestSuiteName_checkboxgroup
> TestFixture.TestSuiteName_definition_list
> TestFixture.TestSuiteName_dlitem
> TestFixture.TestSuiteName_document_title
> TestFixture.TestSuiteName_duplicate_id
> TestFixture.TestSuiteName_empty_heading
> TestFixture.TestSuiteName_frame_title
> TestFixture.TestSuiteName_frame_title_unique
> TestFixture.TestSuiteName_heading_order
> TestFixture.TestSuiteName_hidden_content
> TestFixture.TestSuiteName_href_no_hash
> TestFixture.TestSuiteName_html_has_lang
> TestFixture.TestSuiteName_html_lang_valid
> TestFixture.TestSuiteName_image_alt
> TestFixture.TestSuiteName_image_redundant_alt
> TestFixture.TestSuiteName_input_image_alt
> TestFixture.TestSuiteName_label
> TestFixture.TestSuiteName_label_title_only
> TestFixture.TestSuiteName_layout_table
> TestFixture.TestSuiteName_link_in_text_block
> TestFixture.TestSuiteName_link_name
> TestFixture.TestSuiteName_list
> TestFixture.TestSuiteName_listitem
> TestFixture.TestSuiteName_marquee
> TestFixture.TestSuiteName_meta_refresh
> TestFixture.TestSuiteName_meta_viewport
> TestFixture.TestSuiteName_meta_viewport_large
> TestFixture.TestSuiteName_object_alt
> TestFixture.TestSuiteName_p_as_heading
> TestFixture.TestSuiteName_radiogroup
> TestFixture.TestSuiteName_region
> TestFixture.TestSuiteName_scope_attr_valid
> TestFixture.TestSuiteName_server_side_image_map
> TestFixture.TestSuiteName_tabindex
> TestFixture.TestSuiteName_table_duplicate_name
> TestFixture.TestSuiteName_table_fake_caption
> TestFixture.TestSuiteName_td_has_header
> TestFixture.TestSuiteName_td_headers_attr
> TestFixture.TestSuiteName_th_has_data_cells
> TestFixture.TestSuiteName_valid_lang
> TestFixture.TestSuiteName_video_caption
> TestFixture.TestSuiteName_video_description
Each of these tests will have a mocha body that will run both Test1 and Test2,
then run the a11y audit on the state of the page after each test setup.
[Googler only link to design doc](http://go/a11y-chrome-settings-design). (Doc
has a lot of overlap with this page, but goes more into design considerations)
Debugging
Axe-Core has an extension that can be used to debug by running the same library
that we run the tests with.
[Official
Extension](https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd)
[Googler Extension](http://go/axe-core-chrome-extension)
([source](http://go/axe-core-chrome-extension-source))
You must pass in the --extensions-on-chrome-urls flag in order to run these
extensions on chrome:// urls. This is a security risk, so only run extensions
you trust when doing this.
Running Locally
The axe-core a11y audit is part of browser_tests. Instructions for building are
the same as that target.
Note: axe-core tests are NOT part of DEBUG builds to prevent timeouts on test
bots.
|
