1 | <html> |
---|
2 | <head> |
---|
3 | <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> |
---|
4 | <title>SimpleTest for PHP test runner and display documentation</title> |
---|
5 | <link rel="stylesheet" type="text/css" href="docs.css" title="Styles"> |
---|
6 | </head> |
---|
7 | <body> |
---|
8 | <div class="menu_back"><div class="menu"> |
---|
9 | <a href="index.html">SimpleTest</a> |
---|
10 | | |
---|
11 | <a href="overview.html">Overview</a> |
---|
12 | | |
---|
13 | <a href="unit_test_documentation.html">Unit tester</a> |
---|
14 | | |
---|
15 | <a href="group_test_documentation.html">Group tests</a> |
---|
16 | | |
---|
17 | <a href="mock_objects_documentation.html">Mock objects</a> |
---|
18 | | |
---|
19 | <a href="partial_mocks_documentation.html">Partial mocks</a> |
---|
20 | | |
---|
21 | <span class="chosen">Reporting</span> |
---|
22 | | |
---|
23 | <a href="expectation_documentation.html">Expectations</a> |
---|
24 | | |
---|
25 | <a href="web_tester_documentation.html">Web tester</a> |
---|
26 | | |
---|
27 | <a href="form_testing_documentation.html">Testing forms</a> |
---|
28 | | |
---|
29 | <a href="authentication_documentation.html">Authentication</a> |
---|
30 | | |
---|
31 | <a href="browser_documentation.html">Scriptable browser</a> |
---|
32 | </div></div> |
---|
33 | <h1>Test reporter documentation</h1> |
---|
34 | This page... |
---|
35 | <ul> |
---|
36 | <li> |
---|
37 | Displaying <a href="#html">results in HTML</a> |
---|
38 | </li> |
---|
39 | <li> |
---|
40 | Displaying and <a href="#other">reporting results</a> |
---|
41 | in other formats |
---|
42 | </li> |
---|
43 | <li> |
---|
44 | Using <a href="#cli">SimpleTest from the command line</a> |
---|
45 | </li> |
---|
46 | <li> |
---|
47 | Using <a href="#xml">Using XML</a> for remote testing |
---|
48 | </li> |
---|
49 | </ul> |
---|
50 | <div class="content"> |
---|
51 | |
---|
52 | <p> |
---|
53 | SimpleTest pretty much follows the MVC pattern |
---|
54 | (Model-View-Controller). |
---|
55 | The reporter classes are the view and the model is your |
---|
56 | test cases and their hiearchy. |
---|
57 | The controller is mostly hidden from the user of |
---|
58 | SimpleTest unless you want to change how the test cases |
---|
59 | are actually run, in which case it is possible to |
---|
60 | override the runner objects from within the test case. |
---|
61 | As usual with MVC, the controller is mostly undefined |
---|
62 | and there are other places to control the test run. |
---|
63 | </p> |
---|
64 | |
---|
65 | <p><a class="target" name="html"><h2>Reporting results in HTML</h2></a></p> |
---|
66 | <p> |
---|
67 | The default test display is minimal in the extreme. |
---|
68 | It reports success and failure with the conventional red and |
---|
69 | green bars and shows a breadcrumb trail of test groups |
---|
70 | for every failed assertion. |
---|
71 | Here's a fail... |
---|
72 | <div class="demo"> |
---|
73 | <h1>File test</h1> |
---|
74 | <span class="fail">Fail</span>: createnewfile->True assertion failed.<br> |
---|
75 | <div style="padding: 8px; margin-top: 1em; background-color: red; color: white;">1/1 test cases complete. |
---|
76 | <strong>0</strong> passes, <strong>1</strong> fails and <strong>0</strong> exceptions.</div> |
---|
77 | </div> |
---|
78 | And here all tests passed... |
---|
79 | <div class="demo"> |
---|
80 | <h1>File test</h1> |
---|
81 | <div style="padding: 8px; margin-top: 1em; background-color: green; color: white;">1/1 test cases complete. |
---|
82 | <strong>1</strong> passes, <strong>0</strong> fails and <strong>0</strong> exceptions.</div> |
---|
83 | </div> |
---|
84 | The good news is that there are several points in the display |
---|
85 | hiearchy for subclassing. |
---|
86 | </p> |
---|
87 | <p> |
---|
88 | For web page based displays there is the |
---|
89 | <span class="new_code">HtmlReporter</span> class with the following |
---|
90 | signature... |
---|
91 | <pre> |
---|
92 | class HtmlReporter extends SimpleReporter { |
---|
93 | public HtmlReporter($encoding) { ... } |
---|
94 | public makeDry(boolean $is_dry) { ... } |
---|
95 | public void paintHeader(string $test_name) { ... } |
---|
96 | public void sendNoCacheHeaders() { ... } |
---|
97 | public void paintFooter(string $test_name) { ... } |
---|
98 | public void paintGroupStart(string $test_name, integer $size) { ... } |
---|
99 | public void paintGroupEnd(string $test_name) { ... } |
---|
100 | public void paintCaseStart(string $test_name) { ... } |
---|
101 | public void paintCaseEnd(string $test_name) { ... } |
---|
102 | public void paintMethodStart(string $test_name) { ... } |
---|
103 | public void paintMethodEnd(string $test_name) { ... } |
---|
104 | public void paintFail(string $message) { ... } |
---|
105 | public void paintPass(string $message) { ... } |
---|
106 | public void paintError(string $message) { ... } |
---|
107 | public void paintException(string $message) { ... } |
---|
108 | public void paintMessage(string $message) { ... } |
---|
109 | public void paintFormattedMessage(string $message) { ... } |
---|
110 | protected string _getCss() { ... } |
---|
111 | public array getTestList() { ... } |
---|
112 | public integer getPassCount() { ... } |
---|
113 | public integer getFailCount() { ... } |
---|
114 | public integer getExceptionCount() { ... } |
---|
115 | public integer getTestCaseCount() { ... } |
---|
116 | public integer getTestCaseProgress() { ... } |
---|
117 | } |
---|
118 | </pre> |
---|
119 | Here is what some of these methods mean. First the display methods |
---|
120 | that you will probably want to override... |
---|
121 | <ul class="api"> |
---|
122 | <li> |
---|
123 | <span class="new_code">HtmlReporter(string $encoding)</span><br> |
---|
124 | is the constructor. |
---|
125 | Note that the unit test sets up the link to the display |
---|
126 | rather than the other way around. |
---|
127 | The display is a mostly passive receiver of test events. |
---|
128 | This allows easy adaption of the display for other test |
---|
129 | systems beside unit tests, such as monitoring servers. |
---|
130 | The encoding is the character encoding you wish to |
---|
131 | display the test output in. |
---|
132 | In order to correctly render debug output when |
---|
133 | using the web tester, this should match the encoding |
---|
134 | of the site you are trying to test. |
---|
135 | The available character set strings are described in |
---|
136 | the PHP <a href="http://www.php.net/manual/en/function.htmlentities.php">html_entities()</a> |
---|
137 | function. |
---|
138 | </li> |
---|
139 | <li> |
---|
140 | <span class="new_code">void paintHeader(string $test_name)</span><br> |
---|
141 | is called once at the very start of the test when the first |
---|
142 | start event arrives. |
---|
143 | The first start event is usually delivered by the top level group |
---|
144 | test and so this is where <span class="new_code">$test_name</span> |
---|
145 | comes from. |
---|
146 | It paints the page titles, CSS, body tag, etc. |
---|
147 | It returns nothing (<span class="new_code">void</span>). |
---|
148 | </li> |
---|
149 | <li> |
---|
150 | <span class="new_code">void paintFooter(string $test_name)</span><br> |
---|
151 | Called at the very end of the test to close any tags opened |
---|
152 | by the page header. |
---|
153 | By default it also displays the red/green bar and the final |
---|
154 | count of results. |
---|
155 | Actually the end of the test happens when a test end event |
---|
156 | comes in with the same name as the one that started it all |
---|
157 | at the same level. |
---|
158 | The tests nest you see. |
---|
159 | Closing the last test finishes the display. |
---|
160 | </li> |
---|
161 | <li> |
---|
162 | <span class="new_code">void paintMethodStart(string $test_name)</span><br> |
---|
163 | is called at the start of each test method. |
---|
164 | The name normally comes from method name. |
---|
165 | The other test start events behave the same way except |
---|
166 | that the group test one tells the reporter how large |
---|
167 | it is in number of held test cases. |
---|
168 | This is so that the reporter can display a progress bar |
---|
169 | as the runner churns through the test cases. |
---|
170 | </li> |
---|
171 | <li> |
---|
172 | <span class="new_code">void paintMethodEnd(string $test_name)</span><br> |
---|
173 | backs out of the test started with the same name. |
---|
174 | </li> |
---|
175 | <li> |
---|
176 | <span class="new_code">void paintFail(string $message)</span><br> |
---|
177 | paints a failure. |
---|
178 | By default it just displays the word fail, a breadcrumbs trail |
---|
179 | showing the current test nesting and the message issued by |
---|
180 | the assertion. |
---|
181 | </li> |
---|
182 | <li> |
---|
183 | <span class="new_code">void paintPass(string $message)</span><br> |
---|
184 | by default does nothing. |
---|
185 | </li> |
---|
186 | <li> |
---|
187 | <span class="new_code">string _getCss()</span><br> |
---|
188 | Returns the CSS styles as a string for the page header |
---|
189 | method. |
---|
190 | Additional styles have to be appended here if you are |
---|
191 | not overriding the page header. |
---|
192 | You will want to use this method in an overriden page header |
---|
193 | if you want to include the original CSS. |
---|
194 | </li> |
---|
195 | </ul> |
---|
196 | There are also some accessors to get information on the current |
---|
197 | state of the test suite. |
---|
198 | Use these to enrich the display... |
---|
199 | <ul class="api"> |
---|
200 | <li> |
---|
201 | <span class="new_code">array getTestList()</span><br> |
---|
202 | is the first convenience method for subclasses. |
---|
203 | Lists the current nesting of the tests as a list |
---|
204 | of test names. |
---|
205 | The first, most deeply nested test, is first in the |
---|
206 | list and the current test method will be last. |
---|
207 | </li> |
---|
208 | <li> |
---|
209 | <span class="new_code">integer getPassCount()</span><br> |
---|
210 | returns the number of passes chalked up so far. |
---|
211 | Needed for the display at the end. |
---|
212 | </li> |
---|
213 | <li> |
---|
214 | <span class="new_code">integer getFailCount()</span><br> |
---|
215 | is likewise the number of fails so far. |
---|
216 | </li> |
---|
217 | <li> |
---|
218 | <span class="new_code">integer getExceptionCount()</span><br> |
---|
219 | is likewise the number of errors so far. |
---|
220 | </li> |
---|
221 | <li> |
---|
222 | <span class="new_code">integer getTestCaseCount()</span><br> |
---|
223 | is the total number of test cases in the test run. |
---|
224 | This includes the grouping tests themselves. |
---|
225 | </li> |
---|
226 | <li> |
---|
227 | <span class="new_code">integer getTestCaseProgress()</span><br> |
---|
228 | is the number of test cases completed so far. |
---|
229 | </li> |
---|
230 | </ul> |
---|
231 | One simple modification is to get the HtmlReporter to display |
---|
232 | the passes as well as the failures and errors... |
---|
233 | <pre> |
---|
234 | <strong>class ShowPasses extends HtmlReporter { |
---|
235 | |
---|
236 | function paintPass($message) { |
---|
237 | parent::paintPass($message); |
---|
238 | print "&<span class=\"pass\">Pass</span>: "; |
---|
239 | $breadcrumb = $this->getTestList(); |
---|
240 | array_shift($breadcrumb); |
---|
241 | print implode("-&gt;", $breadcrumb); |
---|
242 | print "-&gt;$message<br />\n"; |
---|
243 | } |
---|
244 | |
---|
245 | function _getCss() { |
---|
246 | return parent::_getCss() . ' .pass { color: green; }'; |
---|
247 | } |
---|
248 | }</strong> |
---|
249 | </pre> |
---|
250 | </p> |
---|
251 | <p> |
---|
252 | One method that was glossed over was the <span class="new_code">makeDry()</span> |
---|
253 | method. |
---|
254 | If you run this method, with no parameters, on the reporter |
---|
255 | before the test suite is run no actual test methods |
---|
256 | will be called. |
---|
257 | You will still get the events of entering and leaving the |
---|
258 | test methods and test cases, but no passes or failures etc, |
---|
259 | because the test code will not actually be executed. |
---|
260 | </p> |
---|
261 | <p> |
---|
262 | The reason for this is to allow for more sophistcated |
---|
263 | GUI displays that allow the selection of individual test |
---|
264 | cases. |
---|
265 | In order to build a list of possible tests they need a |
---|
266 | report on the test structure for drawing, say a tree view |
---|
267 | of the test suite. |
---|
268 | With a reporter set to dry run that just sends drawing events |
---|
269 | this is easily accomplished. |
---|
270 | </p> |
---|
271 | |
---|
272 | <p><a class="target" name="other"><h2>Extending the reporter</h2></a></p> |
---|
273 | <p> |
---|
274 | Rather than simply modifying the existing display, you might want to |
---|
275 | produce a whole new HTML look, or even generate text or XML. |
---|
276 | Rather than override every method in |
---|
277 | <span class="new_code">HtmlReporter</span> we can take one |
---|
278 | step up the class hiearchy to <span class="new_code">SimpleReporter</span> |
---|
279 | in the <em>simple_test.php</em> source file. |
---|
280 | </p> |
---|
281 | <p> |
---|
282 | A do nothing display, a blank canvas for your own creation, would |
---|
283 | be... |
---|
284 | <pre> |
---|
285 | <strong>require_once('simpletest/simple_test.php');</strong> |
---|
286 | |
---|
287 | class MyDisplay extends SimpleReporter {<strong> |
---|
288 | </strong> |
---|
289 | function paintHeader($test_name) { |
---|
290 | } |
---|
291 | |
---|
292 | function paintFooter($test_name) { |
---|
293 | } |
---|
294 | |
---|
295 | function paintStart($test_name, $size) {<strong> |
---|
296 | parent::paintStart($test_name, $size);</strong> |
---|
297 | } |
---|
298 | |
---|
299 | function paintEnd($test_name, $size) {<strong> |
---|
300 | parent::paintEnd($test_name, $size);</strong> |
---|
301 | } |
---|
302 | |
---|
303 | function paintPass($message) {<strong> |
---|
304 | parent::paintPass($message);</strong> |
---|
305 | } |
---|
306 | |
---|
307 | function paintFail($message) {<strong> |
---|
308 | parent::paintFail($message);</strong> |
---|
309 | } |
---|
310 | } |
---|
311 | </pre> |
---|
312 | No output would come from this class until you add it. |
---|
313 | </p> |
---|
314 | |
---|
315 | <p><a class="target" name="cli"><h2>The command line reporter</h2></a></p> |
---|
316 | <p> |
---|
317 | SimpleTest also ships with a minimal command line reporter. |
---|
318 | The interface mimics JUnit to some extent, but paints the |
---|
319 | failure messages as they arrive. |
---|
320 | To use the command line reporter simply substitute it |
---|
321 | for the HTML version... |
---|
322 | <pre> |
---|
323 | <?php |
---|
324 | require_once('simpletest/unit_tester.php'); |
---|
325 | require_once('simpletest/reporter.php'); |
---|
326 | |
---|
327 | $test = &new TestSuite('File test'); |
---|
328 | $test->addTestFile('tests/file_test.php'); |
---|
329 | $test->run(<strong>new TextReporter()</strong>); |
---|
330 | ?> |
---|
331 | </pre> |
---|
332 | Then invoke the test suite from the command line... |
---|
333 | <pre class="shell"> |
---|
334 | php file_test.php |
---|
335 | </pre> |
---|
336 | You will need the command line version of PHP installed |
---|
337 | of course. |
---|
338 | A passing test suite looks like this... |
---|
339 | <pre class="shell"> |
---|
340 | File test |
---|
341 | OK |
---|
342 | Test cases run: 1/1, Failures: 0, Exceptions: 0 |
---|
343 | </pre> |
---|
344 | A failure triggers a display like this... |
---|
345 | <pre class="shell"> |
---|
346 | File test |
---|
347 | 1) True assertion failed. |
---|
348 | in createnewfile |
---|
349 | FAILURES!!! |
---|
350 | Test cases run: 1/1, Failures: 1, Exceptions: 0 |
---|
351 | </pre> |
---|
352 | </p> |
---|
353 | <p> |
---|
354 | One of the main reasons for using a command line driven |
---|
355 | test suite is of using the tester as part of some automated |
---|
356 | process. |
---|
357 | To function properly in shell scripts the test script should |
---|
358 | return a non-zero exit code on failure. |
---|
359 | If a test suite fails the value <span class="new_code">false</span> |
---|
360 | is returned from the <span class="new_code">SimpleTest::run()</span> |
---|
361 | method. |
---|
362 | We can use that result to exit the script with the desired return |
---|
363 | code... |
---|
364 | <pre> |
---|
365 | <?php |
---|
366 | require_once('simpletest/unit_tester.php'); |
---|
367 | require_once('simpletest/reporter.php'); |
---|
368 | |
---|
369 | $test = &new TestSuite('File test'); |
---|
370 | $test->addTestFile('tests/file_test.php'); |
---|
371 | <strong>exit ($test->run(new TextReporter()) ? 0 : 1);</strong> |
---|
372 | ?> |
---|
373 | </pre> |
---|
374 | Of course we don't really want to create two test scripts, |
---|
375 | a command line one and a web browser one, for each test suite. |
---|
376 | The command line reporter includes a method to sniff out the |
---|
377 | run time environment... |
---|
378 | <pre> |
---|
379 | <?php |
---|
380 | require_once('simpletest/unit_tester.php'); |
---|
381 | require_once('simpletest/reporter.php'); |
---|
382 | |
---|
383 | $test = &new TestSuite('File test'); |
---|
384 | $test->addTestFile('tests/file_test.php'); |
---|
385 | <strong>if (TextReporter::inCli()) {</strong> |
---|
386 | exit ($test->run(new TextReporter()) ? 0 : 1); |
---|
387 | <strong>}</strong> |
---|
388 | $test->run(new HtmlReporter()); |
---|
389 | ?> |
---|
390 | </pre> |
---|
391 | This is the form used within SimpleTest itself. |
---|
392 | </p> |
---|
393 | |
---|
394 | <p><a class="target" name="xml"><h2>Remote testing</h2></a></p> |
---|
395 | <p> |
---|
396 | SimpleTest ships with an <span class="new_code">XmlReporter</span> class |
---|
397 | used for internal communication. |
---|
398 | When run the output looks like... |
---|
399 | <pre class="shell"> |
---|
400 | <?xml version="1.0"?> |
---|
401 | <run> |
---|
402 | <group size="4"> |
---|
403 | <name>Remote tests</name> |
---|
404 | <group size="4"> |
---|
405 | <name>Visual test with 48 passes, 48 fails and 4 exceptions</name> |
---|
406 | <case> |
---|
407 | <name>testofunittestcaseoutput</name> |
---|
408 | <test> |
---|
409 | <name>testofresults</name> |
---|
410 | <pass>This assertion passed</pass> |
---|
411 | <fail>This assertion failed</fail> |
---|
412 | </test> |
---|
413 | <test> |
---|
414 | ... |
---|
415 | </test> |
---|
416 | </case> |
---|
417 | </group> |
---|
418 | </group> |
---|
419 | </run> |
---|
420 | </pre> |
---|
421 | You can make use of this format with the parser |
---|
422 | supplied as part of SimpleTest itself. |
---|
423 | This is called <span class="new_code">SimpleTestXmlParser</span> and |
---|
424 | resides in <em>xml.php</em> within the SimpleTest package... |
---|
425 | <pre> |
---|
426 | <?php |
---|
427 | require_once('simpletest/xml.php'); |
---|
428 | |
---|
429 | ... |
---|
430 | $parser = &new SimpleTestXmlParser(new HtmlReporter()); |
---|
431 | $parser->parse($test_output); |
---|
432 | ?> |
---|
433 | </pre> |
---|
434 | The <span class="new_code">$test_output</span> should be the XML format |
---|
435 | from the XML reporter, and could come from say a command |
---|
436 | line run of a test case. |
---|
437 | The parser sends events to the reporter just like any |
---|
438 | other test run. |
---|
439 | There are some odd occasions where this is actually useful. |
---|
440 | </p> |
---|
441 | <p> |
---|
442 | A problem with large test suites is thet they can exhaust |
---|
443 | the default 8Mb memory limit on a PHP process. |
---|
444 | By having the test groups output in XML and run in |
---|
445 | separate processes, the output can be reparsed to |
---|
446 | aggregate the results into a much smaller footprint top level |
---|
447 | test. |
---|
448 | </p> |
---|
449 | <p> |
---|
450 | Because the XML output can come from anywhere, this opens |
---|
451 | up the possibility of aggregating test runs from remote |
---|
452 | servers. |
---|
453 | A test case already exists to do this within the SimpleTest |
---|
454 | framework, but it is currently experimental... |
---|
455 | <pre> |
---|
456 | <?php |
---|
457 | <strong>require_once('../remote.php');</strong> |
---|
458 | require_once('../reporter.php'); |
---|
459 | |
---|
460 | $test_url = ...; |
---|
461 | $dry_url = ...; |
---|
462 | |
---|
463 | $test = &new TestSuite('Remote tests'); |
---|
464 | $test->addTestCase(<strong>new RemoteTestCase($test_url, $dry_url)</strong>); |
---|
465 | $test->run(new HtmlReporter()); |
---|
466 | ?> |
---|
467 | </pre> |
---|
468 | The <span class="new_code">RemoteTestCase</span> takes the actual location |
---|
469 | of the test runner, basically a web page in XML format. |
---|
470 | It also takes the URL of a reporter set to do a dry run. |
---|
471 | This is so that progress can be reported upward correctly. |
---|
472 | The <span class="new_code">RemoteTestCase</span> can be added to test suites |
---|
473 | just like any other group test. |
---|
474 | </p> |
---|
475 | |
---|
476 | </div> |
---|
477 | References and related information... |
---|
478 | <ul> |
---|
479 | <li> |
---|
480 | SimpleTest project page on <a href="http://sourceforge.net/projects/simpletest/">SourceForge</a>. |
---|
481 | </li> |
---|
482 | <li> |
---|
483 | SimpleTest download page on <a href="http://www.lastcraft.com/simple_test.php">LastCraft</a>. |
---|
484 | </li> |
---|
485 | <li> |
---|
486 | The <a href="http://simpletest.org/api/">developer's API for SimpleTest</a> |
---|
487 | gives full detail on the classes and assertions available. |
---|
488 | </li> |
---|
489 | </ul> |
---|
490 | <div class="menu_back"><div class="menu"> |
---|
491 | <a href="index.html">SimpleTest</a> |
---|
492 | | |
---|
493 | <a href="overview.html">Overview</a> |
---|
494 | | |
---|
495 | <a href="unit_test_documentation.html">Unit tester</a> |
---|
496 | | |
---|
497 | <a href="group_test_documentation.html">Group tests</a> |
---|
498 | | |
---|
499 | <a href="mock_objects_documentation.html">Mock objects</a> |
---|
500 | | |
---|
501 | <a href="partial_mocks_documentation.html">Partial mocks</a> |
---|
502 | | |
---|
503 | <span class="chosen">Reporting</span> |
---|
504 | | |
---|
505 | <a href="expectation_documentation.html">Expectations</a> |
---|
506 | | |
---|
507 | <a href="web_tester_documentation.html">Web tester</a> |
---|
508 | | |
---|
509 | <a href="form_testing_documentation.html">Testing forms</a> |
---|
510 | | |
---|
511 | <a href="authentication_documentation.html">Authentication</a> |
---|
512 | | |
---|
513 | <a href="browser_documentation.html">Scriptable browser</a> |
---|
514 | </div></div> |
---|
515 | <div class="copyright"> |
---|
516 | Copyright<br>Marcus Baker 2006 |
---|
517 | </div> |
---|
518 | </body> |
---|
519 | </html> |
---|