source: trunk/server/www/vendors/simpletest/docs/en/reporter_documentation.html @ 6

Last change on this file since 6 was 6, checked in by sander, 11 years ago

Added SimpleTest? test framework

File size: 22.2 KB
Line 
1<html>
2<head>
3<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
4<title>SimpleTest for PHP test runner and display documentation</title>
5<link rel="stylesheet" type="text/css" href="docs.css" title="Styles">
6</head>
7<body>
8<div class="menu_back"><div class="menu">
9<a href="index.html">SimpleTest</a>
10                |
11                <a href="overview.html">Overview</a>
12                |
13                <a href="unit_test_documentation.html">Unit tester</a>
14                |
15                <a href="group_test_documentation.html">Group tests</a>
16                |
17                <a href="mock_objects_documentation.html">Mock objects</a>
18                |
19                <a href="partial_mocks_documentation.html">Partial mocks</a>
20                |
21                <span class="chosen">Reporting</span>
22                |
23                <a href="expectation_documentation.html">Expectations</a>
24                |
25                <a href="web_tester_documentation.html">Web tester</a>
26                |
27                <a href="form_testing_documentation.html">Testing forms</a>
28                |
29                <a href="authentication_documentation.html">Authentication</a>
30                |
31                <a href="browser_documentation.html">Scriptable browser</a>
32</div></div>
33<h1>Test reporter documentation</h1>
34        This page...
35        <ul>
36<li>
37            Displaying <a href="#html">results in HTML</a>
38        </li>
39<li>
40            Displaying and <a href="#other">reporting results</a>
41            in other formats
42        </li>
43<li>
44            Using <a href="#cli">SimpleTest from the command line</a>
45        </li>
46<li>
47            Using <a href="#xml">Using XML</a> for remote testing
48        </li>
49</ul>
50<div class="content">
51       
52            <p>
53                SimpleTest pretty much follows the MVC pattern
54                (Model-View-Controller).
55                The reporter classes are the view and the model is your
56                test cases and their hiearchy.
57                The controller is mostly hidden from the user of
58                SimpleTest unless you want to change how the test cases
59                are actually run, in which case it is possible to
60                override the runner objects from within the test case.
61                As usual with MVC, the controller is mostly undefined
62                and there are other places to control the test run.
63            </p>
64       
65        <p><a class="target" name="html"><h2>Reporting results in HTML</h2></a></p>
66            <p>
67                The default test display is minimal in the extreme.
68                It reports success and failure with the conventional red and
69                green bars and shows a breadcrumb trail of test groups
70                for every failed assertion.
71                Here's a fail...
72                <div class="demo">
73                    <h1>File test</h1>
74                    <span class="fail">Fail</span>: createnewfile-&gt;True assertion failed.<br>
75                    <div style="padding: 8px; margin-top: 1em; background-color: red; color: white;">1/1 test cases complete.
76                    <strong>0</strong> passes, <strong>1</strong> fails and <strong>0</strong> exceptions.</div>
77                </div>
78                And here all tests passed...
79                <div class="demo">
80                    <h1>File test</h1>
81                    <div style="padding: 8px; margin-top: 1em; background-color: green; color: white;">1/1 test cases complete.
82                    <strong>1</strong> passes, <strong>0</strong> fails and <strong>0</strong> exceptions.</div>
83                </div>
84                The good news is that there are several points in the display
85                hiearchy for subclassing.
86            </p>
87            <p>
88                For web page based displays there is the
89                <span class="new_code">HtmlReporter</span> class with the following
90                signature...
91<pre>
92class HtmlReporter extends SimpleReporter {
93    public HtmlReporter($encoding) { ... }
94    public makeDry(boolean $is_dry) { ... }
95    public void paintHeader(string $test_name) { ... }
96    public void sendNoCacheHeaders() { ... }
97    public void paintFooter(string $test_name) { ... }
98    public void paintGroupStart(string $test_name, integer $size) { ... }
99    public void paintGroupEnd(string $test_name) { ... }
100    public void paintCaseStart(string $test_name) { ... }
101    public void paintCaseEnd(string $test_name) { ... }
102    public void paintMethodStart(string $test_name) { ... }
103    public void paintMethodEnd(string $test_name) { ... }
104    public void paintFail(string $message) { ... }
105    public void paintPass(string $message) { ... }
106    public void paintError(string $message) { ... }
107    public void paintException(string $message) { ... }
108    public void paintMessage(string $message) { ... }
109    public void paintFormattedMessage(string $message) { ... }
110    protected string _getCss() { ... }
111    public array getTestList() { ... }
112    public integer getPassCount() { ... }
113    public integer getFailCount() { ... }
114    public integer getExceptionCount() { ... }
115    public integer getTestCaseCount() { ... }
116    public integer getTestCaseProgress() { ... }
117}
118</pre>
119                Here is what some of these methods mean. First the display methods
120                that you will probably want to override...
121                <ul class="api">
122                    <li>
123                        <span class="new_code">HtmlReporter(string $encoding)</span><br>
124                        is the constructor.
125                        Note that the unit test sets up the link to the display
126                        rather than the other way around.
127                        The display is a mostly passive receiver of test events.
128                        This allows easy adaption of the display for other test
129                        systems beside unit tests, such as monitoring servers.
130                        The encoding is the character encoding you wish to
131                        display the test output in.
132                        In order to correctly render debug output when
133                        using the web tester, this should match the encoding
134                        of the site you are trying to test.
135                        The available character set strings are described in
136                        the PHP <a href="http://www.php.net/manual/en/function.htmlentities.php">html_entities()</a>
137                        function.
138                    </li>
139                    <li>
140                        <span class="new_code">void paintHeader(string $test_name)</span><br>
141                        is called once at the very start of the test when the first
142                        start event arrives.
143                        The first start event is usually delivered by the top level group
144                        test and so this is where <span class="new_code">$test_name</span>
145                        comes from.
146                        It paints the page titles, CSS, body tag, etc.
147                        It returns nothing (<span class="new_code">void</span>).
148                    </li>
149                    <li>
150                        <span class="new_code">void paintFooter(string $test_name)</span><br>
151                        Called at the very end of the test to close any tags opened
152                        by the page header.
153                        By default it also displays the red/green bar and the final
154                        count of results.
155                        Actually the end of the test happens when a test end event
156                        comes in with the same name as the one that started it all
157                        at the same level.
158                        The tests nest you see.
159                        Closing the last test finishes the display.
160                    </li>
161                    <li>
162                        <span class="new_code">void paintMethodStart(string $test_name)</span><br>
163                        is called at the start of each test method.
164                        The name normally comes from method name.
165                        The other test start events behave the same way except
166                        that the group test one tells the reporter how large
167                        it is in number of held test cases.
168                        This is so that the reporter can display a progress bar
169                        as the runner churns through the test cases.
170                    </li>
171                    <li>
172                        <span class="new_code">void paintMethodEnd(string $test_name)</span><br>
173                        backs out of the test started with the same name.
174                    </li>
175                    <li>
176                        <span class="new_code">void paintFail(string $message)</span><br>
177                        paints a failure.
178                        By default it just displays the word fail, a breadcrumbs trail
179                        showing the current test nesting and the message issued by
180                        the assertion.
181                    </li>
182                    <li>
183                        <span class="new_code">void paintPass(string $message)</span><br>
184                        by default does nothing.
185                    </li>
186                    <li>
187                        <span class="new_code">string _getCss()</span><br>
188                        Returns the CSS styles as a string for the page header
189                        method.
190                        Additional styles have to be appended here if you are
191                        not overriding the page header.
192                        You will want to use this method in an overriden page header
193                        if you want to include the original CSS.
194                    </li>
195                </ul>
196                There are also some accessors to get information on the current
197                state of the test suite.
198                Use these to enrich the display...
199                <ul class="api">
200                    <li>
201                        <span class="new_code">array getTestList()</span><br>
202                        is the first convenience method for subclasses.
203                        Lists the current nesting of the tests as a list
204                        of test names.
205                        The first, most deeply nested test, is first in the
206                        list and the current test method will be last.
207                    </li>
208                    <li>
209                        <span class="new_code">integer getPassCount()</span><br>
210                        returns the number of passes chalked up so far.
211                        Needed for the display at the end.
212                    </li>
213                    <li>
214                        <span class="new_code">integer getFailCount()</span><br>
215                        is likewise the number of fails so far.
216                    </li>
217                    <li>
218                        <span class="new_code">integer getExceptionCount()</span><br>
219                        is likewise the number of errors so far.
220                    </li>
221                    <li>
222                        <span class="new_code">integer getTestCaseCount()</span><br>
223                        is the total number of test cases in the test run.
224                        This includes the grouping tests themselves.
225                    </li>
226                    <li>
227                        <span class="new_code">integer getTestCaseProgress()</span><br>
228                        is the number of test cases completed so far.
229                    </li>
230                </ul>
231                One simple modification is to get the HtmlReporter to display
232                the passes as well as the failures and errors...
233<pre>
234<strong>class ShowPasses extends HtmlReporter {
235   
236    function paintPass($message) {
237        parent::paintPass($message);
238        print "&amp;&lt;span class=\"pass\"&gt;Pass&lt;/span&gt;: ";
239        $breadcrumb = $this-&gt;getTestList();
240        array_shift($breadcrumb);
241        print implode("-&amp;gt;", $breadcrumb);
242        print "-&amp;gt;$message&lt;br /&gt;\n";
243    }
244   
245    function _getCss() {
246        return parent::_getCss() . ' .pass { color: green; }';
247    }
248}</strong>
249</pre>
250            </p>
251            <p>
252                One method that was glossed over was the <span class="new_code">makeDry()</span>
253                method.
254                If you run this method, with no parameters, on the reporter
255                before the test suite is run no actual test methods
256                will be called.
257                You will still get the events of entering and leaving the
258                test methods and test cases, but no passes or failures etc,
259                because the test code will not actually be executed.
260            </p>
261            <p>
262                The reason for this is to allow for more sophistcated
263                GUI displays that allow the selection of individual test
264                cases.
265                In order to build a list of possible tests they need a
266                report on the test structure for drawing, say a tree view
267                of the test suite.
268                With a reporter set to dry run that just sends drawing events
269                this is easily accomplished.
270            </p>
271       
272        <p><a class="target" name="other"><h2>Extending the reporter</h2></a></p>
273            <p>
274                Rather than simply modifying the existing display, you might want to
275                produce a whole new HTML look, or even generate text or XML.
276                Rather than override every method in
277                <span class="new_code">HtmlReporter</span> we can take one
278                step up the class hiearchy to <span class="new_code">SimpleReporter</span>
279                in the <em>simple_test.php</em> source file.
280            </p>
281            <p>
282                A do nothing display, a blank canvas for your own creation, would
283                be...
284<pre>
285<strong>require_once('simpletest/simple_test.php');</strong>
286
287class MyDisplay extends SimpleReporter {<strong>
288    </strong>
289    function paintHeader($test_name) {
290    }
291   
292    function paintFooter($test_name) {
293    }
294   
295    function paintStart($test_name, $size) {<strong>
296        parent::paintStart($test_name, $size);</strong>
297    }
298   
299    function paintEnd($test_name, $size) {<strong>
300        parent::paintEnd($test_name, $size);</strong>
301    }
302   
303    function paintPass($message) {<strong>
304        parent::paintPass($message);</strong>
305    }
306   
307    function paintFail($message) {<strong>
308        parent::paintFail($message);</strong>
309    }
310}
311</pre>
312                No output would come from this class until you add it.
313            </p>
314       
315        <p><a class="target" name="cli"><h2>The command line reporter</h2></a></p>
316            <p>
317                SimpleTest also ships with a minimal command line reporter.
318                The interface mimics JUnit to some extent, but paints the
319                failure messages as they arrive.
320                To use the command line reporter simply substitute it
321                for the HTML version...
322<pre>
323&lt;?php
324require_once('simpletest/unit_tester.php');
325require_once('simpletest/reporter.php');
326
327$test = &amp;new TestSuite('File test');
328$test-&gt;addTestFile('tests/file_test.php');
329$test-&gt;run(<strong>new TextReporter()</strong>);
330?&gt;
331</pre>
332                Then invoke the test suite from the command line...
333<pre class="shell">
334php file_test.php
335</pre>
336                You will need the command line version of PHP installed
337                of course.
338                A passing test suite looks like this...
339<pre class="shell">
340File test
341OK
342Test cases run: 1/1, Failures: 0, Exceptions: 0
343</pre>
344                A failure triggers a display like this...
345<pre class="shell">
346File test
3471) True assertion failed.
348    in createnewfile
349FAILURES!!!
350Test cases run: 1/1, Failures: 1, Exceptions: 0
351</pre>
352            </p>
353            <p>
354                One of the main reasons for using a command line driven
355                test suite is of using the tester as part of some automated
356                process.
357                To function properly in shell scripts the test script should
358                return a non-zero exit code on failure.
359                If a test suite fails the value <span class="new_code">false</span>
360                is returned from the <span class="new_code">SimpleTest::run()</span>
361                method.
362                We can use that result to exit the script with the desired return
363                code...
364<pre>
365&lt;?php
366require_once('simpletest/unit_tester.php');
367require_once('simpletest/reporter.php');
368
369$test = &amp;new TestSuite('File test');
370$test-&gt;addTestFile('tests/file_test.php');
371<strong>exit ($test-&gt;run(new TextReporter()) ? 0 : 1);</strong>
372?&gt;
373</pre>
374                Of course we don't really want to create two test scripts,
375                a command line one and a web browser one, for each test suite.
376                The command line reporter includes a method to sniff out the
377                run time environment...
378<pre>
379&lt;?php
380require_once('simpletest/unit_tester.php');
381require_once('simpletest/reporter.php');
382
383$test = &amp;new TestSuite('File test');
384$test-&gt;addTestFile('tests/file_test.php');
385<strong>if (TextReporter::inCli()) {</strong>
386    exit ($test-&gt;run(new TextReporter()) ? 0 : 1);
387<strong>}</strong>
388$test-&gt;run(new HtmlReporter());
389?&gt;
390</pre>
391                This is the form used within SimpleTest itself.
392            </p>
393       
394        <p><a class="target" name="xml"><h2>Remote testing</h2></a></p>
395            <p>
396                SimpleTest ships with an <span class="new_code">XmlReporter</span> class
397                used for internal communication.
398                When run the output looks like...
399<pre class="shell">
400&lt;?xml version="1.0"?&gt;
401&lt;run&gt;
402  &lt;group size="4"&gt;
403    &lt;name&gt;Remote tests&lt;/name&gt;
404    &lt;group size="4"&gt;
405      &lt;name&gt;Visual test with 48 passes, 48 fails and 4 exceptions&lt;/name&gt;
406      &lt;case&gt;
407        &lt;name&gt;testofunittestcaseoutput&lt;/name&gt;
408        &lt;test&gt;
409          &lt;name&gt;testofresults&lt;/name&gt;
410          &lt;pass&gt;This assertion passed&lt;/pass&gt;
411          &lt;fail&gt;This assertion failed&lt;/fail&gt;
412        &lt;/test&gt;
413        &lt;test&gt;
414          ...
415        &lt;/test&gt;
416      &lt;/case&gt;
417    &lt;/group&gt;
418  &lt;/group&gt;
419&lt;/run&gt;
420</pre>
421                You can make use of this format with the parser
422                supplied as part of SimpleTest itself.
423                This is called <span class="new_code">SimpleTestXmlParser</span> and
424                resides in <em>xml.php</em> within the SimpleTest package...
425<pre>
426&lt;?php
427require_once('simpletest/xml.php');
428   
429...
430$parser = &amp;new SimpleTestXmlParser(new HtmlReporter());
431$parser-&gt;parse($test_output);
432?&gt;
433</pre>
434                The <span class="new_code">$test_output</span> should be the XML format
435                from the XML reporter, and could come from say a command
436                line run of a test case.
437                The parser sends events to the reporter just like any
438                other test run.
439                There are some odd occasions where this is actually useful.
440            </p>
441            <p>
442                A problem with large test suites is thet they can exhaust
443                the default 8Mb memory limit on a PHP process.
444                By having the test groups output in XML and run in
445                separate processes, the output can be reparsed to
446                aggregate the results into a much smaller footprint top level
447                test.
448            </p>
449            <p>
450                Because the XML output can come from anywhere, this opens
451                up the possibility of aggregating test runs from remote
452                servers.
453                A test case already exists to do this within the SimpleTest
454                framework, but it is currently experimental...
455<pre>
456&lt;?php
457<strong>require_once('../remote.php');</strong>
458require_once('../reporter.php');
459   
460$test_url = ...;
461$dry_url = ...;
462   
463$test = &amp;new TestSuite('Remote tests');
464$test-&gt;addTestCase(<strong>new RemoteTestCase($test_url, $dry_url)</strong>);
465$test-&gt;run(new HtmlReporter());
466?&gt;
467</pre>
468                The <span class="new_code">RemoteTestCase</span> takes the actual location
469                of the test runner, basically a web page in XML format.
470                It also takes the URL of a reporter set to do a dry run.
471                This is so that progress can be reported upward correctly.
472                The <span class="new_code">RemoteTestCase</span> can be added to test suites
473                just like any other group test.
474            </p>
475       
476    </div>
477        References and related information...
478        <ul>
479<li>
480            SimpleTest project page on <a href="http://sourceforge.net/projects/simpletest/">SourceForge</a>.
481        </li>
482<li>
483            SimpleTest download page on <a href="http://www.lastcraft.com/simple_test.php">LastCraft</a>.
484        </li>
485<li>
486            The <a href="http://simpletest.org/api/">developer's API for SimpleTest</a>
487            gives full detail on the classes and assertions available.
488        </li>
489</ul>
490<div class="menu_back"><div class="menu">
491<a href="index.html">SimpleTest</a>
492                |
493                <a href="overview.html">Overview</a>
494                |
495                <a href="unit_test_documentation.html">Unit tester</a>
496                |
497                <a href="group_test_documentation.html">Group tests</a>
498                |
499                <a href="mock_objects_documentation.html">Mock objects</a>
500                |
501                <a href="partial_mocks_documentation.html">Partial mocks</a>
502                |
503                <span class="chosen">Reporting</span>
504                |
505                <a href="expectation_documentation.html">Expectations</a>
506                |
507                <a href="web_tester_documentation.html">Web tester</a>
508                |
509                <a href="form_testing_documentation.html">Testing forms</a>
510                |
511                <a href="authentication_documentation.html">Authentication</a>
512                |
513                <a href="browser_documentation.html">Scriptable browser</a>
514</div></div>
515<div class="copyright">
516            Copyright<br>Marcus Baker 2006
517        </div>
518</body>
519</html>
Note: See TracBrowser for help on using the repository browser.