Context Navigation

source: trunk/server/www/vendors/simpletest/docs/en/reporter_documentation.html @ 6

Last change on this file since 6 was 6, checked in by sander, 12 years ago
Added SimpleTest? test framework
File size: 22.2 KB

Line
1	<html>
2	<head>
3	<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
4	<title>SimpleTest for PHP test runner and display documentation</title>
5	<link rel="stylesheet" type="text/css" href="docs.css" title="Styles">
6	</head>
7	<body>
8	<div class="menu_back"><div class="menu">
9	<a href="index.html">SimpleTest</a>
10	\|
11	<a href="overview.html">Overview</a>
12	\|
13	<a href="unit_test_documentation.html">Unit tester</a>
14	\|
15	<a href="group_test_documentation.html">Group tests</a>
16	\|
17	<a href="mock_objects_documentation.html">Mock objects</a>
18	\|
19	<a href="partial_mocks_documentation.html">Partial mocks</a>
20	\|
21	<span class="chosen">Reporting</span>
22	\|
23	<a href="expectation_documentation.html">Expectations</a>
24	\|
25	<a href="web_tester_documentation.html">Web tester</a>
26	\|
27	<a href="form_testing_documentation.html">Testing forms</a>
28	\|
29	<a href="authentication_documentation.html">Authentication</a>
30	\|
31	<a href="browser_documentation.html">Scriptable browser</a>
32	</div></div>
33	<h1>Test reporter documentation</h1>
34	This page...
35	<ul>
36	<li>
37	Displaying <a href="#html">results in HTML</a>
38	</li>
39	<li>
40	Displaying and <a href="#other">reporting results</a>
41	in other formats
42	</li>
43	<li>
44	Using <a href="#cli">SimpleTest from the command line</a>
45	</li>
46	<li>
47	Using <a href="#xml">Using XML</a> for remote testing
48	</li>
49	</ul>
50	<div class="content">
51
52	<p>
53	SimpleTest pretty much follows the MVC pattern
54	(Model-View-Controller).
55	The reporter classes are the view and the model is your
56	test cases and their hiearchy.
57	The controller is mostly hidden from the user of
58	SimpleTest unless you want to change how the test cases
59	are actually run, in which case it is possible to
60	override the runner objects from within the test case.
61	As usual with MVC, the controller is mostly undefined
62	and there are other places to control the test run.
63	</p>
64
65	<p><a class="target" name="html"><h2>Reporting results in HTML</h2></a></p>
66	<p>
67	The default test display is minimal in the extreme.
68	It reports success and failure with the conventional red and
69	green bars and shows a breadcrumb trail of test groups
70	for every failed assertion.
71	Here's a fail...
72	<div class="demo">
73	<h1>File test</h1>
74	<span class="fail">Fail</span>: createnewfile->True assertion failed.<br>
75	<div style="padding: 8px; margin-top: 1em; background-color: red; color: white;">1/1 test cases complete.
76	<strong>0</strong> passes, <strong>1</strong> fails and <strong>0</strong> exceptions.</div>
77	</div>
78	And here all tests passed...
79	<div class="demo">
80	<h1>File test</h1>
81	<div style="padding: 8px; margin-top: 1em; background-color: green; color: white;">1/1 test cases complete.
82	<strong>1</strong> passes, <strong>0</strong> fails and <strong>0</strong> exceptions.</div>
83	</div>
84	The good news is that there are several points in the display
85	hiearchy for subclassing.
86	</p>
87	<p>
88	For web page based displays there is the
89	<span class="new_code">HtmlReporter</span> class with the following
90	signature...
91	<pre>
92	class HtmlReporter extends SimpleReporter {
93	public HtmlReporter($encoding) { ... }
94	public makeDry(boolean $is_dry) { ... }
95	public void paintHeader(string $test_name) { ... }
96	public void sendNoCacheHeaders() { ... }
97	public void paintFooter(string $test_name) { ... }
98	public void paintGroupStart(string $test_name, integer $size) { ... }
99	public void paintGroupEnd(string $test_name) { ... }
100	public void paintCaseStart(string $test_name) { ... }
101	public void paintCaseEnd(string $test_name) { ... }
102	public void paintMethodStart(string $test_name) { ... }
103	public void paintMethodEnd(string $test_name) { ... }
104	public void paintFail(string $message) { ... }
105	public void paintPass(string $message) { ... }
106	public void paintError(string $message) { ... }
107	public void paintException(string $message) { ... }
108	public void paintMessage(string $message) { ... }
109	public void paintFormattedMessage(string $message) { ... }
110	protected string _getCss() { ... }
111	public array getTestList() { ... }
112	public integer getPassCount() { ... }
113	public integer getFailCount() { ... }
114	public integer getExceptionCount() { ... }
115	public integer getTestCaseCount() { ... }
116	public integer getTestCaseProgress() { ... }
117	}
118	</pre>
119	Here is what some of these methods mean. First the display methods
120	that you will probably want to override...
121	<ul class="api">
122	<li>
123	<span class="new_code">HtmlReporter(string $encoding)</span><br>
124	is the constructor.
125	Note that the unit test sets up the link to the display
126	rather than the other way around.
127	The display is a mostly passive receiver of test events.
128	This allows easy adaption of the display for other test
129	systems beside unit tests, such as monitoring servers.
130	The encoding is the character encoding you wish to
131	display the test output in.
132	In order to correctly render debug output when
133	using the web tester, this should match the encoding
134	of the site you are trying to test.
135	The available character set strings are described in
136	the PHP <a href="http://www.php.net/manual/en/function.htmlentities.php">html_entities()</a>
137	function.
138	</li>
139	<li>
140	<span class="new_code">void paintHeader(string $test_name)</span><br>
141	is called once at the very start of the test when the first
142	start event arrives.
143	The first start event is usually delivered by the top level group
144	test and so this is where <span class="new_code">$test_name</span>
145	comes from.
146	It paints the page titles, CSS, body tag, etc.
147	It returns nothing (<span class="new_code">void</span>).
148	</li>
149	<li>
150	<span class="new_code">void paintFooter(string $test_name)</span><br>
151	Called at the very end of the test to close any tags opened
152	by the page header.
153	By default it also displays the red/green bar and the final
154	count of results.
155	Actually the end of the test happens when a test end event
156	comes in with the same name as the one that started it all
157	at the same level.
158	The tests nest you see.
159	Closing the last test finishes the display.
160	</li>
161	<li>
162	<span class="new_code">void paintMethodStart(string $test_name)</span><br>
163	is called at the start of each test method.
164	The name normally comes from method name.
165	The other test start events behave the same way except
166	that the group test one tells the reporter how large
167	it is in number of held test cases.
168	This is so that the reporter can display a progress bar
169	as the runner churns through the test cases.
170	</li>
171	<li>
172	<span class="new_code">void paintMethodEnd(string $test_name)</span><br>
173	backs out of the test started with the same name.
174	</li>
175	<li>
176	<span class="new_code">void paintFail(string $message)</span><br>
177	paints a failure.
178	By default it just displays the word fail, a breadcrumbs trail
179	showing the current test nesting and the message issued by
180	the assertion.
181	</li>
182	<li>
183	<span class="new_code">void paintPass(string $message)</span><br>
184	by default does nothing.
185	</li>
186	<li>
187	<span class="new_code">string _getCss()</span><br>
188	Returns the CSS styles as a string for the page header
189	method.
190	Additional styles have to be appended here if you are
191	not overriding the page header.
192	You will want to use this method in an overriden page header
193	if you want to include the original CSS.
194	</li>
195	</ul>
196	There are also some accessors to get information on the current
197	state of the test suite.
198	Use these to enrich the display...
199	<ul class="api">
200	<li>
201	<span class="new_code">array getTestList()</span><br>
202	is the first convenience method for subclasses.
203	Lists the current nesting of the tests as a list
204	of test names.
205	The first, most deeply nested test, is first in the
206	list and the current test method will be last.
207	</li>
208	<li>
209	<span class="new_code">integer getPassCount()</span><br>
210	returns the number of passes chalked up so far.
211	Needed for the display at the end.
212	</li>
213	<li>
214	<span class="new_code">integer getFailCount()</span><br>
215	is likewise the number of fails so far.
216	</li>
217	<li>
218	<span class="new_code">integer getExceptionCount()</span><br>
219	is likewise the number of errors so far.
220	</li>
221	<li>
222	<span class="new_code">integer getTestCaseCount()</span><br>
223	is the total number of test cases in the test run.
224	This includes the grouping tests themselves.
225	</li>
226	<li>
227	<span class="new_code">integer getTestCaseProgress()</span><br>
228	is the number of test cases completed so far.
229	</li>
230	</ul>
231	One simple modification is to get the HtmlReporter to display
232	the passes as well as the failures and errors...
233	<pre>
234	<strong>class ShowPasses extends HtmlReporter {
235
236	function paintPass($message) {
237	parent::paintPass($message);
238	print "&<span class=\"pass\">Pass</span>: ";
239	$breadcrumb = $this->getTestList();
240	array_shift($breadcrumb);
241	print implode("-&gt;", $breadcrumb);
242	print "-&gt;$message<br />\n";
243	}
244
245	function _getCss() {
246	return parent::_getCss() . ' .pass { color: green; }';
247	}
248	}</strong>
249	</pre>
250	</p>
251	<p>
252	One method that was glossed over was the <span class="new_code">makeDry()</span>
253	method.
254	If you run this method, with no parameters, on the reporter
255	before the test suite is run no actual test methods
256	will be called.
257	You will still get the events of entering and leaving the
258	test methods and test cases, but no passes or failures etc,
259	because the test code will not actually be executed.
260	</p>
261	<p>
262	The reason for this is to allow for more sophistcated
263	GUI displays that allow the selection of individual test
264	cases.
265	In order to build a list of possible tests they need a
266	report on the test structure for drawing, say a tree view
267	of the test suite.
268	With a reporter set to dry run that just sends drawing events
269	this is easily accomplished.
270	</p>
271
272	<p><a class="target" name="other"><h2>Extending the reporter</h2></a></p>
273	<p>
274	Rather than simply modifying the existing display, you might want to
275	produce a whole new HTML look, or even generate text or XML.
276	Rather than override every method in
277	<span class="new_code">HtmlReporter</span> we can take one
278	step up the class hiearchy to <span class="new_code">SimpleReporter</span>
279	in the <em>simple_test.php</em> source file.
280	</p>
281	<p>
282	A do nothing display, a blank canvas for your own creation, would
283	be...
284	<pre>
285	<strong>require_once('simpletest/simple_test.php');</strong>
286
287	class MyDisplay extends SimpleReporter {<strong>
288	</strong>
289	function paintHeader($test_name) {
290	}
291
292	function paintFooter($test_name) {
293	}
294
295	function paintStart($test_name, $size) {<strong>
296	parent::paintStart($test_name, $size);</strong>
297	}
298
299	function paintEnd($test_name, $size) {<strong>
300	parent::paintEnd($test_name, $size);</strong>
301	}
302
303	function paintPass($message) {<strong>
304	parent::paintPass($message);</strong>
305	}
306
307	function paintFail($message) {<strong>
308	parent::paintFail($message);</strong>
309	}
310	}
311	</pre>
312	No output would come from this class until you add it.
313	</p>
314
315	<p><a class="target" name="cli"><h2>The command line reporter</h2></a></p>
316	<p>
317	SimpleTest also ships with a minimal command line reporter.
318	The interface mimics JUnit to some extent, but paints the
319	failure messages as they arrive.
320	To use the command line reporter simply substitute it
321	for the HTML version...
322	<pre>
323	<?php
324	require_once('simpletest/unit_tester.php');
325	require_once('simpletest/reporter.php');
326
327	$test = &new TestSuite('File test');
328	$test->addTestFile('tests/file_test.php');
329	$test->run(<strong>new TextReporter()</strong>);
330	?>
331	</pre>
332	Then invoke the test suite from the command line...
333	<pre class="shell">
334	php file_test.php
335	</pre>
336	You will need the command line version of PHP installed
337	of course.
338	A passing test suite looks like this...
339	<pre class="shell">
340	File test
341	OK
342	Test cases run: 1/1, Failures: 0, Exceptions: 0
343	</pre>
344	A failure triggers a display like this...
345	<pre class="shell">
346	File test
347	1) True assertion failed.
348	in createnewfile
349	FAILURES!!!
350	Test cases run: 1/1, Failures: 1, Exceptions: 0
351	</pre>
352	</p>
353	<p>
354	One of the main reasons for using a command line driven
355	test suite is of using the tester as part of some automated
356	process.
357	To function properly in shell scripts the test script should
358	return a non-zero exit code on failure.
359	If a test suite fails the value <span class="new_code">false</span>
360	is returned from the <span class="new_code">SimpleTest::run()</span>
361	method.
362	We can use that result to exit the script with the desired return
363	code...
364	<pre>
365	<?php
366	require_once('simpletest/unit_tester.php');
367	require_once('simpletest/reporter.php');
368
369	$test = &new TestSuite('File test');
370	$test->addTestFile('tests/file_test.php');
371	<strong>exit ($test->run(new TextReporter()) ? 0 : 1);</strong>
372	?>
373	</pre>
374	Of course we don't really want to create two test scripts,
375	a command line one and a web browser one, for each test suite.
376	The command line reporter includes a method to sniff out the
377	run time environment...
378	<pre>
379	<?php
380	require_once('simpletest/unit_tester.php');
381	require_once('simpletest/reporter.php');
382
383	$test = &new TestSuite('File test');
384	$test->addTestFile('tests/file_test.php');
385	<strong>if (TextReporter::inCli()) {</strong>
386	exit ($test->run(new TextReporter()) ? 0 : 1);
387	<strong>}</strong>
388	$test->run(new HtmlReporter());
389	?>
390	</pre>
391	This is the form used within SimpleTest itself.
392	</p>
393
394	<p><a class="target" name="xml"><h2>Remote testing</h2></a></p>
395	<p>
396	SimpleTest ships with an <span class="new_code">XmlReporter</span> class
397	used for internal communication.
398	When run the output looks like...
399	<pre class="shell">
400	<?xml version="1.0"?>
401	<run>
402	<group size="4">
403	<name>Remote tests</name>
404	<group size="4">
405	<name>Visual test with 48 passes, 48 fails and 4 exceptions</name>
406	<case>
407	<name>testofunittestcaseoutput</name>
408	<test>
409	<name>testofresults</name>
410	<pass>This assertion passed</pass>
411	<fail>This assertion failed</fail>
412	</test>
413	<test>
414	...
415	</test>
416	</case>
417	</group>
418	</group>
419	</run>
420	</pre>
421	You can make use of this format with the parser
422	supplied as part of SimpleTest itself.
423	This is called <span class="new_code">SimpleTestXmlParser</span> and
424	resides in <em>xml.php</em> within the SimpleTest package...
425	<pre>
426	<?php
427	require_once('simpletest/xml.php');
428
429	...
430	$parser = &new SimpleTestXmlParser(new HtmlReporter());
431	$parser->parse($test_output);
432	?>
433	</pre>
434	The <span class="new_code">$test_output</span> should be the XML format
435	from the XML reporter, and could come from say a command
436	line run of a test case.
437	The parser sends events to the reporter just like any
438	other test run.
439	There are some odd occasions where this is actually useful.
440	</p>
441	<p>
442	A problem with large test suites is thet they can exhaust
443	the default 8Mb memory limit on a PHP process.
444	By having the test groups output in XML and run in
445	separate processes, the output can be reparsed to
446	aggregate the results into a much smaller footprint top level
447	test.
448	</p>
449	<p>
450	Because the XML output can come from anywhere, this opens
451	up the possibility of aggregating test runs from remote
452	servers.
453	A test case already exists to do this within the SimpleTest
454	framework, but it is currently experimental...
455	<pre>
456	<?php
457	<strong>require_once('../remote.php');</strong>
458	require_once('../reporter.php');
459
460	$test_url = ...;
461	$dry_url = ...;
462
463	$test = &new TestSuite('Remote tests');
464	$test->addTestCase(<strong>new RemoteTestCase($test_url, $dry_url)</strong>);
465	$test->run(new HtmlReporter());
466	?>
467	</pre>
468	The <span class="new_code">RemoteTestCase</span> takes the actual location
469	of the test runner, basically a web page in XML format.
470	It also takes the URL of a reporter set to do a dry run.
471	This is so that progress can be reported upward correctly.
472	The <span class="new_code">RemoteTestCase</span> can be added to test suites
473	just like any other group test.
474	</p>
475
476	</div>
477	References and related information...
478	<ul>
479	<li>
480	SimpleTest project page on <a href="http://sourceforge.net/projects/simpletest/">SourceForge</a>.
481	</li>
482	<li>
483	SimpleTest download page on <a href="http://www.lastcraft.com/simple_test.php">LastCraft</a>.
484	</li>
485	<li>
486	The <a href="http://simpletest.org/api/">developer's API for SimpleTest</a>
487	gives full detail on the classes and assertions available.
488	</li>
489	</ul>
490	<div class="menu_back"><div class="menu">
491	<a href="index.html">SimpleTest</a>
492	\|
493	<a href="overview.html">Overview</a>
494	\|
495	<a href="unit_test_documentation.html">Unit tester</a>
496	\|
497	<a href="group_test_documentation.html">Group tests</a>
498	\|
499	<a href="mock_objects_documentation.html">Mock objects</a>
500	\|
501	<a href="partial_mocks_documentation.html">Partial mocks</a>
502	\|
503	<span class="chosen">Reporting</span>
504	\|
505	<a href="expectation_documentation.html">Expectations</a>
506	\|
507	<a href="web_tester_documentation.html">Web tester</a>
508	\|
509	<a href="form_testing_documentation.html">Testing forms</a>
510	\|
511	<a href="authentication_documentation.html">Authentication</a>
512	\|
513	<a href="browser_documentation.html">Scriptable browser</a>
514	</div></div>
515	<div class="copyright">
516	Copyright<br>Marcus Baker 2006
517	</div>
518	</body>
519	</html>

Note: See TracBrowser for help on using the repository browser.

Download in other formats: