One of the biggest Ruby unit test suites in existance is JRuby’s, which weights (all external suites included) a whopping 39kLOC. It is split into some big subprojects:
- The JRuby test suite (10kLOC)
- The Rubinius test suite (8.4kLOC)
- The Rubicon test suite (5.8kLOC)
- The BFTS (4.3kLOC)
- The Matz Ruby Interpreter test suite (4kLOC)
(The other ~6kLoc are benchmarks and Java unit tests, which are not relevant here.)
I have analyzed each of them according to these factors:
- Number of test cases (contexts)
- Number of tests (specifcations)
- Number of assertions (shoulds)
- how many are negated
- what do they test
I’ll go through each test suite and present the numbers. Please note that they were made with some hacked-together scripts and may not be totally accurate, but they should show the big picture. All numbers are based on checkout r3767, last changed 2007-05-30 07:52:31 +0200.
The JRuby test suite
The JRuby test suite uses minirunit and Test::Unit for its tests. The minirunit part has 2747 assertions, but no specially marked test cases. The most used assertions were:
- test_equal (1695)
- test_ok (400)
- test_exception (277)
- test_tree (118)
- test_no_exception (73)
- test_check (69)
There are a lot non-xUnit assertions defined, which only make up a small part of the suite: the often used ones are test_tree (assertions about the parse tree) and test_marshal.
Negated assertions occur as test_no_exception and test_fail, and make up about 5.6%. (Including 71 cases of test_ok with a negated expression.)
The small mri test suite (minirunit too) has 260 assertions, of which 226 are test_ok, 19 test_equal and 15 test_check.
The Test::Unit part of the suite has 40 test cases with 244 tests. There are 568 assertions, of which 33 (5.8%) are negated. Most used are:
- assert_equal (331)
- assert_raises (38)
- assert_parse_only (38)
- assert_inspect_evaled (28)
- assert_argerr (20)
- assert_cycle (18)
Other non-xUnit assertions are: test_to_yaml, test_eval_inspected, test_path_segments.
The Rubinius test suite
The Rubinius test suite uses a small clone of RSpec called mini_rspec. It contains 117 contexts with 1046 specifications. There are 2513 shoulds, of which 15 are negated (0.6%).
The most used shoulds were:
- should == (1765)
- should == true (304)
- should == false (106)
- should_raise (141)
- should == nil (87)
- should_include (35)
should_receive (12) was the only custom should found.
The Rubicon test suite
The Rubicon test suite uses Test::Unit and has 42 test cases with 480 tests. Of 2316 assertions, 18 are negated (0.8%).
Most used assertions were:
- assert_equal (1941)
- assert_flequal (107)
- assert_nil (102)
- assert_raise (79)
- assert_same (30)
Non-xUnit assertions are assert_bag_equal.
The BFTS
The Big Fucking Test Suite consists of 13 Test::Unit test cases with 359 tests. There are 1914 assertions, 34 are negated (1.8%).
Most used where:
- assert_equal (1706)
- assert_nil (85)
- assert_raises (76)
- assert_same (23)
There are no non-xUnit assertions in the BFTS.
The MRI test suite
The MRI test suite consists of 46 Test::Unit test cases with 166 tests. There are 1178 assertions, of which 28 are negated (2.3%). Most used were:
- assert_equal (982)
- assert_raises (69)
- assert_nil (43)
- assert_instance_of (17)
Non-xUnit assertions are: assert_arity, assert_eval_inspected, assert_inspect_evaled.
Conclusion
N.B.: These unit test suites test language implementations and not highlevel libraries or applications, therefore these results are not directly applicable for general testing frameworks. It would be great if someone could do tests like these for big libraries (Rails) or big applications.
The main assertions used are tests for equality (and especially true, false and nil) and tests for exceptions. Only a small part of assertions are negated, and these mostly check that nothing is raised.
xUnit assertions cover over 90% of all assertions, but custom and user-defined assertions simplify testing by abstracting common code and making test cleaner and clearer.
Future testing frameworks (especially for language test suites) should optimize for these features.
NP: Bob Dylan—Forever Young