Today's Community Meeting on Testing

[Antranig Basman]

Today we had an edifying community meeting on the subject of our new testing infrastructure - now the 3rd in 
a series of probably 4 meetings. This thread was suggested by an early post of Yura's - can be seen at
http://old.nabble.com/Testing-Framework-Community-Meeting-to34626170.html


The goals of the new framework include:

i) To facilitate the testing of demands blocks that may be issued by integrators against components deployed 
in a particular (complex) context
ii) To automate and regularise the work of "setup" and "teardown" in complex integration scenarios, by 
deferring this to our standard IoC infrastructure
iii) To simplify the often tortuous logic required when using the "nested callback style" to test a 
particular sequence of asynchronous requests and responses (via events) issued against a component with 
complex behaviour
iv) To facilitate the reuse of testing code by allowing test fixtures to be aggregated into what are 
becoming the 2 standard forms for our delivery of implementation - a) pure JSON structures which can be 
freely interchanged and transformed, b) free functions with minimum dependence on context and lifecycle


I presented the implementation I have so far, which is now good enough to demonstrate the approach we want 
to take for iii), allowing testing of event sequences. The quadratically increasing complexity of doing this 
by hand typically deters us from writing thorough tests of this kind - the following heroic code written by 
Yura for testing a CollectionSpace component illustrates the route by which this complexity increases:

                 "recordEditorReady.test": {
                     path: "listeners",
                     listener: function (admin) {
                         var recordRenderer = admin.adminRecordEditor.recordRenderer;
                         jqUnit.assertEquals("Selected username is", "Reader", 
locateSelector(recordRenderer, "screenName").val());
                         jqUnit.notVisible("Confiration dialog is invisible initially", 
admin.adminRecordEditor.confirmation.popup);
                         locateSelector(recordRenderer, "screenName").val("New Name").change();
                         admin.adminRecordEditor.confirmation.popup.bind("dialogopen", function () {
                             jqUnit.isVisible("Confirmation dialog should now be visible", 
admin.adminRecordEditor.confirmation.popup);
                             admin.events.onSelect.addListener(function () {
                                 admin.events.recordEditorReady.addListener(function (admin) {
                                     jqUnit.assertEquals("User Name should now be", "Administrator", 
locateSelector(admin.adminRecordEditor.recordRenderer, "screenName").val());
                                     start();
                                 }, undefined, undefined, "last");
                             });
                       .....
(available at https://github.com/collectionspace/ui/blob/master/src/test/js/AdminUsersTest.js#L586-594 )

The outer level shows a testing framework devised by Yura to start to attack this issue - although the outer 
layer of event registration has been unwound, in the body of the listener we can see a set of callbacks 
nested FOUR deep in order to issue an integration test making assertions about a sequence of 4 events.

A declarative structure would allow us to flatten this nesting into a simple array of sequential assertions. 
Here is an example from the test cases I showed today:

fluid.defaults("fluid.tests.asyncTester", {
     gradeNames: ["fluid.test.testCaseHolder", "autoInit"],
     testCases: [ {
         name: "Async test case",
         tests: [{
             name: "Rendering sequence",
             expect: 2,
             sequence: [ {
                 func: "fluid.tests.startRendering",
                 args: ["{asyncTest}", "{instantiator}"]
             }, {
                 listener: "fluid.tests.checkEvent",
                 event: "{asyncTest}.events.buttonClicked"
             }]
         }
         ]
     }]
});

(available in my FLUID-4850 branch at 
https://github.com/amb26/infusion/blob/bc7a6a414d251a640e868aca38a9977f474cc9be/src/webapp/tests/test-core/testTests/js/TestingTests.js#L132-L149 
)

The interesting aspects of this "test fixture holding grade" are as follows -

i) It is a pure configuration grade with no implementation code, offering good potential for reusability
ii) The block of configuration "sequence" is a flat array, where each element of the array corresponds to a 
sequential state of the component under test.

The "sequence" array may hold "fixture directives" of a small variety of types. We can currently imagine a 
repertoire of 4 or 5 record types, which themselves can be assigned to one of two broader categories - 
"executor" records which actively interact with the component under test, and "binder" records which 
register listeners responding to events fired by the component under test. Any sequence of these records may 
appear in any order. The types we imagine, indexed by a "duck typing field" system slightly reminiscent of 
that used by the Fluid Renderer, are as follows:


func (executor): Execute a free function with arguments IoC-resolved against the tree

listener (binder): Register a listener to a Fluid event good for ONE firing only at the appropriate point in 
the sequence

jquery (binder): Register a listener to a jQuery event against a DOM element resolvable in the tree

changeListener (binder): Register a listener to a ChangeApplier event fired by a model-bearing component in 
the tree

jqueryTrigger (executor): Trigger a jQuery event from a DOM node resolvable in the tree


We covered the overall idiom of the test framework in our meeting two weeks ago, but a few points have been 
clarified since then. The overall scheme is that arbitrarily sized chunks of application code under test 
will be embedded together with "fixture holders" of the type shown above together in the same overall 
component tree, corresponding to a "testing environment". The setup and teardown for all of the test cases 
held in these fixtures will proceed by the standard IoC semantics for construction and destruction of 
component trees. The root of this tree will hold a component with the grade "fluid.test.testEnvironment" 
with a specific name chosen by the fixture author which is suitable to issue demands blocks whose scope 
consists of just this environment. The "fixture holders" scattered around the tree arbitrarily each have the 
grade "fluid.test.testCaseHolder" - each of these contains configuration coding for a set of jqUnit (qunit) 
test case "modules" and "test cases" which will be dispatched to the standard jqUnit framework once the 
construction of the overall component tree is complete.

A few issues came up relating to the potential mismatch between this setup/teardown model, at the unit of 
whole component trees and qunit "modules", and the native one operated by qunit which only operates at the 
level of individual test cases. Our tentative decision is to sidestep the native teardown model completely 
in favour of one operated by this new framework in a dedicated way. The native model comprises three parts - 
i) setup/teardown at the level of JavaScript globals, checking for global namespace pollution - ii) 
setup/teardown of markup nested inside a DOM node with the hard-coded ID "qunit-fixture" (formerly "main" in 
earlier versions of qunit) - iii) user-supplied setup/teardown functions to a module which are operated for 
each individual TestCase within the module.

Part i) is healthy enough, although there should be no instances of such global pollution caused by Fluid 
components (unless their execution happens to cause further components to become defined - not expected with 
our current code idioms). ii) will be replaced by a new scheme allowing ANY selector to be registered at the 
root of the environment as the markup to participate in setup/teardown on the lifecycle of the entire 
environment, rather than individual test cases, iii) will be replaced by the overall action of the 
construction and destruction of the component tree.


Together with the presentation, there were a number of interesting questions from the group which I 
reproduce here (not necessarily in order):


Testing onCreate:
================

Justin asked how the system could be used to test the action of the "onCreate" event of a component in the 
tree under test. I replied that the creation of the component under test would need to be deferred by means 
of the "createOnEvent" directive, and then an executor block inserted into the fixture description to 
operate that event within the fixture sequence, as in the following sketch:

fluid.defaults("fluid.tests.myTestTree", {
     gradeNames: ["fluid.test.testEnvironment", "autoInit"],
     events: {
         startCat: null
         },
     components: {
         cat: {
             createOnEvent: "startCat",
             type: "fluid.tests.cat"
         },
         catTester: {
             type: "fluid.tests.catTester"
         }
     },

and then in the tester:


fluid.defaults("fluid.tests.catTester", {
     gradeNames: ["fluid.test.testCaseHolder", "autoInit"],
     testCases: [ {
         name: "Late cat tester",
         tests: [{
             name: "Rendering sequence",
             expect: 2,
             sequence: [ {
                 func: "{myTestTree}.events.startCat.fire",
                 }, {
                 event: "{cat}.events.onCreate",
                 listener: "fluid.tests.catCreationTester"
                 },
                 etc.


Spectrum between unit tests and integration tests:
=================================================

Michelle asked about how appropriate this system was for expressing unit tests as considered against 
integration tests, how these two situations could be identified by those reading the code, and how we would 
make recommendations about what tests to write. After some discussion, we resolved on a few things:

i) There is a spectrum between unit tests and integration tests, which can be roughly identified by the SIZE 
of the component tree appearing in the "component under test" part of the overall environment test. If this 
tree consists of a single component, the test situation would more clearly have the character of a "unit 
test". A large tree or indeed an entire application present in the environment would represent a complex 
integration test.
ii) One purpose of the framework would be to bridge between the worlds of unit and integration tests, 
allowing the same code and idioms to be usable in both worlds. However, it's clear that in general the IoC 
testing system is more appropriate and more economical the closer the situation represents an "integration 
testing scenario"
iii) That said, even a single Fluid component which is event-driven could benefit significantly from having 
its tests written in the new style - this enables us to write tests which are easier to read intent from, as 
well as being more powerful than those we could write before - easily testing a PARTICULAR sequence of 
events, rather than just placing ad hoc constraints on sequencing in the way we often would in our old-style 
tests, usually involving some kind of scrawling into suitably scoped variables - the following style of code 
will be familiar to us all:

         var that = fluid.tests.eventParent3();
         var received = {};
         that.eventChild.events.relayEvent.addListener(function(arg) {
             received.arg = arg;
         });
         that.events.parentEvent1.fire(that); // first event does nothing

iv) Non-event driven code which just implements ALGORITHMS (such as the model transformation system) can 
continue to be profitably written in the plain old jqUnit style - however -
v) Given that "new" testing code is reduced to the status of free functions holding jqUnit assertions, there 
is a lot of scope for sharing and reusing these fixture functions between code written in plain jqUnit style 
implementing unit tests, and IoC style implementing integration tests - for example, the following fixture 
function in the TestingTests we looked at:

fluid.tests.startRendering = function (asyncTest, instantiator) {
     asyncTest.refreshView();
     var decorators = fluid.renderer.getDecoratorComponents(asyncTest, instantiator);
     var decArray = fluid.values(decorators);
     jqUnit.assertEquals("Constructed one component", 1, decArray.length);
     asyncTest.locate("button").click();
};

could just as easily be invoked from within a manual jqUnit test as by the IoC testing framework.

Appropriate unit of reusability:
===============================

Alex expressed the concern that the global namespace might endlessly clutter up with numerous specifically 
named fragments of dedicated testing functions (in the "free" style we just looked at). We turned to an 
analysis of Yura's very complex testing code (near the beginning of this post), and decided that the best 
outcome would be if the ability to write such free functions ended up in a better factoring of testing 
responsibilities rather than proliferating numerous similar functions. In fact, once all the complexity of 
event binding were removed from this code, we would discover that the actual testing assertions issued 
mostly consisted of single jqUnit assertions - which could be issued directly from the fixture sequence 
rather than requiring a new dedicated free function to be written. Other opportunities for reuse could also 
be more clearly identified - for example, the AdminUsersTest.js file shown above would reveal opportunities 
centred around the existing "locateSelector" function as used often in the following pattern:

                         locateSelector(recordRenderer, "screenName").val("New Name").change();

Testing particular instance of event firing:
===========================================

Justin asked a question relating to a situation encountered in Decapod where he needed to test (say) the 
SECOND instance of firing an event (in this case, a rendering event) whilst ignoring the first, and asked 
whether this kind of thing would be assisted by the framework. I replied that this was just the kind of use 
case for which it was designed - a sequence record, say, holding just fluid.identity or jqUnit.assert could 
be supplied for the first event firing, and a more complex one for the second event, verifying that 
particular pieces of markup had indeed been rendered.


Dealing with "dropped sequence" failures:
========================================

JURA highlighted the frequently unhelpful behaviour of (j)qunit on encountering a failure in an asynchronous 
test - this simply causes the UI to hang, without any clear indication of whether there really are more 
tests or what the expected operation which failed to occur was. To be clear, this is a failure cause by some 
path where an asyncTest fails to cause the "QUnit.start()" operation to be invoked, through a particular 
event failing to be fired - rather than a direct failure held within code.

I replied that this problem scenario was to a certain extent a fundamental and irremovable feature of any 
asynchronous testing framework - purely by virtue of being asynchronous, an event's occurence would be 
unexpected and could never be finally deemed not to have occured. However, I suggested that as a result of 
the clear declarative nature of the "sequence" structure, the new framework could make it considerably 
clearer than formerly which the "missing event" was by providing information as to the most recently 
correctly operated sequence point. This could (should) even be highlighted somehow in the existing QUnit UI 
as a separate diagnostic to help the user pinpoint the expected and missing event.

Yura mentioned that another approach could be to assist the user in setting timeouts on events - I commented 
that this was certainly valuable, but could itself be the cause of false positive test failures in the case 
the browser was running slowly. However, it would be something that would be easy to add to the existing 
declarative structure for "binding" sequence records by adding an extra "timeout" field - far easier than 
writing the requisite timeout handling code by hand.



I feel there were some other interesting and useful questions but can't bring them to mind right now, 
perhaps if someone recalls, they could add them in replies.



Implementation status:

I still have some work to do to bring the implementation to a usable status. On the to-do list:

i) Implement the other 3 record types (jquery, changeListener, jqueryTrigger) and apply much more thorough 
testing to ensure that all of the different patterns of sequential appearance are handled correctly in terms 
of the binding and execution sequence chosen by the framework
ii) Fix the current very hacked system for chaining together the execution of several test environments in 
sequence - given the markup setup/teardown system described above, this needs to be properly serialised to 
avoid corrupting the document
iii) Implement the described markup setup/teardown system!
iv) Ensure that something sensible happens when the user selects just a single test, or a test case filter 
to be operated via the HTML UI provided by qunit. This should be possible (perhaps by using our now freed-up 
slot for qunit per-test setup/teardown functions) without needing to hack on the underlying qunit code!

This should be all ready in time for the final testing meeting next week!

Cheers,
Antranig