Community status of jqUnit.js

[Antranig Basman]

To me, I think the economics of the issue are about the following -
i) what are tests FOR -
    to which the answer is - tests are for failing as often (early) as possible, in order to show up 
implementation errors
ii) what do we do when they fail -
    to which the answer isn't quite so clear - although generally involves going hunting for what the cause 
is that made them fail. The tests are valuable if the cost of hunting down the failure in a test case is 
considerably less than the cost of hunting down the failure in real use.

That said, we have to balance our efforts wisely, since errors are likely to occur at any level of the 
framework... I feel it's best if tests are spread between all levels, starting with unit tests to test 
elementary functionality, more complex test to test compound functionality, and higher and higher levels of 
integration tests ending up with integration tests for entire applications.

What we hope will happen is that on encountering a test failure at any level, we'll be able to rule out 
failures at any lower level by checking for failures in simpler tests first. What happens when finding a 
panel full of randomly failing tests (as I frequently do this week!) is to home in on fixing the tests which 
fail at the lowest level first, and then progressively work up to higher level test failures - very often 
most if not all of these will be resolved by fixing the lower level tests. We expect there to be a constant 
effort at characterising the underlying reason for failures in higher level tests, if these can actually be 
explained in terms of a framework failure for which there is no existing lower level test, and writing that 
corresponding lower level test - although this is all naturally controlled by what time budget we have 
available overall....

So what do I think this means for this particular dependency?

In terms of point i) - is there any reasonable chance that our tests may fail less often - I think there 
almost certainly isn't - changes in code which had previously passing tests are overwhelmingly likely to 
make tests fail, rather than the other way round. Also, the parts of the framework which jqUnit now depends 
on are extremely simple and already have good tests - these only consist of a few simple functional 
programming utilities and fluid.registerNamespace.
The IoC testing framework is another matter - but I think we already conceded that it would be impossible to 
have an IoC testing framework that didn't depend on the IoC infrastructure. But under the terms of point 
ii), again, we have moderately good tests for IoC itself, which so far do NOT themselves depend on IoC - 
that is, our IoC tests depend on plain jqUnit, which in turn depends only on the simple parts of Fluid.js, 
which themselves have adequate tests, etc.

So my assessment is that the "onion of testing" which we necessarily depend on in case ii) isn't dangerously 
prejudiced by this new dependency. We can always stand to have better tests, but the area we most urgently 
need them isn't here - it's in the area of having more plain IoC tests - but this can wait until the 
implementation stabilises some more and we have a firm idea of what the IoC system is meant to do in each 
situation.

On 21/01/2013 09:56, Justin Obara wrote:
>>> 	• Will this make our tests more brittle, since changes to the framework could potentially affect it?
>>
>> Can you elaborate on the kinds of brittleness that you're concerned about? Perhaps some examples?
>> How, specifically, is it a big issue?
>
> I'm not sure I have any specific concrete examples. I probably don't know enough about the inner workings of either the framework or the jqUnit extension to really be able to do that. I can try to make some hypothetical ones, hopefully they  may lead to further thoughts and start to tease out some actual potential issues, or prove that the fear is unwarranted.
>
> I suppose the everything is broken cases should be pretty obvious to someone making a change to the framework, so those should be less of an issue. I'd be more worried about the more subtle ones that may cause some tests to appear to work or not just because of some framework issue. Maybe a change in the framework might affect the jqunit extensions sequencing of tests, merging of options, or instantiation of components, component creation order or something else.
>
> The obvious worry is that because the debugging is so difficult right now, if something does arise, it will be hard to track. At the end of the day I just want to make sure that we have thought things through and have our bases covered so that we can prevent our tests from becoming unreliable. If we can do that, I'm fine with this dependency.
>
> Thanks
> Justin
>