asyncTest

is an extension to SDK ceylon.test module with following capabilities:

  • testing asynchronous multithread code
  • executing tests concurrently or sequentially
  • value- and type- parameterized testing
  • organizing complex test conditions into a one flexible expression with matchers
  • initialization and disposing with either functions or test rules
  • test execution control with test runners
  • conditional test execution
  • multi-reporting: several failures or successes can be reported for a one particular test execution (test function), each report is represented as test variant and might be marked with String title
  • reporting test results using charts (or plots)

The extension is based on:

  • AsyncTestExecutor - the test executor which satisfies ceylon.test.engine.spi::TestExecutor and provides an interaction with ceylon.test module.
  • AsyncTestContext which provides an interaction of the test function with the test framework.
  • AsyncPrePostContext which provides an interaction of initialization / disposing logic with the test framework.
  • herd.asynctest.rule package which contains rules used for test initialization / disposing and for modification of the test behaviour.
  • herd.asynctest.runner package which provides a control over a test function execution.
  • herd.asynctest.match package which contains matching API.
  • herd.asynctest.chart package which is intended to organize reporting with charts.

It is recommended to read documentation on module ceylon.test before starting with asyncTest.

The source code and examples are available at GitHub


Content

  1. Test procedure.
  2. Test initialization and disposing.
  3. Instantiation the test container class.
  4. Test suites and concurrent execution.
  5. Test execution cycle.
  6. Value- and type- parameterized testing.
  7. Test runners.
  8. Matchers.
  9. Time out.
  10. Retry test.
  11. Conditional execution.
  12. Reporting test results using charts.

Test procedure

  1. Declare test function, which accepts AsyncTestContext as the first argument:
        test async void doTesting(AsyncTestContext context) {...}
    
    Mark test function or upper level container with ceylon.test::test annotation.
    Mark test function or upper level container with async() annotation.
  2. Code test function according to AsyncTestContext specification:
  3. Run test in IDE or command line using Ceylon test tool.

Notes

Both modules ceylon.test and herd.asynctest have to be imported to run testing.

Test executor blocks the thread until AsyncMessageContext.complete() is called. It means test function has to notify completion to continue with other testing and to report results.

async() annotation is exactly the same as testExecutor(`class AsyncTestExecutor`) annotation.

Test functions

Any function or method marked with test and async annotation.

If function (or upper-level container) is not marked with async annotation it is executed with default ceylon.test executor.

The test function arguments:

  • If no arguments or function takes arguments according to a given test variant provider (see, Value- and type- parameterized testing below) then the tet function is executed in synchronous mode and may report on failures using assertions or throwing some exception.
  • If function takes the first argument of AsyncTestContext type and the other arguments according to a given test variant provider (see, Value- and type- parameterized testing below) then it is executed asynchronously and may report failures, successes and completion using AsyncTestContext.

If a number of test functions are declared within some class just a one instance of the class is used for the overall test runcycle. This opens way to have some test interrelations. Please, remember best-practices say the tests have to be independent.


Test initialization and disposing

  • Top-level functions or methods marked with ceylon.test::beforeTestRun are executed once before starting all tests in its scope (package for top-level and class for methods).
  • Top-level functions or methods marked with ceylon.test::beforeTest are executed each time before executing each test in its scope (package for top-level and class for methods).
  • Top-level functions or methods marked with ceylon.test::afterTestRun are executed once after completing all tests in its scope (package for top-level and class for methods).
  • Top-level functions or methods marked with ceylon.test::afterTest are executed each time after each test in its scope (package for top-level and class for methods) is completed.

All before and after callbacks are called prepost functions below.

Prepost functions may take arguments (excepting a top-level function marked with ceylon.test::beforeTestRun and ceylon.test::afterTestRun which may take only empty argument list):

If prepost function reports on failure (throwing exception or calling AsyncPrePostContext.abort() method) the test procedure for every test function in the scope (package for top-level and class for methods) is interrupted and failure is reported.

All prepost functions are called disregard the failure reporting.

Notes:

  • There is no specific order the prepost functions are executed in.
  • If some initializer reports on failure the test is skipped.
  • Every prepost function is always executed regardless failure reporting since a right to be disposed has to be provided for each.
  • Top-level functions marked with ceylon.test::beforeTestRun or ceylon.test::afterTestRun have to take no arguments! While methods may take (see, below).
  • Test executor blocks current thread until prepost function calls AsyncPrePostContext.proceed() or AsyncPrePostContext.abort().
  • Prepost methods have to be shared! Top-level prepost functions may not be shared.
  • Inherited prepost methods are executed also.
  • Top-level prepost functions are not applicable to methods!

Test initialization and disposing example

    class StarshipTest() {

        // called just a once before all tests ae executed
        shared beforeTestRun void createUniverse(AsyncPrePostContext context) { 
            ...
            context.proceed();
        }

        // called just a once after all tests are completed
        shared afterTestRun void destroyUniverse(AsyncPrePostContext context) {
            ...
            context.proceed();
        } 

        // called before each test function is executed
        shared beforeTest void init() => starship.chargePhasers();

        // called after each test function is completed
        shared afterTest void dispose() => starship.shutdownSystems();

        test async testPhasersAiming() { ... }
        test async testPhasersFire(AsynctestContext context) { ... context.complete(); }
}

Prepost function arguments

arguments() annotation is intended to provide arguments for a one-shot function like test prepost functions are.
The annotation takes a one argument - declaration of top-level function or value which returns a tupple with invoked function arguments:

    [Integer] testUniverseSize = [1K];

    arguments(`value testUniverseSize`)
    beforeTestRun void initializeStarshipTestSync(Integer universeSize) { ... }

    arguments(`value testUniverseSize`)
    beforeTestRun void initializeStarshipTestAsync(AsyncPrePostContext context, Integer universeSize) { ... }

In the above example both functions initializeStarshipTestSync and initializeStarshipTestAsync will be called with argument provided by testUniverseSize. So, for both sync and async versions arguments provider has to return the same arguments list. But async version will additionally be provided with AsyncPrePostContext which has to be the first argument.

arguments() annotation is not applicable to test functions. parameterized() annotation is aimed to perform parameterized testing, see, section Value- and type- parameterized testing below.

Test rules

Test rules provide more flexible way for test initialization / disposing and for modification the test behaviour. See, details in herd.asynctest.rule package.

Order in which prepost functions are invoked is described in Test execution cycle section.


Instantiation the test container class

Sometimes instantiation and initialization of the test container class requires some complex logic or some asynchronous operations. If the class declaration is marked with factory() annotation a given factory function is used to instantiate the class.

If no factory function is provided instantiation is done using metamodel staff calling class initializer with arguments provided with arguments() annotation or without arguments if the annotation is missed.

Just a one instance of the test class is used for the overall test runcycle it may cause several misalignments:

  1. Test interrelation. Please, remember best-practices say the tests have to be independent.
  2. Test isolation. If test class has some mutable properties then a test may get mutated state from previous run but not purely initialized property! Always use test rules or before \ after callbacks for such properties.

From the other side having only one test class instance during overall test runcycle helps to orginaze initialization logic in manner more suitable for asynchronous code testing.


Test suites and concurrent execution

Test functions are collected into suites, which are defined by:

  • ClassDeclaration for methods.
  • Package for top-level functions.

So, each suite contains all top-level test functions in a given package or all test methods of a given class.

This is implicit suite organization!

All test and prepost functions within the given suite are always executed sequentially via a one thread (note: any test function is free to run any number of threads it needs).

By default the suites are executed sequentially also. In order to execute suites concurrently (each suite in separated thread) the suite or upper-level container is to be marked with concurrent() annotation.

Thread pool with fixed number of threads equals to number of available processors (cores) is used to execute tests in concurrent mode.
If package or module is marked with concurrent() all suites it contains are executed in concurrent mode.

For example:

  • A package has three test classes.
  • Two of them are annotated with concurrent() and third is not annotated.
  • Two marked suites are executed via thread pool. Each suite in separated thread if number of available cores admits. But all test functions in the given suite are executed sequentially via a one thread.
  • After completion the test of the first two suites the third one is executed.

Top-level prepost functions are not applicable to methods!


Test execution cycle

  1. Test suite initialization:
  2. Execution of the each test function in the scope for each given variant:
    • Test execution inititalization:
    • Test function invoking:
      • Test function invoking with arguments provided by current test variant.
      • TestStatement application.
    • Test execution disposing:
    • Repeat 2 for the next test function in the scope (package for top-level functions or cla for the methods)
  3. Test suite disposing:

Value- and type- parameterized testing

In order to perform parameterized testing the test function has to be marked with annotation which supports TestVariantProvider interface. The interface has just a one method - variants() which has to provide TestVariantEnumerator. The enumerator produces a stream of the TestVariant's and is iterated just a once. The test will be performed using all variants the enumerator produces.

The enumerator may return test variants lazily, dynamicaly or even non-determenisticaly.
Each TestVariant contains a list of generic type parameters and a list of function arguments.

parameterized() annotation satisfies TestVariantProvider interface and provides parameterized testing based on collection of test variants.

Custom parameterization:

  1. Implement TestVariantEnumerator interface:

    class MyTestVariantEnumerator(...) satisfies TestVariantEnumerator {
        shared actual TestVariant|Finished current => ...;
    
        shared actual void moveNext(TestVariantResult result) {
            if (testToBeCompleted) {
                // set `current` to `finished`
            } else {
                // set `current` to test variant to be tested next
            }
        }
    }
    
  2. Make an annotation which satisfies TestVariantProvider interface:

    shared final annotation class MyParameterizedAnnotation(...)
        satisfies SequencedAnnotation<MyParameterizedAnnotation, FunctionDeclaration>&TestVariantProvider
    {
        shared actual TestVariantEnumerator variants() => MyTestVariantEnumerator(...);
    }
    
    shared annotation MyParameterizedAnnotation myParameterized(...) => MyParameterizedAnnotation(...);
    
  3. Mark test function with created annotation:

    myParameterized(...) void myTest(...) {...}
    

Test runners

Test runners provide a way to control test function execution.
Simply, test runner takes a test function and invokes it. But it may, for example, execute it several times or execute simultaneously in several threads, or modify the function report or something else.
For the details, see, herd.asynctest.runner package.


Matchers

Matchers are intended to organize complex test conditions into a one flexible expression.
Each matcher is represented as requirements specification and verification method which identifies if submitted test value satisfies this specification or not. Matchers may be combined using logical operators.

Details of matching API are described in herd.asynctest.match package.


Time out

timeout() annotation indicates that if test has not been completed during some time it has to be interrupted.
Annotation applied at class, package or module level acts for each function within the scope. Lower-level declaration overrides definitions of upper-level. So, if both function and class annotated with timeout() the function annotation is applied.

timeout() annotation is applicable to every function executed during the test: test function, before or after callbacks, test rule or factory.

Example, function doMyTest will be interrupted if not completed during 1 second:

    timeout( 1K ) test async void doMyTest(...) {...}

Retry test

If overall test runcycle (i.e. before callbacks - test function - test statements - after callbacks) has to be retryed for each given test variant the retry() annotation may be applied to the test function. The annotation forces test framework to retry the overall test execution cycle according to a given repeat strategy.


Conditional execution

Test conditions can be specified via custom annotation which satisfies ceylon.test.engine.spi::TestCondition interface.
Any number of test conditions can be specified at function, class, package or module level.
All conditions at every level are evaluated before test execution started and if some conditions are not met (are unsuccessfull) the test is skipped and all rejection reasons are reported.

For an example, see, ceylon.test::ignore annotation.

Conditions are evaluation up to the first unsatisfied condition. So, there is no guarantee for a condition to be evaluated.


Reporting test results using charts

Chart is simply a set of plots, where each plot is a sequence of 2D points.
Test results can be represented and reported with charts using stuff provided by herd.asynctest.chart package.


Platform: Java
By: Lis
License: The MIT License (MIT) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Packages
herd.asynctest

Run, test, run!

herd.asynctest.chart

Chart is a set of plots, where each plot is a sequence of 2D points, Plot.
Chart may have title, names of each axis and…

herd.asynctest.match

Matchers are intended to organize complex test conditions into a one flexible expression.
Each matcher is requirements…

herd.asynctest.rule

Test rule provides a way to perform initialization / disposing before / after test execution and to modify test…

herd.asynctest.runner

Test runner provides a control over a test function execution.

Dependencies
ceylon.collection1.3.1
ceylon.file1.3.1
ceylon.test1.3.1
java.base8

Run, test, run!

By: Lis
Annotations
argumentsshared ArgumentsAnnotation arguments(FunctionOrValueDeclaration source)

Indicates that test container class or test prepost function have to be instantiated or called using arguments provided by this annotation, see ArgumentsAnnotation.argumentList().

Example:

    [Hobbit] who => [bilbo];
    {[[], [Dwarf]]*} dwarves => {[[], [fili]], [[], [kili]], [[], [balin]], [[], [dwalin]]...};

    arguments(`value who`)
    class HobbitTester(Hobbit hobbit) {
        shared test async
        parameterized(`value dwarves`)
        void thereAndBackAgain(AsyncTestContext context, Dwarf dwarf) {
            context.assertTrue(hobbit.thereAndBackAgain(dwarf)...);
            context.complete();
        }
    }
Parameters:
  • source

    The source function or value declaration which has to take no arguments and has to return a stream of values. The source may be either top-level or tested class shared member.

By: Lis
Since 0.5.0
asyncshared TestExecutorAnnotation async()

The same as testExecutor(`class AsyncTestExecutor`)

By: Lis
Since 0.6.0
concurrentshared ConcurrentAnnotation concurrent()

Indicates that all test suites (each suite contains all top-level test functions in the given package or all test methods of the given class) contained in the marked container have to be executed in concurrent mode.
The functions within each suite are executed sequentially in a one thread while the suites are executed concurrently using thread pool of number of available cores size.

Thread pool with fixed number of threads equals to number of available processors (cores) is used to execute tests in concurrent mode.

By: Lis
Since 0.6.0
factoryshared FactoryAnnotation factory(FunctionDeclaration factoryFunction)

Indicates that class has to be instantiated using a given factory function.
factory() annotation takes declaration of top-level factory function.

Factory function has to take no arguments or take first argument of AsyncFactoryContext type.
If factory function takes AsyncFactoryContext as first argument it is executed asynchronously and may fill the context with instantiated object using AsyncFactoryContext.fill() or may report on error using AsyncFactoryContext.abort(). Test executor blocks the current thread until one of AsyncFactoryContext.fill() or AsyncFactoryContext.abort() is called.
Otherwise factory function doesn't take AsyncFactoryContext as first argument. It is executed synchronously and has to return instantiated non-optional object or throw an error.

Example of synchronous instantiation:

    StarshipTest createStarshipTest() => StarshipTest(universeSize);

    factory(`function createStarshipTest`)
    class StarshipTest(Integer universeSize) {
        ...
    }       

Example of asynchronous instantiation:

    StarshipTest createStarshipTest(AsyncFactoryContext context) {
        context.fill(StarshipTest(universeSize));
    }

    factory(`function createStarshipTest`)
    class StarshipTest(Integer universeSize) {
        ...
    }       

Pay attention:
Asynchronous version has to call AsyncFactoryContext.fill() or AsyncFactoryContext.abort().
Synchronous version has to return non-optional object or throw.

Parameters:
  • factoryFunction

    Function used to instantiate anotated class.
    Has to take no arguments or just a one argument of AsyncFactoryContext type.
    Has to return an instance of the looking class.

By: Lis
Since 0.6.0
parameterizedshared ParameterizedAnnotation parameterized(FunctionOrValueDeclaration source, Integer maxFailedVariants = -1)

Indicates that generic (with possibly empty generic parameter list) test function has to be executed with given test variants.

The annotation provides parameterized testing based on collection of test variants. It takes two arguments:

  1. Declaration of function or value which returns a collection of test variants {TestVariant*}.
  2. Number of failed variants to stop testing. Default is -1 which means no limit.

The test will be performed using all test variants returned by the given stream or while total number of failed variants not exceeds specified limit.

parameterized() annotation may occur multiple times at a given test function.
The variants source may be either top-level or tested class shared member.

Example:

    Value identity<Value>(Value argument) => argument;

    {TestVariant*} identityArgs => {
        TestVariant([`String`], ["stringIdentity"]),
        TestVariant([`Integer`], [1]),
        TestVariant([`Float`], [1.0])
    };

    shared test async
    parameterized(`value identityArgs`)
    void testIdentity<Value>(AsyncTestContext context, Value arg)
        given Value satisfies Object
    {
        context.assertThat(identity<Value>(arg), EqualObjects<Value>(arg), "", true );
        context.complete();
    }

In the above example the function testIdentity will be called 3 times:

  •   testIdentity<String>(context, "stringIdentity");
    
  •   testIdentity<Integer>(context, 1);
    
  •   testIdentity<Float>(context, 1.0);
    

In order to run test with conventional (non-generic function) type parameters list has to be empty:

    [Hobbit] who => [bilbo];
    {TestVariant*} dwarves => {
        TestVariant([], [fili]),
        TestVariant([], [kili]),
        TestVariant([], [balin],
        TestVariant([], [dwalin]),
        ...
    };

    arguments(`value who`)
    class HobbitTester(Hobbit hobbit) {
        shared test async
        parameterized(`value dwarves`, 2)
        void thereAndBackAgain(AsyncTestContext context, Dwarf dwarf) {
            context.assertTrue(hobbit.thereAndBackAgain(dwarf)...);
            context.complete();
        }
    }

In this example class HobbitTester is instantiated once with argument provided by value who and method thereAndBackAgain is called multiply times according to size of dwarves stream. According to second argument of parameterized annotation the test will be stopped if two different invoking of thereAndBackAgain with two different arguments report failure.

Parameters:
  • source

    The source function or value declaration which has to take no arguments and has to return a stream of test variants: {TestVariant*}.
    The source may be either top-level or tested class shared member.

  • maxFailedVariants = -1

    Maximum number of failed variants before stop. Unlimited if <= 0.

By: Lis
See also TestVariant
Since 0.6.0
retryshared RetryAnnotation retry(FunctionOrValueDeclaration source)

Indicates that test function or all functions within test container have to be tested using the given RepeatStrategy (extracted from source).

Overall execution cycle including before, after and testRule callbacks are repeated!

If you need to repeat just test function execution, look on RepeatRunner.

Parameters:
  • source

    Repeat strategy source which has to take no arguments and has to return instance of RepeatStrategy type. Either top-level function or value or test function container method or attribute.

By: Lis
Since 0.6.0
timeoutshared TimeoutAnnotation timeout(Integer timeoutMilliseconds)

Indicates that if test function execution takes more than timeoutMilliseconds the test has to be interrupted.
The annotation is applied to any function called using AsyncTestExecutor: prepost functions, test rules, factory and test functions.

Parameters:
  • timeoutMilliseconds

    Timeout in milliseconds.

By: Lis
Since 0.6.0
ArgumentsAnnotationshared final ArgumentsAnnotation

Annotation class for arguments().

ConcurrentAnnotationshared final ConcurrentAnnotation

Annotation class for concurrent().

FactoryAnnotationshared final FactoryAnnotation

Annotation class for factory().

ParameterizedAnnotationshared final ParameterizedAnnotation

Annotation class for parameterized().

RetryAnnotationshared final RetryAnnotation

Annotation class for retry().

TimeoutAnnotationshared final TimeoutAnnotation

Annotation class for timeout().

Interfaces
AsyncFactoryContextshared AsyncFactoryContext

Factory submited to factory function in order to perform asynchronous instantiation in initialization of test class. When object is instantiated it has to be passed to AsyncFactoryContext.fill() method. If some error has been occurred during instantiation and may be reported using AsyncFactoryContext.abort() method.

The test executor blocks the current thread until one of AsyncFactoryContext.fill() or AsyncFactoryContext.abort() method is called.

AsyncMessageContextshared AsyncMessageContext

Base interface to push test messages to.
The interface is mainly used by test runners AsyncTestRunner.
Test function receives derived interface AsyncTestContext.

AsyncPrePostContextshared AsyncPrePostContext

Allows prepost function functions to interract with test executor.

Prepost function has to call AsyncPrePostContext.proceed() or AsyncPrePostContext.abort() when initialization or disposing is completed or errored, correspondently.
The test executor blocks execution thread until AsyncPrePostContext.proceed() or AsyncPrePostContext.abort() is called.

See details about test initialization / disposing in herd.asynctest.

AsyncTestContextshared AsyncTestContext

Provides interaction with asynchronous test executor.

General test procedure within test function is:

  1. Perform the testing. Notify test executor on failures or successes. Several notifications are allowed. Each failure or success notification is represented as test variant.
  2. Notify test executor on test procedure completion (call AsyncMessageContext.complete()). This step is nesseccary to continue testing with next execution since test executor blocks execution thread until AsyncMessageContext.complete() is called.

Example of tested function:

test async
void doTesting(AsyncTestContext context) {
    // perform test procedure and notify about fails, if no fails notified test is considered successfull
    context.fail(Exception("exception"), "some exception");
    context.fail(AssertionError( "assert"), "some assert");
    context.assertThat(true, IsFalse(), "to be `false`");

    // complete testing
    context.complete("title which is added to test variant name only if test is succeeded");
}

It is not required to notify with success, if test function doesn't notify on failure the test is considered as successfull.

TestVariantEnumeratorshared TestVariantEnumerator

Enumerates test variants.

TestVariantProvidershared TestVariantProvider

Provides enumerator for test variants.
Test execution context looks for annotations of the test function which support the interface and performs testing according to provided variants. As example, see parameterized().

Classes
AsyncTestExecutorshared AsyncTestExecutor

Async test executor.

Capabilities

  • testing asynchronous multithread code
  • running test functions concurrently or sequentialy, see concurrent() annotation and herd.asynctest
  • multi-reporting: several failures or successes can be reported for a one test execution, each report is represented as test variant and might be marked with String title
  • value- and type- parameterized testing with a set of function arguments, see parameterized() for details
  • conditional execution with annotations satisfied ceylon.test.engine.spi::TestCondition interface

In order to utilize this executor capabilities test function has to accept AsyncTestContext as the first argument:

    test async
    void doTesting(AsyncTestContext context) {...}  

Test function may have more arguments if it is annotated with parameterized() annotation.

Running

To run the test using this executor apply async() annotation at function, class, package or module level (alternatively, ceylon.test::testExecutor annotation with `class AsyncTestExecutor` argument is to be applied).
Following procedure is as usual for SDK ceylon.test module - mark tested functions with ceylon.test::test annotation and run test in IDE or command line.

Test logic

When test function taking AsyncTestContext as first argument is executed it is expected the function will do following steps:

  1. Performing the test, reporting on failures via AsyncTestContext. Several error or success reports are allowed. Each failure or success report is represented as test variant.
  2. Notifying test executor on test procedure completion - AsyncMessageContext.complete(). This step is nesseccary to continue testing with next execution since test executor blocks execution thread until AsyncMessageContext.complete() is called.

TestInfoshared final TestInfo

Test info represents and information on currently running test variant.

TestOutputshared final TestOutput

Represents a one report.

TestVariantshared TestVariant

Represents test variant, i.e. test function generic type parameters and arguments.

TestVariantResultshared final TestVariantResult

Results of a one test function run with some arguments.

VariantResultBuildershared VariantResultBuilder

Builds test variant result from several test outputs.

Exceptions
FactoryReturnsNothingshared FactoryReturnsNothing

Exception errored when no object instantiated no error throwed by factory.

TimeOutExceptionshared TimeOutException

Exception which shows that time out has been reached.