asyncTest

is an extension to SDK ceylon.test module with following capabilities:

  • testing asynchronous multithread code
  • executing tests concurrently or sequentially
  • value- and type- parameterized testing
  • organizing complex test conditions into a one flexible expression with matchers
  • initialization and disposing with either functions or test rules
  • test execution control with test runners
  • conditional test execution
  • multi-reporting: several failures or successes can be reported for a one particular test execution (test function), each report is represented as test variant and might be marked with String title
  • reporting test results using charts (or plots)
  • benchmarks

The extension is based on:

It is recommended to read documentation on module ceylon.test before starting with asyncTest.

The source code and examples are available at GitHub


Content

  1. Test procedure.
  2. Test initialization and disposing.
  3. Instantiation the test container class.
  4. Test suites and concurrent execution.
  5. Test execution cycle.
  6. Value- and type- parameterized testing.
  7. Test runners.
  8. Matchers.
  9. Time out.
  10. Retry test.
  11. Conditional execution.
  12. Benchmarking.
  13. Reporting test results using charts.

Test procedure

  1. Declare test function, which accepts AsyncTestContext as the first argument:
        test async void doTesting(AsyncTestContext context) {...}
    
    Mark test function or upper level container with ceylon.test::test annotation.
    Mark test function or upper level container with async() annotation.
  2. Code test function according to AsyncTestContext specification:
  3. Run test in IDE or command line using Ceylon test tool.

Notes

Both modules ceylon.test and herd.asynctest have to be imported to run testing.

Test executor blocks the thread until AsyncTestContext.complete() is called. It means test function has to notify completion to continue with other testing and to report results.

async() annotation is exactly the same as testExecutor(`class AsyncTestExecutor`) annotation.

Test functions

Any function or method marked with test and async annotation.

If function (or upper-level container) is not marked with async annotation it is executed with default ceylon.test executor.

The test function arguments:

  • If no arguments or function takes arguments according to a given test variant provider (see, Value- and type- parameterized testing below) then the tet function is executed in synchronous mode and may report on failures using assertions or throwing some exception.
  • If function takes the first argument of AsyncTestContext type and the other arguments according to a given test variant provider (see, Value- and type- parameterized testing below) then it is executed asynchronously and may report failures, successes and completion using AsyncTestContext.

If a number of test functions are declared within some class just a one instance of the class is used for the overall test runcycle. This opens way to have some test interrelations. Please, remember best-practices say the tests have to be independent.


Test initialization and disposing

  • Top-level functions or methods marked with ceylon.test::beforeTestRun are executed once before starting all tests in its scope (package for top-level and class for methods).
  • Top-level functions or methods marked with ceylon.test::beforeTest are executed each time before executing each test in its scope (package for top-level and class for methods).
  • Top-level functions or methods marked with ceylon.test::afterTestRun are executed once after completing all tests in its scope (package for top-level and class for methods).
  • Top-level functions or methods marked with ceylon.test::afterTest are executed each time after each test in its scope (package for top-level and class for methods) is completed.

All before and after callbacks are called prepost functions below.

Prepost functions may take arguments (excepting a top-level function marked with ceylon.test::beforeTestRun and ceylon.test::afterTestRun which may take only empty argument list):

If prepost function reports on failure (throwing exception or calling AsyncPrePostContext.abort() method) the test procedure for every test function in the scope (package for top-level and class for methods) is interrupted and failure is reported.

All prepost functions are called disregard the failure reporting.

Notes:

  • There is no specific order the prepost functions are executed in.
  • If some initializer reports on failure the test is skipped.
  • Every prepost function is always executed regardless failure reporting since a right to be disposed has to be provided for each.
  • Top-level functions marked with ceylon.test::beforeTestRun or ceylon.test::afterTestRun have to take no arguments! While methods may take (see, below).
  • Test executor blocks current thread until prepost function calls AsyncPrePostContext.proceed() or AsyncPrePostContext.abort().
  • Prepost methods have to be shared! Top-level prepost functions may not be shared.
  • Inherited prepost methods are executed also.
  • Top-level prepost functions are not applicable to methods!

Test initialization and disposing example

    class StarshipTest() {

        // called just a once before all tests ae executed
        shared beforeTestRun void createUniverse(AsyncPrePostContext context) { 
            ...
            context.proceed();
        }

        // called just a once after all tests are completed
        shared afterTestRun void destroyUniverse(AsyncPrePostContext context) {
            ...
            context.proceed();
        } 

        // called before each test function is executed
        shared beforeTest void init() => starship.chargePhasers();

        // called after each test function is completed
        shared afterTest void dispose() => starship.shutdownSystems();

        test async testPhasersAiming() { ... }
        test async testPhasersFire(AsynctestContext context) { ... context.complete(); }
}

Prepost function arguments

arguments() annotation is intended to provide arguments for a one-shot function like test prepost functions are.
The annotation takes a one argument - declaration of top-level function or value which returns a tupple with invoked function arguments:

    [Integer] testUniverseSize = [1K];

    arguments(`value testUniverseSize`)
    beforeTestRun void initializeStarshipTestSync(Integer universeSize) { ... }

    arguments(`value testUniverseSize`)
    beforeTestRun void initializeStarshipTestAsync(AsyncPrePostContext context, Integer universeSize) { ... }

In the above example both functions initializeStarshipTestSync and initializeStarshipTestAsync will be called with argument provided by testUniverseSize. So, for both sync and async versions arguments provider has to return the same arguments list. But async version will additionally be provided with AsyncPrePostContext which has to be the first argument.

arguments() annotation is not applicable to test functions. See package herd.asynctest.parameterization for the parameterized testing.

Test rules

Test rules provide more flexible way for test initialization / disposing and for modification the test behaviour. See, details in herd.asynctest.rule package.

Order in which prepost functions are invoked is described in Test execution cycle section.


Instantiation the test container class

Sometimes instantiation and initialization of the test container class requires some complex logic or some asynchronous operations. If the class declaration is marked with factory() annotation a given factory function is used to instantiate the class.

If no factory function is provided instantiation is done using metamodel staff calling class initializer with arguments provided with arguments() annotation or without arguments if the annotation is omitted.

Just a one instance of the test class is used for the overall test runcycle it may cause several misalignments:

  1. Test interrelation. Please, remember best-practices say the tests have to be independent.
  2. Test isolation. If test class has some mutable properties then a test may get mutated state from previous run but not purely initialized property! Always use test rules or before \ after callbacks for such properties.

From the other side having only one test class instance during overall test runcycle helps to orginaze initialization logic in manner more suitable for asynchronous code testing.


Test suites and concurrent execution

Test functions are collected into suites, which are defined by:

  • ClassDeclaration for methods.
  • Package for top-level functions.

So, each suite contains all top-level test functions in a given package or all test methods of a given class.

This is implicit suite organization!

All test and prepost functions within the given suite are always executed sequentially via a one thread (note: any test function is free to run any number of threads it needs).

By default the suites are executed sequentially also. In order to execute suites concurrently (each suite in separated thread) the suite or upper-level container is to be marked with concurrent() annotation.

Thread pool with fixed number of threads equals to number of available processors (cores) is used to execute tests in concurrent mode.
If package or module is marked with concurrent() all suites it contains are executed in concurrent mode.

For example:

  • A package has three test classes.
  • Two of them are annotated with concurrent() and third is not.
  • Two marked suites are executed via thread pool. Each suite in separated thread if number of available cores admits. But all test functions in the given suite are executed sequentially via a one thread.
  • After completion the test of the first two suites the third one is executed.

Top-level prepost functions are not applicable to methods!


Test execution cycle

  1. Test suite initialization:
  2. Execution of the each test function in the scope for each given variant:
    • Test execution inititalization:
    • Test function invoking:
      • Test function invoking with arguments provided by current test variant.
      • TestStatement application.
    • Test execution disposing:
    • Repeat 2 for the next test function in the scope (package for top-level functions or cla for the methods)
  3. Test suite disposing:

Value- and type- parameterized testing

Parameterized testing is performed using annotations to specify a generator of the test variants. Basically, each test variant is a list of test function type parameters and arguments.
Package herd.asynctest.parameterization provides parameterized testing capability with a number of ways to generate test variants.


Test runners

Test runners provide a way to control test function execution.
Simply, test runner takes a test function and invokes it. But it may, for example, execute it several times or execute simultaneously in several threads, or modify the function report or something else.
For the details, see, herd.asynctest.runner package.


Matchers

Matchers are intended to organize complex test conditions into a one flexible expression.
Each matcher is represented as requirements specification and verification method which identifies if submitted test value satisfies this specification or not. Matchers may be combined using logical operators.

Details of matching API are described in herd.asynctest.match package.


Time out

timeout() annotation indicates that if test has not been completed during some time it has to be interrupted.
Annotation applied at class, package or module level acts for each function within the scope. Lower-level declaration overrides definitions of upper-level. So, if both function and class annotated with timeout() the function annotation is applied.

timeout() annotation is applicable to every function executed during the test: test function, before or after callbacks, test rule or factory.

Example, function doMyTest will be interrupted if not completed during 1 second:

    timeout(1K) test async void doMyTest(...) {...}

Retry test

If overall test runcycle (i.e. before callbacks - test function - test statements - after callbacks) has to be retryed for each given test variant the retry() annotation may be applied to the test function. The annotation forces test framework to retry the overall test execution cycle according to a given repeat strategy.


Conditional execution

Test conditions can be specified via custom annotation which satisfies ceylon.test.engine.spi::TestCondition interface.
Any number of test conditions can be specified at function, class, package or module level.
All conditions at every level are evaluated before test execution started and if some conditions are not met (are unsuccessfull) the test is skipped and all rejection reasons are reported.

For an example, see ceylon.test::ignore annotation.

Conditions are evaluation up to the first unsatisfied condition. So, there is no guarantee for a condition to be evaluated.


Benchmarking

herd.asynctest.benchmark package contains library to perform benchmark testing.


Reporting test results using charts

Chart is simply a set of plots, where each plot is a sequence of 2D points.
Test results can be represented and reported with charts using herd.asynctest.chart package.


Platform: Java
By: Lis
License: The MIT License (MIT) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Packages
herd.asynctest

Run, test, run!

herd.asynctest.benchmark

Library to run benchmarks.

herd.asynctest.chart

Chart is a set of plots, where each plot is a sequence of 2D points.

herd.asynctest.match

Matchers are intended to organize complex test conditions into a one flexible expression.

herd.asynctest.parameterization

Value- and type- parameterized testing.

herd.asynctest.rule

Test rule provides a way to perform test initialization/disposing and a way to modify test behaviour.

herd.asynctest.runner

Test runner provides a control over a test function execution.

Dependencies
ceylon.collection1.3.2
ceylon.file1.3.2
ceylon.test1.3.2
java.base8
java.management8

Run, test, run!

By: Lis
Annotations
argumentsshared ArgumentsAnnotation arguments(FunctionOrValueDeclaration source)

Provides arguments for a one-shot functions. See ArgumentsAnnotation for details.

Parameters:
  • source

    The source function or value declaration which has to return a stream of values. The source may be either top-level or tested class shared member.

By: Lis
Since 0.5.0
asyncshared TestExecutorAnnotation async()

The same as testExecutor(`class AsyncTestExecutor`).

By: Lis
Since 0.6.0
concurrentshared ConcurrentAnnotation concurrent()

Indicates that all test suites contained in the marked container have to be executed in concurrent mode.

By: Lis
Since 0.6.0
factoryshared FactoryAnnotation factory(FunctionDeclaration factoryFunction)

Provides factory function for an object instantiation. See FactoryAnnotation for the details.

Parameters:
  • factoryFunction

    Function used to instantiate anotated class.
    See FactoryAnnotation for the requirements to the factory function arguments.
    Has to return an instance of the looking class.

By: Lis
Since 0.6.0
retryshared RetryAnnotation retry(FunctionOrValueDeclaration source)

Provides retry trategy for the tet function execution. See details in RetryAnnotation.

Parameters:
  • source

    Repeat strategy source which has to take no arguments and has to return instance of RepeatStrategy type. Either top-level function or value or test function container method or attribute.

By: Lis
Since 0.6.0
timeoutshared TimeoutAnnotation timeout(Integer timeoutMilliseconds)

Indicates that if test function execution takes more than timeoutMilliseconds the test has to be interrupted.

Parameters:
  • timeoutMilliseconds

    Timeout in milliseconds.

By: Lis
Since 0.6.0
ArgumentsAnnotationshared final ArgumentsAnnotation

Indicates that test container class or test prepost function have to be instantiated or called using arguments provided by this annotation, see ArgumentsAnnotation.

Example:

    [Hobbit] who => [bilbo];
    {[[], [Dwarf]]*} dwarves => {[[], [fili]], [[], [kili]], [[], [balin]], [[], [dwalin]]...};

    arguments(`value who`)
    class HobbitTester(Hobbit hobbit) {
        shared test async
        parameterized(`value dwarves`)
        void thereAndBackAgain(AsyncTestContext context, Dwarf dwarf) {
            context.assertTrue(hobbit.thereAndBackAgain(dwarf)...);
            context.complete();
        }
    }

Source function may also be marked with arguments annotation.

ConcurrentAnnotationshared final ConcurrentAnnotation

Indicates that all test suites (each suite contains all top-level test functions in the given package or all test methods of the given class) contained in the marked container have to be executed in concurrent mode.
The functions within each suite are executed sequentially in a one thread while the suites are executed concurrently using thread pool of number of available cores size.

Thread pool with fixed number of threads equals to number of available processors (cores) is used to execute tests in concurrent mode.

FactoryAnnotationshared final FactoryAnnotation

Indicates that class has to be instantiated using a given factory function.
factory() annotation takes declaration of top-level factory function.

Factory function arguments

Example of synchronous instantiation:

    StarshipTest createStarshipTest() => StarshipTest(universeSize);

    factory(`function createStarshipTest`)
    class StarshipTest(Integer universeSize) {
        ...
    }       

Example of asynchronous instantiation:

    StarshipTest createStarshipTest(AsyncFactoryContext context) {
        context.fill(StarshipTest(universeSize));
    }

    factory(`function createStarshipTest`)
    class StarshipTest(Integer universeSize) {
        ...
    }       

Pay attention:
Asynchronous version has to call AsyncFactoryContext.fill() or AsyncFactoryContext.abort().
Synchronous version has to return non-optional object or throw.

RetryAnnotationshared final RetryAnnotation

Indicates that execution of a test function or all test functions within annotated test container have to be retryed using the given RepeatStrategy (extracted from source).

Overall execution cycle including before, after and testRule callbacks are repeated!

If you need to repeat just test function execution, look on RepeatRunner.

TimeoutAnnotationshared final TimeoutAnnotation

Indicates that if test function execution takes more than timeoutMilliseconds the test has to be interrupted.
The annotation is applied to any function called using AsyncTestExecutor: prepost functions, test rules, factory and test functions.

Interfaces
AsyncFactoryContextshared AsyncFactoryContext

Factory submited to factory function in order to perform asynchronous instantiation in initialization of test class. When object is instantiated it has to be passed to AsyncFactoryContext.fill() method. If some error has been occurred during instantiation and may be reported using AsyncFactoryContext.abort() method.

The test executor blocks the current thread until one of AsyncFactoryContext.fill() or AsyncFactoryContext.abort() method is called.

AsyncPrePostContextshared AsyncPrePostContext

Allows prepost function functions to interract with test executor.

Prepost function has to call AsyncPrePostContext.proceed() or AsyncPrePostContext.abort() when initialization or disposing is completed or errored, correspondently.
The test executor blocks execution thread until AsyncPrePostContext.proceed() or AsyncPrePostContext.abort() is called.

See details about test initialization / disposing in herd.asynctest.

AsyncTestContextshared AsyncTestContext

Provides interaction with asynchronous test executor.

General test procedure within test function is:

  1. Perform the testing. Notify test executor on failures or successes. Several notifications are allowed. Each failure or success notification is represented as test variant.
  2. Notify test executor on test procedure completion (call AsyncTestContext.complete()). This step is nesseccary to continue testing with next execution since test executor blocks execution thread until AsyncTestContext.complete() is called.

Example of tested function:

test async
void doTesting(AsyncTestContext context) {
    // perform test procedure and notify about fails, if no fails notified test is considered successfull
    context.fail(Exception("exception"), "some exception");
    context.fail(AssertionError( "assert"), "some assert");
    context.assertThat(true, IsFalse(), "to be `false`");

    // complete testing
    context.complete("title which is added to test variant name only if test is succeeded");
}

It is not required to notify with success, if test function doesn't notify on failure the test is considered as successfull.

Classes
AsyncTestExecutorshared AsyncTestExecutor

Async test executor.

Capabilities

  • testing asynchronous multithread code
  • running test functions concurrently or sequentialy, see concurrent() annotation and herd.asynctest
  • multi-reporting: several failures or successes can be reported for a one test execution, each report is represented as test variant and might be marked with String title
  • value- and type- parameterized testing with a set of function arguments, see herd.asynctest.parameterization for details
  • conditional execution with annotations satisfied ceylon.test.engine.spi::TestCondition interface

In order to utilize this executor capabilities test function has to accept AsyncTestContext as the first argument:

    test async
    void doTesting(AsyncTestContext context) {...}   

Running

To run the test using this executor apply async() annotation at function, class, package or module level (alternatively, ceylon.test::testExecutor annotation with `class AsyncTestExecutor` argument is to be applied).
Following procedure is as usual for SDK ceylon.test module - mark tested functions with ceylon.test::test annotation and run test in IDE or command line.

Test logic

When test function taking AsyncTestContext as first argument is executed it is expected the function will do following steps:

  1. Performing the test, reporting on failures via AsyncTestContext. Several error or success reports are allowed. Each failure or success report is represented as test variant.
  2. Notifying test executor on test procedure completion - AsyncTestContext.complete(). This step is nesseccary to continue testing with next execution since test executor blocks execution thread until AsyncTestContext.complete() is called.

Exceptions
FactoryReturnsNothingshared FactoryReturnsNothing

Exception errored when no object instantiated no error throwed by factory.

IncompatibleInstantiationshared IncompatibleInstantiation

Thrown when instantiated class has neither default constructor nor factory function.

MultipleAbortExceptionshared MultipleAbortException

Collects multiple abort reasons in a one exception.

TimeOutExceptionshared TimeOutException

Exception which shows that time out has been reached.