asyncTest

is an extension to SDK ceylon.test module with following capabilities:

  • testing asynchronous multithread code
  • common initialization for a set of test functions
  • storing initialized values on test context and retrieving them during test execution
  • executing tests concurrently or sequentially
  • parameterized testing
  • conditional test execution
  • organizing complex test conditions into a one flexible expression with matchers
  • multi-reporting: several failures or successes can be reported for a one particular test execution (test function), each report is represented as test variant and might be marked with String title
  • reporting test results using charts (or graphs)

The extension is based on:

  • AsyncTestContext interface which test function has to operate with (basically, reports on fails to).
  • AsyncTestExecutor class which satisfies ceylon.test.engine.spi::TestExecutor and used by ceylon.test module to execute test functions.
  • TestInitContext interface and init() annotation which can be used for common initialization of test set.
  • herd.asynctest.chart which is intended to organize reporting with charts

Test procedure

  1. Declare test function, which accepts AsyncTestContext as the first argument:
        test testExecutor(`class AsyncTestExecutor`) void doTesting(AsyncTestContext context) {...}
    
    The other arguments have to be in accordance with ceylon.test::parameters annotation or another annotation which supports ceylon.test.engine.spi::ArgumentListProvider.
    Mark test function or upper level container with ceylon.test::test annotation.
    Mark test function or upper level container with testExecutor(class AsyncTestExecutor) annotation.
  2. Code test function according to AsyncTestContext specification:
  3. Apply ceylon.test::testExecutor annotation at function, class, package or module level.
  4. Run test in IDE or command line using Ceylon test tool.

Test executor blocks the thread until AsyncTestContext.complete() is called. It means test function has to call completion to continue with other testing and to report results.

If test function is marked with testExecutor(class AsyncTestExecutor) but doesn't take AsyncTestContext as first argument it is executed using ceylon.test.engine::DefaultTestExecutor. At the same time the function is executed concurrently if not marked with alone() annotation, see Concurrent or sequential test execution section for details.

Initialization

Common asynchronous initialization for a set of test functions can be performed by marking these functions with init() annotation. Argument of this annotation is declaration of initializer function, which is called just once for all tests.

Initializer function has to take first argument of TestInitContext type. If initializer takes more arguments it has to be marked with ceylon.test::parameters annotation or another annotation which supports ceylon.test.engine.spi::ArgumentProvider.

When initialization is completed TestInitContext.proceed() has to be called.
If some error has occured and test has to be aborted, TestInitContext.abort() can be called.

Initializer can store some values on context using TestInitContext.put() method. These values can be retrieved lately by test functions using AsyncTestContext.get() or AsyncTestContext.getAll().
Alternatively init() annotation can be used at class, package or module level to apply initialization to all test functions of corresponding container.

Initializer may not be marked with init() annotation! Test function, class, package or module should be marked.

Executor blocks current thread until TestInitContext.proceed() or TestInitContext.abort() called.

Just a one initializer is called for a given function selecting that from internal level (function) to external one (module). So if function and package (or module) are both marked with init() only initializer from function annotation is used for the given test function initialization.

If initialization is aborted using TestInitContext.abort() tests initialized with the given initializer are never executed but test abort is reported.

init and ceylon.test::beforeTest are different. First one is called just once for the overall test run, while second is called before each test function invoking.

Example:

    // initialization parameters
    [String, Integer] serverParameters => ["host", 123]; 

    // initializer - binds to server specified by host:port,
    // if successfull proceeds with test or aborted if some error occured
    parameters(`value serverParameters`)
    void setupServer(TestInitContext context, String host, Integer port) {
        Server server = Server();
        server.bind(host, port).onComplete (
            (Server server) {
                // storing server on context and notifying to continue with testing
                context.put("``host``:``port``", server, server.close); 
                context.proceed();
            },
            (Throwable err) {
                // abort initialization since server binding errored
                context.abort(err, "server ``host``:``port`` binding error");
            }
        );
    }

    // test functions, setupServer is called just once - neveretheless the actual number of test functions 
    test testExecutor(`class AsyncTestExecutor`) init(`function setupServer`)
    void firstTest AsyncTestContext context) {
        String serverName = serverParameters.host + ":``port``";
        assert ( exists server = context.get<Server>("serverName") );
        ...
    }

    test testExecutor(`class AsyncTestExecutor`) init(`function setupServer`)
    void secondTest(AsyncTestContext context) {
        String serverName = serverParameters.host + ":``port``";
        assert ( exists server = context.get<Server>("serverName") );
        ...
    }

Conditional execution

Test condition can be specified via custom annotation which satisfies ceylon.test.engine.spi::TestCondition interface.
Any number of test conditions can be specified at function, class, package or module level.
All conditions at every level are evaluated before test execution started and if some conditions are not met (are unsuccessfull) the test is skipped and all rejection reasons are reported.

Parameterized testing

Can be performed using staff provided by module ceylon.test: ceylon.test.engine.spi::ArgumentListProvider or ceylon.test::parameters.
See details in corresponding documentation.

Concurrent or sequential test execution

Test function can be executed:

  • concurrently using fixed size thread pool with number of threads equals to number of available processors (cores)
  • sequentially one-by-one on the main thread

Test executor runs all concurrently executed tests firstly and than runs sequential tests.

In order to run test sequentially mark test function with alone() annotation. If this annotation is omitted test function is executed concurrently.

To run sequentially all functions contained in package or module just mark package or module with alone annotation.

Example:

        testExecutor(`class AsyncTestExecutor`)
        test void doTestOne(AsyncTestContext context) {...}

        testExecutor(`class AsyncTestExecutor`)
        test void doTestOther(AsyncTestContext context) {...}

        testExecutor(`class AsyncTestExecutor`)
        alone test void doTestAlone(AsyncTestContext context) {...}

In the above example, doTestOne and doTestOther are executed concurrently, while doTestAlone is executed just after both doTestOne and doTestOther are completed.

Matchers

Matchers are intended to organize complex test conditions into a one flexible expression.
Basically, matcher is a rule and verification method which identifies if submitted test value satisfies this rule or not.

Details of matching API are described in herd.asynctest.match.

Reporting test results using charts

Chart is simply a set of plots, where each plot is a sequence of 2D points.
Test results can be represented and reported with charts using staff provided by herd.asynctest.chart.

Platform: Java
By: Lis
License: The MIT License (MIT) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Packages
herd.asynctest

Main package, see description in herd.asynctest.

herd.asynctest.chart

Chart is a set of plots, where each plot is a sequence of 2D points, Plot.
Chart may have title, names of each axis and…

herd.asynctest.match

Basically, matcher is a rule and verification method which identifies if submitted test value satisfies this rule or…

Dependencies
ceylon.collection1.2.1
ceylon.file1.2.1
ceylon.test1.2.1
java.base8

Main package, see description in herd.asynctest.

By: Lis
Annotations
aloneshared AloneAnnotation alone()

Requires the test function to be executed alone rather than concurrently.
By default all tests are executed concurrently using fixed size thread pool with number of threads equals to number of available processor (cores).

Test functions marked with alone annotation are executed sequentially one-by-one on the main thread and after all concurrent tests are completed.

To run sequentially all functions contained in package or module just mark package or module with alone annotation.

initshared InitAnnotation init(FunctionDeclaration initializer)

Marks a test function (marked with ceylon.test::test also) with initializer.

Initializer function has to take first argument of TestInitContext type. If initializer takes more arguments it has to be marked with ceylon.test::parameters annotation or another annotation which supports ceylon.test.engine.spi::ArgumentProvider.

AsyncTestExecutor invokes initializers just a once for a test run before test execution started.

Parameters:
  • initializer

    Function which performs initialization.

AloneAnnotationshared final AloneAnnotation

Annotation class for alone().

InitAnnotationshared final InitAnnotation

Annotation class for init().

Interfaces
AsyncTestContextshared AsyncTestContext

Provides interaction with asynchronous test executor.

General test procedure within test function is:

  1. Notify test executor on test procedure starting.
  2. Perform the test itself. Notify test executor on failures or successes. Several notifications are allowed. Each failure or success notification is represented as test variant.
  3. Notify test executor on test procedure completion (call AsyncTestContext.complete()). This step is nesseccary to continue testing with next execution since test executor blocks execution thread until AsyncTestContext.complete() is called.

Example of tested function:

test testExecutor(`class AsyncTestExecutor`)
void doTesting(AsyncTestContext context) {
    // start testing
    context.start();

    // perform test procedure and notify about fails, if no fails notified test is considered successfull
    context.fail(Exception("exception"), "some exception");
    context.fail(AssertionError( "assert"), "some assert");
    context.abort(Exception( "exception"), "test aborted");
    context.assertTrue(true, "to be `false`");
    context.assertNotNull(null, "to be nonull");

    // complete testing
    context.complete("title which is added to test variant name only if test is succeeded");
}

Common initialization for a set of test functions can be performed using init() annotation and TestInitContext.

It is not required to notify with success, if test function doesn't notify on failure the test is considered as successfull.

TestInitContextshared TestInitContext

Allows initializer function to interract with test executor.

Initializer can fill in some values using TestInitContext.put(). The values can be retrieved late from AsyncTestContext.
Additionally callback to listen test run finishing can be added using TestInitContext.addTestRunFinishedCallback().

Initializer has to call TestInitContext.proceed() or TestInitContext.abort() when initialization is completed or errored since executor blocks execution thread until TestInitContext.proceed() or TestInitContext.abort() is called.

Classes
AsyncTestExecutorshared AsyncTestExecutor

Async test executor.

Capabilities

  • testing asynchronous multithread code
  • running test functions concurrently or sequentialy, see alone() annotation and herd.asynctest
  • multi-reporting: several failures or successes can be reported for a one test execution, each report is represented as test variant and might be marked with String title
  • parameterized testing with a set of function arguments, see ceylon.test::parameters annotation and ceylon.test.engine.spi::ArgumentListProvider for details
  • conditional execution with annotations satisfied ceylon.test.engine.spi::TestCondition interface

In order to utilize this executor capabilities test function has to accept AsyncTestContext as the first argument:

    test testExecutor(`class AsyncTestExecutor`)
    void doTesting(AsyncTestContext context) {...}

Test function can have more arguments if it is annotated with ceylon.test::parameters annotation or another one which supports ceylon.test.engine.spi::ArgumentListProvider.

Running

To run the test using this executor ceylon.test::testExecutor annotation with class AsyncTestExecutor argument has to be applied at function, class, package or module level.
Following procedure is as usual for SDK ceylon.test module - mark tested functions with ceylon.test::test annotation and run test in IDE or command line.

Test logic

When test function taking AsyncTestContext as first argument is executed it is expected the function will do following steps:

  1. Notifying test executor on test procedure starting - AsyncTestContext.start().
  2. Performing the test, reporting on failures via AsyncTestContext. Several error or success reports are allowed. Each failure or success report is represented as test variant.
  3. Notifying test executor on test procedure completion - AsyncTestContext.complete(). This step is nesseccary to continue testing with next execution since test executor blocks execution thread until AsyncTestContext.complete() is called.

Test function is responsible to catch all exceptions / assertions and to redirect them to AsyncTestContext.

If test function doesn't take AsyncTestContext as first argument it is executed using ceylon.test.engine::DefaultTestExecutor.

It is not recommended to use ceylon.test::assertXXX functions together with AsyncTestContext, since this functions throws an exception which may lead to immediate testing completion. Report via AsyncTestContext instead.