Test runner provides a control over a test function execution. The package contains staff to make a custom runners as well as some build-in runners.

Each runner has to satisfy AsyncTestRunner interface which contains only one method:

    shared formal void run(AsyncRunnerContext context, void testing(AsyncRunnerContext context), TestInfo info)  

Where:

  • context is context to report test results to.
  • testing is test function to be executed.
  • info is information on the test variant to be executed.

Steps to execute test function with some runner:

  1. Declare value or function which return value satisfied to AsyncTestRunner interface.
    The value or factory function may be either top-level or test function container class attribute / method.
  2. Mark test function or upper-level container with runWith() annotation. Provide as the annotation argument declaration of value / function made under item 1.
  3. Run the test.

Example:

    class MyRunner() satisfies AsyncTestRunner {
        shared actual void run(AsyncRunnerContext context, void testing(AsyncRunnerContext context), TestInfo info) {
            testing(context);
        }
    }

    test runWith(`class MyRunner`)
    void myTest() {...}

Test runner executes only test function. All before, after and testRule (including TestStatement) callbacks are executed outside the runner. If overall execution cycle has to be repeated, see retry().

The test function submitted to the runner blocks current thread until context.complete called.

Built-in runners

Custom runners

Implement AsyncTestRunner interface and apply your runner using runWith() annotation.

Note that AsyncRunnerContext sent by runner into the test function will always be wrapped! It means that calling AsyncRunnerContext.complete always leads to test completion and runner has no way to avoid the test completion.
Reasons:

  1. Test may be interrupted by timeout.
  2. Synchronization of the context methods.
By: Lis
Since 0.6.0
Annotations
runWithshared RunWithAnnotation runWith(FunctionOrValueDeclaration runner)

Provides test runner for the annotated test function. See details in RunWithAnnotation.

Parameters:
  • runner

    Runner source. Top-level function or value or method or attribute of a test function container. The source has to return an instance implementing AsyncTestRunner interface.

By: Lis
Since 0.6.0
RunWithAnnotationshared final RunWithAnnotation

Indicates that the test function has to be run with the given runner.
The runner has to satisfy AsyncTestRunner interface.

If class or top-level container is marked with the annotation each test function of the container is executed with the given runner.

Values
repeatOnceshared repeatOnce repeatOnce

Repeats just once and returns the given test variant result.

By: Lis
Since 0.6.0
Interfaces
AsyncRunnerContextshared AsyncRunnerContext

Base interface to push test messages to.
The interface is mainly used by test runners, see package herd.asynctest.runner for details.
Test function receives AsyncTestContext.

AsyncTestRunnershared AsyncTestRunner

Test runner is applied to execute test function with a one test variant.

Just test function is executed by runner. All before, after and testRule (including TestStatement) callbacks are executed outside the runner!

RepeatStrategyshared RepeatStrategy

Identifies if test has to be repeated or completed.

Pay attention: when implementing new strategy it has to take care to understand when it is started and when it is completed to be ready for the next repeating run.

Classes
ChainedRunnershared ChainedRunner

Provides runners chaining:
the first runner executes the second one and so on, the last runner calls testing directly.

Thread-safe.

CollectAndDelegateContextshared CollectAndDelegateContext

Context which collects results and delegates to another context.
To get results of the test run examine CollectorContext.variantResult.

CollectorContextshared CollectorContext

Context which collects results but not actually reports on.
To get results of the test run examine CollectorContext.variantResult.

ErrorCollectorRunnershared ErrorCollectorRunner

Collects all errors into a one ceylon.test.engine::MultipleFailureException exception.
Ignores success messages.

Thread-safe.

RepeatRunnershared abstract RepeatRunner

Runner which repeats test execution a number of times identified by strategy but at least once.

The runner repeats a number of time according to repeat strategy given by inner RepeatRunner.Repeater class.
Such factory approach is preferable before delegation in multithread environment and helps to avoid errors related to race conditions. Since each runner invoking uses each own instance of the repeat strategy.

In order to implement custom strategy just implement RepeatRunner class and actualize inner RepeatRunner.Repeater class.

The runner repeats only test function execution! All before, after and testRule callbacks are executed once. In order to repeat overal test execution cycle see retry().

Thread-safe.

RepeatUpToFailedRunshared RepeatUpToFailedRun

Repeats up to the first failed run but no more than maxRepeats times.
Reports result from the latest run.

RepeatUpToFailureMessageshared RepeatUpToFailureMessage

Repeats up to the first failure message but no more than maxRepeats times.
Reports the first failure message only.

RepeatUpToSuccessfulRunshared RepeatUpToSuccessfulRun

Repeats up to the first successful run but no more than maxRepeats times.
Reports result from the latest run.

TestInfoshared final TestInfo

Information on currently running test variant.

UpToFailureMessageRepeatershared UpToFailureMessageRepeater

Runner which repeats up to the first failure message but no more then maxRepeats.
Reports the first failure message only

UpToFailureRepeatershared UpToFailureRepeater

Runner which repeats up to the first failed run but no more then maxRepeats.
Reports result from the latest run.

UpToSuccessRepeatershared UpToSuccessRepeater

Runner which repeats up to the first successfull run but no more then maxRepeats.
Reports result from the latest run.

repeatOnceshared repeatOnce

Repeats just once and returns the given test variant result.