Value- and type- parameterized testing.

Base

In order to perform parameterized testing the test function has to be marked with annotation which supports TestVariantProvider interface. The interface has just a one method - variants() which has to provide TestVariantEnumerator. The enumerator produces a stream of the TestVariant's and is iterated just a once. The test will be performed using all variants the enumerator produces.

The enumerator may return test variants lazily, dynamicaly or even non-determenisticaly.
Each TestVariant contains a list of generic type parameters and a list of function arguments.


Custom parameterization

  1. Implement TestVariantEnumerator interface:

    class MyTestVariantEnumerator(...) satisfies TestVariantEnumerator {
        shared actual TestVariant|Finished current => ...;
    
        shared actual void moveNext(TestVariantResult result) {
            if (testToBeCompleted) {
                // set `current` to `finished`
            } else {
                // set `current` to test variant to be tested next
            }
        }
    }
    
  2. Make an annotation which satisfies TestVariantProvider interface:

    shared final annotation class MyParameterizedAnnotation(...)
        satisfies SequencedAnnotation<MyParameterizedAnnotation, FunctionDeclaration>&TestVariantProvider
    {
        shared actual TestVariantEnumerator variants() => MyTestVariantEnumerator(...);
    }
    
    shared annotation MyParameterizedAnnotation myParameterized(...) => MyParameterizedAnnotation(...);
    
  3. Mark test function with created annotation:

    myParameterized(...) void myTest(...) {...}
    

Varianted testing

ParameterizedAnnotation satisfies TestVariantProvider interface and provides type- and value- parameterized testing based on collection of test variants.


Combinatorial testing

Combinatorial testing provides generation of test variants based on data sources of each test function argument.

Therminology

  • Data source is a list of the values to be applied as particular value of the test function argument.
  • Variant generator is a function which generates test variants based on data sources for each test function argument.
  • Argument kind is an indicator which is applied to each argument of test function and which variant generator may used to identify particular strategy of the variants generation.

Usage

  1. Declare test function:
    void combinatorialTest(Something arg1, Something arg2);  
    
  2. Declare data sources.
    Something[] arg1Source =>
    Something[] arg2Source =>
    
  3. Apply data source for each argument of the test function.
    void combinatorialTest(permutationSource(`arg1Source`) Something arg1,
        permutationSource(`arg2Source`) Something arg2);
    
  4. Apply variant generator to test function
    permuting void combinatorialTest(permutationSource(`arg1Source`) Something arg1,
        permutationSource(`arg2Source`) Something arg2);
    
  5. Mark the function with async and test annotations and run test
    async test permuting void combinatorialTest(permutationSource(`arg1Source`) Something arg1,
        permutationSource(`arg2Source`) Something arg2);
    

Argument kind

The kind is used by variant generator in order to identify strategy for the variant generations.
There are two kinds:

  • ZippedKind indicates that the data has to be combined with others by zipping the sources, i.e. combine the values with the same index.
  • PermutationKind indicates that the data has to be combined with others using all possible permutations of the source.

Custom kind has to extend CombinatorialKind.

Data source

DataSourceAnnotation provides for the marked argument of the test function:

  • list of the argument values
  • argument kind

There are three functions to apply the annotation:

  1. dataSource() which instantiates DataSourceAnnotation using both parameters.
  2. zippedSource() which provides a list of the argument values and mark argument as zippedKind.
  3. permutationSource() which provides a list of the argument values and mark argument as permutationKind.

DataSourceAnnotation annotation has to be applied to the each argument of the test function.

Variant generator

Is applied to a test function with CombinatorialAnnotation annotation.
There are four functions to apply the annotation:

Custom variant generator has to take a list of ArgumentVariants and has to return TestVariantEnumerator.

Example

    Integer[] firstArgument => [1,2];
    Integer[] secondArgument => [10,20];
    String[] signArgument => ["+","-"];

    shared test mixing void testMixing (
        AsyncTestContext context,
        zippedSource(`value firstArgument`) Integer arg1,
        permutationSource(`value signArgument`) String arg2,
        zippedSource(`value secondArgument`) Integer arg3
    ) {
        context.succeed( "``arg1````arg2````arg3``" );
        context.complete();
    }

In the above example testMixing is called 4 times with following arguments:

  1. 1, +, 10
  2. 1, -, 10
  3. 2, +, 20
  4. 2, -, 20
By: Lis
Since 0.7.0
Annotations
combinatorialshared CombinatorialAnnotation combinatorial(FunctionDeclaration combinator, Integer maxFailedVariants = -1)

Indicates the test has to be performed with combinatorial test variants.
See details in CombinatorialAnnotation.

Parameters:
  • combinator

    Combinator function which has to return TestVariantEnumerator and takes ArgumentVariants[] which is generated based on DataSourceAnnotation at each test function argument.

    Each test function argument has to be marked with DataSourceAnnotation!

  • maxFailedVariants = -1

    Maximum number of failed variants to stop testing. Unlimited if <= 0.

By: Lis
Since 0.7.0
dataSourceshared DataSourceAnnotation dataSource(FunctionOrValueDeclaration source, ValueDeclaration kind)

Provides values for the marked test function argument. See DataSourceAnnotation for details.

Parameters:
  • source

    The source function or value declaration which has to return a stream of values. The source may be either top-level or tested class shared member.

  • kind

    The test data kind. Returned value has to extend CombinatorialKind.

By: Lis
Since 0.7.0
mixingshared CombinatorialAnnotation mixing(Integer maxFailedVariants = -1)

Returns CombinatorialAnnotation with mixingCombinations() combinator function. I.e. it is shortcut for combinatorial(`function mixingCombinations`).

Note:

Example:

    Integer[] firstArgument => [1,2];
    Integer[] secondArgument => [10,20];
    String[] signArgument => ["+","-"];

    async test mixing void testMixing (
        zippedSource(`value firstArgument`) Integer arg1,
        permutationSource(`value signArgument`) String arg2,
        zippedSource(`value secondArgument`) Integer arg3
    ) {...}

In the above example:

  • mixing() annotation forces to use mixingCombinations as variant generator.
  • First and third argument are zipped.
  • Second argument is permuted.

As result four test variants are generated:

  1. arg1 = 1, arg2 = “+“, arg3 = 10.
  2. arg1 = 1, arg2 = “-“, arg3 = 10.
  3. arg1 = 2, arg2 = “+“, arg3 = 20.
  4. arg1 = 2, arg2 = “-“, arg3 = 20.
Parameters:
  • maxFailedVariants = -1

    Maximum number of failed variants to stop testing. Unlimited if <= 0.

By: Lis
Since 0.7.0
parameterizedshared ParameterizedAnnotation parameterized(FunctionOrValueDeclaration source, Integer maxFailedVariants = -1)

Provides parameters for the parameterized testing. See ParameterizedAnnotation for the details.

Parameters:
  • source

    The source function or value declaration which has to take no arguments and has to return a stream of test variants: {TestVariant*}.
    The source may be either top-level or tested class shared member.

  • maxFailedVariants = -1

    Maximum number of failed variants before stop. Unlimited if <= 0.

By: Lis
See also TestVariant
Since 0.6.0
permutationSourceshared DataSourceAnnotation permutationSource(FunctionOrValueDeclaration source)

Provides permutationKind kind values for the marked test function argument.
See DataSourceAnnotation for details.
The annotation is shortcut for dataSource(`value permutationKind`).

Parameters:
  • source

    The source function or value declaration which has to return a stream of values. The source may be either top-level or tested class shared member.

By: Lis
Since 0.7.0
permutingshared CombinatorialAnnotation permuting(Integer maxFailedVariants = -1)

Returns CombinatorialAnnotation with permutingCombinations() combinator function. I.e. it is shortcut for combinatorial(`function permutingCombinations`).

Example:

    Integer[] firstArgument => [1,2];
    Integer[] secondArgument => [10,20,30]; 

    async test permuting void myTest (
        permutationSource(`value firstArgument`) Integer arg1,
        permutationSource(`value secondArgument`) Integer arg2
    ) {...}

The test will be performed using six test variants:

  1. arg1 = 1, arg2 = 10
  2. arg1 = 1, arg2 = 20
  3. arg1 = 1, arg2 = 30
  4. arg1 = 2, arg2 = 10
  5. arg1 = 2, arg2 = 20
  6. arg1 = 2, arg2 = 30
Parameters:
  • maxFailedVariants = -1

    Maximum number of failed variants to stop testing. Unlimited if <= 0.

By: Lis
Since 0.7.0
zippedSourceshared DataSourceAnnotation zippedSource(FunctionOrValueDeclaration source)

Provides zippedKind kind values for the marked test function argument.
See DataSourceAnnotation for details.
The annotation is shortcut for dataSource(`value zippedKind`).

Parameters:
  • source

    The source function or value declaration which has to return a stream of values. The source may be either top-level or tested class shared member.

By: Lis
Since 0.7.0
zippingshared CombinatorialAnnotation zipping(Integer maxFailedVariants = -1)

Returns CombinatorialAnnotation with zippingCombinations() combinator function. I.e. it is shortcut for combinatorial(`function zippingCombinations`).

Example:

    Integer[] firstArgument => [1,2,3];
    Integer[] secondArgument => [10,20,30]; 

    async test zipping void myTest (
        zippedSource(`value firstArgument`) Integer arg1,
        zippedSource(`value secondArgument`) Integer arg2
    ) {...}

The test will be performed using three test variants:

  1. arg1 = 1, arg2 = 10
  2. arg1 = 2, arg2 = 20
  3. arg1 = 3, arg2 = 30
Parameters:
  • maxFailedVariants = -1

    Maximum number of failed variants to stop testing. Unlimited if <= 0.

By: Lis
Since 0.7.0
CombinatorialAnnotationshared final CombinatorialAnnotation

Indicates that the test function has to be executed with variants provided by combinations of the test function agrument values.
DataSourceAnnotation annotation has to be applied to the each argument of the test function in order to specify a list of possible argument values.
combinator function is responsible to generate test variants and to provide TestVariantEnumerator.

DataSourceAnnotationshared final DataSourceAnnotation

Indicates that the test has to be performed by applying a list of values to the marked test function argument.
source has to provide the values.
kind indicates the kind of the provided values. The kind is used by variant generator in order to identify strategy for the variant generations.

ParameterizedAnnotationshared final ParameterizedAnnotation

Indicates that generic (with possibly empty generic parameter list) test function has to be executed with given test variants.

The annotation provides parameterized testing based on collection of test variants. It takes two arguments:

  1. Declaration of function or value which returns a collection of test variants {TestVariant*}.
  2. Number of failed variants to stop testing. Default is -1 which means no limit.

The test will be performed using all test variants returned by the given stream or while total number of failed variants not exceeds specified limit.

parameterized annotation may occur multiple times at a given test function.
The variants source may be either top-level or tested class shared member.

Example:

    Value identity<Value>(Value argument) => argument;

    {TestVariant*} identityArgs => {
        TestVariant([`String`], ["stringIdentity"]),
        TestVariant([`Integer`], [1]),
        TestVariant([`Float`], [1.0])
    };

    shared test async
    parameterized(`value identityArgs`)
    void testIdentity<Value>(AsyncTestContext context, Value arg)
        given Value satisfies Object
    {
        context.assertThat(identity<Value>(arg), EqualObjects<Value>(arg), "", true);
        context.complete();
    }

In the above example the function testIdentity will be called 3 times:

  • testIdentity<String>(context, "stringIdentity");
  • testIdentity<Integer>(context, 1);
  • testIdentity<Float>(context, 1.0);

In order to run test with conventional (non-generic function) type parameters list has to be empty:

    [Hobbit] who => [bilbo];
    {TestVariant*} dwarves => {
        TestVariant([], [fili]),
        TestVariant([], [kili]),
        TestVariant([], [balin],
        TestVariant([], [dwalin]),
        ...
    };

    arguments(`value who`)
    class HobbitTester(Hobbit hobbit) {
        shared test async
        parameterized(`value dwarves`, 2)
        void thereAndBackAgain(AsyncTestContext context, Dwarf dwarf) {
            context.assertTrue(hobbit.thereAndBackAgain(dwarf)...);
            context.complete();
        }
    }

In this example class HobbitTester is instantiated once with argument provided by value who and method thereAndBackAgain is called multiply times according to size of dwarves stream.
According to second argument of parameterized annotation the test will be stopped if two different invoking of thereAndBackAgain with two different arguments report failure.

Values
permutationKindshared permutationKind permutationKind

Indicates that the test data has to be permuted.

By: Lis
Since 0.7.0
zippedKindshared zippedKind zippedKind

Indicates that the test data has to be zipped.

By: Lis
Since 0.7.0
Functions
mixingCombinationsshared TestVariantEnumerator mixingCombinations(ArgumentVariants[] arguments)

Mixes zip and permutation sources. I.e. some arguments may be provided with zippedSource() and the other arguments are provided with permutationSource().

Generation rule: for each zipped variant every possible permutations are generated.

See example in mixing().

Parameters:
  • arguments

    Variants of the test function arguments.

Throws
  • AssertionError

    Argument variant is not of 'zipped' or 'permutation' kind

By: Lis
Since 0.7.0
permutingCombinationsshared TestVariantEnumerator permutingCombinations(ArgumentVariants[] arguments)

Enumerates test variants by all possible combinations from provided arguments of ArgumentVariants[] which provides list of variants for the test function arguments.
Each function argument has to be of PermutationKind kind.

Note: permutationSource() provides variants of PermutationKind.

See example in permuting().

Parameters:
  • arguments

    Variants of the test function arguments.

Throws
  • AssertionError

    Argument variant is not of 'permutation' kind

By: Lis
Since 0.7.0
zippingCombinationsshared TestVariantEnumerator zippingCombinations(ArgumentVariants[] arguments)

Enumerates test variants by zipping provided arguments of ArgumentVariants[] which provides list of variants for the test function arguments.
Each function argument has to be of ZippedKind kind.

Note: zippedSource() provides variants of ZippedKind.

Zipping means that 'n-th' variant is built from each 'n-th' item of the argument lists.
The variant list finished at the same time when a one of argument lists finishes. If the other lists contain more items then those items are simply ignored.

See example in zipping().

Parameters:
  • arguments

    Variants of the test function arguments.

Throws
  • AssertionError

    Argument variant is not of 'zipped' kind

By: Lis
Since 0.7.0
Interfaces
TestVariantEnumeratorshared TestVariantEnumerator

Enumerates test variants.

TestVariantProvidershared TestVariantProvider

Provides enumerator for test variants.
Test execution context looks for annotations of the test function which support the interface and performs testing according to provided variants. As example, see ParameterizedAnnotation.

Classes
ArgumentVariantsshared final ArgumentVariants

Specifies variants for a test function argument.

CombinatorialKindshared abstract CombinatorialKind

Indicates the kind of the combinatorial source data.

PermutationKindshared abstract PermutationKind

Indicates that the test data has to be permuted.

TestOutputshared final TestOutput

Represents a one test report.

TestVariantshared TestVariant

Represents test variant, i.e. test function generic type parameters and arguments.

TestVariantResultshared final TestVariantResult

Results of a one test function run with some arguments.

ZippedKindshared abstract ZippedKind

Indicates that the test data has to be zipped.

permutationKindshared permutationKind

Indicates that the test data has to be permuted.

zippedKindshared zippedKind

Indicates that the test data has to be zipped.