Source Code

The ceylon.test module is a simple framework to write repeatable tests.

Tests execute the code of the module under test and can make assertions about what it does. For example,

  • do functions, when called with certain arguments, return the expected results?
  • do classes behave as required?
  • etc.

CONTENT

  1. Getting started
  2. Running
  3. Assertions
  4. Lifecycle callbacks
  5. Disabling tests
  6. Tagging tests
  7. Extension points
  8. Parameter resolution

GETTING STARTED

Tests can be written as top level functions …

test
void shouldAlwaysSucceed() {}

… or organized inside classes.

class YodaTest() {

    test
    void shouldBeJedi() {
        assert(yoda is Jedi);
    }

    test
    void shouldHavePower() {
        assert(yoda.midichloriansCount > 1k);
    }

}

Notice the test annotation, which helps the framework to automatically discover tests.


RUNNING

The most convenient way how to run tests is to use IDE integration or via command line tools ceylon test and ceylon test-js.

$ceylon test com.acme.mymodule

Tests can be also run programmatically, via interface TestRunner and its factory method createTestRunner(), but this API is usually not necessary to use directly.


ASSERTIONS

Assertions can be evaluated by using the language's assert statement or with the various assert... functions, for example:

assert(is Hobbit frodo);
assert(exists ring);

assertNotEquals(frodo, sauron);
assertThatException(() => gandalf.castLightnings()).hasType(`NotEnoughMagicPowerException`);

A test function which completes without propagating an exception is classified as a success. A test function which propagates an AssertionError is classified as a failure. A test function which propagates any other type of Exception is classified as an error.


LIFECYCLE CALLBACKS

Common initialization logic can be placed into separate functions, which run before or after each test.

class StarshipTest() {

    beforeTest void init() => starship.chargePhasers();

    afterTest void dispose() => starship.shutdownSystems();

Or it is possible to execute custom code, which will be executed only once before or after whole test run. Such callbacks can be only top level functions.

 beforeTestRun
 void createUniverse() { ... }

 afterTestRun
 void destroyUniverse() { ... }    

Other options how to hook into tests execution, is to implement TestListener and react on concrete events. Or if you have to go deeper, there are several TestExtension points.


DISABLING TESTS

Sometimes you want to temporarily disable a test or a group of tests, this can be done via the ignore annotation.

test
ignore("still not implemented")
void shouldBeFasterThanLight() { ... }

Sometimes the conditions, if the test can be reliable executed, are know only in runtime, in that case one of the assume... functions can be used.

test
void shouldBeFasterThanLight() {
    assumeTrue(isWarpDriveAvailable);
    ...
}

TAGGING TESTS

Tests or its containers can be tagged with one or more tags. Those tags can later be used to filter which tests will be executed.

For example test, which is failing often, but from unknow reasons, can be marked as unstable

test
tag("unstable")
shared void shouldSucceedWithLittleLuck() { ... }

… and then excluded from test execution

$ceylon test --tag=!unstable com.acme.mymodule

… or visa versa, we can execute only tests with this tag

$ceylon test --tag=unstable com.acme.mymodule

EXTENSION POINTS

Test execution can be extended or completely changed by several extension points. All extension points are child of marker interface TestExtension and here is a list of currently available extension points:

Extensions can be registered declaratively on several places: on concreate test, on class which contains tests, on whole package or even module, with help of testExtension annotation, for example:

testExtension(`class DependencyInjectionInstancePostProcessor`,
              `class TransactionTestListener`)
package com.acme;

PARAMETER RESOLUTION

It is possible to write parameterized tests. The responsibility for resolving argument lists is on ArgumentListResolver. It's default implementation find annotation which satisfy ArgumentListProvider or ArgumentProvider interface, collect values from them and prepare all possible combination. Developers can easily implement their own argument providers, currently there exists basic implementation, parameters annotation.

Example:

shared {[Integer, Integer]*} fibonnaciNumbers => {[1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 8] ...};

test
parameters(`value fibonnaciNumbers`)
shared void shouldCalculateFibonacciNumber(Integer input, Integer result) {
    assert(fibonacciNumber(input) == result);
}

By: Tom Bentley, Tomáš Hradec
License: Apache Software License
Packages
ceylon.test

Contains API of the test module.

ceylon.test.annotation

Contains annotations types of the test module.

ceylon.test.engine

Contains test engine.

ceylon.test.engine.spi

Contains SPI of test module.

ceylon.test.event

Contains events of the test module.

ceylon.test.reporter

Contains reporting tools.

Dependencies
ceylon.collection1.3.3
ceylon.file (jvm)1.3.3
ceylon.runtime (jvm)1.3.3
java.base (jvm)7
org.jboss.modules (jvm)1.4.4.Final

Contains API of the test module.

Aliases
TestComparatorSource Codeshared TestComparator=> Comparison(TestDescription, TestDescription)

Alias for functions which compare two tests, used for sorting tests in test plan.

TestFilterSource Codeshared TestFilter=> Boolean(TestDescription)

Alias for functions which filter tests. Should return true if the given test should be run, or false if it should be excluded.

TestSourceSource Codeshared TestSource=> Module|Package|ClassDeclaration|FunctionDeclaration|Class<Anything,Nothing>|FunctionModel<Anything,Nothing>|String

Alias for program elements which can be used as a source for discovering tests.

Annotations
afterTestSource Codeshared AfterTestAnnotation afterTest()

Marks a function which will be run after each test in its scope. It allow to place common initialization logic into separate place.

class StarshipTest() {

    beforeTest 
    void init() => starship.chargePhasers();

    afterTest 
    void dispose() => starship.shutdownSystems();
afterTestRunSource Codeshared AfterTestRunAnnotation afterTestRun()

Marks a toplevel function which will be executed once after all tests, after the test run is finished.

afterTestRun
void stopEmbeddedDatabase() { ... }
beforeTestSource Codeshared BeforeTestAnnotation beforeTest()

Marks a function which will be run before each test in its scope. It allow to place common initialization logic into separate place.

class StarshipTest() {

    beforeTest 
    void init() => starship.chargePhasers();

    afterTest 
    void dispose() => starship.shutdownSystems();
beforeTestRunSource Codeshared BeforeTestRunAnnotation beforeTestRun()

Marks a toplevel function which will be executed once before all tests, before the test run starts.

beforeTestRun
void startEmbeddedDatabase() { ... }
ignoreSource Codeshared IgnoreAnnotation ignore(String reason = "")

Marks a test or group of tests which should not be executed, which will be skipped during test run. It can be set on several places: on concrete test, on class which contains tests, on whole package or even module.

test
ignore("still not implemented")
void shouldBeFasterThanLight() {
Parameters:
  • reason = ""

    Reason why the test is ignored.

parametersSource Codeshared ParametersAnnotation parameters(FunctionOrValueDeclaration source)

Annotations to specify source of argument values for parameterized tests, can be used for whole function or individually for each parameter. As a source, can be used toplevel value or function, which type is compatible with parameters of test function.

A test function may have multiple parameters, each with own value source. The test engine will execute it for each combination of provided values. For example, a function with one parameter whose argument provider yields two values, and second parameter whose argument provider yields three values, will be executed six times.

Example:

shared {[Integer, Integer]*} fibonnaciNumbers => {[1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 8] ...};

test
parameters(`value fibonnaciNumbers`)
shared void shouldCalculateFibonacciNumber(Integer input, Integer result) {
    assert(fibonacciNumber(input) == result);
}
Parameters:
  • source

    The source function or value declaration.

tagSource Codeshared TagAnnotation tag(String+ tags)

Marks a test or group of tests with one or more tags, tags can be used for filtering, which tests will be executed.

For example test, which is failing often, but from unknow reasons, can be marked as unstable

test
tag("unstable")
shared void shouldSucceedWithLittleLuck() { ... }

… and then excluded from test execution

$ceylon test --tag=!unstable com.acme.mymodule

… or visa versa, we can execute only tests with this tag

$ceylon test --tag=unstable com.acme.mymodule
Parameters:
  • tags

    One or more tags associated with the test.

testSource Codeshared TestAnnotation test()

Marks a function as being a test.

Example of simplest test:

test
shared void shouldAlwaysSucceed() {}
testExecutorSource Codeshared TestExecutorAnnotation testExecutor(ClassDeclaration executor)

Annotation to specify custom TestExecutor implementation, which will be used for running test. It can be set on several places: on concrete test, on class which contains tests, on whole package or even module. If multiple occurrences will be found, the most closest will be used.

 testExecutor(`class ArquillianTestExecutor`)
 package com.acme;
Parameters:
testExtensionSource Codeshared TestExtensionAnnotation testExtension(ClassDeclaration+ extensions)

Annotation to specify various TestExtension implementation, which will be used during running test. It can be set on several places: on concrete test, on class which contains tests, on whole package or even module.

testExtension(`class DependencyInjectionInstancePostProcessor`,
              `class TransactionTestListener`)
package com.acme;
Parameters:
testSuiteSource Codeshared TestSuiteAnnotation testSuite({Declaration+} sources)

Annotation to specify test suite, which allow combine several tests or test suites and run them together.

testSuite({`class YodaTest`,
           `class DarthVaderTest`,
           `function starOfDeathTestSuite`})
shared void starwarsTestSuite() {}
Parameters:
  • sources

    The program elements from which tests will be executed.

Functions
assertAllSource Codeshared void assertAll(Anything()[] assertions, String? message = null)

Verify all given assertions and any failures will be reported together.

Example:

assertAll([
    () => assertEquals(agent.id, "007"),
    () => assertEquals(agent.firstName, "James"),
    () => assertEquals(agent.lastName, "Bond")]);
Parameters:
  • assertions

    The group of assertions.

  • message = null

    The message describing the problem.

Throws
assertEqualsSource Codeshared void assertEquals(Anything actual, Anything expected, String? message = null, Boolean compare(Anything val1, Anything val2) = ...)

Fails the test if the given values are not equal according to the given compare function.

Parameters:
  • actual

    The actual value to be checked.

  • expected

    The expected value.

  • message = null

    The message describing the problem.

  • compare = equalsCompare(actual, expected)

    The compare function.

Throws
assertFalseSource Codeshared void assertFalse(Boolean condition, String? message = null)

Fails the test if the condition is true.

Parameters:
  • condition

    The condition to be checked.

  • message = null

    The message describing the problem.

Throws
assertIsSource Codeshared void assertIs(Anything val, ClassOrInterface<Anything> expected, String? message = null)

Fails the test if the given value does not satisfy the provided ClassOrInterface

Parameters:
  • val

    The actual value to be checked.

  • expected

    The class or interface to be satisfied.

  • message = null

    The message describing the problem.

Throws
assertNotEqualsSource Codeshared void assertNotEquals(Anything actual, Anything unexpected, String? message = null, Boolean compare(Anything val1, Anything val2) = ...)

Fails the test if the given values are equal according to the given compare function.

Parameters:
  • actual

    The actual value to be checked.

  • unexpected

    The expected value.

  • message = null

    The message describing the problem.

  • compare = equalsCompare(actual, unexpected)

    The compare function.

Throws
assertNotNullSource Codeshared void assertNotNull(Anything val, String? message = null)

Fails the test if the given value is null.

Parameters:
  • val

    The value to be checked.

  • message = null

    The message describing the problem.

Throws
assertNullSource Codeshared void assertNull(Anything val, String? message = null)

Fails the test if the given value is not null.

Parameters:
  • val

    The value to be checked.

  • message = null

    The message describing the problem.

Throws
assertThatExceptionSource Codeshared ExceptionAssert assertThatException(Throwable|Anything() exceptionSource)

Fails the test if expected exception isn't thrown.

Example:

assertThatException(() => gandalf.castLightnings()).hasType(`NotEnoughMagicPowerException`);
Parameters:
  • exceptionSource

    The checked exception or callback which should throw exception.

Throws
assertTrueSource Codeshared void assertTrue(Boolean condition, String? message = null)

Fails the test if the condition is false.

Parameters:
  • condition

    The condition to be checked.

  • message = null

    The message describing the problem.

Throws
assumeFalseSource Codeshared void assumeFalse(Boolean condition, String? message = null)

Abort test execution if the assumption condition is true.

Example:

test
shared void shouldUseNetwork() {
    assumeFalse(isDisconnected);
    ...
}
Parameters:
  • condition

    The condition to be checked.

  • message = null

    The message describing the assumption.

Throws
assumeTrueSource Codeshared void assumeTrue(Boolean condition, String? message = null)

Abort test execution if the assumption condition is false.

Example:

test
shared void shouldUseDatabase() {
    assumeTrue(isDatabaseAvailable);
    ...
}
Parameters:
  • condition

    The condition to be checked.

  • message = null

    The message describing the assumption.

Throws
createTestRunnerSource Codeshared TestRunner createTestRunner(TestSource[] sources, TestExtension[] extensions = [], TestFilter filter = ..., TestComparator comparator = ...)

Create a new TestRunner for the given test sources and configures it according to the given parameters.

Parameters:
  • sources

    The program elements from which tests will be executed.

  • extensions = []

    The extensions which will be used during the test run.

  • filter = defaultTestFilter

    A filter function for determining which tests should be run. Returns true if the test should be run. The default filter always returns true.

  • comparator = defaultTestComparator

    A comparator used to sort the tests, used tests in certain order. The default comparator runs the tests in alphabetical order.

defaultTestComparatorSource Codeshared Comparison defaultTestComparator(TestDescription description1, TestDescription description2)

Default test comparator sort tests alphabetically.

defaultTestFilterSource Codeshared Boolean defaultTestFilter(TestDescription description)

Default test filter, always return true.

failSource Codeshared void fail(String? message = null)

Throws an AssertionError to fail a test.

Parameters:
  • message = null

    The message describing the problem.

Throws
runTestToolSource Codeshared void runTestTool()

Run function used by ceylon test and ceylon test-js tools, it is not supposed to be call directly from code.

Interfaces
TestListenerSource Codeshared TestListener

Represents a listener which will be notified about events that occur during a test run.

Example of simple listener, which triggers alarm whenever test fails.

shared class RingingListener() satisfies TestListener {
    shared actual void testError(TestErrorEvent event) => alarm.ring();
}

… such listener can be used directly when creating TestRunner

TestRunner runner = createTestRunner{
    sources = [`module com.acme`];
    extensions = [RingingListener()];};

… or better declaratively with usage of testExtension() annotation

testExtension(`class RingingListener`)
module com.acme;
TestRunResultSource Codeshared TestRunResult

Represents a summary result of the test run.

TestRunnerSource Codeshared TestRunner

Represents a facade for running tests.

Instances are usually created via the createTestRunner() factory method. For running tests is more convenient to use command line tool ceylon test or use integration with IDE, so it is not necessary to use this API directly.

Classes
ExceptionAssertSource Codeshared ExceptionAssert

An assertions applicable to exceptions, see assertThatException().

TestDescriptionSource Codeshared TestDescription

Describes a test, or a group of tests, can be arranged in a tree.

TestResultSource Codeshared TestResult

Represents a detailed result of the execution of a particular test.

TestStateSource Codeshared TestState

The result state of test execution.