Library to run benchmarks.

Terminology

  • Benchmark is a set of benches with a unified test parameter type and a set of the parameter values.
  • Bench is an executor of the function to be tested.
  • Bench run statistic is number of operations per time unit, i.e. number of test function calls per time unit.
  • Benchmark run result is collection of bench run statistic for all given benches and all given test parameter values.

Bench

Bench is responsible for the test function execution and calculation performance statistic.
Each bench has to satisfy Bench interface.
Any particular bench may implement its own testing semantic and purposes.

The package contains two implementations:

  • SingleBench is intended to run test function in a single thread.
  • MultiBench is intended to run test function in a multithread environment.

Bench flow

An object which provides more flexible execution control then just bench function. Has to satisfy BenchFlow interface.
Both SingleBench and MultiBench may take either bench function of an object satisfied BenchFlow interface.

There are some impementations of bench flow:

  • SelectiveFlow which reselects bench function each given number of iterations
  • SequentialFlow which reselects bench function one by one from the given sequence
  • RandomFlow which randomly reselects bench function from the given list each given number of iterations
  • RandomDataFlow which randomly chooses bench function argument from the given list each given number of iterations

Both SingleBench and MultiBench benches run test with either benchmark function or BenchFlow.

Writing benchmark result

Benchmark run results may be writed to AsyncTestContext using a number of writers. See functions tagged as Writer.

Benchmark execution

benchmark() is intended to run benchmark. The function executes each given bench with each given parameter and returns results of benchmark test as instance of Result.

Steps to perform benchmark testing:

  1. Define test function or implement BenchFlow interface.
  2. Choose Bench which has to run the benchmark.
  3. Invoke benchmark() with the test options, benches and test parameters.
  4. Write benchmark results.

Example

    Integer plusBenchmarkFunction(Integer x, Integer y) {
        return x + y;
    }
    Integer minusBenchmarkFunction(Integer x, Integer y) {
        return x - y;
    }

    shared test async void plusMinusBenchmark(AsyncTestContext context) {
        writeRelativeToFastest (
            context,
            benchmark (
                Options(NumberOfLoops(20000).or(ErrorCriterion(0.002)), NumberOfLoops(100).or(ErrorCriterion(0.002))),
                [SingleBench("plus", plusBenchmarkFunction),
                SingleBench("minus", minusBenchmarkFunction)],
                [1, 1], [2, 3], [25, 34]
            )
        );
        context.complete();
    }

JIT optimization

JIT optimization may eliminate some code at runtime if it finds that the result of the code execution is not used anywhere.

Mainly there are two eliminations:

  1. Unused results of calculations
  2. Constants

Black hole

In order to avoid unused results elimination results may be passed to black hole using pushToBlackHole() function.

Suppose we would like to test plus operation:

    void plusBenchmark() {
        value x = 2;
        value y = 3;
        value z = x + y;
    }

This function call might be eliminated at all, since it results to nothing. To avoid this black hole might be used:

    void plusBenchmark() {
        value x = 2;
        value y = 3;
        pushToBlackHole(x + y);
    }

All returned values are pushed to black hole by default. So, the above example might be replaced with:

    Integer plusBenchmark() {
        value x = 2;
        value y = 3;
        return x + y;
    }

Constants

There are two constants x and y in the above examples. JIT may find that result of + operation is always the same and replace the last function with

    Integer plusBenchmark() {
        return 5;
    }

So, the plus operation will never be executed. In order to avoid this the parameters might be passed to function as arguments or declared outside the function scope as variables:

    variable Integer x = 2;
    variable Integer y = 3;

    Integer plusBenchmark() {
        return x + y;
    }

Note: returning value is prefer to direct pushing to black hole, since in this case time consumed by black hole is excluded from total time of the test function execution.

By: Lis
Since 0.7.0
Functions
benchmarkshared Result<Parameter> benchmark<Parameter>(Options options, [Bench<Parameter>+] benches, Parameter* parameters)
given Parameter satisfies Anything[]

Runs benchmark testing.
Executes each given bench with each given parameter.
Bench is responsible for the execution details and calculation performance statistic.

Parameters:
  • options

    Options the benches has to be executed with.

  • benches

    A list of benches to be executed.

  • parameters

    A list of parameters the test has to be executed with.

By: Lis
Since 0.7.0
delegateWritershared void delegateWriter<Parameter>({Anything(AsyncTestContext, Result<Parameter>)*} writers)(AsyncTestContext context, Result<Parameter> results)
given Parameter satisfies Anything[]

Delegates writing of the benchmark run results to the given writers.

Parameters:
  • writers

    Writers to be delegated to write benchmark run result.

By: Lis
Since 0.7.0
pushToBlackHoleshared void pushToBlackHole(Anything something)

Prevents JIT to eliminate dependent computations.

By: Lis
Since 0.7.0
writeAbsoluteshared void writeAbsolute<Parameter>(AsyncTestContext context, Result<Parameter> results)
given Parameter satisfies Anything[]

Writes absolute results of the benchmarking to the context in format:
bench.title with parameter: mean=mean value, dev=sample deviation value; error=sample error value%

By: Lis
Since 0.7.0
writeRelativeToFastestshared void writeRelativeToFastest<Parameter>(AsyncTestContext context, Result<Parameter> results)
given Parameter satisfies Anything[]

Writes relative to the fastest results of the benchmarking to the context in format:
bench.title with parameter: mean=mean value; dev=sample deviation value; relative mean value% to fastest; error=sample error value%

By: Lis
Since 0.7.0
writeRelativeToSlowestshared void writeRelativeToSlowest<Parameter>(AsyncTestContext context, Result<Parameter> results)
given Parameter satisfies Anything[]

Writes relative to the slowest results of the benchmarking to the context in format:
bench.title with parameter: mean=mean value, dev=sample deviation value, relative mean value% to slowest; error=sample error value%

By: Lis
Since 0.7.0
Interfaces
Benchshared Bench<in Parameter>
given Parameter satisfies Anything[]

Represents benchmark test parameterized with Parameter type.

BenchFlowshared BenchFlow<in Parameter>
given Parameter satisfies Anything[]

Represents benchmark function + test flow. I.e. functions called before / after test iteration.

BenchResultshared BenchResult

Represents results of a one benchmark.
Contains execution statistic and some additional results.

Clockshared Clock

Measures the time interval.

Usage:

  1. Call Clock.start() before each measure.
  2. Call Clock.measure() to measure time interval from the last Clock.start() call.

Example:

    Clock clock = WallClock();
    for ( item in 0 : totalRuns ) {
        clock.start();
        doOperation();
        Integer interval = clock.measure( TimeUnit.seconds );
    }

Time interval is measured relative to the thread current to the caller.
So, if Clock.measure() is called from thread A, then the interval is measured from the last call of Clock.start() done on the same thread A.

CompletionCriterionshared CompletionCriterion

Identifies if benchmark iterations have to be completed or continued.

Statisticshared Statistic

Provides statistic data for a stream of variate values.

Classes
AbstractSelectiveFlowshared abstract AbstractSelectiveFlow<in Parameter>
given Parameter satisfies Anything[]

Abstract base class for flow which selects another bench function each given number of benchmark iterations.
Bench functions are selected independently for each execution thread.

CPUClockshared CPUClock

CPU clock uses ThreadMXBean.currentThreadCpuTime to measure time interval.
If thread CPU time is not supported measures wall time interval.

ErrorCriterionshared ErrorCriterion

Continues benchmark iterations while relative sample error doesn't exceed maxRelativeError.
I.e. Statistic.relativeSampleError is compared against maxRelativeError.

MultiBenchshared MultiBench

Executes bench functions benches in separated threads.
Number of threads for each bench function is specified by the bench parameter with corresponding index.
I.e. given list of bench function benches and list of Integer (parameter argument of MultiBench.execute()).
For each bench function with index index number of threads to execute the function is specified via item of parameter list with the same index index.
So the bench is parameterized by number of execution threads.

NumberOfLoopsshared NumberOfLoops

Continues benchmark iterations while total number of benchmark loops doesn't exceed numberOfLoops.

Optionsshared Options

Benchmark options.

ParameterResultshared final ParameterResult<Parameter>
given Parameter satisfies Anything[]

Benchmark run result for a given parameter and a set of benches.

RandomDataFlowshared RandomDataFlow<in Argument>

Randomly chooses bench function argument from sourceData and pass selected argument to benchFunction.
The argument is selected independently for each execution thread.

RandomFlowshared RandomFlow<in Parameter>
given Parameter satisfies Anything[]

Randomly selects benchmark function from the given list each time before execution.
Each function is paired with corresponding selection probability. Which identifies the selection frequency or distribution.

For example:
[function1, 0.3], [function2, 0.7] leads to function1 calling of 30% and function2 calling of 70% from total number of executions.

If total sum of probabilities does not equal to 1.0 the probability of any function calling is calculated as ratio of the given probability to the total sum of probabilities.

Bench functions are selected independently for each execution thread.

Resultshared final Result<Parameter>
given Parameter satisfies Anything[]

Benchmark run result for a set of parameters and benches.

SelectiveFlowshared SelectiveFlow<in Parameter>
given Parameter satisfies Anything[]

Flow which selects another bench function each given number of benchmark iterations.
select has to return currently selected bench function.
The bench functions are selected independently for each execution thread.

SequentialFlowshared SequentialFlow<in Parameter>
given Parameter satisfies Anything[]

Sequentially selects bench function from the given list independently for the each running thread.

SingleBenchshared final SingleBench<Parameter = []>
given Parameter satisfies Anything[]

Executes test function in a single thread.

ThreadClockshared abstract ThreadClock

Base for the CPU time interval measurements.
Calculates a factor to scale wall time interval corresponds to measure request.
The factor is moving averaged ratio of CPU time interval to wall time interval.
Ends of CPU time interval are measured as close as possible to ends of wall time interval.

ThreadClock.readCurrentTime() provides particular implementation of CPU or user times.

TimeUnitshared TimeUnit

Specifies time unit.

TotalBenchTimeshared TotalBenchTime

Continues benchmark iterations while overall time accumulated by benchmark iterations doesn't exceed totalTime.
This criterion summarizes only time spent on benchmark function execution for all execution threads jointly.

TotalExecutionTimeshared TotalExecutionTime

Continues benchmark iterations while total execution time doesn't exceed timeLimit.
Execution time is overall time spent on test execution. I.e. it summarizes time spent on benchmark function execution as well as on internal calculations.

The alternative way is take into account time spent on benchmark function execution only see TotalBenchTime.

UserClockshared UserClock

User clock uses ThreadMXBean.currentThreadUserTime to measure time interval.
If thread CPU time is not supported measures wall time interval.

WallClockshared WallClock

Wall clock - uses system.nanoseconds to measure time interval.