Monday, February 1, 2021

Oddity noticed while running a benchmark

I've heard it said that using a trampoline to achieve tail recursion incurs a cost of around a factor of two. I was curious whether that was accurate and where that cost comes from, so I decided to measure it with a micro-benchmark. I chose the Tak benchmark as a starting point because it doesn't do much but function calls and it is fairly well known. The basic Tak benchmark is this:

public static int Tak (int x, int y, int z) {
    return (! (y < x))
        ? z
        : Tak (Tak (x - 1, y, z),
               Tak (y - 1, z, x),
               Tak (z - 1, x, y));
}

Tak (32, 16, 8) => 9
When called with arguments 32, 16, and 8, the benchmark makes an additional 50,510,520 calls to itself, of which 1/4 of these (12,627,630) are self tail calls. One run of the benchmark consists of calling Tak (32, 16, 8) ten times. On my laptop this takes around 900 ms.

I ran the benchmark 250 times and recorded how long it took. Here are the times plotted against the run number:

For the first 50 runs of the benchmark the average time seems to slowly increase from around 830 ms to around 930 ms, then it goes to about 900 ms for the next 50 runs, then it settles down to around maybe 890 ms after that. Mostly it stays in a fairly narrow band around the average time, but now and then the time will jump to over 1000 ms for a couple of runs.

I don't know why the average run time drifts over time like this, but the effect is visible on several different benchmarks. Since it takes the better part of a minute to perform the first 50 runs, the period over which the runtime increases is relatively long — on the order of 30 - 40 seconds. My guess is maybe as the benchmark runs the processor heats up and begins to slow down slightly in order to control the heat.

No comments: