This document contains supplementary materials for the paper:

The file replicate_results.py contains code to replicate the results presented in the paper. It requires our cirank library.

Methods for computing confidence intervals for ranks

We present in this section details on the statistical tests and confidence interval procedures.

Bootstrapping

For the bootstrapping method, we proceed as follows: for \(\nu\) repetitions, re-sample \(n\) test cases from the dataset (with replacement) to obtain \(\nu\) bootstrapped samples of the same size as the original test set. In each of these samples \(k \in [1, \nu]\), get the corresponding set of scores \(S^{(k)}_{ij}\), and compute ranks \(r^{(k)}_i\) on the mean score per-algorithm \(S^{(k)}_{i.} = \frac{1}{n}\sum_{j=1}^{n}S^{(k)}_{ij}\), such that \(r^{(k)}_i = 1\) for the lowest value of \(S^{(k)}_{i.}\). Finally, the \(\alpha\)-level confidence interval for algorithm \(i\) is given by the \(\frac{\alpha}{2}\) and \(1-\frac{\alpha}{2}\) percentiles of the distribution of \(r^{(k)}_i\).

Iman-Davenport-Nemenyi rankings

First, a Iman-Davenport test¹ is performed on all algorithms, with the null hypothesis \(H_0\) being that the distributions of scores for each algorithm is the same. To perform this test, the algorithms are first ranked per-case, so that \(r_{ij} \in [1, m]\) and \(r_{ij} = 1\) means that algorithm \(i\) has the lowest score for test case \(j\). Then, the average ranks per-algorithm \(r_{i.} = \frac{1}{n}\sum_{j=1}^nr_{ij}\) are computed. The test statistic is \(F_F = \frac{(n-1)Q}{n(m-1)-Q}\), where \(Q\) is the statistic of the Friedman test: \(Q = \frac{12n}{m(m+1)}(\sum_{i=1}^m(r_{i.}^2)-\frac{m(m+1)^2}{4})\). Under \(H_0\), \(F_F \sim F_{m-1,(m-1)(n-1)}\) where \(F_{a,b}\) is the Fisher-Snedecor distribution with \(a,b\) degrees of freedom. scipy.stats.f is used to compute the critical values of the F-distribution.

If this hypothesis cannot be rejected (p-value \(\geq \alpha\)), then all algorithms have the same confidence interval for their rankings: \([1, m]\). Note that this is different from saying that all algorithms are tied. That would imply that the result of the test proves that the samples are equal, when it merely shows that we cannot reject the hypothesis that they are. The \([1, m]\) confidence interval is therefore a better presentation of the results than, for instance, putting all algorithms tied as first. It correctly shows that the true rank of the algorithms – if they were computed on the population – cannot be resolved based on the available data.

If the null hypothesis is rejected (p-value \(< \alpha\)), then we perform the Nemenyi post-hoc² to obtain pairwise comparisons. The Nemenyi test comparing algorithms \(i\) and \(i'\) uses the test statistic: \(Y = \frac{\sqrt{2}|r_{i.}-r_{i'.}|}{\sqrt{\frac{m(m+1)}{6n}}}\), which if the two distributions are the same (i.e. under \(H_0\)) should \(\sim q_{m, \inf}\) with \(q_{k, d}\) the Studentized range distribution for \(k\) groups and \(d\) degrees of freedom. Pairs where \(P(q \geq Y) < \alpha\) are marked as significantly different, with the sign of \(r_{i.}-r_{i'.}\) determining which is better or worse. scipy.stats.studentized_range is used to get the critical values of the studentized range.

The confidence intervals \([L, U]\) are then computed as in Al Mohamad et al.³: \[L = 1 + \text{number of significantly better algorithms}\] \[U = m - \text{number of significantly worse algorithms}\]

Iman-Davenport-Wilcoxon rankings

The Iman-Davenport test is first performed as described above. If its null hypothesis is rejected, then we perform the Wilcoxon signed-rank test⁴ on all pairs of algorithms, using either the two-sided or the two one-sided alternatives. scipy.stats.wilcoxon is used for the pairwise tests.

For each algorithm \(i\), we use scipy.stats.wilcoxon to compute the p-values of the one- or two-sided tests with all other algorithms. The p-values are then adjusted using Holm’s procedure, following the recommendation of Holm, 2013⁵: the \(m-1\) p-values are ordered from most to least significant. The first value is multiplied by \(m-1\). If the adjusted p-value is \(> \alpha\), the test and all subsequent tests are deemed non-significant. If it is \(< \alpha\), the test is deemed significant, and the next value is multiplied by \(m-2\). We repeat until either we get to the last test (p-value multiplied by 1), or we find a non-significant test. For the two-sided tests, we also record the sign of the difference between the mean scores of \(i\) and each other. A “significantly better” algorithm is therefore an algorithm such that the average score is higher (assuming higher is better) and the p-value is significant. For the one-sided tests, we don’t need to record the sign, as each side give us either the “significantly better” or “significantly worse” information. The correction for the one-sided tests are applied “per-side”: first, we adjust the p-values of the tests to the left, then of the tests to the right (see last section for an example).

Confidence intervals are then computed as above.

ANOVA-Tukey rankings

For this version of the ranking, a rank transform⁶ is first applied to the scores \(S_{ij}\), so that \(r_{ij}\) is the global rank of \(S_{ij}\) (i.e. \(r_{ij} = 1\) if \(S_{ij}\) is the lowest score across all algorithms and all test cases). A one-way repeated measures ANOVA is then performed on the global ranks⁷. As with the Iman-Davenport tests above, a non-rejection of \(H_0\) for the ANOVA leads to a confidence interval of \([1, m] \forall A_i\).

Tukey’s HSD⁸ is then used, still on the ranks, for the pairwise comparisons, and the confidence intervals are computed based on the numbers of significantly better/worse algorithms.

Rank transformation is performed using scipy.stats.rankdata. Repeated-measures ANOVA is reimplemented using scipy.stats.f for the critical values of the F-distribution. Tukey’s HSD is performed with scipy.stats.tukey_hsd.

Synthetic distribution of results

Shape of challenge results distribution

Note: this section (2.1) of the supplementary materials has been added on May 19th, 2025, as I realized that it was missing from the originally published material. The analysis itself had been done in October 2024.

We gathered the results of four recent challenges on grand-challenge.org with per-image numerical evaluation metrics available: the ACROBAT 2024 registration challenge, the AortaSeg 2024 challenge, the CURVAS challenge and the Selma3D. The public leaderboards of each challenge were scraped on October 21nd, 2024, and all available results at that stage were retrieved:

To focus on the shape of the distributions, all results were standardized by subtracting the mean and dividing by the standard deviation (per team result). Then, all histograms were plotted together (per challenge):

As we can see, the distributions tend to be asymmetrical, with a sharp peak, a long tail and some outliers.

Difficulty distribution

The distribution of the difficulty is a Laplace asymmetric distribution with \(\kappa=2\), as implemented in scipy.stats.laplace_asymmetric. The PDF of that function is shown below:

Generated results

Generated results are obtained by first sampling test cases from the difficulty distribution, then, for each algorithm, adding a sample from a normal distribution. For algorithm \(i\), the mean of that normal distribution is \(i \times f \times \sigma_N\) where \(\sigma_N = \sqrt{\sigma_D}\), and \(\sigma_D\) is the standard deviation of the difficulty distribution, and \(f\) is a separability factor, set at \(0.25, 0.5, 1\) and \(2\) for the different experiments.

So for instance, at separability factor \(0.5\) we would have the scores \(s_{ij} = d_{j} + n_{ij}\), with \(d_{j} \sim L_{\kappa=2}\), \(n_{ij} \sim N(\frac{i\sqrt{\sigma_D}}{2}, \sqrt{\sigma_D})\).

Example of samples of generated results at different separability levels are shown below. Lines show the pairings of the points.

Table of results

FWTI

Power @ \(0.25\sigma_N\) (m=5, n=20)

Power @ \(0.5\sigma_N\) (m=5, n=20)

Power @ \(\sigma_N\) (m=5, n=20)

Power @ \(2\sigma_N\) (m=5, n=20)

Power @ \(0.5\sigma_N\) (m=5, n=40)

Showing in parenthesis the difference with \(0.5\sigma_N\) with \(m=5, n=20\).

Power @ \(0.5\sigma_N\) (m=10, n=20)

Showing in parenthesis the difference with \(0.5\sigma_N\) with \(m=5, n=20\).

Why Nemenyi lacks power at high separation levels

As mentioned above, Nemenyi’s test statistic is \(Y = \frac{\sqrt{2}|r_{i.}-r_{i'.}|}{\sqrt{\frac{m(m+1)}{6n}}} \sim q_{m, \infty}\) under \(H_0\). In the extreme case where all algorithms have a perfectly monotonic relationship between paired scores (i.e. \(S_{ij} < S_{i'j} \forall i < i'\)), then the ranks for each test case will be \([1, 2... m]\), and the average rank per algorithm will also be \([1, 2... m]\). Thus, the value for \(|r_{i.}-r_{i'.}|\) will be |i’-i| (assuming the indices \(i\) are sorted by their average rank). It will be exactly \(1\) for successive pairs of algorithms. We can therefore compute the minimum sample size \(n\) so that \(P(Q \sim q_{m, \infty} \geq Y) \leq \alpha\). Let \(q_{m, \infty, \alpha}\) be the critical value for the studentized range at significance level \(\alpha\). Then we will reject the null hypothesis for two successive algorithms if:

\[\frac{\sqrt{2}1}{\sqrt{\frac{m(m+1)}{6n}}} \geq q_{m, \infty, \alpha}\]

\[\frac{12n}{m(m+1)} \geq q^2_{m, \infty, \alpha}\]

\[n \geq \frac{m(m+1)}{12}q^2_{m, \infty, \alpha}\]

This gives us the minimum value for the sample size in order for Nemenyi to be able to correctly identify the distributions as fully separated (i.e. all p-values are \(< \alpha\)). For \(m=5\), we already have \(n\geq38\). At \(m=10\), it is \(n\geq184\) (see figure below).

Power comparison of the one- and two-sided Wilcoxon tests

Given two algorithms \(A_i\), \(A_j\) such that the \(A_i\) is “better” than \(A_j\) (\(S_{i.} > S_{j.}\)), the two-sided Wilcoxon test will have higher (i.e. less significant) p-values that the one-sided test where \(H_0\) is \(S_{i.} \leq S_{j.}\). In fact, we will always have \(P_{2S} = 2min(P_{1S}^+, P_{1S}^-)\), where \(P_{1S}^+\) and \(P_{1S}^-\) are the p-values for both sides of the one-sided test.

If we compute the pairwise p-values for \(m\) algorithms, and use these to compute CI for ranks without applying Holm’s correction, we will therefore always have more significant differences between algorithms using the one-sided test. We would therefore expect a much higher power and more Type I errors using this method. This, however, isn’t the case in our experiments.

The reason for that is Holm’s correction. Let us look at an example, using 5 algorithms, a sample size of 20 and a separation factor of \(1.0\). We generate the following results:

The “raw” pairwise p-values (without Holm’s correction) in this sample for the two-sided tests are:

For the one-sided test, we have two p-values per pair of algorithms: one for the “test to the left”, one for the “test to the right”. The minimum of those two values is always \(\frac{1}{2}\) as the p-value of the two-sided test. The p-value on the other side are all very close to 1:

In both the one-sided and two-sided version, the CI for the ranks are completely distinct: \([1, 1], [2, 2], [3, 3], [4, 4], [5, 5]\).

Holm’s correction is applied “row-by-row” on these tables. For each row, we start from the lowest p-value and multiply it by \(m-1 = 4\), then the second by \(m-2 = 3\), until we find a p-value \(> 0.05\), in which case we stop and put all subsequent p-values at 1. In the two-sided approach, the lowest p-value will always correspond to the pair of algorithms that are “furthest away” (so for the first two rows, \(A_5\), for the last two, \(A_1\), and in the middle row it will depend on the exact random sample but in this case it’s \(A_5\)). We end up with the following p-values:

For the one-sided test, Holm’s correction is applied separately for each table. This measn that the lowest p-value will be for the “furthest” algorithm on the side being tested:

In this last table, we know have one non-significant comparison between algorithms \(A_1\) and \(A_2\) (bold). The CI using the one-sided test are therefore no longer completely distinct, as we have: \([1, 1], [1, 2], [3, 3], [4, 4], [5, 5]\).

Thus, even though the one-sided tests by themselves are more powerful than their two-sided counterpart, they may lead to less distinctive CI for the rankings after the application of Holm’s correction to the p-values.