First stable release. The interface is settled and breaking changes from here go through a deprecation cycle.
applications is a new dataset: 800 synthetic loan applications, seven
predictors and a default indicator, shipped so that a question about a
ranking has an answer to check against. The effects a ranking ought to
recover are stored on the data frame as attr(applications, "effects").bureau_score and income come from one latent creditworthiness and
correlate at 0.84, so the credit for the signal has to be divided somehow,
and both measures below invert them relative to the truth. prior_arrears is
the largest effect in the data and takes six distinct values: on a forest of
500 trees, impurity importance puts it fourth while permutation importance
puts it first. Over the seven predictors the two agree with each other at a
Kendall tau of 0.71, and against the truth at 0.59 for impurity and 0.88 for
permutation.importance_judges() builds the panel from fitted models along the four
axes of the roadmap, which are method, model, seed and resample, and records
the provenance, raw scores and per-judge weights on the returned judges
object, which consensus_rank() consumes weights and all. Supported engines:
randomForest and ranger; resamples come from rsample::vfold_cv() or
rsample::bootstraps(), refit on the analysis set and scored on the
assessment set.
Importance backends: importance_permutation(), importance_mdi() and
importance_loco() are implemented natively against the supported engines;
importance_shap() delegates to kernelshap. The originally planned SHAP
backend, fastshap, was archived from CRAN on 2026-05-27 (and vip is gone too),
so kernelshap is the dependency of record.
judge_weights() expands method weights over a panel, or weighs each judge
by its mean Emond-Mason tau_x agreement with the rest of the panel
(by = "reliability").
rank_confsets() bootstraps the judges to put a rank confidence set around
the consensus, prob_topk() reports the probability that a variable belongs
to the top k, and rank_select() keeps the variables whose whole interval
clears a threshold. Resampling is done with multinomial judge weights rather
than by rebuilding the panel, which ConsRank treats identically.
rank_confsets(type = "data") resamples the rows instead of the panel:
every replicate draws the data with replacement, refits every model and
rebuilds the panel from scratch. It answers the question the judge bootstrap
cannot, namely whether the ranking would survive another sample, and it needs
a panel from importance_judges(), which now carries the recipe that built
it. A replicate mirrors the recipe: a panel that judged out of sample keeps
judging out of sample, on the rows the bootstrap left behind. A resamples
axis is replaced by that in-bag/out-of-bag split, so each replicate votes with
models x methods judges rather than models x methods x V. Replicates that
fail, the usual case being a response class too rare to survive a draw, are
dropped, counted and warned about instead of killing the run.
item_consensus() scores every judge against the consensus, so that a low
tau_x can be read as a split panel rather than as noise.
autoplot() for rank_confsets draws the consensus ranking with its
intervals.
autoplot() for consensus_rank draws the consensus over every rank the
judges gave, point area being the number of judges at each rank. One number
per variable is equally consistent with a panel that agreed and one that was
split, and this is the difference. It is per variable, where tau_x and
item_consensus() measure it per judge. It was the last entry point still
raising "not implemented"; the helper that raised it is gone with it.
Two vignettes for the parts the task-shaped ones do not cover.
vignette("reference") is the map: what each exported function is for, what
it returns, which mistake it exists to prevent, and a table for picking one
by the question you are asking. vignette("theory") is the argument behind
the code: why the consensus is a Kemeny median rather than an average of
ranks, why ties and non-unique optima are kept rather than broken, why
tau_x and not Kendall's tau_b, what each of the two bootstraps is
estimating, and why the heterogeneity test needs a reference distribution
instead of a silhouette threshold.
vignette("credit-scoring") runs the four axes at once on a synthetic
portfolio of two model families, two methods, three folds and two seeds,
making 24 judges, and is no longer a placeholder. Its point is what the
package refuses to say: the three real drivers all come back with the interval
[1, 3], so rank_select(threshold = 2) returns nothing, and only the
unordered set of three is defensible. judge_clusters() then finds the panel
divided along method rather than model family, which is mean decrease in
impurity preferring the continuous variable to the small count.
vignette("against-set-stability") separates this package from stabm:
two panels with identical Nogueira and Jaccard set stability, one of which has
a completely unordered top three.
algorithm = "auto" uses exact branch-and-bound up to ten variables rather
than twelve. Its cost is not smooth in p and the panels this package
produces are the hard ones, because unimportant variables tie near zero. On
tied panels of thirty judges one exact solve took 0.010 s at p = 10, 0.78 s
at p = 11 and 280 s at p = 12, while "quick" returned the identical
consensus and tau_x on twenty out of twenty tied panels at p = 10.
consensus_rank() no longer reports an arbitrary one of the equally optimal
consensus rankings. The Kemeny median need not be unique and ConsRank
returns every ranking attaining the minimum; taking the first was not neutral,
because which came first depended on the order of the columns. On a symmetric
panel of three indistinguishable variables, permuting the columns changed the
winner, and in simulation a variable of pure noise outranked a real one by
sitting further left. Several optima arose in 57% to 98% of replicates there,
so this was the normal case rather than an edge case. The optima are now
combined, each variable taking its average position over them and those that
come out equal being tied, so variables the objective cannot separate are
reported as equal. The full set stays in consensus_all. The same correction
applies to both bootstrap branches of rank_confsets() and to
item_consensus(), which now scores judges against the consensus the user was
shown.
judge_clusters() is implemented: k-medians in the space of the Kemeny-Snell
distance, started from the exactly optimal set of medoids and refined until
the assignments settle, with each group's centre the Kemeny median of its own
members. It draws nothing from the RNG and returns the same answer on every
call. k = NULL divides the panel only when it splits more sharply than a
single population of judges would, tested against reference panels spread to
match, without which the silhouette alone divided a homogeneous panel 62% of
the time. A panel the test rejects is reported as divided in two, which is
the division the evidence is about: reading k off the largest silhouette
instead attaches an uncalibrated number to a calibrated decision, and it
measured worse everywhere it differed: on ten judges or more it never once
chose k = 2, and recovery of a true two-group partition fell as judges
were added (0.877 at six, 0.747 at ten, 0.690 at sixteen) where it now rises
to 0.943, at an identical false division rate. autoplot() shows the panel in
Kemeny-Snell space. Behaviour is measured in
inst/simulations/cluster-recovery.R rather than asserted.
consensus_rank() no longer leaks ConsRank's console output on degenerate
panels. The previous calling handler never invoked muffleMessage(), so it
suppressed nothing, and the messages that matter are emitted with print()
and cannot be caught by a handler at all. They are captured instead.
consensus_rank() rejects a judge whose best rank is not 1. The previous
check tested any(x < 1), which admitted rankings starting at 2 and admitted
a matrix of importance scores, contradicting the documented contract. The
error now names the offending judges and points at importance_to_rank().
prob_topk() and rank_select() are measured for the first time, in
inst/simulations/select-calibration.R. Both read the bootstrap rankings the
resample bug used to repeat, so neither had ever been seen working. On 300
panels of eight close predictors and eighty rows: prob_topk() is
conservative in the middle of its range and accurate at the ends, so that a
variable given 0.44 is in the top k 56% of the time and one given 0.98 is
there 98% of the time. rank_select() falsely selects at most 3% of what it
selects, while selecting between a third and a half of the variables that
deserved it. Both numbers are now in the documentation.
The warning in ?rank_confsets that a data-bootstrap interval "need not
contain the consensus rank" described the resample bug, not the method. It
rested on one panel with an interval of [3, 5] around a consensus rank of 2,
produced by the bootstrap that repeated its draws and so reported intervals
too narrow to be believed. Re-measured on the fixed machinery over the same
design, it happened in none of 2,400 variable-replicates. What is real, and
now documented in its place, is the drift: a replicate ranks on about 0.632n
distinct rows, so the top of the ranking drifts down and the bottom drifts up
by 0.55 ranks for the second variable and 1.00 for the eighth.
rank_confsets(type = "data") draws a fresh resample for every replicate.
panel_scores() calls set.seed() for the seed axis and used to leave the
session stream parked where the last seed put it; the bootstrap loop draws its
next resample from that stream, so every replicate set out from the same state
and the resamples fell into a cycle a few draws long. Four hundred requested
replicates held about eight distinct ones, and n_boot bought almost nothing
beyond the first few. The panel now puts the caller's stream back where it
found it, so importance_judges() no longer moves it either, and the
bootstrap restores it around each replicate, which keeps the replicates
independent whatever a backend does with the RNG. The judge bootstrap was
never affected: nothing on that path seeds.
?rank_confsets and vignette("stability") report the coverage the intervals
were measured to have rather than leaving the nominal level to speak for
itself, and the two bootstraps miss it in opposite directions. Resampling the
data covered 0.966 to 0.998 of the time across the six cells measured, at or
above the nominal 0.95, and wide: 5.1 of 8 ranks on the hardest cell.
Resampling the judges covered 0.582 to 0.929, and never reached the nominal
level. Fifty replicates are enough for the data bootstrap (0.966 against 0.970
at 200). The simulation ships as inst/simulations/rank-coverage.R.
consensus_rank() retains the panel and the judge weights in the returned
object, as $judges and $weights. rank_confsets() bootstraps the judges
and cannot do that from a ranking alone.importance_judges() records the recipe that built the panel, meaning the
fits, the data, the target, the methods, the axes and the backend settings, as
an attribute of the returned judges object, so that rank_confsets() can
rebuild the panel on a bootstrap sample without being handed them all again.
The panel is correspondingly as large as the objects it refers to.rank_confsets() gained type, and n_boot now defaults by kind: 500
replicates for the judge bootstrap, 50 for the data bootstrap, which refits
every model on every replicate. A long data bootstrap announces its projected
cost, measured on the first replicate.consensus_rank() computes the Kemeny median of a panel of judges, with
ties, optional judge weights, and automatic selection of the ConsRank solver
from the number of variables.importance_to_rank() converts importance scores to rankings, preserving
ties between variables a judge scores equally.print() method for the consensus_rank class, reporting the tau_x
agreement and warning when several consensus rankings are equally optimal.judge_clusters() (F4) was the last.