Selection criteria

Pharmpy tools support multiple different selection criteria, and most tools allow the user to specify which to use.

Objective function value (OFV)

Uses the objective function value as a selection critera, most often \(-2\text{log}\mathcal{L}(\hat{\theta} \mid y)\).

Likelihood ratio test (LRT)

Performs a likelihood ratio test, where \(\alpha\) is the cutoff p-value.

Note

In tools that use likelihood ratio test (e.g. COVSearch and RUVSearch) the models are always nested, and thus LRT is applicable. However, if this is set for some of the other tools this might not be the case, but currently Pharmpy does not detect whether models are nested. Thus, if using this in another tool, LRT might not be applicable for all models in all tools.

Aikaike information criterion (AIC)

The Aikaike information criterion can also be calculated. In order to calculate the AIC in Pharmpy standalone, you can use the pharmpy.modeling.calculate_aic()-function. The AIC is described as follows:

\[AIC = -2\text{log}\mathcal{L}(\hat{\theta} \mid y) + 2dim({\theta})\]

Bayesian information criterion (BIC)

Pharmpy supports multiple types of BIC. In order to calculate the BIC in Pharmpy standalone, you can use the pharmpy.modeling.calculate_bic()-function.

Fixed

For linear models, the BIC [1] is described as follows:

\[BIC_{fixed} = -2\text{log}\mathcal{L}(\hat{\theta} \mid y) + dim(\theta)\text{log} n_{tot}\]

Where \(\theta\) is estimated parameters and \(n_{tot}\) is the number of observations.

Mixed

For mixed effect models, BIC takes the number of random vs. fixed parameters into consideration [2], and is described as follows:

\[BIC_{mixed} = -2\text{log}\mathcal{L}(\hat{\theta} \mid y) + dim({\theta_R})\text{log} N + dim({\theta_F})\text{log} n_{tot}\]

Where \(\theta_R\) is random parameters, \(N\) is the number of subjects, \(\theta_F\) is fixed parameters, and \(n_{tot}\) is the number of observations.

IIV

The BIC for IIV [3] is described as follows:

\[BIC_{IIV} = -2\text{log}\mathcal{L}(\hat{\theta} \mid y) + dim({\omega_{IIV}})\text{log} N\]

Where \(\omega_{IIV}\) is IIV parameters and \(N\) is the number of subjects.

Random

The BIC for random effects is described as follows:

\[BIC_{random} = -2\text{log}\mathcal{L}(\hat{\theta} \mid y) + dim({\theta})\text{log} N\]

Where \(\theta\) is parameters and \(N\) is the number of subjects.

Modified BIC (mBIC)

The mBIC selection criterion [4] is an additional penalty term, and is combined with any of the above mentioned types of BIC. mBIC was introduced to address the problem of multiple testing, and penalizes against the size of the search space and the complexity of the candidate model.

Currently, mBIC supports evaluation of the structural PK model and the IIV model, and there are two different penalty calculations for each of them. This means that when e.g. evaluation the structural model, the ModelSearch tool only uses mBIC for the structural model.

Note

mBIC is currently only supported by ModelSearch, IIVSearch, and IOVSearch.

mBIC for the structural model

In mBIC for the structural model, different components of the model structure penalizes differently (see table below). For the structural model, we can define \(mBIC_{structural}\) with the following equation:

\[mBIC_{structural} = BIC_{mixed} + 2 k_M \text{log}\frac{M}{E}\]

Where \(k_M\) is the number of structural predictors in the candidate model, \(M\) is the number of structural predictors in the search space, and \(E\) is the a priori expected number of predictors. The \(E\)-value is set by the user, and is bounded by \(0 < E < M\) (where 0 is no expected number of predictors and thus infinite penalty, and \(M\) is all predictors in the search space which means no penalty).

The number of predictors is defined accordingly:

Feature	Contribution to \(M\)	Contribution to \(k_M\)
ABSORPTION(FO)	0	0
ABSORPTION(ZO)	0	0
ABSORPTION(SEQ-ZO-FO)	1	1
LAGTIME(OFF)	0	0
LAGTIME(ON)	1	1
TRANSITS(0..x)	\(x\)	1
PERIPHERALS(0..x)	\(x\)	\(x\)
ELIMINATION(FO)	0	0
ELIMINATION(MM)	0	0
ELIMINATION(MIX-FO-MM)	1	0

We can view this as four different categories:

• Mutually exclusive: one of options have to be selected

  • FO/ZO absorption, FO/MM elimination

  • Contribution to \(M\) and \(k_M\) are each 0

• On/off: whether a component is active or not

  • Lagtime or no lag time

  • Contribution to \(M\) and \(k_M\) are each 0 when “off”, 1 when “on”

• Stackable: combinations that can be stacked

  • Peripherals, going from FO/ZO absorption to SEQ-ZO-FO, going from FO/MM elimination to MIX-FO-MM

  • Contribution to \(M\) and \(k_M\) are both increased increased by the number of options (e.g. having 0-2 peripherals in the search space will give \(M=2\) and \(k_M\) of how many peripherals is in candidate)

• Non-stackable: numeric option which is mutually exclusive

  • Transits

  • Contribution to \(M\) is the number of non \(k_M\) are both increased increased by the number of non-stackable options (e.g. having 0-2 transits in the search space will give \(M=2\) and \(k_M\) of 1 in any candidate with transits)

As an example, consider the default search space for an oral PK drug in the AMD tool, and some example candidates:

ABSORPTION([FO,ZO,SEQ-ZO-FO])
ELIMINATION(FO)
LAGTIME([OFF,ON])
TRANSITS([0,1,3,10],*)
PERIPHERALS([0,1])

M = 6

Candidate example 1Candidate example 2Candidate example 3

ABSORPTION(FO)
ELIMINATION(FO)
LAGTIME(OFF)
TRANSITS(0)
PERIPHERALS(1)

k_M = 1

If the search space includes transits but without depot, the contribution to \(M\) is still the same as if the depot was kept:

ABSORPTION([FO,ZO,SEQ-ZO-FO])
ELIMINATION(FO)
LAGTIME([OFF,ON])
TRANSITS([0,1,3,10],NODEPOT)
PERIPHERALS([0,1])

M = 6

If there is only one value in a category, the contribution to \(M\) and \(k_M\) will both be 0, no matter the value (since no choice is being made):

ABSORPTION([FO,ZO,SEQ-ZO-FO])
ELIMINATION(FO)
LAGTIME([OFF,ON])
TRANSITS([0,1,3,10],*)
PERIPHERALS(1)

M = 5

Candidate example 1Candidate example 2Candidate example 3

ABSORPTION(FO)
ELIMINATION(FO)
LAGTIME(OFF)
TRANSITS(0)
PERIPHERALS(1)

k_M = 0

mBIC for the IIV model

In mBIC for the IIV model, the penalty consists of one part for the diagonal omegas and one part for the off-diagonals. For the IIV model, we can define \(mBIC_{IIV}\) with the following equation:

\[mBIC_{IIV} = BIC_{IIV} + 2 k_p \text{log}\frac{p}{E_p} + 2 k_q \text{log}\frac{q}{E_q}\]

Where \(k_p\) and \(k_q\) are the number of diagonal and off-diagonal omega predictors in the candidate model, \(p\) and \(q\) are the number of diagonal and off-diagonal omega predictors in the search space, and \(E_p\) and \(E_q\) is the a priori expected number of diagonal and off-diagonal omega predictors.

If you’re only testing the number of diagonal omegas, the penalty related to the off-diagonal omegas is omitted.

AMD selection criteria

Different subtools in AMD uses different selection criteria, mostly depending on whether the candidate models are nested or not.

Subtool	Selection criteria
ModelSearch	BIC (mixed)
StructSearch	BIC (mixed)
IIVSearch	BIC (IIV)
RUVSearch	LRT
IOVSearch	BIC (random)
COVSearch	LRT

References

[1]

Gideon Schwarz. Estimating the Dimension of a Model. The Annals of Statistics, 6(2):461–464, 1978. Publisher: Institute of Mathematical Statistics.

[2]

Maud Delattre, Marc Lavielle, and Marie-Anne Poursat. A note on BIC in mixed-effects models. Electronic Journal of Statistics, 8(1):456–475, January 2014. Publisher: Institute of Mathematical Statistics and Bernoulli Society. URL: https://projecteuclid.org/journals/electronic-journal-of-statistics/volume-8/issue-1/A-note-on-BIC-in-mixed-effects-models/10.1214/14-EJS890.full (visited on 2025-08-29), doi:10.1214/14-EJS890.

[3]

Maud Delattre and Marie-Anne Poursat. An iterative algorithm for joint covariate and random effect selection in mixed effects models. The International Journal of Biostatistics, pages /j/ijb.ahead–of–print/ijb–2019–0082/ijb–2019–0082.xml, May 2020. doi:10.1515/ijb-2019-0082.

[4]

Xiaomei Chen, Ya-Han Hsu, Rikard Nordgren, Stella Belin, Emilie Schindler, Andrew C. Hooker, and Mats O. Karlsson. A modified Bayesian information criterion (mBIC) with multiple testing correction for population pharmacokinetic model building. 2025. URL: https://www.page-meeting.org/default.asp?abstract=11600.