Iteratively fitted models (IFM) and preferred iterations

This topic covers the concept of iteratively fitted models (IFM) and the related selection mechanism for preferred iterations. For an introduction with application examples, refer to the vignette vignette("ifm").

Iteratively fitted models

We use the term iteratively fitted model for classes of models where the fitting process returns not just one single model (or model parameterization), but rather a sequence of models (or model parameterizations) of increasing structural complexity. We will refer to a single models in this sequence as a (model) iteration.

Prominent examples (and currently the only two instances implemented in the package) are:

• gradient boosting: fm_xgb() is a wrapper for xgb.train() from package xgboost. The iterations arise from successively adding trees to the current fitted model.

• Lasso regression and elastic net models: fm_glmnet() is a wrapper for glmnet() from package glmnet. Here, iterations results from successively lowering the penalty parameter lambda.

In the conceptual framework described here, it is crucial that predictions can be computed from each iteration, such that iteration-wise training and test errors can be calculated, too.

Evaluation log

An evaluation log is essentially a table having one row per iteration and including columns for the training and/or test error of the respective iteration. The evaluation log of a “model” (or fitted model) includes the training error only, while the evaluation log of a cross-validated model (object of class “cv”) has both training and test errors.

The modeltuner package has a function evaluation_log() that computes evaluation logs for models and “cv” objects, and a plot method, plot.evaluation_log().

Selection criteria for iterations

The function cv() has a mechanism for specification of preferred iterations. The user specifies one or several selection criteria in the parameter iter= in a call to cv(). By default (at package startup), the iteration having minimal test error will be selected. This default can however be modified by setting the option "cv_iter" (see modeltuner_options). The effect of determining a preference criterion is that application of cv_performance() to the “cv” object will show the results (training and test error) for the iteration selected based on this criterion.

The preferred iteration can be altered by specifying iter=criterion in cv(), where criterion is a selection rule that is applied internally to the cv's evaluation log, resulting in a choice of the preferred iteration. The minimal test error rule corresponds to setting iter=crit_min(); alternative selection criteria are presented in crit_iter.

Multiple criteria can be requested, as in crit_list(crit_min(), crit_1se()). In this case, cv_performance() will show the results related to first criterion (that listed at first position in crit_list(...)).

A “cv” object's selection criteria and selected iterations are displayed when you print it. You can extract the information related to iterations with extract_pref_iter().

Modifying the preference (criteria) of a “cv” object

set_pref_iter() and expand_pref_iter() offer options to change a cv's preferences with respect to selection of iterations subsequently. Note however that certain analyses remain available only for iterations related to the criteria stated in the iter argument of the original call to cv(). In particular, this is the case for running cv_performance() with alternative metrics. The reason of this limitation is that the full sets of predictions from cross-validation are stored only for the preferred iterations resulting from the originally stated criteria.

Setting keep_fits=TRUE in cv() is an option to circumvent there limitations (but can be excessively memory-consuming). In this case, the model fits from cross-validation are stored in the resulting object, such that predictions from all iterations remain obtainable. In particular, this makes it possible to compute evaluation logs with alternative metrics.

See expand_pref_iter() and set_metric() for more manipulations of “cv” objects.

Fitting an IFM after a cross-validation

When a “cv” object based on an IFM is converted back to a model (with tune() or with extract_model()), the information on preferred iterations is stored in an attribute. Likewise, when fit() is used to obtain the resulting fitted model, the model corresponding to the preferred iteration is returned.