Linear Models

Orthogonal Regression

class skmatter.linear_model._base.OrthogonalRegression(use_orthogonal_projector=True, linear_estimator=None)

Orthogonal regression by solving the Procrustes problem

Linear regression with the additional constraint that the weight matrix must be an orthogonal matrix/projection. It minimizes the Procrustes problem:

\[\min_\Omega ||y - X\Omega\||_F \quad\mathrm{subject\ to}\quad \Omega^T\Omega=I\]
Parameters:

  • use_orthogonal_projector (bool, default=True) –

    Controls if orthogonal projectors are used to predict y fitting on X. If this parameter is set to False X and y are padded with zeros to the larger number of features of X and y. The projection method is similar to the procedure in the computation GFRD in the first version of Ref. [Goscinski2021]. The method has been adapted obtain a full weight matrix.

    The projection can introduce nonanalytic behavior with respect to changes in dimensions of X for cases where X n_features > y n_targets. See examples/OrthogonalRegressionNonAnalytic_no-doc.ipynb

  • linear_estimator (object implementing fit/predict, default=None) – The linear estimator is used when use_orthogonal_projector is set to True, to compute the projection matrix

max_components_

The source X and target y are padded with zeros to match in feature/target dimension, when use_orthogonal_projector is set to False. This attribute is set to the maximum of the feature and target dimension.

Type:

int

coef_

Weight matrix. The shape (max_components, max_components) is used if use_orthogonal_projector is set to False.

Type:

ndarray of shape (n_features,) or (n_targets, n_features) or (max_components, max_components)

Ridge Regression with Two-fold Cross Validation

class skmatter.linear_model._ridge.RidgeRegression2FoldCV(alphas=(0.1, 1.0, 10.0), alpha_type='absolute', regularization_method='tikhonov', scoring=None, random_state=None, shuffle=True, n_jobs=None)

Ridge regression with an efficient 2-fold cross-validation method using the SVD solver. Minimizes the objective function:

\[\|y - Xw\|^2_2 + \alpha \|w\|^2_2,\]

while the alpha value is determined with a 2-fold cross-validation from a list of alpha values. It is more efficient than doing a 2-fold cross-validation using sklearn.linear_model.RidgeCV. The advantage over sklearn.linear_model.RidgeCV using leave-one-out cross-validation (LOOCV) [loocv] needs to be analyzed more in detail. Internal benchmarks suggest that it is more efficient than the LOOCV in sklearn.linear_model.RidgeCV for feature sizes < 600 and in general more accurate, see issue #40. However, it is constraint to a svd solver for the matrix inversion. It offers additional functionalities in comparison to sklearn.linear_model.Ridge: The regularaization parameters can be chosen relative to the largest eigenvalue of the feature matrix as well as regularization method. Details are explained in the Parameters section.

Parameters:

  • alphas (ndarray of shape (n_alphas,), default=(0.1, 1.0, 10.0)) – Array of alpha values to try. Regularization strength; must be a positive float. Regularization improves the conditioning of the problem and reduces the variance of the estimates. Larger values specify stronger regularization.

  • alpha_type (str, default="absolute") – “absolute” assumes that alphas are absolute values and does not scale them, “relative” assumes that alphas as relative values in the range [0,1] and scales them with respect to the largest eigenvalue.

  • regularization_method (str, default="tikhonov") – “tikhonov” uses the alphas with standard Tihhonov regularization, “cutoff” uses the alphas as a cutoff for the eigenvalues of the kernel, The case "cutoff" with "relative" ``alphas has the same effect as the rcond parameter in e.g. numpy.linalg.lstsq. Be aware that for every case we always apply a small default cutoff dependend on the numerical accuracy of the data type of X in the fitting function.

  • random_state (int or RandomState instance, default=None) – Controls the shuffling applied to the data before applying the split. Pass an int for reproducible output across multiple function calls. See random_state glossary from sklearn (external link)

  • shuffle (bool, default=True) – Whether or not to shuffle the data before splitting. If shuffle=False then stratify must be None.

  • scoring (str, callable, default=None) – A string (see model evaluation documentation) or a scorer callable object / function with signature scorer(estimator, X, y). If None, the negative mean squared error is used.

  • n_jobs (int, default=None) – The number of CPUs to use to do the computation. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See n_jobs glossary from sklearn (external link) for more details.

cv_values_

shape (n_samples, n_targets, n_alphas), optional Cross-validation values for each alpha (only available if store_cv_values=True and cv=None). After fit() has been called, this attribute will contain the mean squared errors (by default) or the values of the {loss,score}_func function (if provided in the constructor).

Type:

ndarray of shape (n_samples, n_alphas) or

coef_

Weight vector(s).

Type:

ndarray of shape (n_features) or (n_targets, n_features)

intercept_

Independent term in decision function. Set to 0.0 if fit_intercept = False.

Type:

float or ndarray of shape (n_targets,)

alpha_

Estimated regularization parameter.

Type:

float

best_score_

Score of base estimator with best alpha.

Type:

float

References

[loocv]

Rifkin “Regularized Least Squares.” https://www.mit.edu/~9.520/spring07/Classes/rlsslides.pdf

PCovR

Principal Covariates Regression is a linear model, see PCovR.