user_guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# asgl <img src=\"figures/logo.png\" align=\"right\" height=\"150\" alt=\"funq website\" /></a>\n",
    "\n",
    "[![Downloads](https://pepy.tech/badge/asgl)](https://pepy.tech/project/asgl)\n",
    "[![Downloads](https://pepy.tech/badge/asgl/month)](https://pepy.tech/project/asgl)\n",
    "[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)\n",
    "[![Package Version](https://img.shields.io/badge/version-2.0.0-blue.svg)](https://cran.r-project.org/package=asgl)\n",
    "\n",
    "## Introduction\n",
    "\n",
    "The `asgl` package is a versatile and robust tool designed for fitting a variety of regression models, including linear regression, quantile regression, and various penalized regression models such as Lasso, Group Lasso, Sparse Group Lasso, and their adaptive variants. The package is especially useful for simultaneous variable selection and prediction in both low and high-dimensional frameworks.\n",
    "\n",
    "The primary class available to users is the `Regressor` class, which is detailed later in this document.\n",
    "\n",
    "`asgl` is based on cutting-edge research and methodologies, as outlined in the following papers:\n",
    "\n",
    "* [Adaptive Sparse Group Lasso in Quantile Regression](https://link.springer.com/article/10.1007/s11634-020-00413-8)\n",
    "* [`asgl`: A Python Package for Penalized Linear and Quantile Regression](https://arxiv.org/abs/2111.00472)\n",
    "\n",
    "For a practical introduction to the package, users can refer to the user guide notebook available in the GitHub repository. Additional accessible explanations can be found on [Towards Data Science: Sparse Group Lasso](https://towardsdatascience.com/sparse-group-lasso-in-python-255e379ab892) and [Towards Data Science: Adaptive Lasso](https://towardsdatascience.com/an-adaptive-lasso-63afca54b80d)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies\n",
    "\n",
    "asgl requires: \n",
    "\n",
    "* Python >= 3.9\n",
    "* cvxpy >= 1.2.0\n",
    "* numpy >= 1.20.0\n",
    "* scikit-learn >= 1.0\n",
    "* pytest >= 7.1.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## User installation\n",
    "\n",
    "The easiest way to install asgl is using `pip`:\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "pip install asgl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing\n",
    "\n",
    "After installation, you can launch the test suite from the source directory (you will need to have `pytest >= 7.1.2` installed) by runnig:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "pytest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key features:\n",
    "\n",
    "The `Regressor` class includes the following list of parameters:\n",
    "\n",
    "* model: str, default='lm'\n",
    "  * Type of model to fit. Options are 'lm' (linear regression) and 'qr' (quantile regression).\n",
    "* penalization: str or None, default='lasso'\n",
    "  * Type of penalization to use. Options are 'lasso', 'gl' (group lasso), 'sgl' (sparse group lasso), 'alasso' (adaptive lasso), 'agl' (adaptive group lasso), 'asgl' (adaptive sparse group lasso), or None.\n",
    "* quantile: float, default=0.5\n",
    "  * Quantile level for quantile regression models. Valid values are between 0 and 1.\n",
    "* fit_intercept: bool, default=True\n",
    "  * Whether to fit an intercept in the model.\n",
    "* lambda1: float, default=0.1\n",
    "  * Constant that multiplies the penalization, controlling the strength. Must be a non-negative float i.e. in `[0, inf)`. Larger values will result in larger penalizations.\n",
    "* alpha: float, default=0.5\n",
    "  * Constant that performs tradeoff between individual and group penalizations in sgl and asgl penalizations.\n",
    "        ``alpha=1`` enforces a lasso penalization while ``alpha=0`` enforces a group lasso penalization.\n",
    "* solver: str, default='default'\n",
    "  * Solver to be used by `cvxpy`. Default uses optimal alternative depending on the problem. Users can check available solvers via the command `cvxpy.installed_solvers()`.\n",
    "* weight_technique: str, default='pca_pct'\n",
    "  * Technique used to fit adaptive weights. Options include 'pca_1', 'pca_pct', 'pls_1', 'pls_pct', 'lasso', 'unpenalized', and 'sparse_pca'. For low dimensional problems (where the number of variables is smaller than the number of observations) the usage of the 'unpenalized' weight_technique alternative is encouraged. For high dimensional problems (where the number of variables is larger than the number of observations) the default alternative is encouraged.\n",
    "* individual_power_weight: float, default=1\n",
    "  * Power to which individual weights are raised. This parameter only has effect in adaptive penalizations. ('alasso' and 'asgl').\n",
    "* group_power_weight: float, default=1\n",
    "  * Power to which group weights are raised. This parameter only has effect in adaptive penalizations with a grouped structure ('agl' and 'asgl').\n",
    "* variability_pct: float, default=0.9\n",
    "  * Percentage of variability explained by PCA, PLS, and sparse PCA components. This parameter only has effect in adaptiv penalizations where `weight_technique` is equal to 'pca_pct', 'pls_pct' or 'sparse_pca'.\n",
    "* lambda1_weights: float, default=0.1\n",
    "  * The value of the parameter ``lambda1`` used to solve the lasso model if ``weight_technique='lasso'``\n",
    "* spca_alpha: float, default=1e-5\n",
    "  * Sparse PCA parameter. This parameter only has effect if `weight_technique='sparse_pca'`See scikit-learn implementation for more details.\n",
    "* spca_ridge_alpha: float, default=1e-2\n",
    "  * Sparse PCA parameter. This parameter only has effect if `weight_technique='sparse_pca'`See scikit-learn implementation for more details.\n",
    "* individual_weights: array or None, default=None\n",
    "  * Custom individual weights for adaptive penalizations. If this parameter is informed,\n",
    "        it overrides the weight estimation process defined by parameter ``weight_technique`` and allows the user to\n",
    "        provide custom weights. It must be either `None` or be an array with  non-negative float values and length equal to the number of variables.\n",
    "* group_weights: array or None, default=None\n",
    "  * Custom group weights for adaptive penalizations.  If this parameter is informed,\n",
    "        it overrides the weight estimation process defined by parameter ``weight_technique`` and allows the user to\n",
    "        provide custom weights. It must be either `None` or be an array with  non-negative float values and length equal to the number of groups (as defined by `group_index`)\n",
    "* tol: float, default=1e-3\n",
    "  * Tolerance for coefficients to be considered zero.\n",
    "* weight_tol: float, default=1e-4\n",
    "  * Tolerance value used to avoid ZeroDivision errors when computing the weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: Linear Regression with Lasso Penalization"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:45.599319Z",
     "start_time": "2024-10-01T19:19:44.744175Z"
    }
   },
   "source": [
    "from sklearn.datasets import make_regression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from asgl import Regressor\n",
    "\n",
    "# Generate synthetic regression data\n",
    "X, y = make_regression(n_samples=1000, n_features=50, n_informative=25, bias=10, noise=5, random_state=42)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=250)\n",
    "\n",
    "# Create a Regressor object configured for linear regression with Lasso penalization\n",
    "model = Regressor(model='lm', penalization='lasso', lambda1=0.1)\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions on the test data\n",
    "predictions = model.predict(X_test)\n",
    "\n",
    "# Evaluate the model's performance using mean squared error\n",
    "mse = mean_squared_error(predictions, y_test)\n",
    "print(f\"Mean Squared Error: {mse}\")"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Squared Error: 26.946356773986654\n"
     ]
    }
   ],
   "execution_count": 1
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: Quantile Regression with Adaptive Sparse Group Lasso Penalization\n",
    "\n",
    "Group-based penalizations like Group Lasso, Sparse Group Lasso, and their adaptive variants, assume that there is a group structure within the regressors. This structure can be useful in various applications, such as when using dummy variables where all the dummies of the same variable belong to the same group, or in genetic data analysis where genes are grouped into genetic pathways.\n",
    "\n",
    "For scenarios where the regressors have a known grouped structure, this information can be passed to the `Regressor` class during model fitting using the `group_index` parameter. This parameter is an array where each element indicates the group at which the associated variable belongs.  The following example demonstrates this with a synthetic group_index. The model will be optimized using scikit-learn's `RandomizedSearchCV` function."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:53.953823Z",
     "start_time": "2024-10-01T19:19:45.599319Z"
    }
   },
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "# Generate synthetic regression data\n",
    "X, y = make_regression(n_samples=1000, n_features=50, n_informative=25, bias=10, noise=5, random_state=42)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=250)\n",
    "\n",
    "# Define the group structure\n",
    "group_index = np.random.randint(1, 5, size=50)\n",
    "\n",
    "# Create a Regressor object configured for quantile regression with Adaptive Sparse Group Lasso penalization\n",
    "model = Regressor(model='qr', penalization='asgl', quantile=0.5)\n",
    "\n",
    "# Define the parameter grid for RandomizedSearchCV\n",
    "param_grid = {'lambda1': [1e-4, 1e-3, 1e-2, 1e-1, 1], 'alpha': [0, 0.2, 0.4, 0.6, 0.8, 1]}\n",
    "rscv = RandomizedSearchCV(model, param_grid, scoring='neg_median_absolute_error')\n",
    "rscv.fit(X_train, y_train, **{'group_index': group_index})"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(estimator=Regressor(model='qr', penalization='asgl'),\n",
       "                   param_distributions={'alpha': [0, 0.2, 0.4, 0.6, 0.8, 1],\n",
       "                                        'lambda1': [0.0001, 0.001, 0.01, 0.1,\n",
       "                                                    1]},\n",
       "                   scoring='neg_median_absolute_error')"
      ],
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomizedSearchCV(estimator=Regressor(model=&#x27;qr&#x27;, penalization=&#x27;asgl&#x27;),\n",
       "                   param_distributions={&#x27;alpha&#x27;: [0, 0.2, 0.4, 0.6, 0.8, 1],\n",
       "                                        &#x27;lambda1&#x27;: [0.0001, 0.001, 0.01, 0.1,\n",
       "                                                    1]},\n",
       "                   scoring=&#x27;neg_median_absolute_error&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;RandomizedSearchCV<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.model_selection.RandomizedSearchCV.html\">?<span>Documentation for RandomizedSearchCV</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>RandomizedSearchCV(estimator=Regressor(model=&#x27;qr&#x27;, penalization=&#x27;asgl&#x27;),\n",
       "                   param_distributions={&#x27;alpha&#x27;: [0, 0.2, 0.4, 0.6, 0.8, 1],\n",
       "                                        &#x27;lambda1&#x27;: [0.0001, 0.001, 0.01, 0.1,\n",
       "                                                    1]},\n",
       "                   scoring=&#x27;neg_median_absolute_error&#x27;)</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">best_estimator_: Regressor</label><div class=\"sk-toggleable__content fitted\"><pre>Regressor(alpha=0.4,\n",
       "          group_weights=array([0.00983229, 0.00973591, 0.00634475, 0.0053802 ]),\n",
       "          individual_weights=array([0.09969493, 0.0387666 , 0.14232849, 0.01544392, 0.09329751,\n",
       "       0.01960169, 0.0244537 , 0.01080244, 0.86445367, 0.11119265,\n",
       "       0.35847433, 0.06030927, 0.15469818, 0.079797  , 0.04920872,\n",
       "       0.05745112, 0.01504379, 0.07877179, 0.53581954, 0.06853869,\n",
       "       0.33682509, 0.01463754, 0.0117619 , 0.14399055, 0.01479135,\n",
       "       0.10432424, 0.05957878, 0.12252199, 0.82699305, 0.0603595 ,\n",
       "       0.11267987, 0.07130379, 0.01347789, 0.08094675, 0.02010636,\n",
       "       0.03199713, 0.07654506, 0.05585157, 0.28966196, 0.01444298,\n",
       "       0.08138551, 0.01016078, 0.10598385, 0.03385791, 0.01161466,\n",
       "       0.1085965 , 0.0134545 , 0.25188282, 0.05643086, 0.07706238]),\n",
       "          lambda1=1, model=&#x27;qr&#x27;, penalization=&#x27;asgl&#x27;)</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">Regressor</label><div class=\"sk-toggleable__content fitted\"><pre>Regressor(alpha=0.4,\n",
       "          group_weights=array([0.00983229, 0.00973591, 0.00634475, 0.0053802 ]),\n",
       "          individual_weights=array([0.09969493, 0.0387666 , 0.14232849, 0.01544392, 0.09329751,\n",
       "       0.01960169, 0.0244537 , 0.01080244, 0.86445367, 0.11119265,\n",
       "       0.35847433, 0.06030927, 0.15469818, 0.079797  , 0.04920872,\n",
       "       0.05745112, 0.01504379, 0.07877179, 0.53581954, 0.06853869,\n",
       "       0.33682509, 0.01463754, 0.0117619 , 0.14399055, 0.01479135,\n",
       "       0.10432424, 0.05957878, 0.12252199, 0.82699305, 0.0603595 ,\n",
       "       0.11267987, 0.07130379, 0.01347789, 0.08094675, 0.02010636,\n",
       "       0.03199713, 0.07654506, 0.05585157, 0.28966196, 0.01444298,\n",
       "       0.08138551, 0.01016078, 0.10598385, 0.03385791, 0.01161466,\n",
       "       0.1085965 , 0.0134545 , 0.25188282, 0.05643086, 0.07706238]),\n",
       "          lambda1=1, model=&#x27;qr&#x27;, penalization=&#x27;asgl&#x27;)</pre></div> </div></div></div></div></div></div></div></div></div>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 2
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:53.958487Z",
     "start_time": "2024-10-01T19:19:53.954328Z"
    }
   },
   "source": [
    "rscv.best_params_"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'lambda1': 1, 'alpha': 0.4}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 3
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:53.963055Z",
     "start_time": "2024-10-01T19:19:53.958487Z"
    }
   },
   "source": [
    "rscv.best_score_"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(-238.3365880217471)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 4
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 3: Customizing weights\n",
    "\n",
    "The `asgl` package offers several built-in methods for estimating adaptive weights, controlled via the `weight_technique` parameter. For more details onto the inners of each of these alternatives, refer to the [associated research paper](https://link.springer.com/article/10.1007/s11634-020-00413-8) or to thethe next section for an overview. However, for users requiring extensive customization, the package allows for the direct specification of custom weights through the `individual_weights` and `group_weights` parameters. This allows the users to implement their own weight computation techniques and use them within the `asgl` framework.\n",
    "\n",
    "When using custom weights, ensure that the length of `individual_weights` matches the number of variables, and the length of `group_weights` matches the number of groups. Below is an example demonstrating how to fit a model with custom individual and group weights:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.040092Z",
     "start_time": "2024-10-01T19:19:53.963055Z"
    }
   },
   "source": [
    "# Generate custom weights\n",
    "custom_individual_weights = np.random.rand(X_train.shape[1])\n",
    "custom_group_weights = np.random.rand(len(np.unique(group_index)))\n",
    "\n",
    "# Create a Regressor object with custom weights\n",
    "model = Regressor(model='lm', penalization='asgl', individual_weights=custom_individual_weights, group_weights=custom_group_weights)\n",
    "\n",
    "# Fit the model\n",
    "model.fit(X_train, y_train, group_index=group_index)"
   ],
   "outputs": [],
   "execution_count": 5
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mathematical formulations (with code!)\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For an in-depth analysis of the mathematical formulations we highly encourage to read our original [paper](https://link.springer.com/article/10.1007/s11634-020-00413-8), however, here the basics of the formulations will be covered, including code examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model formulations\n",
    "___\n",
    "Given a response vector $y$ and a predictors matrix $X$, the package implements two possible risk functions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Linear models\n",
    "In the usual linear models the risk function is defined as ,\n",
    "\n",
    "$$ R(\\beta)=\\|y-X\\beta\\|_2^2$$\n",
    "\n",
    "This model can be fit by simply defining\n",
    " * `model=lm`: lm referring to linear model\n",
    " * `penalization=None`: This fits an unpenalized model\n",
    " * `fit_intercept=True`: Default value is `True`. If the intercept of the model is not required, it can be set to `False`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.082187Z",
     "start_time": "2024-10-01T19:19:54.040596Z"
    }
   },
   "source": [
    "lm_model = Regressor(model='lm', penalization=None)\n",
    "lm_model.fit(X=X, y=y)\n",
    "\n",
    "coef = lm_model.coef_\n",
    "intercept = lm_model.intercept_\n",
    "print(np.round(coef, 1))"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.3 -0.1  0.2 61.7 -0.  56.5 74.3 91.3  0.2  0.   0.1  0.2  0.1 -0.1\n",
      "  0.1 10.9 69.   0.1 22.7  0.   7.2 94.7 91.5 -0.4 75.4 -0.3 38.1  5.3\n",
      " -0.   0.2 -0.2  0.  70.4  0.5 59.2  7.   0.  -0.1 12.2 69.4  1.3 86.8\n",
      " 21.1 -0.  96.2 -0.2 78.3 -0.   0.  -0.1]\n"
     ]
    }
   ],
   "execution_count": 6
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output provided in `coef` are the $\\beta$ coefficients of the model. If `intercept` is set to `True` (or if it is left with the default value) then the first element of this array is the intercept."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quantile regression models\n",
    "Quantile regression models are not as well known as linear regression models, but are a very powerful alternative. These models provide an estimation of the conditional quantiles of a response variable, and are specially suited for heteroscedastik datasets. The risk function in these models is defined as,\n",
    "\n",
    "$$R(\\beta)=\\frac{1}{n} \\sum_{i=1}^{n} \\rho_{\\tau}(y_{i}-x_{i}^{t} \\beta)$$\n",
    "\n",
    "where $\\rho_{\\tau}(u)=u(\\tau-I(u<0))$\n",
    "\n",
    "This model can be fit as,\n",
    "* `model=qr`: qr refers to quantile regression\n",
    "* `penalization=None`: This fits an unpenalized model\n",
    "* `tau=0.5`: $\\tau$ must be a value between 0 and 1, it controls the quantile to be fit. A value of 0.5 fits a model for the median of y.\n",
    "* `intercept=True`: It has the same definition as in the lasso penalization."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.231652Z",
     "start_time": "2024-10-01T19:19:54.083192Z"
    }
   },
   "source": [
    "qr_model = Regressor(model='qr', penalization=None, quantile=0.5)\n",
    "qr_model.fit(X=X, y=y)\n",
    "\n",
    "coef = qr_model.coef_\n",
    "print(np.round(coef, 1))"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.3  0.1  0.2 61.7  0.  56.5 74.3 91.2  0.1  0.1 -0.1 -0.1  0.1  0.\n",
      " -0.1 10.8 69.   0.2 22.9  0.2  7.1 94.6 91.6 -0.4 75.2 -0.6 38.3  5.2\n",
      "  0.1  0.3 -0.2 -0.  70.5  0.7 59.2  7.1 -0.1 -0.  12.3 69.6  1.2 86.9\n",
      " 21.   0.1 96.2  0.1 78.3  0.1 -0.1 -0.1]\n"
     ]
    }
   ],
   "execution_count": 7
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the remaining of the document, we will stick to using linear models, but at any time you can switch to quantile regression models by simply stating `model='qr'`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Penalization formulations\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lasso penalization\n",
    "\n",
    "Initially defined in 1996 by Tibshirani [(original paper)](http://statweb.stanford.edu/~tibs/lasso/lasso.pdf), lasso penalization boosts individual sparsity. It is defined as an L1 norm on the coefficients where parameter $\\lambda$ controls the level of sparsity to be applied.\n",
    "\n",
    "$$ \\min R(\\beta) + \\lambda\\sum_{i=1}^p|\\beta_i|$$\n",
    "\n",
    "It can be fit as,\n",
    "\n",
    "* `penalization='lasso'`\n",
    "* `lambda1=0.1`. This parameter is the $\\lambda$ defined in the problem formulation. It controls the sparsity of the solution. Large $\\lambda$ values are associated with more sparse solutions, since the coefficients are more heavily penalized."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.286056Z",
     "start_time": "2024-10-01T19:19:54.231652Z"
    }
   },
   "source": [
    "lasso_model = Regressor(model='lm', penalization='lasso',lambda1=0.1)\n",
    "lasso_model.fit(X=X, y=y)\n",
    "coef = lasso_model.coef_\n",
    "print(np.round(coef, 1))"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.2 -0.   0.1 61.7  0.  56.4 74.3 91.3  0.1  0.   0.   0.2  0.  -0.\n",
      "  0.  10.8 69.   0.  22.7  0.   7.1 94.6 91.5 -0.3 75.3 -0.2 38.   5.2\n",
      " -0.   0.2 -0.2  0.  70.4  0.5 59.1  6.9  0.  -0.  12.1 69.4  1.2 86.7\n",
      " 21.1 -0.  96.2 -0.2 78.3  0.   0.  -0. ]\n"
     ]
    }
   ],
   "execution_count": 8
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Group lasso penalization\n",
    "\n",
    "Proposed in 2006 by Yuan and Li [(original paper)](http://www.columbia.edu/~my2550/papers/glasso.final.pdf), group lasso penalization works assuming that predictors from matrix $X$ have a natural grouped structure. The penalization is defined as,\n",
    "\n",
    "$$ \\min R(\\beta) +\\lambda \\sum_{l=1}^{K} \\sqrt{p_{l}}\\left\\|\\beta^{(l)}\\right\\|_{2}$$\n",
    "\n",
    "where $p_{l}$ is the size of the l-th group. This penalization can be fit by simply defining:\n",
    "\n",
    "* `penalization='gl'` where gl refers to group lasso\n",
    "* `lambda1=0.1`, where $\\lambda$ is the parameter defined in the lasso penalization\n",
    "* `group_index=np.random.randint(1, 5, size=50)`. This should be an array of the same length as the number of variables in matrix $X$. Each element on this array indicates the group at which the associated variable belongs. "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.371353Z",
     "start_time": "2024-10-01T19:19:54.286056Z"
    }
   },
   "source": [
    "group_index = np.random.randint(1, 5, size=50)\n",
    "group_lasso_model = Regressor(model='lm', penalization='gl',lambda1=0.1)\n",
    "group_lasso_model.fit(X=X, y=y, group_index=group_index)\n",
    "coef = group_lasso_model.coef_\n",
    "print(np.round(coef, 1))"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.1 10.9 22.7 94.5 -0.4 75.3  0.  12.1  0.  -0.1  0.2 56.4  0.2  0.1\n",
      " -0.1 68.9 -0.1 -0.2  0.3 -0.1 91.2  0.1 -0.3  5.3 -0.  59.1 69.3  1.3\n",
      " 86.7 -0.  -0.  61.7 -0.  74.2 -0.   0.1  0.1  0.   7.2 91.4 38.   0.2\n",
      " -0.2  0.  70.4  0.5  7.  21.1 96.1 78.3]\n"
     ]
    }
   ],
   "execution_count": 9
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sparse group lasso\n",
    "Defined in 2013 [(original paper)](https://arxiv.org/abs/1001.0736), sparse group lasso is a linear combination of lasso and group lasso penalizations that provide solutions that are both between and within group sparse. The penalization is defined as,\n",
    "\n",
    "$$ \\min R(\\beta) + \\alpha\\lambda\\sum_{i=1}^p|\\beta_i| +(1-\\alpha)\\lambda \\sum_{l=1}^{K} \\sqrt{p_{l}}\\left\\|\\beta^{(l)}\\right\\|_{2}$$\n",
    "\n",
    "where $\\alpha$ is a parameter defined in $[0,1]$ that balances the penalization applied between lasso and group lasso. Values of $\\alpha$ close to 1 produce lasso-like solutions, while values close to 0 produce group lasso-like solutions.. This penalization can be fit defining:\n",
    "\n",
    "* `penalization='sgl'` where sgl refers to sparse group lasso\n",
    "* `lambda1=0.1`, where $\\lambda$ is the parameter defined in the lasso penalization\n",
    "* `alpha=0.5`, where $\\alpha$ is the parameter described above\n",
    "* `group_index=np.random.randint(1, 5, size=50)`, as described in the group lasso."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.453278Z",
     "start_time": "2024-10-01T19:19:54.371353Z"
    }
   },
   "source": [
    "sgl_model = Regressor(model='lm', penalization='sgl',lambda1=0.1, alpha=0.5)\n",
    "sgl_model.fit(X=X, y=y, group_index=group_index)\n",
    "coef = sgl_model.coef_\n",
    "print(np.round(coef, 1))"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.  10.8 22.7 94.6 -0.3 75.3  0.  12.1  0.  -0.   0.2 56.4  0.2  0.\n",
      " -0.  68.9 -0.1 -0.2  0.2 -0.  91.3  0.1 -0.2  5.3 -0.  59.1 69.4  1.2\n",
      " 86.7  0.   0.  61.7  0.  74.3  0.   0.   0.1  0.   7.1 91.5 38.   0.2\n",
      " -0.2  0.  70.4  0.5  7.  21.1 96.1 78.3]\n"
     ]
    }
   ],
   "execution_count": 10
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adaptive lasso\n",
    "\n",
    "The adaptive idea was initially proposed by Zou in 2006 [(original paper)](http://users.stat.umn.edu/~zouxx019/Papers/adalasso.pdf) for an adaptive lasso. This idea is based on the usage of additional weights on the penalization as a way to reduce bias and increase the quality of variable selection and prediction accuracy. This way, adaptive lasso is defined as,\n",
    "\n",
    "$$ \\min R(\\beta) + \\lambda\\sum_{i=1}^p\\tilde{w_i}|\\beta_i|$$\n",
    "\n",
    "where $\\tilde{w_i}$ are weights previously provided by the researcher. This penalization can be fit by defining,\n",
    "\n",
    "* `penalization='alasso'` where the 'a' before 'lasso' stands for adaptive.\n",
    "* `lambda1=0.1`, where $\\lambda$ is the parameter defined in the lasso penalization\n",
    "* `individual_weights=np.repeat(0.5, 50)`. Here, `individual_weights` refers to $\\tilde{w}$, and it should be of the same length as the number of predictors in X (the number of columns in X, in this case 50)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.516412Z",
     "start_time": "2024-10-01T19:19:54.453278Z"
    }
   },
   "source": [
    "individual_weights = np.repeat(0.5, 50)\n",
    "alasso_model = Regressor(model='lm', penalization='alasso',lambda1=0.1, individual_weights=individual_weights)\n",
    "alasso_model.fit(X=X, y=y)\n",
    "coef = alasso_model.coef_"
   ],
   "outputs": [],
   "execution_count": 11
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adaptive group lasso\n",
    "\n",
    "In a similar way, adaptive group lasso is defined as,\n",
    "\n",
    "$$ \\min R(\\beta) +\\lambda \\sum_{l=1}^{K} \\sqrt{p_{l}}\\tilde{v_l}\\left\\|\\beta^{(l)}\\right\\|_{2}$$\n",
    "\n",
    "where $\\tilde{v_l}$ are also additional weights. This penalization can be fit by defining,\n",
    "\n",
    "* `penalization='agl'` where the 'a' before 'gl' stand for adaptive.\n",
    "* `lambda1=0.1`, where $\\lambda$ is the parameter defined in the lasso penalization\n",
    "* `group_index=np.random.randint(1, 5, size=50)`, as described in the group lasso.\n",
    "* `group_weights=np.repeat(1.5, len(np.unique(group_index)))`. Here `group_weights` refers to $\\tilde{v}$ and it should be of the same length as the number of groups considered in `group_index`.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.622370Z",
     "start_time": "2024-10-01T19:19:54.516412Z"
    }
   },
   "source": [
    "group_weights=np.repeat(1.5, len(np.unique(group_index)))\n",
    "agl_model = Regressor(model='lm', penalization='agl',lambda1=0.1, group_weights=group_weights)\n",
    "agl_model.fit(X=X, y=y, group_index=group_index)\n",
    "coef = agl_model.coef_"
   ],
   "outputs": [],
   "execution_count": 12
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adaptive sparse group lasso\n",
    "\n",
    "Finally, just like it happened when we saw the sgl definition, adaptive saprse group lasso is defined as a linear combination of adaptive lasso and adaptive group lasso,\n",
    "\n",
    "$$R({\\beta})+\\alpha\\lambda\\sum_{j=1}^p\\tilde{w}_j\\lvert\\beta_j\\rvert+(1-\\alpha)\\lambda\\sum_{l=1}^K\\sqrt{p_l}\\tilde{v_l}\\left\\|\\beta^{(l)}\\right\\|_{2}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Remarks\n",
    "\n",
    "Observe that by setting $\\tilde{w_i}=1$ and $\\tilde{v_l}=1$ we recover the sparse group lasso formulation. And additionally,by setting $\\alpha=1$ we recover the lasso formulation and by setting $\\alpha=0$ we recover the group lasso formulation. This means that all previous penalizations are particular cases of this adaptive sparse group lasso penalization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will see how this penalization can be fit assuming that we know some potential values for $\\tilde{w_i}$ and $\\tilde{v_l}$. After that, we will discuss alternatives for the estimation of such weights.\n",
    "\n",
    "* `penalization='asgl'` where asgl refers to adaptive sparse group lasso\n",
    "* `lambda1 = 0.1`, where $\\lambda$ is the parameter defined in the lasso penalization\n",
    "* `alpha=0.5`, where $\\alpha$ is the parameter described for the sparse group lasso penalization\n",
    "* `group_index=np.random.randint(1, 5, size=50)`, as described in the group lasso.\n",
    "* `individual_weights=np.repeat(0.5, 50)` where `individual_weights` is the parameter defined for adaptive lasso.\n",
    "* `group_weights=np.repeat(1.5, len(np.unique(group_index)))`. where `group_weights` is the parameter defined for adaptive group lasso."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.719284Z",
     "start_time": "2024-10-01T19:19:54.622370Z"
    }
   },
   "source": [
    "asgl_model = Regressor(model='lm', penalization='asgl',lambda1=0.1, alpha=0.5, individual_weights=individual_weights, group_weights=group_weights)\n",
    "asgl_model.fit(X=X, y=y, group_index=group_index)\n",
    "coef = asgl_model.coef_"
   ],
   "outputs": [],
   "execution_count": 13
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Weights calculation alternatives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have seen how to fit an adaptive sparse group lasso model, but for that we used some random weights where $w=\\vec{0.5}$ and $v=\\vec{0.5}$. Now the question is, **is there some way to obtain an estimation of these weights that actually produce better results than a simple sparse group lasso?** And the answer is, of course! (spoiler: otherwise we would not have created this package). We propose here using principal component analysis (PCA) and partial least squares (PLS) for obtaining these weights. The usage of these techniques in the package, as we will see, is pretty straightforward and does not require any knowledge on how they work internally. You just need to choose the option that best suits you (or compare the results from different options and select the best one).\n",
    "\n",
    "But before we start with that, let us introduce here how these weights are usually defined:\n",
    "\n",
    "$$w_i=\\frac{1}{{\\beta_i}^{\\gamma_1}}, \\quad v_l=\\frac{1}{{\\|\\beta^l\\|_2}^{\\gamma_2}}$$\n",
    "\n",
    "The objective, then is to obtain a value for $\\beta$, and we should keep in mind that we will have two extra parameters, $\\gamma_1$ and $\\gamma_2$, the powers at which the coefficients are risen, that we can modify.\n",
    "\n",
    "A small $\\beta$ coefficient results into a large weight, which is heavily penalized and more likely left outside the final model. On the opposite hand, large $\\beta$ coefficients result into small weights that are likely to remain in the final model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### PCA based on a subset of components\n",
    "\n",
    "This is our proposal for the default weight calculation alternative. Simply put, use PCA in order to reduce the number of dimensions of the problem. Fit a non penalized model using the PCA scores (taking advantage of being in a smaller dimension framework) and then project back into the original space. An in-depth explanation of this process can be read in our original [paper](https://link.springer.com/article/10.1007/s11634-020-00413-8). This can be easily done as part of the `Regressor` object:\n",
    "\n",
    "* `penalization=asgl`\n",
    "* `weight_technique='pca_pct'`. It refers to PCA percentage (because this technique is based on selecting a percentage of PCA components to fit the weights)\n",
    "* `individual_power_weight=1`. This is the $\\gamma_1$ coefficient value. Default value for this parameter is `1`\n",
    "* `group_power_weight=1` This is the $\\gamma_2$ coefficient value Default value for this parameter is `1`\n",
    "* `variability_pct=0.9`. The number of PCA components to use for fitting the weight in terms of the total variability they can explain. Default value for this parameter is `variability_pct=0.9`"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.841316Z",
     "start_time": "2024-10-01T19:19:54.719284Z"
    }
   },
   "source": [
    "asgl_model = Regressor(model='lm', penalization='asgl',lambda1=0.1, alpha=0.5, weight_technique='pca_pct', individual_power_weight=1, group_power_weight=1, variability_pct=0.9)\n",
    "asgl_model.fit(X=X, y=y, group_index=group_index)\n",
    "coef = asgl_model.coef_"
   ],
   "outputs": [],
   "execution_count": 14
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:54.845397Z",
     "start_time": "2024-10-01T19:19:54.841316Z"
    }
   },
   "source": [
    "print(f\"Let's see what the individual weights look like:\\n{np.round(asgl_model.individual_weights, 2)}\")"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Let's see what the individual weights look like:\n",
      "[0.48 0.08 0.57 0.02 0.29 0.03 0.02 0.01 0.16 0.07 0.2  0.15 0.1  0.08\n",
      " 0.07 0.04 0.02 0.09 0.04 0.07 0.11 0.01 0.01 0.13 0.02 0.25 0.04 0.19\n",
      " 0.04 0.14 0.08 0.39 0.02 0.08 0.02 0.04 0.11 4.3  0.05 0.02 0.1  0.01\n",
      " 0.05 0.06 0.01 0.15 0.01 0.09 1.15 0.56]\n"
     ]
    }
   ],
   "execution_count": 15
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### PLS based on a subset of components\n",
    "\n",
    "We propose a similar alternative but built based on partial least squares. Partial least squares is a dimensionality reduction technique that works by maximizing the covariance between the predictors $X$ and the response vector $y$ (as opposed to PCA that work by maximizing the variance of $X$). PLS is then better suitted for prediction purposes, but it is based on least squares regression, so it can be more affected by heteroscedasticity or outliers than the PCA proposal.\n",
    "\n",
    "* `penalization=asgl`\n",
    "* `weight_technique='pls_pct'`. It refers to PLS percentage (because this technique is based on selecting a percentage of PLS components to fit the weights)\n",
    "* `individual_power_weight=1`. This is the $\\gamma_1$ coefficient value. Default value for this parameter is `1`\n",
    "* `group_power_weight=1` This is the $\\gamma_2$ coefficient value Default value for this parameter is `1`\n",
    "* `variability_pct=0.9`. The number of PCA components to use for fitting the weight in terms of the total variability they can explain. Default value for this parameter is `variability_pct=0.9`"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:55.007059Z",
     "start_time": "2024-10-01T19:19:54.845397Z"
    }
   },
   "source": [
    "asgl_model = Regressor(model='lm', penalization='asgl',lambda1=0.1, alpha=0.5, weight_technique='pls_pct', individual_power_weight=1, group_power_weight=1, variability_pct=0.9)\n",
    "asgl_model.fit(X=X, y=y, group_index=group_index)\n",
    "coef = asgl_model.coef_"
   ],
   "outputs": [],
   "execution_count": 16
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### PCA / PLS based on the first component\n",
    "\n",
    "Each PCA is built as a linear combination of the original variables. This means that another alternative for estimating the weights can be defined as simply using the weights from the first principal component as weights for the adaptive sparse group lasso model. In the same way, it is poosible to use the first PLS compponent to obtain weights\n",
    "\n",
    "* `penalization=asgl`\n",
    "* `weight_technique='pca_1'`. It refers to using the first principal component.\n",
    "* `weight_technique='pls_1'`It refers to using the first PLS component.\n",
    "* `individual_power_weight=1`. As defined for PCA based on a subset of components.\n",
    "* `group_power_weight=1`. As defined for PCA based on a subset of components."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:55.103048Z",
     "start_time": "2024-10-01T19:19:55.007959Z"
    }
   },
   "source": [
    "asgl_model = Regressor(model='lm', penalization='asgl',lambda1=0.1, alpha=0.5, weight_technique='pca_1', individual_power_weight=1, group_power_weight=1)\n",
    "asgl_model.fit(X=X, y=y, group_index=group_index)\n",
    "coef = asgl_model.coef_"
   ],
   "outputs": [],
   "execution_count": 17
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### LASSO\n",
    "\n",
    "Another alternative consists on running an initial lasso model, and use the estimates of this model as the initial weight for a second model\n",
    "\n",
    "* `penalization=asgl`\n",
    "* `weight_technique='lasso'`. It refers to using lasso to obtain the weights\n",
    "* `lambda1_weights=1e-2`. It is the $\\lambda$ value used in the lasso estimation of the weights.\n",
    "* `individual_power_weight=1`. As defined for PCA based on a subset of components.\n",
    "* `group_power_weight=1`. As defined for PCA based on a subset of components."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:55.243346Z",
     "start_time": "2024-10-01T19:19:55.104057Z"
    }
   },
   "source": [
    "asgl_model = Regressor(model='lm', penalization='asgl',lambda1=0.1, alpha=0.5, weight_technique='lasso', lambda1_weights=1e-2, individual_power_weight=1, group_power_weight=1)\n",
    "asgl_model.fit(X=X, y=y, group_index=group_index)\n",
    "coef = asgl_model.coef_"
   ],
   "outputs": [],
   "execution_count": 18
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Unpenalized model\n",
    "\n",
    "This alternative can only be used when dealing with a low dimensional dataset (in which the number of observations is larger than the number of variables). In this case, it is possible to fit an initial model with no penalization, and then use these as weights for an adaptive model. We consider two alternatives. An unpenalized linear model and unpenalized quantile regression model. The usage of either one is determined by the parameter `model`.\n",
    "\n",
    "* `model='lm'`\n",
    "* `penalization=asgl`\n",
    "* `weight_technique='unpenalized'`. It refers to using an unpenalized model.\n",
    "* `individual_power_weight=1`. As defined for PCA based on a subset of components.\n",
    "* `group_power_weight=1`. As defined for PCA based on a subset of components.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-01T19:19:55.379017Z",
     "start_time": "2024-10-01T19:19:55.243346Z"
    }
   },
   "source": [
    "asgl_model = Regressor(model='lm', penalization='asgl',lambda1=0.1, alpha=0.5, weight_technique='unpenalized', individual_power_weight=1, group_power_weight=1)\n",
    "asgl_model.fit(X=X, y=y, group_index=group_index)\n",
    "coef = asgl_model.coef_"
   ],
   "outputs": [],
   "execution_count": 19
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What parameter values should you use?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* $\\lambda$: This parameter is used in all the penalizations defined in the package, and it controls the level of sparsity applied to a solution. A typical range of values for this parameter goes from $10^{-3}$ up to $10$. eg:\n",
    "`lambda1=10.0**np.arange(-3, 1.01, 0.2)`\n",
    "\n",
    "* $\\alpha$: This parameter is udes in sparse group lasso and adaptive sparse group lasso techniques. It controls the tradeoff between lasso and group lasso techniques. $\\alpha$ values close to $1$ produce lasso solutions, and close to $0$ produce group lasso solutions. Best solutions are achieved usually somewhere close to one of the limit values, so a typical range  for this parameter concentrates more values on the sides and less values on the center. eg: \n",
    "`alpha=np.r_[np.arange(0.0, 0.3, 0.02), np.arange(0.3, 0.7, 0.1), np.arange(0.7, 0.99, 0.02)]`\n",
    "\n",
    "* $\\gamma_1$ and $\\gamma_2$: These parameters are the powers applied to weights in adaptive penalizations. Usually, these are defined in the interval $[0, 2]$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}