#StackBounty: #regression #lasso #convergence #high-dimensional High-dimensional regression: why is $log p/n$ special?

Bounty: 100

I am trying to read up on the research in the area of high-dimensional regression; when $p >> n$. It seems like the term $log p/n$ appears often in terms of rate of convergence for regression estimators.

For example, here, equation (17) says that the lasso fit, $hat{beta}$ satisfies
$$ dfrac{1}{n}|Xhat{beta} – X beta|_2^2 = O_P left(sigma sqrt{dfrac{log p}{n} } |beta|_1right),.$$

Usually, this also implies that $log p$ should be smaller than $n$.

  1. Is there any intuition as to why this ratio of $log p/n$ is so prominent?
  2. Also, it seems from the literature the high-dimensional regression problem gets complicated when $log p geq n$. Why is it so?
  3. Is there a good reference that discusses the issues with how fast $p$ and $n$ should grow compared to each other?


Get this bounty!!!

#StackBounty: #regression #lasso #convergence #high-dimensional High-dimensional regression: why is $log p/n$ special?

Bounty: 100

I am trying to read up on the research in the area of high-dimensional regression; when $p >> n$. It seems like the term $log p/n$ appears often in terms of rate of convergence for regression estimators.

For example, here, equation (17) says that the lasso fit, $hat{beta}$ satisfies
$$ dfrac{1}{n}|Xhat{beta} – X beta|_2^2 = O_P left(sigma sqrt{dfrac{log p}{n} } |beta|_1right),.$$

Usually, this also implies that $log p$ should be smaller than $n$.

  1. Is there any intuition as to why this ratio of $log p/n$ is so prominent?
  2. Also, it seems from the literature the high-dimensional regression problem gets complicated when $log p geq n$. Why is it so?
  3. Is there a good reference that discusses the issues with how fast $p$ and $n$ should grow compared to each other?


Get this bounty!!!

#StackBounty: #scikit-learn #regression #supervised-learning #hyperparameter #hyperparameter-tuning Optimising Kernel parameters using …

Bounty: 50

I want to optimize the Kernel parameters or hyper-parameters using my training data in GaussianProcessRegressor of Scikit-learn.Following is my query:

My training datasets are:

X: 2-D Cartesian coordinate as input data

y: radio signal strength (RSS) at the 2-D coordinates points as observed output

What I’ve done so far:

I’ve installed python and Scikit-learn software. I’ve successfully tested the sample codes. I’m able to predict RSS at test points using training data. I use squared exponential Kernel.

What I want to do:

I want to train the Kernel parameters (hyper-parameter) with different optimizing algorithms like gradient descent, swarm intelligence, and trust-region-reflective algorithms.

What I learned and What help I am asking for:

I’ve learned that, in the GaussianProcessRegressor class of scikit, the optimizer is an argument where I can use my own optimizing algorithm. Since it is callable, I need to write my own function/method for it. Can I use any inbuilt library (library of optimization algorithm) in GaussianProcessRegressor class? Are there such libraries available for python? Could anybody provide any sample code for using kernel parameter optimization algorithm in GaussianProcessRegressor? I’ve learned that we use the training datasets for optimizing the hyper-parameters. Could anybody provide any insight about relating the training datasets with the optimization algorithm, please?

Thank you!


Get this bounty!!!

#StackBounty: #regression #regression-coefficients #categorical-encoding difference between dummy variable categories that weren't …

Bounty: 50

Assume we have a categorical variable (one-hot encoded) with three or more categories. {race1, race2, ..., race-n}
To avoid the dummy variable trap, assume we omitted race1. Knowing the coefficients of race2,3,...n would help us compare each to race1.

What I’m trying to understand/figure-out is how to compare in the same model without rerunning the regression, the difference between race2 and race3.


Get this bounty!!!

#StackBounty: #regression #regression-coefficients #categorical-encoding difference between dummy variable categories that weren't …

Bounty: 50

Assume we have a categorical variable (one-hot encoded) with three or more categories. {race1, race2, ..., race-n}
To avoid the dummy variable trap, assume we omitted race1. Knowing the coefficients of race2,3,...n would help us compare each to race1.

What I’m trying to understand/figure-out is how to compare in the same model without rerunning the regression, the difference between race2 and race3.


Get this bounty!!!

#StackBounty: #regression #regression-coefficients #categorical-encoding difference between dummy variable categories that weren't …

Bounty: 50

Assume we have a categorical variable (one-hot encoded) with three or more categories. {race1, race2, ..., race-n}
To avoid the dummy variable trap, assume we omitted race1. Knowing the coefficients of race2,3,...n would help us compare each to race1.

What I’m trying to understand/figure-out is how to compare in the same model without rerunning the regression, the difference between race2 and race3.


Get this bounty!!!

#StackBounty: #regression #regression-coefficients #categorical-encoding difference between dummy variable categories that weren't …

Bounty: 50

Assume we have a categorical variable (one-hot encoded) with three or more categories. {race1, race2, ..., race-n}
To avoid the dummy variable trap, assume we omitted race1. Knowing the coefficients of race2,3,...n would help us compare each to race1.

What I’m trying to understand/figure-out is how to compare in the same model without rerunning the regression, the difference between race2 and race3.


Get this bounty!!!

#StackBounty: #regression #regression-coefficients #categorical-encoding difference between dummy variable categories that weren't …

Bounty: 50

Assume we have a categorical variable (one-hot encoded) with three or more categories. {race1, race2, ..., race-n}
To avoid the dummy variable trap, assume we omitted race1. Knowing the coefficients of race2,3,...n would help us compare each to race1.

What I’m trying to understand/figure-out is how to compare in the same model without rerunning the regression, the difference between race2 and race3.


Get this bounty!!!

#StackBounty: #regression #regression-coefficients #categorical-encoding difference between dummy variable categories that weren't …

Bounty: 50

Assume we have a categorical variable (one-hot encoded) with three or more categories. {race1, race2, ..., race-n}
To avoid the dummy variable trap, assume we omitted race1. Knowing the coefficients of race2,3,...n would help us compare each to race1.

What I’m trying to understand/figure-out is how to compare in the same model without rerunning the regression, the difference between race2 and race3.


Get this bounty!!!

#StackBounty: #regression #machine-learning #classification #references #cart Regression trees with multiple input and output levels

Bounty: 100

I am looking for different modelling approaches which are able to build regression trees (i.e. with continuous input and output variables) with multiple input and output levels.

The most common approach (e.g. with CART) is to recursively do binary splits. I am looking for methods that could split the target variable into multiple branches (according to some sensible metric) within every step.

Any hints, i.e. papers, links etc. are very welcome.

(The reason for me asking this question is that I want to find a good method to extend my OneR package for solving regression problems.)

Edit
To clarify: I am looking for a method which could split the respective input variable into $n$ intervals where each interval leads to an interval in the target variable. I guess you would need some constraints (e.g. the max number of intervals) to get useful results.


Get this bounty!!!