Polynomial Regression

The problem of fitting a polynomial curve to the data is called polynomial regression, or better “linear regression with polynomial features”, since the problem is still the same of linear regression, but we have a polynomial function.

Note that the polynomial function is polynomial with respect to the data, but is still linear in the parameters.

$$y_i=\sum _{j=1}^k{a}_j{x}_i^j+b\forall i=1,\dots ,n$$

In matrix notation we can express the function as:

$$\underset{y}{\underbrace{\left(\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right)}}=\underset{\mathbb{X}}{\underbrace{\left(\begin{array}{ccccc}{x}_1^k& x_1^{k-1}& \dots & x_1& 1\\ x_2^k& x_2^{k-1}& \dots & x_2& 1\\ ⋮& ⋮& ⋮& ⋮& ⋮\\ x_n^k& x_n^{k-1}& \dots & x_n& 1\end{array}\right)}}\underset{\theta }{\underbrace{\left(\begin{array}{c}{a}_k\\ a_{k-1}\\ ⋮\\ a_1\\ b\end{array}\right)}}$$

The solution is the same as the linear regression solution, with the requirements that $k<n$.

In the matrix $\mathbb{X}$ we have a row for each data point, and a column for each parameter.

Is polynomial regression all we need?

Since we can approximate every continuous function with a polynomial, as stated by The Stone-Weierstrass theorem, is polynomial regression all we need in order to fit the data?

The answer to this question is NO, since polynomial regression is kind of limited, and we cannot perform some useful thing with it, for example:

• If we want different loss functions than MSE;
• If we want to apply regularization;
• If we have additional priors;
• If we want to compute intermediate features;
• If we want more flexibility;
• If we want to perform classification instead of regression.