线性回归之最小二乘法推导

线性回归之最小二乘法推导

一元线性回归函数:
y = ω x + b y = omega x+b y=ωx+b
假设有n组[x1,x2,x3…xn],[y1,y2,y3…yn]数据，它们之间的关系为 f ( x ) = ω x + b f(x) =ωx+b f(x)=ωx+b 线性回归的目的之让 f ( x ) f(x) f(x) 与 y y y 之间的差值最小当 ω 和 b omega和b ω和b 取何值的时候二者之间的差值最小.设 E ( ω , b ) Ε_{(omega,b)} E(ω,b)​ 为 f ( x ) f(x) f(x) 与 y y y 的差异值 E ( ω , b ) Ε(omega,b) E(ω,b)为损失函数:
E ( ω , b ) = ∑ i = 1 n ( f ( x ) − y i ) 2 = ∑ i = 1 n ( ω x i + b − y i ) 2 Ε_{(omega,b)} = sum_{i=1}^n(f(x)-y_{i})^2 = sum_{i=1}^n(omega x_{i}+b-y_{i})^2 E(ω,b)​=i=1∑n​(f(x)−yi​)2=i=1∑n​(ωxi​+b−yi​)2
最小二乘法来确定 ω 和 b ω和b ω和b 的值因为 E ( ω , b ) Ε_{(omega,b)} E(ω,b)​ 为凸函数求其偏导：
∂ E ( ω , b ) ∂ b = ∑ i = 1 n ( ω x i + b − y i ) 2 ∂ b = 2 ∑ i = 1 n ( ω x i + b − y i ) frac{∂Ε_{(omega,b)}}{∂b} = frac{sum_{i=1}^n(omega x_{i}+b-y_{i})^2}{∂b} = 2sum_{i=1}^n(omega x_{i}+b-y_{i}) ∂b∂E(ω,b)​​=∂b∑i=1n​(ωxi​+b−yi​)2​=2i=1∑n​(ωxi​+b−yi​) = 2 [ ∑ i = 1 n ω x i + ∑ i = 1 n b + − ∑ i = 1 n y i ] =2[sum_{i=1}^nomega x_{i} + sum_{i=1}^nb +-sum_{i=1}^ny_{i}] =2[i=1∑n​ωxi​+i=1∑n​b+−i=1∑n​yi​]
= 2 [ ω ∑ i = 1 n x i + n b − ∑ i = 1 n y i ] =2[omegasum_{i=1}^nx_i+nb-sum_{i=1}^ny_i] =2[ωi=1∑n​xi​+nb−i=1∑n​yi​]令 x ˉ = ∑ i = 1 n x i n , y ˉ = ∑ i = 1 n y i n bar{x}=frac{sum_{i=1}^nx_i}{n},bar{y}=frac{sum_{i=1}^ny_i}{n} xˉ=n∑i=1n​xi​​,yˉ​=n∑i=1n​yi​​
∴ ∂ E ( ω , b ) ∂ b = 2 [ ω n x ˉ + n b − n y ˉ ] ∴frac{∂Ε_{(omega,b)}}{∂b} = 2[omega n bar{x}+nb-nbar{y}] ∴∂b∂E(ω,b)​​=2[ωnxˉ+nb−nyˉ​]
= 2 n [ ω x ˉ + b − y ˉ ] =2n[omega bar{x}+b-bar{y}] =2n[ωxˉ+b−yˉ​]
令 b b b的导数为0
2 n [ ω x ˉ + b − y ˉ ] = 0 2n[omega bar{x}+b-bar{y}] = 0 2n[ωxˉ+b−yˉ​]=0
∵ n > = 1 ∵n>=1 ∵n>=1
∴ ω x ˉ + b − y ˉ = 0 ∴omega bar{x} + b - bar{y} = 0 ∴ωxˉ+b−yˉ​=0
∴ b = y ˉ − ω x ˉ ∴b = bar{y} - omega bar{x} ∴b=yˉ​−ωxˉ对 ω omega ω求导
∂ E ( ω , b ) ∂ ω = ∑ i = 1 n ( ω x i + b − y i ) 2 ∂ ω = 2 ∑ i = 1 n ( ω x i + b − y i ) x i frac{∂Ε_{(omega,b)}}{∂ omega } = frac{sum_{i=1}^n(omega x_{i}+b-y_{i})^2}{∂ omega } = 2sum_{i=1}^n(omega x_{i}+b-y_{i})x_i ∂ω∂E(ω,b)​​=∂ω∑i=1n​(ωxi​+b−yi​)2​=2i=1∑n​(ωxi​+b−yi​)xi​
= 2 [ ∑ i = 1 n ω x i 2 + ∑ i = 1 n b x i − ∑ i = 1 n y i x i ] =2[sum_{i=1}^nomega x_i^2 + sum_{i=1}^nb x_i - sum_{i=1}^n y_ix_i] =2[i=1∑n​ωxi2​+i=1∑n​bxi​−i=1∑n​yi​xi​]
= 2 [ ∑ i = 1 n ω x i 2 + ∑ i = 1 n ( y ˉ − ω x ˉ ) x i − ∑ i = 1 n y i x i ] =2[sum_{i=1}^nomega x_i^2 + sum_{i=1}^n (bar{y} - omega bar{x}) x_i - sum_{i=1}^n y_ix_i] =2[i=1∑n​ωxi2​+i=1∑n​(yˉ​−ωxˉ)xi​−i=1∑n​yi​xi​]
= 2 [ ∑ i = 1 n ω x i 2 + n x ˉ ( y ˉ − ω x ˉ ) − ∑ i = 1 n y i x i ] =2[sum_{i=1}^nomega x_i^2 + nbar{x}(bar{y}- omega bar{x})- sum_{i=1}^n y_ix_i] =2[i=1∑n​ωxi2​+nxˉ(yˉ​−ωxˉ)−i=1∑n​yi​xi​]
= 2 [ ∑ i = 1 n ω x i 2 + n x ˉ y ˉ − n ω x ˉ 2 − ∑ i = 1 n y i x i ] =2[sum_{i=1}^nomega x_i^2 +n bar{x} bar{y} -n omega bar{x}^2- sum_{i=1}^n y_ix_i] =2[i=1∑n​ωxi2​+nxˉyˉ​−nωxˉ2−i=1∑n​yi​xi​]
= 2 [ ω ( ∑ i = 1 n x i 2 − n x ˉ 2 ) + n x ˉ y ˉ − ∑ i = 1 n y i x i ] =2[omega (sum_{i=1}^n x_i^2 - n bar{x}^2) +n bar{x} bar{y} - sum_{i=1}^n y_ix_i] =2[ω(i=1∑n​xi2​−nxˉ2)+nxˉyˉ​−i=1∑n​yi​xi​]
令 ω omega ω的导数为0
ω ( ∑ i = 1 n x i 2 − n x ˉ 2 ) + n x ˉ y ˉ − ∑ i = 1 n y i x i = 0 omega (sum_{i=1}^n x_i^2 - n bar{x}^2) +n bar{x} bar{y} - sum_{i=1}^n y_ix_i = 0 ω(i=1∑n​xi2​−nxˉ2)+nxˉyˉ​−i=1∑n​yi​xi​=0
ω = ∑ i = 1 n y i x i − n x ˉ y ˉ ∑ i = 1 n x i 2 − n x ˉ 2 omega = frac{sum_{i=1}^n y_ix_i - n bar{x} bar{y}}{sum_{i=1}^n x_i^2 - n bar{x}^2} ω=∑i=1n​xi2​−nxˉ2∑i=1n​yi​xi​−nxˉyˉ​​
可化简
∵ x ˉ = ∑ i = 1 n x i n ∵bar{x} = frac{sum _{i=1}^n x_i}{n} ∵xˉ=n∑i=1n​xi​​
∴ ∑ i = 1 n x i = n x ˉ ∴sum _{i=1}^nx_i = nbar{x} ∴i=1∑n​xi​=nxˉ
∴ ∑ i = 1 n x i 2 − n x ˉ 2 = ∑ i = 1 n x 2 − 2 n x ˉ 2 + n x ˉ 2 ∴sum_{i=1}^n x_i^2 - n bar{x}^2 = sum_{i=1}^n x^2 - 2nbar{x}^2+nbar{x}^2 ∴i=1∑n​xi2​−nxˉ2=i=1∑n​x2−2nxˉ2+nxˉ2
= ∑ i = 1 n x i 2 − 2 ∑ i = 1 n x i x ˉ + ∑ i = 1 n x 2 = sum_{i=1}^n x_i^2 - 2sum_{i=1}^nx_ibar{x}+sum_{i=1}^nx^2 =i=1∑n​xi2​−2i=1∑n​xi​xˉ+i=1∑n​x2
= ∑ i = 1 n ( x i − x ˉ ) 2 =sum_{i=1}^n(x_i-bar{x})^2 =i=1∑n​(xi​−xˉ)2
同理可得
∑ i = 1 n y i x i − n x ˉ y ˉ = ∑ i = 1 n y i x i − 2 n x ˉ y ˉ + n x ˉ y ˉ sum_{i=1}^n y_ix_i - n bar{x} bar{y} = sum_{i=1}^n y_ix_i - 2nbar{x}bar{y} + nbar{x}bar{y} i=1∑n​yi​xi​−nxˉyˉ​=i=1∑n​yi​xi​−2nxˉyˉ​+nxˉyˉ​
= ∑ i = 1 n y i x i − ∑ i = 1 n x i y ˉ − ∑ i = 1 n y i x ˉ + ∑ i = 1 n x ˉ y ˉ =sum_{i=1}^ny_ix_i - sum_{i=1}^nx_ibar{y}-sum_{i=1}^ny_ibar{x}+sum_{i=1}^nbar{x}bar{y} =i=1∑n​yi​xi​−i=1∑n​xi​yˉ​−i=1∑n​yi​xˉ+i=1∑n​xˉyˉ​
= ∑ i = 1 n ( x i − x ˉ ) ( y i − y ˉ ) =sum_{i=1}^n(x_i-bar{x})(y_i-bar{y}) =i=1∑n​(xi​−xˉ)(yi​−yˉ​)
∴ ω = ∑ i = 1 n ( x i − x ˉ ) ( y i − y ˉ ) ∑ i = 1 n ( x i − x ˉ ) 2 ∴omega = frac{sum_{i=1}^n(x_i-bar{x})(y_i-bar{y})}{sum_{i=1}^n(x_i-bar{x})^2} ∴ω=∑i=1n​(xi​−xˉ)2∑i=1n​(xi​−xˉ)(yi​−yˉ​)​
∴ b = y ˉ − ω x ˉ , ω = ∑ i = 1 n ( x i − x ˉ ) ( y i − y ˉ ) ∑ i = 1 n ( x i − x ˉ ) 2 ∴b =bar{y} - omega bar{x}, omega =frac{sum_{i=1}^n(x_i-bar{x})(y_i-bar{y})}{sum_{i=1}^n(x_i-bar{x})^2} ∴b=yˉ​−ωxˉ,ω=∑i=1n​(xi​−xˉ)2∑i=1n​(xi​−xˉ)(yi​−yˉ​)​
x ˉ , y ˉ 还 原 可 得 bar{x} , bar{y}还原可得 xˉ,yˉ​还原可得
b = ∑ i = 1 n ( y i − ω x i ) n , ω = ∑ i = 1 n y i x i − ∑ i = 1 n x i y i n ∑ i = 1 n x i 2 + ( ∑ i = 1 n x i ) 2 n b= frac{sum_{i=1}^n(y_i - omega x_i)}{n} , omega =frac{ sum_{i=1}^ny_ix_i-frac{sum_{i=1}^nx_iy_i}{n}}{sum_{i=1}^nx_i^2+ frac{(sum_{i=1}^nx_i)^2}{n} } b=n∑i=1n​(yi​−ωxi​)​,ω=∑i=1n​xi2​+n(∑i=1n​xi​)2​∑i=1n​yi​xi​−n∑i=1n​xi​yi​​​