Chapter 3. Multiple Liear Regressio Data structure ad the model yi 0 1xi1 pxip i, i1,, (Y X ),,, : idepedet with E( ) 0 ad 1 : ukow 0, 1,, p, 0 1 i var( i ) X (1, x,, xp), rak( X) p1, X : give where xj ( x1 j,, xj) 1
Least squares estimates miimize ( yi 0 1xi1 pxip) w.r.t. 0,, p i1 ormal equatio : e ˆ ˆ ˆ i yi ( 0 1xi1 pxip) yi yˆ i (p.57) e 0 e 0 ˆ y ˆ x ˆ x xi1ei 0 ( xi1 x1) ei 0 S ˆ ˆ 111 S1 p p S y1 x 0 ( ) 0 ˆ ˆ ipe i xip xp e i S p11 S pp p S yp i i 0 1 1 p p Sij ( xai xi )( xaj xj ) a1 where. Syj ( ya y)( xaj xj ) a1 least squares regressio fit: y ˆ ˆ ˆ 0 1x1 x ˆ p p estimate (ubiased) of : ˆ 1 1 y y SSE p1 p1 ( i ˆi) i1
Matrix approach For y ( y1,, y) x ( x,, x ) ( j 1,, p), j 1 j j X (1, x,, x p ),, 1 ( 0, 1,, p ), ad ( 1,, ), Model: y X Assumptios : 1,, are idepedet with E( ) 0 ad i var( i ) Least square estimate: ˆ argmi( y X)( y X) ( y X )( y X) yy yx Xy XX ( Xy ) Xy ( y X)( y X) XX Xy Xy 0 Recall that ( cx ) ( xc ) c x x ( yay ) ( A A) y y 3
( y X)( y X) XX Xy Xy 0 ˆ ( XX ) 1 Xy ( ) rak( X X ) rak( XX ) rak( X ) rak( X ) p 1 ˆ 1 1 yˆ X X( X X) X y Py ( P X( X X) X ) I case of p 1, ( x j)( yj) ( xj)( xjyj) 1 ˆ0 j y j ˆ x xj ( xj) ˆ x 1 j xj x j y j xjyj ( xj)( yj) xj ( xj) This is coicidet with the result of simple liear regressio. 4
Method of iferece (1) Properties of estimates Recall that E y X y I yˆ X ˆ X XX Xy P P X XX X 1 1 ( ), var( ) var( ), ( ) y ( ( ) ) E( ˆ ), var( ˆ ) ( X X) ˆ 1 1 E( ) E ( XX ) Xy ( XX ) XE( y) ˆ 1 1 1 var( ) ( XX ) Xvar( y) X( XX ) ( XX ) 1 i. ii. E( ˆ ), E( e) 0, var( e) I P e y yˆ I P y E() e I PE( y) I PX X PX X X 0 X var( e) I Pvar( y) I P I PI I P( I P) ( I P) IP 5
() Iferece uder additioal ormality assumptio Let i. 1 C ( XX) ( c ij ) 0 i, j p ˆ i i se..( ˆ ) i ˆ 1/ ~ t ( p1); se..( i) cii ( i1,, p) Pr ˆ t ( p1; ) se..( ˆ ) 1 i i i ˆ Reject H v.s. 0 0 : i i H 0 1 : i i ˆ 0 i i iff t ( p 1, ) se..( ˆ ) i p-value: ii. ˆ 0 0 1/ ~ t ( p1); se..( 0) c00 ˆ 0 se..( ) ˆ ˆ Similar to ˆi 6
E( Y x ) x where x ( x, x,, x ) with x 1 ˆ 0 0 1 1/ ~ t ( p1), se..( ˆ 0) ˆ x0( XX ) x0 se..( ˆ ) iii. 0 0 0 0 00 01 0 p 00 0 ˆ x ˆ var( ˆ ) x var( ˆ ) x 0 0 0 0 0 C.I. ad Test iv. Predictio for y0 x0 0 ( 0 1,, ) yˆ ˆ 0 x0 ( ˆ 0) y yˆ se..( y ) 0 0 1 1/ ~ t ( p1); se..( y0 y0) (1 x0( XX) x0) 0 yˆ 0 ˆ ˆ 7
Example (Supervisor Performace Data) data (=30, p=6) 8
scatter plot eed to be doe to see the validity of liearity assumptio model settig (eq. (3.3)): Y 0 1X1X 6X6 estimated LS fit (eq. (3.5)) LSE s with s.e. s (Table 3.5): idividually x 1 & x 3 the oly sigificat variables. PROC REG DATA=p054; MODEL y = x1 x x3 x4 x5 x6; RUN; 9
Aalysis of Variace Source DF Sum of Mea Squares Square F Value Pr > F Model 6 3147.96634 54.66106 10.5 <.0001 Error 3 1149.0003 49.95654 Corrected Total 9 496.96667 Root MSE 7.06799 R-Square 0.736 Depedet Mea 64.63333 Adj R-Sq 0.668 Coeff Var 10.9355 Parameter Estimates Variable DF Parameter Stadard Estimate Error t Value Pr > t Itercept 1 10.78708 11.5896 0.93 0.3616 X1 1 0.61319 0.16098 3.81 0.0009 X 1-0.07305 0.1357-0.54 0.5956 X3 1 0.3033 0.1685 1.90 0.0699 X4 1 0.08173 0.148 0.37 0.7155 X5 1 0.03838 0.14700 0.6 0.7963 X6 1-0.1706 0.1781-1. 0.356 10
Measurig the quality of fit ( 3.7) i. Decompositio of sum of squares : ( yi y) ( yi yˆi) ( yˆi y) 1 1 1 SST SSE SSR ( 1) ( p1) ( p) 1 1 Recall, for e y yˆ, e 0, x1e 0,, x e 0; i i i i i i ip i ( x x ) e 0,, ( x x ) e 0 i i ip p i ( y yˆ )( yˆ y) e ˆ ( x x ) ˆ ( x x ) 0 i i i i 1 i1 1 p ip p i1 i1 ii. Multiple correlatio coefficiet (MCC) & Adjusted MCC R SSR SSE 1 SST SST ; 0 R 1 R 1 meas that determiatio of y by liear combiatio of x becomes larger or 11
proportio of variatio of y explaied by x 1, x, p 설명변수의개수가증가하면항상 R ; SSE 가무조건감소! 제약 (costrait) 이있으면 최소값은커지고 ˆ ˆ ˆ i ˆi i 0 1 i1 p ip SSE ( y y ) ( y x x ) i1 i1 mi ( y x x ) ( 0, 1,, p ) i1 i 0 1 i1 p ip 최대값은작아진다. 1
mi ( y x x ) p1 p1 1, 0 1 i 0 1 i1 p ip ( SSE of Model : y x x ) mi ( y x x ) 1 i i 0 1 i1 p 0 1 i1 p ip i ( SSE of Model: y x x ) i 0 i1 p ip ip i SSE( reduced model) SSE( full model) R R ( reduced model) ( full model) R 는설명변수의개수가서로다른모형간의적합도비교에는부적절함. 따라서, 다음의 adjusted R 를고려함. SSE ( p 1) Ra 1 SST ( 1) 설명변수개수가서로다른모형의적합도비교에사용 13
Example (Supervisor Performace Data) Full model: y0 1x1 6x6 (SS) (df) SSR SSE SST 3147.97 6 1149 3 496.97 9 3147.97 1149 3 R 0.73, R a 1 0.66 496.97 496.97 9 Simpler (Reduced) model : y 0 1x13x3 (SS) (df) SSR SSE SST 304.3 154.65 7 496.97 9 304.3 154.65 7 R 0.708, R a 1 0.686 496.97 496.97 9 14
Hypotheses testig i liear regressio model ( 3.9) i. Reduced Model ( 축소모형 ) v.s. Full Model ( 완전모형 ) H 0 : reduced model (RM) v.s. H 1: full model (FM) where RMFM (RM: (q +1) regressio parameters, FM:( p +1) regressio parameters q p r) (example) (FM) y 0 1x16x 6 ( i1,,30) i i i i (RM) (a) H0: y i 0 i v.s. H 1 : full model H0: 1 6 0 v.s. H 1 : ot H 0 (b) H0: y0 0 1xi1 3xi3 i v.s. H 1 : full model H0: 4 5 6 0 v.s. H 1 : ot H 1 15
ii. sums-of-squares i RM & FM df p1 FM FM SSE( FM ) ( yi yˆ i ), yˆ i : l.s.fit uder FM (# of parameters p 1) i1 RM RM SSE( RM ) ( yi yˆi ), yˆi : l.s.fit uder RM (# of parameters q 1) i1 df q1 df p SSR( FM ) SST SSE( FM ) SSR( RM ) SST SSE( RM ) df q, SST ( y y) SSE( FM ) SSE( RM ); SSR( FM ) SSR( RM ) i 0 1 i1 p ip p1 1 mi ( yi 0 1xi1 pxip) SSE( RM) restrictios 1 wrt.. SSE( FM ) mi ( y x x ) SSE( RM ) SSE( FM ): i1 i Degree of freedom ( df ) SSE : (# of parameters) SSR :(# of parameters) 1 SST : 1 reductio i residual s.s. by itroducig p SSR( FM ) SSR( RM ): added amout of explaatio due to the p 16 q more parameters (variables) to RM q more parameters (variables) to RM
iii. F-test statistic for RM vs FM : F SSR( FM ) SSR( RM ) #( FM ) #( RM ) SSE( FM ) #( FM ) ( Rp Rq) {( p1) ( q1)} (1 Rp ) ( p1) SSE( RM ) SSE( FM ) #( FM ) #( RM ) SSE( FM ) #( FM ) F ~ F( pq, p 1) uder H 0 Reject RM vs FM if F F( pq, p 1; ) p-value: p 1 1 1 ESSRFM ( ) SSEFM ( ) ( i ), i p1 p1 i1 i1 17
Example: (Supervisor Performace Data) (FM) iid i 0 1 i1 6 i6 i, i ~ (0, ) y x x N H : 0 v.s. H 1 : ot H 0 0 4 5 6 PROC REG DATA=p054; MODEL y = x1 x x3 x4 x5 x6; TEST x=x4=x5=x6=0; RUN; i.e., RM is yi 0 1xi13xi3 i. SSE(FM)=1149 (df=3) SSE(RM)=154.65 (df=7) F (154.65 1149) (6 ) 0.58; F F(4,3;0.05).8. 1149 3 ( or, F ( RFM RRM ) (6 ) (0.736 0.708) (6 ) (1 RFM ) (30 7) (1 0.736) (30 7) ) Do ot reject H 0 at 5% level! 18
iv. Iferece after adaptig a reduced model : - Test more reduced model v.s. the reduced model ew reduced model v.s. ew full model Example (Supervisor Performace Data) New full model: iid i 0 1 i13 i3 i, i ~ (0, ) y x x N PROC REG DATA=p054; MODEL y = x1 x3; RUN; 19
1 Sigificace of x 1 ad x 3 <Table 3.8> (FM) H1: yi 0 1xi1 3xi3 i v.s. (RM) H0: yi 0 i H0: 1 3 0 v.s. H 1 : ot H 0 0
F ( SSE( RM ) SSE( FM )) / (3 1) ( SST SSE( FM )) / (3 1) SSE( FM )/( 3) SSE( FM )/( 3) ; highly sigificat SSR( FM ) 3.7 F(, 7;0.05) SSE( FM ) 7 <ANOVA Table> Source S.S d.f. Mea square F-test Regressio SSR p MSR = SSR / p F = MSR / MSE Residual SSE -p-1 MSE = SSE / (-p-1) Total SST -1 Sigificace of x 1 : H0: 1 0 v.s. H 1 : ot H 0 - (RM) H0: yi 0 3xi3 i v.s. (FM) H1: yi 0 1xi13xi3 i - Either F-test or t-test 1
- t-test : t ˆ 0 0.6435 se..( ˆ ) 0.1185 1 ; t(7;0.05) 1 5.43 와비교또는 p -value<0.0001 H : v.s. H 1 : ot H 0 3 0 1 3 H 0 : yi 0 1 ( xi1xi3) i v.s. H 1 : yi 0 1xi13xi3 i F ( RFM RRM ) ( 1) (0.708 0.6685) 1 3.65 (1 RFM ) ( 3) (1 0.708) 7 F F(1,7;0.05) 4.1 Do ot reject H 0 at 5% level PROC REG DATA=p054; MODEL y = x1 x3; TEST x1=x3; RUN;
Iterpretatios of regressio coefficiets ( 3.5) yi 0 1xi1 pxip i, i1,, i. 0(costat coef.) : the value of y whe x 1 x x p 0 ii. j (regressio coef.) : the chage of y correspodig to a uit chage i x ( j 1,, p ) j whe x i s (i j) are hold costat (fixed) iii. also called partial regressio coef. e.g.: Yˆ 15.33 0.78X1 0.050X Yˆ 14.38 0.75X eyx 1 1 1 Xˆ 18.97 0.51X ex X 1 1 eˆ 0 0.050e 3 YX 1 XX1 j ( j ): the cotributio of X j ( X ) to the respose variable Y after both variables have bee liearly adjusted for the other predictor variables ( X 1). 3
Graphical relatioship betwee full ad reduced model t Let 1 t P 1 11 1, ad what is a graphical meaig of 1 t χ I P x, χ t t of χχχχy PY? χ Let ( 1 ) t P χχχχ, X (1, x) t ( I1 11 1) x( IP 1 ) x? t Px ad t, ad PX t X XX X. What is a graphical meaig Fid a relatioship amog PYPY, χ, 1 ad PY X. ˆ 0 y y y1 x x x1 ˆ 1 y1 4 1