---
W.L.S. & L.O.F.
1. Weighted Least Squares.
- (all matrix) y = Xb + e where e ~ N(0,sigma^2 * I) [a]
RSS(b) = (y-Xb)'(y-Xb)
- y = Xb + e where e ~ N(0,sigma^2 * summation), var(ei) = sigma^2 * summationii
Thus
RSS(b) = (y-Xb)'summation^-1(y-Xb)
As the summation inverse matrix (a diagonal matrix) increases, RSS will increase as well. However, this is merely a generalized case, the summational matrix doesn't has to be diagonal and sometimes hard to obtain the inverse of.
bhat(OLS) = (X'X)^-1X'y
bhat(WLS) = (X'summation^-1X)'(X'summation^-1y) [the ' can be replaced with ^-1]
Definition: Square Root Matrix
A square root matrix is a matrix that C'C = CC' = Summation^-1. (If exist)
Then we call C the square root of Summation^-1
Var(Ce) = CVar(e)C' = Csigma^2summationC' = sigma^2C(C'C)^-1C' = sigma^2CC^-1(C')^-1C' = sigma^2I
Now Cy = CXb + Ce, we let Z = Cy and M = CX and d = Ce
Then Z = Mb + d --> very similar to the original one unless var(d) = sigma^2I
bhat = (M'M)^-1M'y
---
Side Note: Seems like no one is paying attention to the poor TA. 1 girl sitting in front of me is playing tetris on the computer and another is checking e-mail. The guy sitting beside me is checking Fido's X'mas plan and another guy sitting in the first row is answering web MSN.
NICE CLASS... why did all these people come here so early for?? 9 in the morning is still sleeping time, man.
---
bhat = (M'M)^-1M'Z = (X'C'CX)^-1X'C'Cy = (X'summation^-1X)^-1X'summation^-1y
W.L.S.
W = summation^-1 =
[w1 ... 0]
[0 w2 ... 0]
[0 0 w3 ... 0]
[ ... ]
[ 0 ... wn]
where summation = same matrix as above except all wi replace with 1/ui
* Diagonal Matrix are always symmertic.
In real practice, we may not always know what the sigma matrix is, so what we can do is to replace the exact variance by the sample variance.
---
Side note 2: Sorry, I take back my words. The TA is not so poor, she's quite cute, in the way that she works so passionately even tho so little people showed up and the ones who showed up seems so un-interested in what she's talking about. I mean... it's hard to keep one's spirit up in a situation like this, bravo to her... Bravo...
---
Eh... I think that's the end of this class...
