1
00:00:08,491 --> 00:00:09,490
PROFESSOR: Hi, everyone.

2
00:00:09,490 --> 00:00:10,780
Welcome back.

3
00:00:10,780 --> 00:00:14,050
So today, I'd like to talk about
positive definite matrices.

4
00:00:14,050 --> 00:00:17,650
And specifically, we're going
to analyze several properties

5
00:00:17,650 --> 00:00:19,440
of positive definite matrices.

6
00:00:19,440 --> 00:00:21,860
And specifically,
we're going to look

7
00:00:21,860 --> 00:00:24,710
at why each one of these
following statements is true.

8
00:00:24,710 --> 00:00:29,970
So first off, why every positive
definite matrix is invertible.

9
00:00:29,970 --> 00:00:32,270
Why the only positive
definite projection matrix

10
00:00:32,270 --> 00:00:34,860
is the identity matrix.

11
00:00:34,860 --> 00:00:38,320
If D is a diagonal matrix
with positive entries,

12
00:00:38,320 --> 00:00:41,010
show that it must also
be positive definite.

13
00:00:41,010 --> 00:00:44,260
And then lastly, if S is
a symmetric matrix where

14
00:00:44,260 --> 00:00:46,980
the determinant S
is bigger than 0,

15
00:00:46,980 --> 00:00:48,970
show why this might
not necessarily

16
00:00:48,970 --> 00:00:50,870
imply that it's
positive definite.

17
00:00:50,870 --> 00:00:53,239
So I'll let you think
about these for a moment.

18
00:00:53,239 --> 00:00:54,530
And I'll come back in a second.

19
00:01:08,730 --> 00:01:09,410
Hi, everyone.

20
00:01:09,410 --> 00:01:10,640
Welcome back.

21
00:01:10,640 --> 00:01:11,170
OK.

22
00:01:11,170 --> 00:01:15,820
So let's take a look
at part A. So part A

23
00:01:15,820 --> 00:01:21,140
is asking why every positive
definite matrix is invertible.

24
00:01:21,140 --> 00:01:24,680
Well, let's just recall
that if A is a matrix

25
00:01:24,680 --> 00:01:33,010
and if A is invertible,
then this necessarily

26
00:01:33,010 --> 00:01:37,795
implies that the determinant
of A is non-zero.

27
00:01:40,890 --> 00:01:42,500
And I'm going to
just write out det

28
00:01:42,500 --> 00:01:53,710
A as the product of the
eigenvalues of A. So lambda_1

29
00:01:53,710 --> 00:02:06,180
to lambda_n are the
eigenvalues of A.

30
00:02:06,180 --> 00:02:07,070
OK.

31
00:02:07,070 --> 00:02:18,480
In addition, if A is
positive definite,

32
00:02:18,480 --> 00:02:21,000
what does this say about
the eigenvalues of A?

33
00:02:21,000 --> 00:02:24,900
Well, it says that each
eigenvalue of A, lambda_1,

34
00:02:24,900 --> 00:02:30,700
lambda_2, dot, dot, dot, to
lambda_n, each one of them

35
00:02:30,700 --> 00:02:33,440
must be bigger than 0.

36
00:02:33,440 --> 00:02:36,700
So this statement that
each eigenvalue of A

37
00:02:36,700 --> 00:02:38,790
is bigger than 0 is
completely equivalent

38
00:02:38,790 --> 00:02:46,860
to A being positive definite
for symmetric matrices A.

39
00:02:46,860 --> 00:02:50,260
So if I have a whole bunch of
eigenvalues and each of them

40
00:02:50,260 --> 00:02:52,880
are bigger than 0, what
does this say about det A?

41
00:02:56,530 --> 00:02:58,780
Well, I can take the product
of all these eigenvalues.

42
00:03:03,460 --> 00:03:06,740
And of course, the product of a
whole bunch of positive numbers

43
00:03:06,740 --> 00:03:07,660
must also be positive.

44
00:03:10,540 --> 00:03:14,640
So if the quantity is positive,
then it certainly cannot equal

45
00:03:14,640 --> 00:03:15,800
0.

46
00:03:15,800 --> 00:03:18,770
So this proves that det
A is not equal to 0.

47
00:03:18,770 --> 00:03:22,590
Hence, A must be invertible.

48
00:03:22,590 --> 00:03:23,090
OK.

49
00:03:23,090 --> 00:03:29,850
So for part B,
we're asked to show

50
00:03:29,850 --> 00:03:31,810
that the only positive
definite projection

51
00:03:31,810 --> 00:03:35,070
matrix is the identity matrix.

52
00:03:35,070 --> 00:03:37,220
So again, how do we
tackle this problem?

53
00:03:37,220 --> 00:03:39,930
We're going to look
at the eigenvalues.

54
00:03:39,930 --> 00:03:49,690
So remember, if P
is a projection,

55
00:03:49,690 --> 00:03:52,080
what does it say about
the eigenvalues of P?

56
00:03:52,080 --> 00:04:03,520
Well, it says that
the eigenvalues of P

57
00:04:03,520 --> 00:04:05,370
are either 0 or 1.

58
00:04:09,390 --> 00:04:11,520
So this is point one.

59
00:04:11,520 --> 00:04:25,150
Point two, if P is
a positive definite,

60
00:04:25,150 --> 00:04:27,706
what does that say about
the eigenvalues of P?

61
00:04:27,706 --> 00:04:43,670
Well, as I've noted before,
it means that the eigenvalues

62
00:04:43,670 --> 00:04:44,580
are bigger than 0.

63
00:04:47,210 --> 00:04:51,700
So if P is a projection
and it's positive definite,

64
00:04:51,700 --> 00:04:54,540
the only possible
eigenvalues that

65
00:04:54,540 --> 00:04:59,310
are both 0 and 1 and
bigger than 0 are 1.

66
00:04:59,310 --> 00:05:09,540
So the conclusion is
that the eigenvalues of P

67
00:05:09,540 --> 00:05:12,950
must all equal 1.

68
00:05:12,950 --> 00:05:17,870
So which matrix has eigenvalues
1 and is also symmetric?

69
00:05:17,870 --> 00:05:26,110
Well, the only matrix that
satisfies this property

70
00:05:26,110 --> 00:05:28,660
is the identity matrix.

71
00:05:28,660 --> 00:05:31,270
Now you might ask, how do
you actually show that?

72
00:05:31,270 --> 00:05:35,170
Well, you could
argue as follows.

73
00:05:35,170 --> 00:05:47,350
If P is diagonalizable--
and every symmetric matrix

74
00:05:47,350 --> 00:05:50,980
is diagonalizable, so
I'm not making this up.

75
00:05:50,980 --> 00:05:52,770
So if P is
diagonalizable, then you

76
00:05:52,770 --> 00:05:56,870
can always write P
as some matrix, U,

77
00:05:56,870 --> 00:06:00,530
times a diagonal matrix--
and we know in this case,

78
00:06:00,530 --> 00:06:03,900
the diagonal matrix
has eigenvalues 1,

79
00:06:03,900 --> 00:06:08,907
so it's actually the identity
matrix-- times the inverse

80
00:06:08,907 --> 00:06:09,990
of the eigenvector matrix.

81
00:06:12,530 --> 00:06:17,400
But of course, this is
just U times U inverse,

82
00:06:17,400 --> 00:06:21,660
which then gives me the
identity at the end.

83
00:06:21,660 --> 00:06:23,970
So U times the identity
times U inverse.

84
00:06:23,970 --> 00:06:25,859
This is just U times U inverse.

85
00:06:25,859 --> 00:06:27,900
And of course, U and U
inverse collapse back down

86
00:06:27,900 --> 00:06:29,050
to the identity.

87
00:06:29,050 --> 00:06:32,850
So this shows you that the only
matrix that has eigenvalues

88
00:06:32,850 --> 00:06:35,420
of 1 is the identity matrix.

89
00:06:35,420 --> 00:06:39,550
So that's just to cross all
the T's and dot all the I's.

90
00:06:39,550 --> 00:06:40,050
OK.

91
00:06:40,050 --> 00:06:45,120
For part C, we're given
D as a diagonal matrix

92
00:06:45,120 --> 00:06:47,140
with positive entries
on the diagonal.

93
00:06:47,140 --> 00:06:50,140
Now we have to show that
it's positive definite.

94
00:06:50,140 --> 00:06:50,640
OK.

95
00:06:50,640 --> 00:06:53,102
So let me write D as follows.

96
00:06:53,102 --> 00:06:54,310
I'll just write it like this.

97
00:06:54,310 --> 00:06:57,580
I'm going to use a compact
notation, which is sometimes

98
00:06:57,580 --> 00:07:05,600
seen: diagonal d_1, d_2, d_n.

99
00:07:05,600 --> 00:07:09,010
So D is a diagonal matrix whose
elements along the diagonal

100
00:07:09,010 --> 00:07:13,130
are d_1, d_2, dot,
dot, dot, to d_n.

101
00:07:13,130 --> 00:07:16,220
Now what does it mean for a
matrix to be positive definite?

102
00:07:16,220 --> 00:07:28,150
Well, it means that for
any x, for any vector x,

103
00:07:28,150 --> 00:07:34,950
I have to look at the
product x transpose D*x.

104
00:07:34,950 --> 00:07:38,070
And I have to show that
it's bigger than 0.

105
00:07:38,070 --> 00:07:41,100
And I should
qualify this and say

106
00:07:41,100 --> 00:07:43,330
that the vector we're looking
at is x not equal to 0.

107
00:07:43,330 --> 00:07:46,050
So we're now looking
at the zero vector.

108
00:07:46,050 --> 00:07:47,710
But for D to be
positive definite,

109
00:07:47,710 --> 00:07:51,320
we have to show that x
transpose D*x is bigger than 0.

110
00:07:51,320 --> 00:07:53,894
This is just one way to show
that it's positive definite.

111
00:07:53,894 --> 00:07:54,810
It's not the only way.

112
00:08:00,160 --> 00:08:06,872
So if I write x out using
components, x_1, x_2, dot, dot,

113
00:08:06,872 --> 00:08:15,300
dot, to x_n-- I'll
write it like this--

114
00:08:15,300 --> 00:08:19,050
then you can work out the
quantity x transpose D*x.

115
00:08:19,050 --> 00:08:21,160
And we see that we
get a sum of squares.

116
00:08:21,160 --> 00:08:28,690
We get d_1 times x_1 squared
plus d_2 x_2 squared plus dot,

117
00:08:28,690 --> 00:08:33,694
dot, dot plus d_n x_n squared.

118
00:08:36,960 --> 00:08:41,750
Now by definition, each
coefficient is positive.

119
00:08:41,750 --> 00:08:44,290
A sum of a square is positive.

120
00:08:44,290 --> 00:08:45,980
so-- sorry.

121
00:08:45,980 --> 00:08:49,180
A product of a positive number
with a square is positive.

122
00:08:49,180 --> 00:08:51,220
And then of course, a
sum of positive numbers

123
00:08:51,220 --> 00:08:54,060
is going to be positive.

124
00:08:54,060 --> 00:08:57,340
So this means the whole
thing is positive.

125
00:08:57,340 --> 00:08:59,940
Now there's other
more efficient ways

126
00:08:59,940 --> 00:09:01,950
of getting at this using
other tricks we know.

127
00:09:01,950 --> 00:09:04,230
For example, if we're
given a diagonal matrix,

128
00:09:04,230 --> 00:09:06,700
we know its eigenvalues
are already d_1,

129
00:09:06,700 --> 00:09:09,130
d_2, dot, dot, dot, to d_n.

130
00:09:09,130 --> 00:09:13,240
And we know that a matrix
with positive eigenvalues

131
00:09:13,240 --> 00:09:16,010
is already positive definite.

132
00:09:16,010 --> 00:09:20,350
But this is kind of
starting from the base

133
00:09:20,350 --> 00:09:23,930
to show that it's
positive definite.

134
00:09:23,930 --> 00:09:27,910
And now lastly,
let's look at part D.

135
00:09:27,910 --> 00:09:32,730
So S is a symmetric matrix
with det S bigger than 0.

136
00:09:32,730 --> 00:09:34,510
Show that S might
not necessarily

137
00:09:34,510 --> 00:09:36,110
be positive definite.

138
00:09:36,110 --> 00:09:38,020
So there's lots of
counterexamples.

139
00:09:38,020 --> 00:09:39,480
I only need to construct one.

140
00:09:42,390 --> 00:09:44,837
So I'm looking at S, which
is a symmetric matrix.

141
00:09:44,837 --> 00:09:46,920
So I'm just going to throw
in some numbers on some

142
00:09:46,920 --> 00:09:47,930
off diagonals.

143
00:09:47,930 --> 00:09:50,400
So I'll just pick one.

144
00:09:50,400 --> 00:09:55,040
And now I need to pick some
numbers along the diagonal.

145
00:09:55,040 --> 00:09:56,710
And the easiest way
to do it is just

146
00:09:56,710 --> 00:09:59,900
to pick two negative
numbers on the diagonal.

147
00:09:59,900 --> 00:10:04,170
Because if there's a negative
number on a diagonal,

148
00:10:04,170 --> 00:10:06,972
then we know S can't
be positive definite.

149
00:10:06,972 --> 00:10:08,680
And I'll say more
about that in a second.

150
00:10:12,390 --> 00:10:14,380
So I can just pick
negative 2 and negative 3.

151
00:10:14,380 --> 00:10:15,955
So let's quickly
check what det S is.

152
00:10:21,520 --> 00:10:25,460
Well, it's negative 2
times negative 3 minus 1.

153
00:10:25,460 --> 00:10:29,820
So this gives me 6
minus 1, which is 5.

154
00:10:29,820 --> 00:10:33,610
So by construction, det S
is positive, which is good.

155
00:10:33,610 --> 00:10:37,410
And then, as I mentioned
before, if there's

156
00:10:37,410 --> 00:10:39,700
negative elements along
the diagonal of the matrix,

157
00:10:39,700 --> 00:10:41,500
that matrix can't be
positive definite.

158
00:10:41,500 --> 00:10:43,250
Well, why is that?

159
00:10:43,250 --> 00:10:45,850
Well, suppose I
wanted to take a look

160
00:10:45,850 --> 00:10:52,730
at this upper left
component, negative 3,

161
00:10:52,730 --> 00:10:55,370
and it's negative, how do
I show that that implies

162
00:10:55,370 --> 00:10:58,710
S is not positive definite?

163
00:10:58,710 --> 00:11:03,133
Well, what I can do is I can
look at the product x transpose

164
00:11:03,133 --> 00:11:03,633
S*x.

165
00:11:06,790 --> 00:11:08,750
And what I do is I look at it.

166
00:11:08,750 --> 00:11:12,170
I just take one value of x.

167
00:11:12,170 --> 00:11:15,050
So we know that this has to be
positive for every value of x.

168
00:11:15,050 --> 00:11:17,280
So I can pick any value I want.

169
00:11:17,280 --> 00:11:22,660
So I can take x, say,
[1, 0] transpose.

170
00:11:25,600 --> 00:11:33,930
And when I do this, we end up
getting that x transpose S*x is

171
00:11:33,930 --> 00:11:37,270
equal to negative 3.

172
00:11:37,270 --> 00:11:40,370
So notice how by taking 1
in the first entry and 0

173
00:11:40,370 --> 00:11:43,770
on the second entry, that picks
out the upper left corner,

174
00:11:43,770 --> 00:11:44,990
negative 3.

175
00:11:44,990 --> 00:11:48,010
If I were to take
0 here and 1 here,

176
00:11:48,010 --> 00:11:52,000
it would pick out the
negative 2 entry in S.

177
00:11:52,000 --> 00:11:58,200
So by picking 1 in
entry i of a vector x,

178
00:11:58,200 --> 00:12:01,530
and then computing this
product x transpose S*x,

179
00:12:01,530 --> 00:12:05,640
I pick out the i-th
element along the diagonal.

180
00:12:05,640 --> 00:12:09,740
And since that
element is negative,

181
00:12:09,740 --> 00:12:12,490
this shows me that along
the direction [1, 0],

182
00:12:12,490 --> 00:12:15,440
the product x transpose
S*x is also negative.

183
00:12:15,440 --> 00:12:19,090
And hence, S can't possibly
be positive definite.

184
00:12:19,090 --> 00:12:19,990
OK.

185
00:12:19,990 --> 00:12:21,730
So just to summarize,
we've taken a look

186
00:12:21,730 --> 00:12:25,120
at a couple matrices and a
couple different properties

187
00:12:25,120 --> 00:12:27,090
of positive definite matrices.

188
00:12:27,090 --> 00:12:30,510
And notably, we've
used the eigenvalues

189
00:12:30,510 --> 00:12:33,330
to get a handle of the
positive definite matrices.

190
00:12:33,330 --> 00:12:36,120
And I hope these provide
some useful tricks.

191
00:12:36,120 --> 00:12:38,570
And I'll see you next time.