1
00:00:00,000 --> 00:00:01,470
NARRATOR: The
following content is

2
00:00:01,470 --> 00:00:04,690
provided by MIT OpenCourseWare
under a Creative Commons

3
00:00:04,690 --> 00:00:06,100
license.

4
00:00:06,100 --> 00:00:08,230
Additional information
about our license

5
00:00:08,230 --> 00:00:10,490
and MIT OpenCourseWare
in general

6
00:00:10,490 --> 00:00:14,370
is available at ocw.mit.edu.

7
00:00:14,370 --> 00:00:23,470
PROFESSOR: This is my lecture
on section 6.1 of the notes.

8
00:00:26,340 --> 00:00:30,385
Really, the central problem
of numerical linear algebra is

9
00:00:30,385 --> 00:00:32,750
solving A*x equal
b by elimination.

10
00:00:32,750 --> 00:00:37,860
That's what is used much more
often than anything else,

11
00:00:37,860 --> 00:00:42,420
but we're talking about
sparse matrices, large sparse

12
00:00:42,420 --> 00:00:42,980
matrices.

13
00:00:42,980 --> 00:00:48,140
That's what we have in
mind for a long time now --

14
00:00:48,140 --> 00:00:51,710
probably all lectures coming up
-- thinking of those matrices.

15
00:00:51,710 --> 00:00:56,420
And this is the prototype
of a large sparse matrix.

16
00:00:59,210 --> 00:01:01,250
The main point of
the lecture is,

17
00:01:01,250 --> 00:01:04,860
if you take the unknowns in
whatever order they're given --

18
00:01:04,860 --> 00:01:09,680
here I took them numbering along
the rows -- one, two, three,

19
00:01:09,680 --> 00:01:14,190
four, five, six, seven,
eight and so on --

20
00:01:14,190 --> 00:01:18,180
you got to stop and think
or get the computer to look

21
00:01:18,180 --> 00:01:21,530
at reorderings, because
the reorderings can make

22
00:01:21,530 --> 00:01:25,080
an enormous difference
in the fill-in.

23
00:01:25,080 --> 00:01:30,160
So this is, actually, the key
word for this lecture, is --

24
00:01:30,160 --> 00:01:33,830
and the key concept is the
fact that as elimination goes

25
00:01:33,830 --> 00:01:42,270
forward, a lot of these
zeros become non-zeros.

26
00:01:45,060 --> 00:01:46,880
Let me see it
happen, first of all.

27
00:01:46,880 --> 00:01:48,900
So this is the whole
idea of fill-in.

28
00:01:48,900 --> 00:01:50,970
What's going to be the
very first entries that

29
00:01:50,970 --> 00:01:54,590
are filled in?

30
00:01:54,590 --> 00:01:55,440
Let's see.

31
00:01:55,440 --> 00:01:58,960
I guess that the very
first row operation,

32
00:01:58,960 --> 00:02:03,670
if you took the matrix like
this and did elimination, 4

33
00:02:03,670 --> 00:02:05,310
is the first pivot.

34
00:02:05,310 --> 00:02:09,430
You would add 1/4 of the
first row to the second row

35
00:02:09,430 --> 00:02:12,330
to make that a 0, right?

36
00:02:12,330 --> 00:02:13,940
Then what would happen?

37
00:02:13,940 --> 00:02:20,030
This would change that number,
but this would put a new number

38
00:02:20,030 --> 00:02:20,630
in there.

39
00:02:23,150 --> 00:02:25,400
So a new number
would appear when

40
00:02:25,400 --> 00:02:28,610
we do that first row operation.

41
00:02:28,610 --> 00:02:31,260
And then, also working
with this pivot,

42
00:02:31,260 --> 00:02:36,960
I would operate on that
row to make this a 0,

43
00:02:36,960 --> 00:02:41,360
and then this would change,
that would fill in, right?

44
00:02:41,360 --> 00:02:43,753
I guess that this
already has something,

45
00:02:43,753 --> 00:02:46,110
so that won't be new.

46
00:02:46,110 --> 00:02:52,170
So each of those steps, each
of those row operations filled

47
00:02:52,170 --> 00:02:59,610
in a non-zero and reduced
the sparsity of the matrix,

48
00:02:59,610 --> 00:03:01,790
and the sparsity is
what we're living by.

49
00:03:04,900 --> 00:03:11,440
Now if I continue and
continue, we'd need MATLAB --

50
00:03:11,440 --> 00:03:19,210
to use spy in MATLAB to see what
the picture would look like.

51
00:03:19,210 --> 00:03:21,670
It's very interesting to see it.

52
00:03:21,670 --> 00:03:24,750
Spy shows you the non-zeros.

53
00:03:24,750 --> 00:03:27,040
We're very much
focused on what's 0

54
00:03:27,040 --> 00:03:31,090
and what isn't 0
in matrix entries.

55
00:03:31,090 --> 00:03:35,550
We actually keep track of that,
the structure of a matrix,

56
00:03:35,550 --> 00:03:37,830
as well, of course,
as separately keeping

57
00:03:37,830 --> 00:03:40,430
track of the actual numbers.

58
00:03:40,430 --> 00:03:42,550
One place, we're keeping
the actual numbers,

59
00:03:42,550 --> 00:03:45,770
but in the decisions
on ordering,

60
00:03:45,770 --> 00:03:49,130
we're only interested
in zeros and non-zeros.

61
00:03:49,130 --> 00:03:53,600
So let me look even
more at this fill-in.

62
00:03:53,600 --> 00:03:55,590
Now, you saw it happen up there.

63
00:03:55,590 --> 00:04:00,200
Let me just look at a
single fill-in step.

64
00:04:00,200 --> 00:04:09,970
So I have a row j
and a row i, and this

65
00:04:09,970 --> 00:04:12,930
is going to be the pivot that
I'll use to eliminate this,

66
00:04:12,930 --> 00:04:19,430
but in this row, there's another
non-zero and in this row,

67
00:04:19,430 --> 00:04:23,690
the k-th entry is
not non-zero; it's 0.

68
00:04:23,690 --> 00:04:27,760
So this is what we start
with prior to elimination,

69
00:04:27,760 --> 00:04:29,900
just as we did there.

70
00:04:29,900 --> 00:04:34,150
I just want to see
-- focus on small --

71
00:04:34,150 --> 00:04:38,040
on the fill-in occurring.

72
00:04:40,570 --> 00:04:43,290
Of course, this isn't
on the diagonal.

73
00:04:43,290 --> 00:04:50,920
This is in column k,
which is somewhere far

74
00:04:50,920 --> 00:04:57,260
out in that case, somewhere
close by in that case.

75
00:04:57,260 --> 00:05:00,990
But we're just -- this
is on the diagonal.

76
00:05:00,990 --> 00:05:04,630
That's the pivot, the j-th
pivot, that I'm ready to use,

77
00:05:04,630 --> 00:05:07,680
so I subtract a multiple
of that to remove that.

78
00:05:07,680 --> 00:05:14,990
So what does -- elimination will
not change the pivot row and it

79
00:05:14,990 --> 00:05:19,190
will produce a 0 there,
that's the point.

80
00:05:19,190 --> 00:05:21,260
Then the real point
that I'm making

81
00:05:21,260 --> 00:05:23,880
is that when I multiple
this by the right number

82
00:05:23,880 --> 00:05:26,130
to produce 0 and do
this subtraction,

83
00:05:26,130 --> 00:05:29,800
this 0 disappears,
that fills in.

84
00:05:29,800 --> 00:05:36,850
That's some a_(i,k) new that
-- the new value in row i,

85
00:05:36,850 --> 00:05:39,650
column k is not 0.

86
00:05:42,640 --> 00:05:46,350
so we see that -- as
we saw it for numbers,

87
00:05:46,350 --> 00:05:48,570
here's we're seeing
it symbolically.

88
00:05:48,570 --> 00:05:52,310
Let me see it graphically,
because everybody sort of --

89
00:05:52,310 --> 00:05:56,050
to visualize the big picture.

90
00:05:56,050 --> 00:05:58,850
I think everybody
finds the graph

91
00:05:58,850 --> 00:06:04,120
description of the non-zero
structure the simplest

92
00:06:04,120 --> 00:06:05,200
to imagine.

93
00:06:05,200 --> 00:06:08,740
So the graph for this
little problem would be --

94
00:06:08,740 --> 00:06:13,410
so what's the graph that tells
me where the non-zeros are?

95
00:06:13,410 --> 00:06:16,760
Again, it doesn't tell
me what numbers they are,

96
00:06:16,760 --> 00:06:19,970
just tells me where they
are, which positions.

97
00:06:19,970 --> 00:06:23,960
It's only the positions
that I worry about

98
00:06:23,960 --> 00:06:27,290
in deciding on the
ordering to use.

99
00:06:27,290 --> 00:06:30,510
Then I execute using
the actual numbers,

100
00:06:30,510 --> 00:06:34,220
but in deciding on the ordering,
it's just the position.

101
00:06:34,220 --> 00:06:42,070
So I think of my graph as
having a node for every row.

102
00:06:42,070 --> 00:06:45,440
So that's node j for row j.

103
00:06:45,440 --> 00:06:49,490
This is node i for row i.

104
00:06:49,490 --> 00:06:52,910
Somewhere we'll have a node k.

105
00:06:55,690 --> 00:06:59,219
So, at the start -- so
those are the nodes.

106
00:06:59,219 --> 00:07:00,010
What are the edges?

107
00:07:03,440 --> 00:07:06,080
There is a non -- so the
edges come from the non-zeros

108
00:07:06,080 --> 00:07:06,790
in the matrix.

109
00:07:06,790 --> 00:07:12,040
So there's an edge from j
to k, because this is not 0.

110
00:07:12,040 --> 00:07:14,240
There's an edge from
i to j or j to i.

111
00:07:14,240 --> 00:07:16,240
I'm not worrying
about the direction --

112
00:07:16,240 --> 00:07:18,530
this is not a directed
graph right now.

113
00:07:18,530 --> 00:07:22,850
So this is the start, but
there's no edge here initially,

114
00:07:22,850 --> 00:07:27,230
because a_(i, k) is 0.

115
00:07:27,230 --> 00:07:30,810
Now in this graph
world -- it's so small,

116
00:07:30,810 --> 00:07:33,890
we can just focus on the
part that's going to change

117
00:07:33,890 --> 00:07:34,960
a little.

118
00:07:34,960 --> 00:07:40,380
The rest of all the other
rows have their nodes

119
00:07:40,380 --> 00:07:43,730
and all the other non-zero
entries have their edges.

120
00:07:43,730 --> 00:07:48,670
So this is a tiny
part of the big graph.

121
00:07:48,670 --> 00:07:51,810
What happens to this tiny part?

122
00:07:51,810 --> 00:07:56,700
So again, I want to see the
same elimination step, just

123
00:07:56,700 --> 00:08:00,030
this way, in the graph world.

124
00:08:00,030 --> 00:08:04,870
So after, what edges are there?

125
00:08:04,870 --> 00:08:12,020
So again, it's j, i and k --
and after I do elimination,

126
00:08:12,020 --> 00:08:14,510
j is still connected to k.

127
00:08:14,510 --> 00:08:15,600
That's still there.

128
00:08:15,600 --> 00:08:18,070
No change.

129
00:08:18,070 --> 00:08:21,690
j is no longer connected
to y, because elimination

130
00:08:21,690 --> 00:08:25,250
killed that edge, but
elimination created

131
00:08:25,250 --> 00:08:28,260
a new edge, i to k.

132
00:08:28,260 --> 00:08:31,480
So that's essentially
what elimination does.

133
00:08:31,480 --> 00:08:33,270
Maybe we can say it in words.

134
00:08:36,030 --> 00:08:41,140
When this edge disappears,
at the same time

135
00:08:41,140 --> 00:08:51,550
we create new edges
connecting any two nodes that

136
00:08:51,550 --> 00:08:53,370
were both connected to j.

137
00:08:53,370 --> 00:08:55,190
You see that point?

138
00:08:55,190 --> 00:09:04,970
If I'm using j as the pivot row
and I look on that row and say,

139
00:09:04,970 --> 00:09:07,430
what non-zeros are there?

140
00:09:07,430 --> 00:09:14,140
What edges are connected to
j, and I see one -- and then,

141
00:09:14,140 --> 00:09:17,610
when I do elimination,
I'm going to introduce,

142
00:09:17,610 --> 00:09:23,170
exactly as has happened here, a
new edge between any pair that

143
00:09:23,170 --> 00:09:24,990
were both connected to j.

144
00:09:24,990 --> 00:09:33,030
So we're kind of fighting
against creeping filling.

145
00:09:33,030 --> 00:09:35,390
The question is,
what order should we

146
00:09:35,390 --> 00:09:36,460
do these eliminations?

147
00:09:36,460 --> 00:09:39,830
So now, think of a larger graph.

148
00:09:39,830 --> 00:09:48,770
Actually, I guess we could
somehow think of that --

149
00:09:48,770 --> 00:09:52,410
we can try to visualize
it, but maybe not succeed.

150
00:09:52,410 --> 00:10:02,860
So the graph for that problem --
could that serve as the graph?

151
00:10:02,860 --> 00:10:05,240
I guess it could, really.

152
00:10:05,240 --> 00:10:10,180
I would remove -- I'd
[? whiz ?] away the --

153
00:10:10,180 --> 00:10:13,360
if I'm given the values
on the outer edges,

154
00:10:13,360 --> 00:10:16,810
then I would remove all
those, but essentially,

155
00:10:16,810 --> 00:10:20,840
this is the graph for
that matrix, right?

156
00:10:23,350 --> 00:10:28,860
If I number each of
the nodes 1 to 16,

157
00:10:28,860 --> 00:10:32,660
and I would do the numbering
the way I did to produce that

158
00:10:32,660 --> 00:10:35,240
matrix -- now the
whole question is,

159
00:10:35,240 --> 00:10:39,940
if I do some different
numbering, will --

160
00:10:39,940 --> 00:10:41,080
that'll shift around.

161
00:10:41,080 --> 00:10:43,120
That'll just be a
permutation matrix, right?

162
00:10:43,120 --> 00:10:46,180
If I do a different
numbering -- so matrix-wise,

163
00:10:46,180 --> 00:10:53,820
I have my matrix K2D -- what
happens matrix-wise for --

164
00:10:53,820 --> 00:10:58,630
if I do a numbering, a different
numbering, a reordering?

165
00:10:58,630 --> 00:11:04,660
That just permutes the
unknowns and the equations.

166
00:11:04,660 --> 00:11:08,420
So really, what happens is
some permutation multiplies

167
00:11:08,420 --> 00:11:13,660
K2D on the left and its
transpose does the same thing

168
00:11:13,660 --> 00:11:17,210
to the columns on the right,
because this permutation

169
00:11:17,210 --> 00:11:20,610
will change the order
of the equations.

170
00:11:20,610 --> 00:11:25,610
This permutation will reorder
the columns to come back,

171
00:11:25,610 --> 00:11:27,950
to bring the diagonal
back onto the diagonal,

172
00:11:27,950 --> 00:11:29,590
but in a different order.

173
00:11:29,590 --> 00:11:32,950
We won't know it with all
those 4s, but that will have,

174
00:11:32,950 --> 00:11:36,610
again, all the 4s on the
diagonal, but the minus 1s

175
00:11:36,610 --> 00:11:39,050
are moved around.

176
00:11:39,050 --> 00:11:42,810
Of course, none have disappeared
or been created by that.

177
00:11:42,810 --> 00:11:47,770
It's just that's
giving us a new order.

178
00:11:47,770 --> 00:11:51,490
So that the reordering
matrix-wise.

179
00:11:51,490 --> 00:11:54,600
What's the reordering
graph-wise?

180
00:11:54,600 --> 00:11:58,720
Graph-wise would be, instead of
this numbering of unknowns --

181
00:11:58,720 --> 00:12:02,600
one, two, three, four, five,
six, seven, eight and so on --

182
00:12:02,600 --> 00:12:04,940
nine, ten, eleven, twelve,
thirteen, fourteen, fifteen,

183
00:12:04,940 --> 00:12:08,160
sixteen -- we have a different
order, different numbering.

184
00:12:11,190 --> 00:12:16,910
The computations, the computer
science aspect, you could say,

185
00:12:16,910 --> 00:12:19,130
was not going to be difficult.

186
00:12:19,130 --> 00:12:24,670
Actually, we should think and
maybe we'll think in a minute,

187
00:12:24,670 --> 00:12:30,990
from the computer science
aspect, the actual execution --

188
00:12:30,990 --> 00:12:33,800
I'm probably not -- certainly
not going to write out

189
00:12:33,800 --> 00:12:35,130
a permutation matrix.

190
00:12:35,130 --> 00:12:36,660
That's ridiculous.

191
00:12:36,660 --> 00:12:40,340
I'm not going to draw a
graph and keep that --

192
00:12:40,340 --> 00:12:45,680
I'll keep some list of
positions for non-zeros,

193
00:12:45,680 --> 00:12:48,250
so we'll say a word
about that list later.

194
00:12:51,290 --> 00:12:53,750
Here I'm trying
to say, how do we

195
00:12:53,750 --> 00:12:56,690
get an idea of what the
effective reordering is

196
00:12:56,690 --> 00:12:59,510
and what a good
reordering would be?

197
00:12:59,510 --> 00:13:05,160
So here, this is a
specific reordering

198
00:13:05,160 --> 00:13:08,790
that I want to
mention right away,

199
00:13:08,790 --> 00:13:11,560
because it's so straightforward.

200
00:13:11,560 --> 00:13:17,420
You can get the idea of
the red-black ordering

201
00:13:17,420 --> 00:13:19,740
without any big computations.

202
00:13:19,740 --> 00:13:25,690
This is the --
fundamentally, the winner.

203
00:13:25,690 --> 00:13:32,640
This is what people use all
the time and MATLAB's backslash

204
00:13:32,640 --> 00:13:35,180
is going to have that available.

205
00:13:40,170 --> 00:13:43,450
Inside this, the
codes that are called

206
00:13:43,450 --> 00:13:48,160
are the column or
symmetric matrix,

207
00:13:48,160 --> 00:13:51,490
approximate minimum degree.

208
00:13:51,490 --> 00:14:00,850
So that's what this AMD means
-- approximate minimum degree.

209
00:14:00,850 --> 00:14:04,590
So that's the basic
idea that I'll

210
00:14:04,590 --> 00:14:06,860
come back to as
soon as I've just

211
00:14:06,860 --> 00:14:08,310
drawn the red-black picture.

212
00:14:11,070 --> 00:14:13,980
It's so important that
people don't stop thinking.

213
00:14:17,600 --> 00:14:24,120
So another sort of
high-level picture

214
00:14:24,120 --> 00:14:29,380
of how to do an ordering is
look for graph separators.

215
00:14:29,380 --> 00:14:38,090
That's become a big step in the
world of computational graph

216
00:14:38,090 --> 00:14:41,840
theory, and that leads
to something called

217
00:14:41,840 --> 00:14:45,920
nested dissection, which
is separators of separators

218
00:14:45,920 --> 00:14:50,330
of separators, the usual
keep going with the same idea

219
00:14:50,330 --> 00:14:53,815
that you see in algorithms.

220
00:14:53,815 --> 00:14:54,690
What about red-black?

221
00:14:57,200 --> 00:15:01,010
Red-black, I'm simply going
to imagine that this is

222
00:15:01,010 --> 00:15:04,720
a checkerboard -- it's the
nodes that I'm looking at, not

223
00:15:04,720 --> 00:15:08,180
the squares.

224
00:15:08,180 --> 00:15:11,380
So that's red,
black, red, black.

225
00:15:11,380 --> 00:15:14,220
So I'm going to number first --
I'm going to take all the reds

226
00:15:14,220 --> 00:15:21,855
first, so I'm changing this
number now to take all the reds

227
00:15:21,855 --> 00:15:31,400
-- one, two, three,
four, five, six, seven,

228
00:15:31,400 --> 00:15:37,250
eight and now all the
blacks -- nine, ten, eleven,

229
00:15:37,250 --> 00:15:38,920
twelve and so on.

230
00:15:38,920 --> 00:15:43,610
I'll stop there so your eye
will know how to fill in.

231
00:15:43,610 --> 00:15:45,720
What's the point of that?

232
00:15:45,720 --> 00:15:51,780
If I take a red node and look --
suppose this first one is a red

233
00:15:51,780 --> 00:15:58,480
node, as it is -- then what
do I know about the positions

234
00:15:58,480 --> 00:16:03,220
of those minus 1's, the
non-zeros in the matrix?

235
00:16:03,220 --> 00:16:08,350
They're associated with
black nodes, right?

236
00:16:08,350 --> 00:16:14,910
The neighbors of this
node number one are black.

237
00:16:14,910 --> 00:16:19,340
The black ones have numbers
from nine to sixteen.

238
00:16:19,340 --> 00:16:24,400
So then the black --
everybody sees it.

239
00:16:24,400 --> 00:16:32,160
With this molecule that has a
4 in the center and minus 1's

240
00:16:32,160 --> 00:16:36,510
beside it, those minus 1's
are associated with nodes

241
00:16:36,510 --> 00:16:39,080
of the opposite color.

242
00:16:39,080 --> 00:16:41,190
That's what we see here
in the actual numbering.

243
00:16:41,190 --> 00:16:42,780
So what does the matrix become?

244
00:16:42,780 --> 00:16:46,390
What does this -- if
I do that permutation,

245
00:16:46,390 --> 00:16:49,050
the permutation that's
associated with that

246
00:16:49,050 --> 00:16:53,150
renumbering -- what
will the same matrix --

247
00:16:53,150 --> 00:16:56,720
here's the same matrix, but now
I'm permuting it and so what

248
00:16:56,720 --> 00:16:57,600
does it look like?

249
00:16:57,600 --> 00:16:59,010
Maybe I can draw it smaller.

250
00:16:59,010 --> 00:17:00,330
I think I probably can.

251
00:17:03,580 --> 00:17:09,800
So in this pretty special
situation where the red nodes

252
00:17:09,800 --> 00:17:13,230
are not connected to any other
red nodes, only to themselves,

253
00:17:13,230 --> 00:17:18,730
I think I'll only see the
4I -- this is red to red.

254
00:17:18,730 --> 00:17:22,480
Those are the eight red nodes,
an eight by eight identity

255
00:17:22,480 --> 00:17:25,310
matrix times 4.

256
00:17:25,310 --> 00:17:33,160
The blacks are only connected to
the black and all the minus 1's

257
00:17:33,160 --> 00:17:38,100
are in our connections
between red and black,

258
00:17:38,100 --> 00:17:42,570
that is, this far off the
diagonal, what shall I call it?

259
00:17:42,570 --> 00:17:47,320
B I'll just pick a letter --
B and B transposed probably,

260
00:17:47,320 --> 00:17:51,120
because I know the matrix
is going to stay symmetric.

261
00:17:51,120 --> 00:17:53,600
If it started
symmetric, as it did,

262
00:17:53,600 --> 00:17:58,060
then this operation of
reordering rows and columns

263
00:17:58,060 --> 00:18:00,910
compatibly is going
to leave it symmetric

264
00:18:00,910 --> 00:18:03,670
and it must look
something like this.

265
00:18:03,670 --> 00:18:06,770
Is that matrix more
sparse than before?

266
00:18:06,770 --> 00:18:07,890
No.

267
00:18:07,890 --> 00:18:09,185
It's got the same minus 1's.

268
00:18:09,185 --> 00:18:14,100
They're just all pushed
up into this corner.

269
00:18:14,100 --> 00:18:18,180
Will it have some
fill-in as we go forward?

270
00:18:18,180 --> 00:18:20,410
What's the answer on that one?

271
00:18:20,410 --> 00:18:23,567
As I go forward,
maybe I should --

272
00:18:23,567 --> 00:18:25,150
I haven't really
thought that through.

273
00:18:25,150 --> 00:18:27,750
My instinct is yes.

274
00:18:27,750 --> 00:18:29,260
We'll have fill-in.

275
00:18:29,260 --> 00:18:34,840
Will we have more fill-in
or less fill-in than the row

276
00:18:34,840 --> 00:18:37,520
by row ordering?

277
00:18:37,520 --> 00:18:38,870
I should know the answer.

278
00:18:38,870 --> 00:18:44,470
I think I'd probably ask
the machine to tell me.

279
00:18:44,470 --> 00:18:51,820
My guess is probably
not more and we can --

280
00:18:51,820 --> 00:18:55,480
why do I think probably
this is better?

281
00:18:55,480 --> 00:18:58,610
Of course, I should
know exactly.

282
00:18:58,610 --> 00:19:06,300
In general, it's better to put
off the evil day -- that is,

283
00:19:06,300 --> 00:19:15,200
this goes through the first
half of elimination -- I mean,

284
00:19:15,200 --> 00:19:17,130
you know what I'm thinking here.

285
00:19:17,130 --> 00:19:22,240
All the off-diagonals
have been sort of

286
00:19:22,240 --> 00:19:23,420
pushed down in the matrix.

287
00:19:26,390 --> 00:19:31,490
Let me do an example that
shows why that's good.

288
00:19:31,490 --> 00:19:39,600
Suppose my matrix has a non-zero
diagonal -- and by the way,

289
00:19:39,600 --> 00:19:49,940
the first step in the good
codes for elimination do

290
00:19:49,940 --> 00:19:53,620
a permutation exactly to
produce a non-zero diagonal,

291
00:19:53,620 --> 00:19:56,240
so you're OK to start.

292
00:19:56,240 --> 00:19:58,290
You know where you are.

293
00:19:58,290 --> 00:20:04,720
Now suppose that the
non-zeros all show up --

294
00:20:04,720 --> 00:20:07,890
and I'll make the
matrix symmetric.

295
00:20:07,890 --> 00:20:11,380
That's the non-zero
pattern and my point

296
00:20:11,380 --> 00:20:15,120
is, that's a very
good non-zero pattern.

297
00:20:15,120 --> 00:20:15,990
Why is it good?

298
00:20:15,990 --> 00:20:18,910
It's good because there's
no fill-in whatever.

299
00:20:18,910 --> 00:20:20,830
When I subtract
some multiple, when

300
00:20:20,830 --> 00:20:25,160
I go to make this
0, to eliminate it,

301
00:20:25,160 --> 00:20:27,690
I subtract a multiple
using the first pivot.

302
00:20:27,690 --> 00:20:31,500
It changes that, but it
doesn't fill in anything new.

303
00:20:31,500 --> 00:20:36,800
The only operations required
are operating on the last row,

304
00:20:36,800 --> 00:20:38,890
which is already full.

305
00:20:38,890 --> 00:20:43,990
So there's an example in
which the bandwidth -- see,

306
00:20:43,990 --> 00:20:47,830
normally the bandwidth
is a good guide --

307
00:20:47,830 --> 00:20:51,370
but that's only for
pretty full bands.

308
00:20:51,370 --> 00:20:54,830
Here's a matrix that's
not banded, really.

309
00:20:54,830 --> 00:20:59,470
Its bandwidth is the full five.

310
00:20:59,470 --> 00:21:06,140
I have to go out five
columns until I get the last,

311
00:21:06,140 --> 00:21:09,430
until I get the non-zeros.

312
00:21:09,430 --> 00:21:11,110
So that is a big bandwidth.

313
00:21:11,110 --> 00:21:17,610
So bandwidth w, the
bandwidth w is large, but --

314
00:21:17,610 --> 00:21:20,330
it's the perfect
ordering -- but no fill.

315
00:21:24,330 --> 00:21:29,120
I don't fill anything -- you
realize that for banded matrix,

316
00:21:29,120 --> 00:21:32,890
we won't have fill
in outside the band.

317
00:21:32,890 --> 00:21:37,140
Our concern is for fill-in
inside the band and here,

318
00:21:37,140 --> 00:21:42,090
we don't have any, because
the bad row -- and actually,

319
00:21:42,090 --> 00:21:49,450
it often happens in some least
squares and other problem that

320
00:21:49,450 --> 00:21:53,760
you might -- this isn't quite
as stupid a matrix as you might

321
00:21:53,760 --> 00:21:54,940
think.

322
00:21:54,940 --> 00:21:58,750
These might be blocks
rather than single entries.

323
00:21:58,750 --> 00:22:04,790
You often want to think
block-wise about a matrix.

324
00:22:04,790 --> 00:22:07,660
You might think,
why would you have

325
00:22:07,660 --> 00:22:15,870
a matrix that was well separated
with block diagonal, which

326
00:22:15,870 --> 00:22:23,070
is really good, and then
a few rows that are full.

327
00:22:23,070 --> 00:22:26,980
That seems sort of unlikely, but
I guess I want to say it isn't.

328
00:22:26,980 --> 00:22:31,870
If you had a problem in
least squares minimizing

329
00:22:31,870 --> 00:22:35,860
some expression
involving that matrix,

330
00:22:35,860 --> 00:22:39,280
the block diagonal part,
but with the constraint

331
00:22:39,280 --> 00:22:45,350
that maybe the sum of all
the unknowns has to be 1,

332
00:22:45,350 --> 00:22:48,700
then when you use a
Lagrange multiplier to bring

333
00:22:48,700 --> 00:22:50,470
that constraint
into the problem,

334
00:22:50,470 --> 00:22:55,450
into the matrix, that's
going to show like that.

335
00:22:55,450 --> 00:22:59,280
And the beauty of the
Lagrange multiplier device

336
00:22:59,280 --> 00:23:06,180
is that, also, that it'll
produce a symmetric column

337
00:23:06,180 --> 00:23:07,650
there.

338
00:23:07,650 --> 00:23:13,090
We'll see that in the third part
of the course, but I'm just --

339
00:23:13,090 --> 00:23:19,680
my point is that this example is
not as artificial as it seems.

340
00:23:19,680 --> 00:23:28,980
Now suppose we had taken
the bandwidth as our metric,

341
00:23:28,980 --> 00:23:29,890
as our goal.

342
00:23:29,890 --> 00:23:33,770
In reducing the
bandwidth as our goal,

343
00:23:33,770 --> 00:23:37,410
we would have made
a bad mistake.

344
00:23:37,410 --> 00:23:41,650
If I wanted to do a permutation
that reduces the bandwidth,

345
00:23:41,650 --> 00:23:44,120
I better put this up in
the middle somewhere.

346
00:23:44,120 --> 00:23:49,540
So this is -- that bad row
comes up in the middle and then,

347
00:23:49,540 --> 00:23:56,440
I guess, these are
maybe whatever --

348
00:23:56,440 --> 00:24:02,030
I guess I need
three X's down here.

349
00:24:02,030 --> 00:24:04,480
I'm going to do the same
permutation from the right,

350
00:24:04,480 --> 00:24:09,030
so that will move this
column over to the middle.

351
00:24:09,030 --> 00:24:14,010
This would be a reordered --
this would be a P matrix P

352
00:24:14,010 --> 00:24:17,580
transposed thing, I think.

353
00:24:17,580 --> 00:24:19,780
It's got a better
bandwidth, right?

354
00:24:19,780 --> 00:24:23,350
Now we have -- we're totally
confident of all these zeros

355
00:24:23,350 --> 00:24:26,910
here.

356
00:24:26,910 --> 00:24:30,020
But it will fill in.

357
00:24:30,020 --> 00:24:32,720
Where will we get
fill-in, in that matrix?

358
00:24:32,720 --> 00:24:37,200
So as we start, we're using
elimination to remove those

359
00:24:37,200 --> 00:24:41,410
entries, no problem, because
that upper left corner is

360
00:24:41,410 --> 00:24:43,940
the way this big,
the whole matrix was,

361
00:24:43,940 --> 00:24:51,260
no fill-in up to there, but just
as soon as we reach this one

362
00:24:51,260 --> 00:24:56,280
as our pivot row and use it to
subtract multiples to knock out

363
00:24:56,280 --> 00:24:59,630
those, when I make those
0 by using that pivot,

364
00:24:59,630 --> 00:25:04,500
it's going to bring the whole --
this will be gone at that time,

365
00:25:04,500 --> 00:25:11,610
but this will be non-zero and
that will produce new edges

366
00:25:11,610 --> 00:25:12,510
in the graph.

367
00:25:12,510 --> 00:25:14,720
Might be fun to look
at the graph that

368
00:25:14,720 --> 00:25:19,660
is associated with this
matrix and its permutations.

369
00:25:19,660 --> 00:25:23,187
But anyway, you see that
this will all fill-in.

370
00:25:27,820 --> 00:25:30,630
So again, in that
small problem, we

371
00:25:30,630 --> 00:25:33,500
could really see
what was happening.

372
00:25:33,500 --> 00:25:36,510
In the big problem,
we can almost

373
00:25:36,510 --> 00:25:40,550
see it when we take a very
specific, highly structured

374
00:25:40,550 --> 00:25:47,540
permutation like red-black,
although I haven't pinned down

375
00:25:47,540 --> 00:25:52,030
exactly what the
count is like now.

376
00:25:52,030 --> 00:25:57,030
I'd be glad if somebody would
just run a computer experiment,

377
00:25:57,030 --> 00:26:01,640
count non-zeros, count -- yeah.

378
00:26:01,640 --> 00:26:03,770
So what's the good
thing to count?

379
00:26:06,940 --> 00:26:09,160
How do you see what
the fill-in finally

380
00:26:09,160 --> 00:26:12,190
was when you've
done an elimination?

381
00:26:12,190 --> 00:26:18,240
I guess that the L
and U factors, which

382
00:26:18,240 --> 00:26:20,840
is what elimination
produces, that's

383
00:26:20,840 --> 00:26:22,300
where the fill-in will be.

384
00:26:22,300 --> 00:26:26,220
So maybe I just make
that point, so that you

385
00:26:26,220 --> 00:26:28,300
could make these experiments
that I have in mind

386
00:26:28,300 --> 00:26:30,730
and tell me what I
ought to know about it.

387
00:26:30,730 --> 00:26:37,580
So if I have the matrix
K, whichever it is --

388
00:26:37,580 --> 00:26:42,580
whether it's this one or
this one or any other,

389
00:26:42,580 --> 00:26:48,720
then the effect of elimination
is to factor it into a lower-

390
00:26:48,720 --> 00:26:50,400
times an
upper-triangular matrix.

391
00:26:54,760 --> 00:26:59,610
I'll just mention a
special case while I'm --

392
00:26:59,610 --> 00:27:04,290
special positive
definite case --

393
00:27:04,290 --> 00:27:08,680
we can adjust it so that's
K is L times L transposed.

394
00:27:08,680 --> 00:27:09,950
This is called Cholesky.

395
00:27:16,060 --> 00:27:21,070
We can arrange the
-- it will happen.

396
00:27:21,070 --> 00:27:28,090
We can arrange the scaling so
that by putting the square root

397
00:27:28,090 --> 00:27:32,810
of pivots with L and the other
square root with L transpose --

398
00:27:32,810 --> 00:27:38,720
so Cholesky is a MATLAB
command, C-H-O-L, and actually,

399
00:27:38,720 --> 00:27:46,000
MATLAB -- if you give
MATLAB a symmetric matrix --

400
00:27:46,000 --> 00:27:49,550
if you give MATLAB a matrix to
work with, it will first ask --

401
00:27:49,550 --> 00:27:51,830
it will ask, is it symmetric?

402
00:27:51,830 --> 00:27:57,250
If it is symmetric, it will hope
that it's positive definite,

403
00:27:57,250 --> 00:27:58,830
so it will begin Cholesky.

404
00:28:02,040 --> 00:28:06,820
It'll begin to look for this --
it'll begin the factorization,

405
00:28:06,820 --> 00:28:11,840
which is perfect for
positive definite ones,

406
00:28:11,840 --> 00:28:19,620
until it succeeds and finishes
Cholesky or until it discovers

407
00:28:19,620 --> 00:28:28,180
that nope, a negative pivot
appeared, which signals,

408
00:28:28,180 --> 00:28:32,080
that the matrix is not and
was not originally positive

409
00:28:32,080 --> 00:28:38,830
definite and then MATLAB
has to accept, OK,

410
00:28:38,830 --> 00:28:41,780
now I may have to
do row exchanges,

411
00:28:41,780 --> 00:28:44,570
pivoting operations
and all the rest.

412
00:28:44,570 --> 00:28:47,910
So MATLAB's sort of
optimistic until it --

413
00:28:47,910 --> 00:28:51,640
on a symmetric matrix
until it is told otherwise.

414
00:28:51,640 --> 00:28:53,470
But not all matrices
are symmetric,

415
00:28:53,470 --> 00:28:59,850
so this would be the general
case, almost the general case.

416
00:28:59,850 --> 00:29:04,250
I haven't included
here the pivoting,

417
00:29:04,250 --> 00:29:06,880
the row exchanges
that MATLAB does

418
00:29:06,880 --> 00:29:10,310
as it executes elimination.

419
00:29:10,310 --> 00:29:13,360
So let me just finish
my sentence on L

420
00:29:13,360 --> 00:29:20,070
and U. L will have
the fill in, in it.

421
00:29:20,070 --> 00:29:22,800
L will tell us -- and
U will, certainly.

422
00:29:22,800 --> 00:29:25,440
I mean, they both will.

423
00:29:25,440 --> 00:29:35,440
U will have these -- that guy is
going to be part of U and this

424
00:29:35,440 --> 00:29:42,220
one is -- I expect to
see something there in L.

425
00:29:42,220 --> 00:29:45,610
So L and U will actually
exhibit the fill-in,

426
00:29:45,610 --> 00:29:49,930
so that you could find out how
much fill in there is simply

427
00:29:49,930 --> 00:29:57,300
by counting the non-zeros
in the matrices L and U.

428
00:29:57,300 --> 00:30:00,670
This is to encourage you,
because this is really

429
00:30:00,670 --> 00:30:03,170
a pretty experimental subject.

430
00:30:03,170 --> 00:30:11,030
Nobody -- the best
methods, which --

431
00:30:11,030 --> 00:30:12,800
or the favorite method.

432
00:30:12,800 --> 00:30:17,140
I think nobody can say best,
but this approximate minimum

433
00:30:17,140 --> 00:30:25,050
degree, the favorite method,
we could prove some theorem

434
00:30:25,050 --> 00:30:27,510
to show that it was better
than something else,

435
00:30:27,510 --> 00:30:31,660
but to really see,
is it successful

436
00:30:31,660 --> 00:30:34,510
or not on a wide
variety of problems,

437
00:30:34,510 --> 00:30:36,280
that's really experimental.

438
00:30:36,280 --> 00:30:38,080
Just try it.

439
00:30:38,080 --> 00:30:41,490
Now what does it mean?

440
00:30:41,490 --> 00:30:43,140
So I looked at
that specific one.

441
00:30:43,140 --> 00:30:47,200
Now I'm ready to look at the
approximate minimum degree,

442
00:30:47,200 --> 00:30:51,710
and I'll contrast it with
the perfect minimum degree.

443
00:30:55,220 --> 00:31:00,130
There are codes for that
and those A's become M's.

444
00:31:00,130 --> 00:31:04,280
But it turns out -- and I don't
know exactly why it's the case

445
00:31:04,280 --> 00:31:09,670
-- that the approximate minimum
degree usually, if anything,

446
00:31:09,670 --> 00:31:11,260
it's superior.

447
00:31:11,260 --> 00:31:13,820
So these are the ones to use.

448
00:31:13,820 --> 00:31:16,750
So what does
minimum degree mean?

449
00:31:16,750 --> 00:31:19,500
Minimum degree means
-- let's see --

450
00:31:19,500 --> 00:31:23,980
if I took this problem -- I
should draw the graph for this

451
00:31:23,980 --> 00:31:24,500
matrix.

452
00:31:24,500 --> 00:31:27,910
So what does the graph
look like for this matrix?

453
00:31:27,910 --> 00:31:34,480
I've got six nodes -- let
me put six nodes in --

454
00:31:34,480 --> 00:31:39,670
let me put five and the node in
the middle will be the one that

455
00:31:39,670 --> 00:31:42,220
they're all connected to.

456
00:31:42,220 --> 00:31:45,050
In this ordering, those would
be one, two, three, four,

457
00:31:45,050 --> 00:31:50,780
five and six -- and all the
connections go into six.

458
00:31:50,780 --> 00:31:53,180
So that's the graph
for that matrix.

459
00:31:56,880 --> 00:32:02,450
The graph for this one -- so you
probably see it better than I

460
00:32:02,450 --> 00:32:03,740
do.

461
00:32:03,740 --> 00:32:08,670
What I had originally there --
I guess this is the graph with

462
00:32:08,670 --> 00:32:13,550
the numbering one, two,
three, four, five, six --

463
00:32:13,550 --> 00:32:18,510
what was there before
the fill-in happened?

464
00:32:18,510 --> 00:32:20,630
The graph would look
the same, but the nodes

465
00:32:20,630 --> 00:32:23,160
would have a different
numbering, right?

466
00:32:23,160 --> 00:32:29,350
One, two, three,
four, five, six --

467
00:32:29,350 --> 00:32:32,120
so shall I just put
the bad guy here?

468
00:32:32,120 --> 00:32:36,460
One, two, three, four, five,
six and then what's the point?

469
00:32:36,460 --> 00:32:45,000
As soon as I use number
three to eliminate four --

470
00:32:45,000 --> 00:32:50,190
three was connected to five, so
a new edge is going to appear

471
00:32:50,190 --> 00:32:51,000
there.

472
00:32:51,000 --> 00:32:54,630
Three was connected to six,
three is connected to four.

473
00:32:54,630 --> 00:32:58,170
When I use it, a new four,
six is going to appear.

474
00:32:58,170 --> 00:33:00,360
So those are a couple
of the fill-ins.

475
00:33:00,360 --> 00:33:02,930
Those are the two
fill ins in row four.

476
00:33:02,930 --> 00:33:09,630
Then when I go onto row
five -- I'll fill in there.

477
00:33:09,630 --> 00:33:14,250
So those are the three
below the diagonal

478
00:33:14,250 --> 00:33:17,180
and above the diagonal,
non-zeros that come

479
00:33:17,180 --> 00:33:19,840
from the bad ordering, I think.

480
00:33:19,840 --> 00:33:23,390
I didn't prepare that,
so I hope it's right.

481
00:33:23,390 --> 00:33:26,510
Yeah

482
00:33:26,510 --> 00:33:28,520
So what's minimum degree?

483
00:33:28,520 --> 00:33:35,450
Minimum degree is simply sort
of a greedy type algorithm.

484
00:33:35,450 --> 00:33:38,520
We have no way to
look far, far ahead

485
00:33:38,520 --> 00:33:43,550
and see what non-zeros are
coming way, way down the line,

486
00:33:43,550 --> 00:33:48,370
so let's just take
it where we are.

487
00:33:48,370 --> 00:33:50,480
We have to choose what
should be the next pivot.

488
00:33:54,200 --> 00:33:56,670
Let's say we take
that first pivot.

489
00:33:56,670 --> 00:33:57,170
Good.

490
00:34:02,090 --> 00:34:06,900
So what are the
degrees of these ?

491
00:34:06,900 --> 00:34:10,290
Let me redraw the graph
before the fill-in.

492
00:34:10,290 --> 00:34:12,780
One, two, three,
four, five, six.

493
00:34:12,780 --> 00:34:16,770
So I've got six nodes.

494
00:34:16,770 --> 00:34:19,080
That has a degree 1.

495
00:34:19,080 --> 00:34:21,590
It's connected only
to one other node.

496
00:34:21,590 --> 00:34:24,150
Actually, all these
have degree one

497
00:34:24,150 --> 00:34:26,080
and this one has degree five.

498
00:34:26,080 --> 00:34:27,680
Five edges are going out.

499
00:34:27,680 --> 00:34:33,360
So the whole point of minimum
degree is, don't pick that one.

500
00:34:33,360 --> 00:34:37,130
At each step, create
the new graph.

501
00:34:37,130 --> 00:34:40,660
The graph expresses a fill-in --
and we're not going to be able

502
00:34:40,660 --> 00:34:45,170
to avoid some fill-in,
normally -- but at every point,

503
00:34:45,170 --> 00:34:51,760
take the graph as it is and
choose as the next node to work

504
00:34:51,760 --> 00:34:56,990
on -- that means the next
row to use in elimination,

505
00:34:56,990 --> 00:35:01,760
the next pivot to choose --
choose one that has smallest

506
00:35:01,760 --> 00:35:02,960
degree.

507
00:35:02,960 --> 00:35:04,880
Don't choose this one.

508
00:35:04,880 --> 00:35:10,090
So for this very
special case, we

509
00:35:10,090 --> 00:35:12,760
have five choices
for what comes first.

510
00:35:12,760 --> 00:35:15,880
We make a choice like that one.

511
00:35:15,880 --> 00:35:17,660
We do elimination.

512
00:35:17,660 --> 00:35:20,150
In this case, it doesn't
fill in anything.

513
00:35:20,150 --> 00:35:23,250
Now I'm looking at
the rest of the graph.

514
00:35:23,250 --> 00:35:27,780
I have four nodes there with
degree only 1 and a node

515
00:35:27,780 --> 00:35:30,860
there with high degree,
so I don't choose that.

516
00:35:30,860 --> 00:35:34,540
So I choose 2 and so on.

517
00:35:34,540 --> 00:35:38,220
So of course, minimum
degree, approximate

518
00:35:38,220 --> 00:35:42,650
or perfect or any
dumb idea is going

519
00:35:42,650 --> 00:35:47,880
to create the right ordering
that produces this matrix.

520
00:35:50,660 --> 00:35:53,680
I hope you see what
minimum degree is.

521
00:35:53,680 --> 00:35:59,910
At every stage, you look down
the column, just as here.

522
00:35:59,910 --> 00:36:03,060
Actually, what would
minimum degree give here?

523
00:36:03,060 --> 00:36:05,160
I would love to have
a complete picture.

524
00:36:05,160 --> 00:36:08,590
I think we could at
least see at the start.

525
00:36:08,590 --> 00:36:12,590
Let me erase this
red-black ordering

526
00:36:12,590 --> 00:36:19,092
and prepare to write in a
minimum degree ordering.

527
00:36:19,092 --> 00:36:24,630
If I can re-complete that graph.

528
00:36:24,630 --> 00:36:26,100
So what node do I take first?

529
00:36:29,280 --> 00:36:36,000
Remember, the graph really
does not have these pieces.

530
00:36:36,000 --> 00:36:37,320
I might as well erase them now.

531
00:36:37,320 --> 00:36:40,710
Shall I just erase -- these are
pieces that connect to known

532
00:36:40,710 --> 00:36:42,780
values that are on
the right hand side.

533
00:36:42,780 --> 00:36:46,050
So I think the graph doesn't
have those connections,

534
00:36:46,050 --> 00:36:48,650
those connections -- it's going
to look pretty much as it did

535
00:36:48,650 --> 00:36:54,850
before, but I think now we
have the graph of actual 16,

536
00:36:54,850 --> 00:36:58,490
the graph that we
should be looking for.

537
00:36:58,490 --> 00:37:01,080
That's the graph of
our matrix, the graph

538
00:37:01,080 --> 00:37:03,890
of the nonzero
structure in the matrix.

539
00:37:03,890 --> 00:37:08,860
So what do you take
as the first node?

540
00:37:08,860 --> 00:37:10,430
You take the corners, right?

541
00:37:10,430 --> 00:37:14,280
The corners are going to win,
because they have degree 2.

542
00:37:14,280 --> 00:37:15,800
These guys have degree 3.

543
00:37:15,800 --> 00:37:19,529
There's a degree 2, so I
might take that -- let's see,

544
00:37:19,529 --> 00:37:20,070
what happens?

545
00:37:20,070 --> 00:37:26,810
When I take this one,
then I have to see,

546
00:37:26,810 --> 00:37:34,530
when I use that as the pivot
-- exactly what I did here --

547
00:37:34,530 --> 00:37:37,090
what's going to fill in?

548
00:37:37,090 --> 00:37:40,600
I guess explicitly I know
that two new edges fill in

549
00:37:40,600 --> 00:37:41,350
and what are they?

550
00:37:43,960 --> 00:37:45,580
Maybe they're the same.

551
00:37:45,580 --> 00:37:46,870
I guess they are.

552
00:37:46,870 --> 00:37:49,460
This is edge number
-- on the graph,

553
00:37:49,460 --> 00:37:52,650
it's probably easier to
see than in the matrix.

554
00:37:52,650 --> 00:37:56,400
So I use this guy.

555
00:37:56,400 --> 00:37:59,710
By elimination, I'm
removing the connections,

556
00:37:59,710 --> 00:38:04,800
so it maybe sits
there, surviving

557
00:38:04,800 --> 00:38:08,980
to remind me which
node was numbered one,

558
00:38:08,980 --> 00:38:13,300
but these connections have
been removed by elimination.

559
00:38:13,300 --> 00:38:17,950
That entry became a 0 and
that entry became a 0,

560
00:38:17,950 --> 00:38:23,670
but there was a
fill in here, right?

561
00:38:23,670 --> 00:38:27,120
That's the first step
of minimum degree.

562
00:38:27,120 --> 00:38:32,510
The second step, as
you say, I'm going

563
00:38:32,510 --> 00:38:35,060
to do all the
corners first, right?

564
00:38:35,060 --> 00:38:39,090
Because that has degree 2 and
these still have degree 3.

565
00:38:39,090 --> 00:38:40,450
No improvement.

566
00:38:40,450 --> 00:38:42,210
It's kind of fun to do.

567
00:38:42,210 --> 00:38:45,740
Two and that filled in.

568
00:38:45,740 --> 00:38:48,900
Number three and that filled in.

569
00:38:48,900 --> 00:38:53,120
Number four and that filled in.

570
00:38:53,120 --> 00:38:55,960
What next?

571
00:38:55,960 --> 00:38:59,110
There are going to be
a bunch of choices now.

572
00:38:59,110 --> 00:39:04,160
As far as I know --
I could be wrong --

573
00:39:04,160 --> 00:39:07,390
I don't know whether the --
I don't know what the choice

574
00:39:07,390 --> 00:39:11,320
decision -- how the decision
is made when there is a choice,

575
00:39:11,320 --> 00:39:15,300
when I have one, two, three,
four, five, six, seven,

576
00:39:15,300 --> 00:39:19,230
eight nodes, all with degree 3.

577
00:39:19,230 --> 00:39:21,230
Let me just take this one.

578
00:39:21,230 --> 00:39:25,420
So I'll take this as the next
one, as number five, I guess.

579
00:39:25,420 --> 00:39:26,960
That'll be number five.

580
00:39:26,960 --> 00:39:30,370
That will kill all
those connections

581
00:39:30,370 --> 00:39:34,010
and just leave the node,
but it will put in that one.

582
00:39:34,010 --> 00:39:34,870
Any others?

583
00:39:34,870 --> 00:39:38,160
Maybe only that one.

584
00:39:38,160 --> 00:39:39,040
Did I do that right?

585
00:39:43,247 --> 00:39:44,830
I mean, it would
have put in this one,

586
00:39:44,830 --> 00:39:47,590
because those are both
connected to this,

587
00:39:47,590 --> 00:39:53,950
but that edge is already
in there, so nothing new.

588
00:39:53,950 --> 00:39:58,720
Actually, here I've created
a node that is also degree 3.

589
00:39:58,720 --> 00:40:02,860
So I'm going to have a
lot of degree 3's here.

590
00:40:02,860 --> 00:40:08,270
But then, after a while, I think
when I get into the center --

591
00:40:08,270 --> 00:40:11,880
I'm guessing that the
degree will go up.

592
00:40:11,880 --> 00:40:14,750
I don't think I get away
with degree 3 all the way

593
00:40:14,750 --> 00:40:19,190
until the finish, I don't think.

594
00:40:19,190 --> 00:40:25,610
Anyway, you see the
point that there

595
00:40:25,610 --> 00:40:32,600
are choices to make, among
nodes with equal degree.

596
00:40:32,600 --> 00:40:36,190
Then there's this calculation
to make -- maybe I should --

597
00:40:36,190 --> 00:40:45,320
this is a moment to just
mention, if I can do it,

598
00:40:45,320 --> 00:40:51,960
how you might keep -- how
you might organize this pass

599
00:40:51,960 --> 00:40:55,410
through -- so this pass
through the matrix --

600
00:40:55,410 --> 00:40:57,550
it's really a just a
pass through the graph.

601
00:40:57,550 --> 00:41:00,370
It's not looking at
the actual numbers.

602
00:41:00,370 --> 00:41:04,400
It's just choosing the
ordering and then the next pass

603
00:41:04,400 --> 00:41:07,920
through we follow that
ordering with the numbers,

604
00:41:07,920 --> 00:41:10,870
finding the multipliers
that go into L

605
00:41:10,870 --> 00:41:14,870
and the numbers that
are left in matrix U.

606
00:41:14,870 --> 00:41:21,940
So how would I -- how would
anybody reasonably write down

607
00:41:21,940 --> 00:41:22,810
the original list?

608
00:41:22,810 --> 00:41:26,090
So I have the nodes numbered
one, two, three, four --

609
00:41:26,090 --> 00:41:28,640
up to sixteen.

610
00:41:28,640 --> 00:41:31,000
Those are the node numbers.

611
00:41:31,000 --> 00:41:34,730
Now with node one,
I'll list all the --

612
00:41:34,730 --> 00:41:38,410
so this corresponds to
row one of the matrix,

613
00:41:38,410 --> 00:41:42,370
so I'll list all the nodes
that are connected to row one,

614
00:41:42,370 --> 00:41:44,240
to node one.

615
00:41:44,240 --> 00:41:47,240
Node one itself is -- I don't
know whether to put that

616
00:41:47,240 --> 00:41:48,790
in or not.

617
00:41:48,790 --> 00:41:50,520
Maybe I will.

618
00:41:50,520 --> 00:41:55,200
Number two is connected to
it and number five above it.

619
00:41:55,200 --> 00:41:59,160
I'm taking the
original row ordering.

620
00:41:59,160 --> 00:42:02,570
Then when I look at node two
-- this is going to be a long

621
00:42:02,570 --> 00:42:09,580
list, but I'll have to have
a break there to tell me that

622
00:42:09,580 --> 00:42:13,010
these numbers were
associated with row one.

623
00:42:13,010 --> 00:42:16,570
Now I want to know those
associated with row two.

624
00:42:16,570 --> 00:42:21,260
So row two had the
connection back to one,

625
00:42:21,260 --> 00:42:24,250
so this little group is going
to be all the things connected

626
00:42:24,250 --> 00:42:25,760
to row two.

627
00:42:25,760 --> 00:42:29,050
Remember -- sorry, since I've
erased all those connections,

628
00:42:29,050 --> 00:42:31,670
but you remember them --
two was connected to one,

629
00:42:31,670 --> 00:42:36,360
it's connected forward to three
and above it to probably six.

630
00:42:36,360 --> 00:42:38,640
Maybe that's one,
three, and six.

631
00:42:38,640 --> 00:42:41,580
And then a slash, and then
whatever node three --

632
00:42:41,580 --> 00:42:50,800
so I need -- I guess I need one
more line there to tell me --

633
00:42:50,800 --> 00:42:52,520
I need a pointer.

634
00:42:52,520 --> 00:42:57,530
I need a line, a pointer
that tells me, number one,

635
00:42:57,530 --> 00:43:00,620
start with the first
one -- one, two, three.

636
00:43:00,620 --> 00:43:02,860
Number two, start
with the fourth one.

637
00:43:02,860 --> 00:43:04,840
Number three, start
with the seventh one.

638
00:43:04,840 --> 00:43:09,430
So that number seven is
pointing to the beginning

639
00:43:09,430 --> 00:43:16,820
of the sub-list, the
beginning of the short list

640
00:43:16,820 --> 00:43:21,100
that's associated to
each separate node.

641
00:43:23,790 --> 00:43:27,380
I guess what I'm
saying is that this

642
00:43:27,380 --> 00:43:31,700
is a pretty compact
description of the graph.

643
00:43:31,700 --> 00:43:34,720
This will tell me all
the edges in the graph,

644
00:43:34,720 --> 00:43:37,440
It tells me what
two is connected to

645
00:43:37,440 --> 00:43:44,900
and this tells me where
that little set lies.

646
00:43:44,900 --> 00:43:50,780
Now if I, as I will,
change the ordering,

647
00:43:50,780 --> 00:43:54,490
then I just reorganize
the pointers.

648
00:43:54,490 --> 00:43:56,320
I'm going to speak
roughly and not

649
00:43:56,320 --> 00:44:01,330
the details, just in
general, in the large,

650
00:44:01,330 --> 00:44:09,790
you could imagine that this is
a pretty convenient structure

651
00:44:09,790 --> 00:44:16,770
to use for keeping the record
of what the node numbering is.

652
00:44:20,240 --> 00:44:24,910
The command nnz is a useful one.

653
00:44:24,910 --> 00:44:32,200
That's a MATLAB command that
counts the number of non-zeros.

654
00:44:35,490 --> 00:44:40,910
So n, n and z suggest
exactly what that will do.

655
00:44:40,910 --> 00:44:47,680
So if you pick an ordering,
like approximate minimum degree,

656
00:44:47,680 --> 00:44:52,164
and you pick a matrix, like
this one, and you run --

657
00:44:52,164 --> 00:44:53,580
I've been very
interested in that.

658
00:44:59,320 --> 00:45:03,190
You take the matrix, K2D,
you take the permutation

659
00:45:03,190 --> 00:45:06,450
that comes from approximate
minimum degree, that gives you

660
00:45:06,450 --> 00:45:10,240
this reordered matrix.

661
00:45:10,240 --> 00:45:14,870
We then do elimination, which
factors it in L times U --

662
00:45:14,870 --> 00:45:17,250
probably L times L transposed,
because it's a positive

663
00:45:17,250 --> 00:45:18,530
definite.

664
00:45:18,530 --> 00:45:27,310
Then I would do nnz of L
as a measure of fill-in.

665
00:45:27,310 --> 00:45:30,080
That would count
the fill-in in L,

666
00:45:30,080 --> 00:45:32,900
and with symmetry,
it'll be the same in u.

667
00:45:36,600 --> 00:45:39,410
If you have a better idea than
approximate minimum degree,

668
00:45:39,410 --> 00:45:46,510
that's a measure to
use as the criterion.

669
00:45:46,510 --> 00:45:49,520
I've just a little
time to speak, finally,

670
00:45:49,520 --> 00:45:52,310
about this idea of
graph separators.

671
00:45:52,310 --> 00:45:55,700
So again, I have to
get back to a graph.

672
00:45:55,700 --> 00:45:56,850
Let me draw a graph under.

673
00:45:56,850 --> 00:45:59,130
So now I'm ready for this one.

674
00:45:59,130 --> 00:46:00,830
I'll put a little
star next to it,

675
00:46:00,830 --> 00:46:04,790
to say that it didn't
look for a long time is

676
00:46:04,790 --> 00:46:09,990
if it could compete
with number two,

677
00:46:09,990 --> 00:46:13,800
because it'll be a
different ordering.

678
00:46:13,800 --> 00:46:16,450
You could, in a way -- it's
going to be a block minimum

679
00:46:16,450 --> 00:46:17,710
degree order.

680
00:46:17,710 --> 00:46:20,680
Let me say what it is
and as I draw the graph,

681
00:46:20,680 --> 00:46:23,220
let me say when it might win.

682
00:46:23,220 --> 00:46:29,620
It might win when the
size goes way up, 3D.

683
00:46:29,620 --> 00:46:33,920
In 2D, minimum degree is
-- on sparse matrices,

684
00:46:33,920 --> 00:46:37,380
is quite successful.

685
00:46:37,380 --> 00:46:44,130
So in two dimensions, until
the size really gets big,

686
00:46:44,130 --> 00:46:45,670
that's going to be fine.

687
00:46:45,670 --> 00:46:49,150
In three dimensions -- I'm
still going to draw only two,

688
00:46:49,150 --> 00:46:54,110
but in three dimensions or
very, very large matrices,

689
00:46:54,110 --> 00:46:58,000
I want to think about
this graph separator.

690
00:46:58,000 --> 00:47:03,020
So let me -- I'll make it
just so I really separate it

691
00:47:03,020 --> 00:47:14,190
perfectly -- let me
make the number --

692
00:47:14,190 --> 00:47:19,610
so when you look at that
graph, you might say, if I --

693
00:47:19,610 --> 00:47:21,190
look at that set of nodes.

694
00:47:21,190 --> 00:47:25,920
So a separator is
a set of nodes that

695
00:47:25,920 --> 00:47:29,090
separates the rest of the
nodes into two groups.

696
00:47:29,090 --> 00:47:33,420
So I have a group of nodes
P, a group of nodes Q

697
00:47:33,420 --> 00:47:35,710
and nothing, no
connections between P

698
00:47:35,710 --> 00:47:39,220
and Q. P is this part.

699
00:47:39,220 --> 00:47:42,260
Q is a similar part over there.

700
00:47:42,260 --> 00:47:49,320
S, the separator, is the
part with the connection.

701
00:47:49,320 --> 00:47:52,510
So P is connected
to Q only through S.

702
00:47:52,510 --> 00:47:56,200
So now if I think of ordering,
in what order should I

703
00:47:56,200 --> 00:48:00,740
take the P, the Q and the S?

704
00:48:00,740 --> 00:48:05,840
Our absolute rule is,
everything's connected to S,

705
00:48:05,840 --> 00:48:07,630
put it last.

706
00:48:07,630 --> 00:48:11,860
So you see that this is the
block form of my small example

707
00:48:11,860 --> 00:48:13,720
here.

708
00:48:13,720 --> 00:48:16,210
I didn't particularly
have a P or Q,

709
00:48:16,210 --> 00:48:19,620
but I certainly
wanted to put S last.

710
00:48:19,620 --> 00:48:21,260
Maybe I did.

711
00:48:21,260 --> 00:48:26,690
So maybe the separator here
was -- so what's the separator?

712
00:48:26,690 --> 00:48:30,970
One possible separator
would be like that, as S,

713
00:48:30,970 --> 00:48:36,210
P for these two nodes,
Q for these three nodes,

714
00:48:36,210 --> 00:48:39,980
and the rule would be -- I
haven't said what I'll do

715
00:48:39,980 --> 00:48:43,410
within P or within
Q or within S,

716
00:48:43,410 --> 00:48:46,670
but the rule would
be put Q last.

717
00:48:46,670 --> 00:48:49,830
Of course, that was exactly
the right thing to do.

718
00:48:49,830 --> 00:48:53,920
We're following the same
rule, just block-wise.

719
00:48:53,920 --> 00:48:56,100
So we have the --
so our matrix --

720
00:48:56,100 --> 00:49:00,910
what does our matrix look like
when you choose this ordering?

721
00:49:00,910 --> 00:49:06,710
You have whatever P is,
the block of all these P's.

722
00:49:06,710 --> 00:49:08,870
You have the block of
all these Q's, which

723
00:49:08,870 --> 00:49:13,410
might be connected to each
other and are, but not to P.

724
00:49:13,410 --> 00:49:16,570
So the main point is,
you have these 0 blocks.

725
00:49:16,570 --> 00:49:21,120
And then everything
is connected to S

726
00:49:21,120 --> 00:49:22,850
and S is connected to itself.

727
00:49:22,850 --> 00:49:27,360
So all you've achieved, but
it could be considerable,

728
00:49:27,360 --> 00:49:32,840
is to take an ordering
in which P was --

729
00:49:32,840 --> 00:49:36,590
since P is not connected to
Q, that produced these large 0

730
00:49:36,590 --> 00:49:40,920
blocks and of course, they'll
stay 0 as elimination goes

731
00:49:40,920 --> 00:49:42,980
forward.

732
00:49:42,980 --> 00:49:47,250
those 0's there will -- I
don't have to do any pivoting

733
00:49:47,250 --> 00:49:50,680
operations, any row
operations to make those 0.

734
00:49:50,680 --> 00:49:54,590
They're already 0 and
so the only operations

735
00:49:54,590 --> 00:50:00,110
will be P onto S, Q
onto S, S onto itself.

736
00:50:03,860 --> 00:50:08,140
In many problems, it's
the S onto itself part --

737
00:50:08,140 --> 00:50:12,650
so what will happen then,
when I do elimination on S?

738
00:50:12,650 --> 00:50:19,440
When I do elimination -- I'll
have whatever work is necessary

739
00:50:19,440 --> 00:50:25,030
within P, so I'll be factoring
-- can I call that P,

740
00:50:25,030 --> 00:50:26,970
that part of the matrix?

741
00:50:26,970 --> 00:50:29,360
It's an abuse of notation.

742
00:50:29,360 --> 00:50:31,560
This is the
connections of S to P.

743
00:50:31,560 --> 00:50:36,370
This is the connections of
S to Q, S to itself, Q to S,

744
00:50:36,370 --> 00:50:38,040
P to S -- probably symmetric.

745
00:50:41,130 --> 00:50:45,340
So when I do elimination --
if I don't, if I accept this

746
00:50:45,340 --> 00:50:49,850
ordering -- if I accept this
ordering and just go for it,

747
00:50:49,850 --> 00:50:52,550
elimination will
factor P into L*U --

748
00:50:52,550 --> 00:50:56,060
because it'll work up in the
corner as it always does --

749
00:50:56,060 --> 00:51:00,920
then there's no connection here,
so it'll factor Q into its own

750
00:51:00,920 --> 00:51:05,600
LU, and of course, all that --
all this stuff down here will

751
00:51:05,600 --> 00:51:11,780
change, but effectively these
will become 0's, because then

752
00:51:11,780 --> 00:51:17,940
there's a serious elimination,
P onto S, will make that 0.

753
00:51:17,940 --> 00:51:20,520
Q onto S -- Q will make that 0.

754
00:51:20,520 --> 00:51:24,120
This will be some
-- I'll call it SSS.

755
00:51:24,120 --> 00:51:29,800
It's a much fuller matrix,
block -- smaller though,

756
00:51:29,800 --> 00:51:34,600
hopefully smaller
-- but pretty full.

757
00:51:34,600 --> 00:51:39,890
So actually, in the graph
separator nested dissection

758
00:51:39,890 --> 00:51:47,300
world, most of the time is
spent on that third block,

759
00:51:47,300 --> 00:51:49,220
just working with
it, because it's

760
00:51:49,220 --> 00:51:52,700
the one that's pretty full.

761
00:51:52,700 --> 00:51:56,650
So how do graph separators work?

762
00:51:56,650 --> 00:52:00,210
First of all, you
need an algorithm

763
00:52:00,210 --> 00:52:05,620
that finds the separator and
a lot of computer science

764
00:52:05,620 --> 00:52:07,650
attention has gone onto that.

765
00:52:07,650 --> 00:52:09,840
Given a graph, cut it in half.

766
00:52:13,670 --> 00:52:15,970
We don't want more
nodes in the separator

767
00:52:15,970 --> 00:52:19,700
than necessary, because the
work is going to end up there.

768
00:52:19,700 --> 00:52:24,080
So we want to cut the
graph pretty nearly in half

769
00:52:24,080 --> 00:52:25,360
by a short separator.

770
00:52:30,540 --> 00:52:39,790
If we only did the dissection
once, that's what we need,

771
00:52:39,790 --> 00:52:42,720
but this idea of
nested dissection --

772
00:52:42,720 --> 00:52:45,260
you know what's going
to come before I say it.

773
00:52:45,260 --> 00:52:51,800
The idea of nested dissection
will be, take this,

774
00:52:51,800 --> 00:52:57,100
take the graph -- can I just
draw it this way without

775
00:52:57,100 --> 00:52:59,120
drawing all the edges?

776
00:52:59,120 --> 00:53:01,890
Cut it by a separator S_1.

777
00:53:01,890 --> 00:53:04,700
That gives a P_1 and a Q_1.

778
00:53:04,700 --> 00:53:07,590
But now, what's coming?

779
00:53:07,590 --> 00:53:09,930
You're going to separate P_1.

780
00:53:09,930 --> 00:53:13,180
In order to do the work up
in that left corner for the P

781
00:53:13,180 --> 00:53:18,887
matrix, that's itself
a sparse matrix, so now

782
00:53:18,887 --> 00:53:20,720
I'll look for a separator
and what separator

783
00:53:20,720 --> 00:53:24,790
will I choose for the
left side of this graph?

784
00:53:24,790 --> 00:53:29,490
I'll choose the
horizontal separator.

785
00:53:29,490 --> 00:53:31,960
This will be the
second separator.

786
00:53:31,960 --> 00:53:34,740
It'll separate
the two bits of P.

787
00:53:34,740 --> 00:53:39,310
The third separator will
separate the two bits of Q.

788
00:53:39,310 --> 00:53:44,310
Now I have this part to work
on, a fourth separator will --

789
00:53:44,310 --> 00:53:45,310
you'll see its a nested.

790
00:53:45,310 --> 00:53:48,130
These separators become nested.

791
00:53:48,130 --> 00:53:52,840
There's S_5, there's
S_6 and so on.

792
00:53:52,840 --> 00:53:54,820
I think everybody
sees in principle

793
00:53:54,820 --> 00:54:00,110
what nested dissection should
do and the question is, how does

794
00:54:00,110 --> 00:54:02,050
it compare with minimum degree?

795
00:54:02,050 --> 00:54:04,190
You see, it's just
like a different --

796
00:54:04,190 --> 00:54:09,520
we're off on a
different route here.

797
00:54:09,520 --> 00:54:13,810
You can compute
asymptotically for large N,

798
00:54:13,810 --> 00:54:17,550
what power of N appears
there, what power of N appears

799
00:54:17,550 --> 00:54:22,170
in minimum degree -- but I
would love to know, does the --

800
00:54:22,170 --> 00:54:26,870
I would be happy if somebody
used approximate minimum degree

801
00:54:26,870 --> 00:54:33,330
on our model problem and maybe
also on some other problem.

802
00:54:33,330 --> 00:54:38,490
We could see what power of
N -- that's essentially what

803
00:54:38,490 --> 00:54:39,740
we're looking for.

804
00:54:39,740 --> 00:54:47,590
So it's N to what power for
fill-in and for the operation

805
00:54:47,590 --> 00:54:52,730
count, which -- the
number of flops,

806
00:54:52,730 --> 00:54:56,880
which controls the
cost and the time.

807
00:54:56,880 --> 00:54:59,410
So it's what power
of N appears there

808
00:54:59,410 --> 00:55:03,310
and what power appears
in any other method.

809
00:55:05,940 --> 00:55:10,400
There you have it for
direct elimination methods.

810
00:55:10,400 --> 00:55:13,420
I'll just mention
the name of the --

811
00:55:13,420 --> 00:55:19,900
guy's at University of
Florida, and he's the one who

812
00:55:19,900 --> 00:55:24,290
contributed the code that
MathWorks uses and has just

813
00:55:24,290 --> 00:55:30,670
announced a book coming
on this exact topic,

814
00:55:30,670 --> 00:55:33,680
to come later this
year from Sci Am.

815
00:55:33,680 --> 00:55:37,700
So there will be a
book from the person

816
00:55:37,700 --> 00:55:44,750
I take to be the leader in
this algorithmic development.

817
00:55:44,750 --> 00:55:47,920
Just to remind you,
Friday is my guest,

818
00:55:47,920 --> 00:55:51,240
former PhD student who is
going to speak about numerical

819
00:55:51,240 --> 00:55:54,890
methods for finance --
Black-Scholes equations

820
00:55:54,890 --> 00:56:02,400
and other problems to find the
value of financial derivatives.

821
00:56:02,400 --> 00:56:03,597
See you Friday.