1
00:00:00,000 --> 00:00:00,040

2
00:00:00,040 --> 00:00:02,460
The following content is
provided under a Creative

3
00:00:02,460 --> 00:00:03,870
Commons license.

4
00:00:03,870 --> 00:00:06,910
Your support will help MIT
OpenCourseWare continue to

5
00:00:06,910 --> 00:00:10,560
offer high quality educational
resources for free.

6
00:00:10,560 --> 00:00:13,460
To make a donation or view
additional materials from

7
00:00:13,460 --> 00:00:17,390
hundreds of MIT courses, visit
MIT OpenCourseWare at

8
00:00:17,390 --> 00:00:18,640
ocw.mit.edu.

9
00:00:18,640 --> 00:00:23,440

10
00:00:23,440 --> 00:00:25,930
JOHN TSITSIKLIS: So what we're
going to do is to review what

11
00:00:25,930 --> 00:00:29,390
we have discussed last time.

12
00:00:29,390 --> 00:00:32,380
Then we're going to talk about
the classic application of

13
00:00:32,380 --> 00:00:35,950
Markov chains to analyze
how do you

14
00:00:35,950 --> 00:00:39,130
dimension a phone system.

15
00:00:39,130 --> 00:00:42,400
And finally, there will be
two new things today.

16
00:00:42,400 --> 00:00:44,940
We will see how we can calculate
certain interesting

17
00:00:44,940 --> 00:00:48,950
quantities that have to
do with Markov chains.

18
00:00:48,950 --> 00:00:50,200
So let us start.

19
00:00:50,200 --> 00:00:52,860

20
00:00:52,860 --> 00:00:56,680
We've got our Markov chain and
let's make the assumption that

21
00:00:56,680 --> 00:00:58,800
our chain is kind of nice.

22
00:00:58,800 --> 00:01:01,550
And by nice we mean that
we've got maybe

23
00:01:01,550 --> 00:01:02,800
some transient states.

24
00:01:02,800 --> 00:01:06,380

25
00:01:06,380 --> 00:01:11,580
And then we've got a single
recurrent class

26
00:01:11,580 --> 00:01:15,450
of recurrent states.

27
00:01:15,450 --> 00:01:18,600
So this is a single recurrent
class in the sense that from

28
00:01:18,600 --> 00:01:22,080
any state in that class you can
get to any other state.

29
00:01:22,080 --> 00:01:24,790
So once you're in here you're
going to circulate and keep

30
00:01:24,790 --> 00:01:26,650
visiting all of those states.

31
00:01:26,650 --> 00:01:28,340
Those states appear transient.

32
00:01:28,340 --> 00:01:32,020
The trajectory may move around
here, but eventually one of

33
00:01:32,020 --> 00:01:34,320
these transitions will happen
and you're going to

34
00:01:34,320 --> 00:01:36,040
end up in this lump.

35
00:01:36,040 --> 00:01:38,640

36
00:01:38,640 --> 00:01:42,680
Let's make the assumption that
the single recurrent class is

37
00:01:42,680 --> 00:01:43,930
not periodic.

38
00:01:43,930 --> 00:01:48,170

39
00:01:48,170 --> 00:01:51,740
These are the nicest kind
of Markov chains.

40
00:01:51,740 --> 00:01:54,500
And they're nicest because
they have the following

41
00:01:54,500 --> 00:01:58,900
property, the probability that
you find yourself at some

42
00:01:58,900 --> 00:02:03,100
particular state j at
the time n when that

43
00:02:03,100 --> 00:02:04,800
time is very large.

44
00:02:04,800 --> 00:02:08,130
That probability settles to a
steady state value that we

45
00:02:08,130 --> 00:02:10,139
denote by pi sub j.

46
00:02:10,139 --> 00:02:12,630
And there are two parts
in the statement.

47
00:02:12,630 --> 00:02:15,470
One part is that this
limit exists.

48
00:02:15,470 --> 00:02:19,010
So the probability of state j
settles to something, and

49
00:02:19,010 --> 00:02:23,080
furthermore that probability
is not affected by i.

50
00:02:23,080 --> 00:02:25,550
It doesn't matter where you
started, no matter where you

51
00:02:25,550 --> 00:02:28,310
started, the probability of
state j is going to be the

52
00:02:28,310 --> 00:02:30,720
same in the long run.

53
00:02:30,720 --> 00:02:34,990
Maybe a clearer notation
could be of this form.

54
00:02:34,990 --> 00:02:37,650
The probability of being at
state j given the initial

55
00:02:37,650 --> 00:02:42,570
state being i is equal to
pi(j) in the limit.

56
00:02:42,570 --> 00:02:47,380
Now, if I don't tell you where
you started and you look at

57
00:02:47,380 --> 00:02:52,520
the unconditional probability
of being at state i, you can

58
00:02:52,520 --> 00:02:56,260
average over the initial
states, use the total

59
00:02:56,260 --> 00:03:01,560
expectation theorem and you're
going to get the same answer

60
00:03:01,560 --> 00:03:05,330
pi(j) in the limit.

61
00:03:05,330 --> 00:03:09,400
So this tells you that to the
conditional probability given

62
00:03:09,400 --> 00:03:11,820
the initial state in the limit
is the same as the

63
00:03:11,820 --> 00:03:13,780
unconditional probability.

64
00:03:13,780 --> 00:03:17,200
And that's a situation that we
recognize as being one where

65
00:03:17,200 --> 00:03:18,890
we have independence.

66
00:03:18,890 --> 00:03:24,970
So what this result tells
us is that Xn and Xi are

67
00:03:24,970 --> 00:03:27,410
approximately independent.

68
00:03:27,410 --> 00:03:32,690
They become independent in the
limit as n goes to infinity.

69
00:03:32,690 --> 00:03:35,130
So that's what the steady
state theorem tells us.

70
00:03:35,130 --> 00:03:38,860
The initial conditions don't
matter, so your state at some

71
00:03:38,860 --> 00:03:43,440
large time n has nothing to do,
is not affected by what

72
00:03:43,440 --> 00:03:45,350
your initial state was.

73
00:03:45,350 --> 00:03:47,790
Knowing the initial state
doesn't tell you anything

74
00:03:47,790 --> 00:03:50,830
about your state at time
n, therefore the

75
00:03:50,830 --> 00:03:52,890
states at the times--

76
00:03:52,890 --> 00:03:56,580
sorry that should be a 1,
or it should be a 0 --

77
00:03:56,580 --> 00:04:02,350
so the state is not affected by
where the process started.

78
00:04:02,350 --> 00:04:05,450
So if the Markov chain is to
operate for a long time and

79
00:04:05,450 --> 00:04:08,520
we're interested in the question
where is the state,

80
00:04:08,520 --> 00:04:11,470
then your answer would be, I
don't know, it's random.

81
00:04:11,470 --> 00:04:14,020
But it's going to be a
particular j with this

82
00:04:14,020 --> 00:04:15,500
particular probability.

83
00:04:15,500 --> 00:04:18,550
So the steady state
probabilities are interesting

84
00:04:18,550 --> 00:04:21,060
to us and that raises
the question of how

85
00:04:21,060 --> 00:04:22,470
do we compute them.

86
00:04:22,470 --> 00:04:25,460
The way we compute them is by
solving a linear system of

87
00:04:25,460 --> 00:04:28,580
equations, which are called
the balance equations,

88
00:04:28,580 --> 00:04:31,510
together with an extra equation,
the normalization

89
00:04:31,510 --> 00:04:33,690
equation that has to be
satisfied by probability,

90
00:04:33,690 --> 00:04:37,940
because probabilities must
always add up to 1.

91
00:04:37,940 --> 00:04:40,660
We talked about the
interpretation of this

92
00:04:40,660 --> 00:04:42,280
equation last time.

93
00:04:42,280 --> 00:04:45,500
It's basically a conservation
of probability

94
00:04:45,500 --> 00:04:47,400
flow in some sense.

95
00:04:47,400 --> 00:04:50,140
What comes in must get out.

96
00:04:50,140 --> 00:04:53,190
The probability of finding
yourself at state j at a

97
00:04:53,190 --> 00:04:58,150
particular time is the total
probability of the last

98
00:04:58,150 --> 00:05:01,390
transition taking
me into state j.

99
00:05:01,390 --> 00:05:04,860
The last transition takes me
into state j in various ways.

100
00:05:04,860 --> 00:05:07,890
It could be that the previous
time I was at the particular

101
00:05:07,890 --> 00:05:12,550
state, j and i made a transition
from k into j.

102
00:05:12,550 --> 00:05:15,610
So this number here, we
interpret as the frequency

103
00:05:15,610 --> 00:05:18,190
with which transitions
of these particular

104
00:05:18,190 --> 00:05:21,900
type k to j, occur.

105
00:05:21,900 --> 00:05:25,840
And then by adding over all k's
we consider transitions of

106
00:05:25,840 --> 00:05:29,770
all types that lead
us inside state j.

107
00:05:29,770 --> 00:05:34,240
So the probability of being at
the j is the sum total of the

108
00:05:34,240 --> 00:05:38,690
probabilities of
getting into j.

109
00:05:38,690 --> 00:05:42,800
What if we had multiple
recurrent classes?

110
00:05:42,800 --> 00:05:48,645
So if we take this picture
and change it to this.

111
00:05:48,645 --> 00:05:53,420

112
00:05:53,420 --> 00:05:55,920
So here we got a secondary
recurrent class.

113
00:05:55,920 --> 00:05:57,750
If you're here, you
cannot get there.

114
00:05:57,750 --> 00:06:00,050
If you are here, you
cannot get there.

115
00:06:00,050 --> 00:06:02,370
What happens in the long run?

116
00:06:02,370 --> 00:06:05,990
Well, in the long run, if you
start from here you're going

117
00:06:05,990 --> 00:06:09,360
to make a transition eventually,
either of this

118
00:06:09,360 --> 00:06:12,540
type and you would end up
here, or you will make a

119
00:06:12,540 --> 00:06:16,210
transitional of that type and
you will end up there.

120
00:06:16,210 --> 00:06:21,740
If you end up here, the long
term statistics of your chain,

121
00:06:21,740 --> 00:06:24,860
that is, the probabilities of
the different states, will be

122
00:06:24,860 --> 00:06:27,080
the steady state probabilities
of this

123
00:06:27,080 --> 00:06:30,230
chain regarded in isolation.

124
00:06:30,230 --> 00:06:34,440
So you go ahead and you solve
this system of equations just

125
00:06:34,440 --> 00:06:37,270
for this chain, and these will
be your steady state

126
00:06:37,270 --> 00:06:38,560
probabilities.

127
00:06:38,560 --> 00:06:41,660
If you happened to
get in here.

128
00:06:41,660 --> 00:06:45,650
If, on the other hand, it
happens that you went there,

129
00:06:45,650 --> 00:06:49,710
given that event, then what
happens in the long run has to

130
00:06:49,710 --> 00:06:53,390
do with just this chain
running by itself.

131
00:06:53,390 --> 00:06:55,610
So you find the steady
state probabilities

132
00:06:55,610 --> 00:06:57,340
inside that sub chain.

133
00:06:57,340 --> 00:06:59,860
So you solve the linear system,
the steady state

134
00:06:59,860 --> 00:07:03,520
equations, for this chain
separately and for that chain

135
00:07:03,520 --> 00:07:04,590
separately.

136
00:07:04,590 --> 00:07:08,160
If you happen to start inside
here then the steady state

137
00:07:08,160 --> 00:07:12,650
probabilities for this sub
chain are going to apply.

138
00:07:12,650 --> 00:07:16,420
Now of course this raises the
question, if I start here, how

139
00:07:16,420 --> 00:07:19,500
do I know whether I'm going
to get here or there?

140
00:07:19,500 --> 00:07:21,360
Well, you don't know,
it's random.

141
00:07:21,360 --> 00:07:23,620
It may turn out that you get to
here, it may turn out that

142
00:07:23,620 --> 00:07:24,760
you get there.

143
00:07:24,760 --> 00:07:27,790
So we will be interested in
calculating the probabilities

144
00:07:27,790 --> 00:07:31,240
that eventually you end up here
versus the probability

145
00:07:31,240 --> 00:07:33,570
that eventually you
end up there.

146
00:07:33,570 --> 00:07:36,590
This is something that we're
going to do towards the end of

147
00:07:36,590 --> 00:07:37,840
today's lecture.

148
00:07:37,840 --> 00:07:43,730

149
00:07:43,730 --> 00:07:48,420
So, as a warm up, just to see
how we interpret those steady

150
00:07:48,420 --> 00:07:52,710
state probabilities, let us look
at our familiar example.

151
00:07:52,710 --> 00:07:54,680
This is a 2-state
Markov chain.

152
00:07:54,680 --> 00:07:57,860
Last time we did write down the
balance equations for this

153
00:07:57,860 --> 00:08:01,760
chain and we found the steady
state probabilities to be 2/7

154
00:08:01,760 --> 00:08:04,250
and 5/7 respectively.

155
00:08:04,250 --> 00:08:07,040
So let us try to calculate
some quantities.

156
00:08:07,040 --> 00:08:10,520
Suppose that you start at
state 1, and you want to

157
00:08:10,520 --> 00:08:12,940
calculate to this particular
probability.

158
00:08:12,940 --> 00:08:15,620
So since we're assuming that
we're starting at state 1,

159
00:08:15,620 --> 00:08:19,010
essentially here we are
conditioning on the initial

160
00:08:19,010 --> 00:08:21,790
state being equal to 1.

161
00:08:21,790 --> 00:08:23,920
Now the conditional probability
of two things

162
00:08:23,920 --> 00:08:30,370
happening is the probability
that the first thing happens.

163
00:08:30,370 --> 00:08:34,090
But we're living in the world
where we said that the initial

164
00:08:34,090 --> 00:08:35,720
state was 1.

165
00:08:35,720 --> 00:08:40,000
And then given that this thing
happened, the probability that

166
00:08:40,000 --> 00:08:43,530
the second thing happens.

167
00:08:43,530 --> 00:08:46,200
But again, we're talking about
conditional probabilities

168
00:08:46,200 --> 00:08:50,150
given that the initial
state was 1.

169
00:08:50,150 --> 00:08:53,080
So what is this quantity?

170
00:08:53,080 --> 00:08:57,040
This one is the transition
probability from state 1 to

171
00:08:57,040 --> 00:09:00,800
state 1, so it's P11.

172
00:09:00,800 --> 00:09:04,730
How about the second
probability?

173
00:09:04,730 --> 00:09:08,130
So given that you started at 1
and the next time you were at

174
00:09:08,130 --> 00:09:11,980
1, what's the probability that
at the time 100 you are at 1?

175
00:09:11,980 --> 00:09:15,160
Now because of the Markov
property, if I tell you that

176
00:09:15,160 --> 00:09:17,520
at this time you are
at 1, it doesn't

177
00:09:17,520 --> 00:09:19,190
matter how you get there.

178
00:09:19,190 --> 00:09:22,560
So this part of the conditioning
doesn't matter.

179
00:09:22,560 --> 00:09:29,650
And what we have is the 99 step
transition probability

180
00:09:29,650 --> 00:09:34,820
from state 1 to state 1.

181
00:09:34,820 --> 00:09:38,910
So the probability that you
get to 1 and then 99 steps

182
00:09:38,910 --> 00:09:42,340
later you find yourself again at
one is the probability that

183
00:09:42,340 --> 00:09:45,350
the first transition takes you
to 1 times the probability

184
00:09:45,350 --> 00:09:51,810
that over the next 99
transitions starting from 1,

185
00:09:51,810 --> 00:09:55,930
after 99 steps you end
up again at state 1.

186
00:09:55,930 --> 00:10:00,720
Now, 99 is possibly a big
number, and so we approximate

187
00:10:00,720 --> 00:10:01,730
this quantity.

188
00:10:01,730 --> 00:10:05,810
We're using the steady state
probability of state 1.

189
00:10:05,810 --> 00:10:08,110
And that gives us an
approximation for this

190
00:10:08,110 --> 00:10:10,510
particular expression.

191
00:10:10,510 --> 00:10:14,790
We can do the same thing
to calculate

192
00:10:14,790 --> 00:10:16,490
something of the same kind.

193
00:10:16,490 --> 00:10:19,260
So you start at state 1.

194
00:10:19,260 --> 00:10:23,230
What's the probability that 100
steps later you are again

195
00:10:23,230 --> 00:10:24,560
at state 1?

196
00:10:24,560 --> 00:10:26,000
So that's going to be P11--

197
00:10:26,000 --> 00:10:28,970

198
00:10:28,970 --> 00:10:29,890
not P --

199
00:10:29,890 --> 00:10:31,140
R11.

200
00:10:31,140 --> 00:10:34,830

201
00:10:34,830 --> 00:10:38,560
The 100 step transition
probability that starting from

202
00:10:38,560 --> 00:10:43,980
1 you get to 1, and then after
you get to 1 at time 100

203
00:10:43,980 --> 00:10:46,120
what's the probability that
the next time you find

204
00:10:46,120 --> 00:10:47,870
yourself at state 2?

205
00:10:47,870 --> 00:10:50,580
This is going to be the
probability P12.

206
00:10:50,580 --> 00:10:55,320
And approximately, since 100
is a large number, this is

207
00:10:55,320 --> 00:10:57,890
approximately pi(1) times P12.

208
00:10:57,890 --> 00:11:06,230

209
00:11:06,230 --> 00:11:07,070
OK.

210
00:11:07,070 --> 00:11:12,180
So that's how we can use steady
state probabilities to

211
00:11:12,180 --> 00:11:14,190
make approximations.

212
00:11:14,190 --> 00:11:19,090
Or you could, for example, if
you continue doing examples of

213
00:11:19,090 --> 00:11:23,250
this kind, you could ask for
what's the probability that X

214
00:11:23,250 --> 00:11:32,170
at time 100 is 1, and also X
at time 200 is equal to 1.

215
00:11:32,170 --> 00:11:36,200
Then this is going to be the
transition probability from 1

216
00:11:36,200 --> 00:11:43,140
to 1 in 100 steps, and then over
the next 100 steps from 1

217
00:11:43,140 --> 00:11:46,510
you get again to 1.

218
00:11:46,510 --> 00:11:48,750
And this is going to be

219
00:11:48,750 --> 00:11:51,400
approximately pi(1) times pi(1).

220
00:11:51,400 --> 00:11:57,550

221
00:11:57,550 --> 00:12:01,090
So we approximate multi-step
transition probabilities by

222
00:12:01,090 --> 00:12:05,120
the steady state probabilities
when the number n that's

223
00:12:05,120 --> 00:12:07,510
involved in here is big.

224
00:12:07,510 --> 00:12:11,270
Now I said that's 99
or 100 is big.

225
00:12:11,270 --> 00:12:15,180
How do we know that it's big
enough so that the limit has

226
00:12:15,180 --> 00:12:19,560
taken effect, and that our
approximation is good?

227
00:12:19,560 --> 00:12:23,640
This has something to do with
the time scale of our Markov

228
00:12:23,640 --> 00:12:27,460
chain, and by time scale, I mean
how long does it take for

229
00:12:27,460 --> 00:12:29,740
the initial states
to be forgotten.

230
00:12:29,740 --> 00:12:35,080
How long does it take for there
to be enough randomness

231
00:12:35,080 --> 00:12:37,700
so that things sort of
mix and it doesn't

232
00:12:37,700 --> 00:12:39,330
matter where you started?

233
00:12:39,330 --> 00:12:43,720
So if you look at this chain, it
takes on the average, let's

234
00:12:43,720 --> 00:12:47,660
say 5 tries to make a transition
of this kind.

235
00:12:47,660 --> 00:12:51,970
It takes on the average 2 tries
for a transition of that

236
00:12:51,970 --> 00:12:53,990
kind to take place.

237
00:12:53,990 --> 00:12:58,840
So every 10 time steps or so
there's a little bit of

238
00:12:58,840 --> 00:12:59,660
randomness.

239
00:12:59,660 --> 00:13:03,000
Over 100 times steps there's
a lot of randomness, so you

240
00:13:03,000 --> 00:13:06,420
expect that the initial state
will have been forgotten.

241
00:13:06,420 --> 00:13:07,820
It doesn't matter.

242
00:13:07,820 --> 00:13:10,540
There's enough mixing and
randomness that happens over

243
00:13:10,540 --> 00:13:11,870
100 time steps.

244
00:13:11,870 --> 00:13:16,100
And so this approximation
is good.

245
00:13:16,100 --> 00:13:19,080
On the other hand, if the
numbers were different, the

246
00:13:19,080 --> 00:13:20,880
story would have
been different.

247
00:13:20,880 --> 00:13:26,620
Suppose that this number is
0.999 and that number is

248
00:13:26,620 --> 00:13:33,570
something like 0.998, so that
this number becomes 0.002, and

249
00:13:33,570 --> 00:13:37,910
that number becomes 0.001.

250
00:13:37,910 --> 00:13:40,620
Suppose that the numbers
were of this kind.

251
00:13:40,620 --> 00:13:44,760
How long does it take to forget
the initial state?

252
00:13:44,760 --> 00:13:48,840
If I start here, there's a
probability of 1 in 1,000 that

253
00:13:48,840 --> 00:13:50,770
next time I'm going
to be there.

254
00:13:50,770 --> 00:13:53,840
So on the average it's going
to take me about a thousand

255
00:13:53,840 --> 00:13:58,330
tries just to leave
that state.

256
00:13:58,330 --> 00:14:03,390
So, over roughly a thousand time
steps my initial state

257
00:14:03,390 --> 00:14:05,210
really does matter.

258
00:14:05,210 --> 00:14:08,550
If I tell you that you started
here, you're pretty certain

259
00:14:08,550 --> 00:14:11,320
that, let's say over the next
100 time steps, you

260
00:14:11,320 --> 00:14:12,710
will still be here.

261
00:14:12,710 --> 00:14:15,580
So the initial state
has a big effect.

262
00:14:15,580 --> 00:14:19,600
In this case we say that this
Markov chain has a much slower

263
00:14:19,600 --> 00:14:21,030
time scale.

264
00:14:21,030 --> 00:14:25,350
It takes a much longer time to
mix, it takes a much longer

265
00:14:25,350 --> 00:14:29,150
time for the initial state to
be forgotten, and this means

266
00:14:29,150 --> 00:14:32,670
that we cannot do this kind of
approximation if the number of

267
00:14:32,670 --> 00:14:34,870
steps is just 99.

268
00:14:34,870 --> 00:14:39,620
Here we might need n to be as
large as, let's say, 10,000 or

269
00:14:39,620 --> 00:14:43,320
so before we can start using
the approximation.

270
00:14:43,320 --> 00:14:46,210
So when one uses that
approximation, one needs to

271
00:14:46,210 --> 00:14:50,830
have some sense of how quickly
does the state move around and

272
00:14:50,830 --> 00:14:52,210
take that into account.

273
00:14:52,210 --> 00:14:56,100
So there's a whole sub-field
that deals with estimating or

274
00:14:56,100 --> 00:15:00,030
figuring out how quickly
different Markov chains mix,

275
00:15:00,030 --> 00:15:02,460
and that's the question of
when can you apply those

276
00:15:02,460 --> 00:15:03,710
steady state approximations.

277
00:15:03,710 --> 00:15:06,860

278
00:15:06,860 --> 00:15:12,180
So now let's get a little closer
to the real world.

279
00:15:12,180 --> 00:15:15,590
We're going to talk about a
famous problem that was posed,

280
00:15:15,590 --> 00:15:19,220
started, and solved by
a Danish engineer

281
00:15:19,220 --> 00:15:21,110
by the name of Erlang.

282
00:15:21,110 --> 00:15:25,180
This is the same person whose
name is given to the Erlang

283
00:15:25,180 --> 00:15:27,080
distribution that we
saw in the context

284
00:15:27,080 --> 00:15:28,940
of the Poisson processes.

285
00:15:28,940 --> 00:15:33,950
So this was more than 100 years
ago, when phones had

286
00:15:33,950 --> 00:15:36,060
just started existing.

287
00:15:36,060 --> 00:15:42,900
And he was trying to figure out
what it would take to set

288
00:15:42,900 --> 00:15:47,260
up a phone system that how many
lines should you set up

289
00:15:47,260 --> 00:15:49,920
for a community to be
able to communicate

290
00:15:49,920 --> 00:15:53,020
to the outside world.

291
00:15:53,020 --> 00:15:54,350
So here's the story.

292
00:15:54,350 --> 00:15:57,730
You've got a village, and that
village has a certain

293
00:15:57,730 --> 00:16:03,705
population, and you want
to set up phone lines.

294
00:16:03,705 --> 00:16:06,290

295
00:16:06,290 --> 00:16:09,720
So you want to set up a number
of phone lines, let's say that

296
00:16:09,720 --> 00:16:13,520
number is B, to the
outside world.

297
00:16:13,520 --> 00:16:17,710

298
00:16:17,710 --> 00:16:19,690
And how do you want
to do that?

299
00:16:19,690 --> 00:16:22,000
Well, you want B to
be kind of small.

300
00:16:22,000 --> 00:16:24,130
You don't want to set
up too many wires

301
00:16:24,130 --> 00:16:25,680
because that's expensive.

302
00:16:25,680 --> 00:16:30,300
On the other hand, you want to
have enough wires so that if a

303
00:16:30,300 --> 00:16:33,000
reasonable number of people
place phone calls

304
00:16:33,000 --> 00:16:38,000
simultaneously, they will all
get a line and they will be

305
00:16:38,000 --> 00:16:39,600
able to talk.

306
00:16:39,600 --> 00:16:45,220
So if B is 10 and 12 people want
to talk at the same time,

307
00:16:45,220 --> 00:16:48,490
then 2 of these people would get
a busy signal, and that's

308
00:16:48,490 --> 00:16:50,220
not something that we like.

309
00:16:50,220 --> 00:16:53,010
We would like B to be large
enough so that there's a

310
00:16:53,010 --> 00:16:57,240
substantial probability, that
there's almost certainty that,

311
00:16:57,240 --> 00:17:00,120
under reasonable conditions,
no one is going

312
00:17:00,120 --> 00:17:02,230
to get a busy signal.

313
00:17:02,230 --> 00:17:06,720
So how do we go about modeling
a situation like this?

314
00:17:06,720 --> 00:17:09,890
Well, to set up a model you
need two pieces, one is to

315
00:17:09,890 --> 00:17:17,000
describe how do phone calls
get initiated, and once a

316
00:17:17,000 --> 00:17:21,819
phone call gets started, how
long does it take until the

317
00:17:21,819 --> 00:17:24,530
phone call is terminated?

318
00:17:24,530 --> 00:17:27,530
So we're going to make the
simplest assumptions possible.

319
00:17:27,530 --> 00:17:29,890
Let's assume that phone
calls originate

320
00:17:29,890 --> 00:17:31,920
as a Poisson process.

321
00:17:31,920 --> 00:17:34,210
That is, out of that population
people do not

322
00:17:34,210 --> 00:17:35,290
really coordinate.

323
00:17:35,290 --> 00:17:38,160
At completely random times,
different people with decide

324
00:17:38,160 --> 00:17:39,740
to pick up the phone.

325
00:17:39,740 --> 00:17:42,320
There's no dependencies between
different people,

326
00:17:42,320 --> 00:17:44,790
there's nothing special about
different times, different

327
00:17:44,790 --> 00:17:46,340
times are independent.

328
00:17:46,340 --> 00:17:50,550
So a Poisson model is a
reasonable way of modeling

329
00:17:50,550 --> 00:17:51,410
this situation.

330
00:17:51,410 --> 00:17:55,870
And it's going to be a Poisson
process with some rate lambda.

331
00:17:55,870 --> 00:17:59,870
Now, the rate lambda would be
easy to estimate in practice.

332
00:17:59,870 --> 00:18:02,760
You observe what happens in
that village just over a

333
00:18:02,760 --> 00:18:06,050
couple of days, and you figure
out what's the rate at which

334
00:18:06,050 --> 00:18:09,550
people attempt to place
phone calls.

335
00:18:09,550 --> 00:18:11,930
Now, about phone calls
themselves, we're going to

336
00:18:11,930 --> 00:18:15,120
make the assumption that the
duration of a phone call is a

337
00:18:15,120 --> 00:18:18,580
random variable that has an
exponential distribution with

338
00:18:18,580 --> 00:18:20,630
a certain parameter mu.

339
00:18:20,630 --> 00:18:24,400
So 1/mu is the mean duration
of a phone call.

340
00:18:24,400 --> 00:18:27,570
So the mean duration, , again,
is easy to estimate.

341
00:18:27,570 --> 00:18:31,240
You just observe what's
happening, see on the average

342
00:18:31,240 --> 00:18:33,680
how long these phone
calls are.

343
00:18:33,680 --> 00:18:36,590
Is the exponential assumption
a good assumption?

344
00:18:36,590 --> 00:18:39,990
Well, it's means that most phone
calls will be kind of

345
00:18:39,990 --> 00:18:43,180
short, but there's going to be a
fraction of phone calls that

346
00:18:43,180 --> 00:18:45,640
are going to be larger, and
then a very small fraction

347
00:18:45,640 --> 00:18:47,900
that are going to
be even larger.

348
00:18:47,900 --> 00:18:49,880
So it sounds plausible.

349
00:18:49,880 --> 00:18:57,960
It's not exactly realistic, that
is, phone calls that last

350
00:18:57,960 --> 00:19:02,250
short of 15 seconds are
not that common.

351
00:19:02,250 --> 00:19:04,890
So either nothing happens
or you have to say a few

352
00:19:04,890 --> 00:19:06,630
sentences and so on.

353
00:19:06,630 --> 00:19:10,200
Also, back into the days when
people used to connect to the

354
00:19:10,200 --> 00:19:15,430
internet using dial up modems,
that assumption was completely

355
00:19:15,430 --> 00:19:20,390
destroyed, because people would
dial up and then keep

356
00:19:20,390 --> 00:19:25,730
their phone line busy for a few
hours, if the phone call

357
00:19:25,730 --> 00:19:26,930
was a free one.

358
00:19:26,930 --> 00:19:30,390
So at those times the
exponential assumption for the

359
00:19:30,390 --> 00:19:33,260
phone call duration was
completely destroyed.

360
00:19:33,260 --> 00:19:36,430
But leaving that detail aside,
it's sort of a reasonable

361
00:19:36,430 --> 00:19:41,000
assumption to just get started
with this problem.

362
00:19:41,000 --> 00:19:43,790
All right, so now that we have
those assumptions, let's try

363
00:19:43,790 --> 00:19:46,510
to come up with the model.

364
00:19:46,510 --> 00:19:49,470
And we're going to set up
a Markov process model.

365
00:19:49,470 --> 00:19:52,990
Now the Poisson process runs in
continuous time, and call

366
00:19:52,990 --> 00:19:56,280
durations being exponential
random variables also are

367
00:19:56,280 --> 00:19:59,330
continuous random variables, so
it seems that we are in a

368
00:19:59,330 --> 00:20:01,020
continuous time universe.

369
00:20:01,020 --> 00:20:03,610
But we have only started
Markov chains for the

370
00:20:03,610 --> 00:20:05,310
discrete time case.

371
00:20:05,310 --> 00:20:07,370
What are we going to do?

372
00:20:07,370 --> 00:20:10,470
We can either develop the theory
of continuous time

373
00:20:10,470 --> 00:20:13,600
Markov chains, which
is possible.

374
00:20:13,600 --> 00:20:16,300
But we are not going to
do that in this class.

375
00:20:16,300 --> 00:20:20,460
Or we can discretize time
and work with a

376
00:20:20,460 --> 00:20:22,070
discrete time model.

377
00:20:22,070 --> 00:20:24,850
So we're going to discretize
time in the familiar way, the

378
00:20:24,850 --> 00:20:27,350
way we did it when we started
the Poisson process.

379
00:20:27,350 --> 00:20:31,270
We're going to take the time
axis and split it into little

380
00:20:31,270 --> 00:20:35,130
discrete mini slots, where
every mini slot

381
00:20:35,130 --> 00:20:36,880
has a duration delta.

382
00:20:36,880 --> 00:20:41,520
So this delta is supposed to
be a very small number.

383
00:20:41,520 --> 00:20:44,760
So what is the state
of the system?

384
00:20:44,760 --> 00:20:47,290
So, you look at the situation
in the system at some

385
00:20:47,290 --> 00:20:51,340
particular time and I ask you
what is going on right now,

386
00:20:51,340 --> 00:20:53,680
what's the information
you would tell me?

387
00:20:53,680 --> 00:20:56,990
Well, you would tell me that
right now out of these capital

388
00:20:56,990 --> 00:21:02,160
B lines, 10 of them are busy,
or 12 of them are busy.

389
00:21:02,160 --> 00:21:04,740
That describes the state of
the system, that tells me

390
00:21:04,740 --> 00:21:06,560
what's happening
at this point.

391
00:21:06,560 --> 00:21:10,800
So we set up our states base by
being the numbers from 0 to

392
00:21:10,800 --> 00:21:15,510
B. 0 corresponds to a state in
which all the phone lines are

393
00:21:15,510 --> 00:21:17,260
free, no one is talking.

394
00:21:17,260 --> 00:21:19,590
Capital B corresponds to
a case where all the

395
00:21:19,590 --> 00:21:21,580
phone lines are busy.

396
00:21:21,580 --> 00:21:24,020
And then you've got
states in between.

397
00:21:24,020 --> 00:21:28,050
And now let's look at the
transition probabilities.

398
00:21:28,050 --> 00:21:32,580
Suppose that right so now we
have i-1 lines that are busy.

399
00:21:32,580 --> 00:21:36,280

400
00:21:36,280 --> 00:21:38,380
Or maybe, let me look here.

401
00:21:38,380 --> 00:21:41,460
Suppose that there's i
lines that are busy.

402
00:21:41,460 --> 00:21:44,770
What can happen the next time?

403
00:21:44,770 --> 00:21:48,180
What can happen is that the new
phone call gets placed, in

404
00:21:48,180 --> 00:21:53,670
which case my state moves up
by 1, or an existing call

405
00:21:53,670 --> 00:21:59,060
terminates, in which case my
state goes down by 1, or none

406
00:21:59,060 --> 00:22:03,530
of the two happens, in which
case I stay at the same state.

407
00:22:03,530 --> 00:22:06,930
Well, it's also possible that
the phone call gets terminated

408
00:22:06,930 --> 00:22:10,510
and a new phone call gets placed
sort of simultaneously.

409
00:22:10,510 --> 00:22:14,080
But when you take your time
slots to be very, very small,

410
00:22:14,080 --> 00:22:16,970
this is going to have a
negligible probability order

411
00:22:16,970 --> 00:22:19,850
of delta squared, so
we ignore this.

412
00:22:19,850 --> 00:22:22,190
So what's the probability of
an upwards transition?

413
00:22:22,190 --> 00:22:25,160
That's the probability that the
Poisson process records an

414
00:22:25,160 --> 00:22:29,250
arrival during a mini slot
of duration delta.

415
00:22:29,250 --> 00:22:31,270
By the definition of the
Poisson process, the

416
00:22:31,270 --> 00:22:35,550
probability of this happening
is just lambda delta.

417
00:22:35,550 --> 00:22:39,200
So each one of these upwards
transitions has the same

418
00:22:39,200 --> 00:22:41,000
probability of lambda delta.

419
00:22:41,000 --> 00:22:45,220
So you've got lambda deltas
everywhere in this diagram.

420
00:22:45,220 --> 00:22:49,180
How about, now, phone
call terminations?

421
00:22:49,180 --> 00:22:53,630
If you had the single call that
was active, so if you

422
00:22:53,630 --> 00:22:56,510
were here, what's the
probability that the phone

423
00:22:56,510 --> 00:22:57,820
call terminates?

424
00:22:57,820 --> 00:23:00,840
So the phone call has an
exponential duration with

425
00:23:00,840 --> 00:23:02,820
parameter mu.

426
00:23:02,820 --> 00:23:05,860
And we discussed before that an
exponential random variable

427
00:23:05,860 --> 00:23:08,940
can be thought of as the
first arrival time

428
00:23:08,940 --> 00:23:10,780
in a Poisson process.

429
00:23:10,780 --> 00:23:14,750
So the probability that you get
this event to happen over

430
00:23:14,750 --> 00:23:18,820
a delta time interval is
just mu times delta.

431
00:23:18,820 --> 00:23:22,280
So if you have a single phone
call that's happening right

432
00:23:22,280 --> 00:23:24,780
now, with probability
mu times delta, that

433
00:23:24,780 --> 00:23:27,070
call is going to terminate.

434
00:23:27,070 --> 00:23:30,020
But suppose that we have
i phone calls that

435
00:23:30,020 --> 00:23:31,750
are currently active.

436
00:23:31,750 --> 00:23:35,010
Each one of them has a
probability of mu delta, of

437
00:23:35,010 --> 00:23:39,200
terminating, but collectively
the probability that one of

438
00:23:39,200 --> 00:23:45,850
them terminates becomes
i times mu delta.

439
00:23:45,850 --> 00:23:48,570
So that's because you get the
mu delta contribution --

440
00:23:48,570 --> 00:23:51,540
the probability of termination
from each one of the different

441
00:23:51,540 --> 00:23:54,410
phone calls.

442
00:23:54,410 --> 00:23:58,290
OK, now this is an approximate
calculation, because it

443
00:23:58,290 --> 00:24:01,850
ignores the possibility that two
phone calls terminate at

444
00:24:01,850 --> 00:24:03,110
the same time.

445
00:24:03,110 --> 00:24:09,130
Again, the way to think of why
this is the correct rate, when

446
00:24:09,130 --> 00:24:13,610
you have i phone calls that are
simultaneously running and

447
00:24:13,610 --> 00:24:17,250
waiting for one of them to
terminate, this is like having

448
00:24:17,250 --> 00:24:21,170
i separate Poisson processes
that are running in parallel,

449
00:24:21,170 --> 00:24:23,720
and you ask for the probability
that one of those

450
00:24:23,720 --> 00:24:25,980
processes records an event.

451
00:24:25,980 --> 00:24:28,470
Now when you put all those
process together, it's like

452
00:24:28,470 --> 00:24:32,970
having a Poisson process with
total rate i times mu, and so

453
00:24:32,970 --> 00:24:36,210
i times mu delta is the overall
probability that

454
00:24:36,210 --> 00:24:39,580
something happens in terms of
phone call terminations at

455
00:24:39,580 --> 00:24:40,760
those times.

456
00:24:40,760 --> 00:24:43,470
So in any case, this is the
transition probability for

457
00:24:43,470 --> 00:24:46,280
downwards transitions.

458
00:24:46,280 --> 00:24:49,190
Now that we've got this, we
can analyze this chain.

459
00:24:49,190 --> 00:24:53,220
This chain has the birth death
form that we discussed towards

460
00:24:53,220 --> 00:24:54,770
the end of last lecture.

461
00:24:54,770 --> 00:24:58,360
And for birth death chains, it's
easy to write it out to

462
00:24:58,360 --> 00:25:00,710
find the steady state
probabilities.

463
00:25:00,710 --> 00:25:03,360
Instead of writing down the
balance equations in the

464
00:25:03,360 --> 00:25:07,950
general form, we think in terms
of a conservation of

465
00:25:07,950 --> 00:25:10,630
probabilities or of transitions
by looking at what

466
00:25:10,630 --> 00:25:14,640
happens across a particular
cut in this diagram.

467
00:25:14,640 --> 00:25:18,590
Number of transitions in the
chain that cross from here to

468
00:25:18,590 --> 00:25:21,310
here has to be approximately
equal to the number of

469
00:25:21,310 --> 00:25:24,850
transitions from here to there
because whatever comes up must

470
00:25:24,850 --> 00:25:27,520
come down and then come
up and so on.

471
00:25:27,520 --> 00:25:31,410
So the frequency with which
transitions of this kind are

472
00:25:31,410 --> 00:25:34,070
observed has to be the same as
the frequency of transitions

473
00:25:34,070 --> 00:25:35,690
of this kind.

474
00:25:35,690 --> 00:25:38,400
What's the frequency of how
often the transitions of this

475
00:25:38,400 --> 00:25:40,280
kind happen?

476
00:25:40,280 --> 00:25:45,350
And by frequency I mean quite
percentage of the mini slots

477
00:25:45,350 --> 00:25:48,120
involve a transition
of this kind?

478
00:25:48,120 --> 00:25:51,420
Well, for a transition of that
kind to happen we need to be

479
00:25:51,420 --> 00:25:55,730
at states i-1, which happens
this much of the time.

480
00:25:55,730 --> 00:25:59,360
And then the probability
lambda delta that the

481
00:25:59,360 --> 00:26:01,360
transition is of this kind.

482
00:26:01,360 --> 00:26:06,040
So the frequency of transitions
of with which this

483
00:26:06,040 --> 00:26:11,460
kind of transition is observed
is lambda delta times pi(i-1).

484
00:26:11,460 --> 00:26:17,680
This is the fraction of time
steps at which a transition

485
00:26:17,680 --> 00:26:20,180
from specifically this
state to specifically

486
00:26:20,180 --> 00:26:22,120
that state are observed.

487
00:26:22,120 --> 00:26:24,880
This has to be the same as
the frequency with which

488
00:26:24,880 --> 00:26:28,360
transitions of that kind are
observed, and that frequency

489
00:26:28,360 --> 00:26:32,310
is going to be i mu delta
times pi(i), and then we

490
00:26:32,310 --> 00:26:37,870
cancel the deltas, and we are
left with this equation here.

491
00:26:37,870 --> 00:26:43,260
So this equation expresses pi(i)
in terms of pi(i-1).

492
00:26:43,260 --> 00:26:47,020
So if we knew pi(0) we
can use that equation

493
00:26:47,020 --> 00:26:48,690
to determine pi(1).

494
00:26:48,690 --> 00:26:52,320
Once we know pi(1), we can use
that equation to determine

495
00:26:52,320 --> 00:26:55,490
pi(2), and so on,
you keep going.

496
00:26:55,490 --> 00:26:59,890
And the general formula that
comes out of this, I will not

497
00:26:59,890 --> 00:27:02,350
do the algebra, it's a
straightforward substitution,

498
00:27:02,350 --> 00:27:04,830
you find that pi(i), the steady
state probability of

499
00:27:04,830 --> 00:27:08,650
state i is given by this
expression, which involves the

500
00:27:08,650 --> 00:27:12,020
pi(0) from which we started.

501
00:27:12,020 --> 00:27:13,500
Now what is pi(0)?

502
00:27:13,500 --> 00:27:17,420
Well, we don't know yet, but
we can find it by using the

503
00:27:17,420 --> 00:27:19,520
normalization equation.

504
00:27:19,520 --> 00:27:22,680
The sum of pi(i) has
to be equal to 1.

505
00:27:22,680 --> 00:27:26,430
So the sum of all of those
numbers has to be equal to 1.

506
00:27:26,430 --> 00:27:30,370
And the only way that this can
happen is by setting pi(0) to

507
00:27:30,370 --> 00:27:33,320
be equal to that particular
number.

508
00:27:33,320 --> 00:27:38,970
So if I tell you the value of
capital B, you can set up this

509
00:27:38,970 --> 00:27:43,690
Markov chain, you can calculate
pi(0), and then you

510
00:27:43,690 --> 00:27:47,790
can calculate pi(i), and so you
know what fraction, you

511
00:27:47,790 --> 00:27:51,150
know the steady state
probabilities of this chain,

512
00:27:51,150 --> 00:27:53,920
so you can answer
the question.

513
00:27:53,920 --> 00:27:57,240
If I drop in at a random time,
how likely is it that I'm

514
00:27:57,240 --> 00:28:00,200
going to find the states
to be here, or the

515
00:28:00,200 --> 00:28:01,930
states to be there?

516
00:28:01,930 --> 00:28:03,790
So the steady state
probabilities are

517
00:28:03,790 --> 00:28:07,390
probabilities, but we also
interpret them as frequencies.

518
00:28:07,390 --> 00:28:12,000
So once I find pi(i), it also
tells me what fraction of the

519
00:28:12,000 --> 00:28:16,870
time is the state equal to i.

520
00:28:16,870 --> 00:28:20,030
And you can answer that question
for every possible i.

521
00:28:20,030 --> 00:28:22,180
Now, why did we do
this exercise?

522
00:28:22,180 --> 00:28:25,240
We're interested in
the probability of

523
00:28:25,240 --> 00:28:27,660
the system is busy.

524
00:28:27,660 --> 00:28:31,960
So if a person, a new phone
call gets placed, it just

525
00:28:31,960 --> 00:28:33,020
drops out of the sky.

526
00:28:33,020 --> 00:28:37,560
According to that Poisson
process, that new phone call

527
00:28:37,560 --> 00:28:41,770
is going to find the system
at a random state.

528
00:28:41,770 --> 00:28:45,290
That random state is described
in steady state by the

529
00:28:45,290 --> 00:28:47,440
probabilities pi(i)'s.

530
00:28:47,440 --> 00:28:51,940
And the probability that you
find the system to be busy is

531
00:28:51,940 --> 00:28:55,110
the probability that when you
drop in the state happens to

532
00:28:55,110 --> 00:29:01,380
be that particular number B. So
i sub b is the probability

533
00:29:01,380 --> 00:29:02,680
of being busy.

534
00:29:02,680 --> 00:29:06,496

535
00:29:06,496 --> 00:29:09,470
And this is the probability
that you would like to be

536
00:29:09,470 --> 00:29:11,970
small in a well engineered
system.

537
00:29:11,970 --> 00:29:16,890
So you ask the question, how
should, given my lambda and

538
00:29:16,890 --> 00:29:22,570
mu, my design question is to
determine capital B the number

539
00:29:22,570 --> 00:29:25,720
of phone lines so that
this number is small.

540
00:29:25,720 --> 00:29:31,290

541
00:29:31,290 --> 00:29:36,040
Could we have done, could we
figure out a good value for B

542
00:29:36,040 --> 00:29:39,670
by doing a back of the
envelope calculation?

543
00:29:39,670 --> 00:29:44,410
Let's suppose that lambda is
30 and that mu is 1/3.

544
00:29:44,410 --> 00:29:49,190

545
00:29:49,190 --> 00:29:53,640
So I guess that's, let
us these rates to

546
00:29:53,640 --> 00:29:55,135
be calls per minute.

547
00:29:55,135 --> 00:29:58,400

548
00:29:58,400 --> 00:30:03,100
And this mu, again, is
a rate per minute.

549
00:30:03,100 --> 00:30:07,860
Again, the units of mu are going
to be calls per minute.

550
00:30:07,860 --> 00:30:11,310
So since our time unit is
minutes, the mean duration of

551
00:30:11,310 --> 00:30:13,660
calls is 1/mu minutes.

552
00:30:13,660 --> 00:30:16,930
So a typical call, or
on the average a

553
00:30:16,930 --> 00:30:19,185
call lasts for 3 minutes.

554
00:30:19,185 --> 00:30:23,900

555
00:30:23,900 --> 00:30:27,860
So you get 30 calls
per minute.

556
00:30:27,860 --> 00:30:31,540
Each call lasts for 3 minutes
on the average.

557
00:30:31,540 --> 00:30:36,290
So on the average, if
B was infinite,

558
00:30:36,290 --> 00:30:38,100
every call goes through.

559
00:30:38,100 --> 00:30:42,750
How many calls would be
active on the average?

560
00:30:42,750 --> 00:30:44,570
So you get 30 per minute.

561
00:30:44,570 --> 00:30:48,490
If a call lasted exactly 1
minute, then at any time you

562
00:30:48,490 --> 00:30:51,210
would have 30 calls
being active.

563
00:30:51,210 --> 00:30:54,850
Now a call lasts on the
average for 3 minutes.

564
00:30:54,850 --> 00:30:58,120
So during each minute
you generate 90

565
00:30:58,120 --> 00:31:00,460
minutes of talking time.

566
00:31:00,460 --> 00:31:05,400
So by thinking in terms of
averages you would expect that

567
00:31:05,400 --> 00:31:08,070
at any time there would
be about 90

568
00:31:08,070 --> 00:31:10,540
calls that are active.

569
00:31:10,540 --> 00:31:14,210
And if 90 calls are active on
the average, you could say OK,

570
00:31:14,210 --> 00:31:18,600
I'm going to set up my
capital B to be 90.

571
00:31:18,600 --> 00:31:22,250
But that's not very good,
because if the average number

572
00:31:22,250 --> 00:31:25,950
of phone calls that want to
happen is if the average

573
00:31:25,950 --> 00:31:29,900
number is 90, sometimes you're
going to have 85, sometimes

574
00:31:29,900 --> 00:31:31,460
you will have 95.

575
00:31:31,460 --> 00:31:34,300
And to be sure that the phone
calls will go through you

576
00:31:34,300 --> 00:31:37,790
probably want to choose your
capital B to be a number a

577
00:31:37,790 --> 00:31:40,460
little larger than 90.

578
00:31:40,460 --> 00:31:42,600
How much larger than 90?

579
00:31:42,600 --> 00:31:47,070
Well, this is a question that
you can answer numerically.

580
00:31:47,070 --> 00:31:50,280
So you go through the
following procedure.

581
00:31:50,280 --> 00:31:55,590
I tried different values of
capital B. For any given value

582
00:31:55,590 --> 00:31:59,230
of capital B, I do this
numerical calculation, I find

583
00:31:59,230 --> 00:32:03,530
the probability that the system
is busy, and then I ask

584
00:32:03,530 --> 00:32:07,060
what's the value of B that makes
my probability of being

585
00:32:07,060 --> 00:32:10,510
busy to be, let's say,
roughly 1 %.

586
00:32:10,510 --> 00:32:13,110
And if you do that calculation
with the parameters that they

587
00:32:13,110 --> 00:32:18,620
gave you, you find that B would
be something like 106.

588
00:32:18,620 --> 00:32:21,490
So with the parameters they gave
where you have, on the

589
00:32:21,490 --> 00:32:26,550
average, 90 phone calls being
active, you actually need some

590
00:32:26,550 --> 00:32:30,410
margin to protect against the
[?] fluctuation, if suddenly

591
00:32:30,410 --> 00:32:34,010
by chance more people want to
talk, and if you want to have

592
00:32:34,010 --> 00:32:37,890
a good guarantee that an
incoming person will have a

593
00:32:37,890 --> 00:32:40,860
very small probability of
finding a busy system, then

594
00:32:40,860 --> 00:32:44,860
you will need about
106 phone lines.

595
00:32:44,860 --> 00:32:49,740
So that's the calculation and
the argument that the Erlang

596
00:32:49,740 --> 00:32:52,250
went through a long time ago.

597
00:32:52,250 --> 00:32:55,610
It's actually interesting that
Erlang did this calculation

598
00:32:55,610 --> 00:32:58,590
before Markov chains
were invented.

599
00:32:58,590 --> 00:33:02,250
So Markov's work, and the
beginning of work on Markov

600
00:33:02,250 --> 00:33:06,770
chains, happens about 10-15
years after Erlang.

601
00:33:06,770 --> 00:33:10,180
So obviously he didn't call
that a Markov chain.

602
00:33:10,180 --> 00:33:13,350
But it was something that he
could study from first

603
00:33:13,350 --> 00:33:15,900
principles.

604
00:33:15,900 --> 00:33:19,160
So this is a pretty
useful thing.

605
00:33:19,160 --> 00:33:24,530
These probabilities that come
out of that model, at least in

606
00:33:24,530 --> 00:33:28,230
the old days, they would all
be very well tabulated in

607
00:33:28,230 --> 00:33:32,670
handbooks that every decent
phone company engineer would

608
00:33:32,670 --> 00:33:34,530
sort of have with them.

609
00:33:34,530 --> 00:33:38,890
So this is about as practical
as it gets.

610
00:33:38,890 --> 00:33:42,340
It's one of the sort of
standard real world

611
00:33:42,340 --> 00:33:43,810
applications of Markov chains.

612
00:33:43,810 --> 00:33:47,040

613
00:33:47,040 --> 00:33:53,950
So now to close our subjects,
we're going to consider a

614
00:33:53,950 --> 00:33:57,310
couple of new skills and see how
we can calculate the few

615
00:33:57,310 --> 00:33:59,660
additional interesting
quantities that have to do

616
00:33:59,660 --> 00:34:01,340
with the Markov chain.

617
00:34:01,340 --> 00:34:04,800
So the problem we're going to
deal with here is the one I

618
00:34:04,800 --> 00:34:07,630
hinted that when I was talking
about this picture.

619
00:34:07,630 --> 00:34:09,870
You start at a transient
state, you're going to

620
00:34:09,870 --> 00:34:12,239
eventually end up
here or there.

621
00:34:12,239 --> 00:34:16,389
We want to find the
probabilities of one option of

622
00:34:16,389 --> 00:34:19,210
the two happening or the
other happening.

623
00:34:19,210 --> 00:34:23,880
So in this picture we
have a class of

624
00:34:23,880 --> 00:34:25,680
states that's are transient.

625
00:34:25,680 --> 00:34:29,130

626
00:34:29,130 --> 00:34:34,219
These are transient because
you're going to move around

627
00:34:34,219 --> 00:34:37,170
those states, but there's a
transition that you can make,

628
00:34:37,170 --> 00:34:40,260
and you go to a state from
which you cannot escape

629
00:34:40,260 --> 00:34:41,560
afterwards.

630
00:34:41,560 --> 00:34:43,710
Are you going to end
up here or are you

631
00:34:43,710 --> 00:34:45,370
going to end up there?

632
00:34:45,370 --> 00:34:46,130
You don't know.

633
00:34:46,130 --> 00:34:47,280
It's random.

634
00:34:47,280 --> 00:34:50,940
Let's try to calculate the
probability that you

635
00:34:50,940 --> 00:34:54,800
end up at state 4.

636
00:34:54,800 --> 00:34:59,990
Now, the probability that you
end up at state 4 will depend

637
00:34:59,990 --> 00:35:02,200
on where you start.

638
00:35:02,200 --> 00:35:06,170
Because if you start here, you
probably have more chances of

639
00:35:06,170 --> 00:35:09,720
getting to 4 because you get
that chance immediately,

640
00:35:09,720 --> 00:35:12,390
whereas if you start here
there's more chances that

641
00:35:12,390 --> 00:35:16,260
you're going to escape that way
because it kind of takes

642
00:35:16,260 --> 00:35:17,770
you time to get there.

643
00:35:17,770 --> 00:35:20,810
It's more likely that
you exit right away.

644
00:35:20,810 --> 00:35:26,040
So the probability of exiting
and ending up at state 4 will

645
00:35:26,040 --> 00:35:28,190
depend on the initial state.

646
00:35:28,190 --> 00:35:33,900
That's why when we talk about
these absorption probability

647
00:35:33,900 --> 00:35:38,270
we include an index i that
tells us what the

648
00:35:38,270 --> 00:35:40,310
initial state is.

649
00:35:40,310 --> 00:35:44,350
And we want to find this
absorption probability, the

650
00:35:44,350 --> 00:35:46,820
probability that we end
up here for the

651
00:35:46,820 --> 00:35:48,660
different initial states.

652
00:35:48,660 --> 00:35:52,100
Now for some initial states this
is very easy to answer.

653
00:35:52,100 --> 00:35:55,280
If you start at state 4, what's
the probability that

654
00:35:55,280 --> 00:35:59,160
eventually you end up in
this part of the chain?

655
00:35:59,160 --> 00:35:59,820
It's 1.

656
00:35:59,820 --> 00:36:02,585
You're certain to be there,
that's where you started.

657
00:36:02,585 --> 00:36:06,140
If you start at state 5, what's
the probability that

658
00:36:06,140 --> 00:36:08,880
you end up eventually
at state 4?

659
00:36:08,880 --> 00:36:12,580
It's probability 0, there's
no way to get there.

660
00:36:12,580 --> 00:36:17,930
Now, how about if you start
at a state like state 2?

661
00:36:17,930 --> 00:36:20,840

662
00:36:20,840 --> 00:36:26,350
If you start at state 2 then
there's a few different things

663
00:36:26,350 --> 00:36:27,840
that can happen.

664
00:36:27,840 --> 00:36:32,900
Either you end up at state 4
right away and this happens

665
00:36:32,900 --> 00:36:41,150
with probability 0.2, or you
end up at state 1, and this

666
00:36:41,150 --> 00:36:46,730
happens with probability 0.6.

667
00:36:46,730 --> 00:36:50,200
So if you end up at state
4, you are done.

668
00:36:50,200 --> 00:36:51,540
We are there.

669
00:36:51,540 --> 00:36:56,850
If you end up at state
1, then what?

670
00:36:56,850 --> 00:37:00,980
Starting from state 1 there's
two possibilities.

671
00:37:00,980 --> 00:37:05,360
Either eventually you're going
to end up at state 4, or

672
00:37:05,360 --> 00:37:10,010
eventually you're going
to end up at state 5.

673
00:37:10,010 --> 00:37:14,310
What's the probability
of this happening?

674
00:37:14,310 --> 00:37:19,680
We don't know what it is, but
it's what we defined to be a1.

675
00:37:19,680 --> 00:37:21,130
This is the probability --

676
00:37:21,130 --> 00:37:22,650
a1 is the probability --

677
00:37:22,650 --> 00:37:26,650
that eventually you settle in
state 4 given that the initial

678
00:37:26,650 --> 00:37:28,090
state was 1.

679
00:37:28,090 --> 00:37:30,580
So this probability is a1.

680
00:37:30,580 --> 00:37:34,510
So our event of interest
can happen in two ways.

681
00:37:34,510 --> 00:37:38,350
Either I go there directly,
or I go here

682
00:37:38,350 --> 00:37:39,710
with probability 0.6.

683
00:37:39,710 --> 00:37:43,730
And given that I go there,
eventually I end up at state

684
00:37:43,730 --> 00:37:46,540
4, which happens with
probability a1.

685
00:37:46,540 --> 00:37:52,140
So the total probability of
ending up at state 4 is going

686
00:37:52,140 --> 00:37:54,660
to be the sum of the
probabilities of the different

687
00:37:54,660 --> 00:37:57,230
ways that this event
can happen.

688
00:37:57,230 --> 00:38:04,250
So our equation, in this case,
is going to be, that's a2, is

689
00:38:04,250 --> 00:38:07,420
going to be 0.2 (that's the
probability of going there

690
00:38:07,420 --> 00:38:11,210
directly) plus with probability
0.8 I end up at

691
00:38:11,210 --> 00:38:17,160
state 1, and then from state 1
I will end up at state 4 with

692
00:38:17,160 --> 00:38:19,320
probability a1.

693
00:38:19,320 --> 00:38:25,330
So this is one particular
equation that we've got for

694
00:38:25,330 --> 00:38:28,560
what happens if we start
from this state.

695
00:38:28,560 --> 00:38:32,450
We can do a similar argument
starting from any other state.

696
00:38:32,450 --> 00:38:35,790
Starting from state i the
probability that eventually I

697
00:38:35,790 --> 00:38:39,960
end up at state 4 is, we
consider the different

698
00:38:39,960 --> 00:38:43,350
possible scenarios of where do
I go next, which is my state

699
00:38:43,350 --> 00:38:46,440
j, with probability Pij.

700
00:38:46,440 --> 00:38:50,980
Next time I go to j, and given
that I started at j, this is

701
00:38:50,980 --> 00:38:53,630
the probability that I
end up at state 4.

702
00:38:53,630 --> 00:38:56,920
So this equation that we have
here is just an abstract

703
00:38:56,920 --> 00:39:02,540
version in symbols of what we
wrote down for the particular

704
00:39:02,540 --> 00:39:04,540
case where the initial
state was 2.

705
00:39:04,540 --> 00:39:08,880
So you write down an equation
of this type for every state

706
00:39:08,880 --> 00:39:09,580
inside here.

707
00:39:09,580 --> 00:39:15,520
You'll have a separate equation
for a1, a2, and a3.

708
00:39:15,520 --> 00:39:19,200
And that's going to be a system
of 3 equations with 3

709
00:39:19,200 --> 00:39:25,250
unknowns, the a's inside
the transient states.

710
00:39:25,250 --> 00:39:29,190
So you can solve that 3 by
3 system of equations.

711
00:39:29,190 --> 00:39:33,590
Fortunately, it turns out to
have a unique solution, and so

712
00:39:33,590 --> 00:39:36,070
once you solve it you have found
the probabilities of

713
00:39:36,070 --> 00:39:40,880
absorption and the probability
that eventually you get

714
00:39:40,880 --> 00:39:42,610
absorbed at state 4.

715
00:39:42,610 --> 00:39:51,540

716
00:39:51,540 --> 00:39:56,790
Now, in the picture that we had
here, this was a single

717
00:39:56,790 --> 00:39:59,570
state, and that one was
a single state.

718
00:39:59,570 --> 00:40:06,500
How do things change if our
recurrent, or trapping sets

719
00:40:06,500 --> 00:40:09,600
consist of multiple states?

720
00:40:09,600 --> 00:40:16,820
Well, it doesn't really matter
that we have multiple states.

721
00:40:16,820 --> 00:40:21,470
All that matters is that this
is one lump and once we get

722
00:40:21,470 --> 00:40:24,150
there we are stuck in there.

723
00:40:24,150 --> 00:40:33,170
So if the picture was, let's
say, like this, 0.1 and 0.2,

724
00:40:33,170 --> 00:40:36,960
that basically means that
whenever you are in that state

725
00:40:36,960 --> 00:40:42,620
there's a total probability of
0.3 of ending in that lump and

726
00:40:42,620 --> 00:40:45,100
getting stuck inside
that lump.

727
00:40:45,100 --> 00:40:50,660
So you would take that picture
and change it and make it

728
00:40:50,660 --> 00:40:57,330
instead a total probability of
0.3, of ending somewhere

729
00:40:57,330 --> 00:41:00,270
inside that lump.

730
00:41:00,270 --> 00:41:03,670
And similarly, you take this
lump and you view it as just

731
00:41:03,670 --> 00:41:07,570
one entity, and from any state
you record the total

732
00:41:07,570 --> 00:41:10,090
probability that given
that I'm here I

733
00:41:10,090 --> 00:41:12,140
end up in that entity.

734
00:41:12,140 --> 00:41:15,540
So basically, if the only
thing you care is the

735
00:41:15,540 --> 00:41:18,760
probability that you're going
to end up in this lump, you

736
00:41:18,760 --> 00:41:22,830
can replace that lump with a
single state, view it as a

737
00:41:22,830 --> 00:41:24,600
single state, and calculate

738
00:41:24,600 --> 00:41:26,180
probabilities using this formula.

739
00:41:26,180 --> 00:41:33,830

740
00:41:33,830 --> 00:41:36,780
All right, so now we know
where the chain is

741
00:41:36,780 --> 00:41:37,860
going to get to.

742
00:41:37,860 --> 00:41:39,850
At least we know
probabilistically.

743
00:41:39,850 --> 00:41:42,690
We know with what probability
it is going to go here, and

744
00:41:42,690 --> 00:41:45,160
that also tells us the
probability that eventually

745
00:41:45,160 --> 00:41:47,160
it's going to get there.

746
00:41:47,160 --> 00:41:55,780
Other question, how long is it
going to take until we get to

747
00:41:55,780 --> 00:41:58,300
either this state
or that state?

748
00:41:58,300 --> 00:42:02,160
We can call that event
absorption, meaning that the

749
00:42:02,160 --> 00:42:05,130
state got somewhere into a
recurrent class from which it

750
00:42:05,130 --> 00:42:06,380
could not get out.

751
00:42:06,380 --> 00:42:12,340

752
00:42:12,340 --> 00:42:12,510
Okay.

753
00:42:12,510 --> 00:42:15,570
Let's deal with that question
for the case where we have

754
00:42:15,570 --> 00:42:19,320
only 1 absorbing state.

755
00:42:19,320 --> 00:42:22,410
So here our Markov chain is a
little simpler than the one in

756
00:42:22,410 --> 00:42:23,580
the previous slide.

757
00:42:23,580 --> 00:42:25,870
We've got our transient
states, we've got our

758
00:42:25,870 --> 00:42:28,930
recurrent state, and once you
get into the recurrent state

759
00:42:28,930 --> 00:42:31,770
you just stay there.

760
00:42:31,770 --> 00:42:35,330
So here we're certain that no
matter where we start we're

761
00:42:35,330 --> 00:42:37,270
going to end up here.

762
00:42:37,270 --> 00:42:39,170
How long is it going to take?

763
00:42:39,170 --> 00:42:40,830
Well, we don't know.

764
00:42:40,830 --> 00:42:42,540
It's a random variable.

765
00:42:42,540 --> 00:42:44,660
The expected value of that
random variable,

766
00:42:44,660 --> 00:42:46,890
let's call it mu.

767
00:42:46,890 --> 00:42:50,510
But how long it takes to get
there certainly depends on

768
00:42:50,510 --> 00:42:52,470
where we start.

769
00:42:52,470 --> 00:42:56,390
So let's put in our notation
again this index i that

770
00:42:56,390 --> 00:42:59,220
indicates where we
started from.

771
00:42:59,220 --> 00:43:03,300
And now the argument is going to
be of the same type as the

772
00:43:03,300 --> 00:43:05,480
one we used before.

773
00:43:05,480 --> 00:43:11,620
We can think in terms of a tree
once more, that considers

774
00:43:11,620 --> 00:43:13,930
all the possible options.

775
00:43:13,930 --> 00:43:16,300
So suppose that you
start at state 1.

776
00:43:16,300 --> 00:43:19,060

777
00:43:19,060 --> 00:43:24,110
Starting from state 1, the
expected time until you end up

778
00:43:24,110 --> 00:43:27,100
dropping states is mu1.

779
00:43:27,100 --> 00:43:30,700
Now, starting from state 1, what
are the possibilities?

780
00:43:30,700 --> 00:43:33,090
You make your first transition,
and that first

781
00:43:33,090 --> 00:43:36,550
transition is going to take
you either to state

782
00:43:36,550 --> 00:43:38,610
2 or to state 3.

783
00:43:38,610 --> 00:43:42,010
It takes you to state 2 with
probability 0.6, it takes you

784
00:43:42,010 --> 00:43:48,320
to state 3 with probability
0.4.

785
00:43:48,320 --> 00:43:52,490
Starting from state 2,
eventually you're going to get

786
00:43:52,490 --> 00:43:54,410
to state 4.

787
00:43:54,410 --> 00:43:55,990
How long does it take?

788
00:43:55,990 --> 00:43:58,480
We don't know, it's
a random variable.

789
00:43:58,480 --> 00:44:02,325
But the expected time until
this happens is mu2.

790
00:44:02,325 --> 00:44:05,100

791
00:44:05,100 --> 00:44:08,250
Starting from state 2, how long
does it take you to get

792
00:44:08,250 --> 00:44:10,300
to state 4.

793
00:44:10,300 --> 00:44:13,870
And similarly starting from
state 3, it's going to take

794
00:44:13,870 --> 00:44:17,850
you on the average mu3
time steps until you

795
00:44:17,850 --> 00:44:20,070
get to state 4.

796
00:44:20,070 --> 00:44:24,450
So what's the expected value
of the time until I

797
00:44:24,450 --> 00:44:25,790
end at state 4?

798
00:44:25,790 --> 00:44:31,070

799
00:44:31,070 --> 00:44:37,910
So with probability 0.6, I'm
going to end up at state 2 and

800
00:44:37,910 --> 00:44:42,650
from there on it's going to be
expected time mu2, and with

801
00:44:42,650 --> 00:44:45,850
probability 0.4 I'm going to
end up at state 3, and from

802
00:44:45,850 --> 00:44:48,730
there it's going to take
me so much time.

803
00:44:48,730 --> 00:44:52,790
So this is the expected time
it's going to take me after

804
00:44:52,790 --> 00:44:54,860
the first transition.

805
00:44:54,860 --> 00:44:59,420
But we also spent 1 time step
for the first transition.

806
00:44:59,420 --> 00:45:02,610
The total time to get there
is the time of the first

807
00:45:02,610 --> 00:45:06,330
transition, which is 1, plus
the expected time starting

808
00:45:06,330 --> 00:45:07,600
from the next state.

809
00:45:07,600 --> 00:45:10,210
This expression here is the
expected time starting from

810
00:45:10,210 --> 00:45:13,640
the next state, but we also need
to account for the first

811
00:45:13,640 --> 00:45:15,840
transition, so we add 1.

812
00:45:15,840 --> 00:45:18,010
And this is going
to be our mu1.

813
00:45:18,010 --> 00:45:21,150

814
00:45:21,150 --> 00:45:25,370
So once more we have a linear
equation that ties together

815
00:45:25,370 --> 00:45:27,930
the different mu's.

816
00:45:27,930 --> 00:45:32,070
And the equation starting from
state 4 in this case, of

817
00:45:32,070 --> 00:45:35,620
course is going to be simple,
starting from that state the

818
00:45:35,620 --> 00:45:38,280
expected number of steps it
takes you to get there for the

819
00:45:38,280 --> 00:45:42,220
first time is of course, 0
because you're already there.

820
00:45:42,220 --> 00:45:45,640
So for that state this is fine,
and for all the other

821
00:45:45,640 --> 00:45:48,280
states you get an equation
of this form.

822
00:45:48,280 --> 00:45:51,410
Now we're going to have an
equation for every state.

823
00:45:51,410 --> 00:45:53,760
It's a system of linear
equations, once more we can

824
00:45:53,760 --> 00:45:57,290
solve them, and this gives us
the expected times until our

825
00:45:57,290 --> 00:46:03,890
chain gets absorbed in
this absorbing state.

826
00:46:03,890 --> 00:46:06,650
And it's nice to know that
this system of equations

827
00:46:06,650 --> 00:46:09,510
always has a unique solution.

828
00:46:09,510 --> 00:46:13,220
OK so this was the expected
time to absorption.

829
00:46:13,220 --> 00:46:16,720
For this case where we had this
scene absorbing state.

830
00:46:16,720 --> 00:46:23,740
Suppose that we have our
transient states and that we

831
00:46:23,740 --> 00:46:27,020
have multiple recurrent
classes, or

832
00:46:27,020 --> 00:46:28,590
multiple absorbing states.

833
00:46:28,590 --> 00:46:33,850

834
00:46:33,850 --> 00:46:37,470
Suppose you've got the
picture like this.

835
00:46:37,470 --> 00:46:42,040
And we want to calculate the
expected time until we get

836
00:46:42,040 --> 00:46:44,110
here or there.

837
00:46:44,110 --> 00:46:47,830
Expected time until we get
to an absorbing state.

838
00:46:47,830 --> 00:46:50,320
What's the trick?

839
00:46:50,320 --> 00:46:55,010
Well, we can lump both of these
states together and

840
00:46:55,010 --> 00:47:00,660
think of them as just one bad
state, one place for which

841
00:47:00,660 --> 00:47:03,730
we're interested in how long
it takes us to get there.

842
00:47:03,730 --> 00:47:10,250
So lump them as one state, and
accordingly kind of merge all

843
00:47:10,250 --> 00:47:11,570
of those probabilities.

844
00:47:11,570 --> 00:47:15,470
So starting from here, my
probability that the next I

845
00:47:15,470 --> 00:47:18,690
end up in this lump and they
get absorbed is going to be

846
00:47:18,690 --> 00:47:21,630
this probability plus
that probability.

847
00:47:21,630 --> 00:47:24,010
So we would change
that picture.

848
00:47:24,010 --> 00:47:30,080
Think of this as being
just one big state.

849
00:47:30,080 --> 00:47:36,080
And sort of add those two
probabilities together to come

850
00:47:36,080 --> 00:47:39,360
up with a single probability,
which is the probability that

851
00:47:39,360 --> 00:47:42,590
starting from here next
time I find myself at

852
00:47:42,590 --> 00:47:44,440
some absorbing state.

853
00:47:44,440 --> 00:47:48,510
So once you know how to deal
with a situation like this,

854
00:47:48,510 --> 00:47:52,030
you can also find expected times
to absorption for the

855
00:47:52,030 --> 00:47:55,190
case where you've got multiple
absorbing states.

856
00:47:55,190 --> 00:47:57,990
You just lump all of those
multiple absorbing states into

857
00:47:57,990 --> 00:47:59,240
a single one.

858
00:47:59,240 --> 00:48:01,550

859
00:48:01,550 --> 00:48:05,040
Finally, there's a
kind of related

860
00:48:05,040 --> 00:48:09,600
quantity that's of interest.

861
00:48:09,600 --> 00:48:13,480
The question is almost the same
as in the previous slide,

862
00:48:13,480 --> 00:48:16,950
except that here we do not have
any absorbing states.

863
00:48:16,950 --> 00:48:20,335
Rather, we have a single
recurrent class of states.

864
00:48:20,335 --> 00:48:25,320

865
00:48:25,320 --> 00:48:27,840
You start at some state i.

866
00:48:27,840 --> 00:48:31,780
You have a special state,
that is state s.

867
00:48:31,780 --> 00:48:35,600
And you ask the question, how
long is it going to take me

868
00:48:35,600 --> 00:48:39,040
until I get to s for
the first time?

869
00:48:39,040 --> 00:48:41,140
It's a single recurrent
class of states.

870
00:48:41,140 --> 00:48:43,730
So you know that the state keeps
circulating here and it

871
00:48:43,730 --> 00:48:46,610
keeps visiting all of
the possible states.

872
00:48:46,610 --> 00:48:49,510
So eventually this state
will be visited.

873
00:48:49,510 --> 00:48:52,070
How long does it take
for this to happen?

874
00:48:52,070 --> 00:48:55,290

875
00:48:55,290 --> 00:48:55,425
Ok.

876
00:48:55,425 --> 00:48:58,820
So we're interested in how
long it takes for this to

877
00:48:58,820 --> 00:49:02,080
happen, how long it takes until
we get to s for the

878
00:49:02,080 --> 00:49:03,000
first time.

879
00:49:03,000 --> 00:49:06,480
And we don't care about what
happens afterwards.

880
00:49:06,480 --> 00:49:10,110
So we might as well change this
picture and remove the

881
00:49:10,110 --> 00:49:15,470
transitions out of s and to make
them self transitions.

882
00:49:15,470 --> 00:49:18,140
Is the answer going to change?

883
00:49:18,140 --> 00:49:19,280
No.

884
00:49:19,280 --> 00:49:22,890
The only thing that we changed
was what happens

885
00:49:22,890 --> 00:49:25,760
after you get to s.

886
00:49:25,760 --> 00:49:28,810
But what happens after you
get to s doesn't matter.

887
00:49:28,810 --> 00:49:31,730
The question we're dealing with
is how long does it take

888
00:49:31,730 --> 00:49:33,900
us to get to s.

889
00:49:33,900 --> 00:49:37,010
So essentially, it's after we
do this transformation --

890
00:49:37,010 --> 00:49:40,740
it's the same question as
before, what's the time it

891
00:49:40,740 --> 00:49:43,990
takes until eventually
we hit this state.

892
00:49:43,990 --> 00:49:46,390
And it's now in this new
picture, this state is an

893
00:49:46,390 --> 00:49:48,410
absorbing state.

894
00:49:48,410 --> 00:49:50,710
Or you can just think from
first principles.

895
00:49:50,710 --> 00:49:55,720
Starting from the state itself,
s, it takes you 0 time

896
00:49:55,720 --> 00:49:57,430
steps until you get to s.

897
00:49:57,430 --> 00:50:01,610
Starting from anywhere else,
you need one transition and

898
00:50:01,610 --> 00:50:05,050
then after the first transition
you find yourself

899
00:50:05,050 --> 00:50:09,560
at state j with probability Pij
and from then on you are

900
00:50:09,560 --> 00:50:13,350
going to take expected time
Tj until you get to that

901
00:50:13,350 --> 00:50:14,710
terminal state s.

902
00:50:14,710 --> 00:50:17,340
So once more these equations
have a unique solution, you

903
00:50:17,340 --> 00:50:19,510
can solve them and
find the answer.

904
00:50:19,510 --> 00:50:22,530
And finally, there's a related
question, which is the mean

905
00:50:22,530 --> 00:50:24,210
recurrence time of s.

906
00:50:24,210 --> 00:50:30,490
In that question you start
at s, the chain will move

907
00:50:30,490 --> 00:50:34,090
randomly, and you ask how long
is it going to take until I

908
00:50:34,090 --> 00:50:37,160
come back to s for
the next time.

909
00:50:37,160 --> 00:50:38,390
So notice the difference.

910
00:50:38,390 --> 00:50:42,670
Here we're talking the first
time after time 0, whereas

911
00:50:42,670 --> 00:50:45,500
here it's just the first
time anywhere.

912
00:50:45,500 --> 00:50:51,250
So here if you start from
s, Ts* is not 0.

913
00:50:51,250 --> 00:50:54,450
You want to do at least one
transition and that's how long

914
00:50:54,450 --> 00:50:57,360
it's going to take me until
it gets back to s.

915
00:50:57,360 --> 00:51:00,900
Well, how long does it take
me until I get back to s?

916
00:51:00,900 --> 00:51:06,060
I do my first transition, and
then after my first transition

917
00:51:06,060 --> 00:51:11,920
I calculate the expected time
from the next state how long

918
00:51:11,920 --> 00:51:15,350
it's going to take me until
I come back to s.

919
00:51:15,350 --> 00:51:20,200
So all of these equations that I
wrote down, they all kind of

920
00:51:20,200 --> 00:51:22,500
look the same.

921
00:51:22,500 --> 00:51:23,620
But they are different.

922
00:51:23,620 --> 00:51:27,210
So you can either memorize all
of these equations, or instead

923
00:51:27,210 --> 00:51:30,580
what's better is to just
to get the basic idea.

924
00:51:30,580 --> 00:51:33,140
That is, to calculate
probabilities or expected

925
00:51:33,140 --> 00:51:35,830
values you use the total
probability or total

926
00:51:35,830 --> 00:51:38,260
expectation theorem and
conditional the first

927
00:51:38,260 --> 00:51:40,810
transition and take
it from there.

928
00:51:40,810 --> 00:51:43,230
So you're going to get a little
bit of practice with

929
00:51:43,230 --> 00:51:47,276
these skills in recitation
tomorrow, and of course it's

930
00:51:47,276 --> 00:51:48,730
in your problem set as well.

931
00:51:48,730 --> 00:51:49,980