# Markov Chains

Topics: Markov chain, Queueing theory, Probability theory Pages: 15 (3550 words) Published: January 8, 2013
A Review of Management Science Applications of Discrete
Time Markov Chains

Mehmet Emre GÜLER
*

*

Mehmet Emre Güler, Dokuz Eylul University, Faculty of Economics and Administrative Science, Department of Business Administration, Dokuzcesmeler Campus, 35160, Buca – Izmir/TURKEY E-mail: emre.guler@deu.edu.tr

A Review of Management Science Applications of Discrete
Time Markov Chains
Abstract
A short review of applications of Discrete-Time Markov Chains (DTMCs) is provided in this paper. Rather than covering the whole literature, primarily, we concentrate on applications in Management Science / Operations Research (MS/OR) literature. After citing some representative applications in other disciplines, we review the MS/OR applications of DTMCs with grouping the subject studies into four major areas. These areas include: Queuing Theory applications, Inventory Management applications, Manufacturing Systems applications and Other / General applications. Keywords: Discrete-Time Markov Chains, Stochastic Process

1. Introduction
Consider a finite (or infinitely countable) state stochastic process, where the underlying random variables are observed at discrete time points. When the future stochastic behavior of the process is dependent only on the present state, in other words, if it possesses the well-known Markovian property, then the process represents a DTMC. DTMCs are popular tools for modeling and analyzing complex systems, under uncertainty. We are often concerned with the steady state probabilistic behavior of the system and/or research questions regarding the performance of the system, such as: fraction of time that a server is idle, the probability of reaching a certain state, long-run expected number of lost customers, average occupancy of the system etc.

1
There is a huge literature regarding applications of DTMCs in a wide range of disciplines. This stems from the fact that DTMCs provide us with a very useful tool, applicable to many areas in analyzing the important behaviors of systems of interest. For example, consider the first passage time information (sometimes known as hitting time), which can be revealed with using DTMC methodology. For classical gambler’s ruin type models this variable can represent the duration of the game, for physics this variable is the time period for a particle in a system reaching some state, for reliability engineering studies the expectation of this variable corresponds to the mean time for the first failure and its survivor function is regarded as the reliability of a system under study. As seen from these basic examples, DTMCs are a popular area of interest for many researchers from several different disciplines. Table 1 provides a short summary of some representative applications using DTMCs as a primarily research tool in different originating areas. As mentioned before we’ll mainly focus on MS/OR applications, therefore, any review about these studies are beyond the aim of this paper.

2. Management Science / Operations Research Applications
DTMCs attracted a considerable amount of attention in MS/OR area. For simplicity and a tidy representation we grouped these studies into four categories. These are: Queuing Theory, Inventory Management, Manufacturing Systems and General and other applications; specifically, Project Management, Discrete Event System Modeling, Markov Chain Theory, Risk Analysis, and, Finance and Times Series Analysis/ Forecasting.

2
2.1. Queuing Theory Applications
He [2005], associates a Markov chain with the age of the customer batch already in service and analyzes sojourn times and waiting times in a discrete time queue. Although his idea is used in the past for continuous time studies, the...