Article
Full entry |
PDF
(0.4 MB)
Feedback
PDF
(0.4 MB)
Feedback
Keywords:
optimal stochastic control; dynamic programming method; semi-Markov processes
optimal stochastic control; dynamic programming method; semi-Markov processes
Summary:
This paper studies a class of discrete-time discounted semi-Markov control model on Borel spaces. We assume possibly unbounded costs and a non-stationary exponential form in the discount factor which depends of on a rate, called the discount rate. Given an initial discount rate the evolution in next steps depends on both the previous discount rate and the sojourn time of the system at the current state. The new results provided here are the existence and the approximation of optimal policies for this class of discounted Markov control model with non-stationary rates and the horizon is finite or infinite. Under regularity condition on sojourn time distributions and measurable selector conditions, we show the validity of the dynamic programming algorithm for the finite horizon case. By the convergence in finite steps to the value functions, we guarantee the existence of non-stationary optimal policies for the infinite horizon case and we approximate them using non-stationary $\epsilon$-optimal policies. We illustrated our results a discounted semi-Markov linear-quadratic model, when the evolution of the discount rate follows an appropriate type of stochastic differential equation.
References:
[1] Arnold, L.: Stochastic Differential Equations. John Wiley and Sons, New York 1973. MR 0443083
[2] Ash, R., Doléans-Dade, C.: Probability and Measure Theory. Academic Press, San Diego, 2000. MR 1810041 | Zbl 0944.60004
[3] Bhattacharya, R., Majumdar, M.: Controlled semi-Markov models - the discounted case. J. Statist. Plann. Inference 21 (1989), 3, 365-381. DOI 10.1016/0378-3758(89)90053-0 | MR 0995606
[4] Bertsekas, D., Shreve, S.: Stochastic Optimal Control: The Discrete Time Case. Athena Scientific, Belmont, Massachusetts 1996. MR 0809588 | Zbl 0633.93001
[5] Blackwell, D.: Discounted dynamic programming. Ann. Math. Statist. 36, (1965), 226-235. DOI 10.1214/aoms/1177700285 | MR 0173536
[6] Cani, J. De: A dynamic programming algorithm for embedded Markov chains the planning horizon is infinitely. Management. Sci. 10 (1963), 716-733. DOI 10.1287/mnsc.10.4.716 | MR 0169690
[7] Drenyovszki, R., Kovács, L., Tornai, K., Oláh, A., I., I. Pintér: Bottom-up modeling of domestic appliances with Markov chains and semi-Markov processes. Kybernetika 53 (2017), 6, 1100-1117. DOI 10.14736/kyb-2017-6-1100
[8] Dekker, R., Hordijk, A.: Denumerable semi-Markov decision chains with small interest rates. Ann. Oper. Res. 28 (1991), 185-212. DOI 10.1007/bf02055581 | MR 1105173
[9] García, Y., González-Hernández, J.: Discrete-time Markov control process with recursive discounted rates. Kybernetika 52 (2016), 403-426. DOI 10.14736/kyb-2016-3-0403 | MR 3532514
[10] González-Hernández, J., López-Martínez, R., Pérez-Hernández, J.: Markov control processes with randomized discounted cost. Math. Meth. Oper. Res. 65 (2006), 27-44. DOI 10.1007/s00186-006-0092-2 | MR 2302022 | Zbl 1126.90075
[11] González-Hernández, J., Villarreal-Rodríguez, C.: Optimal solutions of constrained discounted semi-Markov control problems.
[12] Hernández-Lerma, O., Lasserre, J.: Discrete-Time Markov Control Processes. Basic Optimality Criteria. Springer-Verlag, New York 1996. DOI 10.1007/978-1-4612-0729-0_1 | MR 1363487 | Zbl 0840.93001
[13] Hu, Q., Yue, W.: Markov Decision Processes With Their Applications. Springer-Verlag, Advances in Mechanics and Mathematics book series 14, (2008). DOI 10.14736/kyb-2017-1-0059 | MR 2361223
[14] Huang, X., Huang, Y.: Mean-variance optimality for semi-Markov decision processes under first passage criteria. Kybernetika 53 (2017), 1, 59-81. DOI 10.14736/kyb-2017-1-0059 | MR 3638556
[15] Howard, R.: Semi-Markovian decision processes. Bull. Int. Statist. Inst. 40 (1963), 2, 625-652. MR 0173545
[16] Jewell, W.: Markov-renewal programming I: formulation, finite return models, Markov-renewal programming II: infinite return models, example. Oper. Res. 11 (1963), 938-971. DOI 10.1287/opre.11.6.938 | MR 0163375
[17] Luque-Vázquez, F., Hernández-Lerma, O.: Semi-Markov control models with average costs. Appl. Math. 26 (1999), 315-331. DOI 10.4064/am-26-3-315-331 | MR 1726636
[18] Luque-Vásquez, F., Minjárez-Sosa, J. A.: Semi-Markov control processes with unknown holding times distribution under a discounted criterion. Math. Methods Oper. Res. 61 (2005), 455-468. DOI 10.1007/s001860400406 | MR 2225824
[19] Luque-Vásquez, F., Minjárez-Sosa, J., Rosas, L.: Semi-Markov control processes with unknown holding times distribution under an average cost criterion. Appl. Math. Optim. 61, (2010), 317-336. DOI 10.1007/s00245-009-9086-9 | MR 2609593
[20] Schweitzer, P.: Perturbation Theory and Markovian Decision Processes. Ph.D. Dissertation, Massachusetts Institute of Technology, 1965. MR 2939613
[21] Vasicek, O.: An equilibrium characterization of the term structure. J. Financ. Econom. 5 (1977), 177-188. DOI 10.1016/0304-405x(77)90016-2
[22] Vega-Amaya, O.: Average optimatily in semi-Markov control models on Borel spaces: unbounded costs and control. Bol. Soc. Mat. Mexicana 38 (1997), 2, 47-60. MR 1313106
[23] Zagst, R.: The effect of information in separable Bayesian semi-Markov control models and its application to investment planning. ZOR - Math. Methods Oper. Res. 41 (1995), 277-288. DOI 10.1007/BF01432360 | MR 1339493
Search
Partner of
