Article
Full entry |
PDF
(0.3 MB)
Feedback
PDF
(0.3 MB)
Feedback
Keywords:
Markov decision process; dynamic programming; varying discount factor; random horizon
Markov decision process; dynamic programming; varying discount factor; random horizon
Summary:
This paper is related to Markov Decision Processes. The optimal control problem is to minimize the expected total discounted cost, with a non-constant discount factor. The discount factor is time-varying and it could depend on the state and the action. Furthermore, it is considered that the horizon of the optimization problem is given by a discrete random variable, that is, a random horizon is assumed. Under general conditions on Markov control model, using the dynamic programming approach, an optimality equation for both cases is obtained, namely, finite support and infinite support of the random horizon. The obtained results are illustrated by two examples, one of them related to optimal replacement.
References:
[1] Carmon, Y., Shwartz, A.: Markov decision processes with exponentially representable discounting. Oper. Res. Lett. 37 (2009), 51-55. DOI 10.1016/j.orl.2008.10.005 | MR 2488083 | Zbl 1154.90610
[2] Chen, X., Yang, X.: Optimal consumption and investment problem with random horizon in a BMAP model. Insurance Math. Econom. 61 (2015), 197-205. DOI 10.1016/j.insmatheco.2015.01.004 | MR 3324056 | Zbl 1314.91192
[3] Cruz-Suárez, H., Ilhuicatzi-Roldán, R., Montes-de-Oca, R.: Markov decision processes on Borel spaces with total cost and random horizon. J. Optim. Theory Appl. 162 (2014), 329-346. DOI 10.1007/s10957-012-0262-8 | MR 3228530 | Zbl 1317.90316
[4] Vecchia, E. Della, Marco, S. Di, Vidal, F.: Dynamic programming for variable discounted Markov decision problems. In: Jornadas Argentinas de Informática e Investigación O\-pe\-ra\-ti\-va (43JAIIO) XII Simposio Argentino de Investigación Operativa (SIO), Buenos Aires 2014, pp. 50-62.
[5] Feinberg, E., Shwartz, A.: Constrained dynamic programming with two discount factors: applications and an algorithm. IEEE Trans. Automat. Control 44 (1999), 628-631. DOI 10.1109/9.751365 | MR 1680195 | Zbl 0957.90127
[6] Feinberg, E., Shwartz, A.: Markov decision models with weighted discounted criteria. Math. Oper. Res. 19 (1994), 152-168. DOI 10.1287/moor.19.1.152 | MR 1290017 | Zbl 0803.90123
[7] García, Y. H., González-Hernández, J.: Discrete-time Markov control process with recursive discounted rates. Kybernetika 52 (2016), 403-426. DOI 10.14736/kyb-2016-3-0403 | MR 3532514
[8] González-Hernández, J., López-Martínez, R. R., Minjarez-Sosa, J. A.: Adaptive policies for stochastic systems under a randomized discounted criterion. Bol. Soc. Mat. Mex. 14 (2008), 149-163. MR 2667162
[9] González-Hernández, J., López-Martínez, R. R., Minjarez-Sosa, J. A.: Approximation, estimation and control of stochastic systems under a randomized discounted cost criterion. Kybernetika 45 (2009), 737-754. MR 2599109 | Zbl 1190.93105
[10] González-Hernández, J., López-Martínez, R. R., Minjarez-Sosa, J. A., Gabriel-Arguelles, J. A.: Constrained Markov control processes with randomized discounted cost criteria: occupation measures and external points. Risk and Decision Analysis 4 (2013), 163-176.
[11] González-Hernández, J., López-Martínez, R. R., Minjarez-Sosa, J. A., Gabriel-Arguelles, J. A.: Constrained Markov control processes with randomized discounted rate: infinite linear programming approach. Optimal Control Appl. Methods 35 (2014), 575-591. DOI 10.1002/oca.2089 | MR 3262763
[12] González-Hernández, J., López-Martínez, R. R., Pérez-Hernández, J. R.: Markov control processes with randomized discounted cost. Math. Methods Oper. Res. 65 (2007), 27-44. DOI 10.1007/s00186-006-0092-2 | MR 2302022 | Zbl 1126.90075
[13] Guo, X., Hernández-del-Valle, A., Hernández-Lerma, O.: First passage problems for nonstationary discrete-time stochastic control systems. Eur. J. Control 18 (2012), 528-538. DOI 10.3166/ejc.18.528-538 | MR 3086896 | Zbl 1291.93328
[14] Hernández-Lerma, O., Laserre, J. B.: Discrete-time Markov Control Processes: Basic Optimality Criteria. Springer-Verlag, New York 1996. DOI 10.1007/978-1-4612-0729-0 | MR 1363487
[15] Hinderer, K.: Foundations of non-stationary dynamic programming with discrete time parameter. In: Lectures Notes Operations Research (M. Bechmann and H. Künzi, eds.), Springer-Verlag 33, Zürich 1970. DOI 10.1007/978-3-642-46229-0 | MR 0267890 | Zbl 0202.18401
[16] Ilhuicatzi-Roldán, R., Cruz-Suárez, H.: Optimal replacement in a system of $n$-machines with random horizon. Proyecciones 31 (2012), 219-233. DOI 10.4067/s0716-09172012000300003 | MR 2995551 | Zbl 1262.90050
[17] Minjares-Sosa, J. A.: Markov Control Models with unknown random state-action-dependent discounted factors. TOP 23 (2015), 743-772. DOI 10.1007/s11750-015-0360-5 | MR 3407674
[18] Puterman, M. L.: Markov Decision Process: Discrete Stochastic Dynamic Programming. John Wiley and Sons, New York 1994. MR 1270015
[19] Sch{ä}l, M.: Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal. Probab. Theory Related Fields 32 (1975), 179-196. DOI 10.1007/bf00532612 | MR 0378841 | Zbl 0316.90080
[20] Wei, Q., Guo, X.: Markov decision processes with state-dependent discounted factors and unbounded rewards/costs. Oper. Res. Lett. 39 (2011), 369-374. DOI 10.1016/j.orl.2011.06.014 | MR 2835530
[21] Wei, Q., Guo, X.: Semi-Markov decision processes with variance minimization criterion. 4OR, 13 (2015), 59-79. DOI 10.1007/s10288-014-0267-2 | MR 3323274 | Zbl 1310.93087
[22] Wu, X., Guo, X.: First passage optimality and variance minimisation of Markov decision processes with varying discounted factors. J. Appl. Probab. 52 (2015), 441-456. DOI 10.1017/s0021900200012560 | MR 3372085
[23] Wu, X., Zou, X., Guo, X.: First passage Markov decision processes with constraints and varying discount factors. Front. Math. China 10 (2015), 1005-1023. DOI 10.1007/s11464-015-0479-6 | MR 3352898 | Zbl 1317.90319
[24] Wu, X., Zhang, J.: An application to the finite approximation of the first passage models for discrete-time Markov decision processes with varying discount factors. In: Proc. 11th World Congress on Intelligent Control and Automation 2015, pp. 1745-1748. DOI 10.1109/wcica.2014.7052984 | MR 3163332
[25] Wu, X., Zhang, J.: Finite approximation of the first passage models for discrete-time Markov decision processes with varying discounted factors. Discrete Event Dyn. Syst. 26 (2016), 669-683. DOI 10.1007/s10626-014-0209-3 | MR 3557415
[26] Ye, L., Guo, X.: Continuous-time Markov decision processes with state-dependent discount factors. Acta Appl. Math. 121 (2012), 5-27. DOI 10.1007/s10440-012-9669-3 | MR 2966962 | Zbl 1281.90082
[27] Zhang, Y.: Convex analytic approach to constrained discounted Markov decision processes with non-constant discount factors. TOP 21 (2013), 378-408. DOI 10.1007/s11750-011-0186-8 | MR 3068494 | Zbl 1273.90235
Search
Partner of
