| Title:
|
Solutions of semi-Markov control models with recursive discount rates and approximation by $\epsilon$-optimal policies (English) |
| Author:
|
García, Yofre H. |
| Author:
|
González-Hernández, Juan |
| Language:
|
English |
| Journal:
|
Kybernetika |
| ISSN:
|
0023-5954 (print) |
| ISSN:
|
1805-949X (online) |
| Volume:
|
55 |
| Issue:
|
3 |
| Year:
|
2019 |
| Pages:
|
495-517 |
| Summary lang:
|
English |
| . |
| Category:
|
math |
| . |
| Summary:
|
This paper studies a class of discrete-time discounted semi-Markov control model on Borel spaces. We assume possibly unbounded costs and a non-stationary exponential form in the discount factor which depends of on a rate, called the discount rate. Given an initial discount rate the evolution in next steps depends on both the previous discount rate and the sojourn time of the system at the current state. The new results provided here are the existence and the approximation of optimal policies for this class of discounted Markov control model with non-stationary rates and the horizon is finite or infinite. Under regularity condition on sojourn time distributions and measurable selector conditions, we show the validity of the dynamic programming algorithm for the finite horizon case. By the convergence in finite steps to the value functions, we guarantee the existence of non-stationary optimal policies for the infinite horizon case and we approximate them using non-stationary $\epsilon$-optimal policies. We illustrated our results a discounted semi-Markov linear-quadratic model, when the evolution of the discount rate follows an appropriate type of stochastic differential equation. (English) |
| Keyword:
|
optimal stochastic control |
| Keyword:
|
dynamic programming method |
| Keyword:
|
semi-Markov processes |
| MSC:
|
49L20 |
| MSC:
|
93E20 |
| idZBL:
|
Zbl 07144950 |
| idMR:
|
MR4015995 |
| DOI:
|
10.14736/kyb-2019-3-0495 |
| . |
| Date available:
|
2019-11-14T08:37:06Z |
| Last updated:
|
2023-08-22 |
| Stable URL:
|
http://hdl.handle.net/10338.dmlcz/147862 |
| . |
| Reference:
|
[1] Arnold, L.: Stochastic Differential Equations..John Wiley and Sons, New York 1973. MR 0443083 |
| Reference:
|
[2] Ash, R., Doléans-Dade, C.: Probability and Measure Theory..Academic Press, San Diego, 2000. Zbl 0944.60004, MR 1810041 |
| Reference:
|
[3] Bhattacharya, R., Majumdar, M.: Controlled semi-Markov models - the discounted case..J. Statist. Plann. Inference 21 (1989), 3, 365-381. MR 0995606, 10.1016/0378-3758(89)90053-0 |
| Reference:
|
[4] Bertsekas, D., Shreve, S.: Stochastic Optimal Control: The Discrete Time Case..Athena Scientific, Belmont, Massachusetts 1996. Zbl 0633.93001, MR 0809588 |
| Reference:
|
[5] Blackwell, D.: Discounted dynamic programming..Ann. Math. Statist. 36, (1965), 226-235. MR 0173536, 10.1214/aoms/1177700285 |
| Reference:
|
[6] Cani, J. De: A dynamic programming algorithm for embedded Markov chains the planning horizon is infinitely..Management. Sci. 10 (1963), 716-733. MR 0169690, 10.1287/mnsc.10.4.716 |
| Reference:
|
[7] Drenyovszki, R., Kovács, L., Tornai, K., Oláh, A., I., I. Pintér: Bottom-up modeling of domestic appliances with Markov chains and semi-Markov processes..Kybernetika 53 (2017), 6, 1100-1117. 10.14736/kyb-2017-6-1100 |
| Reference:
|
[8] Dekker, R., Hordijk, A.: Denumerable semi-Markov decision chains with small interest rates..Ann. Oper. Res. 28 (1991), 185-212. MR 1105173, 10.1007/bf02055581 |
| Reference:
|
[9] García, Y., González-Hernández, J.: Discrete-time Markov control process with recursive discounted rates..Kybernetika 52 (2016), 403-426. MR 3532514, 10.14736/kyb-2016-3-0403 |
| Reference:
|
[10] González-Hernández, J., López-Martínez, R., Pérez-Hernández, J.: Markov control processes with randomized discounted cost..Math. Meth. Oper. Res. 65 (2006), 27-44. Zbl 1126.90075, MR 2302022, 10.1007/s00186-006-0092-2 |
| Reference:
|
[11] González-Hernández, J., Villarreal-Rodríguez, C.: Optimal solutions of constrained discounted semi-Markov control problems. |
| Reference:
|
[12] Hernández-Lerma, O., Lasserre, J.: Discrete-Time Markov Control Processes. Basic Optimality Criteria..Springer-Verlag, New York 1996. Zbl 0840.93001, MR 1363487, 10.1007/978-1-4612-0729-0_1 |
| Reference:
|
[13] Hu, Q., Yue, W.: Markov Decision Processes With Their Applications..Springer-Verlag, Advances in Mechanics and Mathematics book series 14, (2008). MR 2361223, 10.14736/kyb-2017-1-0059 |
| Reference:
|
[14] Huang, X., Huang, Y.: Mean-variance optimality for semi-Markov decision processes under first passage criteria..Kybernetika 53 (2017), 1, 59-81. MR 3638556, 10.14736/kyb-2017-1-0059 |
| Reference:
|
[15] Howard, R.: Semi-Markovian decision processes..Bull. Int. Statist. Inst. 40 (1963), 2, 625-652. MR 0173545 |
| Reference:
|
[16] Jewell, W.: Markov-renewal programming I: formulation, finite return models, Markov-renewal programming II: infinite return models, example..Oper. Res. 11 (1963), 938-971. MR 0163375, 10.1287/opre.11.6.938 |
| Reference:
|
[17] Luque-Vázquez, F., Hernández-Lerma, O.: Semi-Markov control models with average costs..Appl. Math. 26 (1999), 315-331. MR 1726636, 10.4064/am-26-3-315-331 |
| Reference:
|
[18] Luque-Vásquez, F., Minjárez-Sosa, J. A.: Semi-Markov control processes with unknown holding times distribution under a discounted criterion..Math. Methods Oper. Res. 61 (2005), 455-468. MR 2225824, 10.1007/s001860400406 |
| Reference:
|
[19] Luque-Vásquez, F., Minjárez-Sosa, J., Rosas, L.: Semi-Markov control processes with unknown holding times distribution under an average cost criterion..Appl. Math. Optim. 61, (2010), 317-336. MR 2609593, 10.1007/s00245-009-9086-9 |
| Reference:
|
[20] Schweitzer, P.: Perturbation Theory and Markovian Decision Processes..Ph.D. Dissertation, Massachusetts Institute of Technology, 1965. MR 2939613 |
| Reference:
|
[21] Vasicek, O.: An equilibrium characterization of the term structure..J. Financ. Econom. 5 (1977), 177-188. 10.1016/0304-405x(77)90016-2 |
| Reference:
|
[22] Vega-Amaya, O.: Average optimatily in semi-Markov control models on Borel spaces: unbounded costs and control..Bol. Soc. Mat. Mexicana 38 (1997), 2, 47-60. MR 1313106 |
| Reference:
|
[23] Zagst, R.: The effect of information in separable Bayesian semi-Markov control models and its application to investment planning..ZOR - Math. Methods Oper. Res. 41 (1995), 277-288. MR 1339493, 10.1007/BF01432360 |
| . |
