Markov chain for heart failure diagnostic decision model with irregular Thoracic Fluid Content measurements

Properties of results obtained on Markov processes analysis in medical applications have been illustrated. Calculation was done for measurement of Thoracic Fluid Content by means of impedance cardiograph for chosen medical patients population. Main medical aspect of Markov chain calculation has been underlined to exhibit some constraints in decision making with such math models. Markov chain analysis was done as optimization task. Effective algorithm for irregular data in continuous time has been proposed.


Introduction
Each tissue of the human body is characterized by electrical resistance depending on tissue density, temperature, and molecular composition. Blood conducts electricity better (R blood plasma is around 65 Ω/cm, whole blood is around 130 Ω/cm) than any other thoracic tissues such as muscles, bones, lungs and adipose tissue (R is in the range 200-5000 Ω/cm) [1]. Therefore, the impedance measurement (high frequency, low magnitude current is transmitted through the chest) allows the assessment of many important hemodynamic parameters related to the blood flow in a given body segment and its cyclical changes.
In cardiography the placement of four dual disposable sensors on the neck and chest are used to detect the impedance changes caused by a highfrequency, low magnitude current flowing through the thorax [2,3]. Using data from the impedance cardiograph (ICG) device, we can measure or calculate heart parameters such as: HR (Heart Rate), TFC (Thoracic Fluid Content), SBP (Systolic Blood Pressure), DBP (Diastolic Blood Pressure), Delta TFC (difference between two next values of TFC) and others. The Markov models which are presented in the publication describe evolution of the patient's state determined from the TFC [1/kΩ] parameter (see figure 1). However, the method of proceeding can be applied to all ICG parameters used in medical diagnosis. Fig. 1. Examples of measured data for cardiology. Each section inside measured value (green, white, yellow, red flag) means in fact separated condition of the observed patient. [ Basic form of Markov chain is applied in our model it means just chain with Markov attribute.
All measurements and analysis are part of project exploiting impedance cardiogram (ICG) for heart failure diagnostic [4]. Observed patients population is not so large yet but common feature of observation is that time distance for each measurement inside patient population are different, and irregular. It is the reason we propose couple of Markov chain model to exhibit their differences.
It seems to be quite important question if Markov processes describe properly prediction for medical patient state evolution. First of all medical knowledge is imposed on our calculation with any connection to Markov processes. Second -if we have a look at figure 1 one can see state space (or phase space of Markov process if you will) divided in accordance with actual medical knowledge of TFC observation. So space dimension equal to 5 is imposed to our calculation. On the other hand we have population of different patients with individual properties not obligatory equivalent to 5 dimensional space of state.
Simultaneously it is also well known, that result of calculation based on Markov chains strongly depends on number of states used inside chain construction. Used geometry of chain as well as its dimension, and jumps inside states taken into account influences on final values of calculated probabilities.
Nevertheless in 1983, Beck and Pauker described the use of Markov models for determining prognosis in medical applications [3]. Since that introduction, Markov models have been applied with increasing frequency in published decision analyses [5][6][7][8][9][10] to mention only a few which started big number of similar analysis. Most of publications were joined with decision tree evolving in examined illnesses. So space of decision was natural consequence of contemporary medical procedures.
There is really big amount of papers on Markov processes so we decides point out only basic connections in models for medical decisions.
In TFC measurement one can assume that TFC range divided into 5 parts is also result of medical procedure. Unfortunately set of ICG measurement made on patient population is rather random, and irregular. It seems to us worth to analyze such situation in possible suitable way with Markov chain tool.
Structure of document consists of five parts including introduction. Part 2 describes basic rules assumed for patient data model. Parts 3, 4, 5 contain steps in data analysis with Markov model for the same intervals between measurement, for different interval, and finally for irregular intervals with imposed probability distribution as one of the constraints in optimization.

Application of Markov chains in the description of the evolution of the patient's state
We assumed that the Markov chain is a sequence of random variables X m , 1 , X m,2 , X m,3 … describing the state of the m-th patient in 1,2,3 and subsequent times. We assumed also that the Markov chain for each patient is homogeneous.
In addition, due to the small number of data for individual patients, they were combined into groups. For each group, the matrixes of transitions between states for patients in a given group were assumed identical. As a result, the number of data used to determine the parameters of the chain describing the evolution of the patient's condition from a given group has been increased.
In the first model, we assume that further tests for patients in a given group are carried out at regular intervals. So variable X m,n describe the state of the m-th patient in the n-th test.
Second model describes how patient's condition changes within one day. In this model, the random variable X m, n describes the state of the m-th patient on the n-th day. In this approach random variable X m,n+k describes that test for the mth patient took place on the n-th day and the next will take place in k days.
As the third, we present a model that is an extension of the model describing how the patient's condition changes within one day. In this approach, a constraint was added to the optimization problem that the distribution of probabilities after the endless number of days tends to a particular, chosen probability distribution. This approach allows to increase the number of states in which the patient is staying, and therefore to increase the sensitivity of the prognosis. To obtain value of probability we create one dimensional Markov chain, and for each mentioned model we are looking for optimal solution that the sum of logarithms of probabilities occurrence of the observed data achieves maximum value.

Model for the same intervals between successive events
The model presented in this chapter is applicable if further tests for patients in a given group are carried out with regular intervals.
Let us assume that average value of the TFC parameter for a single patient changed as in Figure 2. The transition matrix for the Markov chain describing the evolution of the patient is a square matrix of the order S (number of states in Markov chain) and depends on the number of states for a given highlighted parameter. In the case of TFC S=5 (see figure 1). When determining individual probabilities in the matrix, we use the fact that the chain is homogeneous. In this case, the probability of transition from state i to state j is equal to the quotient l ij the sum of transitions from state i to state j and the sum of transitions from state i to any state. For TFC, we get the following formula: (1) Because we assumed that for the remaining patients in the analyzed group, the Markov chain contain an identical matrix of transitions, while adding up, we also consider transitions between states for all patients in a given group.

Model for various intervals between successive events
If the tests are performed every certain number of days, the model described in the previous chapter cannot be used. By monitoring the health of patients with heart failure, the intervals between subsequent tests range from one to several dozen days. In this chapter, we present model describing how the patient's condition changes within one day. For the illustration of model building, let us assume that we only consider periods of time not longer than 14 days (k = 14).
Let l ijk denote the sum of transitions from the i-th state to the j-th state in k days. Analogously to the model in the previous chapter, we sum up the transitions measured in patients from the analyzed group.
To simplify calculations, we assume that in one day the state may change by one or not at all. This assumption results from the fact that the appropriate l ij1 coefficients are zero in the experimental data we analyze. The matrix describing the Markov process is as follows: The probability of transition from the state i to the state j in k days p ijk we obtain from the Chapman-Kolmogorov equation and is: In formula (3) s(i) is a vector filled with zeros and one in i-th place and P k is the transition matrix multiplied by k-fold. Therefore, the objective function and restrictions are as follows: Where parameters of the matrix P must follow constraints: In the above model, the ε parameter is a small number that has been introduced to make p ijk ≠0 for all l ijk ≠0. Due to the fact that we are dealing with a smooth and strongly non-linear optimization problem (4-5), a generalized reduced gradient method [11,12,13,and 14] was chosen for solving it.
Using the matrix P determined by solving the optimization problem (4,5), from the formula (3) we can determine the probability of finding each state after k days. The dependence of the probability of a jump from the second state (the TFC value is in the range of 27-33 [1 / kΩ]) to individual states on the number of days is shown in Figure 3. The results based on 120 tests performed by 32 patients. Fig. 3. Probability of jump into particular states of the patient whose TFC was measured from the interval 27-33[1/kΩ] after the number of days shown in left column.

Model with stationary distribution determined from the assumed distribution
In this approach to the optimization problem (4, 5), the condition was added that the distribution of probabilities at the endless number of days (stationary distribution) tends to a particular distribution whose parameters are determined experimentally.
It allows to increase the number of states in which the patient is staying, and therefore to increase the sensitivity of the prognosis. Due to the smaller range of the measured parameter attributable to a single state, we assume that after one day the state can change by two, one or remains unchanged. For example, the matrix of transitions for S=9 is as follows: In addition, the presented method allows solving the problem of the lack of results in critical states. For TFC, it is assumed that a critical condition is one in which the TFC value is over 45 [1/kΩ]. According to the medical recommendations patient in such a state should urgently contact hospital within 2 hours to a significant change in treatment. However, available results show that patients who have been tested are very rarely in critical condition. On the other hand, the annual mortality rate of people with heart failure is respectively in NYHA classes: class I -to 10%, class II -10-20%, class III -20-40%, and class IV -mortality 40-60% [2]. Different mortality in individual groups is therefore included in stationary distributions established for these groups.
The objective function and restrictions are as follows: Where parameters of the matrix P for S=9 must follow constraints: 11  In the above model, a vector r is stationary distribution for the searched Markov chain. Probabilities occurring in it are determined based on the certain assumed distribution. At the same time, we assume that the parameter of this distribution β oscillated in the range determined on the basis of clinical results 〈 − ∆, + ∆〉.

Resume
Results illustrate that Markov processes for irregular TFC measurement can indicate different values in dependency of time marker model. All results have been created as goal function maximization during classical optimization task, and resolved with the same method for properly constructed constraints. So in fact only constraints decide about final probabilities. We get time series of 14 days duration, and probabilities are significantly different in dependency on boundaries with the same goal function. Results are for the same patient population, and for the same TFC measurement. Some similarities can be observed in results especially, that state 2 is preferred in all calculations, and some specific asymmetry in preferred states is observed. We do suppose that such kind of stability in calculations proofs their correctness to some extent.
It seems to us that Markov chains must be applied in medical decision support carefully, and after deep clinical studies. Especially sensitivity of such calculation should be examined during clinical tests.