markkvdb / mdpsolver Goto Github PK

Produce the expected cost for a given state (x1, x2) and production rate (u1, u2) in a given state. Loop over all possible jumps of states.

The policy evaluator should take a vector of size SxS, where an element indicates the action in that particular state, i.e., it should define a (mainAction1, mainAction2, prodRate1, prodRate2) tuple for every element.

We then simulate the MDP over a time horizon and take the average cost.

The Init class should contain all the data that will be used in the problem. A (subset) of all data is given below:

• d_L: maximum deterioration level.
• d_cpm: Cost of preventive maintenance.
• d_ccm: Cost of corrective maintenance.
• d_C: Fixed cost of doing maintenance.
• d_pi: Total production rate of the system.
• d_gamma_mean: Mean of the gamma process.
• d_gamma_std: Standard deviation of the gamma process.
• More...

Setting the starting values of d_valueA in the MDP class to zero results in a nonsensical solution. The value iteration will only see expected values of 0, hence not doing maintenance results in the optimal solution (no direct cost incurred).

This function gives the probability of jumping (0, 1, 2, ...) deterioration levels. Note that this is independent of the current deterioration level.

Specify the skeleton of the MDP class.

Data members:

• ...

Public functions:

• solve()

Private functions:

• exp_cost() #1
• expected_cost(x1, x2) #2
• expected_costs(x1, x2, u1, u2) #3
• converged() #4

When the machine is in the failed state it should not be able to set a production rate.

Question: How do we prevent a state of the system, where we cannot meet demand restrictions due to failed units?

Create a function that will tell whether the value iteration algorithm has converged. This is simply checked by looking at the span.

After running the value iteration algorithm we should obtain the optimal action for every state. Could define a new member with optimal actions.

Write new versions of exp_cost, expected_cost, expected_costs that support this maximum. Note that we should select min(d_nStates - state -1, maxJump).

Then the solve function should consist of two do loops.

In the function void MDP::solve() we copy the whole vector d_valueA = d_valueB. However, we do not need d_valueB anymore after this call so we may swap the vectors. This might help when the state space gets really large.

I can also image that is doesn't help. So postpone this issue such that we can test later on.

Given state (x1, x2), produce the value for the new iteration.

Are we sure that we need & instead of &&?
I personally prefer to use and instead of &&.
The same holds for or instead of ||

Furthermore, you can just return the comparison instead of using an if-statement.

Since going linear through memory in C++ is faster than to create a matrix, we create a hash function that maps a given input (x1, x2) to a 2d vector.

The exp_cost should return a vector with the values corresponding to the new iteration.

Similar to p(u,1) this gives the probability of having a failure starting from states 0, 1, ..., L.