The Mathematics of the Reserve: A Toy Model of Optimal Spending
Contents
- A Mathematical Model of Resource Allocation
- A Small Artificial World With a Clairvoyant Oracle
- Teaching a Machine the Value of Preserving Capacity
- The Program
Part 1 - A Mathematical Model of Resource Allocation
The previous discussion began with a deliberately simple equation, \(A=R/D\). The equation is useful because it exposes an important property of finite resources: if \(R\) units remain and \(D\) periods remain, then the amount that can be consumed per period is constrained by the amount of resource that must survive until the end. A resource spent today is a resource that cannot be used tomorrow.
That simple relationship becomes much less informative as soon as the opportunities encountered by the system differ in value. If every opportunity were equally useful, distributing the remaining resource over the remaining lifetime would be a reasonable strategy. But interesting allocation problems are rarely homogeneous. One opportunity may be nearly worthless while another may be exceptionally valuable. The cost of an opportunity may also vary, and the opportunities that appear in the future may depend upon information revealed by the opportunities that have already appeared.
We can therefore describe the system as a sequence of discrete decisions. At time \(t\), let \(R_t\) denote the discretionary resource remaining and \(D_t\) the number of periods remaining. An opportunity \(O_t\) appears with cost \(C_t\) and value \(V_t\). The system has two basic choices:
If the system spends, its resource becomes
while saving leaves the discretionary resource unchanged:
The expenditure is permitted only when
This formulation deliberately separates discretionary spending from unavoidable expenditure. A real system may have costs associated with survival, maintenance, or continued operation. Those costs do not necessarily belong inside the decision being optimized. They can instead be removed from the total resource before constructing the discretionary allocation problem.
Suppose the system begins with a total budget \(B\), of which a fraction is reserved for unavoidable requirements. If \(B_s\) is the survival allocation, then the discretionary reserve is
The artificial world developed later begins with a total budget of \(250{,}000\), allocates seventy percent to survival, and leaves thirty percent as discretionary reserve. Thus
and
The program expresses this discretionary reserve in resource units of \(500\), so the starting reserve is equivalent to
resource units. The conversion is merely an implementation convenience. The underlying problem is the same whether the resource is represented as dollars, energy units, or abstract capacity units.
The important question is not simply whether an opportunity has positive value. The question is whether spending the resource required by the current opportunity is better than preserving that resource for the opportunities that may arrive later.
To represent this formally, let \(H_t\) denote the observable history of the system. The value of an opportunity can then be written as
and the conditional distribution of future opportunities can depend upon the same history:
This distinction is crucial. If the future were completely independent of the past and every opportunity came from the same distribution, history would provide little predictive information. But if the observed opportunities contain evidence about the state of the environment, then the past becomes useful.
For example, a sequence of unusually valuable opportunities might indicate that the environment is temporarily favorable. A sequence of poor opportunities might suggest that conserving resources is sensible. The decision-maker does not need to observe the hidden state directly. It only needs observations from which the state can be inferred.
The complete information available at a decision can therefore be represented abstractly as
A policy restricted to information available at decision time is a causal policy:
Such a policy cannot depend on \(O_{t+1},O_{t+2},\ldots\), because those observations have not yet occurred. A clairvoyant policy, by contrast, is allowed to know the entire realized future.
The realized utility of a policy is
where the notation treats \(a_t=1\) as SPEND and \(a_t=0\) as SAVE. The policy must also respect the resource constraint.
For a particular realized sequence of opportunities, the clairvoyant oracle can therefore calculate
while having access to the complete realized sequence
This is not the same problem as finding the optimal causal policy under uncertainty. The oracle is optimal for the particular future that actually occurred. It therefore provides a clairvoyant upper bound, rather than a realistic policy that an ordinary decision-maker could implement.
An information-limited decision-maker must instead decide whether to spend using only the current opportunity, its remaining resource, and whatever information about the future can be inferred from the past.
This difference can be expressed through the values of the two possible actions. Let
represent the total value obtainable when the current opportunity is accepted, including the value that remains available afterward. Similarly, let
represent the total value obtainable when the current opportunity is rejected and the resource is preserved.
The central quantity is the advantage of spending:
If
spending is better for that state. If
saving is better.
This gives mathematical form to the phrase "the value of preserving capacity." A resource unit saved today has no special physical property. It is valuable because it preserves a set of possible future actions. The value of that preserved capacity depends on what opportunities remain and how valuable they may be.
The original \(R/D\) equation remains useful in this larger system. If future opportunities have approximately equal value, costs are similar, and the environment is sufficiently homogeneous, distributing the remaining resource over the remaining time provides a sensible baseline.
Once opportunities become heterogeneous, however, the allocation problem changes. The system must decide not merely how quickly to consume its reserve, but when consumption creates more value than preservation.
Part 2 - A Small Artificial World With a Clairvoyant Oracle
The easiest way to study this problem is to construct a world in which every relevant fact is known to the experimenter. Instead of beginning with real financial data, biological observations, or a large machine-learning system, we can create a small artificial environment whose complete future can be generated and then solved exactly.
The artificial world used by the accompanying program has a total budget of \(250{,}000\). Seventy percent is designated as a survival allocation:
The remaining thirty percent becomes the discretionary reserve:
The discretionary reserve is represented internally using resource units of \(500\). Thus the optimizer begins with
resource units.
The artificial lifetime is thirty days. On each day exactly one opportunity appears. Each opportunity has a cost, a value, and an environmental quality:
The cost \(C_t\) determines how much discretionary reserve is consumed if the opportunity is accepted. The value \(V_t\) determines the utility obtained from accepting it. The quality \(Q_t\) is an observable characteristic produced by the environment and used by the opportunity generator to create temporal structure.
The artificial environment contains a hidden regime. Let
where \(L\), \(M\), and \(H\) represent low-, medium-, and high-quality environmental regimes.
Opportunities are generated conditionally on this hidden state:
The hidden regime can evolve over time according to a transition process whose behavior depends on the current regime and observed opportunity quality:
The exact probabilities are implementation parameters rather than part of the mathematical architecture. Their purpose is to create temporal correlation instead of independent and identically distributed opportunities. A high-quality observation can increase the probability that favorable conditions persist, while poor observations can make unfavorable conditions more likely to continue.
The learner is never given \(Z_t\) directly. It must infer something about the hidden environment from observable opportunities. This makes the environment partially observed.
The complete history
may contain more information than the small fixed-size representation given to the neural network. The network therefore does not necessarily receive a sufficient statistic for the hidden state. It is deliberately being asked to make useful decisions from a compressed observation of the environment.
The complete artificial history is a sequence
The oracle receives the entire sequence. The learned agent receives only the information that would have been available on the current day.
Resource scarcity is another important property of the environment. If \(E[C]\) denotes the expected opportunity cost, define the resource ratio
When \(\rho\) is much greater than one, the resource is abundant relative to the opportunities encountered. When \(\rho\) is close to one, resource scarcity becomes important. When \(\rho\) is much less than one, the system faces severe scarcity.
The experiment is most informative in the region where scarcity is meaningful but spending remains worthwhile. If resources are effectively unlimited, preserving capacity has little value. If resources are so scarce that almost nothing is affordable, the decision problem becomes trivial for a different reason.
The exact solution for a generated month can be calculated by dynamic programming. Let \(F(t,r)\) be the maximum utility obtainable from day \(t\) onward when \(r\) resource units remain and the complete future from day \(t\) onward is known.
Saving produces
If the opportunity is affordable, spending produces
The oracle therefore chooses
with the SPEND alternative removed whenever
At the end of the month there are no further opportunities:
The oracle's advantage can now be defined precisely:
Substituting the two action values gives
This equation is the mathematical heart of the experiment. The first term is the immediate value obtained by spending. The difference between the two future-value terms is the effect of consuming \(C_t\) units of reserve.
The advantage is therefore not simply the current opportunity's value. It is the current value minus the opportunity cost of losing future capacity.
If \(A^*(t,r)>0\), the oracle spends. If \(A^*(t,r)<0\), the oracle saves. When the two values are equal, either action produces the same oracle utility for that state.
This recurrence is small enough that the complete optimization can be performed directly. There is no approximation and no learning involved. Given a particular generated month, the oracle knows exactly which sequence of decisions maximizes realized utility.
This creates an unusually useful experimental situation. We can first create a complete artificial future, ask the oracle what should have been done, and then hide that future from a second decision-maker.
The oracle's advantage is informational rather than merely computational. It knows that a mediocre opportunity on one day will be followed by an exceptional opportunity later. It therefore knows whether the resource consumed now is worth more when used immediately or when preserved for the future.
The learned agent does not have that information. It must infer the likely future distribution from observations that have already occurred.
For example, suppose the beginning of a generated month contains several moderate opportunities. A greedy policy might spend repeatedly because each opportunity has positive value. The oracle may behave differently. If the complete future contains several unusually valuable opportunities later, the oracle may reject the early opportunities even though they are individually worth accepting in isolation.
This is the behavior the artificial world is intended to expose. The question is not whether spending can create value. Clearly it can. The question is whether spending now creates more value than retaining the ability to spend later.
There are three conceptually distinct policies in this framework. The first is the clairvoyant oracle, which sees the realized future. The second is the optimal causal policy, which knows the probability model of the environment but is restricted to information available at decision time. The third is the learned policy, which attempts to approximate useful causal decisions from finite training data.
Their expected or realized performance can be conceptually ordered as
provided the policies are evaluated under the same environment and the causal policy is optimized for the same objective.
The first gap represents the value of knowing the realized future:
The second represents the loss caused by the learned policy's approximation, limited representation, training procedure, and finite data:
The current program does not need to calculate the causal optimum explicitly for the oracle-versus-agent experiment to be useful. Nevertheless, this third benchmark is mathematically important. A learned agent falling short of the clairvoyant oracle has not necessarily failed to learn: some of the gap is caused by information that no causal policy can possess.
Part 3 - Teaching a Machine the Value of Preserving Capacity
Once the artificial world and its oracle exist, the learning problem becomes straightforward. Generate many synthetic months, solve each one with the clairvoyant oracle, and record decision states containing only information that would have been available at the time of the decision.
The learner is not trained to reproduce the future. It is trained to approximate the oracle's action advantage using information available in the present and the recent past.
The current implementation gives the network nine normalized inputs. They represent the remaining reserve, the remaining time, the current opportunity's cost and value, its quality, and the costs and values of the previous two opportunities.
In conceptual form, the input is
The quantities are normalized before being passed to the network. This keeps large numerical scales from dominating the learning process merely because the quantities are expressed in different units.
The network itself is intentionally small. It contains nine input quantities, one hidden layer containing thirty-two neurons, and a single scalar output. The hidden neurons use the hyperbolic tangent activation function.
The conceptual training target is the oracle's advantage:
In practice, the implementation may normalize or bound this quantity before training. If the target is transformed by a bounded function \(f\), then the network is technically learning
and approximating
rather than directly approximating the unbounded utility difference. This is appropriate when the primary objective is to learn which action has greater value, but the distinction should be kept clear when interpreting the network's output.
The learned decision rule is consequently based on the sign of the predicted advantage. If
the agent chooses SPEND. If
the agent chooses SAVE.
There is one decision that occurs before either of those alternatives:
An unaffordable opportunity is therefore not treated as a low-confidence SAVE decision. It is a separate feasibility state. The agent cannot spend resource that it does not possess.
Training on the oracle's advantage rather than only on the oracle's binary action preserves more information. Suppose one opportunity has
while another has
Both should result in SPEND, but the second state contains a much larger difference between the available actions.
The same applies to SAVE. A slightly negative advantage and a dramatically negative advantage are both SAVE decisions, but they represent very different opportunity costs.
The network is trained using a Huber loss, which is less sensitive to large individual errors than ordinary squared-error loss. The weights are updated using the Adam optimization algorithm, with gradient clipping to reduce the effect of unusually large parameter updates.
The training data is generated from many synthetic months. For each month, the oracle trajectory is recorded, but additional training states can also be sampled at different reserve levels. This allows the network to learn how the value of spending changes as reserve changes rather than merely memorizing the particular reserve trajectory followed by the oracle.
The result is a small learned approximation to a much larger exact calculation.
The oracle has access to the entire month:
The agent has only the information available up to the current time:
and even that history is compressed into a small fixed representation.
This makes the comparison meaningful. The oracle is not simply a faster version of the neural network. It has information that the neural network does not possess.
The simulation can report the two policies side by side. For each day it can show whether the agent chose SPEND, SAVE, or was UNAFFORDABLE, whether the oracle chose SPEND or SAVE, the predicted advantage or confidence, and the remaining reserves of both systems.
The confidence shown by the program should be interpreted carefully. It is derived from the magnitude of the predicted advantage. Large positive or negative predictions produce higher confidence, while predictions close to zero produce lower confidence. It is therefore not a calibrated probability that the network is correct.
The most useful performance measure is final utility rather than simple decision agreement. The program compares the utility obtained by the agent with the utility obtained by the clairvoyant oracle:
If the agent achieves ninety percent of the oracle's utility, that does not mean that it made ninety percent of the decisions correctly. One incorrect decision can have almost no effect, while another can consume reserve that would have been extremely valuable later.
Decision agreement is therefore useful as a diagnostic, but final utility is the more important measure of resource-allocation performance.
If a causal-optimal benchmark is also implemented, the experiment can report a second efficiency:
The distinction between these measures is important. The oracle gap reflects the value of information about the realized future, while the causal gap reflects the performance lost by approximating the best policy that could actually be implemented without that information.
Consider a month in which the oracle initially chooses SAVE for a long series of moderate opportunities. Later, unusually valuable opportunities begin to appear. The oracle spends heavily because it knows that those opportunities are worth the reserve it preserved.
The learned agent may reproduce much of this behavior, but it does not know that the valuable opportunities are coming. It must infer the state of the environment from the opportunities already observed.
This produces exactly the kind of disagreements that make the experiment interesting. The learner might SPEND too early, leaving less reserve for a later opportunity. It might also SAVE when the oracle spends because the current evidence does not make the future opportunity sufficiently predictable.
Neither behavior is adequately described by asking whether the current opportunity has positive value. The relevant quantity is its value relative to the future capacity consumed by accepting it.
The experiment therefore turns the abstract concept of preserving capacity into something observable. A negative predicted advantage means that the learner estimates the current opportunity is not valuable enough to justify consuming part of its remaining set of future actions.
The oracle makes the corresponding calculation with an unfair informational advantage: it can see the future exactly.
This also explains why the opportunity generator matters. If opportunities are too cheap and sufficiently valuable, both systems will naturally choose SPEND most of the time. The reserve constraint will rarely become interesting.
Conversely, if opportunities are generally too expensive or too poor, both systems will mostly choose SAVE. That is equally uninteresting.
A useful artificial world therefore needs a mixture of opportunity types. Some opportunities should be attractive enough to justify spending. Others should be marginal. Some should be poor. Some should be expensive. And the environment should contain enough temporal structure that history provides evidence about what may happen next.
This is why the program exposes the important environmental parameters as constants. Opportunity cost ranges, value ranges, quality probabilities, regime behavior, temporal correlation, resource allocation, and the initial resource-to-opportunity ratio can be changed without changing the underlying architecture of the experiment.
The program thereby becomes a small experimental laboratory. Changing the constants changes the artificial world while leaving the underlying question intact:
The experiment also provides a precise interpretation of the original \(R/D\) rule. A fixed daily allowance assumes, implicitly, that the future is sufficiently homogeneous that resource should be distributed approximately uniformly across time.
The learned policy is attempting something more difficult. It is trying to identify circumstances in which the future is sufficiently nonuniform that uniform spending is no longer the best strategy.
A day can therefore present an opportunity with positive immediate value while both the oracle and the agent choose SAVE. The opportunity is not worthless. It is simply not worth enough relative to the value of retaining the resource.
Likewise, an expensive opportunity can be worth accepting when its value is large enough and sufficient reserve remains. The decision depends on the entire state, not on cost or value in isolation.
The ultimate purpose of this toy model is not to demonstrate that a thirty-two-neuron network is an effective financial advisor. It is not. The environment is artificial, the horizon is short, the state representation is deliberately small, and the oracle has complete knowledge of the synthetic future.
The purpose is to isolate a more general computational idea.
A finite resource creates an opportunity cost for every action. An opportunity that is accepted consumes some of the system's future ability to act. When future opportunities differ in value, the correct decision cannot be derived from the current opportunity alone. It depends on the relationship between the present opportunity, the remaining resource, the remaining time, and the expected structure of what comes next.
The artificial oracle provides a precise upper bound for the realized allocation problem. A causal-optimal policy, when computable, provides a second benchmark that does not have access to the realized future. The neural network then provides a deliberately constrained approximation to that causal decision problem.
This separation is important because the gap between a learned agent and a clairvoyant oracle has two possible sources. Some of the gap comes from information: no causal agent can know the exact future. The remainder comes from the limitations of the learned policy itself.
The experiment can therefore measure not merely whether a neural network matches an oracle, but, when the additional benchmark is available, how much performance loss is attributable to uncertainty and how much is attributable to imperfect learning.
The experiment can be expanded without changing its fundamental structure. Opportunities can acquire categories, spatial positions, diminishing returns, complementarities, information value, delayed rewards, or longer temporal dependencies. Resource can be replenished. Survival requirements can become time-dependent. Opportunities can disappear, recur, or become more likely after related events.
The oracle can continue to solve the complete synthetic history while the learned policy continues to operate using only causal information.
At that point the artificial world begins to resemble a general resource allocation problem rather than a toy spending exercise. Money can be replaced by energy. Purchases can be replaced by machine jobs. Opportunities can represent prey, maintenance tasks, investments, experiments, or computational workloads. The same underlying question remains.
How should a finite resource be allocated when opportunities arrive over time, when their values differ, and when the future is unknown?
The original \(R/D\) equation remains the natural starting point because it describes the homogeneous case. The artificial oracle represents the next step: a world in which the future has structure and preserving capacity can itself have value.
The learned agent represents the final step in the experiment. It must act without seeing the future and must infer the value of preservation from the evidence contained in the past.
The resulting question is small enough to compute, but broad enough to generalize:
That is the real purpose of the artificial world. It does not attempt to solve the general problem of optimal spending. It creates a controlled environment in which the value of information can be studied, the value of preserving capacity can be observed, and a learned policy can be compared directly with an exact clairvoyant oracle.
The most important object in the experiment is ultimately not the neural network itself. It is the advantage function
It measures the marginal value of consuming scarce capacity at a particular moment. The learner's task is to estimate that quantity, or at least its sign, from information that does not include the future.
The original \(R/D\) rule describes how much resource can be spent on average. The advantage function asks the more difficult question: whether this particular unit of capacity is worth spending now.
That distinction is the mathematical core of the reserve problem.
The Program
#include <algorithm>
#include <array>
#include <cmath>
#include <iomanip>
#include <iostream>
#include <limits>
#include <numeric>
#include <random>
#include <sstream>
#include <string>
#include <vector>
using namespace std;
// =====================================================================================================================
// CONFIGURATION
// =====================================================================================================================
// ---------------------------------------------------------
// Budget
// ---------------------------------------------------------
constexpr int StartingBudget = 250000;
constexpr int SurvivalBudgetPercentage = 70;
constexpr int DiscretionaryBudgetPercentage = 100 - SurvivalBudgetPercentage;
constexpr int MonthDays = 30;
constexpr int SurvivalBudget = StartingBudget * SurvivalBudgetPercentage / 100;
constexpr int DiscretionaryBudget = StartingBudget * DiscretionaryBudgetPercentage / 100;
constexpr int DailySurvivalCost = SurvivalBudget / MonthDays;
constexpr int ResourceUnit = 500;
constexpr int InitialReserveUnits = DiscretionaryBudget / ResourceUnit;
// ---------------------------------------------------------
// Opportunity cost
//
// All opportunity costs are expressed in ResourceUnits.
// Actual dollar cost = CostUnits * ResourceUnit.
//
// Example:
// CostUnits = 10
// ResourceUnit = $500
// Actual cost = $5,000
// ---------------------------------------------------------
constexpr int MaxOpportunityCostUnits = 24;
// Low quality opportunity cost range.
constexpr int LowQualityMinimumCostUnits = 2;
constexpr int LowQualityMaximumCostUnits = 8;
// Medium quality opportunity cost range.
constexpr int MediumQualityMinimumCostUnits = 5;
constexpr int MediumQualityMaximumCostUnits = 16;
// High quality opportunity cost range.
constexpr int HighQualityMinimumCostUnits = 10;
constexpr int HighQualityMaximumCostUnits = MaxOpportunityCostUnits;
// ---------------------------------------------------------
// Opportunity value
//
// Value is calculated as:
//
// cost * multiplier
//
// These constants control how attractive each quality level is.
// ---------------------------------------------------------
constexpr int MaxOpportunityValue = 50000;
constexpr float LowQualityMinimumMultiplier = 0.6f;
constexpr float LowQualityMaximumMultiplier = 1.5f;
constexpr float MediumQualityMinimumMultiplier = 1.1f;
constexpr float MediumQualityMaximumMultiplier = 2.4f;
constexpr float HighQualityMinimumMultiplier = 2.0f;
constexpr float HighQualityMaximumMultiplier = 4.0f;
// ---------------------------------------------------------
// Opportunity quality distribution
//
// Quality:
// 0 = LOW
// 1 = MEDIUM
// 2 = HIGH
//
// These probabilities are the starting probabilities for
// each environmental regime.
// ---------------------------------------------------------
constexpr float BaseHighQualityProbability = 0.10f;
constexpr float BaseMediumQualityProbability = 0.35f;
constexpr float FavorableHighQualityProbability = 0.25f;
constexpr float FavorableMediumQualityProbability = 0.45f;
constexpr float ExcellentHighQualityProbability = 0.45f;
constexpr float ExcellentMediumQualityProbability = 0.40f;
constexpr float HighQualityMomentumBonus = 0.08f;
// ---------------------------------------------------------
// Regime changes
// ---------------------------------------------------------
constexpr float RegimeChangeProbability = 0.18f;
// ---------------------------------------------------------
// Learning
// ---------------------------------------------------------
constexpr int MaxOpportunityCostForInput = MaxOpportunityCostUnits;
constexpr int HistoryLength = 4;
constexpr int InputCount = 9;
constexpr int HiddenLayerSize = 32;
constexpr int TrainingEpisodes = 18000;
constexpr int TrainingEpochs = 5;
constexpr float AdvantageScale = 100000.0f;
constexpr float MaximumNormalizedAdvantage = 1.0f;
constexpr float LearningRate = 0.0015f;
constexpr float AdamBeta1 = 0.9f;
constexpr float AdamBeta2 = 0.999f;
constexpr float AdamEpsilon = 1.0e-8f;
constexpr float GradientClip = 5.0f;
// ---------------------------------------------------------
// Evaluation
// ---------------------------------------------------------
constexpr int EvaluationMonths = 10;
// =====================================================================================================================
// DATA TYPES
// =====================================================================================================================
struct Opportunity
{
int CostUnits = 0;
int Value = 0;
int Quality = 0;
};
struct DayState
{
int Day = 0;
int RemainingReserveUnits = 0;
int RemainingDays = 0;
Opportunity CurrentOpportunity;
array<float, HistoryLength * 2> History{};
};
struct Decision
{
bool Accept = false;
bool Affordable = true;
float Advantage = 0.0f;
float Confidence = 0.0f;
};
struct SimulationResult
{
vector<Opportunity> Opportunities;
vector<bool> OracleActions;
vector<bool> AgentActions;
vector<float> AgentAdvantages;
vector<float> AgentConfidences;
vector<bool> AgentAffordable;
int OracleUtility = 0;
int AgentUtility = 0;
int OracleReserveUnits = 0;
int AgentReserveUnits = 0;
};
struct OracleAnalysis
{
vector<vector<int>> Values;
vector<bool> Actions;
};
struct TrainingExample
{
array<float, InputCount> Input{};
float Target = 0.0f;
};
// =====================================================================================================================
// RANDOM GENERATOR
// =====================================================================================================================
class RandomGenerator
{
private:
mt19937 m_Engine;
public:
RandomGenerator(unsigned int seed = random_device{}()) : m_Engine(seed)
{
}
int Integer(int minimum, int maximum)
{
uniform_int_distribution<int> distribution(minimum, maximum);
return distribution(m_Engine);
}
float Real(float minimum, float maximum)
{
uniform_real_distribution<float> distribution(minimum, maximum);
return distribution(m_Engine);
}
bool Chance(float probability)
{
bernoulli_distribution distribution(probability);
return distribution(m_Engine);
}
mt19937& Engine()
{
return m_Engine;
}
};
// =====================================================================================================================
// OPPORTUNITY GENERATOR
// =====================================================================================================================
class OpportunityGenerator
{
public:
vector<Opportunity> GenerateMonth(RandomGenerator& randomGenerator)
{
vector<Opportunity> opportunities;
opportunities.reserve(MonthDays);
int regime = randomGenerator.Integer(0, 2);
int previousQuality = 0;
for(int day = 0; day < MonthDays; ++day)
{
if(randomGenerator.Chance(RegimeChangeProbability))
{
regime = randomGenerator.Integer(0, 2);
}
int quality = GenerateQuality(randomGenerator, regime, previousQuality);
int costUnits = GenerateCost(randomGenerator, quality);
int value = GenerateValue(randomGenerator, quality, costUnits);
opportunities.push_back({costUnits, value, quality});
previousQuality = quality;
}
return opportunities;
}
private:
int GenerateQuality(RandomGenerator& randomGenerator, int regime, int previousQuality)
{
float highProbability = BaseHighQualityProbability;
float mediumProbability = BaseMediumQualityProbability;
if(regime == 1)
{
highProbability = FavorableHighQualityProbability;
mediumProbability = FavorableMediumQualityProbability;
}
else if(regime == 2)
{
highProbability = ExcellentHighQualityProbability;
mediumProbability = ExcellentMediumQualityProbability;
}
if(previousQuality >= 2)
{
highProbability += HighQualityMomentumBonus;
}
float roll = randomGenerator.Real(0.0f, 1.0f);
if(roll < highProbability)
{
return 2;
}
if(roll < highProbability + mediumProbability)
{
return 1;
}
return 0;
}
int GenerateCost(RandomGenerator& randomGenerator, int quality)
{
if(quality == 0)
{
return randomGenerator.Integer(LowQualityMinimumCostUnits, LowQualityMaximumCostUnits);
}
if(quality == 1)
{
return randomGenerator.Integer(MediumQualityMinimumCostUnits, MediumQualityMaximumCostUnits);
}
return randomGenerator.Integer(HighQualityMinimumCostUnits, HighQualityMaximumCostUnits);
}
int GenerateValue(RandomGenerator& randomGenerator, int quality, int costUnits)
{
float multiplierMinimum = LowQualityMinimumMultiplier;
float multiplierMaximum = LowQualityMaximumMultiplier;
if(quality == 1)
{
multiplierMinimum = MediumQualityMinimumMultiplier;
multiplierMaximum = MediumQualityMaximumMultiplier;
}
else if(quality == 2)
{
multiplierMinimum = HighQualityMinimumMultiplier;
multiplierMaximum = HighQualityMaximumMultiplier;
}
float multiplier = randomGenerator.Real(multiplierMinimum, multiplierMaximum);
int value = static_cast<int>(costUnits * ResourceUnit * multiplier);
return min(value, MaxOpportunityValue);
}
};
// =====================================================================================================================
// CLAIRVOYANT ORACLE
// =====================================================================================================================
class ClairvoyantOracle
{
public:
OracleAnalysis Analyze(const vector<Opportunity>& opportunities)
{
const int dayCount = static_cast<int>(opportunities.size());
OracleAnalysis analysis;
analysis.Values.assign(dayCount + 1, vector<int>(InitialReserveUnits + 1, 0));
vector<vector<bool>> actionTable(dayCount, vector<bool>(InitialReserveUnits + 1, false));
for(int day = dayCount - 1; day >= 0; --day)
{
for(int reserve = 0; reserve <= InitialReserveUnits; ++reserve)
{
int saveValue = analysis.Values[day + 1][reserve];
int spendValue = numeric_limits<int>::min();
if(reserve >= opportunities[day].CostUnits)
{
spendValue = opportunities[day].Value + analysis.Values[day + 1][reserve - opportunities[day].CostUnits];
}
if(spendValue > saveValue)
{
analysis.Values[day][reserve] = spendValue;
actionTable[day][reserve] = true;
}
else
{
analysis.Values[day][reserve] = saveValue;
}
}
}
analysis.Actions.reserve(dayCount);
int reserve = InitialReserveUnits;
for(int day = 0; day < dayCount; ++day)
{
bool accept = actionTable[day][reserve];
if(accept && reserve >= opportunities[day].CostUnits)
{
reserve -= opportunities[day].CostUnits;
}
analysis.Actions.push_back(accept);
}
return analysis;
}
int CalculateUtility(const vector<Opportunity>& opportunities, const vector<bool>& actions)
{
int utility = 0;
for(size_t index = 0; index < opportunities.size(); ++index)
{
if(actions[index])
{
utility += opportunities[index].Value;
}
}
return utility;
}
float CalculateAdvantage(const OracleAnalysis& analysis, const vector<Opportunity>& opportunities, int day, int reserve)
{
int saveValue = analysis.Values[day + 1][reserve];
if(reserve < opportunities[day].CostUnits)
{
return -1.0f;
}
int spendValue = opportunities[day].Value + analysis.Values[day + 1][reserve - opportunities[day].CostUnits];
float advantage = static_cast<float>(spendValue - saveValue) / AdvantageScale;
return clamp(advantage, -MaximumNormalizedAdvantage, MaximumNormalizedAdvantage);
}
};
// =====================================================================================================================
// NEURAL NETWORK
// =====================================================================================================================
class NeuralNetwork
{
private:
array<array<float, HiddenLayerSize>, InputCount> m_WeightsInputHidden{};
array<float, HiddenLayerSize> m_BiasHidden{};
array<float, HiddenLayerSize> m_WeightsHiddenOutput{};
float m_BiasOutput = 0.0f;
array<array<float, HiddenLayerSize>, InputCount> m_MomentInputHidden{};
array<float, HiddenLayerSize> m_MomentBiasHidden{};
array<float, HiddenLayerSize> m_MomentHiddenOutput{};
float m_MomentBiasOutput = 0.0f;
array<array<float, HiddenLayerSize>, InputCount> m_VarianceInputHidden{};
array<float, HiddenLayerSize> m_VarianceBiasHidden{};
array<float, HiddenLayerSize> m_VarianceHiddenOutput{};
float m_VarianceBiasOutput = 0.0f;
float m_Beta1Power = 1.0f;
float m_Beta2Power = 1.0f;
long long m_TrainingStep = 0;
public:
void Initialize(RandomGenerator& randomGenerator)
{
float inputScale = sqrt(6.0f / static_cast<float>(InputCount + HiddenLayerSize));
float outputScale = sqrt(6.0f / static_cast<float>(HiddenLayerSize + 1));
for(auto& row : m_WeightsInputHidden)
{
for(float& weight : row)
{
weight = randomGenerator.Real(-inputScale, inputScale);
}
}
for(float& weight : m_WeightsHiddenOutput)
{
weight = randomGenerator.Real(-outputScale, outputScale);
}
m_BiasHidden.fill(0.0f);
m_BiasOutput = 0.0f;
m_MomentInputHidden = {};
m_MomentBiasHidden = {};
m_MomentHiddenOutput = {};
m_MomentBiasOutput = 0.0f;
m_VarianceInputHidden = {};
m_VarianceBiasHidden = {};
m_VarianceHiddenOutput = {};
m_VarianceBiasOutput = 0.0f;
m_Beta1Power = 1.0f;
m_Beta2Power = 1.0f;
m_TrainingStep = 0;
}
float Predict(const array<float, InputCount>& input)
{
array<float, HiddenLayerSize> hidden{};
for(int hiddenIndex = 0; hiddenIndex < HiddenLayerSize; ++hiddenIndex)
{
float sum = m_BiasHidden[hiddenIndex];
for(int inputIndex = 0; inputIndex < InputCount; ++inputIndex)
{
sum += input[inputIndex] * m_WeightsInputHidden[inputIndex][hiddenIndex];
}
hidden[hiddenIndex] = tanh(sum);
}
float output = m_BiasOutput;
for(int hiddenIndex = 0; hiddenIndex < HiddenLayerSize; ++hiddenIndex)
{
output += hidden[hiddenIndex] * m_WeightsHiddenOutput[hiddenIndex];
}
if(!isfinite(output))
{
return 0.0f;
}
return clamp(output, -MaximumNormalizedAdvantage, MaximumNormalizedAdvantage);
}
void Train(const vector<TrainingExample>& examples, RandomGenerator& randomGenerator)
{
if(examples.empty())
{
return;
}
vector<int> indices(examples.size());
iota(indices.begin(), indices.end(), 0);
for(int epoch = 0; epoch < TrainingEpochs; ++epoch)
{
shuffle(indices.begin(), indices.end(), randomGenerator.Engine());
double totalLoss = 0.0;
int correctDirections = 0;
int usefulExamples = 0;
for(int exampleIndex : indices)
{
float loss = TrainExample(examples[exampleIndex]);
totalLoss += loss;
float prediction = Predict(examples[exampleIndex].Input);
float target = examples[exampleIndex].Target;
if(abs(target) > 0.02f)
{
++usefulExamples;
if((prediction >= 0.0f) == (target >= 0.0f))
{
++correctDirections;
}
}
}
double averageLoss = totalLoss / static_cast<double>(examples.size());
double directionAccuracy = usefulExamples > 0
? static_cast<double>(correctDirections) / static_cast<double>(usefulExamples)
: 0.0;
cout << " Epoch " << epoch + 1 << "/" << TrainingEpochs;
cout << " loss: " << fixed << setprecision(5) << averageLoss;
cout << " directional accuracy: " << fixed << setprecision(1) << directionAccuracy * 100.0 << "%\n";
}
}
private:
static float HuberGradient(float error)
{
if(error > 1.0f)
{
return 1.0f;
}
if(error < -1.0f)
{
return -1.0f;
}
return error;
}
static float HuberLoss(float error)
{
float absoluteError = abs(error);
if(absoluteError <= 1.0f)
{
return 0.5f * error * error;
}
return absoluteError - 0.5f;
}
void ApplyAdam(float gradient, float& weight, float& moment, float& variance)
{
gradient = clamp(gradient, -GradientClip, GradientClip);
moment = AdamBeta1 * moment + (1.0f - AdamBeta1) * gradient;
variance = AdamBeta2 * variance + (1.0f - AdamBeta2) * gradient * gradient;
float correctedMoment = moment / (1.0f - m_Beta1Power);
float correctedVariance = variance / (1.0f - m_Beta2Power);
weight -= LearningRate * correctedMoment / (sqrt(correctedVariance) + AdamEpsilon);
if(!isfinite(weight))
{
weight = 0.0f;
}
}
float TrainExample(const TrainingExample& example)
{
array<float, HiddenLayerSize> hidden{};
for(int hiddenIndex = 0; hiddenIndex < HiddenLayerSize; ++hiddenIndex)
{
float sum = m_BiasHidden[hiddenIndex];
for(int inputIndex = 0; inputIndex < InputCount; ++inputIndex)
{
sum += example.Input[inputIndex] * m_WeightsInputHidden[inputIndex][hiddenIndex];
}
hidden[hiddenIndex] = tanh(sum);
}
float output = m_BiasOutput;
for(int hiddenIndex = 0; hiddenIndex < HiddenLayerSize; ++hiddenIndex)
{
output += hidden[hiddenIndex] * m_WeightsHiddenOutput[hiddenIndex];
}
if(!isfinite(output))
{
output = 0.0f;
}
float error = output - example.Target;
float loss = HuberLoss(error);
float outputGradient = HuberGradient(error);
array<float, HiddenLayerSize> hiddenGradients{};
for(int hiddenIndex = 0; hiddenIndex < HiddenLayerSize; ++hiddenIndex)
{
hiddenGradients[hiddenIndex] = outputGradient * m_WeightsHiddenOutput[hiddenIndex];
}
++m_TrainingStep;
m_Beta1Power *= AdamBeta1;
m_Beta2Power *= AdamBeta2;
for(int hiddenIndex = 0; hiddenIndex < HiddenLayerSize; ++hiddenIndex)
{
float gradient = outputGradient * hidden[hiddenIndex];
ApplyAdam(gradient, m_WeightsHiddenOutput[hiddenIndex], m_MomentHiddenOutput[hiddenIndex], m_VarianceHiddenOutput[hiddenIndex]);
}
ApplyAdam(outputGradient, m_BiasOutput, m_MomentBiasOutput, m_VarianceBiasOutput);
for(int hiddenIndex = 0; hiddenIndex < HiddenLayerSize; ++hiddenIndex)
{
float activationGradient = hiddenGradients[hiddenIndex] * (1.0f - hidden[hiddenIndex] * hidden[hiddenIndex]);
for(int inputIndex = 0; inputIndex < InputCount; ++inputIndex)
{
float gradient = activationGradient * example.Input[inputIndex];
ApplyAdam(gradient, m_WeightsInputHidden[inputIndex][hiddenIndex], m_MomentInputHidden[inputIndex][hiddenIndex], m_VarianceInputHidden[inputIndex][hiddenIndex]);
}
ApplyAdam(activationGradient, m_BiasHidden[hiddenIndex], m_MomentBiasHidden[hiddenIndex], m_VarianceBiasHidden[hiddenIndex]);
}
return loss;
}
};
// =====================================================================================================================
// SPENDING AGENT
// =====================================================================================================================
class SpendingAgent
{
private:
NeuralNetwork m_Network;
public:
void Initialize(RandomGenerator& randomGenerator)
{
m_Network.Initialize(randomGenerator);
}
void Train(const vector<TrainingExample>& examples, RandomGenerator& randomGenerator)
{
m_Network.Train(examples, randomGenerator);
}
Decision Decide(const DayState& state)
{
if(state.RemainingReserveUnits < state.CurrentOpportunity.CostUnits)
{
return {false, false, 0.0f, 1.0f};
}
array<float, InputCount> input = BuildInput(state);
float advantage = m_Network.Predict(input);
float confidence = CalculateConfidence(advantage);
return {advantage > 0.0f, true, advantage, confidence};
}
private:
array<float, InputCount> BuildInput(const DayState& state)
{
array<float, InputCount> input{};
input[0] = static_cast<float>(state.RemainingReserveUnits) / static_cast<float>(InitialReserveUnits);
input[1] = static_cast<float>(state.RemainingDays) / static_cast<float>(MonthDays);
input[2] = static_cast<float>(state.CurrentOpportunity.CostUnits) / static_cast<float>(MaxOpportunityCostForInput);
input[3] = static_cast<float>(state.CurrentOpportunity.Value) / static_cast<float>(MaxOpportunityValue);
input[4] = static_cast<float>(state.CurrentOpportunity.Quality) / 2.0f;
input[5] = state.History[0] / static_cast<float>(MaxOpportunityCostForInput);
input[6] = state.History[1] / static_cast<float>(MaxOpportunityValue);
input[7] = state.History[2] / static_cast<float>(MaxOpportunityCostForInput);
input[8] = state.History[3] / static_cast<float>(MaxOpportunityValue);
return input;
}
float CalculateConfidence(float advantage)
{
float magnitude = abs(advantage);
float confidence = 1.0f - exp(-magnitude * 5.0f);
return clamp(confidence, 0.0f, 1.0f);
}
};
// =====================================================================================================================
// TRAINING DATA GENERATOR
// =====================================================================================================================
class TrainingDataGenerator
{
private:
OpportunityGenerator m_OpportunityWorld;
ClairvoyantOracle m_Oracle;
public:
vector<TrainingExample> Generate(RandomGenerator& randomGenerator, int episodeCount)
{
vector<TrainingExample> examples;
examples.reserve(static_cast<size_t>(episodeCount) * MonthDays * 2);
for(int episode = 0; episode < episodeCount; ++episode)
{
vector<Opportunity> opportunities = m_OpportunityWorld.GenerateMonth(randomGenerator);
OracleAnalysis analysis = m_Oracle.Analyze(opportunities);
int oracleReserve = InitialReserveUnits;
array<float, HistoryLength * 2> history{};
for(int day = 0; day < MonthDays; ++day)
{
AddExample(examples, opportunities, analysis, day, oracleReserve, history);
int minimumReserve = opportunities[day].CostUnits;
int randomReserve = randomGenerator.Integer(minimumReserve, InitialReserveUnits);
AddExample(examples, opportunities, analysis, day, randomReserve, history);
if(analysis.Actions[day] && oracleReserve >= opportunities[day].CostUnits)
{
oracleReserve -= opportunities[day].CostUnits;
}
ShiftHistory(history, opportunities[day]);
}
}
return examples;
}
private:
void AddExample(vector<TrainingExample>& examples, const vector<Opportunity>& opportunities, const OracleAnalysis& analysis, int day, int reserve, const array<float, HistoryLength * 2>& history)
{
DayState state;
state.Day = day + 1;
state.RemainingReserveUnits = reserve;
state.RemainingDays = MonthDays - day;
state.CurrentOpportunity = opportunities[day];
state.History = history;
float target = m_Oracle.CalculateAdvantage(analysis, opportunities, day, reserve);
examples.push_back({BuildInput(state), target});
}
array<float, InputCount> BuildInput(const DayState& state)
{
array<float, InputCount> input{};
input[0] = static_cast<float>(state.RemainingReserveUnits) / static_cast<float>(InitialReserveUnits);
input[1] = static_cast<float>(state.RemainingDays) / static_cast<float>(MonthDays);
input[2] = static_cast<float>(state.CurrentOpportunity.CostUnits) / static_cast<float>(MaxOpportunityCostForInput);
input[3] = static_cast<float>(state.CurrentOpportunity.Value) / static_cast<float>(MaxOpportunityValue);
input[4] = static_cast<float>(state.CurrentOpportunity.Quality) / 2.0f;
input[5] = state.History[0] / static_cast<float>(MaxOpportunityCostForInput);
input[6] = state.History[1] / static_cast<float>(MaxOpportunityValue);
input[7] = state.History[2] / static_cast<float>(MaxOpportunityCostForInput);
input[8] = state.History[3] / static_cast<float>(MaxOpportunityValue);
return input;
}
void ShiftHistory(array<float, HistoryLength * 2>& history, const Opportunity& opportunity)
{
for(int index = HistoryLength * 2 - 1; index >= 2; --index)
{
history[index] = history[index - 2];
}
history[0] = static_cast<float>(opportunity.CostUnits);
history[1] = static_cast<float>(opportunity.Value);
}
};
// =====================================================================================================================
// SIMULATION
// =====================================================================================================================
class Simulation
{
private:
OpportunityGenerator m_OpportunityWorld;
ClairvoyantOracle m_Oracle;
public:
SimulationResult Run(SpendingAgent& agent, RandomGenerator& randomGenerator)
{
SimulationResult result;
result.Opportunities = m_OpportunityWorld.GenerateMonth(randomGenerator);
OracleAnalysis analysis = m_Oracle.Analyze(result.Opportunities);
result.OracleActions = analysis.Actions;
result.AgentActions.reserve(MonthDays);
result.AgentAdvantages.reserve(MonthDays);
result.AgentConfidences.reserve(MonthDays);
result.AgentAffordable.reserve(MonthDays);
int agentReserve = InitialReserveUnits;
int oracleReserve = InitialReserveUnits;
int agentUtility = 0;
int oracleUtility = m_Oracle.CalculateUtility(result.Opportunities, result.OracleActions);
array<float, HistoryLength * 2> history{};
for(int day = 0; day < MonthDays; ++day)
{
DayState state;
state.Day = day + 1;
state.RemainingReserveUnits = agentReserve;
state.RemainingDays = MonthDays - day;
state.CurrentOpportunity = result.Opportunities[day];
state.History = history;
Decision decision = agent.Decide(state);
result.AgentActions.push_back(decision.Accept);
result.AgentAdvantages.push_back(decision.Advantage);
result.AgentConfidences.push_back(decision.Confidence);
result.AgentAffordable.push_back(decision.Affordable);
if(decision.Accept && agentReserve >= result.Opportunities[day].CostUnits)
{
agentReserve -= result.Opportunities[day].CostUnits;
agentUtility += result.Opportunities[day].Value;
}
if(result.OracleActions[day] && oracleReserve >= result.Opportunities[day].CostUnits)
{
oracleReserve -= result.Opportunities[day].CostUnits;
}
ShiftHistory(history, result.Opportunities[day]);
}
result.AgentUtility = agentUtility;
result.OracleUtility = oracleUtility;
result.AgentReserveUnits = agentReserve;
result.OracleReserveUnits = oracleReserve;
return result;
}
private:
void ShiftHistory(array<float, HistoryLength * 2>& history, const Opportunity& opportunity)
{
for(int index = HistoryLength * 2 - 1; index >= 2; --index)
{
history[index] = history[index - 2];
}
history[0] = static_cast<float>(opportunity.CostUnits);
history[1] = static_cast<float>(opportunity.Value);
}
};
// =====================================================================================================================
// FORMATTING
// =====================================================================================================================
string FormatMoney(int amount)
{
stringstream stream;
stream << "$" << fixed << setprecision(0) << amount;
return stream.str();
}
string FormatProbability(float probability)
{
stringstream stream;
stream << fixed << setprecision(0) << probability * 100.0f << "%";
return stream.str();
}
string FormatAdvantage(float normalizedAdvantage)
{
int advantage = static_cast<int>(normalizedAdvantage * AdvantageScale);
stringstream stream;
stream << "$" << fixed << setprecision(0) << advantage;
return stream.str();
}
string FormatAgentAdvantage(const SimulationResult& result, int day)
{
if(!result.AgentAffordable[day])
{
return "N/A";
}
return FormatAdvantage(result.AgentAdvantages[day]);
}
// =====================================================================================================================
// HEADER
// =====================================================================================================================
void PrintHeader()
{
cout << "\n";
cout << "========================================================================================================================\n";
cout << " RESOURCE ALLOCATION LABORATORY\n";
cout << " 30 DAY / $250,000 BUDGET\n";
cout << "========================================================================================================================\n";
cout << "\n";
cout << "Starting budget : " << FormatMoney(StartingBudget) << "\n";
cout << "Survival allocation : " << FormatMoney(SurvivalBudget) << " (" << SurvivalBudgetPercentage << "%)\n";
cout << "Discretionary reserve : " << FormatMoney(DiscretionaryBudget) << " (" << DiscretionaryBudgetPercentage << "%)\n";
cout << "Daily survival cost : " << FormatMoney(DailySurvivalCost) << "\n";
cout << "\n";
cout << "Opportunity cost range:\n";
cout << " Low quality : " << FormatMoney(LowQualityMinimumCostUnits * ResourceUnit) << " - " << FormatMoney(LowQualityMaximumCostUnits * ResourceUnit) << "\n";
cout << " Medium quality : " << FormatMoney(MediumQualityMinimumCostUnits * ResourceUnit) << " - " << FormatMoney(MediumQualityMaximumCostUnits * ResourceUnit) << "\n";
cout << " High quality : " << FormatMoney(HighQualityMinimumCostUnits * ResourceUnit) << " - " << FormatMoney(HighQualityMaximumCostUnits * ResourceUnit) << "\n";
cout << "\n";
cout << "The learned agent estimates the value of SPEND versus SAVE.\n";
cout << "The neural network never receives future opportunities.\n";
cout << "UNAFFORDABLE means the agent does not have enough reserve to SPEND.\n";
cout << "\n";
}
// =====================================================================================================================
// TRAINING INTRODUCTION
// =====================================================================================================================
void PrintTrainingIntroduction()
{
cout << "\n";
cout << "========================================================================================================================\n";
cout << "TRAINING THE VALUE FUNCTION\n";
cout << "========================================================================================================================\n";
cout << "\n";
cout << "The oracle uses the complete future to calculate the exact value of\n";
cout << "SPEND and SAVE for every possible reserve level.\n";
cout << "\n";
cout << "The neural network does not learn the oracle's binary action directly.\n";
cout << "Instead it learns the advantage:\n";
cout << "\n";
cout << " advantage = Q(SPEND) - Q(SAVE)\n";
cout << "\n";
cout << "Positive advantage means SPEND. Negative advantage means SAVE.\n";
cout << "\n";
cout << "Training states are sampled across many reserve levels so the network\n";
cout << "does not depend entirely on following the oracle's trajectory.\n";
cout << "\n";
}
// =====================================================================================================================
// SIMULATION REPORT
// =====================================================================================================================
void PrintSimulation(const SimulationResult& result)
{
cout << "\n";
cout << "========================================================================================================================\n";
cout << "MONTH SIMULATION\n";
cout << "========================================================================================================================\n";
cout << "\n";
cout << left << setw(5) << "Day"
<< setw(12) << "Opportunity"
<< setw(14) << "Value"
<< setw(16) << "Agent"
<< setw(16) << "Oracle"
<< setw(12) << "Confidence"
<< setw(15) << "Advantage"
<< setw(15) << "Agent Reserve"
<< setw(15) << "Oracle Reserve"
<< "\n";
cout << string(120, '-') << "\n";
int agentReserve = InitialReserveUnits;
int oracleReserve = InitialReserveUnits;
int matchingDecisions = 0;
int affordableDecisions = 0;
int unaffordableDays = 0;
for(int day = 0; day < MonthDays; ++day)
{
const Opportunity& opportunity = result.Opportunities[day];
bool agentAffordable = result.AgentAffordable[day];
if(agentAffordable)
{
++affordableDecisions;
if(result.AgentActions[day] == result.OracleActions[day])
{
++matchingDecisions;
}
}
else
{
++unaffordableDays;
}
if(result.AgentActions[day])
{
agentReserve -= opportunity.CostUnits;
}
if(result.OracleActions[day])
{
oracleReserve -= opportunity.CostUnits;
}
string agentAction;
if(!agentAffordable)
{
agentAction = "UNAFFORDABLE";
}
else if(result.AgentActions[day])
{
agentAction = "SPEND";
}
else
{
agentAction = "SAVE";
}
string oracleAction = result.OracleActions[day] ? "SPEND" : "SAVE";
cout << left << setw(5) << day + 1
<< setw(12) << FormatMoney(opportunity.CostUnits * ResourceUnit)
<< setw(14) << FormatMoney(opportunity.Value)
<< setw(16) << agentAction
<< setw(16) << oracleAction
<< setw(12) << FormatProbability(result.AgentConfidences[day])
<< setw(15) << FormatAgentAdvantage(result, day)
<< setw(15) << FormatMoney(agentReserve * ResourceUnit)
<< setw(15) << FormatMoney(oracleReserve * ResourceUnit)
<< "\n";
}
float efficiency = result.OracleUtility > 0
? static_cast<float>(result.AgentUtility) / static_cast<float>(result.OracleUtility)
: 1.0f;
cout << "\n";
cout << "========================================================================================================================\n";
cout << "RESULT\n";
cout << "========================================================================================================================\n";
cout << "\n";
cout << "Oracle utility : " << FormatMoney(result.OracleUtility) << "\n";
cout << "Agent utility : " << FormatMoney(result.AgentUtility) << "\n";
cout << "Agent/oracle utility : "
<< fixed << setprecision(1) << efficiency * 100.0f << "%\n";
cout << "Oracle reserve : "
<< FormatMoney(result.OracleReserveUnits * ResourceUnit) << "\n";
cout << "Agent reserve : "
<< FormatMoney(result.AgentReserveUnits * ResourceUnit) << "\n";
cout << "Matching decisions : "
<< matchingDecisions << "/" << affordableDecisions;
if(affordableDecisions > 0)
{
cout << " (" << fixed << setprecision(1)
<< static_cast<float>(matchingDecisions) / affordableDecisions * 100.0f
<< "%)";
}
cout << "\n";
cout << "Affordable decisions : "
<< affordableDecisions << "/" << MonthDays;
cout << " (" << fixed << setprecision(1)
<< static_cast<float>(affordableDecisions) / MonthDays * 100.0f
<< "%)\n";
cout << "Unaffordable days : "
<< unaffordableDays << "/" << MonthDays;
cout << "\n";
cout << "\n";
if(result.AgentUtility >= result.OracleUtility)
{
cout << "The learned agent matched or exceeded the oracle's realized utility.\n";
}
else
{
cout << "The learned agent left "
<< FormatMoney(result.OracleUtility - result.AgentUtility)
<< " of the oracle's realized utility unexplored.\n";
}
cout << "\n";
}
// =====================================================================================================================
// MENU
// =====================================================================================================================
void PrintMenu()
{
cout << "\n";
cout << "========================================================================================================================\n";
cout << "MENU\n";
cout << "========================================================================================================================\n";
cout << "\n";
cout << " [1] Run another month\n";
cout << " [2] Run 10 evaluation months\n";
cout << " [3] Exit\n";
cout << "\n";
cout << "Select: ";
}
// =====================================================================================================================
// EVALUATION
// =====================================================================================================================
void RunEvaluation(SpendingAgent& agent, RandomGenerator& randomGenerator)
{
Simulation simulation;
double totalAgentUtility = 0.0;
double totalOracleUtility = 0.0;
int totalMatchingDecisions = 0;
int totalAffordableDecisions = 0;
int totalUnaffordableDays = 0;
int totalAgentReserve = 0;
cout << "\n";
cout << "========================================================================================================================\n";
cout << "10 MONTH EVALUATION\n";
cout << "========================================================================================================================\n";
cout << "\n";
for(int month = 0; month < EvaluationMonths; ++month)
{
SimulationResult result = simulation.Run(agent, randomGenerator);
totalAgentUtility += result.AgentUtility;
totalOracleUtility += result.OracleUtility;
totalAgentReserve += result.AgentReserveUnits;
for(int day = 0; day < MonthDays; ++day)
{
if(result.AgentAffordable[day])
{
++totalAffordableDecisions;
if(result.AgentActions[day] == result.OracleActions[day])
{
++totalMatchingDecisions;
}
}
else
{
++totalUnaffordableDays;
}
}
double efficiency = result.OracleUtility > 0
? static_cast<double>(result.AgentUtility) / result.OracleUtility
: 1.0;
cout << "Month " << setw(2) << month + 1;
cout << " | Agent " << setw(8) << FormatMoney(result.AgentUtility);
cout << " | Oracle " << setw(8) << FormatMoney(result.OracleUtility);
cout << " | Efficiency " << fixed << setprecision(1) << setw(5) << efficiency * 100.0 << "%";
cout << " | Reserve " << setw(7) << FormatMoney(result.AgentReserveUnits * ResourceUnit);
cout << "\n";
}
double overallEfficiency = totalOracleUtility > 0.0
? totalAgentUtility / totalOracleUtility
: 1.0;
double decisionAccuracy = totalAffordableDecisions > 0
? static_cast<double>(totalMatchingDecisions) / static_cast<double>(totalAffordableDecisions)
: 1.0;
double affordabilityRate = static_cast<double>(totalAffordableDecisions) / static_cast<double>(EvaluationMonths * MonthDays);
double averageAgentUtility = totalAgentUtility / static_cast<double>(EvaluationMonths);
double averageOracleUtility = totalOracleUtility / static_cast<double>(EvaluationMonths);
double averageReserve = static_cast<double>(totalAgentReserve) / static_cast<double>(EvaluationMonths);
cout << "\n";
cout << "Average agent utility : "
<< FormatMoney(static_cast<int>(averageAgentUtility)) << "\n";
cout << "Average oracle utility: "
<< FormatMoney(static_cast<int>(averageOracleUtility)) << "\n";
cout << "Overall efficiency : "
<< fixed << setprecision(1) << overallEfficiency * 100.0 << "%\n";
cout << "Decision agreement : "
<< fixed << setprecision(1) << decisionAccuracy * 100.0 << "%\n";
cout << "Affordability rate : "
<< fixed << setprecision(1) << affordabilityRate * 100.0 << "%\n";
cout << "Unaffordable days : "
<< totalUnaffordableDays << "/" << EvaluationMonths * MonthDays << "\n";
cout << "Average reserve : "
<< FormatMoney(static_cast<int>(averageReserve * ResourceUnit)) << "\n";
}
// =====================================================================================================================
// MAIN
// =====================================================================================================================
int main()
{
RandomGenerator randomGenerator;
SpendingAgent agent;
TrainingDataGenerator trainingDataGenerator;
PrintHeader();
PrintTrainingIntroduction();
cout << "Generating " << TrainingEpisodes << " synthetic months...\n";
vector<TrainingExample> trainingExamples = trainingDataGenerator.Generate(randomGenerator, TrainingEpisodes);
cout << "Generated " << trainingExamples.size() << " value-function training examples.\n";
cout << "\n";
agent.Initialize(randomGenerator);
cout << "Initializing neural value function...\n";
cout << "Training...\n";
cout << "\n";
agent.Train(trainingExamples, randomGenerator);
cout << "\n";
cout << "Training complete.\n";
cout << "The learned value function is now allowed to make decisions without seeing the future.\n";
while(true)
{
PrintMenu();
string choice;
getline(cin, choice);
if(choice == "1")
{
Simulation simulation;
SimulationResult result = simulation.Run(agent, randomGenerator);
PrintSimulation(result);
}
else if(choice == "2")
{
RunEvaluation(agent, randomGenerator);
}
else if(choice == "3" || cin.eof())
{
cout << "\nExiting resource allocation laboratory.\n";
break;
}
else
{
cout << "\nInvalid selection.\n";
}
}
return 0;
}