PSA Simulation API

1

@jrising @hbenveniste @BerBastien @FrankErrickson @AlexandrePavlov This is another PSA for users of the Simulation API. We have been working to simplify the API a bit from a user-perspective, so there are some small changes coming in the next tagged release of Mimi that you will want to be aware of. I will update all models we are responsible for as well. In most uses cases, all you will have to do is (1) remove calls to set_models! and generate_trials! (2) rewrite your run_sim! functions as run functions and add a few arguments to that call.

The basic API now looks like this:

#samplesize
N = 1000 

# get simulation definition
sim_definition = @defsim ...
end

# get whatever model you are using
m = get_model()

# run simulation and save results to the results_output_dir
sim_instance = run(sim_definition, m, 1000; results_output_dir = "./out") 

# view in-memory results for a variable
results = sim_instance[:comp_name, :variable_name]

All changes are documented in docs and tutorials. Lots of details can be found in the notes here: https://github.com/mimiframework/Mimi.jl/pull/522.

Main Changes:

  1. Types: Mirroring work with models, there are now two types SimulationDefinition and SimulationInstance. The former is what you get from @defsim, the latter is returned by run and holds a (mutated) copy of the SimulationDefinition it’s run function had as an argument.
  2. Run Function: The functions set_models! and generate_trials! have been absorbed into the run_sim function, which has been renamed to simply run with the full signature:
function Base.run(sim_def::SimulationDef{T}, models::Union{Vector{Model}, Model}, samplesize::Int;
                 ntimesteps::Int=typemax(Int), 
                 trials_output_filename::Union{Nothing, AbstractString}=nothing, 
                 results_output_dir::Union{Nothing, AbstractString}=nothing, 
                 pre_trial_func::Union{Nothing, Function}=nothing, 
                 post_trial_func::Union{Nothing, Function}=nothing,
                 scenario_func::Union{Nothing, Function}=nothing,
                 scenario_placement::ScenarioLoopPlacement=OUTER,
                 scenario_args=nothing,
                 results_in_memory::Bool=true) where T <: AbstractSimulationData

Thus, in its simplest form, you may define a SimulationDefinition with @defsim, and then call run(sim_def, model, 100) to run sim_def on model with sample size of 100. You will notice that the output filename for the trial data is thus moved to run as trials_output_filename, and if set your trial data will be saved to disk at that location.

run(sim_def ...) will return a SimulationInstance which contains a mutated copy of the SimulationDefinition argument, a vector of models, and a vector of results. These can be accessed with regular dot syntax:

sim_instance = `run(sim_def, model, 1000)
models = sim_instance.models
results = sim_instance.results

The results are organized as a vector, one item per model, and each item is a dictionary with keys for each saved variable and values are DataFrames. It is easiest to access results with getindex, for example you may get the DataFrame for component comp_name and variable var_name with:

results = sim_def[:comp, :var_name] # returns a DataFrame

If there are scenarios, results will have a field with :scen holding the scenario name.

  1. Results Saving: If you specify a results_output_dir, your results will be saved both in that directory in a folder structure as well as in memory in the results array. This folder structure is organized as scenario folder => model folder => comp_var.csv files. Only if you set the results_in_memory flag to false will memory be cleared after each trial. We have also fixed a bug that did not properly save results (to disk or memory) when multiple scenarios were involved.

  2. Callback Functions: Extra functions like post_trial_func and pre_trial_func now have a SimulationInstance in their signature instead of a Simulation.

Please do not hesitate to ask any questions here. This will be the next tagged version of Mimi but I wanted to post this now in case you have questions and don’t get taken by surprise!