PSA Simulation API

@jrising @hbenveniste @BerBastien @FrankErrickson @AlexandrePavlov This is another PSA for users of the Simulation API. We have been working to simplify the API a bit from a user-perspective, so there are some small changes coming in the next tagged release of Mimi that you will want to be aware of. I will update all models we are responsible for as well. In most uses cases, all you will have to do is (1) remove calls to set_models! and generate_trials! (2) rewrite your run_sim! functions as run functions and add a few arguments to that call.

The basic API now looks like this:

N = 1000 

# get simulation definition
sim_definition = @defsim ...

# get whatever model you are using
m = get_model()

# run simulation and save results to the results_output_dir
sim_instance = run(sim_definition, m, 1000; results_output_dir = "./out") 

# view in-memory results for a variable
results = sim_instance[:comp_name, :variable_name]

All changes are documented in docs and tutorials. Lots of details can be found in the notes here:

Main Changes:

  1. Types: Mirroring work with models, there are now two types SimulationDefinition and SimulationInstance. The former is what you get from @defsim, the latter is returned by run and holds a (mutated) copy of the SimulationDefinition it’s run function had as an argument.
  2. Run Function: The functions set_models! and generate_trials! have been absorbed into the run_sim function, which has been renamed to simply run with the full signature:
function{T}, models::Union{Vector{Model}, Model}, samplesize::Int;
                 trials_output_filename::Union{Nothing, AbstractString}=nothing, 
                 results_output_dir::Union{Nothing, AbstractString}=nothing, 
                 pre_trial_func::Union{Nothing, Function}=nothing, 
                 post_trial_func::Union{Nothing, Function}=nothing,
                 scenario_func::Union{Nothing, Function}=nothing,
                 results_in_memory::Bool=true) where T <: AbstractSimulationData

Thus, in its simplest form, you may define a SimulationDefinition with @defsim, and then call run(sim_def, model, 100) to run sim_def on model with sample size of 100. You will notice that the output filename for the trial data is thus moved to run as trials_output_filename, and if set your trial data will be saved to disk at that location.

run(sim_def ...) will return a SimulationInstance which contains a mutated copy of the SimulationDefinition argument, a vector of models, and a vector of results. These can be accessed with regular dot syntax:

sim_instance = `run(sim_def, model, 1000)
models = sim_instance.models
results = sim_instance.results

The results are organized as a vector, one item per model, and each item is a dictionary with keys for each saved variable and values are DataFrames. It is easiest to access results with getindex, for example you may get the DataFrame for component comp_name and variable var_name with:

results = sim_def[:comp, :var_name] # returns a DataFrame

If there are scenarios, results will have a field with :scen holding the scenario name.

  1. Results Saving: If you specify a results_output_dir, your results will be saved both in that directory in a folder structure as well as in memory in the results array. This folder structure is organized as scenario folder => model folder => comp_var.csv files. Only if you set the results_in_memory flag to false will memory be cleared after each trial. We have also fixed a bug that did not properly save results (to disk or memory) when multiple scenarios were involved.

  2. Callback Functions: Extra functions like post_trial_func and pre_trial_func now have a SimulationInstance in their signature instead of a Simulation.

Please do not hesitate to ask any questions here. This will be the next tagged version of Mimi but I wanted to post this now in case you have questions and don’t get taken by surprise!

Is the most recent version of the Monte Carlo/Simulation system included the last tagged release of Mimi? And are you expecting any future breaking changes?

I need to update the MC system in PAGE-ICE, so I wanted to make sure everything on Mimi’s side is ready to go.


Hi James,

The most recent version is not yet included in the tagged release, it is in the pending PR here, and there is also a coming PR for an explore(sim) function here: The problem is that we are waiting on an update to the streaming capability of CSVFiles.jl, which we have as an open PR on that package, before we can merge these.

Luckily our very own @davidanthoff manages that package, so @davidanthoff would it be possible to get that merged and tagged as soon as possible so we can merge in these updates to the Monte Carlo story and update the various models?