MimiIWG/PAGE2009 - adding new perturbation -> KeyError

Hi all,

Previously, I added a 2060 perturbation for non-co2 gasses (CH4 and N2O) to estimate SC-Xs further into the century (to 2060, specifically). To do this, I:

  1. added the corresponding path of 2060 radiative forcings to the input file: MimiIWG \\ data \\ IWG_inputs \\ PAGE \\ Non-co2 perturbation data - 1 Mt.xlsx.

  2. updated the inputs in MimiIWG \\ src \\ core \\ PAGE_other_ghg.jl to include the new perturbations:

_page_ch4_shocks = Dict()
_page_ch4_shocks[2010] = _page_xf["Sheet1"]["I4:M13"]
...
_page_ch4_shocks[2060] = _page_xf["Sheet1"]["AR4:AV13"]

_page_n2o_shocks = Dict()
_page_n2o_shocks[2010] = _page_xf["Sheet1"]["I18:M27"]
...
_page_n2o_shocks[2060] = _page_xf["Sheet1"]["AR18:AV27"]

This all worked great. Now, I would like to extend this exercise to 2070. However, PAGE timsteps jump 2070 (…,2050, 2060, 2080, 2100…), which is fine, but I believe this just means that I will need to add perturbation data for 2080 (not the 2070 of interest) since MimiIWG interpolates in between timesteps during the SC-X calculation. However, when going about this in a similar fashion as before, I:

  1. added 2080 perturbation data to MimiIWG \\ data \\ IWG_inputs \\ PAGE \\ Non-co2 perturbation data - 1 Mt.xlsx.

  2. updated the inputs in MimiIWG \\ src \\ core \\ PAGE_other_ghg.jl to include the new perturbations:

_page_ch4_shocks = Dict()
_page_ch4_shocks[2010] = _page_xf["Sheet1"]["I4:M13"]
...
_page_ch4_shocks[2080] = _page_xf["Sheet1"]["AY4:BC13"]

_page_n2o_shocks = Dict()
_page_n2o_shocks[2010] = _page_xf["Sheet1"]["I18:M27"]
...
_page_n2o_shocks[2080] = _page_xf["Sheet1"]["AY18:BC27"]

But now, when estimating SC-CH4/N2O through 2070 (or 2080) with run_scc_mcs() or even deterministically with compute_scc(), I am getting the error:

...
start_task at /cygdrive/c/buildbot/worker/package_win64/build/src\task.c:839
ERROR: KeyError: key 2080 not found
Stacktrace:
 [1] getindex(h::Dict{Any, Any}, key::Int64)
   @ Base .\dict.jl:482
 [2] _get_page_forcing_shock(scenario_num::Int64, gas::Symbol, year::Int64)
   @ MimiIWG C:\Users\bparthum\.julia\packages\MimiIWG\37nQw\src\core\PAGE_other_ghg.jl:37
 [3] perturb_marginal_page_emissions!(base::Mimi.Model, marginal::Mimi.Model, gas::Symbol, emissionyear::Int64)
   @ MimiIWG C:\Users\bparthum\.julia\packages\MimiIWG\37nQw\src\core\PAGE_main.jl:352
 [4] page_post_trial_func(mcs::Mimi.LatinHypercubeSimulationInstance, trialnum::Int64, ntimesteps::Int64, tup::Tuple{MimiIWG.scenario_choice, Float64})
   @ MimiIWG C:\Users\bparthum\.julia\packages\MimiIWG\37nQw\src\montecarlo\PAGE_mcs.jl:222
 [5] run(sim_def::Mimi.LatinHypercubeSimulationDef, models::Vector{Mimi.Model}, samplesize::Int64; ntimesteps::Int64, trials_output_filename::Nothing, results_output_dir::Nothing, pre_trial_func::Nothing, post_trial_func::typeof(MimiIWG.page_post_trial_func), scenario_func::typeof(MimiIWG.page_scenario_func), scenario_placement::Mimi.ScenarioLoopPlacement, scenario_args::Vector{Pair{Symbol, Vector{T} where T}}, results_in_memory::Bool)
   @ Mimi C:\Users\bparthum\.julia\packages\Mimi\SnUU7\src\mcs\montecarlo.jl:626
 [6] run_scc_mcs(model::MimiIWG.model_choice; gas::Union{Nothing, Symbol}, trials::Int64, perturbation_years::Vector{Int64}, discount_rates::Vector{Float64}, domestic::Bool, output_dir::Union{Nothing, String}, save_trials::Bool, save_md::Bool, tables::Bool)
   @ MimiIWG C:\Users\bparthum\.julia\packages\MimiIWG\37nQw\src\montecarlo\run_scc_mcs.jl:154
 [7] top-level scope
   @ REPL[151]:1

Is there another location where I need to specify the 2080 input? If so, why would this not be needed for the 2060 addition? Any thoughts on what this error is pointing to?

Thanks!!

enviro:

(MimiIWG-env) pkg> st
      Status `~\environments\MimiIWG-env\Project.toml`
  [b1d80750] MimiIWG v1.0.4-DEV `~\.julia\dev\MimiIWG`
  [295af30f] Revise v3.1.16
  [9a3f8284] Random

Hi there,

It looks like your error is in the _get_page_forcing_shock function’s call to getindex i.e. this call _page_ch4_shocks[year][:, col_num] or this call _page_n2o_shocks[year][:, col_num]. It can’t find the key 2080 in the dictionary.

That’s relatively good news since it’s in a discrete helper function, let me try to replicate your problem!

Lisa

@parthum does this look like the changes you made: Add option for 2080 PAGE perturbatoin by lrennels · Pull Request #41 · rffscghg/MimiIWG.jl · GitHub

and is the call that was giving you the error something like this:

compute_scc(PAGE, :USG1; year = 2080)

As of now with these changes the above call runs fine on my machine, so I’m guessing there’s just some issue where you aren’t picking up the edited version of MimiIWG when you make the call so it doesn’t have an updated dictionary with the 2080 key.

Let me know, if my suspicion is correct that’s a good case because it should be easy to sort out together!

That’s it! Any call will do, but for example:

MimiIWG.compute_scc(PAGE, USG1, gas=:N2O, year=2080, discount=0.03)

Hmm, okay. Getting my edits to be incorporated has been so inconsistent. I eventually started using Revise and it helped a little bit. But I’ve made these changes dozens of times now, with different forks, environments, changing before/after using, in dev and in the main packages. Restarting the repl ~30+ times… working with Mimi has not been exactly efficient when it comes to editing packages :smiley: Clearly I’m missing something. Do you all have workflow suggestions for this?

Although, I suspected this might be the case, so I edited the values in the 2060 index, restarted the repl, re-precompiled, and it did catch those. Will keep experimenting.

Thank you!

Oh boy that’s definitely not the experience we want, we should certainly fix that and I had planned to write something up about suggested workflow but for now let’s walk through it here.

Revise can certainly be helpful, what is your current workflow? My workflow is as follows:


(1) Creating a Designated Environment

For a package I work on developing quite a bit, such as MimiIWG, I have a named shared environment for developing that package. You only need to set this up once, and periodically update it with pkg> up to get latest package versions.

So in the case of MimiIWG I have iwgdev which I created with the following call

pkg> activate --shared iwgdev

and then I added a bunch of packages using the compat file as a guide to add what I need. In this case all the packages are added with no version except MimiFUND is pinned and we want to use the development folder of MimiIWG that is on our machine in .julia/dev/MimiIWG.

pkg> add Distributions, ExcelReaders, Interpolations, Mimi, MimiDICE2010, MimiPAGE2009, StatsBase, XLSX
pkg> add MimiFUND#v3.8.6
pkg> dev MimiIWG

Now I have a designated environment with all the packages and versions I want that I can come back to, and that will automatically link to the development version of my MimiIWG package, held in the .Julia/dev/MimiIWG folder which can be controlled with git commands for branching etc.

If I want to activate it, I just call the same activate function and then can check on the contents and packages versions with st

pkg> activate --shared iwgdev
pkg> st
      Status `~/.julia/environments/iwgdev/Project.toml`
  [31c24e10] Distributions v0.23.12
  [c04bee98] ExcelReaders v0.11.0
  [a98d9a8b] Interpolations v0.13.2
  [e4e893b0] Mimi v1.2.3
  [4f2c5d70] MimiDICE2010 v1.0.1
  [b3ba11de] MimiFUND v3.8.6 `https://github.com/fund-model/MimiFUND.jl.git#v3.8.6`
  [b1d80750] MimiIWG v1.0.4-DEV `~/.julia/dev/MimiIWG`
  [e25d4f70] MimiPAGE2009 v3.1.0
  [2913bbd2] StatsBase v0.33.8
  [fdbf4ff8] XLSX v0.7.6

Once in a while you’ll want to call pkg> up just to get the latest versions of packages in there.

You can, of course have many packages in one environment on their dev branch … but I try to avoid that because it can get really confusing!

(2) Working on your package

So whenever I want to work on MimiIWG, my steps are as follows. These are fairly IDE agnostic but I"ll add advice for VSCode specifically in the next post!

  1. open the .Julia/dev/MimiIWG folder in my IDE for editing

  2. start up a terminal and enter that folder and check which branch I’m on and change if needed ie.

lisarennels@Lisas-MBP-4 MimiIWG % git status
    On branch master
    Your branch is up to date with 'origin/master'.
    nothing to commit, working tree clean

lisarennels@Lisas-MBP-4 MimiIWG % git checkout 2090
    Switched to branch '2090'
  1. open up a julia instance and activate my iwgdev environment with
pkg> activate --shared iwgdev

From there I should be able to edit and re-run, and if I use Revise I shouldn’t have to kill the REPL and start again unless I mess with the type system.


I will continue but first so far does that make sense? Next I will add my editor-specific advice on opening your REPL properly, may I ask what IDE you are using? We use VSCode which I would say right now is by far the best for Julia but if you’re using a different one that’s also of course fine!

1 Like

Alright, so assuming you are working within VSCode, here are some details.

When you open your dev folder, you can start a terminal up with Terminal → New Terminal (or the keyboard shortcut) and you’ll get your typical zsh/bash terminal with your current directory in the dev folder where you can to git commands etc.

Now to open a Julia folder where you can run commands, or interactively run your code line by line with Ctrl-Enter you can run View → Command Pallete → Julia> Start REPL. which also probably has a shortcut on your machine or one can be set up I know that’s what David does. A layman’s shortcut I use is also just to go into your code and run some blank line with Ctrl-Enter to automatically spawn an interactive Julia instance.

Note that if you start a Julia REPL from the terminal with a shortcut you may have on your machine like julia you will be able activate your environment and run commands that you type out or copy/paste without a problem … but if you try to use Ctrl-Enter to interactively run code from your script it will not connect and open a new terminal instead … like what happens above if you call Ctrl-Enter on a blank line. Ask if that’s confusing I mess it up occasionally myself!

Ok so now you have a Julia REPL up and running but you might not be sure which environment you’re in! There are ways to set your defaults, but my tried and true method is just to check. You can just type ] to see the name of the environment you are in to the right of the carrot, and change it with pkg> activate --shared iwgdev (Or whatever you named your environment) if needed. You can also use the VSCode handy trick of looking here at the bottom ribbon:

to see that I’m in environment v1.6 and on git branch 2090. If I want to change this, I can click on the environment which opens the command palette and I select iwgdev:

and then see:

on the bottom. Notice I can see I’m on the branch “2090” in environment iwgdev!

1 Like

The instructions above are specific to wanting to directly edit and develop a package like MimiIWG.

If you aren’t developing a package and for example, working on a project that will serve as a new repository to back up the work for a specific academic paper or presentation (something like pairing FUND and FAIR here), then your workflow is simple a bit and I can walk you through that as well.

This is :fire:, thank you @lrennels.

First, I was eventually able to get the updates/edits to register after several restarts and coding everything before even starting a new instance (just a local VSCode in this case). So thank you, I have things running now and will switch to the cluster (jupyter) to run the iterations.

Second, this is such a helpful rundown of workflow tips. I was doing most of these, but perhaps not in the right sequence or as controlled/systematically. I’ll walk through this a couple of times and, as you mention, I’m sure us Mimi users (and new julia users) will catch on quick once we know the in’s. A writeup like this will likely be very useful to many, so thank you for getting these notes down. Tagging @tammyt123 in case she’d like in on this greatness.

Thanks again!!

I’m so glad! There are variations to these techniques, as described in Julia documentation or by other users, but this is my general workflow which works for me as of now.

Working on packages whose versions you then want to test is more involved than working on projects with set environments, since versioning is trickier. Thus for projects it can be simpler to try to stick to modifying existing models with the modification API, or run a Monte Carlo etc., but that does break down at some point and you need to get into the guts of things … like you are with MimiIWG! I’m glad you’re digging in and more than happy to help.

1 Like

You know, I think there are likely more efficient ways to accomplish what we’ve been trying to accomplish (as the documentation does a good job of directing folks to), but for some reason brute forcing it from the inside out has been more intuitive (and much less eloquent). Going through David’s FUND/SNEASY and the FUND/FAIR is a great way to become acquainted with more efficient handshakes. So thanks for sharing those, too.