# MimiIWG/PAGE2009 - adding new perturbation -\> KeyError

**URL:** <https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175>\
**Category:** Uncategorized\
**Created:** [May 12, 2021, 4:27pm UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175 "2021-05-12T16:27:02Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![parthum](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.mimiframework.org/parthum/32/30_2.png) [@parthum](https://forum.mimiframework.org/u/parthum)\
**Post date:** [May 12, 2021, 4:27pm UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175/1 "2021-05-12T16:27:03Z")

</div>

Hi all,

Previously, I [added a 2060 perturbation](https://github.com/rffscghg/MimiIWG.jl/commit/f6f01b1e326cd7f0d33623d310a3a21cdea9c2ca) for non-co2 gasses (CH4 and N2O) to estimate SC-Xs further into the century (to 2060, specifically). To do this, I:

1. added the corresponding path of 2060 radiative forcings to the input file: `MimiIWG \\ data \\ IWG_inputs \\ PAGE \\ Non-co2 perturbation data - 1 Mt.xlsx`.

2. updated the inputs in `MimiIWG \\ src \\ core \\ PAGE_other_ghg.jl` to include the new perturbations:

```auto
_page_ch4_shocks = Dict()
_page_ch4_shocks[2010] = _page_xf["Sheet1"]["I4:M13"]
...
_page_ch4_shocks[2060] = _page_xf["Sheet1"]["AR4:AV13"]

_page_n2o_shocks = Dict()
_page_n2o_shocks[2010] = _page_xf["Sheet1"]["I18:M27"]
...
_page_n2o_shocks[2060] = _page_xf["Sheet1"]["AR18:AV27"]

```

This all worked great. Now, I would like to extend this exercise to 2070. However, PAGE timsteps jump 2070 (…,2050, 2060, 2080, 2100…), which is fine, but I believe this just means that I will need to add perturbation data for 2080 (not the 2070 of interest) since `MimiIWG` interpolates in between timesteps during the SC-X calculation. However, when going about this in a similar fashion as before, I:

1. added 2080 perturbation data to `MimiIWG \\ data \\ IWG_inputs \\ PAGE \\ Non-co2 perturbation data - 1 Mt.xlsx`.

2. updated the inputs in `MimiIWG \\ src \\ core \\ PAGE_other_ghg.jl` to include the new perturbations:

```auto
_page_ch4_shocks = Dict()
_page_ch4_shocks[2010] = _page_xf["Sheet1"]["I4:M13"]
...
_page_ch4_shocks[2080] = _page_xf["Sheet1"]["AY4:BC13"]

_page_n2o_shocks = Dict()
_page_n2o_shocks[2010] = _page_xf["Sheet1"]["I18:M27"]
...
_page_n2o_shocks[2080] = _page_xf["Sheet1"]["AY18:BC27"]

```

But now, when estimating SC-CH4/N2O through 2070 (or 2080) with `run_scc_mcs()` or even deterministically with `compute_scc()`, I am getting the error:

```auto
...
start_task at /cygdrive/c/buildbot/worker/package_win64/build/src\task.c:839
ERROR: KeyError: key 2080 not found
Stacktrace:
 [1] getindex(h::Dict{Any, Any}, key::Int64)
   @ Base .\dict.jl:482
 [2] _get_page_forcing_shock(scenario_num::Int64, gas::Symbol, year::Int64)
   @ MimiIWG C:\Users\bparthum\.julia\packages\MimiIWG\37nQw\src\core\PAGE_other_ghg.jl:37
 [3] perturb_marginal_page_emissions!(base::Mimi.Model, marginal::Mimi.Model, gas::Symbol, emissionyear::Int64)
   @ MimiIWG C:\Users\bparthum\.julia\packages\MimiIWG\37nQw\src\core\PAGE_main.jl:352
 [4] page_post_trial_func(mcs::Mimi.LatinHypercubeSimulationInstance, trialnum::Int64, ntimesteps::Int64, tup::Tuple{MimiIWG.scenario_choice, Float64})
   @ MimiIWG C:\Users\bparthum\.julia\packages\MimiIWG\37nQw\src\montecarlo\PAGE_mcs.jl:222
 [5] run(sim_def::Mimi.LatinHypercubeSimulationDef, models::Vector{Mimi.Model}, samplesize::Int64; ntimesteps::Int64, trials_output_filename::Nothing, results_output_dir::Nothing, pre_trial_func::Nothing, post_trial_func::typeof(MimiIWG.page_post_trial_func), scenario_func::typeof(MimiIWG.page_scenario_func), scenario_placement::Mimi.ScenarioLoopPlacement, scenario_args::Vector{Pair{Symbol, Vector{T} where T}}, results_in_memory::Bool)
   @ Mimi C:\Users\bparthum\.julia\packages\Mimi\SnUU7\src\mcs\montecarlo.jl:626
 [6] run_scc_mcs(model::MimiIWG.model_choice; gas::Union{Nothing, Symbol}, trials::Int64, perturbation_years::Vector{Int64}, discount_rates::Vector{Float64}, domestic::Bool, output_dir::Union{Nothing, String}, save_trials::Bool, save_md::Bool, tables::Bool)
   @ MimiIWG C:\Users\bparthum\.julia\packages\MimiIWG\37nQw\src\montecarlo\run_scc_mcs.jl:154
 [7] top-level scope
   @ REPL[151]:1

```

Is there another location where I need to specify the `2080` input? If so, why would this not be needed for the `2060` addition? Any thoughts on what this error is pointing to?

Thanks!!

enviro:

```auto
(MimiIWG-env) pkg> st
      Status `~\environments\MimiIWG-env\Project.toml`
  [b1d80750] MimiIWG v1.0.4-DEV `~\.julia\dev\MimiIWG`
  [295af30f] Revise v3.1.16
  [9a3f8284] Random

```

---

<div class="post-metadata">

**Author:** ![lrennels](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.mimiframework.org/lrennels/32/6_2.png) [@lrennels](https://forum.mimiframework.org/u/lrennels)\
**Post date:** [May 12, 2021, 8:11pm UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175/2 "2021-05-12T20:11:06Z")

</div>

Hi there,

It looks like your error is in the `_get_page_forcing_shock` function’s call to `getindex` i.e. this call `_page_ch4_shocks[year][:, col_num]` or this call `_page_n2o_shocks[year][:, col_num]`. It can’t find the key `2080` in the dictionary.

That’s relatively good news since it’s in a discrete helper function, let me try to replicate your problem!

Lisa

---

<div class="post-metadata">

**Author:** ![lrennels](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.mimiframework.org/lrennels/32/6_2.png) [@lrennels](https://forum.mimiframework.org/u/lrennels)\
**Post date:** [May 12, 2021, 8:36pm UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175/3 "2021-05-12T20:36:52Z")

</div>

@parthum does this look like the changes you made: [Add option for 2080 PAGE perturbatoin by lrennels · Pull Request #41 · rffscghg/MimiIWG.jl · GitHub](https://github.com/rffscghg/MimiIWG.jl/pull/41/files)

and is the call that was giving you the error something like this:

```auto
compute_scc(PAGE, :USG1; year = 2080)

```

As of now with these changes the above call runs fine on my machine, so I’m guessing there’s just some issue where you aren’t picking up the edited version of MimiIWG when you make the call so it doesn’t have an updated dictionary with the `2080` key.

Let me know, if my suspicion is correct that’s a good case because it should be easy to sort out together!

---

<div class="post-metadata">

**Author:** ![parthum](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.mimiframework.org/parthum/32/30_2.png) [@parthum](https://forum.mimiframework.org/u/parthum)\
**Post date:** [May 12, 2021, 8:53pm UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175/4 "2021-05-12T20:53:06Z")

</div>

That’s it! Any call will do, but for example:

`MimiIWG.compute_scc(PAGE, USG1, gas=:N2O, year=2080, discount=0.03)`

Hmm, okay. Getting my edits to be incorporated has been so inconsistent. I eventually started using `Revise` and it helped a little bit. But I’ve made these changes dozens of times now, with different forks, environments, changing before/after `using`, in `dev` and in the main `packages`. Restarting the repl ~30+ times… working with Mimi has not been exactly efficient when it comes to editing packages 😃 Clearly I’m missing something. Do you all have workflow suggestions for this?

Although, I suspected this might be the case, so I edited the values in the 2060 index, restarted the repl, re-precompiled, and it did catch those. Will keep experimenting.

Thank you!

---

<div class="post-metadata">

**Author:** ![lrennels](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.mimiframework.org/lrennels/32/6_2.png) [@lrennels](https://forum.mimiframework.org/u/lrennels)\
**Post date:** [May 12, 2021, 9:24pm UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175/5 "2021-05-12T21:24:08Z")

</div>

Oh boy that’s definitely not the experience we want, we should certainly fix that and I had planned to write something up about suggested workflow but for now let’s walk through it here.

Revise can certainly be helpful, what is your current workflow? My workflow is as follows:

* * *

**(1) Creating a Designated Environment**

For a package I work on developing quite a bit, such as MimiIWG, I have a named shared environment for developing that package. You only need to set this up once, and periodically update it with `pkg> up` to get latest package versions.

So in the case of MimiIWG I have `iwgdev` which I created with the following call

```auto
pkg> activate --shared iwgdev

```

and then I added a bunch of packages using the compat file as a guide to add what I need. In this case all the packages are added with no version except MimiFUND is pinned and we want to use the development folder of MimiIWG that is on our machine in `.julia/dev/MimiIWG`.

```auto
pkg> add Distributions, ExcelReaders, Interpolations, Mimi, MimiDICE2010, MimiPAGE2009, StatsBase, XLSX
pkg> add MimiFUND#v3.8.6
pkg> dev MimiIWG

```

Now I have a designated environment with all the packages and versions I want that I can come back to, and that will automatically link to the development version of my MimiIWG package, held in the `.Julia/dev/MimiIWG` folder which can be controlled with git commands for branching etc.

If I want to activate it, I just call the same `activate` function and then can check on the contents and packages versions with `st`

```auto
pkg> activate --shared iwgdev
pkg> st
      Status `~/.julia/environments/iwgdev/Project.toml`
  [31c24e10] Distributions v0.23.12
  [c04bee98] ExcelReaders v0.11.0
  [a98d9a8b] Interpolations v0.13.2
  [e4e893b0] Mimi v1.2.3
  [4f2c5d70] MimiDICE2010 v1.0.1
  [b3ba11de] MimiFUND v3.8.6 `https://github.com/fund-model/MimiFUND.jl.git#v3.8.6`
  [b1d80750] MimiIWG v1.0.4-DEV `~/.julia/dev/MimiIWG`
  [e25d4f70] MimiPAGE2009 v3.1.0
  [2913bbd2] StatsBase v0.33.8
  [fdbf4ff8] XLSX v0.7.6

```

Once in a while you’ll want to call `pkg> up` just to get the latest versions of packages in there.

You can, of course have many packages in one environment on their `dev` branch … but I try to avoid that because it can get really confusing!

**(2) Working on your package**

So whenever I want to work on MimiIWG, my steps are as follows. These are fairly IDE agnostic but I"ll add advice for VSCode specifically in the next post!

1. open the `.Julia/dev/MimiIWG` folder in my IDE for editing

2. start up a terminal and enter that folder and check which branch I’m on and change if needed ie.

```auto
lisarennels@Lisas-MBP-4 MimiIWG % git status
    On branch master
    Your branch is up to date with 'origin/master'.
    nothing to commit, working tree clean

lisarennels@Lisas-MBP-4 MimiIWG % git checkout 2090
    Switched to branch '2090'

```

1. open up a julia instance and activate my `iwgdev` environment with

```auto
pkg> activate --shared iwgdev

```

From there I should be able to edit and re-run, and if I use Revise I shouldn’t have to kill the REPL and start again unless I mess with the type system.

* * *

**I will continue but first so far does that make sense? Next I will add my editor-specific advice on opening your REPL properly, may I ask what IDE you are using? We use VSCode which I would say right now is by far the best for Julia but if you’re using a different one that’s also of course fine!**

---

<div class="post-metadata">

**Author:** ![lrennels](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.mimiframework.org/lrennels/32/6_2.png) [@lrennels](https://forum.mimiframework.org/u/lrennels)\
**Post date:** [May 12, 2021, 10:05pm UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175/8 "2021-05-12T22:05:47Z")

</div>

Alright, so assuming you are working within VSCode, here are some details.

When you open your `dev` folder, you can start a terminal up with **Terminal → New Terminal** (or the keyboard shortcut) and you’ll get your typical zsh/bash terminal with your current directory in the dev folder where you can to git commands etc.

Now to open a Julia folder where you can run commands, or interactively run your code line by line with **Ctrl-Enter** you can run **View → Command Pallete → Julia\> Start REPL.** which also probably has a shortcut on your machine or one can be set up I know that’s what David does. A layman’s shortcut I use is also just to go into your code and run some **blank** line with **Ctrl-Enter** to automatically spawn an interactive Julia instance.

Note that if you start a Julia REPL from the terminal with a shortcut you may have on your machine like `julia` you will be able activate your environment and run commands that you type out or copy/paste without a problem … but if you try to use **Ctrl-Enter** to interactively run code from your script it will not connect and open a new terminal instead … like what happens above if you call **Ctrl-Enter** on a blank line. Ask if that’s confusing I mess it up occasionally myself!

Ok so now you have a Julia REPL up and running but you might not be sure which environment you’re in! There are ways to set your defaults, but my tried and true method is just to check. You can just type `]` to see the name of the environment you are in to the right of the carrot, and change it with `pkg> activate --shared iwgdev` (Or whatever you named your environment) if needed. You can also use the VSCode handy trick of looking here at the bottom ribbon:

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/mimiframework/original/1X/a3b910e6cee5d046f432867758da5a1948ceaf05.png)

to see that I’m in environment v1.6 and on git branch 2090. If I want to change this, I can click on the environment which opens the command palette and I select iwgdev:

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/mimiframework/original/1X/7ee3375cc91435135c20f97a5f2af875cb1a0ef9.png)

and then see:

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/mimiframework/original/1X/39109f27fe3425663cb174571d125c1c84d87a19.png)

on the bottom. Notice I can see I’m on the branch “2090” in environment iwgdev!

---

<div class="post-metadata">

**Author:** ![lrennels](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.mimiframework.org/lrennels/32/6_2.png) [@lrennels](https://forum.mimiframework.org/u/lrennels)\
**Post date:** [May 12, 2021, 11:52pm UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175/9 "2021-05-12T23:52:03Z")

</div>

The instructions above are specific to wanting to directly edit and develop a package like MimiIWG.

If you aren’t developing a package and for example, working on a project that will serve as a new repository to back up the work for a specific academic paper or presentation (something like pairing FUND and FAIR [here](https://github.com/anthofflab/MimiFUND-MimiFAIR-Flat.jl)), then your workflow is simple a bit and I can walk you through that as well.

---

<div class="post-metadata">

**Author:** ![parthum](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.mimiframework.org/parthum/32/30_2.png) [@parthum](https://forum.mimiframework.org/u/parthum)\
**Post date:** [May 13, 2021, 1:46am UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175/10 "2021-05-13T01:46:26Z")

</div>

This is 🔥, thank you @lrennels.

First, I was eventually able to get the updates/edits to register after several restarts and coding everything before even starting a new instance (just a local VSCode in this case). So thank you, I have things running now and will switch to the cluster (jupyter) to run the iterations.

Second, this is such a helpful rundown of workflow tips. I was doing most of these, but perhaps not in the right sequence or as controlled/systematically. I’ll walk through this a couple of times and, as you mention, I’m sure us Mimi users (and new julia users) will catch on quick once we know the in’s. A writeup like this will likely be very useful to many, so thank you for getting these notes down. Tagging @tammyt123 in case she’d like in on this greatness.

Thanks again!!

---

<div class="post-metadata">

**Author:** ![lrennels](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.mimiframework.org/lrennels/32/6_2.png) [@lrennels](https://forum.mimiframework.org/u/lrennels)\
**Post date:** [May 13, 2021, 2:00am UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175/12 "2021-05-13T02:00:42Z")

</div>

I’m so glad! There are variations to these techniques, as described in Julia documentation or by other users, but this is my general workflow which works for me as of now.

Working on packages whose versions you then want to test is more involved than working on projects with set environments, since versioning is trickier. Thus for projects it can be simpler to try to stick to modifying existing models with the modification API, or run a Monte Carlo etc., but that does break down at some point and you need to get into the guts of things … like you are with MimiIWG! I’m glad you’re digging in and more than happy to help.

---

<div class="post-metadata">

**Author:** ![parthum](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.mimiframework.org/parthum/32/30_2.png) [@parthum](https://forum.mimiframework.org/u/parthum)\
**Post date:** [May 13, 2021, 2:11am UTC](https://forum.mimiframework.org/t/mimiiwg-page2009-adding-new-perturbation-keyerror/175/13 "2021-05-13T02:11:08Z")

</div>

You know, I think there are likely more efficient ways to accomplish what we’ve been trying to accomplish (as the documentation does a good job of directing folks to), but for some reason brute forcing it from the inside out has been more intuitive (and much less eloquent). Going through David’s FUND/SNEASY and the FUND/FAIR is a great way to become acquainted with more efficient handshakes. So thanks for sharing those, too.