HoweWeSolveIndShockConsumerType uses but does not define acronym EGM

"EGM" stands for "endogenous grid method"

notebooks/HoweWeSolveIndShockConsumerType.py uses this acronym without defining it.
One has to look around in other notebooks to find a link to the canonical resource.

http://www.econ2.jhu.edu/people/ccarroll/EndogenousGridpoints.pdf

PKM should add Durables code as a branch in DemARK

Model runs indefinitely without converging

KrusellSmith

# Construct the economy, make an initial history, then solve 

KSAgent.getEconomyData(KSEconomy) # Makes attributes of the economy, attributes of the agent

KSEconomy.makeAggShkHist() # Make a simulated history of the economy

# Solve macro problem by finding a fixed point for beliefs

KSEconomy.solve() # Solve the economy using the market method. 
# i.e. guess the saving function, and iterate until a fixed point

Missing file or directory

Micro-and-Macro-Implications-of-Very-Impatient-HHs

# Plot Lorenz curves for model with uniform distribution of time preference
from HARK.cstwMPC.SetupParamsCSTW import SCF_wealth, SCF_weights
from HARK.utilities import getLorenzShares, getPercentiles

pctiles = np.linspace(0.001,0.999,200)
sim_wealth = np.concatenate([ThisType.aLvlNow for ThisType in MyTypes])
SCF_Lorenz_points = getLorenzShares(SCF_wealth,weights=SCF_weights,percentiles=pctiles)
sim_Lorenz_points = getLorenzShares(sim_wealth,percentiles=pctiles)
plt.plot(pctiles,SCF_Lorenz_points,'--k')
plt.plot(pctiles,sim_Lorenz_points,'-b')
plt.xlabel('Percentile of net worth')
plt.ylabel('Cumulative share of wealth')
plt.show(block=False)

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-20e277edf0ec> in <module>
      1 # Plot Lorenz curves for model with uniform distribution of time preference
----> 2 from HARK.cstwMPC.SetupParamsCSTW import SCF_wealth, SCF_weights
      3 from HARK.utilities import getLorenzShares, getPercentiles
      4 
      5 pctiles = np.linspace(0.001,0.999,200)

/dhlib/lib/python3.7/site-packages/HARK/cstwMPC/SetupParamsCSTW.py in <module>
    115 # Import survival probabilities from SSA data
    116 data_location = os.path.dirname(os.path.abspath(__file__))
--> 117 f = open(data_location + '/' + 'USactuarial.txt','r')
    118 actuarial_reader = csv.reader(f,delimiter='\t')
    119 raw_actuarial = list(actuarial_reader)

FileNotFoundError: [Errno 2] No such file or directory: '/dhlib/lib/python3.7/site-packages/HARK/cstwMPC/USactuarial.txt'

NameError: name 'np' is not defined (2)

$ ipython KrusellSmith.py 
Using matplotlib backend: Qt5Agg
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
~/projects/econ-ark/DemARK/notebooks/KrusellSmith.py in <module>
    214     "PermGroFac" : [1.0],                  # Permanent income growth factor
    215 # New Parameters that we need now
--> 216     'MgridBase': np.array([0.1,0.3,0.6,
    217                            0.8,0.9,0.98,
    218                            1.0,1.02,1.1,

NameError: name 'np' is not defined

requirements.txt requires matplotlib 2.1.2 which is not compatible with python 3.8

I doubt that anything would break if we updated the requirement to a later version of matplotlib.

Problem when running Krusell-Smith

I get a warning message when I run a cell below.

# Put all agents into the economy
KSEconomy_sim = CobbDouglasMarkovEconomy(agents = MyTypes, **KSEconomyDictionary) 
KSEconomy_sim.AggShkDstn = KSAggShkDstn # Agg shocks are the same as defined earlier

for ThisType in MyTypes:
    ThisType.getEconomyData(KSEconomy_sim) # Makes attributes of the economy, attributes of the agent

KSEconomy_sim.makeAggShkHist() # Make a simulated prehistory of the economy
KSEconomy_sim.solve()          # Solve macro problem by getting a fixed point dynamic r
--------------------------------------------------------------------------------

**** WARNING: could not execute multiThreadCommands in HARK.core.Market.solveAgents()  so using the serial version instead. This will likely be slower. The multiTreadCommands() functions failed with the following error: 
 <class 'AttributeError'> : Can't pickle local object 'CobbDouglasEconomy.update.<locals>.<lambda>'

Errors while running DemARK notebooks [master branch, py3.7]

The following notebooks gives errors (and why) using current master branch.

Alternative Combinations of Parameter Values - this one has an exercise/interactive notebook as users need to create consumer problems first then run the second cell.
A Gentle Introduction to HARK - this one also has a small exercise.
IncExpectationExample_Roszypal-Schlafmann - data not available in the repo, tries to read f = open('/Volumes/Data/Job/Discuss/2018-07_NBER_Behavioral-Macro/cstwMPC/Code/Python/Results/LCbetaPointNetWorthLorenzFig.txt','r') [should be changed]
Structural-Estimates-From-Empirical-MPCs-Fagereng-et-al- T_kill referenced before assignment. [Fixed]

TractableBufferStock-Interactive

@llorracc says here: econ-ark/HARK#472 (comment)

While we're at it, though, let's change the name of
"TractableBufferStockQuickDemo" to "TractableBufferStock-Interactive" or
something like that. (Open it and you'll see why).

ModuleNotFoundError: No module named 'statsmodels'

$ ipython KeynesFriedmanModigliani.py  
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
~/projects/econ-ark/DemARK/notebooks/KeynesFriedmanModigliani.py in <module>
     31 import datetime as dt
     32 import scipy.stats as stats
---> 33 import statsmodels.formula.api as sm
     34 from copy  import deepcopy
     35 

ModuleNotFoundError: No module named 'statsmodels'

binder/runtime.txt specifies python 2.7

@DrDrij:

We've moved to python 3 as the default, and do not want to guarantee that future content will work on both py27 and py3. Need to:

Make sure all existing content works for py3
Update the runtime.txt to specify py3

Can you take this on?

NameError: name 'np' is not defined

$ ipython GenIncProcessModel.py 
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
~/projects/econ-ark/DemARK/notebooks/GenIncProcessModel.py in <module>
    178 # More parameters specific to "Explicit Permanent income" shock model
    179     "cycles": 0,
--> 180     "pLvlPctiles" : np.concatenate(([0.001, 0.005, 0.01, 0.03], np.linspace(0.05, 0.95, num=19),[0.97, 0.99, 0.995, 0.999])),
    181     "PermGroFac" : [1.0],                  # Permanent income growth factor - long run permanent income growth doesn't work yet
    182 }

NameError: name 'np' is not defined

Restore Uncertainty and the Savings Rate

The Uncertainty and the Savings Rate DemARK depends on HARK.cstwMPC code, and not just on the dataset.

This HARK.cstwMPC code has a nonstandard way of representing shock distributions (i.e., not as a list, even when an infinite horizon is used) that is incompatible with recent revisions to the HARK distribution handling code that's being included in version 0.10.6

This DemARK will soon be pulled out of master because it fails when run.

This issue is for getting it working again once an updated and stable cstwMPC version is available.

Missing file or directory

Structural-Estimates-From-Empirical-MPCs-Fagereng-et-al

# Import needed tools from HARK

from HARK.utilities import approxUniform, getPercentiles
from HARK.parallel import multiThreadCommands
from HARK.estimation import minimizeNelderMead
from HARK.ConsumptionSaving.ConsIndShockModel import *
from HARK.cstwMPC.SetupParamsCSTW import init_infinite

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-2-3fc2b9f99f07> in <module>
      5 from HARK.estimation import minimizeNelderMead
      6 from HARK.ConsumptionSaving.ConsIndShockModel import *
----> 7 from HARK.cstwMPC.SetupParamsCSTW import init_infinite

/dhlib/lib/python3.7/site-packages/HARK/cstwMPC/SetupParamsCSTW.py in <module>
    115 # Import survival probabilities from SSA data
    116 data_location = os.path.dirname(os.path.abspath(__file__))
--> 117 f = open(data_location + '/' + 'USactuarial.txt','r')
    118 actuarial_reader = csv.reader(f,delimiter='\t')
    119 raw_actuarial = list(actuarial_reader)

FileNotFoundError: [Errno 2] No such file or directory: '/dhlib/lib/python3.7/site-packages/HARK/cstwMPC/USactuarial.txt'

More clearly indicate when a DemARK is instructional and cannot run without user input

Several of our notebooks (such as those listed in #37 and #38) require changes from users to run correctly. This can be confusing for people not coming to the notebooks in the context of a class or workshop. How can we more clearly label which notebooks are expected to break if run immediately upon loading?

Structural Estimates : "Warning: Maximum number of function evaluations has been exceeded."

The Structural Estimates DemARK is running suspiciously wrong. The checked in version runs for 40 minutes; local it's running for 20 minutes with the following message:

https://gist.github.com/sbenthall/1f16a6764b17024eb3be3dc9a68852c0

There's a warning: "Warning: Maximum number of function evaluations has been exceeded."
This is associated with the underlying scipy.optimize.fmin method, which has a default "maxfun" argument for maximum function evaluations. This default is being used by the underlying HARK call.

Some ideas:

The objective function FagerengObjFunc defined in the notebook may have too many functions inside it because of the new distribution code or other underlying HARK implementations
The parameters have been reset somehow so that there is no unconstrained minimum for this objective function, causing the fmin method to search endlessly

I'm wondering who has a deep understanding of what this notebook is trying to accomplish such that they can explain what's going on in the Fagereng objective function. The GitHub history indicates that @llorracc and @MridulS are the others who have touched the notebook in the repository.

DCT Illustration

Take the DCT Illustration materials from the BayerLuetticke REMARK and make it into a DemARK

This follows:
econ-ark/REMARK#47

NameError: name 'MyTypes' is not defined

$ ipython Alternative-Combos-Of-Parameter-Values.py 
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
~/projects/econ-ark/DemARK/notebooks/Alternative-Combos-Of-Parameter-Values.py in <module>
    114 
    115 # + {"collapsed": true}
--> 116 for ThisType in tqdm(MyTypes):
    117     ThisType.solve()
    118     ThisType.initializeSim()

NameError: name 'MyTypes' is not defined

problem printing graphs with matplotlib whithin KrusselSmith

After running the first cell of this notebook (KrusslSmith), the matplotlib dont work well.

the first cell is:

Import generic setup tools

This is a jupytext paired notebook that autogenerates KrusellSmith.py

which can be executed from a terminal command line via "ipython KrusellSmith.py"

But a terminal does not permit inline figures, so we need to test jupyter vs terminal

Google "how can I check if code is executed in the ipython notebook"

def in_ipynb():
try:
if str(type(get_ipython())) == "<class 'ipykernel.zmqshell.ZMQInteractiveShell'>":
return True
else:
return False
except NameError:
return False

Determine whether to make the figures inline (for spyder or jupyter)

vs whatever is the automatic setting that will apply if run from the terminal

if in_ipynb():
# %matplotlib inline generates a syntax error when run from the shell
# so do this instead
get_ipython().run_line_magic('matplotlib', 'inline')
else:
get_ipython().run_line_magic('matplotlib', 'auto')

Import the plot-figure library matplotlib

import matplotlib
import matplotlib.pyplot as plt
matplotlib.rcParams['text.usetex'] = True

import sys
import os
from copy import copy
from HARK.utilities import plotFuncs, plotFuncsDer

then if i try to plot (for example: "fig, ax = plt.subplots()")
i got error (very long):

Error in callback <function install_repl_displayhook..post_execute at 0x000001F9F4EE1620> (for post_execute):

FileNotFoundError Traceback (most recent call last)
~.conda\envs\econ-ark\lib\site-packages\matplotlib\texmanager.py in _run_checked_subprocess(self, command, tex)
303 cwd=self.texcache,
--> 304 stderr=subprocess.STDOUT)
305 except FileNotFoundError as exc:

~.conda\envs\econ-ark\lib\subprocess.py in check_output(timeout, *popenargs, **kwargs)
394 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
--> 395 **kwargs).stdout
396

~.conda\envs\econ-ark\lib\subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
471
--> 472 with Popen(*popenargs, **kwargs) as process:
473 try:

~.conda\envs\econ-ark\lib\subprocess.py in init(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)
774 errread, errwrite,
--> 775 restore_signals, start_new_session)
776 except:

~.conda\envs\econ-ark\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)
1177 os.fspath(cwd) if cwd is not None else None,
-> 1178 startupinfo)
1179 finally:

FileNotFoundError: [WinError 2] The system cannot find the file specified

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)
~.conda\envs\econ-ark\lib\site-packages\matplotlib\pyplot.py in post_execute()
107 def post_execute():
108 if matplotlib.is_interactive():
--> 109 draw_all()
110
111 # IPython >= 2

~.conda\envs\econ-ark\lib\site-packages\matplotlib_pylab_helpers.py in draw_all(cls, force)
126 for f_mgr in cls.get_all_fig_managers():
127 if force or f_mgr.canvas.figure.stale:
--> 128 f_mgr.canvas.draw_idle()
129
130 atexit.register(Gcf.destroy_all)

~.conda\envs\econ-ark\lib\site-packages\matplotlib\backend_bases.py in draw_idle(self, *args, **kwargs)
1905 if not self._is_idle_drawing:
1906 with self._idle_draw_cntx():
-> 1907 self.draw(*args, **kwargs)
1908
1909 def draw_cursor(self, event):

~.conda\envs\econ-ark\lib\site-packages\matplotlib\backends\backend_agg.py in draw(self)
386 self.renderer = self.get_renderer(cleared=True)
387 with RendererAgg.lock:
--> 388 self.figure.draw(self.renderer)
389 # A GUI class may be need to update a window using this draw, so
390 # don't forget to call the superclass.

~.conda\envs\econ-ark\lib\site-packages\matplotlib\artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
36 renderer.start_filter()
37
---> 38 return draw(artist, renderer, *args, **kwargs)
39 finally:
40 if artist.get_agg_filter() is not None:

~.conda\envs\econ-ark\lib\site-packages\matplotlib\figure.py in draw(self, renderer)
1707 self.patch.draw(renderer)
1708 mimage._draw_list_compositing_images(
-> 1709 renderer, self, artists, self.suppressComposite)
1710
1711 renderer.close_group('figure')

~.conda\envs\econ-ark\lib\site-packages\matplotlib\image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
133 if not_composite or not has_images:
134 for a in artists:
--> 135 a.draw(renderer)
136 else:
137 # Composite any adjacent images together

~.conda\envs\econ-ark\lib\site-packages\matplotlib\artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
36 renderer.start_filter()
37
---> 38 return draw(artist, renderer, *args, **kwargs)
39 finally:
40 if artist.get_agg_filter() is not None:

~.conda\envs\econ-ark\lib\site-packages\matplotlib\axes_base.py in draw(self, renderer, inframe)
2643 renderer.stop_rasterizing()
2644
-> 2645 mimage._draw_list_compositing_images(renderer, self, artists)
2646
2647 renderer.close_group('axes')

~.conda\envs\econ-ark\lib\site-packages\matplotlib\image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
133 if not_composite or not has_images:
134 for a in artists:
--> 135 a.draw(renderer)
136 else:
137 # Composite any adjacent images together

~.conda\envs\econ-ark\lib\site-packages\matplotlib\artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
36 renderer.start_filter()
37
---> 38 return draw(artist, renderer, *args, **kwargs)
39 finally:
40 if artist.get_agg_filter() is not None:

~.conda\envs\econ-ark\lib\site-packages\matplotlib\axis.py in draw(self, renderer, *args, **kwargs)
1204 ticks_to_draw = self._update_ticks()
1205 ticklabelBoxes, ticklabelBoxes2 = self._get_tick_bboxes(ticks_to_draw,
-> 1206 renderer)
1207
1208 for tick in ticks_to_draw:

~.conda\envs\econ-ark\lib\site-packages\matplotlib\axis.py in _get_tick_bboxes(self, ticks, renderer)
1149 """Return lists of bboxes for ticks' label1's and label2's."""
1150 return ([tick.label1.get_window_extent(renderer)
-> 1151 for tick in ticks if tick.label1.get_visible()],
1152 [tick.label2.get_window_extent(renderer)
1153 for tick in ticks if tick.label2.get_visible()])

~.conda\envs\econ-ark\lib\site-packages\matplotlib\axis.py in (.0)
1149 """Return lists of bboxes for ticks' label1's and label2's."""
1150 return ([tick.label1.get_window_extent(renderer)
-> 1151 for tick in ticks if tick.label1.get_visible()],
1152 [tick.label2.get_window_extent(renderer)
1153 for tick in ticks if tick.label2.get_visible()])

~.conda\envs\econ-ark\lib\site-packages\matplotlib\text.py in get_window_extent(self, renderer, dpi)
888 raise RuntimeError('Cannot get window extent w/o renderer')
889
--> 890 bbox, info, descent = self._get_layout(self._renderer)
891 x, y = self.get_unitless_position()
892 x, y = self.get_transform().transform_point((x, y))

~.conda\envs\econ-ark\lib\site-packages\matplotlib\text.py in _get_layout(self, renderer)
289 _, lp_h, lp_d = renderer.get_text_width_height_descent(
290 "lp", self._fontproperties,
--> 291 ismath="TeX" if self.get_usetex() else False)
292 min_dy = (lp_h - lp_d) * self._linespacing
293

~.conda\envs\econ-ark\lib\site-packages\matplotlib\backends\backend_agg.py in get_text_width_height_descent(self, s, prop, ismath)
199 fontsize = prop.get_size_in_points()
200 w, h, d = texmanager.get_text_width_height_descent(
--> 201 s, fontsize, renderer=self)
202 return w, h, d
203

~.conda\envs\econ-ark\lib\site-packages\matplotlib\texmanager.py in get_text_width_height_descent(self, tex, fontsize, renderer)
446 else:
447 # use dviread. It sometimes returns a wrong descent.
--> 448 dvifile = self.make_dvi(tex, fontsize)
449 with dviread.Dvi(dvifile, 72 * dpi_fraction) as dvi:
450 page, = dvi

~.conda\envs\econ-ark\lib\site-packages\matplotlib\texmanager.py in make_dvi(self, tex, fontsize)
336 self._run_checked_subprocess(
337 ["latex", "-interaction=nonstopmode", "--halt-on-error",
--> 338 texfile], tex)
339 for fname in glob.glob(basefile + '*'):
340 if not fname.endswith(('dvi', 'tex')):

~.conda\envs\econ-ark\lib\site-packages\matplotlib\texmanager.py in _run_checked_subprocess(self, command, tex)
306 raise RuntimeError(
307 'Failed to process string with tex because {} could not be '
--> 308 'found'.format(command[0])) from exc
309 except subprocess.CalledProcessError as exc:
310 raise RuntimeError(

RuntimeError: Failed to process string with tex because latex could not be found

FileNotFoundError Traceback (most recent call last)
~.conda\envs\econ-ark\lib\site-packages\matplotlib\texmanager.py in _run_checked_subprocess(self, command, tex)
303 cwd=self.texcache,
--> 304 stderr=subprocess.STDOUT)
305 except FileNotFoundError as exc:

~.conda\envs\econ-ark\lib\subprocess.py in check_output(timeout, *popenargs, **kwargs)
394 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
--> 395 **kwargs).stdout
396

~.conda\envs\econ-ark\lib\subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
471
--> 472 with Popen(*popenargs, **kwargs) as process:
473 try:

~.conda\envs\econ-ark\lib\subprocess.py in init(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)
774 errread, errwrite,
--> 775 restore_signals, start_new_session)
776 except:

~.conda\envs\econ-ark\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)
1177 os.fspath(cwd) if cwd is not None else None,
-> 1178 startupinfo)
1179 finally:

FileNotFoundError: [WinError 2] The system cannot find the file specified

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)
~.conda\envs\econ-ark\lib\site-packages\IPython\core\formatters.py in call(self, obj)
339 pass
340 else:
--> 341 return printer(obj)
342 # Finally look for special method names
343 method = get_real_method(obj, self.print_method)

~.conda\envs\econ-ark\lib\site-packages\IPython\core\pylabtools.py in (fig)
242
243 if 'png' in formats:
--> 244 png_formatter.for_type(Figure, lambda fig: print_figure(fig, 'png', **kwargs))
245 if 'retina' in formats or 'png2x' in formats:
246 png_formatter.for_type(Figure, lambda fig: retina_figure(fig, **kwargs))

~.conda\envs\econ-ark\lib\site-packages\IPython\core\pylabtools.py in print_figure(fig, fmt, bbox_inches, **kwargs)
126
127 bytes_io = BytesIO()
--> 128 fig.canvas.print_figure(bytes_io, **kw)
129 data = bytes_io.getvalue()
130 if fmt == 'svg':

~.conda\envs\econ-ark\lib\site-packages\matplotlib\backend_bases.py in print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, bbox_inches, **kwargs)
2054 orientation=orientation,
2055 dryrun=True,
-> 2056 **kwargs)
2057 renderer = self.figure._cachedRenderer
2058 bbox_artists = kwargs.pop("bbox_extra_artists", None)

~.conda\envs\econ-ark\lib\site-packages\matplotlib\backends\backend_agg.py in print_png(self, filename_or_obj, metadata, pil_kwargs, *args, **kwargs)
525
526 else:
--> 527 FigureCanvasAgg.draw(self)
528 renderer = self.get_renderer()
529 with cbook._setattr_cm(renderer, dpi=self.figure.dpi), \