breakthrough-energy / postreise Goto Github PK

463 branches in Western have 0 sum powerflow in the base case. Branches are transformer windings (281) and lines (182). There is a need to characterize the lines further to understand this issue.

Minor fixes are needed to Bokeh maps for web compatibility. These include, for all maps:

1. remove tool tips (hover over tips)
2. remove 'save' functionality (or confirm it is working and leave it there if we decide to go that route)
3. remove titles (title will be assigned on the web page)
4. make font larger / legible for all legends or other text labels

• connect blob storage to site (create select dropdown and test)
• add scenario comparison feature and test

E.g. SOx, NOx, PM2.5, PM10, mercury, etc. We can follow the 'simple' method at first to just linearly scale MWh to tons of pollutants, or we can use the heat-rate calculations to translate MMBTu to tons of pollutants.

Data (PowerSimData)

• NOx
• SOx
• PM2.5
• PM10
• Mercury

Analysis (PostREISE)

• NOx
• SOx
• PM2.5
• PM10
• Mercury

Same as for PowerSimData#198

Extract from the MATLAB output files the PF profile for DC lines

This task is to keep track of all the tests that need to be added or updated in CENACS in an effort to burn down technical debt. I don't expect this will be complete by the end of the WTT90s milestone but instead it should be an ongoing process until we are completely caught up.

Kamilah

• collapsible text
• dashboard test (just check that existing tests pass)
• dashboard slice - mostly done, needs update filter function
• all graphs that have "format data"
  • focus on the format data functions since they're isolated
  • investigate an apollo testing guide to see if it's easy to test the useQuery part
  • generation pie
  • generation bar
  • generation stacked line
  • curtailment line
• util
  • sortByTimestamp
  • sortByResourceType
  • increaseUnitSize
• docker - investigate and see if testing is worth the time investment

Merrielle

• header
• select
• graphColumn
• scenario toolbar
• scrollytelling
• bokehGraph (waiting on merge from S Dan: https://github.com/Breakthrough-Energy/CENACS/pull/80)
• graphWrapper

Mapping visualizations that work for WECC and ERCOT do not perform well for Eastern, due to the difference in size and shapes of states in the Eastern states, and due to the difference in geographic scale generally. There's a need to re-parameterize these visualization functions to provide the user with options that will work well for Eastern, and ultimately for the whole US. Here's an example of the problem, using bar plots of emissions per state:

After simulation do transmission line analyse and save results such that data is available when requested for plotting.

Transmission utilization mapping is currently not set up in a way that is suitable for mapping the Eastern Interconnection. Using the settings that work for Western, the Eastern map is not legible (image below). Additionally, map would be improved by adding state boundary lines. :

For the web interface, we are currently showing emission goals color coded by state. We need a way to implement this within the code base. I am putting this in postreise because I think we may want to integrate this with our emissions mapping (either side by side or potentially with emissions on top of the color coded states map):

Generation from storage will be added to the plots

Each interval SolveTime and Optimization time is saved.
We want to extract that data after the simulation to look at latency.

Modification need to be done to PostREISE extract-data.py

Projection tools from one coordinates system to another one in pyproj differs from 1.9.6 currently used and newer versions.

We want to update pyproj because the 1.9.6 wheels are not available for python 3.8.x on Windows. This will certainly be an issue when we open source PostREISE.

Not sure if this is something we should document, but matplotlib install failed until after I added some os packages. In my case, this was using poetry (same packages and versions as requirements.txt) and ubuntu 18.04 in a wsl environment. The error I got had this message

REQUIRED DEPENDENCIES AND EXTENSIONS
                     numpy: yes [version 1.16.0]
          install_requires: yes [handled by setuptools]
                    libagg: yes [pkg-config information for 'libagg' could not
                            be found. Using local copy.]
                  freetype: no  [The C/C++ header for freetype2 (ft2build.h)
                            could not be found.  You may need to install the
                            development package.]
                       png: no  [pkg-config information for 'libpng' could not
                            be found.]
                     qhull: yes [pkg-config information for 'libqhull' could not
                            be found. Using local copy.]

and install worked after running

sudo apt install libpng-dev libfreetype6-dev

Spot check script integration is finished in #264
File directory: Dropbox (IVL)/Explorations/BainanX/spot check script integration/

Researchers are going to use the integrated script for scenario post analysis in short term. In the meantime, we would like to gradually move the features that are implemented in the script into the code base together with corresponding tests.

Before doing so, we have following tasks to clear:

• A big picture design of the architecture and interface of data/calculations/tables/plots in PostREISE, so that the code performance in terms of time and memory space is optimized.
• Refactor existing code, such as AnalyzePG to follow the rules we set up above.

Additional comments repost during the presentation on 05/27/2020:
@danielolsen : (12) might be easier to interpret with a log-scale X-axis
@danlivengood : It would be great if the plots (applicable bars, scatters, etc) could have the capabilities to sort along axes, such as states in alphabet order, which is currently based on the default order in our grid (Eastern, Western, Texas)

Following the renaming of a module in PowerSimData (see Breakthrough-Energy/PowerSimData#264), some import statements need to be updated.

We have been showing plots like this one, which was done using image editing and is not automated. There is a need to do this in an automated fashion, integrated into the code base.

Write functions that check input parameters of analysis and plotting functions. Write tests for these functions.

Delete the MAT-file enclosing the simulation outputs (PG, PF, LMP, CONGU, CONGL) once those have have been extracted and saved as pickle files.

Use generators constants gathered in powersimdata/network/us_tamu/constants/plants.py in analysis and plotting modules of PostREISE when possible.

Various postreise plotting scripts needed to be changed to output a file type that would work with the bokeh_upload_demo.ipynb to upload bokeh map files to blob storage for use on the website.

1. Utilization:
2. risk
3. binding incidents
4. LMP (locational marginal price):
5. Carbon Emissions:
6. Color code by state goals:
7. All transmission, color code by interconnection:

• Implement calculate_averaged_congestion in the postreise.extract.extract_data module that calculates the time averaged CONGU and CONGL for each branch.
• Write tests.
• Use the above function to generate an output pickle file enclosing the statistics, e.g., i_AVERAGED_CONG.pkl where: i is the scenario id. The data frame will have branch_id as index and CONGL and CONGU as columns.
• Implement the get_averaged_cong method in the Analyze object that download/load the file.
• Add the download/load functionality to the OutputData object.

• create a docs folder
• write tutorial
• create a toctree
• write succinct README for the repository as pandas does

Some post calculations themselves are light but could be heavy when the data must be retrieved from the server and loaded into memory on a local computer. Having these calculations on the server and passing the results only when query will speed-up scenario validation and analysis.

Handle communication between server and local machine to pull and push data:

• create InputData object dedicated to input data (demand, hydro/solar/wind profiles and change table);
• create OutputData object dedicated to simulation outputs (PG and PF).
• Rename TransferData object as PullData and add new method to class;
• Create PushData object.

Modify the AnalyzePG class to handle Texas and Texas-Western scenarios

Functions that checks the type and value of input parameters in analysis functions are needed. Some of them can be found in multiple modules with slight variation.

Create a module that gathers all the check functions to avoid duplicates and save time when a new analysis functions is developed.

We need a centralized place on the dashboard to remove filters as necessary

There is often some amount of 'negative curtailment' in our results. I think this is due to our use of the barrier method for solutions (with no time-consuming crossover step at the end), so there is sometimes a small fractional amount more generation from renewable resources than the profile would suggest is possible. However, ANY negative value, no matter how small, makes the stack plot error out. I imagine we could fix this fairly easily with a np.clip call.

Example:

>>> from postreise.plot.analyze_pg import AnalyzePG as apg
>>> from powersimdata.scenario.scenario import Scenario
>>> s = Scenario('398')
SCENARIO: test | EasternBase_2020_1

--> State
analyze
>>> zonelist = [
    'Maine',
    'New Hampshire',
    'Vermont',
    'Massachusetts',
    'Rhode Island',
    'Connecticut',
    'New York City',
    'Upstate New York',
    'New Jersey',
    'Pennsylvania Eastern',
    'Pennsylvania Western',
    'Delaware',
    'Maryland',
    'Virginia Mountains',
    'Virginia Tidewater',
    'North Carolina',
    'Western North Carolina',
    'South Carolina',
    'Georgia North',
    'Georgia South',
    'Florida Panhandle',
    'Florida North',
    'Florida South',
    'Alabama',
    'Mississippi',
    'Tennessee',
    'Kentucky',
    'West Virginia',
    'Ohio River',
    'Ohio Lake Erie',
    'Michigan Northern',
    'Michigan Southern',
    'Indiana',
    'Chicago North Illinois',
    'Illinois Downstate',
    'Wisconsin',
    'Minnesota Northern',
    'Minnesota Southern',
    'Iowa',
    'Missouri East',
    'Missouri West',
    'Arkansas',
    'Louisiana',
    'East Texas',
    'Texas Panhandle',
    'New Mexico Eastern',
    'Oklahoma',
    'Kansas',
    'Nebraska',
    'South Dakota',
    'North Dakota',
    'Montana Eastern',
    'Eastern'
    ]
>>> selected_resources = ['coal', 'dfo', 'geothermal', 'hydro', 'ng', 'nuclear', 'solar', 'wind']
>>> stack = apg(s, \
    ('2016-06-01-00', '2016-06-20-23', 'utc', 'H'), \
    zonelist, \
    selected_resources, \
    'stacked', normalize=False)
--> Loading PG
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading dcline
Loading sub
Loading bus2sub
--> Loading demand
Multiply demand in Maine (#1) by 0.98
Multiply demand in New Hampshire (#2) by 0.98
Multiply demand in Vermont (#3) by 0.98
Multiply demand in Massachusetts (#4) by 0.98
Multiply demand in Rhode Island (#5) by 0.98
Multiply demand in Connecticut (#6) by 0.98
Multiply demand in New York City (#7) by 0.99
Multiply demand in Upstate New York (#8) by 0.99
Multiply demand in New Jersey (#9) by 1.04
Multiply demand in Pennsylvania Eastern (#10) by 1.02
Multiply demand in Pennsylvania Western (#11) by 1.02
Multiply demand in Delaware (#12) by 1.04
Multiply demand in Maryland (#13) by 1.01
Multiply demand in Virginia Mountains (#14) by 1.06
Multiply demand in Virginia Tidewater (#15) by 1.06
Multiply demand in North Carolina (#16) by 1.04
Multiply demand in Western North Carolina (#17) by 1.04
Multiply demand in South Carolina (#18) by 1.04
Multiply demand in Georgia North (#19) by 1.01
Multiply demand in Georgia South (#20) by 1.01
Multiply demand in Florida Panhandle (#21) by 1.04
Multiply demand in Florida North (#22) by 1.04
Multiply demand in Florida South (#23) by 1.04
Multiply demand in Alabama (#24) by 1.01
Multiply demand in Mississippi (#25) by 1.06
Multiply demand in Tennessee (#26) by 1.01
Multiply demand in Kentucky (#27) by 1.04
Multiply demand in West Virginia (#28) by 1.04
Multiply demand in Ohio River (#29) by 1.00
Multiply demand in Ohio Lake Erie (#30) by 1.00
Multiply demand in Michigan Northern (#31) by 1.02
Multiply demand in Michigan Southern (#32) by 1.02
Multiply demand in Indiana (#33) by 1.05
Multiply demand in Chicago North Illinois (#34) by 1.02
Multiply demand in Illinois Downstate (#35) by 1.02
Multiply demand in Wisconsin (#36) by 1.04
Multiply demand in Minnesota Northern (#37) by 1.03
Multiply demand in Minnesota Southern (#38) by 1.03
Multiply demand in Iowa (#39) by 1.06
Multiply demand in Missouri East (#40) by 1.04
Multiply demand in Missouri West (#41) by 1.04
Multiply demand in Arkansas (#42) by 1.04
Multiply demand in Louisiana (#43) by 1.02
Multiply demand in East Texas (#44) by 1.07
Multiply demand in Texas Panhandle (#45) by 1.07
Multiply demand in New Mexico Eastern (#46) by 1.02
Multiply demand in Oklahoma (#47) by 1.04
Multiply demand in Kansas (#48) by 1.02
Multiply demand in Nebraska (#49) by 1.02
Multiply demand in South Dakota (#50) by 1.07
Multiply demand in North Dakota (#51) by 1.06
Multiply demand in Montana Eastern (#52) by 1.05
--> Loading solar
--> Loading wind
--> Loading hydro

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-8f3dae0f2635> in <module>
      3             zonelist,
      4             selected_resources,
----> 5             'stacked', normalize=False)
      6 stack.get_plot(save=False)
      7 data_stack = stack.get_data()

c:\python37\lib\site-packages\postreise\plot\analyze_pg.py in __init__(self, scenario, time, zones, resources, kind, normalize, seed)
    137 
    138         if kind == 'stacked':
--> 139             self._do_stacked(time[0], time[1], time[2])
    140         elif kind == 'comp':
    141             self._do_comp(time[0], time[1], time[2])

c:\python37\lib\site-packages\postreise\plot\analyze_pg.py in _do_stacked(self, start_date, end_date, tz)
    425             self.tz = self.zone2time[z] if tz == 'local' else tz
    426             self._set_date_range(start_date, end_date)
--> 427             self.data.append(self._get_stacked(z))
    428 
    429     def _get_stacked(self, zone):

c:\python37\lib\site-packages\postreise\plot\analyze_pg.py in _get_stacked(self, zone)
    515             ax = pg_stack[list(type2label.keys())].tz_localize(None).rename(
    516                 columns=type2label).plot.area(
--> 517                 color=type2color, alpha=0.7, ax=ax)
    518 
    519             demand.tz_localize(None).plot(color='red', lw=4, ax=ax)

c:\python37\lib\site-packages\pandas\plotting\_core.py in area(self, x, y, **kwargs)
   1306             >>> ax = df.plot.area(x='day')
   1307         """
-> 1308         return self(kind="area", x=x, y=y, **kwargs)
   1309 
   1310     def pie(self, **kwargs):

c:\python37\lib\site-packages\pandas\plotting\_core.py in __call__(self, *args, **kwargs)
    792                     data.columns = label_name
    793 
--> 794         return plot_backend.plot(data, kind=kind, **kwargs)
    795 
    796     def line(self, x=None, y=None, **kwargs):

c:\python37\lib\site-packages\pandas\plotting\_matplotlib\__init__.py in plot(data, kind, **kwargs)
     60             kwargs["ax"] = getattr(ax, "left_ax", ax)
     61     plot_obj = PLOT_CLASSES[kind](data, **kwargs)
---> 62     plot_obj.generate()
     63     plot_obj.draw()
     64     return plot_obj.result

c:\python37\lib\site-packages\pandas\plotting\_matplotlib\core.py in generate(self)
    279         self._compute_plot_data()
    280         self._setup_subplots()
--> 281         self._make_plot()
    282         self._add_table()
    283         self._make_legend()

c:\python37\lib\site-packages\pandas\plotting\_matplotlib\core.py in _make_plot(self)
   1077                 stacking_id=stacking_id,
   1078                 is_errorbar=is_errorbar,
-> 1079                 **kwds
   1080             )
   1081             self._add_legend_handle(newlines[0], label, index=i)

c:\python37\lib\site-packages\pandas\plotting\_matplotlib\core.py in _ts_plot(cls, ax, x, data, style, **kwds)
   1118         ax._plot_data.append((data, cls._kind, kwds))
   1119 
-> 1120         lines = cls._plot(ax, data.index, data.values, style=style, **kwds)
   1121         # set date formatter, locators and rescale limits
   1122         format_dateaxis(ax, ax.freq, data.index)

c:\python37\lib\site-packages\pandas\plotting\_matplotlib\core.py in _plot(cls, ax, x, y, style, column_num, stacking_id, is_errorbar, **kwds)
   1233         if column_num == 0:
   1234             cls._initialize_stacker(ax, stacking_id, len(y))
-> 1235         y_values = cls._get_stacked_values(ax, stacking_id, y, kwds["label"])
   1236 
   1237         # need to remove label, because subplots uses mpl legend as it is

c:\python37\lib\site-packages\pandas\plotting\_matplotlib\core.py in _get_stacked_values(cls, ax, stacking_id, values, label)
   1156             "When stacked is True, each column must be either "
   1157             "all positive or negative."
-> 1158             "{0} contains both positive and negative values".format(label)
   1159         )
   1160 

ValueError: When stacked is True, each column must be either all positive or negative.Solar Curtailment contains both positive and negative values

Enable the AnalyzePG class to make plot in the full U.S grid. This is the las feature that will be developed in that class before refactoring the it along with other analysis and plotting modules in PostREISE.

LMP maps need to be improved for the website.

1. simplify by having only one color bar legend (instead of one for LMP and one for shadowprice).
2. aesthetic changes so that mapping works well for whole united states (dot size, consolidation of buses at same lat/lon)
3. speed up improvements (takes 10min24sec to run for whole USA scenario 462 initially).
4. continuous color bar, scenario independent, to better differentiate between scenarios and show cost gradient more clearly

Save MAT-file that encloses all the information regarding the grid, generation cost, heat rate curve passed to the simulation engine. This file is saved in INPUT_DIR. When using the get_grid method of the Analyze object., this file is loaded and read by the MATReader object.

Some modules are not listed in their respective __init__ files.

Reorganize postreise/analyze.. We are leaning towards the following strucutre:

postreise/analyze/transmission contains congestion.py, mwmiles.py, statelines.py, utilization.py (renamed from transmission.py).
postreise/analyze/generation contains carbon.py, curtailment.py, summarize.py

The helpers module would stay in postreise/analyze for the moment since functions defined therein are grid related and hence could be used in both transmission and congestion analyses.

When communicating with the server from a window machine the path is set window specific.

Fix: Add a slash to the end of all the folder constants in the const files.

'congu' should be based on MU_ST, not MU_SF (that's for 'congl').

https://github.com/intvenlab/PostREISE/blob/93124516cf545d1a7c1d6cb3c83c83ec80e9b636/postreise/extract/extract_data.py#L80

https://github.com/intvenlab/PostREISE/blob/360d4d818fff6937bdb1c7370c4394c0a7cadd9c/postreise/plot/analyze_pg.py#L145

Following changes in the MPC files, there is no more GenMWMin and GenMWMax column in the plant data frame of the Grid object. The GenMWMin(Max) values for renewables, extracted from the aux files, are now accessible through Pmin(max). As a result, some refactoring is necessary.

In addition, the MockGrid object defined in PowerSimData will be used inplace of the one defined in PostREISE.

WECC Congestion Analysis to identify congested transmission lines for potential upgrades in scenarios for 2030.

• find internal participants to give feedback
• schedule meetings (there is uncertainty about how soon people will be free)
• create questions / UI testing process
• run feedback sessions based on scheduling
• consolidate feedback and make an action plan