nrel / lore Goto Github PK

License: Other

C++ 75.22% Python 10.63% C 13.57% CMake 0.08% M4 0.05% Batchfile 0.01% HTML 0.13% CSS 0.15% JavaScript 0.04% Shell 0.08% PowerShell 0.03%

lore's Introduction

Live Operations Recommendations Engine (LORE)

This project develops a user interface for the design, analysis, and optimization of concentrating solar power (CSP) systems, including optimization of both physical design variables and operations and maintenance (O&M) activity. The software is derived and actively developed under funding from the U.S. Department of Energy -- Energy Efficiency and Renewable Energy grant DE-EE00034245 ("Real-time operations optimization software") and was previously funded under grant DE-EE000030338 ("Concurrent optimization of capital cost and expected O&M").

This work is the result of a collaboration between researchers and industry, including:

National Renewable Energy Laboratory
Colorado School of Mines
Northwestern University
This repository, LORE, provides the user interface to assign values to inputs of the computational modules, run the modules in the correct order, and display calculation results. It also includes tools for editing LK scripts, viewing field layout and receiver flux map data, and performing multi-dimensional system optimization.

Quick Steps for Building LORE

These are the general quick steps you need to follow to set up your computer for developing LORE:

Set up your development tools:
- Windows: Visual Studio Code here.
- Linux: g++ compiler available here or as part of the Linux distribution.

Windows distributions require the following DLL's: dbghelp, libcurl, libeay32, libssh2, msvcp120, msvcr120, ssleay32, vcruntime140 These can be located from system path directories or downloaded from NREL's FTP server.

Running LORE

For additional setup to run LORE and view the dashboard, see the loredash folder within this repository.

Repository Access

Access is currently limited only to project partners. To obtain access, please send a request to Mike.

Contributing

License

LORE open source code is copyrighted by the U.S. Department of Energy -- Energy Efficiency and Renewable Energy under a mixed MIT and GPLv3 license. It allows for-profit and not-for-profit organizations to develop and redistribute software based on LORE under terms of an MIT license and requires that research entities including national laboratories, colleges and universities, and non-profit organizations make the source code of any redistribution publicly available under terms of a GPLv3 license.

Project organization

The code in this project is organized as follows:

Folder	Description
./libclearsky	Source code for clear sky irradiance model
./libcluster	Source code for the data clustering model
./libcycle	Source code for the cycle availability and failure model
./liboptical	Source code for the optical degradation and soiling model
./liboptimize	Source code and libraries for the nonlinear design optimization problem
./librtdispatch	Source code for the real-time dispatch optimization model
./libsolar	Source code for the solar field failure model
./loredash	Source code for the LORE mediator

Additional instructions on installing and running LORE, as well as viewing the dashboard, are available in the README within the ./loredash folder.

External libraries

LORE utilizes the following open-source libraries:

Project	Version	Usage
RapidJSON	v1.1.0	JSON read-write library for project files

SSL Requests

LORE utilizes SSL requests to obtain weather data that serve as inputs to the dispatch optimzation model. These requests can cause an SSL error when LORE is run thorugh a virtual network connection, and firewall settings may need to be adjusted to allow an exception for requests to the THREDDS server.

lore's People

Contributors

Stargazers

Watchers

lore's Issues

Forecaster should return rounded DNI

The forecasts aren't accurate, so we can simplify things by rounding the decimal portions of the DNI forecasts:

lore/loredash/mediation/mediator.py

Line 315 in 1cb107b

 # data = data.round({'Clear Sky DNI': 0}) # round away decimal points TODO: move this operation to forecasts 

update time in plots to local plant time

New forecasts bug

I assume this came from a recent commit of yours. Can you please address it? Thanks.

Bokeh: zoom in on a single axis

Remake ssc Mac library

@odow Can you remake the ssc Mac library from the daotk-develop branch? There was a somewhat recent bug fix for the tower model that may affect Lore. I can compile the Windows and Linux libraries.

Remove database methods from forecasts file

Can you move all database methods out of forecasts.py ? Move the data 'add' methods to mediator.py and the 'get' methods to solar_plot/main.py This will give control of the forecasts database population to mediator and remove dependence on if the forecasts webpage is requested or not. (It also more aligns with the overall architecture design.)

With this, I think the getForecast() call should be pulled out of the get_weather_df() call so the returned forecasts are immediately accessible in mediator.py . All database adds can then be at the same level in the code and non-forecast weather data calls are more encapsulated.

Let's discuss if needed to get on the same page.

Fix timezone warning

WARNING: Timesteps in run_once were not properly localized.

SSLError in forecasts.py

Line 116 in forecasts.py throws the following fatal exception when Internet connection is through NREL VPN. May be an issue depending on plant's Internet connection.

Exception has occurred: SSLError
HTTPSConnectionPool(host='thredds.ucar.edu', port=443): Max retries exceeded with url: /thredds/catalog.xml (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1124)')))

Only query OWM if needed

I just blew through the rate limit on OpenWeatherMap because of the learning models I'm running.

The issue is that the database is updated every call, which also generates a call to OWM:

lore/loredash/mediation/forecasts.py

Lines 245 to 255 in 997cc42

 new_data.rename( 

 columns = {k: str(k) for k in forecast_columns}, 

 inplace=True, 

 ) 

 for k in forecast_columns: 

 new_data[str(k)] = new_data[str(k)] * new_data['clear_sky'] 

 owm = self.openweathermap.get() 

 if len(owm) > 0: 

 new_data = new_data.join(owm, rsuffix = '_OWM') 

 # Fill in any missing pressure readings with the default. 

 new_data['pressure'].fillna(self.ambient_pressure(), inplace=True)

It's probably best to move this elsewhere, so it's only hit if we also query the other forecast server (i.e. in real time).

Timezone issues with TMY file

The TMY file has some weird data. Here's the row of June 14, 2013. Which shows a TMY of 500 W/m^2 DNI at 05:00.

However sunrise was at

How are the TMY hours constructed?

Move more submodel parameters to JSONs

separate JSON for each submodel
leave initial states in code for now; these will be queried from plant DB

Tests are failing

Investigate Docker vulnerabilities

Add a formatter for python

x-ref #19 (comment)

reference case study with >0 output

get_weather_df should return forecast containing start-time

Janna says

I’m trying to get the dispatch optimization running from weather forecasts in LORE. I have a couple of quick questions when > you have a minute:
In Mediator.get_weather_df() the first point in the weather dataframe (“data” below) appears to occur after the ‘datetime_start’ supplied to the function. Is that just a timezone or daylight savings time issue? Can I safely assume that the first point in the weather dataframe will be at the beginning of the hour containing datetime_start?

On the issue that you (or possibly Matt) noted in get_weather_df() about the NDFD forecasts sometimes returning values starting after the requested time point: Is that something that will always be an issue, or will it be “fixed” eventually? Using the TMY3 data for the early part of the horizon leads to undefined values for clear-sky DNI. So just wondering if I need to plan on populating those elsewhere.

I think both these points are related.

Installing on clean linux box

Trying to install this on a clean linux box.

So far, I

installed Anaconda

curl https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh -o anaconda
bash anaconda
# follow prompts

Followed the instructions (https://github.com/NREL/lore/tree/develop/pysam) to create a conda environment
Got stuck installing nrel-pysam-dao-tk

(pysam_daotk) odow@nova:~/lore$ pip install nrel-pysam-dao-tk
ERROR: Could not find a version that satisfies the requirement nrel-pysam-dao-tk (from versions: none)
ERROR: No matching distribution found for nrel-pysam-dao-tk

I then went and installed PySAM (https://nrel-pysam.readthedocs.io/en/master/GettingStarted.html), and can confirm that it installs and I can call things.

>>> from PySAM.PySSC import PySSC
>>> x = PySSC()
>>> x.version()
240

Here's my conda:

(pysam_daotk) odow@nova:~/lore$ conda list
# packages in environment at /home/odow/anaconda3/envs/pysam_daotk:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
ca-certificates           2020.6.24                     0  
certifi                   2020.6.20                py37_0  
ld_impl_linux-64          2.33.1               h53a641e_7  
libedit                   3.1.20191231         h14c3975_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
ncurses                   6.2                  he6710b0_1  
nrel-pysam                2.1.4                    pypi_0    pypi
nrel-pysam-dao-tk-stubs   1.0.0                    pypi_0    pypi
nrel-pysam-stubs          2.1.4                    pypi_0    pypi
openssl                   1.1.1g               h7b6447c_0  
pip                       20.1.1                   py37_1  
python                    3.7.7                hcff3b4d_5  
readline                  8.0                  h7b6447c_0  
setuptools                49.2.0                   py37_0  
sqlite                    3.32.3               h62c20be_0  
tk                        8.6.10               hbc83047_0  
wheel                     0.34.2                   py37_0  
xz                        5.2.5                h7b6447c_0  
zlib                      1.2.11               h7b6447c_3

What obvious step did I miss?

Model horizons including both December and January

ssc calls won't work across horizons that span December - January

Tests are failing on `develop`

(loredash) oscar@Oscars-MacBook-Pro loredash % python -m pytest
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
django: settings: loredash.settings (from ini)
rootdir: /Users/oscar/Documents/Projects/lore2/lore/loredash, configfile: pytest.ini
plugins: django-4.3.0
collected 14 items                                                                                                                                                                                         

tests/test_forecasts.py F....                                                                                                                                                                        [ 35%]
tests/test_mediator.py .F.                                                                                                                                                                           [ 57%]
tests/tests.py ......                                                                                                                                                                                [100%]

================================================================================================= FAILURES =================================================================================================
________________________________________________________________________________________ test_forecaster_from_plant ________________________________________________________________________________________

    @pytest.mark.django_db
    def test_forecaster_from_plant():
        forecaster = forecasts.SolarForecast(
>           plant.plant_design['latitude'],
            plant.plant_design['longitude'],
            plant.plant_design['timezone_string'],
            plant.plant_design['elevation'],
        )
E       AttributeError: module 'mediation.plant' has no attribute 'plant_design'

tests/test_forecasts.py:32: AttributeError
___________________________________________________________________________________________ test_get_weather_df ____________________________________________________________________________________________

    def test_get_weather_df():
        m = mediator.Mediator(
            params = mediator.mediator_params,
            plant_config_path = PARENT_DIR + "/data/plant_config.json",
>           plant_design = plant.plant_design,
            weather_file = PARENT_DIR + "/data/daggett_ca_34.865371_-116.783023_psmv3_60_tmy.csv",
            update_interval = datetime.timedelta(seconds = 5),
        )
E       AttributeError: module 'mediation.plant' has no attribute 'plant_design'

tests/test_mediator.py:46: AttributeError
============================================================================================= warnings summary =============================================================================================
../../../../../opt/anaconda3/envs/loredash/lib/python3.8/site-packages/pyutilib/misc/import_file.py:11
  /Users/oscar/opt/anaconda3/envs/loredash/lib/python3.8/site-packages/pyutilib/misc/import_file.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================================= short test summary info ==========================================================================================
FAILED tests/test_forecasts.py::test_forecaster_from_plant - AttributeError: module 'mediation.plant' has no attribute 'plant_design'
FAILED tests/test_mediator.py::test_get_weather_df - AttributeError: module 'mediation.plant' has no attribute 'plant_design'
============================================================================ 2 failed, 12 passed, 1 warning in 63.41s (0:01:03) ============================================================================

Rebuild ssc.dylib

Janna made some changes that need to be updated.

And the ssc.so for Linux

No file `prod.env`

There is a dev.env. Do I need to set an environment variable or something?

(loredash) oscar@Oscars-MacBook-Pro loredash % python manage.py migrate
Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/Users/oscar/opt/anaconda3/envs/loredash/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/Users/oscar/opt/anaconda3/envs/loredash/lib/python3.8/site-packages/django/core/management/__init__.py", line 325, in execute
    settings.INSTALLED_APPS
  File "/Users/oscar/opt/anaconda3/envs/loredash/lib/python3.8/site-packages/django/conf/__init__.py", line 79, in __getattr__
    self._setup(name)
  File "/Users/oscar/opt/anaconda3/envs/loredash/lib/python3.8/site-packages/django/conf/__init__.py", line 66, in _setup
    self._wrapped = Settings(settings_module)
  File "/Users/oscar/opt/anaconda3/envs/loredash/lib/python3.8/site-packages/django/conf/__init__.py", line 157, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/Users/oscar/opt/anaconda3/envs/loredash/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/oscar/Documents/Projects/lore2/lore/loredash/loredash/settings.py", line 29, in <module>
    env_config = Config(RepositoryEnv(os.path.join(BASE_DIR, DOTENV_FILE)))
  File "/Users/oscar/opt/anaconda3/envs/loredash/lib/python3.8/site-packages/decouple.py", line 123, in __init__
    with open(source, encoding=encoding) as file_:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/oscar/Documents/Projects/lore2/lore/loredash/prod.env'

Nonlinear Dispatch

Revisit timezone convention

Now that we're using timezone-aware datetimes, we need to enable time zone support in Django to direct it on how to manage these datetimes, most notably with the database. Since we're using a database that doesn't support timezone-aware datetimes (SQLite), it's strongly recommended here that we store datetimes in UTC.

Can we use UTC times most everywhere in the code?
Where do we use/consider local time, besides displaying in the plots?

I think storing datetimes in the db as UTC is less error prone, although it would be nice not to have to pass a timezone to the Bokeh server somehow and just display the datetimes as-is (in their plant-local timezones). However, I assume we'll want to display the datetimes in daylight saving time, so I guess we'll have to do this conversion regardless. So, I lean toward using UTC everywhere we can, including in the db. Thoughts?

Forecast updates

Correct Ambient Temperature -> Temperature #47
Ambient pressure #48
Relative humidity (or dew point)
Change to FixedOffset #46
Check if forecasts ever go close to clear sky #46
See if we can add weather underground widget

Installing loredash

@Matthew-Boyd a few things

I think I need permission to access matthewtboyd/lore:

(basic_3.8) oscar@Oscars-MBP lore-docker % docker pull matthewtboyd/lore:latest
Error response from daemon: pull access denied for matthewtboyd/lore, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

My docker hub account is odow: https://hub.docker.com/u/odow

I have a linux VM on my MacBook, but I get errors trying to create the database:

(loredash) parallels@parallels-Parallels-Virtual-Platform:~/Documents/lore/loredash$ python manage.py migrate
Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/home/parallels/Documents/lore/loredash/mediator/apps.py", line 13, in ready
    mediator = mediator.Mediator()
  File "/home/parallels/Documents/lore/loredash/mediator/mediator.py", line 22, in __init__
    maps = self.pysam_wrap.ReadMaps()

(loredash) parallels@parallels-Parallels-Virtual-Platform:~/Documents/lore/loredash$ python ./data/import_data.py ./data
Using data file path: /home/parallels/Documents/lore/loredash/data
? Select which SQLite3 database to import to:  (Use arrow keys)                                                                                                                                            
Traceback (most recent call last):
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/cache.py", line 34, in get
    return self._data[key]
KeyError: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./data/import_data.py", line 51, in <module>
    answers = prompt(questions)
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/PyInquirer/prompt.py", line 69, in prompt
    answer = run_application(
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/shortcuts.py", line 625, in run_application
    result = cli.run()
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/interface.py", line 413, in run
    self._redraw()
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/interface.py", line 358, in _redraw
    self.renderer.render(self, self.layout, is_done=self.is_done)
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/renderer.py", line 424, in render
    layout.write_to_screen(cli, screen, mouse_handlers, WritePosition(
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 142, in write_to_screen
    sizes = self._divide_heigths(cli, write_position)
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 177, in _divide_heigths
    dimensions = [get_dimension_for_child(c, index) for index, c in enumerate(self.children)]
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 177, in <listcomp>
    dimensions = [get_dimension_for_child(c, index) for index, c in enumerate(self.children)]
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 175, in get_dimension_for_child
    return c.preferred_height(cli, write_position.width, write_position.extended_height)
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1652, in preferred_height
    return self.content.preferred_height(cli, width, max_available_height)
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 999, in preferred_height
    preferred=self.content.preferred_height(
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 254, in preferred_height
    content = self.create_content(cli, width, None)
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 259, in create_content
    tokens_with_mouse_handlers = self._get_tokens_cached(cli)
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 238, in _get_tokens_cached
    return self._token_cache.get(
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/cache.py", line 37, in get
    value = getter_func()
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 239, in <lambda>
    cli.render_counter, lambda: self.get_tokens(cli))
  File "/home/parallels/anaconda3/envs/loredash/lib/python3.8/site-packages/PyInquirer/prompts/list.py", line 103, in _get_choice_tokens
    tokens.pop()  # Remove last newline.
IndexError: pop from empty list

Any ideas?

	new_data.rename(
	columns = {k: str(k) for k in forecast_columns},
	inplace=True,
	)
	for k in forecast_columns:
	new_data[str(k)] = new_data[str(k)] * new_data['clear_sky']
	owm = self.openweathermap.get()
	if len(owm) > 0:
	new_data = new_data.join(owm, rsuffix = '_OWM')
	# Fill in any missing pressure readings with the default.
	new_data['pressure'].fillna(self.ambient_pressure(), inplace=True)