Comments (53)
sr-murthy
commented on June 3, 2024
Obviously I can install shapely myself beforehand, but it would be good to automate this.
My point with the issue is that the lookup services for different models might have their own custom requirements, which are not part of the OasisLMF requirements. And when you run the generate-oasis-files subcommand for a particular model you are calling at some point its lookup service (to generate the keys file). And if a custom requirement is missing the command will fail.
from oasislmf.
OmegaDroid
commented on June 3, 2024
We can but personally I wouldn't. If people are creating their own model with requirements then they should be creating a virtual env for their model which includes oasislmf as a requirement along with any requirements the package requires.
from oasislmf.
sr-murthy
commented on June 3, 2024
OK I understand. I am getting another error
(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model generate-oasis-files -C ../OasisPiWind/mdk-oasislmf-piwind.json
Traceback (most recent call last):
File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/bin/oasislmf", line 4, in <module>
__import__('pkg_resources').run_script('oasislmf==0.0.0', 'oasislmf')
File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 750, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1527, in run_script
exec(code, namespace, namespace)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/EGG-INFO/scripts/oasislmf", line 7, in <module>
sys.exit(RootCmd().run())
File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/cmd/root.py", line 36, in run
return super(OasisBaseCommand, self).run(args=args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 151, in run
return cmd_cls(sub_command_name).run(args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 151, in run
return cmd_cls(sub_command_name).run(args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 153, in run
return self.action(args) or 0
File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/cmd/model.py", line 361, in action
logger=self.logger,
File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/exposures/manager.py", line 731, in start_files_pipeline
self.transform_source_to_canonical(**kwargs)
TypeError: transform_source_to_canonical() takes at least 2 arguments (1 given)
from oasislmf.
OmegaDroid
commented on June 3, 2024
that will be sorted once i push mu latest code implementing the python transform code (which i will do shortly)
from oasislmf.
sr-murthy
commented on June 3, 2024
OK, fine.
Also, shouldn't the README contain instructions on building an egg and installing it? I mean, there will be many end users who won't know how to do this.
from oasislmf.
OmegaDroid
commented on June 3, 2024
The main docs include instructions for installing from pypi or github, I guess these should be included in the readme too.
from oasislmf.
sr-murthy
commented on June 3, 2024
Once it's on PyPi they can of course do a pip install, and we can assume they will know how to do this. But even so it might be good to add a few lines in the README for those working directly with the repository.
from oasislmf.
OmegaDroid
commented on June 3, 2024
imo an end user should only ever install with pip (whether from pypi or direct from github if they are waiting for a change to be released).
I can see that a developer may want to install the development version into a model env so maybe adding a line about pip installing from the directory may be useful.
from oasislmf.
OmegaDroid
commented on June 3, 2024
FYI ive just pushed the changes to fix the transform issue
from oasislmf.
sr-murthy
commented on June 3, 2024
Thanks, I'll test it and let you know.
from oasislmf.
sr-murthy
commented on June 3, 2024
@OmegaDroid I tried to install the package from a new egg in a new virtual env. The egg was built fine but the install failed (easy_install dist/oasislmf-0.0.0-py2.7.egg).
...
...
Searching for requests-toolbelt
Reading https://pypi.python.org/simple/requests-toolbelt/
Downloading https://pypi.python.org/packages/97/8a/d710f792d6f6ecc089c5e55b66e66c3f2f35516a1ede5a8f54c13350ffb0/requests_toolbelt-0.8.0-py2.py3-none-any.whl#md5=aab9f93459410330d292753e398d51b3
Best match: requests-toolbelt 0.8.0
Processing requests_toolbelt-0.8.0-py2.py3-none-any.whl
Installing requests_toolbelt-0.8.0-py2.py3-none-any.whl to /Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages
writing requirements to /Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/requests_toolbelt-0.8.0-py2.7.egg/EGG-INFO/requires.txt
Adding requests-toolbelt 0.8.0 to easy-install.pth file
Installed /Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/requests_toolbelt-0.8.0-py2.7.egg
Searching for requests
Downloading https://pypi.python.org/packages/86/f9/e80fa23edca6c554f1994040064760c12b51daff54b55f9e379e899cd3d4/requests-toolbelt-0.8.0.tar.gz#md5=1028a6044b175440fdc7671e94430cbe
Best match: requests toolbelt-0.8.0
Processing requests-toolbelt-0.8.0.tar.gz
Writing /var/folders/vx/3ts13s190k50b2gzt_6zyp0m0000gn/T/easy_install-kqoD7a/requests-toolbelt-0.8.0/setup.cfg
Running requests-toolbelt-0.8.0/setup.py -q bdist_egg --dist-dir /var/folders/vx/3ts13s190k50b2gzt_6zyp0m0000gn/T/easy_install-kqoD7a/requests-toolbelt-0.8.0/egg-dist-tmp-jDfbYS
no previously-included directories found matching 'docs/_build'
warning: no previously-included files matching '*.py[cdo]' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
zip_safe flag not set; analyzing archive contents...
removing '/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/requests_toolbelt-0.8.0-py2.7.egg' (and everything under it)
Copying requests_toolbelt-0.8.0-py2.7.egg to /Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages
requests-toolbelt 0.8.0 is already the active version in easy-install.pth
Installed /Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/requests_toolbelt-0.8.0-py2.7.egg
error: The 'requests' distribution was not found and is required by oasislmf
It appears as if requests-toolbelt is installed first, which as you know requires requests, which is listed after requests-toolbelt in requirements.txt. So requests should be listed before requests-toolbelt.
from oasislmf.
sr-murthy
commented on June 3, 2024
This didn't happen the last time because I had already installed all the requirements prior to building and installing the egg.
from oasislmf.
sr-murthy
commented on June 3, 2024
I installed requests and also Shapely (required by PiWind) and re-installed the package. I am getting a new error when trying to run PiWind.
(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model generate-oasis-files -C ../OasisPiWind/mdk-oasislmf-piwind.json
Traceback (most recent call last):
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/bin/oasislmf", line 4, in <module>
__import__('pkg_resources').run_script('oasislmf==0.0.0', 'oasislmf')
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 750, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1527, in run_script
exec(code, namespace, namespace)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/EGG-INFO/scripts/oasislmf", line 7, in <module>
sys.exit(RootCmd().run())
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/cmd/root.py", line 36, in run
return super(OasisBaseCommand, self).run(args=args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree-0.0.5-py2.7.egg/argparsetree/cmd.py", line 159, in run
return cmd_cls(sub_command_name).run(args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree-0.0.5-py2.7.egg/argparsetree/cmd.py", line 159, in run
return cmd_cls(sub_command_name).run(args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree-0.0.5-py2.7.egg/argparsetree/cmd.py", line 161, in run
return self.action(args) or 0
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/cmd/model.py", line 361, in action
logger=self.logger,
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/exposures/manager.py", line 664, in start_files_pipeline
source_exposures_file_path=os.path.join(oasis_files_path, os.path.basename(source_exposures_path))
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/exposures/manager.py", line 405, in _process_default_kwargs
kwargs.setdefault('model_exposures_file_path', oasis_model.resources['oasis_files_pipeline'].model_exposures_path)
KeyError: 'oasis_files_pipeline'
from oasislmf.
OmegaDroid
commented on June 3, 2024
The install process uses requirements-package.in as its dependencies which lists requests before requests-toolbelt, ive also never experienced any issues using pip install . so it looks like there is an issue with using easy_install or eggs.
I can take a look at this if you like but I'm not sure how valuable it is as pip and is recommended over easy_install
from oasislmf.
OmegaDroid
commented on June 3, 2024
I have also fixed the key error if you want to give it another try
from oasislmf.
sr-murthy
commented on June 3, 2024
Understood. I use pip as well, but was just trying out easy_install. So the pip install oasislmf worked fine.
from oasislmf.
sr-murthy
commented on June 3, 2024
Looks like there is still a reference in there somewhere to verbose_logging, which one of your recent commits should have replaced with verbose.
(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model generate-oasis-files -C ../OasisPiWind/mdk-oasislmf-piwind.json
Traceback (most recent call last):
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/bin/oasislmf", line 7, in <module>
sys.exit(RootCmd().run())
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/oasislmf/cmd/root.py", line 38, in run
self.logger.error(str(e))
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/oasislmf/cmd/base.py", line 130, in logger
if self.args.verbose_logging:
AttributeError: 'Namespace' object has no attribute 'verbose_logging'
from oasislmf.
OmegaDroid
commented on June 3, 2024
If you are installing from the source or git repo you will need the -U flag on the pip install command to force it to update as the version number isn't changing
from oasislmf.
sr-murthy
commented on June 3, 2024
Yes, I did try that but it is still failing with the same error. BTW it appears that a tests package is also installed. I expected pip uninstall oasislmf would remove any related packages.
from oasislmf.
OmegaDroid
commented on June 3, 2024
When I install with pip the tests module is not installed (which it shouldn't). Did you try to install in the env where you had previously used easy install or from fresh?
from oasislmf.
OmegaDroid
commented on June 3, 2024
If I install and uninstall with pip nothing is left behind.
from oasislmf.
sr-murthy
commented on June 3, 2024
@OmegaDroid I deleted the virtual env. I created the first time I tried the install, and created a new virtual environment inside a new copy of the OasisLMF repo, then tried the pip install oasislmf. And it appears the tests package is being added to the virtual env (oasislmf-venv/lib/python2.7/site-packages/tests). The pip uninstall appears to remove all of the modules and subpackages in tests except for test_cmd.py[c], which are left behind.
That is a separate issue because I'm still getting the error I posted yesterday. This is after deleting the second virtual env., creating a new one in the latest copy of OasisLMF, and running pip install -U oasislmf.
(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model generate-oasis-files -C ../OasisPiWind/mdk-oasislmf-piwind.json
Traceback (most recent call last):
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/bin/oasislmf", line 7, in <module>
sys.exit(RootCmd().run())
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/oasislmf/cmd/root.py", line 38, in run
self.logger.error(str(e))
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/oasislmf/cmd/base.py", line 130, in logger
if self.args.verbose_logging:
AttributeError: 'Namespace' object has no attribute 'verbose_logging'
from oasislmf.
sr-murthy
commented on June 3, 2024
from oasislmf.
OmegaDroid
commented on June 3, 2024
That tests module is part of the argparsetree package not oasislmf so it wont be removed on removing oasislmf.
The command you are running to install is installing from the package on pypi which is not up to date it is the original one @mpinkerton-oasis created, which was before those changes. To install the latest before the deployment is in place you will need to run pip install -e git+ssh://[email protected]/OasisLMF/OasisLMF.git#egg=oasislmf.
from oasislmf.
sr-murthy
commented on June 3, 2024
OK, I thought it was updated. I installed it from the Git repo but on trying the PiWind model run I'm now getting the following error.
(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model generate-oasis-files -C ../OasisPiWind/mdk-oasislmf-piwind.json
No handlers could be found for logger "root"
Traceback (most recent call last):
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/bin/oasislmf", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/bin/oasislmf", line 7, in <module>
sys.exit(RootCmd().run())
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/cmd/root.py", line 36, in run
return super(OasisBaseCommand, self).run(args=args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 159, in run
return cmd_cls(sub_command_name).run(args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 159, in run
return cmd_cls(sub_command_name).run(args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 161, in run
return self.action(args) or 0
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/cmd/model.py", line 360, in action
logger=self.logger,
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/exposures/manager.py", line 678, in start_files_pipeline
self.get_keys(oasis_model=oasis_model, **kwargs)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/exposures/manager.py", line 395, in get_keys
output_file_path=keys_file_path,
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/keys/lookup.py", line 313, in save_keys
return cls.write_oasis_keys_file(keys, output_file_path)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/keys/lookup.py", line 190, in write_oasis_keys_file
data=[heading_row] + records,
TypeError: can only concatenate list (not "generator") to list
This is probably because get_keys (in oasislmf.keys.lookup.OasisKeysLookupFactory) returns a generator not a list. So in the call to get_keys in save_keys you are getting back a generator which you pass on to write_oasis_keys_file, in which you're appending that generator to a dataframe.
Do you want me to fix this?
BTW in fact, the logic in get_keys could be simplified further from line 267 onwards.
...
for record in lookup.process_locations(model_loc_df):
yield record
The reason for the current more complicated version of this logic is that at one point I wanted to cover the possibility that some lookup services will generate tuples of records rather than individual records. But I've refactored all the lookup services so that they do generate only individual records.
from oasislmf.
OmegaDroid
commented on June 3, 2024
feel free to fix it and if you see something that can be simplified further go for it
from oasislmf.
sr-murthy
commented on June 3, 2024
@OmegaDroid I'm still getting an error (seems the PiWind lookup instance is null)
(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model generate-oasis-files -C ../OasisPiWind/mdk-oasislmf-piwind.json
No handlers could be found for logger "root"
Traceback (most recent call last):
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/bin/oasislmf", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/bin/oasislmf", line 7, in <module>
sys.exit(RootCmd().run())
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/cmd/root.py", line 36, in run
return super(OasisBaseCommand, self).run(args=args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 159, in run
return cmd_cls(sub_command_name).run(args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 159, in run
return cmd_cls(sub_command_name).run(args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 161, in run
return self.action(args) or 0
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/cmd/model.py", line 360, in action
logger=self.logger,
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/exposures/manager.py", line 678, in start_files_pipeline
self.get_keys(oasis_model=oasis_model, **kwargs)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/exposures/manager.py", line 395, in get_keys
output_file_path=keys_file_path,
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/keys/lookup.py", line 307, in save_keys
return cls.write_oasis_keys_file(list(keys), output_file_path)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/keys/lookup.py", line 267, in get_keys
for record in lookup.process_locations(model_loc_df):
AttributeError: 'NoneType' object has no attribute 'process_locations'
I haven't read through all of your command class code, but I'm thinking it might not be parsing the lookup package correctly or loading the lookup package - in the call to get_keys in save_keys the lookup instance should exist.
I will try and fix it myself but I thought the tests would pick up these kinds of errors.
from oasislmf.
sr-murthy
commented on June 3, 2024
So this is from the package I installed from the branch fixes
pip install -e git+ssh://[email protected]/OasisLMF/OasisLMF.git@fixes#egg=oasislmf
from oasislmf.
OmegaDroid
commented on June 3, 2024
The tests do load custom lookups so I'm not sure whats happening here, I assume there is something with piwind that's not compatible
from oasislmf.
sr-murthy
commented on June 3, 2024
I am able to go to the PiWind repo and load the lookup class manually, which shows that there are no issues with the PiWind lookup itself.
(venv)[srm@~/Documents/sandeep/cst/dev/oasis/OasisPiWind/src]$ ipython
Python 2.7.10 (default, Jul 15 2017, 17:16:57)
Type "copyright", "credits" or "license" for more information.
IPython 5.4.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: from keys_server import PiWindKeysLookup
In [2]: kls = PiWindKeysLookup('../keys_data/PiWind/', 'OasisLMF', 'PiWind', '0.0.0.1')
In [3]: kls
Out[3]: <keys_server.PiWind.PiWindKeysLookup.PiWindKeysLookup at 0x11206b1d0>
And I can do the end-to-end model run for PiWind using run_model.py in the MDK repo.
from oasislmf.
sr-murthy
commented on June 3, 2024
Unless I'm mistaken I don't think the lookup class instance is being added to the model object resources dict in the action method in the GenerateOasisFilesCmd class.
...
self.logger.info('Getting model info and creating lookup service instance')
model_info, model_klc = OasisKeysLookupFactory.create(
model_keys_data_path=keys_data_path,
model_version_file_path=model_version_file_path,
lookup_package_path=lookup_package_file_path,
)
self.logger.info('\t{}, {}'.format(model_info, model_klc))
self.logger.info('Creating Oasis model object')
model = OasisExposuresManager().create(
model_supplier_id=model_info['supplier_id'],
model_id=model_info['model_id'],
model_version_id=model_info['model_version_id'],
resources={
'oasis_files_path': oasis_files_path,
'canonical_exposures_profile_json_path': canonical_exposures_profile_json_path,
'source_exposures_validation_file_path': source_exposures_validation_file_path,
'source_to_canonical_exposures_transformation_file_path': source_to_canonical_exposures_transformation_file_path,
'canonical_exposures_validation_file_path': canonical_exposures_validation_file_path,
'canonical_to_model_exposures_transformation_file_path': canonical_to_model_exposures_transformation_file_path,
}
)
The lookup class instance is created but not being set in the model object resources.
from oasislmf.
sr-murthy
commented on June 3, 2024
I accidentally deleted the success_only conditional logic in get_keys (in the lookup factory). That is actually required. So I've added this back and also the corresponding parser argument in the keys generation command class. Without this the keys generation will fail if the lookup class generates records for which the area peril and/or vulnerability lookup was unsuccessful.
from oasislmf.
sr-murthy
commented on June 3, 2024
Looks like I really messed that up as there are more references to success_only than I thought.
from oasislmf.
sr-murthy
commented on June 3, 2024
I'm getting a lookup related when trying the PiWind model run - this is the kind of error I'm familiar with from keys server testing, but in fact it appears related to success_only setting for generating lookup records (this doesn't with generate_keys.py in MDK, which works fine along with the other scripts).
(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model generate-oasis-files -C ../OasisPiWind/mdk-oasislmf-piwind.json
No handlers could be found for logger "root"
Traceback (most recent call last):
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/bin/oasislmf", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/bin/oasislmf", line 7, in <module>
sys.exit(RootCmd().run())
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/cmd/root.py", line 36, in run
return super(OasisBaseCommand, self).run(args=args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 159, in run
return cmd_cls(sub_command_name).run(args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 159, in run
return cmd_cls(sub_command_name).run(args)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 161, in run
return self.action(args) or 0
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/cmd/model.py", line 361, in action
logger=self.logger,
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/exposures/manager.py", line 678, in start_files_pipeline
self.get_keys(oasis_model=oasis_model, **kwargs)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/exposures/manager.py", line 395, in get_keys
output_file_path=keys_file_path,
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/keys/lookup.py", line 311, in save_keys
return cls.write_oasis_keys_file(list(keys), output_file_path)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/src/oasislmf/oasislmf/keys/lookup.py", line 267, in get_keys
for record in lookup.process_locations(model_loc_df):
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisPiWind/src/keys_server/PiWind/PiWindKeysLookup.py", line 73, in process_locations
record = self._get_location_record(loc_df.loc[i])
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisPiWind/src/keys_server/PiWind/PiWindKeysLookup.py", line 117, in _get_location_record
) for k in meta
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisPiWind/src/keys_server/PiWind/PiWindKeysLookup.py", line 117, in <genexpr>
) for k in meta
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/pandas/core/series.py", line 623, in __getitem__
result = self.index.get_value(self, key)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 2574, in get_value
raise e1
KeyError: 'id'
As I said this error does not happen when trigger the PiWind lookup on the shell, or via the MDK repo (generate_keys.py).
I don't know whether this is related but I'm getting a test failure from running the keys lookup tests
(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf/tests/keys]$ python -m unittest -v test_oasiskeylookupfactory
test_keys_path_is_not_supplied___correct_instance_is_created_with_correct_model_info (test_oasiskeylookupfactory.OasisKeysLookupFactoryCreate) ... ok
test_keys_path_is_supplied___correct_instance_is_created_with_correct_model_info_and_keys_path (test_oasiskeylookupfactory.OasisKeysLookupFactoryCreate) ... ok
test_entries_are_dictionaries___all_entries_are_included (test_oasiskeylookupfactory.OasisKeysLookupFactoryGetKeys) ... Falsifying example: test_entries_are_dictionaries___all_entries_are_included(self=<test_oasiskeylookupfactory.OasisKeysLookupFactoryGetKeys testMethod=test_entries_are_dictionaries___all_entries_are_included>, data=[{u'id': 0, u'status': u'failure'}])
You can reproduce this example by temporarily adding @reproduce_failure('3.45.2', 'AAEAAAABAA==') as a decorator on your test case
FAIL
test_model_exposures_are_provided___exposures_are_passed_to_get_model_exposures_result_is_passed_to_lookup_process_locations (test_oasiskeylookupfactory.OasisKeysLookupFactoryGetKeys) ... ok
test_model_exposures_path_is_provided___path_is_passed_to_get_model_exposures_result_is_passed_to_lokkup_process_locations (test_oasiskeylookupfactory.OasisKeysLookupFactoryGetKeys) ... ok
test_no_model_exposures_are_provided___oasis_exception_is_raised (test_oasiskeylookupfactory.OasisKeysLookupFactoryGetKeys) ... ok
test_exposures_string_is_provided___file_content_is_loaded (test_oasiskeylookupfactory.OasisKeysLookupFactoryGetModelExposures) ... ok
test_file_is_provided___file_content_is_loaded (test_oasiskeylookupfactory.OasisKeysLookupFactoryGetModelExposures) ... ok
test_no_file_or_exposures_are_provided___oasis_exception_is_raised (test_oasiskeylookupfactory.OasisKeysLookupFactoryGetModelExposures) ... ok
test_no_model_exposures_are_provided___oasis_exception_is_raised (test_oasiskeylookupfactory.OasisKeysLookupFactoryWriteKeys) ... ok
test_produced_keys_are_passed_to_write_oasis_keys_file (test_oasiskeylookupfactory.OasisKeysLookupFactoryWriteKeys) ... ok
test_records_are_given___records_are_written_to_file_correctly (test_oasiskeylookupfactory.OasisKeysLookupFactoryWriteListKeysFiles) ... ok
test_records_are_given___records_are_written_to_csv_correctly (test_oasiskeylookupfactory.OasisKeysLookupFactoryWriteOasisKeyFile) ... ok
======================================================================
FAIL: test_entries_are_dictionaries___all_entries_are_included (test_oasiskeylookupfactory.OasisKeysLookupFactoryGetKeys)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_oasiskeylookupfactory.py", line 201, in test_entries_are_dictionaries___all_entries_are_included
'id': integers(),
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/hypothesis/core.py", line 1001, in wrapped_test
state.run()
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/hypothesis/core.py", line 817, in run
falsifying_example.__expected_traceback,
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/hypothesis/core.py", line 578, in execute
result = self.test_runner(data, run)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/hypothesis/executors.py", line 58, in default_new_style_executor
return function(data)
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/hypothesis/core.py", line 570, in run
return test(*args, **kwargs)
File "test_oasiskeylookupfactory.py", line 201, in test_entries_are_dictionaries___all_entries_are_included
'id': integers(),
File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/lib/python2.7/site-packages/hypothesis/core.py", line 517, in test
result = self.test(*args, **kwargs)
File "test_oasiskeylookupfactory.py", line 210, in test_entries_are_dictionaries___all_entries_are_included
self.assertEqual(res, data)
AssertionError: Lists differ: [] != [{u'status': u'failure', u'id'...
Second list contains 1 additional elements.
First extra element 0:
{u'status': u'failure', u'id': 0}
- []
+ [{u'id': 0, u'status': u'failure'}]
----------------------------------------------------------------------
Ran 13 tests in 20.082s
FAILED (failures=1)
from oasislmf.
sr-murthy
commented on June 3, 2024
@OmegaDroid I'm still getting that id key error when trying the generate-oasis-files, which is mystifying because no such error occurs when doing the same thing via the generate_oasis_files.py script in the MDK repo.
Another thing I noticed in that in the keys generation command class there is no flag set for the output file, which is -o in MDK. Shouldn't this be present?
from oasislmf.
sr-murthy
commented on June 3, 2024
I think what's happening is that the PiWind model loc. df (model_loc_df) constructed in the lookup factory method get_keys doesn't have an ID column. There's no logging set, and it is hard to see what model loc. file is being loaded. The sample model loc. file in the PiWind repo definitely has the ID column
ID,LAT,LON,COVERAGE,CLASS_1,CLASS_2
10002082046,52.76698052,-0.895469856,1,R,RD
10002082047,52.76697956,-0.89536613,1,R,RD
...
from oasislmf.
OmegaDroid
commented on June 3, 2024
The output file is the first (and only) positional argument passed to the command.
I'll look into the id key issue.
from oasislmf.
sr-murthy
commented on June 3, 2024
OK, you may have to look at the PiWind lookup class as well to understand the context of this error. It uses a meta dictionary to make sense of the loc. data frame in order to load the frame rows as dicts (PiWindKeysLookup._get_location_record)
from oasislmf.
sr-murthy
commented on June 3, 2024
The PiWind lookup doesn't require any changes, it works perfectly fine.
from oasislmf.
OmegaDroid
commented on June 3, 2024
This comes from the python csv translation stuff, in there we add ROW_ID to each row rather than ID which piwind expects. Should this instead be ID?
from oasislmf.
sr-murthy
commented on June 3, 2024
I merged @sambles's PyTrans replacement PR into the MDK master and ran PiWind end-to-end several times. The intermediate files from the post-merge runs look fine - the model exposures file contains an ID column - this is dependent on the lookup service format, which is unique to the lookup service obviously. We don't introduce any new headers into the intermediate files.
from oasislmf.
sr-murthy
commented on June 3, 2024
The ROW_ID field is present in the canonical exp. file, but not in the model exp. file where it should get transformed to ID.
from oasislmf.
OmegaDroid
commented on June 3, 2024
right, it looks like I had the row id appending the wrong way round (it was being appended in the canonical to model rather than source to canonical).
The command now runs without error.
Im just running the tests and then will push.
from oasislmf.
sr-murthy
commented on June 3, 2024
OK thanks. I assume you will push to the fixes branch. I will test it at my end, and then the merge can happen.
from oasislmf.
OmegaDroid
commented on June 3, 2024
Yeah ive pushed to the fixed branch
from oasislmf.
sr-murthy
commented on June 3, 2024
OK, so I've just re-installed the oasislmf package from the fixes branch and then ran oasislmf model generate-oasis-files against PiWind but can't see the generated files and folder structure. Where do they live?
(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model generate-oasis-files -C ../OasisPiWind/mdk-oasislmf-piwind.json
No handlers could be found for logger "root"
from oasislmf.
OmegaDroid
commented on June 3, 2024
it is written to OasisFiles-<timestamp> in the current working directory
from oasislmf.
sr-murthy
commented on June 3, 2024
So far I've been running the CLI in the OasisLMF repo, which is adjacent to the PiWind repo, but I've just found that these OasisFiles folders are being created in the PiWind repository. Is that right?
from oasislmf.
sr-murthy
commented on June 3, 2024
Generated files all look OK for generate-oasis-files, but obviously I will test the other generate subcommands as well, before merging.
from oasislmf.
OmegaDroid
commented on June 3, 2024
ive just updated the path so that its relative to cwd rather than config file (unless specified in the config file)
from oasislmf.
sr-murthy
commented on June 3, 2024
Thanks.
from oasislmf.
sr-murthy
commented on June 3, 2024
I am running the full test suite now, but just FYI I'm getting some test dependency related errors (scandir). I've manually installed this package into the virtual env., but is there some requirements file I could change to pull in all tests dependencies?
Traceback:
tests/api_client/test_client.py:16: in <module>
from pathlib2 import Path
oasislmf-venv/lib/python2.7/site-packages/pathlib2.py:45: in <module>
from scandir import scandir as os_scandir
E ImportError: No module named scandir
from oasislmf.
sr-murthy
commented on June 3, 2024
OK, so I had uninstalled oasislmf before running the test suite that's why it threw that error - should be fine now. Tests all pass.
from oasislmf.
Related Issues (20)
- Add warning message when installing on unsupported platforms.
- Update the the `KeyLookupInterface` class to have access to the `lookup_complex_config_json`
- Pip install fails when running PiWind testing HOT 1
- Bug in latest platform2 release
- Add runtime user supplied secondary factor option to plapy
- Add option to enable or disable Post Loss Amplification
- model settings - correlation settings - allow optional hazard or damage correlation value
- Refactor footprint file format priorities
- Standardise docstrings
- PLT outputs are very slow to produce for equiprobable event sets HOT 1
- Partition parquet vunrability data by vunrability_id
- Update fm supported terms document HOT 2
- Incorect TIV in the summary info files
- Apply limits to aggregated exposures HOT 4
- Performace issue in get_exposure_summary
- Converting exposure files to previous OED version before running model
- Change vulnerability weight data type from 32-bit integer to 32-bit float in gulmc
- Flag required in oasislmf 1.28.X to turn off disaggregation. HOT 2
- Missing parquet library dependencies for gulmc in 1.27 HOT 1
- Net RI losses do not use -z in summarycalc HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from oasislmf.