rhayes777 / pyautoconf Goto Github PK
View Code? Open in Web Editor NEWConfiguration parsing for the PyAuto* family
License: MIT License
Configuration parsing for the PyAuto* family
License: MIT License
I have some config files which look like this:
[figure]
colors=['k', 'g']
Currently, I load these are strings and convert them to a list:
def remove_spaces_and_commas_from_colors(colors):
colors = [color.strip(",") for color in colors]
colors = [color.strip(" ") for color in colors]
return list(filter(None, colors))
Its not clear to me how exactly to build this into PyAutoConf, but obviously would be nicer than my currently code.
The following unit test in PyAutoArray does not pass, because Dictable does not support class variables:
The problem is that these variables are assigned the path {'type': 'abc.ABCMeta'}
.
Can this be updated so we can use Dictable
on such objects?
The PyAutoLens unit tests are failing with the following error:
json_file = path.join(
"{}".format(path.dirname(path.realpath(__file__))), "files", "tracer.json"
)
g0 = al.Galaxy(
redshift=0.5, mass_profile=al.mp.SphIsothermal(einstein_radius=1.0)
)
g1 = al.Galaxy(redshift=1.0)
tracer = al.Tracer.from_galaxies(galaxies=[g0, g1])
> tracer.output_to_json(file_path=json_file)
test_autolens/lens/test_ray_tracing.py:2505:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../PyAutoConf/autoconf/dictable.py:168: in output_to_json
json.dump(self.dict(), f, indent=4)
autolens/lens/ray_tracing.py:134: in dict
tracer_dict = super().dict()
../PyAutoConf/autoconf/dictable.py:82: in dict
return as_dict(self)
../PyAutoConf/autoconf/dictable.py:67: in as_dict
**{
../PyAutoConf/autoconf/dictable.py:68: in <dictcomp>
key: as_dict(value)
../PyAutoConf/autoconf/dictable.py:67: in as_dict
**{
../PyAutoConf/autoconf/dictable.py:68: in <dictcomp>
key: as_dict(value)
../PyAutoConf/autoconf/dictable.py:42: in as_dict
return nd_array_as_dict(obj)
../PyAutoConf/autoconf/dictable.py:17: in nd_array_as_dict
"array": obj.tolist(),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <Quantity 67.74 km / (Mpc s)>
def tolist(self):
> raise NotImplementedError("cannot make a list of Quantities. Get "
"list of values with q.value.tolist()")
E NotImplementedError: cannot make a list of Quantities. Get list of values with q.value.tolist()
/home/jammy/venvs/PyAuto/lib/python3.8/site-packages/astropy/units/quantity.py:1323: NotImplementedError
This is specifically the line where the Tracer
object is output to a .json file, and has arisen since
I have a config as follows:
[figure]
figsize=(7, 7)
I convert it from a string to tuple as follows:
if isinstance(self.kwargs["figsize"], str):
self.kwargs["figsize"] = tuple(
map(int, self.kwargs["figsize"][1:-1].split(","))
)
Lets build this into autconf.
I received the following feebdack on the to and from .json methods froim the EUclid / EDEN team:
you should not open the config files in append mode, this asks for writing permission, which the pipeline (fortunately) doesn't have.
In fact, just completely removing the access mode parameter will default to "r" (https://docs.python.org/3/library/functions.html#open), which will work fine.
Furthermore, in output_to_json(), you also should remove the "+" in with 'open(file_path, "w+") as f:', because as is, if the file already exists, the new json data will be appended at the end of the file and the file won't be readable anymore by standard json. So it's better to use "w" alone for the access mode, so that a possibly already existing file is overwritten and the created one is usable.
This relates to this code:
@classmethod
def from_json(cls, file_path: str) -> "Dictable":
"""
Load the dictable object to a .json file, whereby all attributes are converted from the .json file's dictionary
representation to create the instance of the object
A json file of the instance can be created from the .json file via the `output_to_json` method.
Parameters
----------
file_path
The path to the .json file that the dictionary representation of the object is loaded from.
"""
with open(file_path, "r+") as f:
cls_dict = json.load(f)
return cls.from_dict(cls_dict=cls_dict)
def output_to_json(self, file_path: str):
"""
Output the dictable object to a .json file, whereby all attributes are converted to a dictionary representation
first.
An instane of the object can be created from the .json file via the `from_json` method.
Parameters
----------
file_path
The path to the .json file that the dictionary representation of the object is written too.
"""
with open(file_path, "w+") as f:
json.dump(self.dict(), f, indent=4)
I think I wrote this code myself and these changes are fine (e.g. "r+" -> "r", "w+" -> "w"), but figured I'd check you agree first.
I am outputting a Clocker2D
object (from autocti) to a .json file as follows:
clocker = ac.Clocker2D(
iterations=5,
parallel_express=5,
serial_express=5
)
clocker.output_to_json(file_path="cti_clocker.json")
One of the internal variables is a ndarray, which is not output correctly:
{
"type": "VIS_CTI_Autocti.VIS_CTI_Clocker.two_d.Clocker2D",
"iterations": 5,
"parallel_roe": {
"type": "SHE_ArCTICPy.roe.ROE",
"dwell_times": {
"type": "numpy.ndarray"
},
"empty_traps_between_columns": true,
"empty_traps_for_first_transfers": false,
"force_release_away_from_readout": true,
"use_integer_express_matrix": false
},
"parallel_express": 5,
"parallel_offset": 0,
"parallel_window_start": 0,
"parallel_window_stop": -1,
"parallel_poisson_traps": false,
"parallel_fast_pixels": null,
"serial_roe": {
"type": "SHE_ArCTICPy.roe.ROE",
"dwell_times": {
"type": "numpy.ndarray"
},
"empty_traps_between_columns": true,
"empty_traps_for_first_transfers": false,
"force_release_away_from_readout": true,
"use_integer_express_matrix": false
},
"serial_express": 5,
"serial_offset": 0,
"serial_window_start": 0,
"serial_window_stop": -1,
"serial_fast_pixels": null,
"verbosity": 0,
"poisson_seed": -1,
"euclid_orientation_hack": false
}
It is specifically this part which is the issue:
"type": "SHE_ArCTICPy.roe.ROE",
"dwell_times": {
"type": "numpy.ndarray"
},
Could we add support for ndarrays to the dictable object?
Certain configs are losing capitlization, for example notation.yaml
goes to lower case but includes class names:
[superscript]
ModelComponent0=M0
ModelComponent1=M1
Gaussian=g
Exponential=e
superscript:
exponential: e
gaussian: g
modelcomponent0: M0
modelcomponent1: M1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.