error message of "TypeError: Object of type float32 is not JSON serializable" on RMtools_1D/do_RMsynth_1D.py about rm-tools HOT 5 CLOSED

For whatever reason, I can't reproduce the behaviour on my own machine (I can reproduce @hfhsieh's example, but not within RM-Tools). But since a solution has been suggested, I've implemented it into the JSON-saving code in rmsynth1d, rmclean1d, and qufit. Could someone please try the updated version (47e8915) and confirm that this fixes the problem?

Thanks,
Cameron

from rm-tools.

Hi Ruolan:

Can you send me the data you were running that generated this error, and the settings you used? I'd like to recreate it so that I can better understand where it came from (so as to avoid similar problems in the future).

Thanks,
Cameron

from rm-tools.

@Cameron-Van-Eck - this has bitten me in SPICE-RACS as well. Although, in my case the JSON came from my database format, rather than writing to disk. It's strange we haven't seen this issue earlier. If it's helpful, it can be nice to use the numpy primitives to capture all float (or whatever) types. Here's an example from the SPICE-RACS pipeline:

    # Ensure JSON serializable
    for k, v in mDict.items():
        if isinstance(v, np.float_):
            mDict[k] = float(v)
        elif isinstance(v, np.int_):
            mDict[k] = int(v)
        elif isinstance(v, np.ndarray):
            mDict[k] = v.tolist()
        elif isinstance(v, np.bool_):
            mDict[k] = bool(v)

    do_RMsynth_1D.saveOutput(mDict, aDict, prefix, rm_verbose)

P.S. This can also be handled with a custom JSON encoder - but I think the amount of work is pretty much equivalent

from rm-tools.

@Cameron-Van-Eck I think it is because the built-in json package supports the Python basic type only, not all the numpy datatype. The example below can reproduce the this issue

import json
import numpy as np

for dtype in [np.int8, np.int16, np.int32, np.int64, np.int_]:  # np.int_ == np.int64
#for dtype in [np.float16, np.float32, np.float64, np.float128, np.float_]:    # np.float_ == np.float64
    try:
        json.dump(dtype(np.pi), open("foo", "w"))  # numeric
#        json.dump({"key": dtype(np.pi)}, open("foo", "w"))  # dict

        print("{} passed".format(dtype))

    except TypeError:
        print("{} failed".format(dtype))
#        raise

For integer variables (and values in dict), only numpy.int64 passed for Python 2.7.16, and none for Python 3.10.8. For floating number, only numpy.float64 passed for both Python 2 and 3.

I am not sure why the recipe in the SPICE-RACS pipeline works since this issue occurs when not isinstance(v, np.int_) and not isinstance(v, np.float_)

from rm-tools.

Can you send me the data you were running that generated this error, and the settings you used? I'd like to recreate it so that I can better understand where it came from (so as to avoid similar problems in the future).

@Cameron-Van-Eck

It was just the sample data (Soucre1.dat created using 'mk_test_ascii_data.py') you provided. XD

Ruolan

from rm-tools.