SQL Lab hard fails with "'utf-8' codec can't decode byte 0xe6 in position 0: invalid continuation byte" when reading certain binary types about superset HOT 4 OPEN

To address the UnicodeDecodeError you're encountering when querying binary, non-UTF-8 encodable data from a VARBINARY column in Trino using SQL Lab in Superset, consider implementing a custom JSON serializer that can handle binary data by encoding it in a format like base64. This will allow binary data to be serialized into a string format that JSON can handle without errors.

Here are the steps you can take:

1. Implement a Custom Serializer: Create a custom JSON serializer for handling binary data, ensuring it's encoded in a format such as base64 before serialization.

2. Modify JSON Serialization Calls: Update the json.dumps calls in sqllab/api.py to utilize this custom serializer, particularly where JSON serialization of execution results and data serialization occurs.

3. Adjust Data Decompression Logic: In utils/core.py, ensure the decompression logic can handle binary data correctly, which may involve type checks before decompression and appropriate handling for binary data.

4. Thorough Testing: Test the querying of binary data from VARBINARY columns extensively to confirm the absence of the UnicodeDecodeError.

5. Documentation: Update project documentation to reflect the handling of binary data in SQL Lab queries.

This approach should help you resolve the encoding issues you're facing with binary data in Superset's SQL Lab.

To continue the conversation, mention @dosu.

from superset.

I don't think so, but it's possible. I was trying to allow UTF-8 strings through without them being escaped. It's possible this has to do with something being escaped incorrectly, which may or may not be the same chunk o' code.

from superset.

+1
We're running into this as well. It also affects the Table viz (and possibly others), not just SQL Lab.
@rusackas Could this have anything to do with the stringify stuff you were trying to remove?

from superset.

it seems that simplejson will handle bytes itself, and will not use logic in 'default' function:

import simplejson


def convert_bytes(val):
    if isinstance(val, bytes):
        return 'converted-' + repr(val)
    return val


print(simplejson.__version__)
# 3.17.3

print(simplejson.dumps({'a': b'b'}, default=convert_bytes))
# {"a": "b"}

print(simplejson.dumps({'a': b'\x00\x85'}, default=convert_bytes))
# raises UnicodeDecodeError

so bytes should be handled before they goes into the dumps function.

from superset.