fn.lookup_table giving wrong result for gpu about dali HOT 5 CLOSED

Hi @SunXuan90,

Thank you for reporting this bug. There are two problems with your code. The first one is an actual bug in the lookup table operator (we will fix it soon), and the second one is an issue with your code snippet. Let me explain.

external_source source argument is meant for a callable or an iterable. In your case, you are feeding an iterable that it is interpreted as batches of data. The outermost dimension is interpreted as the number of batches (1 batch in your example) and the innermost dimension is interpreted as batch size (4 in your example). Since there are no more dimensions, each sample is assumed to be a scalar value (0-D). That is, you are processing a batch of 4 scalars. The bug in the GPU operator is when processing scalars (fix coming soon).

Now, I believe that your intention (correct me if I am wrong) was to apply a lookup table to a 2-dimensional matrix. Let me give you a couple of examples that do that:

First, using a 2D constant array, no need for external source:

        scale_mat = fn.lookup_table(
            np.array([[0, 1, 2, 3]]),
            keys=scale_keys,
            values=scale_values,
            device=device,
            dtype=types.INT64,
        )

Second, using a callable to generate batches of random data:

        def get_mat():
            out = [
                (np.random.randint(0, 4, size=(1, 4), dtype=np.int32))
                for _ in range(batch_size)
            ]
            return out
        ids = fn.external_source(source=get_mat, device=device)

        scale_mat = fn.lookup_table(
            ids,
            keys=scale_keys,
            values=scale_values,
            device=device,
            dtype=types.INT64,
        )

The good news is that processing 2D inputs should work, as the bug only applies to scalar inputs.

For the record, I'll let you know here when the fix is merged to the main branch.

from dali.

Hi @SunXuan90,

I can confirm there is a problem with the GPU variant of that operator. Let us investigate it and get back to you with more info soon.

from dali.

Thank you!

from dali.

The fix has been merged. It should be available in the next nightly build and 1.34 release.

from dali.

1.34 is out.

from dali.