[BUG] StringMethods - Jaccard-index fails with long strings about cudf HOT 2 CLOSED

The jaccard API uses hash_character_ngrams internally which produces a list column of integer values. The total number of integers in that list column is the number of ngrams for this strings column. The number of integers exceeds the max size_type and so the function is unable to build the output list column.

So you would need to limit the strings column size so the total number of generated ngrams would not exceed max size_type/int32 individual strings.
Meanwhile I can work on a modifying jaccard to avoid this limit since it is an internal detail for that API.

from cudf.

Even with large-strings support the amount of memory needed to process this example will be significant.
The original df size is 11 rows of 214,748,364 bytes each = ~2.4GB for the total input strings size.
Using a width=5 means each row generates 214,748,368 individual substrings at 5 bytes each = ~1.1GB per row. (11 rows ~ 12GB). The internal code uses hashing which reduces the 5 bytes to 4 bytes = ~859MB per row. (11 rows ~ 9.5GB).
Since the jaccard call here in this example is comparing the df with itself the temporary memory doubles to ~19GB.
Internally the intermediate substrings/hashes are sorted to help with counting the unique values. The sorted output requires a 2nd temporary copy (of the 9.5GB) which gets us to (19+9.5) = 28.5GB peak memory.

So overall jaccard_index would need about 6x the input memory available for processing.

from cudf.