[Feature] Extract Non-ASCII Characters about stringi HOT 3 CLOSED

I am afraid this is too specific to be included in stringi.

Perhaps an easier solution: ?

x <- "test 123 ↓ęœß→óęœ©œ©ęπœęπœ©œπą"
x <- unique(unlist(stringi::stri_enc_toutf32(x)))
x <- x[x>127]
stringi::stri_enc_fromutf32(x)
## [1] "↓ęœß→ó©πą"

from stringi.

1. Function stri_enc_toutf32 does indeed the conversion directly. Unfortunately, I am not an expert in the package stringi.

   • it would break if an UTF-64 was introduced: but then again this should be an internal implementation detail inside another function and would be therefore invisible to the end user;

2. Documentation/Examples with stringi::stri_escape_unicode

• for the last solution: I would include an extra example mentioning how to convert back to Unicode code points or how to escape the characters with stri_escape_unicode;
• alternatively, the function could have an option to either return the code-points or the characters;

3. Utility
   I have frequently encountered this situation: both when trying to extract information from articles on Pubmed, as well as from various reports (e.g. Lab-reports). There are probably sufficiently large communities involved in both types of operations, but having only rudimentary understanding of string encodings. If the (manual) cleaning is too time-consuming, then the most common options are:

• to exclude those inputs, or to delete any non-ASCII characters (if the user understands a little bit more about ASCII);
• both approaches are quite sub-optimal: better tools can come in handy for many simple users;

from stringi.

So, you probably mean:

x <- "test 123 ↓ęœß→óęœ©œ©ęπœęπœ©œπą"
x <- unique(unlist(stringi::stri_enc_toutf32(x)))
x <- as.list(x[x>127])
stringi::stri_escape_unicode(stringi::stri_enc_fromutf32(x))
## [1] "\\u2193" "\\u0119" "\\u0153" "\\u00df" "\\u2192" "\\u00f3" "\\u00a9" "\\u03c0" "\\u0105"

from stringi.