woodgern / confusables Goto Github PK

A python package providing functionality for matching words using different characters but appearing to be a similar/the same word.

License: MIT License

Python 96.81% Makefile 3.19%

confusables's Introduction

Confusables

Confusables is a python package that provides functionality for analyzing and matching words that "appear" to be the same or similar, but use different characters.

Confusables uses the unicode confusable characters list (https://www.unicode.org/Public/security/8.0.0/confusables.txt) along with other methods of matching characters.

This package can be used for any application where detecting words using any unexpected characters to pass filters is required. This could include finding malicious fake website names, analyzing or normalizing text data, or even detecting attempts to get past a profanity filter.

Installation

Confusables can be installed through pip using pip3 install confusables

Usage

The functions in the confusables module focus around comparing and finding strings that can be considered "confusable". This means that they can be humanly interpretable as the same string. Since this deals with human interpretation, the "confusable" definition is loose, and in later versions may be more or less strict.

Currently, confusables provides: is_confusable, confusable_characters, and confusable_regex

is_confusable(string1, string2) takes in 2 strings and outputs whether or not the two are "confusable". Keep in mind that in some cases, a single character can be confusable with 2 characters combined (eg. ‼ is a single character, !! is two)

from confusables import is_confusable

print(is_confusable('rover', 'Ʀỏ𝕍3ℛ'))
# prints True

confusable_characters(char) takes in a character and outputs a list of characters that are confusable with it. In some cases, as mentioned above, a single characters can be confusables with multiple characters, in which case those characters will be inluded in the list in the form of a string.

from confusables import confusable_characters

print(confusable_characters('c'))
# prints ['ċ', 'ᴄ', '𝔠', '𝒄', '𝗰', '𝗖', 'ḉ', 'ℂ', 'Ꮯ', 'ć', 'c̦', '𝑐', '𝓬', '𝚌', '𐌂', 'Ⅽ', 'С', '𝘤', 'ｃ', 'ҫ', '𝖈', '🝌', '𝖢', '𝐂', 'C', '𝓒', 'Ç', '𝘾', 'ç', 'Ⲥ', 'с', 'ⅽ', 'ĉ', '𐔜', 'c', 'ℭ', 'ϲ', '𑣩', 'Ϲ', '𝕮', 'č', '𐊢', 'Ĉ', '𝑪', 'Ｃ', '𑣲', '𐐕', '𐐽', 'ⲥ', '𝐶', 'Ċ', 'C̦', 'ꮯ', '𝒞', '𝕔', '𝘊', 'Č', 'ꓚ', '𝒸', '𝐜', '𝙲', '𝖼', 'Ć', '𝙘', 'Ḉ']

confusable_regex(string, include_character_padding=False) takes a string and outputs a regex string that matches words that are confusable with the input string.

from confusables import confusable_regex
import re

regex_string = confusable_regex('bore', include_character_padding=True)
regex = re.compile(regex_string)

print(regex.search('Sometimes people say that life can be a ь.𝞂.ř.ɜ, but I don\'t agree'))
# prints <_sre.SRE_Match object; span=(40, 47), match='ь.𝞂.ř.ɜ'>

normalize(string, prioritize_alpha=False) takes a string and outputs a list of possible "normal forms". This means that characters in the string get converted to their confusable ascii counterparts. The prioritize_alpha option means the outputted options will prioritize converting characters to characters of the latin alphabet over any others. This option is recommended when natural language is expected.

from confusables import normalize

print(normalize('Ʀỏ𝕍3ℛ', prioritize_alpha=True))
# prints ['rov3r', 'rover']

print(normalize('Ʀỏ𝕍3ℛ', prioritize_alpha=False))
# prints ['r0v3r', 'r0ver', 'ro\'v3r', 'ro\'ver', 'rov3r', 'rover']

Updating to the latest Unicode confusables version

If you find the latest version of this package to have an out of date version of the unicode official confusables.txt, then why not submit a PR to update it!

First, find out what the latest version of unicode confusables is. Then, run

make update VERSION=X.Y.Z

Next, run

make parse

And that's it! Commit your changes and create a pull request.

About confusables

This module is something I put together because I'm interested in the field of language processing. I'm hoping to build out it's functionality, and I'm more than happy to take suggestions!

Additionally, I think the effectiveness of the module could be greatly improved using some machine learning models, and I'm currently on the hunt for some useful data sets. Please let me know if you know of any!

You can contact me through any normal Github means, or using my email, [email protected]

confusables's People

Contributors

Stargazers

Watchers

Forkers

matthew-robertson alwc muhmacit stoufa thibaultamartin doggo4242 thiojoe prestonhale spryfox cl0udz ferstar gshmu freso txtsd

confusables's Issues

Assumtion that uppercase and lowercase versions of characters are confusable

In parse.py when creating the confusable sets, there is an assumption that if a certain character is confusable with another, it will also be confuseable with the upper/lower case version. This isn't always the case.

For example this causes "latin small letter v" (v) to be confusable with "latin capital letter n" (N), as "greek small letter nu" is confusable with "latin small letter v" (03BD) and calling string.upper() on charater 03BD produces "latin capital letter n".

Assets not included in PyPI package

Hi,

I'm playing around with flake8 plugins, including flake8-confusables. But installing that (and as a dependency, this package) raises a FileNotFoundError: [Errno 2] No such file or directory: '/home/yggdrasil/.virtualenvs/svhclient/lib/python3.6/site-packages/confusables/assets/confusable_mapping.json'. Manually copying the assets directory there resolves this.
Might I suggest adding a MANIFEST.in :)

Thanks

Also care about invisble characters

There are a few characters that get display as an empty string like those for example:

\u200b
\u200c
\u200d
\u200e
\u200f

They can be mixed into any string and thus bypass the confusable detection since they are not visible the strings look the same

>>> print("f\u200boo")
foo

Regex meaningful characters not escaped in confusable_regex

When trying to create a confusable regex from a string like "t[e]st", it will fail to match the string "t[e]st". I believe this is because the characters are not escaped when added to the regex.

Confusables Coding Questions

Hello woodgern and thanks for the amazing library. I'm working on porting this code to Golang so that others can benefit from your work, and I had a few questions about the code.

In parse.py on line 50, you check to see if str1 only contains 1 character. However, this appears unnecessary, because all of the characters in the left-most column of confusing.txt contain only 1 character (and 1 code point). Is there some other reason that you have included this?
In parse.py on line 59, you check to see if str2 only contains 1 character. Why exactly is this? For example, consider the following line of confusables.txt:

0191 ;	0046 0326 ;	MA	# ( Ƒ → F̦ ) LATIN CAPITAL LETTER F WITH HOOK → LATIN CAPITAL LETTER F, COMBINING COMMA BELOW	# →F̡→

This seems like it should be included in a list of "look-alike" characters (since it case-inverts into "f̦"), but it is now skipped because it has a length of two. Is this intentional?

On lines 40-48, you add both sides to each other's map entry. However, you don't do the same thing on lines 50-66. In other words, shouldn't

unicode_confusable_map[str1].add(case_change)

be instead:

unicode_confusable_map[str1].add(case_change)
unicode_confusable_map[str2].add(case_change)

confusables.txt is outdated

The latest one at the moment is version 13.0.0. Could be updated also in README.md.

The normalize function doesn't return 'm' for 'rn', 'r' followed by 'n'

When using the normalize function for 'rn', 'r' followed by 'n', it doesn't include 'm' in the returned list despite the fact that applying confusable_characters to 'm' includes 'rn' in the list, presumably normalize is only applying to single characters e.g:

>>> normalize('rn')
['rn']  # should return ['m', 'rn']
>>> confusable_characters('m')
['𝑚', 'μ', 'ᗰ', 'Ḿ', '𝘔', 'Ⲙ', '𝓂', 'ℳ', '𝑴', '𝓜', 'м', '𝚳', '𝚖', '𝙼', '𝜧', 'Ꮇ', '𝑀', '𝘮', '𝕞', '𝓶', '𑜀', 'ｍ', '𝗺', '𝞛', '𝗆', 'ᛖ', '𝛭', 'Ⅿ', 'M', 'rn', 'µ', '𝙢', 'ṃ', '𝐦', 'ꭑ', 'Ṃ', '𝕸', 'ⲙ', '𝒎', 'ḿ', 'ϻ', 'ꓟ', 'Ｍ', 'm', 'ꮇ', 'ⅿ', '𐊰', '𝔐', 'Ϻ', 'Μ', '𝝡', 'Ṁ', '𝖬', '𝐌', '𐌑', '𝔪', '𝙈', '𝕄', 'ṁ', '𑣣', 'М', '𝖒', '𝗠']

Specially crafted string on normalize function returns an abnormally long list

If you run normalize on this string:

abing🪀|C-01 |🍏inv100+ Lv16推赞1300

then it will return an EXTREMELY long (3981312 entries) list.

It must be getting caught on something, because a 3+ million list attempting to normalize a string is absurd.

To reproduce:

import confusables

foo = "abing🪀|C-01 |🍏inv100+ Lv16推赞1300"

x = confusables.normalize(foo)

print(len(x))
print(x)

Skeleton function

Hi!
There used to be confusables for python that I forked and no longer exists. It had one very useful function skeleton(). It returned string that transformed all confusables to normalized values. So you could detect that original strings don't match but normalized strings match. Like A in latin and А in azbuka.

https://github.com/don-mums/confusables/blob/master/confusables.py

Is this something you would like to include into your project? The only thing I found that remotely resembles skeleton() is _asciify() but this also removes all non ascii characters, which is not very useful for us. As you can see on the example below.

'Veľký Krtíš'.encode('ascii', 'ignore').decode('ascii') == 'Vek Krt'