Subdomain analysis is currently done by pre-processing URL-regex to simplify many regexs in the HTTPS Everywhere rules which are variations on matching any subdomain prefix (i.e. [^.]+\.), to allow the regex to be passed to sre_yield without resulting in near-infinite expansions. c.f. google/sre_yield#14

The list of existing prefix regexes can be found at https://github.com/jayvdb/https-everywhere-py/blob/873a5b7/https_everywhere/_fixme.py#L158 , and another non-prefix regexes at https://github.com/jayvdb/https-everywhere-py/blob/873a5b7/https_everywhere/_fixme.py#L202

A partial solution would be to split the regex into subpatterns representing dotted parts, where simple logic can then determine that the first part is a near-infinite expansion. There would be some of the regex which would be computationally difficult to split on the literal \., as it is embedded in multiple complex branches, but these dont need to be split afaics.

There is already rudimentary regex splitting at

I'm pretty sure I already fixed this upstream, but possibly hasnt been released yet.

Need to detect if it has been fixed or not, so that I can run the tests in both simplify and default mode.

Or add a simple hack/fallback for default mode so it doesnt break - might be a better approach as they are likely to let more bad data slip in.

self = <tests.test_rules.TestRules testMethod=test_package__<'CIBC'>>
name = 'CIBC'
    @foreach(_get_enabled_rulesets())
    def test_package(self, name):
        ruleset = rulesets[name]
>       self._check_ruleset(ruleset)
tests/test_rules.py:116: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_rules.py:109: in _check_ruleset
    self._check_https(rv)
tests/test_rules.py:89: in _check_https
    self.assertTrue(url.startswith("https://"))
E   AssertionError: False is not true
___________________ TestRules.test_package__<'Canada Post'> ____________________
self = <tests.test_rules.TestRules testMethod=test_package__<'Canada Post'>>
name = 'Canada Post'
    @foreach(_get_enabled_rulesets())
    def test_package(self, name):
        ruleset = rulesets[name]
>       self._check_ruleset(ruleset)
tests/test_rules.py:116: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_rules.py:109: in _check_ruleset
    self._check_https(rv)
tests/test_rules.py:89: in _check_https
    self.assertTrue(url.startswith("https://"))
E   AssertionError: False is not true
__________________ TestRules.test_package__<<'Pickaweb (...>> __________________
self = <tests.test_rules.TestRules testMethod=test_package__<<'Pickaweb (...>>>
name = 'Pickaweb (partial)'
    @foreach(_get_enabled_rulesets())
    def test_package(self, name):
        ruleset = rulesets[name]
>       self._check_ruleset(ruleset)
tests/test_rules.py:116: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_rules.py:103: in _check_ruleset
    rv = https_url_rewrite(test.url, rulesets=reduced_rulesets)
https_everywhere/_rules.py:856: in https_url_rewrite
    new_url = rule[0].sub(rule[1], url)
/usr/local/lib/python3.8/re.py:325: in _subx
    template = _compile_repl(template, pattern)
/usr/local/lib/python3.8/re.py:316: in _compile_repl
    return sre_parse.parse_template(repl, pattern)
/usr/local/lib/python3.8/sre_parse.py:1015: in parse_template
    addgroup(index, len(name) + 1)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
index = 2, pos = 2
    def addgroup(index, pos):
        if index > state.groups:
>           raise s.error("invalid group reference %d" % index, pos)
E           re.error: invalid group reference 2 at position 29
/usr/local/lib/python3.8/sre_parse.py:980: error
------------------------------ Captured log call -------------------------------
WARNING  https_everywhere._rules:_rules.py:858 failed during rule re.compile('^http://support\\.pickaweb\\.co\\.uk/(assets/)') -> https://app.sirportly.com/\g<2> , input http://pickaweb.co.uk/: invalid group reference 2 at position 29

Load data from https://hstspreload.org/api/v2/pending

Mentioned in _rules.py - need re-analysis

tox.ini should have a few combinations testing with older versions of dependencies.

One exception being requests/urllib3, which should have a minimum version that includes the fixes for the latest security vulnerabilities - no exceptions for running older versions of those.

"cached-property" could be made optional - it is only used by the simplification process, and only because of lazy programming.

"appdirs" likely can have quite a low minimum version

"logzero" also could be low, but should also be re-evalutated whether it is a good choice for library logging. (#24)

AppVeyor images have all versions if Python pre-installed, so tox could run them all.

Tagging and uploading of coverage to codecov is a bit more difficult in that scenario.

Useful in its own right, but also sort of an alternative to
#30

Currently only the includeSubdomain entries of the Chrome HSTS list are loaded. This most notably omits Google sites as they use a different entry structure.

It is helpful because this library is mainly useful for offline rewrites, i.e. bulk replacement of URIs in files and GitHub actions requiring contributors to use only https where it is possible

Python 2.7 is supported, so the newer Python 3 syntax can not be used, or if used, the installation / wheel processes need to strip them.

I've seen a few tools which appear to do the latter, and that would be preferred as the codebase should be "Python 3 first" with horrible hacks/degradation for Python 2.

"logzero" is currently badly configured by default, and emits "info" messages by default which isnt ideal for a library where most people will not care about the inner workings of the rule elimination/simplification process.

Try to control logzero, or switch to a different logging util.

CacheControl doesnt support HEAD method (psf/cachecontrol#216), so it should be detected and GET used instead

The reduction process is currently hiding under check=True - it should have a separate arg.

And the start and end should be logged, with some stats on how many rules were simplified.

Need to remove dependency on https://github.com/EFForg/https-everywhere/tree/master/test/rules/src/https_everywhere_checker , esp. as they want to replace Python with Node.

The local loading code can reside only in tests, but it would be better to support it in the library as XML is a better format for creating/editing custom rulesets.

At the moment the best docs are the tests for session and adapter.

fedmsg.com regularly fails on AppVeyor Windows

https://ci.appveyor.com/project/jayvdb/https-everywhere-py/builds/32107945/job/k6u89a5224mmm7n6?fullLog=true

The same commit on Cirrus has no problems

https://cirrus-ci.com/build/6684190053761024

"dbs.com" is one of the few literal domain name specific logic in _rules.py, and should be somehow made generic or deferred by moving the literal into _fixme.py

There are a bunch of these, which need to be converted to issues, prioritised, etc

Add a simple way to enforce HTTPS-only by blocking HTTP.

The Mozilla HSTS list is useful, if only for comparison, but it also includes negative entries which could add a lot of value.

It isnt clear how each adapter functions differently from the others.

TestCase inheritance was used for the everywhere adapters.

It should also be added to the non-everywhere adapters to reduce duplication of identical behaviour.

Many rulesets have a very limited list of hostnames which match, yet use a *.foo.com target. With the list of hostnames possible extracted from the regex, the wildcard target can be replaced with those hostnames, and often the rule can then also be replaced with a simplified rule.

I cant enable Travis CI for unknown reasons. Need to investigate.

Going to try Cirrus

https://cirrus-ci.org/guide/FreeBSD/

https://github.com/jayvdb/https-everywhere-py/blob/a3f2b42/https_everywhere/_rules.py#L406

There are lots of cases of regex which refer to hosts which are not in the rule targets.

They are currently detected, but are not being rejected, or considered in the tests.
The known ones are stored in _FIXME_BROKEN_REGEX_MATCHES:

_FIXME_BROKEN_REGEX_MATCHES = [
    "affili.de",
    "www.belgium.indymedia.org",
    "m.aljazeera.com",
    "atms00.alicdn.com",
    "i06.c.aliimg.com",
    "allianz-fuercybersicherheit.de",
    "statics0.beauteprivee.fr",
    "support.bulletproofexec.com",
    "wwwimage0.cbsstatic.com",
    "cdn0.colocationamerica.com",
    "www.login.dtcc.edu",
    "ejunkie.com",
    "e-rewards.com",
    "member.eurexchange.com",
    "4exhale.org",
    "na0.www.gartner.com",
    "blog.girlscouts.org",
    "lh0.google.*",  # fixme
    "nardikt.org",
    ".instellaplatform.com",
    "m.w.kuruc.org",
    "search.microsoft.com",
    "static.millenniumseating.com",
    "watchdog.mycomputer.com",
    "a0.ec-images.myspacecdn.com",
    "a0.mzstatic.com",
    "my.netline.com",
    "img.e-nls.com",
    "x.discover.oceaniacruises.com",
    "www.data.phishtank.com",
    "p00.qhimg.com",
    "webassetsk.scea.com",
    "s00.sinaimg.cn",
    "mosr.sk",
    "sofurryfiles.com",
    "asset-g.soupcdn.com",
    "cdn00.sure-assist.com",
    "www.svenskaspel.se",
    "mail.telecom.sk",
    "s4.thejournal.ie",
    "my.wpi.edu",
    "stec-t*.xhcdn.com",  # fixme
    "www.*.yandex.st",  # fixme
    "s4.jrnl.ie",
    "b2.raptrcdn.com",
    "admin.neobookings.com",
    "webmail.vipserv.org",
    "ak0.polyvoreimg.com",
    "cdn.fora.tv",
    "cdn.vbseo.com",
    "edge.alluremedia.com",
    "secure.trustedreviews.com",
    "icmail.net",
    "www.myftp.utechsoft.com",
    "research-store.com",
    "app.sirportly.com",
    "ec7.images-amazon.com",
    "help.npo.nl",
    "css.palcdn.com",
    "legacy.pgi.com",
    "my.btwifi.co.uk",
    "orders.gigenetcloud.com",
    "owa.space2u.com",
    "payment-solutions.entropay.com",
    "static.vce.com",
    "itpol.dk",
    "orionmagazine.com",
    # fix merged, not distributed
    "citymail.com",
    "mvg-mobile.de",
    "inchinashop.com",
    "www.whispergifts",
    # already merged?
    "css.bzimages.com",
    "cdn0.spiegel.de",
]

These are mostly fixed in EFForg/https-everywhere#18949 and EFForg/https-everywhere#18957 , but upstream has difficulty reviewing complex changesets - splitting them might help, but even so the progress on smaller PRs is very slow, so these problems will linger for a while, and need to be fixed in this library.

The regex hosts need to either be tested properly so that the extra hosts can be added to the targets and so be used in the processing logic, and optimised sanely, or the regex should be simplified to remove these extra hosts.

There are already some # pragma: no cover in the code, and a few more will be enough to reach 100% so it can be enforced

Once covered by CI, PyPy should be added to classifiers in setup.py

The HSTS preload list is fetched once and not updated.

Need to add http caching headers, etc

GitHub usually shows the selected license at the top of the repo - it is missing on this repo.

AppVeyor is slow to finish, so at least one Windows job should be run on Cirrus CI.

Then AppVeyor CI can be marked as optional, if that lets the commit go green quicker.

Python 2.7 should still be achievable

Need to use responses or similar to allow the tests to pass even when offline, and even if the problematic websites used as test caess are 'fixed'.

unittest-expander has some old code which will likely break on Python 3.8 zuo/unittest_expander#1

http://github.com/maaku/scenariotest/ might be a suitable replacement.

upstream rules checker also has some bad regex EFForg/https-everywhere#18985

Currently the unregex module needs sre_yield master, however it doesnt need to - the code to skip over leading negative assertions is already in unregex.

Also setup.py can have dependency links to force installation using zips, etc.

I dont want to force a release of sre_yield until it is ready, especially if there is some enhancements there in progress which helps this projects needs.

Currently the simplification results are only in memory, and need to be re-done each time. They should be stored on disk for quicker subsequent start times.

Maybe a sqlite db, or emit a json file with the same structure as the published json.

The current dict is very fast, but requires all hostnames are in memory. A btree of the reversed hostname would be better.

https://pypi.org/project/https-everywhere/ needs some descriptive text.