usrflo / registered-domain-libs Goto Github PK

Detect the registered domain for a given domain name in C, Perl and PHP, based on Mozillas effective TLD listing

Perl 34.59% C 18.20% Shell 0.71% PHP 46.29% Makefile 0.21%

registered-domain-libs's Introduction

Detection of registered domain names by reg-dom libs

The reg-dom libs are available in PHP, Perl and C.

They include recent representations of the effective TLD list available at https://publicsuffix.org/list/effective_tld_names.dat and help to convert an arbitrary domain name to the registered domain name. Upon changes of effective_tld_names.dat this git repository is automatically updated once a day. The library supports Unicode (IDN notation) for domain name input and output. ACE notation is not supported currently.

Sample Use

Signing domains in DKIM signatures can be used on the level of registered domains to rate senders who use e.g. a.spamdomain.tld, b.spamdomain.tld, ... under the most common identifier - the registered domain - finally. Project page: https://www.agitos.de/registered-domain-libs/ Updated library downloads (besides Github): https://services.agitos.de/regdom-lib-downloads/

Pseudo code

registeredDomain = getRegisteredDomain(ingoingDomain);

Return values

NULL if ingoingDomain is a TLD
the registered domain name if TLD is known
just . if is unknown This case was added to support new TLDs in outdated reg-dom libs by a certain likelihood. This fallback method is implemented in the last conversion step and can be simply commented out.

If you like to regenerate the effective TLD tree structure by yourself you can use the script generateEffectiveTLDs.php with the following parameters:

php generateEffectiveTLDs.php php  > PHP/effectiveTLDs.inc.php
php generateEffectiveTLDs.php perl > Perl/effectiveTLDs.pm
php generateEffectiveTLDs.php c    > C/tld-canon.h

License

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to you under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at:
# 
#     http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# </@LICENSE>

Florian Sager, 2009-02-05, [email protected], http://www.agitos.de

registered-domain-libs's People

Contributors

Stargazers

Watchers

registered-domain-libs's Issues

effectiveTLDs.inc.php get empty in commit d76e2a3

Just wanted to tell that TLD Array in registered-domain-libs/PHP/effectiveTLDs.inc.php get empty in this commit d76e2a3

Might be by mistake.

PHP version doesn't work, or am I missing anything?

require_once("registered-domain-libs/effectiveTLDs.inc.php");
require_once("registered-domain-libs/regDomain.inc.php");

getRegisteredDomain('subdomain.domain.co.uk');
// this print "co.uk"
getRegisteredDomain('subdomain.domain.co.uk', $tldTree);
// this print "co.uk"

Mislabeling of eTLD-`n`

As mentioned in publicsuffix/list#1872 (comment)

Multiple psl libraries share the same bug while handling wildcards.

I went over all the FOSS implementations in https://publicsuffix.org/learn/ , and I found 3 of them behave funny for wildcard domains. In particular:

regdom-libs

publicsuffix2

publicsuffixlist.js

Seems to all mislabel eTLD-n x [eTLD minus n] (with n from 0 to infinite), as an eTLD, if another eTLD y is defined as a subdomain of that x eTLD-n.

As an example publicsuffix2 behaviour with the following patch:
 // Linode : https://linode.com
 // Submitted by <[email protected]>
 members.linode.com
-nodebalancer.linode.com
+*.nodebalancer.linode.com
+*.linodeobjects.com
+ip.linodeusercontent.com

Example here:

IDN Support suggestion

It seems any IDN can't get past the regexp in regDomain.class.php:

if (!preg_match("/^([a-z0-9])(([a-z0-9-])*([a-z0-9]))*$/", $domPart)) return FALSE;

If anybody's interested, the library could be augmented to support UTF8 letter characters by adding the \p{L} character set along with a-z0-9 and adding /u at the end:

if (!preg_match("/^([a-z0-9\p{L}])(([a-z0-9-\p{L}])*([a-z0-9\p{L}]))*$/u", $domPart)) return FALSE;

I tested it as such and it worked:

echo (new regDomain())->getRegisteredDomain("example.мон", false);

And it works for that IDN (.xn--l1acc):

example.мон

I could submit a pull request if nobody sees an issue

Problem with amazonaws.com

Hi,
is that normal that none of amazonaws.com hosts are found ?
ex: kinesis.us-east-1.amazonaws.com
each time it gives NULL

bye Fred

Corrections erroru the document regDomain.inc.php

In PHP 5.4 gives errors in the document regDomain.inc.php, as line 42
$ signingDomainParts = split .....;
is obsolete.
I propose to change this string instrument regDomain.inc.php (PHP version) to:
The highlighted line in the picture-

Line 42.