pogzyb / asyncwhois Goto Github PK

https://rdap.org/ :-)

Latest release on PyPI is 0.4.1 but on GitHub it's 0.4.0.

Could you please that the source for 0.4.1?

Thanks

Parsing Whois data is hard, especially because the format differ depending on the TLD. I've a specific issue with registrant value. So I decided to test multiple python whois library (pythonwhoisalt, asyncwhois, whoisit, whoisdomain) against the registrant field using google's domain dataset and check for their speed too. Initially, I was using whoisdomain, so here the initial post on its github: mboot-github/WhoisDomain#21

Here the script I wrote: https://gist.github.com/baderdean/cc4643ecd95d3ccde31dee80ebdbea28

Asyncwhois was the best in term of quality, yet slower than whoisdomain by far. Is it something some one could reproduce to tell if that's related to my specific case or is it generic?

And here the results:

{'asyncwhois': {'count': 49,
                'duration': 285.84409061399947,
                'percentage': 26,},
 'whoisdomain': {'count': 44,
                 'duration': 195.54051797400007,
                 'percentage': 24},
 'whoisit': {'count': 6,
             'duration': 27.91238160300054,
             'percentage': 3,},
'pythonwhoisalt': {'count': 7,
                    'duration': 1055.0711162629996,
                    'percentage': '4%'}}

PS: I've created similar issues in other projects as well.

Hello

Thank you very much for your work, it saved me a lot of time.

This being said, I found out that asyncwhois.servers.IPv4Allocations._allocations data is incomplete, both according to my experience, and according the list of assignments from iana

Hello,

Many thanks for this interesting project, I encountered an infinite loop while using this package. Here is the scenario to reproduce this bug with the latest version of this package (1.0.10).

python3 -m venv venv
venv/bin/python -m pip install asyncwhois
venv/bin/python

from logging import basicConfig
from asyncwhois import rdap_domain
basicConfig(level='DEBUG')
result = rdap_domain('myspreadshop.net')
# infinite loop triggered, raising RecursionError at some point 

The rdap_domain method uses whodap's to_whois_dict method behind the scenes. This library returns slightly different keys for the whois dictionary that asyncwhois does not.

For example:

import asyncwhois

w = asyncwhois.whois_domain("google.com")
r = asyncwhois.rdap_domain("google.com")

w_keys = w.parser_output.keys()
r_keys = r.parser_output.keys()

# see mismatches
print([k for k in r_keys if k not in w_keys])
print([k for k in w_keys if k not in r_keys])

The task is to write a static function in the DomainLookup class:

@staticmethod
def _convert_whodap_keys(parser_output: dict) -> dict:
    # handle mismatched keys
    return parser_output

The above function should be used in the rdap_domain and aio_rdap_domain functions of DomainLookup

whois_dict = response.to_whois_dict()
_self._parser = _self._convert_whodap_keys(whois_dict)
return _self

Network request worked successfully, however response processing failed with the following error:

Traceback (most recent call last):

File "asd.py", line 6, in main
response = await asyncwhois.aio_rdap_domain('clientservices.scenber.com')

File "C:\Users\SAMPLE_USER\AppData\Local\Programs\Python\Python311\Lib\site-packages\asyncwhois_init_.py", line 93, in aio_rdap_domain
result = await DomainLookup.aio_rdap_domain(domain, httpx_client)

File "C:\Users\SAMPLE_USER\AppData\Local\Programs\Python\Python311\Lib\site-packages\asyncwhois\pywhois.py", line 156, in aio_rdap_domain
whois_dict = response.to_whois_dict()

File "C:\Users\SAMPLE_USER\AppData\Local\Programs\Python\Python311\Lib\site-packages\whodap\response.py", line 190, in to_whois_dict
flat.update(self._flat_entities(self.entities))

File "C:\Users\SAMPLE_USER\AppData\Local\Programs\Python\Python311\Lib\site-packages\whodap\response.py", line 238, in _flat_entities
vcard_type = vcard[0]

TypeError: 'DomainResponse' object is not subscriptable

Add a parser class to parse_tld.py for .za domains.

In asyncwhois.parse_tld.DomainParser.parse there is a check to see if the given query output contains "not found" language. Specifically, there is a for loop that looks for matches in DomainParser._no_match_checks (lines 100-101) and raises an exception.
This list does a pretty good job, but could be expanded.

TODO:

• Update the DomainParser._no_match_checks list
  • use https://github.com/cheenanet/whois-servers-list/blob/master/whois.json as inspiration
  • No need to copy all of these; a better strategy may be to use combinations of these keywords

I noticed that contact emails aren't being parsed by default and seems to be commented out in the code, any reasons why?

Should I fork and add email or was there a problem?

Hello,

I noticed that performing several RDAP searches results in downloading data.iana.org/rdap/dns.json each time a RDAP search is made.

In your opinion, can this file be cached for a configurable amount of time instead ? Or maybe I missed something that prevents implementing this behavior.

Thank you

Traceback (most recent call last):
File "C:/D/python/bulk_whois/bulk_whois.py", line 28, in
asyncio.run(main())
File "C:\Users\Drakaris\AppData\Local\Programs\Python\Python37-32\lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Users\Drakaris\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 584, in run_until_complete
return future.result()
File "C:/D/python/bulk_whois/bulk_whois.py", line 23, in main
await asyncio.gather(*tasks)
File "C:\Users\Drakaris\AppData\Local\Programs\Python\Python37-32\lib\site-packages\asyncwhois_init_.py", line 19, in lookup
query_result = await do_async_whois_query(domain)
File "C:\Users\Drakaris\AppData\Local\Programs\Python\Python37-32\lib\site-packages\asyncwhois\query.py", line 21, in do_async_whois_query
stderr=asyncio.subprocess.PIPE
File "C:\Users\Drakaris\AppData\Local\Programs\Python\Python37-32\lib\asyncio\subprocess.py", line 202, in create_subprocess_shell
stderr=stderr, **kwds)
File "C:\Users\Drakaris\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 1503, in subprocess_shell
protocol, cmd, True, stdin, stdout, stderr, bufsize, **kwargs)
File "C:\Users\Drakaris\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 463, in _make_subprocess_transport
raise NotImplementedError
NotImplementedError

I've noticed the RDAP functions have no timeout parameter, is there a specific reason for this ?

Sorry friends - I forgot to add the asyncwhois.servers folder to the latest build, so you might have seen this error.
This will be fixed in 1.1.2

Hey There.

I think we forgot to update the setup.py configuration to include the servers submodule in the distributed package.

diff --git a/setup.py b/setup.py
index ffc90ae..aa7f7a1 100644
--- a/setup.py
+++ b/setup.py
@@ -47,7 +47,7 @@ setuptools.setup(
         "Framework :: AsyncIO",
     ],
     url="https://github.com/pogzyb/asyncwhois",
-    packages=["asyncwhois"],
+    packages=setuptools.find_packages(),
     package_dir={"asyncwhois": "asyncwhois"},
     python_requires=">=3.9",
 )

find_packages should automatically do the trick. Sorry for the misshap.

Hi. Your library is great, but I ran into a problem.

import asyncwhois
asyncwhois.whois_domain("urbanvibes.com").parser_output
{domain_name: None, created: None, updated: None, expires: None, registrar: None, registrar_iana_id: None, registrant_name: None, registrant_organization: None, registrant_address: None, registrant_city: None, registrant_state: None, registrant_zipcode: None, registrant_country: None, registrant_email: None, dnssec: None, status: [], name_servers: [], admin_name: None, admin_id: None, admin_organization: None, admin_city: None, admin_address: None, admin_state: None, admin_zipcode: None, admin_country: None, admin_phone: None, admin_fax: None, admin_email: None, billing_name: None, billing_id: None, billing_organization: None, billing_city: None, billing_address: None, billing_state: None, billing_zipcode: None, billing_country: None, billing_phone: None, billing_fax: None, billing_email: None, tech_name: None, tech_id: None, tech_organization: None, tech_city: None, tech_address: None, tech_state: None, tech_zipcode: None, tech_country: None, tech_phone: None, tech_fax: None, tech_email: None}
asyncwhois.whois_domain("urbanvibes.com").query_output
' Domain Name: URBANVIBES.COM\r\n Registry Domain ID: 1569257607_DOMAIN_COM-VRSN\r\n Registrar WHOIS Server: whois.crazydomains.com\r\n Registrar URL: http://www.crazydomains.com.au\r\n Updated Date: 2020-01-13T07:38:10Z\r\n Creation Date: 2009-09-16T18:00:20Z\r\n Registry Expiry Date: 2025-09-16T18:00:20Z\r\n Registrar: Dreamscape Networks International Pte Ltd\r\n Registrar IANA ID: 1291\r\n Registrar Abuse Contact Email: [email protected]\r\n Registrar Abuse Contact Phone: +61 894 220 890\r\n Domain Status: ok https://icann.org/epp#ok\r\n Name Server: NS1.GKSMDNS.COM\r\n Name Server: NS2.GKSMDNS.COM\r\n DNSSEC: unsigned\r\n URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/\r\n>>> Last update of whois database: 2023-06-02T08:41:25Z <<<\r\n\r\nFor more information on Whois status codes, please visit https://icann.org/epp\r\n\r\nNOTICE: The expiration date displayed in this record is the date the\r\nregistrar's sponsorship of the domain name registration in the registry is\r\ncurrently set to expire. This date does not necessarily reflect the expiration\r\ndate of the domain name registrant's agreement with the sponsoring\r\nregistrar. Users may consult the sponsoring registrar's Whois database to\r\nview the registrar's reported date of expiration for this registration.\r\n\r\nTERMS OF USE: You are not authorized to access or query our Whois\r\ndatabase through the use of electronic processes that are high-volume and\r\nautomated except as reasonably necessary to register domain names or\r\nmodify existing registrations; the Data in VeriSign Global Registry\r\nServices' ("VeriSign") Whois database is provided by VeriSign for\r\ninformation purposes only, and to assist persons in obtaining information\r\nabout or related to a domain name registration record. VeriSign does not\r\nguarantee its accuracy. By submitting a Whois query, you agree to abide\r\nby the following terms of use: You agree that you may use this Data only\r\nfor lawful purposes and that under no circumstances will you use this Data\r\nto: (1) allow, enable, or otherwise support the transmission of mass\r\nunsolicited, commercial advertising or solicitations via e-mail, telephone,\r\nor facsimile; or (2) enable high volume, automated, electronic processes\r\nthat apply to VeriSign (or its computer systems). The compilation,\r\nrepackaging, dissemination or other use of this Data is expressly\r\nprohibited without the prior written consent of VeriSign. You agree not to\r\nuse electronic processes that are automated and high-volume to access or\r\nquery the Whois database except as reasonably necessary to register\r\ndomain names or modify existing registrations. VeriSign reserves the right\r\nto restrict your access to the Whois database in its sole discretion to ensure\r\noperational stability. VeriSign may restrict or terminate your access to the\r\nWhois database for failure to abide by these terms of use. VeriSign\r\nreserves the right to modify these terms at any time.\r\n\r\nThe Registry database contains ONLY .COM, .NET, .EDU domains and\r\nRegistrars.\r\n'
import platform
platform.platform()
'Linux-4.18.0-425.19.2.el8_7.x86_64-x86_64-with-glibc2.31'
import sys
sys.version
'3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]'

But the problem does not reproduce on windows.

import asyncwhois
asyncwhois.whois_domain("urbanvibes.com").parser_output
{domain_name: 'URBANVIBES.COM', created: datetime.datetime(2009, 9, 16, 0, 0), updated: datetime.datetime(2020, 1, 13, 15, 38, 10), expires: datetime.datetime(2025, 9, 16, 0, 0), registrar: 'Dreamscape Networks International Pte Ltd', registrar_iana_id: '1291', registrant_name: 'SPORT & FASHION MANAGEMENT PTE. LTD.', registrant_organization: 'SPORT & FASHION MANAGEMENT PTE. LTD.', registrant_address: '6 SHENTON WAY #18-11 OUE DOWNTOWN 2', registrant_city: 'SINGAPORE', registrant_state: 'SINGAPORE', registrant_zipcode: '068809', registrant_country: 'SG', registrant_email: '[email protected]', dnssec: 'unsigned', status: ['ok https://icann.org/epp#ok'], name_servers: ['NS1.GKSMDNS.COM', 'NS2.GKSMDNS.COM'], admin_name: 'SPORT & FASHION MANAGEMENT PTE. LTD.', admin_id: 'C-008487039-SN', admin_organization: 'SPORT & FASHION MANAGEMENT PTE. LTD.', admin_city: 'SINGAPORE', admin_address: '6 SHENTON WAY, #18-11, OUE DOWNTOWN 2', admin_state: 'SINGAPORE', admin_zipcode: '068809', admin_country: 'SG', admin_phone: '+65.66104279', admin_fax: '+65.', admin_email: '[email protected]', billing_name: None, billing_id: None, billing_organization: None, billing_city: None, billing_address: None, billing_state: None, billing_zipcode: None, billing_country: None, billing_phone: None, billing_fax: None, billing_email: None, tech_name: 'SPORT & FASHION MANAGEMENT PTE. LTD.', tech_id: 'C-008487039-SN', tech_organization: 'SPORT & FASHION MANAGEMENT PTE. LTD.', tech_city: 'SINGAPORE', tech_address: '6 SHENTON WAY, #18-11, OUE DOWNTOWN 2', tech_state: 'SINGAPORE', tech_zipcode: '068809', tech_country: 'SG', tech_phone: '+65.66104279', tech_fax: '+65.', tech_email: '[email protected]'}
import sys
sys.version
'3.11.3 (tags/v3.11.3:f3909b8, Apr 4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)]'
import platform
platform.platform()
'Windows-10-10.0.19044-SP0'

Could you please tag the releases?

This would allow to use the source tarball from GitHub to build packages for distributions with running the tests during the build process. Thanks.

Add black code-style linter that runs on each PR

• run the black formatter on the code base
• create linter.yml under .github/workflows

While using the library to perform mass WHOIS lookups, I found that the aio_whois_domain() function is not respecting the timeout parameter and running the query infinitely.

Example query performed:

loop = asyncio.get_event_loop()
result = loop.run_until_complete(asyncwhois.aio_whois_domain("1stchoicebankca.com"))

As per the source code, default timeout is set to 10 seconds, but surely its taking more than that.

Hi,
I can not successfully got demain information as demoed in README.
I use version 1.0.10 with python3.8.When I check in code I found aio_whois_domian is the method to use with async .But this function return DomainLookup instance.And I cannot find how to got domain expiretime and other information ,can any one help me on this issue?

If you hit a particular WHOIS server too frequently in a certain amount of time, the server may return a "rate limit" response.

In asyncwhois.parse_tld.DomainParser.parse there is a check to see if the given query output contains "not found" language. Specifically, there is a for loop that looks for matches in DomainParser._no_match_checks (lines 100-101) and raises an exception.

The same can be replicated for rate limit responses query output.

TODO:

• Create a new error class (something like "RateLimitError") in errors.py
• Create a list of "rate limit" string checks (use https://github.com/cheenanet/whois-servers-list/blob/master/limit-reached-messages.json for inspiration)
• Add logic (similar to the "not found" checks) for rate limit checks in the DomainParser.parse function to raise the exception
• [Optional] write a test

Implement Github actions to run tox on PR's and merge to main. Remove current reference to travis-ci

On whois of any registered in the past .tk domain, entire thread gets stuck on regex.

For example, let's use asdasdasdasd.tk domain, which is has been registered sometime ago, but now is free.

Whois service return the text data:

   Domain name:
      ASDASDASDASD.TK

   Organisation:
      Freedom Registry, Inc.
      2225 East Bayshore Road #290
      Palo Alto CA 94303
      United States
      Phone: +1 650-681-4172
      Fax: +1 650-681-4173

   Domain Nameservers:
      NS01.FREENOM.COM
      NS02.FREENOM.COM
      NS03.FREENOM.COM
      NS04.FREENOM.COM

   Your selected domain name is a domain name that has been
   cancelled, suspended, refused or reserved at the Dot TK Registry

   It may be available for re-registration at http://www.dot.tk

   In the interim, the rights for this domain have been automatically
   transferred to Freedom Registry, Inc.

   Please be advised that the Dot TK Registry, Freenom and
   Freedom Registry, Inc. cannot be held responsible for any content
   that was previously available at this domain name.

   Due to restrictions in Dot TK 's Privacy Statement personal information
   about the previous registrants of the domain name cannot be released
   to the general public.

   Dot TK is proud to work with numerous governmental law enforcement
   agencies to stop spam, fraud, phishing attempts, child pornography and
   other illicit content on Dot TK websites. These agencies may contact the
   Dot TK Registry directly with any enquiries they may have regarding the
   usage of this domain by previous registrants.

   Record maintained by: Dot TK Domain Registry

So, the problem is in parse_tld.py#L1169, because it search for Owner contact, that doesn't exist.

imho, this regex enough dangerous, but as hotfix, i assume, would be enough to add const available for re-registration at to parse_tld.DomainParser._no_match_checks

Hi, I've got a strange error when try to get whois of .org domains:
result = asyncwhois.whois_domain('iana.org')
Traceback (most recent call last):
File "", line 1, in
...
socket.gaierror: [Errno 11001] getaddrinfo failed

result = asyncwhois.whois_domain('domain.org')
Traceback (most recent call last):
File "", line 1, in
...
socket.gaierror: [Errno 11001] getaddrinfo failed

result = asyncwhois.whois_domain('nlmk.org')
Traceback (most recent call last):
File "", line 1, in
...
socket.gaierror: [Errno 11001] getaddrinfo failed

asynctest is no longer maintained and doesn't work with Python 3.11

Martiusweb/asynctest#163