lanrat / czds Goto Github PK

The recent timeout code change which formed v1.0.1 appears to break the download of the largest zone file (.com) when running even on a reasonably spec'ed connection (AWS EC2 system with "Up to 10Gbps" bandwidth).

Since the update, I am now seeing:

2021/06/21 08:34:45 downloading 'https://czds-api.icann.org/czds/downloads/com.zone'
2021/06/21 08:34:45 downloading 'https://czds-api.icann.org/czds/downloads/photo.zone'
<SNIP>
2021/06/21 08:34:52 downloading 'https://czds-api.icann.org/czds/downloads/softbank.zone'
2021/06/21 08:34:52 downloading 'https://czds-api.icann.org/czds/downloads/kerryproperties.zone'
2021/06/21 08:39:45 [com.zone] err: context deadline exceeded (Client.Timeout or context cancellation while reading body)
2021/06/21 08:39:45 downloading 'https://czds-api.icann.org/czds/downloads/com.zone'

The file left on disk is considered corrupted by the zip tool, presumably because it is incomplete.

Ran with options czds-dl -token czdsToken -out ~/czds-zones I'm getting the following error fairly consistently/daily

could not download https://czds.icann.org/en/download-zone-data/1337?token=czdsToken : 403 Forbidden (403)

Visiting the url with valid token I get error The most recent request for this domain has not been approved.

Is there any way to convert download id (1337) into a human readable zone so I can re-request? I've tried multiple times to re-request access to any denied/pending zones with no luck.

Use latest git tag in Makefile.

I know this is a feature request and not a bug report and if I got the understanding of Go I might add a pull request. But for now I will create this as a tracking request if this might be an option to add.

For me, as a user, I would like to download multiple zones provided as a newline or comma separated list. This could be possible by the following options:

$ czds-dl -zone ./newline-separated-list

or maybe:

$ czds-dl -zone example1,example2,example3

What do you think as the author of this tool?

Hi Ian,

Thank's for providing and maintaining this CZDS tool.
Very helpful !

Testing the new version including the new extend "feature" I noticed something about the output of czds-report

$ ./czds-report -username "$CZDS_LOGIN" -password "$CZDS_PASS" -verbose -file report-a.csv
2022/07/13 17:06:18 Authenticating to https://account-api.icann.org/api/authenticate
2022/07/13 17:06:19 Saving to report-a.csv

I guess from the header, what should be the expected fields, with one single line per TLD.

$ head -1 report-a.csv
"Email","TLD name","Status","Last Updated","Reason","Expire Date"

But some lines into the csv report looks split on multiple lines

"[email protected]","associates","approved","2016-03-17T10:05:46Z","Your request has been approved.

Xxxxx Xxxxxx, CTO
Donuts Inc.","2027-02-28T10:05:46Z"

Checking into the web interface, it looks like the "issue" is when the "Reason" has been provided by the registry on multiple lines.

I guess a multiple lines "Reason" should be displayed on the same line into the csv output.

Hi,

Noticed that czds-status -zone may reply for another "close" zone.

Taking the red zone example in the readme, it replies the kred zone

$ czds-status -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -zone red
ID:	xxx
TLD:	kred (kred)
Status:	approved
....

Same for other like audi but reply the audible

$ czds-status -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -zone audi
ID:	xxx
TLD:	audible (audible)
Status:	approved
...

This is indeed how the "filter" looks working on the ICANN API (audible replied as first result for a audi search)

$ curl 'https://czds-api.icann.org/czds/requests/all' -X POST ....
--data-raw '{"status":"","filter":"audi","pagination":{"size":100,"page":0},"sort":{"field":"created","direction":"desc"}}'

{
  "requests": [
    {
      "requestId": "xxx",
      "tld": "audible",
      "ulable": "audible",
      "status": "Approved",
      "created": "xxx",
      "last_updated": "xxx",
      "expired": "xxx",
      "sftp": false
    },
    {
      "requestId": "xxx",
      "tld": "audi",
      "ulable": "audi",
      "status": "Approved",
      "created": "xxx",
      "last_updated": "xxx",
      "expired": "xxxx",
      "sftp": false
    },
    { ....

I'm not sure how to make the ICANN API reply only for the requested TLD, (may be there is a way? ) if not possible, may be the workaround would be to filter the json results where the tld match exactly what has been requested.

In June 2023, ICANN may change the download URL structure. For individual zones the API does not yet return this URL and uses the hard-coded URL pattern, however this may change.

https://github.com/lanrat/czds/blob/main/cmd/czds-dl/download.go#L122

From ICANN:

Additionally, users that rely on the CZDS REST API to download approved zone files should make sure to use the list of links provided by the CZDS API, as described in section 5.1 of the CZDS API specification, and be mindful that the format of the download URLs is subject to change in the near future to incorporate new query parameters. Users that may still be using a hardcoded form of the download URL as “https://czds-api.icann.org/czds/downloads/tld.zone” should update their implementation no later than June 2023.

Hi,

tested with 1.2.6
(btw it may be useful to have some -version option)

• neustar is expiring within 30 days and so appear as "Extensible"

./czds-status -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -zone neustar
ID:	xxxxx
TLD:	neustar (neustar)
Status:	approved
Created:	xxxx
Updated:	xxxxx
Expires:	Wed Sep 28 23:59:59 2022
AutoRenew:	false
Extensible:	true
ExtensionInProcess:	false
...

• Extending all

$ time ./czds-request -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -extend-all -verbose
2022/08/30 16:08:51 Authenticating to https://account-api.icann.org/api/authenticate
2022/08/30 16:08:52 Requesting extension for all TLDs

real	0m2,720s
user	0m0,081s
sys	0m0,037s

• neustar remains Extensible: true and ExtensionInProcess: falseafter previous command.

$ ./czds-status -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -zone neustar
ID:	xxxx
TLD:	neustar (neustar)
Status:	approved
Created:	xxxx
Updated:	xxxx
Expires:	Wed Sep 28 23:59:59 2022
AutoRenew:	false
Extensible:	true
ExtensionInProcess:	false
...

While I'm downloading zone files, the com.zone sometimes get downloaded partially with no errors.

I have added one more step in my script to validate if the file gets downloaded completely by issuing the following command.
$ gzip --test com.txt.gz
gzip: com.txt.gz: unexpected end of file

Even if I try to run the czds-dl for a second time, I got the following logs:

[com.zone] err: unexpected EOF
downloading 'https://czds-download-api.icann.org/czds/downloads/com.zone'

It would be very handy to check the content-length after each download and log/throw/retry and delete and clean up any corrupted files if any.

this will help the user to trust the downloaded zones.

Is it possible to add download and extract option?

Will there be support for the Access Extension option?

'Zone file access requesters already authorized to access a top-level domain (TLD) zone file in the CZDS system will now have the option to submit an access extension request up to 30 days prior to the expiration date of their currently approved access.

To request an access extension though the CZDS portal, go to the TLD zone access request details page. If the expiration date is within 30 days or less, the option to request an extension will be displayed next to the expiration date. Click the “Request Extension” button and confirm the action in order to submit the extension request to the registry operator.'