Enhancement: Check download URL of Appliances about gns3-registry HOT 6 CLOSED

I already catch some errors with it :P

from gns3-registry.

We probably need to speed up stuff in the futur

from gns3-registry.

Reason for this slowness is that the http HEAD request is only done on the original web address. If this returns a redirection, the python urllib does a GET on the new address. That means, that your check does a full download on all images, that are hosted on sourceforge and other sites, that are using redirection.

So either we should tell this script to do no redirection or it should do HEAD requests also on redirected addresses.

from gns3-registry.

Well, I was wrong that the images are downloaded, when a redirection occurs. Nevertheless a GET is issued, what increases the traffic. But that's not the reason for the slowness.

It seems, that some web sites, especially when using https, are not really fast. If redirection takes place, this increases even more. On my system it takes 36 sec for the checks, what's quite a lot.

I searched with google to a way to remove the redirection. I found something, no idea if it's good code. Additionally I check each web site only once. This brings the time for check.py down to 10 sec. When we want to be even faster, we need parallel requests.

He my quick-and-dirty changes

--- check.py.orig   2015-10-13 20:21:51.000000000 +0200
+++ check.py    2015-10-14 00:22:28.000000000 +0200
@@ -22,14 +22,28 @@
 import subprocess
 import urllib.request

+class MyHTTPRedirectHandler(urllib.request.HTTPRedirectHandler):
+    def redirect_request(self, req, fp, code, msg, hdrs, newurl):
+        return None
+
+urllib.request.install_opener(urllib.request.build_opener(MyHTTPRedirectHandler))

 def check_url(url, appliance):
+    if url in check_url.url_ok:
+        return
     try:
         req = urllib.request.Request(url, method='HEAD')
         urllib.request.urlopen(req)
-    except (urllib.error.HTTPError, urllib.error.URLError):
-        print('Error with url ' + url + ' in appliance ' + appliance)
+    except urllib.error.HTTPError as err:
+        if err.getcode() >= 400:
+            print('Error with url ' + url + ' - ' + str(err))
+            sys.exit(1)
+    except urllib.error.URLError:
+        print('Invalid URL ' + url)
         sys.exit(1)
+    check_url.url_ok[url] = True
+
+check_url.url_ok={}


 def check_appliance(appliance):```

from gns3-registry.

Thanks for the patch. I have removed the cache of check because now I use multiprocessing and I need to rewrite the cache to support shared data between process.

from gns3-registry.

It's also slower because I discover some url wassn't check properly

from gns3-registry.