aclark4life / vanity Goto Github PK

View Code? Open in Web Editor NEW

66.0 6.0 14.0 160 KB

Get package download statistics from PyPI

License: GNU General Public License v2.0

Python 78.22% Makefile 21.78%

vanity's Introduction

Vanity

Get package download statistics from PyPI [1]

Installation

$ pip install vanity

Usage

Enter package name:

$ vanity django
Django-1.1.3.tar.gz    2010-12-23        4,938
Django-1.1.4.tar.gz    2011-02-09       10,259
  Django-1.2.tar.gz    2010-05-17       24,011
Django-1.2.1.tar.gz    2010-05-24       71,479
Django-1.2.2.tar.gz    2010-09-09        4,388
Django-1.2.3.tar.gz    2010-09-11       82,629
Django-1.2.4.tar.gz    2010-12-23       66,223
Django-1.2.5.tar.gz    2011-02-09       82,325
Django-1.2.6.tar.gz    2011-09-10        2,559
Django-1.2.7.tar.gz    2011-09-11       31,833
  Django-1.3.tar.gz    2011-03-23      363,202
Django-1.3.1.tar.gz    2011-09-10      585,745
Django-1.3.2.tar.gz    2012-07-30        7,649
Django-1.3.3.tar.gz    2012-08-01       31,375
Django-1.3.4.tar.gz    2013-03-05        1,974
Django-1.3.5.tar.gz    2012-12-10       16,880
Django-1.3.6.tar.gz    2013-02-19        2,292
Django-1.3.7.tar.gz    2013-02-20       14,756
  Django-1.4.tar.gz    2012-03-23      437,635
Django-1.4.1.tar.gz    2012-07-30      328,418
Django-1.4.2.tar.gz    2012-10-17      326,088
Django-1.4.3.tar.gz    2012-12-10      280,915
Django-1.4.4.tar.gz    2013-02-19       12,453
Django-1.4.5.tar.gz    2013-02-20      117,366
  Django-1.5.tar.gz    2013-02-26      124,429
Django-1.5.1.tar.gz    2013-03-28      150,413
----------------------------------------------
Django has been downloaded 3,182,234 times!

Enter package name with version:

$ vanity pillow==2.0.0
                Pillow-2.0.0.zip    2013-03-15       61,022
    Pillow-2.0.0.win32-py3.3.exe    2013-03-15          593
    Pillow-2.0.0.win32-py3.2.exe    2013-03-15          379
    Pillow-2.0.0.win32-py2.7.exe    2013-03-15          703
    Pillow-2.0.0.win32-py2.6.exe    2013-03-15          308
Pillow-2.0.0.win-amd64-py3.3.exe    2013-03-15          487
Pillow-2.0.0.win-amd64-py3.2.exe    2013-03-15          328
Pillow-2.0.0.win-amd64-py2.7.exe    2013-03-15          500
Pillow-2.0.0.win-amd64-py2.6.exe    2013-03-15          311
    Pillow-2.0.0-py3.3-win32.egg    2013-03-15          421
Pillow-2.0.0-py3.3-win-amd64.egg    2013-03-15          431
    Pillow-2.0.0-py3.2-win32.egg    2013-03-15          353
Pillow-2.0.0-py3.2-win-amd64.egg    2013-03-15          357
    Pillow-2.0.0-py2.7-win32.egg    2013-03-15        1,160
Pillow-2.0.0-py2.7-win-amd64.egg    2013-03-15          620
    Pillow-2.0.0-py2.6-win32.egg    2013-03-15          730
Pillow-2.0.0-py2.6-win-amd64.egg    2013-03-15          395
-----------------------------------------------------------
Pillow 2.0.0 has been downloaded 69,098 times!

Enter multiple package names:

$ bin/vanity --quiet setuptools distribute
setuptools has been downloaded 34,601,114 times!
distribute has been downloaded 29,661,287 times!
setuptools and distribute have been downloaded 64,262,401 times!

Enter search pattern:

C:\>vanity pillow -p "Pillow-3.0.0.*win32.*py3.2|Pillow-3.0.0.*cp32.*win32"
    Pillow-3.0.0.win32-py3.2.exe    2015-10-01          582
Pillow-3.0.0-cp32-none-win32.whl    2015-10-01          591
-----------------------------------------------------------
Pillow has been downloaded 1173 times!

[1]	Based on https://github.com/collective/Products.PloneSoftwareCenter/commit/601558870175e35cfa4d05fb309859e580271a1f

vanity's People

Contributors

Stargazers

Watchers

Forkers

jnrowe-retired-forks alex hugovk urbanophile mattjegan fgallaire m3rs pratyushprakash danilolimas andm plpxsk pombredanne ftldragon bmustiata

vanity's Issues

Download count not incrementing

Hi,
I noticed a few days ago that download counts weren't incrementing anymore. I ran

$ vanity django

twice with about a day in between and got the same download count (17,560,592). It seemed suspicious that no one would download Django in that time period. I tried using PyPIXmlRpc on its own and also found that the download count was the same and hadn't incremented. Maybe this is an issue at PyPI?

Thanks

Consider official deprecation

@aclark4life @hugovk @Nico-ehs In reference to #22 and vinta/awesome-python#918, perhaps consider directing users to https://github.com/ofek/pypinfo

vanity fails on packages with non-canonical names

Packages whose name is not the same as their canonical name don't appear to work:

$ vanity django
No downloads for django.
$ vanity Django
No downloads for django.

MultiCall requests have been deprecated, use individual requests instead.

(Due to warehouse launch)

Traceback (most recent call last):
  File "/usr/local/bin/vanity", line 11, in <module>
    sys.exit(vanity())
  File "/usr/local/lib/python3.6/site-packages/vanity.py", line 234, in vanity
    pattern=args.pattern)
  File "/usr/local/lib/python3.6/site-packages/vanity.py", line 103, in count_downloads
    for urls, data in get_release_info([package], json=json):
  File "/usr/local/lib/python3.6/site-packages/vanity.py", line 182, in get_release_info
    for package, releases in get_releases(packages):
  File "/usr/local/lib/python3.6/site-packages/vanity.py", line 163, in get_releases
    result = mcall()
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/xmlrpc/client.py", line 882, in __call__
    return MultiCallIterator(self.__server.system.multicall(marshalled_list))
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/xmlrpc/client.py", line 1170, in single_request
    return self.parse_response(resp)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/xmlrpc/client.py", line 1342, in parse_response
    return u.close()
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/xmlrpc/client.py", line 656, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault -32500: 'ValueError: MultiCall requests have been deprecated, use individual requests instead.'>

Vanity moved to python-packages

FYI @hugovk and @Nico-ehs I've just moved vanity to the python-packages organization. You should still have write access as members of the @python-packages/developers team.

Python 3 JSON issue: ValueError: Expecting value: line 1 column 1 (char 0)

Python 3 --json issue:

$ python3 vanity.py vanity --json
Traceback (most recent call last):
  File "vanity.py", line 260, in <module>
    vanity()
  File "vanity.py", line 232, in vanity
    verbose=verbose)
  File "vanity.py", line 102, in count_downloads
    for urls, data in get_release_info([package], json=json):
  File "vanity.py", line 173, in get_release_info
    data = get_jsonparsed_data(PYPI_JSON % package)
  File "vanity.py", line 136, in get_jsonparsed_data
    return json.loads(data)
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/__init__.py", line 318, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 343, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 361, in raw_decode
    raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)

Needs dependency on argparse for Python 2.6

Current version of vanity fails under Python 2.6 unless the "argparse" backport is installed; it should probably declare the dependency when built for 2.6.

(For 2.5, it fails on "from future import print_function", of course.)

should ignore case in package name

best way to do this is using setuptools/Distribute: pkg_resources.safe_name(name.lower())

$ vanity django
No downloads. Are you sure `django` exists?

$ vanity Django
Package `Django` has been downloaded 217995 times!

Request for decision on quoted package names

Motivated by @killvung in #33

Currently, vanity results with the following given quoted input:

~$ vanity " Flask "
No downloads for  Flask .


     *** Note: PyPI stats are broken again; we're now waiting for warehouse. https://github.com/aclark4life/vanity/issues/22 ***

This should either, interpret the input as vanity flask or raise an error. I think either course of action would be fine, however PEP8 - Package and Module Names eludes to the existence of whitespace in a package name, so perhaps this should be treated as improper input and result in an error.

vanity is sentenced to death :(

Hey,

Bad news: http://mail.python.org/pipermail/distutils-sig/2013-May/020855.html :(

"As you have have noticed the download counts on PyPI are no longer updating. Originally this was due to an issue with the script that processes these download counts. However I have now removed the download counts from the PyPI webui and their use via the API is considered deprecated."

Best,

Laszlo

Removing "bot download" from counts

What does this do about the fact that PyPi is being crawled very often? It would be great if we could get a better estimate of "actual" counts, or did you do something to account for the number of bots out there?

blessings doesn't work on windows

... and we probably don't need it anyway.

Python 2.6 and 3.2 broken

PR #23 broke Vanity for Python 2.6 and 3.2.

Python 2.6:

Traceback (most recent call last):
 File "vanity.py", line 24, in <module>
    from collections import deque, OrderedDict
ImportError: cannot import name OrderedDict

Python 3.2:

Traceback (most recent call last):
  File "vanity.py", line 38, in <module>
    import requests
  File "/home/travis/virtualenv/python3.2.6/lib/python3.2/site-packages/requests-2.11.0-py3.2.egg/requests/__init__.py", line 64, in <module>
    from .models import Request, Response, PreparedRequest
  File "/home/travis/virtualenv/python3.2.6/lib/python3.2/site-packages/requests-2.11.0-py3.2.egg/requests/models.py", line 856
    http_error_msg = u'%s Client Error: %s for url: %s' % (self.status_code, reason, self.url)
                                                      ^
SyntaxError: invalid syntax

See https://travis-ci.org/hugovk/vanity/builds/154439793

These are on the PR #26 branch.

cc @mattjegan

Formatting is off

As reported by @pjeby in catalog-sig:


(Also, the formatting is off for the most popular downloads, because
the count column isn't wide enough to show 7 significant figures.)

Add function docstrings

Some of the functions in vanity.py have empty docstrings, for example:

def normalize(name):
    """
    """
    http = HTTPSConnection(PYPI_HOST)
    http.request('HEAD', '/pypi/%s/' % name)
    r = http.getresponse()
    if r.status not in (200, 301):
        raise ValueError(r.reason)
    return r.getheader('location', name).split('/')[-1]

It would be great to fill these in.

More info on docstrings:

Enable Travis CI for this repo

@aclark4life Please can you enable Travis CI for your aclark4life/vanity repo?

That would pick up some recent build problems when PRs are submitted for merge, and before they're merged.

Enable it here: https://travis-ci.org/profile

Add tests

There are not unit tests available as of yet in this repo apart from the general "does it run" TravisCI tests. Perhaps this could be opened up and labelled as "hacktoberfest" to draw attention @aclark4life. It may help bring more contributors to the project.

Fix TODO: refactor normalize when given empty string

test_empty has a TODO that needs fixing:

    def test_empty(self):
        """
        TODO: this test is rather slow to run,
        perhaps normalize could be refactored to check this
        and kick out faster.
        """
        normalized = vanity.normalize("")
        self.assertEqual(normalized, "")

So right now normalize is looking up on PyPI for a package called "" and not finding it. The empty string is a special case; there's no point looking for anything so it'd make more sense to validate and to return something sensible right away.

Fix style issues

Six style issues have been found by Landscape:

https://landscape.io/github/aclark4life/vanity/2

It'd be nice to fix them.

Stats broken since January 2016

This is a minor niggle but it looks like vanity is not getting any updated statistics since about January. So for example vanity openyxl is confidently telling me that the package has never been downloaded.

Enable Coveralls for this repo

Now we're adding more tests, it'd be great to track coverage too. The existing test are ready to use it.

@aclark4life Please could you enable coveralls.io for this repo? https://coveralls.io/repos/new

installation of 'requests' is not automatic

I installed vanity in a newly created python virtualenv thus:
$ pip install vanity
(this is a python 3.5 virtualenv)
but it did not work since 'requests' was not installed with it.
I took a look in 'setup.py' and it does specify 'requests' as prereq.

Can this be? Must the package name also be in requirements.txt?

TypeError: vanity() takes exactly 4 arguments (0 given)

master

Python 2.7, macOS Sierra.

Latest master branch has an error:

⌂75% [hugo:~/github/vanity] master 3s 1 ± vanity vanity
Traceback (most recent call last):
  File "/usr/local/bin/vanity", line 11, in <module>
    load_entry_point('vanity==2.2.2', 'console_scripts', 'vanity')()
TypeError: vanity() takes exactly 4 arguments (0 given)

pypi

This doesn't occur with the latest release, 2.2.2:

⌂99% [hugo:~/github/vanity] master 1 ± pip install vanity
Collecting vanity
Installing collected packages: vanity
Successfully installed vanity-2.2.2
⌂88% [hugo:~/github/vanity] master 2s ± vanity vanity
     vanity-1.0.zip    2011-04-14        2,707
   vanity-1.1.0.zip    2011-10-26        2,488
   vanity-1.1.1.zip    2011-10-28        2,898
   vanity-1.1.2.zip    2011-10-28        2,491
vanity-1.2.0.tar.gz    2012-01-30        2,442
vanity-1.2.1.tar.gz    2012-02-16        2,547
   vanity-1.2.2.zip    2012-07-31        2,206
   vanity-1.2.3.zip    2012-08-08        2,259
   vanity-1.2.4.zip    2013-02-19        1,971
   vanity-1.2.5.zip    2013-03-18        1,992
   vanity-2.0.0.zip    2013-05-26        1,724
   vanity-2.0.1.zip    2013-05-27        1,734
   vanity-2.0.2.zip    2013-05-27        1,751
   vanity-2.0.3.zip    2013-06-28        2,719
   vanity-2.0.4.zip    2014-09-03        1,294
   vanity-2.0.5.zip    2014-09-03        3,034
   vanity-2.1.0.zip    2015-07-03        3,473
   vanity-2.2.0.zip    2016-01-06          325
vanity-2.2.0.tar.gz    2016-01-06          471
   vanity-2.2.1.zip    2016-04-27          106
vanity-2.2.1.tar.gz    2016-04-27          174
   vanity-2.2.2.zip    2016-08-21          341
vanity-2.2.2.tar.gz    2016-08-21          135
----------------------------------------------
vanity has been downloaded 41,282 times!


	 *** Note: PyPI stats are broken again; we're now waiting for warehouse. https://github.com/aclark4life/vanity/issues/22 ***

git bisect

Let's find what introduced this:

⌂85% [hugo:~/github/vanity] master 3s ± pip uninstall -y vanity
Uninstalling vanity-2.2.2:
  Successfully uninstalled vanity-2.2.2

⌂77% [hugo:~/github/vanity] master ± git bisect start
Note: checking out '7db58fc9adc868449b44039ac4bcab5ed0094ecf'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 7db58fc... Fix makefile

⌂71% [hugo:~/github/vanity] 7db58fc* ± git bisect good 5cee8b2b3a377b5c8d71faab9554ceb2b897049b # version 2.2.2

[hugo:~/github/vanity] 7db58fc* ± git bisect bad master
Bisecting: 33 revisions left to test after this (roughly 5 steps)
[97eab11e22ecf7f675038e1c24e09c7a782ff3a8] Update README.rst

Then running python setup.py install && vanity vanity gives 6cb611c as the first bad commit, and it did change def vanity(): to def vanity(packages, verbose, json): in PR #56.

6cb611c#diff-bb1685ad67c8864616acac2e83efbdd9L260

vanity() missing 4 required positional arguments

I'm seeing this in master for 6cb611c and newer …

Traceback (most recent call last):
  File "/Users/alexclark/Developer/python-packages/vanity/bin/vanity", line 11, in <module>
    load_entry_point('vanity', 'console_scripts', 'vanity')()
TypeError: vanity() missing 4 required positional arguments: 'packages', 'verbose', 'json', and 'pattern'

Add version filtering

As suggested by @pjeby

Sort JSON results

The XML-RPC API code in vanity seems overly complicated, yet is at least sorting the results correctly whereas the JSON implementation is not. Here's an overview of the process:

vanity() calls normalize(package) then downloads_total(package)
downloads_total calls release_data(package)

If JSON, release_data does this::

    if protocol:
        for package in packages:
            data = get_jsonparsed_data(PYPI_JSON % package)
            for release in data['releases']:
                urls = data['releases'][release]
                yield urls, data['info']
        return

If not, release_data calls package_releases which does this:

def package_releases(packages):
    """
    """
    mcall = xmlrpc.MultiCall(PYPI_XML)
    called_packages = deque()
    for package in packages:
        mcall.package_releases(package, True)
        called_packages.append(package)
        if len(called_packages) == 100:
            result = mcall()
            mcall = xmlrpc.MultiCall(PYPI_XML)
            for releases in result:
                yield called_packages.popleft(), releases
    result = mcall()
    for releases in result:
        yield called_packages.popleft(), releases

So perhaps the challenge is to deque the JSON results, but I failed in trying to get that to work as expected. The "real" issue is a convoluted data model and possibly differing results from different protocols (as far as I can tell.)