Giter VIP home page Giter VIP logo

Comments (11)

femtotrader avatar femtotrader commented on August 25, 2024

That's a very interesting idea

Some sample packages can be found here
https://github.com/datasets/

Some code can be found here
https://github.com/okfn/dptools/blob/master/bin/load-postgresql.py
https://github.com/okfn/dptools/blob/master/bin/load-sqlite.py
https://github.com/okfn/jsontableschema-py
https://github.com/okfn/jts-sql
https://github.com/trickvi/datapackage

see also
trickvi/datapackage#22

from pandas-datareader.

femtotrader avatar femtotrader commented on August 25, 2024

Not a lot of luck with trickvi/datapackage

In [1]: import datapackage

In [2]: datapkg = datapackage.DataPackage('http://data.okfn.org/data/cpi/')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-21b6afb5084e> in <module>()
----> 1 datapkg = datapackage.DataPackage('http://data.okfn.org/data/cpi/')

//anaconda/lib/python3.4/site-packages/datapackage/datapackage.py in __init__(self, *args, **kwargs)
    100         elif len(args) == 1:
    101             self.base = args[0]
--> 102             descriptor = self.get_descriptor()
    103             super(DataPackage, self).__init__(**descriptor)
    104         else:

//anaconda/lib/python3.4/site-packages/datapackage/datapackage.py in get_descriptor(self)
    588         # Load the descriptor json contents
    589         str_descriptor = descriptor.read()
--> 590         json_descriptor = json.loads(str_descriptor)
    591
    592         # Return the descriptor json contents (as the dict json.load returns

//anaconda/lib/python3.4/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    310     if not isinstance(s, str):
    311         raise TypeError('the JSON object must be str, not {!r}'.format(
--> 312                             s.__class__.__name__))
    313     if s.startswith(u'\ufeff'):
    314         raise ValueError("Unexpected UTF-8 BOM (decode using utf-8-sig)")

TypeError: the JSON object must be str, not 'bytes'

from pandas-datareader.

noelmas avatar noelmas commented on August 25, 2024

Sorry to hear that @femtotrader - can you post this issue in https://discuss.okfn.org/c/open-knowledge-labs/data-packages ? someone will sort it for you ASAP

from pandas-datareader.

femtotrader avatar femtotrader commented on August 25, 2024

I'm trying to signup with GitHub and get

Sorry, there was an error authorizing your account. Perhaps you did not approve authorization?

from pandas-datareader.

femtotrader avatar femtotrader commented on August 25, 2024

trickvi/datapackage#44

from pandas-datareader.

femtotrader avatar femtotrader commented on August 25, 2024

odo http://odo.readthedocs.org/ might be considered trickvi/datapackage#45

from pandas-datareader.

trickvi avatar trickvi commented on August 25, 2024

@femtotrader I will have a look at this on my and and try to be of as much assistance as I can (both issues). Thanks for reporting. I will keep you updated here in this issue as well.

from pandas-datareader.

femtotrader avatar femtotrader commented on August 25, 2024

trickvi/datapackage is now fixed for Python 3.

It's possible to do

>> import datapackage
>>
>> # Note trailing slash is important for data.okfn.org
>> datapkg = datapackage.DataPackage('http://data.okfn.org/data/cpi/')
>>
>> print(datapkg.title)
Annual Consumer Price Index (CPI)
>> print datapkg.description
Annual Consumer Price Index (CPI) for most countries in the world. Reference year is 2005.
>> # Weird example just to show how to work with data rows
>> print(sum([row['CPI'] for row in datapkg.data]))
405442.60078415077

so converting to a DataFrame can be done using

df = pd.DataFrame(list(datapkg.data))

but that's probably quite long when processing very long table.

Maybe a method should be add to convert to DataFrame when Pandas is installed

it's now possible to do

>> df['CPI'].sum()
405442.60078415077

from pandas-datareader.

femtotrader avatar femtotrader commented on August 25, 2024

A new Python DataPackage client is available at https://github.com/datapackages/datapackage-py

from pandas-datareader.

danfowler avatar danfowler commented on August 25, 2024

New Python Data Package client: https://github.com/frictionlessdata/datapackage-py

There is also a plugin for reading Tabular Data Package resources into Pandas data frames: https://github.com/frictionlessdata/jsontableschema-pandas-py

from pandas-datareader.

bashtage avatar bashtage commented on August 25, 2024

Seems unnecessary given other packages.

from pandas-datareader.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.