wq / django-rest-pandas Goto Github PK

See http://paste.ubuntu.com/10862242/.

Currently the only way to get around this (short of adding id back to the included fields) is to subclass PandasSerializer and override get_index() to ensure it returns None (or any included field).

A better solution would be to have the default get_index() check for the id field before using it as the index, or to make it return None by default and always require it to be overridden for users that want an index to be set on their DataFrame.

Can you help me make an example site? I'm looking to use this with bokeh.

I would love to serve up this citibike dataframe https://www.wakari.io/sharing/bundle/paddy/pydata_citibike

Add a renderer or two that leverage GeoPandas exports. At a minimum, support GeoJSON and Shapefiles (perhaps as a .zip?).

I use:
Python 2.7
Django 1.9
DRF 3.2.2
Pandas 0.17.1
And i have this problem:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 350, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 324, in execute
    django.setup()
  File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/usr/local/lib/python2.7/dist-packages/django/apps/config.py", line 90, in create
    module = import_module(entry)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "build/bdist.linux-x86_64/egg/rest_pandas/__init__.py", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/rest_pandas/views.py", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 98, in <module>
    class APIView(View):
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 103, in APIView
    authentication_classes = api_settings.DEFAULT_AUTHENTICATION_CLASSES
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/settings.py", line 203, in __getattr__
    val = perform_import(val, attr)
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/settings.py", line 148, in perform_import
    return [import_from_string(item, setting_name) for item in val]
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/settings.py", line 160, in import_from_string
    module = importlib.import_module(module_path)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/authentication.py", line 13, in <module>
    from rest_framework.authtoken.models import Token
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/authtoken/models.py", line 16, in <module>
    class Token(models.Model):
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 94, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 239, in get_containing_app_config
    self.check_apps_ready()
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 124, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

And address any issues that arise.

I can start contributing

If there isn't a 'form' param in request ,I want return an error, but raised an error.

def list(self, request: Request, *args: Any, **kwargs: Any) -> Response:
    if self.request.query_params.get('form') and self.request.query_params.get('format') == 'xlsx':
        return Response({'error' : 'no form'})

error:

Response data is a dict, not a DataFrame!

will be sending a PR

As PandasView inherits from APIListView I would expect that the attribute permission_classes can be used
to set permission on this view. However, the following view can be accessed when logged out

class DownloadFileViewAPI(PandasView):
    queryset = Well.objects.all()
    permission_classes = (MayUploadWriteOrDownloadRead,)
    filter_class = WellFilter
   
    serializer_class        = MetaDataSerializer # extends Serializer
    pandas_serializer_class = FlatteningPandasSerializer
    renderer_classes        = [PandasExcelNoColNoIndexRenderer,
                               PandasCSVNoColNoIndexRenderer,
                               PandasJSONRenderer]

While the following view can only be accessed when logged in

class WellList(generics.ListAPIView):
    queryset = Well.objects.all()
    permission_classes = (MayUploadWriteOrDownloadRead,)
    serializer_class = WellSerializer
    filter_class = WellFilter

The only difference is the serializer class and the View they inherit from so I would say this is something with PandasView, but I cannot guess why. Am I missing something here?

Adding these mixin's restricts the view to logged in users again.

class DownloadFileViewAPI(LoginRequiredMixin, PandasView):

So this is my workaround for now

For personal reasons, I wanted to call the PandasCSVRenderer on a dataframe outside of a view.

Because renderer_context is defaulted to None, the following code will throw TypeError: argument of type 'NoneType' is not iterable:

from rest_pandas import renderers


data_frame = pandas.DataFrame(some_data)
print renderers.PandasCSVRenderer().render(data_frame)
>>>TypeError: argument of type 'NoneType' is not iterable

I suggest two options to fix the following code:

1. Change the if statement:

if renderer_context and 'response' in renderer_context:

2. Add the following at the top of the function:

if renderer_context is None:
    renreder_context = {}
...

Hi,

Currently I am working on receiving files form users (csv, xlsx) and pass this files to rest-pandas so its content can be imported to your models or any custom python object.

Before going further I would like to know if the scope of this project includes importing data from files.

If this is part of scope or if project leaders agree to include this part I will be posting my progress.

Have a nice day!

I'm trying to create an option on all of my endpoints to output XLS files when the appropriate header is set. I may not be understanding correctly, but here's what I've tried to do:

REST_FRAMEWORK = {
    ...
    'DEFAULT_RENDERER_CLASSES': (
        'rest_framework.renderers.JSONRenderer',
        'rest_framework.renderers.BrowsableAPIRenderer',
        'rest_pandas.renderers.PandasExcelRenderer',
    ),
}

I haven't included it in INSTALLED_APPS as I've read about the problems that causes. However, when trying this method, Django's runserver is giving me this error:

ImportError: Could not import 'rest_pandas.renderers.PandasExcelRenderer' for API setting 'DEFAULT_RENDERER_CLASSES'. ImportError: cannot import name 'APIView'.

When I try this, runserver comes up but the OPTIONS only shows application/json and text/html under the renders section:

from rest_pandas.renderers import PandasExcelRenderer

REST_FRAMEWORK = {
...
    'DEFAULT_RENDERER_CLASSES': (
        'rest_framework.renderers.JSONRenderer',
        'rest_framework.renderers.BrowsableAPIRenderer',
        PandasExcelRenderer,
    ),
}

I'm sure I'm missing something simple, and I'll be happy to issue a how-to PR on the README if someone can help me get this working! :)

With a model that has an ID column not named ID, and the following code:

class TestView(PandasView):
    queryset = WeirdIDModel.objects.all()
    serializer_class = WeirdIDModelSerializer

when trying to query the class, you get an error KeyError, "id".

Reading that stack trace and the source I see that this is because by default we run "set_index('id') on the df, and this can be fixed in three obvious ways:

1. set pandas_index on the meta of the serializer
2. override get_index_fields on the serializer
3. override get_dataframe

1. setting the index seems to be needless duplication (I've already written once what the index of this model is)
2. an override to get_index_fields I think would just create behaviour that should be default:
   WeirdIDModelSerializer._meta.pk.name gives the ID, for my version of Django at least

so something like

try:
    id_name = (self.<field for model>)._meta.pk.name
except:
    id_name = 'id'
default_fields = [id_name]

3. Thinking about it more, and reading the source, it seems odd to even set an index, especially without preserving the original column - I think that DRF classes that we're mimicking would by default pass the index through to the client, and since I'm trying to drop-in replace some existing code which might rely on that, I'll have to replicate that manually.

Add support for TemplateHTMLRenderer, probably with a special function on the view to allow specifying an html_context to augment the table generated by to_html() on the DataFrame. This will make it easier to have e.g. mysite.url/chart be the landing page that shows data from mysite.url/chart.csv

To modify the behavior of the csv renderer, I have subclassed PandasCSVRenderer

class PandasCSVNoColNoIndexRenderer(PandasCSVRenderer):
    def get_pandas_kwargs(self, data, renderer_context):
        output = super().get_pandas_kwargs(data, renderer_context)
        output['header'] = False
        output['index'] = False
        # Remark on padding zeros: 
        #   removing padding zeros is platform dependent so we better leave it this way (%-d in linux, %#d in windos)
        #   https://stackoverflow.com/questions/28894172/why-does-d-or-e-remove-the-leading-space-or-zero
        output['sep'] = ';'
        output['date_format'] = '%d-%m-%Y %H:%M'  

        return output

All options work out fine, except for the 'date_format' option. I have set in settings.py

and indeed when inside get_pandas_kwargs

renderer_context['response'].data.iloc[10,3]

returns

Timestamp('1985-07-01 00:00:00')

The string is also correct as

renderer_context['response'].data.iloc[10,3].strftime('%d-%m-%Y %H:%M')

yields the desired output (30-11-1999 01:02)

Do you have any clue why this is happening?

I really would love to use this nice package but I'm getting this error no matter what I do.

I see that no one is really maintaining this package anymore but I really hope someone may clarify this for me.

1. Added 'rest_pandas' in installed app BEFORE 'rest_framework'
2. Added 'rest_pandas.renderers.PandasExcelRenderer' in DEFAULT_RENDERED_CLASS
3. I see an xlsx selectable option in the api django rest framework view
4. Added PandasMixin to my BaseView
5. Response data is a ReturnList, not a DataFrame! Did you extend PandasMixin?

Following my baseApi view:

class BaseApi(mixins.RetrieveModelMixin,
              mixins.ListModelMixin,
              mixins.UpdateModelMixin,
              mixins.DestroyModelMixin,
              PandasMixin,
              BulkCreateModelMixin,
              BulkUpdateModelMixin,
              generics.GenericAPIView):

    filter_backends = (OrderingFilter, SearchFilter, DjangoFilterBackend)
    pagination_class = StandardResultsSetPagination

    def get_list(self, request, *args, **kwargs):

        queryset = self.filter_queryset(self.get_queryset())

        page = self.paginate_queryset(queryset)

        if page is not None:
            serializer = self.get_serializer(page, many=True)
            response = self.get_paginated_response(serializer.data)
        else:
            serializer = self.get_serializer(queryset, many=True)
            response = Response(serializer.data)

        query = {
            "ordering_fields": [],
            "filter_fields": [],
            "search_fields": [],
            "element_count": len(response.data),
            "element_total_count": queryset.count(),
            "previous_page": self.paginator.get_previous_link(),
            "next_page": self.paginator.get_next_link(),
            "page_count": self.paginator.page.paginator.num_pages,
            "current_page_size": self.paginator.get_page_size(request),
            "current_page": self.paginator.page.number,
            "max_page_size": self.paginator.max_page_size,
        }

        for backend in list(self.filter_backends):
            if isinstance(backend(), OrderingFilter):
                query["ordering_fields"] = backend().get_ordering(
                    request=self.request,
                    queryset=self.queryset.all(),
                    view=self
                ) or []
            elif isinstance(backend(), SearchFilter):
                query["search_fields"] = backend().get_search_terms(request)
            elif isinstance(backend(), DjangoFilterBackend):
                for f in self.request.query_params.keys():
                    if hasattr(self, 'filter_fields'):
                        if f in self.filter_fields:
                            query["filter_fields"].append({f: self.request.query_params[f]})
                    elif hasattr(self, 'filter_class'):
                        if f in self.filter_class.Meta.fields:
                            query["filter_fields"].append({f: self.request.query_params[f]})

        for k, v in query.items():
            response[k.replace("_", " ").title().replace(" ", "-")] = json.dumps(v)

        if "debug" in self.request.query_params.keys() and self.request.query_params["debug"] == "true":
            response.data = {
                "query": query,
                "data": response.data,
            }

        return response

    # API BASE VIEWS ###########################
    def get(self, request, *args, **kwargs):
        if 'pk' in kwargs.keys():
            return self.retrieve(request, *args, **kwargs)
        else:
            return self.get_list(request, *args, **kwargs)

    def put(self, request, *args, **kwargs):
        if 'pk' in kwargs.keys():
            return self.update(request, partial=True)
        else:
            return self.partial_bulk_update(request, *args, **kwargs)

    def delete(self, request, *args, **kwargs):
        return self.destroy(request, *args, **kwargs)

    def post(self, request, *args, **kwargs):
        return self.create(request, *args, **kwargs)
    # END API BASE VIEWS #######################

I implemented per instructions and not sure why I'm seeing this error.

Is it possibile to extend a standard DRF ViewSet with a method implementing a DRP PandasView?

e.g. the DataFrameManager. This should probably just work but need to test and identify any necessary changes.

Basically I am trying to generate XLSX File and stream it for download.

My tests are like this:

response = self.client.get('/grades/export?format=csv')
print response
self.assertEqual(response.status_code, status.HTTP_200_OK)
print '===================================================='
response = self.client.get('/grades/export?format=xlsx')
print response
self.assertEqual(response.status_code, status.HTTP_200_OK)

And my Output is this:

ERROR: test_export_grades_to_excel (functional_tests.tests.GradesExportTestCase)
Traceback (most recent call last):
File "C:\Users\lhernandezm\Documents\Git\alvin\functional_tests\tests.py", line 528, in test_export_grades_to_excel
response = self.client.get('/grades/export?format=xlsx')
File "C:\Users\path\lib\site-packages\rest_framework\test.py", line 162, in get
response = super(APIClient, self).get(path, data=data, **extra)
File "C:\Users\path\lib\site-packages\rest_framework\test.py", line 88, in get
return self.generic('GET', path, **r)
File "C:\Users\path\lib\site-packages\django\test\client.py", line 379, in generic
return self.request(**r)
File "C:\Users\path\lib\site-packages\rest_framework\test.py", line 159, in request
return super(APIClient, self).request(**kwargs)
File "C:\Users\path\lib\site-packages\rest_framework\test.py", line 111, in request
request = super(APIRequestFactory, self).request(**kwargs)
File "C:\Users\path\lib\site-packages\django\test\client.py", line 466, in request
six.reraise(*exc_info)
File "C:\Users\path\lib\site-packages\django\core\handlers\base.py", line 164, in get_response
response = response.render()
File "C:\Users\path\lib\site-packages\django\template\response.py", line 158, in render
self.content = self.rendered_content
File "C:\Users\path\lib\site-packages\rest_framework\response.py", line 71, in rendered_content
ret = renderer.render(self.data, media_type, context)
File "C:\Users\path\lib\site-packages\rest_pandas\renderers.py", line 30, in render
"Response data is a %s, not a DataFrame!" % type(data)
Exception: Response data is a <type 'NoneType'>, not a DataFrame!

CSV Test Succeeds! But XLSX gives me the error above.

My current pip freeze is:
colorama==0.3.3
coverage==3.7.1
decorator==4.0.4
diff-match-patch==20121119
Django==1.8.2
django-appconf==1.0.1
django-audit-log==0.8.0
django-autoslug==1.8.0
django-cities-light==3.1.2
django-compressor==1.5
django-cors-headers==1.1.0
django-email-as-username==1.6.7
django-extensions==1.5.5
django-filter==0.10.0
django-guardian==1.3
django-import-export==0.2.8
django-rest-swagger==0.3.4
djangorestframework==3.2.4
djangorestframework-jwt==1.7.2
factory-boy==2.5.2
fake-factory==0.5.2
flake8==2.4.1
ipdb==0.8.1
ipython==4.0.0
ipython-genutils==0.1.0
jdcal==1.0
Markdown==2.6.2
mccabe==0.3.1
numpy==1.10.0
openpyxl==1.8.6
pandas==0.16.2
path.py==8.1.1
pep8==1.5.7
pickleshare==0.5
Pillow==2.8.1
psycopg2==2.6.1
pyflakes==0.8.1
Pygments==2.0.2
PyJWT==1.3.0
python-dateutil==2.4.2
pytz==2015.6
PyYAML==3.11
rest-pandas==0.3.2
selenium==2.47.1
simplegeneric==0.8.1
six==1.9.0
tablib==0.10.0
traitlets==4.0.0
Unidecode==0.4.18
Unipath==1.1
Werkzeug==0.10.4
wheel==0.24.0
XlsxWriter==0.7.6
xlwt==1.0.0

I know this is beta so Docs are for next development phase, please let me know if I am doing something wrong or if it is a bug.

Also I would like to collaborate with Docs.

There has not been a release on pypi since 2019, are there any plans to release? There are features on the main branch that I would like to use, but without tracking main.

When I serialize this model

    site            = models.ForeignKey(Site, on_delete=models.CASCADE)
    name         = models.CharField(max_length=100, )
    start_date_time = models.DateTimeField(auto_now=False, auto_now_add=False, verbose_name='Date and time at which the well is deployed.')

with this serializer

class FileSerializer(serializers.ModelSerializer):
    class Meta:
        model = Well
        fields = ('id', 'name', 'site', )

The resulting dataframe has a column start_date_time but this is a string rather than a datetime object. I work around this by using transform_dataframe and convert the column using pandas.to_datetime function. Is this the way to go or am I missing someting?

Add support for Python 3.

Thanks for this great software, I have been using PandasSimpleView to greatly simplify the rest api and provide different useful formats.

One feature that seems missing to me is the ability to suggest a filename for the browser, currently the views do not set a name and it will be confusing for my user to download several tables of data with the same "download.format" name.

I think it is currently possible to override the renderer and set this, but it would be a nice feature to have as a default option in the view.

Can you please add the license to manifest.in so that it gets included in the pypi tar file.

I am experiencing this error on Windows 7.

WindowsError: [error 32] The process cannot access the file because it is being used by another    process.

When this code is executed:

def transform_dataframe(self, dataframe):
    from pandas.io.excel import ExcelWriter
    writer = ExcelWriter('output.xlsx')
    dataframe.to_excel(writer, 'Page1')
    return dataframe

Output.xlsx is being generated on the root folder of the project.

Hello,
I need to construct my json like :

{
  "x": [
    5.0,
    3.0,
    8.0,
    5.0]
  ], 
  "y": [
    7.0,
    2.0,
    6.0,
    4.0]
}

I have this, which is exactly what to_json gives:
[{"x":5,"y":7},{"x":3,"y":2},{"x":8,"y":6},{"x":5,"y":4}]

Is it possible to change this behavior ?
It might be a pandas question.

Thanks for your efforts ! I understand better how REST works now :)
Thomas

Looks like the name changed case (recently?)

Implement a simple mechanism for caching generated dataframes (probably with Redis as the recommended backend).

DRP currently requires the use of a custom serializer.data method that returns a Pandas dataframe. It might be more flexible to allow whatever serializer people want to use and do the DataFrame creation & manipulation entirely on the view. This would likely mean overridding view.list() and adding a call to a new self.create_dataframe(serializer.data)

Apologies, this is probably not the best place to ask this question, but it might lend itself to some useful enhancements to others and I wasn't sure if there was a better forum for asking questions.

I am extending PandasExcelRenderer so that I can 1) use a permanent file (with a filename based on URL parameters passed to renderer_context) stored on my server instead of a temp file and 2) set the sheetname of the file based on the filename of the permanent file. The filename and sheetname are defined in get_pandas_kwargs based on renderer_context and sheet_name is specified in the return value of get_pandas_kwargs.

Is there any way to access this information in PandasSimpleView? Currently I am duplicating the code form get_pandas_kwargs (to determine filename) in PandasSimpleView.get_pandas_filename(), but was hoping for something cleaner and more DRY.

Any guidance or suggestions would be greatly appreciated. Again, apologies if this is the wrong place to post this question, a redirect to a better forum would be appreciated as well if applicable.

The links to this library just show a page mentioning that the app had been moved to this project already.

Please avoid from packages import * from __init__.py