swistakm / graceful Goto Github PK
View Code? Open in Web Editor NEWElegant Python REST toolkit built on top of falcon
Home Page: https://graceful.readthedocs.org
License: BSD 3-Clause "New" or "Revised" License
Elegant Python REST toolkit built on top of falcon
Home Page: https://graceful.readthedocs.org
License: BSD 3-Clause "New" or "Revised" License
Decide which approach is better:
BaseField
class that support null/blankfield = AllowNull(StringField("foo'))
There is mixed code style:
object
and some not when in python3 there is no such need for explicit object
inheritancesuper(SomeClass, self)
call when new simpler super()
is availableSo decide if what to use and why. If decision is to keep things similar to py2 style consider changing your mind and making this package python2.6/2.7 compatible!
This is also true for forbidden and invalid fields
There are few major issues with current validation error representation.
many=True
quits on first invalid value and does not allow to find out if all values were incorrect or only few of them. Also if not all, we don't know which ones.ValueError
exceptions are treated equally with ValidationError
exceptions. This is great because allows for easier integration with built-in or custom types but on the other hand can leak some implementation details to the user agent. It is equally bad as presenting stack trace to user agent. We should treat ValidationError
exceptions as explicit errors that can be safely presented to the user, but ValueError
exceptions should be replaced with generic and safe information in type of "Could not parse" or "Invalid format".This should be obviously a string:
https://github.com/swistakm/graceful/blob/master/src/graceful/fields.py#L183
Usage of mixin classes may be a bit confusing. Extend the guide/resources.rst
to provide introduction to mixins and how to use them with BaseResource
class.
Additional tasks related with this issue:
Generally having access to falcon's req
and resp
parameters is a nice feature to have when you want to read/manipulate headers or provide some custom per-request context value. Typical use case for per-request context variables are database session/connection objects that cannot be stored as resource instance attributes (needs to be unique for every request).
These are usually implemented as middlewares that are able to update req.context
context dictionary. Unfortunately when using generic API resource classes (e.g. ListCreateAPI, PaginatedListAPI) user is expected to provide only basic resource manipulation methods like list()
, retrieve()
, create()
etc. These methods accept only following set of arguments:
params
: dictionary of deserialised keyword parameters (defined using resource-level param classes)meta
: dictionary with additional metadata that will be included in output responsevalidated
(only on create/update handlers): deserialised resource representation provided by client in request body**kwargs
: additional keyword arguments retrieved from falcon's URI template associated to resource route.Because basic resource manipulation methods (list/retrieve/create etc.) accept only specific set of arguments and none of them represents falcon.Request object. The only place where custom context can be provided is **kwargs
dictionary that was primarily supposed to only hold keyword arguments retrieved from URI template.
Right now, the simplest way to provide custom context to every request is to directly override falcon's HTTP method handlers in custom resource classes (inherited from generic and basic resources classes) and pass them further in **kwargs
dictionary using super()
. Example:
# assume Session is some db layer abstraction
def MyResource(ListCreateAPI):
def on_get(req, resp, **kwargs): # <= overrides default HTTP GET handler
super().on_get(req, resp, db_session=Session(), **kwargs)
def list(req,req, db_session, **kwargs): # <= called by super().on_get()
return db_session.all()
def on_post(req, resp, **kwargs): # <= overrides default HTTP POST handler
super().on_post(req, resp, custom_var="some_value", **kwargs)
def create(req,req, validated, db_session, **kwargs): # <= called by super().on_patch()
return db_session.insert(validated)
Of course this is very impractical because every method handler that requires such additional context needs to be overridden. Also **kwargs
are now only expected to hold only values from URI template and their purpose is documented exactly as URI template keyword values.
Overriding every HTTP method handler in exactly the same way is not a clean approach if specific context is required by every API endpoint. This will require a lot of redundant and messy code. One could reduce amount of boilerplate by providing custom base classes (based on graceful generics) with HTTP method handlers overridden to provide additional context. This will still require a lot of code duplication and will be hard to maintain in the long term.
In many frameworks (including falcon) the custom context is usually provided using two methods:
Both middleware and hooks from falcon can be used in graceful resources but their usage is very limited. The only parts of above features that could be used to provide context are:
process_resource(self, req, resp, resource, params)
method of middleware class: the params
dictionary is the save object unpacked as **kwargs
keyword arguments. We cannot use anything else because req
& resp
are unavailable in basic resource manipulation handlers (list/create/retrieve etc.) and the resource
instance can be shared between multiple worker threads.action
argument of falcon.before(action)
decorator with signature of action(req, resp, resource, params)
: same as for middlewares -- only params
parameter that translates directly to **kwargs
can be used as a container for new context values.Additionally usage of hooks in graceful is limited even further. They can be attached to whole resource:
@falcon.before(context_action)
def MyResource(ListCreateAPI):
pass
But cannot be easily attached to specific resource manipulation method. The falcon.before()
expects decorated function to have signature of a HTTP method handler (i.e. on_get(req, resp, **kwargs)
, on_post(req, resp, **kwargs)
and so on). Due to this the only way to attach falcon.before
hooks right now is through following boilerplate.
def MyResource(ListCreateAPI):
@falcon.before(action)
def on_get(req, resp, **kwargs):
super().on_get(req, resp, **kwargs)
@falcon.before(action)
def on_post(req, resp, **kwargs):
super().on_post(req, resp, **kwargs)
So it is too verbose and also counterintuitive. Note that compatibility with falcon hooks is another feature we would like to have so we could support any falcon contrib package that provides hooks. This anyway should be discussed as a separate issue (see #31 )
Existing ways of handling custom context values are too verbose, require too much boilerplate and generally exploit **kwargs
dictionary of list/retrieve/update/create methods that has completely different purpose in both falcon and graceful.
I in my opinion the best approach would be to expose somehow the Request.context
object in the basic resources. The best solution should:
In some situations when we have many=True
option on parameter provided it might be useful to provide a custom container type/class or method for grouping multiple instances.
Now every param class has only one method handler for single value param. Thanks to this implementation of parameter is short and consistent regardless of the many
option setting. If multiple parameters are allowed then we always use list to contain provided parameters. Custom containers would be useful for additional preprocessing of params.
Here is example how this would work on real world example that allows joining multiple occurrences of solrq.Q
object with specified operator (solrq is example external package):
from graceful.params import StringParam
import operator
from functools import reduce
class FilterQueryParam(StringParam):
"""
Param that represents Solr filter queries logically
joined together depending on value of `op` argument
"""
def __init__(
self,
details,
solr_field,
op=operator.and_,
**kwargs
):
if solr_field is None:
raise ValueError("{} needs a `field` param cannot be None".format(
self.__class__.__name__)
)
self.solr_field = solr_field
self.op = op
super(FilterQueryParam, self).__init__(
details, **kwargs
)
def value(self, raw_value):
return Q({self.solr_field: raw_value})
def container(self, values):
return reduce(self.op, values) if len(values) > 1 else values[0]
With such API whenever many
is set to true then .container()
will be fed with list of values got from multiple calls to .value()
method handler. Additionally there can be an additional container_class
keyword argument stored as Param attribute on initialisation (default container=list
) that will be used by default implementation of container method:
class BaseParam(object):
...
def container(self, values):
return self.container_class(values)
In this way this feature can be used even on existing basic Param classes without the need of creating custom param definitions.
Wrapping them could be easier then.
Hi,
I have sent both an empty email and a normal email to graceful at librelist.com but nothing happens
Now it's always JSON but we should be able to support more in a convenient way.
self-explanatory
Restrict "many" field handling to lists in similar way we handle serialization.
Let's assume we have a resource as implemented below:
import falcon
from graceful.resources.generic import PaginatedListCreateAPI
from graceful.serializers import BaseSerializer
from graceful.fields import StringField
class TestSerializer(BaseSerializer):
testStringField = StringField("Test String Field", source='test')
class TestList(PaginatedListCreateAPI):
serializer = TestSerializer()
def create(self, params, meta, validated, **kwargs):
if validated.get('test') == "None":
print("Is it a bug?")
app = application = falcon.API()
app.add_route("/v1/test", TestList())
When client sends a POST request with the following body:
{
"testStringField": null
}
Then the test
field in the validated become "None"
instead of None
.
Proposal:
It can be a solution to check if data is None
in the from_representation
method of the StringField
.
def from_representation(self, data):
"""Convert representation value to ``str`` if it is not None."""
return str(data) if data is not None else None
While, I have been searching around contract-first development approach, I encountered OpenApi Generator. Basically, it can create client and server boilerplate codes for a given api spec. After generation process, only thing that needs to be done is implement the controller methods themselves.
Do we think of contributing to OpenApi Generator project for a new generator of graceful?
I did not analyze it deeply, but what I understand after a quick look is that, it can be pretty straightforward to generate serializers and controllers, as their structure will already be defined in the contract.
This may move the developer experience of graceful to an another level?
What do you guys think about it? Is it worth to give a shot?
Cheers.
todo: more detailed description
Because we do not plan to use Python2 we can mark all keyword arguments as keyword-only (PEP 3102). This will make easier to extend existing base field functionality and custom field classes.
Now graceful shamelessly states that this is "documentation-centered falcon REST toolkit" but it actually does not have any tool to generate such documentation.
It is true that everything that needs to be documented has describe()
method and resources always respond to OPTIONS requests with some resource definition dict. It makes very easy to put such definition into some html template.
I already use my own documenter that does it's work. It is quite opinionated so I need to consider how it should be shared. Most likely this will be a separate repo. I think about using namespace package inside of graceful like 'graceful.extras'. This will be also a convenient way for future developers to create their own extensions in future so there will be no mess with naming like this is in some mature frameworks like django where many of packages have different install/import names and different naming conventions.
TestBase
from falcon.testing
is now deprecated. We should rewrite the tests to pytest style in order to make graceful more future-proof and all tests more consistent.
It happens in forbidden
section.
Hi, recently I found this repository searching some code about authentication and authorization of requests. Please tell me if you are going to support this feature in the future. Some code about it could help me a lot.
It would simplify some usecases like allowing CORS preflights on authorized resources.
We could use annotations feature in some parts of framework:
Instead of using class level attributes. This is only idea to consider.
Simply rethink how this should be handled. Maybe simply creating new param classes is a way to go or we should follow the same concept as it is done in fields.
Research how other frameworks handle that. Mostly Django REST Framework
We need to define how this should be handled. The easiest approach is to assume that fields with source="*"
argument should return dictionary that will be added to the object_dict
instance. Anyway this will make sense only with custom serializer fields. So maybe it would be better completely remove "*"
option from built-in parameters and maybe delegate set/get-attr/key responsibility to custom fields?
This is an enchancement that will surely break compatibility but it will be a solution to problem with deciding which method should have access to which of this basic data structures.
Because there is always single resource instance on given route no state can be stored in this object between method calls. This means that all required data must be always passed to each method that requires it. Because of that graceful
needs to make very opinionated decision about what is available on each step of processing the request. Although this helps developer to focus on what is important in specific method handler (e.g., retrieve, list, create in generic views) it is very hard to extend if something non usual needs to be done - often it requires overriding of many methods.
We could also consider if this context could be passed to serialisers so more complex things could be done like: field serialisation that depends on query string parameters.
Example header value:
Content-Type: application/json; charset=UTF-8
We should allow parameters to be specified muliple times in query string like:
/some/endpoint/?q=something&q=else
This should then create list of q
strings in params
dict
I defined a on_put
method, 50% of the time the method is not allowed.
This is where I add the routes:
class EndpointExpositor(object):
"""Exposes the endpoints, divided in endpoints for data and metadata. The
endpoints for data are those defined in the challenge description. Endpoints
for metadata are all others.
"""
def __init__(self, falcon_api, titulo_tesouro_crud):
self.falcon_api = falcon_api
print()
print(dir(falcon_api))
print()
titulo_tesouro_request_handler = TituloTesouroRequestHandler(titulo_tesouro_crud)
self.endpoint_mapping = {
'/': None,
'/titulo_tesouro': titulo_tesouro_request_handler,
'/titulo_tesouro/{titulo_id}': titulo_tesouro_request_handler
}
endpoints = list(self.endpoint_mapping.keys())
self.endpoint_mapping['/'] = HelpRequestHandler(endpoints)
def expose(self):
for (endpoint, handler) in self.endpoint_mapping.items():
self.falcon_api.add_route(endpoint, handler)
logging.info('Endpoint "{}" exposed.'.format(endpoint))
logging.info('All endpoints exposed.')
This is the base handler:
class RequestHandler(object):
"""Superclass for all request handlers.
"""
def on_get(self, req, resp):
logging.info('GET request received at endpoint "{}"'.format(req.path))
def on_post(self, req, resp):
logging.info('POST request received at endpoint "{}"'.format(req.path))
def on_delete(self, req, resp):
logging.info('DELETE request received at endpoint "{}"'.format(req.path))
def on_put(self, req, resp):
logging.info('PUT request received at endpoint "{}"'.format(req.path))
And this is the child class:
class TituloTesouroRequestHandler(RequestHandler):
"""Handler for POST in endpoint "titulo_tesouro"
"""
def on_put(self, req, resp, titulo_id):
super().on_put(req, resp)
print('test')
I am using Gunicorn 19.7.0. Am I doing something wrong? Shouldn't the method be allowed once I have a on_put
defined?
Since we are breaking backwards compatibility in 1.0.0 anyway, it is time to revisit some class and function names.
Base
suffixes from class names of serializers, resources, fields etc.Main focus of this issue is the reference documentation (docstrings in the source code).
This should be done using Sphinx paragraph level markup:
.. versionchanged:: version
.. versionadded:: version
Verify this
Maybe this is a good idea since generic resources are now quite opinionated and one might want to utilise serializers even without generic resources.
Server SHOULD include such header if this is applicable (see: RFC 2616 Section 9.2)
This is very common feature in many similar serialization solutions. I see two possible approaches:
Decorator approach is the simplest to implement because does not affect serializers code and does not introduce any backwards incompatible changes.
def unbound_method_field(method):
class CallField(fields.BaseField):
def __init__(self):
super().__init__(
method.__doc__,
label=method.__name__,
read_only=True,
source='*',
)
def to_representation(instance):
return method(instance)
return CallField()
class MySerializer(serializers.BaseSerializer):
@unbound_method_field
def my_field_times_10(instance):
return instance['my_field'] * 10
The problem is that we cannot use real methods but only unbound functions defined in class namespace. This is due to how serializer's metaclass works. This creates a bit counterintuitive definitions within class instance. We could use a bit more complex implementation:
def bound_method_field(method):
class CallField(fields.BaseField):
def __init__(self):
super().__init__(
method.__doc__,
label=method.__name__,
read_only=True,
source='*',
)
def to_representation(self, instance):
return method(self, instance)
return CallField()
class MySerializer(serializers.BaseSerializer):
@bound_method_field
def my_field_times_10(instance):
return instance['my_field'] * 10
But this will be even more counter intuitive because such method will be "bound" to the field instance and not serializer instance.
Pros:
Cons:
classmethod
and staticmethod
dictionariesWe could take an example from Django REST Framework and Marshmallow projects and make all field instances bound to the serializer instance on BaseSerializer.fields
property access. In order to ensure proper binding of inherited fields from base serializer classes we would have also to perform deep copy of all that fields. This would not have a negative impact on performance once we cache fields
dictionary items (see: #54).
Binding fields to the parent is a bit easier in DRF because their serializers are initialized per-object instance. Serializers in graceful are more static and stateless object translators. Still, I believe it can be implemented in similar way.
Pros:
staticmethod
and classmethod
.Cons:
fields
property handling on first access to ensure no performance impact.copy.deepcopy()
call. Also dynamic field manipulation once serializer is initialized will be limited.Related work:
BindingDict
class that ensures fields are always bound to the serializer instancedeepcopy()
during schema initializationSide note: the other advantage of approach based on Marsmallow's code is that we could finally pass fields name in the serializer during binding. Then fields would know their names and serializers would not have to wonder if field has its own source specified (see 1.x code). Still I believe that current API of read_instance
makes instance processing a bit more performant and it is worth leaving as it is. We could even try to cache field sources inside of serializers namespace.
It seems that field binding is currently the best approach even if it results in more complex serializer's code. It should land in 1.0.0 release if we want to implement this. It is not that invasive change but at implementing it in 0.x would later require major changes in actual new field classes. We can optionally introduce this change gradually:
I run on this from time to time. In most cases nested resource can be handled by using custom fields but they cannot be trivially validated/serialized/deserialized. Also those new fields classes cannot be reused as unique resource types so in some cases it would be useful to have support for nesting resource definitions.
I see two approaches:
SerializerField
) that can work like a single field and delegates validation and (de)serialisation to inner serialised stored as a attribute. Pros: more explicit, allows both types of objects to have different apis. Cons: feels like more complex and also will require more coding to define API that uses nested resources.BaseSerializer
a descendant of BaseField
class. This should also work but will require serialisers and fields to have the same api. I'm afraid that this will make auto-describing of API more painful. Pros: shorter API descriptions. Cons: fields and serialisers have slightly different purposes so such coupling may introduce difficulties in future.As noted in #27 PR default values are deserialised and validated every time they are retrieved and this could be obviously improved. Consider getting rid of that limitation or making default value deserialised only once on init.
Consider following example:
class CategoriesList(PaginatedListAPI()):
def list(params, meta, uri_template_variable, **kwargs):
return [uri_template_variable] * 5
api.add_route("/v0/categories/{uri_template_variable}", ListResource())
Will raise exception on accessing /v0/categories/{ui_template_variable}
and this is design inconsistency with other endpoint classes:
TypeError: list() missing 1 required positional argument: 'uri_template_variable'
This will ensure that additional validation on whole resource level can be easily added to serialiser.
When this is done there should be also documentation extended with information how to perform such validation.
The best way I think is to add similar as_bad_request
method to ValidationError
It is very useful feature to allow bulk creation and/or update of objects represented by resources. Initial idea is to add PATCH
handler to ListCreateAPI
class. Maybe also CreateMixin
should be modified to allow bulk creation.
The rationale behind using PATCH
is that this method is used to partially modify resource. List resources represent collections of objects, so passing a list of new representations to such resource may be considered as intention to add/modify them within existing collection.
The PUT
method could be understood as intention to replace whole collection in list resource. Usefulness of that is questionable but it may be explored in future.
The important thing that needs to be discussed is the question if we should separate the bulk creation from bulk update or leave that to the user. In my opinion default "upsert-like" behaviour is more flexible. Developer may easily decide how to handle his case when writing his storage integration code. Anyway this should be decided before implementing because this will affect API naming and documentation.
Falcon development is pretty fast recently. Make sure we are up to date. Make testing a real matrix with different versions of python and different versions of falcon.
Edit: also Python 3.5 is available fro quite long time so make sure we cover it in tests too.
Now it is markdown but should be converted to reStructuredText. Fortunately this does not require additional release AFAIK.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.