Dynamic REST Tutorial

Introduction

This document will take you through the basics (and not-so-basics) of developing Dynamic REST APIs. Dynamic REST (or “DREST”) is an extension to Django REST Framework (“DRF”) that makes it easier to implement uniform REST APIs with powerful features like field inclusion/exclusion, flexible filtering, pagination, and efficient “sideloading” of related/nested data.

Setup

Before starting the tutorial, you’ll need to set up Dynamic REST in your dev environment. Refer to the README for details, but it should look something like:

$ git clone git@github.com:AltSchool/dynamic-rest.git
$ cd dynamic_rest
$ make fixtures
$ make server

This should start up Django at http://localhost:9002.

Minimal API

We’ll first implement a very basic API that exposes a minimal view of a basic model.

Model

We will be creating an API for the following model. The model has conveniently already been created for you, and is listed here for reference (the actual code is in tests/models.py):

class Event(models.Model):
    name = models.TextField()
    status = models.TextField()
    location = models.ForeignKey('Location', null=True, blank=True)
    users = models.ManyToManyField('User')

Minimal serializer

All objects returned by APIs must have a serializer. The serializer is effectively your contract with the API user – it’s where you declare which fields are available, what the field data types are, which fields are deferred, and so on.

Add the following to tests/serializers.py (note that Event needs to be imported at the top):

from tests.models import Event

class EventSerializer(DynamicModelSerializer):
    class Meta:
        model = Event
        name = 'event'
        fields = ('id', 'name', 'status')

The example above is a complete serializer. In this particular example, the Meta class contains all the information the framework needs. It knows what the underlying model is, which fields to include in the output, and what to label the returned data. (If you’re wondering why we don’t have location and users, don’t worry, we’ll get to those in a bit.)

Minimal ViewSet

Next we’ll add a viewset, which implements the API (in tests/viewsets.py).:

from tests.serializers import EventSerializer
from tests.models import Event

class EventViewSet(DynamicModelViewSet):
    """
    Events API.
    """
    queryset = Event.objects.all()
    serializer_class = EventSerializer

This implements a simple API, which provides full CRUD capabilities, as well as list functionality with filtering, field inclusion, sideloading, and pagination.

Map to URL

Update the URLs file (in tests/urls.py):

router.register(r'events', viewsets.EventViewSet)

Note that, as a convention, the URL resource name should be the pluralization of the name declared in the serializer Meta class (for irregular pluralizations, a plural_name attribute can be set in the Meta class).

Try it out

Try out a few URLs (assumes you’re running locally on port 9002):

Serializer with relations

We’ll extend the previous example by adding a couple of relational fields to the serializer:

class EventSerializer(DynamicModelSerializer):
    location = DynamicRelationField('LocationSerializer', deferred=False)
    users = DynamicRelationField(
        'UserSerializer', many=True, deferred=True)

    class Meta:
        model = Event
        name = 'event'
        fields = ('id', 'name', 'status', 'location', 'users')

Here, we’ve added the two relational fields. A few notes:

  • For relational fields, map a field name to a DynamicRelationField object, and then pass in the serializer to use when that field is being serialized.
  • If using a serializer that hasn’t been defined yet, you can use the serializer name (or full import path) as a string.
  • We set deferred=True on users, which means that field will not be returned unless specifically requested. For many-relations, this could yield better performance since that saves a DB query.
  • Relational fields can be a Foreign-Key (one to many), a Many-to-Many, or Many-to-One (a.k.a inverse of a foreign key).

Now try some queries:

Note on optimizations

DREST has reasonable optimization strategies built in, which frees up the API developer from having to understand and employ Django optimization strategies. Some optimizations currently implemented include:

  • Prefetching of sideloaded fields - For example, when we sideloaded users above, DREST internally constructed a Prefetch query so Django only performed 2 queries: one for events, one for users.
  • Automatic prefetching - If we were to turn deferred off on users, DREST will automatically prefetch users (otherwise Django will issue a separate query per event object).
  • Field selection - DREST will only request fields that are necessary from the DB, which could reduce data transfer between Django and the DB.
  • Prefetch filtering for sideloads - When we filtered sideloads, the filtering criteria was converted into a Django query and attached to the prefetch request so that it could be converted into the appropriate SQL query.

Advanced Topics

As you saw, simple APIs can be implemented with very little code. Obviously, life is more complicated than that...

Serializer field aliasing

Sometimes we want serializer fields to be named something other than the underlying model (or Django-ism like *_set). We can do this by using the DRF source field attribute. Try modifying the EventSerializer thusly:

class EventSerializer(DynamicModelSerializer):
    location = DynamicRelationField('LocationSerializer', deferred=False)
    participants = DynamicRelationField(
            'UserSerializer', source='users', many=True, deferred=True)

    class Meta:
        model = Event
        name = 'event'
        fields = ('id', 'name', 'status', 'location', 'participants
')
Everything still works as expected:

Query parameter injection

Sometimes we want to modify DREST’s default behaviors. Perhaps we want default filters applied. Or we want some relations to be sideloaded by default. One easy way to do this is through query parameter injection. Try adding the following method to the EventViewSet:

class EventViewSet(DynamicModelViewSet):
    # …

    def list(self, request, *args, **kwargs):
        # sideload location by default
        request.query_params.add('include[]', 'location.')

        # filter for status=current by default
        status = request.query_params.get('filter{status}')
        if not status:
            request.query_params.add('filter{status}','current')

        return super(EventViewSet, self).list(request, *args, **kwargs)

Checkout the default results: http://localhost:9002/events/

Queryset override

By default, DREST/DRF will query the model declared in the viewset’s serializer, which is to say, all objects in that model are in-scope and query-able. If you want to change that, you can override the get_queryset() method in your viewset. One use-case might be to dynamically apply filters that can’t/shouldn’t be overridden by filter{} params. In a viewset, you might do something like:

def get_queryset(self, *args, **kwargs):
    is_admin = user_is_admin(self.user)
    if is_admin:
        return Foo.objects.all()
    else:
        return Foo.objects.filter(creator=self.user)

In this hypothetical example, this would constrain the scope of query-able objects for non-admin users to only those objects created by them.

(Note: Foo.objects.all() does not actually return any objects. It returns a QuerySet which only gets evaluated when its contents are requested, and until a QuerySet is evaluated, it is possible to keep chaining more filters. Internally, DREST/DRF takes the QuerySet returned by get_queryset and modifies it, before it is eventually evaluated.)

Setting serializer context

The DRF way of setting serializer context works as well (serializer context is accessible within the serializer as self.context).:

class FooViewSet(DynamicModelViewSet):
    # ....
    def get_serializer_context(self):
        context = super(FooViewSet, self).get_serializer_context()
        foo = self.request.query_params.get('foo')
        # modify context
        return context

DREST-ifying Existing ViewSet

When migrating existing APIs, it might be possible to “layer” on DREST into an existing ViewSet by using WithDynamicViewSetMixin. Note that getting the old class to play nice might require some shenanigans (see super below):

from dynamic_rest.viewsets import WithDynamicViewSetMixin

class NewViewSet(WithDynamicViewSetMixin, OldViewSet):
    # …
    def list(self, request, *args, **kwargs):
        # …

        # Skip parent’s list() method
        return super(OldViewSet, self).list(request, *args, **kwargs)

DREST-ifying Existing Serializer

As with ViewSets, there’s a mixin to DREST-ify an existing serializer. Same shenanigans warning applies as above:

class NewFooSerializer(WithDynamicModelSerializerMixin, OldFooSerializer):
    # Must override Meta class with DREST attributes
    class Meta:
        name = 'foo'
        model = Foo

Customizing Serializers

Occasionally, it is useful or necessary to customize serializers themselves. One simple way to customize how objects get serialized in DRF, is to override the to_representation() method:

class FooSerializer(DynamicModelSerializer):
    # …
    def to_representation(self, instance):
        # modify instance here
        # …

        # pass through default serializer:
        data = super(FooSerializer, self).to_representation(instance)

        # modify data (dict) here
        # ...
        return data

Computed Fields

Historically, we’ve implemented computed fields using SerializerMethodField, which led to a proliferation of one-off methods with ad hoc implementations. SerializerMethodFields are also problematic because they may not play nice with standard features (like inclusion/sideloading), and don’t have declared data types. In DREST, we introduced a DynamicComputedField base-class, to encourage developers to define and implement (or use) reusable computed fields.:

from dynamic_rest.fields import DynamicComputedField

class HasPermsField(DynamicComputedField):
    def __init__(self, required_perms, **kwargs):
        self.required_perms = required_perms
        kwargs['field_type'] = bool
        super(HasPermsField, self).__init__(**kwargs)

    def get_attribute(self, instance):
        # Override to get field value
        perm_checker = self.context['permission_checker']
        user = self.context['user']
        return perm_checker.has_perms(user, instance, self.required_perms)

    def to_representation(self, value):
        # Override if we need to convert complex data-type to a
        # primitive data type that’s serializable.
        return bool(value)

# in serializer:
class DocumentSerializer(...):
    can_write = HasPermsField('w')
    can_destroy = HasPermsField('d')

Ephemeral Serializers

Sometimes, the output objects don’t map cleanly to any existing model. One approach is to return adhoc JSON, but that makes it difficult or cumbersome to support features like dynamic sideloading/inclusion or pagination (which in turn leads to inconsistent and unpredictable implementations). DREST attempts to address these issues by providing limited support for serializers that are not backed by models.

One use-case for ephemeral serializers is when we want to represent data that is context-sensitive. Consider the earlier query:

http://localhost:9002/events/?include[]=users.&filter{users|location}=1

Note that Event objects returned by this API call only contain users whose location is 1. However, there is nothing in the object indicating that its user set is incomplete, so if that object is cached, there’s no way to know by looking at the object whether the user-set should be considered complete or not.

An alternative representation of the data might look something like this:

class EventLocationUsersSerializer(DynamicEphemeralSerializer):
    class Meta:
        name = 'event-location-users'

    id = CharField()
    user_location = DynamicRelationField('LocationSerializer')
    users = DynamicRelationField('UserSerializer', many=True)
    event = DynamicRelationField('EventSerializer')

    def to_representation(self, event):
        location = self.context['location']

        # Construct dict representing data we want.
        data = {}
        data['id'] = data['pk'] = "%s--%s" % (event.id, location.id)
        data['user_location'] = location
        data['users'] = list(event.users.all())
        data['event'] = event

        # Construct EphemeralObject instance, and let DREST serialize it.
        event_location = EphemeralObject(data)
        return super(EventLocationSerializer, self).to_representation(
            event_location
        )

This serializer will take an Event object with its users set pre-filtered, and emit an object with a unique ID and context that makes it safe for caching and re-use. If hooked up correctly to a viewset, the resulting API would have support for DREST features like field inclusion/sideloading, auto-generated OPTIONS response, and pagination.

Embedded Fields

The DynamicRelationField’s embed option will ensure that the related objects are always included, and also returned nested in the parent object. This is useful for cases where a nested response is desired for legacy reasons, and/or when the related objects should always be returned with the parent objects, and expecting the caller to always include[] those fields is burdensome.:

class BlogPostSerializer(DynamicModelSerializer):
    # ...

    author = DynamicRelationField(UserSerializer, embed=True)

Fun Times with Default Querysets

Default QuerySets on DynamicRelationField can also be used to do almost* anything QuerySets can do. The following examples are also valid:

class BlogPostSerializer(DynamicModelSerializer):

    # default sort applied
    comments = DynamicRelationField(
        CommentSerializer,
        many=True,
        queryset=Comment.objects.order_by('posted_at')
    )

    # most recent comment
    recent_comment = DynamicRelationField(
        CommentSerializer,
        source='comments',
        queryset=Comment.objects.order_by('posted_at').first()
    )

Almost anything? Yes, some things you shouldn’t do:

  • Anything that would cause the queryset to be evaluated. For instance, this will actually run the queryset when the class is loaded, which is NOT what you want:

    queryset=Foo.objects.filter(foo='bar')[:10]
  • Some queryset operations will conflict with DREST’s internal query optimization. These include (but may not be limited to):
    • only() - DREST also uses only()
    • values_list() - Will probably confuse DREST because the data returned won’t match what it’s expecting.

Dynamic Default Querysets

In the previous examples above, the default querysets are constructed when the module loads. For more dynamic filters that can be constructed at run-time, the queryset attribute can be set to a function (or a lambda):

class BlogPostSerializer(DynamicModelSerializer):

    def get_recent_comment_queryset(field, *args, **kwargs):
        # Return queryset to filter comments made in last 3 hours
        recent = datetime.now() - timedelta(hours=3)
        return Comment.objects.filter(
            posted_at__gte=recent).order_by('posted_at')

    # default sort applied
    comments = DynamicRelationField(
        CommentSerializer,
        many=True,
        queryset=get_recent_comment_queryset
    )

Notes: The function mapped to a queryset should accept one parameter, which is the field (i.e. a DynamicRelationField instance) and return a QuerySet instance. It is also possible to access the parent serializer as field.parent and the child serializer as field.serializer (e.g. in the example above, field.parent refers to a BlogPostSerializer instance, while field.serializer would be a CommentSerializer instance).

DynamicMethodField and Prefetch Hinting

DREST will try pretty hard to optimize queries, specifically by only fetching fields that are required, and by using Django’s prefetch features. In most cases, DREST will automatically do the right thing, but sometimes it doesn’t have all the information it needs to pull the right data. Specific examples include:

  • Serializer method fields - DREST doesn’t know what kind of shenanigans you’re up to in that serializer method field, and so it won’t be able to infer what data you need.
  • Computed properties in models - Basically the same problem as serializer method fields.

To address this issue, DREST fields like DynamicField and DynamicMethodField support a requires attribute that allows you to specify fields that are required. DREST will then incorporate that information in its optimization strategy:

class UserSerializer(DynamicModelSerializer):
    preferred_full_name = DynamicMethodField(
        requires=[
            'profile.preferred_first_name',
            'profile.preferred_last_name'
        ]
    )

    def get_preferred_full_name(self, user):
        return '%s %s' % (
            user.profile.preferred_first_name,
            user.profile.preferred_last_name
        )

NOTE: It is possible to specify require fields as either model fields or serializer fields. If requiring nested serializer fields, DREST will prefetch data the same way it does when those relations are included (i.e. features like default querysets are used).