Writing a Flask Extension for the First Time

Photo by Windows on Unsplash

TL;DR: I have developed a little Flask extension for DynamoDB, using PynamoDB. Here is my GitHub repository, and the documentation.

Introduction

Writing web APIs in Python is fun! I wanted to write a small API with Flask & Flask-RESTX, and my DB was DynamoDB in AWS.

Unfortunately, I am not familiar with DynamoDB at all. What am I going to do? Research for the best solution for integrating with DynamoDB! The best solution would be an ORM-like solution for DynamoDB.

After searching a little bit on Google, I have found the following solutions:
1. Boto3: a Python SDK for AWS services, including DynamoDB! It looks like Amazon maintains it. However, the client is just a simple wrapper of the API: there is no abstraction. Also, there is no integration for Flask. For instance, we cannot configure the library through the application object.
2. Flask-Dynamo. this is better since it is a Flask extension. Yet, this is a simple wrapper for boto3, so we still do not have the desired abstraction.
3. PynamoDB: a Pythonic interface for DynamoDB. It is similar to SQLAlchemy and MongoEngine for DynamoDB. However, this is not a Flask extension, so that we have the same drawbacks as boto3.

As I mentioned, there is no integration of PynamoDB and Flask yet.

So, maybe I can create this library? Honestly, I don’t know how to write a flask extension. Also, I am not so familiar with either PynamoDB or DynamoDB in general. Therefore, I will have to learn a lot to make it happen.

As a result, I have decided to make a plan:
1. Learn about PynamoDB (writing an example should assist).
2. Learn about writing a Flask Extension.
3. Write a PoC for the integration between Flask and PynamoDB.
4. Develop the library!

PynamoDB in a nutshell

PynamoDB is a Pythonic interface to Amazon’s DynamoDB. By using simple, yet powerful abstractions over the DynamoDB API, PynamoDB allows you to start developing immediately.

Every table in DynamoDB is represented by the Model class. The Model class consists of configuration and attributes. The configuration is represented by an inner class called Meta .

class Thread(Model):
    class Meta:
        table_name = 'Thread'
        region = 'us-west-1'
        write_capacity_units = 10
        read_capacity_units = 10
    forum_name = UnicodeAttribute(hash_key=True)
    subject = UnicodeAttribute(range_key=True)
    views = NumberAttribute(default=0)

Do you notice the annoying part? Yes! You have to configure the AWS configuration, like the region, to every model!

The solution of PynamoDB for this issue is to override the settings in /etc/pynamodb/global_default_settings.py, or set PYNAMODB_CONFIG environment variable for custom path file.
Also, PynamoDB supports the boto3 configuration. It makes sense since PynamoDB abstracts boto3’s client.

As far as I’m concerned, we can do better than that! We need to support configuration through app.config and pass it to all defined models.

We will come back to this problem. Yet, we should learn about writing flask extensions!

Writing a Flask Extension!

Fortunately, there is a guide inside Flask docs. Also, we can look at other extensions, and understand from there.

So here are the steps for creating an extension:

1. Create the setup.py for the extension, like every python library.
2. Create a manager class that should expose the following interface:

• __init__ : should get an optional argument of the application. If the application is provided, the extension should be initialized from the application’s config.
• init_app: should get the application as an argument and initializes the application. Usually, developers call init_app in case of initialization after the object has been created, or when using multiple applications.
• The API of the extension! Database extensions usually expose their connection as a property.

So, let’s try to create the essential structure for the extension, and then we will solve the configuration issue.

PynamoDB has a connection object for developers who use the low-level API. So it will be a good idea to add a connection property to the manager class.

PynamoDB Model’s Configuration

How do we inject for each model the base configuration? We should add the configurations as members to the Meta inner class of each model.

Option 1: Force Inheritance

We can force the developer to use inheritance with a custom base for the Meta class. We will set the members of the base class in init_app.

class BaseMeta:
    region = 'us-west-1'
    write_capacity_units = 10
    read_capacity_units = 10
class Thread(Model):
    class Meta(BaseMeta):
        table_name = 'Thread'
    
    forum_name = UnicodeAttribute(hash_key=True)
    subject = UnicodeAttribute(range_key=True)
    views = NumberAttribute(default=0)

I don’t like this solution, since the library forces to add this base class to each model. It is less annoying than before but still annoying.

Option 2: Set The Members Dynamically

We can change the behavior of the function that uses Meta and set the configuration just before its usage.

Who does use the Meta class? It turns out each model has a cached connection, created by the get_connection function.

We can override this function, and set the attributes of Meta just before creating the table connection!

Now, we do not force to have a base for the inner Meta class!

Additional Steps before Publishing

Testing

Testing should be easy: we can use pytest with Flask.testing.Client. Yet, we should mock-up DynamoDB for testing without a real database. To solve that, we can use pytest-mock, and mock up the TableConnection class.

Also, it is important to check the code coverage while testing. Code coverage is just the amount of lines that have been executed while testing. Your goal is to cover your code as much as possible while testing. However, keep in mind this is not related to the quality of testing. Therefore, even if your code coverage is 100%, it does not mean the tests are high quality tests.

Docs

It is not that interesting, but in my opinion, every published library must have good docs or at least docs.

Use docs that can be generated in a readable format. In this project, I have used Sphinx. In my opinion, the docs inside the code should be readable as well, so I chose Google docstrings format.

Linting & Formatting

Linting and code formatting helps to preserve the code clean as well as consistent.

In this case, I use pylint, flake8, for linting. In addition, I use black as a code formatter. For the future, it will be nice to have mypy and bandit as linters as well. Also, it will be great to add a CI/CD, for uploading the library to pypi at every new release, and test the code at every commit to master (at least).

Summary

It was a great experience to develop a Flask extension for the first time. I know it is not perfect, but it is pretty good. Perhaps, someone would like to use it, and even improve it! I would also improve it soon.

Here’s again my GitHub repository, and the documentation.

Thanks for reading!