Write a RESTful API Integration in Python

These days, it becomes popular to use web APIs to connect to the cloud, integrate a particular service inside your application, or automating a process based on information from another platform. Essentially, API is everywhere! But how can we use an API by ourselves?

Read the Docs!

Before jumping into code, we have to understand how we should interact with the API. Let’s read the docs!

General Details

In case we need to choose an API from a bunch of services, it might be a good idea to check the following:

1. HTTP or HTTPS: prefer HTTPS — it is SECURED. When using HTTP, the traffic is not encrypted, thus exposed to attacks.
2. Pricing: you should check the API pricing before jumping into the API docs. There might be better free APIs, while there might be a good paid API.
3. Rate Limit: the number of requests you can send at an amount of time. There may be another API with a higher rate limit, in case you need more requests per second.

Authentication

The way you prove you are some user, to obtain the relevant data or accomplish an action.

There are several authentication methods. Some of which are the following:

1. Basic Authentication: using your username and password for authenticating against the API. Usually, the credentials are encoded by base64 in the Authorization header, i.e., Authorization: Basic <base64 of username and password>.
2. Cookie: several APIs also support using a cookie for authentication. Usually, the cookie appears in the Cookie header.
3. API token: usually, tokens are generated by the user inside a configuration page. Some APIs may add roles/permissions to the API token so that the token will not be accessible to all service’s resources. Every API specifies how to pass the token. For example, it could be as a query parameter in the URL, inside the Authorization header with custom type, a custom-header, etc.
4. OAuth: OAuth is an authorization protocol, mainly for an application to obtain access to a web service, using authorization flows. There are two versions of the protocol (1.0 and 2.0). OAuth 2.0 is a complete rewrite of OAuth 1.0 from the ground up. OAuth 1.0 is deprecated right now (or at least should be deprecated), although some services still support it, like Twitter.

Note: some APIs uses Basic Authentication for passing their API token.

Let’s Develop!

To make it simple, we will see how to create an integration through an example. In our case, we will use the GitHub REST API. We want to obtain all details about the public repositories of the authenticated user.

Note: I know there are existing libraries for GitHub’s API. However, it is just an example of learning about API integrations. For non-educational purposes, I would suggest going with a good library, if it does exist. Yet, you should ensure the library is maintainable and meets your needs.

According to the API docs, we can authenticate via our username and password, a personal access token, or an OAuth application. For our purposes, we will use the access token.

Step 1: Using requests

requests is an elegant Python library for executing HTTP requests.

Let’s use it for our API! The library is very intuitive, so for GET request, use requests.get, for POST use requests.post, etc.

All these functions are essentially wrappers to the request function that receives the following parameters:

• method: the HTTP method to use (GET, POST, PATCH, PUT, DELETE, etc).
• url
• params: the query string parameters (the parameters in the URL).
• data / json: the body of the HTTP request.
• headers: for the HTTP request.
• auth: auth tuple to enable Basic / Digest / custom HTTP authentication.
• Many more! The full list.

Note: actually, the request function accepts **kwargs and forwards it to the Session.request method.

Now, we can write a function that returns all public repositories. According to the docs, we should request it via a GET request to /user/repos endpoint. For public repositories, we have to pass the visibility parameter and set it to public.

Is this code good? I think we can do better than that.

Step 2: Code Improvements

Let’s improve our code! We can notice multiple problems:

Errors & Exceptions Handling

When requesting an HTTP request, things can fail:

• There are networking issues, so we cannot connect to the server.
• The authentication is invalid.
• The data is invalid (for example, the visibility is a random string).

We want to catch that and handle it. A simple solution would be catching the exceptions and raising a known exception (or exceptions). That way, we create a solid interface for our API.

For invalid requests, we can raise an exception when the status code is invalid, using raise_for_status method. The method raises HTTPError when the status code is above or equals to 400.

It is not necessary here, but we can catch JSONDecodeError, in case the API might return a non-JSON response (and it is invalid).

Do Not Repeat Yourself

Let’s say we want to get information about a user. We can use /users/{user} endpoint, and do the same thing as before

We can see we repeat our code. It is just a copy-paste of the original function, with little changes for the current endpoint.

A simple solution will be to create a generic get function for the API.

If we want to do this for multiple methods, such as POST, we might write a generic request function, and write a wrapper for the desired methods (just like requests!).

This code is better than before, but it’s still not that good. What if we want to authenticate from multiple users? The code is not scalable.

Step 3: Writing a client

We can write a class representing an API client. This class should store the unique data of the client, such as the credentials, essential headers, and more. Now, we can create multiple clients!

In our case, the client should receive the following arguments:

• username and token: for the authentication.
• headers: should be used at every request.

We can save this data inside the requests.Session object:

The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance and will use urllib3’s connection pooling. So if you’re making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase.

We don’t need to merge the headers inside get. Also, the performance will be better, thanks to the connection pooling.

Step 4: Improving the client

We can improve the client’s implementation!

First of all, we do not close the Session object. Keeping a resource opened after its usage is a bad practice. We can add a method for closing the session. Yet, a better practice is supporting the context-manger protocol, using __enter__ and __exit__ magic methods.

We might also want to support changing the credentials. A convenient interface may be properties for the username and the token.

Step 5: Writing a generic client

In case you have to develop multiple clients, such as a client for each service of your product, you should consider writing a generic REST client. Thus, you make sure you don’t implement the HTTP logic multiple times.

Summary

Writing a good API integration is not that hard, but takes time for understanding the best solution for your situation. Writing a client object might be an overhead sometimes, whereas it may save time in the future.

The final example is available on GitHub!

The code looks fine right now. Yet, we can improve it more!

• Logging: consider adding logging when executing an HTTP request. Also, log when some error occurs. It will save time when an error occurs in your application that uses the client.
• Validation: add validation for your parameters, such as a validation for the base URL in the REST client, as well as the credentials in the GitHub client. Moreover, consider adding a method for testing the API before actually using it.
• Tests: like every piece of software you develop, you want to ensure the client behaves properly.