Thomas' Blog

Writing a good API, and being a good client

Thomas Clowes -

When rebuilding the search functionality of a product that I am working on I managed to quite effortlessly completely break the mobile applications client integrations with the API.

The main issue was that I was entering some unchartered territory, and trying to do some pretty complex things. I was in a programming 'zone' and all I wanted to do was write the new functionality. I managed to do this without issue. The problem was that having written the new endpoints I was aware that I had almost certainly changed something that would cause 'less than positive user experiences' for users of older app versions.

At least I was aware.. right? ;)

On the API end I had made a two changes which I suspected would cause issues.

  • I had changed the name of the GET parameter for our search endpoint from name to searchTerm (for clarity).
  • I had implemented some complex logic such that location search results were returned from the Google Places API (they had not been previously).

The former (in principle) has a simple fix - the API endpoint should look for a GET property named searchTerm, and if it cannot find one it should search for a property named name.

Whilst this is in fact the resolution I went for, careful consideration should be given. I absolutely do not want my API codebase to become unmaintainable and complex as a result of lots of 'fixes' over time. On careful consideration I decided that this was a one off, and that in future I would pay more attention to choosing semantically accurate property names from the get go.

This did however make me think about alternatives, one of which was an appropriate API versioning methodology.

The problem with clients

The problem with API clients is that as an API provider, you cannot be certain as to if or when your clients will update their implementations. If one could guarantee instant updates to clients when breaking changes were pushed then there would be no issue.

Whilst the API is not currently used by external clients, it is used by our own Android and iOS apps.

The iOS review process means that it is essentially impossible to have an anywhere near instant update to the client integration. Even if it were, this does not actually guarantee that users of the application would actually update it.

As such the API needs to support all historical client integrations.

Initial testing

Out of intrigue I loaded up a previous version of the iOS application and had a play. I executed a search from the main screen of the application, and I was unfortunately met with a crash. Fortunately this was the only crash that I encountered during my hands-on integration tests and as such my fear of breaking changes was limited to a singular breaking change.

Having investigated the Crashlytics report it became apparent that I had been a fool. It was not an API breaking change that was the issue, but rather the client wasn't prepared for potential changes to the API - the client was badly written.

The cause of the crash was a simple one. The search callback was trying to handle a result type of google-location - a type of which it was not aware (it was not defined in the enumeration).

After resolving the issues (see below) I went ahead and refactored the codebase such that it gracefully handles any unexpected situations like this that result from further development of the API. I.E I wrote the client how I should have initially.

API versioning

Given the nature of the issue, API versioning was now a requirement. My client would not work 100% with the new version of the API and given that an instant client update was not possible I needed to implement versioning such that 'old' versions of the apps would not receive results with type google-location. It was not however simply the case that I didn't want to return results of type google-location, but rather that I wanted to return the same API response as when the client version had been designed.

As I was implementing API versioning retrospectively (less than ideal) it was necessary for any API request without a specified version to be considered to be using API version 0 (the legacy API). Any updated app versions would need to explicitly specify the API version that they wanted to use.

This differs from 'normal' implementations where a lack of specificity is indicative of wanting to use the most recent version. It is my intention that in time (once it is apparent that no-one is using the 'legacy' versions) we will transition to the 'normal' approach.

Implementation

Here, there are a number of fantastic (and in depth) answers to best practices when versioning an API.

Having read this, I decided to opt for passing version and response format information through an "Accept" header sent with each request.

The API (which is built on top of the PhalconPHP framwork) reads the header information, extracts the important information (using a regex - see below) and makes it available to API methods by proxy of a static helper class.

  • #application/vnd.app(?:\.v(?<version>[0-9]*))?\+(?<format>json|xml)#

So now the API knows which response version is expected from a particular endpoint, but we still need to implement the logic which guarantees that the correct response is returned.

I implemented this in an interesting way utilising PHPs magic methods (specifically __call).

  • The basic logic is that a versioned API method is renamed. You can use any format you want, but I chose to simply prepend an underscore. As such the method signature search($inputModel) became _search($inputModel).

  • I then implemented the magic method __call as follows:

    public function __call($method, $arguments) {

        $realMethodName = "_" . $method;

        if (method_exists($this, $realMethodName)) {

            $response = $this->callOldVersion($arguments);

            if ($response) {

                return $response;
            }

            return call_user_func_array(array($this, $realMethodName), $arguments); 
        }
    }

  • The endpoint attempts to execute the search method. As this method does not exist, __call is triggered (this is important - __call is only called for unknown methods). The __call function then searches for a method called _search, and if it exists (it does) it executes a method called callOldVersion.

  • The logic within callOldVersion checks as to whether the requested version of the API differs to the current version of the API. If it does it attempts to call the legacy implementation of the method.

    public function callOldVersion($argumentsList) {

        //If they have requested a NON current API version
        if (APIHelpers::$API_VERSION != APIHelpers::CURRENT_API_VERSION) {

            //get the name of the method that they have called
            $actionName = $this->dispatcher->getActionName();

            //build the name of the legacy action method
            $legacyActionName = $actionName . APIHelpers::$API_VERSION;

            //if there is a legacy version..
            if (method_exists($this->oldApiService, $legacyActionName)) {

                //iterate through previous versions, and convert the arguments
                for ($toVersion = APIHelpers::$API_VERSION;  $toVersion < APIHelpers::CURRENT_API_VERSION; $toVersion++) {

                    $fromVersion = $toVersion + 1;
                    //echo "Convert from " . $fromVersion . " to " . $toVersion;

                    //build the convertor method name
                    $convertorName = $actionName . $fromVersion . "to" . $toVersion;

                    //if a convertor with that name exists
                    if (method_exists("ArgumentConvertors", $convertorName)) {

                        //convert the arguments
                        $argumentsList = call_user_func_array(array("ArgumentConvertors", $convertorName), $argumentsList);
                    }

                    //otherwise the legacy version must take the same parameters
                }

                $response = call_user_func_array(array($this->oldApiService, $legacyActionName), $argumentsList);
            
                return $response;
            }

            //indicate that there was no legacy handler so we should just call the normal method
            return false;
        }
    }

Notes

  • The ArgumentConvertors helper class allows for any arguments to the current endpoint method to be converted to the required arguments for the legacy endpoint method

  • If there is no legacy method implementation the fallback is to simply call the current version.

Bloated codebase

My main concern with such an implementation (and perhaps yours?) was that if (over time) lots of endpoint implementations are being changed then oldApiService might become bloated with old implementations of legacy endpoints.

Sadly this is exactly the case.. but logically to return different responses for different version of the API.. you need different code.

In reality however this is not a significant issue. The codebase is fully covered with unit tests. As such, when a current endpoint method becomes a legacy endpoint method we simply transfer the unit tests into oldApiServiceTests, and write new ones for the new implementation.
Complete test coverage guarantees that legacy and current API method calls return the expected responses.. indefinitely.

Furthermore (as alluded to previously), by keeping track of what versions of your API are being utilised by clients, and by implementing staged phase-outs (deprecation) of historical API versions you can keep your oldApiService pruned.

Conclusion

This post outlines a number of interesting things.

  • You should be vigilant when building API consuming clients. Make sure you make them safe.
  • APIs need to support returning appropriate responses to clients for as long as they are requested.
  • Retrospective API versioning can be done, but it is advisable to 'do it right' prior to an initial release.

As a result of this 'learning experience' the codebase for the applications have been improved significantly. The apps are also now more 'future proof' - they are resistant to issues associated with external change.

If you would like further clarification on any of this, do not hesitate to ask - I would be happy to help.