Simple API Development pt 2.

Structure

You’ve got this far and there’s not a child in the house washed. Lets get practical on the things that we really need to think about to actually write some code. These are the things that we really want to think about to get from nowhere to a working skeleton in the shortest amount of time. Most of the points presented here are language and platform agnostic. When I’m developing API layers this is what I do almost every time and I need to have a very strong argument before I deviate from this approach. It may sound a bit dull and a bit unimaginative, but I like to spend my time creatively solving the problem being presented by the application rather than dedicate myself to thinking up new ways of doing common things. I’m funny that way.

Interface & Contract

Starting from the outside in, let’s review some of the points we need to consider when building the consumer facing side of our API.

Use the Standards

There are basically three roads you can take when developing a remote API:

• REST
• Web Services
• Proprietary

There are cases where a proprietary protocol is appropriate; it is often coupled with the need for a fully stateful two way communicating system. I’ve used it in games but never, ever in business applications. I’m not going to look at this approach at all here, and unless you have a really strong argument for doing so you should largely accept that this is not your world. My final word of warning on this, evoking my inner Cassandra, proprietary protocols require specific consideration to be given to scale up / out approaches, load balancing, caching, authentication and coffee making. If you need it you need it, but don’t go down that road alone.

Whew!

On the happier standards road things are much easier. The great mind of the Internet has been defining standards for many years, and that collective wisdom has allowed network hardware and systems manufacturers to build complimentary products that can accelerate, scale and strengthen your API without you writing a single line of code.

So much nicer!

REST vs. Web Services

Before we get started, I’m not going to give any oxygen to the argument that REST is a form of web service. Thank you Captain Obvious we can now move on.
Here are the key differences in my mind; you can decide which approach suits you best:

REST

REST, or Representational State Transfer, looks to overload the concepts that underpin the hypermedia approach of the web (as distinct to the internet) using them for data transfer. You can take it to mean that it uses the HTTP verbs GET, POST, PUT and DELETE to change the state of data. No specific definition for the content of a message is mandated, but JSON has emerged as the de-facto approach for content.

REST is small, light and conceptually both clear and digestible which has helped its meteoric rise in popularity.

REST offers few if any challenges for a developer, with the possible exception of the occasions where the developer tries to use it as a means to implement a Remote Procedure Call (RPC) style of interaction. The service that the REST protocol provides is specific to moving data around which makes it quite limited in supporting semantic requests.

This handicap is best explained in the form of an example. If we wanted to offer a mortgage calculator, for example, which verb would we use to get the results of the calculation? Should we POST the parameters? Or is a GET with parameters more appropriate? The issue here is that there is no state change being materialised in the request. Nothing changes with the calculation hence REST acts like that kid with the saucepan on his head running into every wall.

WS-*

The web service standards, or WS-* standards, is a vast body of work that sought to marry all of the benefits of RPC style coding with the emerging standards of the Internet. If REST was defined by your local hipster designer mate who gets all the girls, the WS-* standards were defined by an actuary who wanted to know everything before anything was done. XML is the lowest common denominator of the WS-* standards allowing a very precise definition of the nature of the services offered, the structure of the data being sent and received, the nature of the elements of the structure and more besides. The WS-* standards have evolved over the years to offer a very rich ecosystem of supporting technologies including service discovery, co-ordination and orchestration, workflow, transaction management, security and trust and encryption to name but a few (honestly).

The weighty overhead of the WS-* specifications such as SOAP and WSDL have given rise to a bad reputation for the standard among the cool kids in the mobile world, but in the enterprise world the WS-* standards are deeply relied upon to build highly robust and scalable solutions.
Web Services particularly shine in exactly the area that REST services suffer. Their underpinning in RPC allows them to support a semantically rich style of integration that goes far beyond the simple transfer of data. Our previous example of a mortgage calculation is handled very cleanly in the WS-* world by simply presenting a mortgage calculation service that offers an endpoint for performing the actual calculation.

Pros and Cons

We’ve discussed the semantic challenge that is the key difference between the two approaches, but really that informs the architecture of a solution rather than acting as a positive or negative.

There are a couple of considerations that are worth keeping in mind before progressing in one direction or the other:

Tooling

REST uses JSON for the most part, which opens a world of tools that are trivial to use for testing endpoints. There are fewer of these tools in the WS-* world, and they tend not to be free.

WS-* tools do tend to be richer in their support for defining data contracts as a result of the support that WSDL offers. By extension, the tools to ease integration into strongly typed languages such as Java and C# are mature and well developed for the same reason. REST / JSON has a long way to go in this space.

Message Size

The gorilla in the WS-* room, there’s no nice way to say it WS-* messages are big, bloated and excruciatingly impenetrable. If you’re on a nice fast, fat and reliable corporate network you might not notice this, but in mobile Internet land where sometimes data isn’t free, and where every byte is precious it’s elegance comes with a price.

Integration and Coordination

The gorilla in the REST room (ha! that was unintentionally funny), a REST service is the lone wolf of the Internet. Whether you like to call it orchestration, co-ordination, collaboration or choreography REST services are naturally happiest dancing in the corner by themselves. If you want to glue two or more together, then you’re going to be doing a lot of that coding yourself.

Think Remote

The most common mistake I see made by novice API developers is the ‘chatty protocol’. Irrespective of what approach you take, you retain control of the messages passed and the responses sent. You can exercise a fair degree of control over this, and you should but only at the appropriate time.

Be Terse

This is the easy one, but is consistently the source of the worst offences. You might be testing your system on a LAN or with a nice big juicy corporate Internet pipe to the cloud, but your user may be sitting on an underground train with poor connectivity. There are additional considerations when dealing with latency and the nature of the connection, look at satellite Internet connections for good examples of the discrepancies that users live with on upload vs. download speeds.

The bad example

The worst example of a chatty service that I’ve seen related to a customer relationship management system, names and places have been changed to protect the innocent.

In this system, the request, in this case to add a customer, was posted through the API layer for insertion in this case to a relational database. The database had been set up with an identity field that generated a unique identifier. What was good in this system was the use of stored procedures to insert data; this SPROC returned the newly minted identity for the record. The designer clearly subsequently fell asleep at the wheel because the system took this identity updated the original message and passed it back in its entirety across the wire.

Under load, you can imagine, the system ground to a halt as the system instigated a denial of service attack on itself through swamping its outbound data connections.
The solution? We simply changed the response to a HTTP 200 (OK) message.

The General Rules

1. The bulk of the content of a message should move in one direction. If you’re sending information to a service for adding / updating then as a general rule the API should keep its responses simple, ideally to a simple HTTP response code or similar.
2. Requests for information should be treated with similar diligence. The parameters to a request, whether it be an Id or a series of filters should be passed in the request in as efficient a manner as possible. The response should meet the requirements exactly and no further.
3. There is always a filter. At a minimum it should limit the number of results you will receive in a single response. Your assumption today that there will never be more than 20 results to a query will assuredly turn into 100 results at some point.
4. Clearly define the filters and don’t filter on anything that doesn’t match the definition.
5. Only ever return a list of results in summary form. Each summary must have enough reference information to subsequently retrieve the full record.
6. Only ever return a full record based on a direct, explicit request for that record.

Think Efficiently

Processing requests efficiently is a well-choreographed dance between the API and its consumer. Every request / response should either be a final complete message or have sufficient information to easily allow for the next logical request.

That all sounds a bit wordy; think about it this way: If your application is requesting a collection of some information, car parts for example, then the next logical operation, whether it is to look in detail or to purchase a single part requires the application to get the details of a single part. It makes sense then that the results returned in the collection have all of the information required to get the fully detailed record.

The REST example

REST, with its roots in hyperlinked documents is particularly good at this approach.
For example, consider a customer retrieval query response:

[
    {
        “id”: 100,
        “FirstName” : “John”,
        “LastName” : “Doe”,
        “link” : http://api.myservice.com/customer/100
    },
    {
        “id”: 101,
        “FirstName” : “Jane”,
        “LastName” : “Doe”,
        “link” : “http://api.myservice.com/customer/101”
    }
]

What you can see here is an array of summary information with each returned record having a ‘link’ field that contains a fully qualified URL to the detailed record. No guess work, no possibility for errors.

This approach has the added advantage that no further intermediate calls are required to get to the next step. All of the information you need is embedded in the response.

Think Coherently

This is the big ask, it’s like saying think better! Each message should have a logical home for the information being communicated. There are a few rules on the structure of messages that I follow, once again others may have other ideas.

The Rules of Message Structure

1. Only have data in the body of the message. This relates to the data that is completely under your control. SOAP uses a mark-up for its messages that you can influence but not necessarily control. Leaving aside these protocol exceptions, you should not include ancillary or metadata in your message body.
2. Only have metadata in the message header. The corollary of rule 1, metadata belongs in the message header not the body.
   The simplest example relates to authentication information. If you want to restrict access to the API to registered, authorised users then there is a need to implement an authentication mechanism. Authentication / authorisation and security as a general rule are unrelated to the actual message contents and as such have no place in the body of your messages. Use the headers.
3. Assume that if you introduce a custom header that the Internet will ignore it. If you need a particular behaviour to take place, for example routing of your message, you should find the correct, standard header for that information and understand the demanded format. You might have all sorts of creative and interesting ways to solve the problem, but the multitude of the devices on the Internet through which your precious little message parcel will work through will happily ignore you with no warning.
4. Use the correct content types and or MIME types when appropriate. There’s no point in declaring a content type of ‘application/pdf’ and then passing XML in your message. Vast sections of the Internet will at best quietly ignore your entreatments whilst silently failing. At worst it might try to actually deal with the message logically.

Think Securely

Your API is a security hole through which you will be attacked. If you don’t believe this to be true, stop writing your API immediately and get a job in a bar somewhere. If you are on the public Internet then your API will be discovered and someone will try and exploit it to cough up its data.

You can’t eliminate the threat, you can merely minimise the range of attacks that can be launched, and work to mitigate the risk of unauthorised access. Whilst all of my previous missives have admonished that they are not exhaustive or definitive this is doubly or more true for security. If you’re really serious about security you must familiarise yourself with a vast body of knowledge, starting with the OWASP pages. If you’re taking card payments you’ll need to be familiar with the PCI standards. You should also expect to be spending quite a bit of time looking at hardening articles for the particular platform that’s running your pet API.

The Very Basic Rules of Security

1. Verify everything. If you are being sent a message comb over every aspect for invalid characters or input of any kind. You should know exactly what is permitted for storage. This extends to foreign languages particularly. IF you can’t define rules then you shouldn’t allow that language for input. Remember there is increasing legislation being passed that holds the service provider responsible for the content being presented as a result of that service. As an API provider, this means you.
2. Clean all parameters. You can’t afford to blindly apply filters to a request as they may contain malicious code that initiates a cross-site scripting attack. Eliminate all unnecessary or nefarious characters before using the filter. Everything could be toxic from a security standpoint so you need to develop a healthy degree of paranoia.
3. No open ended or wildcard queries. This should be an obvious one now, both from an efficiency and scalability perspective and from a security perspective. Return only what you are asked for, and if you cant put a boundary around the results, then the query you are allowing is too wide.
4. Don’t bleed information. Your remote API is hosted in some way, either on a web server, or in its own container. Check to ensure that the container isn’t bleeding information on what it is, what version number it is and what underlying platform hosts the server. All of this information is gold to an attacker who is seeking to mount an attack.
5. Use SSL. This should also be obvious, particularly on mobile systems where many will not allow connections to insecure endpoints.
6. Have a patch plan. Weaknesses are being discovered almost daily. No operating system is safe irrespective of the marketing rhetoric. Patch your host regularly and scan it for viruses at least daily.
7. Change the Default Passwords
   Really? You need to be told this one?

Assume Success

Funnily enough, most developers are a little blindsided when their API gains any kind of traction. Success is its own reward as the saying goes, but nobody wants to talk about the baggage it brings. A little work now laying down controls for the future will save hours, days even weeks of effort in the future. There’s really only one thing that you can predictably do to protect yourself from the future: Version your API. For me this an be a two-stage process:

1. Embed the version number of the API in the URI for the service.
   For example rather than declaring an endpoint that accepts a POST of a customer as:
   https://api.myapi.com/cusomer
   use
   https://api.myapi.com/v1/cutomer

2. Occasionally, indeed increasingly, I embed the version number that the client is using in the request header. The reason that I don’t consistently enforce this approach is simply due to the fact that I’m not going to be the only consumer of my API, so I can’t guarantee that this rule will be followed without requiring extra effort from the consumer. This extra effort make the difference between a developer adopting your API and not.

Simple API Development pt 3.