Background
Let's say; When you first are introduced to a new problem space your presented with a wide open beautiful green field project. Anything you can imagine, the best techniques, the best technology, you want this product to shine! But the timelines demand a more pragmatic approach; you get the project up and running, stable, performant, stake holders are happy! But your not, the project has flaws. Flaws that may or may not be visible to your stake holders, remember you got this system working and everything is running smooth, why fix what ain't broke? Well as a craftsman, taking pride in the product you delivered, you see every minor flaw whether it's stake holder apparent or not. You know it could be better in so many ways. What to do? Version the api? Yes, but how you version it matters, so do it responsibly!
Challenge
Versioning an api in production is not an easy task. It takes careful planning and consideration of all your stake holder's needs. Consider: api contracts, database schemas, message contracts etc. all these items will have to be backward compatible
and interoperable. We'll go through the process I conducted to version one of our api's and show how a touch of emotional intelligence can lead to a more rewarding experience for both the developer and the stake holders.
Solution
First step: Gather usage data
When you start to think about all the shiny new ways you can improve your first draft, version 1, you can get lost and out of touch with how the users actually use the product that's been built and, with any luck, they're happy with the product as is. Usage data will tell you what users are doing and what matters most to your stake holders. Are database columns being ignored? Features not utilized? Response codes inconvenient?
Talk to other engineers that use your product, get their real world feedback, develop a rapport with the teams that engage with your product. This will go a long way in telling you what to prioritize when developing the new feature set.
Analyze What Users Are Doing Most
This information will come primarily from logs and analytics already present in your first version. What endpoints get the most traffic? What response codes cause noise in your analytics. Where are the holes in your data? All this pre-work and research is vital to understanding what your stake holders need and if you've built that rapport with the teams utilizing your service, you'll know what matters to them. What are their pain points? What can you do make integration easier? How can you fill holes in data without comprising ease of usage?
Document Your Plan
One thing I missed when I set out to version our api was notifying stake holders changes were coming. This step is critical and this is where emotional intelligence comes in to play. Every one of these other teams has a backlog and every team is already packed with feature requests and tech debt, nobody is waiting around for your shiny new version to come along. With this in mind, make it worth it for the team to get your migration on their backlog. The start to this has already been discussed, don't solve the problems you think are important, solve the problems your stake holders have, identifying their pain points will lead to solutions. A solid document describing every change and migration path is critical. When a team is integrating with your api remember they have already put in the labor and opportunity cost to integrate with the current version. A document describing the ease of which migration is possible to solve their pain points is critical. Describe the problem your solving for them and describe the solution, provide sample requests and payloads that can be copy/paste into their existing infrastructure.
Pain Points Identified
Through your research you have come to realize the pain points and integration struggles teams have gone through to integrate their systems with yours. Some of the biggest pain points I found were: we needed a PATCH endpoint, we needed more data in our event messages and we needed empty responses instead of hard 404's from query endpoints. These insights did not come from my imagination; they came from talking to engineers who worked with the system and struggled to match their process with ours.
Notify Stake Holders
Another mistake I made. Once you have a document describing the solutions to consumer pain points; don't hesitate to get the word out. You're trying to make things smoother, easier and more performant. But remember the work to integrate with your new version comes at an opportunity cost. Getting that integration work on a backlog, however easy you think you've made it, comes at the cost of other features and work the team could be doing. Always remember, everyone, everyone, is already packed with work AND YOU'RE ADDING TO IT. So get the word out early, give teams time to to get ready for the planned changes. If you have that migration doc in hand to provide, all the better, let teams digest and provide feedback. What problems did you solve? What problems did you overlook? That rapport and integration with your stake holders will inform your next steps.
Implement Version Next
Once the migration doc is circulated and feedback received, execute exactly as planned with little deviation. Your stake holders are now depending on what you've planned and their own planning process has already been started, they're integrating what work needs to be done in their backlog, they're prioritizing based on the benefit the next version will provide. Remember we're working for your stake holders, this not your pet project, we're providing value to the teams that consume the service.
When implementing a new version, know that you're now committing to supporting two parallel versions of an api that may have radically different inputs/outputs/schemas etc. Make sure you don't break what's already available. If a feature exists in the current version don't break it! Migration is a choice and that choice is not yours, that choice is based on the value you provide and the documentation that gets teams from version first to version next.
Messaging Contracts
If your working with a distributed system, you're probably sending/receiving messages from a variety of sources. The format of the messages and their content can seem outdated and misinformed on second look. But remember teams are counting on that contract, again, don't break it, provide value. You provide value by identifying what's missing. One big change we made was adding more data into our standard event message, this allows consumers to forgo an additional api call by providing them with the latest data right in the message. But the contract can only expand, teams are counting on the contract you've already published, don't break it just because it could be better. That mentality doesn't help stake holders, it puts an immediate burden on their workload to convert to your new contract. This is highly inappropriate, maintain backward compatibility for as long as consumers are dependent on that contract. Add metrics and analytics to track usage of the old features, watch that usage decline and then version your contract if your removing items. Don't forget at that stage to get the word out, let teams know old feature usage has ceased and will be removed in the next release. This opens the door for even more valuable feedback and sets you up for success.
Get The New Version in Production
Steps: Research, Feedback, More Research, Implementation. Finally you're ready to hit production. Hopefully you've learned form my mistakes and got the word out that new changes are coming. If you've followed these guidelines going to production will be stress free; you didn't break any existing contract, you made migration a choice based on value and you committed to maintaining multiple versions of your system for as long as the original implementation provides enough value that users keep using it. Now all that's left is hit deploy and announce with enthusiasm that the pain points have been remediated. Bottom line you're here to solve problems, if that's the goal of version next then you're on the right track.
Summary
We've covered my mistakes and some guidelines for smooth delivery of new features to a service. Following this simple outline will ease any transition from one version to the next, remember you can't throw the old version away, it's still providing value and you have committed to supporting both versions. If you've provided value, teams will integrate with your changes when their backlog abides the integration.
