Why Testing Is Important for Distributed Software

Testing is especially important in modern distributed software systems. Learn more at the upcoming APIStrat conference.

As developers, we often hear that tests are important. Automated testing minimizes the number of bugs released to production, helps prevent regression, improves code quality, supplements documentation, and makes code reviews easier. In short, tests save businesses money by increasing system uptime and keeping developers working on new features instead of fighting fires. While software testing has been around for about as long as software has, I would argue that testing is especially important (and unfortunately more challenging) in modern distributed software systems.

“Distributed software” refers to any software that is not run entirely on one computer. Almost all web applications are distributed software as they rely on applications on other servers (eg: remote data stores, internal REST APIs, third-party APIs, content delivery networks, etc.), and most mobile and desktop applications are as well. Distributed software presents new challenges and requires a thoughtful approach to testing. This list includes just some of the reasons that testing is crucial for distributed software:

1. Third Party APIs Can Change Unexpectedly

We would like to think that every REST API we use will adhere to some form of versioning, but this doesn’t always happen. Sometimes APIs break when a maintainer fixes a bug, sometimes breaking changes are overlooked, and sometimes the API just isn’t mature or stable yet. With more companies releasing public APIs, we’re bound to see the number of accidentally breaking releases rise, and tests are a great way to prevent those breaking changes from affecting our applications.

2. Internal API Changes can Affect Your App in Unexpected Ways

Even more commonly, breaking API changes come from within our own organization. For the past few years, I’ve been working with startups where the business requirements change almost as fast as we can implement them, so our internal APIs are rarely stable and sometimes the documentation gets outdated. Slowing down, improving communication between team-members, and writing tests for our internal APIs has helped.

3. Remotely Distributed Open Source Packages are More Popular Than Ever

78% of companies are now running on some form of open source software. This has helped the speed and ease of developing software to increase exponentially, but blindly relying on open source packages has bitten plenty of developers as well (see the left-pad incident of 2016). Once again, we hope that open source packages use semantic versioning, but it’s impossible to guarantee this. Testing the boundaries between packages and our software is one way to help improve reliability.

4. Network Connections Aren’t Perfect

In many server-to-server cases, network connections are pretty reliable, but when you start serving up data to a browser or mobile client via an API, it gets much harder to guarantee a connection. In either case, you should have a plan for failure: Does your app break? Throw an error? Retry gracefully? Adding tests that simulate a bad network connection can be a huge help in minimizing poor user experiences or data loss.

5. Siloed Teams can Lead to Communication Gaps

One of the advantages to distributed systems is that a team can be assigned to each component. This allows each team to become an expert on just one part of the system, enabling the scaling of software organizations like we see at Amazon. The downside to these siloed teams is that communication becomes more difficult, but a good test suite, thorough documentation, and self-documenting APIs can help minimize these gaps.

How Do We Test Distributed Systems?

Distributed software has become more popular as the cost of cloud computing has gone down and network connections have become more reliable. While distributed systems offer unique advantages for scaling and cost savings, they introduce new challenges for testing.

Borrowing from some of Martin Fowler’s ideas on testing microservices and my own experience building layered test plans, I’ll be presenting a strategy for testing distributed systems at this year’s API Strategy & Practice Conference. If you’re interested in learning more about the topic of testing distributed software, or you have questions, you can find me at the conference, or anytime on Twitter.

Learn more at APIStrategy and Practice Conference.