Developing software is one thing, keeping it up and running in production is a whole different challenge. When your software is running in production it will be used in ways you never thought about, and it will behave in ways you’ve never expected it to behave. A common culprit of problems in production is the use of external resources.
What are external resources? Everything that happens outside of the safety of your runtime environment, e.g.:
- Database connections
- Consumed webservices (whether it’s REST or SOAP)
- Inter-process communication
- Network access in general
Treat external resources as slow and unreliable: it should return well formatted XML within 2 seconds, but it might decide to return the latest The Avengers sequel at 50kb/s which takes 5 hours to receive. It seems like an unrealistic scenario but sooner or later, it will happen. To prove this point, a few examples I’ve seen through the years:
- Time-out on a database connection (network problem)
- Broken NFS-mount (after 30 minutes) thanks to a misconfigured firewall
- HTTP contract is not honoured:
- Content-Type says ‘application/json’ but it returns plain HTML
- Content-Length is incorrect leading to half-received responses
- Incorrect XML entities: Microsoft Office generates invalid XML entities in PDF metadata leading to unparsable XML
- An external webservice responding with 10kb/s leading to time-outs and rolled back database transaction on our end
- An external webservice honouring only the second (even) request
- An external webservice crashing on parallel requests (we had to synchronise access on our end)
- A VPN shutting down every night after a few hours of inactivity
There are some grey areas like file access; a local disk could be fine. But hey, it might be an NFS-mount (Network File System) which makes it unreliable since again, networks are unreliable (even though Wikipedia says TCP provides ‘reliable transport’, but we know better). In the end it pays to be protective; don’t trust stuff from the outside.
Much of this awareness has been acquired through my personal bible of software development; Release It!, by Michael T. Nygard. Anyone writing software which is meant to go to production (which probably most of us do) should have read it.