Integration best practices – Service robustness

My previous blog post on integration best practices gave some general thoughts on service development. When designing services one of the most important parts is resilience of your services and make sure that failure of one services doesn’t collapse your entire world.
There are several key features of a service which combined determine the fault tolerance of your service. In this blogpost I will explain which features contribute to creating better services.  Continue reading

Een software audit: Wanneer en waarom?

Soms kan het voorkomen dat er, als gevolg van hooggespannen verwachtingen of veranderende marktomstandigheden, noodzaak ontstaat om eens kritisch te kijken naar de staat van een softwareproduct. Het is dan raadzaam om analytisch en objectief te kijken naar zaken als de codekwaliteit, kwaliteit van de architectuur, schaalbaarheid of het softwareontwikkelproces.  Continue reading

Get more out of Bamboo with Docker – Part 1

If you search in Google for Docker and Bamboo and skip the Atlassian links, then you will find numerous links on how to create a Docker container for a Bamboo remote agent. Nice to know but what is the advantage of doing that? A remote agent that can be started and destroyed quickly? But you are still limited to the number of remote agents allowed. In my opinion, there is a far more powerful way of using Docker with Bamboo.

Continue reading

Using Pedestal with Trapperkeeper

Intro

I am currently working on Dash, developed with Clojure. For this project I recently started using two libraries together: Pedestal and Trapperkeeper. Clojure enables me to pick and combine the libraries I see fit, which is one of the reasons I enjoy writing Clojure. But figuring out how to combine libraries can sometimes be tricky, I had to fiddle a bit before I had this working. Hopefully this blogpost will save someone else some time.

Continue reading

Java LDAP connection pooling

One of our software platforms uses LDAP to manage users and organisations. All application servers and webservers connect to the LDAP servers to verify user credentials and such. Unfortunately, after migrating to a new datacenter with new hardware, we discovered some errors in our logging from time to time:

javax.naming.ServiceUnavailableException: ldaploadbalancer:389; 
socket closed; remaining name '...'
 at com.sun.jndi.ldap.Connection.readReply(Connection.java:454)
 at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639)
 at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562)
 at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)

Continue reading

External Resources Are Unreliable

Developing software is one thing, keeping it up and running in production is a whole different challenge. When your software is running in production it will be used in ways you never thought about, and it will behave in ways you’ve never expected it to behave. A common culprit of problems in production is the use of external resources.

What are external resources? Everything that happens outside of the safety of your runtime environment, e.g.:

  • Database connections
  • Consumed webservices (whether it’s REST or SOAP)
  • Inter-process communication
  • Network access in general

Treat external resources as slow and unreliable: it should return well formatted XML within 2 seconds, but it might decide to return the latest The Avengers sequel at 50kb/s which takes 5 hours to receive. It seems like an unrealistic scenario but sooner or later, it will happen. To prove this point, a few examples I’ve seen through the years:

  • Time-out on a database connection (network problem)
  • Broken NFS-mount (after 30 minutes) thanks to a misconfigured firewall
  • HTTP contract is not honoured:
    • Content-Type says ‘application/json’ but it returns plain HTML
    • Content-Length is incorrect leading to half-received responses
  • Incorrect XML entities: Microsoft Office generates invalid XML entities in PDF metadata leading to unparsable XML
  • An external webservice responding with 10kb/s leading to time-outs and rolled back database transaction on our end
  • An external webservice honouring only the second (even) request
  • An external webservice crashing on parallel requests (we had to synchronise access on our end)
  • A VPN shutting down every night after a few hours of inactivity

There are some grey areas like file access; a local disk could be fine. But hey, it might be an NFS-mount (Network File System) which makes it unreliable since again, networks are unreliable (even though Wikipedia says TCP provides ‘reliable transport’, but we know better). In the end it pays to be protective; don’t trust stuff from the outside.

Much of this awareness has been acquired through my personal bible of software development; Release It!, by Michael T. Nygard. Anyone writing software which is meant to go to production (which probably most of us do) should have read it.

Static dynamic webpages with Baked.JS

Static dynamic webpages, seems like a contradiction? You want to enable non-techies to make changes to the pages of the website, but you don’t want to go through all the problems you’re going to have by implementing a CMS, and for various reasons you haven’t created your website in something like WordPress. Baked.JS to the rescue! In this blog I will explain what it is, how we use it, and how you can profit from this awesome technology too.
Continue reading