This is a blog post I originally wrote more than two years ago, in reaction to “spirited debates” I was having with developers. I didn’t post it, but perhaps I should have! Anyway the ideas within are as true to me now as they were then, so I thought I’d post it today after a bit of revision.

How many times have you had a developer shrug their shoulders at you and say “it’s an upstream bug”?

I heard it today, and it is so, so wrong, that it is practically an admission of guilt.

Do you say that to your customers when their personal data is leaked from your database? When your app crashes their device? No? Good, because it’s your problem.

It’s great that you can use third party libraries to do your job more efficiently, but doing so does not absolve you of responsibility if the product breaks. You made the decision on what library to use, and you are ultimately responsible for delivering functionality.

I first saw this idea articulated on Jeff Atwood’s Coding Horror blog, which is based on this excellent post on Hackernews:

“When you run a business, if your software has a bug, your customers don’t care if it is your fault or Linus’ or some random Rails developer’s. They care that your software is bugged. Everyone’s software becomes my software because all of their bugs are my bugs. When something goes wrong, you need to seek out what is broken, and you need to fix it. […]

True hackers have come to terms with a simple fact: If it runs on my machine, it’s my software. I’m responsible for it. I must understand it. Building from source is the rule and not an exception. I must control my environment and I must control my dependencies.”

Limitations

It’s only fair to state the limitations of this idea. We don’t expect front-end JavaScript developers to be able to write web browsers any more than we expect back-end Java developers to write operating systems. We don’t expect developers to understand everything that runs on their laptop, because in reality, no one does.

But we do expect them to be able to pinpoint and work around or fix any problems in software they depend on.

It would be awesome if all front-end developers could write web browsers and back-end developers could write operating systems, because they’d have a better idea of how to write code for the layers beneath them. The very best ones can. But in reality, this is only a very small percentage, and you probably don’t work with any of them.

Still your responsibility

It’s relatively seldom that specific bugs in a product are caused far from where they’re observed. For example, web browser bugs are going to be widely felt, and if this “web browser bug” is only being felt by users of your product, your developer must be doing something esoteric. If they’re unable to diagnose it or fix it upstream, they’re out of their depth and should stick to simpler solutions.

As you go deeper into the stack, from your own code down through React, JavaScript, the browser, C++ libraries, the C compiler and the operating system, there are more and more users of the code and thus a lot more scrutiny of its suitability.

That is, as long as you don’t stray too far from the beaten path. I doubt that a math library written in Dylan and running on Haiku is going to have as much low-level scrutiny as Java.lang.Math on RedHat Linux. Which is why you should be wary of developers that are always “bleeding edge” and chasing the next shiny new language or framework.

Lower level bugs do happen, and they happen often, but they are much less likely to explain a given problem than a bug in your developer’s code, which has probably only been tested by that QA guy drumming with his headphones on a couple of desks away, and the few end-users that actually bothered to update their profile.

But this is really beside the point, because regardless of where the bug lies, the responsibility is always with your developers.

Read the source

I hope you haven’t heard it as an excuse, but if you ever do hear a developer try to explain that they can’t fix a bug because it lies upstream, and the source code is open, tell them to fork the library, send a pull request, work around it or monkey-patch it at run-time.

Then send them to Coding Horror. And make a mental note to never rely on anything they write.

And if the source is closed? Cut your losses and run.