Flexibility is software's miracle, and its curse.
“When we had electromechanical systems, we used to be able to test them exhaustively,” says Nancy Leveson, a professor of aeronautics and astronautics at the Massachusetts Institute of Technology who has been studying software safety for 35 years. She became known for her report on the Therac-25, a radiation-therapy machine that killed six patients because of a software error. “We used to be able to think through all the things it could do, all the states it could get into.” The electromechanical interlockings that controlled train movements at railroad crossings, for instance, only had so many configurations; a few sheets of paper could describe the whole system, and you could run physical trains against each configuration to see how it would behave. Once you’d built and tested it, you knew exactly what you were dealing with.
Software is different. Just by editing the text in a file somewhere, the same hunk of silicon can become an autopilot or an inventory-control system. This flexibility is software’s miracle, and its curse. Because it can be changed cheaply, software is constantly changed; and because it’s unmoored from anything physical — a program that is a thousand times more complex than another takes up the same actual space — it tends to grow without bound. “The problem,” Leveson wrote in a book, “is that we are attempting to build systems that are beyond our ability to intellectually manage.”
— James Somers (2017, September 27). The Coming Software Apocalypse.
This is a great article form The Atlantic about the flaws of software.
The software did exactly what it was told to do. In fact it did it perfectly. The reason it failed is that it was told to do the wrong thing. Software failures are failures of understanding, and of imagination.
When things go wrong with software it’s always because the software was told to do the wrong thing. This is why it’s so important that product and engineering teams don’t function in silos. How one person interprets a story is totally different to how another may interpret it. If your “product people” can’t clearly articulate how something should work, it doesn’t matter how good an engineer is at communicating logic to a system. This is a risky situation.
This article also serves as a good introduction to TLA+ and model-based design.
28 September, 2017
Subscribe for updates
Subscribe for weekly advice covering product strategy, development operations, building teams and more.
Privacy and terms
I care about privacy as much as you do. I will only use your email address to send you this newsletter or to reach out to you directly, and you can unsubscribe at any time. I will not share, sell, or rent your email address to any third party, though I do store it the software I use to dispatch emails.
The information provided on this blog is for informational purposes only and should not be considered investment advice. The content on this blog is not a substitute for professional financial advice. The views and opinions expressed on this blog are solely those of the author and do not necessarily reflect the views of other organizations. The author makes no representations as to the accuracy, completeness, currentness, suitability, or validity of any information on this blog and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its use. The author may hold positions in the companies or products discussed on this blog. Always conduct your own research and consult a financial advisor before making any investment decisions.