POSTS
Of Debt and Decisions
Back in the salad days of the 1990’s, I was working at a company that was building the hot product of the day: an intranet. Every company wanted an intranet. We’d built several for different client before deciding to create a white-label intranet that would be the delight of all our customers. Give us some money and we’d give you a themed intranet with calendaring, project tracking, task management and more.
We didn’t offer a hosted service instead, we would ship a computer to you with all the software installed that you could run on your network. In addition to being to white-label it, you got to pick which you would be most comfortable supporting: a Linux or Windows NT based install. The Linux version came with RedHat Linux, Apache and MySQL. The NT, in addition to Windows, came with IIS and SQL Server. This post is the story of engineering decisions, both good and bad, that went into the creation of our intranet-in-a-box and my role therein.
A tale of two languages
One of the first questions most engineers will ask of you when discussing a project is what language you used. Our intranet-in-a-box used 2: Perl and Java. Our primary language for doing client projects before we got into software was Perl. In the beginning, the intranet-in-a-box was the work of a single developer. It wasn’t a primary focus of the company. The vast majority of time was spent doing client work. While intranet-in-a-box development slowly progressed on the side, that single developer was tasked with writing a chat server in Java. At the time, chat servers were also a really hot product that everyone wanted and Java and Java Applets were the hot new thing. To call the developer who wrote the chat server smitten with Java and Applets would probably be an understatement. I remember him as being head over heels in love. At that point, the calendaring and a couple other modules of the intranet-in-a-box were already mostly done in Perl when the project switched to using Java. Existing code in those modules wasn’t rewritten and new development in it would be done in Perl, new code however was to be written in Java. The chat server had gotten us into the “shrinkwrap” software business. The intranet-in-a-box would eventually push us fully there. More developers were added to the project and work commenced in with both new Perl and Java being written.
Most communication between the different modules was done by using a database as a pool of mutable shared state. If module A and B needed to talk to one another, they could just modify and read from the same tables.The vast majority of sharing could be handled this way, there was just one problem: the further we got into the project, the more shared base code that we were porting from Java to Perl or Perl to Java. Further, that shared code was getting out of sync. If you had a utility routine “foo” that was used by both the Java and Perl parts of the codebase, any update had to be made in 2 places. As a percentage of the codebase, there wasn’t much of this code, but it was becoming more and more of a problem making sure that the functionality didn’t diverge. My first task when I was added to the intranet-in-a-box project was to solve this problem. My one constraint? It couldn’t involve rewriting all the Perl code in Java so that we had a single language codebase.
I don’t know what anyone else would have done given that task and constraint but I know what I did: I created a cross language bridge that allowed the two sides of the codebase to talk to one another. The bridge involved serializing data on one side of a socket and deserializing it on the other. CORBA and various RPC tools like it were quite popular at the time. I had been learning a lot about them and so, our own little monster was created. I might cringe about it now, but it worked. It worked well, it was damn solid and with a little wrapper code most people didn’t even know it was there.
The other way to be “database agnostic” The time came when we were going to start selling our intranet-in-a-box to our first customers. We installed RedHat Linux, Apache, MySQL and our mixed Perl and Java codebase onto a box and shipped them off to a customer. Before long, we had another customer but their IT staff was comfortable with NT and not Linux so we set about getting everything running on NT with IIS as the webserver. Getting Perl running on Windows NT with IIS was a bit of a pain but it was a known, solved problem. Our real issue was that the customer wanted to use SQLServer. I don’t blame them. They already had a SQLServer install and were familiar with the various administration tools. To understand what I am about to relay, you need to understand a bit about our development and deployment processes at the time.
We barely had any source control, we had no automated testing and all work was done on and tested against individual developer machines. Some like mine were Linux, others Windows. Somehow along the way we ended with code in one of the modules that had been written against SQLServer while being developed. It relied on SQLServer syntax that didn’t exist in MySQL. For the life of me, I don’t remember what the syntax was, I just remember that we had hit a point where we needed to run different sections of code for each database. In a truly amusing twist of fate, it turned out that changing our codebase to account for the differences wasn’t the easiest path forward. Making the changes throughout our shared codebase that lacked an automated build and deploy process was going to take a decent chunk of time. On the other hand, my writing patches for MySQL to support the SQLServer functionality was relatively cheap. Yup, that’s right, I maintained an internal set of patches for MySQL that bridged the SQLServer to MySQL functionality gap. And it took less time than sorting out automated build and deploy was going to take. I’ll admit it, I was young and got a bit of a rush that I was able to help us sidestep having a come to Jesus moment with our “cross platform” strategy by creating a custom version of MySQL. And it gave me a taste maintaining custom patched software.
What’s a few more patches between friends
Somewhere along the line, we got a couple complaints about the software not being fast enough. Given our stack and lack of sophisticated tooling, we had no way to address the issue easily. We didn’t have any fancy profiling tools and most code paths from web request to database and back were going to be fairly short. Optimizing the code was going to be problematic and might not result in much of a gain. Moving the Perl portion of the codebase to Java would have helped with performance in those sections but we had no tests and the project was sure to take quite a while. By this point, we were doing a good business providing customizations to the product and weren’t in a position to stop doing that work to address the performance issues via a rewrite.
I started searching the Internet for ways we could speed up Linux and MySQL. I came across a couple of kernel changes you could make that would in theory provide a speed boost. I patched the Linux source, created a new build and tried it out. Primitive quantitative testing and qualitative testing both pointed to the changes having a real impact. I was already interested in operating systems, network stacks and had been doing quite a bit of reading and exploring both the Linux and NetBSD codebases trying to learn more. The results from our testing were like catnip, I spent the next 3 weekends in the office going through the kernel source, looking for optimizations, creating patches, trying them out and moving on. By the end of those 3 weeks, we had a kernel that was tuned and patched for the boxes we were shipping our product on and happier customers.
The horror story that wasn’t
Hopefully you are horrified by all things we didn’t do that are now considered best practices. No reproducible builds, lack of automated testing, creating and then repeatedly sidestepping technical debt. For a long time, that is how I told these stories: as horror stories that came complete with hand gesticulations to emphasize the “can you believe I had to do that?” nature of it all. Here’s the thing though: we were making money. We were growing and supporting ourselves. We were continually creating value for the company. Yeah we had a lot of technical debt but we kept finding inventive ways to sidestep having to pay it. The debt metaphor that we’ve adopted is misleading. You don’t always have pay off the debt you created; you can work around it and avoid ever having to pay it off. Technical debt only hurts if you have to pay it back.