I’m continuing to talk about how frustrated I am that people believe that public employees are too expensive, public employees are lazy and incompetent, and unions only protect morons.
So now, on to technology and innovation. I hear that the public sector is stagnant and lacks imagination. I hear that the public sector makes horrible choices in technology investments. I wonder where people are that they think the private sector is that much better.
Oh, I agree that ideas move very slowly through the public sector. And there is a horrible trend to buy something that really doesn’t make sense. But this is true in most large organizations, public or private. When a company is small, ideas flourish. When a company is huge, they slowly ooze to the surface if you’re lucky. Enterprises want to buy the one-size-fits-all systems, and scratch their heads wondering why it isn’t working — but they’ll keep trying to push that square peg in all the same.
Harmony Central
Musician’s Friend acquired Harmony Central around 2005, and put me in charge of development after a while. It quickly became clear that MF didn’t want to put time into the project, as I spent less than 20% of my overall time as “lead” developer on HC, and I was not only the lead, but also the only programmer on the project for the vast majority of my time at MF. (This should explain a lot to the HC fans — sorry, I really had some great plans but nobody at the top approved anything major)
When we acquired HC, we realized quickly we needed to change the user reviews system. It consisted of over 100,000 lines of Perl code, where each category (Guitars, Bass, Microphones, etc) had a set of effectively copied and pasted scripts to manage admin actions, user actions, static HTML generation, etc. MF decided to hire ProKarma, an outsourcing firm, to implement the new Harmony Central system with various Java-based technologies for portals, CMS, and so forth. In the meantime, I was authorized to spend time building a little hacked-together Rails app to help the site float by until ProKarma got their solution finished.
I put in six months and got something pretty solid working. It did the basic job of all the Perl scripts in about a tenth the code, and I even built a system to transfer all old reviews into the new system, including those stored in static HTML files instead of a database. The team of outsourced consultants at ProKarma (at least three people, sometimes more) spent even longer and gave us code we were unable to use, and no way to convert old data. Despite the best efforts of many people, and a strong push from above, the system wasn’t usable and the lead said nothing was even able to be salvaged to restart the project in-house.
Even so, management did not like Rails. Consultants told them it was a toy, it wasn’t enterprise, Java was the way to go. Fewer than 1000 man-hours to get something working in Rails vs. over 3000 to achieve nothing, and management was still on the hunt for the new enterprise-approved HC.
Finally, in 2009, MF chose Jive. My HC code for user reviews had grown and changed, and I had even built another Rails app for the HC news system to replace a horrible Mason-based system. But my effort was still nowhere near enough to outdo the silver-tongued consultants.
MF initially spent a hefty sum on Jive. Without mentioning specifics that might violate some contract or another, I’ll just say this: they could have hired a team of ten people at my salary for a full year working full-time on Harmony Central for what they spent on Jive. And Jive was a disaster. The first two attempted launches failed – my Rails app was amazingly robust, and served a ridiculous amount of traffic on two very old blade servers. Jive couldn’t keep up with the traffic, even though they were an all-Java shop.
Eventually Jive got some serious back-end servers, and now it’s running pretty well. But my main point here is that I had done something consultants and Jive failed to do, and I did it in an innovative and rather cheap way. Given a real full-time staff, we could have had something amazing, and I said so more than once. But this idea was rejected because it didn’t fit what the consultants said an enterprise application should be. How’s that for lack of imagination?
Heiler
Java-is-the-only-way continued at MF when they decided to purchase Heiler, a product meant to manage catalogs and workflows. Instead of paying us to do what we so desperately had wanted to do (rewrite the old cobbled-together systems that ran Musician’s Friend), they spent a fortune on Heiler. The product shipped very late, was riddled with major bugs, and even though I left MF almost a year ago, I still hear major complaints about the system. Two employees quit directly because of it, and others considered Heiler just the kick in the ass they needed to start looking for a new job.
The codebase that we worked with looked like somebody’s half-assed high school homework assignment. The internal Heiler developer we had on-site (whose name I won’t mention) was about as good as your typical junior-level software engineer (he never was able to understand the rationale of using a source control management system; he didn’t understand bitmasks even though his product used them for configuration; he never understood why we pushed for subclassing instead of copying an entire class and then slightly tweaking one function; the list goes on).
Instead of trying out something innovative (we had a plan for doing exactly what this system tried — and mostly failed — to do), or at least listening to the people having so much trouble using the system, MF decided instead to ignore all the red flags and keep pushing onward. This ended up meaning a lot of extra hours for many people, and I even felt bad for the Heiler devs who were scrambling to try and fix major flaws in their company’s application. Eventually the scale of the app had to just be toned down and certain manual workarounds were implemented because their system was simply not capable of handling MF’s needs.
I sent an email which detailed my concerns about Heiler. It’s a long read, but well worth it for a laugh at MF’s expense.
The university
On the other hand, at Oregon State, we’re using Rails alongside Java. We’re doing iPhone and iPad apps. We’ve got a mobile-friendly version of our site. We have done some pair programming and we are migrating toward git over svn.
Things move slowly, but at OSU they do move. At MF, the technology decisions were quite literally made at the top. VPs would talk to consultants and tell us how to write software. At OSU, there is a lot of direction from the top, and sometimes it’s just as stupid, but ideas from the peons are not automatically discarded if we can explain how an idea can help.
I disagree with the majority of management’s ideas, and I probably always will no matter where I am, but overall my department is keeping up with technology far better than MF did.
Long time, stranger