Tuesday, December 3, 2013

Dear Media and Conservawanks: Please to Engage in Aural Self-Love Now kthx

Nothing has been more ridiculous and annoying to me in the last many weeks than all this fucking wangst over the ACA website, and I'll tell you why.

It's manufactured. And it's bullshit.

TL;DR/BLUF: The entire contract system is flawed, the process models used by contractors guarantee either severe bugs or perpetual delays, and the media and cock-waving conservatives can suck the dick I don't have.

Allow me to share some tidbits and yarn-spinnin' to elucidate this not opinion but straight up, no shit, absolute fucking FACT.

The software industry has a problem, one which I suspect is shared across the more general tech industry, especially when you're talking safety-critical systems. Because code can be intricate and have unintended side-effects (aka bugs, aka undocumented features hurr hurr) when human error and varied platforms are factored in, the industry has adopted process models to ensure code is created and adjusted, from requirements gathering to release, in an orderly and safe fashion. Regardless of the model of development, there is always some administrative process model around that development. So for example, if you use a spiral development process, you might have meetings at each cardinal point in development, and these meetings might be just a handful of the total number you engage in.

In other words, process models often require a metric shit-ton of man-hours in meetings and coordination and administrative bullshit intended to make you more efficient but actually...don't. I saw this during my internship at BAE Systems, which was working at the time to move from the SEI (Software Engineering Institute of Carnegie Mellon) CMM (Capability Maturity Model) Level IV. They were already at Level III, which roughly translates to "all our processes are documented and we have a shit ton of meetings kthx." I also saw this during my time at Conexant Systems, which wasn't on an official mission to create an official process for all software/firmware/whateverware development. We still had meeting after meeting after meeting, though not as rigid and regimented as at BAE. I'm pretty sure the difference is that BAE is a government contractor and Conexant was making cable modems and shit. BAE made its money by stringing along the government agency requesting their help. Conexant made money by releasing cable modems into the world. See where I'm going?

Cut to my time in Bahrain, where I was hired to administer the GCCS-M system but actually ended up doing a ton of special projects. I was still supposed to handle the message traffic system--at the time, it was JMHS--but my trouble tickets ended up being routed to the IT group, which used two Excel spreadsheets to track trouble tickets, so someone else was getting all my dirty work, and I wouldn't know unless I looked over their shoulder at the spreadsheet. I wasn't okay with that, so I decided to create a helpdesk system everyone would be required to use to submit a trouble ticket for JMHS.

I nosed around, procured an unused desktop, loaded it with Linux and Apache and MySQL and Perl, and went to town creating a web-based system. Two hours later, I realized it would be super easy to open up the options to pull in trouble tickets for all systems the IT department was handling. A webpage would be hella better to deal with than those ridiculous spreadsheets, right? So I asked the chief if he was down with this evil plan, and he said yes. He gave me the parameters and inputs they'd need, and the next day, I had a more robust system created. I took another few days to test it, expand it a bit more, and make it prettier, and then I asked the chief to test it.

Within a week, we had outside testers. Within two weeks, it was released to the entire base, and it was being used. The spreadsheets went away. And then someone asked the question:

Hey, didn't we pay for a system like this?

Yes. Yes, they had. It was called Remedy, and it was already a year late being delivered. I knew my system was a temporary bandaid while we waited for Remedy's delivery, which was supposed to be in four to six months, and I was fine with that. If I'd thought Helpdesk (my system) was going to be more permanent, I'd have actually used some software engineering on it and not just created a kluge pot of code.

Over the next several months, I added functionality and even extended the web forms to allow other requests: funding, awards, reimbursements, etc. Before I knew it, that little desktop was doing a whole fuckload of work, and I was kinda proud of it.

Then one day, two dudes showed up in my trailer. "Hi, we're from whatever company."

"Oooookay," said I.

"We're working on Remedy, and they sent us out here to find out if you have new requirements we need to incorporate."

Of course they did. More meetings to create more code to lengthen the amount of work they're putting in to get more money from the government.

They gave me a condescending smile. "We heard you wrote a little program to track trouble tickets. Would you mind showing us? We'd like to see what you're working with so we know where our deliverables stand."

I gave them a condescending smile. "Sure. But you should know it's much more than trouble tickets. Let me show you." I gave them a five minute quickie tour of the ticket system, the reporting, the graphs and statistics, etc. Then I gave them a tour of the awards requests, the budget tracker, equipment acquisition requests, and all the other doodads I'd added to the system.

One of the dudes cleared his throat. "So, how long did it take you to develop this?"

I had to think for a minute. It had kind of grown organically over the past three or four months since the Iraq invasion, but it was only one small thing that took up my time. "It took about a week for just the ticket tracker. The rest has just grown over time. I couldn't really say, but I've been working on it since January."

They -- and I'm not fucking shitting you here -- literally paused, looked at each other, did some of that silent nerd-speak, and then looked back at me. "A week?"

"Just for the ticket tracker. But that includes testing."

Then the other dude kinda does that little snort-laugh of ultimate dudebro dismissiveness. "How many people are on your team?"

Ha! A team. I wished. "I don't have a team. It's just me."

And then I'm not shitting you when I say they took over the extra cubicle in the trailer and sat with their heads together the rest of the day. They glanced at me a few times, noodled around on our helpdesk system a while, and then left. I never saw them again.

It took another four years for Remedy to be delivered. FOUR. YEARS.

Why? Because this is process development with government contractors. They come up with requirements, divide the work, pull in subcontractors to deal with pieces of code they don't have the time or manning to deliver on, and then commence the endless cycle of meetings and confab.

Here's what's particularly tasty: I was told the base didn't want to move to Remedy because its functionality didn't match the helpdesk's, and it wasn't as stable--it crashed. Keep in mind I had not been there for two years at that point and hadn't upgraded any of the helpdesk pieces. I'd tried to teach some folks Perl so they'd be able to tweak code and fix bugs, but I was told it *just ran*. This freaked me out because I hadn't built it for longevity, and they were really, truly relying on that system, but Remedy appeared to be a step down for them.

Do you see where I'm going here?

When you're talking about software systems delivered by contractors being paid every time they have to reevalute the requirements and add new functionality, of course they're going to stretch out the time to delivery. But even if they're on a tight deadline and know they have an absolute, no-shit delivery date, there are still so many moving parts to the process model between a contractor and all its subcontractors that it is impossible to properly implement and test a system before delivery.

I repeat: When there is a fixed, no-shit delivery date, it is impossible for government contractors to properly implement and test a system before delivery.

Software contractors are so accustomed to a way of doing business (where maximizing profit involves extending the delivery of the product) that when they have a very rigid timeframe for the development and testing of the deliverable, they cannot fit all the fuckdickulous process elements (meetings, development and testing cycles, etc.) into that timeframe.

It's the nature of the beast and the way government contracts work. The entire fucking system is flawed, and those who rely on the end products are regularly screwed over as a result. YodaMan was involved in a crapton of acquisitions while we were there, and I got to see what happens when we deal in contracts with civilian vendors. THE SYSTEM IS FUCKED. It's set to protect the vendor, tax dollars be damned. And once a company has you snared, they can keep you snared. For, apparently, half a decade. Or more. Gods only know how bad it is when you're talking about really complex mechanical/software systems, like an entire spy plane or some shit.

So when I watch CNN and Fox News, and when I read wankfests online, all pointing fingers at "Obama's website" and talking about what a black eye this is for the administration, I want to junkpunch some fuckers. Because here's what I GUARANFUCKINGTEE happened:

  1. The ACA comes with a deadline for mandatory enrollment.
  2. Republicans have been doing everything in their power to turn off the ACA.
  3. There had to be a firm deadline for delivery OR ELSE.
  4. Pressure came down like a waterfall, and the poor, unsuspecting contractor who literally doesn't know how to function outside of that process model choked. They delivered something they knew was fucked up because they could not delay.
Where does Obama blame come in? Pressure delivery. Not coming in and saying "we have to delay rollout." That's it. But you know what? EVERYTHING DELIVERED HAS BUGS. Microsoft Windows has bugs. Apple's shit has bugs that drive people off cliffs and into trees, apparently. Code has bugs. But prior to code, other systems had bugs. There are a lot of moving parts to every new system implemented by a large group of humans since the beginning of time, and this insistence by media and conservawanks that the enrollment run perfectly from day one is ridiculous. 

I will totally point and laugh at the contractor, though, for being a typical software company and an even more typical government contractor. They are to blame for this mess in the end because, like most companies of their ilk, they lack perspective and the willingness to transcend their process model.

But all this political shit-throwing and blame is bullshit, and every time a pundit opens his/her maw and waxes on about the ACA website, I want to kick that person in the mouth. They have no fucking clue what they're talking about, have no idea how software is built, and have no idea how flawed the contract system is. If they did, they'd know this isn't a political football to be tossed around, but an indictment of an industry in serious need of an overhaul.

Also, business as usual since no rollout ever has been perfectly smooth, kthx now stfu.


liberal army wife said...

YES!!! all those who say "amazon can do this/that"... don't get that this has to be done to GUBMINT standards, also incorporating existing GUBMINT programs. thank you for saying it with the authority of someone who has been there, done that, designed it.

Anchored Away said...

LAW, not only that, but Amazon rolled out their website--which did NOT look like it does now--fifteen or so years ago. It has evolved. Their database has evolved. Their development builds on top of the last round of development and debugging. AND STILL Amazon will shit the bed because someone tweaked something. So saying "Amazon can do this and that" is a variation on the theme of You Don't Know What the Fuck You're Talking About So STFU Now. Which you know. I'm just so fucking pissy about all of this shit. Errbody shut up while the contractor unfucks itself. Meanwhile, let's fix the system.

Open source FTW!

Ananta Androscoggin said...

Yah! I had a feeling it was more about this kind of thing.

I mean, just how incompetent are those software companies (I heard on one news show that there were 3 involved in the ACA site) to deliver something so screwed up, and how can us taxpayers demand that they DON'T GET PAID FOR shoddy work?