Sean Gallagher, Ars Technica‘s IT editor and a former Navy officer, explains, “Why US government IT fails so hard, so often.”
Despite efforts to make government IT systems more modern and efficient, many agencies are stuck in a technology time warp that affects how projects like the healthcare exchange portal are built. Long procurement cycles for even minor government technology projects, the slow speed of approval to operate new technologies, and the vast installed base of systems that government IT managers have to deal with all contribute to the glacial adoption of new technology. With the faces at the top of agency IT organizations changing every few years, each bringing some marquee project to burnish their résumés, it can take a decade to effect changes that last.
That inertia shows on agency networks. The government lags far behind current technology outside the islands of modernization created by high-profile projects. In 2012, according to documents obtained by MuckRock, the Drug Enforcement Agency’s standard server platform was still Windows Server 2003.
Magnifying the problem is the government’s decades-long increase in dependency on contractors to provide even the most basic technical capabilities. While the Obama administration has talked of insourcing more IT work, it has been mostly talk, and agencies’ internal IT management and procurement workforce has continued to get older and smaller.
Over 50 percent of the federal workforce is over 48 years old—and nearly a quarter is within five years of retirement age. And the move to reliance on contractors for much of IT has drained the government of a younger generation of internal IT talent that might have a fresher eye toward what works in IT.
But even the most fresh and creative minds might go numb at the scale, scope, and structure forced on government IT projects by the way the government buys and builds things in accordance with “the FAR”—Federal Acquisition Regulations. If it isn’t a “program of record,” government culture dictates, it seems it’s not worth doing.
So, how bad is government IT, anyway? Well, the abysmal launch of the Obamacare website is a pretty good indicator. But that’s a brand new system being operated by a swarm of untrained, external users for the first time during a mad deluge of interest. Internally, it’s much worse.
The government has kept using Windows XP and Server 2003 despite warnings from the National Security Agency that “Windows XP and Windows Server 2003 lack critical security features and are near the end of their extended support lifecycle.” Still, the federal government (much like most of the business world) largely took a pass on Windows Vista. And even though the National Institute of Standards and Technology added Windows 7 to the US Government Configuration Baseline a year after its release, most agencies didn’t start migrating off Windows XP until 2011 or later. While the Army and Air Force had adopted Windows Vista, XP was still fairly widespread when the Army began migrating to Windows 7 in July of 2012.
It’s not as if the Powers That Be don’t realize there’s a problem.
That’s not to say that the DOD hasn’t tried to make its IT more modern and efficient—the DOD has long been among the government’s IT “haves,” at least budget-wise, and the military is counting on IT efficiency to help reduce costs. But the scale of the changes it has to make in those efforts is staggering, and the process takes years to implement.
The US Army’s Enterprise Email (EE) program is a case in point. EE, which the DOD’s CIO calls “Army’s #1 information technology efficiency initiative,” is a unified e-mail system for the DOD. The Army, its first customer, had over 440 separate networks—each of them with their own e-mail systems and directories on a variety of different software platforms—when it started looking for an Army-wide solution in 2009. The Army gave up on trying to find its own answer and turned to the Defense Information Systems Agency. The agency, which is in essence the DOD’s internal IT and telecommunications provider, started rolling out its internally built system based on Microsoft Exchange to the Army in February of 2011.
The migration was finally (mostly) completed in July, two-and-a-half years after it started, after a few fits and starts (and a Congressionally mandated cooling-off period). It now reaches over 1.43 million users on the Army’s unclassified network, and 115,000 on the secret SIPRNET network. That’s a tiny fraction of the over 425 million users that Google serves with Gmail, but it’s the biggest Exchange deployment outside of Microsoft’s Office 365 cloud—and it all has to run within the DOD’s security specifications on DOD hardware.
So, what gives?
Part of the reason is the metrics that the government uses for “success” in its IT programs and part is how they get bought in the first place.
The DOD CIO’s IT Dashboard entry for EE rates it as a top program with a metrics score of 5 out of 5. But the Army hasn’t begun to measure one of its key metrics—the speed of mail delivery. And service availability, rated at 99.998 percent, is measured based on “a collective average using the Exchange servers in all the pods”—not on the user experience.
There’s a similar disconnect in metrics over at the Department of Health and Human Services. Take the HealthCare.gov Plan Finder program, part of the HealthCare.gov site that, as described by HHS’ project dashboard, is “a portal that allows consumers to search for both public and private health coverage options through an easy to use health insurance finder tool. Based on answers to a series of questions, the coverage finder produces a menu of potential coverage choices personalized for the user.”
[…]
The bottom line is that federal IT programs’ success is measured by things that have nothing to do with how successful they are or by the metrics most of the world uses. While the business world (and Web companies in particular) now monitor user experience and productivity as a metric for IT success, the government keeps throwing out numbers that mask the truth: the only people who would use their systems are the ones that are forced to.
I’ve been in my own little IT circle of hell since re-entering federal service on August 26–seven weeks ago. That morning, I posted here that
I’ll likely be off the grid for a day or three owing to one of the Catch-22-s of working for the Defense Department: You can’t start working until you have a Common Access Card and you can’t get a CAC until you start working. Presumably, I’ll fill out the paperwork and get my photo taken today but it’ll likely be a bit before the card gets issued and, if this is typical of my previous DoD experience, I won’t be able to get the process rolling to get set up with a work station with Internet access until then.
Alas, it was not typical of my previous DoD experience. When I last worked for the federal government, it was as a contractor for DISA back in 2004. The private firm that actually paid my salary was able to arrange for my to get my security badge on my first day in the building but it took some time to get my CAC issued and then a couple more days to get access to a personal workstation. It took nearly a week before I was really able to do my job, which struck me as absurdly inefficient.
Things are much, much worse now.
On the day I reported in, they told us that we wouldn’t be able to get our CACs issued until we were “in the system,” which would likely be several working days. Why didn’t they already have us “in the system,” given that they’d known for quite some time that we were going to be onboarding? Well, sometimes people don’t show up for work and then the time spend getting them “in the system” would have been wasted. Far better to waste the time of new employees, who are in many cases being paid far more for not working during this interval than those responsible for putting them “in the system” are for working than risk having some time wasted in the rare event someone who went through all the hassles of jumping a hundred hoops to get a government job didn’t report for work.
Roughly ten days later, I was “in the system.” That allowed me to set up an appointment with the people who issue a CAC; they won’t do it until then. So, I called over and was told that the first appointment was several weeks later. Most new employees dutifully sign up for an appointment but, anxious to actually be able to get work done at the office, I instead showed up early the next morning and endured the long wait for those without an appointment. Two hours later, I was next in line. Forty minutes later, I still hadn’t been called. Finally, they announced that “the system” (a different system, I gather than the previous system) was down. Sometimes, they explained, it would be back up in a little while and sometimes it would be down for a couple days in a row. I wasted another hour or so to see if it was the former and then gave up.
I came back the next morning, a Friday, and was thankfully able to get my CAC issued. At that point, they asked what my DoD email address was, so they could put it on the card. I told them that my agency had told me that they wouldn’t issue me an email until I got my CAC. Which was true and, indeed, the policy. So the CAC-issuing people told me that, once I got an email address, I’d have to come back over and have it added to my CAC. Great.
So, CAC in hand, I went to my IT office. The woman in charge of getting me started on the process of getting an email address wasn’t in. The system (different than the two aforementioned systems) had just changed, so no one else was able to help.
The next day, the woman who understood how to apply for an email address told me that I would need to sign up for an online system that allowed me to take two classes, which I would have to retake on an annual basis, that would enable me to apply for an email address.
It took nearly two weeks to get approved to take said classes.
Access in hand, I dutifully went over to the library to take them. The system didn’t work, so I finally gave up in frustration after a wasted hour. I tried again at home that afternoon, figuring my superior broadband connection would do the trick. It did not. I tried the library again the next morning. Same deal. Finally, I went back to the IT department and they let me take the classes there. Essentially, they told me stuff that we either obvious to people who had ever used email or which were absurdly silly policies that were so impractical that few seem to actually follow them. Regardless, half a day later, I had passed the classes and printed out the certificates to prove it.
That allowed me to apply for an email address.
A week and a half later, the person in the agency responsible for sending up the request to whoever it is that processes said requests in fact sent up the request. Three days later, those people input it into their system. I was told that an email address would be forthcoming the next Tuesday. Or, certainly, the next Wednesday.
The next Tuesday, the government shut down.
The Monday after that, I reported back to work, as my part of the government was back in business. Still no email address. Ditto Tuesday. And Wednesday. And Thursday. So, finally, they called over to the email issuing place and were told that my stuff, which they told us had been entered into the system two Fridays earlier, was in fact not in the system. So, the IT people sent my stuff back over via email.
That’s where I stand now. I’ll check back in when the place opens back up on Tuesday.
Note that this is just to get an email and access to the ordinary Internet on an outdated PC with Windows XP installed. It’s much, much more cumbersome to get access to systems which can access classified materials.
Aside from all the issues Gallagher points to, a lot of the nonsense is overreaction to various legitimate concerns like cyber attacks, the spread of viruses, and protection of privileged personal information. And, of course, the recent whole-scale theft of classified information by Edward Snowden and the person formerly known as PFC Bradley Manning. But, of course, Snowden and Manning made it through all these hoops and then some.
