The Iowa Caucuses Debacle
It's more embarrassing than we thought.
I was somewhat amused to wake up yesterday morning to find out that the Iowa Caucuses results were delayed indefinitely because they were having trouble tallying a meager number of votes, which they had typically handled with little difficulty, because they relied on a faulty phone app. Even more so since I had, just the day before, posted about the foolish choice of West Virginia officials to conduct the 2020 general election partly in that fashion.
The next morning, we still have only partial results. They should have been able to get it done via carrier pigeon by now.
The NYT is rubbing salt in the wound with a story headlined “Faulty Iowa App Was Part of Push to Restore Democrats’ Digital Edge.”
The faulty smartphone app behind the chaotic aftermath of Iowa’s Democratic caucuses was the work of a little-known company called Shadow Inc. that was founded by veterans of Hillary Clinton’s unsuccessful presidential campaign, and whose previous work was marked by a string of failures, including a near bankruptcy.
The app grew out of a broader push by Democrats, backed by tens of millions of dollars in donor money, to match the Republicans’ prowess in digital advertising and organizing after the 2016 election. Much of the energy and investment have gone into enterprises that are intended to both boost the Democrats’ digital game and turn a profit, like Shadow.
Yet instead of showcasing how far the Democrats had come since the 2016 defeat, the disarray surrounding the Iowa caucuses raised new questions about how the party hopes to compete in 2020 with the Trump campaign, a digital juggernaut that is churning out ads and raising record sums of money.“It’s the exact opposite of the Trump team approach — bring the engineers in house, figure out exactly what we need, we build it, we test it, we own it,” said David Goldstein, chief executive of Tovo Labs, a progressive digital consulting firm.
That is, to say the least, embarrassing.
As noted in the West Virginia post, and especially the excellent discussion in comments, computer scientists and election experts are almost unanimous that we’re not ready to risk elections of this technology. Error rates that are acceptable in consumer financial transactions are unacceptable in deciding who governs. And a whole load of safeguards that exist in banking and commerce don’t and can’t exist in voting systems.
Having decided to do it anyway, the system should have been tested. And tested. And then tested some more. But no. Nick Statt reports for The Verge that “The app that broke the Iowa Caucuses was sent out through beta testing platforms.”
The app was not deployed through traditional app stores or even sideloaded using an enterprise certificate. Instead, it was deployed through mobile testing platforms, including Apple’s TestFlight and a similar service that services both iOS and Android called TestFairy. Both platforms are for apps that are not yet finalized.
Testing platforms are common for mobile apps, and are one of many ways in which independent app developers and large software makers can deploy beta software without going through the sometimes rigorous App Store and Play Store review processes. This is primarily to let developers squash bugs and ensure the app can run on a variety of different devices, some of which may be using outdated operating systems and powered by older, less powerful components that may render the app sluggish or just plain inoperable.
In this case, however, it looks like Shadow used test platforms for the app’s public distribution. Motherboard obtained screenshots showing a TestFairy download link for Android, while The Wall Street Journal reported Tuesday that Shadow used TestFlight for iOS devices.
Installing software through a test platform or sideloading onto your device manually both come with security risks, as app store review processes are designed to discover whether a piece of software is hiding malware or does something behind the scenes it’s not supposed to. In the event you do sideload an app or try installing an unofficial version, your smartphone typically warns you of the risks and asks if you want to proceed. It’s also a less stable model for deploying software at scale, which might explain the difficulty precinct chiefs had in downloading the program.
That’s beyond embarrassing. It’s damned near criminal.
I agree that if one is going to use an app, it ought to be tested thoroughly. But it is an extremely challenging and expensive task to test an app that is going to be used one time by new users in a somewhat stressful situation. Given the legitimate security concerns, probably supplying the hardware and software would have been a better solution.
If a realistic testing plan, and the advice to use secure devices controlled by the party, had been presented to the Iowa state party, the estimated cost would far exceed the paltry $60K that was apparently spent on this debacle. My guess from having been on the vendor side of these kinds of discussions is that someone told them that’s what it should cost and was dismissed out of hand when a lowball vendor came in with that $60K price. Nobody can make money on a quality, tested solution at that price point, so it’s no surprise that they got a shoddy product, with a terrible implementation (I read that the precinct captains were supposed to sideload the app, which means they had to go through a multi-step process that included accepting some pretty scary security warnings.)
Some things, like calling in results from ~1700 precincts, are just not a priority for automation when a realistic price is put on the table.
The person who got the most hosed, IMHO, is Buttigieg. This should have been a big moment for him — a small mayor winning Iowa! And instead it’s bogged down in partial results and crackpot conspiracy theories.
Well done, Democrats.
The whole thing is a disaster that *did not need to happen.*
I cannot believe that someone or multiple someones got talked into purchasing vaporware for this. If you’re first, you must must must get it right.
The Atlantic has reported that the company that botched this so very badly was formed only months ago. There was no need for an app, AT ALL.
@Hal_10000: Agreed. The Buttigieg campaign should have been riding high today, instead, this cluster-you-know-what is the headline.
FWIW, the most recent St. Anselm’s College poll has Sanders and Biden tied here in NH, with Buttigieg within the MOE for the poll.
I think this is part of a long-term trend in which the newest tech is seen as de facto the best, which means you either use it or are behind the times.
Right now it’s apps. Not long ago it was touchscreens (to an extent it still is).
The touchscreen craze brought us such disasters as Windows 8. Apps are showing what they can wreck now.
There is some justification. Computers greatly enhanced services like billing and banking. Touchscreens are great for phones, and even tablets. But it’s not absolute. Some technologies are not appropriate for some uses. Touchscreens, BTW, are making their way into aircraft, which frightens me. Think how many times you wanted to do something on your phone while driving, assuming it is on a holder, and how often you wind up doing something different because in a moving vehicle it’s hard to hit one tiny spot on a screen.
I don’t know the particulars of this app, but overall I’ve found entering data on phones is cumbersome and easy to get wrong. A PC with a keyboard is much better, even if it is outdated technology from before people even lived in caves.
As someone who works in the civic tech space, this is very much a result of the democratization (small “d”) of product/software design coupled with an unquestioning embrace of a certain type of Silicon Valley “creative disruption” — i.e. move fast and break things.
There are a lot of great organizations working very hard and very carefully to try and develop tools like this one to improve elections. Most are not for-profit organizations staffed by relative neophytes in the tech space.
Unfortunately our work, because it’s slow, tends not to attract the same type of attention. And young disruptive start-ups who over promise and under-deliver are much sexier to fund.
The Iowa party is to blame, but so is the organization who promised to develop a fully realized caucus app in 2 MONTHS! without even thinking through roll-out. No ethical organization should have accepted that deal.
https://www.theatlantic.com/technology/archive/2020/02/bad-app-not-russians-plunged-iowa-into-chaos/606052/
@mistermix:
100% this. However, given that this tool was apparently going to be used for the Nevada caucuses as well, they might have been double dipping ($60Kx2) to try and recover costs.
All that said, $120K for a design effort like this is still way under-budgeted. And that’s before we get to the fact that they were committing to delivering and testing this platform in under four months. It’s also clear no time was spent thinking through the more service design aspects of the roll out (i.e. ok, this is a great app, but how do we design the overall ecosystem that it’s working within, including contingency plans).
@mattbernius: It’s insane. I can’t believe that there wasn’t someone, somewhere who could have told them that this was all out of whack–a handful of months and $60K? Really? How did they not know intuitively that something was amiss?
These might not be “tech” people, but they have plenty of exposure to tech and costs. Most of what they do needs to be done with tech–fundraising, for example, needs to be able to interact with the FEC reporting functions. They know what they spent on that fully developed, designed and tested software.
The mind, it boggles.
There was some more reporting on Acronym and Shadow done by The Outline.
https://theoutline.com/post/8636/acronym-shadow-iowa-caucus-results-disaster
And another thing…
Who the heck thought “Shadow” was a good name for ANYTHING political? Operating in the shadows has a sinister connotation to it and I’ve had a negative reflexive reaction every time I hear the name. It’s AWFUL.
@Jen:
There are a lot of reasons for making that type of decision — you might have funders offsetting the costs. Or you’re taking the $60K as a loss to develop a platform that you can monetize on the backend.
In terms of the Iowa Party, I can easily see how that happens. I’ve seen that with clients all the time. The agency sells themselves as the smartest people in the room and convinces the client that they’re the sharpest for being smart enough to hire that agency. And if you’ve never been involved with a tech project then you really don’t imagine how it can go wrong.
Again, I work on the reg with Valley folks (and I’m exposed to Valley funders). The reality is that it’s a system that’s pretty much set up to incentivize ignoring glaring warning signs. Again “move fast and break things” is seen as a virtue.
(And funding models definitely incentivize bad behaviors.)
@Mike in Arlington:
I wish I could say anything in that article surprised me. I don’t now Acronym, but that sounds pretty similar to what often happens when people with absolutely no tech background create “disruptive tech” organizations.
@mattbernius: Yeah, I get the glitz/sell the sizzle that surrounds tech.
But I’ve worked at both a tech company with developers, and I’ve worked at a political party. There’s still a lot of just irresponsible/dumb behavior on display here on the party’s part.
When I was at the political party job (I was not in Iowa), purchases like software/hardware (this was long before apps) were thoroughly vetted. There was even resistance to purchasing the very latest versions of anything because no one wanted to spend thousands of dollars only to find out that the programs were buggy or the hardware was glitchy.
To make a purchase like this, so significant, with such a short turnaround/time frame is bizarre. I have to think that someone higher up (at the committee level–state parties don’t operate in a vacuum) was pushing for it for some reason, whether it’s because they wanted the bling of saying “there’s an app for that” or were pressured by a campaign or whatever–it’s unreal.
Gawd.
As ever, The Onion nails it.
There’s a huge, huge difference between a voting app and this.
The voting app needs to be put into potentially a million hands, and needs to do a fully automated authentication of the user, and cannot have hands-on training, and needs to maintain the secrecy of the ballot AND the integrity of the ballot.
This uploads a couple of numbers that will be publicly accessible anyway.
Worlds of difference. This could be relatively safely automated. Maybe not worth the cost, but totally doable.
Obviously, they screwed this up. But there aren’t insurmountable technological barriers.
As I mentioned in another thread, I just don’t understand why Iowa, of all places, 90% white and 84% religious is where we start our presidential election process. It made sense at some point. No so much now.
Tangentially, all you need to know about Iowa voters in encapsulated in this clip:
https://www.cnbc.com/2020/02/04/watch-iowa-caucus-voter-ends-pete-buttigieg-support-after-hearing-he-is-gay.html
For those who don’t want to click on the link, it features a woman trying to pull back her ballot for Pete Buttegeig when she is told he is gay, which she DID NOT KNOW. Now, this isn’t some rando off the street. This is a caucus voter. This is someone who agrees with Buttegeig on issues, but DIDN’T KNOW HE WAS GAY!!! WTF??? Seriously? How the heck do you know know Mayor Pete is gay?
My apologies. I meant to post my comment above in the Post Iowa forum post.
Just before I started typing this, Wednesday at 1pm Eastern, saw at CNN that we are now at 71% counted since roughly 10pm central Monday night.
At this point and given the cluster the Iowa Caucus (Cluster-cus?) was, my own conclusion is that this is a sad state of affairs for the state I lived in for a couple years. The results are tainted, questioned and this is all on the leaders of the Iowa Democratic Party. The rank and file members of the party in Iowa deserve better and they should call for changes in how that party does things.
I certainly hope they drop the antiquated caucus system. A regular primary with a paper trail is the best way to go.
I am impressed that they still only have 71% of precinct results. All of this was supposed to have been written down before uploaded, and there are only 1600 precincts, so they should have been able to do it by hand, or a spreadsheet.
Even with the extra numbers that they are reporting this year.
I almost suspect some precincts didn’t use the paper worksheets, and data was lost and everyone is shitting in their pants — but I cannot imagine that many people shitting their pants and no one talking to the press.
In all likelihood it is people who are spending way more time to get the broken system working than they would just doing it manually. God knows how many times I’ve done that.
@Jen:
Those are all potential reasons. I also think that its one thing to be involved with purchasing off-the-shelf enterprise solutions and getting involved the new product development. Two very different types of people are involved (the scariest people I ever met were corporate purchasers).
I don’t write any of this to defend the Iowa Democratic party. They done eff’d up. But I also can understand (if not excuse) why that might have happened.
@Kathy:
I’m going to assume that you’re not the driver 😉 .
Let’s all just blame Hollywood. After all, they’ve spent a couple decades convincing people that tech is magic, and you can hack the Pentagon in 15 seconds of techno-babble.
More seriously, it continues to baffle me how, with all our consumer experience of computers by now, how many people still just think the newest shiny thing will “just work.”
As a side gripe, since someone mentioned touch screens in cars above, that’s one of my favorite examples of how screwed up our approach to apps has gotten. A touch screen is a TERRIBLE interface for a driver you want keeping their eyes on the road. The tactile feedback of knobs and buttons is far superior. But 100 years of UI experience about what works in a car got thrown out for “new! shiny! app!”. Fortunately that trend seems to be partially reversing, but it never should have gotten past the drawing board in the first place. A more subtle example was a hybrid rental car I had once, with a dash display on how much energy you were recovering when braking. Complete with target lines and colors from red (you’re braking inefficiently) to green (good job on getting max efficiency back!). It was like a game. And was pretty cool at first. Then you realized you’re braking and watching your dash and not the car also braking in front of you.
Just because apps can, doesn’t mean they should.
And you kids get off my lawn!
@Just Another Ex-Republican:
Amen.
For the record, though, this is not exactly new. Just better (worse) engagement with the driver.
My dad, back in the 70s, had an AMC (remember them?) station wagon, which had a big gauge right next to the speedometer labeled “FUEL ECONOMY.” It had a range from green to red arranged in a circle, and a needle that moved more towards either end when fuel economy was good or bad.
Later on I owned GM car, I forget the model, which had a “computer” with a display near the cup holder area. You could check fuel economy, ETA, temperature, and other things. My current car displays time elapsed, fuel economy (instant), fuel economy (average), average speed, temperature, and range, in a little plain LCD display within the speedometer.
And today who doesn’t have Waze or Maps running almost all the time one drives?
@Just Another Ex-Republican:
Still, you have to wonder.
1,681 locations, each one has to send a few sheets of “paper” to a central location, and each sheet has a list of candidates and vote counts. Maybe the central location preloads part of the sheet contents for each round in the ranked-choice voting system. Writing it as a native app for various versions of iOS and Android, reportedly moving large images and doing OCR, is a pain; writing it as something that runs as a Web client-server thing with the client running in a compliant browser, entering numbers with a keypad, and the server doing some easy calculations seems like a “you can see the first version this Friday” sort of problem. Lots of the security problems have already been solved. It’s even straightforward to gin up load testing.
@Kathy:
Me!
I find it annoying and intrusive and just kind of don’t care if I get somewhere on time. Sitting in my car, with the seat warmer on calming my bad back and singing along to music that I love… seems like a fine way to spend some time, whether I’m making good time or stuck in traffic.
A friend of mine hates being in the car with me. When he drives from A to B, he has a complicated route with dozens of turns that is meant to avoid as many traffic lights as possible, even if it means zipping down alleys and cutting through loading zones. Meanwhile, I’m happy my route has only three turns.