Ace of Spades HQ

July 14, 2012

The Real Cause of the San Diego Big Boom Fireworks Fiasco
— rdbrewer

According to the LA Times, an "overzealous computer backup" caused all 7000 fireworks to go off at once. When they hit "enter" a backup file was executed instead of the primary file.

The trouble? Well, it's complicated. The explanation from Garden State Fireworks, the company responsible for the mishap, is so convoluted no one seems to be sure exactly what happened. It had something to do with an overzealous backup computer file that overlapped with the primary file to accidentally set off all the fireworks at once.
"Contrary to numerous opinions and theories, the simultaneous ignition was not due to any sabotage or massive computer hardware failure," Garden State Fireworks wrote in a lengthy statement that may require a PhD in firework-ology to fully understand. "The display sequence started exactly when it was expected to and the systems executed the file the way it appeared. Sadly, the file executed was the double file created in the back-up plan download process."

Video shot at the time, however, reveals the computer had nothing to do with it. It was "human" error.

Follow me on Twitter.

Posted by: rdbrewer at 12:36 PM | Comments (46)
Post contains 205 words, total size 2 kb.

1 So, your backup plan was to fire them off all at once? Yeah, right.

Posted by: t-bird at July 14, 2012 12:39 PM (FcR7P)

2 It's okay people....nothing to see here......

Posted by: Craig Poe at July 14, 2012 12:42 PM (BVkEs)

3 So, they made a mistake? No one was hurt. It was funny. It was historic. It was American. We can laugh at ourselves. Though I'm sure heads are rolling.....

Posted by: Craig Poe at July 14, 2012 12:50 PM (BVkEs)

4 So are the reimbursing for the cost of the fireworks?

Posted by: yankeefifth at July 14, 2012 12:51 PM (Z9EHQ)

I figured the instructions were just a bad google translation from the original Chinese.

Posted by: Invictos at July 14, 2012 12:55 PM (OQpzc)

6 There is no such thing as an "overzealous computer file" and computer files do not "overlap". That is a weakass bunch of crap to poorly attempt to cover they fucked up.

Posted by: Brainpimp at July 14, 2012 12:58 PM (mwlsF)

7 Next, they will probably get the contract to run our healthcare system.

Posted by: GnuBreed at July 14, 2012 01:01 PM (ccXZP)

8 Oops!

Posted by: Rick Perry at July 14, 2012 01:03 PM (p18em)

9 Kennedys, alcohol, and motor vehicles do not mix;

http://tinyurl.com/76b63s4

The only surprise is that she was arrested at all. Kennedys write laws, obeying them is for the serfs

Posted by: kbdabear at July 14, 2012 01:04 PM (Y+DPZ)

Oh yeah.....blame it on 'the computer'.

For all their virtues and various uses.....Computers really *have* become the biggest cloaking device for incompetence, ever invented.

Posted by: wheatie at July 14, 2012 01:06 PM (MbB0O)

11 Here, hold my beer and watch this.

Posted by: GnuBreed at July 14, 2012 01:06 PM (ccXZP)

12 Posted by: kbdabear at July 14, 2012 05:04 PM (Y+DPZ)

Not to worry, this will be quietly swept under the rug like so many other hundreds of Kennedy family high-jinks and escapades!

Posted by: Hrothgar at July 14, 2012 01:06 PM (Cnqmv)

13 Amateurs. I could show you some REAL fireworks!

Posted by: Kate Upton at July 14, 2012 01:08 PM (p18em)

14 Kennedys, alcohol, and motor vehicles do not mix; http://tinyurl.com/76b63s4 The only surprise is that she was arrested at all. Kennedys write laws, obeying them is for the serfs Posted by: kbdabear at July 14, 2012 05:04 PM (Y+DPZ) I am pushing a law in NY to have "DUI" changed to "K" simply "kennedying"

Posted by: yankeefifth at July 14, 2012 01:09 PM (Z9EHQ)

15 Sorry for OT Just finished watching the undefeated,,, again. Trying to gather the strength and courage to face the trials I am facing at work. In the grander scheme of things it seems so trivial. But I seem to stand alone. And so tired and beaten down from standing against the bureaucracy and things that are a microcosm of all that's wrong with this country. God bless you and your family Sarah. And God bless this once great nation. May she rise to greatness again.

Posted by: teej at July 14, 2012 01:13 PM (F1lOF)

16 I'm gonna take a shot in the dark and say alcohol was involved.

Posted by: katya, the designated driver at July 14, 2012 01:14 PM (bMLFV)

So, they made a mistake? No one was hurt. It was funny. It was historic. It was American. We can laugh at ourselves.

But it was ffffffffffffffiiiiiiiiiiiiiiirrrrrrrrrrrrrrrreworks! Someone COULD HAVE DDDDDDDIIIIIIIIIIIIIEEEEEEEEEEEEEED!

Posted by: generally unhinged liberal who believes everything they see and hear on tv at July 14, 2012 01:16 PM (bMLFV)

Computers really *have* become the biggest cloaking device for incompetence, ever invented

"My computer crashed" has become the new "the dog ate my homework".

Posted by: katya, the designated driver at July 14, 2012 01:18 PM (bMLFV)

19 Trying to gather the strength and courage to face the trials I am facing at work. Teej, look for a new job inmediately & never look back. I just spent 8 years in that kind of hell. Trust me, it doesn't just look like they don't want to fix it. They don't want to fix it, & are pissed that you are rocking the boat. It's not worth it. Find a new job ASAP & bail.

Posted by: bebe's boobs destroy at July 14, 2012 01:22 PM (M8oJM)

20 Computers really *have* become the biggest cloaking device for incompetence, ever invented

"My computer crashed" has become the new "the dog ate my homework". Posted by: katya, the designated driver at July 14, 2012 05:18 PM

If it's a major fuckup that can't be ignored or covered by friends in the ObamaMedia, it's called a "glitch"

Posted by: kbdabear at July 14, 2012 01:22 PM (Y+DPZ)

21 So there were not any kennedy's involved in running those fireworks?

Posted by: yankeefifth at July 14, 2012 01:23 PM (Z9EHQ)

22 29 So there were not any kennedy's involved in running those fireworks?

Posted by: yankeefifth at July 14, 2012 05:23 PM (Z9EHQ)

Why? Did someone drown?

Posted by: buzzion at July 14, 2012 01:24 PM (GULKT)

23 One'll get you ten that it was a Windows-based system that bombed. Anyone doing this on a machine like this is a dope.

Posted by: Jeffersonian at July 14, 2012 01:31 PM (gxCRt)

24 Enjoyed the clip.

Posted by: Count de Monet at July 14, 2012 01:33 PM (BAS5M)

25 I think San Diego should do that every year.

Posted by: rdbrewer at July 14, 2012 01:34 PM (Iyg03)

26 Our only failure, really, was not providing a story...

Posted by: Obama State Fireworks at July 14, 2012 01:44 PM (FcR7P)

27 "...the file executed was the double file created in the back-up plan download process." I paint this!

Posted by: Zombie Dali at July 14, 2012 01:45 PM (C8mVl)

28 The French did just about the same thing with a very big rocket a few years back. Not as entertaining.

Posted by: pawn at July 14, 2012 01:49 PM (iMsF6)

29 #32

Bullshit. The OS had nothing to do with it. By all claims the system did exactly what it was told to do. The problem was the existence of a script file that had every timer set to zero. (Yes, I have a little familiarity with some of these pyrotechnics systems. I remember one from way back on the Atari ST that used MIDI as a cheap networking method and coding system so the guy could design displays like a musical performance.) I'm willing to bet that the software produces such a file whenever a new project is started and the designer gave not thought to what would happen if only certain fields were populated while the timers were left at a default of zero. A better design would require a value to be entered before a tube could execute.

How would a different OS prevent that?

Posted by: epobirs at July 14, 2012 01:49 PM (kcfmt)

30 30 29 So there were not any kennedy's involved in running those fireworks? Posted by: yankeefifth at July 14, 2012 05:23 PM (Z9EHQ) Why? Did someone drown? Posted by: buzzion at July 14, 2012 05:24 PM (GULKT) Buzz! I was mostly thinking it sounded like an alcohol thing. or ambien.

Posted by: yankeefifth at July 14, 2012 01:53 PM (Z9EHQ)

31 "The display sequence started exactly when it was expected to and the systems executed the file the way it appeared. Sadly, the file executed was the double file created in the back-up plan download process." This is the most brilliant non-technical techno-babble I have ever seen. I'm going to start using this in my responses to Service Tickets for the Working As Designed responses.

Posted by: David C. at July 14, 2012 01:54 PM (ZtMso)

32 #40

I have to wade through such garbage frequently. My interpretation is that a backup file was created early in the process, as many apps do automatically. (Thus the .BAK files one sees when certain apps are present.) Due to bad design it was created at a point when the locations had all been entered but no timing values.

If the company re-uses the same mortar tube layout a lot they may have a generic file they use at beginning of the design phase that then gets Saved As under a name specific to the project. If the timer fields that were blank or zero were accepted as valid, then boom. A manually created backup that wasn't updated would explain much.

A big problem with software that has a very small expert user base is that it rarely has the same kind of fool-proofing that a widely used app would receive. Considering what gets out there in products with millions of users, just imagine how little testing a highly vertical app like this might get. It may be limited to just the company or a small number of companies. Right now some coder is having conniptions because he assumed nobody would ever do what these people did. It could be worse. There could have been an injury or death involved.

Posted by: epobirs at July 14, 2012 02:11 PM (kcfmt)

Would anyone here be able to tell me the best way to keep roots from my main drainpipe (Please, Keep the Jokes!) from returning?

Tried Salt..Tried other not so approved minerals. In the Heat, They grow back faster than I can get the clean out cap put back on.

Thanks for any advise.

Posted by: just another dave at July 14, 2012 02:13 PM (rDwTR)

34 What happened? I'll tell you.

There was a big evening prepared; they had mood music, they were waiting for the right lighting; and they promised nearly a half an hour of the best night you've had in a while... yeah baby, you know what I mean.

Then, just when things were getting ready to kick off, they got overexcited, and blew their entire load in the first 15 seconds... before anyone was even ready to start.

And then there was the disappointment recriminations, and apologies that always follow such an encounter.

But don't worry guys; it happens to everyone, it's not that big a deal... I'm sure they'll still respect and love you in the morning.

What? Oh we're talking about the fireworks? Ok yeah, that too I guess.

Posted by: gekkobear at July 14, 2012 02:27 PM (FYX6y)

35 Well now we know what it looks like when the whole show goes off at once.

Posted by: DaveA at July 14, 2012 02:30 PM (yVsIQ)

42OT
Would anyone here be able to tell me the best way to keep roots from my main drainpipe (Please, Keep the Jokes!) from returning?
Tried Salt..Tried other not so approved minerals. In the Heat, They grow back faster than I can get the clean out cap put back on.
Thanks for any advise.

----------

Have you tried straight bleach?

That's what the city uses here....to clear the roots out of the sewage lines.

Posted by: wheatie at July 14, 2012 02:31 PM (MbB0O)

37 I wish I'd been there. I'd have called it the best fireworks show ever. I don't care much for artsy fartsy choreographed stuff, I just want to see shit blow up.

Posted by: schizoid at July 14, 2012 02:34 PM (N9Y+W)

38 Apparently these people don't understand the necessity of a dry run before an event. When computers and electronics are involved, it is doubly necessary.

Posted by: Ostral B Heretic at July 14, 2012 02:35 PM (oSbSN)

Yes bleach, how would I go about a self dispensing system?

Toilet?

I cant pour bleach down the drain every hour or so.

Posted by: just another dave at July 14, 2012 02:42 PM (rDwTR)

40 #41 - I concur, and there was no 'overlapping' files... They executed the file from a backup copy that ONLY had locations in it (and firing instructions, but NO sequencing or timing). Basically just a map of where everything was connected and fire commands/methods. Totally blowing smoke (see what I did there)?

Posted by: Evil_Bun_Bun at July 14, 2012 02:50 PM (8qoUN)

/sock off

And a new post is up!

Posted by: Stateless_Infidel at July 14, 2012 03:00 PM (8UijS)

42 I read through the whole "explanation" and technical bullshit detector was running full tilt boogie the whole time. Whenever someone keeps saying "its really complicated, you wouldn't understand", what they're saying is they're full of shit.

If the transmitted primary/secondary files weren't sent with validation CRC or checksums and were being merged on the fly, then their extensively tested procedure was crap.

This some CYA ploy so they can punt responsibility off on someone else and still try to GET PAID.

Posted by: Unicorn mating rituals at July 14, 2012 03:03 PM (/5cfa)

43 Obama is a stuttering clusterf*ck of a miserable failure.

Posted by: steevy at July 14, 2012 04:17 PM (Xb3hu)

44 How would a different OS prevent that?

Posted by: epobirs at July 14, 2012 05:49 PM (kcfmt)

Windows might crash. So its at least got that going for it.

Posted by: Not Drinking Nearly Enough at July 14, 2012 04:22 PM (HtUdo)

45 " When they hit "enter" a backup file was executed instead of the primary file."

I don't think I'm gonna stop laffing for 2 days...unless I catch a glimpse of the Hatchet Faced Hack David Gregory tmw morning...

Posted by: JewishOdysseus at July 14, 2012 04:40 PM (1X/a4)

46 9 There is no such thing as an "overzealous computer file" and computer files do not "overlap". Posted by: Brainpimp ^ this I'm no pro, but I've been making computers my b.. uh.. programming since before there was MS-DOS. Hardware has generally been astonishingly dependable at doing just what the software tells it. At the grocer when their new system doesn't recognize the self-serve lanes (night before last), when the bar quits serving because the cash registers are down (I almost got kicked out for harrassing the owner about that one), there is one song, one mantra, one great truth to shout aloud: BLAME THE PROGRAMMERS!

Posted by: A Mindful Webworker at July 14, 2012 07:14 PM (lBpVP)

Hide Comments | Add Comment | Refresh | Top

Comments are disabled. Post is locked.

89kb generated in CPU 0.0345, elapsed 0.2124 seconds.
64 queries taking 0.1885 seconds, 174 records returned.
Powered by Minx 1.1.6c-pink.