Approximate 8 minute read
Have you ever had a show or a church service “crash and burn” technically? I don’t mean a little hiccup, I mean a major failure! The type that EVERYONE in the audience knows “something’s definitely wrong.” The type that gets you called into someone’s office for a meeting afterward?
If you asked me what show I worked on last week I would have a hard time telling you without looking at a schedule. If a show goes well my mind just drags all the details into its Recycle Bin. Unless I work with that client soon after, or at the same venue again, most of the details are long lost after the invoice is sent.
But if a show had the slightest problem I can remember details with remarkable clarity. The worse a show goes, the better my memory. Why is that?
One phrase sometimes heard on headset during a problematic show is “foam the runway!” In these three short words someone can communicate a much more complicated message: “I’m going to land this plane/show somehow. It may not be pretty. It’s likely to get bumpy. We may even put it down without all the landing gear in place. So stow your tray tables, tighten your seatbelt low and tight across your lap, put your seat-backs in the upright position and brace for impact!”
I have to admit, I’ve had to foam the runway a couple of times and I remember each like it was yesterday.
My worst crash & burn was in 2004. I was representing the company I work for providing all the TV technical facilities for the Miss Universe Pageant originating from Quito Ecuador. In that role I was responsible for four Avid edit rooms, a dub station, eleven camera live-live production to NBC, satellite transmission, screens switching and seven crew members. Simultaneously I was also serving as Technical Manager for the Miss Universe Organization, which in that role I was responsible for the technical set-up of all other equipment including projection, PA, lighting, etc. and a crew of about fifty people. Lastly I was also coordinating the technical needs of international and local Ecuadorian broadcast licensees. I had a lot on my plate.
Not enough pressure?
For the Miss Universe Pageant we typically have a world-wide audience of one billion people. If I made a mistake Id’e be doing it in front of one of the largest worldwide television audiences. To put things in perspective, the 2017 MTV Video Music Awards has an audience of about six million people in the United States. The 2018 Tony Awards has about 6.3 million viewers in the US. The Miss Universe worldwide audience is 15 times larger!
We were in Quito for about two weeks. All was going reasonably well. The big challenge was that digital audio consoles were just gaining acceptance in the broadcast TV world. The on-air mix was scheduled to be done on a Calrec Zeta digital board. The TV A1 audio engineer however had recently been bitten on a few shows by a Neve Libra Live digital console. So, even though he had no prior bad experience with the Calrec Zeta, he requested that he have a back-up desk standing by so if the primary desk failed he could swivel his chair and continue mixing on the back-up console.
Now doing a broadcast TV show with two on-air desks backing each other up is no small feat. There are hundreds of inputs in various signal formats that need to go to both desks. Worse, main program stereo to transmission, international audio to transmission, feeds from the TV A1 going to FOH and monitor mixers, feeds to control room speakers, the A1’s speakers, various PFL speakers, feeds to licensee broadcasters, feeds to edit rooms and upwards of eighteen record VTRs all need to be salvo’d using a router to send the correct analog, AES and MADI to the right destination no matter which desk the A1 was working from.
The night before show day an executive came to me and said that the NBC Today Show wanted to have the Miss Universe Pageant host, Billy Bush and Executive Producer, Donald Trump as guests live from Ecuador on their morning show. The executive asked if I could arrange to use our “main show control room,” cameras and audio mix facilities for the NBC Today Show’s live insert.
Well, equipment-wise we were fine. We had more than enough equipment resources to do a morning show live shot. The problem was that the crew had been rehearsing every evening for more than a week in preparation for our live broadcast in the evening. To do a morning-show insert meant that we had to get some crew back to the venue by 5:30 AM and then they would be awake for fourteen and a half hours before we even started our main show. It would be sixteen and a half hours before we’re off the air, and twenty-four hours around the clock before we completed the strike. It would be a very long day with some crew members getting as little as three and a half hours sleep before the morning live-shot.
Despite the risks and with the intent of being a “can do” person, I said yes. Would you tell Donald Trump he couldn’t appear on the Today Show? … No, I didn’t think so.
Before show time we tested our “911 router salvo” making sure that we could indeed go back and forth between audio consoles. After going back and forth a few times everything seemed ok.
Our show open was a pre-tape package where each delegate introduced herself in alphabetical order. For example, “Telma Sonhi, twenty-one years old, Miss Angola!” followed by, “Ann-Marie Brown, twenty years old, Miss Antigua & Barbuda” all the way to Miss Vietnam. Shortly after the show open rolled the Television Director, Technical Director, Associate Director and Producers started calling out that there was no audio! The A1 had faded up the correct pots. One billion people at home could indeed hear the introductions, but there was no sound coming out of the control room speakers! But the producers did not know the problem was only in the control room! And besides, doing a live international entertainment special without being able to hear is no one’s idea of fun.
I ran into the Engineering room and told our Engineer in Charge what happened. He had been one of those that only had three and a half hours of sleep. He quickly checked our transmission feed and net return from NBC and concluded we were good going out to the world and the audio problem was local to the control room. We then went into the audio room and concluded that the router 911 salvo had switched over all of the out-bound sends, but had not switched over the control room speakers. The control room was still routed to receive audio from the back-up audio console! Since the A1 had no faders up on the back-up console, the control room could not hear.
Once realizing what was wrong it was a relatively easy fix to get the control room sound again, but nerves were now frayed and we still had another one hour and fifty-five minutes of live TV programming to do.
My cell phone then started ringing. It was NBC New York. I thought they were calling to confirm that our transmission to them was good but instead they said, “We think you have a problem with your Profile” (a hard drive video playback device). “Why?” I asked. “Well, we’ve seen Miss Germany three times already.” Uh-oh, that can’t be good. I knew that on the pre-taped show intro package each delegate introduced herself once, not three times!
I told the Engineer in Charge what NBC said and he went back to his router panel and started punching buttons, followed by a loud “Oh Sh*t!”
“What happened?” I asked. It seems that when I came to him a few minutes earlier regarding the control room speakers, when he checked the transmission feeds on his router, he accidentally sent NBC’s “net-return” (we were receiving from NBC) accidentally back to NBC on our outgoing transmission path.
We essentially created a perpetual video loop between Ecuador and New York. Miss Germany played out of the Profile, was transmitted to NBC, a few seconds later it was received by NBC’s Network Operations Center (which then fed that signal to 30 Rockefeller Plaza for commercial integration, then out to the network affiliates) and transmitted back to us in Ecuador as our “confidence” we were actually on the air. We inadvertently routed that same signal back to NBC. Miss Germany would have appeared to introduce herself for the next two hours if we did not fix the issue!
All else seemed to settle down until it came time for our guest musical artist to perform. The latin pop star Gloria Estefan waited just off-stage for her intro and background music track to start. The musicians on stage would be “strum-syncing” along with the backing track but the vocalists would be performing and would be heard live.
After the Miss Germany problem earlier in the show and not fully understanding that the Miss Germany problem was caused as a result of human “pilot-error” miss-routing, the TV A1 decided that he did not trust the Profile to playback Gloria’s background track and made the decision instead to have his assistant play Gloria’s track off a computer. Just before the track was to roll there was confusion on PL between the TV A1, the PA FOH mixer and the Stage Monitor mixer as to where the soundtrack would actually originate from and what tie lines would carry it.
Billy Bush intro’d Gloria Estefan and as she walked on-stage her background track could be heard on-air in viewer’s homes, and in the video control room, and through the PA system for the audience in the venue, but not through the stage audio monitors or in-ear monitors. To the viewers at home, all one billion of them, Gloria and the band just stood there waiting for the music to start while everyone at home heard what the band, the vocalists and Gloria could not hear.
So, what lessons can be learned?
My first lesson was that no matter how capable the technical facilities I design are, unless the operators are well rested and comfortable with their primary and back-up systems, failure is inevitable. It was my responsibility to protect them from being over-extended and in that responsibility I failed.
Second: Crash and burn failures are most often not the result of a single major bad decision or equipment malfunction, but is rather the culmination of many smaller or minor poor decisions. In my case, we had planned for the complete loss of a transmission path, for loss of a camera chain or a microphone, we had even planned for the loss of the video production switcher or an audio console … but we hadn’t planned for the loss of a control room speaker or human error.
Third: To be effective, a back-up contingency plan must be comprehensive and rehearsed. Yes, we did run the “911 salvo” a few times before the show. In hindsight, it should have been run daily, or at least often enough that everyone knew and trusted the plan. A plan that does not have the participant’s trust invites deviation. In my case it was a crew member’s last minute deviation that caused Gloria to be on-stage, live on TV without her backing soundtrack.
Fourth: Failure reminds us of our own weakness. Awareness of our own weakness results in true humility. The ability to handle failure in all its various forms is a vital part of spiritual life and a sign of maturity. The Bible is brutally honest regarding some of the greatest figures of Scripture experiencing failures. However these failures often led later in life to more effective service.
Fifth: After a failure, leadership is likely far less concerned about what technically caused the screw up than they are concerned that you as a technical artist take ownership of the problem and resolve to make changes to reduce the likelihood of a similar problem occurring again.
After my Miss Universe debacle I had to write a report to the executives at NBC and the Miss Universe Organization detailing what happened. I did so, attempting earnestly to just state facts and not throw any of the technicians or engineers under the bus. A few weeks later a follow up meeting was planned with the President of the company I work for, the executive who requested we do the NBC morning show live-shot, and myself. I’ll admit, I was nervous.
After some chit-chat the executive took out my ten page report and said, “OK, summarize for me what happened?” I looked over at my boss, then looked at the executive and said, “I was in charge. I screwed up. It was my fault,” and left it there.
My boss looked at me with shock that I chose to give such a decidedly short and “non-Apprentice” contestant answer. There was silence for what seemed like forever. The executive then put away the report and said, “then I guess there are no further questions.” We went back to small talk.
Some would say I dived on the sword. I would rather say that I communicated that regardless of what technically failed, it happened on my watch and therefore I was responsible. Period.
Epilogue: I continue to do work for the Miss Universe Organization. In fact, I’ve worked on the vast majority of Miss Teen USA, Miss USA and Miss Universe Pageant telecasts since 1999. I also continue to do a large amount of work for NBC.
The original version of this article was published November 7, 2013. Title photo attribution – Drake Goodman