Approximate 9 minute read
Have you ever had a church service or a production “crash and burn?” I don’t mean a small technical hiccup, I mean a major failure! The type where EVERYONE in the audience knows something’s definitely wrong. The type that gets you called into someone’s office for a meeting afterward?
If you asked me what shows I worked on last month I would have a hard time telling you without looking at a schedule. If a show goes well my mind just drags all the details into its cerebral Recycle Bin. Unless I work with that client or at the same venue soon after, most of the details are long lost after the invoice is sent. But if a show had the slightest problem I can remember details with remarkable clarity. The worse a show goes, the better my memory. Why is that?
One phrase sometimes heard on headset during problematic shows is “foam the runway!” In these three short words someone can communicate an intricate nuanced message: “I’m going to land this somehow. It may not be pretty. It’s likely to get bumpy. We may even put it down without all the landing gear in place… so stow your tray tables, fasten your seatbelts low and tight across your laps, put your seat-backs in the upright position and brace for impact!” I’ve had to foam the runway a couple of times and I can remember each like it was yesterday.
My worst crash & burn was in 2004. I was representing the company providing all the TV technical facilities for the Miss Universe Pageant which was to originate from Quito, Ecuador. In that role I was responsible for four Avid edit rooms, a dub station, eleven camera live-live production with satellite transmission to NBC in the United States, screens switching and seven crew members. In addition I was also serving as Technical Manager for the Miss Universe Organization which in that role I was responsible for the technical set-up of all other equipment including projection, PA, lighting, etc. and a crew of about fifty people. Lastly I was also coordinating the technical needs of the international and local Ecuadorian broadcast licensees. I had a lot on my plate.
Not enough pressure?
For the Miss Universe Pageant we typically have a world-wide audience of one billion people. If I made a mistake Id’e be doing it in front of one of the largest worldwide television audiences. To put things in perspective, the 2018 MTV Video Music Awards has an audience of about 5.23 million people in the United States. The 2018 Tony Awards has about 6.3 million viewers in the US. Miss Universe’s worldwide audience is 15 times larger!
We were in Quito for about two weeks and all was going reasonably well. One of biggest challenges was that digital audio consoles were just gaining acceptance in the broadcast TV world. The on-air mix was planned to be done on a Calrec Zeta digital board. The broadcast A1 audio engineer however recently had bad experiences with the Neve Libra Live digital console. So, even though he had no prior poor experience with the Calrec Zeta, he requested that he have a back-up desk set-up on stand-by so if the primary desk failed he could just swivel his chair and continue mixing the live show on the back-up console.
Facilitating a live broadcast TV show with two on-air audio consoles backing each other up is no small feat. Aside from splits which normally occur between Front-of-House and Monitors, just within the broadcast compound there are hundreds of inputs, all in various signal formats that would need to go to both broadcast desks. Even more challenging, main program stereo audio to transmission, international audio to transmission, feeds from the broadcast A1 returning to FOH and the monitor mixers, feeds to control room speakers, the broadcast A1’s own speakers, various PFL speakers, feeds to licensee broadcasters, feeds to edit rooms and upwards of eighteen record VTRs (eight audio channels each) would all need to be fed from either the primary or back-up broadcast audio console within a moment’s notice. The only solution would be a router salvo to send the correct analog, AES and MADI to the correct destination no matter which broadcast audio console the A1 was working from.
The night before the live show an executive approached me and said that the NBC Today Show wanted to have the Miss Universe Pageant host Billy Bush, and the show’s Executive Producer Donald Trump, as guests live from Ecuador during their morning show. The executive asked if I could facilitate using our “main show control room,” a few cameras and audio mix capabilities for the early morning NBC Today Show insert.
Well, equipment-wise we were fine. We had more than enough production and transmission equipment resources to do a morning show live shot. The problem was that the crew had been rehearsing every evening for more than a week in preparation for an evening live broadcast. To also do a morning-show insert meant that we had to get some crew back to the venue by 5:30 AM (plus travel time) and then they would be awake for fourteen and a half hours before we even started our main show. It would be sixteen and a half hours before we’re off the air and twenty-four hours around the clock before we completed the post-show strike. The impact of facilitating the morning show live-hit would be a very long day with some crew members only getting as little as three and a half hours sleep before the morning live-shot and the main broadcast that evening.
Despite the risks and with the intent of being a “can do” person, I said yes. Would you tell Donald Trump he couldn’t appear on the Today Show? … No, I didn’t think so.
The Today Show live-shot with Mr. Trump and Billy Bush went fine. Afterward, in between the morning show live shot and the main Miss Universe telecast we tested our 911 router salvo making sure we could indeed go back and forth between the primary and back-up broadcast audio consoles. After going running the salvo a few times everything seemed ok.
The Miss Universe telecast show opener was a pre-taped package where each delegate, in alphabetical order introduced herself. For example, “Telma Sonhi, twenty-one years old, Miss Angola!” followed by, “Ann-Marie Brown, twenty years old, Miss Antigua & Barbuda” all the way to Miss Vietnam.
Shortly after the show opener rolled the Television Director, Technical Director, Associate Director and Producers started calling out that there was no audio! The broadcast A1 had faded up the correct pots. One billion people at home could indeed hear the introductions, but there was no sound coming out of the control room speakers! What the producers did not know was the problem was only in the control room. Regardless, doing a live international entertainment special without being able to hear is no one’s idea of fun. This had to be fixed immediately!
I ran into the engineering room and told our Engineer in Charge what happened. He was one of those who only had three and a half hours of sleep. He quickly confirmed our transmission feed to NBC as well as the “net return” from NBC back to us and concluded audio was indeed fine on-air therefore the audio problem must be local to our facilities in Quito. We then went into the broadcast audio mix room … and heard sound. The conclusion was the 911 router salvo had switched over all in-bound and out-bound audio sends, but had not switched over the control room speakers. The control room was still routed to receive audio from the back-up audio console! Since the broadcast A1 was mixing on the primary desk and had no faders up on the back-up console, the control room could not hear.
Once realizing what was wrong it was a relatively easy and quick fix to get the control room hearing sound again, but nerves were now frayed and we still had hours of live TV programming to do.
My cell phone then started ringing. It was NBC New York. I thought they were calling to confirm that our transmission to them was good but instead they said, “We think you have a problem with your Profile” (a hard drive video playback device). “Why?” I asked. “Well, we’ve seen Miss Germany three times already” (uh-oh, that can’t be good). I knew that on the pre-taped show opener each delegate introduced herself only once, not three times!
I told the Engineer in Charge what NBC said and he went back to his router panel and started punching buttons, followed by a loud exclamation, “Oh Sh*t!”
“What happened?” I asked. It seems that when I came to him a few minutes earlier regarding the control room speakers when he was checking the outbound transmission feeds on his router he accidentally sent NBC’s “net-return” (that which we were receiving from NBC to confirm we were on the air) accidentally back to NBC on our outgoing transmission path.
We essentially created a perpetual video loop between Ecuador and New York. Miss Germany played out of the Profile, was transmitted from Quito to NBC in New York, then a few seconds later it was received via satellite by NBC’s Network Operations Center (which sent the signal on to commercial integration, and then out to the network affiliates) and simultaneously NBC transmitted the net-return back to us in Ecuador. We inadvertently routed that same signal back to NBC. Not unlike Bill Murray’s character in Groundhog Day, Miss Germany would have appeared to introduce herself over and over and over again (to our one billion viewers) indefinitely if we did not fix the issue!
Once NBC was correctly receiving our PGM (not net-return), all else seemed to settle down until it came time for our guest musical artist to perform. The latin pop star Gloria Estefan waited just off-stage for the show’s host to intro her and the background music track to start playing. Her band on stage would be “strum-syncing” along with the backing track but the lead and background vocals would be performed and would be heard live.
After the control room speaker and Miss Germany problem earlier in the show, and not fully understanding that these were human “pilot-error” failures (not necessarily equipment malfunctions), and not fully trusting the Profile video playback device to play Gloria’s music backing track, the broadcast A1 decided to have his audio assistant playback the music track via a computer in the audio booth instead of coming off of the Profile controlled by the videotape ops as rehearsed. Just before the track was to roll there was confusion on PL between the broadcast A1, the FOH PA mixer and the Monitor mixer as to where the soundtrack would actually originate from and what tie lines would carry it.
Billy Bush introduced Gloria Estefan and she walked on-stage. Her music track could be heard on-air in viewer’s homes, as well as in the video control room, in the audio booth and through the PA system for the audience in the venue, however the Monitor engineer did not not track the audio assistant’s computer and therefore there was no backing track in the stage audio monitors or in-ears the band and Gloria were listening to. To the viewers at home (all one billion of them) Gloria and the band just stood there waiting for the music to start (live on TV) while everyone at home heard what the band, the backup vocalists and Gloria could not hear.
So, what lessons can be learned?
My first lesson was that no matter how capable the technical facilities, unless the operators are well rested and comfortable with their primary and back-up systems, failure is inevitable. It was my responsibility to protect the crew from being over-extended and in that responsibility I failed.
Second: Crash and burn failures are most often not the result of a single bad decision or equipment malfunction but are rather the culmination of many smaller poor decisions. In my case, we had planned for the complete loss of a transmission path, for loss of a camera chain or a microphone; we had even planned for the loss of the video production switcher and the audio console … but we hadn’t planned for the loss of a control room speaker feed, human “pilot error,” or the lack of confidence by a crew member.
Third: To be effective, a back-up contingency plan must be comprehensive and rehearsed. Yes, we did run the “911 salvo” a few times. In hindsight, it should have been run daily, or at least often enough that everyone knew and trusted the plan. A plan that does not have the participant’s trust invites deviation. In my case it was a crew member’s last minute deviation that caused Gloria to be on-stage on TV without hearing her backing soundtrack.
Fourth: Failure reminds us of our own weakness. Awareness of our own weakness, therefore having an objectively accurate view of ourselves results in humility. The Bible is brutally honest in it’s accounts regarding some of the greatest figures in Scripture experiencing failures. These failures often led later in life to more effective service.
Fifth: After a failure, leadership is likely far less concerned with what the technical reasons were that may have caused a screw up than they are concerned that you as a technical artist or manager take ownership of the problem and resolve to make changes to mitigate the likelihood of a similar problem occurring again.
After my 2004 Miss Universe debacle I had to write a report to the executives at NBC and the Miss Universe Organization detailing what happened. I did so, attempting earnestly to just state facts and not throw any of the technicians or engineers under the bus. A few weeks later a follow up meeting was planned with the President of the company I worked for, the executive who requested we do the NBC morning show live-shot, and myself. I’ll admit, I was nervous.
After some chit-chat the executive took out my ten page report and said, “OK, summarize for me what happened?” I looked over at my boss, then looked at the executive and said, “I was in-charge. I screwed up. It was my fault,” and left it there.
My boss looked at me in shock that I chose to give such a decidedly short and “non-Apprentice show contestant” answer. There was silence in the room for what seemed like eternity. The executive then folded and put away the report and said, “well, I guess there are no further questions then.”
We went back to small talk.
Some would say I dived on the sword. I would rather say that I communicated that regardless of what technical equipment or personnel failed, it happened on my watch and therefore I was responsible. Period.
Tom D’Angelo has worked in television production and AVL corporate theater for nearly four decades. He is Emmy Award nominated (Best Director category, Mid-Atlantic) and has been part of various teams nominated or winning national Emmys. As the Media Director at a megachurch in the 1980’s he developed a love for the Church and church performing and technical artists.