Back to IndieWire

Pilot Error: Top TV Critics on Reviewing New Shows in the Binge-Watching Era

Pilot Error: Top TV Critics on Reviewing New Shows in the Binge-Watching Era

In the binge-watching era, it’s become common to think of TV shows as novels — not least among the people who make them. Just as you wouldn’t judge a book by its first few pages, the argument goes, so you can’t evaluate a series after a single episode, or even a handful of them. But with network TV largely sticking to the model of airing episodes one at a time, that’s still how most people judge them, and that includes TV critics, too.

As the broadcast networks roll out their new fall lines, that often means filing a review based on a single pilot episode, usually produced before a network has ordered a show to series. Pilots, which are often shot before a series has the money to build sets or hire all the actors they need, can be startlingly different from the shows they ostensibly represent. Even a diehard “Seinfeld” fan might not find much of worth in the first episode of what was then called “The Seinfeld Chronicles,” with Michael Richards playing a character named “Kessler” and no Elaine in sight.

But while cable networks like FX and HBO now regularly send out a handful of episodes (anywhere from two to six) before a new show airs, broadcast networks still give critics a scant 22 or 44 minutes to pass judgement, which leaves some of the people who make those shows wishing they got the same treatment as their cable brethren. After BuzzFeed’s Kate Aurthur calledDr. Ken” the worst new show of the fall TV season, star/namesake Ken Jeong tweeted, “[D]on’t judge a book by its pilot cover.” 

When I pointed out that ABC had given critics no choice but to judge “Dr. Ken” by its cover, Jeong had another episode sent out, and while it’s not much different than the pilot, it does feature an engaging B-story that sketches the conflicts between first-, second- and third-generation Korean-Americans with a specificity otherwise lacking from the show’s utterly generic proceedings.

Given how little pilots end up resembling the shows they spawn, one might ask why critics bother reviewing them at all. A new study by Netflix suggests that the practice of binge-watching has greatly reduced the importance of that initial chapter to hooking viewers; the tipping point past which viewers end up finishing a season falls, most of the time, between episodes four and eight. (It’s worth pointing out that Netflix’s spin on this data is entirely congruent with their binge-watching brand, and if pilots aren’t the deciding fact in whether casual viewers become completists, that doesn’t mean they’re not important.)

But while streaming, bingeing and DVRing continue to gain ground over traditional viewing methods, the fact remains that much of the audience still watches shows as they air, and a lousy pilot can smother a show in its cradle before it has a chance to get better. As the Hollywood Reporter’s Tim Goodman wrote to the networks last month, “You are not guaranteed a second chance. Nothing has been more true in what is now a completely revamped modern television landscape. You absolutely can’t have a crappy-ass pilot and believe that viewers will forgive your learning curve and stick with you until you get it right.”

Even as the TV landscape shifts, first impressions still matter. So I asked Goodman, along with HitFix’s Alan Sepinwall and newly appointed Variety critic Maureen Ryan, how they deal with the tricky process of evaluating pilot episodes, and whether they’ve adjusted their approach as the way people watch TV has changed.

How do you review a pilot, and has the way you approach them changed over the years?

Maureen Ryan, Variety: I look at a pilot as a test case: It has to explain the premise and introduce the world and the characters, so there’s a good chance it’ll be somewhat different from what will transpire week to week going forward. Given that there are so many boxes a pilot has to check, I might give it a little more leeway than an episode that’s in the middle of a show’s run. All I really ask of a pilot is to get me to want to return.

Tim Goodman, Hollywood Reporter: Everything has completely changed. I got burned a few times way back when (“Flash Forward,” ABC, “The Nine,” CBS); after that I started to have a caveat in reviews. I thought this would change as networks realized sending out pilots alone was not helping them, but it didn’t. Now I mention it each time — every review that’s only a pilot. In the past, I would circle back if a show got better and add a column, expand the review somehow, etc. Not with bad ones, but good ones. Now I only do that in extremely rare instances, for two reasons: 1) It’s not how “real people” watch television. In this super-crowded environment we’re in, they watch once and, if you’re lucky, they like it enough to watch the next one. If it does nothing for them, they never come back. That’s the real world. And that’s a better reflection of what networks have brought on themselves. 2) If they don’t change, why enable them? Sometimes people — and networks — have to learn the hard way that there are no second chances. Certainly not to make a better first impression than your first impression. Solution: Make better pilots.

As for the exceptions, I’ll circle back if the show got exceptionally better and if it is now in the zeitgeist being talked about for its improvement.

Alan Sepinwall, HitFix: It really hasn’t changed that much, other than the fact that I’m more annoyed having to do it now than I was 20 years ago. Now it mostly only happens with the broadcast networks — and even then, we occasionally get two or (praise Jeebus) three episodes in advance — which is in part a reflection on the production schedule of the networks (whereas cable seasons are often in the can before a single episode airs), but also a reflection of how mired in tradition the whole network fall premiere process is.

But I watch the pilot, figure out what I liked and didn’t like about it, and then try to extrapolate that into the larger question of what the series as a whole might be. Does the premise feel sustainable? Do the things I don’t like feel like they can be easily fixed, and were the parts that worked easily repeatable? What is the best possible version of what this show is trying to be, and what is the worst?

A lot of the time, I have to go outside the pilot itself to find answers. How well do I know the work of the creators, for instance, and does their track record indicate I should trust what’s happening here? Are the actors good enough that in time they might be entertaining enough to watch even if the writing doesn’t catch up to them. The “Parks and Rec” pilot, for instance, has a lot of problems, but I knew Daniels and Schur’s work from “The Office” — and had seen Daniels improve that show dramatically after a pilot I really didn’t like — and figured between them and the presence of Amy Poehler, I would give the show a good long leash. It turned out to be one of the best sitcoms ever made.

The TCA press tour can be enormously helpful in this regard. (I actually find it more useful for pilot prognostication than I do as a source of interviews and quotes.) Just listening to how showrunners answer questions about elements of their pilots can offer big clues for the future. If they’re asked about an element I didn’t like and act like it’s their favorite part of the pilot, that’s a trouble sign, but if they agree that it needs work — and, better, can articulate how to fix it — then that offers hope.

A good example I often use was the year that both NBC and Fox had serialized kidnapping dramas on their fall schedules: respectively, “Kidnapped” and” Vanished.” In both cases, the critics asked a bunch of questions about how these shows would function in both the short and long-term, and where the “Kidnapped” creator had smart answers for all of them (including what form the show would have taken in the event of a second season), while the “Vanished” creator seemed mystified by every single one. Both shows were ultimately canceled, but “Kidnapped” was good and “Vanished” wasn’t, for the exact ways that were obvious in those two panels.

To what extent do you limit your comments on a pilot to the episode itself, and how much do you try to look to what the show might be rather than what it might become?

Goodman: This part is very difficult. While I note in the reviews that it’s only the pilot and therefore who the hell knows what will happen next — and by the way, I find that very tiring as a critic to have to do, which is basically forfeiting an opinion because the networks are unable or unwilling to deliver more episodes — I do try to comment on “potential.” If it’s there, it gives me hope. Also, if you have a great cast and a great showrunner but the pilot somehow implodes, I take that into account. It could get better. On the other hand, if a pilot is flat-out horrible — and there were a lot this fall — there’s a level of certainty when you tell people that future episodes are unlikely to improve.

Sepinwall: It’s a mix of both, and even more now than in the past. As you may have heard, it’s Peak TV in America, and people’s time is valuable. So you want to give them a sense of what kind of reward might await them if they start watching this new show, or warn them of how quickly this show with a fun pilot might fly off the rails.

Ryan: If I’m only reviewing the pilot, I try to keep myself to writing only about that, but I often fail. It is very hard not to speculate on what a show might become. That was a big part of my reaction to “Jane the Virgin,” for example — it created such a distinctive tone and world in that first episode, which was magical. If the show could sustain that vision, it would be wonderful. If not, it could have easily been a disaster. Fortunately the latter situation didn’t come about, but it’s funny — I reviewed “Jane the Virgin” pilot and the first episode of “The Affair” together, because they had such distinctive premises. One season turned out great, because it delivered on the promise of the pilot. With “The Affair,” unfortunately, it squandered a lot of what was interesting about its foundational elements by midway through its first season. Ah well.

Have you considered not reviewing pilots at all, given how much and how quickly shows can change? Put more bluntly, what’s the point of reviewing a pilot, especially when fewer people are watching episodes once a week as they air?

Sepinwall: I have occasionally done that, particularly when the pilot leaves me at a loss of anything to say. “Halt and Catch Fire” was the rare recent example of a cable show where we only got one episode in advance, and that first episode gave me almost no sense of what the actual show was, or if I should stick with it beyond an interest in the actors and the AMC of it all. So I didn’t write a proper review of the show until the season was almost done. The last few falls, I’ve reviewed most of the new network shows, but not all of them, and have often just handled them in glorified capsule review form — not just because I only had the pilot, but because those pilots weren’t interesting enough to merit more.

The point of reviewing a pilot is more about the consumer watchdog end of the job, as I alluded to before. People look to me in part to know what new things they should be watching. Binge culture has changed things so that the most valuable thing for some viewers is to hear about a show they should watch (or avoid) when it comes to Netflix, but there are still lots of people looking to sample new shows as they debut, or not.

Ryan: Honestly, it’s hard to review anything off one episode — even two is far better than one, because you get more of a sense of how the show will work when all that premise pressure has lessened. And there’s no real reason anymore networks can’t give us more than one episode — when they don’t these days, I tend to wonder if the network doesn’t believe in the show in question or whether subsequent episodes have problems. Of course, that’s not always the case, but reviewing based on multiple episodes is more often the norm these days. I’m truly grateful for that, but I have to also point out that it’s the broadcast networks that are too often the ones that are stingy with episodes. I think that’s an unfortunate situation, but nobody (except you!) asks my opinion on these things.

Goodman: I believe in reviewing what’s given. I feel a responsibility to review, as does THR, all new scripted shows. I’m not concerned at all with how many people will watch or when they will watch, I’m concerned with the content of that pilot. Often it takes six months to make it, so it theoretically should be an indicator of what the show will be like or at least what they wanted it to be at the start. I think that’s fair game and essential to review. I should also note that I have a policy of not “reviewing” any late night shows on their first night or first weeks, etc. I wait on those, They are different animals than scripted series. And since you have this dialogue going, percentages-wise, VERY few bad pilots shift gears and become very good shows. It does happen, but given the vast volume of what’s produced, not statistically often.

What’s the most “wrong” you’ve ever been about a show based on the pilot, either one that looked like it had nothing to offer and turned out to be great, or that seemed promising and ended up not to be?

Goodman:
I would say the most recent example is “You’re the Worst” on FX. There have been others – mostly pilots I liked that were the best things those series ever did, hence the policy to put all kinds of qualifiers in a review. But in the case of “You’re the Worst,” I thought it was important enough — how much better the show got and how wrong I found myself, because I did really love it after that — to go public and say I was wrong.

Sepinwall: I’m sure there are good examples, but most of what’s coming to mind right now are shows like “BoJack Horseman” or “You’re the Worst” where I saw multiple episodes upfront and still changed my mind later. ABC’s “The Nine” is often cited as the quintessential example of “Great pilot, mediocre show,” but (and I just checked my initial review of it to be sure) I was skeptical from the start if the things that were good about the pilot could work as part of a series.

Ryan: I will always be thankful that a good editor (who had also seen the “Studio 60 on the Sunset Strip” pilot) got me to dial back back my overly gushy first review of that show before it came out. I honestly can’t recall if we got more than one episode of “Studio 60” in advance, but I did think the first episode was really good. Within a few weeks, of course, it had devolved into the train wreck it became. At some point, your shame center burns out as a critic — if it didn’t, the job wouldn’t be doable. We have to take chances all the time when it comes to what we say about a show, because it’s possible to be spectacularly “wrong” on a regular basis. Then again, I like that TV is a moving target, and something that starts out as leaden as “Fringe” did can become entirely enthralling in its second season, for instance. But yeah, there’s far less chance of feeling particularly foolish if we get three or four episodes for review. The greatest feeling is getting an entire season ahead of time; even six episodes is a very nice number to have. Then you can make far more firm and decided comments and critiques, which, for me as a writer, is a real relief. I just hate writing wishy-washy reviews that could be summed up as, “Well, maybe it’ll be good — or maybe it won’t be.” I can’t imagine many people like reading that kind of thing either.

This Article is related to: Television and tagged , , , , , , , , , ,