Were the Tiers Well Calibrated?
| Tier | Average R% | Actual Renewal Rate | Renewals/Total |
|---|---|---|---|
| Dark Red (0-19%) | 9% | 25% | 2/8 |
| Light Red (20-39%) | 29% | 50% | 3/6 |
| White (40-59%) | 53% | 50% | 5/10 |
| Light Green (60-79%) | 70% | 64% | 7/11 |
| Dark Red (80-100%) | 95% | 95% | 39/41 |
At first glance, this looks like a much bigger year for reach renewals than last year. There was at least one more renewal than expected in both the dark red and light red tiers. I would point out a couple shows like Bless the Harts (dark red) and The Blacklist (light red) that are graded in their respective tiers based on renewal timing but spent very little of their actual seasons there, which might throw the numbers a bit off kilter. (But there's also Zoey's Extraordinary Playlist, which was dark red all season and barely snuck into light red at the end.)
The last three tiers (where the nets were extremely generous last year) were much better calibrated. The bubble was truly a bubble and there were about the right amount of cancellations in the light green and dark green tiers. We even got a freak cancellation in the very dark green territory (Hawaii Five-0) after not having one for a couple years. As I said in past years, it was probably overdue, and I'm just glad to have it at 97% and not 100%. That three point difference is supposed to mean something, and we saw it this year!
One thing that was nice to see this year was that the overall picture was much better calibrated, following a 2018-19 in which the misses were mostly in one direction. Last year we basically had three networks that were much more generous than expected, leading to about a six-show gap between expected and actual renewals. This year, that was greatly improved; the sum of R% at the time of decision was 54.25 renewals, and the networks renewed 56 shows. If Katy Keene gets cancelled, that would mean we were only off by one renewal across the five networks (great!), and if she gets renewed it would be a little over two. Pretty good either way.
Results By Network
| Net | Sum of R% | Renewals |
|---|---|---|
| ABC | 11.05 | 11 |
| CBS | 18.51 | 16 |
| NBC | 8.02 | 10 |
| Fox | 6.95 | 9 |
| CW | 9.71 | 10 |
ABC: For the second year in a row, the model did an incredible job of setting the renewal bar for ABC. The alphabet renewed exactly the number of shows expected. What was a little more questionable was the relative standing of the shows; ideally, Renewology probably would've wanted one more renewal out of the glut of comedies in the 50-60% range and a cancellation of Stumptown.
CBS: I've talked before about the two ways to grade this model: the "bar" (total number of shows renewed) and the "curve" (the relative standing of shows). ABC did great with the bar but a bit shakier with the curve. CBS was much more of a bar problem, as the network reversed course from last year (when they were generous) and cancelled 2-3 more shows than expected. Of course, some of this is explained by freakish incidents like the cancellations of Hawaii Five-0 and (to a much lesser extent) Man with a Plan. But CBS was harsher than expected even if you completely take those out of the equation.
It wasn't really a "curve" problem; other than Five-0, the cancellations were all out of a fairly closely bunched tier of the nine weakest shows on the network. The problem was that this tier was way too high across the board: of the eight shows from The Unicorn through Man with a Plan, we expected a 73% renewal rate (most likely six renewals) and it should've been far closer to 50% (as only four survived).
My main theory on this is that the loss of The Big Bang Theory depressed the network's average, but CBS did not really depress its renewal standards to match. So we may need to adjust CBS' network generosity to account for that.
NBC: If there's one consistency year after year, it's that NBC renews more shows than expected. I keep jacking up NBC's expected generosity, and they keep taking it to another level. This year, the main culprit was Zoey's Extraordinary Playlist, then (as explained above) we got a little unlucky that they renewed The Blacklist at one of its relative low points, then they renewed two of the three bubble shows (Manifest, Good Girls, Lincoln Rhyme: Hunt for the Bone Collector) and didn't touch anything in the green.
Fox: The model did pretty well with Fox... if you throw out the animated newbies. Maybe it was a little unlucky that Bless the Harts got renewed after week three, because it was never in the dark red after that point and actually improved all the way to renewal favorite territory after its last couple post-football episodes. But there's no such consolation with Duncanville, which was in the teens throughout. I think this was a pretty hard one to see coming. They have no DVRing, so looking at delayed numbers doesn't explain it. In what small sample size we have for Fox animation, there is very little history of the network going out of their way to make reach renewals, much less two in one season. It's hard for me to envision a version of the model, with the info available prior to this season, in which these wouldn't have been big misses. Shrug.
CW: Basically everything was listed as a renewal lock and everything got renewed. Yawn. Throw in the still-undecided Katy Keene (currently listed at 65%) and you're talking about 10.4 renewals projected, and we're going to end up at either 10 or 11. Hard to do much better than that, especially considering (like with All American last year) it seems that there was at least some chance of a Katy cancel, based on the decision coming this late. So whatever the result, I feel all right with asserting that it was not a dark green type of show. (And incidentally, this is a show where not having the un-roundeds makes a real difference. If we'd known it was getting very high 0.1's in the early part of the run, followed by south of 0.10 at the end, my guess is it'd probably have been rated much more consistently around 55-60%ish.) From a modeling perspective, I'm kind of rooting for a cancellation, just because I think it would help justify this model's reason for existing with respect to the CW. It'd be nice to know they still can cancel something.
Final Thoughts
I am consistent every year (even when I do well) in mentioning that simple "win-loss record" is a bad way to judge something like this that puts out percentages. Evaluating calibration is much more important. That being said, this was a relatively bad year. The final record was 60-16 (79%). I've always said I need a little luck to be competitive with the people who are including many factors beyond ratings, and the luck ran out this time. All three of the NBC bubble shows mentioned above were "wins" for the majority of their seasons and slipped into "loss" territory in their final weeks, and there was really only one example in the other direction (Deputy). I was only 5-5 on the ten bubble shows, barely above .500 (15-12) combining light red, white and light green, and there were also more losses than usual in the dark red/dark green territory. A lot of this I view as reversion to the mean; we had a couple lucky seasons and this one was maybe a little unlucky.
I did not apply any sort of theory as to how COVID-19 would affect these decisions, and considering the model ended up very close to the mark in terms of total big five renewals, with misses in both directions, that was probably a good decision. If I had to evaluate after the fact, I would be more inclined to say that it was a better year than usual for the "conventional wisdom" and the "internal favorites." You could say that this is just part of a progression over time of ratings mattering less and less, but my guess is that the general craziness in the country exacerbated it a bit this year. This model has usually had some value even when compared against people who are just regurgitating the insiders' bubble watch articles, but I don't think that was really the case this year. Perhaps we saw more networks throwing up their hands, considering the whole ratings landscape a bit of a wash, and renewing the stuff they liked.
For years I've been trying to evaluate whether it's worth going through all the trouble of adding a Renewology component based on actual DVR ratings. This has to clear a couple of bars: 1) Will it actually improve the results? and 2) Will it improve the results enough to be worth going through all the logistical hoops?
In past years, there have always been counter-examples where using DVR ratings would hurt the result, perhaps most famously Man with a Plan (a very lightly DVRed show that was a narrow favorite even with the True formula, and almost certainly would've been a "loss" after DVRing). While MWAP got the axe this year, ironically after it finally scored high in Renewology, there were still some counter-examples this year; the big ones that spring to mind are CBS' All Rise and the much-aforementioned Fox cartoons.
This might be the year that finally tipped me to being fully convinced on #1. But clearing the bar for question #2 is not as easy as you might think, because the True formula already makes a guess at how difficult timeslots are. Particularly comparing across hours, that translates pretty well in a broad sense to how much DVRing occurs. So, as an example, I'm not sure a DVR component would help that much with something like Good Girls, because it's already judged to be in a brutal timeslot. It definitely wouldn't explain most of the freak misses this year, like Five-0, Zoey, or the Fox cartoons. But it seems like there are a lot of shows along the margins where there would be value added. It probably would've helped some with Stumptown, but even more with stuff like Manifest and Prodigal Son that are judged to be in "easy" timeslots yet still have good DVRing. There were also shows in "hard" timeslots that under-indexed in DVRing, like Tommy and The Baker and the Beauty.
If I actually put something like this together, it would probably be less about overhauling the True formula (which seems like way too much trouble at this point) and more about something separate. Maybe I could translate what was once called "Sitch" (a measurement of timeslot difficulty) into a guess for how much Live+7 would result, then make a simple adjustment to the True number once the actual Live+7 comes in. I'm not making any promises, but that's just an idea for something that could happen. Thanks for reading!
No comments:
Post a Comment