Were the Tiers Well Calibrated?
| Tier | Average R% | Actual Renewal Rate | Renewals/Total |
|---|---|---|---|
| Dark Red (0-19%) | 8% | 11% | 1/9 |
| Light Red (20-39%) | 32% | 30% | 3/10 |
| White (40-59%) | 52% | 73% | 8/11 |
| Light Green (60-79%) | 74% | 83% | 10/12 |
| Dark Green (80-100%) | 94% | 100% | 45/45 |
So there are a couple things that I would like to see happen, in an ideal world. The first, simplest hope is that the actual renewal rate increases with every tier. That happened, and I don't think that achievement should be underestimated. It's a good metric for measuring the truly fundamental question: do the ratings really mean anything? Generally, as you go from tier to tier, better ratings meant more likely to get renewed. That's good in terms of justifying the continued existence of something like this.
Beyond that, the other thing I would ideally like to see is that the actual renewal rates fall close to the average R%. Those numbers look quite good in the red tiers, though it's worth noting the only dark red renewal was very close to being light red (Blindspot, the biggest miss of the season at 19%) and all three of the light red renewals (The Orville, Madam Secretary and How to Get Away with Murder) were in the high-30's, very close to becoming toss-ups. If the model were just a bit more generous, they might've snuck into the toss-up zone, leaving us with a near-complete wipeout in the light red tier. Overall (with the caveat that there are still some CBS/NBC shows pending) this looks like a rather light year for reach renewals.
But the broadcasters made up for it by going very generous in all of the next three tiers. On average we would've expected probably two more cancels in the toss-up zone, another one in the light green, and yes, even two or three of the dark greens to get the axe. Running the table in the dark-green tier might seem like a good thing to the casual observer, but it worries me because it will probably make the model even more confident (too confident?) going forward.
Overall, the broadcasters were definitely more generous than we expected. But they went about it in sort of an odd way; usually I would think that means there were a lot of reach renewals, but instead we got this situation where the reaches were axed but the toss-ups and higher were renewed at a crazy rate. There aren't a ton of individual renewals you can point at and say that this is the one bad decision; you almost have to look at them in buckets. Did CBS really need God Friended Me AND Magnum P.I. AND Madam Secretary? Did NBC really need Blindspot AND The Blacklist AND Good Girls? Did the CW really need Dynasty AND All American AND In the Dark?
Results By Network
| Net | Sum of R% | Renewals |
|---|---|---|
| ABC | 14.88 | 15 |
| CBS | 16.12 | 18 |
| NBC | 10.26 | 13 |
| Fox | 8.26 | 8 |
| CW | 11.40 | 13 |
ABC: Renewology has historically found ABC to be harsher than expected, and I think many would've expected them to be harsher this year as well. But this time, we really nailed it, expecting 15 renewals and getting 15. Maybe I just got lucky that they had bad development and/or a management change; they definitely picked up less new product than usual.
But this was good. A particular piece of contrarian vindication for the model was its rather stubborn insistence all season long that Single Parents was about a 70% favorite and Speechless (a show many believed would be saved for syndication reasons) was about a 30% underdog. Maybe it was too generous to consider The Kids Are Alright and Splitting Up Together as toss-ups, but Bless This Mess meant ABC renewed one of its three shows at that level, and ABC cancelled out some of the toss-up carnage with a minor reach renewal for How to Get Away with Murder, a show that I still think was kind of fooling the True formula by being incompatible with A Million Little Things. (I definitely think Murder got weaker in the winter, but I don't think it should've gone from 93% to 38%.)
CBS: CBS has probably been the model's most well-calibrated network historically, but this year they were a good bit more generous than expected. Assuming we can throw in The Code and The Red Line as cancels (upping the sum of R% to 16.4 or 16.5), they renewed about one and a half more shows than expected. Not a great miss, but not a catastrophe. I still maintain it was kind of crazy to renew Magnum P.I. and God Friended Me so early, and then they pretty much ran the table on a huge batch of second-tier shows (Criminal Minds, Bull, MacGyver, SEAL Team, S.W.A.T., Madam Secretary) that all had reasons they could have been cancelled. It avoids being a huge miss for Renewology because they were harsher with comedies, axing two of the three toss-up-ish shows in Fam and Life in Pieces.
NBC: NBC is always more generous than expected, and this year was no exception. They still have four undecided shows hanging out there, which would add another nearly full renewal to their Sum of R%, so it may not be quite as bad as it looks right now, but they'll still have renewed a couple more shows than expected. A big part of the miss this year is tied up in Blindspot, and I can't say I feel too guilty about having that with a low R% considering it got freaking pulled for May Sweeps. Fortunately, since it's heading into an announced final season, that won't be a problem moving forward.
Beyond that, it's just a question of whether they needed to run the table on the toss-up-ish stuff like Brooklyn Nine-Nine, Good Girls and The Blacklist. The first two seem to be big in the streaming world, and Blacklist really looked better in the ratings in the spring, so I can't really argue with any of them individually, but the net sure has a lot of stuff waiting for midseason.
Fox: This was a really good showing for the model in its attempt to quantify a one-time transition year for Fox. The reality was, as expected, somewhere between "normal network" and the all-out scripted apocalypse some were predicting. The light green tier proved to be what it should be: a likely renewal (The Resident and the cartoons) but not a lock (Star). The light red tier proved to be what it should be: a likely cancel (The Passage, The Cool Kids, Lethal Weapon) but not a lock (The Orville). As good as this was, this perception of the network is likely not something that can be applied going forward; now that they have inventory that's more in line with their new scheduling needs, they should act more like a "normal" network in the future.
CW: Kinda the same story as CBS and NBC; in renewing everything, the CW renewed one to two more shows than expected. We would've expected something out of the All American / Dynasty / In the Dark group to get the axe. I'm fine with having All American as something less than a lock; I think if it had always been a 100% lock it would've been in the first batch of early renewals. There was some uncertainty there. But what does bug me here is that maybe I should've done something different with the way I calculate the CBS studio target, since two of their three returnees were in pre-announced final seasons. Maybe I should've had some way of portraying all the CBS shows as locks because the studio had a quota to meet.
I'm not sure how it would work, though, and it's far from a guarantee it would really improve anything; for example, you could argue a similar thing with the CBS network comedies, dialing down their targets since they're losing The Big Bang Theory this year. But doing something like that would've hurt the results, inflating the numbers on Fam and Life in Pieces!
Some Final Thoughts
As I've said a million times, I don't think "win-loss record" is a good way of evaluating these things. I think it should be about how well your confidence is calibrated: what I discussed in the first section. I started this largely because I was sick of people saying things were 95% or 100% to happen because of non-ratings reasons and seeing them be wrong constantly, so I wanted to see if a "ratings-only" approach could do better. In the last two years, we have just two misses (and 107 wins) in the "dark" zones: Kevin Can Wait's cancellation last year (81%) and Blindspot's renewal this year (19%), both of which were just barely in the dark zones. In reality, this probably means the model is underconfident, but maybe some underconfidence is a nice antidote to the overconfidence that largely existed before it.
With all that being said, comparing confidence with "the other guys" is tough to do, since some people use percentages and some use smiley faces and some use a simple thumbs up or thumbs down. So I will just mention the win-loss record; this year, Renewology went 77-10 if you look at each show's R% at the time of renewal. This appears to be pretty good, tying the reaper's number of misses.
I think that Renewology has been pretty lucky with toss-ups; it was 9-2 last year including a "loss" on the exactly 50% show (LA to Vegas was 50.2% in the week it was cancelled). This year it was 7-4 including a "win" on the exactly 50% show (Fam was 49.9% in the week it was cancelled) and a few shows that slipped over 50% just in time to get renewed like Good Girls, In the Dark and God Friended Me (before later slipping back below).
But even if there's been a little luck, I don't feel too ambitious in asserting that this model has proven to be rather competitive with the pundits who are applying a bunch of non-ratings assumptions in their predictions. Considering the pervasive narrative about how ratings are becoming irrelevant to decision-making, that's... interesting. Maybe it means the ratings can still unlock the secrets of decision-making if approached more rigorously, or maybe it just means you should be skeptical of the non-ratings assumptions.
For me, the real question is if there's a hybrid approach that can do even better. After all, out of my 10 misses and the reaper's 10 misses, there were only three shows in common (Blindspot, Star, The Kids Are Alright). Can the best ratings analysis be combined with some non-ratings information to wipe out some of the blindspots (pun intended) that result from just looking at ratings?
It's possible, but it's tricky to know what is actual knowledge and what isn't. I would want to stick with low-hanging fruit that has been proven out for many years and continued to hold up even after it was first hypothesized. I've always believed Madam Secretary qualifies under this banner, as I've said from the very first Renewology post! But how much of the "knowledge" out there would actually improve your record (vs. just looking at ratings) long-term? Speechless and Life in Pieces cancellations don't seem like great news for the "Third Season Rule" and the quest for 88 episodes and syndication. If you want a model that reacts to news developments, that means you have to kill All American when it's not in the first batch of CW renewals, Criminal Minds when it only gets 15 episodes, Blindspot when it's pulled for May Sweeps.
If anything, a safer bet for improvement might actually involve looking at more ratings. Specifically, at delayed viewing ratings. Part of the reason the True formula works is that it makes some assumptions about timeslot difficulty that are largely borne out in the delayed ratings. But those assumptions don't always hold; they're not as good as having the actual numbers. To name just a few examples: having the actual Live+7 would really show you that the gap between The Orville and Star was not as big as the Live+SD numbers indicated. They would more clearly identify The Kids Are Alright as a dud, and probably give How to Get Away with Murder more of a chance. Wouldn't be a perfect approach; it would hurt a show like Man with a Plan, and probably further inflate some high-DVRed cancellations like Designated Survivor. I would have to see if it actually makes a net improvement across a bunch of shows I'm not even thinking about. But that's a possibility for the future: a delayed version of the formula that has access not to DVR ratings as well as the un-rounded Live+SD ratings. One of my biggest worries looking forward is having to do this with ratings to only one decimal point.
No comments:
Post a Comment