So this week, I've got a couple posts looking back on the predictions. This is all probably not very good for my credibility, but I'm hoping that holding myself to these numbers will both motivate me to improve as well as give me some pointers on how to improve.
We'll start with all the many Best Case/Worst Case predictions generated over the course of the season. I tried to weed out burn-off numbers that would've deflated the averages, though I may not have gotten all of them. I'll start with some highlights and lowlights of the BC/WC seasons, and then I'll look at some of the larger statistics.
Best Picks:
| Show | Best | Worst | Likeliest | Actual | %Diff |
| Community | 1.40 | 1.05 | 1.24 | 1.24 | -0% |
| Grey's Anatomy | 3.50 | 2.74 | 3.10 | 3.10 | -0% |
| Dancing with the Stars Mon Spring | 2.71 | 2.05 | 2.30 | 2.29 | -0% |
| How to Live | 2.45 | 1.45 | 1.90 | 1.91 | +0% |
| Survivor Spring | 2.69 | 2.41 | 2.55 | 2.56 | +1% |
| CSI: NY | 1.50 | 1.30 | 1.40 | 1.41 | +1% |
| The Amazing Race Spring | 2.56 | 2.23 | 2.33 | 2.35 | +1% |
| The Big Bang Theory | 5.70 | 4.80 | 5.20 | 5.25 | +1% |
| Last Resort | 2.50 | 1.13 | 1.40 | 1.38 | -2% |
| Castle | 2.24 | 1.85 | 2.08 | 2.04 | -2% |
| Sports: Sunday Night Football | 8.78 | 7.52 | 8.00 | 8.16 | +2% |
| Family Guy | 2.94 | 2.58 | 2.76 | 2.70 | -2% |
| Dateline Fri | 1.42 | 1.20 | 1.32 | 1.29 | -2% |
| Bones | 2.22 | 1.91 | 2.09 | 2.14 | +2% |
These are all the picks within 2% of the mark, with another 11 within 3%. Community and Grey's Anatomy were my two picks nailed to the hundredth. For some reason I have been the Grey's Whisperer, as I was just 0.07 off in 2011 and just 0.01 off in 2010.
The best picks of this bunch include TBBT and Sunday Night Football, both shows with huge numbers and somewhat unpredictable outcomes that I was very close to nailing; TBBT continued its improbable raw numbers growth, but only to a small extent, while SNF finally dropped after years of raw growth. One of my best new show picks was Last Resort, one of the few shows I had a really strong opinion on coming into the season. And credit to me for making a strong adjustment on the late spring reality shows; was very close on Survivor, The Amazing Race and DWTS.
Worst Picks to the Negative:
| Show | Best | Worst | Likeliest | Actual | %Diff |
| The Mindy Project | 3.10 | 2.00 | 2.82 | 1.56 | -45% |
| The Mob Doctor | 1.90 | 1.20 | 1.50 | 0.90 | -40% |
| Smash | 1.84 | 1.27 | 1.50 | 0.94 | -37% |
| Vegas | 3.08 | 2.00 | 2.60 | 1.68 | -35% |
| Suburgatory | 3.64 | 2.71 | 3.25 | 2.14 | -34% |
| 90210 | 0.63 | 0.50 | 0.56 | 0.37 | -33% |
| The Job | 1.65 | 1.05 | 1.20 | 0.80 | -33% |
| Red Widow | 1.90 | 1.20 | 1.65 | 1.10 | -33% |
| The New Normal | 2.70 | 1.55 | 2.30 | 1.55 | -32% |
| Made in Jersey | 1.62 | 1.30 | 1.39 | 0.95 | -32% |
| Do No Harm | 1.60 | 1.00 | 1.15 | 0.80 | -30% |
These are all the shows that did at least 30% worse than expected. Not surprisingly, most of these are new shows, which are so much more difficult to gauge because there is no raw data whatsoever going in. For me, the toughest part of gauging new shows is that even if you get the end result right, it's tough to nail the extent; I thought shows like The Mob Doctor, Do No Harm and even returnees like Smash and 90210 were going to fail, and they did fail, it's just the numbers were a lot more ugly. I've observed this difficulty even in looking at the ad rates. Advertising rates suggest almost nothing gets pegged as a total meltdown out of the gate, and yet many of those exist every year. Maybe the lesson of Last Resort and of these shows is that I should have even stronger opinions about the shows I'm not high on.
Still, there were several big whiffs even in terms of outcome. I thought The Mindy Project and Vegas would be legit successes, and I really thought Suburgatory would benefit from the Modern Family lead-in.
Worst Picks to the Positive:
| Show | Best | Worst | Likeliest | Actual | %Diff |
| Chicago Fire | 2.00 | 1.19 | 1.40 | 1.83 | +30% |
| The Voice Tue Fall | 3.90 | 3.00 | 3.20 | 4.08 | +27% |
| Scandal | 2.48 | 1.60 | 2.01 | 2.50 | +24% |
| Family Tools | 1.35 | 0.65 | 0.90 | 1.12 | +24% |
| America's Next Top Model Fall | 0.45 | 0.35 | 0.41 | 0.51 | +24% |
| The Biggest Loser Spring | 2.40 | 1.64 | 1.90 | 2.32 | +22% |
| The Bachelor Spring | 2.57 | 2.03 | 2.26 | 2.75 | +22% |
| The Voice Mon Fall | 4.40 | 3.40 | 3.66 | 4.38 | +20% |
| The Taste | 1.91 | 1.15 | 1.35 | 1.61 | +19% |
| The Following | 2.75 | 1.30 | 2.30 | 2.63 | +14% |
| Golden Boy | 2.00 | 1.05 | 1.25 | 1.42 | +13% |
| Revolution | 3.30 | 1.60 | 2.27 | 2.57 | +13% |
| The Vampire Diaries | 1.19 | 1.02 | 1.11 | 1.24 | +12% |
| Supernatural | 0.95 | 0.73 | 0.84 | 0.92 | +10% |
| Grimm | 1.61 | 1.30 | 1.45 | 1.59 | +10% |
| Rock Center | 1.16 | 0.75 | 0.93 | 1.02 | +10% |
These are all the shows that did at least 10% better than expected. Once again, there's a disproportionate volume of new shows, with Chicago Fire the biggest overachiever. This is a fun way to try to get a gauge on who had the "best" season on TV. Scandal is certainly a show many would agree is in contention for that title.
Worst Returning Show Picks:
| Show | Best | Worst | Likeliest | Actual | %Diff |
| Smash | 1.84 | 1.27 | 1.50 | 0.94 | -37% |
| Suburgatory | 3.64 | 2.71 | 3.25 | 2.14 | -34% |
| 90210 | 0.63 | 0.50 | 0.56 | 0.37 | -33% |
| New Girl | 3.71 | 2.58 | 3.23 | 2.30 | -29% |
| Private Practice | 2.10 | 1.70 | 1.84 | 1.33 | -28% |
| The Voice Tue Fall | 3.90 | 3.00 | 3.20 | 4.08 | +27% |
| Apartment 23 | 1.90 | 1.40 | 1.56 | 1.16 | -26% |
| Happy Endings | 2.30 | 1.49 | 1.76 | 1.33 | -24% |
| Scandal | 2.48 | 1.60 | 2.01 | 2.50 | +24% |
| America's Next Top Model Fall | 0.45 | 0.35 | 0.41 | 0.51 | +24% |
| The Mentalist | 2.48 | 1.96 | 2.25 | 1.72 | -23% |
| Touch | 1.20 | 0.80 | 0.90 | 0.70 | -22% |
| Dancing with the Stars Mon Fall | 3.12 | 2.50 | 2.90 | 2.26 | -22% |
| The Biggest Loser Spring | 2.40 | 1.64 | 1.90 | 2.32 | +22% |
| The Bachelor Spring | 2.57 | 2.03 | 2.26 | 2.75 | +22% |
Weeding out the always highly unpredictable new shows, here were the biggest whiffs by percentage. It was a season marked by some very disappointing sophomore seasons, most notably Suburgatory and New Girl, but also 2 Broke Girls which was just off the list. I expected much more out of these those shows, but the few actual "sophomore bounces" were elsewhere (Scandal and to a lesser extent Grimm and Person of Interest).
Big Picture Stats:
88 of 126 "Likeliest" picks this season were too low (70%), 36 were too high, and two were nailed to the hundredth.
The median miss size (in other words, the absolute value of the percent difference) was 13%.
The median miss (in other words, lining up all of the %Diffs from +30% to -48%) was -9%.
In case it is not yet obvious, I was pretty consistently too optimistic this season. It seems almost everyone who predicts ratings suffers from that, but it should be a problem I can lessen in future seasons with a more developed sense about the collective entertainment average decline.
It is worth noting, in my defense, that even a more perfect process probably would've been too optimistic, since this year saw a larger collective decline than usual. You could argue the league average was up to 5% lower than expected, which makes up over half of that median -9% miss. Adding 5% to every show's ratings would change the too low/too high balance to 72 too low (57%) and 54 too high.
But even those "adjusted" numbers would've been too optimistic, and honestly the adjustment was probably too much. The league average was probably really only about 3% lower than expected. Bottom line: I should have been more pessimistic.
It's hard to know how good of a job I actually did since there aren't many other people out there doing this exact thing. I suppose I could create some kind of "control group" where I just predict a 10% drop for everything and compare my picks with that. Maybe I'll do that at some point! But mostly, my goal for 2013-14 is to do better than in 2012-13.
The most distressing stat to me was the number of shows that went below the "Worst Case" pick. A whooping 44 picks did that, or 35% of the total picks!!! (Plus another five went above the "Best Case.") And that wasn't even all about having tons of horrendous picks. About a third of those 44 were on picks were the Likeliest missed by 15% or less. There was something like 48 Hours, which averaged a 1.00, just 9% off my 1.10 pick, but my BC/WC range was 1.15 to 1.04. Ultimately, I probably just shouldn't get so precise with the outer ranges. Even though 48 Hours and the other newsmagazine-type shows usually have fairly predictable ratings, I can't imagine there's ever a show where the best case and the worst case are just 0.11 points apart.
Here's a link to the full list of predictions/results.
11 comments:
I think you did a good job overall. I don't think you should alter the system by just predicting a 10% drop for everything, because I (and I assume other readers as well) enjoy looking at the best/worst/most likely predictions.
I used to be an optimist… and then I got sucker-punched in the face by this fall's ratings. Now, I'm usually pessimistic. Fun fact: I have the lowest %Over in The Question out of all the serious players.
Of course, it bites me sometimes. Take yesterday, when I predicted Under the Dome would get half the ratings points it got. ;)
I think the Best Case/Worst Case hit the mark pretty well, especially because it hasn't been used as widely before. You're not that optimistic. If you want to see too optimistic, you should see the polls over at TV by the Numbers. The crazy people over there vote for the highest option every time (What is it about that site that always attracts the crazies?). Anyway, the BC/WC worked pretty well, and I look forward to it improving next year.
Out of all the amazing features of your website, the BC/WC is probably my favorite t read and I go back to it quite often during the season. It really makes me admire you more for having the guts to do the accountability at the end of the season. I think you did very well. I think that if you want to get into more technical analysis, one thing that would help you with returning shows would be to look more at late season data for returning shows when coming up with the percentage decline. That is something I felt since the beginning when reading your New Girl's prediction for instance. In the first half of the its first season, it averaged a 3.7. On the second half, a 2.7. This year, it averaged a 2.30. So although your prediction of it being even would still be off, it would be way less off than predicting it to stay even with the entire first season, which clearly had too much inflated results from early on. For instance, how useful are Scandal's fall ratings from this season useful to predict its future season results? With new shows it is clearly more difficult, and I am very respectful of you for having the guts to do it, the only thing I do is try to guess renewals and cancellations and even on that I fail quite often!Keep it up!
I agree. I wouldn't call it my favorite feature (that would probably be A18-49+ for me), but I do refer back to it frequently during the season.
Yeah, I also love that one. Especially the "Now" thing (it is the same information, it is just more easy for my lazy brain to process)
Thirded about the feature.
The interesting part is that the prediction are thoughtful and rational, not their accuracy.
Exactly!
Well put, that's exactly it!
I didn't mean that I would start doing that in BC/WC, just that I wanted to quickly throw those together as something to compare against, to see if my "instincts" are adding any value.
By the way, I didn't put this in the post, but I lined up a bunch of the 2012 predictions at ShowBuzzDaily.com, which were just for fall ratings (so I took originals only through 12/31/12). They were optimistic more often (~75% of the time vs. my 70%) and their median miss size was 16.4% vs. my 12.9%, while their median miss was -14% vs. my -9%. So I guess I did better than someone, at least!
That is good advice, and I generally do that, which is why I was correctly down on shows like HIMYM and 2.5 Men that were inflated by the fall results. I just thought New Girl had more momentum and would be helped by the new shows around it (both of which were among my favorites of the season).
Yeah that makes sense. I also thought New Girl would do better than it did on a vacuum. However, seeing the performance of the shows surrounding it and of NBC's fueled the voice tuesdays for at least the fall, its numbers are not as bad as they would seem pre-season (together with the overall league average decline of course)
Post a Comment