On Total QBR And More (8/16/11)

By Paul Bessire

Tuesday, August 16 at 11:47 PM ET
With the NFL Preview published and having had a great day with the baseball picks yesterday (actually the last two days by the time this was published), it's a great day to do a blog. The timing of the baseball performance is a bit of a fluke, but after analyzing and scrutinizing 32 NFL rosters in a general state of 'round the clock flux for the last three weeks, I'm happy to take a breath, bring up a few, short (at least for me) thoughts I've come across in the last couple of weeks and breakdown the baseball performance thus far in August... And then, it will be back to it. With 152 NFL and FBS rosters to maintain (not to mention preparing for leagues that may or may not begin in October), there's always something to do. In fact, for those who have access to our College Football O/U Win Total Picks, check out the site tomorrow for updates on every team with the most recent lines and appropriate rosters. Below you will find my thoughts on the Houston Texans vs. Indianapolis Colts, Michael Vick, "Total QBR," August MLB performance and (like one sentence on) the economy.

"Total QBR":
I am not someone who enjoys poking fun at or putting down ESPN. Too many people in the media outside of the network seem to be the opposite. I have great respect for the vast majority of people who work there - many of whom I work with and some of whom I would call friends. I also appreciate the advances, most of them in a positive/appropriate manner, that the organization has made to integrate intelligent analysis in its TV, online, radio and print coverage. And I fully understand that it would be nearly impossible to do my job (and to enjoy it) without the existence of an entity like ESPN... However...

With so many smart and respectable in-house analysts - like Dean Oliver, who wrote the book, Basketball on Paper, that served as the foundation of my Master's thesis - and external partners, I have no idea what the organization is trying to accomplish with the creation and purveyance of the new "Total Quarterback Rating." My only rationalization is that, while attempting to integrate analytics with and appeal more to the "talent" that played on the field - maybe my only consistent criticism of ESPN is its reliance on previous players who are often clueless to what they are actually discussing - the entity designed the only possible system that can justify Trent Dilfer's existence as an NFL quarterback and analyst beyond "the guy who didn't screw things up for the 2000 Baltimore Ravens' defense."

ESPN is choosing not to disclose the proprietary formula. That's totally cool with me; we do it with most of our algorithms too. There are business reasons to do that. In our case, however, we can let our prediction accuracy speak to the benefit of the algorithms. ESPN cannot. Instead, it allows itself to continuously adjust the inputs so the results match what it perceives is the public perception of which quarterbacks are best. (The "our formula is right because it says player X is awesome and everyone knows he is" justification will never fly with me.) In lieu of the actual formula, we get an explanation from Oliver whom I presume was forced to write this because he waffles a bit between the "Total QBR" end-all-be-all terminology and alluding to the fact that this is a total work in progress. You can read through the full article as you wish. Here are two "catastrophic failures" that jumped out at me from this "total" formula:


  • Opponent Is Ignored: I could not disagree with this decision more. As I tweeted, "Any "total rating" in team sports that ignores opponent is almost completely irrelevant. #anotherworthlessQBRating" I firmly believe that the single most important reason why we are successful at predicting game outcomes is not the simulation engine but the way in which we strength-of-schedule adjust team and player data. Context is absolutely critical to evaluating any player or team. Does anyone believe that the Green Bay Packers and Kansas City Chiefs were equally as good in 2010 because they had the same 10-6 record? I would hope not. The Chiefs played one of the easiest schedules we have ever seen and were incredibly lucky (partially because San Diego was incredibly unlucky). Green Bay overcame a much more difficult schedule and a ridiculous amount of bad luck. True analysis should be able to use this context to appropriately evaluate the difference between these two teams.
  • Total QBR Is Heavily Reliant On "Clutch": What does "clutch" mean? Teams and players are likely to win games because they are better than their opponents at using their strengths to exploit opponents' weaknesses. They win or lose games based on how well they live up to expectations in that specific iteration of the game. How that happens is important. When that happens is immaterial. This is further confounded by the fact that opponent is ignored. If the Green Bay Packers are favored by 17 points at home against the Carolina Panthers and Aaron Rodgers throws five first half, uncatchable (by his receivers) interceptions on first or second down at the Packers 40 yard-line, only to lead a last second comeback to win the game 7-6 on a 90-yard Hail Mary that bounces off of all 11 defenders as it travels down the field, only to be caught in the endzone by a hustling Derek Sherrod, who was just trying to force a fumble after another interception, Aaron Rodgers did not have a good day.


For what it's worth, I have been interested in the referenced win probability work done by Brian Burke at AdvancedNFLStats.com, but a) the explanation does not make it sound as if this work is appropriately applied to "clutch" and b) I still vastly prefer our version of win probability added that considers the opponent by simulating the game in real-time from the culmination of every play until the end of the game to truly determine the value in win percentage added (or subtracted) from the previous play.

Houston Texans vs. Indianapolis Colts:
Since we just published the NFL Preview yesterday, I'm going to use this space to address a couple of the observations for which we have already received the most feedback. One, we already outlined in the article, while the other, we have not touched. The item that I have already noted, yet want to revisit is that we have the Houston Texans in the playoffs as the most likely AFC South division winner, while the Indianapolis Colts, who have made the playoffs the last nine straight seasons, are not in.

Houston is not better than Indianapolis. In our Preseason Power Rankings, the Colts rank three spots higher than the Texans. However, according to our efficiency numbers, the Texans should have scored 29.7 points-per-game last season and allowed 27.1, which should have resulted in about 8.7 wins (against a league-average schedule). Using similar analysis, Indianapolis should have scored 26.9 points-per-game and allowed 24.3, which should have resulted in 8.9 wins. Houston appears to be a little better in 2011, while Indianapolis is about the same (if not just a little worse overall).

Since it was already that close despite their records, the strength-of-schedule advantage Houston gains by playing a third place AFC South schedule (getting Oakland and Miami) instead of a second place AFC South schedule (facing San Diego and the New York Jets) - Indianapolis takes on Kansas City and New England with its first place AFC South schedule - is enough to win the Texans the division more often than any other team. The biggest loser in this division is Jacksonville Jaguars. Indianapolis plays a first place schedule almost every season. For Jacksonville, due to a fluke Hail Mary victory at home over the Texans, the Jaguars now face a schedule that is even more difficult than the Colts', which projects to the worst record in the league. If Houston had Jacksonville's schedule, the Texans would be looking at another 7-9 or 8-8 season without much chance at the playoffs and the Colts would be a relative lock (again) to win the division.

Michael Vick:
While it seems most fantasy football pundits are jumping all over Michael Vick this season, our Fantasy Projections (over 350 players) rank him sixth amongst quarterbacks and 29th overall. Considering Michael Vick's story and career trajectory, quarterbacking the second best team in the NFL to 11+ wins and accounting for 3,750 yards with 27 total touchdowns is almost hard to believe. Yet, many have already noted that our projections are too LOW for a player who did what he did playing in just 12 games for a new team (as a starter at least) in 2010. As simply as I can put it, it is almost a statistical impossibility (let alone an improbability) for a guy with career numbers of 53.7% completion percentage, 6.6 yards-per-attempt, 4.1% touchdown/pass attempt and 3.0% interception/pass attempt to improve upon a season where he had 62.6% completion percentage, 8.1 yards-per-attempt, 5.6% touchdown/pass attempt and 1.6% interception/pass attempt. Keep in mind that Michael Vick is 31 years old, runs the ball almost seven times a game and has only taken about 80% of his team's quarterback snaps in the six seasons in which he was considered his team's starting quarterback.

August MLB Performance:
On the whole, August has not been as strong as June or July in baseball pick performance. Given how strong those months were, that shouldn't be surprising. However, we have continued our strong performance among "normal" or better plays. Over the last three months, highlighted "normal+" plays in baseball are 36-16 (69%, +$917 for a normal $50 player using play value recommendations). As I write this, I am getting more excited for our new features on the site for this upcoming football season, particularly the Play Analyzer and TrendFinder. It's not that I will abandon blogging during the season, but it will be nice to have a constant log of our performance as well as a means by which all subscribers can always get the most up-to-date pick information possible (watch for emails and/or web videos soon about the Play Analyzer).

Here are the June-August MLB performance figures (return based on play value recommendations for $50 player):

  • 69%, +$917 "Normal" or better Money-Line and O/U plays
  • 54%, +$787 Return for all playable Money-Line plays
  • 54%, +$556 Return for all playable Over/Under plays
  • 54%, +$326 Return for all playable Run-Line plays


Here are the August-only MLB performance figures:

  • 67%, +$206 (8-4) "Normal" or better Money-Line and O/U plays
  • 53%, -$205 Return for all playable Money-Line plays
  • 51%, +$374 Return for all playable Over/Under plays
  • 51%, +$25 Return for all playable Run-Line plays


The Economy:
And lastly, likely because of my finance background and my propensity to use big words - or just because it's just about all anyone outside of the sports world has talked about since the temperatures dipped back down below 90 degrees on the East Coast - several people have asked for my opinion on the economy and stock market. I have one very generalized comment to sum up my thoughts and, stimulate some thoughts on efficiency, teamwork and forests beyond trees. This is also probably the deepest thing that I have written or said publicly since my graduation speech on the "Coke Is It" guy who lived next to me in college (I buried it so people would get this far down the page without immediately falling asleep, leaving the page or hurting their brains)... Every decision any human being makes is designed to maximize perceived net present value - with "value" defined by the individual. The closer perceived net present value comes to actual net present value, the more successful the decision... In other words, as is often the disconnect in politics that creates long-reaching ramifications in many walks of life, what is believed to be perceived is not actually presently nearly as important as what is real. Every decision we make impacts every other decision we make. And, the best solutions are those that yield the greatest net positive results to the most for the longest.

As usual, if you have any of your own comments about this article or suggestions about how to improve the site, please do not hesitate to contact us at any time. We respond to every support contact as quickly as we can (usually within a few hours) and are very amenable to suggestions. I firmly believe that open communication with our customers and user feedback is the best way for us to grow and provide the types of products that will maximize the experience for all. Thank you in advance for your suggestions, comments and questions.