Few new “themes” or trends have emerged in the last week (and I refuse to acknowledge what happened last night), so I felt it was useful to review some of the metrics we use, why we use them, and the pitfalls of relying too much on any one metric… or your “eyes” for that matter.

Shooting Efficiency – The Basics

Metric: Field Goal Percentage (FG%)
How its calculated: Field goals made / Field goal attempts (FTM/FTA)
Why you should use it: You shouldn’t. Ever. Ban it.
The drawbacks: It does not recognize the 50% higher value a three pointer has. For example, a player with a FG% of 33% could have the same impact on a game as someone with a 50% FG% if the former only shot threes. That’s a huge difference and why the metric is useless. It also doesn’t recognize a player’s ability to get to the line (where typically they shoot a much higher percentage)

So you should use…
Metric: Effective Field Goal Percentage (eFG%)
How its calculated: (FieldGoalsMade + 0.5 x 3ptFieldGoalsMade) / FieldGoalsAttempted.
Why you should use it: Because a 3 point shot is worth 50% more points than a 2 point shot. You can make less, but contribute more.
The drawbacks: Few, but in some cases True Shooting percentage (below) gives a fuller picture.

or….
Metric: True Shooting Percentage (TS%)
How its calculated: Points / (2 x (FieldGoalsAttempted + 0.44 x FreeThrowsAttempted)
Why you should use it: Because a 3 point shot is worth 50% more points than a 2 point shot. You can make less, but contribute more. Players that are able to draw fouls and shoot a high percentage from the line get “properly credited”.
The drawbacks: None. It gives the player full credit whether they shot a 2, 3 or were fouled.

“Holy Grail” models – Searching for the Ultimate Player Ranking

Metric: Player Efficiency Rating (PER)
How its calculated: Simply. Well, not so much.
Why you should use it: Generally speaking, rankings meet the “eye test”…
The drawbacks: … but the “eye test” is often biased towards scorers. PER has two major issues: 1) it tends to overrate high volume, inefficient shooters and 2) it relies on box score statistics, several of which have flaws – many other contributions to scoring and defence are not recognized in the box score (more below)

Metric: Wins Produced (WP; WP48)
How its calculated: Dr. David Berri, its author, recently updated his methodology here.
Why you should use it: Translates box score statistics into what matters most “how much is Player X contributing to wins”. Attempts to correct PER’s inefficient shooter bias, among other factors.
The drawbacks: 1) Also relies on box score statistics [see below] 2) Makes a team adjustment for defence – making the assumption “defence is essentially a team activity” and no one individual makes above or below average contributions. 3) Values rebounds highly (due to change in possession) and gives full credit to the outcome, but does not give any credit to how that rebound was generated (e.g. perhaps by excellent individual defense that forced the missed shot).

Metric: Adjusted +/-, Regularized adjusted +/-, and others.
How its calculated: Don’t ask…
Why you should use it: It attempts to measure the player’s overall impact on the floor and removes the issues resulting from box score statistics. How does it do this? It “adjusts” for the quality of the players one is playing with as well as the quality of the players one is playing against on the floor.
The drawbacks: It can be very “noisy”. One usually needs a couple seasons’ worth of data for it to be utilized properly.

What are my issues with box score statistics?


Several pioneers in the field of basketball analytics have done an excellent job at focusing attention on possessions and highlighting the pitfalls of past trends like “paying for points”. Researchers like Oliver (Four Factors, Win Shares), Hollinger (PER) and Berri (WP) created new metrics to better recognize other aspects of the game.

However, relying entirely on box score statistics can lead to a few problems.

The simple issue I have is box score numbers are largely “result based”, i.e., you get the “check mark” based on the end result and no (or little) value is assigned to the “components” of, for example, how a field goal was made, or a defensive rebound was obtained. Here are some of my (not unique) concerns about various box score data:

  • Assists – a few things: 1) its obvious, but your great passes mean little if your teammates are not efficient shooters/scorers 2) a huge issue I have: you do not get credit for an assist if the play ends in a shooting foul [the exception being “and 1″ of course] 3) its still a judgement call by the scorekeepers.
  • Blocks – when a defender blocks a ball out of bounds, it does not change the possession. Some feel this makes it overrated. However, it still: results in a “miss”, could result in intimidation (e.g. more hesitant to drive next time), and usually takes seconds off the clock (team forced to inbound the ball and “reset” offence).
  • Steals – steals are based on a successful outcome. Thus, a defender could gamble and attempt several steals without being successful. The “missed steal” gamble is can be expensive as the defender is often well out of position.
  • Charges – are not recognized, yet change the possession (and often ignite the team/fans).
  • Defensive rebound – rebounding is an incredibly important part of the game. However, a defensive rebound is a function of: 1) a missed shot 2) blocking out your opponent and 3) luck [how the ball comes of the rim – direction and distance]
  • Personal fouls – context can be important – e.g weak foul 23 feet from the basket, moving screens, intentional at the end of game to stop the clock or a foul as a result of help defence (that would have been an easy basket).

Overall: The largest challenge is there is no adjustment for the quality of players someone is playing with nor against. A player that is productive in “garbage time” may not be nearly as productive in “crunch” time. Points guards playing with efficient scorers will have more assists. Bigs playing with defensive minded wings will have more defensive rebounds.

What is more than a bit odd is a Wins Produced post, with the following argument.

The absolutely most incorrect way to judge a decision is solely by the outcome.

We agree, and also wonder why Wins Produced (largely) only measures outcomes.

The challenge I have with some of these metrics is similar to the Wins Produced authors’ criticism of advanced +/-

You are not measuring what you think you are measuring.

The Curious Case of Reggie Evans


First of all, I’m a fan of Reggie Evans. Despite being a one-dimensional player, he can be very effective in the right lineups. However, he also highlights a challenge with Wins Produced.


Source: data from http://www.thenbageek.com/players

Last year, according to his WP48 score, he should have been considered a top ten player in the league. I wrote about it previously.

The response to some of my curiosity on why he was rated so highly was “so even though the shooting is terrible, there isn’t much terrible happening … It’s perfect: if you can’t shoot, don’t!”

This is the root of the problem. Someone has to shoot the darn ball. I wish I still had the screen grab, but I captured Evans (several times, but this was the best example) with the ball at the top of the key. His defender was no where to be found – he double teammed Bargnani right away. Evans had as clear as shot as one can possibly have and not from a great distance. He also had a fairly clear lane to the basket for a layup. What happened? Well, as WP likes, he didn’t do anything terrible like shoot or turn the ball over. The shot clock continued to count down and Evans had a hard time passing the ball and, because Bargnani was now double teamed, he had one less option to pass the ball. I cannot recall who ultimately got the ball, but 5 or 6 seconds has drained from the clock leaving only a few seconds left. And we know FG% drops significantly in this type of situation.

The challenge is obvious, but not doing anything terrible on paper, its clear his lack of shooting ability hurt the team in this case. He doesn’t get any “knocks” against his WP score, but his teammate likely did.

Simply put: Wins produced has an allocation problem.

After refusing to acknowledge this for some time (The article noted yet again: “I don’t want to get into another rehash of the endless ‘WP overvalues rebounds’ argument”), Dr. Berri (sort of) conceded somewhat and has altered the WP calculation: “although diminishing returns – as detailed in Stumbling on Wins — certainly exists for defensive rebounds (but not for offensive rebounds), the size of the effect is ‘small’.”

However, the “before and after” data shows some fairly dramatic changes:

Dr. Berri massages these large changes with the phrase “the players who ranked towards the top of the league before still rank towards the top now.” Sure, generally speaking, the players are are in the same zip code.
However, the data also suggest rebounds may have been “overvalued” previously. For example, Blake Griffin’s WP48 was 48% higher than after the adjustment. That’s quite material. To use another example, Kevin Garnett was thought to “produce” 24% higher WP48 than Paul Gasol before this adjustment. Now, when rebounds are viewed differently, Gasol is viewed as producing 4% higher WP48. That is pretty significant and somewhat validates previous criticisms.

Ultimately we are encouraged Dr. Berri made the adjustments. However, the issue of allocating some credit to those that may have forced few extra “misses” remains. Nonetheless, the Wins Produced team is appears open to continuous improvement to address its flaws. Continuous improvement will move away from unfortunate posts like calling cancer survivor (no cancer survivor is an idiot – trust me, as I’ve been witness to both positive and negative outcomes), who’s a 1000 win coach and has led his team to Western’s Conference’s second best record.

Bring the Noise! Adjusted +/-


I fully admit to appreciating the concept of adjusted +/-. Why? it attempts to measure ultimately what matters – how much does that player influences the production at both ends of the court, adjusting the for quality of players he’s playing with and against. It indirectly measures all the intangibles: charges taken, tipped balls, help defense, running the lanes, setting good picks etc. The problem is the data is often very “noisy”. It doesn’t mean its worthless, it means you want to ensure the sample size is reasonable. As well, researchers like Joe Sill (who now works for the Wizards, introduced Regularized Adjusted +/-) and Dapo Omidiran (A New Look at Adjusted Plus/Minus for Basketball Analysis) continue to improve upon this analysis.

It would be great for these pioneers to spend less time focused on criticizing each other’s work and more time working through flaws found in each.

So Your “Eyes” are Better then, RIGHT?


No. Don’t get smug yet.

The VAST majority of fans, scouts, GMs etc cannot possibly watch and remember enough tape to accurately access all players. Sure, some get most of the way there by scouting one opponent at a time. But we know that the vast majority remember the “buzzer beater” and yet completely ignore (or most likely forget) that player shot 4 for 12 in the game and had 4 turnovers. Some make more errors in judgment than others.

Eyes Wide Open, Mind Wide Open

The data helps, sometimes a lot. The contributions of Oliver, Hollinger, Pelton, Rosenbaum, Winston, Berri, etc are very significant. Whether you agree with their theories or not, they – at a minimum – enhanced the discussion (even if a few are not overly open to debate).

You may believe your eyes when you say Hedo Turkoglu is clutch when the data allows us to know he isn’t. Or you may accept that Dwight Howard’s contribution to defense is identical to Hedo Turkoglu’s and simply make a “team defense adjustment” – or your eyes tell you Howard is a much superior defender. You may complain about your “go to” player shooting 38% FG%. Or you may realize most of those shots were threes, he got to the line 12 times and was very effective. Your eyes may tell you Russell Westbrook is an all-star based on 22 ppg, but Wins Produced tell you he’s more of an average PG given how often he turns the ball over (4x a game!) and has so few (relative) assists despite having several top offensive weapons. The metrics may look great, but your eyes tell you he’s playing against inferior bench players and it won’t translate to starters minutes.

The best analysis of players and/or teams combine a number of metrics as well as your well trained eyes. Your “eyes” don’t see every play of every game and record it accurately. Nor does any one metric tell the whole story, such that one can make conclusive statements (as much as some may try). Basketball is a complex game and thus anyone who doesn’t utilize several tools is bound to miss the full picture.

Questions? Arsenalist introduced a forum thread dedicated to “Statophile Q&A”. If you prefer to send questions privately, you’re welcome to email me at tomliston [at] gmail [dot] com or find me on Twitter (@Liston).

  • Ppellico

    Wow…very nice job and
    this was worth every word.

    The only issue with it is
    it made me depressed. It made me feel even less able to understand the
    game…in a way.

    In another way, it made
    me feel good about what I myself sees in players…not just the stats others
    throw around.

    I remember one player
    everybody spoke badly of and repeated the same old stuff written…he was out
    of shape to play this year. This was in spite of the fact he returned halfway
    through the year after a broken foot. I asked…who here could return mid year
    after a broken foot? I asked…how could you stay in shape with a freaking
    broken foot!?

    It went nowhere….

    I guess this is like
    pornography. I really can’t define it…but I know it when I see it.

    Ditto for players.

    Oh…and one more stat
    that doesn’t exist…team leadership.

    I know of sooooo many
    players traded and then their teams tanked. These so called bench players were
    the leaders in the locker rooms. They tutored the young guys and talked up
    players in tough spots.

    THIS is a stat found
    nowhere.

  • mountio

    Great read – very well done. A great job of capturing what I think was a swing too far in the advanced stats direction (on this site and elsewhere). People were convinced that because they could quote PER or WS/48, they had “cracked the code” in evaluating players and could ingore the other factors (like whether or not a guy can create his own shot, (or shoot at all in the case of Reggie), whether a teamate chokes on a wide-open shot that you set up, a rebound isnt a rebound, isnt a rebound and so on) .
    As you correctly state, these are very interesting and helpful metrics .. but can not be looked at in isolation and need to be considered with the “eye test” to truly understand the impact of a player.
    To me, the most obvious example on the Raps is Amir vs AB from last year. The advanced stats would suggest Amir was a better player (some would suggest a MUCH better player), yet the eye test told you that Amir only survives as a 4th or 5th option where he can thrive on put backs and pic n rolls with a floor spread by shooters. There is no way he can be a main option and create his own shot. And .. when he doesnt have to, he can be a great player. BUT – when he gets the ball with less than 10 on the clock, look out.
    AB on the other hand, is clearly a talented scorer, and can easily be a 1st or 2nd scoring option on a good NBA team, but is asked to score in tougher situations because on that talent. All of which is fair game (AB isnt expected to do some of the things Amir is) .. but its simply hard for stats to pick this stuff up.
    Fast forward to this year .. and to me, the results are not that surprising. With less talent around (esp with AB out), and a slower, half court game, Amir suffers. AB on the other hand, starts to show what the “eye test” has been showing all along.
    Anyways .. I dont claim to have the answer either. I think these metrics certainly highlight undervalued guys and make you look at certain players in a different light. Kevin Love is a solid player .. but hes sure as hell not the best (or most valuable) player in the league. Nor is Humphries. But – perhaps, on the off chance you under value those guys with the eye test, something like WP/48 would push you in the other direction…

    • Statement

      In the case of Bargnani, I feel like I have to defend myself as a disparager of Bargnani’s game for the prior 5 years.

      Be clear that I am very happy with what he has done this year and that he is playing great, though IMHO it remains to be seen if he is physically strong enough to play hard on both ends of the floor for a sustained season. 

      If I remember correctly, he was able to play major minutes in the past few years because of his lacklustre defensive effort, IMHO.

      It isn’t that there was any 1 stat that said Bargnani was poor, it was nearly ALL of them (aside from the overrated PPG stat). 

      As Tom pointed out, relying on 1 stat doesn’t tell a good enough picture.  Relying on several of them helps.

      I made a post last year using WP/48, Adjusted +/-, 82games on-court off court and some other ones which indicated that Bargs WAS a negative player. 

      Sure, the talent was there, but it wasn’t being used.

      Andrea Bargnani:

      2010-2011 WP/48 = minus .126 (thenbageek)
      2010-2011 net PER = minus 4.9 (82games)
      2010-2011 Net points per 100 posessions = minus 2.6
      2010-2011 2 year adjusted +/- = minus 5.46
      2010-2011 WS/48 = 0.053 (average is .100)

      As can be seen, the data was indicating that Bargnani was a poor player, not just one piece, but several.

      • mountio

        I think we are saying the same thing .. I realize the advance stats (many of them) were saying AB is a poor player. And those same advanced stats (many of them) were saying Amir is a great player.
        But – the “eye test” says otherwise.
        To me this is almost entirely due to their role on the team (primary scorer vs hustle guy and garbage collector).
        Knowing what you know now, would you still say that Amir is a better player than AB (this year, last year or any year?) The advanced stats would say yes. I would say no …  

        • Nilanka15

          If we ignore this season, the eye test was bang on with the advanced stats.

          • mountio

            So, you subscribe to the theory that advanced stats were right about Amir and AB, and this season is just a total aboration for both of them? If so, Im glad I have my eyes and not yours .. because what we are seeing this season should not be a surprise at all ..

            • Nilanka15

              The advanced stats weren’t claiming to make any predictions about player’s future performances. 

              The stats showed that Bargnani wasn’t a productive player last year, while Amir was.  And the eye test last year showed the same.  Simple as that.

              • mountio

                Fair enough .. I would say the opposite, that with Amir and AB, based on their play last year, it was very clear to me who was a better player. (same conclusion if you compared their play at any year they have been in the league)

                This year its perhaps more obvious .. but I struggle with objective observers ever coming to a different conclusion.
                I guess you can say that I am a better “predictor” of talent / performance and AB really sucked last year and Amir was really great .. I would say it was obvious all along ..

                • Nilanka15

                  We don’t have enough of a sample size to produce statistical confidence. 

                  Is Bargnani, through 10-15 games, playing better than he’s capable of for a full season?  And on the flipside, Amir is clearly playing worse than what we saw last year. 

                  I don’t think we can draw any conclusions based on this season (so far) alone.

                • mountio

                  Its clear where we diverge. I dont need a stastically significant set of data to figure out whether an NBA player is good or not. This is clear after watching a handful of games to me. (Of course players improve and regress .. but for the most part .. you can figure out whether a guy has it or not from what would clearly be considered not statistically significant in any real statistic excercise.) Thats teh difference here .. these are NBA players, not simply random data points that require a certain amount of data to make any sense of.
                  You want to rely on stats to tell you the answer, whereas I think the answer is apparent.
                  Thats not to say stats arent useful (I think they are quite useful) .. its just that I dont need them to draw conclusions .. and I certainly dont let them sway my common sense…

                • Nilanka15

                  I agree that Bargnani has far more skills than Amir.  I don’t need stats to tell me that either.  But being “skilled” and being “productive” aren’t necessarily synonymous.

                  I’m just saying that you seem to have concluded that Bargnani is the more productive player using just a handful of games as your eye test this season.  But it’s possible that Bargnani could be playing above his level of statistical normalcy, while Amir could be playing below his. 

                  The point is that neither you or I know what “normal” play is for either of these guys.  We need more games to determine if last year was a fluke for Amir, or if this year is a fluke for Bargnani.

                  But the difference in skill between the players has never been in doubt.

            • Nilanka15

               This year, the advanced stats say the exact opposite, and the eye test confirm it also.

        • Statement

          I would say that Amir is a better defender, harder worker and talented offensive rebounder.

          So overall, I would say he is a better overall player. 

          That said, Bargs is playing at an elite offensive level now.  We’ll have to see if he regresses to his career norms or not.

          • mountio

            Hmmm. You are certainly entitled to your opinion and I respect you for sticking with it despite pretty compelling evidence to the contrary this year (albeit in a crazy, shortened season). [I would say the evidence is pretty compelling since they entered the league .. but obviously its more glaring this year]
            We will see how things play out …  

          • CaseyDaMan

             lmao
            “So overall, I would say he is a better overall player.”

            This is the kind of stuff on here that makes a guy wonder…. is he baiting, or is he serious?

            I’d be willing to bet anything, anything at all, that you couldn’t get a single NBA GM or coach that would take Amir at half AB’s price, if they could have AB at twice Amir’s price.

  • http://www.facebook.com/people/Atlas-Ryder/19218774 Atlas Ryder

    “Continuous improvement will move away from unfortunate posts like calling cancer survivor (no cancer survivor is an idiot – trust me, as I’ve been witness to both positive and negative outcomes)”

    I don’t think anyone’s going to argue that cancer is a tragic disease that devastates its victims, and my heart does go out to them.  On the other hand, after reading through the “unfortunate post,” it’s clear that the author is assessing the judgement of Mr. Karl qua his coaching ability, not as a human being.  If the post is unjustified, it would be just as unjustified if Mr. Karl had never had cancer.  Similarly, if the post is justified, Mr. Karl’s cancer, while tragic, will not negate the points the author has made. 

    If there are legitimate issues with a post, it’s very gratifying to see the community address them and work towards better models.  Nobody is elevated above all criticism by cancer and succumbing to it or surviving it.

    • http://twitter.com/Liston Tom Liston

      Fair comment, fine. Wasn’t the focus of the article. It was about defining these metrics and the pros and cons of using each one.

      I still do not think its entirely fair to call a 1000 game winner an “idiot”.  Denver is also second in the West (well, LAC are officially a snick ahead on a % basis I update this – though they’ve played a lot less road games). There is a way to say “he may not be optimally attributing metrics based on WP48, but we also cannot say with certainty Chris Andersen would maintain a high shooting efficiency with the additional shots required (by him – or perhaps his teammates) with his additional minutes”

      Point is that Karl currently has them playing some pretty good basketball in a tough conference. Thus the “unfortunate post” reference that as well. No reason not to show a little respect to someone who battles in and out of the arena.

      And really, this wasn’t the point of the article. Its that many metrics (as well as our “eyes”) have some flaws and we should try to recognize them.

      I stand by my statement that making conclusive arguments based on a single metric is hazardous.

  • Big_chris1

    Great article.

    You are looking at the wrong stats on Westbrook though.  He’s taking a hit in efficiency for the team by carrying 33% of the offense.  He has the largest role on the offensive end and defensive end for the best team in the league.

    • http://twitter.com/Liston Tom Liston

      I was simply pointing out an example.  But I’ll defend WP’s “average” score for him – at least somewhat. He turns the ball over a lot. That doesn’t help his team.  No does his 19.2 FGA per 36 min – leading his team – despite shooting a *much* worse (only 27.1%) 3 pt% than Durant (35.1%), Harden (36.2%), Cook (38.7%), Sefolosha (48.1%).

      Yes, OKC is 17-4 and a terrific team.  But the selfish play of Westbrook is still inexcusable.  You have 4 or 5 amazing scorers (efficient ones at that) on your team. Distribute a bit more.  6.0 assists per 36 min is pretty sad for a PG on a team with such great offensive weapons.

      • Pizzaman

        Tom I agree with you 100%. 
        Great article above, and shows the value of being able to hit the three and everything else.
        On Westbrook there is no denying he is a hugely talented player, but he is a ball hog nonetheless, especially on a team of great players like he has and I for one would not want him on my future team because I still want a traditional PG like Nash who can score and does score but makes everyone on his team much better because he understands that’s his role and he has the talent to do so.

    • http://twitter.com/Liston Tom Liston

      Related: The Wins Produced crew had a very good post on Harden yesterday (highlighting several metrics…)
      http://wagesofwins.com/2012/02/02/james-harden-the-nbas-best-2-guard/ 

  • A-Wanna-Be-Mathematician

    Nice, clearly articulated post – thanks!  Do you know if anybody has used, or is using, or is developing the use of systems dynamic modeling for understanding the flow of a basketball game *as a complex system*?  Here’s the wiki article – http://en.wikipedia.org/wiki/System_dynamics.  Now I fully admit, I don’t have the mathematical knowledge to follow along with all the details of the theory, but conceivably, the one advantage of systems dynamics is “SD models solve the problem of simultaneity (mutual causation) by updating all variables in small time increments with positive and negative feedbacks and time delays structuring the interactions and control.”  So the advantage for basketball is something like accounting for the interactions b/w players in a non-linear model.  Again, I might be way off, but thought you might be able to help me with a “yea” or “nay”.

  • Nilanka15

    Great read Tom.  One of the best Statophiles in recent memory.  Thanks for clearing up so many questions about advanced metrics.

  • jorvay

    I think you might have a typo in your eFG% equation. You have: (FieldGoalsMade + 0.5 x 3ptFieldGoalsMade) / FieldGoalsAttempted.
    I think it’s supposed to be (FieldGoalsMade + 1.5 x 3ptFieldGoalsMade) / FieldGoalsAttempted.

    Example: 3/6 2-pointers and 2/6 3-pointers, which Should yield 50%.
    Total attempts = 12
    (3 + 0.5(2))/12 = 33.3%
    (3 + 1.5(2))/12 = 50%

  • http://twitter.com/Liston Tom Liston

    Interesting timing – a good example of what’s a “good” foul from last night.
    Help defense by Mozgov prevents an otherwise guaranteed two points (and a rocking crowd).
    http://www.laneviolation.com/good-for-mozgov/ 

    Blake Griffin only shoots 51% from the line. He missed both free throws (airballed the first).  

    It prevented momentum and resulted in two points for LAC.  However, some metrics would penalize a player for this PF.

  • Guest

    Great post! That is all.

  • treboR

    Awesome summation, Tom – I enjoy reading Wages of Wins, but it baffles me how blind some of the authors over there can be to the possibility that a regression analysis could be flawed when the input data is so sketchy.

    • http://twitter.com/Liston Tom Liston

      Berri and the team have written several good posts. I do sometimes have issues with making strong conclusions when – as you say – the input data isn’t ideal.