Featured image: Courtesy of nasljerseys.com

In 2008, I wrote the following article for a blog post on BigSoccer. With the recent publication of The Numbers Game: Why Everything You Know About Soccer Is Wrong (Anderson, Chris and David Sally), as well as last year’s Moving The Goalposts by Rob Jovanovic (referenced heavily in my series on top U.S. club sides last winter), I thought it would be a good time to revisit some of the concepts I discussed five years ago and see if there was any interest in further examining a better way to evaluate goalkeepers.

Unlike pretty much every other sport in America, soccer is not one that lends itself to a lot of statistical analysis. In other countries, player statistics are limited to games played and goals scored or, if you are a goalkeeper, games played and “clean sheets.” There are also “match ratings,” a completely subjective evaluation of a player’s performance in games. Otherwise, there is not much else to base a fantasy league on.

Of course, since we are Americans, we have done our best to foist statistics upon the Beautiful Game. This may be an attempt to make the game more palatable to the infamous “casual fan,” or has perhaps come about because, as soccer fans, we are jealous of fans of other sports’ ability to babble on incessantly about so-and-so’s on-base percentage or what’s-his-face’s quarterback rating. In any event, only in America will you find such nebulous statistics as “touches,” “times fouled,” “chances created,” and other data designed to more fully measure a player’s contribution to a team’s performance. As Billy Beane becomes more involved with the San Jose Earthquakes, we may see more of these “moneyball”-type concepts enter scouting reports and other player evaluations.

Even before these New Age stats came about, however, North American soccer has always included two statistics not seen in the rest of the world: assists, and goals against average. Both borrowed from ice hockey, neither has found great acceptance among the world football community. While this is somewhat surprising in the case of assists–as fans appreciate great vision and service from players, one would think that a stat rewarding such talents would be in regular use by now–it is completely understandable when it comes to GAA.

It is universally accepted among soccer fans that goals against average is an especially deceiving statistic. Unlike in hockey, where it is easier to create quality shots on goal and, thus, make a goalie earn his money, in soccer a low GAA can be attributable to a number of factors which have little to do with the goalkeeper’s value. For example, was his team so dominant in possession that the other team had few shots on goal? Did his team utilize an aggressive offside trap, limiting the other team’s offensive chances? Did he play during a particularly defensive era? Any one of these factors can make a goalkeeper statistically impressive. Conversely, a goalkeeper saddled with a particularly porous defense can suffer from a high GAA which would make him look bad, even though fans could see that he was a tremendously acrobatic player who kept his side in games with more tremendous saves than one had a right to expect

As a result, evaluation of goalkeepers in MLS (and, before it, the North American Soccer League) has been largely anecdotal. Among my memories of the NASL is the constant lauding of goalkeepers such as Mick Poole and Jan Van Beveren as among the best in the world, even though their GAAs were decidedly average, at best, while perennial league leaders in GAA such as Shep Messing or Phil Parkes were deemed overrated.

Even with the wider acceptance of GAA in hockey, however, similar debates raged. A few famous examples include Denis Herron and Gilles Meloche, two goalies who labored for the worst teams in NHL history (the Kansas City Scouts and California Golden Seals, respectively). Each often suffered with inordinately high GAAs but, claimed most pundits, either would have each been the second coming of Glenn Hall if they just had a defense in front of them.

Two individuals who questioned the true value of GAAs in hockey were Jeff Klein and Karl-Eric Reif. In their seminal work, The Hockey Compendium, the two created a new statistic designed to measure and reward a goalie’s “perseverance.” Although the NHL had adopted the “save percentage” statistic, this, too, was found wanting, as it overly-credited a goalkeeper who was fortunate enough to face fewer shots. For example, if you have two goaltenders with a .900 save percentage, who deserves the greater credit: one who saved 9 out of 10 shots, or one who saved 45 out of 50?

To try to address these issues, Klein and Reif devised a “perseverance rating” statistic. Basically taking total number of shots faced and shots faced per game, the duo constructed the formula as follows:

To find Efficiency:

(Saves divided by Total Shots Faced) X 100

To find Shots Faced per Game:

Total Shots Faced divided by (Minutes Played divided by 60)

To find Perseverance Rating:

(Efficiency X 6) + Shots Faced per Game = µ

µ divided by .6 = Perseverance Rating

Using this formula, Klein and Reif were able to determine which goaltenders were able to earn their money, and which were simply benefiting from strong, close-checking defenses.

****

I thought Klein and Reif’s approach was as close to perfect as one could get, and set about applying it to soccer goalkeepers (obviously adjusting the formula for minutes played and multipliers along the way). However, I immediately encountered a problem: the perseverance numbers tended to skew wildly from year to year, and, especially, when comparing the NASL with MLS. Whether the NASL was a more offensive league or the league’s statisticians were overly generous with saves, the result was that the worst NASL goalkeeper in a given year achieved a higher Perseverance Rating than the best MLS goalkeeper in a given year. This was especially problematic for me, as I like to write about and compare American soccer players from different leagues and eras.

As a result, I borrowed yet another concept from hockey: plus/minus. Specifically, I calculated a league’s seasonal Perseverance Rating, and then calculated a PR plus/minus by taking an individual goalkeeper’s PR and subtracting the overall season PR from that. In this way, one could more accurately judge a goalkeeper’s performance in a season, but could also use the +/- rating to judge his performance historically. Specifically, the +/- figure tended to equalize differences in shots faced which might be attributable to either generous stat keeping (as in the NASL) or different “fads” in styles of play (or, in MLS’ case, the gradual improvement in the quality of play) which might effect statistics from year to year.

(Note: after first writing this in 2008, I came across the revised edition of the Compendium, wherein Klein and Reif also recognized the problem and adopted the above adjustment—to give credit where it’s due, they did it first. I just didn’t know it in 2008)

Applying the modified Klein/Reif Perseverance Rating to soccer goalkeepers revealed some interesting results, and tended to indicate that it would more accurately a goalkeeper’s performance.

A good example is found in MLS’ 2004 season. That year, the league’s leading goalkeepers were as follows:

min. 1000 minutes

If GAA were the only indicator, we would believe that Rimando was a key element in D.C. United’s title run, while Henry Ring was a below average performer for Chicago.

However, the Perseverance Ratings reveal the following leaders:

He’s an unquestioned great today, but in 2004 Rimando was kind of along for the ride in D.C.’s MLS Cup run

The Perseverance Ratings suggest that D.C.’s title was largely attributable to a high-possession offense which resulted in few quality opportunities on goal for the opposition, which, apparently, was a good thing, given the dismal PRs registered by Troy Perkins and Nick Rimando. Ring, on the other hand, reveals himself to be a goalkeeper who did his best in the face of a heavier workload.

But does this mean that Ring had a great year, or that Rimando had a really bad one? Not necessarily, when it is seen that Ring’s PR +/- was +7.05; in other words, he was just above average. Rimando, with a PR +/- of -93.18, was absolute rubbish–notwithstanding his league-leading 1.00 GAA. In fact, Rimando’s +/- may be the worst ever for an MLS goalkeeper.

Does this mean Rimando is an overrated stiff, unworthy of playing in MLS? Let’s take a look at 2007. The GAA leaders for that year were:

min. 1000 minutes

As we can see, Mr. Rimando is smack dab in the middle of the pack, and nothing to write home about as far as GAA is concerned. His PR rating shows, however, that his performance in goal for the Royals was much better than his GAA would indicate:

Thus, according to PR, Rimando had a great year playing for the next-to-worse team in MLS. In addition, it appears the Red Bulls made a real mistake in giving Ronald Waterreus so much playing time, as Jon Conway’s PR was the best in the league. Also surprising is Brad Guzan’s rather mediocre PR, despite 13 shutouts; apparently, he got to sit back and enjoy the view in many of the Goats’ matches.

Furthermore, both Rimando and Conway significantly outperformed the league average, best represented by Matt Reis’ 745.25 PR (a -7.86).

The ever-acrobatic Bob Rigby

As anticipated, the PR +/- was most useful in comparing goalkeeper performances in different eras. For example, Bob Rigby’s rookie season in 1973 for the Philadelphia Atoms resulted in the current U.S. Division One professional record for GAA in a season, 0.62. As might be expected, Rigby also led the NASL in PR that year, with a 918.13 rating. As this puts him well ahead of the league-leading PRs posted by Cannon and Conway in 2004 and 2007, respectively, one would think it proves that Rigby did, in fact, have one of the greatest seasons for a goalkeeper ever. However, Rigby’s 1973 PR was only +56.62 above the 1973 PR average; thus, in comparison with their respective peers, Cannon’s +69.44 in 2004 and Conway’s +59.02 in 2007 were statistically equal to, if not better than, Rigby’s 1973 performance.

Do you want an example of a truly dominant season? Tino Lettieri’s breathtaking 1983 performance with the Vancouver Whitecaps appears to be the greatest season ever by a goalkeeper in a major professional league. That season, he not only led the NASL with a 0.87 GAA, but also led all qualifying goalkeepers with a 874.77 PR–a +112.68 for the season. This, by itself, is stunning; the fact that it was 82.28 points higher than the next best PR reveals that Lettieri was truly in a class by himself in the North American Soccer League’s penultimate season. By comparison, Rigby’s 1973 PR was only 20.38 points better than his closest competitor, Dallas’ Ken Cooper…or, as we now have to refer to him, Ken Cooper, Sr.

(By the way, I humbly submit that, if statistics for the era were available, Findlay Kerr’s 1924-25 season for the Fall River Marksmen—when he posted 20 shutouts and an 0.86 GAA in an extremely offensive-minded year which saw, among other things, Archie Stark score 67 goals in 44 games—would be the most statistically dominating single-season performance by a goalkeeper in a major American league. As it is impossible to reconstruct save figures from game reports of the era, however, this will forever remain conjecture.)

Speaking of the NASL, remember how I remarked how goalkeepers like Van Beveren and Poole were lauded by pundits of the era as true greats, while Shep Messing was considered by those same pundits to be little more than an overrated showboater and Phil Parkes was said to benefit from playing behind great defenses? The following are the PRs of all NASL goalkeepers with at least 10,000 minutes played, along with the PR +/- compared with the NASL’s career PR.

Shep Messing–Bronx-born U.S. soccer legend…and the best goalkeeper in NASL history

Surprisingly, using PR, we see that Shep Messing was, in fact, the best goalkeeper in NASL history, sporting not only the best lifetime PR, but also a +/- which puts him over 25 points higher than the next best keeper, the often (and, as it turns out, justly) lauded Hubert Birkenmeier. Meanwhile, it turns out the perception of Parkes was entirely accurate; although he has the best career GAA of the group, it appears he did little to earn it as he was slightly below average at stopping the shots he had to face.  Most surprisingly, Jan Van Beveren–who routinely finished near the top of most Soccer Digest “best goalkeeper” polls–is revealed to be incredibly overrated, more than 25 points behind the NASL lifetime average. Similarly, Arnie Mausser–long time U.S. National Team goalkeeper and the only Hall of Fame member on the list–is revealed to be a slightly-below average goalkeeper. This begs the question: if Mausser is in the Hall, aren’t Messing, Birkenheimer, Lettieri and the largely-forgotten Zeljko Bilecki also deserving of induction?

Is the Perseverance Rating and the PR Plus/Minus the final word on evaluating goalkeepers? Given the fact that saves remain a rather fluid statistic, and the fact that team play in soccer can effect the number of shots on goal much more than in hockey, both probably remain less than perfect. Also, a goalkeeper who enjoys little work because of his team’s aggressive offside trap may find himself facing much more dangerous opportunities (i.e., breakaways) once the opposition has broken the trap, resulting in a higher goal-to-shot ratio (which might explain, but not totally excuse, Rimando’s dismal 2004 PR numbers). The effect of penalty kicks is also hard to judge. All told, however, PR and PR Plus/Minus remain far superior to GAA as an evaluator of a goalkeeper’s true worth to his team, and it would be nice to see the PR and PR +/- adopted as a soccer statistic.

(So what’s the point of revisiting all this five years later? Join us next time, when we apply the metric to the goalkeepers the Union have employed in their short history.  The results will shock you…)