In part 2 of 2 in our one day series on change in the Philadelphia Union organization, Chris Sherman looks at the data to see if changing things up may help or hurt the Union. Check out Part 1, in which Tim Jones gives his thoughts on the benefit of a consistent system.
Narratives in sports often focus on change within a team, or lack-thereof. Many fans believe the Philadelphia Phillies held onto their championship roster for too long. The Philadelphia Eagles, on the other hand, baffled fans during Chip Kelly’s tenure as he nearly flipped an entire roster that most pundits deemed talented.
For the last several seasons, Philadelphia Union have faced these narratives heavily. While many fans clamored for a coaching change or a game-to-game lineup switches, others are now begginning to Trusty the Process. Sporting Director Earnie Stewart has personally made it clear that he does not believe in “change for the sake of change,” but rather sees a benefit in staying the course.
This question brought me to investigate whether change, in and of itself, is harmful or beneficial (or irrelevant).
Executive Summary
- Manager experience counts for something, and a non-insignificant factor over the course of a season.
- This seems to indicate that, while a clear upgrade in manager could have a greater impact than the experience, replacing a manager without an assuredly better manager could do more harm than good.
- Keeping your lineup the same from match to match appears to help team cohesion as common narratives suggest. A lot of change, however, doesn’t appear to be more harmful than medium amounts of change.
More Detailed Analysis: Manager
Last year, many fans began to question at what point Union head coach Jim Curtin would lose his job. Though the first game shone bright for the Union this year, many fans – right or wrong – will assuredly raise the same question when the Union fall into a rough patch, as nearly all clubs do in an MLS season. The club on the other hand maintains that developing coaches was a part of their plan, and that takes time.
Let’s then begin with how a manager’s experience relates to performance:
*Data is from 2013 through 2017 and is for MLS matches only.
*Managers who were already in place with their club before 2013 had their game count added in manually by me.
*Some of the pre-2013 data (for managers in their position prior to 2013) was hard to nail down, and so is probably not exactly-correct, but near enough.
While it doesn’t seem like a lot to add 1.45 points per season, this indicates Jim Curtin’s 3.5 seasons with Philadelphia is worth around 5 points which could certainly determine a playoff spot.
However, the problem with the above analysis is that only good managers are likely to survive in their position long enough to have high amounts of games managed. Therefore this data is biased.
To offset this, we need to control for the manager, himself. To do this, we are essentially grading each manager upon their own curve rather than that of others in the league. This way, we can actually measure whether their experience in their job contributed to higher performance than their own career average.
* In the below, when the dots or lines are above the black line (zero), it indicates that the manager performed worse than expected. When the dots or lines are below the black line, it indicates that the manager performed better than expected.
This shows that managers tend to perform worse than their career-for-club average at the very beginning (1st season or so) and actually continue to get better as time goes on.
More Detailed Analysis: Player Lineup
Another point of contention last season was switching lineups from game-to-game. Indeed the question is on many minds as to whether Jim Curtin will continue to stick with a consistent lineup all season or whether he will change depending on opponent.
Let’s see if there is any data backing up the theory that players perform better when they get used to playing with one another:
The next chart shows how clubs perform over a season compared with how often they change their player-minutes distribution match-by-match.
We can clearly see a trend where greater volatility leads to fewer points.
However this isn’t quite a fair analysis because teams that win (and are already good) will inherently be less likely to try out different players, and teams that lose may need to experiment more.
So the next chart breaks it down by match and simply looks to how the team performs compared with how their minutes given changed from the previous match.
We can continue to see that greater change between matches leads to worse results.
However this is still not quite right, because, as before, better teams will still be less likely to change their lineups in any given match.
What we do next is to try what we did for the manager analysis above and control for the quality of players (among the home/away/opponent factors).
What I did was, for each season and team (i.e. 2017 Philadelphia Union), I chose the five most common starting lineups. With that in mind, I graded each match’s performance on its own starting lineup’s curve, to try and measure how change, itself, affected results, even among the same starting lineup.
*As with the manager analysis above, dots and lines above zero indicate worse-than-expected performance, and below zero better-than-expected performance.
As the blue line shows, the difference between medium amounts of change and large amounts of change has no noticeable affect of the team’s performance, but it does appear that any changes from match to match does, in fact, appear to harm team performance.
Conclusion
This is certainly an indication that retaining Jim Curtin was a good idea by Earnie Stewart in lieu of a clear upgrade at his position.
Furthermore, I don’t actually blame Jim Curtin for his refusal to play youth when the club is out of contention or his substitution choices.
Since Jim Curtin has no job security or fan support to allow him to disobey an instruction from Earnie Stewart, then these decisions must be getting made by Stewart, himself (or at least a shocking amount of negligence on Stewart’s part, which would still lead me to blame him). Furthermore, I expect that Curtin is probably very personable and good with managing the players in the background for the 95% of his job that we can’t observe.
The lineup-change analysis does also bolster the theory that playing the same lineup is likely to improve team cohesion.
Fun fact in this analysis: only 11.4% of distinct starting lineups in MLS were repeated for a second match within a season (this represented only about a quarter of matches that involved the same lineup as a different match).
So it looks like Philadelphia isn’t the only club to throw out numerous starting lineup changes.
However, it does appear that if health, timing, and quality allow, it would be beneficial to trot out the same lineups (or maybe one player switch) in repeated matches to see if they can perform better with enhanced team chemistry.
On both accounts in MLS, it seems, change is bad, in of itself, without clear upgrades in players or managers.
Appendix
Stability issues aside, do you not think the Union could benefit from a better coach?
.
Btw, was Curtin trying out a new formation by putting in Simpson? Jay was meant for a 2 striker system, was Curtin trying to capitalize on that? Were we ultimately playing a 4-4-2 (in the emptiest of buckets sort of way) by putting him in.
I do think the Union could benefit from a better coach, I’m just not sure who would be available. And, especially given the data, I’m not in favor of taking a gamble on a comparable coach.
–
Yeah, I really don’t know. Maybe the organization is trying to justify his cost a little, but that didn’t seem to work
I think a clear pick (if they can get to him first which seems unlikely) would be Caleb Porter. He always coached an attractive style of soccer and wasn’t afraid to make changes. Unfortunately, as long as Stewart is with the U, so Curtin will be also.
.
Unfortunately, by the time Stewart thinks that the partnership has run its course (which seems unlikely to ever happen), Porter will have been snapped up by Orlando or NYCFC or NYRB after they get tired of having talent with no results.
.
(Of all those above, Jesse Marsch and Jason Kreis both seem to be the ones in the “danger zone”.
Mark Lowry. Without question will be coaching in MLS shortly…playing a beautiful brand of possession footy.
In God we trust. All others, bring data.
.
Great job Chris.
Interesting stuff, but I’m not sure your analysis leads to your stated conclusions. Sure, I’ll buy that more lineup change doesn’t equal more points. If you replaced Blake with McCarthy for injury on an extended run, we would probably not have as many points. But when you look at the why a lineup needs freshening or that a player could benefit from rest but isn’t getting it, how did you capture for that? So Pontius should have seen pine during his run of futility, the U continued on their 1.22ppg run, lineup didn’t change and we missed playoffs. If your baseline trajectory is still low, what’s the real risk of not changing?
.
Also, your “in lieu of someone better, Earnie was right to keep Jim” statement- sure. Ok. Without an opening we didn’t look to upgrade. But if someone is a better coach, it’s expect better results than what Curtin has delivered. Remove his first few games (dead cat bounce) in 2014 and he’s quietly chugged along at 1.20-1.23ppg. Solidly in “not good enough territory “. Any analysis that excludes his actual performance but states managers “continue to get better as time goes on” feels incomplete and borderline misleading. He hasn’t actually improved.
I think those are some very good questions, scottymac. To account for what I think you’re asking about (in that, of course changing from Blake to McCarthy should result in worse performance, since McCarthy is not as good), I was measuring a single match’s performance against the average result of the same(ish) lineup vs the change from the previous match. So if McCarthy lineups score 1.1 ppg on average, and Blake’s lineups score 1.3 ppg, then we chart a Draw as -0.1 pts for McCarthy and -0.3 pts for Blake when we compare against the degree of change.
–
I think we’re in agreement that there are plenty of valid reasons to change one’s lineup, due to health, a new signing, rotten form, a backup’s recent excellent form, or if the training staff assesses that a player is too tired to play better than their backup. Good reasons could also be to do more experimentation when you’re already out of contention since match results matter less. What we get to see in this data, however, is that some rotation or experimentation that isn’t related to one of the above reasons may make a team more likely to leak points and justify some aversion-to-change.
–
I’m not sure it’s fair to assume that for (Quick Wikipedia stats, not at my home where my data is) 133 games managed for his career, a few games at the start could skew the sample. Even if you erase all of 2014 (also unfair), Curtin’s average only drops from 1.346 ppg to 1.325 ppg.
–
That said, I do think it’s fair to question whether Curtin got any better with experience. My analysis was based upon MLS in general, is not guaranteed to work the same for everyone, and could also be due to other related factors such as organizational stability leading to player investment, rather than a manager’s actual contribution. While the data in the appendix does indicate that Curtin & Hackworth(2013+) did seem to get better with experience, that could also be due to them getting more investment in players they like and/or Earnie Stewart’s involvement, rather than their innate coaching ability.
–
Still, it does appear as if change-for-change’s-sake would set the Union back unless they have an attractive replacement willing to take a subordinate-to-Stewart-role within the Union’s budget.
2017 – 1.24, 2016 – 1.24, 2015-1.09ppg.
.
Are you counting USOC “points”?
Yes, it appears I was, good catch
This is great analysis, Chris. The stuff about managers getting better over time is particularly nice — shows what we might have suspected, but it’s always good to have the data to prove it.
–
And as for our own, well, we certainly have a deeper squad this year. So I guess we’ll get to see what he can do with that.
Alex Ferguson would’ve been fired his first season if he were held to the standards of modern day coaches – it’s kind of silly.
It’s interesting to see the data. I do think it’s important though to look at individual clubs and their situations at any given time. Looking at the big view with all teams over 5 years shows a general trend, but it will certainly miss micro view nuances. Pontius needing to sit, giving a little more time to younger players towards the end, changing a player to take advantage of a particular teams weakness or to protect against a strength. These are things Curtin should rightly be asked about, and things that won’t be seen on a larger view trend.
I totally agree, these strategic change-choices, which might have resulted in greater benefit than the stability did, are part of Jim’s and Earnie’s jobs and are totally fair to call into question
This has been the main reason I’ve supported keeping Curtin. It’s not so much that I think he’s going to win us a ton of games in which we’re under-matched. It’s more that changing coaches typically has a net negative effect all else equal. Unless you have a Tata-level coach waiting in the wings, you don’t waste your energy on a change.
“It’s not so much that I think he’s going to win us a ton of games” – could’ve stopped right there.
Players win matches, coaches lose matches and referees ruin them. 😉