Ben Bishop entered his second season with the Dallas Stars assuming things would be different. In Year 1, his coach was Ken Hitchcock, who took the team's defense from 29th in the NHL to seventh. But Hitch's reunion with Dallas lasted only one season. Jim Montgomery left the University of Denver to replace the legendary coach earlier this year, taking his first NHL head-coaching gig with the Stars.
"It's different," Bishop told ESPN. "You get a chance to play for Ken Hitchcock, a Hall of Fame coach, all those wins. And then Monty comes in and he's got a young voice, first-year coach in the league, brings a lot of energy and a different style of play. More offensive. More upbeat. Hitch was more defensive. They both work in their own ways, but it's a new coach, new approach and a new attitude, I guess, around the locker room.
"Hitch's system really limited the shots from other teams. You're not really complaining about that, but sometimes you'd like a little more action. There's pros and cons to every coach I've ever had. But you have to go out there and stop the puck. It doesn't really matter what systems they're running. At the end of the day, the goalie has to do his job, and that's keeping the puck out of the net."
But as Bishop said, each coach is different. Witness Montgomery's recent decision to pull Bishop with eight minutes left in a game against the Ottawa Senators, seeking an offensive boost while down by two goals.
"I wasn't upset at all. It kind of surprised me. But we had the puck in their end for nearly two minutes," Bishop said. "You're seeing that trend a little bit around the league. We have such a potent power play, you could see how he'd want to do that."
The analytics say that pulling a goalie early is the right move, especially for an offensively gifted team. The analytics say a lot about goalies, actually, including how effective they are beyond traditional stats like save percentage and goals-against average.
But when the analytics speak to a goalie's prowess, Ben Bishop has his fingers jammed in his ears, refusing to hear them. "I don't look into analytics at all. They're not even right, to be honest," he said.
Over the past few seasons, goalie stats on sites like Corsica Hockey have become more readily available to the public and more widely consumed. A check of Bishop's numbers through seven games this season finds him having given up 14 goals at 5-on-5 in 172 shot attempts. His save percentage at even strength is .919, while his "expected save percentage" is .917. His "goals saved above average," i.e., his numbers versus what a league-average goalie would post on the Stars, is 0.36, just out of the negative. Then there are situational save percentages: low-danger, or pedestrian shots that should be stopped (.975); medium-danger (.873); and high-danger, those shots from right in front of the crease (.865), which ranked him seventh overall in the NHL early in the season.
But Bishop sees much of this as junk science because of the quality of data being used.
"Last year, we were looking at some of the goalie analytics. And they have, like, high-danger save percentage, low-danger save percentage. If you look at the NHL clips, it has all the goals. We pulled up some of the goals, even on myself, and there are goals that they say are from the red line that are actually from the slot, because they clipped it wrong," he said. "They're like, 'This goal is from a low-danger spot.' But then when you watch the goal, it's a tip in front of the net. So it's clipped wrong."
Emmanuel Perry, who runs Corsica Hockey, explained that the shot categories are tiered according to "expected goals" on his site, and they're only as reliable as the shot-quality models that define them.
"One of my biggest pet peeves is the pervasive fallacy that an imperfect model is a bad one," Perry told ESPN via email. "The features common between most public expected goals models are shot type, distance, angle, game state and often indicators for things like rebounds and rushes imputed from the official play-by-play. Due to data limitations, many factors that we intuitively believe are important influencers of shot quality, such as screens and puck movement, are not possible to include, in the absence of manually obtained, crowdsourced or purchased tracking data.
"To answer the question of whether I believe this makes metrics like low- and high-danger save percentage unreliable, I'll pose one of my own: What is the alternative? The notion that incorporating an approximate or incomplete version of shot quality is worse than none at all is bold, to say the least. If you are able to get your hands on tracking data, there is no doubt in my mind that these metrics can be improved upon. However, that does not mean that the publicly available versions do not have value."
This is where Bishop and the analytics community have some common concerns: Who is tabulating goalie stats during games, and why aren't they more consistent?
"I don't take them too seriously because this is someone's opinion. Especially those high-danger [saves]. That's someone who's looking from the press box that sees a play and has an opinion," Bishop said. "When it's some old hockey fan that doesn't even get the shots right on the game, how are they going to get that right? You go to every different building, and sometimes they count the dump-ins as shots on goals. Well, half the buildings don't count those, because they don't know the rules. When you go to some buildings, and they don't count it, and there are four or five shots that don't count. ... That's what's frustrating. That's what you get paid on. That's your job."
But even if these data points were accurate, the current collection of public goalie metrics fail to tell the full story, according to Bishop. It's like in the NFL when the quarterback is credited with an incomplete pass: What was the defense doing at the line of scrimmage or in the secondary? Did the receiver run the wrong route, or whiff on the ball? None of it is conveyed by an incomplete pass.
"There's so much more that goes into a read for a goalie: Who's shooting it, from where, who's in front of you, who's backdoor, what are his outlets? So many more variables than who scored a goal and from where," Bishop said. "The goalie analytics ... I don't even think they're true, in my opinion. It's completely unrealistic."
Kevin Woodley, a goalie analyst for NHL.com and InGoal Magazine, agrees that the current analytics fall short in telling the full story on each shot.
"I don't think they're junk," he said. "Every layer of context we add has value, but I do believe there are limits in the publicly available statistics that are primarily location-based. I know there are other variables that go into them, but none include whether the play made a goalie move laterally, and to me, that is a key factor in scoring, maybe the biggest factor."
As Perry noted, the best hope for fans seeking to use contextual and accurate public goalie analytics would be player-tracking technology. But then, would that data even be made public by the NHL?
"At a very basic level, being able to identify traffic in front of a goaltender, or when passes occur directly before a shot, could massively improve shot-quality modelling," he wrote. "Beyond that, there are almost endless possibilities with event sequencing and the positioning of the goalies themselves. Unfortunately, I don't envision this type of data becoming available to the public, which is why I believe some of the best research in this area is being done privately."
The issue for hockey fans seeking to evaluate goaltenders through available stats is that the best ones aren't available publicly. Woodley cites former NHL goalie Steve Valiquette's Clear Sight Analytics as one company producing good data. NHL teams, meanwhile, generate their own analytics for goalies that includes context like pre-shot movement of shooters. That includes the Washington Capitals, the reigning Stanley Cup champions.
In two of the past three seasons, Capitals goalie Braden Holtby led the league in wins. It's a stat that many analytics devotees sneer at, much in the way they do plus/minus for skaters. But Bishop says it's the only number that matters.
"In my opinion, it comes down to the wins," he said. "That's how you can find out how I'm doing. For a goalie to get a win, you need your teammates. But you can battle. If it's 1-0 or 6-5, you battle for a win. Save percentage and goals against can be a little skewed. Sometimes it's just a bad bounce, too. The law of averages means that things will even out, that you play through tough stretches. But as far as stats, you have to look at the wins. At the end of the day, that's what you're paid to do."
Perhaps someday, goalie analytics will get to a point where they paint an accurate picture of how someone like Bishop is playing. Until then, the Dallas goalie wants to be judged by his record.
"Wins can be overlooked as a stat. But that's the goal at the end of the day. You try to win games. You can't really worry about the numbers. Just the W's," he said.
"As far as the analytics ... if they were 100 percent correct, maybe. But when it's someone else's opinion, and it's not 100 percent correct, I'm not going to waste time looking at it."