FOXBOROUGH, Mass. -- The Baltimore Ravens didn't need a Tom Brady comeback to remind them they need to improve their secondary. The 35-31 loss to the New England Patriots on Saturday just underscored how much the Ravens have to significantly upgrade their pass defense in order to become a championship team again.

Even though the Patriots capitalized on trickery, the AFC divisional playoff game was lost on missed tackles and blown coverages by cornerback Rashaan Melvin and defensive back Matt Elam. The Ravens defense failed to hold three leads -- including two 14-point ones -- because the deficiencies of a depleted secondary were exposed by 408 yards passing.

The Ravens got within two wins of reaching the Super Bowl when their pass rush harassed quarterbacks. But the Ravens couldn't get to the quarterback enough Saturday, and Brady had time to pick apart a secondary that allowed the most passing yards in the franchise's 19-year existence.

The secondary couldn't slow down Brady in the red zone, in which he was 3-of-5 with two touchdowns. The Ravens couldn't stop Brady in the fourth quarter, when he completed eight of nine passes for 72 yards.

When it mattered the most, the Ravens couldn't put away the Patriots.

"I thought the guys all played their hearts out, played hard and played well," coach John Harbaugh said. "They made plays, so give them credit for making some plays. So, I don't want, I really don't want to disparage our guys."

While Harbaugh didn't want to point any fingers, the statistics certainly do.

Joe Flacco threw four touchdowns and put the Ravens in position where most teams win. Over the past 10 postseasons, teams were 42-8 (.840) when scoring 30 or more points prior Saturday's game.

The Ravens were able to run the ball (Justin Forsett gained 129 yards) and stop the run (the Patriots' 14 rushing yards are the fewest by a winning team in NFL postseason history). Brady didn't even hand the ball off in the second half.

The one area in which the Ravens stumbled was pass defense. The Patriots dropped back to pass on 54 of their 66 plays Saturday (81.8 percent), and teams were 3-59 over the past five seasons (including playoffs) when throwing more than 80 percent of the time. Against the Ravens defense, it proved to be a winning formula. The Patriots saw the weakness and attacked it repeatedly.

"The offense scored 31 points. For us, they held up their end of the bargain," defensive end Chris Canty said. "Defensively, we didn't do enough. We've got to help our guys on the back end, and we didn't today."

It's equally admirable and remarkable that the Ravens advanced this far with this patchwork defensive backfield. The Ravens started seven cornerbacks this season due to injuries and poor play, and they went through four safeties. This defense has had to overcome the season-ending foot injury to cornerback Jimmy Smith, the struggles of banged-up cornerback Lardarius Webb and the poor play of Elam, a first-round pick in 2013.

The Ravens just didn't have the players to match up with Brady. The Patriots tied the game at 14 when Elam missed a tackle on wide receiver Danny Amendola at the 13-yard line. The Patriots tied the game at 28 when Melvin allowed Amendola to run past him on a trick play, on which wide receiver Julian Edelman took a lateral from Brady and hit a wide-open Amendola for a 51-yard touchdown.

With the Ravens ahead by three points in the fourth quarter, the secondary once again flopped. Brady targeted Melvin on three of his last four passes, including a 23-yard touchdown to Brandon LaFell. Melvin, who was out of football for five weeks and spent time on two practice squads this season, ran stride for stride with Melvin, but he didn't have the awareness to turn around to make a play.

"You live and you learn," Melvin said.

The Ravens have shown they can fix their weaknesses swiftly. It took only a year for the Ravens to go from having one of the worst offensive lines to one of the best.

Now, the focus has to be on the secondary, especially at cornerback. Smith and Asa Jackson are expected back from season-ending injuries. Webb is under contract, although the Ravens might want to do something about his $12 million salary-cap hit.

The Ravens need to improve the secondary through the draft and free agency. Unlike last offseason, when the Ravens didn't address cornerback sufficiently, they need to do all they can so the defense can avoid feeling it let the team down.

"They wanted it more," linebacker Pernell McPhee said. "Our offense played great but our defense didn't play well. It seemed like they wanted it more than we did."