The Hoyas (5-4, 1-3) scored first on Gunther Johnson's 28-yard keeper, driving 76 yards in nine plays after Colgate missed a 41-yard field goal.
But Diaco replied with a 28-yard keeper on fourth-and-1 to tie it at 7 at halftime before the Raiders (2-8, 1-3) cashed in on Jackson Saffold's fumble to set up Chris Puzzi's go-ahead 25-yard field goal in the third quarter.
Daramy-Swaray's 36-yard punt return and Grant Breneman's 38-yard keeper set up Mathews' 1-yard TD run for a 17-7 lead in the third quarter, then Daramy-Swaray scored on an 11-yard run for a 24-7 lead.
Johnson's 7-yard TD pass to Joshua Thomas in the fourth capped the scoring.
Breneman was 12 of 19 for 93 yards passing and gained 70 yards on 18 carries.
Johnson was 10 for 23 for 148 yards and gained 42 on eight carries.