The result sees the Bundesliga side advance to the semifinals on the back of a 4-3 aggregate win across the two legs, while Barca's hopes of a European trophy in 2021-22 were ended.
The visitors, roared on by massive travelling support at the Camp Nou, got off to a dream start when Barca's Eric Garcia hauled down Jesper Lindstrom in the area and Kostic calmly beat Marc-Andre ter Stegen from the spot to give Frankfurt the lead inside of three minutes.
Xavi Hernandez's side settled quickly after conceding and began to force the issue, with Ousmane Dembele setting up Pierre-Emerick Aubameyang for a header that went over the crossbar and Ronald Araujo volleying straight at Frankfurt keeper Kevin Trapp in the following minutes.
Despite the hosts' pressure, Frankfurt scored again to double their lead, this time courtesy of Rafael Borre's absolute thunderbolt from distance that blistered past Ter Stegen in the Barcelona goal.
"I am speechless," Trapp said after the match. "Nobody expected this, to be honest. Everyone expected us to suffer here.
"We needed to take our chances and today we were very efficient. We are in the last four and we want to get as far as possible. We showed we don't have to hide from anybody, but no doubt, now we want to make the final."
With Barcelona trailing 2-0, the match was wide open from the start of the second half. Aubameyang forced a point-blank save from Trapp in the 48th minute and Ter Stegen did well to keep out a powerful Lindstrom shot at the other end soon after.
Xavi brought on Adama Traore for Aubameyang just past the hour mark and substituted Sergino Dest in place of Oscar Mingueza in hopes of jumpstarting the attack, but Kostic scored his second of the night with a low drive past Ter Stegen in the 67th minute to effectively seal the victory for Frankfurt.
Sergio Busquets scored for Barca as second-half stoppage time began and Memphis Depay added one from the spot just before the final whistle, but the deficit was too much to overcome as Frankfurt saw out a famous result.
"What we did was insufficient today despite the heroic attempt in the end," Xavi said. "It is very disappointing. We did not play well. We had control and possession, but found no spaces to hurt them."