Midfielder Ilkay Gundogan gave Germany the lead with a penalty on 33 minutes at the Khalifa International Stadium in Doha, and forward Kai Havertz had the ball in the back of the net just ahead of half-time but the goal was ruled out for offside.
The shock result was a repeat of their 2018 World Cup nightmare when, as defending champions, they lost their opener to Mexico and, after another defeat by South Korea, were condemned to a surprise group-stage exit.
Germany players placed their hands over their mouths during their prematch team photo, while the country's interior minister Nancy Faeser sat next to FIFA president Gianni Infantino wearing the "OneLove" armband as FIFA's threat of sanctions over protest continued.
Hansi Flick's side enjoyed early dominance and were rewarded when Joshua Kimmich picked out David Raum in acres of space in the box and goalkeeper Shuichi Gonda clumsily brought him down as he turned, with Gundogan dispatching the penalty in the 33rd minute.
There was initially little change to the pattern in the second half as Jamal Musiala shot over after a penetrating run and Gundogan clipped a post.
Germany goalkeeper Manuel Neuer, appearing in his fourth World Cup, was alert to save from Hiroki Sakai and did well again soon after to block from Takumi Minamino, only for fellow substitute Doan to smash in the loose ball.
Suddenly buoyed, Japan pushed on and Asano showed great strength to bring down a high free kick and hold off defender Nico Schlotterbeck and fantastic technique to hammer the ball into the smallest of spaces.
Germany launched all-out attack from then on, to no avail, and now face a potentially must-win game next against Spain.
Japan next face Costa Rica and will already be dreaming of the last 16.
Information from Reuters contributed to this report.