The International Whaling Commission rejected Japan's proposal, which claimed that the whale stocks had recovered sufficiently. The vote was defeated 41 to 27 on Friday after a five-day meeting in Brazil.
The commission also rejected attempts to weaken its decision-making rules and set catch limits for commercial whaling.
Japan, Norway and Iceland continue to hunt whales every year in defiance of the ban.
Australia has been one of Japan's biggest criticisms on the issue and was one of three countries out of a total of 75 nations present to send a ministerial representative to the conference in Brazil.
"The Australian government worked hard with the partner countries on the commission to achieve this result," Sen. Anne Ruston, deputy minister for International Development and the Pacific, said in a statement on Friday.
"The strong defense of our delegation was successful in rejecting the Japanese proposal to resume whaling, after intense lobbying by member nations and non-governmental representatives from many parts of the world," he said.
After the vote, Japan's Agriculture Minister, Masaaki Taniai, warned his country that he could leave the commission.
"If the scientific evidence and diversity are not respected, if the commercial whaling based on science is completely denied, and if there is no possibility that the different positions and points of view coexist with understanding and mutual respect, Japan will be pressured to undertake a fundamental re-evaluation of his position as a member of the IWC, "he said.
Despite the victory, the Greens have criticized the government for not exerting enough pressure on Japan and for sending Ms. Ruston, an assistant minister, instead of Environment Minister Melissa Price.
Greens Environment spokeswoman Sarah Hanson-Young called on the government to take "urgent measures to stop Japan's intention to slaughter whales in the Southern Ocean under the vine leaf of scientific research."
In a press release, he said: "The world expects Australia to challenge Japan's plan to slaughter whales in the Southern Ocean this summer, we must prepare to send a ship to our southern waters to investigate our legal options."