Victoria announced a $600 million fund to remove the dangerous material from about 500 buildings found by a task force to be high-risk.

But a request that Canberra split the cost, and pay $300 million has already been refused.

"I think it's fair to say the Commonwealth do have a liability and responsibility... but I'm not going to take no for an answer," the state's Treasurer Tim Pallas said on Wednesday.

If they don't, Victorian Premier Daniel Andrews plans to increase the building levy on developments worth more than $800,000 to cover the shortfall.

"We're not picking up the bill for what is a state responsibility," federal Treasurer Josh Frydenberg said.

Victoria's Liberal-Nationals opposition is calling for a parliamentary inquiry into the Victorian Building Authority, saying there has been a total regulatory failure.

Opposition planning spokesman, Tim Smith, also said $600 million was unlikely to be sufficient given the scale of the cladding problem.

While there was a role for the Commonwealth in regulating the building industry, it also wasn't the Morrison government's job to bail out state governments, he added.

The cladding crisis will be front and centre of a meeting of the country's building ministers in Sydney on Thursday.

It includes federal Industry Minister, Karen Andrews, who on Tuesday said any regulatory failures were of states and territories' own making and that the Commonwealth "wasn't an ATM".

But federal Labor has demanded the Morrison government pay a share of rectification works.

"It is appalling that the (federal) treasurer does not consider the safety of Australians a federal responsibility and is only too willing to pass the buck on this issue," Labor's Brendan O'Connor said in a statement on Wednesday.

The head of the Housing Industry Association in Victoria, Fiona Nield, argued the cost of rectification works should not be passed on to property buyers.

Australian Associated Press