OK this is the word from my DH (he used to work for the Environmental Protection Agency in a botany-type area).
Oleander is toxic but not massively so you should be able to remove it safely as long as you don't let pets / bubs chew on it, or lick your hands etc. But you should wear gloves. Google "Yellow oleander" - the first link that comes up is from
Qld Health poisons info service.
DH reckons if you're replanting with Aust natives, might be better to make it all bare then replant all at once, so you'll have a garden full of natives sooner, otherwise might take a long time till it's all sorted.
Umbrella trees he reckons are native to SE Qld but they grow as a weed outside of their home territories. So prob'ly not much of a concern and perhaps even not worth removing. They don't reproduce like rabbits unlike
Chinese Elms which is an invasive species i.e. it grows aggressively in new areas so you should get of it quick.
HTH