You can’t separate morality from intentionality. Morality isn’t concerned so much with the actual consequences of an action as with the intentions of the actor. This is why we hold someone who plots an act of terror to be immoral even if the attack is thwarted; and why we judge people who unintentionally cause an accident not to be morally culpable. So the measure of immorality isn’t harm, but the intention to cause harm.
Harm is difficult to define in absolute terms. On the grand scale of the universe, there is just the constant churn of change, none of it problematic. Things are just as they are. Zoom in on a local level, though, and you find that change is disruptive. insofar as the well-being of some beings is invested in things being a certain way, then messing with that state of affairs will cause suffering; you now have harm.
From an absolute perspective, you could say that we all just need to accept that everything changes, that there’s no state of affairs that one can pin one’s well-being to without having that pulled out from under you at some point, so everyone just needs to go with the flow. Maybe it’s true that if all beings could do this, we’d find that there’s no more suffering, and so no more harm. Just the churn of change.
But we live our lives on a local level, where the relative matters as much as the absolute. It’s at this relative level that morality comes into play. Our lives are shaped by our human mind, which has the peculiarity of being attuned to what others feel. It resonates with other minds. On some level, it suffers when others suffer, and is happy when others are happy. Metaphorically, we’re all swimming in the same pool of feeling.
If we required of each other that nobody do anything that might potentially cause a disruptive change that will negatively impact some being down the road, we would indeed be paralyzed. But that’s not what we ask. Instead, we seek a balance between the absolute and the relative. We accept that change is inevitable, and that no one can always have things as they like. But we also demand that each person, to the best of their ability, make a good-faith effort to be a good citizen of this common pool of feeling.
You can justify anything if you see your actions as “Shiva, the Destroyer and Creator”, but that errs in the direction of the absolute. Or you can get so bogged down in the knowledge that you can’t do anything without impacting another being that you cannot act at all; but that is to err in the direction of the relative. Instead we have to thread a path that allows us to function as healthy beings, while giving as much consideration as possible to the well-being of others. It’s not always easy to know how to do that.