So much does depend on how people define “harm.” It is a concept difficult to define because harm is comes in so many varied ways, but the basic idea that harming other people is wrong is fundamental to any moral philosophy. It is interesting that Libertarians do not subscribe to that harm principle that is part of every moral system. They instead use this non-aggression principle which superficially resembles the harm principle but is in fact quite different. Aggression is also difficult to define but by focusing on aggression, Libertarians divert attention from tangible harms suffered by people to perceived aggressive acts. That allows them to intellectually disregard the suffering of those harmed by immoral acts, and consider only acts they deem aggressive. This move partially explains why Libertarians do not consider systemic social discrimination to be a moral harm, but consider taxation to be an egregious aggression. Or, as you point out, it allows Libertarians to claim that criticism of Libertarianism is “aggression” even though they are in no way harmed by open dialogue about Libertarianism. Though, as you also point out, to too easily take offense at criticism is a very human tendency.