I've always said "the best martial arts is knowing when to cross the street."

I've had 2 important instructors who come to mind. The 1st was a San Soo practitioner, a kung fu style known for its brutality & he echoed precisely what you have said; only do as much as is needed to terminate the conflict with the least amount of harm.

The 2nd was a practitioner who also taught Chinese medicine. He suggested that for someone to start a physical conflict, there must be some form of imbalance in their body influencing their aggressive behavior. A skilled martial artist should strive to apply a "treatment" to restore balance to the aggressor. If sufficiently skilled, the aggressor would ultimately stand down & then query "why were we fighting?" (Perhaps a fanciful concept but illustrative of your point)