Soit dit is a shortcut to cela soit dit and introducing here a clause that make the whole sentence close to an apophasis, i.e. you say something that you wouldn’t have say in the first place. A literary equivalent of soit dit sans vouloir te froisser is sauf votre respect (with all due respect).

You can find soit dit in the similar idioms soit-dit en passant (by the way, incidentally) and soit dit entre nous (don’t tell anyone).