Bonne nuit = Late evening

Bonne nuit is often used just before going to bed, even if you don’t sleep in the same bed as the person. You say that when you know the person will go to bed just after you leave.

Bonne soirée = Early evening

Bonne soirée is said when you leave the person and you know they will not go to bed immediately. As it is too early to go to bed, the person will have other activities (apéro, dinner…).

As you guessed, bonne nuit ! is only used when you know the person is going to his/her bedroom. It is then common in a family, or when you leave people going back to their rooms in a hotel. Otherwise, bonne soirée ! is used and means “enjoy your evening”.

Saying bonne nuit ! too early is not really rude. Maybe like saying good afternoon when you meet someone at 10am, but you are unlikely to have it echoed back to you as a response, instead perhaps merci, mais je ne vais pas me coucher tout de suite.

When late at night, a safe bet is bonne fin de soirée !.