Du doesn’t mean “of the” here but is closer to “some” in “give me some”.

The balai is used as a metaphor. The person to whom du balai is addressed is asked to be swept out by this imaginary broom.

The expression seems to have first appeared by the end of the 19th century in the play “Une opposition” (In Un Client sérieux, 1897) by Georges Courteline, a famous writer of Théâtre de boulevard. Maybe Courteline created it or perhaps it was already used.