In this sentence hasard has its primary meaning of "chance". It means by chance, by accident, haphazardly.

• On ne s’était pas donné rendez-vous. Je l’ai rencontré par hasard an allant faire les courses.
  → We hadn’t planned a meeting. I met him by chance on my way to the shops.

• « C’est souvent du hasard que nait l’opinion. » (a famous quote from Jean de la Fontaine)
  → It is often by chance that opinion is born¹.

Au hasard is a set phrase that means at random/randomly.

• Si tu ne sais pas lequel choisir prends en un au hasard.
  → If you don’t know which one to choose take one at random.

That’s how I understand those lyrics:
" At first it was by chance (by accident/haphazardly), sometimes you come home a little late…"

^{1 My literal translation. Walter Thornbury’s translation in his translation of the Fables in verse: "Opinion is the child of Chance".}

Hasard is uncountable: du hasard = "de le" hasard = some hasard.

I’d translate it "At first, it was random", i.e. it was happening unpredictably.

The underlying meaning is that the late arrivals were initially rare but as time went on, they became more and more common and therefore predictable.

C’était au hasard wouldn’t have work well because there was no actual randomness in what happened, it was just perceived as such by the person speaking.