It means “What’s up?”, “What’s happening?”. Translated literally, it means “what is there?”: “il y a” means “there is” and you turn it into a question. In this context, the friend is asking what the other called about, stating that he just left work (explaining why he couldn’t answer the phone).

Note that the meaning can change a little depending on the context. For example if you see someone looking upset and you ask “Qu’est-ce qu’il y a ?”, you’re asking if there is something making them upset (so it would be closer to “What’s wrong?”).

It’s not rude, but I think it’s a little bit informal. If you were to message back your CEO after you had missed her call, you probably wouldn’t start your message with “Qu’est-ce qu’il y a ?” but rather something a bit more formal and explicit about what you’re asking (“Pourquoi m’avez vous appelé ?” or something like that).

“Qu’est-ce qu’il y a?” is familiar and is used to ask what was the reason of your call. You may also say:

Qu’est-ce qu’il se passe ?
Qu’est-ce que tu voulais / Tu voulais quoi ? (both familiar)

So there are 2 cases :

1. The person is worried about what happens: do you need his/her help ? Is there an important issue ?
2. The person doesn’t want to be bothered, since (s)he was at work, so (s)he’s hopping you had a good reason to disturb him/her.

You could translate this with “What’s going on?”, basically. Depending on the context, the meaning will automatically follow

The English equivalent would be “What is it?”, as in “Why did you call me?”.

In a more formal context, you would use “Qu’y a-t-il ?”. The two phrases are based on “il y a”, which means “there is”