It means “What’s up?”, “What’s happening?”. Translated literally, it means “what is there?”: “il y a” means “there is” and you turn it into a question. In this context, the friend is asking what the other called about, stating that he just left work (explaining why he couldn’t answer the phone).

Note that the meaning can change a little depending on the context. For example if you see someone looking upset and you ask “Qu’est-ce qu’il y a ?”, you’re asking if there is something making them upset (so it would be closer to “What’s wrong?”).

It’s not rude, but I think it’s a little bit informal. If you were to message back your CEO after you had missed her call, you probably wouldn’t start your message with “Qu’est-ce qu’il y a ?” but rather something a bit more formal and explicit about what you’re asking (“Pourquoi m’avez vous appelé ?” or something like that).

“Qu’est-ce qu’il y a?” is familiar and is used to ask what was the reason of your call. You may also say:

Qu’est-ce qu’il se passe ?
Qu’est-ce que tu voulais / Tu voulais quoi ? (both familiar)

So there are 2 cases :

1. The person is worried about what happens: do you need his/her help ? Is there an important issue ?
2. The person doesn’t want to be bothered, since (s)he was at work, so (s)he’s hopping you had a good reason to disturb him/her.

You could translate this with “What’s going on?”, basically. Depending on the context, the meaning will automatically follow 🙂

The English equivalent would be “What is it?”, as in “Why did you call me?”.

In a more formal context, you would use “Qu’y a-t-il ?”. The two phrases are based on “il y a”, which means “there is” 😉