Il est rectangulaire is correct if it is clear you are talking about a specific book, C’est en papier is not unless you haven’t already tell it is a book, or if you are talking about what is a book in general, i.e. not a specific book.

• Ce livre, il est rectangulaire.
• Un livre, c’est rectangulaire.

More generally, c’est is used when the object is not yet identified, for example when you are guessing what it could be or when you are telling a what an object is.

Il est or Elle est is used when you know what object it is, and then the gender you need to use for the following adjective.

To summarize, if the first sentence is C’est un livre, the rule is very simple: in the sentences following it, always use il est as livre is masculine and never c’est as ce is neutral here so can only refer to unnamed or generic objects.

• C’est un livre: correct (because of the article indéfini which is un)
• Il est orange, rouge, et bleu: correct (because you are introducing an adjective)
• C’est en papier: incorrect (because what you are talking about is undefined. It can be defined if you complete it with something like this: C’est en papier qu’est faite la poupée)
• Il est assez grand: correct (for the same reason as explained via the second bullet point above)
• Il est rectangulaire: correct (for the same reason as explained via the second bullet point: rectangulaire is an adjective)

So my question is when describing a shape of an object, do you use
“c’est” or “il/elle est”?

You can use both c’est and il/elle est, depending on the context:

1. C’est un bus: it is a bus
2. Elle est grande: It is big (Elle replaces, for instance, maison meaning a house)

In example 2. we know what are we talking about (a house), so we can use Elle to refer to it.

In example 1. we are referring to a specific object.

C’est is followed by a self-contained description, normally a noun phrase. It can be used both for an object that’s already been mentioned and for an object that is being introduced to the conversation.

C’est un livre.
C’est un roman.
C’est un objet rectangulaire.
C’est un objet en papier.
C’est son cadeau d’anniversaire.
C’est le plus beau livre que j’aie jamais lu.

Il est describes a characteristic of an object. Because il is a definite personal pronoun, the object is one that is already the topic of the conversation.

Il est orange, rouge et bleu.
Il est en papier.
Il est rectangulaire.
Il est passionnant.

For answering my question: you use il/elle est when describing an object because it’s an adjective. So it would be like: C’est un stylo. Il est cylindrique.