When the auxiliary used to form the past tense is avoir, the past participle agreement is done with the direct object (provided that it appears before the verb, otherwise no agreement is done). In this sentence tu is the subject. The object is me, which refers to a woman, and therefore the past participle is correctly written quittée.