"Annoying" describes the person, place, or thing that causes one's feelings. "Annoyed" says how one feels. Something is annoying, so one feels annoyed. Remember: the cause of the feelings ends in -ing. The way one feels ends in -ed. Here's a list of other word pairs that follow the same pattern:

Examples:
▶ That was an exhausting trip. I feel exhausted!
▶ What shocking news! I'm shocked.
▶ The lecture was boring. I was so bored I almost fell asleep.
▶ These driving directions are confusing. I'm confused.
▶ The massage was energizing. I felt energized afterwards.
▶ This is exciting news! I'm so excited.