To make a memorable impression or have a strong effect on someone; to impact someone's feelings, especially when such an effect is not expected or wanted.
rile up
verb
To rile; to aggravate.
How often have the words get under someone's skin and rile up occurred in a corpus of books? (source: Google Ngram Viewer )