To make a memorable impression or have a strong effect on someone; to impact someone's feelings, especially when such an effect is not expected or wanted.
To irritate someone.
How often have the words drive around the bend and get under someone's skin occurred in a corpus of books? (source: Google Ngram Viewer )