To annoy (someone) so much that they cannot concentrate.
get under someone's skin
verb
To irritate someone.
To make a memorable impression or have a strong effect on someone; to impact someone's feelings, especially when such an effect is not expected or wanted.
How often have the words drive to distraction and get under someone's skin occurred in a corpus of books? (source: Google Ngram Viewer )