To annoy (someone) so much that they cannot concentrate.
To tax the patience of; irk, frustrate, vex, provoke, annoy; to make angry.