get on someone's wick vs try one's patience

To annoy or upset someone, especially by repeated disagreeable actions. 

To be difficult to deal with; annoy.