Something (especially a minor thing) which ruins or spoils everything else, or makes it less pleasant; a nuisance or problem; a disagreeable or unpleasant detail.
stumbling block
noun
A hindrance, obstacle or impediment; an opportunity for error.
How often have the words fly in the ointment and stumbling block occurred in a corpus of books? (source: Google Ngram Viewer )