Something (especially a minor thing) which ruins or spoils everything else, or makes it less pleasant; a nuisance or problem; a disagreeable or unpleasant detail.
worm in the apple
noun
A serious flaw in something otherwise good; a problem that ruins everything else.
How often have the words fly in the ointment and worm in the apple occurred in a corpus of books? (source: Google Ngram Viewer )