This expression dates as far back as man has been stuffing his junk into farm animals. You fuck one goat. Meaning there are some things we do that we must carry with us, no matter the good deeds you do or the people who love you, this can all be overshadowed by one horrific deed.
An old farmer walking through his field with his grandson.
Old farmer- "Billy, you see that fence there? Your father and I built that fence with our own two hands.
Billy-''Yeah... That's great grandpa.
Old farmer- "And that barn Billy, do you see that barn? My father and I built that barn long ago together. And these fields Billy...
Billy- uh huh, didnt you fuck some goats once though?
Old farmer- "it was one goat! Geez you fuck one goat...