A pirate walks into a bar with a steering wheel in his fly. The bartender asks why it is there, to which the pirate replies "Yar, I do not know myself, but it's been driving me nuts"
By Merrie
Driving Me Up My Nuts
Conflation of the expressions drive me up a wall and driving me nuts. Basically when someone is annoying you so intensely it's almost like they're kicking you in the balls. Hence the reference to the word "nuts."
