You have a few options.
- Accept it and move on. The question is not egregiously bad, so once a few days have passed, people will stop noticing it, and it will stop being downvoted. This is what I recommend, honestly. You have been posting enough good contributions that the downvotes on that post are just a drop in the bucket.
- Flag the post, select "other (needs ♦ moderator attention)", and describe your issue. I am not sure whether the community managers generally do anything about this type of situation, but it can't hurt to check. This will not cost you reputation - see below.
- Ask Yuttadhammo if he would be okay with deleting his answer. If he says yes and chooses to delete his answer, you will then be able to delete your question (assuming other people do not also post answers).
- You may request that your account be dissociated from the post altogether by contacting the Stack Exchange team (select "Other" from the menu). This is sort of a "nuclear option" and I strongly recommend against it unless you think the question is so incredibly bad that you cannot bear to have your name associated with it. (Actually, I am not even sure if it is possible to request this on a post-by-post basis rather than on a whole-account level.)
Note: I don't think it is appropriate that people are downvoting your question - it is not bad, per se; just off-topic.
Note also: the only time flags affect reputation is if a post is flagged as spam or offensive, abusive, or hate speech. If enough of those flags are cast on a post, the post will be auto-deleted and the person who made that post will suffer a -100 reputation penalty. Other types of flags do not directly affect your reputation.