If you ask me a question I don't know, I'm not going to answer it.

If you ask me anything I don't know, I'm not going to answer.

I refuse to answer that question on the grounds that I don't know the answer

If you don't know what to do with an answer, don't ask the question.

It's a very good question, very direct, and I'm not going to answer it.

You have every right to ask me that. It's your prerogative. But I have the right to not answer.

I don't answer questions about conjectures.

I'm trying to think of all the reasons that are appropriate for me to refuse to answer that question.

I felt the question was stupid so I didn't answer.

His shoulder bumped mine again. "Can I ask you something?"
I didn't answer. He was going to ask me anyway. People don't say that if they don't want to pry something out of you.

There are questions that you don't ask because you're afraid of the answers to them.

Don't tell me what I'm doing; I don't want to know.

don't ask me a lot of questions if you don't like the answers

Why me, lord? Don't answer that!