Ha! I love this question. It's like when I make soup for the kids and ask them later what did you have for dinner, and they said "We ate soup", we all have a good laugh, because they actually "drank" soup.
I personally think it's okay to say that you read the book because the person who you are talking to will know that the book in question does not have any words. But I am not sure what is the 'correct' way of answering this question.