Personally, I think listening to a book requires a lot more concentration and focus than actually holding a book and reading it. Let me explain. If I was listening to a book, I am bound to be multi-tasking, even if it means I am listening while driving. I am sure that my thoughts will go astray, and I will need to re-listen to bits and pieces if I lose track of the plot. However, if my focus goes adrift while reading a book, all I need to do is just shift my eyes two paras up, and I'll have reached the parts my brain actually registered.
Anyhow, I think there is a difference, but if you are good at listening to books, then the main point is that you know what the book is all about - doesn't matter whether you read it or listened to it.