Did my doctor get annoyed with me? Was I being rude?

My doctor was telling me that he's going on holiday next week to Morocco. I said to him "please take me with you" but I meant it as a joke. Then he said "that's where me and my wife went for our honeymoon - that's why we're going again". He kind of glared at me and didn't smile at all. Then he just stopped talking about the holiday and started talking about my symptoms...I wasn't trying to be rude or flirtatious. I was just joking. Do you think he was a bit offended? I'm 17 by the way...