You LOST Her?!?!

Whether he looking for a toy, his shoes, his favorite pillow, his "pink princess gloves" or a person if Brogan cannot almost immediately find it he comes to me, asks me where it is and if I reply "I don't know."
He'll respond, "You lost it?"
Which, irrationally, makes me rather annoyed. I know he's not actually accusing me of losing it, it's just how he phrases sentences and questions. "You" replaces I, we, me, us, she, he. So in reality he's more than likely telling me he can't find whatever it is because he lost it. But, because I am a hormonal mother I take it personally and throw back "I didn't lose anything but maybe you did." Then go help him find whatever it is he feels is lost.
The other evening I was in the laundry room attending to the sea of clothes when I heard this exchange:
Brogan, "Where's mommy?"
Jerry, "I'm not sure."
Brogan (accusingly), "You lost HER?!?!"
Jerry (slightly defensively) "I didn't lose her I just don't know where she is."
It amused me to hear someone else in that situation and to hear they react pretty much the same way I do. Is that wrong?