Allergies cause inflammation, and repeated exposure to food allergens reduces your guts ability to absorb nutrients and can be very very dangerous.
I'd argue that's the best case scenario. Witnessing someone going into anaphylactic shock from eating something with honey is a whole different level of nightmare fuel.
Another thing to be aware of: there is lactose intolerance and milk allergy. I had a friend who constantly got served lactose free catering at conferences etc because he announced way beforehand that he has a milk allergy. He always felt like an asshole not eating the obviously milk protein containing foods that were prepared especially for him. But, like, when the alternative is going to the hospital, politeness won't be your choice.