When you and your husband did partake, how much would you smoke? Were you heavy users?
This may not be your experience, but for me it seems to be about moderation (as with most things) to avoid adverse issues with imbibing.
But if it makes you feel good to assume that pot smokers are losers that you're superior too... *shrug* You do you I guess.