who wants to stir up shit! I do know that compose can have a slight odor for a short period of time, since I have one here, but it is not much and you have to get close to it for it to be a problem. Certainly not enough to keep everyone indoors and locked up.
I understand all about the idiot rules with zoning. I lived in a subdivision with community rules that made no sense at all, just as you are running into. Is this a lot next to your own house? Maybe you have done the right thing to get approval for a garage....piss them off about your garden and you can get a variance to put in a garage. I would not let the "community" have the garden if that is not what you want to do with YOUR property. There are things that make no sense....but we already know that.
I suppose that you should have gotten the variance before you did anything with the property, since the neighbors would not have been up in arms. But that is water under the bridge. Don't let them intimidate you and don't let them get you angry. Go for the permits for special use and be totally sane about the insanity that they are throwing your way. And good luck. Let us know how it works out.