Was it signed, or was it some coward who sent it anonymously? I feel for you----it is always difficult to garden in the city.
But....is there truth to it? Did you start gardening on it without proper zoning changes? Or is there a restriction for gardens in your area? One thing I have learned about living in town is that there are a lot of hoops to jump through these days. And is it true that you put down cow manure that still had a lot of odor to it? From what I have read and been told, when it is ready to be put on a garden, the odor should be gone (although I have never used manure). Does it really stink, or is someone being overly sensitive? If it is really stinky, that is an issue that I would also have if I lived next door to it---homes are just too close in the city.
But I think that it is all something that should be do-able in a city, and I would prefer it to the two neighbors that I have who have yet to mow their "lawns" this year, and have grasses and weeds up to my thigh (although it doesn't bother me as much as it does the rest of the neighborhood, but I am not too picky and have some unconventional landscaping choices myself). Give me a garden next door any day instead of an untended lot!
But some people will just never be happy---they like being outraged.