There is an easier fix. Respect your doctors opinion, and respect your own opinion, and proceed to eat whatever you want. If you otherwise like your doctor, no reason to change because you disagree on diet, that is a very small part of their practice.
If you have a personality clash with them, and are in need for a new doctor you can always check out http://primaldocs.com/
A quick search for Queens, NY brought up http://primaldocs.com/listing/cheriee-slobodsky/
Consider a Naturopathic doctor (ND) over a regular MD for non surgical treatments and care.
Politely agree to disagree with him on the subject of nutrition, keeping in mind that doctors are generally uneducated and misinformed about diet. Remind him you eat a lot of other plant foods that provide you with plenty of vitamins minerals and fiber.