500 isn't an unreasonably low level for your 20's, also free testosterone might be a more telling biomarker. If free T is also low, look into your all around lifestyle. Not enough sleep, poor mental health, nutrient deficiencies (e.g. zinc, magnesium, vitamin D), infections, high levels of body fat, inactivity, and lots of less than healthy aspects of a man's life can lower T.
As far as dietary tweaks, lots of protein seems to lower testosterone: http://www.ncbi.nlm.nih.gov/pubmed/3573976. This might not be the case, but I figured it was more likely than other things given common paleo diets. I would consider protein intake of secondary of importance to looking at previously mentioned unhealthy conditions or behaviors which may be affecting your T levels.