So, what keeps each "family" limited to 20 bushes to cultivate?

Are silver fruit difficult to transport or store?

In a real world econostate, I would've expected each family to deal with their surplus family members by explanding their farm land and number of crops to expand feed. Eventually this would lead to a worker class of pecan-pickers er fruit-pickers who were paid to work the fields, allowing the land owners to specialize into other pursuits. Eventually, we'd leave individual estates and have a communal grove maintained by a worker caste, allowing the non workers to get on with their business.

For your conceit to work, there has to be some arbitrary constraint on why each family is limited to the max number of cultivated plants. That's not a bad thing, but as a player I'd want to know what that constraint was.

Each plant requires a large amount of surrounding space to itself with no other silver fruit plants nearby
Each grove self limits to a size of about 20 plants and dies off if another grove is planted nearby.

Something like that maybe.