I would just tell the player that downtime activities are just that, down time activities. Anything that can't be solved with a few simple rolls and no major consequence for failure is not a down time activity, it's an adventure. And when I'm DMing, adventures have to include the rest of the group.