It was only a couple miles there and back, and honestly the encounter(s) happened very quickly. It's only midday when you return to the tavern where you met, and Ezren is not around.