I think the main drive for space colonies is "I don't want to live on this planet anymore!", rather than economic concerns. Mining on independent outposts doesn't happen because the people thought it's a great way to make a profit, but because they already decided they want to live there and now need something to trade for the imported goods that they need. Since this form of mining is very inefficient compared to a giant mining company processing entire asteroids, the profit margins for independent outpost mining are laughable, making those outpost very poor. This is one of the main tensions I see to create conflict.

Vegetable factories are absolutely doable today, and amazingly productive. The main thing keeping them from being hugely profitable are the costs in lighting and fertilizer. and in dry regions water purification. All of these things become much, much cheaper with access to fusion power. (Which is why commercial fusion power will be the most important technological leap since the steam engine.)

Government funded colonies and independent settlements have their free picks for planets. And why green planets are extremely rare, the number of star systems and planets means that there's still an abundance of those. If you are looking to start a new home for people, those would always be the first choice. Nobody would live in an inhospitable place because they like the weather.
While fusion power is cheap, it's not free. Especially the construction of reactors is very expensive. Poor colonies might be able to afford one reactor to produce electricity for home and some farming and mining equipment. But for growing food, they could still fall back on the actually free sunlight and rain, and make use of recycled compost to reduce their fertilizer consumption. When budgets are tight, this can make a huge difference. (In reality, the most attractive thing about vegetable factories is that you can build them in the middle of a city and cut down on transportation costs between production and consumers.)

Settlements on inhospitable planets still happen, but these are specifically mining camps. People come there to work and plan to leave once their tour is completed. And after some 100 years, the resources are depleted and the place shut down by the company. I was reminded yesterday of adding Outland on my list of movies to get ideas from.