More important than the total number of slaves is the number of new slaves per year. If these slaves get treated relatively well and on average survive 20 years of service you're looking at 2000 new slaves being born or brought in per year. If you go onto that market trying to buy 4000 slaves within a month you'll be raising prices a lot. If you're paying enough you might still be able to get them, but you're basically using up the supply of the next two years, and prices won't go back to normal for longer than that.

If the life expectancy of a slave in this city is closer to two years that's 20000 coming in per year. If you buy up a few thousand of those you'll be noticed, but the market can regulate itself well enough that you'll have no significant problems (outside of the usual problems with buying thousands of people from slavers and transporting them).