Unfortunately I cannot find a complete clip of that scene at this time to fully refute you there, only one that starts midway through the relevant discussion, but that was very much not my takeaway from that scene. There was no suggestion that it is only the use of the Time Stone that results in timelines varying, only that removing one would result in a timeline split, since the past would have been changed such that it would no longer become the Avengers' original future.