We have never worried about them in any game I was in, with the exception of expensive ones like diamond dust. It generally wouldn't have made a difference anyway, I haven't seen many casters without component pouches.