The object glows so that players can't make coins or gems to spend them in shops. To claim a conjured sword breaks when used to attack because it took damage is being grossly unfair. The fighter's sword isn't taking damage. The great weapon master barbarian's great sword isn't taken damage when he attacks. The conjured sword is being used as designed.

To deny the sword use is to say the enemy automatically knows a Silent Image of a wall of stone is an illusion or at least automatically gets an intelligence check where as if Wall of Stone was cast the enemy would accept it as real and behave accordingly. That's the DM metagaming an ability to uselessness for fear the players is trying to get away with something. The DM knows the source because he has to. That doesn't mean the NPC/monster knows.

That's not to say there should be no DM adjudication at all. As much as a DM shouldn't always assume players are trying to get away with something, neither should players actually do try to get away with something. Call it a house rule if you must, but to say a conjured thing can't be used as a spell component is reasonable. Particular spells use up an expensive component on purpose. Accepting the PC saw the real thing once, always having it available for the spell to be cast almost at will (spell slot limited) might lead to unforeseen consequences. Playability of the game matters.

The Conjurer otherwise creating an object to be used as the object is supposed to be used is the whole point of the ability. A conjured door stop doesn't stop being a door stop when it blocks the door from closing because oh noes it got scratched from the friction thus took damage so sorry so sad the door slams shut.