Reasons for ruined castles:
1. The family dies out and the claim isn't clear, or the family lives elsewhere and doesn't have enough money to keep "the old house" in good repair.
2. It was damaged during a siege and never repaired
3. The border changed enough that it was no longer on a frontier and therefore no longer needed for defense.
4. The central government ordered it slighted, either as a preemptive move against a potentially rebellious lord or as punishment afterward.
5. The stone was taken for other projects.
Chateau Gaillard had several of those happen. It sits above the Seine, controlling river traffic between Paris and the sea. It was taken by lengthy sieges several times, left uninhabited, occupied by bandits, slighted in the late 1500s, stone taken for use by the local monastery.