In that last case, a half dead party member of the last party sent into the fortress, running by screaming and yelling about wild mobs of hairy sentient killer apes would get them on their toes. A few well-timed spot- and listen checks and they'll be doing double guards in no time.

One time in a wheel of time adventure I was running, I had the party tracked by a fairly low-level animal, a sort of hairy, walking bird who was a scavenger type. The animal was really good in tracking, and was following the adventurers night and day because of the left-overs of their meals and the waste of their hunted animals. They moved at day, the thing cought up with them in the evening and was circeling them at night. The creature was not from that continent but was used to people (it was half trained as a hunt-dog, eh, bird by invaders). I had them spot the creature's huge eyes at night when the moon was shining on it. They heard things moving in the bushes, and sometimes they saw some movement on the track behind them. I kept it all quite eerie and mysterious, never letting them see the whole beast. It totally freaked them out. Since there were other thing to do in that vast forest, I used it as a sort of mood-setter and kept it going for 3 evenings in a row. By then half the party couldn't sleep, one was riding backwards, and all were really freaked out by the big eyes and strange noises.

In the end they smacked it silly, kept it alive and trained it as a pet. The thing wouldn't really do tricks (can;t have them have a natural 20 tracker doing their bidding all the time) but it would guard and be creepy to townsfolk. They loved it.