I know, I've read the entire comic, some parts of it multiple times.

The issue for Pa'anuri, specifically, is that they are very big and the Book is (for them) very, very small. Without extensive help from baryonic matter creatures, Pa'anuri would have no way to actually tell where the Book is, what the Book is, nevermind why the Book does anything. So while they could trivially destroy it, it would be near-impossible for them to detect and pin-point, before it would annihilate them too.