Your touch does the damage. The attack roll is to determine if you successfully touch your opponent. Just because you're grappling them doesn't mean you're automatically touching them.