add robots directive to all directive pages