New spider Adsbot-Google to check landing page quality

A new Google spider is likely to be showing up in your logs soon - AdsBot-Google will be used to monitor AdWords landing page quality.

While we strongly recommend against restricting our system's automatic review of your landing page, you can edit your site's robots.txt file to avoid a review.

The file must explicitly exclude your page from our system visits as follows:

To prevent AdsBot-Google from accessing your site, add the following
to your robots.txt file:

User-agent: AdsBot-Google
Disallow: /

To prevent AdsBot-Google from accessing parts of your site, add
the following to your robots.txt file:

User-agent: AdsBot-Google
Disallow: /exclude/

Where exclude represents the directories you don't want the AdWords system to visit.

Read more