Detecting and compensating for link scanners/spam checkers/clickbots for Marketo users

The Problem

Link scanners / spam filters / click bots are network security devices enterprise IT deploys to block links to risky websites. These devices will click on every link within an email you send before the email is delivered to the recipient’s email server. These phantom clicks are sometimes followed by real human clicks, but often not. This type of phantom click is a pain for marketers because it makes analysis of and reporting on email campaign performance virtually impossible since you can’t distinguish what’s real and what’s not. Worse yet, these phantom clicks rack up high behavior scores for those leads and cause them to become MQLs prematurely and repeatedly, compounding the age-old sales complaint that marketing sends them bad leads.

This is a problem that affects every marketing automation user. Just do a quick search for the above terms in the Marketo online forum, and you’ll see many discussions of this topic, as well as all kinds of ways Marketo users have tried to hack around it, including:

• Manually maintaining a blacklist of domains based on manual review of activity logs
• Adding a link that is visible to machine but invisible to human
• Using a secondary landing page after a click through

Unfortunately, none of these methods work reliably or are scalable.

We saw the same problem here at Openprise and set out to solve the problem for ourselves. We believe we have come up with a reliable and scalable solution, and would like to share that with you.

Our Solution

It is actually fairly easy for a human to spot a lead/domain using link scanner by just examining the sequence of email activities for a given email sent to a lead. Here is an example from Openprise’ own Marketo database:

This webinar email to this lead was sent, delivered, and clicked on 4 times BEFORE it was ever opened.  This means the link scanner first clicked on all the links in this email before delivering it to the email server, which confirmed receipt. There was no more Click Email activity after the email was opened, so the human recipient never clicked on the webinar link. Every email sent to this individual exhibited this behavior, so this was not a one-time data glitch.

By the way, this lead had a behavior score of 5,000 because “it” clicked on every link in every email we sent him.

As we like to say at Openprise, if you can write down your logic and tasks, we can likely automate it. So we built out an automation recipe to identify these types of out-of-sequence email activity patterns:

1. Click Email activity happens before Email Open or Email Delivered

2. Email Bounces, yet it still shows Email Open or Email Delivered

This recipe produces two lists:

1. List of emails / leads that exhibited these behaviors more than 3 times in the last 6 months
2. List of domains that exhibited these behaviors more than 3 times in the last 6 months

The threshold is adjustable. We think 3 is a good number. You can also include as much activity data as you wish in this analysis. For our own deployment, we are using the entire 3+ years of Marketo activity data we archived in Openprise.

This fully automated process consists of these steps:

1. Import Marketo activity data
2. Append and clean activity data
3. Identify these data patterns from the cleaned and appended activity data
4. Export data back into Marketo

We have this process run daily, so the list of leads and domain is up-to-date.

For Openprise customers, you can find this new Recipe in your Openprise account and here is a Cookbook explaining how to use the Recipe.

How to Leverage These Lists

Once you have this always up-to-date lists of leads and domains exhibiting link scanner and click bot behavior, how can you leverage that to remedy your scoring program and reports? We recommend the following:

1. Create a custom field in Marketo to flag if a lead is using link scanner, say “Is Using Link Scanner”. Openprise can automatically update this flag.
2. Create a list in Marketo to hold these leads that have been flagged. You can either have Openprise automatically populate a static list for you, or use a smart list based on the flag above.

Now that Marketo knows which lead is using a link scanner, you can:

1. Exclude these leads from your email program performance analysis
2. Compensate for the click in your scoring program

For scoring, you can go with a fairly basic algorithm of just not score these leads for email clicks or deduct a fixed score. You can also implement a more sophisticated scoring program to still accurately score a human click after machine clicks. This can be a complicated setup in Marketo. If you have good ideas on how to do this in Marketo efficiently, we would love to hear it.

Alternatively, you can just use Openprise to do the scoring instead of Marketo, which is what we have done. Using Openprise do the scoring not only makes it simple for us to accommodate for these type of tricky scenarios, we also get account score.

This solution definitely has room to improve. There are surely more patterns we can add to our detection algorithm or more effective ways of dealing with a lead that has been identified as a click bot or as using a link scanner. We’d love to hear what you think of this solution and possible ways to improve it.