"Today's Internet is like the Wild West of old. While it's packed  with new and exciting places to go and things to do,  it's also rife with hidden threats." —Karel Obluk, CTO and main AVG 8 strategist Webmasters, this is a stickup!   Hand over your bandwidth.

AVG Watch

Since 13 July 2008, three other websites we own logged
10,907 unique IP addresses that use LinkScanner.  

What do you call a web surfer who uses an AVG product   with the new LinkScanner component activated? a)   Someone with a grudge against webmasters, who feels that all        webmasters should pay for extra bandwidth. b)   A script kiddie who wants to hit a website with a denial of service        attack, and finally has a multi-threaded script from a respected        dot-com that provides perfect cover. ("I was just checking for        dangerous pages, and it seemed stuck so I kept clicking.") c)   A typical surfer who believes rave software reviews written by        dot-com pundits.

Here's how to do a DoS (denial of service) attack with your free AVG 8 download with LinkScanner, available from CNET's download.com:

1. Set your Google preference to 100 links per page.
2. Search for site:www.byebyesucker.com in Google.
3. Now AVG's LinkScanner downloads every one of Google's links for that site at a rate of about 2 to 12 site pages per second, depending on the page size.
4. Refresh Google's search results when LinkScanner is nearly finished. There is no caching by LinkScanner, so you can rinse, lather, and repeat.

We tried this on our own site from an average DSL connection, and then looked at our log. LinkScanner grabbed 600 complete pages (but no images) in three minutes flat. This included 230 downloads of the home page and 370 downloads of deep pages that averaged 50K each. The home page pig-out is a LinkScanner specialty — it was presented with a home-page link by Google only twice, which should have meant just two downloads instead of 230.

Of course, if you try this the webmaster might detect it and track you down through your IP address. Here's what you tell the judge:

"Gosh, Your Honor, I kept getting this gray checkmark from AVG and I tried to get it to turn green by clicking again and again. I don't dare visit a website that isn't all green. As you know, Your Honor, the Internet is not a safe place for God-fearing people like us. It's full of porn and other dangerous stuff. I didn't think I was doing anything wrong. Please look at all these recommendations for AVG's security products from respected high-tech reviewers that I've collected!"

No one has attacked us, but just from the normal use of the AVG LinkScanner by enthusiastic but clueless AVG customers, we saw our traffic spike on one of our sites. On 2008-06-03 we started counting when a home-page image is fetched. This new method eliminates bots. We had to do this because the previous method of counting the page fetch itself included bots, and the numbers were getting suspicious. Sure enough, a few days later it went crazy. This graph shows the old-style counts on top, and the new-style counts on a blue line below it:


At first we suspected a script kiddie was using our graph page to test the growth of his nascent botnet, because the IP addresses for the extra traffic came from all over the world. But looking closer, we discovered that the culprit is AVG's LinkScanner component in the new version of their product. They think it's really clever to scan every link returned by a search on Google or Yahoo or Live.com, by downloading every site's home page. This happens even when the searcher never intends to click on any links. In order to foil the bad guys, AVG tries to make the scans appear to be normal traffic from that person's browser. It's called "real-time link scanning" and a lot of webmasters are furious. In the end, it's a lesson in how to stop the bad guys by becoming a bigger bad guy. We had to throw out our old-style graphs, which tracked home page traffic beginning 2006-09-01.

If you are a webmaster, currently there are four user-agents in use. The first two in the samples below are very common, the third is much less common, and the fourth is rare. The fifth one is a HEAD request, which fetches only the headers. The purpose of this is to see if you are redirecting LinkScanner. If you are not redirecting, the HEAD request is immediately followed by a GET request for the entire page.

166.82.153.56 - - [04/Jul/2008:06:01:45 -0400] "GET / HTTP/1.1" 200 6909 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
82.74.194.58 - - [04/Jul/2008:06:03:30 -0400] "GET / HTTP/1.1" 200 6909 "-" "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
151.80.8.182 - - [04/Jul/2008:06:59:58 -0400] "GET / HTTP/1.1" 200 6909 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"
216.221.63.215 - - [04/Jul/2008:08:53:56 -0400] "GET / HTTP/1.1" 200 6909 "-" "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"
79.39.18.71 - - [04/Jul/2008:09:24:22 -0400] "HEAD / HTTP/1.1" 200 - "-" "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

Recommended .htaccess file RewriteEngine on Rewritecond %{HTTP_USER_AGENT} ".*MSIE 6.0; Windows NT 5.1; SV1.$" [OR] Rewritecond %{HTTP_USER_AGENT} ".*MSIE 6.0; Windows NT 5.1;1813.$" RewriteCond %{HTTP_REFERER} ^$ ReWriteCond %{HTTP:Accept-Encoding} ^$ RewriteCond %{REQUEST_METHOD} ^GET$ RewriteRule ^.* http://freeforum.avg.com/?LinkScannerSucks [R=307,L] search your log for " 307 " (spaces are important) to count the hits

This LinkScanner component is not a minor issue. AVG claims five million downloads of this free security product during the last week of May 2008, and 70 million users worldwide.


The situation is improving, but...
Since we started this site, the hits from LinkScanner have fluctuated. Beginning July 5 they decreased. The graph below is from one of our sites that is not often associated with us, has not been changed for three years, and is very stable in Google. It averages 200 real visitors per day, and is an ideal site for tracking what's up with the AVG prefetches.

On July 5, 2008 Peter Cameron, a managing director of AVG for Australia and New Zealand, said that the AVG free edition is already fixed, and a new commercial version will be released on July 9. He pointed to a new version of free AVG 8 (build 138, July 4) that "addressed and rectified the issue," and also said that the old version of free AVG 8 was currently getting auto-updated with the new code, but that this usually takes a few days to propagate to the users.

We tested the 138 version and it no longer prefetches from websites. It looks the same as the earlier version to the user, and pretends to fetch every link from Google. It even says, "This page contains no active threats" when you mouseover the green check mark. But it's not fetching the page. All it's doing is the DNS lookup. So while it still hammers your local DNS provider pointlessly, and then lies about what it just did, at least it doesn't steal bandwidth from websites.

This graph compares the number of redirects we did for LinkScanner on our test site, and shows the dramatic drop in hits when AVG disabled LinkScanner. Our test site hits continue to be tracked on another page. The level of prefetches from LinkScanner is still unacceptable. AVG should publicly issue a recall of LinkScanner, because that's probably the best way to make sure that the hits will decline to zero.