Bing, Yandex, Baidu, Google, Facebook, Sogou, Twitter and Yahoo all do the following; they respect and play by the rules when it comes to crawling sites. When these companies wish to crawl your site, they ask for permission via a request to your robots.txt. These companies respect the sites rules and follow them. Each of these companies identify their crawler with published user agents. These known agents alert to their identity and intent. These user agent are useful determining if the crawlers seeks the mobile or desktop version of your site. Of course, these user agents are easy to spoof, which is why each of these companies backs their user agent with an rdns entry confirming their identify.
Intel Security / McAfee works by their own set of rules and as a security company pretty much acts like a rogue agent attacking your site. Intel/McAfee does not respect robots. Intel/McAfee uses a generic user agent that is common from bad actor states like the Ukraine, Russia and many countries in Asia; “Java/1.8.0_51” There is no way to distinguish a rogue crawler from Intel/McAfee. After much research we have determined that Intel/McAfee use IP's assigned to such names as pathdefender, or Digital Ocean. How many strikes does a webmaster need to block Intel/McAfee as rogue traffic.
After receiving unwanted traffic from the domain 104.131.0.0/104.131.255.255 we blocked all of Digital Ocean. Standing up a new site using our common blocked traffic database, Intel/McAfee was blocked as a bad actor in this domain, failing to identifying themselves and using user agents that are known for attacks, they were justifiably blocked.
Here is the rub and where the hubris and true arrogance of this purported security company surfaces. Intel/McAfee is blocked from access to our established sites, yet denied access, they labeled our new site as suspicious, they said they scanned our site and found viruses and malware and they presented a highly inflammatory warning to go back to safety; a load of excrement. The fact is we blocked them, so the message about the malware and viruses they found was a flat out lie.
Through inference and trial and error we determined who was behind the rogue crawls from 104.131.94.252 and 72.5.66.72 with the known hacker user agents, IP's we had blocked as bad actors; Intel! We had no choice but to succumb to Intel/McAfee's extortion; either we allow their rogue agents from unknown IP's onto our site and install their site seal, or they would block our site. Pay up or get blocked, we had no choice but to allow their rogue traffic and install their site seal; extortion.
We find Intell/McAfee to be bad actors that do not properly identify themselves, and if you do not allow their rogue agents onto your site, they will maliciously and without merit label your site as dangerous when in fact they are the problem.
We urge Intell/McAfee to follow the example of Google, Microsoft, Yahoo, Twitter, Baidu, Facebook and Sogou and establish and publish a branded user agent, then back it up with published and identifiable ip addresses via rdns. As security company, shame on you Intel for acting like the very actors you purport to block with your security software.
How Many Badges Can You Collect?
