Don’t look that it is a nice Amazon spider, it might be stealing your content!

Comments

1. Matt says

   September 5, 2022 at 5:34 am

   Scraper bots have been driving me crazy and causing massive damage to my news artciles. I need RSS on as I have a Google News approved website that shows in top stories every day. I tired everything, cloudflare rules, Bot fight mods, premium security plugins and all at the same time.

   Nothing was stopping these disrespeful lazy crooks using my own content to cause caninical issues and rank above me causing the demotion of my own original content.

   The biggiest question, with all the technology Google has, they can’t stop scapped stolen content from indexing and ranking? Apparently they cant. I think (They wont) and you are forced to spend hours of your day filling out the Google copyright form. Which can taken them over a week to remove the stolen content (by then its too late because you lose all your trending headline traffic and may not get your original rightful ranking positon back either

   Anyways. I found one of the best ways to stop 95% of this rubbish was to actualy just use Wordfence,
   Wordfence wont automatically take care of this by default. So you gotta set this up manually.

   Here is how

   Most of these scrapers are hosted and coming in via AWS and Linode hosting. So just jump into wordfence and select ‘Blocking” and then choose “Custom Pattern”

   From there make two blocking rules.

   Under the custom pattern enter these two in the “Hostname” section

   *.amazonaws.com
   And
   *.linode.com

   This will block a a massive amount of scrapers inlcuding RSS feed scraping.

   You can also do this via cloudflare. With the free account you only get 5 ruiles. I always lock down my wp-admin and wp-login.php to my personal dedicated IP address for access. You should always at least have those two rules in place. Or at least lock down those admin URLs to your country only

   However, I have the $20 per month premum account becuase you really need much more than just 5 rules and the other protections cloudflare WAF offers if you have a good website that scammrs and crooks want a peice of.

   So. What I do, I also have a WAF rule in cloudflare blocking *.amazonaws.com and *.linode.com

   This blocks these scrapers both at the edge and locally

   Note. This is of course no good if you are actually hositng your own website with one of these. You will block yourself : ) So make sure not to take note what I wrote here of you are hosting with any of these these companies.

   Now. There are some manual content crooks out there. They will come and do a simple copy and paste. This is can be stopped by using a a few different plugins.

   Eg. Hide My WP Ghost, or you can simply search for many other copy & paste blocking plugins. The only problem with these copy blocking plugins. Every single one of them has a performance imapct on your website due to java script I think.

   Well. This is the best way to stop a huge amount of scapers.

   Just use wordfence to block aws and linode host names
   You also block aws and linode via a cloudflare waf rule.

   No need to add all those IP to block lists. However, nice list, If you are going to use them it would a good idea to check if they are stil allocated to AWS. It could be outdated by now

   Reply
   • Michael (the author) says

     September 6, 2022 at 5:19 am

     Yes, modern web application firewalls and security plugins help dealing with this. By the way, I reviews a while ago both Wordfence and Sucuri WAF and other premium security options on this page.
     Thanks for such a detailed comment, Matt!

     Reply
2. Jen says

   May 17, 2018 at 2:34 pm

   Can you give an update on how to handle AWS? You say a firewall but what settings?

   Reply
   • Michael (the author) says

     May 17, 2018 at 3:45 pm

     Hi Jen,
     if you mean the firewall I mentioned in the end of the article, then it handles malicious traffic automatically. You don’t need any special settings in there.
     Moreover, using a firewall like that is a preferred method since blocking the ranges of IPs is too straight forward method which works only in a short-run. Malicious traffic sources do not stay on the same IPs for months.
     Meanwhile you still can block IPs using the described method with editing .htaccess. However for a more solid solution it’s advised to use a firewall.

     Reply
3. Jo-Ann Armstrong says

   March 9, 2018 at 6:38 pm

   In my opinon, you should only all major search spiders on your site, why? major search engines have a legit reason to be on your site, all other spiders serve no real purpose, and worst case is they maybe crawling your content, and scooping it up as they go to serve some other low life that does not have the ability or money in order to come up with his/her own content. Your biggest and only friend is google bot, no other search engine legit or scam will bring in traffic that Google can provide, there is no legit reason why spiders overseas should be on my site, so they get blocked, there is no reason for amazonaws to be on my site, so they are blocked too. It does not matter if spiders are harmless, if they are not helping me such as google spider, then those spiders are not useful.

   Reply
   • Michael (the author) says

     March 10, 2018 at 3:08 am

     Hi Jo-Ann,
     Thanks for your comment.
     What you say does make sense.
     However, in my opinion this is too strict approach. I don’t mind my site to be indexed by alternative search engines (which names may change through time), as well as by some other services like Ahrefs.
     Using a firewall (e.g. the one I mentioned in the article) seems like an easier solution to me.
     Besides, when you talk about allowing only selected bots to crawl your site, you probably mean robots.txt, which may be neglected by harmful bots. So restricting access by robots.txt will probably affect only good bots. Again, using a firewall service looks like a better idea.

     Reply
     • Jo-Ann Armstrong says

       March 10, 2018 at 9:30 am

       There are two sides to every story, another option is to do nothing. Google has done a good job placing fear into the minds of webmasters concerning this or that. If your content is going to be stolen, then it will be stolen, even if you stop amazonaws, then there are tons of other spiders that can and will slip by, and do what they are designed to do. I know webmasters would prefer to keep their content on their site, and not others, however, this is nothing to lose sleep over as your content is going to be scrapped at some point anyways. Do a google search on “is duplicate content penalty a myth?” and you will find multiple authority sites that explain that duplicate content penalty is a myth, however,if you have a new site, and a larger site stills your content, then that larger site will benefit since they have more authority, however, if the smaller site steals content from the authority site, then it will not have any effect on the larger sites since that larger site already has authority and trust in the eyes of Google,not only that, I have seen authroity sites based on something but duplicate content, and half the internet has duplicate content.

       Reply
       • Michael (the author) says

         March 10, 2018 at 1:20 pm

         To keep it short: I agree 🙂

         Reply
4. _removed says

   September 22, 2017 at 2:12 pm

   _removed are providing amazon scraping tools without IP blocked and Banned.Using that tools any one can scrape million of records easily.
   Below is Few Tools we provide

   1.Amazon Scraping and Reprice tools
   2.Amazon competitor products monitor tools
   3.FBA scraping tools
   4.Buybox Scraping tools
   5.Amazon title modifications alert tools
   6.Amazon to Ebay Price comparisons
   7.Amazon to Ebay automatic scraping and listing tools and maintain price and stocks
   8.Aliexpress to Ebay Automatic listing tools and maintain price and stocks
   9.Walmart,Bhphotovideo,best buy and many other website to Ebay listing tools and maintain price and stocks
   10.Ebay scraping tools and Tracking tools
   11.ASIN track tools
   12.Ebay Listing tools
   13.Scrape million of data from any website. etc…..
   based on your needs i can develop or modify this tools
   Contact us for demo

   #1 Web Scraping Software – _removed |‎ Free Developer Support

   Reply
   • Michael (the author) says

     September 22, 2017 at 3:23 pm

     I just leave this spam comment a bit edited (I’ve removed the URL) as a reminder for everyone that there are lots of bots stealing your bandwidth and data. In order to be protected from such activities I recommend using website firewalls. For example, Sucuri WAF which I use.

     Reply
5. murad abuseta says

   July 8, 2015 at 10:35 am

   Hi professional researcher
   you save me three times
   what you did here is not just talk you just search and give us the most important this in all subject and content thank you

   Reply
   • Michael (the author) says

     July 8, 2015 at 12:30 pm

     Hey Murad,
     Thanks for your words.
     I try my best to deliver as much value as possible for my readers.

     Reply
6. Manik says

   September 16, 2014 at 2:24 am

   Hi Michael,
   Thanks for this valuable post with the great list of Web Scrapers IP.

   Reply
   • Michael (the author) says

     September 16, 2014 at 3:20 am

     Hey Manik, no problem!

     Reply
7. Leena Shah says

   September 8, 2014 at 6:59 pm

   This is Great Your Content Article Guide Thanks For Sharing me

   Reply
   • Michael (the author) says

     September 9, 2014 at 3:12 am

     Leena, glad I could be helpful.

     Reply
     • Leena Shah says

       September 10, 2014 at 9:05 pm

       Thanks Add My Comments

       Reply
       • Michael (the author) says

         September 11, 2014 at 2:05 am

         No problem. English is not my native language as well, so I understand.

         By the way, you can use a free translation service “Google translate” (translate.google.com) to translate from your language to English if you want. It works pretty well.

         Reply
8. Rajiv says

   August 24, 2014 at 3:02 am

   Wow Thanks for making us aware of this treat. Does adding this code in the .htaccess file affect the SEO?

   Please tell.

   Reply
   • Michael (the author) says

     August 24, 2014 at 6:51 am

     Hi Rajiv,

     it affects SEO in a good way 🙂

     The main SEO advantage of using this code is to help your content from being stolen by web spiders that use AWS IPs. Otherwise stolen and thus duplicated content may drop your rankings.

     Thanks for your question!

     Reply
9. Emmanuel says

   August 21, 2014 at 6:50 am

   Hello Michael,
   It is a great sharing this message about. This is my first time hearing of this spider though and I obviously need to stop it as well.

   So you mean adding #Blocking AWS scrapers will do the magic? Thanks for this and do have a great day!

   Reply
   • Michael (the author) says

     August 21, 2014 at 1:20 pm

     Hi Emmanuel,

     Thanks for stopping by!

     Yes, just add the code to your .htaccess file and the spiders from these IPs will not be able to access your website.

     Reply
10. SEO Arbiter says

    August 14, 2014 at 10:05 pm

    Hi Michael,

    That’s interesting. So basically it’s people using AWS services to scrape your content and use it, not actually Amazon, right?

    Have you found it made a big difference (In terms of saving bandwidth etc)?

    I have never heard of people using AWS to scrape sites so I’ll have to do some more reading on this.

    Thanks for the post.

    Robert.

    Reply
    • Michael (the author) says

      August 15, 2014 at 12:55 am

      Hi Robert,
      Thanks for your questions.

      > So basically it’s people using AWS services to scrape your content and use it, not actually Amazon, right?
      Absolutely.

      > Have you found it made a big difference (In terms of saving bandwidth etc)?
      As regards saving bandwidth, there are just 3 weeks have passed after I implemented AWS blocking (not much time to judge), but anyway here are the results:

      – I’ve compared the period of 22 days from AWS blocking implementation till now (i.e. from the 24th of Jul till the 14th of Aug) with the period of 22 days before the implementation (from the 2nd of Jul till the 23th of Jul).
      – I’ve got 83% more visitors (with the same pageviews per visitor stats) [according to Google Analytics]
      – I’ve got just 30% increase of bandwidth. [according to AWStats]
      Thus, roughly, I’ve saved 40% of bandwidth ((1.83-1.3)/1.3 = 40%).

      However, I believe, that websites with a different popularity can get a different bandwidth savings.

      Reply
    • Adam says

      February 22, 2018 at 9:48 am

      Not exactly, amazon is letting users rebind IP very easily. So amazon could be aware of this feature and could know that rebinding IP adress on AWS has single purpose only – scraping.

      Reply
      • Michael (the author) says

        February 23, 2018 at 12:33 pm

        Hi Adam,
        Thanks for your comment.
        I’d agree with you. I wrote this article almost 3 years ago. So the part which still stays actual for sure now is the recommendation to use a firewall service from the threats like this.
        Thanks for your comment!

        Reply
11. Robinsh says

    August 4, 2014 at 2:24 pm

    Hey Michael, thanks for sharing these IPs that we should block in order to host a more powerful and working webpage focusing upon the basic business. I also read Neil Patel’s blog but couldn’t see that specific post in past about what you were talking in this update.

    Anyway thanks for the update, that helped me.

    Reply
    • Michael (the author) says

      August 5, 2014 at 1:14 am

      Hi Robinsh, Neil’s post is over here. And the part which says about Amazon Server scrapers has a headline “Lesson #2: Stop scraper sites”.
      Thanks for your comment!

      Reply