OpenLinkprofiler is an excellent free tool that shows you the backlinks of any website. Just pop the url in the searchbox and see what they have instore for you. You should note the link alert feature on the leftbar. A very handy feature to get alerts on new backlinks going to your website.
By doing that, you’re basically moving deeper into their funnel. Their paid product is a full blown SEO toolset called SEOprofiler. Check it out yourself!
OK enough about them lets get to the meat of this post. A lot of webmasters block bots from Moz, Majestic, Ahrefs, etc. and this should be another one they should block too.
You can either do it via .htaccess or your WordPress plugin. Here’s the user agent of their bot.
# Open Link Profiler # Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+) # http://OpenLinkProfiler.org/bot User-agent: spbot Disallow: /
Are you sure that bots blocks via .htaccess ? As I remember “Disallow: /” command adding in robots.txt
robots.txt is just a directive. The bots can choose to ignore it. When you block via .htaccess, apache config or firewall, you have better control by giving it a 403. The quoted block is in robots.txt format. I’m just displaying it to show the User Agent.
Please bear in mind that User Agent can be easily faked. If you want to be sure, double reverse the IP and compare with a IP database like MaxMind’s GeoIP2 Precision Services then block the IP as it’s much harder to spoof IP.
Of course if you’re just trying to block spbot, UA blocking is enough as they had publicly disclosed their UA and stated that their bots plays by the rules. And if they really play by the rules, doing it via robots.txt is enough 😉 If they deliberately choose to ignore robots.txt, there’s no stopping them thus why I’m recommending to use .htaccess, apache config or firewall rules.
Got around 800k+ requests from spbot, and my site got suspended because of too much resource usage. Thanks for the tip!