What is your bot / user-agent called?
Our user-agent is MBCrawler/1.0.
This might be useful if you are getting 403 page statuses on your links due to your own bot / crawler / firewall policy. By whitelisting our user-agent, you would allow us to properly crawl your site.
This might be useful if you are getting 403 page statuses on your links due to your own bot / crawler / firewall policy. By whitelisting our user-agent, you would allow us to properly crawl your site.
Updated on: 24/08/2019
Thank you!