Technical SEO and server logs
Many recent articles from SEO experts including Rand Fishkin have been giving importance to the “technical” part of SEO. Infact, Moz’s blog has a full category on technical SEO. Recently the more advanced technical SEO folks have started using server logs for analysis. Refer MOZ, Internet Marketing Podcast. When google or bing index your site, they use “bots” or crawlers. The only data you have of their visits is the server log. It is during this indexing process the search engines decide how to rank you. If they see a “Page Not Found” error on a page or a slow server performance, they will note and hurt your site rankings.
Google and other search engines visit your site regularly
The way your website shows up in search results in google is that google “indexes” your website. Technical SEO experts will tell you to that there are factors in your control such as robots.txt, meta tags, quality back links and sitemaps. In order to keep its index uptodate, google uses “bots” or crawlers to traverse your site. The bots would honor the robots.txt and sitemap, but would generally visit a page, get its content, find the “href” tags and traverse those pages as well. This technique is used by other search engines such as Microsoft Bing and yahoo. Other bots too visit the site – such as Alexa.
BOT traffic is not seen in google analytics
But do you or your SEO experts have the insight to what the google bots are actually seeing? Did they get a 404 not found error? How much time did your site respond in to the google bot?
Some of these are seen in Google webmaster tools search console, but the server has consolidated data from all crawlers. You cannot see this in the google analytics dashboard. The google analytics dashboard is created from the access to the site via the browser. Google bots do not use a browser to “crawl” your website.
Google is not the only crawler of your site – and you possibly need to know these hidden visitors and what they are seeing.
luroConnect Insight presents the “BOT Report”.
What is luroConnect Insight?
luroConnect Insight is a cloud platform that monitors Magento servers, giving insights with dashboards, alerts and reports.
What is the BOT report?
The BOT report tells you everything about the bots that visit your site.
- Which bots visited your site in the last 24 hours
- Which bots visited your site in the last one month
- How fast did the server respond to on an average to all bots and to a specific bot
- Were any errors seen by the bots?
- What pages were visited by the bots?
These reports are generated daily and either visible on a dashboard or sent as an email.
What can I do with the BOT report?
- Know what bots are indexing your site.
- Act on errors – some may be old links. You can correlate this information with the google dashboard as well.
- Review the response time on URLs – some URLs such as category filters maybe too slow. If the speed cannot be fixed, the robots.txt or your sitemap.xml may be modified in consultation with your SEO expert.
- If the server may be consuming resources that would be better left for your real traffic. In consultation with an SEO expert, slow down crawling for selective BOTs.
How do I get my BOT report?
Easy! If you have a apache or nginx log file from your hosting, go to www.luroconnect.com/botreport/sample and upload the file and choose the type of report you want. You will receive the selected report in email in about ½ hour or less.
Alternative, signup for luroConnect Insights at luroconnect.com/signup. Fill in the preferred mode of contact and time. We will contact you. Based on your hosting configuration, we will quickly tell you if this service will work for you out-of-the-box. Our onboarding team will help you install the agent on the server (or with your permission install the agent). Wait for 24 hours and you will be able to download the BOT report.
