What is log file tracking (analysis) in SEO and how to do it?

I am sure that if you have ever visited one of those big apartment complexes as a guest, you would first be stopped by security.

Usually, this security guard will diligently note down your name, phone number, vehicle number (if you have one), what time you entered and more.

Now, imagine having someone do that for your website.

Everytime someone visits your website, be it a crawler or a user, they would note down every information possible.

What could you do with such information? 

Turns out, you can do quite a lot if you can get your hands on this information.

And that’s what log file analysis is all about. 

Log file analysis or tracking allows you to understand the behaviour of search engine crawlers on your website and help you tremendously with your SEO.

So, what is log file tracking in seo? Where can you find it? How to analyse it?

Let’s find out.

What is a log file?

In extremely simple terms, log files are the records of who visited your website and what content they accessed. It can be a search engine bot or a person going through your site. 

You can use log files to find out search engines’ actual behaviour on your site. What’s more, you can acquire data that would help you take data-driven decisions to improve your site with valuable optimisation.   

Now, you might be wondering how you can access it? Where is it stored?

How to access log files?

Only a webmaster  (a person who maintains the website) has the access to log files.

What you are looking for is a specific file, very aptly called “access log” 

To access it, you will need to log into your cPanel that your hosting has provided you with.

Here is how you can access it on Bluehost: https://www.bluehost.com/help/article/raw-access-logs-overview

The methods to access these files can vary depending on the hosting solution.

 If you want to know how to do this for your hosting, just google “how to find access log <insert your hosting provider>”

If you are using a CDN, please check with them how you can access them. (Unfortunately, Cloudflare gives this feature only in their enterprise plan 🙁 )

How does a log file look like?

This is what my log file looks like: 

I know, I know it looks like you are staring into the Matrix, not understanding a word.

Let’s take a deep breath, and take out a single file from this file and see what it has to tell us.

What data does the access log file contain?

Like your friendly security guard at the apartment complex , the access log file basically registers who visited your website, when did they visit, what did they request for, and whether they got what they requested for, and much more.

Let’s take an example entry from my access log file: 

172.106.16.190 66.249.66.206 – [07/Nov/2022:08:36:30 -0600] “GET /blog/ HTTP/1.1” 200 29197 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” “-” 0 

Let’s break it down:

Who visited: 172.106.16.190 is the IP address

When did they visit: 07/Nov/2022:08:36:30 -0600 is the timestamp

How did they visit: There are two most common HTTP ( the Hypertext Transfer Protocol) methods, GET and POST.

GET method is used to get data from a specified source

POST method is used to send data to a survey to create or update a source

In this GET is the Method

What did they request: /blog/ is the URL that was requested

Did they find it? 200 is the status code indicating that the request was successful

How much data was transferred? 29197 is the bytes transferred

Where did they come from? “-” is the referrer URL or the source but it’s empty because this request was made by a crawler

More information: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) is the user agent. In computing, a user agent is any software, acting on behalf of a user ( in your case its your browser)

But don’t worry you don’t have to go through each line to get some insights.

We have some tools for them, let’s find out more about that.

What tools to use to analyse log files?

Log analysis tools or network log analysis tools make your work easier by extracting meaningful data from logs to show if there is any app or system error. There are paid, free, and open-source log file analysis tools. Splunk, Paper tail, Power BI, Logs.io., and Screaming Frog are some of the most popular log file analysers.

SEO Log File Analyser

How to analyse log files?

How often is the log file updated?

https://www.contentkingapp.com/academy/log-file-analysis/#log-file-history

What is log file analysis in SEO

By tracking and analysing these log files, SEOs focus on understanding what search engines are actually doing on their site. 

There are different log files in which the system keeps valuable information. Tracking and analysing them can help you build your SEO strategy. 

But before knowing how to analyse them let me help you understand what server logs or log files are. 

As I told you before, at first glance log files can be quite overwhelming. So avoid making the mistake of going deep into them to find issues. Rather ask yourself questions to save yourself from falling into a black hole. 

You can start by questing 

• Which search engine is crawling your site?
• Which content is crawled the most?
• How much of your site is actually getting crawled?
• How fast is your site being crawled?
• How fast the resources are downloaded?
• How deep is your site crawled?

What insights log files can give?

Here are some important insights that you can procure with the help of log file analysis:

– Orphaned pages

Orphaned pages are pages that are out of your site structure. They don’t have internal links from other pages. Hence, no crawl simulations will be able to discover these pages, and they are forgotten easily.

 If the search engine is crawling them, your log files will show. This will help you take an action like adding them to your site structure, removing them completely, or redirecting them.

– Indexability

You can check how search engines are crawling indexable pages on your site and compare the insight with non-indexable pages. When you combine log file data with a crawl of your site, you will also be able to find out if there are unnecessary pages that might be wasting your crawl budget. 

– Comparison of the most and least crawled pages

Log files will show what pages are most crawled, more so with the large sites. You will also see that search engines are spending more time crawling pages that are of little value. You can then take action and make changes to your robot.txt file, navigation system, and internal linking structure. 

This will also help you ensure that your key pages are found and crawled, the new pages get noticed and updated pages are crawled timely. 

– Page Errors

Log files help you identify 5XX error response codes which can be used in further investigation.

Conclusion

If you have a large website, log file analysis is an unavoidable task. Instead of rushing in, you should start by separating data and monitoring the changes for a while. 

Regular analysis of logs will help you better understand how your site is crawled by search engines and build SEO strategies based on the procured data.

I hope the information I have provided helped you understand how to track and analyse log files. Try it for yourself and share with me how your experience was by commenting below. 

FAQs