Automatic navigation based on search queries. How to quickly parse keywords and get conversions from search engines

Explains how to save time when collecting keywords for your site using clustering (and explains what it is and why you might need it).

In this article we will talk about accelerating the creation of a semantic core for the subsequent creation of landing pages and their optimization. The faster we collect, clean and group our semantics, the faster we will attract search traffic. One of the longest and most difficult stages of working with the kernel is grouping or clustering queries. Using automation at this stage allows you to save a lot of time. Therefore, I decided to consider in detail all the issues and nuances of this process in this material.

What is the semantic core

Let's start with the basics. The semantic core (semantics) is an ordered set of queries that characterize a service, product or type of activity presented on the site in the most accurate way. Collecting semantics is the first step on the path to search traffic. Queries distributed across site pages help both understand what exactly the resource pages are about and offer these web pages to users in accordance with their search queries.

Search words collected in different ways are grouped, and based on the analysis of the resulting groups, a mental map is compiled with the distribution of words on specific pages. After this, in accordance with the resulting scheme, materials for the site are created and pages are optimized.

The process of combining keywords from the core into topic groups is called clustering.

Why group requests?

Clustering (grouping) of keywords is the ungrouping of queries into clusters (groups united by one topic) automatically.

The main advantages of clustering:

You can quickly understand which keywords to put on one web page and promote simultaneously and together, and which to promote separately

You can quickly distribute a huge number of keywords from the assembled core into clusters

You can increase the accuracy of grouping and get rid of the “human factor”

How were they grouped before?

Previously, queries were grouped manually “by eye”, combining them into groups based on semantic similarity. Let me give you a hypothetical real estate example. Thus, one group included the queries “rent an apartment”, “how much does it cost to rent an apartment” and “rent a one-room apartment”.

As a rule, when grouping, queries with the same intent (user intent) that differed in semantics were not taken into account. For this reason, the keyword “rent apartments” often did not fall into the cluster with the keyword “rent an apartment.”

According to some sources, the number of erroneous distributions with this method reached up to thirty percent, which reduced the speed and efficiency of promotion.

Clustering by search tops

The most popular method of grouping queries is based on the principle of similarity of Yandex or Google search results automatically. This method, in addition to the main advantages of speed and time saving, has an important property. When clustering, requests for which the page cannot be promoted will not be included in one group.

Disadvantages include a decrease in the quality of clustering when the quality of search results for a specific request or for a selected niche as a whole is not high enough, and the difficulty of independently implementing the method due to the need to use a multi-stage algorithm and collect a large amount of data from search results.

Clustering methods

When clustering based on tops, search queries can be combined in two ways:

When using the first method, the main key query is taken as the basis for the group, and the rest are compared with it, analyzing the number of common URLs in Yandex results. An additional keyword is included in the group if the number of shared URLs exceeds a specified threshold.

When using the hard method, requests are included in one group only if for all requests there is the same set of URLs shown for each of the keywords in the results.

One of the main parameters of clustering is the “threshold”. This term refers to the number of common URLs to obtain a cluster. The higher it is, the more accurate the clusters are, but the smaller they become.

When working with lower metrics, too many inappropriate requests end up in groups. Obviously, different clustering methods give different results. Thus, Soft gives excellent completeness indicators, but insufficient accuracy indicators.

Does this mean that Soft should not be used? No, it all depends on your goals. When promoting traffic and the need to place the largest number of requests on a page, Soft will be the optimal solution. But if you need pure and very precise semantics on the page, you only need to use Hard.

Review of programs and services

Using free tools usually involves grouping requests manually.

MS Excel, Google Docs, OpenOffice

These tools do not cluster search queries, but only make the process easier for the optimizer. This method allows you to achieve high quality grouping of key phrases by processing the results by an SEO specialist.

Advantages:

Free (except Excel)

Ability to work online (in the case of Google Docs)

Universal – make it possible to use formulas and so on

More precise grouping due to manual work

Flaws:

Low data processing speed due to the manual method

The need to make backups (with the exception of Google Docs)

Service Coolakov.ru

Allows automatic grouping of previously collected search queries. The breakdown is based on the similarity of the Yandex top 10. Processes up to a thousand requests for free; if you need more, you need to order the service separately via email, indicating the region and providing a semantic core. In this case, the cost will be 20 kopecks per request.

Advantages:

Free (up to a thousand requests)

Works online

Flexible choice of clustering threshold (from 1 to 10 intersections)

Flaws:

Clustering queries only by Yandex

Groups need to be further adjusted manually

There is no way to bind the cluster to the site URL

KeyAssort

A desktop program that allows you to perform collection and clustering, create a site structure and find leaders in the selected niche. The user can structure the semantics by creating categories and sorting queries into them. The cost of the program is 1,900 rubles. You can test the functionality for free, the only limitation is the inability to export queries.

Advantages:

Possibility of collecting top 5, top 10 (up to top 50)

Availability of filters for manual refinement

There is a free demo version

Flaws:

• Desktop program for Windows only; on MacOS it can be launched via a virtual machine

  The need for manual modification of clusters

RushAnalytics

One of the online clustering services based on the top 10. It is possible to set the method and strength of the group. Also, when clustering, you can use manual markers based on intents. The results are provided in an Excel file on two tabs: the first is clusters, the second is non-clustered queries.

Advantages:

Works online

Fast collection speed

User-friendly interface

Additional functionality (indexation check, etc.)

Flaws:

• Relatively expensive

  Just like everywhere else, clusters need to be additionally sorted manually

Semparser

Clustering is performed automatically; after all requests are ungrouped, a window opens for the user in which errors can be corrected. The clustering results are downloaded in an Excel file, in which there are several tabs - on the first - the resulting groups with details, on the second - only groups, on the third - top topics. The service makes it possible to determine the strength of the group.

The work algorithm is classic - based on search results. If a group cannot be found for the query, then the step is repeated again, but the required number of intersections is reduced.

Advantages:

Works online

Saving projects

Intent accounting

Test mode (clustering of 50 search queries after registration)

Flaws:

• Manual correction of groups is required

Just Magic

An automatic service for grouping requests using the Hard method based on the top 10 search results of Yandex and Google. By default, the Moscow region is used, but you can change it by entering the necessary data. The clusterer solves the problem of which requests can be promoted on one page, separating commercial and informational ones, for the main and internal pages, and more.

Advantages:

Working online

Information content of the results (groups of clusters, number of main pages in SERP, subject of phrases, geo-dependence, check for “commerciality”)

Possibility of clustering based on region

Determining relevant pages if a site is specified

Flaws:

High cost of clustering

Access to the clustering section is provided only to registered users

What to do after clustering

If you choose automatic clustering, the data obtained as a result of using services or programs must be modified manually. In the process of manual refinement, based on logic, some queries or clusters are deleted, others are separated or, conversely, merged.

Each group corresponds to a separate page on the site. For each page, you need to prepare a Title, Description, H1 and URL, which will use search queries from the cluster, as well as an alt attribute for the img tag and provide for the use of queries in other zones.

Non-clustered queries

Requests that were not included in any cluster do not need to be deleted. You can add them to separate pages of the site (for example, in the “Articles” or “Blog” sections) or include them in one of the existing clusters according to their meaning.

Final check

The final check is done at the stage of drawing up a content plan - the correspondence of requests in the cluster to user intentions and the possible completeness of the topic are determined.

Conclusion

I recommend always using clustering when promoting a site, regardless of the number of requests being promoted. The only exceptions are topics in which competition is extremely low - high-quality grouping of queries using the top method in such topics is practically impossible due to the lack of relevant answers in the output.

The main advantage of using automatic clustering is, first of all, speeding up the work, which is especially important when parsing large kernels. Using clusterers, an SEO specialist can ungroup a huge number of queries in just a few hours, which previously could have taken weeks or even months to complete the same amount of work.

Automatic clustering does not give 100% accurate results; in most cases, clusters must be modified manually. But it significantly simplifies the work of the optimizer, allows you to create the most correct site structure and prepare competent technical specifications.

For whom: for beginner SEO specialists

Level of training: elementary

Every webmaster or website owner has probably encountered this problem. Many people are familiar with the situation when, after installing a plugin or updating a CMS, the next day they suddenly notice a sharp drop in traffic. It’s good when the cause is known - it can be quickly eliminated. But unfortunately, it happens that traffic suddenly drops on its own, despite the fact that no work has been done on the site before.

In this article we will tell you how to correctly determine the reasons for the drop in traffic and what to do in this case.

Stage 1. Make sure that it is organic traffic that has dropped

If the type of sagging traffic was initially determined incorrectly, you will waste time and effort fixing non-existent problems. In addition, these actions may make the situation worse.

To prevent this from happening, we suggest you use our recommendations.

Let's go to Yandex.Metrica and choose « Reports – Standard reports – Sources – Sources, summary"

Then select the item "Transfers from search engines " and put a tick next to the word "Yandex ».

After that, we'll see what search traffic looks like from just that search engine:

We see that traffic has dropped specifically from the Yandex search engine.

You can check for a drop in traffic in another way. Choose “Reports – Standard reports – Sources – Search engines » .

In the list that appears, select the item "Yandex ».

If in the end we see a drop in traffic, it really is a drop in organic traffic from Yandex.

Step 2: Check for sanctions or filters

Once we have determined that there has been a drop in organic traffic, we need to find out the reason for it. To do this, we need to make sure that the site is not subject to any sanctions.

2.1. We check through the beta version of Webmaster

We go to the beta version (new version) of Yandex.Webmaster.

If Webmaster has any messages about fatal errors, then your site is under sanctions.

For example, thanks to such a message, it becomes clear in which direction to work next - you need to remove low-quality links.

2.2. We write a letter to Yandex

The site may be subject to restrictions, information about which does not appear in the Webmaster’s messages. In this case, you need to write to technical support.

From the main page of Yandex.Webmaster, go to "Feedback ».

After this we go to point "Site Ranking"

Select the following items in turn:

Then check the box “ Ask a question to support":

• choose your site;
• indicate the region through which we are moving;
• We give an example of a request for which the site is not visible;
• We write a letter describing the problem.

If Yandex indicates in a letter that your site violates any requirements (for example, the site is over-optimized), then with a 99% probability this is the reason for the drop in traffic.

2.3. We check with the serviceSeolib

Stage 3. Look for the cause of the traffic drop

If the previous steps did not provide clarity, then you need to do the following:

3.1. Checking with search trafficGoogle

First, let's compare traffic with the Google search engine. It is quite possible that the Google search engine experienced the same drop.

If you see the same drop in the Google search engine, then this can usually be due to technical errors.

There are also several options here:

Option 1. Traffic dropped to 0

This means that the site has critical errors that affected the functioning of the site as a whole. Need to check:

• Is the site accessible?
• Is the entire site blocked from indexing?
• Are there any incorrect redirects on the site?

Option 2. Traffic dropped to 0

It follows from this that the site has some critical errors that did not affect the functioning of the site. Need to check:

• Are the Title and Description tags displayed correctly?
• whether sections or pages of the site that brought significant traffic are closed from indexing;
• Are there any incorrect redirects on the site?
• Is the content reflected correctly on the pages (text, images);
• server response speed.

3.2. We are looking for the reason for the drop in traffic in Yandex

If traffic has dropped from the Yandex search engine, then to diagnose the problem we need to understand which sections or pages have lost traffic.

Pages

Let's look at the report " Login pages » with segmentation for the Yandex search engine.

If we see that traffic has dropped only to certain sections, then we need to pay attention specifically to these sections.

Need to check:

• Correct filling of Title and Description;
• correct content optimization (unique text and images, absence of spam);
• page accessibility;
• the presence of duplication of these pages (filter pages, pagination).

Requests

Let's look at the report "Search phrases" with segmentation for the Yandex search engine.

If we see that traffic has dropped only for certain phrases, we need to pay attention specifically to these phrases.

There are cases when traffic drops not only on specific pages, but only for a limited number of requests. In this case, you need to check:

• signs of over-optimization (under-optimization) by keywords;
• the presence of several pages that can respond to the same request;
• matching the TOP 10 pages for the keywords you are interested in.

Important points:

1. Branded and non-branded traffic. If you notice that brand traffic has not dropped (vital queries), then most likely the problem lies in optimization. It is necessary to look for optimization errors first.
2. If you have several login pages for one request, you need to check these pages: perhaps they are duplicates or the landing page is not optimized enough for the request, and Yandex cannot determine the most relevant page.

We described the main reasons that lead to a sharp drop in organic traffic and indicated the areas of work that need to be done on the site to eliminate them.

We wish you a rapid return of traffic!

Subscribe to our newsletter

Write to support service

Select a question to find a solution.

Choose

The data does not match the statistics of another system

The number of internal transitions has increased

No referrals from a specific site

Problem with calculating data using UTM tags

Yandex.Metrica receives information about other advertising systems from labels - parameters in the ad link. If the advertising system does not add a tag to the link to your site, you can create a link yourself using UTM or Openstat tags. For a complete list of values, see the Advertising Systems Report section.

There may be several reasons.\\n

\\n \\n \\n \\n \\n

Let's imagine the path of a website visitor from clicking on a link to\\n Yandex.Metrica reports:

\\n \\n \\n \\n

Reading tag data and analyzing reports.

\\n

As a result, between the first and last stages, data loss\\n is possible at each stage.

\\n \\n \\n

\\n

The labels are compiled incorrectly: required values ​​are missing\\n, parameter names are specified\\n incorrectly, there are extra question marks or\\n spaces, etc.

\\n \\n

\\n

\\n \\n \\n \\n

\\n

The visitor stopped loading the page and closed\\n the page before it opened. For example, this could be\\na random click on a mobile phone or\\ntablet.

\\n

The page did not load due to a problem with\\n the performance of the site or did not load correctly\\n in a certain browser.

\\n

The ad specifies a URL, and when you click on it, you will be redirected to a page on which the Yandex.Metrica counter is not installed.

\\n

The ad specifies a URL, and when clicked,\\n it will redirect to the page, and\\n the UTM tags will be lost.

\\n

When you go to the site, loading other scripts on\\n the site blocks the operation of the counter.

\\n

The ad contains an incorrect URL,\\n which does not open.

\\n

\\n

\\n \\n \\n \\n

\\n

The Yandex.Metrica counter is not installed on landing pages or is installed incorrectly.

\\n

The counter did not load on the site because a plugin\\n was installed in the visitor's browser (or on the corporate proxy server) that prohibits loading counters.

\\n

The visitor's antivirus is blocking the loading\\n of the counter.

\\n

\\n

\\n \\n \\n \\n

\\n \\n \\n \\n\\n

\\\\n \\\\n \\\\n

\\\\n\\\\n ")]))\\">

\\n\\n ")]))\">

There may be several reasons.

Reading tag data and analyzing reports.

Let's imagine the path of a website visitor from clicking on a link to Yandex.Metrica reports:

Reading tag data and analyzing reports.

As a result, between the first and last stages, there may be data loss at each stage.

Reasons at the stage of label creation

Labels are composed incorrectly: required values ​​are missing, parameter names are specified incorrectly, there are extra question marks or spaces, etc.

Reason at the stage of transition to the site

The visitor stopped loading the page and closed the page before it opened. For example, this could be an accidental click on a mobile phone or tablet.

The page did not load due to a problem with the site's performance or did not load correctly in a certain browser.

The ad specifies a URL, which when clicked will redirect you to a page on which the Yandex.Metrica counter is not installed.

The ad contains a URL that, when clicked, redirects to the page and loses the UTM tags.

When you go to the site, loading other scripts on the site blocks the counter from working.

The ad contains an incorrect URL that does not open.

Reasons at the stage of loading the counter code

Reasons at the report analysis stage

\\n\\n ")]))\">

Yandex.Metrica and other web analytics systems or advertising accounts record visitor interactions with the site in different ways and calculate statistics. For example, in an advertising account, a click can be considered not only to go to a website, but also to open more detailed information about an ad or go to a group page on a social network. Yandex.Metrica will record only those visitors who went to the site.

In addition, check that:

In Yandex.Metrica and another system, correct data for the same period is compared.

In Yandex.Metrica reports, the default attribution is Last Transition. If necessary, change the attribution settings in Yandex.Metrica so that they match the attribution settings of your system.

\\n

In Yandex.Metrica, in the section Settings → Filters, too harsh\\n filters are connected. As a result, the data is not collected completely.

\\n

Yandex.Metrica has a robot filtering system. Perhaps another statistics system also took into account visits from robots. You can check the presence\\n of robot traffic on the site in the report Standard reports→ Monitoring → Robots.

\\n

The counter did not load on the site because a plugin\\n was installed in the visitor's browser (or on the corporate proxy server) that prohibits loading counters. In addition,\\n an antivirus with high\\n security settings may prevent\\n counters from loading.

\\n

Also make sure that the Yandex.Metrica counter\\n is installed on all domains and subdomains for which\\n statistics are taken into account in another system.\\n You can view data on domains on which\\n the Yandex.Metrica counter is installed in report Standard reports→ Contents → Popular.

Please note: Customer Support responds to questions\\\\nby email only. Do not follow the instructions of people who call you and\\\\n introduce themselves as Yandex.Metrica support service.\\\\n

\\\\n \\\\n \\\\n

\\\\n\\\\n ")]))\\">

Please note: Customer Support only answers questions via email. Do not follow the instructions of people who call you and introduce themselves as Yandex.Metrica support service.

This may happen for the following reasons:

\\n

In Yandex.Metrica, in the section Settings → Filters Robot filtering is disabled. As a result, reports display information on all robots that visited the site.

\\n

Another system has filters that are too harsh.

\\n

The counter of another system is not installed on all\\n domains, including mobile versions\\n of the site.

\\n

Please note: Customer Support responds to questions\\\\nby email only. Do not follow the instructions of people who call you and\\\\n introduce themselves as Yandex.Metrica support service.\\\\n

\\\\n \\\\n \\\\n

\\\\n\\\\n ")]))\\">

Please note: Customer Support only answers questions via email. Do not follow the instructions of people who call you and introduce themselves as Yandex.Metrica support service.

\\n\\n ")]))\">

This may happen for the following reasons:

In Yandex.Metrica, in the section Settings → Filters, too harsh filters are connected. As a result, the data is not fully collected.

Yandex.Metrica has a robot filtering system. Perhaps another statistical system also took into account robot visits. You can check the presence of robot traffic on the site in the report Standard reports→ Monitoring → Robots.

The counter did not load on the site because a plugin was installed in the visitor's browser (or corporate proxy server) that prohibits loading counters. In addition, an antivirus with high-level security settings may prevent the counters from loading.

Also make sure that the Yandex.Metrica counter is installed on all domains and subdomains for which statistics are taken into account in another system. You can view data on domains on which the Yandex.Metrica counter is installed in the report Standard reports→ Contents → Popular.

Please note: Customer Support only answers questions via email. Do not follow the instructions of people who call you and\\n introduce themselves as Yandex.Metrica support service.\\n

\\n\\n ")]))\">

Please note: Customer Support only responds to questions via email. Do not follow the instructions of people who call you and introduce themselves as Yandex.Metrica support service.

This may happen for the following reasons:

In Yandex.Metrica, in the section Settings → Filters Robot filtering is disabled. As a result, the reports display information on all robots that visited the site.

Another system has filters that are too harsh.

The counter of another system is not installed on all domains, including mobile versions of the site.

Please note: Customer Support only answers questions via email. Do not follow the instructions of people who call you and\\n introduce themselves as Yandex.Metrica support service.\\n

\\n\\n ")]))\">

Please note: Customer Support only responds to questions via email. Do not follow the instructions of people who call you and introduce themselves as Yandex.Metrica support service.

This may happen for the following reasons:

The counter code is not installed on one of the site login pages.

There may be JavaScript code on the site that runs before the counter code and does a redirect within the site. For example, this could be AB testing code. The presence of such code can be detected by a redirect that occurs on the site (usually, this is a redirect to the same page of the site with additional parameters in the URL). To resolve this issue, contact your webmaster.

Your hosting provider may have enabled protection against DDOS attacks. Such protection can add redirects before going to the site. Contact your hosting provider and ask them to disable such protection.

Please note: Customer Support only answers questions via email. Do not follow the instructions of people who call you and\\n introduce themselves as Yandex.Metrica support service.\\n

\\n\\n ")]))\">

Please note: Customer Support only responds to questions via email. Do not follow the instructions of people who call you and introduce themselves as Yandex.Metrica support service.

Information from referral source reports is needed to know where a visitor is coming to a site from. Yandex metrics uses the following terms: internal transitions and direct visits. Sources can be search engines, search phrases, social networks and many others.

You can choose the display of the report according to your taste and needs by clicking on the icons on the right.

For each type of source of visits there is a definition in the Yandex Metrics help ]]> ]]>

What are “direct calls”?

Direct user entry all visits to the site are called directly from browser bookmarks and from a manually typed URL in the browser. This is traffic without a referrer.

Referrers are the pages from which a link to your site was made.

Direct calls Transitions from communication programs, chats, e-books and courses are also counted.

In the statistics you will also find data on transitions:

From search engines– hits from search results based on the user’s search results; the query is also recorded in this report.

From saved pages - first the user saves the page on his computer, and later uses it to access the site. In this case, the referrer will be something like this: C:\\Documents\Vasya\page.html).

Cmailings.

In order for such transitions to count, you need to use .

And there is another term “source undefined”.

You should pay close attention to the number of direct visits. The more direct visits, the more popular your web resource.

We need to strive to increase the rate of direct visits, because... it ultimately directly affects.

What are "internal transitions"?

If a person came to the page, looked through it, and then left for lunch for another hour, the page remains open. In this case, the session is terminated if the user is absent for more than 30 minutes. He returned and continued to look further - he was assigned a new visit. This is it called internal transition. It comes in as new, but not from an external site, not from bookmarks, but from the same web page.

Internal transition It also happens when a visitor lands on a 404 page or another that does not have a Yandex metrics counter. And then it goes to some blog page. In this case, the entry point is not visible.

Internal transition It will also be when the tab with the site is open, but not active. At this time, the person looks at another browser tab and does not return to your site for half an hour or more.

The question arises, what is the impact of such inactive half-hour visits to the site? It is logical to assume that the time spent is counted, and we also receive a second visit from the same person. And the “visits” indicator is what we mainly focus on. This means that the behavioral factor will improve.

As you can see, using Yandex metrics opens your eyes to the site in many aspects.

Good luck with your analysis and results!