In this article, we’ll cover some of the most important things in setting your website to appear in Internet search engine results – & nbsp; web indexing .
I’m sure we all want other users to find our hard-built websites with interesting content, such as Google, List, and more. However, if we don’t have the indexing set up correctly, our site will never appear in search results.
Robots.txt file – the cornerstone of search engines
Undoubtedly, the most important thing you can do to index your site is to set up the & nbsp; robots.txt file correctly. Every Internet search engine accesses this file and searches for it directly in the root of your website.
This file provides search engines with basic information on how to access your site. In short, you will tell him where he can and where he cannot. If you deny search engines access everywhere in this file, no one will find your site in the search results. For example, a properly set up robots.txt file might look like this:
User-agent & nbsp; tells who the following rules apply to (in this case, everyone). The Allow directive allows access everywhere from the root of your site, and the Disallow directive forbids robots from accessing site administration (wp-admin for a WordPress site). p>
It is also recommended to add a link to the Sitemap here ( sitemap.xml ). This is especially necessary if your sitemap is not at the standard & nbsp; /sitemap.xml address.
Meta tag robots
Meta tag robots are located in the source code in the <head> and it might look something like this:
This allows us to set how search engines treat specific pages on your site. The basic directives we can use for the content attribute include the following:
- Index – Allows search engines to index this page, making it appear in search results
- Noindex – Disables the site index the page for search engines to display in search results
- follow – search engines can follow links on the page and give them authority ( more about link authority )
- nofollow – Search engines will not follow links on the page and give them authority
- all – Abbreviated notation for use
index, follow - none – abbreviated notation for use
noindex, nofollow, all disabled
WordPress indexing settings
If you use the WordPress content management system , you can set up a robots meta tag directly in the web administration. You can find this option in the Settings > Impressions.
It is important to mention that if you have this option unchecked (ie indexing enabled), WordPress will not list the meta tag robots in the source code at all, so the robots will understand that they have everything enabled. This option also has no effect on your robots.txt settings.
A situation where we don’t want to index the site
Finally, I’d like to mention situations in which we don’t want search engines to index our site.
In practice, I most often encounter a situation where we develop websites on a test server with a test URL. Here we definitely do not want our test server to appear in search results, on the one hand users would go to the wrong version of the website and especially search engines will start penalizing the website (and the production one) for duplicate content and can thus unpleasantly affect SEO. Another example is when we have a website with private content and we don’t want anyone to know about it.
If you want to learn more about the issue of web indexing, for example, the article Web Crawling and Indexing Guide on ContentKing.
Leave a Reply