What Is Robots.txt? (And What Can You Do With It?)

What’s a robots.txt document?

Robots.txt is a brief textual content document that instructs internet crawlers (e.g. Googlebot) what they’re allowed to move slowly for your website online.

From the search engine marketing viewpoint, robots.txt is helping to move slowly an important pages first and stops bots from visiting pages that aren’t necessary.

Right here’s how robots.txt can seem like:

robots.txt example

The place to search out robots.txt

Discovering robots.txt information is lovely easy – cross to any area homepage and upload “/robots.txt” on the finish of it. 

It is going to display you an actual, running robots.txt document, right here’s an instance:


Robots.txt document is a public folder that may be checked nearly on any website online – you’ll be able to even in finding it on websites equivalent to Amazon, Fb, or Apple.

Why is robots.txt necessary?

The aim of the robots.txt document is to inform crawlers which portions of your website online they may be able to get right of entry to and the way they will have to have interaction with the pages.

Normally talking, it will be significant that the content material at the website online may also be crawled and listed first – serps have to search out your pages sooner than they may be able to seem as seek effects. 

Then again, in some circumstances, it’s higher to prohibit internet crawlers from visiting sure pages (e.g. empty pages, login web page to your website online, and many others.).

This may also be completed by means of the use of a robots.txt document this is at all times checked by means of crawlers first sooner than they in truth get started crawling the website online.

Word: Robots.txt document can save you serps from crawling, however now not from indexing.

Even if crawlers may well be prohibited from visiting a definite web page, serps would possibly nonetheless index it if some exterior hyperlinks are pointing to it. 

This listed web page can subsequently seem as a seek end result, however with none helpful content material – since crawlers couldn’t move slowly any information from the web page:

indexed page blocked by robots.txt

To forestall Google from indexing your pages, use different appropriate strategies (e.g. noindex meta tag) for indicating that you just don’t need some portions of your website online to seem as seek effects.

But even so the basic objective of the robots.txt document, there also are some search engine marketing advantages that may well be helpful in sure eventualities.

1. Optimize move slowly price range

The move slowly price range determines the collection of pages that internet crawlers equivalent to Googlebot will move slowly (or re-crawl) inside a definite length.

Many better internet sites typically comprise lots of unimportant pages that don’t want to be steadily (or in no way) crawled and listed.

The use of robots.txt tells serps which pages to move slowly, and which to keep away from altogether – which optimizes the potency and frequency of crawling.

2. Arrange replica content material

Robots.txt assist you to keep away from the crawling of equivalent or replica content material for your pages. 

Many internet sites comprise some type of replica content material – whether or not there are pages with URL parameters, www vs. non-www pages, an identical PDF information, and many others.

By means of mentioning those pages by the use of robots.txt, you’ll be able to set up content material that doesn’t want to be crawled and assist the hunt engine to move slowly handiest the ones pages that you need to seem in Google Seek.

3. Save you server overload

The use of robots.txt would possibly assist save you the website online server from crashing. 

Normally talking, Googlebot (and different first rate crawlers) are typically just right at figuring out how briskly they will have to move slowly your website online with out overwhelming its server capability.

Then again, chances are you’ll want to block get right of entry to to crawlers which are visiting your web page an excessive amount of and too incessantly.

In those circumstances, robots.txt can inform crawlers on which specific pages they will have to be that specialize in, leaving different portions of the website online by myself and thus combating web page overload.

Or as Martin Splitt, the Developer Recommend at Google defined:

That’s the move slowly fee, principally how a lot rigidity are we able to put for your server with out crashing anything else or affected by killing your server an excessive amount of.

As well as, chances are you’ll want to block sure bots which are inflicting web page problems – whether or not that may be a “unhealthy” bot overloading your web page with requests, or block scrapers which are looking to reproduction your entire website online’s content material.

How does the robots.txt document paintings?

The elemental rules of the way robots.txt document paintings is lovely easy – it is composed of two fundamental parts that dictate which internet crawler will have to do one thing and what precisely that are supposed to be:

  • Consumer-agents: specify which crawlers will likely be directed to keep away from (or move slowly) sure pages
  • Directives: tells user-agents what they will have to do with sure pages.

This is the most straightforward instance of the way the robots.txt document can seem like with those 2 parts:

Consumer-agent: Googlebot
Disallow: /wp-admin/

Let’s take a better take a look at either one of them.


Consumer-agent is a reputation of a particular crawler that will likely be prompt by means of directives about tips on how to move slowly your website online. 

For instance, the user-agent for the overall Google crawler is “Googlebot”, for Bing crawler it’s “BingBot”, for Yahoo “Slurp”, and many others.

To mark all sorts of internet crawlers for a definite directive without delay, you’ll be able to use the logo ” * ” (known as wildcard) – it represents all bots that “obey” the robots.txt document. 

Within the robots.txt document, it could seem like this:

Consumer-agent: * 
Disallow: /wp-admin/

Word: Needless to say there are lots of sorts of user-agents, every of them that specialize in crawling for various functions. 

If you want to peer what user-agents Google use, take a look at this evaluate of Google crawlers.


Robots.txt directives are the foundations that the required user-agent will apply.

By means of default, crawlers are prompt to move slowly each and every to be had webpage – robots.txt then specifies which pages or sections for your website online will have to now not be crawled.

There are 3 maximum commonplace regulations which are used:

  • Disallow – tells crawlers to not get right of entry to anything else this is specified inside this directive. You’ll assign a couple of disallow directions to user-agents.
  • Permit – tells crawlers that they may be able to get right of entry to some pages from the already disallowed web page segment.
  • Sitemap – when you’ve got arrange an XML sitemap, robots.txt can point out internet crawlers the place they may be able to in finding pages that you just want to move slowly by means of pointing them for your sitemap.

Right here’s an instance of the way robots.txt can seem like with those 3 easy directives:

Consumer-agent: Googlebot
Disallow: /wp-admin/ 
Permit: /wp-admin/random-content.php 
Sitemap: https://www.instance.com/sitemap.xml

With the primary line, we’ve decided that the directive applies to a particular crawler – Googlebot.

In the second one line (the directive), we instructed Googlebot that we don’t need it to get right of entry to a definite folder – on this case, the login web page for a WordPress web page.

Within the 3rd line, we added an exception – even supposing Googlebot can’t get right of entry to anything else this is beneath the /wp-admin/  folder, it will probably consult with one particular cope with.

With the fourth line, we prompt Googlebot the place to search out your Sitemap with a listing of URLs that you just want to be crawled.

There also are a couple of different helpful regulations, that may be carried out for your robots.txt document – particularly in case your web page comprises 1000’s of pages that want to be controlled.

* (Wildcard)

The wildcard * is a directive that signifies a rule for matching patterns.

The guideline is particularly helpful for internet sites that comprise lots of generated content material, filtered product pages, and many others.

For instance, as an alternative of disallowing each and every product web page beneath the /merchandise/ segment for my part (as it’s within the instance beneath):

Consumer-agent: * 
Disallow: /merchandise/footwear?
Disallow: /merchandise/boots?
Disallow: /merchandise/shoes?

We will be able to use the wildcard to disallow them suddenly:

Consumer-agent: * 
Disallow: /merchandise/*?

Within the instance above, the user-agent is prompt not to move slowly any web page beneath the /merchandise/ segment that comprises the query mark “?” (incessantly used for parameterized product class URLs).


The  $  image is used to signify the tip of a URL – crawlers may also be prompt that they shouldn’t (or will have to) move slowly URLs with a definite finishing:

Consumer-agent: *
Disallow: /*.gif$

The “ $ “ signal tells bots that they’ve to forget about all URLs that finish with “.gif“.


The  #  signal serves simply as a remark or annotation for human readers – it has no have an effect on on any user-agent, nor does it function a directive:

# We are not looking for any crawler to consult with our login web page! 
Consumer-agent: *
Disallow: /wp-admin/

Learn how to create a robots.txt document

Growing your personal robots.txt document isn’t rocket science.

If you’re the use of WordPress to your web page, you are going to have a fundamental robots.txt document already created – very similar to those proven above.

Then again, when you plan to make some further adjustments at some point, there are a couple of easy plugins that assist you to set up your robots.txt document equivalent to:

Those plugins make it simple to regulate what you need to permit and disallow, with no need to jot down any sophisticated syntax on your own.

Then again, you’ll be able to additionally edit your robots.txt document via FTP – if you’re assured in gaining access to and modifying it, then importing a textual content document is lovely simple.

Then again, this technique is much more sophisticated and will temporarily introduce mistakes.

Learn how to test a robots.txt document

There are lots of tactics how you’ll be able to test (or check) your robots.txt document – originally, you will have to attempt to in finding robots.txt by yourself.

Until you may have said a particular URL, your document will likely be hosted at “https://yourdomain.com/robots.txt” – if you’re the use of every other website online builder, the precise URL may well be other.

To test whether or not serps like Google can in truth in finding and “obey” your robots.txt document, you’ll be able to both:

  • Use robots.txt Tester – a easy instrument by means of Google that assist you to in finding out whether or not your robots.txt document purposes correctly.
  • Take a look at Google Seek Console – you’ll be able to search for any mistakes which are led to by means of robots.txt within the “Protection” tab of Google Seek Console. Make certain that there are not any URLs which are reporting messages “blocked by means of robots.txt” by accident.
Google Search Console - blocked by robots.txt example

Robots.txt highest practices

Robots.txt information can simply get complicated, so it’s best to stay issues so simple as imaginable.

Listed here are a couple of pointers that assist you to with developing and updating your personal robots.txt document:

  • Use separate information for subdomains – in case your website online has a couple of subdomains, you will have to deal with them as separate internet sites. All the time create separated robots.txt information for every subdomain that you just personal.
  • Specify user-agents simply as soon as – attempt to merge all directives which are assigned to a particular user-agent in combination. This may occasionally determine simplicity and group to your robots.txt document.
  • Make certain specificity – be sure you specify precise URL paths, and take note of any trailing slashes or particular indicators which are provide (or absent) to your URLs.

Leave a Reply

Your email address will not be published. Required fields are marked *