Integrating the robots.txt, .htaccess ( +GZIP compression ) and sitemaps.xml files is a fundamental step for SEO optimization on our WordPress site. Below I explain in a simple way how to configure and add these files without much hassle.
▶ Creating XML Sitemaps
The sitemap.xml file is used to outline all existing pages of a website.
Google designed this system to facilitate indexing and to directly send a summary of all the URLs on our site to Google’s indexing bot so that they can be indexed more easily and quickly.
In Google’s support forum they explain how to create a sitemap from scratch, but… Why complicate things if there’s a great plugin that solves it for us?
The best option is undoubtedly to generate the sitemap automatically by installing the SEO YOAST plugin. Once the plugin is activated, our sitemap.xml will be available at ../sitemap_index.xml.
To check it, click on your WordPress vertical menu / SEO -> XML Sitemaps, then on the button “You can find your XML Sitemap here: XML Sitemap.” If you get a 404 Error Page Not Found, go back to the WordPress menu / Settings / Permalinks and simply save your permalinks again.
If for some strange and unknown reason we don’t want to install SEO Yoast, we can also generate the sitemap with Google XML Sitemaps Generator, and it will be automatically created in the root of our WordPress with the name sitemap.xml.
It’s that easy.
▶ Configuring the robots.txt file
The robots.txt file is a plain text file that provides recommendations for all search engine crawlers and robots to follow. This article explains clearly and accurately what it is and how to configure it.
To simplify things, here is the robots.txt file I usually use for WordPress:
Disallow: /wp-admin Disallow: /wp-includes/ Disallow: /*/feed/ Disallow: /*/trackback/ Disallow: /*/attachment/ Disallow: /author/ Disallow: /*/feed/ Disallow: /tag/*/page/ Disallow: /tag/*/feed/ Disallow: /xmlrpc.php Disallow: /*?s= Disallow: /*/*/*/feed.xml Disallow: /?attachment_id* Sitemap: http://gooddesign.es/sitemap.xml
You just need to copy this content into a plain text file, save it as robots.txt, and upload it to the root directory of your WordPress via FTP.
If you don’t want to use FTP, you can also generate the robots.txt with the WP Robots Txt plugin by Christopher Davis.
▶ .htaccess File (with gzip compression)
The .htaccess file is necessary for the proper functioning of WordPress. It is mainly used to specify security restrictions on the website: prevent directory listing, restrict access to specific files or folders, create custom error pages, prevent bandwidth theft, or block users or sites, among other options.
From the WordPress Codex we get the basic .htaccess file for WordPress:
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
Within this basic .htaccess file we can add the GZIP compression line. GZIP compression will make our website lighter in KB, which will lead to two major advantages: Bandwidth savings and increased website speed.
php_value output_handler ob_gzhandler
With the new GZIP line, the final .htaccess file would look like this:
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
php_value output_handler ob_gzhandler
# END WordPress
Now we can add the lines to tell it where our robots.txt and sitemap.xml files are located. So our FINAL GZIP FILE would look like this:
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
php_value output_handler ob_gzhandler
# END WordPress
# canonical robots.txt
<IfModule mod_rewrite.c>
RewriteBase /
RewriteCond %{REQUEST_URI} !^/robots.txt$ [NC]
RewriteCond %{REQUEST_URI} robots.txt [NC]
RewriteRule .* http://your-site.com/robots.txt [R=301,L]
</IfModule>
# canonical sitemaps
<IfModule mod_alias.c>
RedirectMatch 301 /sitemap\.xml$ http://example.com/sitemap.xml
RedirectMatch 301 /sitemap\.xml\.gz$ http://example.com/sitemap.xml.gz
</IfModule>
Copy this text into a plain text file, save it as htaccess.txt (without the dot), and upload it via FTP to the root folder of your WordPress. Once uploaded, rename it to .htaccess (with the dot and no extension). The next step is to make this file read-only to avoid major issues. This is done by editing the file permissions in your FTP client and changing them to 444.
Everything should be working fine now, but let’s make sure GZIP compression is actually enabled.
At GIDNETWORK TOOLS we have a tool to test our compression. Just enter your URL.
You may also be interested in: Web Optimization and SEO Analysis in WordPress
