{"id":14822,"date":"2015-07-05T12:35:08","date_gmt":"2015-07-05T10:35:08","guid":{"rendered":"https:\/\/marabelia.com\/?p=14822"},"modified":"2025-06-20T12:46:55","modified_gmt":"2025-06-20T10:46:55","slug":"robots-txt-htaccess-gzip-compression-and-sitemaps-xml","status":"publish","type":"post","link":"https:\/\/marabelia.com\/en\/robots-txt-htaccess-gzip-compression-and-sitemaps-xml\/","title":{"rendered":"robots. txt, .htaccess, GZIP compression and sitemaps.xml"},"content":{"rendered":"
[vc_row][vc_column][vc_column_text]\n

Integrating the robots.txt, .htaccess<\/em> ( +GZIP compression ) and sitemaps.xml<\/em> files is a fundamental step for SEO optimization on our WordPress<\/em> site. Below I explain in a simple way how to configure and add these files without much hassle.<\/p>\n

\u25b6 <\/span> Creating XML Sitemaps<\/h6>\n

The sitemap.xml<\/em> file is used to outline all existing pages of a website.<\/p>\n

Google designed this system to facilitate indexing and to directly send a summary of all the URLs on our site to Google’s indexing bot so that they can be indexed more easily and quickly.<\/p>\n

In Google\u2019s support forum<\/em> they explain how to create a sitemap<\/a> from scratch, but\u2026 Why complicate things if there\u2019s a great plugin that solves it for us?<\/em><\/p>\n

The best option is undoubtedly to generate the sitemap automatically by installing the SEO YOAST plugin.<\/a> Once the plugin is activated, our sitemap.xml will be available at ..\/sitemap_index.xml.<\/p>\n

To check it, click on your WordPress vertical menu \/ SEO -> XML Sitemaps, then on the button \u201cYou can find your XML Sitemap here: XML Sitemap.\u201d If you get a 404 Error Page Not Found<\/strong>, go back to the WordPress menu \/ Settings \/ Permalinks and simply save your permalinks again.<\/p>\n

If for some strange and unknown reason we don\u2019t want to install SEO Yoast, we can also generate the sitemap with Google XML Sitemaps Generator<\/a>, and it will be automatically created in the root of our WordPress with the name sitemap.xml<\/em>.<\/p>\n

It\u2019s that easy.<\/p>\n

\u25b6 <\/span> Configuring the robots.txt<\/em> file<\/h6>\n

The robots.txt<\/em> file is a plain text file that provides recommendations for all search engine crawlers and robots to follow. This article<\/a> explains clearly and accurately what it is and how to configure it.<\/p>\n

To simplify things, here is the robots.txt<\/em> file I usually use for WordPress<\/em>:<\/p>\n

Disallow: \/wp-admin\r\nDisallow: \/wp-includes\/\r\nDisallow: \/*\/feed\/\r\nDisallow: \/*\/trackback\/\r\nDisallow: \/*\/attachment\/\r\nDisallow: \/author\/\r\nDisallow: \/*\/feed\/\r\nDisallow: \/tag\/*\/page\/\r\nDisallow: \/tag\/*\/feed\/\r\nDisallow: \/xmlrpc.php\r\nDisallow: \/*?s=\r\nDisallow: \/*\/*\/*\/feed.xml\r\nDisallow: \/?attachment_id*\r\nSitemap: http:\/\/gooddesign.es\/sitemap.xml\r\n<\/pre>\n
You just need to copy this content into a plain text file, save it as robots.txt<\/em>, and upload it to the root directory of your WordPress<\/em> via FTP.<\/p>\n
If you don\u2019t want to use FTP, you can also generate the robots.txt<\/em> with the WP Robots Txt<\/em> plugin by Christopher Davis<\/a><\/cite>.<\/p>\n
\u25b6 <\/span> .htaccess<\/em> File (with gzip<\/em> compression)<\/h6>\n
The .htaccess<\/em> file is necessary for the proper functioning of WordPress. It is mainly used to specify security restrictions on the website: prevent directory listing, restrict access to specific files or folders, create custom error pages, prevent bandwidth theft, or block users or sites, among other options.<\/p>\n
From the WordPress Codex<\/em><\/a> we get the basic .htaccess<\/em> file for WordPress:<\/p>\n
# BEGIN WordPress\r\n<IfModule mod_rewrite.c>\r\nRewriteEngine On\r\nRewriteBase \/\r\nRewriteRule ^index\\.php$ - [L]\r\nRewriteCond %{REQUEST_FILENAME} !-f\r\nRewriteCond %{REQUEST_FILENAME} !-d\r\nRewriteRule . \/index.php [L]\r\n<\/IfModule>\r\n# END WordPress\r\n<\/pre>\n
Within this basic .htaccess<\/em> file we can add the GZIP compression<\/em> line. GZIP compression will make our website lighter in KB, which will lead to two major advantages: Bandwidth savings and increased website speed.<\/p>\n
php_value output_handler ob_gzhandler<\/pre>\n
With the new GZIP line, the final .htaccess<\/em> file would look like this:<\/p>\n
# BEGIN WordPress\r\n<IfModule mod_rewrite.c>\r\nRewriteEngine On\r\nRewriteBase \/\r\nRewriteRule ^index\\.php$ - [L]\r\nRewriteCond %{REQUEST_FILENAME} !-f\r\nRewriteCond %{REQUEST_FILENAME} !-d\r\nRewriteRule . \/index.php [L]\r\nphp_value output_handler ob_gzhandler\r\n# END WordPress\r\n<\/pre>\n
Now we can add the lines to tell it where our robots.txt<\/em> and sitemap.xml<\/em> files are located. So our FINAL GZIP FILE<\/strong> would look like this:<\/p>\n
# BEGIN WordPress\r\n<IfModule mod_rewrite.c>\r\nRewriteEngine On\r\nRewriteBase \/\r\nRewriteRule ^index\\.php$ - [L]\r\nRewriteCond %{REQUEST_FILENAME} !-f\r\nRewriteCond %{REQUEST_FILENAME} !-d\r\nRewriteRule . \/index.php [L]\r\nphp_value output_handler ob_gzhandler\r\n# END WordPress\r\n\r\n# canonical robots.txt\r\n<IfModule mod_rewrite.c>\r\nRewriteBase \/\r\nRewriteCond %{REQUEST_URI} !^\/robots.txt$ [NC]\r\nRewriteCond %{REQUEST_URI} robots.txt [NC]\r\nRewriteRule .* http:\/\/your-site.com\/robots.txt [R=301,L]\r\n<\/IfModule>\r\n\r\n# canonical sitemaps\r\n<IfModule mod_alias.c>\r\nRedirectMatch 301 \/sitemap\\.xml$ http:\/\/example.com\/sitemap.xml\r\nRedirectMatch 301 \/sitemap\\.xml\\.gz$ http:\/\/example.com\/sitemap.xml.gz\r\n<\/IfModule>\r\n<\/pre>\n
Copy this text into a plain text file, save it as htaccess.txt<\/em> (without the dot), and upload it via FTP to the root folder of your WordPress<\/em>. Once uploaded, rename it to .htaccess<\/em> (with the dot and no extension). The next step is to make this file read-only to avoid major issues. This is done by editing the file permissions in your FTP client and changing them to 444.<\/p>\n
Everything should be working fine now, but let\u2019s make sure GZIP compression is actually enabled.<\/p>\n
At GIDNETWORK TOOLS<\/em><\/a> we have a tool to test our compression. Just enter your URL.<\/p>\n
You may also be interested in: Web Optimization and SEO Analysis in WordPress<\/a><\/strong><\/p>\n[\/vc_column_text][\/vc_column][\/vc_row]\n<\/div>","protected":false},"excerpt":{"rendered":"[vc_row][vc_column][vc_column_text] Integrating the robots.txt, .htaccess ( +GZIP compression ) and sitemaps.xml files is a fundamental step for SEO optimization on our WordPress site. Below I explain in a simple way how to configure and add these files without much hassle. \u25b6 Creating XML Sitemaps The sitemap.xml file is used to outline all existing pages of […]\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[96,89,98],"tags":[],"class_list":["post-14822","post","type-post","status-publish","format-standard","hentry","category-seo","category-web-optimization","category-wordpress-2"],"yoast_head":"\nrobots. txt, .htaccess, GZIP compression and sitemaps.xml • Marabelia | Agencia Digital en Valencia<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/marabelia.com\/en\/robots-txt-htaccess-gzip-compression-and-sitemaps-xml\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"robots. txt, .htaccess, GZIP compression and sitemaps.xml • Marabelia | Agencia Digital en Valencia\" \/>\n<meta property=\"og:description\" content=\"[vc_row][vc_column][vc_column_text] Integrating the robots.txt, .htaccess ( +GZIP compression ) and sitemaps.xml files is a fundamental step for SEO optimization on our WordPress site. Below I explain in a simple way how to configure and add these files without much hassle. \u25b6 Creating XML Sitemaps The sitemap.xml file is used to outline all existing pages of [...]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/marabelia.com\/en\/robots-txt-htaccess-gzip-compression-and-sitemaps-xml\/\" \/>\n<meta property=\"og:site_name\" content=\"Marabelia | Agencia Digital en Valencia\" \/>\n<meta property=\"article:published_time\" content=\"2015-07-05T10:35:08+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-20T10:46:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/marabelia.com\/wp-content\/uploads\/2025\/09\/Portada-Marabelia-sitio-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"800\" \/>\n\t<meta property=\"og:image:height\" content=\"800\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"admin-mbl\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin-mbl\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"robots. txt, .htaccess, GZIP compression and sitemaps.xml • Marabelia | Agencia Digital en Valencia","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/marabelia.com\/en\/robots-txt-htaccess-gzip-compression-and-sitemaps-xml\/","og_locale":"en_US","og_type":"article","og_title":"robots. txt, .htaccess, GZIP compression and sitemaps.xml • Marabelia | Agencia Digital en Valencia","og_description":"[vc_row][vc_column][vc_column_text] Integrating the robots.txt, .htaccess ( +GZIP compression ) and sitemaps.xml files is a fundamental step for SEO optimization on our WordPress site. Below I explain in a simple way how to configure and add these files without much hassle. \u25b6 Creating XML Sitemaps The sitemap.xml file is used to outline all existing pages of [...]","og_url":"https:\/\/marabelia.com\/en\/robots-txt-htaccess-gzip-compression-and-sitemaps-xml\/","og_site_name":"Marabelia | Agencia Digital en Valencia","article_published_time":"2015-07-05T10:35:08+00:00","article_modified_time":"2025-06-20T10:46:55+00:00","og_image":[{"width":800,"height":800,"url":"https:\/\/marabelia.com\/wp-content\/uploads\/2025\/09\/Portada-Marabelia-sitio-1.jpg","type":"image\/jpeg"}],"author":"admin-mbl","twitter_card":"summary_large_image","twitter_misc":{"Written by":"admin-mbl","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[]}},"_links":{"self":[{"href":"https:\/\/marabelia.com\/en\/wp-json\/wp\/v2\/posts\/14822","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marabelia.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marabelia.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marabelia.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/marabelia.com\/en\/wp-json\/wp\/v2\/comments?post=14822"}],"version-history":[{"count":2,"href":"https:\/\/marabelia.com\/en\/wp-json\/wp\/v2\/posts\/14822\/revisions"}],"predecessor-version":[{"id":14891,"href":"https:\/\/marabelia.com\/en\/wp-json\/wp\/v2\/posts\/14822\/revisions\/14891"}],"wp:attachment":[{"href":"https:\/\/marabelia.com\/en\/wp-json\/wp\/v2\/media?parent=14822"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marabelia.com\/en\/wp-json\/wp\/v2\/categories?post=14822"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marabelia.com\/en\/wp-json\/wp\/v2\/tags?post=14822"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}