Rap Genius started in 2009 as a lyrics and text annotation site. So if you needed the lyrics to a Beyonce banger, you’d be sure to find the exact lyrics and a critical analysis of the same on their site. All seemed well, and the company was headed for success until Google noticed an anomaly.
Just four years into its launch, in 2013, Google deindexed all Rap Genius from its search results. Just like that, anyone wanting to search for the site would have an incredibly hard time finding it, probably on the sixth page. How devastating for a website that was ranked first in every lyric search!
Rap Genius’s mistake was that they used manipulative SEO tricks to get to the top of Google’s search results. More specifically, they promoted internal links to external blogs in exchange for these blogs linking back to their site.
Fun Read: Buying Backlinks Online: Should You Do It in 2023?
Google deindexed their web page for about two weeks, after which they helped Rap Genius rectify the situation. But this blackout period cost them user traffic and had a negative impact on their business.
Image Credits: mediaheroes.com
That’s the tea on deindexing. It isn’t all detrimental, though, as we’ll see later, as some website owners actually deindex pages to grow their brands. Join us as we take you through how to deindex a page from Google.
But first…
Why Does Google Deindex Pages?
Google has built its reputation as a search engine authority for many years. It has also spent resources, time, and energy into ensuring that this reputation is upheld. One of the ways to optimally ensure this is to permanently or temporarily remove useless, irrelevant and low-quality pages from its site.
The following are some of the reasons why Google will deindex pages:
1. Spammy Pages
This may be intentional or otherwise. Regardless, Google views any SEO strategies that are in contravention of its algorithm or SEO best practices as sufficient ground for deindexing.
Image Credits: liberotecnologia.com
Here’s a sobering fact: Google finds over 25 billion spammy pages daily. Google considers spam in terms of links (like what happened to Rap Genius), user-generated, and hacked website spams.
2. Keyword Stuffing
The rise of keyword placing in web content was a paradigm shift in the content creation industry, specifically for blogs and websites. Placing relevant keywords in an article ensured that it rose to the top of search engine results.
Image Credits: rankhigheragency.com
However, knowing what is enough, and when it is enough, is the trick. Keyword stuffing involves placing a specific keyword excessively in a piece of content.
3. Duplicate Content
Reusing content is a giant no in Google. Whether it’s copying content from other websites or reusing your own, Google terms this as plagiarism and may deindex your pages.
4. Auto-Generated Content
Spinning content saves a lot of time. However, it could be disastrous and lead to deindexing. Google aims to ensure that its users get value from their content; it has to be contextual and without error.
5. Cloaking
Just as the word sounds, cloaking is when a site owner displays malicious or irrelevant content to their users, yet on Google’s side, what they see is search-optimized content.
Image Credits: bigtrunkcommunications.com
Another example is when a webpage shows text to search engine bots yet displays images to a human user. Cloaking violates Google’s rules and will cause your website to be deindexed.
Why People Deindex Their Pages From Google
There are multiple reasons why you would want to deindex your page on Google (and other search engines), and some of them include:
- You’re done with your website and are looking to completely remove it from Google search engine results.
- You want to improve your rankings and authority by removing irrelevant content and placing more informative and on-intent content.
- Google indexed some of your pages mistakenly, such as a Thank You page. Ask yourself: Of what use is this obsolete page showing up on the first page of your site’s search while more important pages could take precedence?
- You have duplicate content. Having the same title and meta description for two different articles automatically rings a warning bell.
If you are looking to exit your business and clear your online presence in the process, we have the solution for you. At Indexsy, we buy incredible businesses that want to finally take a much-needed rest.
How to Deindex Pages From Google
Now that we’ve gone through some of the reasons why your web pages could be deindexed and why you’d want to deindex your page let us dive into the actual ways to deindex pages from Google.
1. Using a robots.txt file
This URL removal tool allows you to add your page’s URL to the robots.txt file. Google will understand that you don’t want them to crawl and index this specific page on your website.
Image Credits: seobility.com
This will only work for search engines that follow the robots.txt protocol. This is how this file works.The file contains instructions for spiders (web crawlers) to restrict and remove access to certain pages.
Every time search engine bots crawl your website, they will first look for this file. It stands as the gatekeeper, so to speak, so if you don’t want your site to be indexed or accessed, it knows that from the get-go.
There are several ways to use this file. It entails changing the description, that is, manipulating the instruction depending on what you want to achieve. Here are some of the ways:
- Allow all bots to crawl and index the entire site. You can use this code in the robots.txt file:User-agent: *Disallow:
- Block all bots from crawling and indexing the whole site using this code:User-agent: *Disallow: /
- Block a specific bot from crawling and indexing your entire site, such as Google’s bots, using the following lines of code:User-agent: GooglebotDisallow: /
- Block a specific folder or page using these lines:User-agent: *Disallow: /folder/Disallow: /page.html
Note: Some bots will outright ignore the robots’ meta tag and will go ahead to index pages (and create indexed URLs) that you wish were deindexed. To avoid this challenge, consider using the noindex tag.
2. Using a noindex meta tag
Here, you’ll use the noindex tag in the HTML code of the page you want to deindex. Google will automatically know not to index the page.
The meta tag will be placed in the <head> section of the page. For example, this code can be placed in the meta tag of your HTML code.
<meta name=”robots” content=”noindex”>
Image Credits: seobility.com
If you are using Google Chrome to open your website, right-click on any blank space on your web page and select “View page source.” You can also click CTRL+U on your keyboard on any search engine, and it will open a new tab with your web page’s original code.
From this point, you can now insert the above line of code. It will automatically deindex your page from Google’s search results.
And voila, you just became a coder while deindexing your page at the same time. How suave!
3. Using Google Search Console
For website owners with access to this tool, you can use the “Remove URLs” tool to request Google to remove specific URLs from its search results. However, this method is not deindexing but rather removing the URL from Google’s search results.
Image Credits: searchengineland.com
4. Natural Deindexing
Any time you make changes to your website, it may take some time for Google to update its indexes. The same applies if you delete web pages. In some cases, the page may be deindexed automatically.
5. HTTP 410 Code
Providing an HTTP 410 code or HTTP 404 code on the deleted pages you intend to be deindexed shows Google that this is a removal request.
Image Credits: wetopi.com
You may use this error code:
- If you want to intentionally remove or delete pages from websites permanently.
- If all content is being removed from the website when it is shutting down.
- If an old page is being replaced with a new one, the old one will be unavailable permanently.
In short, using this code on your page means that that specific URL will not be used again. An apt example is a sign-in page for a one-time event such as a concert or marathon. So every time a visitor comes to this site after the event, they will find this code.
Deindexing such a page benefits your website by removing irrelevant and useless material. If this page is indexed, and your visitors find themselves on it when searching for your content, they will feel short-changed and duped. This will hurt your website’s credibility and possibly reduce search rankings.
How to Check if a Page is Deindexed
The first obvious way to check if your page has been deindexed is by doing a manual Google search. In Google’s search bar, enter the URL of the page you want to check. If it doesn’t show up in the search results, or if it does but is numerous pages away, then the page could be deindexed.
Image Credits: softway.com
If you have access to Google’s Search Console, you can also check the Coverage report. The page should be listed as “excluded” if you want it deindexed. “Valid” means that it is still one of your indexed pages. This method is convenient if you have multiple URLs.
You can also check your cache. A deindexed page will not have a cached version. Prefix your website’s name with the word “cache” on the search bar to check this status. For example, if your website is called Dog Food, you can type “cache:dogfood.com/page-url” with the specific URL of the page you want to check if deindexed or not.
Jacky Chou is an electrical engineer turned marketer. He is the founder of Indexsy, Far & Away, Laurel & Wolf, a couple FBA businesses , and about 40 affiliate sites. He is a proud native of Vancouver, BC, who has been featured on Entrepreneur.com, Forbes, Oberlo and GoDaddy.
