Case Study 2, Month 6: Solving Indexing Issues for a Programmatic SEO Website
Yoyao & Dim
Hey there, everyone,
Yoyao here with a story about some indexing issues we had (and how we fixed them)!
Dim mentioned indexing issues in the case study update for month three. We thought they had resolved on their own, but it turned out they hadn’t.
After last month’s update and the April Update was over, I figured enough was enough. Time to try something else!
How I Usually Resolve Indexing Issues
One thing I succeeded with in the past was tapping into the Google Indexing API directly. The Indexing API is meant for short-lived pages, job postings, or livestream videos, where content freshness is important.
Google Search Central’s documentation says the Indexing API can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject.
But, the API can technically work with any type of page. There’s no checking system in place that I’ve seen.
When certain pages on a site are stubborn and have not been indexed for some time, I’ll post those URLs to the Indexing API to try and get them indexed.
How to Connect to Google’s Indexing API
The easiest way of connecting to the Indexing API is with the Rank Math Instant Indexing Plugin. However, you’ll still need to perform a few extra steps.
First, create a project on the Google Cloud Platform and enable access to the API.
Then, you’ll need to set up additional service accounts and JSON keys to use with Google Search Console and the plugin.
Once everything is set up, enter your list of URLs that are not indexed, and they’ll be instantly submitted to Google for indexing.
Results
On May 4, 2023, 48 URLs were not indexed. Their status was either “Discovered – currently not indexed” or “Crawled – currently not indexed.”
I submitted all 48 of those to the Indexing API.
On May 5th, I checked, and there were only 8 that needed to be indexed. This means we effectively got 40 URLs indexed with the plugin in 24 hours!
I submitted those 8 again, but they didn’t index. Not sure why, but we’ll continue to work on them to see if there are things we can do to get them indexed.
As of writing this article (May 13th), there are 15 articles that need to be indexed, meaning 7 were de-indexed. Not sure why the deindexing occurred, but we’ll need to look into it.
Most of these non-indexed pages use two templates, but other pages use them, too, and they are indexed OK.
We will keep you up to date on how the indexing goes and what other methods we use!
April 2023 Metrics
Content
Dim here. Now that we’ve covered Yoyao’s indexing wizardry (and the 7 articles that mysteriously keep disappearing from the index), let’s talk about content production!
As I mentioned in last month’s update, March and April were going to be slow months for reasons unrelated to the case study.
In April 2023, we published 1 new template that covered 8 existing entities in 1 existing topical cluster, bringing the total number of articles published on the site to 104 (a milestone!).
| Topical Clusters | 2 |
| Entities Covered | 10 |
| Templates Live | 11 |
| Posts Published | 104 |
Another milestone is the case study website reaching six months of age.
So, what’s next?
We’re ramping up content production from late May and beyond.
We’ll use more generative AI to assist this process, which has already become a staple for content creators.
We’ll also review the already-published templates to try to find patterns and identify what’s working and what isn’t — so stay tuned.
Traffic
The site survived Google’s April 2023 reviews update and had 160 clicks and 12.8k impressions in Google’s SERPs, with an average CTR of 1.2% and an average position of 14:
If you expand the timeline to look at the trend for the past six months, the line is definitely in the right direction:
Of course, the Web is as uncertain as ever.
From the new Google’s algorithm updates to search engines generating their own answers instead of giving you results, the future of this business model looks less bright when you consider that we’re competing for user attention with the platforms that are supposed to be sending them to us.
Anyone who remembers the social media boom and the downfall of the organic traffic from Facebook (and then Pinterest) knows that this usually doesn’t end well for publishers, who will have to find new ways to stay afloat.
Who We Are
Yoyao
Yoyao manages his portfolio of niche and authority sites, publishes the Niche Surfer weekly newsletter, and has a topical map service that helps sites build topical authority and drive organic traffic.
Learn more about NicheSurfer.com and TopicalMap.com
Dim
Dim started his first site in 2007, when he stumbled upon a blog about making money online. He’s been buying, growing, and selling sites ever since. Fast-forward to today, and Dim owns an indie media company and runs Publetise, a weekly email newsletter for online publishers.
Subscribe to Dim’s newsletter at Publetise.com
Google has been not indexing or de-indexing some of my articles. One thing I found is that if they won’t index it, I’ll switch the article to draft and then republish it with today’s date and a new url slug. Like my-awesome-post becomes my-awesome-post-2. This usually works for me.
Thanks for sharing, John!
Yeah, Yoyao and I will definitely give this technique a try if the articles keep getting de-indexed.
I’ve had success with this on other sites, too. Google’s random like that sometimes.
Dim
Yeah! Google has been unindexing my article as well. I had to put it into reindexing in webmasters. May be it is because of Google’s new update. But this is an awesome information. I am bookmarking your page. Thanks!