Website owners often struggle to get their content indexed on time. Unless you own a large website, chances are search engines might only crawl your site for new URLs once every few days.
To solve this problem and improve crawl efficiency, Microsoft Bing and Yandex collaborated to create a new API-based protocol called IndexNow.
Keep reading to learn:
How Leading CDNs Are Supporting the Latest Bing Initiative
The Difference Between IndexNow and the Google Indexing API
IndexNow vs XML Sitemaps: Why API-Based Approach Crawling Is the Future
How To Get Started With IndexNow
Final Thoughts — IndexNow: The Future of Search Engine Crawling & Indexing
What Is IndexNow?
IndexNow is a protocol that lets you notify search engines about changes on your website with an HTTP request. With every API call, you can provide search engines using IndexNow with a list of up to 10,000 URLs that have changed on your site.
IndexNow aims to make the internet more efficient. It’s an attempt to help search engines improve crawl efficiency by eliminating the need for exploratory crawls.
How Leading CDNs Are Supporting the Latest Bing Initiative
Fabrice Canel, Principal Program Manager at Microsoft Bing, believes that Content Delivery Networks (CDNs) are crucial to adopting IndexNow.
CDNs are often the first to learn when users update content on their websites. Moreover, they have insights into crawler behavior to leverage IndexNow effectively.
Cloudflare, the world’s largest proxy service as per W3Techs, already supports IndexNow. They’ve integrated it into their Crawler Hints service and made it available to their users for free.
Similarly, Akamai, another leading CDN service provider, has also announced support for IndexNow so their customers can enjoy easier notifications to search engines when their cached content changes.
With the support of industry-leading CDNs, it’s likely that search engine crawling will soon shift from being proactive to reactive.
The Difference Between IndexNow and the Google Indexing API
Google launched its Indexing API back in 2018. Although IndexNow takes a similar approach, it brings it to the masses instead of a small subset of the internet.
Google’s API is only meant for pages with job postings and live stream content. In contrast, IndexNow supports all content types.
It accepts up to 10,000 URLs per API call, whereas Google only supports 100. Moreover, IndexNow’s implementation process is simple, whereas Google’s is significantly more complicated.
IndexNow vs XML Sitemaps: Why API-Based Approach Crawling Is the Future
The traditional approach to indexation is outdated. You update your sitemap, ping search engines and wait for them to crawl your sitemap to discover changes before they crawl and index them.
IndexNow solves this problem by presenting search engines with the information they need upfront. Search engine crawlers don’t need to check and recheck sitemaps for new information.
And unlike an XML sitemap which only helps you redirect crawlers to pages on your website, IndexNow lets you inform search engines of non-200 status code pages like 403, 301 and 302.
But that doesn’t mean that sitemaps are irrelevant. Bing and Yandex only represent a small fraction of the internet. The most dominant search engine, Google, is still testing IndexNow and hasn’t announced any plans to adopt it.
Google has the infrastructure it needs to build an alternative to IndexNow. They only need to modify and upgrade it. But, no matter their decision, it’s clear that a future where indexing shifts to an API-based approach isn’t far off.
How To Get Started With IndexNow
Implementing the IndexNow protocol is easy. You can do it yourself using this four-step process:
1. Generate an API Key
Use Bing’s online key generation tool to generate an API key needed to verify ownership of your domain.
2. Host the API Key
Host the generated key in a UTF-8 encoded .txt file in the root directory of your website or another location on your server. If you choose the latter, you need to specify the key file location each time you submit an URL.
3. Submit URLs
Submit URLs using POST JSON requests for each search engine in the recommended format.
4. Monitor Your Site
Use Bing Webmaster Tools to monitor the URLs you’ve submitted and those discovered by Bing.
Final Thoughts — IndexNow: The Future of Search Engine Crawling & Indexing
IndexNow lays the foundation for the future of crawling and indexing — one that’s faster and more efficient. And by collaborating with market leaders like Cloudflare, Microsoft and Yandex have perfectly positioned it for mass adoption.
Make sure you’re ready to leverage IndexNow with a future-proof website built on Bluehost’s hosting infrastructure.
