Keep up to date on the latest web design & SEO Projects

Incorrect URLs in Bing IndexNow from CloudFlare

Are you experiencing site errors in your Google Search Console? These errors can include pages with redirects, pages blocked by robots.txt, soft 404s, 404s, and pages with proper canonical tags. It can be frustrating to see these errors, especially if you thought you had already set directives on the affected pages. But why are these errors occurring, and what can you do to fix them?

Based on a discussion here on the CloudFlare community forum about the issue here is a bit more of an outline and summary and how to resolve these incorrect page URLs from getting into the IndexNow data.

In this article, we will explore the possible causes of these errors and provide solutions for resolving them. Some possible causes may include the use of Cloudflare’s “index now” feature or changes to Google’s search engine results pages (SERPs). We will also discuss the benefits of using your own “index now” feature or a plugin, as well as the potential drawbacks of blocking certain search engines. By following these tips, you can help ensure that your site is free of errors and fully optimised for search engines.

  • The “Index Now” Feature from Cloudflare
  • Using Your Own “Index Now” Feature or a Plugin
  • Blocking Certain Search Engines
  • Stay Up-to-Date on Changes to Google’s SERPs
Incorrect URLs in Bing IndexNow from CloudFlare -

What is “IndexNow”?

IndexNow is a tool that helps websites owners quickly notify search engines about updates to their content. By using IndexNow, website owners can send a signal to search engines that a URL and its associated content have been added, modified, or removed. This allows search engines to reflect these changes in their search results more quickly, rather than waiting for their regular crawling schedule to discover the updates. Without IndexNow, it can take several days to weeks for search engines to detect changes to a website’s content. IndexNow helps prioritise the crawl of changed URLs, reducing the need for search engines to crawl all URLs to discover new content.

The “Index Now” Feature from Cloudflare

The “Index Now” feature from Cloudflare is a tool that allows users to submit URLs to be indexed by search engines, specifically Bing and Yandex. While this feature can be useful for quickly getting new pages indexed, it can also cause problems if not used correctly.

One issue with the “index now” feature is that it submits all entered URLs to Bing and Yandex, even if they are mistyped or behind firewall rules. This can lead to Google reporting errors on pages that were requested to be crawled by other search engines. For example, if a page is blocked by robots.txt, Google will not crawl it, but if it is requested to be crawled by Bing or Yandex through the “index now” feature, Google may still report it as an error.

To avoid this problem, you could use the “index now” feature from your own site rather than relying on Cloudflare. There are also plugins for CMS such as WordPress, that can support the “index now” feature and help ensure that only valid URLs are submitted for indexing. This can save you time and effort in fixing errors in your Google Search Console.

It is also worth noting that while the “index now” feature can be useful for quickly getting new pages indexed, it may not be the most effective solution for all websites. Some experts recommend using more traditional methods, such as submitting a sitemap or using internal linking, to get pages indexed. As with any tool, it is important to carefully consider the pros and cons before using it on your site.

Downloading a list of URLs submitted by CloudFlare IndexNow

It is possible to download a list of URLs that have been submitted to Bing through CloudFlare’s IndexNow feature using Bing Webmaster Tools. This can be useful for website owners who want to review the URLs that are being submitted for indexing and ensure that only valid pages are being crawled by Bing.

To download a list of URLs submitted by CloudFlare IndexNow from Bing Webmaster Tools, you will need to have a verified Bing Webmaster account and be logged in. Once logged in, follow these steps:

  1. Navigate to the “Reports & Data” section in the main menu.
  2. Click on the “URL submissions” option.
  3. Under the “URL submissions” tab, select the “Index Now” option.
  4. Use the filters on the page to select the date range for the URLs you want to download.
  5. Click the “Export” button in the top right corner of the page.
  6. Choose the format you want to export the data in (e.g., CSV, Excel, etc.).
  7. Click “Export” to download the file.
Incorrect URLs in Bing IndexNow from CloudFlare -

The downloaded file will contain a list of all the URLs that have been submitted to Bing through CloudFlare IndexNow, as well as information about the date and time of the submission.

Using this information, website owners can review the URLs that are being submitted for indexing and take any necessary actions to ensure that only valid pages are being crawled by Bing. This can help prevent errors in search engine results and improve the overall search engine optimisation of the site.

Using Your Own “IndexNow” Feature or a Plugin

Using your own “index now” feature or a plugin can be a effective way to avoid errors in your Google Search Console caused by the “index now” feature from Cloudflare. By submitting URLs directly from your own site or using a plugin, you can control which URLs are submitted for indexing and ensure that only valid pages are being crawled by search engines.

One option for implementing your own “index now” feature is to use a plugin, such as the WordPress SEO plugin. These plugins often include tools for submitting sitemaps and individual URLs to search engines, as well as other features for optimising your site for search engines. Using a plugin can save you time and effort in manually submitting URLs and can provide additional benefits, such as helping you to identify and fix other SEO issues on your site.

Another option is to use the “index now” feature directly from your own site. Most search engines, including Google, Bing, and Yandex, have tools for manually submitting URLs for indexing. These tools can be accessed through your search engine webmaster accounts and typically allow you to submit individual URLs or entire sitemaps.

While using your own “index now” feature or a plugin can be effective in preventing errors caused by the Cloudflare “index now” feature, it is still important to carefully consider the pros and cons of each approach. Using a plugin may be more convenient, but it may also introduce additional dependencies and require ongoing maintenance. Manually submitting URLs through search engine webmaster tools can be more time-consuming, but it can also give you more control and potentially provide a more stable solution in the long run.

Blocking Certain Search Engines

Blocking certain search engines, such as Bing, Seznam, and Yandex, in your robots.txt file can be a effective way to prevent errors in your Google Search Console caused by the “index now” feature from Cloudflare but it’s not a recommended solution. By blocking these search engines, you can prevent them from crawling pages on your site that may be causing errors in Google.

To block a search engine in your robots.txt file, you will need to add a line of code specifying which search engine you want to block and which pages or directories you want to block. For example, to block Bing from crawling your entire site, you would add the following line of code to your robots.txt file:

User-agent: Bingbot
Disallow: /

You can also block specific pages or directories by specifying their paths, such as:

User-agent: Bingbot
Disallow: /private/

It is important to note that blocking a search engine in your robots.txt file does not guarantee that the search engine will not crawl your site. Some search engines may ignore the directives in your robots.txt file, or they may crawl your site despite the block. However, using the robots.txt file can be a useful way to signal to search engines which pages or directories on your site you do not want them to crawl.

Blocking certain search engines can be a effective way to prevent errors in your Google Search Console, but it is important to carefully consider the potential drawbacks. By relying on only one search engine, such as Google, you may be more vulnerable to changes in their algorithms or policies. It is generally recommended to have as many alternatives as possible to help mitigate any potential impact on your site’s ranking.

Checking for 404 errors with Screaming Frog

Using Screaming Frog SEO Spider is a useful way to check a website for 404 errors. A 404 error occurs when a user tries to access a webpage that does not exist or has been removed. These errors can be detrimental to a website’s user experience and search engine optimisation, as they may cause users to leave the site and can negatively impact the site’s ranking in search results.

Using Screaming Frog we can be sure that the website is not full of 404 errors.

Incorrect URLs in Bing IndexNow from CloudFlare -

Stay Up-to-Date on Changes to Google’s SERPs

Staying up-to-date on changes to Google’s search engine results pages (SERPs) is an important part of optimising your site for search engines and preventing errors in your Google Search Console. Google frequently makes changes to its algorithms and ranking factors, which can have a significant impact on the visibility and ranking of your site in search results.

To stay informed about changes to Google’s SERPs, you can sign up for alerts from Google Webmaster Central or follow SEO industry blogs and news sources. These sources can provide updates on new features, changes to ranking algorithms, and other important information that can help you optimise your site for search engines.

In addition to staying informed about changes to Google’s SERPs, it is also important to regularly review and optimise your site for search engines. This can include identifying and fixing errors in your Google Search Console, optimising your site’s content and meta tags, and building high-quality backlinks. By regularly reviewing and optimising your site, you can help ensure that it is fully optimised for search engines and free of errors.

Overall, staying up-to-date on changes to Google’s SERPs and regularly optimising your site can help you avoid errors in your Google Search Console and improve your site’s visibility and ranking in search results. By following these tips, you can help ensure that your site is fully optimised for search engines and attract more traffic and customers.

Conclusion

In conclusion, site errors in your Google Search Console can be frustrating, but there are steps you can take to fix them. One possible cause of these errors is the use of Cloudflare’s “index now” feature, which can submit URLs to Bing and Yandex even if they are mistyped or behind firewall rules. To prevent these errors, you can use your own “index now” feature or a plugin, or block certain search engines in your robots.txt file.

Using your own “index now” feature or a plugin can provide more control over which URLs are submitted for indexing, while blocking certain search engines can help prevent errors caused by the Cloudflare “index now” feature. However, it is important to carefully consider the pros and cons of each approach, as well as the potential drawbacks of relying on only one search engine.

Staying up-to-date on changes to Google’s search engine results pages (SERPs) and regularly optimising your site can also help prevent errors and improve your site’s visibility and ranking in search results. By following these tips, you can help ensure that your site is fully optimised for search engines and free of errors.