Tag Archive | "Indexing"

How Mobile-First Indexing Disrupts the Link Graph

Posted by rjonesx.

It’s happened to all of us. You bring up a webpage on your mobile device, only to find out that a feature you were accustomed to using on desktop simply isn’t available on mobile. While frustrating, it has always been a struggle for web developers and designers alike to simplify and condense their site on mobile screens without needing to strip features or content that would otherwise clutter a smaller viewport. The worst-case scenario for these trade-offs is that some features would be reserved for desktop environments, or perhaps a user might be able to opt out of the mobile view. Below is an example of how my personal blog displays the mobile version using a popular plugin by ElegantThemes called HandHeld. As you can see, the vast page is heavily stripped down and is far easier to read… but at what cost? And at what cost to the link graph?

My personal blog drops 75 of the 87 links, and all of the external links, when the mobile version is accessed. So what happens when the mobile versions of sites become the primary way the web is accessed, at scale, by the bots which power major search engines?

Google’s announcement to proceed with a mobile-first index raises new questions about how the link structure of the web as a whole might be influenced once these truncated web experiences become the first (and sometimes only) version of the web Googlebot encounters.

So, what’s the big deal?

The concern, which no doubt Google engineers have studied internally, is that mobile websites often remove content and links in order to improve user experience on a smaller screen. This abbreviated content fundamentally alters the link structure which underlies one of the most important factors in Google’s rankings. Our goal is to try and understand the impact this might have.

Before we get started, one giant unknown variable which I want to be quick to point out is we don’t know what percentage of the web Google will crawl with both its desktop and mobile bots. Perhaps Google will choose to be “mobile-first” only on sites that have historically displayed an identical codebase to both the mobile and desktop versions of Googlebot. However, for the purposes of this study, I want to show the worst-case scenario, as if Google chose not only to go “mobile-first,” but in fact to go “mobile-only.”

Methodology: Comparing mobile to desktop at scale

For this brief research, I decided to grab 20,000 random websites from the Quantcast Top Million. I would then crawl two levels deep, spoofing both the Google mobile and Google desktop versions of Googlebot. With this data, we can begin to compare how different the link structure of the web might look.

Homepage metrics

Let’s start with some descriptive statistics of the home pages of these 20,000 randomly selected sites. Of the sites analyzed, 87.42% had the same number of links on their homepage regardless of whether the bot was mobile- or desktop-oriented. Of the remaining 12.58%, 9% had fewer links and 3.58% had more. This doesn’t seem too disparate at first glance.

Perhaps more importantly, only 79.87% had identical links on the homepage when visited by desktop and mobile bots. Just because the same number of links were found didn’t mean they were actually the same links. This is important to take into consideration because links are the pathways which bots use to find content on the web. Different paths mean a different index.

Among the homepage links, we found a 7.4% drop in external links. This could mean a radical shift in some of the most important links on the web, given that homepage links often carry a great deal of link equity. Interestingly, the biggest “losers” as a percentage tended to be social sites. In retrospect, it seems reasonable that one of the common types of links a website might remove from their mobile version would be social share buttons because they’re often incorporated into the “chrome” of a page rather than the content, and the “chrome” often changes to accommodate a mobile version.

The biggest losers as a percentage in order were:

  1. linkedin.com
  2. instagram.com
  3. twitter.com
  4. facebook.com

So what’s the big deal about 5–15% differences in links when crawling the web? Well, it turns out that these numbers tend to be biased towards sites with lots of links that don’t have a mobile version. However, most of those links are main navigation links. When you crawl deeper, you just find the same links. But those that do deviate end up having radically different second-level crawl links.

Second-level metrics

Now this is where the data gets interesting. As we continue to crawl out on the web using crawl sets that are influenced by the links discovered by a mobile bot versus a desktop bot, we’ll continue to get more and more divergent results. But how far will they diverge? Let’s start with size. While we crawled an identical number of home pages, the second-tier results diverged based on the number of links found on those original home pages. Thus, the mobile crawlset was 977,840 unique URLs, while the desktop crawlset was 1,053,785. Already we can see a different index taking shape — the desktop index would be much larger. Let’s dig deeper.

I want you to take a moment and really focus on this graph. Notice there are three categories:

  • Mobile Unique: Blue bars represent unique items found by the mobile bot
  • Desktop Unique: Orange bars represent unique items found by the desktop bot
  • Shared: Gray bars represent items found by both

Notice also that there are there are four tests:

  • Number of URLs discovered
  • Number of Domains discovered
  • Number of Links discovered
  • Number of Root Linking Domains discovered

Now here is the key point, and it’s really big. There are more URLs, Domains, Links, and Root Linking Domains unique to the desktop crawl result than there are shared between the desktop and mobile crawler. The orange bar is always taller than the gray. This means that by just the second level of the crawl, the majority of link relationships, pages, and domains are different in the indexes. This is huge. This is a fundamental shift in the link graph as we have come to know it.

And now for the big question, what we all care about the most — external links.

A whopping 63% of external links are unique to the desktop crawler. In a mobile-only crawling world, the total number of external links was halved.

What is happening at the micro level?

So, what’s really causing this huge disparity in the crawl? Well, we know it has something to do with a few common shortcuts to making a site “mobile-friendly,” which include:

  1. Subdomain versions of the content that have fewer links or features
  2. The removal of links and features by user-agent detecting plugins

Of course, these changes might make the experience better for your users, but it does create a different experience for bots. Let’s take a closer look at one site to see how this plays out.

This site has ~10,000 pages according to Google and has a Domain Authority of 72 and 22,670 referring domains according to the new Moz Link Explorer. However, the site uses a popular WordPress plugin that abbreviates the content down to just the articles and pages on the site, removing links from descriptions in the articles on the category pages and removing most if not all extraneous links from the sidebar and footer. This particular plugin is used on over 200,000 websites. So, what happens when we fire up a six-level-deep crawl with Screaming Frog? (It’s great for this kind of analysis because we can easily change the user-agent and restrict settings to just crawl HTML content.)

The difference is shocking. First, notice that in the mobile crawl on the left, there is clearly a low number of links per page and that number of links is very steady as you crawl deeper through the site. This is what produces such a steady, exponential growth curve. Second, notice that the crawl abruptly ended at level four. The site just didn’t have any more pages to offer the mobile crawler! Only ~3,000 of the ~10,000 pages Google reports were found.

Now, compare this to the desktop crawler. It explodes in pages at level two, collecting nearly double the total pages of the mobile crawl at this level alone. Now, recall the graph before showing that there were more unique desktop pages than there were shared pages when we crawled 20,000 sites. Here is confirmation of exactly how it happens. Ultimately, 6x the content was made available to the desktop crawler in the same level of crawl depth.

But what impact did this have on external links?

Wow. 75% of the external, outbound links were culled in the mobile version. 4,905 external links were found in the desktop version while only 1,162 were found in the mobile. Remember, this is a DA 72 site with over twenty thousand referring domains. Imagine losing that link because the mobile index no longer finds the backlink. What should we do? Is the sky falling?

Take a deep breath

Mobile-first isn’t mobile-only

The first important caveat to all this research is that Google isn’t giving up on the desktop — they’re simply prioritizing the mobile crawl. This makes sense, as the majority of search traffic is now mobile. If Google wants to make sure quality mobile content is served, they need to shift their crawl priorities. But they also have a competing desire to find content, and doing so requires using a desktop crawler so long as webmasters continue to abbreviate the mobile versions of their sites.

This reality isn’t lost on Google. In the Original Official Google Mobile First Announcement, they write…

If you are building a mobile version of your site, keep in mind that a functional desktop-oriented site can be better than a broken or incomplete mobile version of the site.

Google took the time to state that a desktop version can be better than an “incomplete mobile version.” I don’t intend to read too much into this statement other than to say that Google wants a full mobile version, not just a postcard.

Good link placements will prevail

One anecdotal outcome of my research was that the external links which tended to survive the cull of a mobile version were often placed directly in the content. External links in sidebars like blog-rolls were essentially annihilated from the index, but in-content links survived. This may be a signal Google picks up on. External links that are both in mobile and desktop tend to be the kinds of links people might click on.

So, while there may be fewer links powering the link graph (or at least there might be a subset that is specially identified), if your links are good, content-based links, then you have a chance to see improved performance.

I was able to confirm this by looking at a subset of known good links. Using Fresh Web Explorer, I looked up fresh links to toysrus.com which is currently gaining a great deal of attention due to stores closing. We can feel confident that most of these links will be in-content because the articles themselves are about the relevant, breaking news regarding Toys R Us. Sure enough, after testing 300+ mentions, we found the links to be identical in the mobile and desktop crawls. These were good, in-content links and, subsequently, they showed up in both versions of the crawl.

Selection bias and convergence

It is probably the case that popular sites are more likely to have a mobile version than non-popular sites. Now, they might be responsive — at which point they would yield no real differences in the crawl — but at least some percentage would likely be m.* domains or utilize plugins like those mentioned above which truncate the content. At the lower rungs of the web, older, less professional content is likely to have only one version which is shown to mobile and desktop devices alike. If this is the case, we can expect that over time the differences in the index might begin to converge rather than diverge, as my study looked only at sites that were in the top million and only crawled two levels deep.

Moreover (this one is a bit speculative), but I think over time that there will be convergence between a mobile and desktop index. I don’t think the link graphs will grow exponentially different as the linked web is only so big. Rather, the paths to which certain pages are reached, and the frequency with which they are reached, will change quite a bit. So, while the link graph will differ, the set of URLs making up the link graph will largely be the same. Of course, some percentage of the mobile web will remain wholly disparate. The large number of sites that use dedicated mobile subdomains or plugins that remove substantial sections of content will remain like mobile islands in the linked web.

Impact on SERPs

It’s difficult at this point to say what the impact on search results will be. It will certainly not leave the SERPs unchanged. What would be the point of Google making and announcing a change to its indexing methods if it didn’t improve the SERPs?

That being said, this study wouldn’t be complete without some form of impact assessment. Hat tip to JR Oakes for giving me this critique, otherwise I would have forgotten to take a look.

First, there are a couple of things which could mitigate dramatic shifts in the SERPs already, regardless of the veracity of this study:

  • A slow rollout means that shifts in SERPs will be lost to the natural ranking fluctuations we already see.
  • Google can seed URLs found by mobile or by desktop into their respective crawlers, thereby limiting index divergence. (This is a big one!)
  • Google could choose to consider, for link purposes, the aggregate of both mobile and desktop crawls, not counting one to the exclusion of the other.

Second, the relationships between domains may be less affected than other index metrics. What is the likelihood that the relationship between Domain X and Domain Y (more or less links) is the same for both the mobile- and desktop-based indexes? If the relationships tend to remain the same, then the impact on SERPs will be limited. We will call this relationship being “directionally consistent.”

To accomplish this part of the study, I took a sample of domain pairs from the mobile index and compared their relationship (more or less links) to their performance in the desktop index. Did the first have more links than the second in both the mobile and desktop? Or did they perform differently?

It turns out that the indexes were fairly close in terms of directional consistency. That is to say that while the link graphs as a whole were quite different, when you compared one domain to another at random, they tended in both data sets to be directionally consistent. Approximately 88% of the domains compared maintained directional consistency via the indexes. This test was only run comparing the mobile index domains to the desktop index domains. Future research might explore the reverse relationship.

So what’s next?: Moz and the mobile-first index

Our goal for the Moz link index has always been to be as much like Google as possible. It is with that in mind that our team is experimenting with a mobile-first index as well. Our new link index and Link Explorer in Beta seeks to be more than simply one of the largest link indexes on the web, but the most relevant and useful, and we believe part of that means shaping our index with methods similar to Google. We will keep you updated!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Related Website Articles

Posted in Latest NewsComments Off

How Does Mobile-First Indexing Work, and How Does It Impact SEO?

Posted by bridget.randolph

We’ve been hearing a lot about mobile-first indexing lately, as the latest development in Google’s ever-continuing efforts to make the web more mobile-friendly and reflect user behavior trends.

But there’s also a lot of confusion around what this means for the average business owner. Do you have to change anything? Everything? If your site is mobile-friendly, will that be good enough?

IS THIS GOING TO BE ANOTHER MOBILEGEDDON?!!

In this post I’ll go over the basics of what “mobile-first indexing” means, and what you may need to do about it. I’ll also answer some frequently asked questions about mobile-first indexing and what it means for our SEO efforts.

What is “mobile-first indexing”?

Mobile-first indexing is exactly what it sounds like. It just means that the mobile version of your website becomes the starting point for what Google includes in their index, and the baseline for how they determine rankings. If you monitor crawlbot traffic to your site, you may see an increase in traffic from Smartphone Googlebot, and the cached versions of pages will usually be the mobile version of the page.

It’s called “mobile-first” because it’s not a mobile-only index: for instance, if a site doesn’t have a mobile-friendly version, the desktop site can still be included in the index. But the lack of a mobile-friendly experience could impact negatively on the rankings of that site, and a site with a better mobile experience would potentially receive a rankings boost even for searchers on a desktop.

You may also want to think of the phrase “mobile-first” as a reference to the fact that the mobile version will be considered the primary version of your website. So if your mobile and desktop versions are equivalent — for instance if you’ve optimized your content for mobile, and/or if you use responsive design — this change should (in theory) not have any significant impact in terms of your site’s performance in search results.

However it does represent a fundamental reversal in the way Google is thinking about your website content and how to prioritize crawling and indexation. Remember that up until now the desktop site was considered the primary version (similar to a canonical URL) and the mobile site was treated as an “alternate” version for a particular use case. This is why Google encouraged webmasters with a separate mobile site (m.domain.com) to implement switchboard tags (which indicated the existence of a mobile URL version with a special rel=alternate tag). Google might not even make the effort to crawl and cache the mobile versions of all of these pages, as they could simply display that mobile URL to mobile searchers.

This view of the desktop version as the primary one often meant in practice that the desktop site would be prioritized by SEOs and marketing teams and was treated as the most comprehensive version of a website, with full content, structured data markup, hreflang (international tags), the majority of backlinks, etc.; while the mobile version might have lighter content, and/or not include the same level of markup and structure, and almost certainly would not receive the bulk of backlinks and external attention.

What should I do about mobile-first indexing?

The first thing to know is that there’s no need to panic. So far this change is only in the very earliest stages of testing, and is being rolled out very gradually only to websites which Google considers to be “ready” enough for this change to have a minimal impact.

According to Google’s own latest guidance on the topic, if your website is responsive or otherwise identical in its desktop and mobile versions, you may not have to do anything differently (assuming you’re happy with your current rankings!).

That said, even with a totally responsive site, you’ll want to ensure that mobile page speed and load time are prioritized and that images and other (potentially) dynamic elements are optimized correctly for the mobile experience. Note that with mobile-first indexing, content which is collapsed or hidden in tabs, etc. due to space limitations will not be treated differently than visible content (as it may have been previously), since this type of screen real estate management is actually a mobile best practice.

If you have a separate mobile site, you’ll want to check the following:

  • Content: make sure your mobile version has all the high-quality, valuable content that exists on your desktop site. This could include text, videos and images. Make sure the formats used on the mobile version are crawlable and indexable (including alt-attributes for images).
  • Structured data: you should include the same structured data markup on both the mobile and desktop versions of the site. URLs shown within structured data on mobile pages should be the mobile version of the URL. Avoid adding unnecessary structured data if it isn’t relevant to the specific content of a page.
  • Metadata: ensure that titles and meta descriptions are equivalent on both versions of all pages.
    • Note that the official guidance says “equivalent” rather than “identical” – you may still want to optimize your mobile titles for shorter character counts, but make sure the same information and relevant keywords are included.
  • Hreflang: if you use rel=hreflang for internationalization, your mobile URLs’ hreflang annotations should point to the mobile version of your country or language variants, and desktop URLs should point to the desktop versions.
  • Social metadata: OpenGraph tags, Twitter cards and other social metadata should be included on the mobile version as well as the desktop version.
  • XML and media sitemaps: ensure that any links to sitemaps are accessible from the mobile version of the site. This also applies to robots directives (robots.txt and on-page meta-robots tags) and potentially even trust signals, like links to your privacy policy page.
  • Search Console verification: if you have only verified your desktop site in Google Search Console, make sure you also add and verify the mobile version.
  • App indexation: if you have app indexation set up for your desktop site, you may want to ensure that you have verified the mobile version of the site in relation to app association files, etc.
  • Server capacity: Make sure that your host servers can handle increased crawl rate.
    • (This only applies for sites with their mobile version on a separate host, such as m.domain.com.)
  • Switchboard tags: if you currently have mobile switchboard tags implemented, you do not need to change this implementation. These should remain as they are.

Common questions about mobile-first indexing

Is mobile-first indexing adding mobile pages to a separate mobile index?

With mobile-first indexing, there is only one index (the same one Google uses now). The change to mobile-first indexing does not generate a new “mobile-first” index, nor is it creating a separate “mobile index” with a “desktop index” remaining active. Instead, it simply changes how content is added to the existing index.

Is the mobile-first index live and affecting my site now? If not, when does it go live?

Google has been experimenting with this approach to indexing on a small number of sites, which were selected based on perceived “readiness”. A wider rollout is likely going to take a long time and in June 2017, Gary Illyes stated that it will probably take a few years before “we reach an index that is only mobile-first.”

Google has also stated the following on the Webmasters Blog, in a blog post dated Dec 18 2017:

“We will be evaluating sites independently on their readiness for mobile-first indexing based on the above criteria and transitioning them when ready. This process has already started for a handful of sites and is closely being monitored by the search team.

“We continue to be cautious with rolling out mobile-first indexing. We believe taking this slowly will help webmasters get their sites ready for mobile users, and because of that, we currently don’t have a timeline for when it’s going to be completed.”

Will Google only use my mobile site to determine my rankings?

Mobile-first means that the mobile version will be considered the primary version when it comes to how rankings are determined. However, there may be circumstances where the desktop version could be taken into consideration (for instance, if you don’t have a mobile version of a page).

That being said, you will potentially still see varying ranking results between mobile search results and desktop search results, so you’ll still want to track both. (In the same way that now, Google primarily uses the desktop site to determine rankings but you still want to track mobile rankings as these vary from desktop rankings based on user behavior and other factors).

When might Google use the desktop site to determine rankings vs. the mobile site?

The primary use case I’ve seen referred to so far is that they will use the desktop site to determine rankings when there is no mobile version.

It is possible that for websites where the desktop version has additional ranking information (such as backlinks), that information could also be taken into consideration – but there is no guarantee that they will crawl or index the desktop version once they’ve seen the mobile version, and I haven’t seen any official statements that this would be the case.

Therefore one of the official recommendations is that once the mobile-first indexing rollout happens, if you’re in the process of building your mobile site or have a “placeholder” type mobile version currently live it would actually be better to have no mobile site than a broken or incomplete one. In this case, you should wait to launch your mobile site until it is fully ready.

What if I don’t have a mobile version of my site?

If you don’t have a mobile version of your site and your desktop version is not mobile-friendly, your content can still be indexed; however you may not rank as well in comparison to mobile-friendly websites. This may even negatively impact your overall rankings on desktop search as well as mobile search results because it will be perceived as having a poorer user experience than other sites (since the crawler will be a “mobile” crawler).

What could happen to sites with a large desktop site and a small mobile site? Will content on your desktop site that does not appear on the mobile version be indexed and appear for desktop searches?

The end goal for this rollout is that the index will be based predominantly on crawling mobile content. If you have a heavily indexed desktop version, they’re not going to suddenly purge your desktop content from the existing index and start fresh with just your thin mobile site indexed; but the more you can ensure that your mobile version contains all relevant and valuable content, the more likely it is to continue to rank well, particularly as they cut back on crawling desktop versions of websites.

How does this change ranking factors and strategy going forward?

This may impact a variety of ranking factors and strategy in the future; Cindy Krum at Mobile Moxie has written two excellent articles on what could be coming in the future around this topic.

Cindy talks about the idea that mobile-first indexing may be “an indication that Google is becoming less dependent on traditional links and HTML URLS for ranking.” It seems that Google is moving away from needing to rely so much on a “URL” system of organizing content, in favor of a more API type approach based on “entities” (thanks, structured data!) rather than URL style links. Check out Cindy’s posts for more explanation of how this could impact the future of search and SEO.

Is there a difference between how responsive sites and separate mobile sites will be treated?

Yes and no. The main difference will be in terms of how much work you have to do to get ready for this change.

If you have a fully responsive site, you should already have everything present on your mobile version that is currently part of the desktop version, and your main challenge will simply be to ensure that the mobile experience is well optimized from a user perspective (e.g. page speed, load time, navigation, etc).

With a separate mobile site, you’ll need to make sure that your mobile version contains everything that your desktop site does, which could be a lot of work depending on your mobile strategy so far.

Will this change how I should serve ads/content/etc. on my mobile site?

If your current approach to ads is creating a slow or otherwise poor user experience you will certainly need to address that.

If you currently opt to hide some of your mobile site content in accordions or tabs to save space, this is actually not an issue as this content will be treated in the same way as if it was loaded fully visible (as long as the content is still crawlable/accessible).

Does this change how I use rel=canonical/switchboard tags?

No. For now, Google has stated that if you have already implemented switchboard tags, you should leave them as they are.


Has this overview helped you to feel more prepared for the shift to mobile-first indexing? Are there any questions you still have?

I’d love to hear what you’re thinking about in the comments!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Posted in Latest NewsComments Off

Does Googlebot Support HTTP/2? Challenging Google’s Indexing Claims – An Experiment

Posted by goralewicz

I was recently challenged with a question from a client, Robert, who runs a small PR firm and needed to optimize a client’s website. His question inspired me to run a small experiment in HTTP protocols. So what was Robert’s question? He asked…

Can Googlebot crawl using HTTP/2 protocols?

You may be asking yourself, why should I care about Robert and his HTTP protocols?

As a refresher, HTTP protocols are the basic set of standards allowing the World Wide Web to exchange information. They are the reason a web browser can display data stored on another server. The first was initiated back in 1989, which means, just like everything else, HTTP protocols are getting outdated. HTTP/2 is one of the latest versions of HTTP protocol to be created to replace these aging versions.

So, back to our question: why do you, as an SEO, care to know more about HTTP protocols? The short answer is that none of your SEO efforts matter or can even be done without a basic understanding of HTTP protocol. Robert knew that if his site wasn’t indexing correctly, his client would miss out on valuable web traffic from searches.

The hype around HTTP/2

HTTP/1.1 is a 17-year-old protocol (HTTP 1.0 is 21 years old). Both HTTP 1.0 and 1.1 have limitations, mostly related to performance. When HTTP/1.1 was getting too slow and out of date, Google introduced SPDY in 2009, which was the basis for HTTP/2. Side note: Starting from Chrome 53, Google decided to stop supporting SPDY in favor of HTTP/2.

HTTP/2 was a long-awaited protocol. Its main goal is to improve a website’s performance. It’s currently used by 17% of websites (as of September 2017). Adoption rate is growing rapidly, as only 10% of websites were using HTTP/2 in January 2017. You can see the adoption rate charts here. HTTP/2 is getting more and more popular, and is widely supported by modern browsers (like Chrome or Firefox) and web servers (including Apache, Nginx, and IIS).

Its key advantages are:

  • Multiplexing: The ability to send multiple requests through a single TCP connection.
  • Server push: When a client requires some resource (let’s say, an HTML document), a server can push CSS and JS files to a client cache. It reduces network latency and round-trips.
  • One connection per origin: With HTTP/2, only one connection is needed to load the website.
  • Stream prioritization: Requests (streams) are assigned a priority from 1 to 256 to deliver higher-priority resources faster.
  • Binary framing layer: HTTP/2 is easier to parse (for both the server and user).
  • Header compression: This feature reduces overhead from plain text in HTTP/1.1 and improves performance.

For more information, I highly recommend reading “Introduction to HTTP/2” by Surma and Ilya Grigorik.

All these benefits suggest pushing for HTTP/2 support as soon as possible. However, my experience with technical SEO has taught me to double-check and experiment with solutions that might affect our SEO efforts.

So the question is: Does Googlebot support HTTP/2?

Google’s promises

HTTP/2 represents a promised land, the technical SEO oasis everyone was searching for. By now, many websites have already added HTTP/2 support, and developers don’t want to optimize for HTTP/1.1 anymore. Before I could answer Robert’s question, I needed to know whether or not Googlebot supported HTTP/2-only crawling.

I was not alone in my query. This is a topic which comes up often on Twitter, Google Hangouts, and other such forums. And like Robert, I had clients pressing me for answers. The experiment needed to happen. Below I’ll lay out exactly how we arrived at our answer, but here’s the spoiler: it doesn’t. Google doesn’t crawl using the HTTP/2 protocol. If your website uses HTTP/2, you need to make sure you continue to optimize the HTTP/1.1 version for crawling purposes.

The question

It all started with a Google Hangouts in November 2015.

When asked about HTTP/2 support, John Mueller mentioned that HTTP/2-only crawling should be ready by early 2016, and he also mentioned that HTTP/2 would make it easier for Googlebot to crawl pages by bundling requests (images, JS, and CSS could be downloaded with a single bundled request).

“At the moment, Google doesn’t support HTTP/2-only crawling (…) We are working on that, I suspect it will be ready by the end of this year (2015) or early next year (2016) (…) One of the big advantages of HTTP/2 is that you can bundle requests, so if you are looking at a page and it has a bunch of embedded images, CSS, JavaScript files, theoretically you can make one request for all of those files and get everything together. So that would make it a little bit easier to crawl pages while we are rendering them for example.”

Soon after, Twitter user Kai Spriestersbach also asked about HTTP/2 support:

His clients started dropping HTTP/1.1 connections optimization, just like most developers deploying HTTP/2, which was at the time supported by all major browsers.

After a few quiet months, Google Webmasters reignited the conversation, tweeting that Google won’t hold you back if you’re setting up for HTTP/2. At this time, however, we still had no definitive word on HTTP/2-only crawling. Just because it won’t hold you back doesn’t mean it can handle it — which is why I decided to test the hypothesis.

The experiment

For months as I was following this online debate, I still received questions from our clients who no longer wanted want to spend money on HTTP/1.1 optimization. Thus, I decided to create a very simple (and bold) experiment.

I decided to disable HTTP/1.1 on my own website (https://goralewicz.com) and make it HTTP/2 only. I disabled HTTP/1.1 from March 7th until March 13th.

If you’re going to get bad news, at the very least it should come quickly. I didn’t have to wait long to see if my experiment “took.” Very shortly after disabling HTTP/1.1, I couldn’t fetch and render my website in Google Search Console; I was getting an error every time.

My website is fairly small, but I could clearly see that the crawling stats decreased after disabling HTTP/1.1. Google was no longer visiting my site.

While I could have kept going, I stopped the experiment after my website was partially de-indexed due to “Access Denied” errors.

The results

I didn’t need any more information; the proof was right there. Googlebot wasn’t supporting HTTP/2-only crawling. Should you choose to duplicate this at home with our own site, you’ll be happy to know that my site recovered very quickly.

I finally had Robert’s answer, but felt others may benefit from it as well. A few weeks after finishing my experiment, I decided to ask John about HTTP/2 crawling on Twitter and see what he had to say.

(I love that he responds.)

Knowing the results of my experiment, I have to agree with John: disabling HTTP/1 was a bad idea. However, I was seeing other developers discontinuing optimization for HTTP/1, which is why I wanted to test HTTP/2 on its own.

For those looking to run their own experiment, there are two ways of negotiating a HTTP/2 connection:

1. Over HTTP (unsecure) – Make an HTTP/1.1 request that includes an Upgrade header. This seems to be the method to which John Mueller was referring. However, it doesn’t apply to my website (because it’s served via HTTPS). What is more, this is an old-fashioned way of negotiating, not supported by modern browsers. Below is a screenshot from Caniuse.com:

2. Over HTTPS (secure) – Connection is negotiated via the ALPN protocol (HTTP/1.1 is not involved in this process). This method is preferred and widely supported by modern browsers and servers.

A recent announcement: The saga continues

Googlebot doesn’t make HTTP/2 requests

Fortunately, Ilya Grigorik, a web performance engineer at Google, let everyone peek behind the curtains at how Googlebot is crawling websites and the technology behind it:

If that wasn’t enough, Googlebot doesn’t support the WebSocket protocol. That means your server can’t send resources to Googlebot before they are requested. Supporting it wouldn’t reduce network latency and round-trips; it would simply slow everything down. Modern browsers offer many ways of loading content, including WebRTC, WebSockets, loading local content from drive, etc. However, Googlebot supports only HTTP/FTP, with or without Transport Layer Security (TLS).

Googlebot supports SPDY

During my research and after John Mueller’s feedback, I decided to consult an HTTP/2 expert. I contacted Peter Nikolow of Mobilio, and asked him to see if there were anything we could do to find the final answer regarding Googlebot’s HTTP/2 support. Not only did he provide us with help, Peter even created an experiment for us to use. Its results are pretty straightforward: Googlebot does support the SPDY protocol and Next Protocol Navigation (NPN). And thus, it can’t support HTTP/2.

Below is Peter’s response:


I performed an experiment that shows Googlebot uses SPDY protocol. Because it supports SPDY + NPN, it cannot support HTTP/2. There are many cons to continued support of SPDY:

    1. This protocol is vulnerable
    2. Google Chrome no longer supports SPDY in favor of HTTP/2
    3. Servers have been neglecting to support SPDY. Let’s examine the NGINX example: from version 1.95, they no longer support SPDY.
    4. Apache doesn’t support SPDY out of the box. You need to install mod_spdy, which is provided by Google.

To examine Googlebot and the protocols it uses, I took advantage of s_server, a tool that can debug TLS connections. I used Google Search Console Fetch and Render to send Googlebot to my website.

Here’s a screenshot from this tool showing that Googlebot is using Next Protocol Navigation (and therefore SPDY):

I’ll briefly explain how you can perform your own test. The first thing you should know is that you can’t use scripting languages (like PHP or Python) for debugging TLS handshakes. The reason for that is simple: these languages see HTTP-level data only. Instead, you should use special tools for debugging TLS handshakes, such as s_server.

Type in the console:

sudo openssl s_server -key key.pem -cert cert.pem -accept 443 -WWW -tlsextdebug -state -msg
sudo openssl s_server -key key.pem -cert cert.pem -accept 443 -www -tlsextdebug -state -msg

Please note the slight (but significant) difference between the “-WWW” and “-www” options in these commands. You can find more about their purpose in the s_server documentation.

Next, invite Googlebot to visit your site by entering the URL in Google Search Console Fetch and Render or in the Google mobile tester.

As I wrote above, there is no logical reason why Googlebot supports SPDY. This protocol is vulnerable; no modern browser supports it. Additionally, servers (including NGINX) neglect to support it. It’s just a matter of time until Googlebot will be able to crawl using HTTP/2. Just implement HTTP 1.1 + HTTP/2 support on your own server (your users will notice due to faster loading) and wait until Google is able to send requests using HTTP/2.


Summary

In November 2015, John Mueller said he expected Googlebot to crawl websites by sending HTTP/2 requests starting in early 2016. We don’t know why, as of October 2017, that hasn’t happened yet.

What we do know is that Googlebot doesn’t support HTTP/2. It still crawls by sending HTTP/ 1.1 requests. Both this experiment and the “Rendering on Google Search” page confirm it. (If you’d like to know more about the technology behind Googlebot, then you should check out what they recently shared.)

For now, it seems we have to accept the status quo. We recommended that Robert (and you readers as well) enable HTTP/2 on your websites for better performance, but continue optimizing for HTTP/ 1.1. Your visitors will notice and thank you.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Posted in Latest NewsComments Off

Google begins mobile-first indexing, using mobile content for all search rankings

While called an ‘experiment,’ it’s actually the first move in Google’s planned shift to looking primarily at mobile content, rather than desktop, when deciding how to rank results.

The post Google begins mobile-first indexing, using mobile content for all search rankings appeared first on Search…



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in Latest NewsComments Off

All About App Search: Indexing, Ranking Factors, Universal Links, and More – Whiteboard Friday

Posted by Tom-Anthony

App search is growing and changing, and there’s more opportunity than ever to both draw customers in at the top of the funnel and retain them at the bottom. In today’s special British Whiteboard Friday, Tom Anthony and Will Critchlow of Distilled dig into everything app search and highlight a future where Google may have some competition as the search engine giant.

App Search Whiteboard

Click on the whiteboard image above to open a high resolution version in a new tab!

Video Transcription

Tom: Howdy, and welcome to another British Whiteboard Friday. I’m Tom Anthony, head of the R&D Department here at Distilled. This is Will Critchlow, founder and CEO. Today we’re going to be talking about app search. App search is really, really important at the moment because research shows that the average user is spending 85% of their time in apps on their mobile phone.
Will, tell us a bit about app search.

Will: When we say “app search,” we could potentially mean three things. The first is App Store Optimization or ASO, which is not what we’re going to be talking about today. It’s an important area, and it’s got its own quirks and intricacies, but it’s pretty far down the funnel. Most of the searches in app stores are either branded or high-level category searches.

What we want to spend more of our time on today is…

App indexing

This is right at the top of the funnel typically, and it’s taking over the opportunities to rank in long-tail search. So this gives you the opportunity to acquire new users via search really for the first time in app marketing.
The third element that we’ll touch on later is the personal corpus, which is the idea right down at the bottom of the funnel and it’s about retaining the users once you have them.

The critical thing is app indexing. That’s what we want to spend most of our time on. What are the basics, Tom? What are the prerequisites for app indexing?

Tom: The first thing, the most important thing to understand is deep links.

Close-up of App Search whiteboard: a tree graph depicting Deep Links leading to the Distilled Twitter account.

Tom: People sometimes struggle to understand deep links, but it’s a very simple concept. It’s the parallel of what a normal URL is for a web page. A URL takes you to a specific web page rather than a website. Deep links allow you to open a specific screen in an app.
So you might click a deep link. It’s just a form of a URL. It might be on a web page. It might be in another app. It can open you to a specific point in an app, for example the @Distilled page in the Twitter app.
There’s been various competing standards for how deep links should work on different platforms. But what’s important to understand is that everyone is converging on one format. So don’t bother trying to learn all the intricacies of it.
The important format is what we call universal links. Will, tell us a bit about them.

Will: Universal links — this is actually Apple’s terminology, but it is, as Tom said, spreading everywhere — which is the idea that you can take a URL just like we use to a regular HTTP or HTTPS URL and this URL would normally open up the web page on the desktop.

Close-up of App Search whiteboard: a URL pointing at a web page

Will: Now if instead we were on a mobile device — and we’ve brought our mobile whiteboard again to demonstrate this concept — then if you clicked on this same link on your mobile device, same URL, it would open up the deep view within the app like Tom mentioned.
So the critical thing about the universal link is that the form of this link is the same, and it’s shared across those different devices and platforms.

Now before that was the case, in the world where we had different kinds of links, different kinds of link formats for the different devices and platforms, it was important that we mapped our web pages to those mobile URLs. There were various ways of doing that. So you could use Schema.org markup on your web pages. You could use JSON-LD. You could match them all up in your robots.txt. Or you could use rel=”alternate” links.

Tom: This is much like how you would’ve done the same thing for the mobile version of a desktop web page.

Will: Right. Yeah, if you had a different mobile website, an m-dot website for example, you would use rel=”alternate” to match those two together. In the old world of deep links, where there were the application-specific links, you could use this rel=”alternate” to map them together.

Close-up of whiteboard: a normal desktop page on the left with a two-sided arrow with "alternate" written underneath, a drawing of a mobile phone to the right

If you’re using universal links, it’s not so much about this mapping anymore. It’s not about saying it’s over there. But it’s about advertising the fact that there is an app, that you have an app that can open this particular view or web page. That’s kind of important obviously to get that indexed and to get that app ranking.

Tom: Google and Co. are encouraging you to have parity at the moment between your app. So you’ve got your desktop site, your mobile site, and then you’ve got the same screen in the mobile application.

Will: Absolutely, and they’d like that all to be on these universal URLs. Now all of this so far is pretty familiar to us as search marketers. We understand the concept of having these URLs, having them crawled, having them indexed. But in the app world there’s more opportunity than just crawling because both Google and Apple on iOS have opened up APIs, which means that you can push information to the search engine about how the app is actually being used, which opens up all kinds of interesting possibilities.

Tom: Absolutely. The first one is new types of ranking factor, the big one being engagement. Apple have already confirmed that they’re going to use engagement as a ranking factor. We anticipate that Google will do the same thing.
This is the idea that users opening your app, using your app, spending time in your app is a clue of the value of that app. So it’s more likely to appear in search results. There are two layers to this. The first is appearing in personalized search results. If I use a specific app a lot, then I’ll expect to see that more.
Then, there’s the second level, which is the aggregated user statistics, which is where they see that most people like this app for this thing, so other people will see that in the search results.

The second point is taking us back to what Will mentioned at the start.

The personal corpus

This is the idea where you get search results specific to yourself coming from your data. So you might run a search and you’ll see things such as your messages, entries in your calendar, photos from your gallery. I’d see different results to Will, and I’d see them all in the same interface as where I’d see the public search results.

So I might do a search for a restaurant. I might see a link to the restaurant’s website in the public search results, but I might also see that Will sent me a message about going for dinner at that restaurant, and there might be an entry in my calendar, which other people wouldn’t see. It’s a really interesting way that we might start to appear in search results in a new format.

Then the third interesting thing here is the idea of app-only indexing.

Closeup of whiteboard: Showing the top of the funnel (app indexing) and the bottom of the funnel (a personal corpus).

With universal links, we talked about needing parity between the desktop site, the mobile site, the app. With app-only indexing, we could be looking at a model where there are screens in apps that don’t have a web equivalent. So you might start to see search results where there’s no possibility of a website actually appearing for that. That’s also a fascinating new model. Apple already do this. Google have confirmed that they’re going to be doing this. So it’s definitely coming.

Then further out into the future one of the important things is going to be app streaming. So Will, are you going to tell us a bit about that?

Will: Right. App streaming, this is another thing that Google has announced. It’s kind of available in limited trials, but we think it’s going to be a bigger thing because they’re trying to attack this core problem, which is that to use an app and for an app to appear in search results, if you haven’t already got it, you have to download it and you have to install it. That’s both a slow process and a data-hungry process. If you’re just kicking the tires, if this is an app you’ve never seen before, it’s a little bit too much to ask you to do this multi-megabyte download and then install this app, just to try it out.

So what they’re trying with app streaming is saying, “We can simplify that process. This is an app you’ve not used before. Let’s preview it for you.” So you can use it. You can see it. You can certainly check out the public areas of the app and then install it if it’s useful to you.

The current setup is a little bit of a kind of a kludge; they’re running in a virtual machine in the cloud and streaming. It’s all very weird. We think the details are going to change.

Tom: Yeah.

Will: Fundamentally, they’re going to figure out a way to make this streamlined and smooth, and it will become much easier to use apps for the first time, making it possible to expose them in a much broader array of search results. Then there’s all kinds of other things and stuff coming in the future. I mean, Tom’s passionate about the personal assistant.

Tom: Yeah. The intelligent personal assistant thing is really, really exciting to me. By intelligent personal assistant, I mean things like Siri, Cortana, Google Now, and the up-and-coming ones — Facebook M and SoundHound’s Hound app. What’s fascinating about personal assistants is that when you do a search, you do a search for weather in Siri for example, you just get a card about the weather for where you are. You don’t get taken to a list of results and taken elsewhere. You just get a direct answer.
Most of the personal assistants are already able to answer a lot of search queries using this direct answer methodology. But what we think is exciting about apps is that we anticipate a future where you can store an app and it allows the personal assistants to tap into that app’s data to answer queries directly. So you can imagine I could do a search for “are the trains running on time.” Siri taps into my train app, pulls that data, and just lets me know right there. So no longer am I opening the app. What’s important is the app is actually sort of a gateway through to a data source in the backend. We start to get all this data pulled into a central place.

Will: It’s fascinating. You mentioned a whole bunch of different tools, companies, platforms coming up there. The final thing that we want to point out is that this is a really interesting space because Google’s had a lock on web search for what feels like forever.
App search is a whole new area. Obviously, Google has some advantages just through the fact that the Android devices and they’ve got the apps installed in so many places and it’s part of people’s habits. But there are certainly opportunities. It’s the first crack. It’s first chink in the armor that means that maybe there are some upcoming players who will be interesting to watch and interesting for us as marketers to pay attention to.

Thank you for joining us here in Distilled’s London HQ. It’s been great talking to you. Thank you for taking the time. Bye.

Tom: Bye.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Posted in Latest NewsComments Off

SearchCap: Google assistant, AMP stats & Firebase app indexing

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.

The post SearchCap: Google assistant, AMP stats & Firebase app indexing appeared first on Search Engine Land.



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in Latest NewsComments Off

SearchCap: Google PLA test, Google app indexing study & more

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.

The post SearchCap: Google PLA test, Google app indexing study & more appeared first on Search Engine Land.



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Related Articles

Posted in Latest NewsComments Off

SearchCap: Google Smarter Search, Facebook Google App Indexing & Foursquare With Apple Maps

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.

The post SearchCap: Google Smarter Search, Facebook Google App Indexing & Foursquare With Apple Maps appeared first on Search Engine Land.



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in Latest NewsComments Off

Google Search Adds Support For App Indexing In Safari On iOS 9

By the end of October, searching on Google in Safari on iOS 9 will provide deep links into apps, when appropriate and where publishers have taken action.

The post Google Search Adds Support For App Indexing In Safari On iOS 9 appeared first on Search Engine Land.



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in Latest NewsComments Off

SearchCap: Google On Links, RTBF In US & Tweets Indexing

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.

The post SearchCap: Google On Links, RTBF In US & Tweets Indexing appeared first on Search Engine Land.



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Find More Articles

Posted in Latest NewsComments Off

Advert