New and improved website designs might impress visitors, but they may not provide crawlers with all the information they need.

Companies with better websites have a search advantage over brands with dated domains, according to Google’s Matt Cutts. In a recent Webmaster Help Channel video, Cutts said stale sites with older templates can easily be outranked by pages with fancier bells and whistles because they’re more aligned with users’ expectations. However, sites with infinite scroll may not be sending all the right signals to search crawlers.

No crawling = No search presence

A recent Google Webmaster Central Blog explained that pages with infinite scroll (the ones where web content continues to appear as you scroll down) rather than traditional pagination may not consistently be crawled. When Googlebots visit those pages, they can only crawl the links that are already loaded on the screen and will generally only show those in search results. This means brands’ valuable content may be virtually invisible unless a visitor has already landed on the page and loaded all the content.

As the post details, “Crawlers cannot always emulate manual user behavior – like scrolling or clicking a button to load more items – so they don’t always access all individual items in the feed or gallery.”

“Crawlers cannot always emulate manual user behavior – like scrolling or clicking a button to load more items”

Chunk content for easy crawling

To make infinite scroll sites search-friendly, webmasters must chunk their content into pieces that crawlers can access and index. According to the post, a chunk should contain the amount of information that can be seen when JavaScript is disabled. They should then create structured URLs for each “page” of content without overlapping.

To get specific, there should be a “Page 1” and a “Page 2,” but not a “Page 1.5” that shows all the content falling between the first and second pages. A crawler might mistake this additional information as duplicated content, the post added.

Confused? Talk to your webmaster

A lot of marketers get lost when it comes to technical website development, and they may need to query their webmasters to make sure infinite scroll pages are optimized for search. If troubleshooting isn’t a problem, they can use the following process to test their website chunks:

There is fine line marketers must walk in today’s search environment between user optimization and designing for search crawlers. 

“[Make sure] each component page (the URL) works to take anyone directly to the content and is accessible/referenceable in a browser without the same cookie or user history,” the post advised.

Google points to resources that might help, such as a video by Webmaster Trends Analyst John Mueller. As a whole, this news (combine with Cutts’ recent guidance about updating Web 2.0 designs) implies there is fine line marketers must walk in today’s search environment between user optimization and designing for search crawlers.

SEO isn’t dead, and companies are smart to consider crawlers (and their limitations) when designing pages that will rank.

Lauren Kaye is a Marketing Editor at Brafton Inc. She studied creative and technical writing at Virginia Tech before pursuing the digital frontier and finding content marketing was the best place to put her passions to work. Lauren also writes creative short fiction, hikes in New England and appreciates a good book recommendation.