Wednesday 24 June 2015

Unlock More Volume with Broad Match

A Bing product called Broad Match, a method of query-keyword matching, promises to get marketers more conversions and ensure they lose less clicks, while reaching a wider audience.

Saturday 20 June 2015

How to Estimate the Total Volume and Value of Keywords in a Given Market or Niche - Whiteboard Friday

Posted by randfish

To get a sense for the potential value of keywords in a certain niche, we need to do more than just look at the number of searches those keywords get each month. In today's Whiteboard Friday, Rand explains what else we should be looking at, and how we can use other data to prioritize some groups over others.

How to Estimate the Total Volume and Value of Keywords in a Given Market or Niche Whiteboard

For reference, here's a still of this week's whiteboard. Click on it to open a high resolution image in a new tab!

Video transcription

Howdy, Moz fans, and welcome to another edition of Whiteboard Friday. This week I want to chat about how you can estimate the total volume and value of a large set of keywords in a market or a niche.

Look, we're going to try and simplify this and reduce it to something that is actually manageable, because you can go way, way deep down a well. You could spend a year trying to figure out whether Market A or Market B is better to enter or better to chase keywords in, better to create content in. But I want to try and make it a little simple without reducing it to something that is of no value whatsoever, which unfortunately can be how some marketers have looked at this in the past.

Asian noodle keywords

So let's try this thought exercise. Let's say I'm a recipe site or a food site and I'm thinking I want to get into the Asian noodles scene. There's a lot of awesome Asian noodles out there. I, in fact, had Chow fun for lunch from Trove on Capitol Hill. When you come to MozCon, you have to try them. It's awesome.

So maybe I'm looking at Chow fun and sort of all the keyword sets around those, that Chinese noodle world. Maybe I'm looking at pad Thai, a very popular Thai noodle, particularly in the U.S., and maybe Vietnamese rice noodles or bun. I'm trying to figure out which of these is the one that I should target. Should I start creating a lot of pad Thai recipes, a lot of Chow fun recipes? Should I go research one or the other of these? Am I going to chase the mid and long tail keywords?

I'm about to invest a large amount of effort and really build up a brand around this. Which one of these should I do?

Side note, this is getting more and more important as Google is moving to these topic modeling and sight specific, topic authority models. So if Google starts to identify my site as being an authority on Chow fun, I can expect to rank for all sorts of awesome stuff around it, versus if I just kind of dive in and out and have one-offs of Chow fun and 50 different other kinds of noodles. So this gets really important.

The wrong way to look at AdWords data

A massively oversimplified version, that a lot of people have done in the past, is to look broadly at kind of AdWords groups, the ones that AdWords selects for you, or individual keywords and say, "Oh, okay. Well, Chow fun gets 22,000 searches a month, Pad Thai gets 165,000, and rice noodles, which is the most popular version of that query -- it could also be called Vietnamese noodles or bun noodles or something like that -- gets 27,000. So there you go, one, two, three.

This is dead wrong. It's totally oversimplified. It's not taking into account all the things we have to do to really understand the market.

First off, this isn't going to include all the variations, the mid and long tail keywords. So potentially there might be a ton of variations of rice noodles that actually add up to as much or more than pad Thai. Same thing with Chow fun. In fact, when I looked, it looked like there's a ton of Chow fun modifications and different kinds of things that go in there. The Pad Thai list is a little short. It's like chicken, vegetable, shrimp, and beef. Pretty simplistic.

There's also no analysis of the competition going on here. Pad Thai, yeah it's popular, but it also has 50 recipe sites all bidding for it, tons of online grocers bidding for it, tons of recipes books that are bidding on that. I don't know. Then it could be that Chow fun has almost no competition whatsoever. So you're really not considering that when you look in here.

Finally, and this can be important too, these numbers can be off by up to 200% plus or minus this number. So if you were to actually bid on Chow fun, you might see that you get somewhere in the 22,000 impressions per month, assuming your ad consistently shows up on page one, but you could see as little as 11,000. I've seen as much as 44,000, like huge variations and swings in either direction and not always totally consistent between these. You want them to be, but they're not always.

A better process

So because of that, we have to go deeper. These problems mean that we have to expend a little more energy. Not a ton. It doesn't have to be massive, but probably a week or two of work at least to try and figure this out. But it's so important I think it's worth it every time.

1) Keyword research dive

First off, we're going to conduct a broad keyword research dive into each one of these. Not as much as we would do if we knew, hey, Chow fun is the one we're going to target. We're going to go deep. We're going to find every possible keyword. We're going to do kind of what I call a broad dive, not a deep dive into each market. So I might want to go, hey, I'm going to look at the AdWords suggestions and tally those up. I'm going to look at search suggest and related searches for some of the queries that I get from AdWords, some of the top ones anyway, and I'm going to do a brief competitive analysis. Maybe I'll put the domains that I'm seeing most frequently around these specific topics into SEMrush or another tool like that -- SpyFu, Key Compete or whatever your preference might be -- and see what other terms and phrases they might be ranking on.

So now I've got a reasonable set. It probably didn't take me more than a few hours to put that together, if that. If I've got an efficient process for this already, maybe even less.

2) Bid on sample keyword sets

Now comes the tricky part. I want you to take a small sample set, and we've done this a few times. Random might be not the right word. It's a small considered set of keywords and bid on them through AdWords. When I say "considered," what I mean is a few from the long tail, a few from the chunky middle, and a few from the head of the demand curve that are getting lots and lots of searches. Now I want to point each of those to some new, high-quality pages on your site as a test.

So I might make maybe one, two, or three different landing pages for each of these different sets. One of them might be around noodles. One might be around recipes. One might be around history or uses in cuisine or whatever it is.

Then I am going to know from that exercise three critically important things. I'm going to know accuracy of AdWords volume estimates, which is awesome. Now I know whether these numbers mean anything or not, how far off they were or not. I could probably run for between 10 and 15 days and get a really good sense for the accuracy of AdWords. If you're feeling like being very comprehensive, run for a full month, especially if you have the budget, because you can learn even more over time, and you'll rule out any inconsistencies due to a particular spike, like maybe The New York Times recipe section features Chow fun that week and suddenly there's a huge spike or whatever it is.

You can also learn relative price competition in click-through rate. This is awesome. This means that I know it costs a lot more per visitor that I'm trying to get on pad Thai. There are two really good things to know there. When a click costs more money, that also usually means there are more advertisers willing to pay for that traffic.

If you're primarily on the organic side and you believe you can compete with the folks in the organic ranking, a very high bid price and payment price that you have to pay to AdWords is a good thing.

If you're on the other side of that, where you think, "Hey, look, we're not going to compete organically right now. We just don't have the domain authority to do it. It's going to take us a while," then a high price is a bad thing. You want that cheaper traffic so you can start to build up that brand through paid as you're growing the organic side. So it really depends on who you are and what situation you're in.

Then finally you can figure out some things around click-through rate as well, which is great to know. So you can build some true model estimates and then go into your board meeting or your client pitch or whatever it is and say, "Hey, here are the numbers."

Lastly, you're going to learn the difficulty of content creation, like how hard was it for you to create these kinds of things. Like, "Wow, when we write about Chow fun, it's just easy. It just rolls off. Pad Thai we have a really hard time creating unique value because everything has been done in that world. We're just not as passionate about those noodles as we are about Chow fun." Cool. Great, you know that.

Also, assuming your test includes this, which it doesn't always have to, you can guess from sort of engagement rate, browse rate, time on site, all those kinds of things, but you can look at search conversion as well. So let's say you have some action to complete on the page -- subscribe to our email newsletter, sign up to get updates when we send them out about this recipe, or create an account so you can sign in and save this recipe. All that kind of stuff or a direct ecommerce conversion, you can learn that through your bidding test.

3) Analyze groups based on relevant factors

Awesome. That's great. Now we really, really know something. Based on that, we can do a true analysis, an accurate analysis of the different groups based on:

  • Relative value
  • Difficulty
  • Opportunity
  • Growth rate
  • ROI

Growth rate might be an interpreted thing, but you can look at the Google trends to kind of figure out over time whether a broad group of terms is getting more or less popular. You could use something like Mention.net or Fresh Web Explorer from Moz to look at mentions as well.

Now, you can be happy here. I might have chosen chow fun because I looked and I said, "Hey, you know what, it did not have the most volume overall, but it did have the lightest competition, the highest return on investment. We were great at creating the content. We were able to engage our visitors there, had lots of mid and long tail terms. We think it's poised for big growth with the growth of Chinese noodles overall and the fact that the American food scene hasn't really discovered Chow fun the way they have Vietnamese noodles and pad Thai. So that is where we're placing our bet."

Great. Now you have a real analysis. You have numbers behind it. You have estimates you can make. This process, although a little heavy, is going to get you so much further than this kind of simplistic thinking.

All right, everyone, I look forward to hearing from you about how you've done analyses like these in the past, and we'll see you again next week for another edition of Whiteboard Friday. Take care.

Video transcription by Speechpad.com


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

Thursday 18 June 2015

The Colossus Update: Waking The Giant

Posted by Dr-Pete

Yesterday morning, we woke up to a historically massive temperature spike on MozCast, after an unusually quiet weekend. The 10-day weather looked like this:

That's 101.8°F, one of the hottest verified days on record, second only to a series of unconfirmed spikes in June of 2013. For reference, the first Penguin update clocked in at 93.1°.

Unfortunately, trying to determine how the algorithm changed from looking at individual keywords (even thousands of them) is more art than science, and even the art is more often Ms. Johnson's Kindergarten class than Picasso. Sometimes, though, we catch a break and spot something.

The First Clue: HTTPS

When you watch enough SERPs, you start to realize that change is normal. So, the trick is to find the queries that changed a lot on the day in question but are historically quiet. Looking at a few of these, I noticed some apparent shake-ups in HTTP vs. HTTPS (secure) URLs. So, the question becomes: are these anecdotes, or do they represent a pattern?

I dove in and looked at how many URLs for our 10,000 page-1 SERPs were HTTPS over the past few days, and I saw this:

On the morning of June 17, HTTPS URLs on page 1 jumped from 16.9% to 18.4% (a 9.9% day-over-day increase), after trending up for a few days. This represents the total real-estate occupied by HTTPS URLs, but how did rankings fare? Here are the average rankings across all HTTPS results:

HTTPS URLs also seem to have gotten a rankings boost – dropping (with "dropping" being a positive thing) from an average of 2.96 to 2.79 in the space of 24 hours.

Seems pretty convincing, right? Here's the problem: rankings don't just change because Google changes the algorithm. We are, collectively, changing the web every minute of the day. Often, those changes are just background noise (and there's a lot of noise), but sometimes a giant awakens.

The Second Clue: Wikipedia

Anecdotally, I noticed that some Wikipedia URLs seemed to be flipping from HTTP to HTTPS. I ran a quick count, and this wasn't just a fluke. It turns out that Wikipedia started switching their entire site to HTTPS around June 12 (hat tip to Jan Dunlop). This change is expected to take a couple of weeks.

It's just one site, though, right? Well, historically, this one site is the #1 largest land-holder across the SERP real-estate we track, with over 5% of the total page-1 URLs in our tracking data (5.19% as of June 17). Wikipedia is a giant, and its movements can shake the entire web.

So, how do we tease this apart? If Wikipedia's URLs had simply flipped from HTTP to HTTPS, we should see a pretty standard pattern of shake-up. Those URLs would look to have changed, but the SERPS around them would be quiet. So, I ran an analysis of what the temperature would've been if we ignored the protocol (treating HTTP/HTTPS as the same). While slightly lower, that temperature was still a scorching 96.6°F.

Is it possible that Wikipedia moving to HTTPS also made the site eligible for a rankings boost from previous algorithm updates, thus disrupting page 1 without any code changes on Google's end? Yes, it is possible – even a relatively small rankings boost for Wikipedia from the original HTTPS algorithm update could have a broad impact.

The Third Clue: Google?

So far, Google has only said that this was not a Panda update. There have been rumors that the HTTPS update would get a boost, as recently as SMX Advanced earlier this month, but no timeline was given for when that might happen.

Is it possible that Wikipedia's publicly announced switch finally gave Google the confidence to boost the HTTPS signal? Again, yes, it's possible, but we can only speculate at this point.

My gut feeling is that this was more than just a waking giant, even as powerful of a SERP force as Wikipedia has become. We should know more as their HTTPS roll-out continues and their index settles down. In the meantime, I think we can expect Google to become increasingly serious about HTTPS, even if what we saw yesterday turns out not to have been an algorithm update.

In the meantime, I'm going to melodramatically name this "The Colossus Update" because, well, it sounds cool. If this indeed was an algorithm update, I'm sure Google would prefer something sensible, like "HTTPS Update 2" or "Securageddon" (sorry, Gary).

Update from Google: Gary Illyes said that he's not aware of an HTTPS update (via Twitter):

No comment on other updates, or the potential impact of a Wikipedia change. I feel strongly that there is an HTTPS connection in the data, but as I said – that doesn't necessarily mean the algorithm changed.


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

What Social Search and New Domain Extensions Could Mean for Start-Ups

Brands like Google and Amazon have plans to launch their own dot brands in the next two years. More marketers should follow their lead as social and search become more connected.

Wednesday 17 June 2015

Search in Asia Goes Well Beyond Google

Google and Yahoo might be staple search engines in the West, but for consumers in markets like Japan and China there are a plethora of other options.

Tuesday 16 June 2015

Can You Rank in Google Without Links? New Data Says Slim Chance

Posted by Cyrus-Shepard

For years now, we've heard the drumbeat from Google that marketers should stop focusing on building links. While it's accepted wisdom that you should avoid manipulative link building to rank higher in search results, the popular narrative would have us believe that external links aren't important in Google's ranking algorithms at all, and that link building can be safely ignored.

Is there any truth to this?

To find out, we mined new information from our upcoming biannual ranking correlation study, conducted by Moz's scientist, Dr. Matthew Peters.

External Link Definition

https://moz.com/learn/seo/external-link

Moz's study examined the top 50 Google search results for approximately 15,000 keywords. This allowed us to examine not only what factors correlate with higher search rankings, but also how frequently those characteristics are seen.


At this point I must insert the usual caveat that correlation is not causation. Simply because a feature is strongly related to high rankings, this doesn't prove or disprove that Google actually uses it in its algorithm. That said, it sure is a hint!


The relationship between external links and rankings

When we look at what the study found about links, we find a strong relationship.

The correlation between higher rankings and the number of linking websites (root domains) sits at .30. This number seems small, but it's actually one of the highest correlations the study found. (Smaller correlations are also not surprising—with over 200 ranking signals, Google specifically designed their algorithm so that one factor doesn't dominate the others.)

Even more telling is the number of websites we found in the top results that had external backlinks, or rather, the lack thereof.

Out of the top results, a full 99.2% of all websites had at least one external link. (The remaining .8% is well within the margin of error expected between Mozscape and Google's own link index.) The study found almost no websites ranking for competitive search phrases that didn't have at least a single external link pointing at them, and most had significantly more links.

The Relationship Between Google Rankings And Links

In other words, if you're looking for a site that ranks well with no external links, be prepared to look for a very long time.

That said, the study did find numerous examples where individual pages ranked just fine without specific external links, as long as the website itself had external links pointing at it. For example, consider when The New York Times publishes a new page. Because it's new, it has no external links yet. But because The New York Times' website itself has tons of external links, it's possible for the new page to rank.

In all, 77.8% of individual pages in the top results had at least one external link from another site, which means 22.2% of individual pages ranked with no external links.

What the data says about links and Google rankings

There are a number of conclusions you can reasonably draw from these numbers.

1. External links are almost always present for competitive searches

If you want to rank for anything that's even remotely competitive, the chances of finding a website ranking without external links is very rare indeed.

2. It's possible to rank individual pages without links

As long as your website itself is linked externally, it appears more than possible to rank individual pages on your site, even if those pages themselves don't have external links. That said, there's a strong relationship between links to a page, and that pages performance in search—so it's much better if the page actually does have external links.

To put this in layman's terms, if a lot of people link to your website homepage, it's possible for other pages to rank as well, but it's even better if those pages also have external links pointing at them.

Although not examined in this study, it's likely most of the pages without external links at least had internal links pointing at them. While not as strong as an external link, internal links remain a decent way to pass authority, relevancy and popularity signals to pages on the same site.

3. More links correlate with higher rankings

It seems obvious, but the study confirmed the long-standing correlation between higher rankings and the number of external links found from unique websites.

Indeed, out of all the data points the ranking correlation study looked at, the number of unique websites linking to a page was one of the highest correlated relationships we found.

4. When can you rank without links?

Despite the fact that we found almost no websites ranking without external links, it is still possible?

Absolutely, but there's a catch.

The 15,000 keyword phrases used in this study were, for the most part, competitive. This means that lots of other people and websites are trying to rank for the same term. Think of phrases like "Galaxy s6" and "New York car insurance."

Non-competitive phrases, by their nature, are much easier to rank for. So if you want your website to rank without obtaining any backlinks, you might succeed by targeting more obscure phrases like "Oregon beekeeper ballet emporium" or "Batman flux platypus." These phrases have much lower competition, and by default, much lower traffic (and in many cases, none.)

There are other edge cases where it's possible to rank without links, such as when the user is searching for your website specifically, or when you offer something very unique that can't be found anywhere else. Even in these cases, it helps tremendously to actually have links pointing at you.

Proceed with caution

There's good reason people believe link building is dead, as readers of this blog know well. For readers less familiar with this concept, or those newer to SEO...

A link isn't always a link.

Google Penalty

In the past 10 years, after people spammed the heck out of link building to gain higher rankings, Google began cracking down in a serious way starting in 2012. First with its Penguin algorithm, then by de-indexing several link networks, and then by cracking down on guest blogging.

Today, even slight deviations from Google's guidelines on manipulative links can land webmasters in penalty jail.

The web is filled with links. Billions of them. Many are built by robots, some are paid for by advertisers, some are good old fashioned editorial links. The challenge for Google is to separate the good from the bad in its ranking algorithm.

When Google finds a link pointing at your website, it can choose to do one of 3 things:

  1. Count it in its ranking algorithm
  2. Ignore it - or not give it any weight in boosting your rankings
  3. Penalize you - if it thinks the link is manipulative

In fact, most people would be surprised to learn how many links don't actually help you to rank, or can actually hurt. To play within Google's good graces, it's best to understand Google's guidelines on manipulative link building, and knowing what types of links to avoid.

The safest link building is simply link earning, and to get your content in front of the right people.

But trying to rank in Google without any links at all?

Fuhgeddaboudit.


Photo Credit: Geographically Accurate Paris Metro Map by Nojhan under Creative Common License


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

Wednesday 10 June 2015

Apple to Upgrade Siri and Update Maps

At Apple's keynote during the Worldwide Developers' Conference, the company introduced plans to upgrade Siri's voice search capabilities and make Maps more efficient.

Saturday 6 June 2015

Should I Use Relative or Absolute URLs? - Whiteboard Friday

Posted by RuthBurrReedy

It was once commonplace for developers to code relative URLs into a site. There are a number of reasons why that might not be the best idea for SEO, and in today's Whiteboard Friday, Ruth Burr Reedy is here to tell you all about why.

Relative vs Absolute URLs Whiteboard

For reference, here's a still of this week's whiteboard. Click on it to open a high resolution image in a new tab!

Let's discuss some non-philosophical absolutes and relatives

Howdy, Moz fans. My name is Ruth Burr Reedy. You may recognize me from such projects as when I used to be the Head of SEO at Moz. I'm now the Senior SEO Manager at BigWing Interactive in Oklahoma City. Today we're going to talk about relative versus absolute URLs and why they are important.

At any given time, your website can have several different configurations that might be causing duplicate content issues. You could have just a standard http://www.example.com. That's a pretty standard format for a website.

But the main sources that we see of domain level duplicate content are when the non-www.example.com does not redirect to the www or vice-versa, and when the HTTPS versions of your URLs are not forced to resolve to HTTP versions or, again, vice-versa. What this can mean is if all of these scenarios are true, if all four of these URLs resolve without being forced to resolve to a canonical version, you can, in essence, have four versions of your website out on the Internet. This may or may not be a problem.

It's not ideal for a couple of reasons. Number one, duplicate content is a problem because some people think that duplicate content is going to give you a penalty. Duplicate content is not going to get your website penalized in the same way that you might see a spammy link penalty from Penguin. There's no actual penalty involved. You won't be punished for having duplicate content.

The problem with duplicate content is that you're basically relying on Google to figure out what the real version of your website is. Google is seeing the URL from all four versions of your website. They're going to try to figure out which URL is the real URL and just rank that one. The problem with that is you're basically leaving that decision up to Google when it's something that you could take control of for yourself.

There are a couple of other reasons that we'll go into a little bit later for why duplicate content can be a problem. But in short, duplicate content is no good.

However, just having these URLs not resolve to each other may or may not be a huge problem. When it really becomes a serious issue is when that problem is combined with injudicious use of relative URLs in internal links. So let's talk a little bit about the difference between a relative URL and an absolute URL when it comes to internal linking.

With an absolute URL, you are putting the entire web address of the page that you are linking to in the link. You're putting your full domain, everything in the link, including /page. That's an absolute URL.

However, when coding a website, it's a fairly common web development practice to instead code internal links with what's called a relative URL. A relative URL is just /page. Basically what that does is it relies on your browser to understand, "Okay, this link is pointing to a page that's on the same domain that we're already on. I'm just going to assume that that is the case and go there."

There are a couple of really good reasons to code relative URLs

1) It is much easier and faster to code.

When you are a web developer and you're building a site and there thousands of pages, coding relative versus absolute URLs is a way to be more efficient. You'll see it happen a lot.

2) Staging environments

Another reason why you might see relative versus absolute URLs is some content management systems -- and SharePoint is a great example of this -- have a staging environment that's on its own domain. Instead of being example.com, it will be examplestaging.com. The entire website will basically be replicated on that staging domain. Having relative versus absolute URLs means that the same website can exist on staging and on production, or the live accessible version of your website, without having to go back in and recode all of those URLs. Again, it's more efficient for your web development team. Those are really perfectly valid reasons to do those things. So don't yell at your web dev team if they've coded relative URLS, because from their perspective it is a better solution.

Relative URLs will also cause your page to load slightly faster. However, in my experience, the SEO benefits of having absolute versus relative URLs in your website far outweigh the teeny-tiny bit longer that it will take the page to load. It's very negligible. If you have a really, really long page load time, there's going to be a whole boatload of things that you can change that will make a bigger difference than coding your URLs as relative versus absolute.

Page load time, in my opinion, not a concern here. However, it is something that your web dev team may bring up with you when you try to address with them the fact that, from an SEO perspective, coding your website with relative versus absolute URLs, especially in the nav, is not a good solution.

There are even better reasons to use absolute URLs

1) Scrapers

If you have all of your internal links as relative URLs, it would be very, very, very easy for a scraper to simply scrape your whole website and put it up on a new domain, and the whole website would just work. That sucks for you, and it's great for that scraper. But unless you are out there doing public services for scrapers, for some reason, that's probably not something that you want happening with your beautiful, hardworking, handcrafted website. That's one reason. There is a scraper risk.

2) Preventing duplicate content issues

But the other reason why it's very important to have absolute versus relative URLs is that it really mitigates the duplicate content risk that can be presented when you don't have all of these versions of your website resolving to one version. Google could potentially enter your site on any one of these four pages, which they're the same page to you. They're four different pages to Google. They're the same domain to you. They are four different domains to Google.

But they could enter your site, and if all of your URLs are relative, they can then crawl and index your entire domain using whatever format these are. Whereas if you have absolute links coded, even if Google enters your site on www. and that resolves, once they crawl to another page, that you've got coded without the www., all of that other internal link juice and all of the other pages on your website, Google is not going to assume that those live at the www. version. That really cuts down on different versions of each page of your website. If you have relative URLs throughout, you basically have four different websites if you haven't fixed this problem.

Again, it's not always a huge issue. Duplicate content, it's not ideal. However, Google has gotten pretty good at figuring out what the real version of your website is.

You do want to think about internal linking, when you're thinking about this. If you have basically four different versions of any URL that anybody could just copy and paste when they want to link to you or when they want to share something that you've built, you're diluting your internal links by four, which is not great. You basically would have to build four times as many links in order to get the same authority. So that's one reason.

3) Crawl Budget

The other reason why it's pretty important not to do is because of crawl budget. I'm going to point it out like this instead.

When we talk about crawl budget, basically what that is, is every time Google crawls your website, there is a finite depth that they will. There's a finite number of URLs that they will crawl and then they decide, "Okay, I'm done." That's based on a few different things. Your site authority is one of them. Your actual PageRank, not toolbar PageRank, but how good Google actually thinks your website is, is a big part of that. But also how complex your site is, how often it's updated, things like that are also going to contribute to how often and how deep Google is going to crawl your site.

It's important to remember when we think about crawl budget that, for Google, crawl budget cost actual dollars. One of Google's biggest expenditures as a company is the money and the bandwidth it takes to crawl and index the Web. All of that energy that's going into crawling and indexing the Web, that lives on servers. That bandwidth comes from servers, and that means that using bandwidth cost Google actual real dollars.

So Google is incentivized to crawl as efficiently as possible, because when they crawl inefficiently, it cost them money. If your site is not efficient to crawl, Google is going to save itself some money by crawling it less frequently and crawling to a fewer number of pages per crawl. That can mean that if you have a site that's updated frequently, your site may not be updating in the index as frequently as you're updating it. It may also mean that Google, while it's crawling and indexing, may be crawling and indexing a version of your website that isn't the version that you really want it to crawl and index.

So having four different versions of your website, all of which are completely crawlable to the last page, because you've got relative URLs and you haven't fixed this duplicate content problem, means that Google has to spend four times as much money in order to really crawl and understand your website. Over time they're going to do that less and less frequently, especially if you don't have a really high authority website. If you're a small website, if you're just starting out, if you've only got a medium number of inbound links, over time you're going to see your crawl rate and frequency impacted, and that's bad. We don't want that. We want Google to come back all the time, see all our pages. They're beautiful. Put them up in the index. Rank them well. That's what we want. So that's what we should do.

There are couple of ways to fix your relative versus absolute URLs problem

1) Fix what is happening on the server side of your website

You have to make sure that you are forcing all of these different versions of your domain to resolve to one version of your domain. For me, I'm pretty agnostic as to which version you pick. You should probably already have a pretty good idea of which version of your website is the real version, whether that's www, non-www, HTTPS, or HTTP. From my view, what's most important is that all four of these versions resolve to one version.

From an SEO standpoint, there is evidence to suggest and Google has certainly said that HTTPS is a little bit better than HTTP. From a URL length perspective, I like to not have the www. in there because it doesn't really do anything. It just makes your URLs four characters longer. If you don't know which one to pick, I would pick one this one HTTPS, no W's. But whichever one you pick, what's really most important is that all of them resolve to one version. You can do that on the server side, and that's usually pretty easy for your dev team to fix once you tell them that it needs to happen.

2) Fix your internal links

Great. So you fixed it on your server side. Now you need to fix your internal links, and you need to recode them for being relative to being absolute. This is something that your dev team is not going to want to do because it is time consuming and, from a web dev perspective, not that important. However, you should use resources like this Whiteboard Friday to explain to them, from an SEO perspective, both from the scraper risk and from a duplicate content standpoint, having those absolute URLs is a high priority and something that should get done.

You'll need to fix those, especially in your navigational elements. But once you've got your nav fixed, also pull out your database or run a Screaming Frog crawl or however you want to discover internal links that aren't part of your nav, and make sure you're updating those to be absolute as well.

Then you'll do some education with everybody who touches your website saying, "Hey, when you link internally, make sure you're using the absolute URL and make sure it's in our preferred format," because that's really going to give you the most bang for your buck per internal link. So do some education. Fix your internal links.

Sometimes your dev team going to say, "No, we can't do that. We're not going to recode the whole nav. It's not a good use of our time," and sometimes they are right. The dev team has more important things to do. That's okay.

3) Canonicalize it!

If you can't get your internal links fixed or if they're not going to get fixed anytime in the near future, a stopgap or a Band-Aid that you can kind of put on this problem is to canonicalize all of your pages. As you're changing your server to force all of these different versions of your domain to resolve to one, at the same time you should be implementing the canonical tag on all of the pages of your website to self-canonize. On every page, you have a canonical page tag saying, "This page right here that they were already on is the canonical version of this page. " Or if there's another page that's the canonical version, then obviously you point to that instead.

But having each page self-canonicalize will mitigate both the risk of duplicate content internally and some of the risk posed by scrappers, because when they scrape, if they are scraping your website and slapping it up somewhere else, those canonical tags will often stay in place, and that lets Google know this is not the real version of the website.

In conclusion, relative links, not as good. Absolute links, those are the way to go. Make sure that you're fixing these very common domain level duplicate content problems. If your dev team tries to tell you that they don't want to do this, just tell them I sent you. Thanks guys.


Video transcription by Speechpad.com


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!