How to Boost Your Site’s SEO

I’ve consulted with hundreds of business, and I’ve answered thousands of SEO questions.

When I sit down with a new client, I always ask about their goals.

Most of the time, I hear the same ones:

“I want to optimize my content for keywords.”

“I want to rank higher on Google.”

“I want to drive more traffic to my site.”

But there’s a much more fundamental need that every SEO needs to address.

Think about it. What is SEO?

It’s an obvious question but bear with me.

SEO is optimizing your site for search engines, right? That means it’s important to consider how search engines interact with your site.

This is a building block of SEO, but it doesn’t get as much air time as keywords or rankings even though it should.

There are many factors that affect how search engines treat your content.

Crawl budget is one of those factors.

If you understand crawl budget, you can use some techniques to make sure your site gets crawled effectively.

That means you’ll have better SERP results and better SEO overall.

Don’t worry––there’s nothing too technical here. It’s easy for anyone to pick up, and it can give you a serious SEO advantage.

What is crawl budget?

In the past few years, the term “crawl budget” made its rounds in SEO circles. Lots of people speculated about what exactly it meant, but there was never an official definition.

Those two parts are crawl rate limit and crawl demand.

Here’s how Google defines crawl rate limit:

Basically, Google doesn’t want to overload your website by crawling it too heavily.

Crawl rate limit stops Googlebot from making too many requests and slowing down your site.

If no one’s using your site, it’ll respond to Googlebot quickly, and so Googlebot will most likely crawl it more.

Got it? Awesome.

Now let’s look at the definition of crawl demand:

This has huge implications for SEO. Google is saying that popular content and fresh content has a higher crawl demand (and we’ll talk about this later on).

Google sums everything up to give a definition for crawl budget:

“Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl.”

Pretty simple, right?

Ideally, you want all of your pages to get crawled, and you want Googlebot to want to crawl your site.

So now it’s time to address the question that’s on your mind: How do you leverage crawl budget to get better SEO?

It all comes down to giving Google exactly what it wants.

Google has one mission: to provide the best searching experience for users.

If you can help them with that mission, you’ll be rewarded.

Let’s look at how exactly you can do that.

Improving site speed

Site speed is crucial for good SEO, and that’s true for crawl budget too.

It makes sense that if your site is faster and performs better overall, it’ll be able to handle more requests from Googlebot and human users at the same time.

Google notes this at the bottom of their blog post on crawl budget:

You can test your speed from four locations. I recommend using all of them to get an idea of how users from around the world are experiencing your site.

This may take anywhere from several seconds to a couple of minutes depending on the size of your site.

Once it’s finished, you’ll see a helpful summary:

ny longer than that and you run the risk of Google paying less attention to your site.

Google’s John Mueller has said that if a page on your site takes over 2 seconds to load, Google may crawl your site more slowly.

2 seconds is pretty fast, right? For Google, it’s not fast enough.

Your site should also have as few requests as possible.

Googlebot can make more requests and crawl your site faster if you have a low number of requests to start with.

So how few is good enough?

1. Declutter your design

This might be a tough one.

It’s 2017 and everybody’s trying to get the trendiest designs possible.

And if you have any unnecessary images, I suggest removing them entirely.

When you add new images, try to limit the file size if you can.

Finally, you should make some adjustments to your code if you are able to do so.

Excess code can weigh down a website like an elephant. It not only causes a poor user experience but also forces your site to take longer to respond.

If you’re a programmer or know a developer, it’s worth your time to improve your code.

You want to do two things: 1) Make JavaScript asynchronous and 2) combine as many CSS files as possible.

That may sound foreign to you, but these steps are just streamlining your code.

So remember, when it comes to design, you want a harmonious balance of fashion and function. Don’t let appearance make you forget about performance.

All of this will help you increase your crawl budget and allow Googlebot to send more requests. This will make sure your site is crawled more often.

2. Watch out for crawl errors

Here’s another tip from Google:

Crawl errors happen when Googlebot is crawling your site and runs into an issue.

Obviously, you want 0 crawl errors, but how do you monitor them?

You might not know that Google will actually inform you of any errors it encounters while crawling your site.

It’s a feature from one of my favorite tools, Google Search Console.

To get to the Crawl Errors section, open Search Console and select the property you want to analyze.

3. Clean up your low-value URLs

In that blog post, Google specifically calls out “low-value URLs.” These “can negatively affect a site’s crawling and indexing.”

It makes sense. Google wants high-quality content––the best of the best. Low-value URLs act as filler and by definition are not high quality whatsoever.

irst, make sure all of your pages are working correctly.

You want as few 404 pages (especially soft 404s) as possible, and you should only use redirects when you absolutely need to.

Second, avoid duplicate content.

You probably already know that Google will slap you on the wrist for duplicate content, but it can also negatively affect your crawl budget.

Third, avoid infinite spaces.

These happen when links go on and on. Google gives the example of a calendar with a “Next Month” link. Googlebot could theoretically follow those links forever.

As Google puts it, “crawling those URLs may use unnecessary bandwidth and could result in Googlebot failing to completely index the real content on your site.”

Increase your popularity

Every site wants to be popular. It’s part of all SEO strategies.

As it turns out, popularity is important to Google too.

Check out this little snippet from Google: So even though it’s not that obvious at first, a good content marketing strategy can increase your crawl budget.

You want people to be sharing your content like crazy. The more links to your site, the more Google will crawl you.

eep everything fresh

Another way to increase your crawl budget is to give Google what it wants: fresh content.

This goes hand-in-hand with popularity. The more fresh, comprehensive content you put out, the more people will share it and the more of a crawl budget you’ll get.

There are a few different ways of refreshing your site.

The first way is to create new content. I know that’s obvious, but I have to mention it because it’s the most effective way of keeping your site fresh.

Conclusion

Crawl budget isn’t talked about a lot but it’s at the core of SEO.

It makes you think about what your site is contributing to the Internet.

It’s also a brutally honest metric. You can’t take shortcuts to increase your crawl budget.

You have to create awesome content and let everyone know about it.

After all, Google has its users in mind. What do Google users want? The best and most relevant results.

By extension, Google wants to increase crawl budgets.

It wants users to find the best content, so if it sees sites doing things the right way, it will reward those sites.

On the other hand, if your site isn’t contributing much, then you’re going to have a hard time getting a bigger crawl budget.

At the end of the day, crawl budget is all about quality. And that’s what you should be focusing on.

So by all means, use these techniques to up your crawl budget. But never forget to bring something to the table.

How are you going to increase your crawl budget? What tips do you have to get a site crawled more?