Maybe it’s crossed your mind once or twice before: You know, this would be a lot easier if I just knew how to program. But it’s an intimidating subject, especially if you’re not sure of your technical expertise, and there’s so much to learn that it’s hard to know where to start.
In today’s Whiteboard Friday, master technical SEO Paul Shapiro shares why it’s so important for SEOs and marketers to take the programming plunge, explains key concepts, and helps you determine the best course of action for you to get started when it comes to leveling up your technical prowess.
Click on the whiteboard image above to open a high resolution version in a new tab!
Howdy, Moz fans. Paul Shapiro here, Head of SEO at Catalyst. I’m here to talk to you today about programming for SEOs and marketers.
Why should you learn how to program?
I think there are really several key benefits to learning how to program.
1. Improved developer relations
First, being developer relations. As SEOs, we’re constantly working with developers to implement our recommendations. Understanding why they make certain decisions, how they think is really pivotal to working with them better.
2. Become a better technical SEO
Understanding how to program makes you a better technical SEO. Just understanding the construction of websites and how they operate really helps you do a lot better with your SEO. Automation. As marketers, as SEOs, we all sometimes do very repetitive tasks, and being able to cut down on the time spent to do those repetitive tasks is really key.
It really opens up the opportunity to do things and focus more on strategy and the other things that you can’t leave to automation.
3. Leveling up your data analysis
If anyone is familiar with this number, 1,048,576, that’s the row limit in Microsoft Excel.
As marketers, we’re swimming in a sea of data. It’s very easy to work with a dataset that well exceeds that. I often work with hundreds of millions of rows of data. Utilizing a program language like R or Python is a really good way of handling that amount of data.
It’s becoming really, really more common in the States to be taught how to program in elementary school. So by learning how to program, you’re on equal footing with the children of the world, people that may enter the workplace in the future. So you don’t even have to learn how to program in depth. But I do recommend you at least understand the concepts and logic behind programming.
Which language should you learn?
Oftentimes I hear people say, “I did a little bit of programming in college or high school. I learned so-and-so language.” To them, I say, “You’re in great shape. Stick to whichever programming language you’re comfortable with.” You don’t have to start from square one.
A lot of the programming languages share a common logic. But if you are starting from square one and you need to just decide on which programming language I’m going to learn today, I have two recommendations.
If you’re going down the path of data analysis, your primary reason for learning how to program is to work with data and do more sophisticated things with data, then I think there’s no better language than Python.
Python is very well-equipped. There are lots of libraries designed specifically for data analysis, and it’s a very much more robust language than something like R.
Now I want to go through some basic programming concepts so that you walk away feeling a little bit more comfortable with the idea of learning a program so it’s a little less intimidating.
The first concept I want to go through is the idea of a variable. These are just like algebra, like basic algebra.
So you can assign x is equal to 2 or any other value, and then we can use that later. So x plus 2 is 4. Variables can have any name. We’re using Python syntax as an example. So the first variable we have is a variable called “animal,”and it’s equal to the value “cat.”
This is a string, which is just a bit of text that we assign to it. Now variables could be of many different types. So the variable “number” can be equal to 2, an integer. Or the variable “colors” can be a list, which is a type of Python array. Arrays are just variables with multiple values. So in this instance, colors is equal to red, blue, and green, and it’s just denoted with the brackets.
The next concept I’d like you to understand is conditions, so if/else being a basic condition that we would work with. It reads a lot like English. So if the variable “animal” is equal to “cat,” which it is, print out the text “MEOW!” If “animal” wasn’t equal to “cat,” say it was equal to “dog,”then we would print out “Woof!”
Then the output, since “animal” is equal to “cat,” is “MEOW!” Loops. There are many different types of loops. I’m going to use a for loop as an example. Again, it reads a little bit like the English language. So we have a variable “colors,”which we know is equal to red, blue, and green.
So we want to say for every value in that variable “colors,”print out that value. So for x in colors, print (x). It will go through each one, one at a time and print it out. So the first value is red. It gets printed out. The second value is blue. It gets printed out.
The last value is green. It gets printed out, and the code ceases. Now the last concept I want to explain is functions. Functions very simply are reusable snippets of code. So we have a very basic function here, which we define as moz, so the function moz, which has the value one line of code print (“WBF!”) for Whiteboard Friday.
If we execute the function moz, it will print out the value “WBF!” So all these concepts in themselves aren’t very useful. But when you start really programming and you start stringing them all together, you’re doing all sorts of sophisticated things, and it becomes very, very powerful building blocks to doing much greater things.
So now that you understand programming and why you should do it, I want to leave you with some resources to actually learn.
The first resource I recommend is Lynda. It got rebranded LinkedIn Learning. The reason why I recommend Lynda is because many, many public libraries offer you a subscription for free.
When I was learning to program originally, I actually went to the library and had to take out books and try to do it myself. Nowadays, there are tons of other resources, like Codecademy.
Python for Data Analysis
It also acts as an invaluable reference guide. If you’re interested in learning Python for data analysis, there’s one book that I highly recommend. It is “Python for Data Analysis” by McKinney. That’s an O’Reilly book. McKinney was the creator of Pandas, which is a very well used Python library for data analysis. So hopefully you’ve walked away a little less scared of programming and are excited to learn.
Leave your comments in the section below. Thanks for watching. Till next time.
Whether you consult with teams within your company or with outside clients, the chances are fairly high that at least once, you’ve left a meeting frustrated by the actions of others, even asking yourself: “why would they do that?”
It’s easy to walk into a project thinking of it as a simple matter of “they brought me in to fix a problem.” But the reality is rarely so simple. Consulting with other teams always entails organizational and emotional nuance that you may not be privy to.
Every interpersonal relationship is unique, and hopefully the circumstances I’m discussing won’t apply to many engagements or projects you take part in. However, when you do end up in a difficult consulting situation, it’s helpful to have a bit of empathy for those you’re working with.
I’ve found that remembering these 3 points can help me put myself in the shoes of my point of contact and interact with them in a way that is sensitive to what they may be dealing within their environment.
1. Your point of contact may not have asked for your help
It is entirely possible that the person you are trying to help may not want to be helped.
Management has its own ideas sometimes and internal communication isn’t always perfect at any company. This can lead to situations where your point of contact may feel defensive, especially if their job functions seem like they might cover what you are consulting on. The best intentions of a manager who wants to help by bringing in more resources may look like distrust or undermining to the employee who didn’t get a say.
At one point during my stint as an in-house SEO, I actually found myself in this exact position. Leadership brought in an outside agency to help with SEO during a domain migration, and while their intentions may have been to provide more help, they didn’t effectively communicate that to me.
As a result, since I was the one who was responsible for that area, it made me feel insecure about how management viewed me and my skills. I was lucky enough to work with a great consultant who was able to support me and help move forward the many projects that were already in-flight. But because I initially felt like they were undermining my credibility by being involved in the first place, it took a while to build that trust and be able to get things done effectively.
The best way to deal with this potential issue is to ensure that you respect the context and institutional knowledge that the team you are helping possesses. Work to have a collaborative relationship instead of an authoritative one. The more context and communication you have, the better the recommendations you can contribute.
2. If they did ask for help, they may be feeling vulnerable or insecure
Step back for a second and think about why a team might bring in an outside consultant, to begin with. There are tons of specific issues they could need assistance with, but all of this boils down to a problem that they presumably want or need help to solve — a problem that they couldn’t solve on their own. Regardless of whether they couldn’t solve it because of knowledge, resources, or even office politics, your contributions add something that they couldn’t contribute themselves — and that can be hard to deal with.
This isn’t something that needs to be discussed with the client or another team, but it is something that you should acknowledge and keep front-of-mind when you communicate with them. Respect the vulnerability of seeking out help, and appreciate the trust that they have placed in you.
3. Your client is accountable for the results of their project
When planning a long-term strategy, making tactical recommendations, or accessing the results of a marketing campaign that you helped execute, it’s easy to feel invested or accountable for the results of a project. However, it’s important to remember that your point of contact is usually far more accountable for results than you are. Their job, success, and emotions are all on the line much more than yours.
As an outside subject matter expert, your job is to give them all the information and resources to make the best decision. At the end of the day, the choice is theirs. I know how hard it can be to see your recommendations or projects rejected, but it’s important to try not to take it personally if they, having all the facts, make what they believe to be the best decision.
If they seem like they are questioning everything you say, maybe it’s because they want to be 100 percent sure it’s the best approach. Perhaps their micromanaging comes from a place of good intentions — just wanting to follow through and get the best outcome with every aspect of a project. Even what can come off as argumentative or difficult could be them playing devils advocate to ensure that everything has been considered.
All this being said, perhaps none of these circumstances apply to the client that you are finding it hard to work with. People can have bad days, hard years, or even just generally prickly dispositions. But more empathy and compassion in the world is never a bad thing. So, I would encourage anyone who works with other teams to avoid the impulse to judge a harsh response, and instead, consider what may be behind it.
Have you ever been faced with a complicated consulting situation? Share what helped you navigate it in the comments below!
Search can’t live in a silo. If you want to see success, cross-collaboration across your organic, content, and paid teams is absolutely key. But that takes a huge amount of effort, from untangling communication to cross-training to getting buy-in from everyone involved. What’s a search marketer to do?
If you missed her talk this year at MozCon 2019, here’s your chance to make up for it! In today’s edition of Whiteboard Friday, Heather Physioc shares her framework for successfully integrating your organic, paid, and content practices for a smoother search experience.
Click on the whiteboard image above to open a high resolution version in a new tab!
Hey, everybody, and welcome back to Whiteboard Friday. My name is Heather Physioc, and I’m Group Director of Discoverability at VMLY&R. Today we’re going to talk about nine tips to help integrate your organic search, paid search, and content practices.
1. Announce change all at once, but roll out changes one at a time
So your first tip is that you want to announce change all at once, but then you want to roll out the changes one at a time.
It can be overwhelming to integrate practices and change processes. So you don’t want to try to do everything all at the same time. It’s like trying to boil the ocean, and it’s too much to stick. So while you want to get everybody on board and aligned to the benefits and challenges they’ll be facing as you integrate, then you just progressively roll out the changes iteratively over time.
2. Document new products & processes
Next, as you develop new capabilities and processes and offerings together, you’re going to document those processes in a shared, living wiki, because those processes are going to continue to change.
So my team uses Confluence, where we document our shared workflows, but everybody on the team has access and total trust to continue refining those in the ways that they see are best for the team.
3. Make recommendations and report together
Your next step should be obvious, but a lot of people are not doing it. You should be making recommendations and reporting together. So a lot of times we’ll collect all our data for reporting from all our different channels.
We’ll smash some slides together at the last minute before we throw it over the fence to the client. It ends up with a pretty shallow, almost meaningless set of data that doesn’t tell a story. So we should be getting together, sharing our insights, observations, and findings in the room together to find the story that is the most meaningful and help prioritize for our clients the best marketing decisions they can make from that data.
4. Cross-train to build advocacy across teams
So your next tip is to cross-train so you can build advocacy across the teams. We host a lot of workshops and hands-on training. We’ve even done job swaps where we had SEOs writing performance content for product detail pages. It creates this wonderful sense of empathy and understanding for what others need in order to do their jobs well.
But it also creates these great mental checks where you ask yourself, “Am I including the right people at the right times? Is there anyone else who could add value here? Could my work be impacting someone else?” So the purpose here is not necessarily to know how to do each other’s jobs so much as it is to empower people to be able to advocate for, speak about, and cross-sell your other teams.
5. Reintroduce the team or capability
Next, when you’ve done your integration of processes and people, everyone else in the organization may not necessarily know what that means for them. So you’ll want to reintroduce your team or your new capability to the rest of the organization. Put faces with names.
Talk about what the new capability is and does and the value it brings to the organization. Tell people how to engage with that new offering and what it means for their project or initiative or client.
6. Market the joint wins
Up next, we’re going to market the joint wins. As you’re continuously integrating, you should always be looking for wins or warnings that you can share with others so they can learn how to better engage with your offerings.
So if you have a great case study, where you integrated paid and organic or organic and content, make sure you’re marketing those stories out to your colleagues, your clients, your bosses, and of course your team.
7. Hold roundtables to deep-dive search opportunities
Up next, we’re going to do roundtables so we can deep dive search opportunities with other departments. So of course it makes sense to have roundtables between organic search and paid search or organic search and performance content, but also think beyond your immediate team.
Think about other marketing teams, like social media and pairing search behavior insights with social listening data. Or think about geographic teams. What if you sat your organic search team down with your Europe group to figure out what opportunities make the most sense for that region? Or even sales and IT and finding those areas of intersection, where you can do great search work that supports more parts of the organization.
8. Host mutual lunch & learns to cross-pollinate
Next, think about hosting mutual lunch and learns so you can start to cross-pollinate different skill sets. So similar to the roundtables, this is where you’re going to bring different groups together to talk about capabilities. But think about more than just presenting your capabilities to other people. Also be sure to invite them to present their capabilities to your group. For example, we’ve invited the project management team or the client engagement team to make us stronger in our search work through the value that they bring.
9. Give ownership of change to others
And finally, as you’re making all of these changes, it can’t just come from the top, one person just handing change down for everyone else to implement. It has to be organic, pardon the pun, and everybody should have ownership over the direction that we’re heading together. So when we make changes to products or processes or we start to integrate different groups or spin up little teams to work on specific objectives, we make sure that those individuals from each side have ownership to make those decisions together and roll it out to the rest of the group.
It helps make sure we’ve considered all the angles and greatly impacts our ability to get buy-in across the team. So those are nine quick tips to integrate organic search, paid search, and content practices. Let us know what you think in the comments below. I want to hear your tips too, and we’ll see you next time on Whiteboard Friday.
Take a moment to think about how you’ve used the internet today. Which posts made you stop scrolling through your Instagram feed? What webpages did you spend the most time on? What content did you enjoy?
If you’re like most of us, there’s a good chance that videos played a factor in your answers to all three of those questions. So it’s no surprise that marketing experts have been encouraging brands to use more video for years now.
Despite all this hype, many small to medium brands still use very few videos in their marketing, if they use any at all. In our experiences with clients, we’ve seen companies struggle in four major areas:
Talent. Most small marketing teams don’t have people experienced with creating or starring in videos.
Buy-in. Companies that use video well have many people across the organization committed to leveraging video content, including leadership, salespeople, customer service representatives, and subject-matter experts. But achieving this level of buy-in across an organization can be very difficult.
Consistency. While many brands have figured out how to produce frequent and consistent written content, few have figured out how to do so with video.
With challenges this significant, it’s no wonder why most brands still don’t include video as a major component of their marketing efforts. The good news is that these barriers to entry give you an opportunity to beat your competitors to the punch – but that window is closing quickly.
To help our clients reap everything video marketing has to offer, we’ve put together a framework that makes building a video marketing strategy much easier to approach and manage.
The ABCs of video content
The first thing that must happen before a marketing team can successfully use video in today’s world is usually a mental shift: If your marketing team or company leadership thinks about producing video the same way they did a decade ago, it will be very difficult to create enough video content to truly make a difference because you’ll constantly face the challenges mentioned above.
The greatest thing about creating video content today is that it doesn’t always have to be this large-scale production. The days of spending thousands of dollars and weeks of time on every video you create are officially over.
We believe that every brand needs a strong mix of video content across three levels: A-level, B-level, and C-level.
A-level video content
A-level videos are the videos that most brands are already used to creating. These videos are polished and well-produced, and therefore the most expensive to create. If your company has ever created a television commercial or a brand overview video for your website, it was probably an A-level video.
A-level videos work best when you need to create a strong impression on the viewer. If it’s the first time someone is interacting with your brand or another situation when you need to convey that your company is professional and credible, an A-level video will likely work best. This is what makes them great for commercials, product videos, and company overviews.
Don’t use A-level videos when your primary goal is to convey authenticity or build a relationship with your viewer. A-level videos also aren’t cost-effective for most brands to use as consistent, regular video content to support your social media, video SEO, email communications, or blog.
A brand will need much fewer A-level videos than B or C. As such, the key to getting the most from your investment in A-level video content is repurposing. You should always consider how you can use clips or footage from your A-level content for things like social media posts, presentations, across your website, or as a quick way to add a little polish to a B or C-level video.
To create A-level videos, most brands will need to work with a third party video company or marketing agency. These videos will be scripted, shot with high-end cameras by people who really know how to use them, will often feature paid on-camera talent, and will be professionally edited.
When a brand produces high volumes of video content, the majority of it is usually B-level. These videos are planned, but not perfect. Most of the how-to videos and vlogs you watch would fit into this category.
B-level videos work great when your goal is to build a relationship with your video viewers because they allow you to show more authenticity than A-level videos, and their lower cost makes them perfect for a consistent video strategy. This level works well for educational content, social media videos, a video series, team or personal intro videos, simple product demos, and video testimonials.
The best thing about B-level videos is that they can often be created by your own staff. Even if you decide to outsource them, they will be much cheaper than A-level videos because you can produce multiple videos at the same time or engage a third-party for just one part of the production process, such as editing.
To create a B-level video, all you’ll need is a basic script outline (bullet points work great), someone on your staff willing to get on camera, some basic video equipment, and an entry-level video editing program. If you don’t have people on your staff who are comfortable shooting video from a smartphone and editing it together, we recommend seeking training or considering hiring a student or recent graduate with those skills.
The keys to success with B-level videos are authenticity, volume, and consistency.
Authenticity. Brands that achieve success with these videos aren’t worried about memorizing lines and being perfect on camera. You’d be amazed at how much the occasional “um” will make you sound more human and help you connect with your audience. With that said, you do want to feature someone who is comfortable on camera, even if it takes them some practice to get there.
Volume. As long as you’re creating valuable content, the more videos you have, the better.
Consistency. Just like with a blog, magazine, or newsletter, publishing videos at a consistent interval allows you to more easily build an engaged audience of return viewers.
Examples of B-level video include this one from Solo, Simple Strat (hi!), and Moz, below:
C-level video content
This is the level where brands can really connect with their audience and stand out from the competition. C-level videos are raw, unpolished, and extremely effective in humanizing your brand and the team behind it.
To create a C-level video, all you need to do is pull out your cell phone or turn on your webcam, press record, and start talking. You may want to prepare a few quick bullet points of what you’re going to talk about, but even that is often unnecessary. These videos are great for sharing lessons on-location from industry events, making key employees shine on their social media channels, helping your salespeople stand out in their prospects’ email inboxes, and adding a personal touch to your customer service communications.
The most powerful aspects of C-level videos are that they can be personalized for individual people and they can help you get information out in almost realtime. You can use them to pack an extra punch in any email you send or to share lessons as you learn them or think about them — which is often when you’re most passionate about them and before the competition has a chance to talk about them.
C-level videos still require good lighting and audio quality, so we do recommend purchasing a cheap portable light and lapel microphone, but you can easily get everything you need for under $50. These videos don’t require anyone else to shoot them, and you often don’t need to do any editing beyond occasionally cutting out or combining a couple of video clips.
Now that you know the different levels of video content you’ll need, it’s time to put together your plan. Thinking about these levels as you begin to determine your video topics and schedules will make it easier to determine the resources you’ll need, your ideal number and frequency of videos, and how each video will fit into your larger marketing strategy and goals. Just remember your ABCs and get ready to experience the difference that video will make for your brand.
Are you currently working on a video marketing strategy for the year? What have you found useful (or not) so far?
Editor’s note: This post first appeared in April of 2017, but because SEO (and Google) changes so quickly, we figured it was time for a refresh!
Meta tags represent the beginning of most SEO training, for better or for worse. I contemplated exactly how to introduce this topic because we always hear about the bad side of meta tags — namely, the keywords meta tag. One of the first things dissected in any site review is the misuse of meta tags, mainly because they’re at the top of every page in the header and are therefore the first thing seen. But we don’t want to get too negative; meta tags are some of the best tools in a search marketer’s repertoire.
There are meta tags beyond just description and keywords, though those two are picked on the most. I’ve broken down the most-used (in my experience) by the good, the bad, and the indifferent. You’ll notice that the list gets longer as we get to the bad ones. I didn’t get to cover all of the meta tags possible to add, but there’s a comprehensive meta tag resource you should check out if you’re interested in everything that’s out there.
It’s important to note that in 2019, you meta tags still matter, but not all of them can help you. It’s my experience, and I think anyone in SEO would agree, that if you want to rank high in search, your meta tags need to accompany high-quality content that focuses on user satisfaction.
My main piece of advice: stick to the core minimum. Don’t add meta tags you don’t need — they just take up code space. The less code you have, the better. Think of your page code as a set of step-by-step directions to get somewhere, but for a browser. Extraneous meta tags are the annoying “Go straight for 200 feet” line items in driving directions that simply tell you to stay on the same road you’re already on!
The good meta tags
These are the meta tags that should be on every page, no matter what. Notice that this is a small list; these are the only ones that are required, so if you can work with just these, please do.
Meta content type – This tag is necessary to declare your character set for the page and should be present on every page. Leaving this out could impact how your page renders in the browser. A few options are listed below, but your web designer should know what’s best for your site.
Meta description – The infamous meta description tag is used for one major purpose: to describe the page to searchers as they read through the SERPs. This tag doesn’t influence ranking, but it’s very important regardless. It’s the ad copy that will determine if users click on your result. Keep it within 160 characters, and write it to catch the user’s attention. Sell the page — get them to click on the result. Here’s a great article on meta descriptions that goes into more detail.
Different sites will need to use these in specific circumstances, but if you can go without, please do.
Social meta tags – I’m leaving these out. OpenGraph and Twitter data are important to sharing but are not required per se.
Robots – One huge misconception is that you have to have a robots meta tag. Let’s make this clear: In terms of indexing and link following, if you don’t specify a meta robots tag, they read that as index,follow. It’s only if you want to change one of those two commands that you need to add meta robots. Therefore, if you want to noindex but follow the links on the page, you would add the following tag with only the noindex, as the follow is implied. Only change what you want to be different from the norm.
<meta name="robots" content="noindex" />
Specific bots (Googlebot) – These tags are used to give a specific bot instructions like noodp (forcing them not to use your DMOZ listing information, RIP) and noydir (same, but instead the Yahoo Directory listing information). Generally, the search engines are really good at this kind of thing on their own, but if you think you need it, feel free. There have been some cases I’ve seen where it’s necessary, but if you must, consider using the overall robots tag listed above.
Language – The only reason to use this tag is if you’re moving internationally and need to declare the main language used on the page. Check out this meta languages resource for a full list of languages you can declare.
Keywords – Yes, I put this on the “indifferent” list. While no good SEO is going to recommend spending any time on this tag, there’s some very small possibility it could help you somewhere. Please leave it out if you’re building a site, but if it’s automated, there’s no reason to remove it.
Refresh – This is the poor man’s redirect and should not be used, if at all possible. You should always use a server-side 301 redirect. I know that sometimes things need to happen now, but Google is NOT a fan.
Site verification – Your site is verified with Google and Bing, right? Who has the verification meta tags on their homepage? These are sometimes necessary because you can’t get the other forms of site verification loaded, but if at all possible try to verify another way. Google allows you to verify by DNS, external file, or by linking your Google Analytics account. Bing still only allows by XML file or meta tag, so go with the file if you can.
The bad meta tags
Nothing bad will happen to your site if you use these — let me just make that clear. They’re a waste of space though; even Google says so (and that was 12 years ago now!). If you’re ready and willing, it might be time for some spring cleaning of your <head> area.
Author/web author – This tag is used to name the author of the page. It’s just not necessary on the page.
Revisit after – This meta tag is a command to the robots to return to a page after a specific period of time. It’s not followed by any major search engine.
Rating – This tag is used to denote the maturity rating of content. I wrote a post about how to tag a page with adult images using a very confusing system that has since been updated (see the post’s comments). It seems as if the best way to note bad images is to place them on a separate directory from other images on your site and alert Google.
Expiration/date – “Expiration” is used to note when the page expires, and “date” is the date the page was made. Are any of your pages going to expire? Just remove them if they are (but please don’t keep updating content, even contests — make it an annual contest instead!). And for “date,” make an XML sitemap and keep it up to date. It’s much more useful.
Copyright – That Google article debates this with me a bit, but look at the footer of your site. I would guess it says “Copyright 20xx” in some form. Why say it twice?
Abstract – This tag is sometimes used to place an abstract of the content and used mainly by educational pursuits.
Distribution – The “distribution” value is supposedly used to control who can access the document, typically set to “global.” It’s inherently implied that if the page is open (not password-protected, like on an intranet) that it’s meant for the world. Go with it, and leave the tag off the page.
Generator – This is used to note what program created the page. Like “author,” it’s useless.
Cache-control – This tag is set in hopes of controlling when and how often a page is cached in the browser. It’s best to do this in the HTTP header.
Resource type – This is used to name the type of resource the page is, like “document.” Save yourself time, as the DTD declaration does it for you.
There are so many meta tags out there, I’d love to hear about any you think need to be added or even removed! Shout out in the comments with suggestions or questions.
I’m always fascinated with the marketing budgets of enterprise-level companies that are ready to pay astronomical sums to contractors. A recent shmooze in the community was thanks to Hertz that paid 32M to Accentura agency, which (so far) hasn’t resulted in any substantial changes to their site.
Though I personally don’t work with client’s who throw around millions of dollars, that doesn’t affect the quality of services that I provide. My average client wants to get the maximum by spending as little as possible. It might sound like a tough job for me and indeed it is, but I love the challenges that a small budget brings, as it helps me stay creative and reach new professional heights.
So while the budget isn’t a challenge, changing my client’s mindset is, and that’s because all of my clients are victims of one of the biggest misconceptions about content marketing: They think that once they start publishing content pieces regularly, inbound traffic will hit their site like a meteorite.
And it’s not just the traffic — links are a subject to a similar misconception. Each time I share studies like the one by Brian Dean that clearly shows that links don’t come on its own, there’s always someone that’s going to say: “That’s because their content’s just not good enough.” When I have a call with clients that ask for quality content with zero focus on links.
The bottom line is, traffic and links don’t just show up out of thin air. Regardless of how good your content is, how well structured and valuable it may seem, it has nearly zero chances of getting attention in today’s overcrowded digital space.
In this post, I want to share with you five bulletproof tactics that help me boost content linkability without having a big fat budget to waste.
A note on content and modern-day link building
Before we dive into the best ways to boost your content without breaking the bank, it’s important to touch on what link-building is today. Links are a digital marketing currency — which you need to earn and spend wisely. And to earn them, you need to build relationships.
A while ago, I noticed a shift in a client’s mindset: After a few projects delivered together, they started to ask for in-depth forms of content like how-to’s, case studies, and guides — which (according to Brian’s research) is exactly the type of content that has the highest chances of getting links. But that’s not necessarily the number one reason why people allocate links.
Links are inherently relationships. And if you agree that linking to a strategic partner brings more benefits compared to referring to a random stranger, then you’ll find appreciate Robbie Richards methods.
Robbie’s roundups are a textbook definition of highly linkable content. A post about the best keyword research tools published not that long ago on his blog attracted nearly 300 referring domains and a decent organic traffic share:
What’s his secret?
Robbie made sure to target the experts within his business circle. In a nutshell, his roundup posts work as part of a well-delivered outreach strategy that has a strong focus on gaining links by leveraging existing relationships. This is the key to modern-day link-building — a combination of content, links, and partnerships.
Without further ado, let’s talk the best ways to promote content that doesn’t involve any where-do-I-get-the-money-for-it drama.
5 bulletproof ways to blow up your content without breaking the bank
If you’re creating quality content with zero focus on links, you won’t be getting optimal traffic. The only chance to make your content stand out is to focus on its potential linkability even before you actually start writing it. Here are some of favorite ways to get your content seen.
1. Adding expert quotes
Quoting an expert is one of my favorite ways to boost content linkability and shareability. It’s quick, easy, and doesn’t require a significant time investment. When you write out of your expertise area, adding a quote of a thought-leader grants your content more credibility and value, not to mention boosting its linking potential.
Depending on how influential your company is, you can either select an existing quote or reach out to the experts and ask for a new one.
Here’s a tip: If you decide to go with a pre-existing quote, contact the expert in advance to confirm it. This way, you can make sure that they still stand by that opinion, plus, they’re okay with you quoting them.
Remember, while quoting experts is a good idea, you also need to find the right expert andthe right quote. Here’s how to do that:
If your brand has a big audience, I recommend starting by checking your current followers and subscribers across various channels, including social media. You might not know it, but there’s a good chance you’ll find real influencers among people who follow your brand’s pages. To speed up the process of spotting influencers among your Twitter followers, you can use Followerwonk. This tool allows you to export all your followers to a list and sort them by the size of their audience.
Another way is to analyze the websites that link back to your site. To do that, you can use Moz Link Explorer that will show the list of URLs that are referring to your site. Chances are, some of those authors are pretty influential in their niche.
Finally, you could use BuzzSumo to find relevant influencers to contact. For example, you could export a list of bloggers who are contributing to the industry-leading blogs.
The last option is less suitable for link building purposes, as the influencers that you find have no idea of your business existence and are hard to get on board. However, it’s not impossible. Before getting in touch, make sure to scratch their backs: Share their content on your social media, sign up for their newsletter, etc. To find the influencer’s most recent pieces, search on BuzzSumo Content Analyzer by “Author: [INSERT NAME].” This helps build a bridge and create the right first impression.
Don’t forget that expert quotes need to be allocated in content with special formatting which means you need to involve a designer/developer.
Here are a few examples that I personally find quite visually appealing:
And another one:
2. Strategically linking back to blogs that you’re interested in
Strategic link building is like playing poker while blindfolded. A strategic approach always pays off in the long run in almost any area, but when applied to link building, it depends on how well you can spot linking opportunities. Based on this, your chances of acquiring links are either very high or very low.
If you want industry leaders to link back to your content someday, you have to prove that your content deserves their attention. The best way to get your foot in the door is to link back to them.
You need to find the right experts to link back to. How do you do that?
The mechanic behind finding the right sites to refer to is similar to the one that I shared in a section about expert quotes. However, there’re a few more strategies that I want to add:
Are you a part of any industry groups on Facebook? If so, go and check the members of those groups and find people that are also involved in link building. Now, you have a legit reason to contact them (since you’re both a part of one group on Facebook/LinkedIn) and ask whether they’re interested in getting a link in your upcoming post. Please note, that you shouldn’t skip this step, as by this you’re making them aware that you’re expecting for the favor to be returned.
Have you ever participated in any roundups? If yes, then reach out to the experts that were also featured in this post.
Finally, check your current blog subscribers, clients, and partners. The chances that they’re also interested in partnering up on a link building side are quite high.
3. Adding good images/GIFs and hiring a designer for professional-looking visuals
In 2019, using stock images in your content is a big no. After all, they are easily recognizable for their abstract nature and give away the fact that the author didn’t invest much into creating custom visuals.
However, there is a way to adapt it to your unique brand style and still make it work. And to do it, you don’t even need to hire a designer right away.
The drag-&-drop tools like Vengagge, Canva, or Visme make it easy to create pretty nice graphics. For example, Canva has a lot of great grids and predefined templates, which makes the whole design process really fast.
What you need to do is take a good-looking cover image, for example, like the ones we use in our blog, and cheer it up with custom-made designs in Canva. You can add your picture, your brand’s logo, or anything else your heart desires. Such an approach allows us to maintain our own unique style while staying within the budget.
Static images are not the only way to pretty up your content. One of my favorite visual elements is GIFs. They are perfect for visualizing step-by-steps and how-tos and can easily demonstrate how to perform something in a digital tool. You can even use them to tell a story. At one of my recent presentations, I used a GIF to explain why simply posting on Twitter is not enough to get attention to a brand.
I saw many posts that were able to acquire loads of links and social shares thanks to good graphics, for instance, this post that featured the SEO experts in Halloween costumes.
Without a doubt, this requires a little bit of a budget, but I’d say it’s 100 percent worth it because it’s creating value. The last time our company did something like this for a client, we hired a designer who charged us $30 USD for one image. It’s not too bad since custom-made images make it way easier to pitch your posts to other blogs to get more links!
Hint: When you’re looking for custom graphics that won’t make your wallet cry, you can always find freelancers on sites like Upwork or on freelancing Facebook groups.
4. Delivering email outreach by targeting the “low hanging fruits”
We’ve done a lot of email outreach campaigns here at Digital Olympus, and so, I’ve noticed that we have a fast turnaround rate when our outreach targets are in the “right state of mind,” meaning they’re interested in cooperating with us.
There are many reasons why they might show interest. For example, perhaps they’ve recently published a piece and are now invested in promoting it. To spot content marketers and authors like these, you can use Pitchbox. Pitchbox lets you create a list of posts that were published within the last 24 hours based on the keywords of your choice.
The biggest bonus of Pitchbox is that it not only pulls together a list of content pages but it also provides contact details. In addition to this, Pitchbox automates the whole outreach process.
Another tool that can pull together a list of posts published within the last 24 hours is Buzzsumo. Here’s a great piece by Sujan Patel that shows how to deliver outreach the right way.
There can be many speculations about which email outreach techniques work and which don’t, but the truth remains: It’s a very hard time-consuming job that requires lots of skill and practice. In one of my recent posts, I write about proven email outreach techniques and how to master them.
5. Adding stats that don’t involve a huge time investment
You’ve heard that a picture is worth a thousand words. How about this: A number knocks out 10 thousand words. By adding statistics to your piece, you can simply mark out the whole process of having to refer to another page.
But fresh, relevant stats don’t grow on trees. You need to know where you can find them.
The easiest and the cost-efficient way of adding numbers to your piece is by running Twitter polls. They can collect up to 1k results for only $100 USD of properly paid promotion efforts. The biggest plus of running polls on Twitter is that you can create a specific list of people (aka a tailored audience) that will see your ad. For a detailed explanation on how to work with tailored audiences, I recommend checking this post.
Besides running Twitter polls, you can use survey tools that will help you collect answers for a fee:
Survata will show your survey across their online publisher’s network with the average cost per answer starting from 1 USD;
Surveymonkey market research module starts from $1.25 for 200 complete responses. As you can see from a screenshot below, it allows you to set up a more laser-targeted group by selecting a particular industry.
Another quick hack that I use from time to time is comparing already existing data sets to reveal new insights. Statista is a great site for getting data on any topic. For instance, on one graph you can show the revenue growth on the major SMM platforms as well as the growth of their audience. Plus, don’t forget that while the numbers are good, the story is key. Statistics tend to be dry without a proper story that they are wrapped in. For inspiration, you can use this great post that shares many stories that were built on numbers.
It doesn’t always have to be serious. Numbers draw more attention than written copy, so you can create a fun poll, for example, whether your followers are more into dogs or cats.
Creating captivating content is hard work and often a hella lot of money, but there are ways to spare a few bucks here and there. By utilizing the strategies that I shared, you can make sure that your content gets the audience it needs without time waste, huge costs, and stress. The amount of backend work you put into research and advertising is what makes your audience not only scroll through your content but actually read it. This is what will differentiate your piece from millions of similar ones.
Create a strategy and go for it! Whether it’s polling, graphics, emails, quotes, or backlinks, make a game plan that will promote your content the right way. Then your site will rock.
Do you have any other tips or suggestions? Tell me below in the comments!
Log File Analysis should be a part of every SEO pro’s tool belt, but most SEOs have never conducted one. Which means most SEOs are missing out on unique and invaluable insights that regular crawling tools just can’t produce.
Let’s demystify Log File Analysis so it’s not so intimidating. If you’re interested in the wonderful world of log files and what they can bring to your site audits, this guide is definitely for you.
What are Log Files?
Log Files are files containing detailed logs on who and what is making requests to your website server. Every time a bot makes a request to your site, data (such as the time, date IP address, user agent, etc.) is stored in this log. This valuable data allows any SEO to find out what Googlebot and other crawlers are doing on your site. Unlike regular crawlings, such as with the Screaming Frog SEO Spider, this is real-world data — not an estimation of how your site is being crawled. It is an exact overview of how your site is being crawled.
Having this accurate data can help you identify areas of crawl budget waste, easily find access errors, understand how your SEO efforts are affecting crawling and much, much more. The best part is that, in most cases, you can do this with simple spreadsheet software.
In this guide, we will be focussing on Excel to perform Log File Analysis, but I’ll also discuss other tools such as Screaming Frog’s less well-known Log File Analyser which can just make the job a bit easier and faster by helping you manage larger data sets.
Note: owning any software other than Excel is not a requirement to follow this guide or get your hands dirty with Log Files.
How to Open Log Files
Rename .log to .csv
When you get a log file with a .log extension, it is really as easy as renaming the file extension .csv and opening the file in spreadsheet software. Remember to set your operating system to show file extensions if you want to edit these.
How to open split log files
Log files can come in either one big log or multiple files, depending on the server configuration of your site. Some servers will use server load balancing to distribute traffic across a pool or farm of servers, causing log files to be split up. The good news is that it’s really easy to combine, and you can use one of these three methods to combine them and then open them as normal:
Use the command line in Windows by Shift + right-clicking in the folder containing your log files and selecting “Run Powershell from here”
Then run the following command:
copy *.log mylogfiles.csv
You can now open mylogfile.csv and it will contain all your log data.
Or if you are a Mac user, first use the cd command to go to the directory of your log files:
Then, use the cat or concatenate command to join up your files:
cat *.log > mylogfiles.csv
2) Using the free tool, Log File Merge, combine all the log files and then edit the file extension to .csv and open as normal.
3) Open the log files with the Screaming Frog Log File Analyser, which is as simple as dragging and dropping the log files:
(Please note: This step isn’t required if you are using Screaming Frog’s Log File Analyser)
Once you have your log file open, you’re going to need to split the cumbersome text in each cell into columns for easier sorting later.
Excel’s Text to Column function comes in handy here, and is as easy as selecting all the filled cells (Ctrl / Cmd + A) and going to Excel > Data > Text to Columns and selecting the “Delimited” option, and the delimiter being a Space character.
Once you’ve separated this out, you may also want to sort by time and date — you can do so in the Time and Date stamp column, commonly separating the data with the “:” colon delimiter.
Your file should look similar to the one below:
As mentioned before, don’t worry if your log file doesn’t look exactly the same — different log files have different formats. As long as you have the basic data there (time and date, URL, user-agent, etc.) you’re good to go!
Understanding Log Files
Now that your log files are ready for analysis, we can dive in and start to understand our data. There are many formats that log files can take with multiple different data points, but they generally include the following:
Date and time
Server request method (e.g. GET / POST)
HTTP status code
More details on the common formats can be found below if you’re interested in the nitty gritty details:
Apache and NGINX
Amazon Elastic Load Balancing
How to quickly reveal crawl budget waste
As a quick recap, Crawl Budget is the number of pages a search engine crawls upon every visit of your site. Numerous factors affect crawl budget, including link equity or domain authority, site speed, and more. With Log File Analysis, we will be able to see what sort of crawl budget your website has and where there are problems causing crawl budget to be wasted.
Ideally, we want to give crawlers the most efficient crawling experience possible. Crawling shouldn’t be wasted on low-value pages and URLs, and priority pages (product pages for example) shouldn’t have slower indexation and crawl rates because a website has so many dead weight pages. The name of the game is crawl budget conservation, and with good crawl budget conversion comes better organic search performance.
See crawled URLs by user agent
Seeing how frequently URLs of the site are being crawled can quickly reveal where search engines are putting their time into crawling.
If you’re interested in seeing the behavior of a single user agent, this is easy as filtering out the relevant column in excel. In this case, with a WC3 format log file, I’m filtering the cs(User-Agent) column by Googlebot:
And then filtering the URI column to show the number of times Googlebot crawled the home page of this example site:
This is a fast way of seeing if there are any problem areas by URI stem for a singular user-agent. You can take this a step further by looking at the filtering options for the URI stem column, which in this case is cs-uri-stem:
From this basic menu, we can see what URLs, including resource files, are being crawled to quickly identify any problem URLs (parameterized URLs that shouldn’t be being crawled for example).
You can also do broader analyses with Pivot tables. To get the number of times a particular user agent has crawled a specific URL, select the whole table (Ctrl/cmd + A), go to Insert > Pivot Table and then use the following options:
All we’re doing is filtering by User Agent, with the URL stems as rows, and then counting the number of times each User-agent occurs.
With my example log file, I got the following:
Then, to filter by specific User-Agent, I clicked the drop-down icon on the cell containing “(All),” and selected Googlebot:
Understanding what different bots are crawling, how mobile bots are crawling differently to desktop, and where the most crawling is occurring can help you see immediately where there is crawl budget waste and what areas of the site need improvement.
Find low-value add URLs
Crawl budget should not be wasted on Low value-add URLs, which are normally caused by session IDs, infinite crawl spaces, and faceted navigation.
To do this, go back to your log file, and filter by URLs that contain a “?” or question mark symbols from the URL column (containing the URL stem). To do this in Excel, remember to use “~?” or tilde question mark, as shown below:
A single “?” or question mark, as stated in the auto filter window, represents any single character, so adding the tilde is like an escape character and makes sure to filter out the question mark symbol itself.
Isn’t that easy?
Find duplicate URLs
Duplicate URLs can be a crawl budget waste and a big SEO issue, but finding them can be a pain. URLs can sometimes have slight variants (such as a trailing slash vs a non-trailing slash version of a URL).
Ultimately, the best way to find duplicate URLs is also the least fun way to do so — you have to sort by site URL stem alphabetically and manually eyeball it.
One way you can find trailing and non-trailing slash versions of the same URL is to use the SUBSTITUTE function in another column and use it to remove all forward slashes:
=SUBSTITUTE(C2, “/”, “”)
In my case, the target cell is C2 as the stem data is on the third column.
Then, use conditional formatting to identify duplicate values and highlight them.
However, eyeballing is, unfortunately, the best method for now.
See the crawl frequency of subdirectories
Finding out which subdirectories are getting crawled the most is another quick way to reveal crawl budget waste. Although keep in mind, just because a client’s blog has never earned a single backlink and only gets three views a year from the business owner’s grandma doesn’t mean you should consider it crawl budget waste — internal linking structure should be consistently good throughout the site and there might be a strong reason for that content from the client’s perspective.
To find out crawl frequency by subdirectory level, you will need to mostly eyeball it but the following formula can help:
The above formula looks like a bit of a doozy, but all it does is check if there is a trailing slash, and depending on the answer, count the number of trailing slashes and subtract either 2 or 1 from the number. This formula could be shortened if you remove all trailing slashes from your URL list using the RIGHT formula — but who has the time. What you’re left with is subdirectory count (starting from 0 from as the first subdirectory).
Replace C2 with the first URL stem / URL cell and then copy the formula down your entire list to get it working.
Make sure you replace all of the C2s with the appropriate starting cell and then sort the new subdirectory counting column by smallest to largest to get a good list of folders in a logical order, or easily filter by subdirectory level. For example, as shown in the below screenshots:
The above image is subdirectories sorted by level.
The above image is subdirectories sorted by depth.
If you’re not dealing with a lot of URLs, you could simply sort the URLs by alphabetical order but then you won’t get the subdirectory count filtering which can be a lot faster for larger sites.
See crawl frequency by content type
Finding out what content is getting crawled, or if there are any content types that are hogging crawl budget, is a great check to spot crawl budget waste. Frequent crawling on unnecessary or low priority CSS and JS files, or how crawling is occurring on images if you are trying to optimize for image search, can easily be spotted with this tactic.
In Excel, seeing crawl frequency by content type is as easy as filtering by URL or URI stem using the Ends With filtering option.
Quick Tip: You can also use the “Does Not End With” filter and use a .html extension to see how non-HTML page files are being crawled — always worth checking in case of crawl budget waste on unnecessary js or css files, or even images and image variations (looking at you WordPress). Also, remember if you have a site with trailing and non-trailing slash URLs to take that into account with the “or” operator with filtering.
Spying on bots: Understand site crawl behavior
Log File Analysis allows us to understand how bots behave by giving us an idea of how they prioritize. How do different bots behave in different situations? With this knowledge, you can not only deepen your understanding of SEO and crawling, but also give you a huge leap in understanding the effectiveness of your site architecture.
See most and least crawled URLs
This strategy has been touched up previously with seeing crawled URLs by user-agent, but it’s even faster.
In Excel, select a cell in your table and then click Insert > Pivot Table, make sure the selection contains the necessary columns (in this case, the URL or URI stem and the user-agent) and click OK.
Once you have your pivot table created, set the rows to the URL or URI stem, and the summed value as the user-agent.
From there, you can right-click in the user-agent column and sort the URLs from largest to smallest by crawl count:
Now you’ll have a great table to make charts from or quickly review and look for any problematic areas:
A question to ask yourself when reviewing this data is: Are the pages you or the client would want being crawled? How often? Frequent crawling doesn’t necessarily mean better results, but it can be an indication as to what Google and other content user-agents prioritize most.
Crawl frequency per day, week, or month
Checking the crawling activity to identify issues where there has been loss of visibility around a period of time, after a Google update or in an emergency can inform you where the problem might be. This is as simple as selecting the “date” column, making sure the column is in the “date” format type, and then using the date filtering options on the date column. If you’re looking to analyze a whole week, just select the corresponding days with the filtering options available.
Crawl frequency by directive
Understanding what directives are being followed (for instance, if you are using a disallow or even a no-index directive in robots.txt) by Google is essential to any SEO audit or campaign. If a site is using disallows with faceted navigation URLs, for example, you’ll want to make sure these are being obeyed. If they aren’t, recommend a better solution such as on-page directives like meta robots tags.
To see crawl frequency by directive, you’ll need to combine a crawl report with your log file analysis.
(Warning: We’re going to be using VLOOKUP, but it’s really not as complicated as people make it out to be)
To get the combined data, do the following:
Get the crawl from your site using your favorite crawling software. I might be biased, but I’m a big fan of the Screaming Frog SEO Spider, so I’m going to use that.
If you’re also using the spider, follow the steps verbatim, but otherwise, make your own call to get the same results.
Export the Internal HTML report from the SEO Spider (Internal Tab > “Filter: HTML”) and open up the “internal_all.xlsx” file.
From there, you can filter the “Indexability Status” column and remove all blank cells. To do this, use the “does not contain” filter and just leave it blank. You can also add the “and” operator and filter out redirected URLs by making the filter value equal “does not contain → “Redirected” as shown below:
This will show you canonicalized, no-index by meta robots and canonicalized URLs.
Copy this new table out (with just the Address and Indexability Status columns) and paste it in another sheet of your log file analysis export.
Now for some VLOOKUP magic. First, we need to make sure the URI or URL column data is in the same format as the crawl data.
Log Files don’t generally have the root domain or protocol in the URL, so we either need to remove the head of the URL using “Find and Replace” in our newly made sheet, or make a new column in your log file analysis sheet append the protocol and root domain to the URI stem. I prefer this method because then you can quickly copy and paste a URL that you are seeing problems with and take a look. However, if you have a massive log file, it is probably a lot less CPU intensive with the “Find and Replace” method.
To get your full URLs, use the following formula but with the URL field changed to whatever site you are analyzing (and make sure the protocol is correct as well). You’ll also want to change D2 to the first cell of your URL column
down the formula to the end of your Log file table and get a nice list of full URLs:
Now, create another column and call it “Indexability Status”. In the first cell, use a VLOOKUP similar to the following: =VLOOKUP(E2,CrawlSheet!A$1:B$1128,2,FALSE). Replace E2 with the first cell of you “Full URL” column, then make the lookup table into your new. crawl sheet. Remember to sue the dollar signs so that the lookup table doesn’t change as you. apply the formula to further roles. Then, select the correct column (1 would be the first column of the index table, so number 2 is the one we are after). Use the FALSE range lookup mode for exact matching. Now you have a nice tidy list of URLs and their indexability status matched with crawl data:
Crawl frequency by depth and internal links
This analysis allows us to see how a site’s architecture is performing in terms of crawl budget and crawlability. The main aim is to see if you have far more URLs than you do requests — and if you do then you have a problem. Bots shouldn’t be “giving up” on crawling your entire site and not discovering important content or wasting crawl budget on content that is not important.
Tip: It is also worth using a crawl visualization tool alongside this analysis to see the overall architecture of the site and see where there are “off-shoots” or pages with poor internal linking.
To get this all-important data, do the following:
Crawl your site with your preferred crawling tool and export whichever report has both the click depth and number of internal links with each URL.
In my case, I’m using the Screaming Frog SEO Spider, going exporting the Internal report:
Use a VLOOKUP to match your URL with the Crawl Depth column and the number of Inlinks, which will give you something like this:
Depending on the type of data you want to see, you might want to filter out only URLs returning a 200 response code at this point or make them filterable options in the pivot table we create later. If you’re checking an e-commerce site, you might want to focus solely on product URLs, or if you’re optimizing crawling of images you can filter out by file type by filtering the URI column of your log file using the “Content-Type” column of your crawl export and making an option to filter with a pivot table. As with all of these checks, you have plenty of options!
Using a pivot table, you can now analyze crawl rate by crawl depth (filtering by the particular bot in this case) with the following options:
To get something like the following:
Better data than Search Console? Identifying crawl issues
Search Console might be a go-to for every SEO, but it certainly has flaws. Historical data is harder to get, and there are limits on the number of rows you can view (at this time of writing it is 1000). But, with Log File Analysis, the sky’s the limit. With the following checks, we’re going to be discovered crawl and response errors to give your site a full health check.
Discover Crawl Errors
An obvious and quick check to add to your arsenal, all you have to do is filter the status column of your log file (in my case “sc-status” with a W3C log file type) for 4xx and 5xx errors:
Find inconsistent server responses
A particular URL may have varying server responses over time, which can either be normal behavior, such as when a broken link has been fixed or a sign there is a serious server issue occurring such as when heavy traffic to your site causes a lot more internal server errors and is affecting your site’s crawlability.
Analyzing server responses is as easy as filtering by URL and by Date:
Alternatively, if you want to quickly see how a URL is varying in response code, you can use a pivot table with the rows set to the URL, the columns set to the response codes and counting the number of times a URL has produced that response code. To achieve this setup create a pivot table with the following settings:
This will produce the following:
As you can see in the above table, you can clearly see “/inconcistent.html” (highlighted in the red box) has varying response codes.
View Errors by Subdirectory
To find which subdirectories are producing the most problems, we just need to do some simple URL filtering. Filter out the URI column (in my case “cs-uri-stem”) and use the “contains” filtering option to select a particular subdirectory and any pages within that subdirectory (with the wildcard *):
For me, I checked out the blog subdirectory, and this produced the following:
View Errors by User Agent
Finding which bots are struggling can be useful for numerous reasons including seeing the differences in website performance for mobile and desktop bots, or which search engines are best able to crawl more of your site.
You might want to see which particular URLs are causing issues with a particular bot. The easiest way to do this is with a pivot table that allows for filtering the number of times a particular response code occurs per URI. To achieve this make a pivot table with the following settings:
From there, you can filter by your chosen bot and response code type, such as image below, where I’m filtering for Googlebot desktop to seek out 404 errors:
Alternatively, you can also use a pivot table to see how many times a specific bot produces different response codes as a whole by creating a pivot table that filters by bot, counts by URI occurrence, and uses response codes as rows. To achieve this use the settings below:
For example, in the pivot table (below), I’m looking at how many of each response code Googlebot is receiving:
Diagnose on-page problems
Websites need to be designed not just for humans, but for bots. Pages shouldn’t be slow loading or be a huge download, and with log file analysis, you can see both of these metrics per URL from a bot’s perspective.
Find slow & large pages
While you can sort your log file by the “time taken” or “loading time” column from largest to smallest to find the slowest loading pages, it’s better to look at the average load time per URL as there could be other factors that might have contributed to a slow request other than the web page’s actual speed.
To do this, create a pivot table with the rows set to the URI stem or URL and the summed value set to the time taken to load or load time:
Then using the drop-down arrow, in this case, where it says “Sum of time-taken” and go to “Value Field Settings”:
In the new window, select “Average” and you’re all set:
Now you should have something similar to the following when you sort the URI stems by largest to smallest and average time taken:
Find large pages
You can now add the download size column (in my case “sc-bytes”) using the settings shown below. Remember that the set the size to the average or sum depending on what you would like to see. For me, I’ve done the average:
And you should get something similar to the following:
Bot behavior: Verifying and analyzing bots
The best and easiest way to understand bot and crawl behavior is with log file analysis as you are again getting real-world data, and it’s a lot less hassle than other methods.
Find un-crawled URLs
Simply take the crawl of your website with your tool of choice, and then take your log file an compare the URLs to find unique paths. You can do this with the “Remove Duplicates” feature of Excel or conditional formatting, although the former is a lot less CPU intensive especially for larger log files. Easy!
Identify spam bots
Unnecessary server strain from spam and spoof bots is easily identified with log files and some basic command line operators. Most requests will also have an IP associated with it, so using your IP column (in my case, it is titled “c-ip” in a W3C format log), remove all duplicates to find each individual requesting IP.
From there, you should follow the process outlined in Google’s document for verifying IPs (note: For Windows users, use the nslookup command):
Conclusion: Log Files Analysis — not as scary as it sounds
With some simple tools at your disposal, you can dive deep into how Googlebot behaves. When you understand how a website handles crawling, you can diagnose more problems than you can chew — but the real power of Log File Analysis lies in being able to test your theories about Googlebot and extending the above techniques to gather your own insights and revelations.
What theories would you test using log file analysis? What insights could you gather from log files other than the ones listed above? Let me know in the comments below.
They say history repeats itself. In the case of the great 301 vs 302 vs rel=canonical debate, it repeats itself about every three months. And in the case of this Whiteboard Friday, it repeats once every two years as we revisit a still-relevant topic in SEO and re-release an episode that’s highly popular to this day. Join Dr. Pete as he explains how bots and humans experience pages differently depending on which solution you use, why it matters, and how each choice may be treated by Google.
Click on the whiteboard image above to open a high-resolution version in a new tab!
Hey, Moz fans, it’s Dr. Pete, your friendly neighborhood marketing scientist here at Moz, and I want to talk today about an issue that comes up probably about every three months since the beginning of SEO history. It’s a question that looks something like this: Aren’t 301s, 302s, and canonicals all basically the same?
So if you’re busy and you need the short answer, it’s, “No, they’re not.” But you may want the more nuanced approach. This popped up again about a week [month] ago, because John Mueller on the Webmaster Team at Google had posted about redirection for secure sites, and in it someone had said, “Oh, wait, 302s don’t pass PageRank.”
John said, “No. That’s a myth. It’s incorrect that 302s don’t pass PR,” which is a very short answer to a very long, technical question. So SEOs, of course, jumped on that, and it turned into, “301s and 302s are the same, cats are dogs, cakes are pie, up is down.” We all did our freakout that happens four times a year.
So I want to get into why this is a difficult question, why these things are important, why they are different, and why they’re different not just from a technical SEO perspective, but from the intent and why that matters.
I’ve talked to John a little bit. I’m not going to put words in his mouth, but I think 95% of this will be approved, and if you want to ask him, that’s okay afterwards too.
Why is this such a difficult question?
So let’s talk a little bit about classic 301, 302. So a 301 redirect situation is what we call a permanent redirect. What we’re trying to accomplish is something like this. We have an old URL, URL A, and let’s say for example a couple years ago Moz moved our entire site from seomoz.org to moz.com. That was a permanent change, and so we wanted to tell Google two things and all bots and browsers:
First of all, send the people to the new URL, and, second,
pass all the signals. All these equity, PR, ranking signals, whatever you want to call them, authority, that should go to the new page as well.
So people and bots should both end up on this new page.
A classic 302 situation is something like a one-day sale. So what we’re saying is for some reason we have this main page with the product. We can’t put the sale information on that page. We need a new URL. Maybe it’s our CMS, maybe it’s a political thing, doesn’t matter. So we want to do a 302, a temporary redirect that says, “Hey, you know what? All the signals, all the ranking signals, the PR, for Google’s sake keep the old page. That’s the main one. But send people to this other page just for a couple of days, and then we’re going to take that away.”
So these do two different things. One of these tells the bots, “Hey, this is the new home,” and the other one tells it, “Hey, stick around here. This is going to come back, but we want people to see the new thing.”
So I think sometimes Google interprets our meaning and can change things around, and we get frustrated because we go, “Why are they doing that? Why don’t they just listen to our signals?”
Why are these differentiations important?
The problem is this. In the real world, we end up with things like this, we have page W that 301s to page T that 302s to page F and page F rel=canonicals back to page W, and Google reads this and says, “W, T, F.” What do we do?
We sent bad signals. We’ve done something that just doesn’t make sense, and Google is forced to interpret us, and that’s a very difficult thing. We do a lot of strange things. We’ll set up 302s because that’s what’s in our CMS, that’s what’s easy in an Apache rewrite file. We forget to change it to a 301. Our devs don’t know the difference, and so we end up with a lot of ambiguous situations, a lot of mixed signals, and Google is trying to help us. Sometimes they don’t help us very well, but they just run into these problems a lot.
In this case, the bots have no idea where to go. The people are going to end up on that last page, but the bots are going to have to choose, and they’re probably going to choose badly because our intent isn’t clear.
How are 301s, 302s, and rel=canonical different?
So there are a couple situations I want to cover, because I think they’re fairly common and I want to show that this is complex. Google can interpret, but there are some reasons and there’s some rhyme or reason.
1. Long-term 302s may be treated as 301s.
So the first one is that long-term 302s are probably going to be treated as 301s. They don’t make any sense. If you set up a 302 and you leave it for six months, Google is going to look at that and say, “You know what? I think you meant this to be permanent and you made a mistake. We’re going to pass ranking signals, and we’re going to send people to page B.” I think that generally makes sense.
Some types of 302s just don’t make sense at all. So if you’re migrating from non-secure to secure, from HTTP to HTTPS and you set up a 302, that’s a signal that doesn’t quite make sense. Why would you temporarily migrate? This is probably a permanent choice, and so in that case, and this is actually what John was addressing in this post originally, in that case Google is probably going to look at that and say, “You know what? I think you meant 301s here,” and they’re going to pass signals to the secure version. We know they prefer that anyway, so they’re going to make that choice for you.
If you’re confused about where the signals are going, then look at the page that’s ranking, because in most cases the page that Google chooses to rank is the one that’s getting the ranking signals. It’s the one that’s getting the PR and the authority.
So if you have a case like this, a 302, and you leave it up permanently and you start to see that Page B is the one that’s being indexed and ranking, then Page B is probably the one that’s getting the ranking signals. So Google has interpreted this as a 301. If you leave a 302 up for six months and you see that Google is still taking people to Page A, then Page A is probably where the ranking signals are going.
So that can give you an indicator of what their decision is. It’s a little hard to reverse that. But if you’ve left a 302 in place for six months, then I think you have to ask yourself, “What was my intent? What am I trying to accomplish here?”
Part of the problem with this is that when we ask this question, “Aren’t 302s, 301s, canonicals all basically the same?” what we’re really implying is, “Aren’t they the same for SEO?” I think this is a legitimate but very dangerous question, because, yes, we need to know how the signals are passed and, yes, Google may pass ranking signals through any of these things. But for people they’re very different, and this is important.
2. Rel=canonical is for bots, not people.
So I want to talk about rel=canonical briefly because rel=canonical is a bit different. We have Page A and Page B again, and we’re going to canonical from Page A to Page B. What we’re basically saying with this is, “Look, I want you, the bots, to consider Page B to be the main page. You know, for some reason I have to have these near duplicates. I have to have these other copies. But this is the main one. This is what I want to rank. But I want people to stay on Page A.”
So this is entirely different from a 301 where I want people and bots to go to Page B. That’s different from a 302, where I’m going to try to keep the bots where they are, but send people over here.
So take it from a user perspective. I have had in Q&A all the time people say, “Well, I’ve heard that rel=canonical passes ranking signals. Which should I choose? Should I choose that or 301? What’s better for SEO?”
That’s true. We do think it generally passes ranking signals, but for SEO is a bad question, because these are completely different user experiences, and either you’re going to want people to stay on Page A or you’re going to want people to go to Page B.
Why this matters, both for bots and for people
So I just want you to keep in mind, when you look at these three things, it’s true that 302s can pass PR. But if you’re in a situation where you want a permanent redirect, you want people to go to Page B, you want bots to go to Page B, you want Page B to rank, use the right signal. Don’t confuse Google. They may make bad choices. Some of your 302s may be treated as 301s. It doesn’t make them the same, and a rel=canonical is a very, very different situation that essentially leaves people behind and sends bots ahead.
So keep in mind what your use case actually is, keep in mind what your goals are, and don’t get over-focused on the ranking signals themselves or the SEO uses because all off these three things have different purposes.
So I hope that makes sense. If you have any questions or comments or you’ve seen anything weird actually happen on Google, please let us know and I’ll be happy to address that. And until then, we’ll see you next week.
If the last day of MozCon felt like it went too fast or if you forgot everything that happened today (we wouldn’t judge — there were so many insights), don’t fret. We captured all of day three’s takeaways so you could relive the magic of day three.
Don’t forget to check out all the photos with Roger from the photobooth! They’re available here in the MozCon Facebook group. Plus: You asked and we delivered: the 2019 MozCon speaker walk-on playlist is now live and available here for your streaming pleasure.
Cindy Krum— Fraggles, Mobile-First Indexing, & the SERP of the Future
If you were hit with an instant wave of nostalgia after hearing Cindy’s walk out music, then you are in good company and you probably were not disappointed in the slightest by Cindy’s talk on Fraggles.
First learning of the day: Fraggles. Fragment + Handles. A piece of information and an anchor that scrolls directly to the information on the page @Suzzicks#MozCon
“Fraggles” are fragments + handles. A fragment is a piece of info on a page. A handle is something like a bookmark, jump link, or named anchor — they help people navigate through long pages to get what they’re looking for faster.
Ranking pages is an inefficient way to answer questions. One page can answer innumerable questions, so Google’s now can pull a single answer from multiple parts of your page, skipping sections they don’t think are as useful for a particular answer.
The implications for voice are huge! It means you don’t have to listen to your voice device spout off a page’s worth of text before your question is answered.
Google wants to index more than just websites. They want to organize the world’s information, not websites. Fraggles are a demonstration of that.
Luke Carthy — Killer Ecommerce CRO and UX Wins Using A SEO Crawler
Luke Carthy did warn us in his talk description that we should all flex our notetaking muscles for all the takeaways we would furiously jot down — and he wasn’t wrong.
Traffic doesn’t always mean sales and sales don’t always mean traffic!
Custom extraction is a great tool for finding missed CRO opportunities. For example, Luke found huge opportunity on Best Buy’s website — thousands of people’s site searches were leading them to an unoptimized “no results found” page.
You can also use custom extraction to find what product recommendations you or your customers are using at scale! Did you know that 35% of what customers buy on Amazon and 75 percent of what people watch on Netflix are the results of these recommendations?
For example, are you showing near-exact products or are you showing complementary products? (hint: try the latter and you’ll likely increase your sales!)
Custom extraction from Screaming Frog allows you to scrape any data from the HTML of the web pages while crawling them.
Andy Crestodina — Content, Rankings, and Lead Generation: A Breakdown of the 1% Content Strategy
Next up, Andy of Orbit Media took the stage with a comprehensive breakdown of the most effective tactics for turning content into a high-powered content strategy. He also brought the fire with this sound advice that we can apply in both our work life and personal life.
If you write an amazing, high-traffic blog post, the people visiting it don’t have commercial intent. People landing on a sales page have 50x higher intent. #mozcon@crestodina
Blog visitors often don’t have commercial intent. One of the greatest ways to leverage blog posts for leads is by using the equity we generate from links to our helpful posts and passing that onto our product and service pages.
If you want links and shares, invest in original research! Not sure what to research? Look for unanswered questions or unproven statements in your industry and provide the data.
Original research may take longer than a standard post, but it’s much more effective! When you think about it this way, do you really have time to put out more, mediocre posts?
Give what you want to get. Want links? Link to people. Want comments? Comment on others people’s work.
To optimize content for social engagement, it should feature real people, their faces, and their quotes.
Collaborating with other content creators on your content not only gives it built-in amplification, but it also leads to great connections and is just generally more fun.
Rob Ousbey — Running Your Own SEO Tests: Why It Matters & How to Do It Right
Google’s algorithms have changed a heck of a lot in recent years — what’s an SEO to do? Follow Rob’s advice — both fashion and SEO — who says that the answer lies in testing.
in head terms, it’s about user engagement metrics. However, links are more correllated with long tail searches. @RobOusbey#Mozcon
“This is the way we’ve always done it” isn’t sufficient justification for SEO tactics in today’s search landscape.
In the earlier days of the algorithm, it was much easier to demote spam than it was to promote what’s truly good.
Rob and his team had a theory that Google was beginning to rely more heavily on user experience and satisfaction than some of the more traditional ranking factors like links.
Through SEO A/B testing, they found that:
Google relies less heavily on link signals when it comes to the top half of the results on page 1.
Google relies more heavily on user experience for head terms (terms with high search volume), likely because they have more user data to draw from.
In the process of A/B testing, they also found that the same test often produces different results on different sites. The best way to succeed in today’s SEO landscape is to cultivate a culture of testing!
Greg Gifford — Dark Helmet’s Guide to Local Domination with Google Posts and Q&A
If you’re a movie buff, you probably really appreciated Greg’s talk — he schooled us all in movie references and brought the fire with his insights on Google Posts and Q&A
Google Posts allows you to drive conversions without driving people to your site – convert on zero-click searches! @GregGifford#MozCon
The man behind #shoesofmozcon taught us that Google is the new home page for local businesses, so we should be leveraging the tools Google has given us to make our Google My Business profiles great. For example…
Images should be 1200×900 on google posts
Images are cropped slightly higher than the center and it’s not consistent every time
The image size of the thumbnail is different on desktop than it is on mobile
You can also upload videos. The file size limit is 100mb and/or 30 seconds
Add a call-to-action button to make your Posts worth it! Just know that the button often means you get less real estate for text in your Posts
Don’t share social fluff. Attract with an offer that makes you stand out
Make sure you use UTM tracking so you can understand how your Posts are performing in Google Analytics. Otherwise, it’ll be attributed as direct traffic.
Anyone can ask and answer questions — why not the business owner! Control the conversation and treat this feature like it’s your new FAQ page.
This feature works on an upvote system. The answer with the most upvotes will show first.
Don’t include a URL or phone number in these because it’ll get filtered out.
A lot of these questions are potential customers! Out of 640 car dealerships’ Q&As Greg evaluated, 40 percent were leads! Of that 40 percent, only 2 questions were answered by the dealership.
Emily Triplett Lentz — How to Audit for Inclusive Content
Emily of Help Scout walked dropped major knowledge on the importance of spotting and eliminating biases that frequently find their way into online copy. She also hung out backstage after her talk to cheer on her fellow speakers. #GOAT. #notallheroeswearcapes.
As content creators, we’d all do well to keep ableism in mind: discrimination in favor of able-bodied people. However, we’re often guilty of this without even knowing it.
One example of ableism that often makes its way into our copy is comparing dire or subideal situations with the physical state of another human (ex: “crippling”).
While we should work on making our casual conversation more inclusive too, this is particularly important for brands.
Create a list of ableist words, crawl your site for them, and then replace them. However, you’ll likely find that there is no one-size-fits-all replacement for these words. We often use words like “crazy” as filler words. By removing or replacing with a more appropriate word, we make our content better and more descriptive in the process.
At the end of the day, brands should remember that their desire for freedom of word choice isn’t more important than people’s right not to feel excluded and hurt. When there’s really no downside to more inclusive content, why wouldn’t we do it?
Curious about image optimization and visual search? Joelle has the goods for you — and was blowing people’s minds with her tips for visual optimization and how to leverage Google Lens, Pinterest, and AR for visual search.
“Visual search is easier when you don’t know what you’re looking for, when you’re looking to match a particular style, and when your search way is too long or complicated.” @joelleirvine#MozCon
Visual search is not the same thing as searching for images. We’re talking about the process of using an image to search for other content.
Visual search like Google Lens makes it easier to search when you don’t know what you’re looking for.
Pinterest has made a lot of progress in this area. They have a hybrid search that allows you to find complimentary items to the one you searched. It’s like finding a rug that matches a chair you like rather than finding more of the same type of chair.
62 percent of millennials surveyed said they would like to be able to search by visual, so while this is mostly being used by clothing retailers and home decor right now, visual search is only going to get better, so think about the ways you can leverage it for your brand!
Joy Hawkins — Factors that Affect the Local Algorithm that Don’t Impact Organic
Proximity varies greatly when comparing local and organic results — just ask Joy of Sterling Sky, who gets real about fake listings while walking through the findings of a recent study.
GMB landing pages (AKA: the website URL you link to from your GMB account)
Joy tested linking to the home page (which had more authority/prominence) vs. linking to the local landing page (which had more relevance) and found that traffic went way up when linking to the home page.
Before you go switching all your GMB links though, test this for yourself!
Joy wanted to know how much reviews actually impacted ranking, and what it was exactly about reviews that would help or hurt.
She decided to see what would happen to rankings when reviews were removed. This happened to a business who was review gating (a violation of Google’s guidelines) but Joy found that reviews flagged for violations aren’t actually removed, they’re hidden, explaining why “removed” reviews don’t negatively impact local rankings.
Organic results can get filtered because of duplicate content, whereas local results can get filtered because they’re too close to another business in the same category. This is called the Possum filter.
Keywords in a business name
This is against Google’s guidelines but it works sadly
For example, Joy tested adding the word “salad bar” to a listing that didn’t even have a salad bar and their local rankings for that keyword shot up.
Although it works, don’t do it! Google can remove your listing for this type of violation, and they’ve been removing more listings for this reason lately.
New listings can rank even if they have no website, authority, citations, etc. simply because they keyword stuffed their business name. These types of rankings can happen overnight, whereas it can take a year or more to achieve certain organic rankings.
Spend time reporting spam listings in your clients’ niches because it can improve your clients’ local rankings.
Britney Muller — Featured Snippets: Essentials to Know & How to Target
Closing out day three of MozCon was our very own Britney, Sr. SEO scientist extraordinaire, on everyone’s favorite SEO topic: Featured snippets!
Why are we so concerned about traffic? What about branding/messaging/share of voice? You can’t NOT target a keyword just because it has a Featured Snippet @BritneyMuller#MozCon
We’re seeing more featured snippets than ever before, and they’re not likely going away. It’s time to start capitalizing on this SERP feature so we can start earning brand awareness and traffic for our clients!
Know what keywords trigger featured snippets that you rank on page 1 for
Know the searcher’s intent
Provide succinct answers
Add summaries to popular posts
Identify commonly asked questions
Leverage Google’s NLP API
Monitor featured snippets
If all else fails, leverage ranking third party sites. Maybe your own site has low authority and isn’t ranking well, but try publishing on Linkedin or Medium instead to get the snippet!
There’s lots of debate over whether featured snippets send you more traffic or take it away due to zero-click results, but consider the benefits featured snippets can bring even without the click. Whether featured snippets bring you traffic, increased brand visibility in the SERPs, or both, they’re an opportunity worth chasing.
Aaaand, that’s a wrap!
Thanks for joining us at this year’s MozCon! And a HUGE thank you to everyone (Mozzers, partners, and crew) who helped make this year’s MozCon possible — we couldn’t have done it without all of you.
What was your favorite moment of the entire conference? Tell us below in the comments! And don’t forget to grab the speaker slides here!
While SEOs have been doubling-down on content and quality signals for their websites, Google was building the foundation of a new reality for crawling — indexing and ranking. Though many believe deep in their hearts that “Content is King,” the reality is that Mobile-First Indexing enables a new kind of search result. This search result focuses on surfacing and re-publishing content in ways that feed Google’s cross-device monetization opportunities better than simple websites ever could.
For two years, Google honed and changed their messaging about Mobile-First Indexing, mostly de-emphasizing the risk that good, well-optimized, Responsive-Design sites would face. Instead, the search engine giant focused more on the use of the Smartphone bot for indexing, which led to an emphasis on the importance of matching SEO-relevant site assets between desktop and mobile versions (or renderings) of a page. Things got a bit tricky when Google had to explain that the Mobile-First Indexing process would not necessarily be bad for desktop-oriented content, but all of Google’s shifting and positioning eventually validated my long-stated belief: That Mobile-First Indexing is not really about mobile phones, per se, but mobile content.
I would like to propose an alternative to the predominant view, a speculative theory, about what has been going on with Google in the past two years, and it is the thesis of my 2019 MozCon talk — something we are calling Fraggles and Fraggle-based Indexing.
I’ll go through Fraggles and Fraggle-based indexing, and how this new method of indexing has made web content more ‘liftable’ for Google. I’ll also outline how Fraggles impact the Search Results Pages (SERPs), and why it fits with Google’s promotion of Progressive Web Apps. Next, I will provide information about how astute SEO’s can adapt their understanding of SEO and leverage Fraggles and Fraggle-Based Indexing to meet the needs of their clients and companies. Finally, I’ll go over the implications that this new method of indexing will have on Google’s monetization and technology strategy as a whole.
Ready? Let’s dive in.
Fraggles & Fraggle-based indexing
The SERP has changed in many ways. These changes can be thought of and discussed separately, but I believe that they are all part of a larger shift at Google. This shift includes “Entity-First Indexing” of crawled information around the existing structure of Google’s Knowledge Graph, and the concept of “Portable-prioritized Organization of Information,” which favors information that is easy to lift and re-present in Google’s properties — Google describes these two things together as “Mobile-First Indexing.”
Fraggles represent individual parts (fragments) of a page for which Google overlayed a “handle” or “jump-link” (aka named-anchor, bookmark, etc.) so that a click on the result takes the users directly to the part of the page where the relevant fragment of text is located. These Fraggles are then organized around the relevant nodes on the Knowledge Graph, so that the mapping of the relationships between different topics can be vetted, built-out, and maintained over time, but also so that the structure can be used and reused, internationally — even if different content is ranking.
More than one Fraggle can rank for a page, and the format can vary from a text-link with a “Jump to” label, an unlabeled text link, a site-link carousel, a site-link carousel with pictures, or occasionally horizontal or vertical expansion boxes for the different items on a page.
The easiest way for an SEO to think about a Fragment is within the example of an AJAX expansion box: The piece of text or information that is fetched from the server to populate the AJAX expander when clicked could be described as a Fragment. Alternatively, if it is indexed for Mobile-First Indexing, it is a Fraggle.
We have also recently discovered that Google has begun to index URLs with a # jump-link, after years of not doing so, and is reporting on them separately from the primary URL in Search Console. As you can see below from our data, they aren’t getting a lot of clicks, but they are getting impressions. This is likely because of the low average position.
Why index fragments & Fraggles?
If you’re used to thinking of rankings with the smallest increment being a URL, this idea can be hard to wrap your brain around. To help, consider this thought experiment: How useful would it be for Google to rank a page that gave detailed information about all different kinds of fruits and vegetables? It would be easy for a query like “fruits and vegetables,” that’s for sure. But if the query is changed to “lettuce” or “types of lettuce,” then the page would struggle to rank, even if it had the best, most authoritative information.
This is because the “lettuce” keywords would be diluted by all the other fruit and vegetable content. It would be more useful for Google to rank the part of the page that is about lettuce for queries related to lettuce, and the part of the page about radishes well for queries about radishes. But since users don’t want to scroll through the entire page of fruits and vegetables to find the information about the particular vegetable they searched for, Google prioritizes pages with keyword focus and density, as they relate to the query. Google will rarely rank long pages that covered multiple topics, even if they were more authoritative.
With featured snippets, AMP featured snippets, and Fraggles, it’s clear that Google can already find the important parts of a page that answers a specific question — they’ve actually been able to do this for a while. So, if Google can organize and index content like that, what would the benefit be in maintaining an index that was based only on per-pages statistics and ranking? Why would Google want to rank entire pages when they could rank just the best parts of pages that are most related to the query?
To address these concerns, historically, SEO’s have worked to break individual topics out into separate pages, with one page focused on each topic or keyword cluster. So, with our vegetable example, this would ensure that the lettuce page could rank for lettuce queries and the radish page could rank for radish queries. With each website creating a new page for every possible topic that they would like to rank for, there’s lot of redundant and repetitive work for webmasters. It also likely adds a lot of low-quality, unnecessary pages to the index. Realistically, how many individual pages on lettuce does the internet really need, and how would Google determine which one is the best? The fact is, Google wanted to shift to an algorithm that focused less on links and more on topical authority to surface only the best content — and Google circumvents this with the scrolling feature in Fraggles.
Even though the effort to switch to Fraggle-based indexing, and organize the information around the Knowledge Graph, was massive, the long-term benefits of the switch far out-pace the costs to Google because they make Google’s system for flexible, monetizable and sustainable, especially as the amount of information and the number of connected devices expands exponentially. It also helps Google identify, serve and monetize new cross-device search opportunities, as they continue to expand. This includes search results on TV’s, connected screens, and spoken results from connected speakers. A few relevant costs and benefits are outlined below for you to contemplate, keeping Google’s long-term perspective in mind:
Why Fraggles and Fraggle-based indexing are important for PWAs
What also makes the shift to Fraggle-based Indexing relevant to SEOs is how it fits in with Google’s championing of Progressive Web Apps or AMP Progressive Web Apps, (aka PWAs and PWA-AMP websites/web apps). These types of sites have become the core focus of Google’s Chrome Developer summits and other smaller Google conferences.
The answer is because PWA’s require ServiceWorkers, which uses Fraggles and Fraggle-based indexing to take the burden off crawling and indexing of complex web content.
ServiceWorkers and SEO
For a PWA to be indexed, Google requires webmasters to ‘register their app in Firebase,’ but they used to require webmasters to “register their ServiceWorker.” Firebase is the Google platform that allows webmasters to set up and manage indexing and deep linking for their native apps, chat-bots and, now, PWA’s.
Direct communication with a PWA specialist at Google a few years ago revealed that Google didn’t crawl the ServiceWorker itself, but crawled the API to the ServiceWorker. It’s likely that when webmasters register their ServiceWorker with Google, Google is actually creating an API to the ServiceWorker, so that the content can be quickly and easily indexed and cached on Google’s servers. Since Google has already launched an Indexing API and appears to now favor API’s over traditional crawling, we believe Google will begin pushing the use of ServiceWorkers to improve page speed, since they can be used on non-PWA sites, but this will actually be to help ease the burden on Google to crawl and index the content manually.
It’s important to remember that this is how AMP, Schema, and many other types of powerful SEO functionalities have started with a limited launch; beyond that, some great SEO’s have already tested submitting other types of content in the API and seen success. Submitting to APIs skips Google’s process of blindly crawling the web for new content and allows webmasters to feed the information to them directly.
It is possible that the new Indexing API follows a similar structure or process to PWA indexing. Submitted URLs can already get some kinds of content indexed or removed from Google’s index, usually in about an hour, and while it is only currently officially available for the two kinds of content, we expect it to be expanded broadly.
How will this impact SEO strategy?
Of course, every SEO wants to know how to leverage this speculative theory — how can we make the changes in Google to our benefit?
The first thing to do is take a good, long, honest look at a mobile search result. Position #1 in the organic rankings is just not what it used to be. There’s a ton of engaging content that is often pushing it down, but not counting as an organic ranking position in Search Console. This means that you may be maintaining all your organic rankings while also losing a massive amount of traffic to SERP features like Knowledge Graph results, Featured Snippets, Google My Business, maps, apps, Found on the Web, and other similar items that rank outside of the normal organic results.
These results, as well as Pay-per-Click results (PPC), are more impactful on mobile because they are stacked above organic rankings. Rather than being off to the side, as they might be in a desktop view of the search, they push organic rankings further down the results page. There has been some great reporting recently about the statistical and large-scale impact of changes to the SERP and how these changes have resulted in changes to user-behavior in search, especially from Dr. Pete Meyers, Rand Fishkin, and JumpTap.
Dr. Pete has focused on the increasing number of changes to the Google Algorithm recorded in his MozCast, which heated up at the end of 2016 when Google started working on Mobile-First Indexing, and again after it launched the Medic update in 2018.
Rand, on the other hand, focused on how the new types of rankings are pushing traditional organic results down, resulting in less traffic to websites, especially on mobile. All this great data from these two really set the stage for a fundamental shift in SEO strategy as it relates to Mobile-First Indexing.
The research shows that Google re-organized its index to suit a different presentation of information — especially if they are able to index that information around an entity-concept in the Knowledge Graph. Fraggle-based Indexing makes all of the information that Google crawls even more portable because it is intelligently nested among related Knowledge Graph nodes, which can be surfaced in a variety of different ways. Since Fraggle-based Indexing focuses more on the meaningful organization of data than it does on pages and URLs, the results are a more “windowed” presentation of the information in the SERP. SEOs need to understand that search results are now based on entities and use-cases (think micro-moments), instead of pages and domains.
Google’s Knowledge Graph
To really grasp how this new method of indexing will impact your SEO strategy, you first have to understand how Google’s Knowledge Graph works.
Since it is an actual “graph,” all Knowledge Graph entries (nodes) include both vertical and lateral relationships. For instance, an entry for “bread” can include lateral relationships to related topics like cheese, butter, and cake, but may also include vertical relationships like “standard ingredients in bread” or “types of bread.”
Lateral relationships can be thought of as related nodes on the Knowledge Graph, and hint at “Related Topics” whereas vertical relationships point to a broadening or narrowing of the topic; which hints at the most likely filters within a topic. In the case of bread, a vertical relationship-up would be topics like “baking,” and down would include topics like “flour” and other ingredients used to make bread, or “sourdough” and other specific types of bread.
SEOs should note that Knowledge Graph entries can now include an increasingly wide variety of filters and tabs that narrow the topic information to benefit different types of searcher intent. This includes things like helping searchers find videos, books, images, quotes, locations, but in the case of filters, it can be topic-specific and unpredictable (informed by active machine learning). This is the crux of Google’s goal with Fraggle-based Indexing: To be able to organize the information of the web-based on Knowledge Graph entries or nodes, otherwise discussed in SEO circles as “entities.”
Since the relationships of one entity to another remain the same, regardless of the language a person is speaking or searching in, the Knowledge Graph information is language-agnostic, and thus easily used for aggregation and machine learning in all languages at the same time. Using the Knowledge Graph as a cornerstone for indexing is, therefore, a much more useful and efficient means for Google to access and serve information in multiple languages for consumption and ranking around the world. In the long-term, it’s far superior to the previous method of indexing.
Examples of Fraggle-based indexing in the SERPs
Google has dramatically increased the number of Knowledge Graph entries and the categories and relationships within them. The build-out is especially prominent for topics for which Google has a high amount of structured data and information already. This includes topics like:
TV and Movies — from Google Play
Food and Recipe — from Recipe Schema, recipe AMP pages, and external food and nutrition databases
Science and medicine — from trusted sources (like WebMD)
Businesses — from Google My Business.
Google is adding more and more nodes and relationships to their graph and existing entries are also being built-out with more tabs and carousels to break a single topic into smaller, more granular topics or type of information.
As you can see below, the build-out of the Knowledge Graph has also added to the number of filters and drill-down options within many queries, even outside of the Knowledge Graph. This increase can be seen throughout all of the Google properties, including Google My Business and Shopping, both of which we believe are now sections of the Knowledge Graph:
Other similar examples include the additional filters and “Related Topics” results in Google Images, which we also believe to represent nodes on the Knowledge Graph:
The Knowedge Graph is also being presented in a variety of different ways. Sometimes there’s a sticky navigation that persists at the top of the SERP, as seen in many media-oriented queries, and sometimes it’s broken up to show different information throughout the SERP, as you may have noticed in many of the local business-oriented search results, both shown below.
Since the launch of Fraggle-based indexing is essentially a major Knowledge Graph build-out, Knowledge Graph results have also begun including more engaging content which makes it even less likely that users will click through to a website. Assets like playable video and audio, live sports scores, and location-specific information such as transportation information and TV time-tables can all be accessed directly in the search results. There’s more to the story, though.
Companies who want to leverage the Knowledge Graph should take every opportunity to create your own assets, like AR models and AMP Stories, so that Google will have no reason to do it. Beyond that, companies should submit accurate information directly to Google whenever they can. The easiest way to do this is through Google My Business (GMB). Whatever types of information are requested in GMB should be added or uploaded. If Google Posts are available in your business category, you should be doing Posts regularly, and making sure that they link back to your site with a call to action. If you have videos or photos that are relevant for your company, upload them to GMB. Start to think of GMB as a social network or newsletter — any assets that are shared on Facebook or Twitter can also be shared on Google Posts, or at least uploaded to the GMB account.
You should also investigate the current Knowledge Graph entries that are related to your industry, and work to become associated with recognized companies or entities in that industry. This could be from links or citations on the entity websites, but it can also include being linked by third-party lists that give industry-specific advice and recommendations, such as being listed among the top competitors in your industry (“Best Plumbers in Denver,” “Best Shoe Deals on the Web,” or “Top 15 Best Reality TV Shows”). Links from these posts also help but are not required — especially if you can get your company name on enough lists with the other top players. Verify that any links or citations from authoritative third-party sites like Wikipedia, Better Business Bureau, industry directories, and lists are all pointing to live, active, relevant pages on the site, and not going through a 301 redirect.
While this is just speculation and not a proven SEO strategy, you might also want to make sure that your domain is correctly classified in Google’s records by checking the industries that it is associated with. You can do so in Google’s MarketFinder tool. Make updates or recommend new categories as necessary. Then, look into the filters and relationships that are given as part of Knowledge Graph entries and make sure you are using the topic and filter words as keywords on your site.
Featured Snippets or “Answers” first surfaced in 2014 and have also expanded quite a bit, as shown in the graph below. It is useful to think of Featured Snippets as rogue facts, ideas or concepts that don’t have a full Knowledge Graph result, though they might actually be associated with certain existing nodes on the Knowledge Graph (or they could be in the vetting process for eventual Knowledge Graph build-out).
Featured Snippets seem to surface when the information comes from a source that Google does not have an incredibly high level of trust for, like it does for Wikipedia, and often they come from third party sites that may or may not have a monetary interest in the topic — something that makes Google want to vet the information more thoroughly and may prevent Google from using it, if a less bias option is available.
Like the Knowledge Graph, Featured Snippets results have grown very rapidly in the past year or so, and have also begun to include carousels — something that Rob Bucci writes about extensively here. We believe that these carousels represent potentially related topics that Google knows about from the Knowledge Graph. Featured Snippets now look even more like mini-Knowledge Graph entries: Carousels appear to include both lateral and vertically related topics, and their appearance and maintenance seem to be driven by click volume and subsequent searches. However, this may also be influenced by aggregated engagement data for People Also Ask and Related Search data.
The build-out of Featured Snippets has been so aggressive that sometimes the answers that Google lifts are obviously wrong, as you can see in the example image below. It is also important to understand that Featured Snippet results can change from location to location and are not language-agnostic, and thus, are not translated to match the Search Language or the Phone Language settings. Google also does not hold themselves to any standard of consistency, so one Featured Snippet for one query might present an answer one way, and a similar query for the same fact could present a Featured Snippet with slightly different information. For instance, a query for “how long to boil an egg” could result in an answer that says “5 minutes” and a different query for “how to make a hard-boiled egg” could result in an answer that says “boil for 1 minute, and leave the egg in the water until it is back to room temperature.”
The data below was collected by Moz and represents an average of roughly 10,000 that skews slightly towards ‘head’ terms.
SEO strategy for featured snippets
All of the standard recommendations for driving Featured Snippets apply here. This includes making sure that you keep the information that you are trying to get ranked in a Featured Snippet clear, direct, and within the recommended character count. It also includes using simple tables, ordered lists, and bullets to make the data easier to consume, as well as modeling your content after existing Featured Snippet results in your industry.
This is still speculative, but it seems likely that the inclusion of Speakable Schema markup for things like “How To,” “FAQ,” and “Q&A” may also drive Featured Snippets. These kinds of results are specially designated as content that works well in a voice-search. Since Google has been adamant that there is not more than one index, and Google is heavily focused on improving voice-results from Google Assistant devices, anything that could be a good result in the Google Assistant, and ranks well, might also have a stronger chance at ranking in a Featured Snippet.
People Also Ask & Related Searches
Finally, the increased occurrence of “Related Searches” as well as the inclusion of People Also Ask (PAA) questions, just below most Knowledge Graph and Featured Snippet results, is undeniable. The Earl Tea screenshot shows that PAA’s along with Interesting Finds are both part of the Knowledge Graph too.
The graph below shows the steady increase in PAA’s. PAA results appear to be an expansion of Featured Snippets because once expanded, the answer to the question is displayed, with the citation below it. Similarly, some Related Search results also now include a result that looks like a Featured Snippet, instead of simply linking over to a different search result. You can now find ‘Related Searches’ throughout the SERP, often as part of a Knowledge Graph results, but sometimes also in a carousel in the middle of the SERP, and always at the bottom of the SERP — sometimes with images and expansion buttons to surface Featured Snippets within the Related Search results directly in the existing SERP.
Boxes with Related Searches are now also included with Image Search results. It’s interesting to note that Related Search results in Google Images started surfacing at the same time that Google began translating image Title Tags and Alt Tags. It coincides well with the concept that Entity-First Indexing, that Entities and Knowledge Graph are language-agnostic, and that Related Searches are somehow related to the Knowledge Graph.
This data was collected by Moz and represents an average of roughly 10,000 that skews slightly towards ‘head’ terms.
SEO STRATEGY for PAA and related searches
Since PAAs and some Related Searches now appear to simply include Featured Snippets, driving Featured Snippet results for your site is also a strong strategy here. It often appears that PAA results include at least two versions of the same question, re-stated with a different language, before including questions that are more related to lateral and vertical nodes on the Knowledge Graph. If you include information on your site that Google thinks is related to the topic, based on Related Searches and PAA questions, it could help make your site appear relevant and authoritative.
Finally, it is crucial to remember that you don’t have a website to rank in Google now and SEO’s should consider non-website rankings as part of their job too.
If a business doesn’t have a website, or if you just want to cover all the bases, you can let Google host your content directly — in as many places as possible. We have seen that Google-hosted content generally seems to get preferential treatment in Google search results and Google Discover, especially when compared to the decreasing traffic from traditional organic results. Google is now heavily focused on surfacing multimedia content, so anything that you might have previously created a new page on your website for should now be considered for a video.
Google My Business (GMB) is great for companies that don’t have websites, or that want to host their websites directly with Google. YouTube is great for videos, TV, video-podcasts, clips, animations, and tutorials. If you have an app, a book, an audio-book, a podcast, a movie, TV show, class or music, or PWA, you can submit that directly to GooglePlay (much of the video content in GooglePlay is now cross-populated in YouTube and YouTube TV, but this is not necessarily true of the other assets). This strategy could also include books in Google Books, flights in Google Flights, Hotels in Google Hotel listings, and attractions in Google Explore. It also includes having valid AMP code, since Google hosts AMP content, and includes Google News if your site is an approved provider of news.
Changes to SEO tracking for Fraggle-based indexing
The biggest problem for SEOs is the missing organic traffic, but it is also the fact that current methods of tracking organic results generally don’t show whether things like Knowledge Graph, Featured Snippets, PAA, Found on the Web, or other types of results are appearing at the top of the query or somewhere above your organic result. Position one in organic results is not what it used to be, nor is anything below it, so you can’t expect those rankings to drive the same traffic. If Google is going to be lifting and representing everyone’s content, the traffic will never arrive at the site and SEOs won’t know if their efforts are still returning the same monetary value. This problem is especially poignant for publishers, who have only been able to sell advertising on their websites based on the expected traffic that the website could drive.
The other thing to remember is that results differ — especially on mobile, which varies from device to device (generally based on screen size) but also can vary based on the phone IOS. They can also change significantly based on the location or the language settings of the phone, and they definitely do not always match with desktop results for the same query. Most SEO’s don’t know much about the reality of their mobile search results because most SEO reporting tools still focus heavily on desktop results, even though Google has switched to Mobile-First.
As well, SEO tools generally only report on rankings from one location — the location of their servers — rather than being able to test from different locations.
The only thing that good SEO’s can do to address this problem is to use tools like the MobileMoxie SERP Test to check what rankings look like on top keywords from all the locations where their users may be searching. While the free tool only provides results with one location at a time, subscribers can test search results in multiple locations, based on a service-area radius or based on an uploaded CSV of addresses. The tool has integrations with Google Sheets, and a connector with Data Studio, to help with SEO reporting, but APIs are also available, for deeper integrations in content editing tools, dashboards and for use within other SEO tools.
At MozCon 2017, I expressed my belief that the impact of Mobile-First Indexing requires a re-interpretation of the words “Mobile,” “First,” and “Indexing.” Re-defined in the context of Mobile-First Indexing, the words should be understood to mean “portable,” “preferred,” and “organization of information.” The potential of a shift to Fraggle-based indexing and the recent changes to the SERPs, especially in the past year, certainly seems to prove the accuracy of this theory. And though they have been in the works for more than two years, the changes to the SERP now seem to be rolling-out faster and are making the SERP unrecognizable from what it was only three or four years ago.
SEOs need to consider the opportunities and change the way we view our overall indexing strategy, and our jobs as a whole. If Google is organizing the index around the Knowledge Graph, that makes it much easier for Google to constantly mention near-by nodes of the Knowledge Graph in “Related Searches” carousels, links from the Knowledge Graph, and topics in PAAs. It might also make it easier to believe that featured snippets are simply pieces of information being vetted (via Google’s click-crowdsourcing) for inclusion or reference in the Knowledge Graph.
Fraggles and Fraggled indexing re-frames the switch to Mobile-First Indexing, which means that SEOs and SEO tool companies need to start thinking mobile-first — i.e. the portability of their information. While it is likely that pages and domains still carry strong ranking signals, the changes in the SERP all seem to focus less on entire pages, and more on pieces of pages, similar to the ones surfaced in Featured Snippets, PAAs, and some Related Searches. If Google focuses more on windowing content and being an “answer engine” instead of a “search engine,” then this fits well with their stated identity, and their desire to build a more efficient, sustainable, international engine.
SEOs also need to find ways to serve their users better, by focusing more on the reality of the mobile SERP, and how much it can vary for real users. While Google may not call the smallest rankable units Fraggles, it is what we call them, and we think they are critical to the future of SEO.