You may not want certain pages of your site crawled because they might not be useful to users if found in a search engine's search results. If you do want to prevent search engines from crawling your pages, Google Search Console has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain. For more information on robots.txt, we suggest this Webmaster Help Center guide on using robots.txt files13.


To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]
Content gaps – make an inventory of the site’s key content assets, are they lacking any foundational/cornerstone content pieces, non-existent content types, or relevant topic areas that haven’t been covered? What topics or content are missing from your competitors? Can you beat your competitors’ information-rich content assets? Useful guides on Content Gap Analysis:
There is no magic formula for content marketing success, despite what some would have you believe. For this reason, vary the length and format of your content to make it as appealing as possible to different kinds of readers. Intersperse shorter, news-based blog posts with long-form content as well as video, infographics and data-driven pieces for maximum impact.
Search engine optimization (SEO) is often about making small modifications to parts of your website. When viewed individually, these changes might seem like incremental improvements, but when combined with other optimizations, they could have a noticeable impact on your site's user experience and performance in organic search results. You're likely already familiar with many of the topics in this guide, because they're essential ingredients for any web page, but you may not be making the most out of them.
Excellent post Brian. I think the point about writing content that appeals to influencers in spot on. Could you recommend some good, manual strategies through which I can spot influencers in boring niches *B2B* where influencers are not really talking much online? Is it a good idea to rely on newspaper articles to a feel for what a particular industry is talking about? Would love to hear your thoughts on that.
Some features on the Service require payment of fees. If you elect to sign up for these features, you agree to pay Us the applicable fees and any taxes as described on the Service. All payments due are in the U.S. dollars unless otherwise indicated. Upon payment, You will have access to the chosen features immediately. If Your use of the Service is terminated for any reason, whether by You or by Us, You will lose and forfeit any time remaining on Your account with Us.

Hello Brian, really such an informative article and is more meaningful as you provided screen shots. I have noticed that articles with images bring more value to understand the things. I have just started my career in this industry and thus keep looking for some good articles/blogs that are meaningful and help me to implement tips in my work apart from my seniors instructions. I guess this was I can prove them about my caliber 🙂
To give you an example, our domain authority is currently a mediocre 41 due to not putting a lot of emphasis on it in the past. For that reason, we want to (almost) automatically scratch off any keyword with a difficulty higher than 70%—we just can’t rank today. Even the 60% range as a starting point is gutsy, but it’s achievable if the content is good enough.

Fortunately, Google puts more weight on the anchor text of external links anyway. So as long as some of your external links have your target anchors, you’re probably OK with a “Home” button. In fact, I’ve ranked homepages with a “Home” anchor text nav button for some seriously competitive terms. So it’s not a make-or-break ranking signal by any means.
Ask a marketer or business owner what they’d like most in the world, and they’ll probably tell you “more customers.” What often comes after customers on a business’ wish list? More traffic to their site. There are many ways you can increase traffic on your website, and in today’s post, we’re going to look at 25 of them, including several ways to boost site traffic for FREE.

Tablet - We consider tablets as devices in their own class, so when we speak of mobile devices, we generally do not include tablets in the definition. Tablets tend to have larger screens, which means that, unless you offer tablet-optimized content, you can assume that users expect to see your site as it would look on a desktop browser rather than on a smartphone browser.
Promoting your websites by publishing articles to various article directories is by no means a new idea but still an extremely effective way to drive traffic. If you write content and publish it to websites like Article Base, and Article Dashboard website owners will pick it up and post it. This idea is similar to guest blogging except that you only have to write one piece of content that can end up on hundreds of even thousands of blogs and websites. The same rule applies here: don’t be boring – be creative and interesting and use common keywords in your article and title so website owners can find it!
A breadcrumb is a row of internal links at the top or bottom of the page that allows visitors to quickly navigate back to a previous section or the root page. Many breadcrumbs have the most general page (usually the root page) as the first, leftmost link and list the more specific sections out to the right. We recommend using breadcrumb structured data markup28 when showing breadcrumbs.

Historical refreshes of content is a good thing, especially if some of your content has expired. Note, this does not mean re-doing your content; simply refreshing it to bring it current if it isn't already evergreen content. Look at ways you can update outdated content on your site to drive more traffic through visibility on search engines like Google.
Search engines use complex mathematical algorithms to interpret which websites a user seeks. In this diagram, if each bubble represents a website, programs sometimes called spiders examine which sites link to which other sites, with arrows representing these links. Websites getting more inbound links, or stronger links, are presumed to be more important and what the user is searching for. In this example, since website B is the recipient of numerous inbound links, it ranks more highly in a web search. And the links "carry through", such that website C, even though it only has one inbound link, has an inbound link from a highly popular site (B) while site E does not. Note: Percentages are rounded.
It’s not enough to just share content through social channels – you need to actively participate in the community, too. Got a Twitter account? Then join in group discussions with relevant hashtags. Is your audience leaving comments on your Facebook posts? Answer questions and engage with your readers. Nothing turns people off quicker than using social media as a broadcast channel – use social media as it was intended and actually interact with your fans.

Influencer marketing is a great way to get your message out to millions of people without having your own large audience. You can leverage influencers on Twitter, Instagram and any number of other social networks. Find people who are entrenched in your niche or industry and reach out to them. This won't be cheap, but it could potentially bring you droves of customers and sales in the near term.
I’ve just taken the SEO role at my agency full time and, whilst it can be difficult at times, I am liking the challenge. I wonder if you had any suggestions when it came to finding “opportunity keywords” for term/subjects that don’t necessarily have massive search volumes associated to them? I use a few tools and utilise Google’s related terms already, but wondered if there were any tricks for finding new markets?
It’s free to be active in online groups and on websites that are relevant to your business and community—and it helps you to obtain more traffic. Comment on blogs and social media posts, answer questions people are posting, and participate in conversations about your industry. The more you engage with your community, the more exposure and profile visits you get.
Headlines are one of the most important parts of your content. Without a compelling headline, even the most comprehensive blog post will go unread. Master the art of headline writing. For example, the writers at BuzzFeed and Upworthy often write upward of twenty different headlines before finally settling on the one that will drive the most traffic, so think carefully about your headline before you hit “publish.”
Influencer marketing is a great way to get your message out to millions of people without having your own large audience. You can leverage influencers on Twitter, Instagram and any number of other social networks. Find people who are entrenched in your niche or industry and reach out to them. This won't be cheap, but it could potentially bring you droves of customers and sales in the near term.

Hi Brian, Awsome content as ever! I’m very interested in your idea of creating an ‘uber’ resource list or expert roundup post i.e. linking out to lots of to other authorities in my niche within one post. But should you always create ‘no-follow’ links to these authority sites to prevent juice from passing to them? And similarly if you sprinkle a few outbound authority links in other posts should they all be ‘no-follow’ or do you think big G ignores ‘no follow’ these days?
Hey Brian, love your site + content. Really awesome stuff! I have a question about dead link building on Wikipedia. I actually got a “user talk” message from someone moderating a Wikipedia page I replaced a dead link on. They claimed that “Wikipedia uses nofollow tags” so “additions of links to Wikipedia will not alter search engine rankings.” Any thoughts here?
There were some great tips in this article. I notice that many people make the mistake of making too many distracting images in the header and the sidebar which can quickly turn people off content. I particularly dislike google ads anchored in the centre of a piece of text. I understand that people want to make a revenue for ads but there are right ways and wrong ways of going about this. The writing part of the content is the important part, why would you take a dump on it by pouring a load of conflicting media in the sides?
×