You may not want certain pages of your site crawled because they might not be useful to users if found in a search engine's search results. If you do want to prevent search engines from crawling your pages, Google Search Console has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain. For more information on robots.txt, we suggest this Webmaster Help Center guide on using robots.txt files13.
Make Call to Actions a value to your audience and keep them simple - We have all seen the fill-in and submit forms on sites that tend to end in an endless cycle of junk email and eventual un-subscription. People are becoming more and more weary of giving out their personal information so if you want a visitor to give something to you, offer them something in return. Maybe you offer a special, one-time coupon or access to information and education. With an offer of some sort of perceived value, visitors will feel more mutual about the transaction and may be more likely to convert to customers in the long run. Also, stick to name and email at the most for fill-in forms. While it is nice to have a lot of information about leads, giving out too much information can be a big turn-off for visitors.

In December 2009, Google announced it would be using the web search history of all its users in order to populate search results.[33] On June 8, 2010 a new web indexing system called Google Caffeine was announced. Designed to allow users to find news results, forum posts and other content much sooner after publishing than before, Google caffeine was a change to the way Google updated its index in order to make things show up quicker on Google than before. According to Carrie Grimes, the software engineer who announced Caffeine for Google, "Caffeine provides 50 percent fresher results for web searches than our last index..."[34] Google Instant, real-time-search, was introduced in late 2010 in an attempt to make search results more timely and relevant. Historically site administrators have spent months or even years optimizing a website to increase search rankings. With the growth in popularity of social media sites and blogs the leading engines made changes to their algorithms to allow fresh content to rank quickly within the search results.[35]


By relying so much on factors such as keyword density which were exclusively within a webmaster's control, early search engines suffered from abuse and ranking manipulation. To provide better results to their users, search engines had to adapt to ensure their results pages showed the most relevant search results, rather than unrelated pages stuffed with numerous keywords by unscrupulous webmasters. This meant moving away from heavy reliance on term density to a more holistic process for scoring semantic signals.[13] Since the success and popularity of a search engine is determined by its ability to produce the most relevant results to any given search, poor quality or irrelevant search results could lead users to find other search sources. Search engines responded by developing more complex ranking algorithms, taking into account additional factors that were more difficult for webmasters to manipulate. In 2005, an annual conference, AIRWeb (Adversarial Information Retrieval on the Web), was created to bring together practitioners and researchers concerned with search engine optimization and related topics.[14]
Use the right anchor text. Using our previous example: if you wanted to internally link to the “how to make money” blog post, you can write a sentence in another blog, like “Once you have mastered [how to make money], you can enjoy as much luxury as you can dream.” In this case, the reader has a compelling case for clicking on the link because of both the anchor text (“how to make money”) and the context of the sentence. There is a clear benefit from clicking the link.
Use clean backgrounds. The background textures and color you choose have the ability to drastically affect the overall appeal of the website. Lots of texture and graphics in the background can be distracting. If you are going to use a color on the background, you should make sure there is significant contrast between the background color and the text. Be careful when using brighter and darker colors such as red or yellow. They cause visual fatigue (temporary loss of strength and energy resulting from hard physical or mental work) and the reader will lose their focus on the text.
To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]
×