AngularJS SEO with Prerender.io

Introduction

AngularJS is an excellent framework for building websites and apps. Built-in routing, data-binding, and directives among other features enable AngularJS to completely handle the front-end of any type of application.

The one pitfall to using AngularJS (for now) is Search Engine Optimization (SEO). In this tutorial, we will go over how to make your AngularJS website or application crawlable by Google.

The Problem

Search engines crawlers (or bots) were originally designed to crawl the HTML content of web pages. As the web evolved, so did the technologies powering websites and JavaScript became the de facto language of the web. AJAX allowed for asynchronous operations on the web. AngularJS fully embraces the asynchronous model and this is what creates problems for Google’s crawlers.

If you are fully utilizing AngularJS, there is a strong possibility that you will only have one real HTML page that will be fed HTML partial views asynchronously. All the routing and application logic is done on the client-side, so whether you’re changing pages, posting comments, or performing other CRUD operations, you are doing it all from one page.

The Solution

Rest assured, Google does have a way of indexing AJAX applications, and your AngularJS app can be crawled, indexed, and will appear in search results just like any other website. There are a few caveats and extra steps that you will need to perform, but these methods are fully supported by Google. To read more about Google’s guidelines for crawlable AJAX content visit Google’s Webmaster AJAX Crawling Guidelines.

What We’ll Be Building

Our application will be able to be rendered by Google bot and all his friends (Bing bot). This way, we won’t run into the problem shown in the picture above. We’ll get nice search results as our users expect from us.

How It Works

• When a search engine crawler visits your app and sees the <meta name="fragment" content="!"> it will add an ?_escaped_fragment_= tag to your URL.
• Your server will intercept this request and send it to the middleware that will handle the special crawler request. For this article, we have chosen Prerender.io so the next step is specific to Prerender.io.
• Prerender.io will check to see if the requested page has an existing snapshot (or cached page), if it does, it will serve that page to the crawler, if it does not, Prerender will make a call to PhantomJS which will render the page in its entirety and show it to the crawler.
• Non-cached pages that require the call to PhantomJS will take longer to render leading to a much longer response time, so it’s a good idea to cache pages often.
• There are additional ways to do this!

Alternatives:

• Set up your own Prerender service using Prerender.io open-source code
• Use a different existing service such as BromBone, Seo.js or SEO4AJAX
• Create your own service for rendering and serving snapshots to search engines

About Prerender.io

Prerender.io is a service that is compatible across a variety of different platforms including Node, PHP, and Ruby. The service is fully open-source but they do offer a hosted solution if you do not want to go through the hassle of setting up your own server for SEO. The folks over at Prerender believe that SEO is a right, not a privilege and they have done some great work extending their solution, adding a lot of customizable features and plugins.

Node Setup package.json

We will be building a simple Node/AngularJS application that has multiple pages with dynamic content flowing throughout. We will use Node.js as our backend server with Express. Check out the Node package.json file below to see all of our dependencies for this tutorial. Once you are ready, sign up for a free prerender.io account and get your token.

    // package.json
    {
      "name": "Angular-SEO-Prerender",
      "description": "...",
      "version": "0.0.1",
      "private": "true",
      "dependencies": {
        "express": "latest",
        "prerender-node": "latest"
      }
    }

Now that we have our package.json ready to go, let’s install our Node dependencies using npm install.

Node Setup server.js

The setup here is pretty standard. In our server.js file we will require the Prerender service and connect to it using our prerender token.

    // server.js

    var express = require('express');

    var app = module.exports = express();

    app.configure(function(){
      // Here we require the prerender middleware that will handle requests from Search Engine crawlers
      // We set the token only if we're using the Prerender.io service
      app.use(require('prerender-node').set('prerenderToken', 'YOUR-TOKEN-HERE'));
      app.use(express.static("public")); app.use(app.router);
    });

    // This will ensure that all routing is handed over to AngularJS
    app.get('*', function(req, res){
      res.sendfile('./public/index.html');
    });

    app.listen(8081);
    console.log("Go Prerender Go!");

Main Page index.html

The main page is also pretty standard. Write your code like you normally would. The big change here will simply be adding <meta name="fragment" content="!"> to the <head> of your page. This meta tag will tell search engine crawlers that this is a website that has dynamic JavaScript content that needs to be crawled.

Additionally, if your page is not caching properly or it’s missing content you can add the following script snippet: window.prerenderReady = false; which will tell the Prerender service to wait until your entire page is fully rendered before taking a snapshot. You will need to set window.prerenderReady = true once you’re sure your content has completed loading. There is a high probability that you will not need to include this snippet, but the option is there if you need it.

That’s it! Please see the code below for additional comments.

    <!-- index.html -->
    <!doctype html> <!-- We will create a mainController and bind it to HTML which will give us access to the entire DOM -->
    <html ng-app="prerender-tutorial" ng-controller="mainController"> <head>

      <meta name="fragment" content="!">
      <!-- We define the SEO variables we want to dynamically update -->
      <title>Scotch Tutorial | {{ seo.pageTitle }}</title>
      <meta name="description" content="{{ seo.metaDescription }}">

      <!-- CSS-->
      <link rel="stylesheet" type="text/css" href="/assets/bootstrap.min.css">
      <style>
          body { margin-top:60px; }
      </style>

      <!-- JS -->
      <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.2.10/angular.min.js"></script>
      <script src="http://code.angularjs.org/1.2.10/angular-route.min.js"></script>
      <script src="/app.js"></script>

    </head>
    <body>
    <div class="container">

      <!-- NAVIGATION BAR -->
      <div class="bs-example bs-navbar-top-example">
          <nav class="navbar navbar-default navbar-fixed-top">
              <div class="navbar-header">
                  <a class="navbar-brand" href="/">Angular SEO Prerender Tutorial</a>
              </div>

              <ul class="nav navbar-nav">
                  <li><a href="/">Home</a></li>
                  <li><a href="/about">About</a></li>
                  <li><a href="/features">Features</a></li>
              </ul>
          </nav>
      </div>

      <h1 class="text-center">Welcome to the Angular SEO Prerender Tutorial</h1>
      <!-- where we will inject our template data -->
      <div ng-view></div>


    </div>
    </body>
    </html>

Angular Setup app.js

In our app.js, the page where we define our AngularJS code, we will need to add this code to our routes config: $locationProvider.hashPrefix('!');. This method will change the way your URLs are written.

If you are using html5Mode you won’t see any difference, otherwise, your URLs will look like http://localhost:3000/#!/home compared to the standard http://localhost:3000/#/home.

This #! in your URL is very important, as it is what will alert crawlers that your app has AJAX content and that it should do its AJAX crawling magic.

    // app.js

    var app = angular.module('prerender-tutorial', ['ngRoute'])

    .config(function($routeProvider, $locationProvider){

      $routeProvider.when('/', {
        templateUrl : 'views/homeView.html',
        controller: 'homeController'
      })

      $routeProvider.when('/about', {
          templateUrl : '/views/aboutView.html',
          controller: 'aboutController'
      })

      $routeProvider.when('/features', {
          templateUrl : '/views/featuresView.html',
          controller : 'featuresController'
      })

      $routeProvider.otherwise({
              redirectTo : '/'
      });

      $locationProvider.html5Mode(true);
      $locationProvider.hashPrefix('!');
    });

    function mainController($scope) {
      // We will create an seo variable on the scope and decide which fields we want to populate
      $scope.seo = {
        pageTitle : '', pageDescription : ''
      };
    }

    function homeController($scope) {
      // For this tutorial, we will simply access the $scope.seo variable from the main controller and fill it with content.
      // Additionally you can create a service to update the SEO variables - but that's for another tutorial.
      $scope.$parent.seo = {
        pageTitle : 'AngularJS SEO Tutorial',
        pageDescripton: 'Welcome to our tutorial on getting your AngularJS websites and apps indexed by Google.'
      };
    }

    function aboutController($scope) {
      $scope.$parent.seo = { pageTitle : 'About',
        pageDescripton: 'We are a content heavy website so we need to be indexed.'
      };
    }

    function featuresController($scope) {
      $scope.$parent.seo = { pageTitle : 'Features', pageDescripton: 'Check out some of our awesome features!' };
    }

In the above code, you can see how we handle Angular routing and our different pageTitle and pageDescription for the pages. These will be rendered to crawlers for an SEO-ready page!

So What Happens?

When a crawler visits your page at http://localhost:3000/#!/home, the URL will be converted to http://localhost:3000/?escaped_fragment=/home, once the Prerender middleware sees this type of URL, it will make a call to the Prerender service. Alternatively, if you are using HTML5mode, when a crawler visits your page at http://localhost:3000/home, the URL will be converted to http://localhost:3000/home/?escaped_fragment=.

The Prerender service will check and see if it has a snapshot or already rendered page for that URL, if it does, it will send it to the crawler, if it does not, it will render a snapshot on the fly and send the rendered HTML to the crawler for correct indexing.

Making Sure It Worked

Prerender provides a dashboard for you to see the different pages that have been rendered and crawled by bots. This is a great tool to see how your SEO pages are working.

Caveats

I recently got a chance to chat with the creator of Prerender.io and asked him for some tips on getting your single-page app indexed. This is what he had to say:

• Serve the crawlers prerendered HTML, not JavaScript,
• Don’t send soft 404s
• If you’re sticking with #s for your URLs, make sure to set the hashPrefix(‘!’) so that the URLs are rewritten as #!s
• If you have a lot of pages and content, be sure to include a sitemap.xml and robots.txt
• Google crawls only a certain number of pages per day, which is dependent on your PageRank. Including a sitemap.xml will allow you to prioritize which pages get indexed.
• When testing to see how your AngularJS pages render in Google Webmaster Tools, be sure to add the #! or ?escaped_fragment= in the right place as the manual tools do not behave exactly as the actual crawlers do.

Conclusion

Hopefully, you won’t let the SEO drawback of Angular applications hold you back from using the great tool. There are services out there like Prerender and ways to crawl AJAX content. Make sure to look at the Google Webmaster AJAX Crawling Guidelines and have fun building your SEO-friendly Angular applications!