tasty_var

Scalable content modeling for Contentful and Gatsby

@joonaskykkanen — Tue, 23 Feb 2021 00:00:00 GMT

This guide will tell you how to create a scalable and flexible content model with Contentful and Gatsby.

Contentful provides a flexible and loved headless content backend for many Gatsby applications. Getting started with Gatsby and Contentful is easy with the official gatsby-source-contentful plugin, and editing content types and adding new content is a breeze using Contentful's web app.

However, it's also easy for teams to paint themselves into a corner when they introduce new content types without thinking about the overall architecture of their content model.

The purpose of this guide is to describe a technique to model your content in a way that's scalable and allows content editors to not only add new blog posts but also build new landing pages without the help of developers. The architectural pattern that is presented here could be described as a block-based design or block-based architecture.

Sidenote: I originally learned about blocks from a presentation given by one of Contentful's own developers. Back then I had finished my first Contentful project where we had applied a somewhat block-based architecture but lacked a proper term to describe what we were trying to achieve.

The five categories of content types

Your content types are all going to fall into the following five categories:

Applications
Instead of multiple application content types, you are likely to have just one application content type and one entry where you hold all the general editable config of your site. Multi-site setups have multiple entries of the application content type but not multiple application content types (unless needed).

Layouts
The layout content type allows content editors to define the repeating elements of different pages (such as the navbar and footer). This is an optional layer of design since while many sites have multiple page types, they often have just one layout which means the repeating elements can be "hard coded" inside Gatsby.

Pages
A page content type is almost like a page template. Each page type gets its own content type.

Blocks
Blocks are the different headers, sections and other UI elements that content editors can add on layouts and pages.

Values
Value content types take a group of fields and extract them into new concepts that can be reused across the site. A typical value content type is an author with name and image that blog post entries can link to.

In the next chapters I'll show you what these different content types look like in practice. Instead of using Contentful's web UI to create the content types, I will use Contentful's migration scripts that can be run using the Contentful CLI.

You can apply the same changes using Contentful's web app but the migration files make it easier to describe the different types and fields in written form. In addition, if you aren't using migrations yet, I would really recommend you to look into them; once you learn how to write Contenful migration scripts, you will keep using them especially when you need to refactor something in your current content model.

You can find all the migration scripts presented in this guide in their full form here. To read more about how to do migration scripting, see this tutorial.

Application content types

There's usually some general site config that you want content editors to be able to change by themselves. For example, we might want content editors to be able to change the base meta title of our meta title template or the site logo. In order to achieve this, we need to have some place where we can store this information that doesn't belong to any specific site.

To add a new content type to our content model, we create a new migration file. Contentful's migration files all export a function that takes a migration object as its parameter:

module.exports = function (migration) {};

We create a content type for the application with id set to application, name set to Application, and display field (or entry title field) set to name field.

module.exports = function (migration) {
  const application = migration
    .createContentType("application")
    .name("Application")
    .displayField("name");
};

Let's add the name field that we are using as the display field:

module.exports = function (migration) {
  // ...

  const name = application.createField("name");
  name.name("Name").type("Symbol").required(true);
};

The other fields in your application content type will be very specific to the needs of your site's setup. But let's add that base meta title as an example to our application content type:

module.exports = function (migration) {
  // ...

  const baseMetaTitle = application.createField("baseMetaTitle");
  baseMetaTitle.name("Base Meta Title").type("Symbol").required(true);
};

After creating the application content type, create a new entry for it and add the initial config by filling out the content fields.

Block content types

Our layout and page content types require us to have some block content types ready. Because of this, I'm showing you how to create the block content types at this point—before creating any layout or page content types.

When building a site, you are probably not able to add all the block content types needed prior to adding any layout and page content types. This is okay; you can keep adding new blocks and updating block fields in your layout and page content types accordingly.

Generic and specific blocks

If you create a separate block content type for each one of your page elements, you might run into your tier's content type limits. In addition to this, having tons of different block content types can make it difficult for content editors to remember what each content type does and teach others in their team how to manage the content.

One of the most effective ways of keeping the number of block content types under control is to identify which UI elements get constantly reused and refurbished with new content and which elements might get reused but don't necessarily see any changes in their content or other config.

An example of a UI element that get reused and refurbished often is a hero block that contains a heading and a background image. Each landing page usually has one but the text and image inside the heroes changes based on the purpose of the landing page.

An example of a UI element that might get added on every page, but doesn't change much in terms of its content, is a footer block. Footers aren't usually modified to fit specific pages but instead contain the same privacy policy links, copyright notices, and other info from one page to another.

What we want is to have a specific content type for hero blocks but a generic content type for footer blocks. Let's start with the migration script for our example hero block.

Adding specific block content types

As an example of the specific, non-generic block content type, let's add a hero block to our content model.

We start by creating a new content type heroBlock with a name field:

module.exports = function (migration) {
  const heroBlock = migration
    .createContentType("heroBlock")
    .name("Hero Block")
    .displayField("name");

  const name = heroBlock.createField("name");
  name.name("Name").type("Symbol").required(true);
};

Our application content type had also a name field that was used as the content's entry title. I have found it to be a good practice to give every content type a name field that allows content editors to name different blocks or pages in a way that makes it easy for them to find specific entries from a long list of content.

To complete our hero block, let's add fields for the hero heading and image:

module.exports = function (migration) {
  // ...

  const heading = heroBlock.createField("heading");
  heading.name("Heading").type("Symbol").required(true);

  const image = heroBlock.createField("image");
  image
    .name("Image")
    .type("Link")
    .linkType("Asset")
    .validations([{ linkMimetypeGroup: ["image"] }]);
};

Adding the generic block content type

As mentioned earlier, in order to keep our content model more manageable for content editors, we don't necessarily want to create a new block content type for each UI element.

The generic block content type is a content type that can be used to add individual entries for different types of UI elements. Each entry of the generic content type is mapped to a specific React component.

Because generic block entries are mapped to React components, these block entries are added by developers and not content editors. In other words, a content editor can't add a new block entry for a generic block content type without the help of a developer.

However, as mentioned before, it's not likely that content editors need to add for example a second footer or a second navbar to the site. It's often enough that the blocks for the footer and navbar exist as entries and that the content editors can link to them inside different layouts and pages.

It's still possible for content editors to add a second footer using the generic block content type without developers if they really need to. It's just not going to be as user-friendly as it is to add a second hero block.

Let's start building a new migration script for the generic block content type by creating the content type and adding a name field to it:

module.exports = function (migration) {
  const block = migration
    .createContentType("block")
    .name("Block")
    .displayField("name");

  const name = block.createField("name");
  name.name("Name").type("Symbol").required(true);
};

We add a field for the React component and include some validation logic to check that the field value is a valid React component name:

module.exports = function (migration) {
  // ...

  const component = block.createField("component");
  component
    .name("Component")
    .type("Symbol")
    .required(true)
    .validations([{ regexp: { pattern: "^[A-Z][a-zA-Z0-9]*$" } }]);
};

Finally, we add a field for the component properties. These properties are saved as a JSON object.

module.exports = function (migration) {
  // ...

  const properties = block.createField("properties");
  properties.name("Properties").type("Object");
};

Component properties allow content editors to change things like marketing copy or color theme. Editing JSON is not going to be as easy as editing content fields but the assumption is that editors rarely need to do this type of editing.

Deciding which blocks get their own content type and which don't seems to be a balancing act between different concerns. Remember that having a content type for each one of your UI elements can make your content model more confusing to use or cause issues with the content type limits of your current Contenful plan.

Layout content types

The layout content type is going to enable you to extract some of the repeating page elements into their own entries. This is an optional layer of the content model design and it might be enough for you to simply hardcode navbars and footers to your pages inside Gatsby.

If you do want to add layouts to your content model, you need only one content type. Let's create this content type by creating a layout content type with a name field:

module.exports = function (migration) {
  const layout = migration
    .createContentType("layout")
    .name("Layout")
    .displayField("name");

  const name = layout.createField("name");
  name.name("Name").type("Symbol").required(true);
};

Layouts contain the blocks that are repeated from page to page. These blocks are placed before or after the actual content of a given page. We add separate fields for these before and after areas:

module.exports = function (migration) {
  // ...

  const before = layout.createField("before");
  before
    .name("Blocks added before page content")
    .type("Array")
    .items({
      type: "Link",
      linkType: "Entry",
      validations: [{ linkContentType: ["heroBlock", "block"] }],
    });

  const after = layout.createField("after");
  after
    .name("Blocks added after page content")
    .type("Array")
    .items({
      type: "Link",
      linkType: "Entry",
      validations: [{ linkContentType: ["heroBlock", "block"] }],
    });
};

Notice that we have to list the IDs of our block content types when defining the validation logic for both the before and after fields. By validating the content type of each linked item we can make sure that content editors are not able to add any other content besides blocks to these fields.

These lists of valid block IDs can be updated every time you add a new block content type.

Page content types

Each page content type could be thought of as a separate page template. In this guide, we will create one page content type for articles and one default page content type that can be used for things like the homepage and the about page.

Adding a default page content type

Let's start with the default page content type. We create a migration that creates the page content type and adds the name field to it:

module.exports = function (migration) {
  const page = migration
    .createContentType("page")
    .name("Page")
    .displayField("name");

  const name = page.createField("name");
  name.name("Name").type("Symbol").required(true);
};

Next, we create a field for the page slug (slug is a part of the URL path that is used to identify different pages from each other). We also change the appearance of this field to a slug field control:

module.exports = function (migration) {
  // ...

  const slug = page.createField("slug");
  slug
    .name("Slug")
    .type("Symbol")
    .required(true)
    .validations([{ unique: true }]);
  page.changeFieldControl("slug", "builtin", "slugEditor");
};

We add a layout field to link each page to a specific layout:

module.exports = function (migration) {
  // ...

  const layout = page.createField("layout");
  layout
    .name("Layout")
    .type("Link")
    .linkType("Entry")
    .required(true)
    .validations([{ linkContentType: ["layout"] }]);
};

Finally, we add a field for the actual content of the page which in our case consists of different blocks. As is the case with the layout content type, we have to make sure that content editors can only add blocks (and not other content) to the field by listing all the IDs of the available block content types inside our item validations:

module.exports = function (migration) {
  // ...

  const blocks = page.createField("blocks");
  blocks
    .name("Blocks")
    .type("Array")
    .items({
      type: "Link",
      linkType: "Entry",
      validations: [{ linkContentType: ["heroBlock", "block"] }],
    });
};

Adding an article page content type

We can add another page content type for article pages to demonstrate how we might structure other, more specific page content types.

We first create the content type itself and add the same name, slug, and layout fields that we added to the default page content type:

module.exports = function (migration) {
  const articlePage = migration
    .createContentType("articlePage")
    .name("Article Page")
    .displayField("name");

  const name = articlePage.createField("name");
  name.name("Name").type("Symbol").required(true);

  const slug = articlePage.createField("slug");
  slug
    .name("Slug")
    .type("Symbol")
    .required(true)
    .validations([{ unique: true }]);
  articlePage.changeFieldControl("slug", "builtin", "slugEditor");

  const layout = articlePage.createField("layout");
  layout
    .name("Layout")
    .type("Link")
    .linkType("Entry")
    .required(true)
    .validations([{ linkContentType: ["layout"] }]);
};

Instead of the blocks field, we want to add a rich text field for the article content. This field will make it easier for content editors to create article pages that consist mainly of written text with different heading levels and paragraphs.

We can call this field body:

module.exports = function (migration) {
  // ...

  const body = articlePage.createField("body");
  body.name("Body").type("RichText");
};

We still want to give editors the option to add blocks to specific article pages if they want to. To achieve this, we add before and after fields to the article page content type. Both of these fields are used to link specific blocks to the page. The before field is reserved for blocks that should be displayed before the body text and the after field for blocks that should be displayed after the body text:

module.exports = function (migration) {
  // ...

  const before = articlePage.createField("before");
  before
    .name("Blocks added before body text")
    .type("Array")
    .items({
      type: "Link",
      linkType: "Entry",
      validations: [{ linkContentType: ["heroBlock", "block"] }],
    });

  const after = articlePage.createField("after");
  after
    .name("Blocks added after body text")
    .type("Array")
    .items({
      type: "Link",
      linkType: "Entry",
      validations: [{ linkContentType: ["heroBlock", "block"] }],
    });
};

Value content types

You may want to abstract some of the content into their own content types so that you can reuse them inside other content entries. For example, if we want to add authors to our article pages, we should create a new author content type so that author content entries can be linked to multiple different articles.

These value content types are not blocks because they don't really correspond to an UI element that content editors can add on their pages between other blocks. Value content types simply take some set of fields and combine them to form a new concept to our content model.

As an example, let's add authors to our article pages.

First, we create the author content type with author name and avatar:

module.exports = function (migration) {
  const author = migration
    .createContentType("author")
    .name("Author")
    .displayField("name");

  const name = author.createField("name");
  name.name("Name").type("Symbol").required(true);

  const avatar = author.createField("avatar");
  avatar
    .name("Avatar")
    .type("Link")
    .linkType("Asset")
    .validations([{ linkMimetypeGroup: ["image"] }]);
};

Next, we need to add a field to our article page that links to author entries. Let's edit the existing article page content type inside the current migration script:

module.exports = function (migration) {
  // ...

  const articlePage = migration.editContentType("articlePage");
  const authorField = articlePage.createField("author");
  authorField
    .name("Author")
    .type("Link")
    .linkType("Entry")
    .validations([{ linkContentType: ["author"] }]);
};

Conclusion

This guide laid out a way for you to build a scalable and flexible content model for a site built with Gatsby and Contentful. I hope you found it helpful. If so, you can also help others by sharing this guide on Twitter or wherever else.

You can find all the migration files presented in this guide here.

Deploying a Ruby on Rails 6 application on Google App Engine Standard with Travis CI

@joonaskykkanen — Tue, 27 Oct 2020 00:00:00 GMT

This guide will walk you through how to deploy a Ruby on Rails 6 application with a PostgreSQL database on Google App Engine standard environment using a Travis CI pipeline.

App Engine standard vs. flexible environment

Google added Ruby support for App Engine standard environment in August 2019. Before this, Rails developers could only deploy their apps on App Engine flexible environment.

If you are currently running a Rails application using the flexible environment, you might want to consider switching to the standard environment if you want to bring down your deployment times (deployment times are around 4-7 minutes on the flexible environment and 1-3 minutes on the standard environment) or use scale-to-zero (instances can be scaled down when no one is using them).

As of September 2020, App Engine standard environment supports only Ruby 2.5. If downgrading or upgrading to 2.5 is not possible for you, you are not able to deploy your app on App Engine standard environment.

If you are starting a new Rails application or moving your application for example from Heroku to App Engine, it's likely that the standard environment will be your pick. However, there are lots of other technical differences between the two environments that you might need to take into consideration in the context of your project. For a more in-depth comparison of App Engine standard vs. flexible environment, see GCP's article Choosing an App Engine environment.

Prerequisites

This guide assumes that you have a working Rails project installed on your computer and that you have a user account in the following services:

Signing up for all of these services is free. However, as of September 2020 GCP will charge you for your PostgreSQL instance. It's important to clean up your GCP resources after finishing this guide if you don't want these costs to continue running. This guide has separate instructions on how to do this and you can find them in chapter Clean-up.

If you don't have Rails installed or a working Rails application on your computer, GoRails has excellent guides on how to install and start a new Rails application on Ubuntu, macOS, or Windows. You can find them here.

Setting up Travis CI

Before setting up a new App Engine application, we will initialize our Travis CI pipeline.

If you have signed in to Travis CI using your GitHub account, all you need to do to start your Travis CI builds is to add the .travis.yml file to your project root and push the changes to your GitHub repository. Travis CI will start a new build automatically on each push.

Configuring Travis CI

Travis CI pipelines are configured using a YAML file that lives inside your project root directory.

Create a new file called .travis.yml inside your project's root directory and add the following content to it (make sure to call your file .travis.yml and not .travis.yaml):

dist: xenial
language: ruby
cache:
  bundler: true
  yarn: true
  directories:
    - "node_modules"
services:
  - postgresql
install:
  - bundle install --jobs=3 --retry=3 --deployment
  - nvm install
  - bin/yarn
before_script:
  - bin/rails db:prepare
script:
  - bin/rails test

Let's briefly go through these different lines.

dist: xenial and language: ruby tells Travis CI to use the Ubuntu Xenial and Ruby build environments.

cache tells Travis CI to cache our dependencies for Bundler (bundler: true) and Node.js (yarn: true and "node_modules" under directories).

Under services we add postgresql to run a PostgreSQL instance during our build. We need this database instance to run tests (and later on in the guide to precompile assets). This database instance is not our production database. It's just a temporary instance that gets thrashed every time our Travis CI build finishes.

During the install phase we install all our dependencies: First, we install our Bundler dependencies with Travis CI's default bundler command bundle install --jobs=3 --retry=3 --deployment. After that, we instruct Node Version Manager to install the correct Node.js version with nvm install. Finally, we run the command bin/yarn to install Node.js dependencies.

In our before_script step we prepare our database for the tests. The command bin/rails db:prepare was introduced in Rails 6 and it either runs migrations or creates the database, loads the schema, and seeds the database if no database exists.

Sidenote: Travis CI doesn't store our database between builds and therefore bin/rails db:prepare will always create a new database and seed it. This means that we could easily replace bin/rails db:prepare with bin/rails db:setup. But I thought that this is a great opportunity to get more familiar with this new Rails 6 database task.

Finally, we run our tests to make sure everything is working during the script phase with the command bin/rails test.

Specifying Ruby and Node.js versions

We could specify the required Ruby and Node.js versions inside .travis.yml (see Specifying Ruby versions and Specifying Node.js versions) but we will instead specify the required versions by adding .ruby-version and .nvmrc files inside our project root. Travis CI is able to pick up the correct Ruby and Node.js versions from these files.

Why add these config files? It's likely that other developers working on the project are using Ruby Version Manager (RVM) (or rbenv) and Node Version Manager (nvm) to manage their Ruby and Node.js versions. Just like Travis CI, these version managers are able to read the version files automatically and instruct your fellow developers to install and use the correct language versions while working on the project.

To add the .ruby-version file, run

$ echo "ruby-2.5.8" > .ruby-version

Sidenote: Remember that as of September 2020 App Engine standard environment supports only Ruby 2.5.

In addition to specifying a Ruby version with a .ruby-version file, we need to make sure our Gemfile doesn't restrict the Ruby version to a specific version. This is because App Engine might update the patch version.

Inside your Gemfile, define the required Ruby version as:

ruby '~> 2.5.0'

Finally, let's define our Node.js version. To add the .nvmrc file with the latest LTS version (as of September 2020), run

$ echo "12.18.4" > .nvmrc

First Travis CI build

Save the .travis.yml file and commit your changes by running

$ git add .travis.yml
$ git commit -m "Add .travis.yml"

After the commit, push your changes to GitHub by running

$ git push origin master

If you go to https://travis-ci.com/, you should see your new build appear. If your build errors, double check your typing and look for clues inside the build logs to figure out what's causing the build to error. If your build doesn't appear on Travis CI even after waiting for couple of minutes, make sure you have correctly named the config file .travis.yml: the filename begins with a dot and its extension is .yml – not .yaml.

Setting up App Engine

Creating a new App Engine application

Before we can add a deploy step to our CI/CD pipeline, we need to initialize our application on App Engine.

First, log in to the Google Cloud console at https://console.cloud.google.com/. Select or create a new project in the Cloud console. Make sure that billing is enabled for your project.

After that, go to App Engine. Create a new application in a region that suits your needs. After selecting the region, you can optionally select the language and environment for your application. Our application's language will be Ruby and environment Standard.

Instead of using the Cloud console, you can also create the App Engine application using the Google Cloud SDK by running

$ gcloud app create

For more information on how to install the Cloud SDK on your computer, see https://cloud.google.com/sdk/docs/quickstarts.

Configuring the App Engine application

App Engine applications are configured using an app.yaml file which is located inside the root directory of your project.

Create the app.yaml file and add the following to it:

runtime: ruby25
entrypoint: bundle exec rails server -p $PORT
env_variables:
  RAILS_ENV: production

Let's go through these lines.

runtime defines the runtime environment for your App Engine application. As of September 2020, the only available standard environment Ruby runtime is ruby25 (Ruby 2.5).

entrypoint is the command that starts your application. The value for the environment variable PORT is set up by App Engine on our behalf.

Finally, env_variables allows us to define custom environment variables for our instance.

Important: don't add any sensitive information (such as passwords or API keys) inside your app.yaml file.

In this guide we will commit app.yaml to our git history and push it to GitHub. If your GitHub repository is public, your unencrypted passwords and API keys specified inside app.yaml are available to everyone who stumbles across your repository. Even if your GitHub repository is private, committing sensitive information in your git history without any kind of encryption is not a good practice.

We will discuss more about managing sensitive information in chapter Configuring database connections.

Choosing which files to deploy to App Engine

When you deploy a new version of your application to App Engine, the gcloud command-line tool (part of the Google Cloud SDK) uploads your current project files to App Engine where a new version of your application gets installed. You can't deploy a new version directly from GitHub. Instead, you need to do your deployment from an environment that has Google Cloud SDK installed. This environment can be your computer or a virtual machine such as Travis CI.

If we don't upload our files directly from git history but instead from our current project directory, we will upload lots of extra files that App Engine doesn't really need. These extra files include for example our node_modules and tmp folders.

gcloud does ignore some files during App Engine deployments by default. However, this default ignore list will actually contain files that we do want to upload to App Engine. These "ignored but needed" files are our precompiled assets which we will get to in chapter Preparing assets.

Luckily, we can add a .gcloudignore file inside the root directory of our project. This file will tell gcloud what files to skip when uploading project files to App Engine. .gcloudignore will also override gcloud's default ignore list.

If you are familiar with the syntax of .gitignore, you should have no problem adding new rules to .gcloudignore: the .gcloudignore syntax is heavily inspired by .gitignore. To read more about .gcloudignore, see https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore.

Here is an example .gloudignore file for a new Rails 6 project:

# Ignore itself
.gcloudignore

# Ignore git files
.git
.gitignore

# Ignore gems installed by Travis CI
/vendor/bundle

# Ignore other local Rails files
/.bundle
/log/*
/tmp/*
!/log/.keep
!/tmp/.keep
/tmp/pids/*
!/tmp/pids/
!/tmp/pids/.keep
/storage/*
!/storage/.keep
.byebug_history
/public/packs-test
/node_modules
/yarn-error.log
yarn-debug.log*
.yarn-integrity

If you have added new rules to your .gitignore file, you should add them also to .gcloudignore.

Our .gitignore and .gcloudignore are not one-to-one in terms of ignore rules:

.gcloudignore ignores itself (.gitignore) and git history (.git and .gitignore).
.gcloudignore doesn't ignore /public/assets and /public/packs. These directories contain our precompiled assets that we need to upload to App Engine.
.gcloudignore doesn't ignore /config/master.key. This is the key that allows us to access different application credentials on App Engine. Important: never commit this file to your git history (keep it in .gitignore)!
Travis CI will install gems in /vendor/bundle inside our project. If we don't ignore this directory, we will end up uploading extra copies of our gems to App Engine.

If at any point gcloud errors because your deployment has too many files, you should double check your .gcloudignore file. When debugging this error, you might find it also helpful to view the current deployed source code by navigating to App Engine > Dashboard > Debug in Cloud console and seeing if your source contains any files that shouldn't be there.

Adding gcloud to Travis CI

Installing Google Cloud SDK

As mentioned before, we need the gcloud command-line tool to deploy our application to App Engine. The gcloud command-line tool is part of the Google Cloud SDK.

Since we want to deploy the application using our CI/CD pipeline, we need to install the Google Cloud SDK for our Travis CI virtual machine. We will also cache this install to make our builds faster.

First, add a new cached directory for Google Cloud SDK inside .travis.yml:

cache:
    bundler: true
    yarn: true
    directories:
        - "node_modules"
        - "$HOME/google-cloud-sdk/"

After that, add a new before_install step before the install step inside .travis.yml:

before_install:
    - if [ ! -d ${HOME}/google-cloud-sdk ]; then
            curl https://sdk.cloud.google.com | bash;
        fi

This command checks if an existing Google Cloud SDK installation exists in our virtual machine and installs it if needed.

Authorizing gcloud to access Google Cloud Platform

Our Travis CI pipeline now has gcloud installed but it's not authorized to access our Google Cloud project. In order to authorize Travis CI to run our App Engine deployments, we need to add a service account for it.

Inside Cloud console, go to IAM & Admin > Service Accounts and select Create service account. You can give whatever name and ID you want for the service account (e.g. I have named my service account "Travis CI" and given it the ID "travis-ci"). After that, add the roles App Engine Admin, Cloud Build Service Account, and Storage Admin to the service account.

Create new credentials for the service account by selecting the account from the list of service accounts and then selecting Add key > Create new key under the Keys section. When asked for the key type, choose JSON.

The credentials are automatically downloaded on your computer. Rename the downloaded file as service-account.json and move it to your project directory.

Important: Do not commit this file to your git history. To be extra sure you won't accidentally do this now or in the future, add the following line to your .gitignore:

service-account.json

Feel free to also delete service-account.json from your computer after we have finished encrypting it (don't delete it yet).

In order to safely upload these credentials to GitHub and Travis CI, we need to encrypt them using the Travis CI Command Line Client.

You can install the client by running:

$ gem install travis

After installing the client, authorize it with Travis CI by running:

$ travis login --com

To read more about the Travis Client, see https://github.com/travis-ci/travis.rb.

Encrypt the credentials file by running:

$ travis encrypt-file service-account.json --com

Sidenote: the --com option tells the client to use travis-ci.com and not travis-ci.org. All new users and projects should only use travis-ci.com.

The output of the encrypt-file command will contain instructions on how to also decrypt service-account.json. Copy the decrypt command from the instructions and add it as the first command in the before_install step inside .travis.yml:

openssl aes-256-cbc -K $encrypted_ENCRYPTION_ID_key -iv $encrypted_ENCRYPTION_ID_iv -in service-account.json.enc -out service-account.json -d

Finally, we can authorize gcloud by adding the following line as the last before_install step command:

gcloud auth activate-service-account --key-file service-account.json

Here is what our final before_install step should look like:

before_install:
    - openssl aes-256-cbc -K $encrypted_9f3b5599b056_key -iv $encrypted_9f3b5599b056_iv -in service-account.json.enc -out service-account.json -d
    - if [ ! -d ${HOME}/google-cloud-sdk ]; then
            curl https://sdk.cloud.google.com | bash;
        fi
    - gcloud auth activate-service-account --key-file service-account.json

Enabling App Engine APIs

In addition to creating and authorizing a service account to deploy our App Engine application, we need to enable few APIs using the Cloud console.

Start by navigating to APIs & Services > Library in the cloud console.

The APIs that we need to enable are:

App Engine Admin API
Cloud Build API

Search for each of these APIs on the library page and enable them for your project.

Connecting Cloud SDK to the Cloud project

Each App Engine application lives inside a Cloud project. We need to tell gcloud which Cloud project we are using to run our application in.

Add the following command to the install step in .travis.yml (replace MY_PROJECT with your project ID):

gcloud config set project MY_PROJECT

Here is what our final install step should look like:

install:
    - gcloud config set project MY_PROJECT
    - bundle install --jobs=3 --retry=3 --deployment
    - nvm install
    - bin/yarn

Deploying the app to App Engine

Before we can deploy the first version of our app to App Engine, we need to share our master.key securely with App Engine and precompile our assets. After that, we are ready to deploy our app.

Managing secrets

If you have previously used services like Heroku to deploy your Rails applications, you have probably managed sensitive information such as passwords and API keys using environment variables and setting them using an admin panel.

App Engine doesn't provide us with such a GUI for setting environment variables. Instead, you define all your environment variables inside app.yaml which in our case gets committed to our git history and shared to GitHub (possibly for the whole world to see if our repo is public).

Fortunately, Rails comes with the credentials feature that allows us to encrypt our secrets for safe sharing. Secrets are stored inside config/credentials.yml.enc and encrypted/decrypted using config/master.key (which is ignored by git by default). To read more about credentials, see the official documentation.

config/master.key is a highly sensitive file. Anyone who has access to your master key, will have access to your encrypted credentials. In order to safely upload the config/master.key file to App Engine, we are going to encrypt it the same way we encrypted the service-account.json file earlier:

$ travis encrypt-file config/master.key config/master.key.enc --com

Again, the terminal output will contain the command we need to run to decrypt config/master.key. Copy the command and add it as the first command to a new step called before_deploy in .travis.yml:

before_deploy:
    - openssl aes-256-cbc -K $encrypted_ENCRYPTION_ID_key -iv $encrypted_ENCRYPTION_ID_iv -in config/master.key.enc -out config/master.key -d

Preparing assets

In order to precompile production assets, add the following command to before_deploy in .travis.yml:

RAILS_ENV=production bin/rails assets:precompile

To learn more about how prepare your asset pipeline for production, see the official asset pipeline documentation.

Here is what our final before_deploy step should look like:

before_deploy:
    - openssl aes-256-cbc -K $encrypted_8ad82cc635a3_key -iv $encrypted_8ad82cc635a3_iv -in config/master.key.enc -out config/master.key -d
    - RAILS_ENV=production bin/rails assets:precompile

Writing the deploy script

Travis CI has a ready-made App Engine integration that we could use to deploy our Rails app (read more about it here). However, this integration will install a second version of Google Cloud SDK instead of using our existing, cached version. In order to keep our Travis CI builds fast, we will write our own deploy script and make use of our existing Google Cloud SDK installation instead.

Why do we need our own Google Cloud SDK version? Later on in this guide we will start running migration tasks on App Engine using the appengine gem. This step requires our own gcloud installation.

Add a new file deploy to the bin directory with the following content:

#!/bin/bash
gcloud -q app deploy app.yaml

This gcloud command deploys a new version to App Engine using the app.yaml config file. The -q flag is there to disable user prompts.

Make this file executable by running:

$ chmod +x ./bin/deploy

Add a new deploy step to .travis.yml:

deploy:
  provider: script
  script: ./bin/deploy
  skip_cleanup: true
  on:
    branch: master

provider: script tells Travis CI to use a custom deploy script. script: ./bin/deploy tells Travis CI where to find this deploy script. skip_cleanup: true prevents Travis CI from removing the files generated during the previous steps (mainly config/master.key and precompiled assets). branch: master prevents this deploy step from being run on any other branch than our main branch.

First deployment

Commit your changes and push them to GitHub. If you go see your build at https://travis-ci.com/ you should see it running successfully. See the log from gcloud's deploy command to see where you can preview your app (the URL will have the format of YOUR_PROJECT_ID.YOUR_REGION_ID.r.appspot.com).

If you go preview your app right now, you will probably encounter an error page. This is because your app is still missing a database. Let's set up one in the next chapter.

Setting up PostgreSQL with Cloud SQL

Creating a database

Cloud SQL is GCP's product name for its managed database instances. This guide assumes you are using PostgreSQL but it's also possible to use Cloud SQL with MySQL and SQL Server.

In the Cloud console, navigate to SQL and click Create instance. Select PostgreSQL and continue to the creation form.

Choose an ID for your database instance (this ID will be public) and a password for the default postgres user. Save the password somewhere safe for later use.

Select a region that's closest to your App Engine region (according to Google Cloud's documentation, it's okay to leave the zone as Any in most cases).

Finally, choose a PostgreSQL version you want to use and click Create.

You can also configure things like your backup strategy and machine type using this form but this beyond the scope of this guide; the default configuration should work for you just fine especially if you are working with a demo app.

If you are experiencing problems setting up Cloud SQL or if you want to create a PostgreSQL instance using gcloud, you can check out Google Cloud's documentation on creating PostgreSQL instances here.

Wait for GCP to create your new database instance and after that, select it from the list of instances. Click Databases and Create database.

Give the database a name such as myapp_production and click Create.

Configuring database connections

We need to tell our Rails app how to connect to our new PostgreSQL instance. Part of this configuration contains sensitive information (the password for the postgres user) and we need to keep it private since we are uploading it to our possibly public GitHub repo.

As mentioned in chapter Managing secrets, we can store encrypted secrets inside config/credentials.yml.enc which we can then commit to our git history.

To add new secrets, open the credentials file by first running:

$ bin/rails credentials:edit

Add the username and password of your PostgreSQL instance to the file and save it:

db:
  username: postgres
  password: <YOUR PASSWORD>

Rest of the configuration contains less sensitive information that we can include in app.yaml as is.

Open app.yaml and add two new environment variables, DATABASE_NAME and DATABASE_HOST, to it:

runtime: ruby25
entrypoint: bundle exec rails server -p $PORT
env_variables:
  RAILS_ENV: production
  DATABASE_NAME: YOUR_DATABASE_NAME
  DATABASE_HOST: /cloudsql/YOUR_INSTANCE_CONNECTION_NAME

You can find the instance connection name of your PostgreSQL instance in Cloud console by navigating to SQL and finding the column Instance connection name from the instances list. The instance connection name should have the format of PROJECT:REGION:INSTANCE_NAME.

Now that we have credentials and environment variables set up, we can finally add them to our database configuration.

Open config/database.yml and add the following production configuration to it:

production:
  <<: *default
  database: <%= ENV["DATABASE_NAME"] %>
  host: <%= ENV["DATABASE_HOST"] %>
  username: <%= Rails.application.credentials.db.fetch(:username) %>
  password: <%= Rails.application.credentials.db.fetch(:password) %>

At this point we need to also share our config/master.key with Travis CI so that it will be able to read our database credentials during database config initialization. Travis CI won't actually use the production config, but it will try to parse it nonetheless.

Log in to Travis CI and select your project. Navigate to the project settings and add a new environment variable RAILS_MASTER_KEY with the value of the contents of your config/master.key file.

Commit your changes to config/credentials.yml.enc, app.yaml and config/database.yml, and push them to GitHub.

Running migrations

In addition to creating a database in Cloud SQL and establishing a connection to it, we need to be able to run our migration files as we keep developing our Rails application.

Using the appengine gem, we can run migration tasks on Google Cloud instead of connecting your computer to the production database and running them from your local environment.

First, add appengine to your Gemfile:

gem 'appengine'

Install the dependency by running

$ bundle

After that, add a migration script to our before_deploy step inside .travis.yml:

before_deploy:
    - openssl aes-256-cbc -K $encrypted_8ad82cc635a3_key -iv $encrypted_8ad82cc635a3_iv -in config/master.key.enc -out config/master.key -d
    - RAILS_ENV=production bin/rails assets:precompile
    - bundle exec rake appengine:exec -- bin/rails db:migrate

You can now try to add a new migration file and deploy your application.

Sidenote: At the time of writing, my migrations fail every now and then when I use this setup. However, whenever I restart a failed build on Travis, the migrations run successfully. I haven't unfortunately been able to figure out what's causing these occasional fails.

Finishing up

We have now successfully deployed a Rails 6 application on Google App Engine Standard. Below you will find the full Travis CI file for reference. You can also find an example project running this setup at https://github.com/jonaskay/rails-app-engine-standard.

But before we finish up, let's clean up our GCP resources so that we won't end up with surprise costs.

Clean-up

The safest way to make sure that you have deleted all the billable resources is to delete your Cloud project.

You can find the official documentation on how to do this here.

Full .travis.yml file

dist: xenial
language: ruby
cache:
  bundler: true
  yarn: true
  directories:
    - "node_modules"
    - "$HOME/google-cloud-sdk/"
services:
  - postgresql
before_install:
  - openssl aes-256-cbc -K $encrypted_9f3b5599b056_key -iv $encrypted_9f3b5599b056_iv -in service-account.json.enc -out service-account.json -d
  - if [ ! -d ${HOME}/google-cloud-sdk ]; then
      curl https://sdk.cloud.google.com | bash;
    fi
  - gcloud auth activate-service-account --key-file service-account.json
install:
  - gcloud config set project sandbox-jonaskay
  - bundle install --jobs=3 --retry=3 --deployment
  - nvm install
  - bin/yarn
before_script:
  - bin/rails db:prepare
script:
  - bin/rails test
before_deploy:
  - openssl aes-256-cbc -K $encrypted_8ad82cc635a3_key -iv $encrypted_8ad82cc635a3_iv -in config/master.key.enc -out config/master.key -d
  - RAILS_ENV=production bin/rails assets:precompile
  - bundle exec rake appengine:exec -- bin/rails db:migrate
deploy:
  provider: script
  script: ./bin/deploy
  skip_cleanup: true
  on:
    branch: master