Change to two-stage builds, and display build ID

Author: dazuma

.toys.rb

@@ -123,6 +123,27 @@ def run
   end
 end
 
+tool "predeploy" do
+  flag :project, "--project=VALUE", "-p VALUE"
+  flag :name, "--name=VALUE", "-n VALUE", default: "tanx"
+  flag :yes, "--yes", "-y"
+
+  include :exec, exit_on_nonzero_status: true
+  include :terminal
+
+  def run
+    project = get(:project) || capture(["gcloud", "config", "get-value", "project"]).strip
+    exit(1) unless yes || confirm("Prebuild tanx dependencies in #{project}? ", default: true)
+
+    puts("Building base images...", :bold, :cyan)
+    exec(["gcloud", "builds", "submit",
+          "--project", project,
+          "--config", "deploy/build-base.yml",
+          "."])
+    puts("Done", :bold, :cyan)
+  end
+end
+
 tool "deploy" do
   flag :project, "--project=VALUE", "-p VALUE"
   flag :tag, "--tag=VALUE", "-t VALUE", default: ::Time.now.strftime("%Y-%m-%d-%H%M%S")
@@ -134,13 +155,13 @@ def run
 
   def run
     project = get(:project) || capture(["gcloud", "config", "get-value", "project"]).strip
-    image = "gcr.io/#{project}/#{name}:#{tag}"
-    exit(1) unless yes || confirm("Build #{image} and deploy to GKE in project #{project}? ", default: true)
+    exit(1) unless yes || confirm("Deploy build #{tag} in project #{project}? ", default: true)
 
+    image = "gcr.io/#{project}/#{name}:#{tag}"
     puts("Building image: #{image} ...", :bold, :cyan)
     exec(["gcloud", "builds", "submit",
           "--project", project,
-          "--config", "deploy/cloudbuild.yml",
+          "--config", "deploy/build-tanx.yml",
           "--substitutions", "_BUILD_ID=#{tag}",
           "."])
     puts("Updating deployment...", :bold, :cyan)

README.md

@@ -80,31 +80,49 @@ Google Kubernetes Engine.
 
 ### Building
 
-We will use Google's [Cloud Build](https://cloud.google.com/cloud-build/)
-service to build the Tanx application in a Docker image. Notice that there is
-a Dockerfile provided. It uses Distillery to produce an OTP release, and
-installs it in a Docker image based on bitwalker's Alpine-Erlang image. We
-will just hand this to Google Cloud Build to build an image and upload it to
-your project's container registry.
+Tanx builds are done in two stages. First, you build base images that include
+precompiled dependencies, and build tools such as brunch. This build can take
+several minutes to complete. Then, a normal application build uses those base
+images and just builds the app itself, which is usually pretty quick.
+Generally, you must perform a base build once at the beginning, and then any
+time your dependencies change. Otherwise, you can perform normal builds.
 
-To perform the first build.
+Tanx uses Google's [Cloud Build](https://cloud.google.com/cloud-build/)
+service to build Docker images in the cloud. The builds are based on
+bitwalker's Alpine-Erlang image, and use Distillery to produce an OTP release
+that can be run by Kubernetes or any other container orchestration system.
 
-    gcloud builds submit --config deploy/cloudbuild.yml .
+To perform a base build:
+
+    gcloud builds submit --config deploy/built-base.yml .
 
 The period at the end is required; it is the root directory for the application
-you are building.
+you are building. This builds the base images (including precompiling the hex
+dependencies and node module tools) and uploads them to your project.
+
+Once that is done, you can perform an application build using the base images:
+
+    gcloud builds submit --config deploy/build-tanx.yml .
+
+This builds the application image itself and uploads it to your project. The
+image is uploaded as `gcr.io/${PROJECT_ID}/tanx:latest`.
 
-This will build your image in the cloud, and upload it to your project's image
-registry.
+Thereafter, you can rebuild the application by just performing application
+builds. Another base build is required whenever the dependencies change.
 
 ### Local builds (optional)
 
-You may also build an image locally using, for example:
+You may also build images locally using Docker. To build the base images:
 
-    docker build -f deploy/Dockerfile -t tanx .
+    docker build -f deploy/Dockerfile-builder-base -t tanx-builder-base .
+    docker build -f deploy/Dockerfile-runtime-base -t tanx-runtime-base .
 
 Note that Docker 17.05 or later is required.
 
+Then, to perform an application build using those base images:
+
+    docker build -f deploy/Dockerfile-tanx -t tanx .
+
 You may run a local build using:
 
     docker run --rm -p 8080:8080 tanx
@@ -185,10 +203,10 @@ at that URL to view the running application.
 ### Updating the app
 
 To update the tanx app to reflect changes you have made, rebuild with a new
-version tag. For example, if your original build image was tagged
-`gcr.io/${PROJECT_ID}/tanx:v1`, you might do a new build as:
+version tag. To supply a version tag, pass a `_BUILD_ID` substitution into the
+build command. For example, to tag a build as `v2`, do this:
 
-    gcloud builds submit --config deploy/cloudbuild.yml \
+    gcloud builds submit --config deploy/build-tanx.yml \
       --substitutions _BUILD_ID=v2 .
 
 Now the new image `gcr.io/${PROJECT_ID}/tanx:v2` will be available in your
@@ -199,6 +217,15 @@ to the new image:
 
 This performs a "rolling" update for zero downtime deploys.
 
+If you do not provide a `_BUILD_ID`, then the tag will default to `latest`. For
+example, in the initial application build we did previously, we did not provide
+a `_BUILD_ID`, so that image was tagged as `gcr.io/${PROJECT_ID}/tanx:latest`.
+
+Remember that if dependencies have been updated, you need to perform a base
+build first, to prebuild the dependencies and update your base images. Do not
+pass a `_BUILD_ID` to the base build. Just perform the base build as described
+earlier, then perform an application build setting `_BUILD_ID`.
+
 ### Cleanup and tearing down a deployment
 
 To clean up a deployment of tanx and stop incurring hosting costs, do the

apps/tanx_web/assets/css/app.scss

@@ -86,3 +86,6 @@ body {
 #client-node-name {
   font-weight: bold;
 }
+#client-build-id {
+  font-weight: bold;
+}

apps/tanx_web/assets/js/lobby.js

@@ -115,6 +115,7 @@ class Lobby {
     this._lobbyChannel.on("update", update => {
       this._updateGameTable(update.g)
       $('#client-node-name').text(update.d);
+      $('#client-build-id').text(update.b);
       this._gameInfo = {};
       update.g.forEach((game) => {
         this._gameInfo[game.i] = game;

apps/tanx_web/lib/tanx_web/channels/lobby_channel.ex

@@ -88,6 +88,8 @@ defmodule TanxWeb.LobbyChannel do
         %{i: meta.id, n: meta.display_name, d: meta.node}
       end)
 
-    push(socket, "update", %{g: games, d: Node.self()})
+    build_id = System.get_env("TANX_BUILD_ID") || "local"
+    node_name = Node.self()
+    push(socket, "update", %{g: games, d: node_name, b: build_id})
   end
 end

apps/tanx_web/lib/tanx_web/controllers/page_controller.ex

@@ -2,8 +2,7 @@ defmodule TanxWeb.PageController do
   use TanxWeb, :controller
 
   def index(conn, _params) do
-    build_id = System.get_env("TANX_BUILD_ID")
-    build_id = if build_id == nil, do: "local", else: build_id
+    build_id = System.get_env("TANX_BUILD_ID") || "local"
 
     conn
     |> assign(:build_id, build_id)

apps/tanx_web/lib/tanx_web/templates/page/index.html.eex

@@ -55,6 +55,7 @@
         <div class="col-12">
           <h4>Player Info</h4>
           <div>Connected to node: <span id="client-node-name"></span></div>
+          <div>Running build: <span id="client-build-id"></span></div>
           <div class="form-inline">
             <span class="mr-2">as</span>
             <input type="text" id="tanx-name-field" class="form-control input-sm" size="30" maxlength="30" placeholder="(player name)">

deploy/Dockerfile

@@ -0,0 +1,14 @@
+FROM elixir:alpine
+WORKDIR /opt/base-app
+COPY . .
+ENV MIX_ENV=prod \
+    REPLACE_OS_VARS=true \
+    TERM=xterm
+RUN apk update \
+    && apk --no-cache --update add build-base nodejs nodejs-npm python2 git \
+    && mix local.rebar --force \
+    && mix local.hex --force \
+    && cd /opt/base-app \
+    && mix do deps.get, deps.compile \
+    && cd apps/tanx_web/assets/ \
+    && npm install

deploy/Dockerfile-builder-base

@@ -0,0 +1,3 @@
+FROM alpine:latest
+RUN apk update \
+    && apk --no-cache --update add bash openssl-dev

deploy/Dockerfile-runtime-base

@@ -0,0 +1,25 @@
+FROM tanx-builder-base
+WORKDIR /opt/app
+COPY . .
+RUN mv /opt/base-app/_build _build \
+    && mv /opt/base-app/deps deps \
+    && mv /opt/base-app/apps/tanx_web/assets/node_modules apps/tanx_web/assets/node_modules \
+    && mix compile \
+    && cd apps/tanx_web/assets \
+    && node_modules/brunch/bin/brunch build -p \
+    && cd .. \
+    && mix phx.digest
+RUN mix release --env=prod --verbose \
+    && mv _build/prod/rel/tanx /opt/release \
+    && mv /opt/release/bin/tanx /opt/release/bin/start_server
+
+FROM tanx-runtime-base
+ARG build_id=local
+ENV PORT=8080 \
+    MIX_ENV=prod \
+    REPLACE_OS_VARS=true \
+    TANX_BUILD_ID=${build_id}
+WORKDIR /opt/app
+COPY --from=0 /opt/release .
+EXPOSE ${PORT}
+CMD ["/opt/app/bin/start_server", "foreground"]

deploy/Dockerfile-tanx

@@ -0,0 +1,13 @@
+steps:
+  - name: "gcr.io/cloud-builders/docker"
+    args: ["build", "-f", "deploy/Dockerfile-builder-base",
+           "-t", "gcr.io/$PROJECT_ID/tanx-builder-base:latest",
+           "."]
+  - name: "gcr.io/cloud-builders/docker"
+    args: ["build", "-f", "deploy/Dockerfile-runtime-base",
+           "-t", "gcr.io/$PROJECT_ID/tanx-runtime-base:latest",
+           "."]
+
+images:
+  - "gcr.io/$PROJECT_ID/tanx-builder-base:latest"
+  - "gcr.io/$PROJECT_ID/tanx-runtime-base:latest"

deploy/build-base.yml

@@ -0,0 +1,20 @@
+steps:
+  - name: 'gcr.io/cloud-builders/docker'
+    args: ['pull', 'gcr.io/$PROJECT_ID/tanx-builder-base:latest']
+  - name: 'gcr.io/cloud-builders/docker'
+    args: ['tag', 'gcr.io/$PROJECT_ID/tanx-builder-base:latest', 'tanx-builder-base']
+  - name: 'gcr.io/cloud-builders/docker'
+    args: ['pull', 'gcr.io/$PROJECT_ID/tanx-runtime-base:latest']
+  - name: 'gcr.io/cloud-builders/docker'
+    args: ['tag', 'gcr.io/$PROJECT_ID/tanx-runtime-base:latest', 'tanx-runtime-base']
+  - name: "gcr.io/cloud-builders/docker"
+    args: ["build", "-f", "deploy/Dockerfile-tanx",
+           "-t", "gcr.io/$PROJECT_ID/tanx:$_BUILD_ID",
+           "--build-arg", "build_id=$_BUILD_ID",
+           "."]
+
+substitutions:
+  _BUILD_ID: latest
+
+images:
+  - "gcr.io/$PROJECT_ID/tanx:$_BUILD_ID"