fix: Merge branch 'dev' into 'feed-diffs'.

Author: binh-dam-ibigroup

docs/img/feed-profile.png

@@ -2,9 +2,6 @@
 
 The IBI Transit Data Tools suite provides web-based tools for creating, managing, evaluating, and publishing transit data, specifically data stored in the General Transit Feed Specification (GTFS) format.
 
-![screenshot](img/public-portal.png)
+![screenshot](img/feed-profile.png)
 
-The following documentation is available (see full table of contents at left):
-
-- **User Documentation** covering basic use and operation of the tools
-- **Developer Documentation** covering installation/deployment of the software and project development
+To get started, click a topic from the table of contents on the left pane.

docs/img/otp-deployment-diagram.png

@@ -0,0 +1,48 @@
+# Adding an OTP Deployment Server in Data Tools
+
+Assumptions:
+
+* [X] You are an admin for Data Tools.
+* [X] You have [set up OTP UI and backend servers on AWS](./setting-up-aws-servers.md).
+* [X] You have a private key file (usually ends in `.pem`) to connect to that AWS environment and EC2 servers via ssh.
+
+From `Administration > Deployment servers`, click on `+ Add Server`.
+
+## General Server Properties
+
+| Property | Description |
+|----------|-------------|
+| Name     | A descriptive display name for the server.        |
+| Public URL | The URL where the public can access the Data Tools UI, e.g. `https://otp-mod-ui.ibi-transit.com`. It is typcially the  CNAME of the CloudFront mirror of the AWS S3 bucket you created or picked for this deployment.         |
+| Internal URLs (Optional) | The URL(s) based on the UI server IP address(es). |
+| S3 bucket name | The name of the AWS S3 bucket you created or picked for this deployment, where Data Tools will share files with the OTP servers. |
+| Admin access only? | Check this option to only allow logins from Data Tool admins. |
+| Project specific? (Optional) | Select a project to only allow the GTFS feeds of that project (e.g. within a region) to be deployed to this server. Leave blank to remove the project restriction. |
+| Use elastic load balancer (ELB) | **We recommend using an elastic load balancer (ELB)**. Behind the scenes, a new server is initialized and added to the load balancer, and old servers are removed and destroyed without interruption to the user. <br><br>The load balancer also allows instantiating multiple OTP servers on large deployments. (You can start, add, or remove more than one OTP server to the load balancer.)
+
+## Load Balancer Properties
+
+(Applies to Data Tools versions prior to October 2019.)
+
+| Property | Description |
+|----------|-------------|
+| Instance type (Optional) | The AWS server type (defines the size, CPU, memory) for all instances on the load balancer. Defaults to `t2.medium`. <br><br> For large deployments (e.g., large metro or statewide), you may need to scale up the instance type beyond the default. Information about instance type sizes and costs can be found at the helpful resource https://ec2instances.info or in the [AWS docs](https://aws.amazon.com/ec2/instance-types/). <br><br>**Note:** Some instance types are not available in certain regions or for VPC setups. If there is a failure during the deployment process (either due to an AWS system error or an out of memory error related to the OTP graph build process), you should be notified and may need to adjust the instance type.
+| AMI ID (Optional) | Specify an AWS Machine Image (AMI) if it is not the default one. |
+| Instance count | Specifies the number of servers to create under this load balancer. Defaults to 1. |
+| Target group ARN | From the AWS Target Groups view (`EC2 > Left Pane > Load Balancing > Target Groups`), find the target group of the desired load balancer, and enter the ARN value under the `Description` tab. |
+| Subnet ID (Optional) | The first portion of the first AWS availability zone, e.g. if the zone value is `subnet-0123456789 - us-east-xx`, only enter `subnet-0123456789`. <br><br>The AWS availability zone is found under the load balancer `Description` tab for the desired load balancer in the `AWS Load Balancer` view. |
+| Security group ID (Optional) | The security group of the AWS load balancer that supports HTTP, HTTPS, and SSH, e.g. `sg-0123456789abcdef`. <br><br>The security group ID is found under the load balancer `Description` tab for the desired load balancer in the `AWS Load Balancer` view.|
+| IAM instance profile ARN | From the `AWS IAM Dashboard > Roles`, pick a role to assign to the deployment servers. Open the role details in AWS to see the IAM instance profile ARN, , e.g. `arn:aws:iam::0123456789:instance-profile/example-role`. |
+| Key file name | Enter a valid private key file name, without the `.pem` extension (copy value from another deployment server). |
+
+If there are EC2 instances running for the desired load balancer, you can click `Terminate EC2 Instances` before proceeding.
+
+Proceed to save the new deployment server (The `Save` button is at the top of the dialog). The new server will  appear in the list of available servers when deploying feeds.
+
+## For test deployments (from old notes)
+
+A flexible approach for test deployments is to create a stable EC2
+instance with a good amount of memory. Identify one or more `internalUrls`
+which is the URL where the OTP server is listening for a `buildGraphOverWire`
+request. (Note: the `publicUrl` should point to a user interface
+for the trip planner.)

docs/img/public-portal.png

@@ -0,0 +1,35 @@
+# Deploying GTFS Feeds to OTP
+
+Assumptions:
+
+* [X] You have [loaded a GTFS feed into a project](./managing-projects-feeds.md).
+* [X] You have a deployment server available [(example: AWS)](./add-deployment-server.md).
+* [X] [An osm-lib server has been set up](https://github.com/conveyal/osm-lib) for Data Tools to fetch Open Streets Map (OSM) data.
+
+## Executing a deployment
+
+To deploy or update GTFS feeds to OTP:
+
+1. Open a project.
+2. Click on the `Deployments` tab.
+3. (Optional) To create a new deployment, click `+ New Deployment`, enter a name, then press or click Enter.
+4. Click the name of the deployment to execute. A summary of feeds and existing deployments (if available) are shown for your review.
+5. Remove the feeds you don't need from the deployment. For the remaining feeds, select the correct feed version.
+6. In the `OTP Configuration` pane:
+ * Select the latest OTP version (the first one in the list).
+ * Check `Build graph only` to only generate and output a graph file on EC2 to the S3 server (no OTP server will be running after the graph is generated).
+ * The R5 option is not used.
+7. If you select `Custom` under `Build configuration` or `Router configuration`, enter the desired configuration settings.
+8. Click the `Deploy` dropdown at the top of the main pane, then pick the server on which to perform the deployment. Existing deployments on that server will be discarded.
+
+## Watching deployments take place
+
+After click Deploy, you can watch the deployment progress from the right-hand panel:
+
+1. The data bundle is uploaded to S3.
+2. One EC2 server is commissioned.
+3. The EC2 server downloads data and begins building the graph.
+4. The EC2 server uploads the graph to S3.
+ - If you check `Build graph only`, the process stops here and EC2 server is discarded.
+5. The OTP process is started on EC2 to load graph.
+6. If more EC2 servers are designated, these will be started and told to download and load graph.

docs/index.md

@@ -0,0 +1,29 @@
+# OTP Deployment Guide
+
+In this guide:
+
+- [OTP Deployment Guide](#otp-deployment-guide)
+  - [Overview](#overview)
+  - [OTP Deployment Architecture](#otp-deployment-architecture)
+  - [Performing an OTP Deployment](#performing-an-otp-deployment)
+
+## Overview
+
+This guide describes how to configure and deploy OTP servers using OTP Data Tools, and is for intermediate to advanced OTP Data Tools administrators.
+
+The deployment architecture diagram below depicts how OTP servers are managed by Data Tools and can be used with elastic load balancers. The user interface is deployed on Amazon S3 servers and optionally mirrored by CloudFront, a high-bandwidth content delivery mechanism. Data Tools prepares and sends a data bundle and configuration properties to initialize OTP servers. The data bundle includes a set of GTFS feeds and OpenStreetMap data. DataTools makes the request to the osm-lib server and then creates a bundle of the resulting OSM and GTFS data. Data Tools does not manage UI deployments at this time.
+
+The steps to perform an OTP deployment describe how to set up and link OTP servers to load balancers, S3 servers to CloudFront, and tie these various AWS resources in Data Tools. Administrators can also find how to configure optional subdomains (i.e. public URLs) for OTP servers.
+
+## OTP Deployment Architecture
+
+The figure below depicts the OTP deployment architecture.
+
+![OTP Deployment Diagram](../img/otp-deployment-diagram.png)
+[Source link](https://ibigroup-my.sharepoint.com/:p:/p/binh_dam/EV_e-3qGZzxIgxJy06StsuIB8TW1A50D_DeKF-aV99jIVQ?e=GMjMh7)
+
+## Performing an OTP Deployment
+
+1. [Setting up OTP UI and backend servers on AWS](./setting-up-aws-servers.md)
+2. [Adding a deployment server from Data Tools](./add-deployment-server.md)
+3. [Deploying GTFS feeds to OTP](./deploying-feeds.md)

docs/user/add-deployment-server.md

@@ -0,0 +1,77 @@
+# Setting Up OTP Deployment Servers on Amazon Web Services (AWS)
+
+This document describes how to:
+
+1. [Create an OTP UI server (AWS S3 and CloudFront)](#ui-server-s3-bucket-and-cloudfront)
+2. [Create an OTP backend load balancer (AWS EC2)](#backend-server-load-balancer)
+
+Assumptions for IBI-hosted deployments:
+
+1. You are logged into an IBI Group AWS environment.
+2. You have selected a region (e.g. [US East (N. Virginia)](https://console.aws.amazon.com/console/home?region=us-east-1)).
+3. A Virtual Private Cloud (VPC) with two subnets exists in that AWS environment.
+
+## UI Server: S3 Bucket and CloudFront
+
+The OTP user interface is delivered using a plain HTTP file server that does not perform any computations. The file server consists of one S3 bucket. For fast high-bandwidth file delivery, we mirror the S3 bucket using CloudFront.
+
+### Create an S3 Bucket
+
+1. From [AWS S3](https://console.aws.amazon.com/s3/home), click `Create Bucket`. Each deployment uses its own bucket.
+2. Specify a name (write down the name for use in Data Tools later).
+3. When specifying options, uncheck `Block All Public Access`.
+
+### Create a CloudFront instance
+
+1. From [AWS CloudFront Home](https://console.aws.amazon.com/cloudfront/home), click `Create Distribution` [(direct link)](https://console.aws.amazon.com/cloudfront/home?#create-distribution:).
+2. Select `Web Distribution`, then click `Next`.
+3. Under `Origin Settings`, select the `DNS name` of the S3 bucket you created above.
+4. Under `Cache Behavior`, select `Redirect HTTP to HTTPS`.
+5. Under `Distribution Settings`, select `Custom SSL certificate`, and select the `*.ibi-transit.com` certificate.
+6. Under `Distribution Settings`, set the `Default Root Object` value to be `index.html`.
+7. (Optional) Enter a comment to make the distribution easy to search. Leave other parameters as is, and click `Create Distribution`.
+8. Open the properties of this CloudFront instance, and copy the `Domain Name` value (e.g. `abcdef0123456789.cloudfront.net`).
+
+### Create a CNAME (i.e. subdomain) for CloudFront
+
+1. Go to the [AWS hosted zone for ibi-transit.com](https://console.aws.amazon.com/route53/home#resource-record-sets:Z37ATQUY9Y96RY).  
+   (It should be under [AWS Route 53](https://console.aws.amazon.com/route53/home), [Hosted Zones](https://console.aws.amazon.com/route53/home#hosted-zones:).)
+2. Select `Create Record Set`. Fill in the subdmain (e.g. `otp-mod-ui.ibi-transit.com`). The OTP UI will be available at this URL. Set the `Record Type` to `CNAME`.
+3. In the value field, paste the `Domain Name` value of the CloudFront instance above.
+4. Click `Create`.
+5. Return to [AWS CloudFront Home](https://console.aws.amazon.com/cloudfront/home).
+6. Edit the properties of the CloudFront instance you created:
+   * In `Alternate Domain Names (CNAMEs)`, paste the CNAME created above.
+
+### Upload files for OTP UI
+
+Upload the files [referenced here](https://github.com/ibi-group/trimet-mod-otp#to-build-a-production-bundle-for-deployment) to the S3 bucket created above.
+
+If updating the UI files, remember to invalidate the CloudFront instance (this forces an update of the files on CloudFront).
+
+## Backend server: Load balancer
+
+We recommend using an elastic load balancer (ELB) for deploying/upgrading an OTP server instance without interrupting the current one. In doing so, a new server is set up in the background, and when ready (preprocessing done), the new server is added to the load balancer, and the old server removed and destroyed.
+
+The load balancer also allows instantiating multiple OTP servers on large deployments. (You can start, add, or remove more than one OTP server to the load balancer.)
+
+### Create the load balancer in AWS
+
+1. Go to [Create Application Load Balancer](https://console.aws.amazon.com/ec2/home#V2CreateELBWizard:type=application:)  
+   (Under [AWS EC2 Load Balancers view](https://console.aws.amazon.com/ec2/home#LoadBalancers:), click `Create Load Balancer` then `Application Load Balancer`.)
+2. Enter a name, add a listener for HTTPS (443), pick a VPC with two subnets/availability zones available, and select two of the subnets. (Leave other params as is, the HTTP(80) listener should be there by default). Click `Next`.
+3. Choose a certificate in ACM, pick the IBI Group certificate. (Leave other params as is). Click `Next`.
+4. Create a new security group, or use an existing one that supports HTTP, HTTPS, SSH. (Leave other params as is). Click `Next`.
+5. Create a new target group (pick a name). Click `Next`.
+6.  Do not register any targets. Click `Finish`.
+7.  From the [Load Balancer view](https://console.aws.amazon.com/ec2/home?#LoadBalancers:), select the row corresponding to your new load balancer.
+8.  Under the `Listeners` tab, there should be two listeners, one for HTTP and one for HTTPS. Add the HTTPS listener if it is not there.
+9.  Open the load balancer properties, and, under the `Description` tab, copy the load balancer's `DNS name` (e.g. `ibi-dev-otp-1234567890.us-east-1.elb.amazonaws.com`).
+
+### Create a CNAME (i.e. subdomain) for the load balancer
+
+1. Go to the [AWS hosted zone for ibi-transit.com](https://console.aws.amazon.com/route53/home#resource-record-sets:Z37ATQUY9Y96RY).\
+   (It should be under [AWS Route 53](https://console.aws.amazon.com/route53/home), [Hosted Zones](https://console.aws.amazon.com/route53/home#hosted-zones:).)
+2. Select `Create Record Set`. Fill in the subdmain (e.g. `otp-mod-dev.ibi-transit.com`). Set the `Record Type` to `CNAME`.
+3. In the value field, paste the `DNS Name` of the load balancer instance above.
+4. Click `Create`.