Feat: improve proof-of-concept app #116

lalver1 · 2025-05-30T21:23:14Z

Closes #114

Adds the stations Streamlit app to the repo (before it was only a local branch) and wires up Django and Streamlit in an environment-dependent way so that the proof-of-concept districts app can run both locally and in the Cloud.

~~Currently this branch is on top of ci/cloud-infra but it will be rebased on main once PR #111 is merged.~~

Notes

AWS Copilot services

Originally, the streamlit service was defined as a copilot Backend Service. For an initial proof-of-concept this is not a convenient service type since the service is unreachable from the internet (recall that the web service embeds streamlit in an iframe, so streamlit needs to be reachable from the internet). For this PR, we ~~will change~~ changed streamlit to a Load Balanced Web Service. As we get closer to production, the architecture ~~will likely~~ may change to:

Configure web to act as a reverse proxy
- streamlit-related iframes in web will point to a path in web but this path will be set up in the ngingx configuration to forward any requests to streamlit over the private network
Place streamlit in a private network (back to Backend Service) and configure it to deny all incoming traffic except for requests from the private IP address of web.

Django database fixtures

For AWS copilot deployments, we load the fixtures from S3 on each deploy by:

Adding the pems-db addon to the web service
This S3 bucket addon was created using
```
copilot storage init --name pems-db --storage-type S3 --workload web --lifecycle workload
```
and the output of the command was an addon manifest file. The S3 bucket's lifecycle is tied to the web service, so it will be deleted if the web service is deleted. Once the deployment process stabilizes, the S3 bucket's lifecycle will instead be tied to the copilot environment.
Setting up the aws CLI in the app container
The aws CLI will be needed for step 3, so it is installed in the app container. There is no need to re-install it in the dev container so the installation step in the dev container's Dockerfile has been removed.
Running start_aws.sh using the web service manifest's CMD definition
Since the web service manifest's CMD definition overrides the Dockerfile's CMD, we use this feature to run start_aws.sh which copies the fixtures from the S3 bucket and loads them into the SQLite database, while still keeping the original behavior for local builds of the app container (local development will still use start.sh in CMD)

Reviewing

Local behavior

To ensure that all the compose services still work locally, you can run bin/build.sh followed by starting up the dev container. Then you can start a debugging session for both Django: PeMS Client and Streamlit, no sidebar and see the locally running app in your browser.

AWS behavior

Running

copilot svc deploy -n streamlit

copilot svc deploy -n web

from inside the infra folder redeploys the services to AWS. Currently the dev environment is running at http://pems-d-publi-tdvc9tnwbm5n-1785238641.us-west-2.elb.amazonaws.com/ but it can change at any time until the environment is stabilized.

github-actions · 2025-05-30T21:24:21Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
pems
settings.py
pems/core
context_processors.py
streamlit_app
main.py
Project Total

_{This report was generated by python-coverage-comment-action}

lalver1 · 2025-06-12T21:04:10Z

pems/districts/templates/districts/district.html

-            <h2>Map</h2>
+    <div class="row" style="min-height: 450px;">
+        <div class="col-lg-12 border">
+            <iframe class="streamlit-app" src="{{ streamlit_host }}/stations--stations?embed=true&district_number={{ current_district.number }}">


~~rename class to streamlit-service~~
Realized that the custom streamlit-app class was not necessary since it can be replaced by Bootstrap's w-100 h-100.

lalver1 · 2025-06-12T21:05:45Z

pems/core/context_processors.py

+def streamlit_host(request):
+    """Context processor to add the Streamlit host URL part to the context."""
+
+    return {"streamlit_host": settings.STREAMLIT_HOST}


consider top-level streamlit context

lalver1 · 2025-06-12T21:06:24Z

pems/settings.py

+
+# Streamlit settings
+STREAMLIT_LOCAL_PORT = os.environ.get("STREAMLIT_LOCAL_PORT", "")
+STREAMLIT_HOST = os.environ.get("STREAMLIT_HOST", f"http://localhost:{STREAMLIT_LOCAL_PORT}")


collapse these 2 variables to 1

lalver1 · 2025-06-12T21:08:31Z

pems/static/css/styles.css

@@ -10,3 +10,8 @@ header {
    }
  }
 }
+
+.streamlit-app {


is there a class in Bootstrap already that already sets these 2 properties? if so, use that one

lalver1 · 2025-06-12T21:13:23Z

streamlit_app/Dockerfile

@@ -35,6 +35,8 @@ EXPOSE 8501

 COPY streamlit_app streamlit_app

+COPY .streamlit .streamlit


move this to before COPY streamlit_app streamlit_app to take advantage of layer caching

thekaveman

I was able to run this locally without any issues, thank you for the great write up!

Since I still need to renew my DOT account password, I didn't run anything against AWS. But the changes look good 👍

The only small comment would be to remove the pems/streamlit_sample app and corresponding streamlit_app/apps/sample since you have a real streamlit app now. No worries either way.

thekaveman · 2025-06-18T15:43:00Z

@lalver1 thanks for removing the sample app!

Do you want to try to squash that pre-commit commit into the commit that introduced those large text files? It seems like pre-commit is having an issue with the ~~line endings~~ extra whitespace, would be nice to get rid of that extra commit if it's easy enough.

pems/urls.py

lalver1 · 2025-06-18T15:47:28Z

Thanks for the review @thekaveman! That was a good suggestion to remove the pems/streamlit_sample app and corresponding streamlit_app/apps/sample to keep things clean 🧹 .
Yep, let me do a little more cleanup on that pre-commit commit and /streamlit_service base URL, it would be nice to keep those changes in this PR 😄

the District Stations Streamlit app is a simple proof-of-concept that displays the traffic data collection stations in a district.

the django and streamlit apps are now wired up in a way that works both for a local dev environment as well as a Cloud deployment.

since the web and streamlit services run on the same load balancer in AWS, the http path defined in the manifest of each service must be different from each other. the http path of the streamlit service (streamlit_service) is also passed to the streamlit run command in the entrypoint.sh script. also, the healthcheck path must be defined in the manifest so that the service does not continually restart.

this setup was causing an error when deploying to AWS since the Django pems app resides in a different container. when running a local debugger though, this error was not present since the local dev container does have the pems app. since at this moment we are not using Django models in the streamlit app, we are removing this ability.

copy the .streamlit directory into the streamlit service's image so that the streamlit configuration is available to the container. before, the configuration was only available to the local dev container.

for AWS copilot deployments, load the fixtures from S3 on each deploy: 1. add the pems-db addon to the web service 2. set up the aws CLI in the app container (remove it from the dev container since it'll already be available) 3. run start_aws.sh using the web service manifest's CMD definition (local development will still use start.sh in CMD)

also remove sample app from streamlit_app since we are not using it anymore.

since the streamlit_sample Django app was removed, the load balancer can now use /streamlit as the path to serve the streamlit copilot service and not worry about path naming conflicts.

thekaveman

Nice work 🚀 🚀

I left a few comments for follow-up, but this looks great for now!

thekaveman · 2025-06-18T21:01:57Z

infra/copilot/streamlit/manifest.yml

+http:
+  # Requests to this path will be forwarded to your service.
+  # To match all requests you can use the "/" path.
+  path: "/streamlit"


thekaveman · 2025-06-18T21:03:40Z

appcontainer/Dockerfile

@@ -52,6 +52,23 @@ RUN useradd --create-home --shell /bin/bash $USER && \
    apt-get install -qq --no-install-recommends build-essential nginx gettext && \
    python -m pip install --upgrade pip


Let's take a follow-up to add the Buildkit cache mounting feature to these Dockerfiles, for apt, pip, etc.

Perfect, I was also thinking that I should add this feature soon 😄

thekaveman · 2025-06-18T21:04:43Z

infra/copilot/web/manifest.yml

+platform: linux/x86_64 # See https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#platform
+count: 1 # Number of tasks that should be running in your service.
+exec: true # Enable running commands in your container.
+command: bin/start_aws.sh


thekaveman · 2025-06-18T21:07:18Z

bin/start_aws.sh

+
+echo "Downloading $S3_FIXTURE_PATH from bucket $S3_BUCKET_NAME"
+aws s3 cp "s3://${S3_BUCKET_NAME}/${S3_FIXTURE_PATH}" "${LOCAL_FIXTURE_PATH}"
+echo "Download complete"


As a follow-up, let's look into mounting the bucket as a volume that presents as a readable directory in the container. I assume this is possible with our setup (we do something similar in Azure).

This seems good for now 👍

thekaveman · 2025-06-18T21:08:22Z

pems/core/context_processors.py

+def streamlit(request):
+    """Context processor adds Streamlit-related information."""
+
+    return {"streamlit": {"url": settings.STREAMLIT_URL}}


lalver1 self-assigned this May 30, 2025

lalver1 force-pushed the feat/app-stations branch from b32158d to 31a45b5 Compare June 4, 2025 21:47

lalver1 force-pushed the feat/app-stations branch 4 times, most recently from 271fa52 to 394f3da Compare June 12, 2025 20:27

lalver1 commented Jun 12, 2025

View reviewed changes

lalver1 force-pushed the feat/app-stations branch 3 times, most recently from 2af3987 to 1e95879 Compare June 16, 2025 23:29

lalver1 marked this pull request as ready for review June 16, 2025 23:41

lalver1 requested a review from a team as a code owner June 16, 2025 23:41

thekaveman previously approved these changes Jun 18, 2025

View reviewed changes

lalver1 dismissed thekaveman’s stale review via a0dbdd9 June 18, 2025 15:32

lalver1 force-pushed the feat/app-stations branch 2 times, most recently from 0864cb8 to 17aa7a4 Compare June 18, 2025 15:37

thekaveman reviewed Jun 18, 2025

View reviewed changes

pems/urls.py Show resolved Hide resolved

lalver1 added 6 commits June 18, 2025 16:03

feat: add District Stations Streamlit app and iframe

1f1b235

the District Stations Streamlit app is a simple proof-of-concept that displays the traffic data collection stations in a district.

chore: wire up django and streamlit in an environment-dependent way

c47dcad

the django and streamlit apps are now wired up in a way that works both for a local dev environment as well as a Cloud deployment.

chore: enable streamlit configuration for standalone streamlit service

802c361

copy the .streamlit directory into the streamlit service's image so that the streamlit configuration is available to the container. before, the configuration was only available to the local dev container.

chore: normalize streamlit http path name

3dc0416

lalver1 added 4 commits June 18, 2025 16:03

chore: organize infrastructure tools under the same directory

8569888

chore(streamlit): remove streamlit_sample Django app

34d3dfc

also remove sample app from streamlit_app since we are not using it anymore.

chore(infra): update streamlit http path name

3578cfd

since the streamlit_sample Django app was removed, the load balancer can now use /streamlit as the path to serve the streamlit copilot service and not worry about path naming conflicts.

lalver1 force-pushed the feat/app-stations branch from e29a2b0 to 3578cfd Compare June 18, 2025 16:52

thekaveman approved these changes Jun 18, 2025

View reviewed changes

lalver1 merged commit bfa7ac5 into main Jun 18, 2025
9 checks passed

lalver1 deleted the feat/app-stations branch June 18, 2025 21:53

lalver1 mentioned this pull request Jun 18, 2025

Fix the deploy workflow #134

Closed

		@@ -35,6 +35,8 @@ EXPOSE 8501

		COPY streamlit_app streamlit_app

		COPY .streamlit .streamlit

		@@ -52,6 +52,23 @@ RUN useradd --create-home --shell /bin/bash $USER && \
		apt-get install -qq --no-install-recommends build-essential nginx gettext && \
		python -m pip install --upgrade pip

Feat: improve proof-of-concept app #116

Feat: improve proof-of-concept app #116

Uh oh!

Conversation

lalver1 commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes

AWS Copilot services

Django database fixtures

Reviewing

Local behavior

AWS behavior

Uh oh!

github-actions bot commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

lalver1 Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lalver1 Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thekaveman left a comment

Choose a reason for hiding this comment

Uh oh!

thekaveman commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

lalver1 commented Jun 18, 2025

Uh oh!

thekaveman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lalver1 commented May 30, 2025 •

edited

Loading

github-actions bot commented May 30, 2025 •

edited

Loading

lalver1 Jun 12, 2025 •

edited

Loading

lalver1 Jun 12, 2025 •

edited

Loading

thekaveman commented Jun 18, 2025 •

edited

Loading