Skip to content

Commit 0f6d30f

Browse files
jhamongithub-advanced-security[bot]
jhamon
and
github-advanced-security[bot]
authored
Add workflow to process changes after merge to master (#440)
## Problem We would like to trigger some actions based on changes to this repository ## Solution Introduce a new workflow that is run whenever changes are merged to `master` branch (the default branch). The workflow is defined in `.github/workflows/post-merge.yaml` and uses git with `diff-filter`s to group changes by whether they are additions, deletions, or modifications. Then it invokes specific actions to handle each type of change, although for now these actions are just stubs. More work will be needed to implement the real contents of each action. - The names of added notebooks are passed to the `.github/actions/post-merge-additions/process-additions.py` script. - The names of removed notebooks are passed to the `.github/actions/post-merge-deletions/process-deletions.py` script - The names of changed notebooks are passed to the `.github/actions/post-merge-modifications/process-modifications.py` For simplicity, renames are handled as deletion + addition, so when a file is renamed both actions will be triggered. ## Next steps Arjun should set an API key in the `ARJUN_PINECONE_API_KEY` repository secret (referenced in since that is what is being fed through to the actions as `PINECONE_API_KEY`. Currently I just stuck a placeholder value in to verify the jobs and actions are wired up correctly, but wasn't using Pinecone for anything yet inside these actions so there was no need for an actual key. For iterating on these scripts mentioned above, it's probably easiest to run them locally and just poke in different values yourself via environment variable, e.g. run something like this from the terminal: `PINECONE_API_KEY='yourkey' NOTEBOOK='docs/it-threat-detection.ipynb' .github/actions/post-merge-modifications/process-modifications.py` This will allow you to iterate on the contents of the `process-modifications.py` script without having to do a bunch of unnecessary pushing and merging. ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update - [ ] Infrastructure change (CI configs, etc) - [ ] Non-code change (docs, etc) - [ ] None of the above: (explain here) --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
1 parent 61942c5 commit 0f6d30f

File tree

7 files changed

+260
-0
lines changed

7 files changed

+260
-0
lines changed

‎.github/actions/post-merge-additions/action.yml

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: "Post Merge: Process Added Notebook"
2+
description: "Invoke script to process added notebook"
3+
4+
inputs:
5+
added_notebook:
6+
description: "The notebook that has been added"
7+
required: true
8+
PINECONE_API_KEY:
9+
description: "The Pinecone API key"
10+
required: true
11+
12+
runs:
13+
using: 'composite'
14+
steps:
15+
- name: Set up Python
16+
uses: actions/setup-python@v5
17+
with:
18+
python-version: '3.11'
19+
20+
- name: Install dependencies
21+
shell: bash
22+
run: |
23+
pip install --upgrade pip
24+
pip install nbformat requests pinecone
25+
26+
- id: process-changes
27+
shell: bash
28+
name: Process addition of ${{ inputs.added_notebook }}
29+
run: |
30+
python .github/actions/post-merge-additions/process-additions.py
31+
env:
32+
PINECONE_API_KEY: ${{ inputs.PINECONE_API_KEY }}
33+
NOTEBOOK: ${{ inputs.added_notebook }}

‎.github/actions/post-merge-additions/process-additions.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
import os
2+
import nbformat
3+
from pinecone import Pinecone
4+
5+
notebook_added = os.environ['NOTEBOOK']
6+
7+
print(f"Processing new notebook {notebook_added}")
8+
9+
with open(notebook_added, 'r') as f:
10+
notebook = nbformat.read(f, as_version=4)
11+
print(notebook)
12+
13+
pc = Pinecone() # Reads PINECONE_API_KEY from environment variable
14+
15+
# TODO: add embeddings related to new notebook

‎.github/actions/post-merge-deletions/action.yml

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: "Post Merge: Process Deleted Notebook"
2+
description: "Invoke script to process deleted notebook"
3+
4+
inputs:
5+
deleted_notebook:
6+
description: "The notebook that has been deleted"
7+
required: true
8+
PINECONE_API_KEY:
9+
description: "The Pinecone API key"
10+
required: true
11+
12+
runs:
13+
using: 'composite'
14+
steps:
15+
- name: Set up Python
16+
uses: actions/setup-python@v5
17+
with:
18+
python-version: '3.11'
19+
20+
- name: Install dependencies
21+
shell: bash
22+
run: |
23+
pip install --upgrade pip
24+
pip install nbformat requests pinecone
25+
26+
- id: process-changes
27+
shell: bash
28+
name: Process deletion of ${{ inputs.deleted_notebook }}
29+
run: |
30+
python .github/actions/post-merge-deletions/process-deletions.py
31+
env:
32+
PINECONE_API_KEY: ${{ inputs.PINECONE_API_KEY }}
33+
NOTEBOOK: ${{ inputs.deleted_notebook }}

‎.github/actions/post-merge-deletions/process-deletions.py

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
import os
2+
import nbformat
3+
from pinecone import Pinecone
4+
5+
notebook_deleted = os.environ['NOTEBOOK']
6+
7+
print(f"Processing deletions to {notebook_deleted}")
8+
9+
pc = Pinecone() # Reads PINECONE_API_KEY from environment variable
10+
11+
# TODO: remove embeddings related to deleted notebook

‎.github/actions/post-merge-modifications/action.yml

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: "Post Merge: Process Modified Notebook"
2+
description: "Invoke script to process changes to a notebook"
3+
4+
inputs:
5+
modified_notebook:
6+
description: "The notebook that has changed"
7+
required: true
8+
PINECONE_API_KEY:
9+
description: "The Pinecone API key"
10+
required: true
11+
12+
runs:
13+
using: 'composite'
14+
steps:
15+
- name: Set up Python
16+
uses: actions/setup-python@v5
17+
with:
18+
python-version: '3.11'
19+
20+
- name: Install dependencies
21+
shell: bash
22+
run: |
23+
pip install --upgrade pip
24+
pip install nbformat requests pinecone
25+
26+
- id: process-changes
27+
shell: bash
28+
name: Process changes to ${{ inputs.modified_notebook }}
29+
run: |
30+
python .github/actions/post-merge-modifications/process-modifications.py
31+
env:
32+
PINECONE_API_KEY: ${{ inputs.PINECONE_API_KEY }}
33+
NOTEBOOK: ${{ inputs.modified_notebook }}

‎.github/actions/post-merge-modifications/process-modifications.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
import os
2+
import nbformat
3+
from pinecone import Pinecone
4+
5+
notebook_changed = os.environ['NOTEBOOK']
6+
7+
print(f"Processing modified notebook {notebook_changed}")
8+
9+
with open(notebook_changed, 'r') as f:
10+
notebook = nbformat.read(f, as_version=4)
11+
print(notebook)
12+
13+
pc = Pinecone() # Reads PINECONE_API_KEY from environment variable
14+
15+
# TODO: update embeddings to reflect new contents of notebook

‎.github/workflows/post-merge.yaml

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
name: "Post Merge"
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
8+
permissions:
9+
contents: read
10+
11+
jobs:
12+
detect-changes:
13+
runs-on: ubuntu-latest
14+
outputs:
15+
modified_notebooks: ${{ steps.set-modifications.outputs.modified_notebooks }}
16+
has_modifications: ${{ steps.set-modifications.outputs.has_modifications }}
17+
deleted_notebooks: ${{ steps.set-deletions.outputs.deleted_notebooks }}
18+
has_deletions: ${{ steps.set-deletions.outputs.has_deletions }}
19+
added_notebooks: ${{ steps.set-additions.outputs.added_notebooks }}
20+
has_additions: ${{ steps.set-additions.outputs.has_additions }}
21+
steps:
22+
- uses: actions/checkout@v4
23+
with:
24+
fetch-depth: 0 # Required for git diff
25+
26+
- name: Detect changed notebooks
27+
id: set-modifications
28+
run: |
29+
# Get list of changed .ipynb files
30+
# --diff-filter=M: Only modifications (non-deletions)
31+
CHANGED_NOTEBOOKS=$(git diff --diff-filter=M --name-only --no-renames HEAD^ HEAD | grep '\.ipynb$' || true)
32+
if [ -z "$CHANGED_NOTEBOOKS" ]; then
33+
echo "No notebook modifications detected"
34+
echo "has_modifications=false" >> $GITHUB_OUTPUT
35+
echo "modified_notebooks={\"notebook\":[]}" >> $GITHUB_OUTPUT
36+
else
37+
# Convert newlines to JSON array format
38+
NOTEBOOK_LIST=$(echo "$CHANGED_NOTEBOOKS" | jq -R -s -c 'split("\n")[:-1]')
39+
echo "has_modifications=true" >> $GITHUB_OUTPUT
40+
echo "modified_notebooks={\"notebook\":$NOTEBOOK_LIST}" >> $GITHUB_OUTPUT
41+
fi
42+
43+
- name: Detect deleted notebooks
44+
id: set-deletions
45+
run: |
46+
# Get list of deleted .ipynb files
47+
# --diff-filter=D: Only deletions
48+
DELETED_NOTEBOOKS=$(git diff --diff-filter=D --name-only --no-renames HEAD^ HEAD | grep '\.ipynb$' || true)
49+
if [ -z "$DELETED_NOTEBOOKS" ]; then
50+
echo "No notebook deletions detected"
51+
echo "has_deletions=false" >> $GITHUB_OUTPUT
52+
echo "deleted_notebooks={\"notebook\":[]}" >> $GITHUB_OUTPUT
53+
else
54+
# Convert newlines to JSON array format
55+
NOTEBOOK_LIST=$(echo "$DELETED_NOTEBOOKS" | jq -R -s -c 'split("\n")[:-1]')
56+
echo "has_deletions=true" >> $GITHUB_OUTPUT
57+
echo "deleted_notebooks={\"notebook\":$NOTEBOOK_LIST}" >> $GITHUB_OUTPUT
58+
fi
59+
60+
- name: Detect added notebooks
61+
id: set-additions
62+
run: |
63+
# Get list of added .ipynb files
64+
# --diff-filter=A: Only additions
65+
ADDED_NOTEBOOKS=$(git diff --diff-filter=A --name-only --no-renames HEAD^ HEAD | grep '\.ipynb$' || true)
66+
if [ -z "$ADDED_NOTEBOOKS" ]; then
67+
echo "No notebook additions detected"
68+
echo "has_additions=false" >> $GITHUB_OUTPUT
69+
echo "added_notebooks={\"notebook\":[]}" >> $GITHUB_OUTPUT
70+
else
71+
# Convert newlines to JSON array format
72+
NOTEBOOK_LIST=$(echo "$ADDED_NOTEBOOKS" | jq -R -s -c 'split("\n")[:-1]')
73+
echo "has_additions=true" >> $GITHUB_OUTPUT
74+
echo "added_notebooks={\"notebook\":$NOTEBOOK_LIST}" >> $GITHUB_OUTPUT
75+
fi
76+
77+
process-modified-notebooks:
78+
needs:
79+
- detect-changes
80+
if: needs.detect-changes.outputs.has_modifications == 'true'
81+
runs-on: ubuntu-latest
82+
strategy:
83+
fail-fast: false
84+
matrix: ${{ fromJSON(needs.detect-changes.outputs.modified_notebooks) }}
85+
steps:
86+
- uses: actions/checkout@v4
87+
- uses: ./.github/actions/post-merge-modifications
88+
with:
89+
modified_notebook: ${{ matrix.notebook }}
90+
PINECONE_API_KEY: ${{ secrets.ARJUN_PINECONE_API_KEY }} # TODO: update for arjun project
91+
92+
process-deleted-notebooks:
93+
needs:
94+
- detect-changes
95+
if: needs.detect-changes.outputs.has_deletions == 'true'
96+
runs-on: ubuntu-latest
97+
strategy:
98+
fail-fast: false
99+
matrix: ${{ fromJSON(needs.detect-changes.outputs.deleted_notebooks) }}
100+
steps:
101+
- uses: actions/checkout@v4
102+
- uses: ./.github/actions/post-merge-deletions
103+
with:
104+
deleted_notebook: ${{ matrix.notebook }}
105+
PINECONE_API_KEY: ${{ secrets.ARJUN_PINECONE_API_KEY }} # TODO: update for arjun project
106+
107+
process-added-notebooks:
108+
needs:
109+
- detect-changes
110+
if: needs.detect-changes.outputs.has_additions == 'true'
111+
runs-on: ubuntu-latest
112+
strategy:
113+
fail-fast: false
114+
matrix: ${{ fromJSON(needs.detect-changes.outputs.added_notebooks) }}
115+
steps:
116+
- uses: actions/checkout@v4
117+
- uses: ./.github/actions/post-merge-additions
118+
with:
119+
added_notebook: ${{ matrix.notebook }}
120+
PINECONE_API_KEY: ${{ secrets.ARJUN_PINECONE_API_KEY }} # TODO: update for arjun project

0 commit comments

Comments
 (0)