Build OpenSearch on Windows (#767)

* Refactored TemporaryFile and GitRepository into context managers.

Signed-off-by: dblock <[email protected]>

* Remove mkdtemp.

Signed-off-by: dblock <[email protected]>

* Fix tests on Windows.

Signed-off-by: dblock <[email protected]>

* Execute commands using bash.

Signed-off-by: dblock <[email protected]>

* Support windows zip for OpenSearch-min.

Signed-off-by: dblock <[email protected]>

Author: dblock

.github/workflows/tests.yml

@@ -17,6 +17,7 @@ jobs:
         os:
           - ubuntu-latest
           - macos-latest
+          - windows-latest
     runs-on: ${{ matrix.os }}
     env:
       PYTHON_VERSION: 3.7

.gitignore

@@ -16,3 +16,5 @@ out.txt
 
 /artifacts/
 /bundle/
+
+/.vscode/

DEVELOPER_GUIDE.md

@@ -24,7 +24,7 @@ Fork this repository on GitHub, and clone locally with `git clone`.
 
 #### Pyenv
 
-Use pyenv to manage multiple versions of Python. This can be installed with [pyenv-installer](https://github.com/pyenv/pyenv-installer).
+Use pyenv to manage multiple versions of Python. This can be installed with [pyenv-installer](https://github.com/pyenv/pyenv-installer) on Linux and MacOS, and [pyenv-win](https://github.com/pyenv-win/pyenv-win#installation) on Windows.
 
 ```
 curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
@@ -42,7 +42,7 @@ Python 3.7.11
 If you are using pyenv.
 
 ```
-pyenv install 3.7.12
+pyenv install 3.7.12 # use 3.7.9 on Windows, the latest at the time of writing this
 pyenv global 3.7.12
 ```
 
@@ -57,7 +57,10 @@ $ pipenv --version
 pipenv, version 19.0
 ```
 
+On Windows, run `pyenv rehash` if `pipenv` cannot be found. This rehashes pyenv shims, creating a `pipenv` file in `/.pyenv/pyenv-win/shims/`.
+
 #### NVM and Node
+
 Install [nvm](https://github.com/nvm-sh/nvm/blob/master/README.md) to use the Node 10.24.1 version as it is required
 
 ```
@@ -66,6 +69,7 @@ nvm install v10.24.1
 ```
 
 #### Yarn
+
 [Yarn](https://classic.yarnpkg.com/en/docs/install) is required for building and running the OpenSearch Dashboards and plugins
 
 ```

Pipfile.lock

@@ -20,4 +20,4 @@ export PIPENV_PIPFILE="$DIR/Pipfile"
 python3 -m pipenv install
 
 echo "Running "$1" ${@:2} ..."
-python3 -m pipenv run "$1" ${@:2}
+python3 -m pipenv run python "$1" ${@:2}

run.sh

@@ -72,18 +72,42 @@ mkdir -p $OUTPUT/maven/org/opensearch
 # Copy maven publications to be promoted
 cp -r ./build/local-test-repo/org/opensearch "${OUTPUT}"/maven/org
 
+# Assemble distribution artifact
+# see https://github.com/opensearch-project/OpenSearch/blob/main/settings.gradle#L34 for other distribution targets
+
 [ -z "$PLATFORM" ] && PLATFORM=`uname -s` | awk '{print tolower($0)}'
 [ -z "$ARCHITECTURE" ] && ARCHITECTURE=`uname -m`
 
-# Assemble distribution artifact
-# see https://github.com/opensearch-project/OpenSearch/blob/main/settings.gradle#L34 for other distribution targets
+case "$(uname -s)" in
+    Linux*) 
+        PACKAGE="tar"
+        EXT="tar.gz"
+        ;;
+    Darwin*)
+        PACKAGE="tar"
+        EXT="tar.gz"
+        ;;
+    CYGWIN*)
+        PACKAGE="zip"
+        EXT="zip"
+        ;;
+    MINGW*)
+        PACKAGE="zip"
+        EXT="zip"
+        ;;
+    *)
+        echo "Unsupported system: $(uname -s)"
+        exit 1
+        ;;
+esac
+
 case $ARCHITECTURE in
     x64)
-        TARGET="$PLATFORM-tar"
+        TARGET="$PLATFORM-$PACKAGE"
         QUALIFIER="$PLATFORM-x64"
         ;;
     arm64)
-        TARGET="$PLATFORM-arm64-tar"
+        TARGET="$PLATFORM-arm64-$PACKAGE"
         QUALIFIER="$PLATFORM-arm64"
         ;;
     *)
@@ -96,7 +120,7 @@ esac
 
 # Copy artifact to dist folder in bundle build output
 [[ "$SNAPSHOT" == "true" ]] && IDENTIFIER="-SNAPSHOT"
-ARTIFACT_BUILD_NAME=`ls distribution/archives/$TARGET/build/distributions/ | grep "opensearch-min.*$QUALIFIER.tar.gz"`
+ARTIFACT_BUILD_NAME=`ls distribution/archives/$TARGET/build/distributions/ | grep "opensearch-min.*$QUALIFIER.$EXT"`
 mkdir -p "${OUTPUT}/dist"
 cp distribution/archives/$TARGET/build/distributions/$ARTIFACT_BUILD_NAME "${OUTPUT}"/dist/$ARTIFACT_BUILD_NAME
 

scripts/components/OpenSearch/build.sh

@@ -12,6 +12,6 @@
 class BundleOpenSearch(Bundle):
     def install_plugin(self, plugin):
         tmp_path = self._copy_component(plugin, "plugins")
-        cli_path = os.path.join(self.archive_path, "bin/opensearch-plugin")
+        cli_path = os.path.join(self.archive_path, "bin", "opensearch-plugin")
         self._execute(f"{cli_path} install --batch file:{tmp_path}")
         super().install_plugin(plugin)