Description
Describe the change you'd like to see
A clear and concise description of what you want to happen.
We are deploying ensemble learning model with InferenceGraph Ensemble node, and would like to provide our python code and yaml file for document example.
This example would be helpful for understanding how to deploy another node in InferenceGraph and how ensemble node gathers parallel isvc outputs.
And, it can be adapted by anyone who has similar model output format, which is a list of probabilities of each label, to use our code directly for convenience. (v1 for now)
For example, will provide yaml and python code similar as following after we finsih our work and change our classifiers to more simple ones.
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: avg-vote
spec:
predictor:
containers:
- name: avg-vote
image: {avg-vote-image}
args:
- --model_name=avg-vote
---
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: dummy-classifier-1
spec:
predictor:
containers:
- name: dummy-classifier-1
image: {dummy-classifier-1-image}
args:
- --model_name=dummy-classifier-1
---
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: dummy-classifier-2
spec:
predictor:
containers:
- name: dummy-classifier-2
image: {dummy-classifier-2-image}
args:
- --model_name=dummy-classifier-2apiVersion: "serving.kserve.io/v1alpha1"
kind: "InferenceGraph"
metadata:
name: "ensemble-2-avg-vote"
spec:
nodes:
root:
routerType: Sequence
steps:
- nodeName: ensemble-2
name: ensemble-2
- serviceName: avg-vote
name: avg-vote
data: $response
ensemble-2:
routerType: Ensemble
steps:
- serviceName: dummy-classifier-1
name: dummy-classifier-1
- serviceName: dummy-classifier-2
name: dummy-classifier-2import argparse
from typing import Dict, Union
import numpy as np
from kserve import (
Model,
ModelServer,
model_server,
InferRequest,
InferOutput,
InferResponse,
logging,
)
from kserve.errors import InvalidInput
from kserve.model_server import app
from kserve.utils.utils import get_predict_response
class AvgVoting(Model):
def __init__(self, name: str):
super().__init__(name)
self.model = None
self.ready = False
self.load()
def load(self):
self.ready = True
def predict(self, payload: Union[Dict, InferRequest], headers: Dict[str, str] = None) -> Union[Dict, InferResponse]:
# assume payload as ensemble node output: {"sklearn-iris":{"predictions":[1,1]},"xgboost-iris":{"predictions":[1,1]}}
tmp = []
for isvcName, output in payload.items():
prediction = output['predictions']
tmp.append(prediction)
result = [sum(x)/len(tmp) for x in zip(*tmp)] # assume same number of label
return get_predict_response(payload, result, self.name)
parser = argparse.ArgumentParser(parents=[model_server.parser])
args, _ = parser.parse_known_args()
if __name__ == "__main__":
if args.configure_logging:
logging.configure_logging(args.log_config_file)
model = AvgVoting(args.model_name)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
ModelServer().start([model])Additional context
Add any other context or screenshots about the feature request here.
Should be related to #288
any suggestion or telling somebody already doing this that i didnt notice would be nice.