Scheduler cache does not remove node when remove label from node

[Monkeytesting]

Description

We use node-selector in volcano scheduler.

  containers:
  - args:
    - --node-selector=train:true

When the volcano-scheduler is running, we remove the label from **the node A **, but volcano still schedules the pod to the node A.

func (sc *SchedulerCache) SyncNode(nodeName string) error {
	node, err := sc.nodeInformer.Lister().Get(nodeName)
	if err != nil {
		if errors.IsNotFound(err) {
			deleteErr := sc.RemoveNode(nodeName)
			if deleteErr != nil {
				klog.Errorf("Failed to delete node <%s> and remove from cache: %s", nodeName, deleteErr.Error())
				return deleteErr
			}

			klog.V(3).Infof("Node <%s> was deleted, removed from cache.", nodeName)
			return nil
		}
		klog.Errorf("Failed to get node %s, error: %v", nodeName, err)
		return err
	}

	if !sc.nodeCanAddCache(node) {
		return nil
	}
	nodeCopy := node.DeepCopy()
	csiNode, err := sc.csiNodeInformer.Lister().Get(nodeName)
	if err == nil {
		sc.setCSIResourceOnNode(csiNode, nodeCopy)
	} else if !errors.IsNotFound(err) {
		return err
	}
	return sc.AddOrUpdateNode(nodeCopy)
}

Is the code missing removing node from cache in **if !sc.nodeCanAddCache(node) ** logic ?

Steps to reproduce the issue

Describe the results you received and expected

remove node when remove "the label" from node

What version of Volcano are you using?

master

Any other relevant information

No response

[hajnalmt]

Before an other PR on this issue is submitted.

I think we shall have a proper mitigation plan for this, and actual verification. My problem is that we cannot just remove the node from the cache as it can still contain pods scheduled by volcano on it, and we should be able to preempt/reclaim them. I think that's the reason why the node is not removed in the current implementation in the first place (as this can create surprising behaviours if this happens, legacy running pods in the cluster that should be killed manually, allocated queue resources that the scheduler cannot free up etc...).

What I see as a somewhat proper solution is that we actually check if there are still volcano scheduled pods on the node, and remove it from the cache only if this is not the case. Additionally we can approach this from the scheduling side as well and mark the node as unschedulable from volcano's point of view in allocate.