ci: Exclude HTTP 5xx metrics from comparisons (#7671)

## Which problem is this PR solving?
- Part of #7617 

## Description of the changes
- This PR introduces the changes to filter out 503 calls to reduce the
flakiness in the metrics comparision.

## How was this change tested?
- I used the artifacts form
https://github.com/jaegertracing/jaeger/actions/runs/19589467818/job/56105108460
and
https://github.com/jaegertracing/jaeger/actions/runs/19596403385/job/56121587792?pr=7666
to manually test the script. Before changes the 503 calls were listed as
metrics added/deleted but after changes they are being filtered
correctly.

Before changes
<img width="1885" height="299" alt="image"
src="https://github.com/user-attachments/assets/0bac1ceb-f8a9-47f3-b841-41639c4841c8"
/>

After changes
<img width="1805" height="77" alt="image"
src="https://github.com/user-attachments/assets/96e6d79c-352a-43a9-9da6-868791634277"
/>



## Checklist
- [x] I have read
https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
- [x] I have signed all commits
- [ ] I have added unit tests for the new functionality
- [x] I have run lint and test steps successfully
  - for `jaeger`: `make lint test`
  - for `jaeger-ui`: `npm run lint` and `npm run test`

---------

Signed-off-by: Tushar Anand <tusharannand@gmail.com>

Author: neoandmatrix

scripts/e2e/compare_metrics.py

@@ -26,6 +26,41 @@
     # }
 }
 
+METRIC_EXCLUSION_RULES = {
+    # excluding HTTP 5xx responses as these can be flaky
+    'http_5xx_errors': {
+        'condition': 'label_match',
+        'label': 'http_response_status_code',
+        'pattern': r'^5\d{2}$',
+    },
+    
+}
+
+
+def should_exclude_metric(metric_name, labels):
+    """
+    Determines if a metric should be excluded from comparison based on configured rules.
+    
+    Args:
+        metric_name: The name of the metric
+        labels: Dictionary of labels for the metric
+        
+    Returns:
+        tuple: (should_exclude: bool, reason: str or None)
+    """
+    for rule_name, rule_config in METRIC_EXCLUSION_RULES.items():
+        condition = rule_config['condition']
+        
+        if condition == 'label_match':
+            label = rule_config['label']
+            pattern = rule_config['pattern']
+            if label in labels and re.match(pattern, labels[label]):
+                return True
+                
+    
+    return False
+
+
 def suppress_transient_labels(metric_name, labels):
     """
     Suppresses transient labels in metrics based on configured patterns.
@@ -58,9 +93,11 @@ def parse_metrics(content):
     for family in text_string_to_metric_families(content):
         for sample in family.samples:
             labels = dict(sample.labels)
-            #simply pop undesirable metric labels
-            labels.pop('service_instance_id',None)
-            
+
+            if should_exclude_metric(sample.name, labels):
+                continue
+
+            labels.pop('service_instance_id', None)
             labels = suppress_transient_labels(sample.name, labels)
 
             label_pairs = sorted(labels.items(), key=lambda x: x[0])