MongoDB logs repeated DuplicateKey (E11000) errors for cluster_leader lock with multiple Graylog nodes

[peppepace]

Expected Behavior

Follower Graylog nodes should not cause continuous DuplicateKey (E11000) warnings in MongoDB logs during normal, stable leader election. Lock acquisition attempts by non-leader nodes should not generate write operations that cause MongoDB unique index violations when the lock is already held by another node.

Current Behavior

Every time a follower node is running in a Graylog cluster (leader election mode AUTOMATIC, 2+ nodes), and another node is already the leader, MongoDB logs the following warning at every leader election polling interval (e.g., every 30 seconds):

E11000 duplicate key error collection: <db>.cluster_locks index: resource_1 dup key: { resource: "cluster-leader" }

This happens continuously—not just at startup. The cluster remains healthy and leader/follower roles are maintained. However, MongoDB logs are spammed and unnecessary write contention occurs.

Possible Solution

Graylog should avoid upserting the lock if another node already owns it. Consider a lock acquisition mechanism that checks lock status before attempting upsert/write, or uses a conditional update which does not trigger E11000 for non-owner attempts.

Steps to Reproduce (for bugs)

1. Deploy MongoDB 8.0.13 (single node or replica set).
2. Deploy Graylog 6.3.7 (Docker) with 2 nodes, leader election mode AUTOMATIC, both pointed to the same MongoDB DB.
3. Set leader_election_lock_polling_interval=30s and lock_service_lock_ttl=300s.
4. Start node A (becomes leader).
5. Start node B (follower).
6. Observe MongoDB logs: repeated DuplicateKey warnings every polling interval.

Context

This produces significant MongoDB log noise and some write overhead from follower nodes repeatedly failing to upsert the lock. Cluster operates correctly, but this is suboptimal for operational visibility and clean logs.

Relevant MongoDB log sample:

{
  "s": "W",
  "c": "WRITE",
  "msg": "Plan executor error",
  "attr": {
    "operation": "findAndModify",
    "error": {
      "code": 11000,
      "codeName": "DuplicateKey",
      "errmsg": "E11000 duplicate key error collection: <db>.cluster_locks index: resource_1 dup key: { resource: \"cluster-leader\" }",
      "keyValue": { "resource": "cluster-leader" }
    },
    "stats": {
      "stage": "UPDATE",
      "nMatched": 0,
      "failed": true,
      "inputStage": {
        "stage": "FETCH",
        "filter": {
          "locked_by": { "$eq": "<follower-node-id>" }
        }
      }
    }
  }
}

Only one lock document exists for cluster-leader (owned by the leader node):

db.cluster_locks.find({resource:"cluster-leader"})
[
  {
    "_id": ObjectId("<redacted>"),
    "locked_by": "<leader-node-id>",
    "resource": "cluster-leader",
    "updated_at": ISODate("<timestamp>")
  }
]

Your Environment

• Graylog Version: 6.3.7+79219a1 (Docker)
• Java Version: (OpenJDK 17, Eclipse Adoptium)
• OpenSearch Version: 2.19.3
• MongoDB Version: 8.0.13
• Operating System: Ubuntu 22.04.5 LTS (jammy)
• Browser version: n/a

Checklist

[] This issue fix need to be backported.
[] Does this issue have security implications?