Unable to lock keystore file

[chidg]

Describe the bug
I have a validator running with Nimbus on Goerli. When I attempt to exit the validator, using the following command:

/usr/local/bin/nimbus_beacon_node deposits exit --data-dir=/media/storage/nimbus --validator=0x123...

I receive this error message:

ERR 2022-10-04 13:28:32.438+00:00 Unable to lock keystore file               key_path=/media/storage/nimbus/validators/0x123.../keystore.json error_msg="Resource temporarily unavailable"
FAT 2022-10-04 13:28:32.438+00:00 Unable to continue without decrypted signing key

Running lslocks gives:

nimbus_beacon_n 47638 POSIX  660B WRITE 0          0          0 /media/storage/nimbus/validators/0x123.../keystore.json

It appears that nimbus is acquiring a lock on the keystore file, preventing the exit command from also doing so.

To Reproduce
Steps to reproduce the behavior:

1. Platform details (OS, architecture): Ubuntu AMD64
2. Branch/commit used: Production version 22.10.0, also attempted with previous versions

Validator was set up using Somer Esat's guide, and is running Geth as the execution client. Validator has been running about 1 month and has been functioning well and attesting successfully. I have asked for help in the Ethstaker discord and Nimbus discord but haven't had any success there.

[arnetheduck]

We've tracked this down to the fact that running a beacon node with the validator loaded prevents the exit command from loading the same keystore for creating the exit signature. For now, the workaround is to move the validator/secret folders from the data directory, restart the beacon node then point the exit command to the moved keys/secrets.

[vahir2016]

I had the same issue also on GOERLI, I tried to move the file without success (my password was incorrect).

So I tried to reimport my validators keys, I deleted "files in secrets" directory and directory in "validator".

I don't why but I was able to send the EXIT signal.

https://goerli.beaconcha.in/validator/0x9871a9ec7be0c04935d7608f7e967d42459a9726f8ada8e9c365d2bedf9057da384e092616cbd587100051646c456a16#attestations

It was not a pleasant experience.

[Mostafizor]

Any progress toward fixing this withdrawal issue?

I tried moving the validators/secrets folders but this didn't work. I received an error along the lines of "public key 0x.... could not be found in keystore".

However, @vahir2016's solution worked for me.

[maxtreyver]

I ran into the same issue, which was fairly surprising (when one attempts to exit a validator, presumably one would still be running the nimbus validator so as to continue sending attestations until the exit completes). It would be good to fix this before withdrawals go live on mainnet.

I managed to get around this by copying $DATA_DIR/validators/* AND $DATA_DIR/secrets/* to a separate temporary directory, and running nimbus_beacon_node deposits exit --data-dir=$TMP --validator=0x123... successfully.

Without the /secrets/* directory copied over as well, I continued getting password incorrect like @vahir2016

[arnetheduck]

We've discussed this internally at length, and the core problem is that without locking, it's exceedingly easy to run two processes accessing the same validator keys (for example a BN and a VC pointing to the same data directory) - the cross-platform options are also limited when it comes to file locking.

We'll be updating and re-documenting the exit command so that instead of working with the internal keystore of nimbus, it'll be possible to use the keys as generated by the staking cli (same as running the deposit import command essentially.