'lightning-rpc': Connection refused

[Sjors]

This started happening somewhere in the past few days on Ubuntu and is still happening with the latest master.

cli/lightning-cli help
lightning-cli: Connecting to 'lightning-rpc': Connection refused

The log is happily spitting out lightningd and lightning_gossipd messages, but I can't use RPC.

ps aux | grep lightningd
bitcoin  21805  0.0  0.1  26184  2980 ?        Ss   19:28   0:00 SCREEN -S lightningd
bitcoin  22008  0.2  0.5  26128 12220 pts/3    S+   19:28   0:00 lightningd/lightningd --network=bitcoin --ipaddr=... --log-level=debug --log-file=/home/bitcoin/.lightning/debug.log
bitcoin  22015  0.0  0.1  11112  2776 pts/3    S+   19:28   0:00 /home/bitcoin/lightning/lightningd/lightning_hsmd
bitcoin  22016  4.6  0.9  28264 20428 pts/3    S+   19:28   0:11 /home/bitcoin/lightning/lightningd/lightning_gossipd

I think it's still catching up so that might be related, but that didn't use to matter. I also noticed that when running with -daemon it doesn't daemonize, at least not in the first minute that I waited.

[cdecker]

This is likely due to some pre-startup processing. The line starting with "Server started with public key" in the logs is about the last thing that is done before we actually enter the processing loop, so if that message isn't printed yet, we are still in the warmup phase.

[Sjors]

Well, it's been warning up for 18 hours... Indeed no Server started message.

[cdecker]

What log messages are you seeing?

[Sjors]

I tried connecting it to a full node instead of the local pruned bitcoind node, which made the problem go away and it immediately daemonized. So it seems related to #1297, though that doesn't explain why it recently started happening.

I attached yesterdays log, which consists of a few sessions.

debug-pruned.log

[cdecker]

In that case it probably was attempting to rescan some blocks that were recently pruned from your bitcoind node. This should also be improved a lot once #1179 is fixed and we no longer go back to the wallet birth blockheight.

[Sjors]

Turns out there was a prune event yesterday before I ran into this:

cat ~/.bitcoin/debug.log | grep Prune
2018-03-27T10:18:05Z Prune configured to target 5000MiB on disk for block and undo files.
2018-04-10T12:52:24Z Prune: UnlinkPrunedFiles deleted blk/rev (01196)

pruneheight is 511782: March 3rd. The node was created on March 27th, so it doesn't seem to be wallet birth height.