Grafana issue on Synology NAS

[jaydkay]

Hi,

I tried setting up the Powerwall-Dashboard with Docker on a Synology NAS. I'm not sure if this is even supposed to work, but the setup script is running fine and three containers seem to start up okay. Only grafana keeps shutting down with the following entries in the docker log:

GF_PATHS_DATA='/var/lib/grafana' is not writable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migration-from-a-previous-version-of-the-docker-container-to-5-1-or-later
mkdir: cannot create directory '/var/lib/grafana/plugins': Permission denied

Any idea what is causing this or how to get grafana running?

Best regards,
Jochen

[jasonacox]

Hi @jaydkay - When grafana starts, it attempts to install the required plugins as indicated by the error. It uses user 1000 group 1000 for this directory. The link you pasted talks about this as being a user permission issue. Some ideas I have:

Is it possible you already had grafana (older version) installed? Try removing it and re-install:

# stop and remove
docker stop grafana
docker rm grafana

# reinstall
docker-compose -f powerwall.yml up -d

I have seen permission error if you run thesetup.sh as root. If you did this, try to remove the Powerwall-Dashboard folder and reinstall using the local user. You will need to make sure your local user has docker permissions:

# Add your user to docker group
sudo usermod -aG docker $USER

If that wasn't the case, you could try to change the permission manually:

# run a shell in grafana interactively
docker exec -ti --user root grafana /bin/bash
ls -la /var/lib/grafana/
chown 1000:1000 /var/lib/grafana/

[jaydkay]

Dear @jasonacox,

thanks for your quick reply. I tried following your suggestions, alas, no success so far...

Is it possible you already had grafana (older version) installed? Try removing it and re-install:

No, it's a completely new installation from scratch. In fact, it's my first contact with docker.

I have seen permission error if you run thesetup.sh as root. If you did this, try to remove the Powerwall-Dashboard folder and reinstall using the local user. You will need to make sure your local user has docker permissions:

Yes, I needed to start ./setup.sh via sudo - it seems that on Synology DSM, docker is typically not available for the normal user. Command usermod also doesn't exist, but there are corresponding commands, so was able to add the current user to the docker group and start docker with a standard user account. Unfortunately, the error remained the same. Additionally, now also Telegraf refuses to start with the following log entry:

I! Using config file: /etc/telegraf/telegraf.conf
E! [telegraf] Error running agent: Error loading config file /etc/telegraf/telegraf.conf: open /etc/telegraf/telegraf.conf: permission denied

Finally, I also tried your final suggestion:

If that wasn't the case, you could try to change the permission manually:

That didn't work either, because the container already refuses to start, so the command you suggested complains about grafana not yet running (but it never will):

Error response from daemon: Container ac20100690faff31b7c2df6bcb6fa9079eea9a496efdbbcf2fed6b8144731aae is restarting, wait until the container is running

If you have any more suggestions, please let me know. Thanks a lot again for your help!

[jasonacox]

Since you originally used sudo the file permission are likely in bad shape. I suggest removing the Powerwall-Dashboard directory and reinstall from scratch now that you have your local user in the docker group:

# remove the old install
rm -fr Powerwall-Dashboard

# git clone or pull down the repository again
git clone https://github.com/jasonacox/Powerwall-Dashboard.git

# run setup
cd Powerwall-Dashboard
./setup.sh

Thanks for the feedback on this. Hopefully we can figure this out for anyone else using a Synology NAS to host their dashboard. :)

[jaydkay]

Dear @jasonacox,

after a few more tries and errors, I guess I finally got it working! Here are the steps that were required - at least in my case:

Log in via SSH to Synology DSM. Create a usergroup docker:
sudo synogroup --add docker

Change ownership of docker to this group:
sudo chown root:docker /var/run/docker.sock

Make the user a member of the newly created group:
sudo synogroup --member docker <user>

Finally, edit file powerwall.yml and change the line

user: "1000:1000"

in the section grafana to the uid of the user on the Synology. I got that by running id while logged in via SSH.

After this, I was able to issue ./setup.sh, finish setting it up and all containers are starting up.

So far it seems to be working, although I didn't do much more beyond starting it up, yet.

[jasonacox]

Awesome!! Nice job @jaydkay ! Thanks for posting the helpful instructions. I'll reference that in the troubleshooting tips.

[BuongiornoTexas]

Hi @jasonacox, @jaydkay,

I've just run into either the same problem or a closely related one while setting up Powerwall-Dashboard under a rootless docker. I was getting the same messages as Jochen GF_PATHS_DATA='/var/lib/grafana' is not writable. and mkdir: cannot create directory '/var/lib/grafana/plugins': Permission denied.

It took a while for me to track it down - as far as I can tell for the rootless version, it comes down to the fact that grafana runs with user:group ids of 472:0 in the rootless container, and this results in name space mapping issues when creating/writing files on the host machine (this explanation isn't quite right, but is probably close enough to the truth allow resolution of the problem).

I found two possible solutions:

1. The first solution is probably safe in a rootless system, but I really wouldn't recommend it as a) only probably safe and b) if you ever switch to a normal docker, it becomes very not safe. It's simple - run the rootless container as "root" by changing the grafana user line in powerwall.yml to user: "0:0". This works as docker rootless maps root to the user running the docker installation. Again, I would really not recommend this approach - I'm just including it as someone else may find this step useful in their own troubleshooting.

2. The second solution is somewhat cleaner, although I'm pretty sure this is still just a work around for a limitation on name space mapping for rootless docker.

• Change the grafana user line in powerwall.yml to user: "472:0" (you may be able to delete it altogether - haven't tried this).

• Get the subuid entry for the user xxxx running docker from /etc/subuid on the host machine. In my case it is xxxx:100000:65536 - from this, we can calculate the uid mapping of the container uid grafana user to a subuser user id on the host machine as: first number in the subuid (100000) + 471 (0 offset of the grafana uid) - so for me this is 100471.

• Change the ownership of the grafana directory in Powerwall-Dashboard on the host machine to this subuid:

  sudo chown 100471 ./grafana (Fails without sudo, but this is a subuid in the xxxx user name space, so has the same privileges as xxxx - effectively no change in permissions)

Running ls -ld ./grafana should result in:

drwxr-xr-x 6 100471 xxxx 4096 Sep 21 10:54 ./grafana (where xxxx is the username running docker - same as the subuid file).

• Kill the current container and restart.

docker stop grafana
docker rm grafana
docker compose -f powerwall.yml up -d

I hope this saves someone else a day or so of digging in the future. Picking up docker this way was definitely a baptism by fire.

Finally, @jasonacox, thanks again for the time you have put into a great monitoring tool!

[jasonacox]

Thanks for this great information, @BuongiornoTexas ! Is this on a Synology NAS or something else?

[BuongiornoTexas]

Is this on a Synology NAS or something else?

Raspberry Pi 4B - so quite the journey getting docker running at all even without dealing with getting it rootless and the grafana permissions issue.

Dropped in here as the error is identical to Jochen's and almost certainly related, but if you want to include it as a tip, I'd put it in a section on grafana permissions and/or rootless docker.

[jasonacox]

Thanks @BuongiornoTexas !!!- I'm running on a Raspberry Pi as well (Raspbian GNU/Linux 11 bullseye). Can you cat /etc/os-release for anyone following this?

For my setup, I had to run something like this:

# Install docker
curl -fsSL https://get.docker.com/ -o get-docker.sh
sudo sh get-docker.sh
sudo apt install -y docker-compose

# Add your user to docker group
sudo usermod -aG docker $USER

# Set docker to start on boot
sudo systemctl enable docker.service
sudo systemctl enable containerd.service

# install docker-compose
sudo pip3 install docker-compose

I wonder if I can add notes or setup.sh instructions to help.

[BuongiornoTexas]

Like you, I'm on bullseye (64 bit).

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian

I think the difference between our installs is that I'm running docker as a non-privileged (rootless) user. So I don't have a docker group at all.

The process for setup is a bit messier, but works pretty well. It's detailed at: https://docs.docker.com/engine/security/rootless/

The steps I took were:

• Use apt -qq list yyy to check for the following pre-req packages, and if not, installed them.

  uidmap
  dbus-user-session
  fuse-overlayfs
  slirp4netns (0.4 or later)
  

• Run the Without packages install script curl -fsSL https://get.docker.com/rootless | sh. At which point docker engine is installed.

• Added the two environment lines at the end of the script to my .bashrc:

 export PATH=/home/username/bin:$PATH
 export DOCKER_HOST=unix:///run/user/1000/docker.sock

• Installed v2 of docker compose manually per the process at https://docs.docker.com/compose/install/linux/ (there are binaries for arm64 on github now, aarch64 file identifier from uname -m!) :

  DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
  mkdir -p $DOCKER_CONFIG/cli-plugins
  curl -SL https://github.com/docker/compose/releases/download/v2.11.0/docker-compose-linux-aarch64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
  

I think everything was running after this.

[jasonacox]

Thanks @BuongiornoTexas ! This is a great 'get starting' for anyone wanting to run docker as a non-privileged (rootless) user. I'll capture this in the troubleshooting section for anyone else wanting to do the same.

Thank you!

[BuongiornoTexas]

No probs. One more step I forgot - I created a file called docker-compose in ~/bin containing:

docker compose "$@"

And ran chown +x ~/bin/docker-compose. An alias that emulates v1 with v2.

Alternatively, users can just run docker compose manually.