"Bad Gateway" after adding a proxy host and a certificate

[guillaumesoucy94]

Checklist

• Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
  • No
• Are you sure you're not using someone else's docker image?
  • No
• Have you searched for similar issues (both open and closed)?
  • Yes for two hours, finds posts but could not fix my issue.

Describe the bug

"Bad Gateway" when trying to login. Was able to login and add an proxy host and when I add the certificate to the host (certificate obtained but with a time out error" every hosts entries get gone and when I log off and try to log back in, it gave me that "bad gateway" error.

The proxy is still working, just can't login and make changes.

Nginx Proxy Manager Version

v2.11.2

To Reproduce
Steps to reproduce the behavior:
I go to my NPM instance using its IP address like the usual then I try to login.

Expected behavior

The management page as usual.

Screenshots

Operating System

Ubuntu Server 20.04 LTS (Upgraded to 22.04 LTS to see if it would fix it but no).

Additional context

[guillaumesoucy94]

Yeah this what I thought. Things breaks right after attempting to create a certificate.

I forgot that the sub domain wasn't created yet on my registrar before requesting the certificate.

Do you know now how I can fix this mess?

Thanks,

Guillaume

[guillaumesoucy94]

What happened is that I requested the certificate without having created the dns entry for the subdomain at first.

So, it fails to request it, possibility because of that.

I realized the mistake, I quickly proceeded with the DNS entry creation from my domain registration interface, but it didn't fix the issue.

Guillaume

Edit

If someone from the dev team can please also give a look at the issue.

This morning, I noticed that some of my certificates just expired. Affected websites and hosted apps are now throwing certificate issues.

And also @LukeHoneyball thanks for at least getting a look into the issue. I really appreciate it 🙏

[guillaumesoucy94]

Hello,

I still need help with the issue. It's still saying "bad gateway" when trying to login.

Thanks,

Guillaume

[guillaumesoucy94]

fallback_error.log file is full of lines like:

2024/07/11 22:20:26 [error] 347#347: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.2.100, server: nginxproxymanager, request: "POST /api/tokens HTTP/1.1", upstream: "http://127.0.0.1:3000/tokens", host: "192.168.2.136:81", referrer: "http://192.168.2.136:81/login"

[guillaumesoucy94]

Any ideas? Please? A lot of my sites aren't working properly now...

[tinetoalves]

Hello, I have the same issue happening in my environment. If I stack a new npm, I can access and configure all services/sites. However, when I issued a new certificate for a site, if I logout the admin page, I cannot access again and get the 'bad gateway' with the identical message on fallback_error.log.

[riruigit]

me too , and i not good idea to deal with

[guillaumesoucy94]

Hello,

Just want to add that I will soon running NPM from a physical machine rather than on a VM, for capacity related reasons. I will also need to know how to migrate every hosts to another instance of NPM. It is something possible to do? Or, how to obtain a list of every hosts so I will be able to recreate them manually? As I cannot remember every of them.

Regards,

Guillaume

[discokugel]

I have the same problem, acme certificate did not recreate, no login into the ui (Bad Gateway) possible.

What helped for the recreation was:

#1499

docker exec -it npm-db-1 /bin/sh
cd /var/lib/mysql
chown -R mysql:mysql npm
exit

But after a reboot the same problem occured- but sites worg (till next recreation)

[guillaumesoucy94]

I suspect it may related to the database who is not connected any more to NPM.

MySQL is running:
`service mysql status
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2024-11-30 08:41:09 EST; 2 days ago
Process: 45157 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 45165 (mysqld)
Status: "Server is operational"
Tasks: 39 (limit: 2219)
Memory: 341.8M
CPU: 16min 2.524s
CGroup: /system.slice/mysql.service
└─45165 /usr/sbin/mysqld

Nov 30 08:40:58 proxy-dc systemd[1]: Starting MySQL Community Server...
Nov 30 08:41:09 proxy-dc systemd[1]: Started MySQL Community Server.`

However, the docker-compose.yml look like this:

version: '3' services: app: image: 'jc21/nginx-proxy-manager:latest' restart: unless-stopped ports: - '80:80' - '81:81' - '443:443' environment: DB_MYSQL_HOST: "localhost" DB_MYSQL_PORT: 3306 DB_MYSQL_USER: "npm" DB_MYSQL_PASSWORD: "npm" DB_MYSQL_NAME: "npm" volumes: - ./data:/data - ./letsencrypt:/etc/letsencrypt db: image: 'jc21/mariadb-aria:latest' restart: unless-stopped environment: MYSQL_ROOT_PASSWORD: 'npm' MYSQL_DATABASE: 'npm' MYSQL_USER: 'npm' MYSQL_PASSWORD: 'npm' volumes: - ./data/mysql:/var/lib/mysql

The database npm was not existing when doing:

SHOW DATABASES;

I recreate it manually and:

GRANT ALL PRIVILEGES ON *.* TO 'npm'@'localhost' WITH GRANT OPTION;

so this: SHOW GRANTS FOR 'npm'@'localhost'; returns me that: +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Grants for npm@localhost | +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO npm@localhostWITH GRANT OPTION | | GRANT APPLICATION_PASSWORD_ADMIN,AUDIT_ABORT_EXEMPT,AUDIT_ADMIN,AUTHENTICATION_POLICY_ADMIN,BACKUP_ADMIN,BINLOG_ADMIN,BINLOG_ENCRYPTION_ADMIN,CLONE_ADMIN,CONNECTION_ADMIN,ENCRYPTION_KEY_ADMIN,FIREWALL_EXEMPT,FLUSH_OPTIMIZER_COSTS,FLUSH_STATUS,FLUSH_TABLES,FLUSH_USER_RESOURCES,GROUP_REPLICATION_ADMIN,GROUP_REPLICATION_STREAM,INNODB_REDO_LOG_ARCHIVE,INNODB_REDO_LOG_ENABLE,PASSWORDLESS_USER_ADMIN,PERSIST_RO_VARIABLES_ADMIN,REPLICATION_APPLIER,REPLICATION_SLAVE_ADMIN,RESOURCE_GROUP_ADMIN,RESOURCE_GROUP_USER,ROLE_ADMIN,SENSITIVE_VARIABLES_OBSERVER,SERVICE_CONNECTION_ADMIN,SESSION_VARIABLES_ADMIN,SET_USER_ID,SHOW_ROUTINE,SYSTEM_USER,SYSTEM_VARIABLES_ADMIN,TABLE_ENCRYPTION_ADMIN,TELEMETRY_LOG_ADMIN,XA_RECOVER_ADMIN ON *.* TOnpm@localhost WITH GRANT OPTION | +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2 rows in set (0.00 sec)

I don't know what to do next...

Guillaume

[guillaumesoucy94]

I also rebooted, the bad gateway error is still here.

Also upgraded to v2.12.1 with theses: docker compose pull
docker compose up -d without any changes in the issue.

[guillaumesoucy94]

I just noticed those two lines in fallback_error.log

2025/01/18 22:41:48 [alert] 310#310: cache manager process 339 exited on signal 9

and

2025/01/18 22:41:48 [alert] 310#310: worker process 338 exited on signal 9

It looks like if a process crashed or something. Is this could be related the the issue?

Another thing is, if I rename docker-compose.yml doing then recreating it containing the defaults values and doing a docker compose up -dI I'm able to login with the default credentials:

At the moment I put back the original docker-compose.yml file and doing docker compose up -d, I'm getting 'Bad gateway' error again.

Here the content of docker-compose.yml

services: app: image: 'jc21/nginx-proxy-manager:latest' restart: unless-stopped ports: - '80:80' - '81:81' - '443:443' environment: DB_MYSQL_HOST: "127.0.0.1" DB_MYSQL_PORT: 3306 DB_MYSQL_USER: "npm" DB_MYSQL_PASSWORD: "npm" DB_MYSQL_NAME: "npm" volumes: - ./data:/data - ./letsencrypt:/etc/letsencrypt db: image: 'jc21/mariadb-aria:latest' restart: unless-stopped environment: MYSQL_ROOT_PASSWORD: 'npm' MYSQL_DATABASE: 'npm' MYSQL_USER: 'npm' MYSQL_PASSWORD: 'npm' volumes: - ./data/mysql:/var/lib/mysql

[guillaumesoucy94]

@discokugel

I did tried chown -R mysql:mysql npm

The only thing that I'd changed was the docker exec -it npm-db-1 /bin/sh by docker exec -it root-app-1 /bin/sh to match my setup.

Unfortunately it still won't work.