Scalelite: Nginx - Connection refused

[Marco-Total]

Hello together,

we are facing the problem that since yesterday our scalelite server has stopped working.

We have not installed any updates. The Lets Encrypt certificate is still valid for one month.

When trying to call the API we get a Refused:

root@cmpvideolb:/etc/systemd/system# curl https://videolb.cmp.de/bigbluebutton/api
curl: (7) Failed to connect to videolb.cmp.de port 443: Connection refused

The three Docker containers (NGINX, API, POLLER) started fine:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f47bd4baad79 blindsidenetwks/scalelite:v1.3.5-nginx "/etc/nginx/start -g..." 30 minutes ago Up 30 minutes 111.11.111.111:80->80/tcp, 111.11.111.111:443->443/tcp scalelite- nginx
af8b627c4a80 blindsidenetwks/scalelite:v1.3.5-poller "bin/start-poller" 54 minutes ago Up 54 minutes scalelite-poller
d61577319eab blindsidenetwks/scalelite:v1.3.5-api "bin/start" 54 minutes ago Up 54 minutes 3000/tcp scalelite-api

In docker log I can see that a file "/etc/nginx/conf.d/scalelite.template" is missing:

root@cmpvideolb1:/etc/systemd/system# docker logs f47bd4baad79
Startup script was run as init, re-executing using tini.
Generating templated nginx configuration...
Using non-SSL configuration template.
**/etc/nginx/start: line 32: can't open /etc/nginx/conf.d/scalelite.template: no such file**
Starting nginx periodic reload process...
Starting nginx...
root@cmpvideolb1:/etc/systemd/system#

In the docker directoy i can't find a folder "conf.d"

root@cmpvideolb1:/etc/systemd/system# docker exec -it f47bd4baad79 ls /etc/nginx
dhparam.pem     http.d          nginx.conf      start
fastcgi.conf    mime.types      scgi_params     uwsgi_params
fastcgi_params  modules         ssl

The systemd on my server looks like this:

root@cmpvideolb1:/etc/systemd/system# cat /etc/systemd/system/scalelite-nginx.service
[Unit]
Description=Scalelite Nginx
After=network-online.target
Wants=network-online.target
Before=scalelite.target
PartOf=scalelite.target
After=scalelite-api.service
Requires=scalelite-api.service
After=remote-fs.target
[Service]
EnvironmentFile=/etc/default/scalelite
ExecStartPre=-/usr/bin/docker kill scalelite-nginx
ExecStartPre=-/usr/bin/docker rm scalelite-nginx
ExecStartPre=/usr/bin/docker pull blindsidenetwks/scalelite:${SCALELITE_TAG}-nginx
ExecStart=/usr/bin/docker run --log-opt max-size=10m --log-opt max-file=5 --name scalelite-nginx --env-file /etc/default/scalelite --network scalelite --publish 111.11.111.111:80:80 --publish 111.11.111.111:443:443 --mount type=bind,source=${SCALELITE_RECORDING_DIR}/published,target=/var/bigbluebutton/published,readonly $SCALELITE_NGINX_EXTRA_OPTS blindsidenetwks/scalelite:${SCALELITE_TAG}-nginx
ExecStartPost=/usr/bin/docker image prune -f
[Install]
WantedBy=scalelite.target
root@cmpvideolb1:/etc/systemd/system#

The configuration file for the Scalelite Docker looks like this:

root@cmpvideolb1:/etc/systemd/system# cat /etc/default/scalelite
URL_HOST=videolb.cmp.de
SECRET_KEY_BASE=xxx
LOADBALANCER_SECRET=xxx
DATABASE_URL=postgresql://scalelite:xxx@192.168.100.10/scalelite
REDIS_URL=redis://:xxx@192.168.100.10

SCALELITE_TAG=v1.3.5
SCALELITE_RECORDING_DIR=/var/local/bbb

NGINX_SSL=true
SCALELITE_NGINX_EXTRA_OPTS=--mount type=bind,source=/etc/letsencrypt,target=/etc/nginx/ssl,readonly

RECORDING_DISABLED=true
RECORDING_IMPORT_POLL=false
root@cmpvideolb1:/etc/systemd/system#

Can anyone help me, i have no idea, how i can solve the problem.

Thanks!
Marco

Marco

[schorschroch]

Hello all,
Same issue here since 1.3.5 release.
Switching back to 1.3.4 was our workaround.

Bernhard

[Bevito]

Hi,
Same here. I'll followed Bernhard advice and I rolled back to 1.3.4 too.
I hope an update or a patch will be out soon.
Best regards.

[flyinghuman]

same here. please fix this. we also rolled back for the moment.

Edit: seems that this commit is the cause for the Error: f71012d

- RUN rm /etc/nginx/conf.d/default.conf
+ RUN rm /etc/nginx/http.d/default.conf

- COPY nginx/conf.d /etc/nginx/http.d/

The Command COPY nginx/conf.d /etc/nginx/http.d/was removed. but why?

[marmaou]

@jfederico Folder /etc/nginx/http.d is empty. File /etc/nginx/start still refers to missing folder /etc/nginx/conf.d. Moreover, the missing file /etc/nginx/conf.d/scalelite.template has references to the missing folder. Actually, the same is true for all files in the original /etc/nginx/conf.d folder.

[dannyplace]

Same problem here, switched back to 1.3.4

[NiTRoeSE]

Same here, switched back to 1.3.4

[debuglevel]

Does this software have any basic automated integration tests? As far as I see, this would be as simple as "connects to port 80".

[defnull]

Docker based deployments are not tested prior a release apparently. This is not the first time a new release breaks things unfortunately, so we started maintaining our own fork and usually wait a couple of days after each release to be sure to not break things in production.

[jfederico]

@defnull, with all due respect, that is a very blunt statement, since you are not part of the people doing the releases or development.

The images are tested and used in several deployments before release. Demo is one of them (although very different as it runs on kubernetes and the front-end proxy is not the image by default). But as you may know there are many ways of deploying SL, and yes, the tests we run this time did not cover the systemd deployment. But keep in mind that we support several versions, several images, and several deployments.

If you want to get involved though, you are welcome to join us. I am sure this project will be better with your help. Ping me privately when you have a chance.

Now, to everybody else. Before being released as "stable" (which is the more stable we can get it), it is available at least for 2 weeks as beta. We ask our customers and close contributors (with whom we maintain a close and friendly relationship since the inception of SL) to test their own deployment. We fix what is reported, and then we release.

But, if nobody in the (free and open source) community try it out and wait until is tagged as "stable", then these situations happen.

So again, why don'y instead of waiting to see if nothing breaks after it is released as "stable", you do the same with the beta.1 release? you are doing it anyways. That would help everybody.

[jfederico]

Anyways, at a first glance this seems to be the cuplit after the update of base alpine image. L32 f71012d#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557L32

Patch is on the way