Firebird server stops accepting new connections after some time

[agx4ever]

I have server that runs FB3 and I want to migrate to FB4. I have created new test server and installed latest FB4. It works fine until one day. It can work few days or max 2 weeks without problems and then suddenly firebird server just stops accepting new connections. On server I can see firebird in process list, but it simply doesn't accept new connections. When I stop and then start firebird - it works fine again. Error log does not show anything unusual.

I tried same installation and same configuration on different server, to exclude hardware problems or software misconfigurations - and the result is same - FB process stops accepting new connections after some time.

OS: Linux, CentOS Stream release 8
Firebird 4.0.2 - Firebird-4.0.2.2816-0.amd64

--- firebird.conf ---
TempDirectories = /mnt/data0/fb4/tmp/
DefaultDbCachePages = 2048
UseFileSystemCache = true
TempBlockSize = 8M
TempCacheLimit = 64M
InlineSortThreshold = 2048
AuthServer = Srp256
AuthClient = Srp256, Srp
UserManager = Srp
ReadConsistency = 0
RemoteServicePort = 3050
LockMemSize = 1M
LockHashSlots = 8191
ServerMode = SuperClassic

--- databases.conf ---
dev_main = /mnt/data0/fb4/dev_main.fdb
{
DatabaseGrowthIncrement = 128M
DeadlockTimeout = 10
DefaultDbCachePages = 32768
FileSystemCacheThreshold = 1048576
GCPolicy = combined
LockHashSlots = 49999
LockMemSize = 40M
}
--- no replication configuration ---

Last time when the problem occurred I made fbguard and firebird process dumps with "gcore" command. I can send those dumps in email (or other convenient way, just tell how).
If there is anything else I can do, to provide more information, please tell me.

[sim1984]

The current database level configuration is more suitable for SuperServer mode than SuperClassic. The following values are too large:

DefaultDbCachePages = 32768
GCPolicy = combined

[agx4ever]

This is configuration for my test server. For database I am using pagesize 32768 and together with "DefaultDbCachePages = 32768" - each connection takes ~1GB of RAM. I have 40GB RAM installed and usually there are few connections simultaneously (for test server). If you think that this setting could be reason for this problem - I can reduce it.

For GCPolicy - I will change it to "cooperative". I have turned sweep process off and sweeping is done manually with gfix - I thought that it doesn't affect anything important.

[hvlad]

On server I can see firebird in process list, but it simply doesn't accept new connections.

What kind of connection string you use ?
What error is returned by application ?
Does you tried to connect using isql in such moment ?
Could you try embedded (hostless) connection ?

GCPolicy = combined

Doesn't matters for non-SS architectures

[agx4ever]

What kind of connection string you use ?

From IBExpert:
connect 'my_dev_dns/3050:dev_main' user "SYSDBA" password 'VerySecurePass';

What error is returned by application ?

Mostly I am using Java and Jaybird driver (4.0.8.java11).
When Firebird hangs - I get errors:

java.sql.SQLNonTransientConnectionException: Unable to complete network request to host "xx.xx.xx.xx". [SQLState:08006, ISC error code:335544721]
        at org.firebirdsql.gds.ng.FbExceptionBuilder$Type$5.createSQLException(FbExceptionBuilder.java:598)
        at org.firebirdsql.gds.ng.FbExceptionBuilder$ExceptionInformation.toSQLException(FbExceptionBuilder.java:492)
        at org.firebirdsql.gds.ng.FbExceptionBuilder.toSQLException(FbExceptionBuilder.java:223)
        at org.firebirdsql.gds.ng.wire.WireConnection.socketConnect(WireConnection.java:236)
        at org.firebirdsql.gds.ng.wire.FbWireDatabaseFactory.performConnect(FbWireDatabaseFactory.java:50)
        at org.firebirdsql.gds.ng.wire.FbWireDatabaseFactory.connect(FbWireDatabaseFactory.java:39)
        at org.firebirdsql.gds.ng.wire.FbWireDatabaseFactory.connect(FbWireDatabaseFactory.java:32)
        at org.firebirdsql.jca.FBManagedConnection.<init>(FBManagedConnection.java:145)
        at org.firebirdsql.jca.FBManagedConnectionFactory.createManagedConnection(FBManagedConnectionFactory.java:599)
        at org.firebirdsql.jca.FBStandAloneConnectionManager.allocateConnection(FBStandAloneConnectionManager.java:65)
        at org.firebirdsql.jdbc.FBDataSource.getConnection(FBDataSource.java:109)
        at org.firebirdsql.jdbc.FBDriver.connect(FBDriver.java:114)
        at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
        at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)

Also when connecting from IBExpert - it waits around one minute and then says connection failed.

Does you tried to connect using isql in such moment ?
Could you try embedded (hostless) connection ?

Not yet - I will try this next time when server will freeze.
It's sad, that I can't reproduce this problem on demand.

[agx4ever]

From Java I'm using this connection string:

jdbc:firebirdsql://%s:%s/%s?charSet=%s&encoding=%s&roleName=%s&TRANSACTION_READ_COMMITTED=isc_tpb_read_committed,isc_tpb_nowait,isc_tpb_rec_version

Today FB4 server again froze. So I tried to connect from isql tool from same server.

[root@dev1 fb402]# ./isql
Use CONNECT or CREATE DATABASE to specify a database
SQL> connect 127.0.0.1:dev_main user SYSDBA password 'VerySecurePass';
Statement failed, SQLSTATE = 08004
connection rejected by remote interface
SQL>

It took around 1 minute before error message. After that I tried embedded connection without specifying user/pass.

[root@dev1 fb402]# ./isql
Use CONNECT or CREATE DATABASE to specify a database
SQL> connect /data/fb4/dev_main.fdb ;

It never connected - there was no error messages or anything.

[hvlad]

Could you provide full memory dump of firebird process and another one of hung isql (with embedded connection) ?

[agx4ever]

Could you provide full memory dump of firebird process and another one of hung isql (with embedded connection) ?

Today I encountered same problem. I made asked memory dumps.
isql dump was quite small, but firebird process dump took a while and it is ~140GB large. Compressed file is ~230MB large. I can send dropbox share link but only privately (as it may contain sensitive data).
My guess would be, that Firebird freezes due to memory allocation problems, but then it should write according error message or something like that. Also server memory and swap space wasn't fully used.
Where I can send share link?

[AlexPeshkoff]

This core file should go to me, but how was it compressed? Please make sure you've used tar with --sparse switch to process core dump, in other case I may have problems decompressing it. Also xz may be used - it automatically detects sparse files.
Link to be send to peshkoff@mail.ru.

[agx4ever]

This core file should go to me, but how was it compressed? Please make sure you've used tar with --sparse switch to process core dump, in other case I may have problems decompressing it. Also xz may be used - it automatically detects sparse files. Link to be send to peshkoff@mail.ru.

I just sent link for dump files.
Dump was compressed with "gzip -9 " command.

[AlexPeshkoff]

Please use xz next time instead gzip for core dumps (file was decompressing >hour - due to disk load). It's not an issue with compressed size, it's about sparse core dump.

[AlexPeshkoff]

Also I need he following libraries from your box:
ld-linux-x86-64.so.2
libc.so.6
libdl.so.2
libgcc_s.so.1
libgpm.so.2
libm.so.6
libpthread.so.0
librt.so.1
libstdc++.so.6
libthread_db.so.1
libtommath.so.0
libz.so.1

[agx4ever]

Sorry for using wrong compression method.
I just added required libs from Linux server to the same shared folder. I used tar xz compression method this time.
If you need anything else - just ask.

[AlexPeshkoff]

Definitely wrong libraries:
Error while mapping shared library sections:
/opt/lib64/libdl.so.2': Shared library architecture i386 is not compatible with target architecture i386:x86-64. Error while mapping shared library sections: /opt/lib64/libm.so.6': Shared library architecture i386 is not compatible with target architecture i386:x86-64.
Error while mapping shared library sections:
/opt/lib64/libgcc_s.so.1': Shared library architecture i386 is not compatible with target architecture i386:x86-64. Error while mapping shared library sections: /opt/lib64/libpthread.so.0': Shared library architecture i386 is not compatible with target architecture i386:x86-64.
Error while mapping shared library sections:
/opt/lib64/libc.so.6': Shared library architecture i386 is not compatible with target architecture i386:x86-64. Error while mapping shared library sections: /opt/lib64/libstdc++.so.6': Shared library architecture i386 is not compatible with target architecture i386:x86-64.

[agx4ever]

Sorry. It looks I have copied some wrong files from 32bit folder.
I just uploaded correct libraries from x64 directory. Check once more shared folder.

[AlexPeshkoff]

Sooner of all your hang is already fixed in current codebase. Please try current snapshot. In any case it should provide more informative core dumps.

PS. If snapshot anyway hangs (with current dump it's hard to diagnose exact reason) please do not try to attach to server 16000 times - almost all core dump (>90% size) contains stacks of attach threads waiting in same place.