The backlog configuration parameter sets a queue length for incoming connection
indications (a request to connect) for a server socket. If a connection indication arrives when the queue is full,
the connection is refused.
Depending on how your OS is configured, you might still hit a limit at 128 or so. This is probably due to the kernel config parameter which has a default value lesser than one you specified for the backlog. So try setting it to the same value:
$ sysctl -w kern.ipc.somaxconn=1024
$ sysctl net.core.somaxconn=1024
The ioThreadCount configuration parameter sets the max number of threads handling asynchronous IO operations in client connections. By default, it is set to value of doubled number of processors/cores available, which should be enough for normal operation modes. However, there can be an issue with slow clients fetching big file chunks simultaneously, which in conjunction with slow disk read speed may cause other connections lagging. Try increasing the value of this configuration parameter to some sensible one.
Write spin count¶
The writeSpinCount configuration parameter is used to control channel data writing behaviour in attempts to send a buffer’s data into underlying socket. The matter is that a write from buffers to the underlying sockets may not transfer all data in one try, and the parameter sets maximum loop count for a write operation until the channel’s write() method writes some amount. There is a balance between how much time the IO thread can spend attempting to fully write a single buffer to the socket and it serving other sockets instead and coming back when success is guaranteed. If that buffer is not fully written then IO thread must register for the writability change event to be notified when the underlying socket becomes writable. If there are many parallel channels served by restricted amount of IO threads, it may have more sense to let the IO thread to switch more frequently from temporarily non-writable channel to one which is available for writing.
The handlersThreadCount configuration parameter sets the max number of threads handling API request processing logic. Some operations may require more time to complete than others. For instance, listing of a directory with many subdirectories and files in it may take significantly more time than acquiring properties of a certain file. Like a thread from IO pool executes one IO task at a time, the request handling thread runs one operation at a time. Thus ideally, the value of handlersThreadCount must be at least as a max number of concurrent API requests is required to be served simultaneously.
File descriptors are operating system resources used to represent connections and open files, among other things. Should you have queries resulting in simultaneous locking of too many files, or should the server manage serving a large number of connections, try increasing the FD ulimts in the shell for the server application (ulimit -n number).
Since Linux Kernel INotify API does not support recursive listening on a directory, SourceAgent adds an inotify watch
on every subdirectory under the watched directory. This process takes a time which is linear to the number of
directories in the tree being recursively watched, and requires system resources, namely INotify watches, which are
limited (by default to 8192 watches per processes).
If you observe an error “No space left on device” in logs it may mean the native inotify watch limit per process has been reached. The solution is to increase the limits by editing respective files in
When running on Linux, Source Agent utilizes kernel INotify API for getting file system modification events. However, INotify does not work with directories on NFS shares. In case you have such directories defined with roots.<container>.path directives, you need to force Source Agent to use directory polling for these by specifying values for roots.<container>.polling.* configuration parameters for each such directory.