Tuning

Backlog queue

The backlog configuration parameter sets a queue length for incoming connection indications (a request to connect) for a server socket. If a connection indication arrives when the queue is full, the connection is refused.
Depending on how your OS is configured, you might still hit a limit at 128 or so. This is probably due to the kernel config parameter which has a default value lesser than one you specified for the backlog. So try setting it to the same value:

OS X:

$ sysctl -w kern.ipc.somaxconn=1024

Linux:

$ sysctl net.core.somaxconn=1024

ioThreadCount

The ioThreadCount configuration parameter sets the max number of threads handling asynchronous IO operations in client connections. By default, it is set to value of doubled number of processors/cores available, which should be enough for normal operation modes. However, there can be an issue with slow clients fetching big file chunks simultaneously, which in conjunction with slow disk read speed may cause other connections lagging. Try increasing the value of this configuration parameter to some sensible one.

Write spin count

The writeSpinCount configuration parameter is used to control channel data writing behaviour in attempts to send a buffer’s data into underlying socket. The matter is that a write from buffers to the underlying sockets may not transfer all data in one try, and the parameter sets maximum loop count for a write operation until the channel’s write() method writes some amount. There is a balance between how much time the IO thread can spend attempting to fully write a single buffer to the socket and it serving other sockets instead and coming back when success is guaranteed. If that buffer is not fully written then IO thread must register for the writability change event to be notified when the underlying socket becomes writable. If there are many parallel channels served by restricted amount of IO threads, it may have more sense to let the IO thread to switch more frequently from temporarily non-writable channel to one which is available for writing.

handlersThreadCount

The handlersThreadCount configuration parameter sets the max number of threads handling API request processing logic. Some operations may require more time to complete than others. For instance, listing of a directory with many subdirectories and files in it may take significantly more time than acquiring properties of a certain file. Like a thread from IO pool executes one IO task at a time, the request handling thread runs one operation at a time. Thus ideally, the value of handlersThreadCount must be at least as a max number of concurrent API requests is required to be served simultaneously.

FD limits

File descriptors are operating system resources used to represent connections and open files, among other things. Should you have queries resulting in simultaneous locking of too many files, or should the server manage serving a large number of connections, try increasing the FD ulimts in the shell for the server application (ulimit -n number).

INotify

Since Linux Kernel INotify API does not support recursive listening on a directory, SourceAgent adds an inotify watch on every subdirectory under the watched directory. This process takes a time which is linear to the number of directories in the tree being recursively watched, and requires system resources, namely INotify watches, which are limited (by default to 8192 watches per processes).
If you observe an error “No space left on device” in logs it may mean the native inotify watch limit per process has been reached. The solution is to increase the limits by editing respective files in /proc/sys/fs/inotify/max_user_watches directory.

NFS-mounted directories

When running on Linux, Source Agent utilizes kernel INotify API for getting file system modification events. However, INotify does not work with directories on NFS shares. In case you have such directories defined with roots.<container>.path directives, you need to force Source Agent to use directory polling for these by specifying values for roots.<container>.polling.* configuration parameters for each such directory.

Native libraries unavailability

If you are running SourceAgent on a host where directory for temporary files (/tmp) is mounted with noexec option, you may encounter an issues with unavailability of native Zlib decompression libraries. By default, SourceAgent process uses the system-provided temporary directory to extract required libraries to, but such security settings prevent it to launch the extracted libraries from the directory, which results in java.lang.UnsatisfiedLinkError with message “failed to map segment from shared object: Operation not permitted” which can be observed in error logs.
To resolve the issue, please provide another temporary directory to be used for storing and executing bundled native libraries, and specify its location with “jna.tmpdir” system property for SourceAgent in its bin/sa.env.sh script. Alternatively, you might specify full paths to required libraries at executable location as described in System properties section.