Friday, March 17, 2023

Large File Cache

I've come to realize that the gateway does not always clean up after itself :O

Really it's probably Java's garbage collection, but either way if you see your partition filling up and can't find the files responsible this may help.

Out of the box the gateway will cache large message attachments to disk while it processes the request/response, if they exceed a certain size.  The attachment.diskThreshold cluster property is the limit if over which the attachment will be cached to disk.  You can increase this value to keep more attachments in memory and not have them written to disk, but keep in mind that means they are kept in RAM, so make sure you have enough if increasing this property.

A buildup of files on the opt partition should not effect gateway performance... except for those services that are using this attachments cache, though when there is no room on the opt partition the gateway will also stop writing log files, and really I don't trust that it wouldn't effect anything else given the importance of the opt partition.

You can map the attachments directory off of the opt partition to prevent blocking the logs (and whatever else might need space there), and you can also clear the files themselves.  These attachments are not required once the request/response have completed, but experience has shown me that they are not cleaned up in a timely fashion and this results in the full disk.  Even if you delete the file the disk space is not released by the running process because it is still holding the file as open which continues to reserve the space.  The process will eventually release them, but I've seen it take more than a week to do this (it probably waits until it's not busy, which may or may not happen for a while).  You can force it to release the files by restarting the process, but if your gateway is so busy that it can't release the files on its own then I'm guessing this would not be your preferred choice.  Here I detail another option; removing the files (which it usually does on its own even though it doesn't release them), and then truncate the reference (making the reserved space equal to 0).

The below commands will map the attachments folder over to the tmp partition without changing any setting or configuration of the gateway application itself so that if it does fill up then it won't effect running processes.

service ssg stop
cd /tmp
mkdir /tmp/attachments
chmod 775 attachments
chown gateway:layer7 attachments
cd /opt/SecureSpan/Gateway/node/default/var/
rm -f ./attachments/*
rmdir attachments/
ln -s /tmp/attachments attachments
chown -h gateway:layer7 attachments
service ssg start

The below script will delete the attachments older than 15 minutes and then detect which files are referenced by the gateway's java process and truncate the references to the deleted files.

(If you don't map over to /tmp/attachments then you'll have to change the below find to /opt/SecureSpan/Gateway/node/default/var/attachments/ instead.)

#!/bin/sh
#set -x
#export PATH=/bin:/usr/bin:/bin/sh
export PID=$(</opt/SecureSpan/Gateway/node/default/var/ssg.pid);
find /tmp/attachments/ -name "*" -type f -mmin +15 -delete

export deleted="ls -l /proc/$PID/fd 2> /dev/nul | grep deleted | sed -r \"s/.* ([0-9]+) \-.*/\/proc\/$PID\/fd\/\1/g\""
export listing=$(eval "$deleted")

for fn in $listing
do
    truncate -s 0 $fn
done

Schedule with crontab as required.


Ref:

Miscellaneous Cluster Properties (broadcom.com)

Change Gateway default log file location (broadcom.com)