Large File Cache

I've come to realize that the gateway does not always clean up after itself :O

Really it's probably Java's garbage collection, but either way if you see your partition filling up and can't find the files responsible this may help.

Out of the box the gateway will cache large message attachments to disk while it processes the request/response, if they exceed a certain size.  The attachment.diskThreshold cluster property is the limit if over which the attachment will be cached to disk.  You can increase this value to keep more attachments in memory and not have them written to disk, but keep in mind that means they are kept in RAM, so make sure you have enough if increasing this property.

A buildup of files on the opt partition should not effect gateway performance... except for those services that are using this attachments cache, though when there is no room on the opt partition the gateway will also stop writing log files, and really I don't trust that it wouldn't effect anything else given the importance of the opt partition.

You can map the attachments directory off of the opt partition to prevent blocking the logs (and whatever else might need space there), and you can also clear the files themselves.  These attachments are not required once the request/response have completed, but experience has shown me that they are not cleaned up in a timely fashion and this results in the full disk.  Even if you delete the file the disk space is not released by the running process because it is still holding the file as open which continues to reserve the space.  The process will eventually release them, but I've seen it take more than a week to do this (it probably waits until it's not busy, which may or may not happen for a while).  You can force it to release the files by restarting the process, but if your gateway is so busy that it can't release the files on its own then I'm guessing this would not be your preferred choice.  Here I detail another option; removing the files (which it usually does on its own even though it doesn't release them), and then truncate the reference (making the reserved space equal to 0).

The below commands will map the attachments folder over to the tmp partition without changing any setting or configuration of the gateway application itself so that if it does fill up then it won't effect running processes.

service ssg stop
cd /tmp
mkdir /tmp/attachments
chmod 775 attachments
chown gateway:layer7 attachments
cd /opt/SecureSpan/Gateway/node/default/var/
rm -f ./attachments/*
rmdir attachments/
ln -s /tmp/attachments attachments
chown -h gateway:layer7 attachments
service ssg start

The below script will delete the attachments older than 15 minutes and then detect which files are referenced by the gateway's java process and truncate the references to the deleted files.

(If you don't map over to /tmp/attachments then you'll have to change the below find to /opt/SecureSpan/Gateway/node/default/var/attachments/ instead.)

#!/bin/sh
#set -x
#export PATH=/bin:/usr/bin:/bin/sh
export PID=$(</opt/SecureSpan/Gateway/node/default/var/ssg.pid);
find /tmp/attachments/ -name "*" -type f -mmin +15 -delete

export deleted="ls -l /proc/$PID/fd 2> /dev/nul | grep deleted | sed -r \"s/.* ([0-9]+) \-.*/\/proc\/$PID\/fd\/\1/g\""
export listing=$(eval "$deleted")

for fn in $listing
do
    truncate -s 0 $fn
done

Schedule with crontab as required.

Ref:

Miscellaneous Cluster Properties (broadcom.com)

Change Gateway default log file location (broadcom.com)

Why is space not being freed from disk after deleting a file in Red Hat Enterprise Linux? - Red Hat Customer Portal

Efficiently Purging Audits

Cleaning up the audit tables will release some disk space and may be required to shrink the database enough to perform the dump and sftp for a manual database re-sync when replication is broken. You should stop replication and run the commands on each node. DO NOT perform this if replication is broken and you do not intend to run the replication restart shell script, otherwise this may prevent automatic recovery by removing records that are referenced in pending relay logs.

Clean up the tables that have no constraints first to shorten the time between drop/add constraints:

truncate table audit_detail_params;

truncate table audit_message;

truncate table audit_admin;

truncate table audit_system;

Drop the constraints that block truncating other tables:

alter table audit_system drop foreign key `audit_system_ibfk_1`;

alter table audit_admin drop foreign key `audit_admin_ibfk_1`;

alter table audit_message drop foreign key `audit_message_ibfk_1`;

alter table audit_detail drop foreign key `audit_detail_ibfk_1`;

alter table audit_detail_params drop foreign key `audit_detail_params_ibfk_1`;

Truncate all the audit tables to be sure that there will be no records preventing re-adding the constraints:

truncate table audit_main;

truncate table audit_detail;

truncate table audit_detail_params;

truncate table audit_message;

truncate table audit_admin;

truncate table audit_system;

Re-add the constraints:

alter table audit_system add CONSTRAINT `audit_system_ibfk_1` FOREIGN KEY (`goid`) REFERENCES `audit_main` (`goid`) ON DELETE CASCADE ON UPDATE NO ACTION;

alter table audit_admin add CONSTRAINT `audit_admin_ibfk_1` FOREIGN KEY (`goid`) REFERENCES `audit_main` (`goid`) ON DELETE CASCADE ON UPDATE NO ACTION;

alter table audit_message add CONSTRAINT `audit_message_ibfk_1` FOREIGN KEY (`goid`) REFERENCES `audit_main` (`goid`) ON DELETE CASCADE ON UPDATE NO ACTION;

alter table audit_detail add CONSTRAINT `audit_detail_ibfk_1` FOREIGN KEY (`audit_goid`) REFERENCES `audit_main` (`goid`) ON DELETE CASCADE ON UPDATE NO ACTION;

alter table audit_detail_params add CONSTRAINT `audit_detail_params_ibfk_1` FOREIGN KEY (`audit_detail_goid`) REFERENCES `audit_detail` (`goid`) ON DELETE CASCADE ON UPDATE NO ACTION;

You can run all of these commands at once and more than once. If there is any errors on the re-add then run all the commands again.