Showing posts with label RedHat. Show all posts
Showing posts with label RedHat. Show all posts

Friday, March 17, 2023

Large File Cache

I've come to realize that the gateway does not always clean up after itself :O

Really it's probably Java's garbage collection, but either way if you see your partition filling up and can't find the files responsible this may help.

Out of the box the gateway will cache large message attachments to disk while it processes the request/response, if they exceed a certain size.  The attachment.diskThreshold cluster property is the limit if over which the attachment will be cached to disk.  You can increase this value to keep more attachments in memory and not have them written to disk, but keep in mind that means they are kept in RAM, so make sure you have enough if increasing this property.

A buildup of files on the opt partition should not effect gateway performance... except for those services that are using this attachments cache, though when there is no room on the opt partition the gateway will also stop writing log files, and really I don't trust that it wouldn't effect anything else given the importance of the opt partition.

You can map the attachments directory off of the opt partition to prevent blocking the logs (and whatever else might need space there), and you can also clear the files themselves.  These attachments are not required once the request/response have completed, but experience has shown me that they are not cleaned up in a timely fashion and this results in the full disk.  Even if you delete the file the disk space is not released by the running process because it is still holding the file as open which continues to reserve the space.  The process will eventually release them, but I've seen it take more than a week to do this (it probably waits until it's not busy, which may or may not happen for a while).  You can force it to release the files by restarting the process, but if your gateway is so busy that it can't release the files on its own then I'm guessing this would not be your preferred choice.  Here I detail another option; removing the files (which it usually does on its own even though it doesn't release them), and then truncate the reference (making the reserved space equal to 0).

The below commands will map the attachments folder over to the tmp partition without changing any setting or configuration of the gateway application itself so that if it does fill up then it won't effect running processes.

service ssg stop
cd /tmp
mkdir /tmp/attachments
chmod 775 attachments
chown gateway:layer7 attachments
cd /opt/SecureSpan/Gateway/node/default/var/
rm -f ./attachments/*
rmdir attachments/
ln -s /tmp/attachments attachments
chown -h gateway:layer7 attachments
service ssg start

The below script will delete the attachments older than 15 minutes and then detect which files are referenced by the gateway's java process and truncate the references to the deleted files.

(If you don't map over to /tmp/attachments then you'll have to change the below find to /opt/SecureSpan/Gateway/node/default/var/attachments/ instead.)

#!/bin/sh
#set -x
#export PATH=/bin:/usr/bin:/bin/sh
export PID=$(</opt/SecureSpan/Gateway/node/default/var/ssg.pid);
find /tmp/attachments/ -name "*" -type f -mmin +15 -delete

export deleted="ls -l /proc/$PID/fd 2> /dev/nul | grep deleted | sed -r \"s/.* ([0-9]+) \-.*/\/proc\/$PID\/fd\/\1/g\""
export listing=$(eval "$deleted")

for fn in $listing
do
    truncate -s 0 $fn
done

Schedule with crontab as required.


Ref:

Miscellaneous Cluster Properties (broadcom.com)

Change Gateway default log file location (broadcom.com)

Friday, July 29, 2016

OS root account unlock and reset

As many of you know the Gateway is built on top of RedHat Enterprise Linux running on Oracle Fire hardware, so some these procures are specific to vendors and not so much the gateway itself.
This was performed on a hardware appliance, commands for VM vary slightly.

Oracle Integrated Lights-Out Manager root password reset

  1. Connect to the iLOM serial port and login using the 'default' account (see reference for default account password).
  2. You will be prompted to press the physical presence button (located on the back of the server next to the iLOM serial port).
  3. After login is completed you may then change the password for the iLOM 'root' user:
    • set /SP/users/root password
Ref:
http://docs.oracle.com/cd/E24707_01/html/E24528/z400203c1534691.html

RedHat Enterprise Linux root password reset

  1. Connect to the system console and start (or restart if running) the OS.
  2. When prompted press any key to enter the boot loader menu.
  3. Press 'p' to enter the GRUB password (see referenced article for default password).
  4. Press 'e' to edit the kernel boot command.
  5. Select the 'kernel ...' line and press 'e' to edit.
  6. Append to the end of the line:
    • init=/bin/bashAnd if connected to a hardware appliance via the iLOM serial port then also append to the end of the line:
    • console=ttyS0Press Enter to save the changes
    • If you run into issues with the above parameters then the two following parameters may be useful in troubleshooting:
      panic=0
      --system noinitrd
  7. Then press 'b' to boot the kernel with the modified command parameters.
  8. Mount the root file system with the following command:
    • mount -o remount,rw /
  9. Change the root password:
    • passwd
    • Note: your new password should adhere to the system password policy.
  10. Re-mount the root file system with the following command:
    • mount -o remount,ro /
  11. Save the changes and restart the appliance:

    • sync; reboot -f

  12. Note: the above password reset will not unlock the root account
    • but this command will:
      /sbin/pam_tally2 --reset --user root
Ref:
http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec0000001436.aspx

admin password reset

(The most direct way is ssgconfig menu option 2, option 8.  However if that is not working then you may use the following approach.)
(This section copied directly from the referenced article.)
  1. Login into the command shell of a gateway database node.
  2. Elevate to the root account (option 3).
  3. Run the following command:
    • /opt/SecureSpan/Appliance/bin/resetAdmin.sh root [MySQL root account password]
  4. Enter the Admin account name when prompted. This is the Administrative user name that was created when the Gateway was initially configured.
  5. When the script is complete, the password for the Administrative user is reset back to password. You should now log into the Policy Manager and change the password to a more secure password.
Ref:
https://docops.ca.com/ca-api-gateway/9-0/en/troubleshoot/troubleshooting-password-issues#TroubleshootingPasswordIssues-ResettingtheAdministrativePassword