Advanced Checkpoint Gaia CLI Commands (Tips and Tricks)

With my most populous post "Basic Checkpoint Gaia CLI Commands (Tips and Tricks)", I would like to
collect some more advanced troubleshooting commands used in my daily work into this post. Actually, some of commands are not only for Checkpoint Gaia, it will be for SPLAT or IPSO platform as well. This post will keep updating as soon as I have something new.

1. fw ctl chain

Check Checkpoint Security Gateway packet inspection order/chain. For more details, check the post "How Firewalls (Security Gateways) Handle the Packets?"

in chain (18):
        0: -7f800000 (f28854f0) (ffffffff) IP Options Strip (in) (ipopt_strip)
        1: -7d000000 (f1796f10) (00000003) vpn multik forward in
        2: - 2000000 (f177cb70) (00000003) vpn decrypt (vpn)
        3: - 1fffff8 (f1787c00) (00000001) l2tp inbound (l2tp)
        4: - 1fffff6 (f2886ca0) (00000001) Stateless verifications (in) (asm)
        5: - 1fffff5 (f28bce30) (00000001) fw multik misc proto forwarding
        6: - 1fffff2 (f17a4df0) (00000003) vpn tagging inbound (tagging)
        7: - 1fffff0 (f177a150) (00000003) vpn decrypt verify (vpn_ver)
        8: - 1000000 (f29049c0) (00000003) SecureXL conn sync (secxl_sync)
        9:         0 (f282f810) (00000001) fw VM inbound  (fw)
        10:         1 (f28a6b30) (00000002) wire VM inbound  (wire_vm)
        11:   2000000 (f177b5e0) (00000003) vpn policy inbound (vpn_pol)
        12:  10000000 (f2902cb0) (00000003) SecureXL inbound (secxl)
        13:  7f600000 (f287ab70) (00000001) fw SCV inbound (scv)
        14:  7f730000 (f2a13500) (00000001) passive streaming (in) (pass_str)
        15:  7f750000 (f2c0bef0) (00000001) TCP streaming (in) (cpas)
        16:  7f800000 (f2885890) (ffffffff) IP Options Restore (in) (ipopt_res)
        17:  7fb00000 (f2fac050) (00000001) HA Forwarding (ha_for)
out chain (15):
        0: -7f800000 (f28854f0) (ffffffff) IP Options Strip (out) (ipopt_strip)
        1: -78000000 (f1796ef0) (00000003) vpn multik forward out
        2: - 1ffffff (f1779a10) (00000003) vpn nat outbound (vpn_nat)
        3: - 1fffff0 (f2c0bd70) (00000001) TCP streaming (out) (cpas)
        4: - 1ffff50 (f2a13500) (00000001) passive streaming (out) (pass_str)
        5: - 1ff0000 (f17a4df0) (00000003) vpn tagging outbound (tagging)
        6: - 1f00000 (f2886ca0) (00000001) Stateless verifications (out) (asm)
        7:         0 (f282f810) (00000001) fw VM outbound (fw)
        8:         1 (f28a6b30) (00000002) wire VM outbound  (wire_vm)
        9:   2000000 (f1779c30) (00000003) vpn policy outbound (vpn_pol)
        10:  10000000 (f2902cb0) (00000003) SecureXL outbound (secxl)
        11:  1ffffff0 (f17887b0) (00000001) l2tp outbound (l2tp)
        12:  20000000 (f177d5b0) (00000003) vpn encrypt (vpn)
        13:  7f700000 (f2c0e340) (00000001) TCP streaming post VM (cpas)
        14:  7f800000 (f2885890) (ffffffff) IP Options Restore (out) (ipopt_res)

2. Proxy Arp

a. Use the Gaia portal.
Network Management -> Arp -> Proxy ARP

b. Use the command line (in Gaia):
add arp proxy ipv4-address 172.16.0.8 interface eth0 real-ipv4-address 172.16.0.22

Actually the GAIA command above convert it automatically to a file called local.arp

c. Use the command line (in expert mode):
Then insert the information directly to /opt/CPsuite-R76/fw1/conf/local.arp
echo "172.16.0.8 00:0c:29:f1:b7:74 172.16.0.22" >> $FWDIR/conf/local.arp


Verify the changes after a policy push with command "fw ctl arp":

[Expert@CP1:0]# fw ctl arp  (10.9.3.21) at 00-1c-7f-32-cc-15  (10.9.3.53) at 00-1c-7f-32-cc-15  (10.9.3.35) at 00-1c-7f-32-cc-15  (10.9.3.26) at 00-1c-7f-32-cc-15  (10.9.3.29) at 00-1c-7f-32-cc-15  (10.9.3.80) at 00-1c-7f-32-cc-15  (191.24.11.13) at 00-1c-7f-33-07-ae interface 191.24.11.116  (10.9.3.25) at 00-1c-7f-32-cc-15  (10.9.3.61) at 00-1c-7f-32-cc-15  (10.9.3.28) at 00-1c-7f-32-cc-15  (10.9.3.24) at 00-1c-7f-32-cc-15  (10.9.3.27) at 00-1c-7f-32-cc-15 FW-GAIA> show arp proxy all IP Address              MAC Address / Interface         Real IP Address 200.0.0.102           eth0

Reference: Checkpoint SPLAT Manual Proxy ARP Configuration Example

3. fw ctl zdebug drop

lists all dropped packets in real time gives an explanation why the packet is dropped

4. TCPDUMP

• tcpdump port 257   , <– on the firewall, this will allow you to see if the logs are passing from the firewall to the manager, and what address they are heading to.
• tcpdump -i WAN.15  <- to capture everything on this interface
• tcpdump -i eth1.16 icmp  <– to capture just PINGs on this interface
• tcpdump -i  Mgmt -vvv -s0 -w tcpdumpfile.log   <– this captures the FULL packets to a file usefull for wireshark the -s0 stops the files being shortened
• tcpdump -i INT port 67   <– view dhcp requests
• tcpdump -eP -nni any host 10.9.4.30 <-disable both name and service port resolution while performing a capture, by using the -nn option; -e Print the link-level header on each dump line. This can be used, for example, to print MAC layer addresses for protocols such as Ethernet and IEEE 802.11. -p--no-promiscuous-mode.
• tcpdump -i any  <- any can be used to tell tcpdump to listen on all interfaces
• tcpdump -n  <- disable to lookup and translate hostnames and ports.

Reference: Understanding TCPDUMP Output

5. FW Monitor

• fw monitor -e 'accept host(192.168.1.12);'   <-- Show packets with IP 192.168.1.12 as SRC or DST
• fw monitor -e 'accept src=192.168.1.12 and dst=192.168.3.3;' <--Show all packets from 192.168.1.12 to 192.168.3.3
• fw monitor -pi ipopt_strip -e 'accept udpport(53);' <--Show UDP port 53 (DNS) packets, pre-in position is before 'ippot_strip'
• fw monitor -m O -e 'accept udp and (sport>1023 or dport>1023);' <-- Show UPD traffic from or to unprivileged ports, only show post-out
• fw monitor -e 'accept net(192.168.1.0,24) and tracert;' <--Show Windows traceroute (ICMP, TTL<30) from and to network 192.168.1.0/24
• fw monitor -v 23 -e 'accept tcpport(80);' <--Show Capture web traffic for VSX virtual system ID 23
• fw monitor -e 'accept ip_p=50 and ifid=0;' <--Show all ESP (IP protocol 50) packets on the interface with the ID 0. (List interfaces and corresponding IDs with fw ctl iflist)
• srfw monitor -o output_file.cap <--Show traffic on a SecuRemote/SecureClient client into a file. srfw.exe is in $SRDIR/bin (C:\Program Files\CheckPoint\SecuRemote\bin)

6. VPN tu

vpn tu  or  vpn tunnelutil

********** Select Option **********

(1)  List all IKE SAs
(2)  List all IPsec SAs
(3)  List all IKE SAs for a given peer (GW) or user (Client)
(4)  List all IPsec SAs for a given peer (GW) or user (Client)
(5)  Delete all IPsec SAs for a given peer (GW)
(6)  Delete all IPsec SAs for a given User (Client)
(7)  Delete all IPsec+IKE SAs for a given peer (GW)
(8)  Delete all IPsec+IKE SAs for a given User (Client)
(9)  Delete all IPsec SAs for ALL peers and users
(0)  Delete all IPsec+IKE SAs for ALL peers and users

(Q)  Quit 



7. Disk/File/Folder Commands

Checkpoint SK60080 displays some solutions to resolve excessive disk consumption on SPLAT/Gaia/IPSO/Lunix OS system. Here are some helpful commands:
a. df -h  (view the partition table and its associated utilization)
b. du -h --max-depth=1 /opt | sort -n -r   (examine disk space utilization at directory-level)
c. ls -1 $FWDIR/conf/db_versions/repository/ | wc -l   (check the number of database revisions on a Security Management server)
d. ls -l $RTDIR/distrib/* | wc -l  (counts the number of records)
e. evstop & evstart (Stop / start the Eventia / SmartEvent)
f. rm -r $RTDIR/distrib/* (Purge this directory of stale records)

g. ls -lR /var/log/dump/usermode/   (Find and delete old core dump files)

h. ls -lR /var/crash/  (Find and delete old core dump files)

i. rm $FWDIR/log/2009*.log*  (removes all old log files for year 2009)

8. Connections

CP-1> fw tab -t connections -s
HOST                  NAME                               ID #VALS #PEAK #SLINKS
localhost             connections                      8158    77   948     179

Note: 
The NAME Id is the actual table number. 
The VALS colum is the current number of connections that are in the connections table at the time the command was run. 
The PEAK number is the max number of connections that have been recorded since the last reboot. 
The SLINKS table is a table of symbolic link that point to the real connection entry. There are usually 4 symbolic links per connection. This way no matter which direction the packet comes, there will be an entry for it. There is more to it than that, but that is the general idea.

CP-1> fw ctl pstat

System Capacity Summary:
  Memory used: 8% (62 MB out of 696 MB) - below watermark
  Concurrent Connections: 0% (79 out of 24900) - below watermark
  Aggressive Aging is in detect mode

Hash kernel memory (hmem) statistics:
  Total memory allocated: 71303168 bytes in 17408 (4096 bytes) blocks using 1 pool
  Total memory bytes  used:  9703728   unused: 61599440 (86.39%)   peak: 18891512
  Total memory blocks used:     2665   unused:    14743 (84%)   peak:     4705
  Allocations: 198489371 alloc, 0 failed alloc, 198382561 free

System kernel memory (smem) statistics:
  Total memory  bytes  used: 117769900   peak: 120093268
  Total memory bytes wasted:   996590
    Blocking  memory  bytes   used:  2530356   peak:  2557584
    Non-Blocking memory bytes used: 115239544   peak: 117535684
  Allocations: 433810 alloc, 28 failed alloc, 432937 free, 0 failed free
  vmalloc bytes  used: 114086588 expensive: no

Kernel memory (kmem) statistics:
  Total memory  bytes  used: 56103032   peak: 66020104
  Allocations: 198922588 alloc, 28 failed alloc
               198815489 free, 0 failed free
  External Allocations: 0 for packets, 0 for SXL

Cookies:
        90753187 total, 0 alloc, 0 free,
        7839 dup, 2107678 get, 160176 put,
        91154457 len, 0 cached len, 0 chain alloc,
        0 chain free

Connections:
        231169 total, 7807 TCP, 4665 UDP, 182351 ICMP,
        36346 other, 0 anticipated, 3 recovered, 79 concurrent,
        948 peak concurrent

Fragments:
        0 fragments, 0 packets, 0 expired, 0 short,
        0 large, 0 duplicates, 0 failures

NAT:
        80509/0 forw, 5266/0 bckw, 85750 tcpudp,
        16 icmp, 10440-949656 alloc

Sync:
        Version: new
        Status: Able to Send/Receive sync packets
        Sync packets sent:
         total : 864451,  retransmitted : 0, retrans reqs : 15,  acks : 1826
        Sync packets received:
         total : 3614413,  were queued : 30, dropped by net : 15
         retrans reqs : 0, received 11745 acks
         retrans reqs for illegal seq : 0
         dropped updates as a result of sync overload: 0
        Callback statistics: handled 11588 cb, average delay : 1,  max delay : 5

9. Check Point SecureXL

To enable SecureXL, run the command:
CP[admin]# fwaccel on

To disable SecureXL, run the command:
CP[admin]# fwaccel off

Note: The fwaccel off command is not persistent and SecureXL will be enabled again after a reboot of the system. SecureXL can be permanently disabled through the CPconfig utility.

To check the number of accelerated connection and other SecureXL statistics: CP[admin]# netstat -f
To check the number of accelerated SA (VPN traffic): CP[admin]# netstat -s
To check overall SecureXL statistics:  CP[admin]# fwaccel stat

10.View Checkpoint Log from CLI

expert mode 

fw log -n | morefw log -n -f | https

or 

normal mode without pipe

11. Revision Control Versions Location on Management Server

[Expert@CP-Management]# cd /opt/CPsuite-R75.20/fw1/conf/db_versions/repository/

[Expert@CP-Management]# ls1  11  12  13  14  15  16  17  18  2  3  4  5  6  7  8  9

All version are in those number directories. Actual version info is in versioning_db.fws

[Expert@CP-Management]# cd database/ 

[Expert@CP-Management]# lsversioning_db.fws

12. Change user cli between BASH and CLISH

HostName> set user admin shell /bin/bash
HostName> save config

[Expert@HostName]# dbset passwd:admin:shell /etc/cli.sh
[Expert@HostName]# dbset :save

or

chsh -s /bin/bash admin

chsh -s /etc/cli.sh admin

for SPLAT the default shell is /bin/cpshell which is not showing from /etc/shells file.

chsh -s /bin/cpshell admin

[admin@CP-Management ~]$ cat /etc/shells

/bin/sh

/bin/bash

/bin/bash2

/bin/tcsh

/bin/csh

[admin@CP-Management ~]$ chsh

Changing shell for root.

New shell [/bin/cpshell]: /bin/bash

Shell changed.

13. Enable SFTP in Gaia

[Expert@HostName]# vi  /etc/ssh/sshd_config 

Uncomment the 'sftp-server' line by deleting the pound '#' character: 

from

#Subsystem sftp /usr/libexec/openssh/sftp-server
to
Subsystem sftp /usr/libexec/openssh/sftp-server

[Expert@HostName]# /etc/init.d/sshd restart

Note: Please check my previous post: Enable SFTP to Checkpoint Gaia OS System for more details.

14. Installation of Hotfixes on Gaia or SPLAT

[Expert@HostName]# tar -zxvf Check_Point_Hotfix_VERSION_OS_sk104443.tgz
[Expert@HostName]# ./SecurePlatform_HOTFIX_NAME
[Expert@HostName]# reboot

Steps to Installation a Jumbo Hotfix for R77.20 on Cluster Environment:
a. install a hotfix on standby cluster member (CP2) then reboot it
b. failover from active cluster member (CP1) to standby cluster (CP2) after standby cluster finished rebooting
c. install hotfix on CP1 and reboot it.

[Expert@FW-CP2:0]# md5sum Check_Point_R77.20.linux.tgz  d788583cf44389b83b0dd6990cb53f63  Check_Point_R77.20.linux.tgz [Expert@FW-CP2:0]# tar -zxvf Check_Point_R77.20.linux.tgz  Actions/ Actions/cpconfig Actions/CheckPackage Actions/CRSValidator Actions/GetPa ...... [Expert@FW-CP2:0]# ./UnixInstallScript  *********************************************************** Welcome to Check Point R77_20_JUMBO_HF installation  *********************************************************** Verifying installation environment for R77_20_JUMBO_HF...Done! The following components will be installed: * R77_20_JUMBO_HF Installation program is about to stop all Check Point Processes. Do you want to continue (y/n) ? y Stopping Check Point Processes...Done! Installing Security Gateway / Security Management R77_20_JUMBO_HF...Done! Installing GAIA R77_20_JUMBO_HF...Done! Installing Performance Pack R77_20_JUMBO_HF...Done! Installing Mobile Access R77_20_JUMBO_HF...Done! ************************************************************************ Package Name                                                    Status ------------                                                    ------ Security Gateway / Security Management R77_20_JUMBO_HF          Succeeded GAIA R77_20_JUMBO_HF                                            Succeeded Performance Pack R77_20_JUMBO_HF                                Succeeded Mobile Access R77_20_JUMBO_HF                                   Succeeded ************************************************************************ Installation program completed successfully. Do you wish to reboot your machine (y/n) ? y Broadcast message from admin (pts/2) (Mon Oct 26 16:37:44 2015): The system is going down for reboot NOW! Broadcast message from admin (pts/2) (Mon Oct 26 16:37:44 2015): The system is going down for reboot NOW! [Expert@FW-CP2:0]#

15. SSH Timeout Solutions

a. Increasing the timeout
set inactivity-timeout 720

b. Ignore Hangup
[Expert@R76GaiaGate1:0]# fw monitor -e "accept host(172.16.0.1);" -o test.cap & [1] 27524
[Expert@R76GaiaGate1:0]# ps -aux | grep "fw monitor"
admin 27524 0.7 2.1 88268 21256 pts/2 S 14:09 0:00 fw monitor -e accept host(172.16.0.1); -o test.cap

Disown the process with this command, specifying the PID:

disown 27524

Closing, or having the SSH session end due to timeout will no longer send a hangup to this process, since it is no longer a child process of the SSH session.

A new SSH session or console session can be started later, and the process can be killed manually to stop it.

kill 27524

[Expert@R76GaiaGate1:0]# nohup fw ctl kdebug -T -f -o debug.txt -m 10 -s 50000 & [1] 30209
nohup: appending output to 'nohup.out'

This example is creating cyclic debug files, 10 files, with a maximum of 50000Kb. Again, the PID is displayed, and the output of the command is piped to the text file 'nohup.out'.

The SSH session can be ended with "exit" or timed out, and the hangup sent to this child process will be ignored, the debug will continue running until we log in again and manually kill the PID.

16. ClusterXL Troubleshooting

16.1 Force a failover

This creates a pnote (problem notification) that is in problem state at current cluster member and force a failover to another member:

cphaprob -d fail -s problem -t 0 register

Verify it's in problem state with

cphaprob stat

and

cphaprob -i list

(you should see 'fail' in problem state)

Once you've finished your testing, run these two to reset it:

cphaprob -d fail -s ok reportcphaprob -d fail unregister

Reference: CheckPoint HA: How to force a failover (ClusterXL/VRRP)

16.2 cphaprob commands and troubleshooting ClustXL Problem

FW-CP2 is fine. But FW-CP1 shows problem on the clustxl status.

[Expert@FW-CP2:0]# cphaprob -a if  Required interfaces: 5 Required secured interfaces: 1 eth1       UP                    non sync(non secured), multicast eth2       UP                    sync(secured), multicast Mgmt       UP                    non sync(non secured), multicast eth3       UP                    non sync(non secured), multicast  (eth3.106  ) eth3       UP                    non sync(non secured), multicast  (eth3.102  ) Virtual cluster interfaces: 6 eth1            2.13.11.60       eth2            10.1.90.14       Mgmt            10.1.72.14       eth3.104        10.1.104.14       eth3.106        10.1.106.14       eth3.102        10.1.102.14

FW-CP1> cphaprob -i list Built-in Devices: Device Name: Interface Active Check Current state: problem Device Name: HA Initialization Current state: OK Device Name: Recovery Delay Current state: OK Registered Devices: Device Name: Synchronization Registration number: 0 Timeout: none Current state: OK Time since last report: 64196.3 sec Device Name: Filter Registration number: 1 Timeout: none Current state: OK Time since last report: 63492.1 sec Device Name: cphad Registration number: 2 Timeout: none Current state: OK Time since last report: 2.68138e+06 sec Device Name: fwd Registration number: 3 Timeout: none Current state: OK Time since last report: 2.68137e+06 sec Device Name: routed Registration number: 4 Timeout: none Current state: OK Time since last report: 62898.8 sec

Usually it was caused by the connection between firewall interface port and  switch port. UDP port 8116 will help us to find out which one is not sending the keep-alive packets:

Cluster Control Protocol (CCP) runs on UDP port 8116, and allows cluster members to report their own states and learn about the states of other members, by sending keep-alive packets (applies only to ClusterXL clusters). Also CCP keeps cluster member sync state.

Following tcpdump shows cluster member 1 (00:00:00:00:fe:00) and cluster member 2 (00:00:00:00:fe:01) both are sending 8116 CCP packets. That is normal. If you only see one sending, you will have to check another one's switch port vlan configuration. You may miss one vlan on switch trunk port, which has happened to me.

[Expert@FW-CP2:0]# tcpdump -enni eth3.102 port 8116 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth3.102, link-type EN10MB (Ethernet), capture size 96 bytes 11:13:17.497801 00:00:00:00:fe:01 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 92: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 50 11:13:17.597743 00:00:00:00:fe:01 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 76: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 34 11:13:17.676067 00:00:00:00:fe:00 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 76: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 34 11:13:17.676182 00:00:00:00:fe:00 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 92: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 50

Note: Previous Troubleshooting Post - Checkpoint Cluster Member Down because interfaces show partially up

17. Permanent Change Global Kernel Parameters Value
Global kernel parameters exist to control (customize) the behavior of Security Gateway (kernel parameters are located in $FWDIR/boot/modules/fw*mod* kernel modules).

This control (customization) can be done on-the-fly using the fw ctl set int command (change takes effect immediately). However, the value of the kernel parameter returns to its default value after a reboot. At times, it may be required to control (customize) the behavior of Security Gateway permanently. In addition, it is necessary for some kernel parameters to be changed upon boot. fwkern.conf file is the one which holds all those kernel parameters value. If it is not existing in your system, you will need to create it manually.

The Security Gateway must be rebooted after any change in the $FWDIR/boot/modules/fwkern.conf file.

[Expert@CP1:0]# cat /opt/CPsuite-R77/fw1/boot/modules/fwkern.conf  fwha_mac_magic=40 fwha_mac_forward_magic=41 fw_allow_simultaneous_ping=1 fwha_forw_packet_to_not_active=1

Useful Checkpoint KBs:

1. sk97638 - Check Point Processes and Daemons
2. sk98348 - Best Practices - Security Gateway Performance

Reference:

1. Check Point/SPLAT/Network Debug Cheat Sheet
2. A tcpdump Primer with Examples
3. Check Point fw monitor cheat sheet – 20141028
4. Check Point CLI Reference Card – 20150617 by Jens Roesen
5. Upgrading ClusterXL Deployments(R77)