There are always bad days during your life. The only thing we could do is to face it and find the solution. Just like today, it supposed to be a good weekend after a quick patch. But things quickly turned to bad way. There was a Checkpoint firewall not coming back after a hotfix installed. System crashed and kept rebooting during loading the policy from local host. (Root cause for this issue is another story.)
Since system crashed and no way for administrator to log in, what we could do is to log in to maintenance mode , either restore from previous backup / image (hopefully you have one, usually I will have a snapsot monthly and remote backup weekly), or uninstall the hotfix.
Usually uninstallation script will save your huge amounts of time from this awkward situation, the worst case is to get into maintenance mode to restore image you took before. Let me list all steps I experienced today:
1. System crushed during rebooting after applied a hotfix from Check Point
INIT: Entering runlevel: 3
Applying Intel CPU microcode update: [ OK ]
Starting sysstat: Calling the system activity data collector (sadc):
[ OK ]
Running UP accel driver check.
IP series driver not present
Starting background readahead: [ OK ]
Checking for hardware changes [ OK ]
Configuring ipv6 kernel support: [ OK ]
Starting kdump:[ OK ]
Inserting ipsctlmod.2.6.18.cp.i686: [ OK ]
CKP: Loading SecureXL: [ OK ]
CKP: Loading FW-1 IPv4 Instance 0: [ OK ]
CKP: Loading VPN-1 Instance 0: [ OK ]
CKP: Loading FW-1 IPv4 Instance 1: [ OK ]
CKP: Loading VPN-1 Instance 1: [ OK ]
FW1: Starting cpWatchDog
Starting wrp:
[ OK ]
Starting auditd: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Fulcrum switch not installed
Update Interfaces in Database: 0 bindings were imported
[ OK ]
Generating vrfs: [ OK ]
Configuring NetAccess: [ OK ]
Generating NTP configuration: [ OK ]
Generating Time Zone configuration: [ OK ]
Generating domain name configuration: [ OK ]
Generating keyboard mapping configuration: [ OK ]
Generating hostname configuration: [ OK ]
Configuring Interfaces: [ OK ]
Generating /etc/monitor_mode: [ OK ]
Generating /etc/fonic_pairs: [ OK ]
Configuring NDP: [ OK ]
Generating hosts.conf: [ OK ]
Generating resolv.conf: [ OK ]
Generating dhclient.conf: [ OK ]
Generating pwcontrol.conf [ OK ]
Generating passwd + shadow [ OK ]
Generating group + gshadow [ OK ]
Generating routed.conf [ OK ]
Generating routed0.conf [ OK ]
Generating extended commands: [ OK ]
Generating MOTD: [ OK ]
Generating banner message: [ OK ]
Generating /etc/raddb/server: [ OK ]
Generating TACACS+ configuration: [ OK ]
Generating /etc/msmtp.conf: [ OK ]
Generating /etc/pam.d/system-auth: [ OK ]
Generating /etc/sysconfig/external.if: [ OK ]
Generating /etc/lldpd.conf: [ OK ]
Generating DHCP server configuration: Write DSTATE called
ServerConfigured = 1
DdnsConfigured = 0
[ OK ]
Generating /etc/adjust_radius: [ OK ]
Running /bin/arp_xlate: [ OK ]
Generating SNMP configuration: [ OK ]
Generating Job Scheduler configuration: [ OK ]
Updating general configuraion file: [ OK ]
Updating syslogd configuration: Reloading syslogd...[ OK ]
Reloading klogd...[ OK ]
[ OK ]
Updating httpd2 configuration: [ OK ]
Updating httpd-ssl configuration: [ OK ]
Applying NetFlow configuration [ OK ]
Configuring PPPoE: [ OK ]
CPshell initialization: [ OK ]
Initializing CP Process Manager..
Starting cp_pm_rl2: [ OK ]
Starting cp_pm_rl3: [ OK ]
Starting cp_pm_rl4: [ OK ]
Starting acpi daemon: [ OK ]
Starting sshd: [ OK ]
Starting arp: <not configured>
Starting xinetd: [ OK ]
Starting bp_init: [ OK ]
Starting bypass_off: [ OK ]
Starting crond: [ OK ]
Starting cpri_d: cpridstart: Starting cprid
[1] 7382
[ OK ]
Starting cpboot: cpstart: Power-Up self tests passed successfully
cpstart: Starting product - SVN Foundation
SVN Foundation: cpWatchDog already running
SVN Foundation: Starting cpd
Multiportal daemon: starting mpdaemon
SVN Foundation started
cpstart: Starting product - VPN-1
FireWall-1: starting external VPN module -- OK
cpwd_admin:
Process CPHAMCSET started successfully (pid=8208)
FireWall-1: Starting fwd
SecureXL disabled, cannot use affinity commands
SecureXL will be started after a policy is loaded.
FireWall-1: Fetching policy
Installing Security Policy Internet-CP-Cluster on all.all@Pub-cp2
wdt stop function not defined
Oops: 0000 [#1]
SMP
last sysfs file: /devices/pci0000:00/0000:00:00.0/class
Modules linked in: w83627ehf(U) hwmon_vid(U) hwmon(U) button(U) xfrm_nalgo(U) crypto_api(U) 8021q(U) wrpmodmod(PU) vpn_1(PU) fw_1(PU) vpn_0(PU) fw_0(PU) simmod(PU) bridge(U) llc(U) ipsctlmod(PU) parport_pc(U) lp(U) parport(U) sg(U) pcspkr(U) bypass_sb_gpio(U) i2c_i801(U) bypass_class(U) igb(U) i2c_core(U) e1000e(U) serio_raw(U) ip_srs_apic(U) dm_snapshot(U) dm_zero(U) dm_mirror(U) dm_mod(U) ata_piix(U) libata(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) ehci_hcd(U) ohci_hcd(U) uhci_hcd(U)
CPU: 1
EIP: 0060:[<f13bf15b>] Tainted: P VLI
EFLAGS: 00010202 (2.6.18-92cp #1)
EIP is at cphwd_api_init+0x82b/0xe90 [simmod]
eax: 5505b527 ebx: 00000005 ecx: 00000000 edx: 00000080
esi: 00000001 edi: f1685580 ebp: f1683120 esp: e2e5b984
ds: 007b es: 007b ss: 0068
Process fw_full (pid: 8553, ti=e2e58000 task=ef452c70 task.ti=e2e58000)
Stack: f1441ac0 00000002 00000000 80405d5a f40e3c74 00000000 f40e3e80 00000000
f13be930 e2e5b9cc f40e3c74 00000000 f2d2eb97 e2e5b9cc f338ae30 00000060
00000202 f40e3e80 00000000 00000000 00000000 00000001 00000002 00000000
Call Trace:
[<e2e5b990>] <0> [<80405d5a>] common_interrupt+0x1a/0x20
[<e2e5b9a4>] <0> [<f13be930>] cphwd_api_init+0x0/0xe90 [simmod]
[<e2e5b9b4>] <0> [<f2d2eb97>] cphwd_api_init_+0x97/0x100 [fw_0]
[<e2e5b9bc>] <0> [<f338ae30>] fwhamultik_validate_not_locked+0x0/0x90 [fw_0]
[<e2e5b9e8>] <0> [<f2d1b0c4>] cphwd_start+0x2174/0x2cc0 [fw_0]
[<e2e5ba64>] <0> [<804388a9>] update_process_times+0x59/0x90
[<e2e5ba74>] <0> [<f2eaa135>] hmem_global_receive_returned_blocks+0x65/0xd0 [fw_0]
[<e2e5ba78>] <0> [<8041e50a>] smp_apic_timer_interrupt+0x7a/0x80
[<e2e5ba84>] <0> [<80405deb>] apic_timer_interrupt+0x1f/0x24
2. Enter into Maintenance Mode
Following Steps will bring your CheckPoint appliance into maintenance mode:
- Connect to the machine over console (serial).
- Reboot the machine (power cycle).
- During the boot, press a key on the "Press any key to see the boot menu" screen. This should open the Check Point Boot Menu. By default, user has only 5 seconds to press any key.
- Choose the "Start in maintenance mode" and press Enter.
- Enter the Admin credentials and press Enter.
3. Uninstall the hotfix from /opt/CPsuite-R77 folder
sh-3.1# fw verList all installed hotfix. You will see that problem one marked with red color:
This is Check Point's software version R77.10 - Build 243
sh-3.1# cpinfo -y
Error: 'Couldn't connect to /tmp/xgets: Connection refused
'.
------------------------
Hotfix versions
------------------------
[FW1]
HOTFIX_R77_10
HOTFIX_R77_HF_HA10_005
HOTFIX_GYPSY_LTE_HF_001
[PPACK]
HOTFIX_R77_10
[SecurePlatform]
HOTFIX_R77_10_GAIA_GHOST_833
[CVPN]
HOTFIX_R77_10
[CPinfo]
No hotfixes..
[SmartLog]
HOTFIX_R77_10
Go to /opt/CPsuite-R77 folder:
Note: Usually it is the parent folder $FWDIR. Based on the version you are having on your Checkpoint Device, the real folder directory is different. In this case, it is Gaia R77.10, and folder is /opt/CPsuite-R77.
sh-3.1# cd CPsuite-R77
sh-3.1# ls
CPinstall fw1_wrapper_HOTFIX_GYPSY_LTE_HF_001_bcp.tgz
LICENSE.TXT fw1_wrapper_HOTFIX_GYPSY_LTE_HF_001_bcp.tgz.new.txt
conf fw1_wrapper_HOTFIX_R77_HF_HA10_005_bcp.tgz
fg1 fw1_wrapper_HOTFIX_R77_HF_HA10_005_bcp.tgz.new.txt
fw1 uninstall_fw1_wrapper_HOTFIX_GYPSY_LTE_HF_001
fw1_wrapper uninstall_fw1_wrapper_HOTFIX_R77_HF_HA10_005
sh-3.1# ls -ali
total 122712
328062 drwxrwx--x 7 admin bin 4096 Mar 15 10:26 .
65537 drwxr-xr-x 19 admin root 4096 Aug 6 2014 ..
328064 drwxrwx--- 2 admin bin 4096 Aug 6 2014 CPinstall
328066 -rwxrwx--- 1 admin bin 38604 Jan 16 2014 LICENSE.TXT
328067 drwxrwx--- 2 admin bin 4096 Aug 6 2014 conf
328069 drwxrwx--- 9 admin bin 4096 Nov 9 01:37 fg1
328095 drwxrwx--x 30 admin bin 4096 Mar 15 12:35 fw1
852062 drwxr-x--- 3 admin bin 4096 Apr 7 2014 fw1_wrapper
327694 -rw-rw---- 1 admin root 72317473 Mar 15 10:25 fw1_wrapper_HOTFIX_GYPSY_LTE_HF_001_bcp.tgz
327692 -rw-rw---- 1 admin root 763 Mar 15 10:24 fw1_wrapper_HOTFIX_GYPSY_LTE_HF_001_bcp.tgz.new.txt
329068 -rw-rw---- 1 admin root 53080782 Aug 6 2014 fw1_wrapper_HOTFIX_R77_HF_HA10_005_bcp.tgz
329067 -rw-rw---- 1 admin root 187 Aug 6 2014 fw1_wrapper_HOTFIX_R77_HF_HA10_005_bcp.tgz.new.txt
327700 -rwxr-x--- 1 admin bin 18224 Nov 9 01:37 uninstall_fw1_wrapper_HOTFIX_GYPSY_LTE_HF_001
329069 -rwxr-x--- 1 admin bin 18218 Apr 7 2014 uninstall_fw1_wrapper_HOTFIX_R77_HF_HA10_005
sh-3.1# ./uninstall_fw1_wrapper_HOTFIX_GYPSY_LTE_HF_001
Validating uninstall archive...
Do you want to proceed with uninstallation of
Security Gateway Power/UTM R77.10 GYPSY_LTE_HF_001 on this computer?
If you choose to proceed, uninstall will perform CPSTOP.
To proceed type y to cancel type n :
y
cpwd_admin: Failed to submit request to cpWatchDog
cvpnd: no process killed
dbwriter: no process killed
cvpnproc: no process killed
MoveFileServer: no process killed
CvpnUMD: no process killed
Mobile Access: Stopping MoveFileDemuxer service (if needed)
cpwd_admin: Failed to submit request to cpWatchDog
Mobile Access: MoveFileDemuxer is not running
Exception: connect() failed - Network is unreachable
Multiportal daemon is not running
Pinger: no process killed
Mobile Access: Successfully stopped Mobile Access services
cpwd_admin: Failed to submit request to cpWatchDog
SmartView Monitor: Unable to find CpWatchDog - run cpstart
FloodGate-1 is already stopped.
Unable to open '/dev/fw0': No such file or directory
fw_syncn_set: failed to set off synchronization
cpwd_admin: Failed to submit request to cpWatchDog
Unable to open '/dev/fw0': No such file or directory
Failed to notify kernel: No such file or directory
HA not stopped.
VPN-1/FW-1 stopped
Multi portal stopped
fw: Unable to open '/dev/fw0': Unknown error 4294967295
fw: Set operation failed: failed to get parameter
fw: set: Operation failed: Unknown error 4294967295
SVN Foundation: cpd is not running
Multiportal daemon: mpdaemon is not running
cpwd_admin: Failed to submit request to cpWatchDog
SVN Foundation: cpWatchDog is not running
SVN Foundation stopped
Launching pre-uninstall utility
Removing gx.lf file from registry...
****************
Security Gateway Power/UTM R77.10
Security Gateway Power/UTM R77.10 GYPSY_LTE_HF_001
Uninstall completed successfully.
****************
***********************************************************
Don't forget to reboot the machine!!
***********************************************************
sh-3.1# reboot
Preforming soft reboot
INIT: Sending processes the TERM signal
INIT: Starting killall: [ OK ]
Starting bypass_on: [ OK ]
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Saving random seed:
Syncing hardware clock to system time
Turning off swap:
Unmounting file systems:
mount: /proc is busy
Please stand by while rebooting the system...
Restarting system.
4. Verify Hotfix uninstalled
You will find HOTFIX_GYPSY_LTE_HF_001 has gone from the list.[Expert@Pub-CP1:0]# cpinfo -y
------------------------
Hotfix versions
------------------------
[FW1]
HOTFIX_R77_10
HOTFIX_R77_HF_HA10_005
[SecurePlatform]
HOTFIX_R77_10_GAIA_GHOST_833
[PPACK]
HOTFIX_R77_10
[CVPN]
HOTFIX_R77_10
[CPinfo]
No hotfixes..
[SmartLog]
HOTFIX_R77_10
[rtm]
No hotfixes..
No comments:
Post a Comment