Sunday 29 December 2013

VCS SPLIT BRAIN IN VCS 6.X ON SOLARIS 11

1)TO SEE RESERVATION KEYS
-bash-4.1# vxfenadm -s all -f /etc/vxfentab
Device Name: /dev/rdsk/c3d24s2
Total Number Of Keys: 2
key[0]:
        [Numeric Format]:  86,70,48,48,54,52,48,48
        [Character Format]: VF006400
   *    [Node Format]: Cluster ID: 100   Node ID: 0   Node Name: EPEP2SP
key[1]:
        [Numeric Format]:  86,70,48,48,54,52,48,49
        [Character Format]: VF006401
   *    [Node Format]: Cluster ID: 100   Node ID: 1   Node Name: EPEP2SF
Device Name: /dev/rdsk/c3d23s2
Total Number Of Keys: 2
key[0]:
        [Numeric Format]:  86,70,48,48,54,52,48,48
        [Character Format]: VF006400
   *    [Node Format]: Cluster ID: 100   Node ID: 0   Node Name: EPEP2SP
key[1]:
        [Numeric Format]:  86,70,48,48,54,52,48,49
        [Character Format]: VF006401
   *    [Node Format]: Cluster ID: 100   Node ID: 1   Node Name: EPEP2SF
Device Name: /dev/rdsk/c3d22s2
Total Number Of Keys: 2
key[0]:
        [Numeric Format]:  86,70,48,48,54,52,48,48
        [Character Format]: VF006400
   *    [Node Format]: Cluster ID: 100   Node ID: 0   Node Name: EPEP2SP
key[1]:
        [Numeric Format]:  86,70,48,48,54,52,48,49
        [Character Format]: VF006401
   *    [Node Format]: Cluster ID: 100   Node ID: 1   Node Name: EPEP2SF

-bash-4.1# hastop -all
VCS ERROR V-16-1-10600 Cannot connect to VCS engine

-bash-4.1# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   563f0d membership 01
Port a gen   563f0d   jeopardy ;1

-bash-4.1# /etc/init.d/vxfen.rc stop
-bash: /etc/init.d/vxfen.rc: No such file or directory
[[[[[[[[[ OR  ]]]]]]]]]]]]
vxfen-shutdown

-bash-4.1# /opt/VRTSvcs/vxfen/bin/vxfenclearpre
         ******** WARNING!!!!!!!! ********
This script recovers from a preexisting split brain condition.
This script removes the SCSI-3 registrations on the coordinator
disks contained in the file /etc/vxfentab on this node EPEP2SF.
This script also removes the SCSI-3 registrations and reservations
on the data disks contained in shared disk groups.
This script will remove all the keys from the coordinator and data disks.
VERIFY ALL OTHER NODES ARE POWERED OFF OR INCAPABLE OF
ACCESSING SHARED STORAGE.
If this is not the case, data corruption will result.
This script will clear keys from the disks as follows.
/dev/rdsk/c3d22s2: VF006400 VF006401
EMC%5FSYMMETRIX%5F000295700464%5F6401A4A008: VXFEN vxfenadm ERROR V-11-2-1116 Cannot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4A008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F000295700464%5F6401A4A008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned
/dev/rdsk/c3d23s2: VF006400 VF006401
EMC%5FSYMMETRIX%5F000295700464%5F6401A4B008: VXFEN vxfenadm ERROR V-11-2-1116 Cannot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4B008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F000295700464%5F6401A4B008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned
/dev/rdsk/c3d24s2: VF006400 VF006401
EMC%5FSYMMETRIX%5F000295700464%5F6401A4C008: VXFEN vxfenadm ERROR V-11-2-1116 Cannot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4C008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F000295700464%5F6401A4C008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned

Do you still want to continue: [y/n] (default : n)?yy
Please respond with (y/n)y
Cleaning up the coordinator disks ...
VXFEN vxfenadm ERROR V-11-2-1116 Cannot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4A008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F000295700464%5F6401A4A008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned
VXFEN vxfenadm ERROR V-11-2-1116 Cannot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4A008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F000295700464%5F6401A4A008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned
VXFEN vxfenadm ERROR V-11-2-1116 Cannot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4B008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F000295700464%5F6401A4B008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned
VXFEN vxfenadm ERROR V-11-2-1116 Cannot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4B008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F000295700464%5F6401A4B008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned
VXFEN vxfenadm ERROR V-11-2-1116 Cannot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4C008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F000295700464%5F6401A4C008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned
VXFEN vxfenadm ERROR V-11-2-1116 Cannot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4C008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F000295700464%5F6401A4C008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned
Cleared keys from 6 out of 3 disks
Successfully removed SCSI-3 persistent registrations
from the coordinator disks.
Cleaning up the data disks for all shared disk groups ...

Successfully removed SCSI-3 persistent registration and reservations
from the shared data disks.
You can retry starting fencing module. In order to restart the whole
product, you might want to reboot the system.


-bash-4.1#  /opt/VRTSvcs/vxfen/bin/vxfenclearpre
         ******** WARNING!!!!!!!! ********
This script recovers from a preexisting split brain condition.
This script removes the SCSI-3 registrations on the coordinator
disks contained in the file /etc/vxfentab on this node EPEP2SP.
This script also removes the SCSI-3 registrations and reservations
on the data disks contained in shared disk groups.
This script will remove all the keys from the coordinator and data disks.
VERIFY ALL OTHER NODES ARE POWERED OFF OR INCAPABLE OF
ACCESSING SHARED STORAGE.
If this is not the case, data corruption will result.
This script will clear keys from the disks as follows.
/dev/rdsk/c3d22s2:
EMC%5FSYMMETRIX%5F000295700464%5F6401A4A008: VXFEN vxfenadm ERROR V-11-2-1116 Ca                                                                             nnot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4A008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F0002957                                                                             00464%5F6401A4A008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned
/dev/rdsk/c3d23s2:
EMC%5FSYMMETRIX%5F000295700464%5F6401A4B008: VXFEN vxfenadm ERROR V-11-2-1116 Ca                                                                             nnot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4B008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F0002957                                                                             00464%5F6401A4B008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned
/dev/rdsk/c3d24s2:
EMC%5FSYMMETRIX%5F000295700464%5F6401A4C008: VXFEN vxfenadm ERROR V-11-2-1116 Ca                                                                             nnot open: EMC%5FSYMMETRIX%5F000295700464%5F6401A4C008
VXFEN vxfenadm ERROR V-11-2-1132 Open of file failed, errno = 2
VXFEN vxfenadm ERROR V-11-2-1205 READ_KEYS failed for: EMC%5FSYMMETRIX%5F0002957                                                                             00464%5F6401A4C008
VXFEN vxfenadm ERROR V-11-2-1133 Error returned

Disks are already clear of expected keys.

-bash-4.1#  vxfenadm -s all -f /etc/vxfentab
Device Name: /dev/rdsk/c3d23s2
Total Number Of Keys: 0
No keys...
Device Name: /dev/rdsk/c3d22s2
Total Number Of Keys: 0
No keys...
Device Name: /dev/rdsk/c3d24s2
Total Number Of Keys: 0
No keys...

-bash-4.1#  cpsadm -s 10.128.45.215  -a list_membership  -c EP2_CLUSTER
Connection failed for host 10.128.45.215 on port 14250
Failed to create connection
Error in connecting with CPS
-bash-4.1#  cpsadm -s 10.128.45.215  -a list_membership  -c EP2_CLUSTER
Connection failed for host 10.128.45.215 on port 14250
Failed to create connection
Error in connecting with CPS
-bash-4.1#  cpsadm -s 10.128.45.215  -a list_membership  -c EP2_CLUSTER
Connection failed for host 10.128.45.215 on port 14250
Failed to create connection
Error in connecting with CPS
-bash-4.1#  cpsadm -s 10.128.45.215  -a list_membership  -c EP2_CLUSTER
Connection failed for host 10.128.45.215 on port 14250
Failed to create connection
Error in connecting with CPS


 

No comments:

Post a Comment