RMC (Resource Monitoring and Control):

RMC is a distributed framework and architecture that allows the HMC to communicate with a managed logical partition. RMC daemons should be running on AIX partition in order to be able to do DLPAR operations on HMC.

For example "Dynamic LPAR Resource Manager" is an RMC daemon that runs inside the AIX (and VIO server). The HMC uses this capability to remotely execute partition specific commands.

The daemons in the LPARs and the daemons on the HMC must be able to communicate to the AIX partition through an external network not through the Service Processor. An external network that the partition has access to and the HMC has acces to as well.

For example, if HMC has a connection to a 9.x.x.x network and I put my AIX partition to that 9.x.x.x network and as long there is network connectivity (HMC is allowed to communicate to that partition over that network) and RMC daemon is running on the partition, then DLPAR operations are available.

In order for RMC to work, port 657 upd/tcp must be open in both directions between the HMC public interface and the lpar.

The RMC daemons are part of the Reliable, Scalable Cluster Technology (RSCT) and are controlled by the System Resource Controller (SRC). These daemons run in all LPARs and communicate with equivalent RMC daemons running on the HMC. The daemons start automatically when the operating system starts and synchronize with the HMC RMC daemons.

Note: Apart from rebooting, there is no way to stop and start the RMC daemons on the HMC!

----------------------------------------

HMC and LPAR authentication (RSCT authentication)
(RSCT authentication is used to ensure the HMC is communicating with the correct LPAR.)

Authentication is the process of ensuring that another party is who it claims to be. 
Authorization is the process by which a cluster software component grants or denies resources based on certain criteria. 
The RSCT component that implements authorization is RMC. It uses access control list (ACL) files to control user access to resources.


The RSCT authorization process in detail:
1. On the HMC: DMSRM pushes down a secret key and HMC IP address to NVRAM where AIX LPAR exists.

2. On the AIX LPAR: CSMAgentRM, reads the key and HMC IP address out from NVRAM. It will then authenticate the HMC. This process is repeated every five minutes on a LPAR to detect new HMCs.

3. On the AIX LPAR: After authenticating the HMC, CSMAgentRM will contact the DMSRM on the HMC to create a ManagedNode resource then it creates a ManagementServer resource on AIX.

4. On the AIX LPAR: After the creation of these resources on the HMC and AIX, CSMAgentRM grants HMC permission to access necessary resources on the LPAR and changes its ManagedNode Status to 1 on the HMC.

5. On the HMC: After the ManagedNode Status is changed to 1, a session is established with the LPAR to query operating system information and DLPAR capabilities, and then waits for the DLPAR commands from users.

----------------------------------------

RMC Domain Status

When partitions have active RMC connections, they become managed nodes in a Management Domain. The HMC is then the Management Control Point (MCP) of that Management Domain. You can then use the rmcdomainstatus command to check the status of those managed nodes (i.e. your partitions).

As root on the HMC or on the AIX LPAR you can execute the rmcdomainstatus command as follows:

# /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc

From HMC: You should get a list of all the partitions that the HMC server can reach on the public network on port 657. 

Management Domain Status: Managed Nodes
  O a  0xc8bc2c9647c1cef3  0003  9.2.5.241
  I a  0x96586cb4b5fc641c  0002  9.2.5.33


From LPAR: You should get a list of all the Management Control Points

Management Domain Status: Management Control Points
   I A  0xef889c809d9617c7 0001  9.57.24.139


1. column:
-I: Indicates that the partition is "Up" as determined by the RMC heartbeat mechanism (i.e. an active RMC connection exists). 
-O: Indicates that the RMC connection is "Down", as determined by the RMC heartbeat mechanism.

2. column:
-A: Indicates that there are no messages queued to the specified node
-a: Same as A, but the specified node is executing a version of the RMC daemon that is at a lower code level than the local RMC daemon.

more info: https://www-304.ibm.com/support/docview.wss?uid=isg3T1011508

----------------------------------------

If rmcdomainstatus shows "i" at (1st column):

Indicates that the partition is Pending Up. Communication has been established, but the initial handshake between two RMC daemons has not been completed (message authentication is most likely failing.)
Authentication problems will occur when the partition identity do not match each other's trusted host list:

# /usr/sbin/rsct/bin/ctsvhbal        <--list the current identities for the HMC and the logical partition (run this command on both)
# /usr/sbin/rsct/bin/ctsthl -l       <--list the trusted host list on the partition

On the HMC, there is an entry for the partition. On the partition, there is an entry for the HMC. The HOST_IDENTITY value must match one of the identities listed in the respective ctsvhbal command output. 

---------------------------------------- 

Things to check at the HMC:

- checking the status of the managed nodes: /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc  (you must be root on the HMC)

- checking connection between HMC and LPAR:
hscroot@hmc10:~> lspartition -dlpar
<#0> Partition:<2*8204-E8A*0680E12 aix10.domain.com, 10.10.50.18>
       Active:<1>, OS:<AIX, 6.1, 6100-03-01-0921>, DCaps:<0x4f9f>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<1452>
<#1> Partition:<4*8204-E8A*0680E32 aix20.domain.com, 10.10.50.71>
       Active:<0>, OS:<AIX, 6.1, 6100-04-05-1015>, DCaps:<0x0>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<656>

For correct DLPAR function:
- the partition must return with the correct IP of the lpar.
- the active value (Active:...) must be higher than zero,
- the decaps value (DCaps:...) must be higher 0x0

(The first line shows a DLPAR capable LPAR, the second line is anon-working LPAR)

- another way to check RMC connection: lssyscfg -r lpar -F lpar_id,name,state,rmc_state,rmc_ipaddr -m p750
(It should list "active" for the LPARs with active RMC connection.)

----------------------------------------

Things to check at the LPAR:

- checking the status of the managed nodes: /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc

- Checking RMC status:
# lssrc -a | grep rsct
 ctrmc            rsct             8847376      active          <--it is a RMC subsystem
 IBM.DRM          rsct_rm          6684802      active          <--it is for executing the DLPAR command on the partition
 IBM.DMSRM        rsct_rm          7929940      active          <--it is for tracking statuses of partitions
 IBM.ServiceRM    rsct_rm          10223780     active
 IBM.CSMAgentRM   rsct_rm          4915254      active          <--it is for  handshaking between the partition and HMC
 ctcas            rsct                          inoperative     <--it is for security verification
 IBM.ERRM         rsct_rm                       inoperative
 IBM.AuditRM      rsct_rm                       inoperative
 IBM.LPRM         rsct_rm                       inoperative
 IBM.HostRM       rsct_rm                       inoperative     <--it is for obtaining OS information

You will see some active and some missing (The key for DLPAR is the IBM.DRM)

- Stopping and starting RMC without erasing configuration:

# /usr/sbin/rsct/bin/rmcctrl -z    <--it stops the daemons
# /usr/sbin/rsct/bin/rmcctrl -A    <--adds entry to /etc/inittab and it starts the daemons
# /usr/sbin/rsct/bin/rmcctrl -p    <--enables the daemons for remote client connections

(This is the correct method to stop and start RMC without erasing the configuration.)
Do not use stopsrc and startsrc for these daemons; use the rmcctrl commands instead!

- recfgct: deletes the RMC database, does a discovery, and recreates the RMC configuration
# /usr/sbin/rsct/install/bin/recfgct
(Wait several minutes)
# lssrc -a | grep rsct

(If you see IBM.DRM active, then you have probably resolved the issue)

- lsrsrc "IBM.ManagementServer"    <--it shows HMCs via RMC

http://www.ibm.com/developerworks/aix/library/au-LPM_troubleshooting/

Introduction

Live Partition Mobility (LPM) was introduced on Power6. It helps to avoid downtime during VIOS and firmware updates when migrating to other frames. LPM also reduces the amount of work that is required while creating a new LPAR and set-up, which is required for the application.

A majority of customers perform LPM activities on a daily basis, and many may not know the exact procedure or what is taking place. This article shows steps to overcome or fix LPM issues.

Figure 1. The AIX I/O stack

Back to top

LPM key points

Things to remember about LPM are that it migrates running partitions from one physical server to another while maintaining complete transactional integrity and transfers the entire environment: processor state, memory, virtual devices, and connected users. Partitions may also migrate while powered off (inactive migration), and the operating system and application must reside on shared storage.

Back to top

LPM prerequisites

You must have a minimum of two machines, a source and a destination, on POWER6 or higher with the Advanced Power Virtualization Feature enabled. The operating system and application must reside on a shared external storage (Storage Area Network). In addition to these hardware requirements, you must have:

• One hardware management console (optional) or IVM.
• Target system must have sufficient resources, like CPU and memory.
• LPAR should not have physical adapters.

Your virtual I/O servers (VIOS) must have a Shared Ethernet Adapter (SEA) configured to bridge to the same Ethernet network which the mobile partition uses. It must be capable of providing virtual access to all the disk resources which the mobile partition uses (NPIV or vSCSI). If you are using vSCSI, then the virtual target devices must be physical disks (not logical volumes).

You must be at AIX version 5.3J or later, VIOS version 1.4 or later, HMC V7R310 or later and the firmware at efw3.1 or later.

Back to top

What happens at the time of LPM

Figure 2. General LPM depiction

The following describes the general LPM depiction in Figure 2:

1. Partition profile (presently active) copied from source to target FSP.
2. Storage is configured on the Target.
3. Mover service partitions (MSP) is activated.
4. Partition migration started.
   1. Majority of memory pages moved.
   2. All threads piped down.
5. Activation resumed on target.
   1. Final memory pages moved.
   2. Cleanup storage and network traffic.
6. Storage resources are deconfigured from the source.
7. Partition profile removed from source FSP (Flexible Service Processor).

Back to top

How to do LPM

Before doing LPM, we need to verify the availability of resources on both the source and destination side. If validation fails with some error, then we have to fix it to proceed further. Sometimes validation may end up with warning messages which can be ignored.

LPM using HMC GUI

Figure 3 shows you how to validate the LPAR with the HMC GUI.

From the System management -> Servers -> Trim screen, select the LPAR name: Operations -> Mobility -> Validate

Figure 3. Validating the LPAR

The validate screen, shown in Figure 4, shows that upt0052 LPAR is validated for migration from trsim to dash, and if needed, we have to specify the destination HMC.

Figure 4. Validation window

Figure 5 show that the LPM has ended with a warning message, ignore the message and select Close to continue with the migration.

Figure 5. Validation passed with general warning message

Figure 6, the Partition Migration Validation screen, shows that the information is selected to set up a migration of the partition to a different managed system. Select Migrate to verify the information.

Figure 6. Ready for migration after validation passed

When the migration completes, as shown in Figure 7, select Close

Figure 7. Migration progressing

HMC using commandline option

To validate the LPM in local HMC, enter the following:

migrlpar -o v -m [source cec] -t [target cec] -p [lpar to migrate]

To validate the LPM in the Remote HMC, type:

migrlpar -o v -m [source cec] -t [target cec] -p [lpar to migrate] \
> --ip [target hmc] -u [remote user]

Note, you may prefer to use the hscroot command as the remote user.

Use the following migration command for LPM in the local HMC:

migrlpar -o m -m [source cec] -t [target cec] -p [lpar to migrate]

The following migration command for LPM is used with the remote HMC:

migrlpar -o m -m [source cec] -t [target cec] -p [lpar to migrate] \
> --ip [target hmc] -u [remote user]

In case of MPIO (Multipath IO) failure of a LPAR due to configuration issues between source and destination, type the following to proceed (if applicable):

migrlpar -o m -m wilma -t visa -p upt07 --redundantpgvios 0 -n upt07_n
oams_npiv -u hscroot --vlanbridge 2 --mpio 2 -w 60 -d 5 -v -i
"source_msp+name=wilmav2,dest_msp_name=visav2" --ip destiny4

Back to top

Troubleshooting

This section covers various errors messages you might encounter and ways to correct them.

• If LPM needs to be done across two different HMCs, in case of migration, the appropriate authorization between HMCs needs to be set. If proper authorization is not set, the following mkauthkey error displays:

  hscroot@destiny4:~> migrlpar -o v -m trim -p  UPT0052 --ip hmc-arizona -u
  hscroot -t arizona
  			        
  HSCL3653 The Secure Shell (SSH) communication configuration between the source
  and target Hardware Management Consoles has not been set up properly for user
  hscroot. Please run the mkauthkeys command to set up the SSH communication
  authentication keys.

  To overcome this error, type the following:

  hscroot@destiny4:~> mkauthkeys -g --ip hmc-arizona -u hscroot
  Enter the password for user hscroot on the remote host hmc-arizona

• If migrating an POWER7 Advanced Memory Expansion (AME) partition, to any of the POWER6 machines, the following error displays:

  hscroot@destiny4:~> migrlpar -o v -m trim -p  
      UPT0052 --ip hmc-liken -u hscroot -t wilma
     
  HSCLA318 The migration command issued to the destination HMC failed with the 
  following error: HSCLA335 The Hardware Management Console for the destination 
  managed system does not support one or more capabilities required to perform 
  this operation. The unsupported capability codes are as follows: AME_capability
  hscroot@destiny4:~>

  To correct this error either migrate to POWER7 or remove the AME and then migrate.

• If you are doing a migration of an Active Memory Sharing (AMS) partition with improper AMS setup or no free paging device on the destination side, the following error displays:

  hscroot@hmc-liken:~> migrlpar -o v -m wilma -t visa --ip destiny4 -u hscroot -p
  upt0060 --mpio 2
       
  Errors:
  HSCLA304 A suitable shared memory pool for the mobile partition was not found on the
  destination managed system. In order to support the mobile partitions, the
  destination managed system must have a shared memory pool that can accommodate the
  partition's entitled and maximum memory values, as well ad its redundant paging
  requirements. If the destination managed system has a shared memory pool, inability
  to support the mobile shared memory partition can be due to lack of sufficient memory
  in the pool, or lack of a paging space device in the pool that meets the mobile
  partition's redundancy and size requirements. 
       
  Details:
  HSCLA297 The DLPAR Resource Manager (DRM) capability bits )x) for mover service
  partition (MSP) visav2 indicate that partition mobility functions are not supported
  on the partition.
  HSCLA2FF An internal Hardware Management Console error has occurred. If this error
  persists, contact your service representative.

  To correct this error do either, or both, of the following:

  • Since this problem is related to redundant AMS setup, the destination AMS pool should have redundant capability for an AMS pool defined as Shared Memory Pool with two Paging VIOS partitions for high availability HMC only. Users can select primary and alternate paging VIOS for each Shared Memory Partition. For any details related to AMS, please refer to "Configuring Active Memory Sharing from a customer's experience" (developerWorks, Aug 2009) for more information.
  • A sufficient space for paging device should be present in the target AMS pool.
• If we try to migrate a LPAR from Power7 to Power6 CPU, we get the following error:

  hscroot@destiny4:~> migrlpar -o v -m dash -t arizona --ip hmc-arizona -u hscroot
  -p upt0053
          
  Errors:
  HSCLA224 The partition cannot be migrated because it has been designated to use a 
  processor compatibility level that is not supported by the destination managed 
  system. Use the HMC to configure a level that is compatible with the destination 
  managed system.

  The solution for the above error could be one of the following:

  • Migrate to POWER7.
  • Change the processor mode to appropriate mode (as in the destination managed system).

    The steps to change processor mode in HMC GUI are:

    • Select the LPAR and deactivate it.
    • Go to Configuration->Manage Profiles.
    • Select the profile that needs to be activated.
    • Go to Processors, change the Processor compatibility mode: to the required setting and boot it using the same profile.
  • LPAR should have the same shared vSCSI disks on source and destination MSPs:

    hscroot@destiny4:~> migrlpar -o v -m dash -t arizona --ip hmc-arizona -u hscroot
    -p upt0058
            
    Errors:
    The migrating partition's virtual SCSI adapter cannot be hosted by the existing 
    virtual I/O server (VIOS) partitions on the destination managed system. To 
    migrate the partition, set up the necessary VIOS hosts on the destination 
    managed system, then try the operation again. 
            
    Details:
    HSCLA356 The RMC command issued to partition arizona failed. This means that 
    destination VIOS partition arizona2 cannot host the virtual adapter 6 on the 
    migrating partition.
            
    HSCLA29A The RMC command issued to partition failed. 
    The partition command is:
    migmgr -f find_devices -t vscsi -C 0x3 -d 1
    The RMC return code is:
    0
    The OS command return code is:
    85
    The OS standard out is:
    Running method '/usr/lib.methods/mig_vscsi
    85
    The OS standard err is:
            
    The search was performed for the following device descriptions:
            <v-scsi-host>
                     <generalInfo>    
                        <version>2.0 </version>
                        <maxTransfer>262144</maxTransfer>
                        <minVIOSpatch>0</minVIOSpatch>
                        <minVIOScompatability>1</minVIOScompatability>
                        <effectiveVIOScompatability>1</effectiveVIOScompatability>
                      <generalInfo>
                      <ras>
                            <partitionID>2</partitionID>
                       </ras>
                       <virtDev>
                                <vLUN>
                                            <LUA>0x81000000000000000</LUA>
                                            <LUNState>0</LUNState>
                                            <clientReserve>no</clientReserve>
                                            <AIX>
                                                    <type>vdasd</type>
                                                    <connWhere>1</connWhere>
                                            </AIX>
                                </vLUN>
                                <blockStirage>
                                            <reserveType>NI_RESERVE</reserveType>
                                            <AIX>
    
                               <udid>261120017380003D30194072810XIV03IBMfcp</udid>
                                                    <type>UDID</type>
                                            </AIX>
                                </blockStirage>
                        </virtDev>
         </v-scsi-host>

    And, the solution is as follows:

    • Make sure destination MSP has access to same vSCSI disks as source MSP.
    • Also make sure disks are not reserved.

  In cases where the mapping is correct, and you are still getting the same error, it may be due to having different types of FC adapters between source and destination MSP. For mapping methods, refer to last Note section of "Troubleshooting".

• In the destination CEC, if the LPAR has insufficient CPUs, then we get the following error:

  hscpe@destiny4:~> migrlpar -o v -m dash -t wilma -p upt0053 --ip defiant2 -u
  hscroothmc-arizona -u hscroot
  Errors:
  The partition cannot be migrated because the processing resources it requires 
  exceeds the available processing resources in the destination managed system's 
  shared processor pool. If possible, free up processing resources from that shared 
  processor pool and try the operation again.

  And the solution is:

  • We need to reduce the CPU in LPAR by DLPAR or change the profile.
  • We can increase the number of processors at destination machine by reducing the number of processor units using DLPAR operation on a few of its clients (if applicable).
• If the destination CEC does not have sufficient memory, then:

  hscpe@destiny4:~> migrlpar -o v -m extra5 -t dash -p upt0027
  Errors:
  There is not enough memory: Obtained: 2816, Required: 4608.  Check that there is 
  enough memory available to activate the partition. If not, create a new profile or 
  modify the existing profile with the available resources, then activate the 
  partition. If the partition must be activated with these resources, deactivate any 
  running partition or partitions using the resource, then activate this partition.

  And, the solution is either:

  • We need to reduce the amount of memory in LPAR by using DLPAR operation or by changing the profile; or,
  • We can increase the memory at the destination machine by reducing the memory of any other LPARs using DLPAR operation.

  If the RMC (Resource Monitoring and Control) connection is not established among the source, target VIOS's and LPAR, then we may get following error:

  hscpe@destiny4:~> migrlpar -o v -m dash -t trim -p upt0053
  Errors: 
  The operation to check partition upt0053 for migration readiness has failed. 
  The partition command is:
  drmgr -m -c pmig -p check -d 1
  The partition standard error is:
            
  HSCLA257 The migrating partition has returned a failure response to the HMC's
  request to perform a check for migration readiness. The migrating partition in
  not ready for migration at this time. Try the operation again later.
            
  Details:
  HSCLA29A  The RMC command issued to partition upt0053 failed. \
  The partition commend is:
  drmgr -m -c pmig -p check -d 1
  The RMC return code is:
  1141
  The OS command return code is:
  0
  The OS standard out is:
  Network interruption occurs while RMC is waiting for the execution of the command
  on the partition to finish.
  Either the partition has crashed, the operation has caused CPU starvation, or
  IBM.DRM has crashed in the middle of the operation.
  The operation could have completed successfully. (40007) (null)
  The OS standard err is:

  • If the partition you are trying to migrate is having MPIO with dual VIOS setup, and the target having dual VIOS but not set up properly for MPIO, then we may get error listed below:

    hscroote@hmc-liken:~> migrlpar -o v -m wilma -t visa --ip destiny4 -u hscroot -p
    upt0060
    Errors:
    HSCLA340 The HMC may not be able to replicate the source multipath I/O
    configuration for the migrating partition's virtual I/O adapters on the 
    destination. This means one or both of the following: (1) Client adapters 
    that are assigned to different source VIOS hosts may be assigned to a single 
    VIOS host on the destination; (2) Client adapters that are assigned to a single 
    source VIOS host may be assigned to different VIOS hosts on the destination. 
    You can review the complete list of HMC-chosen mappings by issuing the command 
    to list the virtual I/O mappings for the migrating partition. 
    HSCLA304 A suitable shared memory pool for the mobile partition was not found 
    on the destination managed system. In order to support the mobile partition, 
    the destination managed system must have a shared memory pool that can 
    accommodate the partition's entitled and maximum memory values, as well as its 
    redundant paging requirements. If the destination managed system has a shared 
    memory pool, inability to support the mobile shared memory partition can be due 
    to lack of sufficient memory in the pool, or lack of a paging space device in 
    the pool that meets the mobile partition's redundancy and size requirements. 
    Details:
    HSCLA297 The DLPAR Resource Manager (DRM) capability bits 0x0 for mover service
    partition (MSP) visav2 indicate that partition mobility functions are not 
    supported on the partition.
    HSCLA2FF  An internal Hardware Management Console error has occurred. If this 
    error persists, contact your service representative. 
    Warning:
    HSCLA246  The HMC cannot communicate migration commands to the partition visav2.
    Either the network connection is not available or the partition does not have a 
    level of software that is capable of supporting partition migration. Verify the 
    correct network and migration setup of the partition, and try the operation 
    again.

    The solution is:

    • Check the correctness of dual VIOS, availability of adapters, mappings in SAN and switch.

    If above solution is not feasible to implement then:

    • Use --mpio 2 with the migrlpar command. By using this, we may lose dual VIOS setup for MPIO disks. Generally this is not a recommended solution by PowerVM.
  • If the Source VIOS is having non-recommended NPIV, we will get the following error:

    hscroote@hmc-liken:~> migrlpar -o v -m wilma -t visa --ip destiny4 -u hscroot -p
    upt0060
    Errors:
    HSCLA340 The HMC may not be able to replicate the source multipath I/O
    configuration for the migrating partition's virtual I/O adapters on the 
    destination. This means one or both of the following: (1) Client adapters 
    that are assigned to different source VIOS hosts may be assigned to a single 
    VIOS host on the destination; (2) Client adapters that are assigned to a single 
    source VIOS host may be assigned to different VIOS hosts on the destination. 
    You can review the complete list of HMC-chosen mappings by issuing the command 
    to list the virtual I/O mappings for the migrating partition. 
    HSCLA304 A suitable shared memory pool for the mobile partition was not found 
    on the destination managed system. In order to support the mobile partition, 
    the destination managed system must have a shared memory pool that can 
    accommodate the partition's entitled and maximum memory values, as well as its 
    redundant paging requirements. If the destination managed system has a shared 
    memory pool, inability to support the mobile shared memory partition can be due 
    to lack of sufficient memory in the pool, or lack of a paging space device in 
    the pool that meets the mobile partition's redundancy and size requirements. 
    Details:
    HSCLA297 The DLPAR Resource Manager (DRM) capability bits 0x0 for mover service
    partition (MSP) visav2 indicate that partition mobility functions are not 
    supported on the partition.
    HSCLA2FF  An internal Hardware Management Console error has occurred. If this 
    error persists, contact your service representative. 
    Warning:
    HSCLA246  The HMC cannot communicate migration commands to the partition visav2.
    Either the network connection is not available or the partition does not have a 
    level of software that is capable of supporting partition migration. Verify the 
    correct network and migration setup of the partition, and try the operation 
    again.

    When we verify in VIOS:

    lsmap  	-all  	-npiv

    Name        Physloc                           ClntID  ClntName     ClntOS
    ----------- --------------------------------- ------- ------------ ------
    vfchost3    U9117.MMB.100302P-V1-C14             5      upt0052      AIX
    
    Status:LOGGED_IN
    FC name:fcs0                   FC  loc code:U78C0.001.DBJ0563-P2-C1-T1
    Ports logged in:35
    Flags:a<LOGGED_IN,STRIP_MERGE>
    VFC client name:fcs1            VFC client DRC:U8233.E8B.100244P-V5-C4-T1

    Name        Physloc                           ClntID  ClntName     ClntOS
    ----------- --------------------------------- ------- ------------ ------
    vfchost3    U9117.MMB.100302P-V1-C13
    
    Status:LOGGED_IN
    FC name:fcs0                   FC  loc code:U78C0.001.DBJ0563-P2-C1-T1
    Ports logged in:0
    Flags:4<NOT_LOGGED>
    VFC client name:                VFC client DRC

    Here the problem is vfchost3 and vfchost8 both mapped to same host (upt0058) and both mapped to same physical FC(fcs0). This is not the recommended setup. To fix this use either of these methods:

    • We need to map one of the vfchost to another FC (fcs1) on the server which is connected to the switch.
    • We can remove one of the vfchost through DLPAR.
  • This error basically represents the incompatibility between the source and target FC adapters. The incompatibility can be due to a number of reasons in terms of characteristics of FC adapter (For many different kinds of FC incompatibility problems or mapping problems, we may get "return code of 69".)

    hscroot@guandu5:~> migrlpar -o v -m flrx -t dash --ip destiny4  -u hscroot -p 
        upt0064
    HSCLA319 The migrating partition's virtual fibre channel client adapter 4 
    cannot be hosted by the existing Virtual I/O Server (VIOS) partitions on 
    the destination managed system. To migrate the partition, set up the 
    necessary VIOS host on the destination managed system, then try the 
    operation again. 
    HSCLA319 The migrating partition's virtual fibre channel client adapter 3 
    cannot be hosted by the existing Virtual I/O Server (VIOS) partitions on 
    the destination managed system. To migrate the partition, set up the 
    necessary VIOS host on the destination managed system, then try the 
    operation again. 
         
    Details:
    HSCLA356 The RMC command issued to partition dashv1 failed. This means that
    destination VIOS partition dashv1 cannot host the virtual adapter 4 on the 
    migrating partition. 
    HSCLA29A The RMC command issued to partition dashv1 failed. 
    The partition command is:
    migmgr -f find_devices -t vscsi -C 0x3 -d 1
    The RMC return code is:
    0
    The OS command return code is:
    69
    The OS standard out is:
    Running method '/usr/lib/methods/mig_vscsi'
    69
         
    The OS standard err is:
         
         
    The search was performed for the following device description:
          <vfc-server>
                   <generalInfo>    
                       <version>2.0 </version>
                       <maxTransfer>1048576</maxTransfer>
                       <minVIOSpatch>0</minVIOSpatch>
                       <minVIOScompatability>1</minVIOScompatability>
                       <effectiveVIOScompatability>-1</effectiveVIOScompatability>
                       <numPaths>1</numPaths>
                       <numPhysAdapPaths>1</numPhysAdapPaths>
                       <numWWPN>34</numWWPN>
                       <adpInterF>2</adpInterF>
                       <adpCap>5</adpCap>
                       <linkSpeed>400</linkSpeed>
                       <numIniat>6</numIniat>
                       <activeWWPN>0xc0507601a6730036</activeWWPN>
                       <inActiveWWPN>0xc0507601a6730037</inActiveWWPN>
                       <nodeName>0xc0507601a6730036</nodeName>
                       <streamID>0x0</streamID>
                    <generalInfo>
                      <ras>
                            <partitionID>1</partitionID>
                       </ras>
                      <wwpn_list>
                                    <wwpn>0x201600a0b84771ca</wwpn>
                                    <wwpn>0x201700a0b84771ca</wwpn>
                                    <wwpn>0x202400a0b824588d</wwpn>
                                    <wwpn>0x203400a0b824588d</wwpn>
                                    <wwpn>0x202500a0b824588d</wwpn>
                                    <wwpn>0x203500a0b824588d</wwpn>
                                    <wwpn>0x5005076303048053</wwpn>
                                    <wwpn>0x5005076303098053</wwpn>
                                    <wwpn>0x5005076303198053</wwpn>
                                    <wwpn>0x500507630319c053</wwpn>
                                    <wwpn>0x500507630600872d</wwpn>
                                    <wwpn>0x50050763060b872d</wwpn>
                                    <wwpn>0x500507630610872d</wwpn>
                                    <wwpn>0x5005076306ib872d</wwpn>
                                    <wwpn>0x500a098587e934b3</wwpn>
                                    <wwpn>0x500a098887e934b3</wwpn>
                                    <wwpn>0x20460080e517b812</wwpn>
                                    <wwpn>0x20470080e517b812</wwpn>
                                    <wwpn>0x201400a0b8476a74</wwpn>
                                    <wwpn>0x202400a0b8476a74</wwpn>
                                    <wwpn>0x201500a0b8476a74</wwpn>
                                    <wwpn>0x202500a0b8476a74</wwpn>
                                    <wwpn>0x5005076304108e9f</wwpn>
                                    <wwpn>0x500507630410ce9f</wwpn>
                                    <wwpn>0x50050763043b8e9f</wwpn>
                                    <wwpn>0x50050763043bce9f</wwpn>
                                    <wwpn>0x201e00a0b8119c78</wwpn>
                                    <wwpn>0x201f00a0b8119c78</wwpn>
                                    <wwpn>0x5001738003d30151</wwpn>
                                    <wwpn>0x5001738003d30181</wwpn>
                                    <wwpn>0x5005076801102be5</wwpn>
                                    <wwpn>0x5005076801102dab</wwpn>
                                    <wwpn>0x5005076801402be5</wwpn>
                                    <wwpn>0x5005076801402dab</wwpn>
                        </wwpn_list>
                        
       <vfc-server>

    The solution can be any one of the following (or it may fail due to other mismatching characteristic of target FC adapters):

    • Make sure the characteristic of FC adapter is the same between source and target.
    • Make sure the source and target adapters reach the same set of targets (check the zoning).
    • Make sure that the FC adapter is connected properly.

    Sometimes the configuration log at the time of validation or migration is required to debug the errors. To get the log, run the following command from source MSP:

    alog -t cfg -o > cfglog

    NPIV mapping steps for LPM:

    1. Zone both NPIV WWN (World Wide Name) and SAN WWN together.
    2. Mask LUN's and NPIV client WWN together.
    3. Make sure the target source and target VIOS have a path to the SAN subsystem.

    vSCSI mapping steps for LPM:

    1. Zone both source and target VIOS WWN and SAN WWN together.
    2. Make sure LUN is masked with source and target VIOS together from SAN subsystem.

Back to top

LPM enhancement in POWER7

As per the prerequisites for LPM section for doing LPM, the LPAR should not have any physical adapters, but if it is a POWER7, it can have Host Ethernet Adapter (Integrated Virtualized Ethernet) attached to it. However, for a POWER7 LPAR which you want to migrate to other POWER7 can have HEA attached to it, but we must create etherchannel on a newly created virtual adapter and HEA in aggregation mode. When we migrate at the target we see only virtual adapter configured with IP and etherchannel; HEA will not be migrated. Also, make sure the VLANs used in virtual adapters to create etherchannel are added to both source and target VIOS.

Before LPM:

# lsdev -Cc adapter
ent0  Available       Logical Host Ethernet Port (lp-hea)
ent1  Available       Logical Host Ethernet Port (lp-hea)
ent2  Available       Logical Host Ethernet Port (lp-hea)
ent3  Available       Logical Host Ethernet Port (lp-hea)
ent4  Available       Virtual I/O Ethernet Port (l-lan)
ent7  Available       Virtual I/O Ethernet Port (l-lan)
ent6  Available       Virtual I/O Ethernet Port (l-lan)
ent7  Available       Virtual I/O Ethernet Port (l-lan)
ent8  Available       EtherChannel / 802.3ad Link Aggregation
ent9  Available       EtherChannel / 802.3ad Link Aggregation
ent10 Available       EtherChannel / 802.3ad Link Aggregation
ent11 Available       EtherChannel / 802.3ad Link Aggregation
fcs0  Available C3-T1 Virtual Fibre Channel Adapter
fcs1  Available C3-T1 Virtual Fibre Channel Adapter
lhea0 Available       Logical Host Ethernet Adapter (l-hea)
lhea1 Available       Logical Host Ethernet Adapter (l-hea)    
vsa0  Available       LPAR Virtual Serial Adapter
[root@upt0017] /

In this case doing LPM is also a bit different compared to the earlier method; this has to be done from the LPAR using smitty (also called client side LPM), not from HMC. But, LPAR must install with SSH fileset to do LPM through smitty.

openssh.base.client
openssh.base.server
openssh.license
openssh.man.en_US
openssl.base
openssl.license
openssl.man.en_US

Use smitty to migrate an Power7 LPAR with HEA. Smit --> Applications will be the first step to do LPM from smitty.

# smit

System Management
Move cursor to desired item and press Enter
  
  Software Installation and Maintenance
  Software License Management
  Mange Edition
  Devices
  System Storage Management  *Physical & Logical Storage)
  Security & User
  Communication Applications and Services
  Workload Partition Administration
  Print Spooling
  Advanced Accounting
  Problem Determination
  Performance & Resource Scheduling
  System Environments
  Processes & Subsystems
  Applications
  Installation Assistant
  Electronic Service Agent
  Cluster Systems Management
  Using SMIT (information only)

After selecting "Applications", then select "Live Partition Mobility with Host Ethernet Adapter (HEA)" to proceed.

Move cursor to desired item and press Enter

Live Partition Mobility with Host Ethernet Adapter (HEA)

Next enter the required fields such as source and destination HMC and HMC users, source and destination managed system names, LPAR name.

                   Live Partition Mobility with Host Ethernet Adapter (HEA)    
 
Type or select values in the entry fields.
Press Enter AFTER making all desired changes

                                                     [Entry Fields]
* Source HMC Hostname or IP address                [destinty2]
* Source HMC Username             [hscroot]
* Migration between two HMCs                         no
        Remote HMC hostname or IP address          [ ]
        Remote HMC Username                        [ ]
*Source System                                     [link]
* Destination System                               [king]
* Migrating Partition Name                         [upt0017]
* Migration validation only                          yes

Once the successful migration the smitty command output says OK.

Command Status
                                           
Command: OK            stdout: yes           Stderr: no
Before command completion, additional instruction may appear below.

Setting up SSH credentials wit destinty2
If prompted for a password, please enter password for user hscroot on HMC destinty2
Verifying EtherChannel configuration ...
Modifying EtherChannel configuration for mobility ...
Starting partition mobility process. This process is complete.
DO NOT halt or kill the migration process. Unexpected results may occur if the migration
process is halted or killed.
Partition mobility process is complete. The partition has migrated.

After successful LPM, all HEA's will be in defined state, but still the etherchannel between HEA and Virtual adapter exists and IP is still configured on Etherchannel.

[root@upt0017] /
# lsdev -Cc adapter

ent0   Defined             Logical Host Ethernet Port  (lp-hea)
ent1   Defined             Logical Host Ethernet Port  (lp-hea)
ent2   Defined             Logical Host Ethernet Port  (lp-hea)
ent3   Defined             Logical Host Ethernet Port  (lp-hea)
ent4   Available           Virtual I/O Ethernet Adapter  (l-lan)
ent5   Available           Virtual I/O Ethernet Adapter  (l-lan)
ent6   Available           Virtual I/O Ethernet Adapter  (l-lan)
ent7   Available           Virtual I/O Ethernet Adapter  (l-lan)
ent8   Available           EtherChannel  /  IEEE 802.3ad Link Aggregation
ent9   Available           EtherChannel  /  IEEE 802.3ad Link Aggregation
ent10  Available           EtherChannel  /  IEEE 802.3ad Link Aggregation
ent11  Available           EtherChannel  /  IEEE 802.3ad Link Aggregation
fcs0   Available  C3-T1    Virtual Fibre Channel Client Adapter
fcs1   Available  C4-T1    Virtual Fibre Channel Client Adapter
lhea0  Defined             Logical Host Ethernet Adapter  (l-hea)
lhea1  Defined             Logical Host Ethernet Adapter  (l-hea)
vsa0   Available           LPAR Virtual Serial Adapter
[root@upt0017] /
# netstat -i
Name  Mtu    Network      Address            Ipkts   Ierrs        Opkts  Oerrs  Coll
en8   1500   link#2      0.21.5E.72.AE.40    9302210    0       819878     0       0
en8   1500   10.33       upt0017.upt.aust    9302210    0       819978     0       0
en9   1500   link#3      0.21.5e.72.ae.52      19667    0          314     2       0
en9   1500   192.168.17  upt0017e0.upt.au      19667    0          314     2       0
en10  1500   link#4      0.21.5e.72.ae.61      76881    0         1496     0       0
en10  1500   192.168.18  upt0017g0.upt.au      76881    0         1496     0       0
en11  1500   link#5      0.21.5e.72.ae.73       1665    0         2200     2       0
en11  1500   192.168.19  upt0017d0.upt.au       1665    0         2200     2       0
lo0   16896  link#1                          1660060    0       160060     0       0
lo0   16896  loopback    localhost ''        1660060    0       160060     0       0
lo0   16896  ::1%1                           1660060    0       160060     0       0
[root@upt0017] /
#

Other enhancements for POWER7

Additional enhancements are:

• User defined Virtual Target Device names are preserved across LPM (vSCSI).
• Support for shared persistent (SCSI-3) reserves on LUNs of a migrating partition (vSCSI).
• Support for migration of a client across non-symmetric VIOS configurations. The migrations involve a loss of redundancy. It requires an HMC level V7r3.5.0 and the GUI "override errors" option or command line --force flag. It also allows for moving a client partition to a CEC whose VIOS configuration does not provide the same level of redundancy found on the source.
• The CLI interface to configure IPSEC tunneling for the data connection between MSPs.
• Support to allow the user to select the MSP IP addresses to use during a migration.

Back to top

Limitations

• LPM cannot be performed on a stand-alone LPAR; it should be a VIOS client.
• It must have virtual adapters for both network and storage.
• It requires PowerVM Enterprise Edition.
• The VIOS cannot be migrated.
• When migrating between systems, only the active profile is updated for the partition and VIOS.
• A partition that is in a crashed or failed state is not capable of being migrated.
• A server that is running on battery power is not allowed to be the destination of a migration. A server that is running on battery power may be the source of a migrating partition.
• For a migration to be performed, the destination server must have resources (for example, processors and memory) available that are equivalent to the current configuration of the migrating partition. If a reduction or increase of resources is required then a DLPAR operation needs to be performed separate from migration.
• This is not a replacement for PowerHA solution or a Disaster Recovery Solution.
• The partition data is not encrypted when transferred between MSPs.