Replacing failed Disk on Zpool and Error:Component system is busy, try again: failed to offline:

                     Replacing failed Disk on Zpool

Error: Component system is busy, try again: failed to offline

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

1.Check the zpool status

# zpool status

 pool: rpool
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scan: scrub repaired 0 in 1h12m with 0 errors on Mon Aug 21 03:12:50 2017
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         DEGRADED     0     0     0
          mirror-0    DEGRADED     0     0     0
            c1t1d0s0  ONLINE       0     0     0
            c1t0d0s0  DEGRADED     0     0 7.37K  too many errors

errors: No known data errors

2.Offline the Disk in zpool

# zpool offline rpool  c1t0d0s0

# zpool status
  pool: rpool
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scan: scrub repaired 0 in 1h12m with 0 errors on Mon Aug 21 03:12:50 2017
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         DEGRADED     0     0     0
          mirror-0    DEGRADED     0     0     0
            c1t1d0s0  ONLINE       0     0     0
            c1t0d0s0  OFFLINE      0     0 7.37K

3.Unconfigure the Disk

 # cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c1                             scsi-sas     connected    configured   unknown
c1::dsk/c1t0d0                 disk         connected    configured   unknown
c1::dsk/c1t1d0                 disk         connected    configured   unknown
c2                             fc-fabric    connected    configured   unknown
 # cfgadm -c unconfigure c1::dsk/c1t0d0
Aug 24 00:33:45 corpt710 rcm_daemon[3599]: rcm script es_rcm.pl: VxVM vxdmpadm ERROR V-5-1-13080 Attempt to disable all paths through portid and enclosure failed. Last path to the disk can not be disabled.
cfgadm: Component system is busy, try again: failed to offline:
     Resource              Information
------------------  -------------------------
/dev/dsk/c1t0d0s2   Device being used by VxVM

4.Here is the error

cfgadm: Component system is busy, try again: failed to offline:

5.Cause :

This host uses ZFS to manage internal disks and Veritas Volume Manager (VxVM) to manage SAN attached disks. VxVM keeps track of the internal disks – even if it doesn’t actually manage them – and may not allow you to unconfigure them. To get around this restriction below are the steps.

# vxdmpadm getsubpaths ctlr=c1
NAME         STATE[A]   PATH-TYPE[M] DMPNODENAME  ENCLR-TYPE   ENCLR-NAME   ATTRS
================================================================================
c1t0d0s2     ENABLED(A)    -          disk_0       Disk         disk            -
c1t1d0s2     ENABLED(A)    -          disk_1       Disk         disk            -

# vxdmpadm disable path=c1t0d0s2

VxVM vxdmpadm ERROR V-5-1-13080 Attempt to disable all paths through portid and enclosure failed. Last path to the disk can not be disabled.

# vxdmpadm -f disable path=c1t0d0s2

 corpt710 vxdmp: NOTICE: VxVM vxdmp V-5-0-111 [Warn] disabled dmpnode 295/0x8

# vxdmpadm getsubpaths ctlr=c1

NAME         STATE[A]   PATH-TYPE[M] DMPNODENAME  ENCLR-TYPE   ENCLR-NAME   ATTRS
================================================================================
c1t0d0s2     DISABLED(M)    -          disk_0       Disk         disk            -
c1t1d0s2     ENABLED(A)    -          disk_1       Disk         disk            -

6.Now we can un configure the disk

# cfgadm -c unconfigure c1::dsk/c1t0d0

# cfgadm -al|grep -i c1t0d0
c1::dsk/c1t0d0                 disk         connected    unconfigured unknown

7.Replace the Disk physically

8.Now configure the disk

 # devfsadm

 # cfgadm -al|grep -i c1t0d0
c1::dsk/c1t0d0                 disk         connected    unconfigured unknown
# cfgadm -c configure c1::dsk/c1t0d0
# cfgadm -al|grep -i c1t0d0
c1::dsk/c1t0d0                 disk         connected    configured   unknown

9.Replace the zpool with new disk

# zpool replace rpool c1t0d0s0
cannot replace c1t0d0s0 with c1t0d0s0: device is too small

Note: Here we need to layout the new disk

# prtvtoc /dev/dsk/c1t1d0s2 > /tmp/vtoc_root.out

# fmthard -s /tmp/vtoc_root.out /dev/rdsk/c1t0d0s2

fmthard:  New volume table of contents now in place.

10.Now replace the disk and online

# zpool replace rpool c1t0d0s0

# zpool online rpool c1t0d0s0

 # zpool status
  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Thu Aug 24 00:54:08 2017
    461M scanned out of 111G at 15.4M/s, 2h2m to go
    457M resilvered, 0.41% done
config:

        NAME                STATE     READ WRITE CKSUM
        rpool               DEGRADED     0     0     0
          mirror-0          DEGRADED     0     0     0
            c1t1d0s0        ONLINE       0     0     0
            replacing-1     DEGRADED     0     0     0
              c1t0d0s0/old  OFFLINE      0     0 7.37K
              c1t0d0s0      ONLINE       0     0     0  (resilvering)

errors: No known data errors

Note: Wait till the New disk sync and online , its depends on the disk size

# zpool status rpool ;date
  pool: rpool
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on older software versions.
 scan: resilvered 111G in 1h30m with 0 errors on Thu Aug 24 02:24:25 2017
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            c1t1d0s0  ONLINE       0     0     0
            c1t0d0s0  ONLINE       0     0     0

errors: No known data errors