how to see the .vmx file in esxi?

I needed to check for sure the chain of snapshot files for my VM. I could see the disks in vCenter, and could see the .vmsd file, but before breaking the file lock on an old snapshot file and forcing a delete, I needed to be sure. In ESX you just ssh to the host and vi the .vmx file. ESXi doesn't allow you to do that on a running VM. So, what to do? create a vm-support bundle of the host, then unzip and untar it. find the vmfs directory and one more directory under there are the VMFS datastores listed with their .vmx files intact. Open them in your fave editor, and enjoy!

what is vmname-vss_manifestss9.zip?

Never seen this before in the home directory of my VMs.  Related to the VSS snapshot created for the VADP backup that is having issues?

It's 32K in size, but other than that all I know is nothing on Google search, nothing on VMware.com or support.emc.com searches either.

No lock on the file, so I downloaded it and had a look:

So, looking inside, this manifest is to do with the VSS writer and the VADP backup.  Probably only left this file behind 'cause of problems with the backup.  Lots of DLLs listed in the writer xlm file:

<?xml version="1.0"?>
-<WRITER_METADATA version="1.1" xmlns="x-schema:#VssWriterMetadataInfo"><IDENTIFICATION dataSource="OTHER" usage="BOOTABLE_SYSTEM_STATE" friendlyName="System Writer" instanceId="b4973554-e918-490d-a887-46fc0a85c5a5" writerId="e8132975-6f93-4464-a53e-1050253ae220"/><RESTORE_METHOD rebootRequired="yes" writerRestore="never" method="REPLACE_AT_REBOOT"/>-<BACKUP_LOCATIONS>-<FILE_GROUP componentFlags="0" selectableForRestore="no" selectable="no" notifyOnBackupComplete="no" restoreMetadata="no" caption="System Files" componentName="System Files"><FILE_LIST filespecBackupType="3855" recursive="yes" filespec="*" path="C:\WINDOWS\system32\CatRoot\{127D0A1D-4EF2-11D1-8608-00C04FC295EE}"/><FILE_LIST filespecBackupType="3855" recursive="yes" filespec="*" path="C:\WINDOWS\system32\CatRoot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}"/><FILE_LIST filespecBackupType="3855" recursive="yes" filespec="*" path="C:\WINDOWS\system32\CatRoot2\{127D0A1D-4EF2-11D1-8608-00C04FC295EE}"/><FILE_LIST filespecBackupType="3855" recursive="yes" filespec="*" path="C:\WINDOWS\system32\CatRoot2\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}"/><FILE_LIST filespecBackupType="3855" filespec="acgenral.dll" path="c:\windows\apppatch"/><FILE_LIST filespecBackupType="3855" filespec="aclayers.dll" path="c:\windows\apppatch"/><FILE_LIST filespecBackupType="3855" filespec="acres.dll" path="c:\windows\apppatch"/><FILE_LIST filespecBackupType="3855" filespec="acspecfc.dll" path="c:\windows\apppatch"/><FILE_LIST filespecBackupType="3855" filespec="acxtrnal.dll" path="c:\windows\apppatch"/><FILE_LIST filespecBackupType="3855" filespec="admwprox.dll" path="c:\windows\system32"/><FILE_LIST filespecBackupType="3855" filespec="admwprox.dll" path="c:\windows\syswow64"/><FILE_LIST filespecBackupType="3855" filespec="adsiis.dll" path="c:\windows\system32\inetsrv"/><FILE_LIST filespecBackupType="3855" filespec="adsiis.dll" path="c:\windows\syswow64\inetsrv"/><FILE_LIST filespecBackupType="3855" filespec="ahui.exe"

the backup.xml file isn't so much windoze gobeldy gook:

-<WRITER_COMPONENTS writerId="a6ad56c2-b509-4e6c-bb19-49d8f43532f0" instanceId="1c8717c4-c53e-4aac-8738-b510483836f8"><COMPONENT backupSucceeded="yes" componentType="filegroup" componentName="WMI"/></WRITER_COMPONENTS>

So, the backup reports as suceeded, but these files weren't cleaned up.

KC

An error occurred while consolidating disks: msg.fileio.lock.(Can't consolidate VM snapshot)

1.  Tried clicking on consolidate -fail
2.  Tried creating and deleting snapshot -  succeeded, but didn't allow me to consolidate snapshot
3.  Tried creating and snapshot with memory state unticked -- same as above, no-go.
4.  Tried cloning the VM. The clone had consolidated disks, but didn't want the outage of switching VMs, plus the hassle of new mac address on the cloned VM with ghost vNIC issue.

5.  But then tried storage vMotioning the VM.  Bingo!  success!

This originating problem was caused by VADP backup.  Seems vSphere 5.5 handles snapshots differently, or maybe, as I read here.

******************
UPDATE: 13/11/2014
******************

overlooked two important facts here:

1) the old snapshots on your old disk are not cleaned up when you sVMotion to a new disk.
2) Trying to delete them from DataStore browser doesn't work.  Are these files still locked?  Shouldn't be as the .vmsd file is empty and the disks referenced as active by the VM is no longer the delta VMDK.  More to come.

I hope this helps someone avoid an outage for their VMs.

KC

Migrating the Whole Stack

Goal:  to migrate as many of our varied services and VMs with the least interruption to our customers.  vMotion and storage vMotion whenever possible.  Similar to my previous post here.

Preparation

New vBlock with all storage, compute, network and virtualisation components above was installed fresh so no in-place upgrades were needed as this method is simpler and has less risk.    

Prepare ESX4 migration storage

Storage: Create a VMFS3 datastore on the new vBlock.  Old vSphere v4 won’t recognise the new VMF5 so this is one of many “hops” that mean an extra step to avoid downtime on the VMs.

Prepare ESX4 migration host

Storage: Split HBAs so one is mapped to the old storage for the VMs before migration (unchanged) Then map the new VMFS3 datastore to the second HBA .  This is making the first migration host a “bridge” between the old storage and the new storage, which the VMs step across via vMotion..  

Prepare ESXi5 migration host 1

CPU: put in VMware cluster with EVC mode set to “Neehelam”.  This is so you can vMotion the VMs from the old CPU chipset on ESX4 to new CPU chipset on ESXi5.  This migration host is an extra step to avoid you needing to shutdown the VM to move it to newer CPUs.

LAN: Split pNICs on the 1st ESXi5 migration host by removing one (we have two) from the vDSwitch and assigning it to standard virtual switch and port groups identical to the ESX4 VM networking.

Prepare ESXi5 migration host 2

LAN: Split pNICs on the  2nd ESXi5 migration host by removing one from the vDSwitch and assigning it to standard virtual switch and port groups identical to the ESX4 VM networking.

Steps:

1.  vMotion first batch of VMs to the IBM/NetApp/ESX4 migration host on vCenter4

disconnect migration host from vCenter4 and connect to vCenter5

2.  sVMotion VMs to temporary VMFS3 datastore on the vBlock

configure vBlock vMotion pNIC on ESX4 host to enable vMotion to vBlock host

3.  migrate VMs to ESXi5 migration host with vMotion to get from old CPU to new CPU and vSphere5

4.  migrate VMs to 2nd ESXi migration host with vMotion to get from EVC to access new CPU features 

5.  use network migration wizard to move VMs from standard vDSwitch port groups to vDSwitch port groups with same vLANs.

6.  finally, migrate VMs to “permanent” ESXi5 host 

check cluster settings for VMs  (HA, DRS) in final vSphere5 clusters

7.  reconfigure vMotion disabling vBlock pNIC and enabling IBM/DataCore configured pNIC for vMotion ready for next batch of VMs to be migrated

8.  disconnect ESXi4 migration host from vCenter5 and reconnect to vCenter4.

repeat steps 1-11 until VMs are migrated

Next:  Exchange, SQL Server, and other VMs with RDM storage

EMC Networker BMR Bare Metal Recovery

Preparation
Disconnect the NIC of the machine you're about to recover, if it's not completely dead yet.  I always build a new machine identical to the one we're recovering.  Since they're always virtual, it's easy to set the same OS, vRAM, vNICs (mac address, if needed), vCPUs and VMDKs.

No need for an operating system or NW client to be installed on the new client as the BMR will do that.

Steps 
EMC has a video that shows this in action here or search YouTube for the same video.
Also, see page 634 of the Networker Administration Guide.

I thought it glosses over a few details which might be of interest.

Gotcha
My learning was around the version of the Wizard/ISO needed.  If your Networker server is v8.1.1 and your client is still running NW764, then don't try the Windows with NW 8.1 BMR.  It seems to work in loading and letting you fill out the fields for the wizard, and even formats the partition on your recovery client.  But it bombs right after trying to restore the files/folders to the partitions, with no real error message.  The logs are pretty unhelpful.

Learnings
You might need to wipe your recovery server if you need to make a few attempts. If the tool bombs and dumps you to DOS prompt and you try to restart it, you may have issues.  We got error that there was already a machine on the network with this ip address (even though it had not recovered the files/folders successfully at that stage).  Trashing the new recovery server's C drive from VMware and creating a new one got around this easily enough.

There doesn't seem to be a Linux version of this tool either, although you could make one easily enough.

I thought this tool might just bring back the crucial registry and disk partitions and OS needed to boot, then you might need to do another restore to get the rest of the data, but it did restore everything for me.  Nice.

Is the EMC documentation any good in your view?  There's a lot there, like:

Note:  By default, the Windows 2012 System Writer does not report Win32 Service Files as a part
of systems components. As a result, the volumes that contain Win32 Service Files are not
considered critical and the DISASTER_RECOVERY:\ save set will not include a volume that
contains files for an installed service. To configure the Windows 2012 server to report
Win32 Service Files as a part of system components, set the ReportWin32ServicesNonSystemState registry sub key to 0. Microsoft KB article 2792088 provides more information.

It mentions Windows storage spaces, storage pools, synthetic full backup as well, which I've not learned about yet.

My experience is that BMR formats and restores C drive and one more.  You'll need to run the Networker client software to restore any other data disks in the usual way.

Another thing you probably already noticed is that BMR doesn't know anything about the Virutal Machine.  That means the vNIC.  If you recover with BMR to a new VM, and the software on your server cares if the MAC address changes, then you'll do an extra step of changing the mac address of your vNIC to "manual" and use the copy/paste

Before using PowerPath, we wanted a way to check/fix primary fibre (even fiber) storage paths that were active across our interswitch link.

We can go in to vCenter and click on poperties for a datastore (or lun if it's an RDM) and change the active path:

But how do you do this automatically for hundreds of luns, or just check to see if they're wrong.  NetApp has a command to run on the filer to tell you if traffic is going down the wrong path, but these ESX commands will tell you if they're set wrong, even if no traffic is going down them (like on servers in a cluter that aren't running the VMs).

It's not simple, but a few key commands will get the info needed:

list all your luns by NAAid:
naa_list=`esxcfg-mpath -l | grep "naa\." | grep "Device: " | cut -d: -f2 | sort -u`

loop through all of them to check the primary path currently assigned, and whether it matches what it should be.

for naa_identifier in `echo $naa_list` ; do
   echo $naa_identifier
   echo do other commands here
done



Find the RunTime Name (VHBA-Controller-Target-LUNID) which is the active path:

# esxcli nmp fixed getpreferred --device  naa.60a98000375334364a2b42436c754239

or

# esxcli nmp fixed getpreferred --device  ${naa_identifier}

which gives:
vmhba1:C0:T3:L200

Is that on the correct initiator (SAN HBA)?

# esxcfg-mpath -L -P  vmhba1:C0:T3:L200
vmhba1:C0:T3:L200 state:active naa.60a98000375334364a2b42436c754239 vmhba1 0 3 200 NMP active san fc.20000024ff03607f:21000024ff03607f fc.500a09808d310651:500a09819d310651

See in bold above, that's the unique part of our NAAid which shows which filer owns the storage.

If it's right, you've confirmed so, if it's wrong, you need to get the correct RunTime Name and change the active path to match it.  You'll need to find a few other things first:

lun_ID=`esxcli nmp fixed getpreferred --device ${naa_identifier}  | cut -d: -f4 | tr -d "L" `

naa_identifier (see above)

correct_pripathwwn (this is one thing you have to figure out and set as variables in your script based on your unique identifiers reported by ESX for your SAN HBAs)

esxcfg-mpath -m | grep -i ${naa_identifier}

vmhba2:C0:T3:L200 vmhba2 fc.2000001b329cdc1d:2100001b329cdc1d fc.500a09808d310651:500a09829d310651 naa.60a98000375334364a2b42436c754239

vmhba1:C0:T3:L200 vmhba1 fc.20000024ff03607f:21000024ff03607f fc.500a09808d310651:500a09819d310651 naa.60a98000375334364a2b42436c754239

vmhba2:C0:T2:L200 vmhba2 fc.2000001b329cdc1d:2100001b329cdc1d fc.500a09808d310651:500a09818d310651 naa.60a98000375334364a2b42436c754239

vmhba1:C0:T2:L200 vmhba1 fc.20000024ff03607f:21000024ff03607f fc.500a09808d310651:500a09828d310651 naa.60a98000375334364a2b42436c754239

[root@VHB24 scripts]#

see how similar this info above is to the screen grab of what vCenter shows?  it's all four paths for our lun, with the full RunTime Name (VHBA-Controller-Target-LUNID) for eath of the paths to the lun.

With all that, you can get the bit needed to change the active path with this command:

correct_pripath=`[root@VHB24 scripts]# esxcfg-mpath -m | grep -i naa.60a98000375334364a2b42436c754239 | grep -i vmhba[1-2]:C0:T[0-9]:L${lun_ID} | grep ${correct_pripathwwn}`

so you can fix the active path with:

#esxcli nmp fixed setpreferred  --device ${naa_identifier} --path  ${correct_pripath}

We do have to run this on each of our hosts, but if running the script takes 20 minutes per host, imagine how long it would take to manually fix these paths on once for each of our 200 luns on our dozen hosts! 

Anyone want to put this in a powershell script?  I'm a bit embarrassed about putting my final script here as it's messy and not optimised beautifully.

The RDM part of my script looks a little like this:

-KC

reboot needed for vCenter Windoze VM

"Clone virtual machine A general system error occurred: Failed to write to C:\ProgramData\VMware\VMware VirtualCenter\journal\1398968714.70: Error writing file. There is not enough space on the disk. Creating snapshot of Virtual Machine"

I checked and there is plenty of GBytes of free space on that disk partition.

So that's yet one more bogus error message.



https://communities.vmware.com/message/1528814 says to reboot vCenter, which sorted the problem for me.

Script for Setting Perennial Reservations on RDM luns

RDM's must have the perennial reservation flag set to true.

Unfortunately the four bits of info (perennial reservations setting, NAA ID, LUN_ID and Data Store Name, aren't all available from one command, so you need to do a little work.

NAAID and Perennial Reservation status:

ssh to host and run:

# esxcli storage core device list > /tmp/esxcli_reservations_naaID.txt

where you'll see the NAA id and Perennially Reserved status for all luns on that host:

naa.60060160b1b02d00559a83248cc0e311
   Display Name: DGC Fibre Channel Disk (naa.60060160b1b02d00559a83248cc0e311)
   Has Settable Display Name: false
   Size: 122880
   Device Type: Direct-Access
   Multipath Plugin: PowerPath
   Devfs Path: /vmfs/devices/disks/naa.60060160b1b02d00559a83248cc0e311
   Vendor: DGC
   Model: VRAID
   Revision: 0532
   SCSI Level: 4
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters: VAAI_FILTER
   VAAI Status: supported
   Other UIDs: vml.020003000060060160b1b02d00559a83248cc0e311565241494420
   Is Local SAS Device: false
   Is Boot USB Device: false

LUN_ID and NAAID

# esxcli storage core device list | grep -C12 "Reserved: false" | grep "Path: /vmfs/devices" | cut -d. -f2 | tee /tmp/NAA_perennial_false.txt
to get the list of NAA ids of luns without perennial reservation set.

List all LUN numbers:
~ # esxcfg-mpath -l | grep "LUN:" | cut -d: -f4,5 | awk '{print $2,$3}' | sort -
u | sort -n -k2
LUN: 0
LUN: 1
LUN: 2
LUN: 3
LUN: 8
LUN: 9
LUN: 10
LUN: 11
LUN: 12
LUN: 13
LUN: 14
LUN: 15
LUN: 42
LUN: 132
LUN: 149
LUN: 151
LUN: 198
LUN: 200
LUN: 201
LUN: 202
LUN: 203
LUN: 204

NAAid and Data Store name

But most of these are VMFS datastores, which are set correctly when they have perennial reservation as false (default is correct, so don't change):

~ # esxcfg-scsidevs -m | awk '{print $1,$5}' > /tmp/esxcfg-scsidevs-m-naaID_DataStore.txt

naa.6000144000000010a0245b40d0472b3a:1 SAN_DS1
~ #

You can manually verify the RDMs and build the command below to change from False to True on RDMs:

esxcli storage core device setconfig -d naa.id --perennially-reserved=true

for example:

# esxcli storage core device setconfig -d naa.6000144000000010a0245b40d047369b --perennially-reserved=true

# esxcli storage core device setconfig -d naa.6000144000000010a0245b40d047369e --perennially-reserved=true

Or just run this script to build the commands for you:
_____________________________________________________________________________
esxcfg-scsidevs -m > /tmp/esxcfg-scsidevs-m-naaID_DataStore.txt
esxcli storage core device list | grep -C12 "Reserved: false" | \
grep "Path: /vmfs/devices" | cut -d. -f2 | while read naaID ; do
#echo "checking ${naaID} for datastore" ;
grep ${naaID} /tmp/esxcfg-scsidevs-m-naaID_DataStore.txt >/dev/null
RC=$?
if [ ${RC} != 0 ] ; then
echo -n "${naaID} does not have datastore, assuming RDM, "
echo "which needs perennial reservation set with command shown:"
echo
echo -n "esxcli storage core device setconfig -d "
echo "naa.${naaID} --perennially-reserved=true"
echo
echo
else
DataStore=`grep ${naaID} /tmp/esxcfg-scsidevs-m-naaID_DataStore.txt |\
awk '{print $1,$5}' | awk '{print $2}'`
echo -n "NAAID ${naaID} has datastore ${DataStore}, so does not need "
echo "perennial reservation changed"
fi
done
__________________________________________________________________________

Example run of script follows:

60060160b1b02d00559a83248cc0e311 does not have datastore, assuming RDM, which needs perennial reservation set with command shown:

esxcli storage core device setconfig -d  naa.60060160b1b02d00559a83248cc0e311 --perennially-reserved=true

NAAID 6000144000000010a0245b40d047339f has datastore SAN_DS1, so does not need perennial reservation changed

References:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1016106

vCOPs Backups (VMware VCenter Operations Manager) backups

Seems this isn't documented very well or discussed much online from what I can see.

Here's what I found so far:

commands to backup postgress database on the Analytics VM (dbase VM):

#su - postgres
#pg_dumpall | gzip -c > /data/pgsql/vcops_pg_dump_all_gz

Then I'll backup the VMs.

To automate this I'll put the postgress dumpall command in to run under the postgres cron scheduler.

Kudos to postgress help from NixCraft.

VMware Docs (see 'round page 116)

So to restore I'd recover the VMs using my backup software, and if needed go back to the database dump and recover it with:

(stop vCOPs service if not already stopped first) from admin linux login, run "vcops-admin start"

#gunzip /data/pgsql/vcops_pg_dump_all.gz
#psql -f /data/pgsql/vcops_pg_dump_all postgres

verifying reload of syslog server

This post is gonna be a bit messy.  I'm trying (again) to start learning about PowerCLI so there are more things that I don't know than things that I do.  No worries, post comments and help me as all I know is Bourne Shell scripting.  (^;

I'm checking my syslog settigns are all consistent and correct.  Of course we can check and change them from vCenter (or new vCenter web client--shudder!), but more than a few hosts are best done by scripts/cli.

I can check this from PowerVCLI:

get-vmhost | Get-VMHostAdvancedConfiguration -Name Syslog.global.logHost

or specific hosts:

Get-VMHostAdvancedConfiguration -Name Syslog.global.logHost -VMHost host1


Name                           Value
----                           -----
Syslog.global.logHost          udp://1.2.3.4:514

and I can even fix it with a script using these commands

Set-VMHostAdvancedConfiguration -Name Syslog.global.logHost -Value 'udp://1.1.1.1:514' -VMHost host1

$esxcli = Get-EsxCli -VMHost host1
    $esxcli.system.syslog.reload()

thanks to this cool script kindly provided by Caleb here.

So, the reloading of the syslog service.  This isn't the usual linux service there's a command for reloading:
from the ESXi host (ssh/putty/DUI session):

~ # esxcli system syslog reload

the VMware kbase article(s) say to check the syslog is running with the good ol' linux ps command:

~ # ps | grep vmsyslogd

8666 8666 vmsyslogd            /bin/python

8667 8666 vmsyslogd            /bin/python

8668 8666 vmsyslogd            /bin/python

But how do you verify whether either of these methods have worked (other than the absence of error messages?).  

Seems /var/log/hostd.log is where this is recorded:

reload issued via PowerCLI:

2014-02-19T11:39:14.045Z [32B81B90 info 'Solo.VmwareCLI'] Dispatch reload

2014-02-19T11:39:14.114Z [32B81B90 info 'Solo.VmwareCLI'] Dispatch reload done

reload issued directly from ESXi linux login session:
2014-02-19T11:52:30.677Z [2A9DEB90 verbose 'Hostsvc.SyslogConfigProvider'] Running '/sbin/localcli system syslog config logger list'
2014-02-19T11:52:30.678Z [2A9DEB90 info 'SysCommandPosix'] ForkExec(/sbin/localcli)  393025
2014-02-19T11:52:30.830Z [2A9DEB90 verbose 'Hostsvc'] Received advanced config change notification

Can you shed any light on this?

Join the discussion and add your perspective.   Thanks, KC

Migrating ESXi4 to ESXi5 and changing vDistSw port groups without downtime

Scope: 

To move VMs from ESXi4 to ESXi5 without downtime means you won't be updating VMware tools (evidently future versions won't require a reboot--they promise this is the last). You can migrate to VMFS5 or upgrade datastore in place, but that won't be covered here either.

We're moving from ESXi4 hosts and vDist Switch port groups to ESXi5 hosts with different vDist Switch labels (but the same VLANs).

The method is different for VMs running on VMFS.  VMs with RDMs can't VMotion, but ours are part of Microsoft Clusters so we only have a small outage while they service is failed over from one VM to another.

Prerequisites and Setup before starting:

We needed an extra ESXi4 hosts and an extra ESXi5 host which I call "migration hosts" (or stepping stone hosts).  That's just to get the vDist Switches changed without interrupting the VMs network connections.


First we needed to move the VMKernel and VMotion off the vDist Switch so the host is not dependent on it (as we'll be removing the vDistSwitch as this host is moved to vCenter5).

Make sure you have one or two spare uplinks, you might need to remove a few spares from the vDist Switch.  Move the VMkernel configuration from the vDistSwitch to a newly created local port group from vCenter: Host > Configuration > Networking > vSphere Distributed Switch > Manage Virtual Adapters and "Migrate" (do VMkernel first, then do VMotion after you've finished VMKernel--two separate steps). 


 
who said VMKernel was dead in ESXi?  (^;

ESXi4 migration host: needs local port groups created and assigned VLANs for all VMs being migrated with vMotion.  Local port groups do not need the same name, but they need the same VLAN IDs so the VMs traffic will flow.  I did this a text file with all my port group names and VLAN IDs and a script that ran these commands on my ESXI4 migration host.

____________________________________________________
cat vDistPortGroups-new-only.txt | while read LINES ; do
pgn=`echo ${LINES} | awk '{print $1}'`
vlanid=`echo ${LINES} | cut -c1-3`
#echo $vlanid is vlanid and $pgn is portgroup name
echo esxcfg-vswitch -A "${pgn}" vSwitch0
echo esxcfg-vswitch -v ${vlanid} -p "${pgn}" vSwitch0
done
____________________________________________________

The ESXi4 migration host needs to be configured with the vDistSwitch used by your VMs that will be migrated and these new local port groups you've created with the commands from the script above.  the ESXi4 migration host is your stepping stone from VMs on ESXi4 with vDistSw to VMs on ESXi4 with local port groups.

Run the same commands on your ESXi5 migration host to get it ready as it will be a stepping stone from ESXi4 on local port groups to ESXi5 with local port groups (it will also be the stepping stone form ESXi5 on local port groups to ESXi5 on vDistSw).

Migrating VMs

ESXi4 to ESXi4 migration host:

1.  vMotion VMs from ESXI4 host to ESXi4 migration host.

2.  Migrate VMs to local port groups with Wizard; go to the Networking section of vCenter and highlight the vDist Switch and right click on "Migrate Virtual Machine Networking" to start the wizard.  Select the vDistSwitch port groups and VMs on your host.  I have this all mapped out in a spreadsheet as it can be confusing and I don't want to miss any VMs or NICs.

3.  Remove the vDistSw from your ESXi4 migration host

4.  Disconnect your ESXi4 migration host from vCenter4

5.  Remove your ESXi4 host from vCenter4.  Warning says you will loose VMs and resource pool info, but it's info in vCenter, not info on the ESXi host (read carefully, it's accurate, worrying, but no risk to VMs that are running).  Of course at this point if you have a host failure you won't get any HA benefits because you've moved the host from the HA cluster by removing it from vCenter).

6.  On vCenter5 (completely different hosts and vCenter and Nexus1K VMs, etc.) add the ESXi4 migration host which is running your migrating VMs.

7.  As our Nexus1K vDistSw VEM code on our ESXi4 and ESXi5 hosts are incompatible, we need an ESXi5 migration how in this step.  It needs the same local port groups as ESXI4 migration hosts to be configured, and you just vMotion your VMs from ESXi4 migration host to ESXi5 migration host (controlled by vCenter5).  Now you have your migration VMs on an ESXi5 host but they're still running on local port groups.  As the ESXi5 migration host has both local port groups and the new ESXi5 vDistSw port groups, you can use the same Virtual Machine Network Migration Wizard to move the VMs to their final vDistSw port groups.

8.  Finally, migrate the VMs with a regular vMotion to the finally "puka" ESXi5 host with the good vDistSw port groups.  The only difference between this host and the previous one is that there are no local port groups on this host.

9.  disconnect your ESXI4 migration host from vCenter5, reconnect it to vCenter4, "lather, rinse & repeat"

My checklist also includes making sure the HA/DRS settings are all correct and updated as new VMs are migrated into vCenter5 and cleaning up the migration hosts.


Migrating Microsoft Cluster VMs involves shutting down the passive/inactive node/VM, recording then removing the RDM's (remove, do not remove and delete) from the VM configuration (.vnx).  Disconnect/remove from vCenter4 the host with the passive RDM VM, add to vCenter5.  migrate to ESXi5 host while powered down, change VLANs for vNICS, add RDM's back by adding existing disks and browsing and selecting them--ensuring same SCSI ID is assigned.  Power up VM, failover MS Cluster so you can "lather, rinse & repeat" same steps on the other node and you're done!  This method does mean short interruption of service as MS cluster is failed over.

good luck!

Quick & Dirty: Replacing NetApp Disk

This doesn't include steps for a MetroCluster, I'll add that later if I do that

1.      Verify failed disk
SnapVault> vol status -f shows failed disks

  also, 
  SnapVault> disk show 0a.02.03
 Shows failed disk  
    where 02 is the shelf , see LED number on front, left (shelf 1 is top, 2 is middle, 3 is bottom)
    where 03 is bay (see printed numbers associated with each disk location)

The disk is amber instead of green and the shelf indicates a fault as well.  To make the disk blink to be sure it's the correct one, run

SnapVault> priv set advanced
SnapVault*> blink on 0a.02.03
SnapVault*> blink off 0a.02.03

2.       Now you need to physically replace disk

3.       Assign the newly replaced disk so it becomes a hotspare:

SnapVault> disk assign 0a.02.03

4.  Verify all is well

SnapVault> disk show 0a.02.03
    to verify it's assigned as spare
SnapVault> vol status -f shows failed disks
 Note: OnCommand GUI can do some of this too under "Storage", "Disks"

deleting backups from Networker

Deleting records from Networker is easy enough, but you have to use the CLI:

(For me it's always helpful to start DOS with a "Run as Admin" option)

Then you "just" delete each backup, by SSID, from the Networker server like this:

1.  (NetworkerDos)# nsrmm -d -y -S 123456789

In human, that's something like "networker meadiamanagement command delete, answer yes to any prompts (like, are you sure?) and the SSID is 123456789

But you have to find the SSIDs of the backup jobs you want to delete first.
And, of course, be careful as you don't want to delete the wrong backups!

My process was to list all backup records (SSIDs) more than 8 months old:
(NetworkerDos)# mminfo -q "savetime<01/01/2013" > c:\temp\delete-2012\ssids-2012-only.txt

In human, that's something like, "give me info from the media mgt database where records are before January 1st 2013.  

The < (less than sign) logic isn't really intuitive or obvious at all, so see this post to understand it.

You can add the backup clients as well with:
(NetworkerDos)# mminfo -q "client=client.domain.org,client=client2.domain.org,savetime<01/31/2013" > c:\temp\delete-2012\ssids-2012-clients-list-1.txt


The way my backups retention is configured I had to get a list of all backup jobs for two sets of clients (those backed up weekly, then those backed up daily).  So I had to build two lists of these clients, and run the command twice, one for those backed up weekly and the other for the clients backed up daily:

Next is some user intensive bits that I couldn't automate as nicely as I'd have liked to.  I had lists of every backup job with the usual information including the SSID.  But I needed to remove from the list the backups I didn't want to delete.  And those were the all but the first backup of every Friday of the month (for weekly backups).  For the daily backups I wanted to keep all the Friday backups.

I needed Linux or Cygwin to manipulate these files as I still haven't learned powershell.

So I looked at a calendar of 2012 and made a list of the backups with the dates that I wanted to keep.  For example, I saw that May 4th 2012 was one of the dates that I didn't want to delete from my backups.  So I grep'ed all the backups from that date out of my delete-ssid.cmd script file:

$ grep -v "04/05/2012" ssids-2012-weekly-delete-list-removed.txt > ssids-2012-weekly-4-May-removed.txt


And after doing this for all the dates I wanted to keep, I did some spot-checking  before building a dos batch file to do the nsrmm -d command on all the reccords I wanted to remove.


It took my dedicated (physical) IBM X3650  2x2.39 GHz, 32 GB RAM Windows 2008 R2 Networker server  about 10 hours to remove some 12000 records.

2.  Then you run nsrim -X (I guess a Networker database check).  This took less than an hour, if memory serves...

3.  Finally, you're ready to run the Data Domain "clean".  You can do it from the CLI but it's works fine from the GUI too.  It's a 12 step process with most of the steps being building a list, step 11 being copying, and the last step seems to be doing some checks.  This step took up to 12 hours to run.

References:

cygwin echo with a tab separators: $ echo -e "test \t\tabcdetest"

http://nsrd.moab.be/
nsrvalley.com 
Data Domain Overview of Cleaning Phases, Document ID:1071

Collecting vBlock Logs--the MotherLoad

 Before testing preparations:

 MDS switches:

 # clear counters interface all
# debug system internal clear-counters all
   Cleared counters for module 1
# terminal length 0
# show hardware internal packet-flow dropped
`show hardware internal packet-flow dropped`

        Module: 01      Dropped Packets: NO

#

 ESXi: 

  #  date ; esxtop -b -a -d 5 -n 2000 | gzip -9c > /tmp/VH1-create-lun-i-o-esxtopoutput.csv.gz ; date

EMCget (from Windows)

VNX:

start NAR file collection

VPLEX:

VPLEXcli: collect-diagnostics
writes to /daigs/collect-diagnostics-out

After testing:

MDS switches:

set logging in putty for everything
 # terminal length 0
 # show tech-support details

UCS Cisco chassis/blades

log into UCS web-interface and select Admin tab near top left and right click "All" under filter anc Create and Download TechSupport Files in the main pane under "Actions".  Then select uscm, etc. and the location where you want the logs stored so you can copy them and upload them for analysis.

http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-manager/115023-visg-tsfiles-00.html

EMCgrab (from Windows):

dos# emcgrab -h vmhostnaem -user username -password passw0rd

files are placed in install directory of EMCgrab, i.e..   

d:\ESXiGrab-1.3.1\EMC-ESXi-GRAB-1.3.1\outputs\

 

VNX:

stop and collect NAR files
generate and gather SP-Collects

VPLEX:

VPLEXcli: collect-diagnostics
writes to /daigs/collect-diagnostics-out

Extending VPLEX Virtual Distributed Volume online

If you've paid enough for a VPLEX then you probably don't want an outage to extend a virtual distributed volume.  It's not quick and it's a bit fiddly, but here's how I did it (with help from some great folks at EMC).

Shown below is a distributed volume “DB_13_1_1” in consistency group “Cluster-2”.

1.  Check consistency group as it should be cluster-2 for this volume

and Check the Rule Set as it should be "Cluster-2-detaches"

 2.  Verify the distributed volume is assigned to both cluster-1 and cluster-2 storage views as the non-active side will be removed to rebuild the extended mirror:

Cluster-1 DB13:

3.  ESX Visibility of Volumes before extending

cluster-2  host that runs active database on cluster-2 DB_13:

4.  Remove cluster-2 DB13 from Cluster-2 consistency group:

5.  Remove inactive volume (mirror with no I/O) from storage view not needed:

6.  ssh into VPLEX and run vplexcli username/password again so you can break the mirror, removing the side that is inactive and that will be added back with a new larger size:

device detach-mirror --device DB_13_1_1 --mirror device_VNX_DB_13_1_1 --discard --force device detach-mirror --device EU01_Exch_DB_13_1 --mirror device_VNX_DB_13_1_1  --discard --force

7.  Refresh list of Distributed Devices and confirm the removed side of the mirror is gone

8.  Click on Cluster-1 devices and delete the device that's been removed from the mirror

9.  Click on Cluster-1 extend and delete the extent that's been removed from the mirror

10.  on Cluster-1 click on Storage Volumes, highlight the claimed volume that's been removed from mirror and click on "Unclaim"

11. collapse the live volume that's serving I/O to get it ready to give it a new larger mirror:

VPlexcli:/clusters/cluster-1/devices> device collapse --device DB_13_1 _1 drill-down device device_ VNX_DB_13_1_1

12.  Type the set visibility local command to change the visibility of the device to local and then validate the change with another ll.

set visibility local DB_13_1 cd  /clusters/cluster-2/devices/device-VNX_DB_13_1_1  ll

13.  Expand active/live (non-mirrored) volume in GUI

Remember the storage that's being added into the live volume to make it a raid-c cluster needs to have a physical volume but not a virtual volume or extent or device.

14.  Add Capacity from Virtual Volume in Cluster-2 choosing the new 50 Gb IWB_ExchDB13_EXT device

15.  Confirm whether needed to rescan hosts at Cluster-2 as DB13 on vSphere

16.  Expand the datastore to use the new space in as added to the lun from vSphere
17.  Create a Virtual Distributed Volume with the newly extended 300+50 Gb Cluster-2-DB_13 volume and the new 350 Gb Cluster-1DB_13_1_EXT

18.  Confirm  Consistency Group (Cluster2) is as should be when done

19.  Add volume back into Storage View (Cluster-1)

20.  Rescan hosts in Cluster-1 and Cluster-2  vCenter as needed.