Microsoft Crappy VSS

Mark Bellows on the EMC forums posted some good stuff about toubleshooting some bad backup software with this great list that happens to show how much M$ VSS sucks too:

VSS snapshot creation may fail after a LUN resynchronization on a computer that is running Windows 7 or Windows Server 2008 R2 - November 25, 2009
http://support.microsoft.com/kb/976099
Backup fails with VSS Event ID 12292 and 11 on Windows Server 2008 and Windows Server 2008 R2 - January 20, 2010
http://support.microsoft.com/kb/2009513
No VSS writers are listed when you run the “vssadmin list writers” command in Windows Server 2008 R2
http://support.microsoft.com/kb/2009550
http://support.microsoft.com/kb/2009533
Windows 2008 R2 64-bit backup failed
http://solutions.emc.com/emcsolutionview.asp?id=esg108377
System State backup using Windows Server Backup fails with error: System writer is not found in the backup
http://support.microsoft.com/kb/2009272

A VSS hardware snapshot database keeps growing with duplicated ...

(959476) - ... VSS) requestor instances with VSS hardware provider to delete snapshots in Windows Server 2008 ... Important Windows Vista and Windows Server 2008 hotfixes ...
http://support.microsoft.com/kb/959476

A snapshot may become corrupted when the Volume Shadow Copy ...

(975688) - Fixes a problem in Windows 7 and in Windows Server 2008 R2 in which a snapshot may become corrupted when the VSS snapshots providers takes more than 10 ...
http://support.microsoft.com/kb/975688

A virtual machine online backup fails in Windows Server 2008 R2 ...

This issue occurs because the Hyper-V Volume Shadow Copy Service (Hyper-V VSS ... Important Windows 7 hotfixes and Windows Server 2008 R2 hotfixes are included in ...
http://support.microsoft.com/kb/2521348

You cannot safely remove volumes after you perform a VSS backup ...

(2487341) - ... files after you perform a Volume Shadow Copy Service (VSS) backup operation in Windows Server 2008 SP2. ... Windows Vista hotfixes and Windows Server 2008 hotfixes ...
http://support.microsoft.com/kb/2487341

Also, please note that not all of these fixes come with Windows update and are used on a case by case situation.

Now, from the client side, one thing you can do is to check on the status of VSS from the command line using vssadmin.

Here are a couple of MS KB articles to get you started:

Vssadmin usage

http://technet.microsoft.com/en-us/library/cc754968(WS.10).aspx

Manage Volume Shadow Copy Service from the Vssadmin Command-Line

http://technet.microsoft.com/en-us/magazine/dd348398.aspx

How to enable the Volume Shadow Copy service's debug tracing features in Microsoft Windows Server 2003 and Windows 2008

http://support.microsoft.com/kb/887013

Oh - one other thing you do from both the server and the client (I would start with the client to see if it is showing any errors) is to render the daemon.raw file.

To do this, make a copy of the /nsr/logs/daemon.raw file - I typically rename the copy to be daemon_YYYYmonthnameDD.raw.

From the command line, navigate to the directory the copied daemon.raw file is in, then run:

nsr_render_log daemon_YYYYmonthnameDD.raw >daemon_YYmonthnameDD.log

ie. nsr_render_log daemon_2012Jan27.raw >daemon_2012Jan27.log

You can then open the file through windows explorer with note pad or word pad and read the contents.  It may take a little time to get to the portion you need, but know when the backup started will help.

NetApp OSSV Notes

So, Open System Snap Vault is NetApp's answer to backing up everything that you can't backup with SMVI (VMware VMs) or Snap Manager for Exchange/SQL Server/SharePoint.  Here's a table a notes about the different backup methods I'm finding out about from NetApp.  Quick overview from NetApp here.  Note, FAQ says that Linux OS backup is not supported.

It's software you install on a physical or virtual machine that does NetApp block level backup of any data (NetApp or otherwise) to a NetApp SnapVault node.

I thought it simply used NDMP protocol, but it seems it sends the data with QSM (Qtree Snap Mirror) and seems to setup the relationship or do the backup management with NDMP:


NDMP port (default value is 10000)
FILESERVER port-10555.
QSMSERVER port-10566.

Components:

• NetApp Host Agent (install this first)
• Host Agent Plugin (comes with Host Agent)
• NetApp OSSV

Host Agent

Windows: agentsetup-2-7-win32.exe

Linux: ./agentsetup-2-7-linux.bin

Unpacking files needed for the installation ...

Beginning the installation ...

Starting agent.

        You may now point your browser to

            http://127.0.0.1:4092/welcome

        to configure the agent software.

This is NTAPagent service (which runs /opt/NTAPagent/ntap_agent) on linux 

or ntap_agent.exe on Windoze

Like this diagram shows, set a password for the "admin" login on the Host Agen application on your backup client.  This is unhelpfully referred to as the Mgt API Password here.










Install Host Agent Plugin

Windows runs this automatically: 

c:\Program Files\netapp\snapvault\manageability\InstallHostAgentPlugins.exe

But for Linux you need to run:

./ossv/manageability/InstallHostAgentPlugins.sh

before you can get DFM to connect to OSSV on your backup client.

OSSV

License OSSV on your SnapVault servers with these generic licenses available from NetApp Now website.

Install OSSV on your backup client

uses port 10000 for NDMP backup between client and NetApp

configure NDMP user/password and SnapVault filers (names separated by commas)

Runs on most OS'es (Windows, Linux, Solaris, etc.) but you can only create luns from Windows OSSV.

Seems to be different versions of OSSV for Windows 2003 and Windows 2008, but same for Host Agent. 

Gotcha: SnapVault relationship wouldn't create on my Linux backup client until:
1.  stop service on linux backup client
service snapvault stop
2. backup and edit /usr/snapvault/config/snapvault.cfg:

changing:

[QSM:Check Access List]
Type= CheckBox
value= TRUE

to:


[QSM:Check Access List]
Type= CheckBox
value= FALSE



3.  And starting service again.  
service snapvault start
This was much easier than getting the syntax for the svsetstanza command.  (^;

Installing OSSV on Linux

mv ossv_linux_v3.0.1.tar.gz ossv_linux_v3.0.1.tar
tar -xvf ossv_linux_v3.0.1.tar
cd ossv
[root@linux ossv]# ./install
Installer invoked in /home/kevin/ossv
Using default /tmp as the temporary directory
Expanding distribution file
OSSV
3_0_1_2011FEB17_RC
Have you read and agreed to the terms of the license?
(y = yes, n = no, d = display license) (y n d) [d] : y
Please enter the path where you would like
the SnapVault directory to be created [/usr/snapvault] :
Enter the User Name to connect to this machine
via the NDMP protocol : backups
Please enter the password to connect to this machine
via the NDMP protocol :
Confirm password:
Enter the NDMP listen port [10000] :
Enter the hostname or IP address of the SnapVault secondary
storage system(s) allowed to backup this machine.
Multiple hostnames or IP addresses must be comma seperated.
> : SnapVaultHost
NetApp Host Agent is recommended for managing OSSV.
OSSV could not detect NetApp Host Agent on this system.
If you install this software later, you can install OSSV plugins using
/usr/snapvault/manageability/InstallHostAgentPlugins.sh
Writing ndmp password to password file
checkinstall running
CHOSEN_CLASSES=ossvcore
PKG_BASE=/usr/snapvault
This is a new installation - not an upgrade
Trace Directory = /usr/snapvault/trace
Temp  Directory = /usr/snapvault/tmp
preinstall running
Installing ossvcore
Copying uninstallation scripts
postinstall running
HOST_OS=Linux
Creating Trace Directory /usr/snapvault/trace
NV_UPGRADE=FALSE
Creating database directory /usr/snapvault/db
Installing libraries
libsvdb.so
libsvgui.so
libsvndmp.so
libsvplugin.so
libsv.so
libsvxctl.so
libsvxpm.so
Creating symbolic links
Installing base npk's
Checking for components in /home/kevin/ossv/packages
Installing additional npk's
Install normal package
Successfully installed '/home/kevin/ossv/packages/ossv2300.npk'
Checking for extra components in /home/kevin/ossv/extrapackages
Copying non-installable additional npk's
Setting gathered configuration values
Installing OSSV Services
Will link to '/usr/snapvault/etc/startup.sh'
Starting OSSV Services
Checking install validity
SnapVault home directory: '/usr/snapvault'
SnapVault database directory: '/usr/snapvault/db'
SnapVault temporary directory: '/usr/snapvault/tmp'
SnapVault Database and Temporary directories have 71% space left (47182Mb)
SnapVault service is running
SnapVault listener is running
Snapvault NDMP interface on port 10000:
   Vendor     : Netapp
   Product    : SnapVault
   Version    : 3_0_1_2011FEB17_RC
   Host       : maserati
   Host Id    : CED5936CC94D727724CFDEE23D77C48F
   OS Type    : Linux
   OS Version : 2.6.9-5.ELsmp
   IPV6       : Ndmp Server is responding correctly
   IPV4       : Ndmp Server is responding correctly
Snapvault QSM interface on port 10566:
   IPV6       : QSM Server is responding correctly
   IPV4       : QSM Server is responding correctly
Validating filesystems:
   Mount point / (/dev/mapper/VolGroup00-LogVol00) is suitable for backup
   Mount point /proc (none) is a special mount, unsuitable for backup
   Mount point /sys (none) is a special mount, unsuitable for backup
   Mount point /dev/pts (none) is a special mount, unsuitable for backup
   Mount point /proc/bus/usb (usbfs) is a special mount, unsuitable for backup
   Mount point /boot (/dev/cciss/c0d0p1) is suitable for backup
   Mount point /dev/shm (none) is a special mount, unsuitable for backup
   Mount point /proc/sys/fs/binfmt_misc (none) is a special mount, unsuitable for backup
   Mount point /var/lib/nfs/rpc_pipefs (sunrpc) is a special mount, unsuitable for backup
NetApp Host Agent is installed on this system.
Check Succeeded
Installation appears valid
Installation completed successfully
[root@linux ossv]#

Bits & Bobs:

LREP: allows out-of-band data transfers to keep load off normal channel, can be used to transfer data to portable disk for physical transfer to another location.

Enable OSSV compression on SnapVault if required.

DFM setup:

Go to OSSV tab in NetApp Mgt Console on DFM server

Click "Add" 

have NDMP username and password ready

password needs entering as it is shown from the command on the SnapVault server from the ndmpd command:

ndmpd password backup

where NDMP user is called "backup"

CLI snapvault check:

test206:C:\                                              SnapVault:/vol/sva_OSSV_test/test206_OSSV_test_test206_C__

         Uninitialized  -          Transferring fes206:SystemState                                      SnapVault:/vol/sva_OSSV_test/test206_OSSV_test_fes206_SystemState

         Uninitialized  -          Transferring

SnapVault> snapvault status "SnapVault:/vol/sva_OSSV_test/test206_OSSV_test_test206_C__"

Snapvault is ON.

Source                Destination                                                State          Lag        Status

test206:C:\            SnapVault:/vol/sva_OSSV_test/fes206_OSSV_test_test206_C__  Uninitialized  -          Transferring

SnapVault>snapvault status -l "destination volume/path"

NetApp Backup Offerings

I'm just starting to get to know the complicated and varied ways to do backups in the NetApp world.  Here's some notes and a table to try to lay it out concisely:

Product	NetApp DFM	NetApp SnapManager for SQL/Exchange/SharePoint	OSSV
Managed by	itself	backup clients	DFM
Depends on		NetApp SnapDrive	DFM/SnapVault SnapVault on NetApp NetApp Host Agent on backup client NetApp Host Agent Plugin on backup client
Can Backup:	any NetApp storage	NetApp Host Agent	Windows or Linux, SAN or local disks Oracle
strengths	centrally managed, very quick and saves lots of storage space thanks to storage snapshots	saves storage and gives very fast backups of SAN data via storage snapshots	Doesn't need to be NetApp storage Data owners can manage backups try LREP for migrating data to new location Can do file/folder level backups file/folder backup exclusion supported
weaknesses	not sure how to run pre-post exec scripts on backup clients for individual backups.	doesn't allow DFM to manage How to run pre-post scripts? does not backup CIFS or NFS

vCLI (vSphere PowerShell) Primer

Connect-VIServer -Server vsphere_vCenter

vihostupdate –server HOSTNAME –install –bundle c:\folder\name_of_file.zip

to install drivers, etc. but host needs to be in maint mode

get-vm | Get-CDDrive | select @{'n' = 'Name'; 'e' = { $_.Parent.Name} }, @{'n' = 'IsConnected'; 'e' = {$_.ConnectionState.Connected}}, @{'n' = "ISOFile"; 'e' = {$_.ISOPath}} | where {$_.IsConnected}

write-output "disconnecting Idle vCenter session connnections"

(following examples used by ESX5 Deployment Tool)
PowerCLI> Get-DeployRuleSet
PowerCLI> Get-VMHost vh1t.mits.apmn.org | Get-VMHostAttributes

M$ PowerShell:

http://technet.microsoft.com/en-us/library/ee176949.aspx