NetApp ONTAP 9.8 with SRM 8.4 Configuration Guide

By all means I am not an expert with NetApp hardware or VMware SRM. However over earlier this year I got the task to get SRM installed and working with our new NetApp AFF arrays. I figured this would be a great place to share the steps taken to get everything up and running.  As with any deployment always check the compatibility matrixes to ensure you are going to be supported by your vendors.

So lets get started. 

The Prerequisites

1. A pair of NetApp Arrays. For my setup I was working with a pair of NetApp A400 AFF running ONTAP 9.8 
   
   
2. NetApp ONTAP Tools for VMware (Formally the VSC) Installed.
   ONTAP tools for VMware vSphere 9.8 Documentation Center (netapp.com)
   ONTAP Tools for VMware 9.8 Download (NetApp Support Login Required)
   
   
3. VMware SRM Installed. for the purposes of this I deployed the virtual appliance version SRM 8.4 as VMware has discontinued support for the Windows version. If you need a guide on installing SRM, VMware has a great document with step by step instructions.  VMware SRM 8.4 Documentation
   
   
4. Have download the NetApp SRA. ONTAP Tools for VMware 9.8 Download (NetApp Support Login Required)

Configuring ONTAP Tools

Now that you have the prerequisites done we can configure ONTAP Tools and get the arrays added. 

• Enable TLS 1.2. By default TLS is not enabled and it needs to be enabled for the NetApp SRA to be able to communicate with the ONTAP Tools for VMware.

1. From the console of the  ONTAP Tools VM, login using the Maint Login. 
2. Select Option 1 "Application Configuration" 
3. Select Option 13 "Enable TLS Protocol" 
4. Acknowledge the warning that the operation will restart ONTAP Tools for VMware. 
5. Type the protocol you wish to enable. I used just TLSv1.2
6. Press Enter 
7. Follow the Prompts to Exit" 

• Creating a VSC/ONTAP Tools User on the NetApp Array. Starting in ONTAP 9.7 and above the roles JSON file can be downloaded from the appliance. Older versions of ONTAP use the RBAC tool from NetApp along with the updated XML privilege's file. If you already have the VSC deployed and configured, you may just need to update the existing users permissions to get everything working. 

1. download the privilege's file from the ONTAP tools appliance https://<IP-Address>:9083/vsc/config/VSC_ONTAP_User_Privileges.zip
2. Extract the JSON file from the zip.
3. Logon to your NetApp Array
1. Navigate to Cluster / Settings / Users and Roles
2. In the User pane, click Add
3. Select Virtualization Products
4. Upload the JSON 
5. From the Product Compatibility drop-down chose either VSC and SRA or VSC,VASA Provider, SRA. This is important to ensure the SRA and ONTAP Tools has the privilege's it needs to function properly.
6. Enter a Username and Password the account. 
7. Leave the suggested privilege's checked 
8. Click Add

• Adding your Array to ONTAP Tools for VMware.
  Each VM that resides on an NetApp served datastore, whether its block or NFS must be added to the ONTAP tools storage system so SRM can properly discover and communicate with the storage array. You can either add the SVM directly or connect to the cluster mgmt. IP.

1. Logon to your VCenter and navigate to the NetApp ONTAP Tools plugin 
2. Select Storage Systems from the left hand menu. 
3. Click ADD. 
1. Enter the IP or DNS Name of the SVM or Array
2. Enter the VSC/ONTAP Tools username created above (case sensitive) and password. 
3. Click ADD
4. Wait a few min and the array will be listed. 

ONTAP Tools Storage Systems

 

Configuring Site Recovery Manager Storage Replication Adapter

Once you have installed VMware Site Recovery Manager (SRM), you will need to take a few steps to configure SRM to to communicate with the NetApp Storage Replication Adapter (SRA) and VSC. Lets get started. 

1. Download the NetApp SRM SRA if you have not already. 
2. Login to each of the SRM Appliance Management page: https://FQDN:5480
1. Click Storage Replication Adapters
2. Click New Adapter
3. Click Upload and browse to the location of the NetApp SRA. This should be a tar.gz file.
   
   
   
3.  With the SRA installed we must configure the SRA to connect to our ONTAP Tools deployment.
1. Using Putty or your preferred SSH client, connect to the SRM appliance.
2. Elevate to root using the command: su root
3. Get the docker CONTAINER ID used by the SRA using the following command: docker ps -l 
   
   
   
4. Login to the docker container using the following command: docker exec -it -u srm <container id> sh
5.  Configure the SRA with the ONTAP Tools IP address and the administrator password using the following command: perl command.pl -I <ONTAP Tools - IP> administrator <ONTAP Tools - Password>.  

Configuring Site Recovery Manager

Now that we have deployed and configured ONTAP Tools, SRM, and the SRA, we can verify that SRM shows our SRA and that its status is OK, and that we can configure our Array Pairs. 

1. Login vCenter and from the Menu launch Site Recovery.
2. Create your Site Pair if you have not already
3. from the left hand menu under Configure / Array Based Replication / Storage Replication Adapters. verify that the NetApp Storage Replication Adapter for ONTAP shows a status of OK.
   
   

   
   
4. Verify the same output at the recovery site.
5. From the left hand menu, Click Array Pairs
1. Click ADD
2. Select your NetApp Storage Replication Adapter. Click Next
3. Fill out the Local Array Manager Info:
• Enter a name for the Array Manager on the "Local vCenter"
• Enter the IP or FQDN Hostname of the NetApp Cluster or the SVM
• Enter a list of  IP for the NFS LIFS on the Array.  (leave blank if using SAN)
• Enter the Name of the SVM hosting the Disks. leave blank if you are connecting directly to an SVM.
• Enter a list volumes to discover. leave blank to discover all volumes on SVM
• Enter a list of volumes to exclude from discovery
• Enter the username created when you during section for configuring ONTAP Tools. 
• Enter the password for the account in previous step.
4. Click Next
5. Repeat step 3 for the recovery site.
6. Click Next
7. Click Finish
   

Notes:

• Hide the snapshot directories on your source volumes to prevent warnings when performing your failovers.
• Make sure your SVMs at your protected site and at your recovery site are not the same. If they are, the Array Pairing will fail. 

UCS Blade Firmware 4.2(1d) upload issue

There is a known Cisco bug ID CSCvy96101 in which when trying to upload the 4.2(1d)B bundle you will get an error on the FSM saying "Remote Invocation Result: Sw Defect"  with a description "Cannot execute image unpack (invalid Image)#" and the progress status will be stuck at 39%. 

to address the issue you will need to remove the failed Download Task, and upgrade the Infrastructure code first to 4.2(1d) then upload your blade bundle.

Screenshot:

Happy Upgrading

Microsoft Teams no Camera found

There are a lot of various articles out there on how to try to fix a missing camera in the Microsoft Teams desktop app on Windows 10. 

I found the issue was the laptop had an in place upgrade from windows 7 to 10 and the camera was was installed under the Camera Node in Device Manager. 

To fix the issue I had to uninstall the camera and remove the installed drivers. Once I did that and scanned for hardware changes, the camera reinstalled but under the Imaging Devices node. Relaunched Teams and the camera was now being detected.

NSX Management Service Operation Failed. (Initialization of Admin Registration Service Provider failed. Root Cause: The SSL certificate of the STS service cannot be verified)

The Problem

So I am new to supporting VMware from an infrastructure prospective and do to some departures at our company, our large VMware deployment is now my responsibility. 

A few weeks ago, I updated our expiring Vcenter custom machine certificate using the VMware SSL certificate tool. This Vcenter is still windows but thanks to these two articles, we were able to fix this issue:

Recently replaced our Vcenter custom machine certificate using the SSL cert tool. This Vcenter is still windows but thanks to these two articles, we were able to fix this issue:

https://techie.cloud/blog/2019/04/21/issue-setting-up-lookup-service-in-nsx-manager-6.4.4/ https://www.vembu.com/blog/replacing-vcenter-ssl-certificate-lookup-service/
VMware KB2121689

Outlook 2010 - Error While Preparing to Send Sharing Message

We recently satarted to see an eror when users try to send a calender sharing invitation to users from their Outlook 2010 client. the error prenented reads as follows "Error While Preparing to Send Sharing Message" Our orginzation at the time of writing this is runing Exchange 2016 CU4 and Outlook 2010 SP2 April 2017 Updates and Outlook is configured to use MAPI over HTTP. the issue looks to be tied to the permissions on the calendar and permisisons being present for users who nolong are with the orginization and their mailboxes have been disabled.
to ressolve this issue review the permission list on the calendar and remove any users who nolonger have a mailbox within Exchange. 
You can as workaround, use Outlook on the Web to share the calendar and that will work just fine

Outlook On The Web Error when Saving Calendar Appearance / Reminders Options

I have been meaning to post this for a while now and well life and work sometimes get in the way of sharing the nuggets of info from my job.
So here is the scenario we were running into:
upon migrating an Exchange 2013 Mailbox to Exchange 2016 CU1 some users when adjusting their calendar appearance or setting the option to send a daily agenda email would get an error when saving their settings. The user would get an error that "There was a problem saving your changes. Please try again." you would then see a correlating error in the event viewer with Event ID: 4999 on the server the user was connected to. After working for several days with support, we found that within the mailbox calendar configuration the weather location bookmark was set to a -1 rather than a 0. This setting is currently only used in the O365 and we are not sure as why when migrating a user this bit gets flipped for some users.
you can see this property value by running a Get-MailboxCalendarConfiguration -Identity |FT ID*,Weather* we found that the only way to reset that value was to use MFC Mapi and delete the IPM.Configuration.OWA.useroptions from the users mailbox.

To find all Mailboxes affected you can run:
Get-Mailbox -Server <ServerName> -ResultSize unlimited | Get-MailboxCalendarConfiguration | Where {$_.WeatherLocationBookmark -ne "0"}| FL Id*,WeatherLocationBookmark

to Resolve:
1. Download the latest version of MFC Mapi
2. Grant yourself Full access to the effected users mailbox (Add-mailboxPermission -Identity -User -AccessRights FullAccess)
3. Ensure the user has MAPI access enabled
4. Open MFC Mapi

a. Click Session > Logon

b. Locate the mailbox on the right hand side and double-click it

c. Right-Click the Root Container and chose Open Associated Contents Table

d. In the new window that opened scroll to the right till you see the Message Class Column

e. Scroll down till you locate the IPM.Configuration.OWA.UserOptions Message

f. Right-Click IPM.Configuration.OWA.useroptions and chose Delete Message

g. In the Delete Item Window click OK

h. Close all MFC windows
5. Verify issue is resolved by running Get-MailboxCalendarConfiguration -Identity <username>  | FT WeatherLocationBookmark

 

Mailbox Export Request and MapiExceptionUnknownUser: Unable to make connection to the server. (hr=0x80004005, ec=1003)

As I was preparing to export several mailboxes to PST for a Cross Forest Exchange 2010 to 2016 Migration as a part of an acquisition, I ran in to an issue that when I would run a New-MailboxExportRequest and get an error "Couldn't connect to the source mailbox" When running the cmdlet with a -Verbose, would see the error "MapiExceptionUnknownUser: Unable to make connection to the server. (hr=0x80004005, ec=1003)" After going through some troubleshooting steps:
-Verifying MAPI is enabled on the accounts.
-Verifying that the Exchange Trusted SubSystems group had R/W Share Permissions.
-tried a different users.

After nothing fixed my issue, I called support. One troubleshooting step that they had me do that never crossed my mind was to create a new mailbox database. after doing that I moved a user successfully to that database and was able to export it with no issues. upon comparing the mailbox database settings Get-MailboxDatabase -Identity I noticed that the new database did not have a Public folder configured and the other database did but it looked as if someone had deleted it. Using ADSI we removed the Public folder path from the database and were able to export mailboxes without any issues.

Removing Public Folder path from Mailbox Database:
1. Launch ADSI and enter the configuration Node.
2. Navigate to :
Services > Microsoft Exchange > Exchange OrgID > Administrative Groups > Exchange Administrative Groups (FYDIBOHF23SPDLT) > Databases
3. Right-Click the Database with the public folder you want to remove and click Properties
4. From the Attribute Editor tab select the attribute msExchHomePublicMDB
5. Click Edit
6. Copy that value to a text file as a backup. :)
7. Click Clear
8. Click OK
9. Click OK to close the attribute editor
10. Close ADSI Edit