Category: SCVMM

SCVMM VM template creation fails with Error 2927 / 0x80338048

Another error in SCVMM I just recently stumbled across.

After upgrading a SCVMM 2019 environment to SCVMM 2022, VM template creation failed with Error (2927) – Unknown error (0x80338048).

Detailed error message:

Error (2927)
A Hardware Management error has occurred trying to contact server ‘server.domain.com’ .

WinRM: URL: [http://server.domain.com:5985], Verb: [INVOKE], Method: [HWScoutOffline], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/P2VServerJob]

Unknown error (0x80338048)

Recommended Action
Check that WinRM is installed and running on server ‘server.domain.com’. For more information use the command “winrm helpmsg hresult” and http://support.microsoft.com/kb/2742275.

After a quick Google search, I found an older article, referring to a smiliar error in SCVMM 2008 :

This can occur if the WinRM setting “MaxEnvelopeSizekb” is mismatched between the System Center Virtual Machine Manager server and the host.

https://techcommunity.microsoft.com/t5/system-center-blog/kb-storage-migration-of-a-running-virtual-machine-fails-with/ba-p/344996

I checked MaxEnvelopeSizeKB on Hyper-V Host, SCVMM management server and SCVMM Library Server by running:

winrm get winrm/config

It turned out that MaxEnvelopeSizekb was set to 500KB on the SCVMM management server:

Comparing MaxEnvelopeSizekb value on other SCVMM Fabric server, I found out that the SCVMM default size should be set to 8192KB.

By executing the command below, I changed back MaxEnvelopeSizekb to 8192KB:

winrm set winrm/config @{MaxEnvelopeSizekb="8192"}

After rebooting SCVMM management node, template creation is working again.

What happened?

After SCVMM 2022 upgrade, I also did an inplace-upgrade of the SCVMM management server from WS2019 to WS2022. It seems like the inplace-upgrade reset MaxEnvelopeSizekb.

Unable to create or access VMMDKM

A common error message one can get during SCVMM setup is:

Unable to create or access the Active Directory container ‘CN=VMMDKM,CN=System,DC=domain,DC=local’. Access is denied. Specify the distinguished name for the container and verify that you have GenericRead|CreateChild|WriteProperty rights on the container.

Well, it took me some time to find the root cause. And as this happened twice in the last few months, I decided to write it down and share it with you.

Why do I need a DKM container?

SCVMM needs a DKM container to encrypt several data like Run As Account credentials. You can either create a new DKM container when you setup SCVMM for the first time or you use an existing DKM container if you upgrade your installation from SCVMM 2016 to 2019, for example. Some further information can be found here: https://docs.microsoft.com/en-us/system-center/vmm/plan-install?view=sc-vmm-2019#distributed-key-management

Back to troubleshooting…

So in my case, I have a highly available setup containing of 2 SCVMM nodes. The environment was initially setup with SCVMM 2016 and a legacy service account (username & password). During the upgrade from 2016 to 2019 I wanted to replace the existing service account with a gMSA account, which is supported in SCVMM 2019.

So first I double-checked permissions on the DKM container in Active Directory. Both SCVMM Service Accounts (gMSA for VMM Service and legacy service account for database connection in my case) have full control permissions on the container and also on the objects itself. But still the setup failed with error message “Access is denied”.

Applies to “This object and all descendant objects” should be selected.

I also checked that inheritance is set to ‘This object and all descendant objects’, but still I got an access denied error. VMM Setup log under C:\ProgramData\VMMLogs also wasn’t helpful at all.

After some time I realized, that Domain Admins group was set as an owner on the DKM container. After replacing it with the actual service account (the gMSA one…), the setup succeeded without an error. Please make sure to replace the owner on all descendant objects as well!

Accidentally ‘Domain Admins’ group was set as an owner
Replace owner information with actual service account

Conclusion

When facing DKM access denied issues during SCVMM setup, please make sure, that:

  • new SCVMM service account has full controll on DKM container and on all descendant objects
  • new SCVMM service account is Owner of the DKM container and its descendant objects

I hope this information is useful and safes some time during your next SCVMM setup 🙂

Fix for Synology SMI-S provider & SCVMM

If you have a Synology NAS in your environment (like me in my home lab) and it’s connected to SCVMM via SMI-S, you may have already faced the issue that storage discovery doesn’t work when NAS is running DSM 6.1. Although status is “Responding”, you can’t see any capacity information in SCVMM console:

Luckily there’s a fix available for that. Synology just released an update for its SMI-S Provider plugin (version 1.0.1-0020) which fixes exactly this issue.

After updating the plugin on my Synology box, I just ran a rescan via the SCVMM console and now all storage information is available via SCVMM again:

Hope this helps 🙂

How to remove write lock in SCVMM

If you are administering System Center Virtual Machine Manager (SCVMM) you probably came to that point, that you received the following error message below as a job result:

"Unable to acquire a 'Delete' lock on object 'c216053f-22c8-4135-aaa5-53a017a254e5' of type 'VM' because it is locked by task '896ef157-fa79-4fb3-9ed0-294f7d75ce21' 'Unknown' with a 'Write' lock."
SCVMM error message

Normally, SCVMM automatically resolves a write lock. But in some cases, the write lock can’t be removed. In this case, the only way to remove the write lock is to manually delete it from the database.

!!! Please make sure you back up SCVMM database first !!!

Open SQL Management studio and connect to SCVMM database. Run the following query to double-check the write lock: (replace TaskID with the ID in your error message!)

select * from tbl_VMM_Lock where TaskID = '896ef157-fa79-4fb3-9ed0-294f7d75ce21'

If this query returns an output, you can run the following query below to manually remove the write lock: (replace TaskID with the ID in your error message!)

exec prc_VMM_ReleaseAllLocksByTaskID '896ef157-fa79-4fb3-9ed0-294f7d75ce21'

The write lock should have been removed now.

In my case, one VM was in state “Under Migration” although migration already has been finished. A manual failover to another Hyper-V host within the cluster didn’t refresh the state in SCVMM. After manually deleting the write lock, SCVMM was able the refresh the VM state again.