Revert volume to snapshot failures

[nvazquez]

ISSUE TYPE

• Bug Report
• Improvement Request
• Enhancement Request

COMPONENT NAME

Storage

CLOUDSTACK VERSION

4.16.1 and main

CONFIGURATION

Set snapshot.backup.to.secondary to true

OS / ENVIRONMENT

Tested KVM on NFS storage

SUMMARY

Revert volume to snapshot after migrating a volume to another storage fails with error:

2022-04-05 14:52:25,411 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-e7abb0ff job-48) (logid:8178b033) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.RevertSnapshotCmd
com.cloud.utils.exception.CloudRuntimeException: Cannot find an entry for snapshot 1 on primary storage pools
	at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.revertSnapshot(SnapshotServiceImpl.java:433)
	at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.revertSnapshot(DefaultSnapshotStrategy.java:393)
	at com.cloud.storage.snapshot.SnapshotManagerImpl.revertSnapshot(SnapshotManagerImpl.java:330)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy218.revertSnapshot(Unknown Source)
	at org.apache.cloudstack.api.command.user.snapshot.RevertSnapshotCmd.execute(RevertSnapshotCmd.java:101)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)

STEPS TO REPRODUCE

- Take volume snapshots
- Migrate VM with volume / migrate volume / reinstall VM
- Try revert volume to snapshot

EXPECTED RESULTS

Revert succeeds

ACTUAL RESULTS

Failure (logs above)

[nvazquez]

@slavkap @weizhouapache @Pearl1594 this might be related to #5433 and should find a solution for both issues

[slavkap]

@nvazquez, this can also be hit when migrating a volume to another storage.

[nvazquez]

Thanks @slavkap I've updated the description

[nvazquez]

@slavkap @Pearl1594 opening this for discussion: since #1740 snapshots on primary storage are cleaned up after migration. Internally a new volume DB record is created after the volume is migrated and the old one is destroyed, along with snapshots and that seems to cause the issue (for the reinstall VM case I think that is also the case).

As an example, before the volume migration:

MariaDB [cloud]> select id, store_id, store_role, snapshot_id, state, volume_id from snapshot_store_ref where snapshot_id = 5;
+----+----------+------------+-------------+-------+-----------+
| id | store_id | store_role | snapshot_id | state | volume_id |
+----+----------+------------+-------------+-------+-----------+
|  9 |        1 | Primary    |           5 | Ready |        13 |
| 10 |        1 | Image      |           5 | Ready |        13 |
+----+----------+------------+-------------+-------+-----------+
2 rows in set (0.00 sec)

After volume migration:

MariaDB [cloud]> select id, store_id, store_role, snapshot_id, state, volume_id from snapshot_store_ref where snapshot_id = 5;
+----+----------+------------+-------------+-------+-----------+
| id | store_id | store_role | snapshot_id | state | volume_id |
+----+----------+------------+-------------+-------+-----------+
| 10 |        1 | Image      |           5 | Ready |        14 |
+----+----------+------------+-------------+-------+-----------+
1 rows in set (0.00 sec)

As discussed offline @slavkap proposed the storage driver checks if the references on snapshot_store_ref should be deleted for Primary role - @Pearl1594 what do you think?

[Pearl1594]

I think that makes sense @nvazquez @slavkap

[nvazquez]

Ping @weizhouapache @DaanHoogland @rohityadavcloud @andrijapanicsb what's your take on the approach to be taken for solving this issue?

[weizhouapache]

@nvazquez @slavkap @Pearl1594

my opinion, for kvm with NFS storage,
(1) the record should be removed from DB, because no snapshot exists in primary storage.
(2) the primary datastore info should be optional (not required) when revert snapshot on KVM, because kvm agent tries to copy snapshot from secondary storage. the operation should succeed even if primary datastore info is not provided. however, from code it looks it is required.
(3) it looks ceph storage and managed storage are not impacted. correct me if I am wrong