
Snapshots are not deleted according to the snapshot policy
We configured CloudStack to take snapshots daily and delete all old snapshots to keep only the 2 most recent snapshots. But the old snapshots are not deleted.
From the log on Secondary Storage VM, we can see the CleanupSnapshotBackupCommand was sent to Secondary Storage VM and the Secondary Storage VM agent received this request.
2013-04-25 00:01:37,978 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Request:Seq 25-300817340: { Cmd , MgmtId: 153252621541, via: 25, Ver: v1, Flags: 100011, [{"CleanupSnapshotBackupCommand":{"secondaryStoragePoolURL":"nfs://172.18.30.12/export/secondary","dcId":1,"accountId":2,"volumeId":2422,"validBackupUUIDs":["/snapshots/1/2/2422/gitfraud01_DATA-2396_20130311171422","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130322114746","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130414110421","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130415110421","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130416110053","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130417110053","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130418110053","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130419110053","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130420110053","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130421110053","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130422110053","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130423110053","/snapshots/1/2/2422/gitfraud01_DATA-2396_20130424110406"],"wait":0}}] }
From the log of Secondary Storage VM, as shown below, it seems the command was successfully executed.
2013-04-25 00:01:37,978 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Processing command: com.cloud.agent.api.CleanupSnapshotBackupCommand
2013-04-25 00:01:37,979 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-4:) Executing: mount
2013-04-25 00:01:37,980 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-4:) Execution is successful.
2013-04-25 00:01:37,980 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Seq 25-300817340: { Ans: , MgmtId: 153252621541, via: 25, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
However, the snapshots that are not listed in the validBackupUUIDs were not deleted.
In addition, the validBackupUUIDs were wrong as well. We configured to only keep the most recent 2 snapshots, but the validBackupUUIDs sent by CloudStack Manager also contain several old snapshots. But when we view the snapshots for that VM from CloudStack Web UI, it only shows the 2 most recent snapshots.
Anyone has seen this issue before? Any idea on what the causes are? How to fix it?

