Community

CloudPlatform 4.x

New post New comment

284 followers

Markus Waag

How to delete networks that are in state 'shutdown'?

January 13, 2017 14:52

Hi,

we have a network-tier which is in state 'shutdown' and cannot delete it.

We tried it via Webgui and via API (plus via API with option 'forced=true')

Here are the results:

Webgui: Error-Message after ten(!) minutes: "Delete network failed", in management-server-log we following messages:

2017-01-12 17:33:36,653 WARN [c.c.u.d.Merovingian2] (API-Job-Executor-85:ctx-84d76baf job-155960 ctx-369e542c) (logid:7022cef7) Timed out on acquiring lock networks866 . Waited for 600seconds com.cloud.utils.exception.CloudRuntimeException: Timed out on acquiring lock networks866 . Waited for 600seconds at com.cloud.utils.db.Merovingian2.acquire(Merovingian2.java:142) at com.cloud.utils.db.TransactionLegacy.lock(TransactionLegacy.java:389) at com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1028)

2017-01-12 17:33:36,654 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-85:ctx-84d76baf job-155960 ctx-369e542c) (logid:7022cef7) Unable to acquire lock for the network null as a part of network shutdown

API-Call (cloudmonkey, profile: root): delete network id=3b52f0aa-101b-4949-b2e6-a8ea3c58e5d0

=same result as via webgui

API-Call (cloudmonkey, profile: root): delete network id=3b52f0aa-101b-4949-b2e6-a8ea3c58e5d0 forced=true

= Timeout after 70 minutes (!!!)

Entries in Management-Server-Log:

2017-01-13 13:27:51,162 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-114:ctx-85c471bb job-156235) (logid:fac743a5) Add job-156235 into job monitoring

2017-01-13 13:27:51,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-5:ctx-b0fa64c4 ctx-9b49db49 ctx-1a878da6) (logid:607056a5) submit async job-156235, details: AsyncJobVO {id:156235, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo: {"cmdEventType":"NETWORK.DELETE","ctxUserId":"2","signatureversion":"3","httpmethod":"GET","apiKey":"xxx","id":"3b52f0aa-101b-4949-b2e6-a8ea3c58e5d0","response":"json","ctxDetails":"{\"com.cloud.network.Network\":\"3b52f0aa-101b-4949-b2e6-a8ea3c58e5d0\"}","expires":"2017-01-13T12:37:51+0000","ctxAccountId":"2","uuid":"3b52f0aa-101b-4949-b2e6-a8ea3c58e5d0","ctxStartEventId":"5267400","signature":"Pdmh6r1aZBssaYxD5OYIN6BBKXA\u003d","forced":"true"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345049465082, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}

2017-01-13 13:27:51,168 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235) (logid:b64e2f8f) Executing AsyncJobVO {id:156235, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo: {"cmdEventType":"NETWORK.DELETE","ctxUserId":"2","signatureversion":"3","httpmethod":"GET","apiKey":"CuYlSHIFbEjRvMTTvGxxR1Zvx1tSetTtR1_uOQMHTMqW5lypFMlvd-XBaBYCyBjkucdRHCrAHHjXeJXVr-K58Q","id":"3b52f0aa-101b-4949-b2e6-a8ea3c58e5d0","response":"json","ctxDetails":"{\"com.cloud.network.Network\":\"3b52f0aa-101b-4949-b2e6-a8ea3c58e5d0\"}","expires":"2017-01-13T12:37:51+0000","ctxAccountId":"2","uuid":"3b52f0aa-101b-4949-b2e6-a8ea3c58e5d0","ctxStartEventId":"5267400","signature":"Pdmh6r1aZBssaYxD5OYIN6BBKXA\u003d","forced":"true"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345049465082, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}

2017-01-13 13:29:22,251 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-c67882ac) (logid:76f072f7) Task (job-156235) has been pending for 91 seconds

[...]many "pending" messages [...]

2017-01-13 13:37:36,298 WARN [c.c.u.d.Merovingian2] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Timed out on acquiring lock networks866 . Waited for 600seconds com.cloud.utils.exception.CloudRuntimeException: Timed out on acquiring lock networks866 . Waited for 600seconds at com.cloud.utils.db.Merovingian2.acquire(Merovingian2.java:142) at com.cloud.utils.db.TransactionLegacy.lock(TransactionLegacy.java:389) at com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1028)

[...]java backtrace[...]

2017-01-13 13:37:36,299 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Unable to acquire lock for the network null as a part of network shutdown

2017-01-13 13:37:36,304 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Releasing 0 port forwarding rules for network id=866

2017-01-13 13:37:36,305 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Releasing 0 static nat rules for network id=866

2017-01-13 13:37:36,306 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) There are no port forwarding rules to apply for network id=866

2017-01-13 13:37:36,307 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) There are no port forwarding rules to apply for network id=866

2017-01-13 13:37:36,308 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) There are no static nat rules to apply for network id=866

2017-01-13 13:37:36,309 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) There are no static nat rules to apply for network id=866

2017-01-13 13:37:36,312 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Successfully released rules for network id=866 and # of rules now = 0

2017-01-13 13:37:36,312 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Successfully cleaned up portForwarding/staticNat rules for network id=866

2017-01-13 13:37:36,313 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Found 0 lb rules to cleanup

2017-01-13 13:37:36,313 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Successfully cleaned up load balancing rules for network id=866

2017-01-13 13:37:36,325 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Releasing 0 firewall rules for network id=866

2017-01-13 13:37:36,327 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) There are no firewall rules to apply

2017-01-13 13:37:36,329 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Successfully released firewall rules for network id=866 and # of rules now = 0

2017-01-13 13:37:36,337 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Successfully cleaned up firewallRules rules for network id=866

2017-01-13 13:37:36,341 DEBUG [c.c.n.v.NetworkACLManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Releasing 2 Network ACL Items for network id=866

2017-01-13 13:37:36,344 DEBUG [c.c.n.v.NetworkACLManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Applying NetworkACL for network: 866 with Network ACL service provider

2017-01-13 13:37:36,350 DEBUG [c.c.n.e.VpcVirtualRouterElement] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Virtual router elemnt doesn't need to apply firewall rules on the backend; virtual router doesn't exist in the network 866

2017-01-13 13:37:36,351 DEBUG [c.c.n.v.NetworkACLManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Successfully released Network ACLs for network id=866 and # of rules now = 2

2017-01-13 13:37:36,351 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Successfully cleaned up NetworkACLs for network id=866

2017-01-13 13:37:36,354 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Sending destroy to com.cloud.network.element.VpcVirtualRouterElement@64f3e89

2017-01-13 13:37:36,372 DEBUG [c.c.n.NetworkModelImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Service SecurityGroup is not supported in the network id=866

2017-01-13 13:37:36,382 DEBUG [c.c.n.NetworkModelImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Service SecurityGroup is not supported in the network id=866

2017-01-13 13:37:36,398 DEBUG [c.c.a.m.ClusteredAgentAttache] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Seq 44-1379508860859036098: Forwarding Seq 44-1379508860859036098: { Cmd , MgmtId: 345049465082, via: 44(172.27.228.92), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.SetupGuestNetworkCommand":{"dhcpRange":"10.10.11.1","networkDomain":"cs64cloud.internal","isRedundant":false,"add":false,"nic":{"deviceId":3,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"67c7933b-ab32-4ca1-ac1c-0b25592218f9","uuid":"3b52f0aa-101b-4949-b2e6-a8ea3c58e5d0","ip":"10.10.11.1","netmask":"255.255.255.0","gateway":"10.10.11.1","mac":"02:00:54:77:00:02","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://3320","isolationUri":"vlan://3320","isSecurityGroupEnabled":false,"name":"dvSwitch1,,vmwaredvs"},"accessDetails":{"router.guest.ip":"10.10.11.1","guest.vlan.tag":"3320","guest.network.gateway":"10.10.11.1","guest.bridge":"10.10.11.255","router.name":"r-1821-VM","router.ip":"172.27.6.59"},"wait":0}}] } to 345049489649

2017-01-13 13:38:22,251 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-250bed93) (logid:37744aa9) Task (job-156235) has been pending for 631 seconds

[...] again many many "pending" messages until [...]

2017-01-13 14:37:36,399 INFO [c.c.u.e.CSExceptionErrorCode] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions

2017-01-13 14:37:36,404 WARN [c.c.a.m.AgentAttache] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Seq 44-1379508860859036098: Timed out on Seq 44-1379508860859036098: { Cmd , MgmtId: 345049465082, via: 44(172.27.228.92), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.SetupGuestNetworkCommand":{"dhcpRange":"10.10.11.1","networkDomain":"cs64cloud.internal","isRedundant":false,"add":false,"nic":{"deviceId":3,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"67c7933b-ab32-4ca1-ac1c-0b25592218f9","uuid":"3b52f0aa-101b-4949-b2e6-a8ea3c58e5d0","ip":"10.10.11.1","netmask":"255.255.255.0","gateway":"10.10.11.1","mac":"02:00:54:77:00:02","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://3320","isolationUri":"vlan://3320","isSecurityGroupEnabled":false,"name":"dvSwitch1,,vmwaredvs"},"accessDetails":{"router.guest.ip":"10.10.11.1","guest.vlan.tag":"3320","guest.network.gateway":"10.10.11.1","guest.bridge":"10.10.11.255","router.name":"r-1821-VM","router.ip":"172.27.6.59"},"wait":0}}] }

2017-01-13 14:37:36,404 DEBUG [c.c.a.m.ClusteredAgentAttache] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Seq 44-1379508860859036098: Forwarding to peer to cancel due to timeout

2017-01-13 14:37:36,404 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Seq 44--1: MgmtId 345049465082: Req: Routing to peer

2017-01-13 14:37:36,404 DEBUG [c.c.a.m.AgentAttache] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Seq 44-1379508860859036098: Cancelling.

2017-01-13 14:37:36,404 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Timed Out com.cloud.exception.OperationTimedoutException: Commands 1379508860859036098 to Host 44 timed out after 3600 at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:437) at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:442) at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:966)

[...]java backtrace[...]

2017-01-13 14:37:36,407 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-114:ctx-85c471bb job-156235 ctx-0a7b8e61) (logid:b64e2f8f) Unable to complete destroy of the network due to element: VpcVirtualRouter com.cloud.exception.AgentUnavailableException: Resource [Host:44] is unreachable: Host 44: Unable to send commands to virtual router at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3851) at com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.setupVpcGuestNetwork(VpcVirtualNetworkApplianceManagerImpl.java:374) at com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.removeVpcRouterFromGuestNetwork(VpcVirtualNetworkApplianceManagerImpl.java:338)

[...]java backtrace[...]

2017-01-13 14:37:36,417 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235) (logid:b64e2f8f) Complete async job-156235, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to delete network"}

2017-01-13 14:37:36,419 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235) (logid:b64e2f8f) Publish async job-156235 complete on message bus

2017-01-13 14:37:36,419 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235) (logid:b64e2f8f) Wake up jobs related to job-156235

2017-01-13 14:37:36,419 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235) (logid:b64e2f8f) Update db status for job-156235

2017-01-13 14:37:36,420 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235) (logid:b64e2f8f) Wake up jobs joined with job-156235 and disjoin all subjobs created from job- 156235

2017-01-13 14:37:36,424 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-114:ctx-85c471bb job-156235) (logid:b64e2f8f) Done executing org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd for job-156235

2017-01-13 14:37:36,425 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-114:ctx-85c471bb job-156235) (logid:b64e2f8f) Remove job-156235 from job monitoring

Is here anyone else with similar problems and how can it be fixed?

We hoped that the parameter "forced=true" (found it here: https://issues.apache.org/jira/browse/CLOUDSTACK-5861) would help, but it doesn't.

2 comments

Please sign in to leave a comment.

Previous 2 comments

Date Votes

Prashanth Reddy Mandadi

January 16, 2017 04:20

Hi Markus,

I see the following in the logs

Resource [Host:44] is unreachable: Host 44: Unable to send commands to virtual router

Can you please check the status of the host with id=44? You can do "select status from cloud.host where id=44". Seems like the host is not Up and CloudPlatform is unable to send command to Virtual router running on that host.

To resolve the issue, make sure the host is Up in cloudplatform or you can delete the Virtual router associated with this network and then proceed with deleting the network itself.

Thanks

Prashanth

Markus Waag

January 16, 2017 18:51

Hi Prashanth,

thanks for your reply.

This host seems to be up:

mysql> select status from cloud.host where id=44;
+--------+
| status |
+--------+
| Up |
+--------+
1 row in set (0,00 sec)

Host seems up and iam not sure about deleting him because there is another tier in this net, which we do need. I sent management-server-logs to the support-team already and will wait, what they can tell me.