Correction - upon further testing it seems the egress policies are being applied correctly, but the source NAT entry is not being created. Egress policies are correctly being created.

I have also noted that the VR is getting 5 NICs assigned, with 3 of them in the public traffic vlan (1x in a port group reflecting the bandwidth assignment, and 2x in another cp-created port group with a 0 where the bandwidth limit usually goes)

I've also noted this in the management server log during the VR creation:

2014-07-16 14:08:45,297 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-322:ctx-4b3460ba <<ESXHOST_IP>>) Failed to find DomR VIF to associate/disassociate IP with.

2014-07-16 14:08:45,297 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-322:ctx-4b3460ba <<ESXHOST_IP>>) Unexpected exception: com.cloud.exception.InternalErrorException: Failed to find DomR VIF to associate/disassociate IP with. will shortcut rest of IPAssoc commands

com.cloud.exception.InternalErrorException: Failed to find DomR VIF to associate/disassociate IP with.

                at com.cloud.hypervisor.vmware.resource.VmwareResource.assignPublicIpAddress(VmwareResource.java:1910)

                at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2092)

                at com.cloud.hypervisor.vmware.resource.VmwareResource.executeNetworkElementCommand(VmwareResource.java:431)

                at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:502)

                at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:215)

                at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:50)

                at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)

                at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)

                at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)

                at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:47)

                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

                at java.util.concurrent.FutureTask.run(FutureTask.java:166)

                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)

                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                at java.lang.Thread.run(Thread.java:701)

Thanks all again,

Eric

And after even further testing, it seems this issue does not occur with a VPC VR, only with a isolated network VR.

Has anyone encountered anything similar to this?

Ok, so to troubleshoot this I have created completely fresh install of CloudPlatform 4.3.0.1 with a fresh vSphere 5.5 environment - no modifications to any network service offerings, no modifications to anything for that matter.

Created an isolated network and sure enough the problem presents itself; however doesn't appear with VPC.

The vSphere notes the following error when trying to create the VR:

Task Name: Reconfigure Distributed Switch

Target: cloud.public.3004.0.1-<DVSWITCHNAME> (3004 is the public traffic vlan in this case)

Status: Cannot complete operation due to concurrent modification by another operation.

The management server log entry i've already posted above and just to confirm that again, as before, the VR is created with 2x public NICs - one in the port group "cloud.public.3004.0.1-<DVSWITCHNAME>" and one in "cloud.public.3004.200.1-<DVSWITCHNAME>".

And as before, when looking at the VR in CloudPortal it only shows 3 NICs - Public, Control, Isolated - as opposed to the actual 4 NICs that the VR is built with.

I'm abandoning this for now, but hopefully it'll help someone with troubleshooting whatever bug is in this area of the system.

Hi Tim,

Sadly, no. The bug is reportedly addressed in Apache CloudStack 4.4, but there isn't an equivalent CloudPlatform release yet.

One thing of note - looking at various developer discussions, etc. - would be that VPC seems to be the primary future direction for per-tenant networks; as such we've decided to proceed exclusively with VPC and thus work around the problem.

Hope this helps somewhat at least...