The message is not a problem itself, rather it indicates a problem occurred earlier in the job. There is no avoid set flag to clear as it is only a temporary state. Look earlier in the job for the "real" error.

Best regards,
Kirk

You are absolutely correct, after trawling through the log file it seems that my issue is with the network configuration. The correct interfaces have not been assigned to the xapi0 bridge, and so I can't mount secondary storage.

The full story:

Primary storage is provisioned from local (lvm)
Secondary storage is provisioned from NFS volume

The process i have followed is as follows:

1. Install base xenserver on all nodes
2. Apply updates on all nodes
3. Apply network config to all nodes (via Xencenter)

Physical networks as follows:

NIC0 - Internal ops network
NIC1 - Management interface (10.103.0.0/24) - each server has an IP assigned on this interface (e.g. 10.103.0.10, gateway = 10.103.0.254); xenserver label = management
NIC2 - Interface for guest and public; no IP assigned; xenserver label = guest-public
NIC3 + NIC8 - bonded for storage (active/passive) 10.103.6.0/24; IP assigned on this interface (e.g. 10.103.6.10, gateway = 10.103.6.254) xenserver label = cloud-storage

Secondary storage = 10.103.6.5/vol/secondaryVol (600GB) - Restricted to VLAN 100 on NFS server (NetApp Filer)

Once networking configured on one server in each pool/cluster, create the pools in xencenter and all networks get propagated as expected

In CloudPlatform I then created the zone, pod, cluster as follows:

Zone name =
Type = Advanced
DNS1 = 10.103.0.3
DNS2 =
Guest CIDR = 10.103.7.0/24
Hypervisor = Xenserver
Public = Yes

Pod name =
Reserved system gateway = 10.103.0.254
Reserved system netmask = 255.255.255.0
Start/End Reserved IP Range = 10.103.0.150 - 10.103.0.199

Physical networks configured as follows:

Network = Management
Traffic type = Management
Broadcast Domain = Native
Xenserver traffic label = management
IP Range = as per pod

Network = Storage
Traffic Type = Storage
Broadcast Domian Type = Native
Xenserver traffic label = cloud-storage
GW = 10.103.6.254
NM = 255.255.255.0
VLAN = 100
Start IP = 10.103.6.10
End IP = 10.103.6.253

Network = Guest
Broadcast Domain = ZONE
Xenserver Traffic Label = guest-public

Network = Public
Traffic Type = Public
Broadcast Domain = Vlan
Xenserver Traffic label = guest-public
GW = 10.103.11.254
NM = 255.255.252.0
VLAN = 102
Start IP = 10.103.8.1
End IP = 10.103.11.254

When i complete the configuration and deploy the appropriate networks appear in Xencenter on each node, eg:

Network = VLAN-f0adcdf6-2157-5b7c-bc06-b51224d87875-100
NIC = Bond 3+8
VLAN = 100

Network = VLAN-5adaa36a-16bb-18c8-b5af-ecf2ed699a40-102
NIC = NIC 1
VLAN = 102

Following this System VMs started without issue once "systemvm.use.local" was set, and i was able to download a Debian 7 ISO, thus proving the SSVM is functioning.

I then created a test Service Offering with storage set to local, and no HA

I then tried to deploy a VM using this service offering, the default centos template, and no disk... just to prove deployment works. This VM was successfully deployed onto devhost13

When i tried to repeat the activity the VM failed. The root cause seems to be the inability to mount secondary storage.

Having investigated further i identified some discrepancies in the interface config on each host in the cluster/pool.

Running "brctl show"; on devhost13 (the member server that hosted the VM that was deployed) gives:

xapi00000.d4ae52e8c7dcnoeth3
eth8
xapi3
xapi24
xapi210000.b6c5ced8d447no
xenbr00000.d4ae52e8c7d6noeth0
xenbr10000.d4ae52e8c7d8noeth1
xenbr100000.90e2ba2aac54noeth10
xenbr110000.90e2ba2aac55noeth11
xenbr20000.d4ae52e8c7danoeth2
xapi25
vif5.0
xapi23
xenbr40000.90e2ba2aacf0noeth4
xenbr50000.90e2ba2aacf1noeth5
xenbr60000.90e2ba2aacf4noeth6
xenbr70000.90e2ba2aacf5noeth7
xenbr90000.90e2ba2aac51noeth9

however on every other host (including the master), the same command returns:

bridge namebridge idSTP enabledinterfaces
xapi00000.d4ae52e8cfd7noeth3
eth8
xapi210000.92f51fdae344no
xenbr00000.d4ae52e8cfd1noeth0
xenbr10000.d4ae52e8cfd3noeth1
xenbr100000.90e2ba2aaa3cnoeth10
xenbr110000.90e2ba2aaa3dnoeth11
xenbr20000.d4ae52e8cfd5noeth2
xenbr40000.90e2ba2aa858noeth4
xenbr50000.90e2ba2aa859noeth5
xenbr60000.90e2ba2aa85cnoeth6
xenbr70000.90e2ba2aa85dnoeth7
xenbr90000.90e2ba2aaa39noeth9

so it seems that the correct interfaces have not been assigned to the xapi0 bridge... so the hosts can't communicate with the secondary storage server, thus can't pull down the template, thus can't deploy a VM.

this seems very strange that a member server has the correct settings, but no other host (including the master) is incorrect.

I have tried mounting from the console and this fails, and also a ping fails... this is expected as the communication should go over VLAN 100, which doesn't exist, and the NFS server is restricted to VLAN 100.

Is there a way to remedy this without rebuilding?

I have managed to resolve this

By going into xenserver and removing the storage IP address configured on the 'cloud-storage' bonded network

I then added the IP address back but this time on the CloudPlatform created, VLAN tagged, network.