Problem
We noticed NSX Manager had become disonnected from both the lookup service and vCenter
Troubleshooting
Check Connectivity
From NSX Manager
# debug connection {IP_of_vCenter}
PING 10.63.240.1(10.63.240.1): 56 data bytes
64 bytes from 10.63.240.1: icmp_seq=0 ttl=64 time=0.205 ms
64 bytes from 10.63.240.1: icmp_seq=1 ttl=64 time=0.134 ms
64 bytes from 10.63.240.1: icmp_seq=2 ttl=64 time=0.096 ms
— 10.63.208.5 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.096/0.145/0.205/0.045 ms
10.63.240.1 reachable
10.63.240.1 reachable over port 443
10.63.240.1 not reachable over port 902
From the NSX Manager logs:
# show logs
I found this snippet – which showed the nsx-user had User account expired
2019-03-06 10:25:31.302 GMT ERROR http-nio-127.0.0.1-7441-exec-5 SoapBindingImpl:133 - SOAP fault javax.xml.ws.soap.SOAPFaultException: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: nsx-user, Domain: vsphere.local}
Resolution
Reset the password for the nsx-user in vCenter. Both lookup service and vcenter connections were instantly restored.