Troubleshooting¶
If you find Enclave isn't working as expected, here's a simple set of troubleshooting checklists to follow. You can also search our developer community forum to see if any issue you might be experiencing have been observed by other users.
Your first port of call should always be to run enclave self-test
to see if any installation or runtime problems are detected.
Platform connectivity problems¶
If you think Enclave is having trouble reaching our SaaS services, please work through the following checklist to identify the problem.
-
Check https://status.enclave-networks.com/ for any disruptions or service outages
-
Check your systems are enrolled and showing as both connected and approved in the portal
-
Check you're running on the latest version of Enclave with
enclave version
-
Check the output of
enclave status
doesn't contain any warnings or errors -
Check the output of
enclave status
lists at least onePeer
and the state shows asUp
-
Check network traffic is allowed out to
tcp/*:443
-
Check network traffic is allowed out to
udp/*:1024-65355
-
Check network traffic isn't forced through a SOCKS proxy, which is currently unsupported
-
Check local anti-virus software is not interfering with Enclave by temporarily disabling it
Problems enrolling new peers¶
-
Check https://status.enclave-networks.com/ for disruptions or service outages
-
If you're running on Linux check that the
ca-certificates
package is up to date -
Check that the date and time on the enrolling system is accurate
Traffic not flowing between peers¶
If Enclave appears to be connected to our SaaS services and other peers, but you can't get network traffic to cross the tunnel, please work through the following checklist to identify the problem.
-
Check the Enclave network interface is online using
ipconfig /all
.Look for
Ethernet adapter Universe
in the list and check thatMedia State
is not shown asMedia disconnected
. If it is, try runningenclave restart
and consider sharing local Enclave logs with support@enclave.io to help us investigate the problem.C:\> ipconfig /all Ethernet adapter Universe: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Enclave Virtual Network Port Physical Address. . . . . . . . . : 00-FF-2C-FD-0F-12 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes IPv4 Address. . . . . . . . . . . : 100.117.177.98(Preferred) Subnet Mask . . . . . . . . . . . : 255.192.0.0 Default Gateway . . . . . . . . . : 100.127.255.254 DNS Servers . . . . . . . . . . . : 100.117.177.98
-
Check both connected peers can each ping their own Enclave IP address listed in
enclave status
as theLocal address:
. -
Check that the
Peer state
of the peer you're trying to reach is showing asUp
in theenclave status
output. -
When trying to ping another peer connected via Enclave, check that your ping requests are targeting the
Virtual address
of the destination peer and not theLocal address
. -
Check the routing table on each peer is configured correctly.
The routing table is configured automatically by Enclave so unlikely to be the source of a problem unless there are other conflicting routes already in place. Check for duplicate entries or conflicting routes in your routing table, or other network interfaces also using the
100.64.0.0/64
subnet.C:\> route print | findstr 100.64.0.0 IPv4 Route Table =========================================================================== Active Routes: Network Destination Netmask Gateway Interface Metric 100.64.0.0 255.192.0.0 On-link 100.117.177.98 281
-
Check the MTU configured in the
.profile
files matches on both sides of the tunnel. Each time Enclave starts it will set the interface MTU according to the value configured in the relevant.profile
so if you change of the the MTU manually, be sure reflect this change in the.profile
file, otherwise it will be overwritten each time Enclave is restarted. You may also use native OS tooling to examine configured MTUs. On Windows usenetsh interface ipv4 show subinterfaces
. On Linux useip addr | grep mtu
and on MacOS, useifconfig -a
.
Using Ping to verify connectivity¶
If you're using ping tests to verify connectivity, it's important to check that the host-local firewall is not dropping traffic arriving on the Enclave network interface.
-
Check that the host-local firewall on both peers is not obstructing traffic flows either to, or from the Enclave network interfaces.
On Linux the Enclave network interface is likely to be named
tap0
(or similar) and on Windows the Enclave network interface is usually calledUniverse
orEnclave Virtual Network Port
. On MacOS, the network interface is likely namedutun0
.The easiest way to verify that the host-local firewall is not interfering with Enclave traffic is to temporarily disable it, but if that's not possible for in environment, you should ensure ping traffic (icmp type 8, code 0) is permitted by the host-local firewall.
-
On Windows, you should check that the Enclave network interface is correctly classified by the Windows Firewall as
Private
and if required, create an ACL permitting inbound ICMP traffic using PowerShell:New-NetFirewallRule -DisplayName "ICMPv4 (In)" -Profile Private -Direction Inbound -Protocol ICMPv4 -Program Any -Action Allow`
-
Check that the ACLs defined by your Policies are allowing ICMP traffic to flow on both systems by examining the ACLs reported by
enclave status
on each peer.For example, if your attempts to ping a target system are timing out, check that the system you're sending the pings from has an allow ACL like
allow [X] from local -> peer
whereX
is either[any]
or includes the wordicmp
.Critically you're checking that the Enclave
local -> peer
ACL on the sender-side of the tunnel permitsicmp
(orany
) to be sent. If it does, that means thepeer -> local
ACL on the receiver side of the tunnel also permits that traffic to be received too.ACLs in Enclave are symmetric, so if the sender-side is allowed to send ICMP traffic, the receiver-side will implicitly be allowed to receive it.
DNS resolution not working¶
-
Obtain your Enclave local IP address using the
enclave get-ip
command.That's the IP address which the local DNS stub resolver will bind to (
udp\53
).C:\> enclave get-ip 100.117.177.98
-
Use
nslookup
to send a test query to the Enclave nameserver identified in step 1 by attempting to resolve the special DNS name_my.id.enclave
. It should resolve to your own local Enclave IP address, the same IP as the nameserver you're querying.C:\> nslookup _my.id.enclave 100.117.177.98 Server: UnKnown Address: 100.117.177.98 Name: _my.id.enclave Address: 100.117.177.98
-
If the DNS query fails, check there isn't other software bound to your Enclave
Local address
on portudp/53
-
If you're running Enclave on a Linux operating system or inside a container, check that you've correctly configured DNS forwarding to ensure DNS queries from the operating system are reaching the local Enclave DNS stub resolver.
-
Some ISPs hijack DNS queries to serve ads. Check that your primary recursive nameserver correctly returns a
SRVFAIL
response when queried with a hostname it should be unable to resolve. Some ISPs run nameservers which return IP addresses for ad servers instead of aSRVFAIL
for non-existent domains. The ping request shown below to the non-existing hostnamenull.enclave.io
should fail. If it does not, your ISPs nameservers may be hijacking your DNS requests to serve ads, in which case we recommend contacting your ISP to disable this function or switching to another primary resolver.C:\> ping null.enclave.io Ping request could not find host null.enclave.io. Please check the name and try again.
DNS queries slow to complete¶
- On Windows, check that the Enclave network interface doesn't list more than one DNS nameserver. If it does, please remove the DNS server that does not match the Enclave
Local address
shown by theenclave get-ip
command.
Enclave Gateway not working¶
See our Enclave Gateway troubleshooting guide.
Advanced troubleshooting¶
-
Try running Enclave as a foreground process with high log verbosity enabled
sudo enclave run -v 5
to inspect traffic flows. -
Try running
tcpdump
orWireShark
to capture from the tap0 interface (orEnclave Virtual Network Port
on Windows, usually also calledUniverse
) and inspect the traffic flows.
If you're still having problems after following this checklist, please contact support@enclave.io or join one of our community support channels to get help and advice.
Last updated May 5, 2022