If you find Enclave isn't working as expected, here's a simple set of troubleshooting checklists to follow. You can also search our developer community forum to see if any issue you might be experiencing have been observed by other users.
Your first port of call should always be to run
enclave self-test to see if any installation or runtime problems are detected.
- If you encounter the error
Could not find file "C:\Program Files\Enclave Networks\Enclave\Agent\ext\enclavetap6.inf"while running the Windows installer, add
https://release.enclave.ioto your firewall's allowed domains list and retry the installation.
Platform connectivity problems¶
If you think Enclave is having trouble reaching our SaaS services, please work through the following checklist to identify the problem.
Check https://status.enclave.io/ for any disruptions or service outages
Check your systems are enrolled and showing as both connected and approved in the portal
Check you're running on the latest version of Enclave with
Check the output of
enclave statusdoesn't contain any warnings or errors
Check the output of
enclave statuslists at least one
Peerand the state shows as
Check network traffic is allowed out to
Check network traffic is allowed out to
Check network traffic isn't forced through a SOCKS proxy, which is currently unsupported
Check local anti-virus software is not interfering with Enclave by temporarily disabling it
Problems enrolling new peers¶
Check https://status.enclave.io/ for disruptions or service outages
If you're running on Linux check that the
ca-certificatespackage is up to date
Check that the date and time on the enrolling system is accurate
Traffic not flowing between peers¶
If Enclave appears to be connected to our SaaS services and other peers, but you can't get network traffic to cross the tunnel, please work through the following checklist to identify the problem.
Check the Enclave network interface is online using
Ethernet adapter Universein the list and check that
Media Stateis not shown as
Media disconnected. If it is, try running
enclave restartand consider sharing local Enclave logs with firstname.lastname@example.org to help us investigate the problem.
C:\> ipconfig /all Ethernet adapter Universe: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Enclave Virtual Network Port Physical Address. . . . . . . . . : 00-FF-2C-FD-0F-12 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes IPv4 Address. . . . . . . . . . . : 100.117.177.98(Preferred) Subnet Mask . . . . . . . . . . . : 255.192.0.0 Default Gateway . . . . . . . . . : 100.127.255.254 DNS Servers . . . . . . . . . . . : 100.117.177.98
Check both connected peers can each ping their own Enclave IP address listed in
enclave statusas the
Check that the
Peer stateof the peer you're trying to reach is showing as
When trying to ping another peer connected via Enclave, check that your ping requests are targeting the
Virtual addressof the destination peer and not the
Check the routing table on each peer is configured correctly.
The routing table is configured automatically by Enclave so unlikely to be the source of a problem unless there are other conflicting routes already in place. Check for duplicate entries or conflicting routes in your routing table, or other network interfaces also using the
C:\> route print | findstr 100.64.0.0 IPv4 Route Table =========================================================================== Active Routes: Network Destination Netmask Gateway Interface Metric 100.64.0.0 255.192.0.0 On-link 100.117.177.98 281
Check the MTU configured in the
.profilefiles matches on both sides of the tunnel. Each time Enclave starts it will set the interface MTU according to the value configured in the relevant
.profileso if you change of the the MTU manually, be sure reflect this change in the
.profilefile, otherwise it will be overwritten each time Enclave is restarted. You may also use native OS tooling to examine configured MTUs. On Windows use
netsh interface ipv4 show subinterfaces. On Linux use
ip addr | grep mtuand on MacOS, use
Using Ping to verify connectivity¶
If you're using ping tests to verify connectivity, it's important to check that the host-local firewall is not dropping traffic arriving on the Enclave network interface.
Check that the host-local firewall on both peers is not obstructing traffic flows either to, or from the Enclave network interfaces.
On Linux the Enclave network interface is likely to be named
tap0(or similar) and on Windows the Enclave network interface is usually called
Enclave Virtual Network Port. On MacOS, the network interface is likely named
The easiest way to verify that the host-local firewall is not interfering with Enclave traffic is to temporarily disable it, but if that's not possible for in environment, you should ensure ping traffic (icmp type 8, code 0) is permitted by the host-local firewall.
On Windows, you should check that the Enclave network interface is correctly classified by the Windows Firewall as
Privateand if required, create an ACL permitting inbound ICMP traffic using PowerShell:
New-NetFirewallRule -DisplayName "ICMPv4 (In)" -Profile Private -Direction Inbound -Protocol ICMPv4 -Program Any -Action Allow`
Check that the ACLs defined by your Policies are allowing ICMP traffic to flow on both systems by examining the ACLs reported by
enclave statuson each peer.
For example, if your attempts to ping a target system are timing out, check that the system you're sending the pings from has an allow ACL like
allow [X] from local -> peerwhere
[any]or includes the word
Critically you're checking that the Enclave
local -> peerACL on the sender-side of the tunnel permits
any) to be sent. If it does, that means the
peer -> localACL on the receiver side of the tunnel also permits that traffic to be received too.
ACLs in Enclave are symmetric, so if the sender-side is allowed to send ICMP traffic, the receiver-side will implicitly be allowed to receive it.
DNS resolution not working¶
Obtain your Enclave local IP address using the
That's the IP address which the local DNS stub resolver will bind to (
C:\> enclave get-ip 100.117.177.98
nslookupto send a test query to the Enclave nameserver identified in step 1 by attempting to resolve the special DNS name
_my.id.enclave. It should resolve to your own local Enclave IP address, the same IP as the nameserver you're querying.
C:\> nslookup _my.id.enclave 100.117.177.98 Server: UnKnown Address: 100.117.177.98 Name: _my.id.enclave Address: 100.117.177.98
If the DNS query fails, check there isn't other software bound to your Enclave
Local addresson port
If you're running Enclave on a Linux operating system or inside a container, check that you've correctly configured DNS forwarding to ensure DNS queries from the operating system are reaching the local Enclave DNS stub resolver.
Some ISPs hijack DNS queries to serve ads. Check that your primary recursive nameserver correctly returns a
SRVFAILresponse when queried with a hostname it should be unable to resolve. Some ISPs run nameservers which return IP addresses for ad servers instead of a
SRVFAILfor non-existent domains. The ping request shown below to the non-existing hostname
null.enclave.ioshould fail. If it does not, your ISPs nameservers may be hijacking your DNS requests to serve ads, in which case we recommend contacting your ISP to disable this function or switching to another primary resolver.
C:\> ping null.enclave.io Ping request could not find host null.enclave.io. Please check the name and try again.
DNS queries slow to complete¶
- On Windows, check that the Enclave network interface doesn't list more than one DNS nameserver. If it does, please remove the DNS server that does not match the Enclave
Local addressshown by the
Gateway IP Connectivity not working¶
If you find an Enclave Gateway isn't working as expected, please start by follow this troubleshooting checklist:
Check your systems (sender systems and gateways) are all enrolled, connected and approved in the portal.
Check that the sender systems(s) can ping the Gateway using the Gateway's Enclave address.
Check the subnets which the Gateway is advertising.
Check the output of
enclave statuson all systems has the correct
Check that the Gateway itself can reach (i.e. ping) other devices on its local subnet.
Check the routing table has been correctly configured on the relevant Sender systems of the policy.
The routing table is configured automatically by Enclave so it's unlikely to be the source of a problem unless there are other conflicting routes already in place. The
Interfaceaddress is the client's local Enclave IP address.
C:\> route print | findstr 172.26.0.0 IPv4 Route Table =========================================================================== Active Routes: Network Destination Netmask Gateway Interface Metric 172.26.0.0 255.255.240.0 On-link 100.119.20.243 26
Check that iptables is correctly installed on the Enclave Gateway system (
sudo apt install iptables) and correctly configured.
sudo iptables -t nat -n -L ENCLAVE --line-numbers.
In particular pay attention to the
to:field on the postrouting chain, which should be the local (non-enclave) IP address of your Enclave Gateway.
user@gateway:~$ sudo iptables -t nat -n -L ENCLAVE --line-numbers Chain ENCLAVE (1 references) num target prot opt source destination 1 ACCEPT all -- 100.64.0.0/10 100.64.0.0/10 2 SNAT all -- 100.64.0.0/10 172.26.0.0/20 to:172.26.0.3
Check that the iptables
bytescounters are incrementing.
If they're not then the iptables configuration may be incorrect or the routing table on the sender system may not be correct.
We may also ask you to try running Enclave as a foreground process with high log verbosity to capture detailed diagnostic information.
Run enclave directly with
sudo enclave run -v 5to inspect traffic flows on the sender and Enclave Gateway.
We may also ask you to try running
tcpdumpon your Enclave Gateway to catpure a packet trace.
Run a tcpdump trace on your Enclave Gateway to capture traffic sent to the host you're trying to communicate with via the Enclave Gateway, for example a printer at
Using tcpdump on the network interface which your Gateway would use to reach target the target subnet (in this example, we're capturing on
eth0. In this example, you can see an successful ICMP echo request originate from an Enclave sender on the Gateway's eth0 interface (
172.26.0.3) destined for the printer (
172.26.0.250), followed by an ICMP echo reply showing the Gateway is working.
$ sudo tcpdump -ni eth0 host 172.26.0.250 11:28:12.444590 IP 172.26.0.3 > 172.26.0.250: ICMP echo request, id 1, seq 4208, length 40 11:28:12.444995 IP 172.26.0.250 > 172.26.0.3: ICMP echo reply, id 1, seq 4208, length 40
Gateway DNS not working¶
If there are issues resolving the names of systems behind an Enclave gateway, try these steps.
Check that the systems you wish to resolve addresses for are reachable by a configured Enclave policy. Gateways will only return results for systems they can route to.
If you have a
.localdomain suffix in your network, check our information on gateway DNS and the
Check whether any other VPN or Zero Trust Network Access software is installed on the system running Enclave Gateway. Consider removing it and restarting the system to rule out interoperability issues affecting DNS resolution.
Try running Enclave as a foreground process with high log verbosity enabled
sudo enclave run -v 5to inspect traffic flows.
WireSharkto capture from the tap0 interface (or
Enclave Virtual Network Porton Windows, usually also called
Universe) and inspect the traffic flows.
In order to help with troubleshooting, our support team may ask you to provide logs. The amount of information recorded in a log is determined by the
Log Level, which we may ask you to increase.
Enclave log levels are as follows:
|minimal||Always shown; critical problems, startup and shutdown messages.|
|informational||Operational state changes, some warnings. (this is the default value).|
|networkevents||Enclave connectivity events, including tunnel attempts.|
|protocolevents||Detailed protocol events, including keep-alives and tunnel negotiation.|
|debug||Debug tracing, including DNS queries and low-level socket events.|
|framedata||All network frames to/from this system are traced to the log. Significant performance impact.|
To collect logs at a given level for a brief period, you can set the Enclave log level temporarily, until Enclave restarts. This does not require admin rights.
To do this, run
enclave loglevel <desired-log-level> in a terminal.
To collect logs during Enclave start-up, or for a longer period, you can set the configured profile log level to ensure it is retained after a restart of the Enclave process. This is useful if, for example, the issue only happens when starting Enclave or we need to restart Enclave a few times for the issue to manifest.
Configuring the profile log level requires admin/elevated rights:
Windows: Open a command prompt window as administrator:
enclave set-config loglevel <desired-log-level>
Linux/Mac: Run this command in your terminal:
sudo enclave set-config loglevel <desired-log-level>
Once you're ready to send the logs you can identify the current log file by running
enclave status in a terminal.
Look for this line in the
Local Identity section at the top of
Log file location . : C:\Program Files\Enclave Networks\Enclave\Agent\logs\fabric.universe.20230315.pid.5452.log
From here you can forward this file over to our support team who will be happy to help.
Be aware you may you need to provide additional log files. We often recommend sending the entire contents of your "logs" directory, to ensure that we have as much information as possible to resolve your issue.
On mobile the process is slightly different:
Open the Enclave app, ensuring that enclave is first running, and then select the hamburger menu in the top left.
settingsand enable debug logging. From here, use the app as you would normally to reproduce the bug.
Once you've reproduced the bug you can now submit the logs. Navigate back to the
settingspage by again selecting the hamburger menu and then
Submit Enclave Logs. Once the logs have submitted you'll receive a pop up stating that the logs were successfully sent.
Last updated March 16, 2023