Troubleshooting¶

If you find Enclave isn't working as expected, here's a simple set of troubleshooting checklists to follow. You can also search our developer community forum to see if any issue you might be experiencing have been observed by other users.

Your first port of call should always be to run enclave self-test to see if any installation or runtime problems are detected.

Installation Issues¶

Windows Installer¶

If you encounter the error Could not find file "C:\Program Files\Enclave Networks\Enclave\Agent\ext\enclavetap6.inf" while running the Windows installer, add https://release.enclave.io to your firewall's allowed domains list and retry the installation.

Platform connectivity problems¶

If you think Enclave is having trouble reaching our SaaS services, please work through the following checklist to identify the problem.

Check https://status.enclave.io/ for any disruptions or service outages
Check your systems are enrolled and showing as both connected and approved in the portal
Check you're running on the latest version of Enclave with enclave version
Check the output of enclave status doesn't contain any warnings or errors
Check the output of enclave status lists at least one Peer and the state shows as Up
Check network traffic is allowed out to tcp/*:443
Check network traffic is allowed out to udp/*:1024-65355
Check network traffic isn't forced through a SOCKS proxy, which is currently unsupported
Check local anti-virus software is not interfering with Enclave by temporarily disabling it
Check whether the system in question may have been cloned, duplicating it's private key material

Problems enrolling new peers¶

Check https://status.enclave.io/ for disruptions or service outages
If you're running on Linux check that the ca-certificates package is up to date
Check that the date and time on the enrolling system is accurate

Traffic not flowing between peers¶

If Enclave appears to be connected to our SaaS services and other peers, but you can't get network traffic to cross the tunnel, please work through the following checklist to identify the problem.

Check the Enclave network interface is online using ipconfig /all.

Look for Ethernet adapter Universe in the list and check that Media State is not shown as Media disconnected. If it is, try running enclave restart and consider sharing local Enclave logs with support@enclave.io to help us investigate the problem.

C:\> ipconfig /all

Ethernet adapter Universe:

Connection-specific DNS Suffix  . :
Description . . . . . . . . . . . : Enclave Virtual Network Port
Physical Address. . . . . . . . . : 00-FF-2C-FD-0F-12
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 100.117.177.98(Preferred)
Subnet Mask . . . . . . . . . . . : 255.192.0.0
Default Gateway . . . . . . . . . : 100.127.255.254
DNS Servers . . . . . . . . . . . : 100.117.177.98

Check both connected peers can each ping their own Enclave IP address listed in enclave status as the Local address:.
Check that the Peer state of the peer you're trying to reach is showing as Up in the enclave status output.
When trying to ping another peer connected via Enclave, check that your ping requests are targeting the Virtual address of the destination peer and not the Local address.
Enclave honours the local firewall configuration. Check that the traffic you're trying to send is permitted to be received by any host-based firewalls on the target system. Pay particular attention to Windows Firewall; although Enclave sets its interface classification in Windows Firewall to Private, occasionally we see misclassification as Public instead. Also note that if you're expecting to be able to PING a target system, and are able to do so over the bearer network but not via Enclave, it's possible that there is an ICMP rule in the Windows firewall that allows ICMP packets in only on Domain classified interfaces, for example, and not Private interfaces too.
Check the routing table on each peer is configured correctly.

The routing table is configured automatically by Enclave so unlikely to be the source of a problem unless there are other conflicting routes already in place. Check for duplicate entries or conflicting routes in your routing table, or other network interfaces also using the 100.64.0.0/64 subnet.
```
C:\> route print | findstr 100.64.0.0

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination    Netmask        Gateway         Interface    Metric
100.64.0.0             255.192.0.0    On-link    100.117.177.98       281
```
Check that the MTU configured in the .profile files matches on both sides of the tunnel. Each time Enclave starts, it will set the interface MTU according to the value configured in the relevant .profile, so if you change the MTU manually, be sure to reflect this change in the .profile file; otherwise, it will be overwritten each time Enclave is restarted. You may also use native OS tools to examine configured MTUs. On Windows, use netsh interface ipv4 show subinterfaces. On Linux, use ip addr | grep mtu, and on macOS, use ifconfig -a.

Using ping to verify connectivity¶

If you're using ping tests to verify connectivity, it's important to check that the host-local firewall is not dropping traffic arriving on the Enclave network interface.

Check that the host-local firewall on both peers is not obstructing traffic flows either to, or from the Enclave network interfaces.

On Linux the Enclave network interface is likely to be named tap0 (or similar) and on Windows the Enclave network interface is usually called Universe or Enclave Virtual Network Port. On MacOS, the network interface is likely named utun0.

The easiest way to verify that the host-local firewall is not interfering with Enclave traffic is to temporarily disable it, but if that's not possible for in environment, you should ensure ping traffic (icmp type 8, code 0) is permitted by the host-local firewall.
On Windows, you should check that the Enclave network interface is correctly classified by the Windows Firewall as Private and if required, create an ACL permitting inbound ICMP traffic using PowerShell:
```
New-NetFirewallRule -DisplayName "ICMPv4 (In)" -Profile Private -Direction Inbound -Protocol ICMPv4 -Program Any -Action Allow`
```
Check that the ACLs defined by your Policies are allowing ICMP traffic to flow on both systems by examining the ACLs reported by enclave status on each peer.

For example, if your attempts to ping a target system are timing out, check that the system you're sending the pings from has an allow ACL like allow [X] from local -> peer where X is either [any] or includes the word icmp.

Critically you're checking that the Enclave local -> peer ACL on the sender-side of the tunnel permits icmp (or any) to be sent. If it does, that means the peer -> local ACL on the receiver side of the tunnel also permits that traffic to be received too.

ACLs in Enclave are symmetric, so if the sender-side is allowed to send ICMP traffic, the receiver-side will implicitly be allowed to receive it.

Relayed connections between peers¶

If you observe that some or all of your enrolled systems are using Enclave's traffic relay servers to connect, you may want to explore reconfiguring the bearer network to facilitate peer-to-peer (P2P) connections. Relay servers are a fallback mechanism, used when direct connections fail due to restrictive NAT settings, high-port blocks, or threat inspection modules on routers and firewalls.

Relay servers are shared across the platform and have throughput restrictions, which can lead to increased latency and reduced bandwidth. Follow the steps below to troubleshoot and potentially resolve relayed connections.

Check NAT (Network Address Translation) settings on routers and firewalls. NAT configurations can either support or block direct peer-to-peer connections. Depending on the type of NAT, you may need adjustments to improve connectivity. Full-Cone NAT allows external hosts to contact internal hosts using the same port as the original outbound connection, making it ideal for direct connections. Also, Address-Restricted Cone NAT and Port-Restricted Cone NAT are also favorable for peer-to-peer connectivity as they typically allow direct communication in most scenarios. Symmetric NAT is highly restrictive however, as it assigns a different public port for each outbound destination, which complicates direct peer-to-peer communication and often necessitates relayed connections.

Verify that the firewall rules and NAT settings on both systems allow outbound UDP and TCP traffic on the ports used by Enclave. If possible, adjust NAT modes to a less restrictive type, such as Full-Cone or Port/Address-Restricted Cone NAT. See also notes about traffic relays.
Firewalls can block outbound connections, particularly on high-numbered ports, which are necessary for successful peer-to-peer communication. Verify that outbound UDP and TCP connections on high-numbered ports (typically above 1024) are allowed on both systems. To check if outbound connectivity on high-numbered ports is available, you can use the http://portquiz.net/ website. If outbound connectivity on high ports fails, update firewall settings to permit outbound traffic on these ports. See also the related guidance on firewall ports for establishing peer-to-peer connectivity.
Ensure threat detection or deep packet inspection rules are not interfering with Enclave traffic, and check for any explicit P2P detection and blocking on your network firewalls. See also using Enclave with your firewall.

DNS resolution not working¶

Obtain your Enclave local IP address using the enclave get-ip command.

That's the IP address which the local DNS stub resolver will bind to (udp\53).
```
C:\> enclave get-ip
100.117.177.98
```
Use nslookup to send a test query to the Enclave nameserver identified in step 1 by attempting to resolve the special DNS name _my.id.enclave. It should resolve to your own local Enclave IP address, the same IP as the nameserver you're querying.
```
C:\> nslookup _my.id.enclave 100.117.177.98
Server:   UnKnown
Address:  100.117.177.98

Name:     _my.id.enclave
Address:  100.117.177.98
```
If the DNS query fails, check there isn't other software bound to your Enclave Local address on port udp/53
If you're running Enclave on a Linux operating system or inside a container, check that you've correctly configured DNS forwarding to ensure DNS queries from the operating system are reaching the local Enclave DNS stub resolver.
Some ISPs hijack DNS queries to serve ads. Check that your primary recursive nameserver correctly returns a SRVFAIL response when queried with a hostname it should be unable to resolve. Some ISPs run nameservers which return IP addresses for ad servers instead of a SRVFAIL for non-existent domains. The ping request shown below to the non-existing hostname null.enclave.io should fail. If it does not, your ISPs nameservers may be hijacking your DNS requests to serve ads, in which case we recommend contacting your ISP to disable this function or switching to another primary resolver.
```
C:\> ping null.enclave.io
Ping request could not find host null.enclave.io. Please check the name and try again.
```

DNS queries slow to complete¶

On Windows, check that the Enclave network interface doesn't list more than one DNS nameserver. If it does, please remove the DNS server that does not match the Enclave Local address shown by the enclave get-ip command.

Enclave Gateway¶

IP connectivity not working¶

If you find an Enclave Gateway isn't routing traffic as expected, please start by follow this troubleshooting checklist:

Check your systems (sender systems and gateways) are all enrolled, connected and approved in the portal.
Check that the sender systems(s) can ping the Gateway using the Gateway's Enclave address.
Check the subnets which the Gateway is advertising.
Check the output of enclave status on all systems has the correct Gateway for subnets.
Check that the Gateway itself can reach (i.e. ping) other devices on its local subnet.
Check the routing table has been correctly configured on the relevant Sender systems of the policy.

The routing table is configured automatically by Enclave so it's unlikely to be the source of a problem unless there are other conflicting routes already in place. The Interface address is the client's local Enclave IP address.
```
C:\> route print | findstr 172.26.0.0
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination         Netmask     Gateway         Interface    Metric
172.26.0.0            255.255.240.0     On-link    100.119.20.243        26
```
Check that iptables is correctly installed on the Enclave Gateway system (sudo apt install iptables) and correctly configured.

Run sudo iptables -t nat -n -L ENCLAVE --line-numbers.

In particular pay attention to the to: field on the postrouting chain, which should be the local (non-enclave) IP address of your Enclave Gateway.
```
user@gateway:~$ sudo iptables -t nat -n -L ENCLAVE --line-numbers
Chain ENCLAVE (1 references)
num  target     prot opt source               destination
1    ACCEPT     all  --  100.64.0.0/10        100.64.0.0/10
2    SNAT       all  --  100.64.0.0/10        172.26.0.0/20            to:172.26.0.3
```
Check that the iptables pkts and bytes counters are incrementing.

If they're not then the iptables configuration may be incorrect or the routing table on the sender system may not be correct.
We may also ask you to try running Enclave as a foreground process with high log verbosity to capture detailed diagnostic information.

Run enclave directly with sudo enclave run -v 5 to inspect traffic flows on the sender and Enclave Gateway.
We may also ask you to try running tcpdump on your Enclave Gateway to catpure a packet trace.

Run a tcpdump trace on your Enclave Gateway to capture traffic sent to the host you're trying to communicate with via the Enclave Gateway, for example a printer at 172.26.0.250.

Using tcpdump on the network interface which your Gateway would use to reach target the target subnet (in this example, we're capturing on eth0. In this example, you can see an successful ICMP echo request originate from an Enclave sender on the Gateway's eth0 interface (172.26.0.3) destined for the printer (172.26.0.250), followed by an ICMP echo reply showing the Gateway is working.
```
$ sudo tcpdump -ni eth0 host 172.26.0.250
11:28:12.444590 IP 172.26.0.3 > 172.26.0.250: ICMP echo request, id 1, seq 4208, length 40
11:28:12.444995 IP 172.26.0.250 > 172.26.0.3: ICMP echo reply, id 1, seq 4208, length 40
```

Periodic disconnection¶

If you find an Enclave Gateway is working as expected, but periodically disconnects from the Enclave platform, please follow this troubleshooting checklist:

Check there are no IP conflicts on the local network. We suggest using arp-scan. To detect IP conflicts, run the following:

$ sudo arp-scan –I eth0 -l

As an output example, in this instance, the `192.168.1.39 IP is in conflict since it appears twice.

192.168.1.10   00:1b:a9:63:a2:4c       BROTHER INDUSTRIES, LTD.
192.168.1.30   00:1e:8f:58:ec:49       CANON INC.
192.168.1.33   00:25:4b:1b:10:20       Apple, Inc
192.168.1.37   10:9a:dd:55:d7:95       Apple Inc
192.168.1.38   20:c9:d0:27:8d:56       (Unknown)
192.168.1.39   d4:85:64:4d:35:be       Hewlett Packard
192.168.1.39   00:0b:46:e4:8e:6d       Cisco (DUP: 2)
192.168.1.40   90:2b:34:18:59:c0       (Unknown)

DNS not working¶

If there are issues resolving the names of systems behind an Enclave gateway, try these steps.

Check that the systems you wish to resolve addresses for are reachable by a configured Enclave policy. Gateways will only return results for systems they can route to.
Check if latency to the Enclave Gateway host is reporting as greater than 1000ms. Windows allows 1 second for local DNS queries to complete. If the gateway is not able to recieve and respond to DNS queries from the local OS in 1 second, Windows will failover to the next available resolver, giving the appearance that Enclave Gateway is not correctly resolving hostnames.
If you have a .local domain suffix in your network, check our information on gateway DNS and the .local domain.
Check whether any other VPN or Zero Trust Network Access software is installed on the system running Enclave Gateway. Consider removing it and restarting the system to rule out interoperability issues affecting DNS resolution.

Advanced troubleshooting¶

It may be necessesary to gain a deeper level of understanding into Enclave behaviour. To do enable richer log messages it may be useful to run Enclave as a foreground process with high log verbosity using sudo enclave run -v 5 to inspect output in realtime. Beware, this will also increase the size of your log files. You may also find it useful to run tcpdump or WireShark to capture from tap0 or tun0 interfaces on Linux, or Enclave Virtual Network Port and Universe adatpers on Windows to inspect traffic flows.

If you're still having problems after following this checklist, please contact support@enclave.io or join one of our community support channels to get help and advice.

Collecting logs¶

Enclave is composed of two separate parts, the Fabric, which is the background process responsible for the creating and moving packets across the virtual network interface, and a desktop user interface on Windows that communicates with the Fabric using local API calls. Our support team may ask you to collect logs from the Fabric, the desktop app, or both, depending on the nature of the problem you're encountering.

Enclave Fabric¶

In order to help with troubleshooting on Windows, Linux or MacOS, our support team may ask you to provide logs from the Enclave Fabric. The amount of information recorded in a log is determined by the Log Level, which we may ask you to increase.

Enclave log levels are set as follows:

Level
minimal	Always shown; critical problems, startup and shutdown messages.
informational	Operational state changes, some warnings. (this is the default value).
networkevents	Enclave connectivity events, including tunnel attempts.
protocolevents	Detailed protocol events, including keep-alives and tunnel negotiation.
debug	Debug tracing, including DNS queries and low-level socket events.
framedata	All network frames to/from this system are traced to the log. Significant performance impact.

To collect logs from the Encalve Network Fabric at a given level for a brief period, you can set the Enclave log level temporarily, until Enclave restarts. This does not require admin rights.

To do this, run enclave loglevel <desired-log-level> in a terminal.

To collect logs during Enclave start-up, or for a longer period, you can set the configured profile log level to ensure it is retained after a restart of the Enclave process. This is useful if, for example, the issue only happens when starting Enclave or we need to restart Enclave a few times for the issue to manifest.

Configuring the profile log level requires admin/elevated rights:

Windows: Open a command prompt window as administrator: enclave set-config loglevel <desired-log-level>
Linux/Mac: Run this command in your terminal: sudo enclave set-config loglevel <desired-log-level>

Once you're ready to send the logs you can identify the current log file by running enclave status in a terminal.

Look for this line in the Local Identity section at the top of enclave status:

Log file location . : C:\Program Files\Enclave Networks\Enclave\Agent\logs\fabric.universe.20230315.pid.5452.log

From here you can forward this file over to our support team who will be happy to help.

Note

Be aware you may you need to provide additional log files. We often recommend sending the entire contents of your "logs" directory, to ensure that we have as much information as possible to resolve your issue.

Enclave desktop app¶

If we suspect that the source of a bug lies within the desktop app, rather than the supervisor service, we might ask you to collect the desktop app application logs instead. These can be found at %TEMP%\enclave-tray-logs.

The desktop app log level is elevated independently of the supervisor service. To do this, first close down the application by right clicking on the desktop app icon and selecting Exit. Then, run enclave-tray --profile <profile-name> --verbosity <desired-integer-log-level> in the terminal, where:

<profile-name> is the name of your profile.
<desired-log-level-as-integer> is an integer between 0 and 5, where 0 corresponds to the minimal log level. The higher the value, the higher the log level.

Enclave on mobile¶

On mobile devices the process for collecting logs from the Enclave Fabric is slightly different:

Open the Enclave app, ensuring that enclave is first running, and then select the hamburger menu in the top left.
Select settings and enable debug logging. From here, use the app as you would normally to reproduce the bug.
Once you've reproduced the bug you can now submit the logs. Navigate back to the settings page by again selecting the hamburger menu and then settings.
Now select Submit Enclave Logs. Once the logs have submitted you'll receive a pop up stating that the logs were successfully sent.

Contact support¶

If you've followed our troubleshooting guides and are still unable to resolve the problem, please reach out to our support team for assistance.

Last updated September 26, 2024