A Comprehensive Guide to FreeBSD Monitoring and Troubleshooting

FreeBSD is an operating system that has been around for over three decades. Thanks to its advanced networking, storage, and security features, it is widely used in servers, embedded systems, and networking equipment. However, like any powerful system, FreeBSD requires careful attention to maintain optimal performance.

To ensure its smooth operation, it’s important to actively monitor its health and status, as well as quickly troubleshoot any detected issues or bottlenecks. This guide walks you through the steps and techniques to do just that.

Understanding FreeBSD monitoring

FreeBSD monitoring is an important process that tracks system performance, resource usage, and network activity to gauge the overall health and efficacy of a system. Through monitoring, administrators can identify issues before they become critical, and quickly gather the required context to diagnose and resolve them.

Here are some additional reasons why FreeBSD monitoring is important:

• It helps spot CPU spikes, memory leaks, or disk I/O slowdowns before they degrade performance.
• FreeBSD is widely used in networking roles, such as firewalls and routers. Monitoring tracks packet loss, bandwidth usage, and connection errors, which helps maintain reliable communication.
• FreeBSD’s Zettabyte File System (ZFS) is a powerful feature, but it requires monitoring to detect issues like pool fragmentation, disk failures, or degraded performance. Regular checks can prevent data loss
• Unusual spikes in system activity or unauthorized access attempts could signal security threats. By monitoring logs and system behavior, you can detect and respond to these risks.
• FreeBSD’s kernel is known for its reliability, but monitoring helps detect anomalies like kernel panics, resource exhaustion, or hardware failures that could disrupt operations.

Key FreeBSD metric categories to monitor

These are the metric categories to focus on:

1. CPU usage

The CPU is one of the most critical resources in any FreeBSD system. High CPU usage can slow down applications, cause delays in processing, and lead to system instability. By tracking CPU-related metrics, you can identify workload trends and detect performance bottlenecks before they affect system efficiency. Here are some examples of CPU usage metrics:

• Load average: The number of processes waiting for CPU time. A consistently high value suggests CPU congestion.
• User vs. system CPU time: Differentiates between application processing time (user) and system operations (kernel).
• Idle time: Measures unused CPU capacity. Low idle time may indicate system overload.

2. Memory utilization

Memory usage directly affects system performance, especially when running multiple applications, virtual machines, or jails. If FreeBSD runs out of available memory, it may start swapping data to disk, which significantly slows down performance. Important memory metrics include:

• Active, inactive, and free memory: Tracks how RAM is allocated and whether enough is available for new tasks.
• Swap usage: Indicates if the system is relying on disk-based memory (swap), which can slow down performance.
• Wired memory: Shows memory locked by the kernel that cannot be swapped out. High wired memory may indicate excessive kernel resource usage.

3. Network activity

Problems such as high latency, dropped packets, or unexpected traffic patterns can indicate misconfigurations, hardware failures, or security threats. To avoid disruptions, monitor these network-related metrics:

• Packet loss: Measures the percentage of lost packets.
• Interface traffic (RX/TX bytes): Tracks the volume of data sent and received on network interfaces.
• Errors and collisions: Indicates issues with data transmission, which could be due to misconfiguration or hardware problems.
• Connections and active sockets: Helps monitor the number of open connections, which can reveal abnormal traffic patterns or potential attacks.

4. Disk and storage performance

Disk performance is crucial for FreeBSD systems, especially those running databases, file servers, or other data-intensive applications. Monitor storage to prevent disk overload, an issue that can lead to severe performance bottlenecks and data corruption.

• Disk I/O (read/write speed): Measures how fast data is read from and written to disk. Slow speeds indicate potential bottlenecks.
• ZFS ARC efficiency: Tracks how well data is cached in memory.
• Disk space usage: Ensures that partitions and volumes do not run out of space.
• Filesystem mount health: Monitors whether filesystems are correctly mounted and accessible.

5. System logs and errors

Logs contain valuable information about the system’s health, security, and operational status. Regularly check system logs to detect hardware failures, security incidents, and software crashes before they become major issues. Pay special attention to:

• Kernel messages (dmesg): Displays system boot logs and hardware-related errors.
• Syslog (/var/log/messages): Stores system-wide logs that help diagnose issues.
• Authentication logs (/var/log/auth.log): Tracks login attempts and authentication failures.

6. Processes and Services

FreeBSD runs multiple processes and services, from web servers to background daemons. Monitor active processes to make sure that system resources are used efficiently and that critical services remain available.

• Running processes (top, ps): Shows active processes and their resource usage.
• Zombie processes: Indicates processes that have completed execution but are still in the process table.
• Service availability (service status): Checks if critical services (e.g., SSH, web servers, database servers) are running as expected.

Tools for FreeBSD monitoring

FreeBSD has several built-in tools for monitoring system performance, resource usage, and network activity. Let’s explore some of the best ones:

top

The top command provides a real-time overview of system performance. It shows CPU usage, memory consumption, and active processes. Use it for quickly identifying resource-heavy processes. Simply open a terminal and enter:

top

Then:

• Press q to quit.
• Press m to sort by memory usage.
• Press P to sort by CPU usage.

sysctl

sysctl allows you to view and modify kernel parameters, including system performance data.

sysctl -a | grep cpu

This command lists all CPU-related system parameters. For example:

kernel.hardlockup_all_cpu_backtrace = 0
kernel.hung_task_all_cpu_backtrace = 0
kernel.oops_all_cpu_backtrace = 0
kernel.perf_cpu_time_max_percent = 25
kernel.softlockup_all_cpu_backtrace = 0
kernel.watchdog_cpumask = 0-3
net.core.flow_limit_cpu_bitmap = 0
vm.percpu_pagelist_high_fraction = 0

Vmstat

vmstat provides a summary of CPU, memory, and disk activity. It’s great for diagnosing system slowdowns.

vmstat 5

This command updates system statistics every 5 seconds.

iostat

iostat is useful for monitoring disk I/O performance and identifying slow disk operations.

iostat -x 5

This command displays extended disk I/O statistics, with the output updating every 5 seconds.

netstat

netstat helps track active network connections, interface statistics, and routing tables.

netstat -an

This command lists all active network connections.

ping and traceroute

These tools help diagnose network connectivity issues and detect slow or unreachable hosts.

ping google.com

This command checks if a remote server is reachable.

traceroute google.com

This one shows the path packets take to reach the destination.

DTrace

DTrace is a powerful FreeBSD tool that provides deep insights into system performance. Use it for real-time debugging of kernel and user-space activity.

dtrace -n 'syscall:::entry { @num[execname] = count(); }'

This command monitors system calls made by running processes.

ZFS monitoring tools

zpool and zfs commands can be used to track storage health and performance.

zpool status

This command checks the status of ZFS pools.

zfs list

This command displays information about available ZFS data sets.

Log monitoring with tail and grep

The tail and grep commands help analyze log files in real time.

tail -f /var/log/messages

This continuously displays new log entries as they appear in the /var/log/messages file.

grep "error" /var/log/messages

This command filters logs to show only error messages in the file.

Site24x7

Alternatively, if you want a dedicated monitoring solution that covers all key metrics, including CPU, memory, disk, network, and logs, Site24x7 is a great option. It provides real-time monitoring, alerting, and performance analysis for FreeBSD systems, along with visual dashboards for easy tracking.

FreeBSD issue troubleshooting guide

Even with proper monitoring, FreeBSD systems can encounter issues that affect performance, stability, and functionality. Let’s discuss some common FreeBSD problems and how to solve them.

Performance issues

Performance slowdowns can arise due to high resource usage, misconfigured settings, or inefficient processes.

High CPU Usage

You experience consistently high CPU usage.

Symptoms:

• System response time is slow.
• CPU usage remains high even when no resource-intensive applications are running.

Troubleshooting:

• Use top or htop to identify processes consuming the most CPU.

top -P

• If a specific process is consuming too much CPU, try renicing or killing it:

renice 10   # Lowers process priority
kill -9     # Force kill the process

• Check for runaway or zombie processes using ps aux.
• Review /var/log/messages for any warnings related to CPU spikes.
• Consider enabling process limits using login.conf to prevent excessive CPU consumption.

Memory utilization spikes

Excessive memory consumption can lead to slowdowns and application crashes.

Symptoms:

• The FreeBSD system runs out of memory, causing applications to crash.
• System starts using swap excessively.

Troubleshooting:

• Check memory usage using top or vmstat:

vmstat -s

• Identify memory-hungry processes using:

ps aux | sort -nk4

• Once you've identified potential culprits, investigate the behavior of those applications. Look for memory leaks, inefficient algorithms, or misconfigurations that could be contributing to excessive memory usage.
• Consider setting resource limits for processes to prevent them from consuming excessive memory.

Slow disk performance

Disk I/O bottlenecks can slow down system operations and file access.

Symptoms:

• Delays in reading/writing files.
• High disk I/O wait time in iostat.

Troubleshooting:

• Use iostat -x to check disk performance metrics.
• Identify large read/write operations with du -sh /*.
• Optimize file system performance using tunefs.
• Check for failing drives using smartctl.

Networking issues

Network problems can cause connectivity failures and slow data transfers.

No internet connectivity

Network misconfigurations can prevent the system from connecting to the internet.

Symptoms:

• Unable to ping external sites.
• ifconfig shows no assigned IP address.

Troubleshooting:

• Use ifconfig to check network interfaces and ensure that they are up.
• Restart the network service with service netif restart.
• Use nslookup or dig to see if DNS is working.
• Use netstat -rn to inspect the routing table. This command displays the network routes configured on the system. Ensure that a default gateway is present and that routes to external networks are correctly configured.
• Use traceroute to trace the path of network packets to a destination. This can help identify where connectivity is failing.

Slow network speeds

High latency or bandwidth issues can cause slow data transfers.

Symptoms:

• Slow response when accessing remote servers.
• High packet loss in ping tests.

Troubleshooting:

• Use ping -c 5 <destination> to check for packet loss.
• Run netstat -i to analyze network interface errors.
• Check firewall rules with pfctl -sr to ensure traffic isn’t being blocked.
• Test internet speed using iperf.

Scalability problems

Scaling issues can affect system performance under increased workloads.

Process limits reached

Too many running processes can exhaust system resources.

Symptoms:

• New processes fail to start.
• Errors related to maxproc in system logs.

Troubleshooting:

• Check the current process count with sysctl kern.maxproc.
• Increase process limits in /etc/sysctl.conf.
• Use pkill -9 <process> to terminate unnecessary processes.
• If a critical service is consuming excessive resources, restart it to temporarily fix the issue and debug its code for a more long-term fix.

File system issues

File system problems can lead to data loss, corruption, or access failures.

Disk space running out

Low disk space can cause system instability and prevent you from creating files.

Symptoms:

• df -h shows disk usage at 100%.
• New files cannot be saved.

Troubleshooting:

• Use du -sh /* to identify large files.
• Remove unnecessary logs. For example, use this command to clear up /var/log:

rm -rf /var/log/*.gz

• Clear old snapshots if using ZFS (zfs list -t snapshot).
• If nothing else works, extend disk space or add a new storage device.

File system corruption

Corrupted file systems can prevent booting and lead to data loss.

Symptoms:

• Errors when accessing or modifying files.
• System fails to boot due to file system inconsistencies.

Troubleshooting:

• Run fsck -y /dev/ada0 to check and repair file system errors.
• If using ZFS, use zpool scrub to detect and fix corruption.
• Check system logs (dmesg) for disk-related errors.
• If corruption is severe, your only option may be to restore a backup.

Kernel panics and crashes

Critical errors in the kernel can cause system instability or crashes.

Kernel panic on boot

Boot failures due to kernel issues can prevent the system from starting.

Symptoms:

• System hangs or reboots during startup.
• Kernel panic messages appear on screen.

Troubleshooting:

• Boot into single-user mode and check logs with less /var/log/messages.
• Verify kernel modules with kldstat.
• Roll back to a previous kernel using boot -s and nextboot.
• File system corruption can sometimes cause kernel panics. In single-user mode, run fsck -y / to check and repair the root file system. Run fsck on any other partitions as well.
• Check the /boot/loader.conf file for any custom kernel parameters or module loading configurations that could be causing conflicts.
• Search online for the specific kernel panic messages to find potential solutions. The FreeBSD handbook and forums are excellent resources.

System freezes randomly

Unstable drivers or hardware failures can cause unexpected freezes.

Symptoms:

• System becomes unresponsive without warning.
• No logs recorded before the freeze.

Troubleshooting:

• Disable unneeded kernel modules with kldunload.
• Look for overheating issues using sysctl -a | grep temperature.
• If not done already, upgrade FreeBSD to the latest stable version.
• Investigate any recent software installations or updates that might be causing conflicts.
• Disable any aggressive power management settings that may be causing the system to become unstable.
• Faulty network cards or drivers can sometimes cause system freezes. Disable the network interface to test.

Best practices for a healthy FreeBSD environment

To prevent several of the above-mentioned issues from happening in the first place, adhere to the following best practices:

Regularly monitor system health

As discussed earlier, regular monitoring will help you detect, even forecast, issues before they become full-blown problems. Remember to:

• Use top, vmstat, and iostat to track CPU, memory, and disk usage.
• Set up automated alerts with tools like Site24x7.
• Check logs (/var/log/messages) for warning signs.

Tune kernel parameters for performance

FreeBSD’s kernel is highly tunable, and optimizing its parameters can significantly improve performance and stability. You can use sysctl (or edit the /etc/sysctl.conf file) to adjust kernel settings based on your workload. For example:

• Set process limits by modifying the kern.maxproc property.
• Fine-tune network buffers with net.inet.tcp.recvbuf_max and net.inet.tcp.sendbuf_max.

Keep FreeBSD and installed packages updated

Regular updates improve security and performance. Here are some tips:

• Update the base system using freebsd-update fetch install.
• Upgrade installed packages with pkg upgrade.
• Regularly check Errata Notices and Security Advisories from the FreeBSD project.

Use ZFS for advanced file system management

ZFS provides reliability and efficiency for storage management. Make sure to:

• Enable automatic snapshots (zfs snapshot) for quick recovery.
• Use zfs scrub to detect and fix data corruption.
• Optimize performance by tuning ARC cache size (vfs.zfs.arc_max).

Implement proper security measures

A secure system prevents unauthorized access and data breaches. Remember to:

• Disable unnecessary services with service <service> disable.
• Use pf (Packet Filter) firewall to control network traffic.
• Regularly audit security configurations.

Automate maintenance tasks

Automation reduces manual work and ensures consistent upkeep. You should

• Schedule periodic cleanup scripts for logs and cache.
• Automate system updates using cron or periodic.conf.
• Use configuration management tools like Ansible for system provisioning.

Optimize network configuration

Fine-tune network settings to improve performance and reliability.

• Adjust TCP settings (sysctl net.inet.tcp.delayed_ack=0) for low-latency applications.
• Use netstat -i to monitor interface errors and optimize routing.
• Enable fastforwarding (sysctl net.inet.ip.fastforwarding=1) for faster packet processing.

Conclusion

FreeBSD is a stable and high-performance operating system that has been widely used for decades in networking, storage, and security-critical environments. However, as we have established, proactive monitoring and timely troubleshooting are important to keep it running smoothly. We hope that this guide has equipped you with the tools and insights to do that.

If you are looking for an all-in-one FreeBSD monitoring solution, don’t forget to try out the FreeBSD monitoring tool by Site24x7.