High CPU usage due to short bond heartbeat period (0.1s) in nav2_util::LifecycleNode

[zz990099]

Bug report

Required Info:

• Operating System:
  • Ubuntu-24.04
• Computer:
  • 12th Gen Intel(R) Core(TM) i5-12600KF
• ROS2 Version:
  • rolling
• Version or commit hash:
  • 52334b0
• DDS implementation:
  • Fast-RTPS

Description

We encountered this issue while deploying Navigation2 on an RK3588 platform. We observed that the navigation node group was using a high amount of CPU even when idle, which affected the overall stability of the system. After investigating, we identified that the default heartbeat period in nav2_util::LifecycleNode was set too low, leading to the excessive CPU usage.

1. With the current heartbeat confirmation period set to 0.1s, the nav2_container node group's CPU usage is around 140% while idle.
2. When we increase the confirmation period to 1.0s, the idle CPU usage of the nav2_container node group drops to about 75%.
3. If we disable the composition feature and run individual node processes, with a period interval of 0.1s, each process uses around 15% CPU; increasing the period to 1.0s brings the usage down to 7% per process.

This issue may partly explain why Nav2 performs poorly on hardware with limited computing resources.

CPU Usage Screenshots

• nav2_container cpu usage with heartbeat period 0.1s.

• Single-node process cpu usage with heartbeat period 0.1s.

• nav2_container cpu usage with heartbeat period 1.0s.

• Single-node process cpu usage with heartbeat period 1.0s.

Suggested Modification Strategy

1. Increase the bond_heartbeat_period in nav2_util::LifecycleNode to 1.0s.
   

   navigation2/nav2_ros_common/include/nav2_ros_common/lifecycle_node.hpp

   Line 70 in 52334b0

   bond_heartbeat_period = this->declare_or_get_parameter<double>("bond_heartbeat_period", 0.1);

2. Parameterize the hard-coded bond_heartbeat_period in nav2_lifecycle_manager::LifecycleManager, and update its default value to 1.0s.

3. Remove the connection establishment check in the createBondConnection function of nav2_lifecycle_manager::LifecycleManager to avoid prolonged startup times.
   

   navigation2/nav2_lifecycle_manager/src/lifecycle_manager.cpp

   Lines 273 to 300 in 52334b0

   	LifecycleManager::createBondConnection(const std::string & node_name)
   	{
   	const double timeout_ns =
   	std::chrono::duration_cast<std::chrono::nanoseconds>(bond_timeout_).count();
   	const double timeout_s = timeout_ns / 1e9;
   	if (bond_map_.find(node_name) == bond_map_.end() && bond_timeout_.count() > 0.0) {
   	bond_map_[node_name] =
   	std::make_shared<bond::Bond>("bond", node_name, shared_from_this());
   	bond_map_[node_name]->setHeartbeatTimeout(timeout_s);
   	bond_map_[node_name]->setHeartbeatPeriod(0.10);
   	bond_map_[node_name]->start();
   	if (
   	!bond_map_[node_name]->waitUntilFormed(
   	rclcpp::Duration(rclcpp::Duration::from_nanoseconds(timeout_ns / 2))))
   	{
   	RCLCPP_ERROR(
   	get_logger(),
   	"Server %s was unable to be reached after %0.2fs by bond. "
   	"This server may be misconfigured.",
   	node_name.c_str(), timeout_s);
   	return false;
   	}
   	RCLCPP_INFO(get_logger(), "Server %s connected with bond.", node_name.c_str());
   	}
   	return true;
   	}

Thanks

Thank you very much for your hard work and dedication to maintaining and improving Navigation2. We are submitting this issue to share our recent experience and observations, and hope that it can help make Nav2 better. We greatly appreciate your time and any suggestions.

[mini-1235]

Interestingly, I am also working on an issue this week that may improve Nav2 performance, so I can take a deeper look at this

When we increase the confirmation period to 1.0s, the idle CPU usage of the nav2_container node group drops to about 75%.

Do you mean you modified the following lines:

navigation2/nav2_ros_common/include/nav2_ros_common/lifecycle_node.hpp

Line 70 in 52334b0

bond_heartbeat_period = this->declare_or_get_parameter<double>("bond_heartbeat_period", 0.1);

navigation2/nav2_lifecycle_manager/src/lifecycle_manager.cpp

Line 283 in 52334b0

bond_map_[node_name]->setHeartbeatPeriod(0.10);

Or just the lifecycle node?

[zz990099]

Thank you for your follow-up. To achieve the reduced CPU usage at idle, we found it necessary to modify both of the code locations you referenced.

[mini-1235]

Hi @zz990099, we discussed this today. In short, we agree that the bond_heartbeat_period in lifecycle manager could be parameterized in the same way lifecycle nodes are.

After that, could you provide metrics for 0.1, 0.2, 0.25, 0.5, and 1.0 so we can review the trade-offs and better inform the default choice? It's possible that the major performance gains occur at values much smaller than 1.0, which would suggest adjusting the default accordingly.

If it turns out that a value as large as 1.0 is required for certain use cases(e.g. embedding systems), we should document this explicitly.

[zz990099]

Absolutely, I'd be happy to run those tests. I'll collect the data for the specified periods and report back.

[zz990099]

Hi! I've completed the tests and recorded the CPU usage data. Please note that the data of nav2_container may have a variation of around ±5%. Submitting for your reference.

period	nav2_container	single_process
0.1s	140%	13%
0.2s	110%	10%
0.25s	100%	9%
0.5s	85%	8%
1.0s	80%	8%
2.0s	80%	8%

[SteveMacenski]

I think for your case that you want to optimize for performance, increasing it makes sense - though 0.5s seems to get you the bulk of what you want with a faster response time when things fail.

As for defaults, I think 0.25s makes sense on the performance trade-off if we see a benefit in performance on a typical x86 or more powerful ARM computer like a Jetson.

12th Gen Intel(R) Core(TM) i5-12600KF

What's the difference on this computer of 0.1 vs 0.25?

The main thing we need to analyze are the safety consequences. 10 hz results in a delay of 100ms when things go down to responding after the timeout period. Admittedly since that is set relatively large at 4s, moving to 250ms seems OK to me if there are consequential savings on all platforms. If only low-power embedded sees a difference and that may need even coarser, then I don't think its necessarily worth updating by default.

However, I think it would be nice in the configuration / tuning guide to have a page for tuning for low-power embedded systems. Things like this would be important notes to make.

[zz990099]

Thank you for your detailed response and guidance! Apologies for the delayed response.

My apologies for the lack of clarity. The issue was first observed on the RK3588, but we used the i5 for benchmarking when creating the issue. The data I provided earlier was from tests on the i5-12600KF CPU, using the latest code from the main branch built with the command:

colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release

Interestingly, on our RK3588 platform, where we deployed code from the humble-main branch, we observed that the nav2_container's idle CPU usage was significantly lower. Even with the heartbeat period set to 0.1s, starting only the core Nav2 nodes resulted in about 40% CPU usage for the container. When we increased the period to 1.0s, the CPU usage dropped to around 20%.

Our application involves many functional nodes, most based on nav2_util::LifecycleNode and managed by the LifecycleManager. We've observed a correlation: as the total number of LifecycleNode instances in the system increases, the idle CPU usage per node also increases. Disabling the bond heartbeat check returns CPU usage to normal. Given the large number of LifecycleNode in our system, increasing the heartbeat period became necessary to reduce the overall load.

Unfortunately, we haven't had the capacity to investigate the root cause of this phenomenon further, but we wanted to share this observation with you.

We plan to conduct further tests to better understand how the heartbeat period and the number of nodes collectively impact CPU usage across different platforms.

Thank you for your time and consideration.

[mini-1235]

I tested this today on my computer:

OS: Ubuntu 24.04.3 LTS (x86_64)
Kernel: 6.8.0-88-generic
CPU: Intel i7-12700H (20 cores)
GPU: Intel Iris Xe + NVIDIA RTX 3050 Ti Mobile
Memory: 32 GB

ros2 launch nav2_bringup tb4_simulation_launch.py

Period	nav2_container CPU Usage
0.1 s	4.4%
0.2 s	3.3%
0.25 s	3.3%
0.5 s	2.8%
1.0 s	2.6%

For me, it would probably make the most sense to set 0.2 or 0.25 s as the default