https://www.aerosota.com/blog/feed.xml

Can an LLM Create an Effective Motion Planner?

2025-01-28

TLDR; I prompted an LLM (Claude) to create an RRT-star implementation in Rust, optimized for initial solve time. I compared its performance with the RRT-star planner in the Open Motion Planning Library (OMPL). For my toy example, the LLM planner (after I fixed it!) was faster, found shorter paths, and had a higher success rate than the OMPL planner. Pure naivety or an opportunity? Maybe both!


RRT-star via Claude

Like the rest of the world, I am thinking about how large language models (LLMs) can improve my work. As a curiosity, I used Claude 3.5 Sonnet to generate a two-dimensional RRT-star planner using the following prompt:

You are an expert on motion planning and Rust. Please create a single-file implementation of a highly-optimized for time RRT* module that should support defining a starting point, a goal, and a circular obstacle in a 2D workspace. Do not use unsafe code for your optimization, but make any other optimizations you can, even if the code is obfuscated. The test should be a benchmark of 50 planner runs from the same start and goal, and report average time run, total successes, and average memory used.

I am no prompt engineer, but I almost got a working planner out of this. Claude struggled with moving resources, which I fixed manually. Additionally, the planner did not perform the crucial rewiring step of RRT-star, so I prompted the LLM to fix it. I finally changed the benchmark to run 500 instantiations of the planner, set the start and goal locations, and adjusted the release profile settings to increase execution speed (e.g. opt-level=3, lto=true).

I ran the bencmhark on my M2Pro Macbook and visualized the results: RRT-star Claude

The LLM planner had a 100% success rate, took an average of 2.33 milleseconds to compute each of the 500 paths, and each path had an average length of 17.43 units. The planner worked almost too well.

Comparison with OMPL

The Open Motion Planning Library (OMPL) is a go-to open-source planning toolbox with a few planners available, among those RRT-star. I decided to setup the same problem using the OMPL implementation of RRT* and run 500 times. After a quick brew install ompl and small CMake project with some optimizations (thanks again to Claude!), I was almost ready to go.

Setting up a direct comparison was a tricky for two reasons:

  1. The LLM planner terminates after the first valid path is found, or if a max number of iterations is reached. The OMPL planner terminates after a specified amount of time by default;
  2. The LLM planner uses a constant radius (3.0 units) around a point for rewiring. The OMPL planner uses either an adaptive k-nearest neighbors or search radius.

To overcome 1., I began by setting the maximum solve time of the OMPL planner to the mean time of the LLM planner (2.5ms). The OMPL planner had a 0% success rate. I bumped it up to 5ms and the success rate increased to 62.8%. Increasing the solve time to 10ms saw a 98.4% success rate (492/500 paths). I could have also setup a callback to terminate the planner when the first solution was found, but that is more involved than I wanted.

To somewhat overcome 2., I first used an adaptive radius rewiring factor of 0.1. This was my best attempt at using this method, as the documentation for it is wanting. After adjusting this value and not seeing any discernable effects in the success rate, I implemented the adaptive radius method that OMPL uses in calculate_rewire_radius, again with little effect.

Here are the results visualized using the OMPL RRT-star planner: RRT-star OMPL

The OMPL planner had a 98.4% success rate, with a slightly higher mean path length of 18.0. I was surprised it could not find a path every time especially given 10 milliseconds of planning time. I found a couple of related issues here and here, indicating the potential for improving this planner.

Or, this toy problem is too simple to warrant any serious discussion around the performance on OMPL RRT-star. The thinking continues.

Conclusion

I found some success in using an LLM to create an RRT-star planner in a very simple setting. Out of the black-box the planner did not compile or work, but half an hour of debugging resolved the issues.

Would I use the LLM planner for my robotics application? Not without more verification and analysis of its performance in more diverse settings. However, its performance relative to the OMPL implementation has me cautiously optimistic about the potential for improvements in OMPL.

I hope you enjoyed reading. Please get in touch if you have any comments or ideas. Here is the code I used in this experiment for the curious.

Other Notes

The LLM planner initially used the motion step size (0.05) as the distance threshold for determining if the goal was reached. I changed this to f64::EPSILON to be in line with std::numeric_limits<double>::epsilon(), although this has little effect since both planners have a 5% chance of sampling the goal state.

I mucked around with different RRT-star settings exposed by the OMPL interface, such as enabling pruning, delaying constraint checking, and trying different rewiring factors. None of it made a significant difference on the simple problem setup.