Tree traversal in Rust vs Borrow Checker - data-structures

I'm attempting to implement a tree structure in Rust, traverse it, and modify it, and I'm running into trouble with the borrow checker. My setup is more or less the following:
#![feature(slicing_syntax)]
use std::collections::HashMap;
#[deriving(PartialEq, Eq, Hash)]
struct Id {
id: int, // let’s pretend it’s that
}
struct Node {
children: HashMap<Id, Box<Node>>,
decoration: String,
// other fields
}
struct Tree {
root: Box<Node>
}
impl Tree {
/// Traverse the nodes along the specified path.
/// Return the node at which traversal stops either because the path is exhausted
/// or because there are no more nodes matching the path.
/// Also return any remaining steps in the path that did not have matching nodes.
fn traverse_path<'p>(&mut self, mut path: &'p [Id]) -> (&mut Box<Node>, &'p [Id]) {
let mut node = &mut self.root;
loop {
match node.children.get_mut(&path[0]) {
Some(child_node) => {
path = path[1..];
node = child_node;
},
None => {
break;
}
}
}
(node, path)
}
}
I have mutable references here because I want to be able to mutate the node returned by the method. For example, an add method would call traverse_path and then add nodes for the remainder of the path that did not have matching nodes.
This produces these errors:
s.rs:28:19: 28:32 error: cannot borrow `node.children` as mutable more than once at a time
s.rs:28 match node.children.get_mut(&path[0]) {
^~~~~~~~~~~~~
s.rs:28:19: 28:32 note: previous borrow of `node.children` occurs here; the mutable borrow prevents subsequent moves, borrows, or modification of `node.children` until the borrow ends
s.rs:28 match node.children.get_mut(&path[0]) {
^~~~~~~~~~~~~
s.rs:39:6: 39:6 note: previous borrow ends here
s.rs:25 fn traverse_path<'p>(&mut self, mut path: &'p [Id]) -> (&mut Box<Node>, &'p [Id]) {
...
s.rs:39 }
^
s.rs:31:21: 31:38 error: cannot assign to `node` because it is borrowed
s.rs:31 node = child_node;
^~~~~~~~~~~~~~~~~
s.rs:28:19: 28:32 note: borrow of `node` occurs here
s.rs:28 match node.children.get_mut(&path[0]) {
^~~~~~~~~~~~~
s.rs:38:10: 38:14 error: cannot borrow `*node` as mutable more than once at a time
s.rs:38 (node, path)
^~~~
s.rs:28:19: 28:32 note: previous borrow of `node.children` occurs here; the mutable borrow prevents subsequent moves, borrows, or modification of `node.children` until the borrow ends
s.rs:28 match node.children.get_mut(&path[0]) {
^~~~~~~~~~~~~
s.rs:39:6: 39:6 note: previous borrow ends here
s.rs:25 fn traverse_path<'p>(&mut self, mut path: &'p [Id]) -> (&mut Box<Node>, &'p [Id]) {
...
s.rs:39 }
^
error: aborting due to 3 previous errors
I understand why the borrow checker doesn't like this code, but I don't know how to make this work.
I also attempted an alternate implementation using an iterator using code like the following:
struct PathIter<'a> {
path: &'a [Id],
node: &'a mut Box<Node>
}
impl<'a> Iterator<Box<Node>> for PathIter<'a> {
fn next(&mut self) -> Option<Box<Node>> {
let child = self.node.get_child(&self.path[0]);
if child.is_some() {
self.path = self.path[1..];
self.node = child.unwrap();
}
child
}
}
The errors here ended up being lifetime-related:
src/http_prefix_tree.rs:147:27: 147:53 error: cannot infer an appropriate lifetime for autoref due to conflicting requirements
src/http_prefix_tree.rs:147 let child = self.node.get_child(&self.path[0]);
^~~~~~~~~~~~~~~~~~~~~~~~~~
src/http_prefix_tree.rs:146:3: 153:4 help: consider using an explicit lifetime parameter as shown: fn next(&'a mut self) -> Option<Box<Node>>
src/http_prefix_tree.rs:146 fn next(&mut self) -> Option<Box<Node>> {
src/http_prefix_tree.rs:147 let child = self.node.get_child(&self.path[0]);
src/http_prefix_tree.rs:148 if child.is_some() {
src/http_prefix_tree.rs:149 self.path = self.path[1..];
src/http_prefix_tree.rs:150 self.node = child.unwrap();
src/http_prefix_tree.rs:151 }
Another thing I'm interested in is to collect the values of the decoration field for matching nodes and display these values if the path was fully exhausted. My very first thought was to have backlinks from the nodes to their parents, but the only example of this I found was Rawlink in DList, which scared me off. My next hope is that the iterator implementation (if I can get it to work) would lend itself naturally to something like that. Is that the right track to pursue?

Here's a variant of your first approach, using recursion to avoid borrowing conflicts. The iterative equivalent fails to compile because Rust is too strict when dealing with mutable borrowed pointers to mutable values.
impl Node {
fn traverse_path<'p>(&mut self, mut path: &'p [Id]) -> (&mut Node, &'p [Id]) { // '
if self.children.contains_key(&path[0]) {
self.children[path[0]].traverse_path(path[1..])
} else {
(self, path)
}
}
}
impl Tree {
/// Traverse the nodes along the specified path.
/// Return the node at which traversal stops either because the path is exhausted
/// or because there are no more nodes matching the path.
/// Also return any remaining steps in the path that did not have matching nodes.
fn traverse_path<'p>(&mut self, mut path: &'p [Id]) -> (&mut Node, &'p [Id]) { // '
self.root.traverse_path(path)
}
}
Note that I've changed the return type from &mut Box<Node> to &mut Node; you don't need to reveal to your users that you're using a Box in your implementation. Also, see how Node::traverse_path first checks if there's a value in the map using contains_key(), then retrieving the value using indexing. This means that the value is looked up twice, but that's the only way I've found to make this work without requiring unsafe code.
P.S.: You can change the root in Tree to be a Node, rather than a Box<Node>.

Related

How to move a String out of a for-loop embedded within a Closure? Rust

New to Rust.
I am attempting to solve: https://leetcode.com/problems/longest-common-prefix/
My own solution is:
impl Solution {
pub fn longest_common_prefix(strs: Vec<String>) -> String {
let first = &strs[0];
let result = strs.iter() // [&string1, &string2, ..., &stringn] for &string in strs
.map(|&string| {
for (i, c) in first.chars().enumerate() {
println!("c is {}", &c);
let mut partial_res = String::new();
if c == string.chars().nth(i).unwrap() {
println!("c is {}", &c);
partial_res.push(c);
}
}
partial_res
}
)
.min_by_key(|string| string.len()).unwrap();
result
}
}
The idea is that for each string in strs, we first iter() them, and map a closure to each of the string.
The closure takes in a &string and compare all the characters of &string and first (which is the first element / string of strs).
Finally, to search for the shortest string in result Iterator and returns the shortest string.
I encounter this error:
Line 16, Char 29: cannot find value `partial_res` in this scope (solution.rs)
|
16 | ... partial_res
| ^^^^^^^^^^^ not found in this scope
For more information about this error, try `rustc --explain E0425`.
error: could not compile `prog` due to previous error
mv: cannot stat '/leetcode/rust_compile/target/release/prog': No such file or directory
Therefore, my question is: > How can I move a String out of a for-loop embedded within a Closure?
Note:
I understand the approach to solving the problem is not ideal, please neglect the algorithmic aspect of the approach here (O(N^2) instead of O(N) algorithm).

How to optimize brainf*ck instructions

I'm trying to write an optimisation feature for my brainf*ck interpreter.
It basically combines same instructions into 1 instruction.
I wrote this function but It doesn't work properly:
pub fn optimize_multiple(instructions: &Vec<Instruction>) -> Vec<OptimizedInstruction> {
let mut opt: Vec<OptimizedInstruction> = Vec::new();
let mut last_instruction = instructions.get(0).unwrap();
let mut last_count = 0;
for instruction in instructions.iter() {
if instruction == last_instruction {
last_count += 1;
}
else if let Instruction::Loop(i) = instruction {
opt.push(OptimizedInstruction::Loop(optimize_multiple(i)));
last_count = 1;
}
else {
opt.push(OptimizedInstruction::new(last_instruction.clone(), last_count));
last_instruction = instruction;
last_count = 1;
}
}
opt
}
Here's the OptimizedInstruction enum and the "new" method:
(The Instruction::Loop hand is just a place holder, I didn't used it)
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum OptimizedInstruction {
IncrementPointer(usize),
DecrementPointer(usize),
Increment(usize),
Decrement(usize),
Write,
Read,
Loop(Vec<OptimizedInstruction>),
}
impl OptimizedInstruction {
pub fn new(instruction: Instruction, count: usize) -> OptimizedInstruction {
match instruction {
Instruction::IncrementPointer => OptimizedInstruction::IncrementPointer(count),
Instruction::DecrementPointer => OptimizedInstruction::DecrementPointer(count),
Instruction::Increment => OptimizedInstruction::Increment(count),
Instruction::Decrement => OptimizedInstruction::Decrement(count),
Instruction::Write => OptimizedInstruction::Write,
Instruction::Read => OptimizedInstruction::Read,
Instruction::Loop(_i) => OptimizedInstruction::Loop(Vec::new()),
}
}
}
I ran it with this input:
++-[+-++]
But it gave me this output:
[Increment(2), Loop([Increment(1), Decrement(1)])]
Insted of this:
[Increment(2), Decrement(1), Loop([Increment(1), Decrement(1), Increment(2)])]
I've been trying to solve it for 2 days and still, I don't have any idea about why it doesn't work.
~ Derin
First off, just to rain a little on your parade I just want to point out that Wilfred already made a brainf*ck compiler in Rust that can compile to a native binary through LLVM (bfc). If you are getting stuck, you may want to check his implementation to see how he does it. If you ignore the LLVM part, it isn't too difficult to read through and he has a good approach.
When broken down into its core components, this problem revolves around merging two elements together. The most elegant way I have come across to solve this is use an iterator with a merge function. I wrote an example of what I imagine that would look like below. I shortened some of the variable names since they were a bit long but the general idea is the same. The merge function has a very simple job. When given two elements, attempt to merge them into a single new element. The iterator then handles putting them through that function and returning items once they can no longer be merged. A sort of optional fold if you will.
pub struct MergeIter<I: Iterator, F> {
iter: I,
func: Box<F>,
held: Option<<I as Iterator>::Item>,
}
impl<I, F> Iterator for MergeIter<I, F>
where
I: Iterator,
F: FnMut(&<I as Iterator>::Item, &<I as Iterator>::Item) -> Option<<I as Iterator>::Item>,
{
type Item = <I as Iterator>::Item;
fn next(&mut self) -> Option<Self::Item> {
let mut first = match self.held.take() {
Some(v) => v,
None => self.iter.next()?,
};
loop {
let second = match self.iter.next() {
Some(v) => v,
None => return Some(first),
};
match (self.func)(&first, &second) {
// If merge succeeds, attempt to merge again
Some(v) => first = v,
// If merge fails, store second value for next iteration and return result
None => {
self.held = Some(second);
return Some(first);
}
}
}
}
}
pub trait ToMergeIter: Iterator {
fn merge<F>(self, func: F) -> MergeIter<Self, F>
where
Self: Sized,
F: FnMut(&Self::Item, &Self::Item) -> Option<Self::Item>;
}
impl<I: Sized + Iterator> ToMergeIter for I {
fn merge<F>(self, func: F) -> MergeIter<Self, F>
where
Self: Sized,
F: FnMut(&Self::Item, &Self::Item) -> Option<Self::Item>,
{
MergeIter {
iter: self,
func: Box::new(func),
held: None,
}
}
}
Then we can apply this process recursively to get our result. Here is a brief example of what that would look like. It isn't quite as memory efficient since it makes a new Vec each time, but it makes the process of specifying what instructions to merge way easier for you and helps make your work easier to read/debug.
pub fn optimize(instructions: Vec<Instruction>) -> Vec<Instruction> {
instructions
.into_iter()
// Recursively call function on loops
.map(|instruction| match instruction {
Instruction::Loop(x) => Instruction::Loop(optimize(x)),
x => x,
})
// Merge elements using the merge iter
.merge(|a, b| {
// State if any two given elements should be merged together or not.
match (a, b) {
(Instruction::IncPtr(x), Instruction::IncPtr(y)) => {
Some(Instruction::IncPtr(x + y))
}
(Instruction::DecPtr(x), Instruction::DecPtr(y)) => {
Some(Instruction::DecPtr(x + y))
}
(Instruction::Increment(x), Instruction::Increment(y)) => {
Some(Instruction::Increment(x + y))
}
(Instruction::Decrement(x), Instruction::Decrement(y)) => {
Some(Instruction::Decrement(x + y))
}
// Etc...
_ => None,
}
})
// Collect results to return
.collect()
}
playground link

Rust - adding event listeners to a webassembly game

I'm attempting to create a game in web assembly. I chose to prepare it in rust and compile it using cargo-web. I managed to get a working game loop, but I have a problem with adding MouseDownEvent listener due to rust borrowing mechanisms. I would very much prefer to write "safe" code (without using "unsafe" keyword)
At this moment the game simply moves a red box from (0,0) to (700,500) with speed depending on the distance. I would like to have the next step to use user click update the destination.
This is the simplified and working code of the game.
static/index.html
<!DOCTYPE html>
<html lang="en">
<head>
<title>The Game!</title>
</head>
<body>
<canvas id="canvas" width="600" height="600">
<script src="game.js"></script>
</body>
</html>
src/main.rs
mod game;
use game::Game;
use stdweb::console;
use stdweb::traits::*;
use stdweb::unstable::TryInto;
use stdweb::web::document;
use stdweb::web::CanvasRenderingContext2d;
use stdweb::web::html_element::CanvasElement;
use stdweb::web::event::MouseDownEvent;
fn main()
{
let canvas: CanvasElement = document()
.query_selector("#canvas")
.unwrap()
.unwrap()
.try_into()
.unwrap();
canvas.set_width(800u32);
canvas.set_height(600u32);
let context = canvas.get_context().unwrap();
let game: Game = Game::new();
// canvas.add_event_listener(|event: MouseDownEvent|
// {
// game.destination.x = (event.client_x() as f64);
// game.destination.y = (event.client_y() as f64);
// });
game_loop(game, context, 0f64);
}
fn game_loop(mut game : Game, context : CanvasRenderingContext2d, timestamp : f64)
{
game.cycle(timestamp);
draw(&game,&context);
stdweb::web::window().request_animation_frame( |time : f64| { game_loop(game, context, time); } );
}
fn draw(game : &Game, context: &CanvasRenderingContext2d)
{
context.clear_rect(0f64,0f64,800f64,800f64);
context.set_fill_style_color("red");
context.fill_rect(game.location.x, game.location.y, 5f64, 5f64);
}
src/game.rs
pub struct Point
{
pub x: f64,
pub y: f64,
}
pub struct Game
{
pub time: f64,
pub location: Point,
pub destination: Point,
}
impl Game
{
pub fn new() -> Game
{
let game = Game
{
time: 0f64,
location: Point{x: 0f64, y: 0f64},
destination: Point{x: 700f64, y: 500f64},
};
return game;
}
pub fn cycle(&mut self, timestamp : f64)
{
if timestamp - self.time > 10f64
{
self.location.x += (self.destination.x - self.location.x) / 10f64;
self.location.y += (self.destination.y - self.location.y) / 10f64;
self.time = timestamp;
}
}
}
The commented out part of main.rs is my attempt of adding a MouseDownEvent listener. Unfortunately it generates a compilation error:
error[E0505]: cannot move out of `game` because it is borrowed
--> src\main.rs:37:15
|
31 | canvas.add_event_listener(|event: MouseDownEvent|
| - ----------------------- borrow of `game` occurs here
| _____|
| |
32 | | {
33 | | game.destination.x = (event.client_x() as f64);
| | ---- borrow occurs due to use in closure
34 | | game.destination.y = (event.client_y() as f64);
35 | | });
| |______- argument requires that `game` is borrowed for `'static`
36 |
37 | game_loop(game, context, 0f64);
| ^^^^ move out of `game` occurs here
I would very much like to know how to properly implement a way of reading user input into a game. It doesn't need to be asynchronous.
I think that the compiler error message is pretty clear in this case. You're trying to borrow the game in the closure for the 'static lifetime and then you're also trying to move the game. It isn't allowed. I'd recommend to read The Rust Programming Language book again. Focus on chapter 4 - Understanding Ownership.
To make it shorter, your question boils down to something like - how to share a state, which can be mutated. There're plenty of ways how to achieve this goal, but it really depends on your needs (single or multi thread, etc.). I'm going to use Rc & RefCell for this problem.
Rc (std::rc):
The type Rc<T> provides shared ownership of a value of type T, allocated in the heap. Invoking clone on Rc produces a new pointer to the same value in the heap. When the last Rc pointer to a given value is destroyed, the pointed-to value is also destroyed.
RefCell (std::cell):
Values of the Cell<T> and RefCell<T> types may be mutated through shared references (i.e. the common &T type), whereas most Rust types can only be mutated through unique (&mut T) references. We say that Cell<T> and RefCell<T> provide 'interior mutability', in contrast with typical Rust types that exhibit 'inherited mutability'.
Here's what I did to your structures:
struct Inner {
time: f64,
location: Point,
destination: Point,
}
#[derive(Clone)]
pub struct Game {
inner: Rc<RefCell<Inner>>,
}
What does this mean? Inner holds the game state (same fields as the old Game). New Game has just one field inner, which contains the shared state.
Rc<T> (T is RefCell<Inner> in this case) - allows me to clone inner multiple times, but it won't clone the T
RefCell<T> (T is Inner in this case) - allows me to borrow T immutably or mutably, checking is done in the runtime
I can clone the Game structure multiple times now and it won't clone the RefCell<Inner>, just the Game & Rc. Which is what the enclose! macro is doing in the updated main.rs:
let game: Game = Game::default();
canvas.add_event_listener(enclose!( (game) move |event: MouseDownEvent| {
game.set_destination(event);
}));
game_loop(game, context, 0.);
Without the enclose! macro:
let game: Game = Game::default();
// game_for_mouse_down_event_closure holds the reference to the
// same `RefCell<Inner>` as the initial `game`
let game_for_mouse_down_event_closure = game.clone();
canvas.add_event_listener(move |event: MouseDownEvent| {
game_for_mouse_down_event_closure.set_destination(event);
});
game_loop(game, context, 0.);
Updated game.rs:
use std::{cell::RefCell, rc::Rc};
use stdweb::traits::IMouseEvent;
use stdweb::web::event::MouseDownEvent;
#[derive(Clone, Copy)]
pub struct Point {
pub x: f64,
pub y: f64,
}
impl From<MouseDownEvent> for Point {
fn from(e: MouseDownEvent) -> Self {
Self {
x: e.client_x() as f64,
y: e.client_y() as f64,
}
}
}
struct Inner {
time: f64,
location: Point,
destination: Point,
}
impl Default for Inner {
fn default() -> Self {
Inner {
time: 0.,
location: Point { x: 0., y: 0. },
destination: Point { x: 700., y: 500. },
}
}
}
#[derive(Clone)]
pub struct Game {
inner: Rc<RefCell<Inner>>,
}
impl Default for Game {
fn default() -> Self {
Game {
inner: Rc::new(RefCell::new(Inner::default())),
}
}
}
impl Game {
pub fn update(&self, timestamp: f64) {
let mut inner = self.inner.borrow_mut();
if timestamp - inner.time > 10f64 {
inner.location.x += (inner.destination.x - inner.location.x) / 10f64;
inner.location.y += (inner.destination.y - inner.location.y) / 10f64;
inner.time = timestamp;
}
}
pub fn set_destination<T: Into<Point>>(&self, location: T) {
let mut inner = self.inner.borrow_mut();
inner.destination = location.into();
}
pub fn location(&self) -> Point {
self.inner.borrow().location
}
}
Updated main.rs:
use stdweb::traits::*;
use stdweb::unstable::TryInto;
use stdweb::web::document;
use stdweb::web::event::MouseDownEvent;
use stdweb::web::html_element::CanvasElement;
use stdweb::web::CanvasRenderingContext2d;
use game::Game;
mod game;
// https://github.com/koute/stdweb/blob/master/examples/todomvc/src/main.rs#L31-L39
macro_rules! enclose {
( ($( $x:ident ),*) $y:expr ) => {
{
$(let $x = $x.clone();)*
$y
}
};
}
fn game_loop(game: Game, context: CanvasRenderingContext2d, timestamp: f64) {
game.update(timestamp);
draw(&game, &context);
stdweb::web::window().request_animation_frame(|time: f64| {
game_loop(game, context, time);
});
}
fn draw(game: &Game, context: &CanvasRenderingContext2d) {
context.clear_rect(0., 0., 800., 800.);
context.set_fill_style_color("red");
let location = game.location();
context.fill_rect(location.x, location.y, 5., 5.);
}
fn main() {
let canvas: CanvasElement = document()
.query_selector("#canvas")
.unwrap()
.unwrap()
.try_into()
.unwrap();
canvas.set_width(800);
canvas.set_height(600);
let context = canvas.get_context().unwrap();
let game: Game = Game::default();
canvas.add_event_listener(enclose!( (game) move |event: MouseDownEvent| {
game.set_destination(event);
}));
game_loop(game, context, 0.);
}
P.S. Please, before sharing any code in the future, install and use the rustfmt.
In your example game_loop owns game, as it is moved into the loop. So anything that should change game needs to happen inside game_loop. To fit event handling into this, you have multiple options:
Option 1
Let the game_loop poll for events.
You create a queue of events and your game_loop will have some logic to get the first event and handle it.
You will have to deal with synchronization here, so I suggest that you read up on Mutex and Concurrency in general. But it should be a fairly easy task once you get the hang of it. Your loop gets one reference and each event handler gets one, all try to unlock the mutex and then access the queue (vector probably).
This will make your game_loop the monolithic one truth of them all, which is a popular engine design because it is easy to reason about and start with.
But maybe you want to be less centralized.
Option 2
Let events happen outside the loop
This idea would be a bigger refactor. You would put your Game in a lazy_static with a Mutex around it.
Every invocation of the game_loop it will try to get the lock on said Mutex and then perform game calculations.
When an input event happens, that event also tries to get the Mutex on the Game. This means while the game_loop is processing, no input events are handled, but they will try to get in between ticks.
A challenge here would be to preserve input order and to make sure that inputs are processed quick enough. This might be a bigger challenge to get completely right. But the design will give you some possibilities.
A fleshed out version of this idea is Amethyst, which is massively parallel and makes for a clean design. But they employ a quite more complex design behind their engine.

Safely return multiple references to internal nodes, while still allowing mutation of other nodes

Suppose, for example, I have a linked list which does not allow removal of nodes.
Would it be possible to return shared references to values which have already been inserted, while still allowing the relative order of the nodes to be changed, or new nodes inserted?
Even mutation through one of the nodes should be safe "on paper" as long as only one node is used to mutate the list at a time. Is it possible to represent this in rust's ownership system?
I'm specifically interested in doing so without runtime overhead (potentially using unsafe in the implementation, but not in the interface).
EDIT: As requested, here is an example that gives the outline of what I am thinking of.
let list = MyLinkedList::<i32>::new()
let handle1 = list.insert(1); // Returns a handle to the inserted element.
let handle2 = list.insert(2);
let value1 : &i32 = handle1.get();
let value2 : &i32 = handle2.prev().get(); // Ok to have two immutable references to the same element.
list.insert(3); // Also ok to insert and swap nodes, while the references are held.
list.swap(handle1,handl2);
foo(value1,value2);
let exclusive_value: &mut i32 = handle1.get_mut(); // While this reference is held, no other handles can be used, but insertion and permutation are ok
handle5 = list.insert(4);
list.swap(handle1, handle2);
In other words, the data contained inside the nodes of the list is treated as one resource that can be borrowed shared/mutably, and the links between the nodes are another resource that can be borrowed shared/mutably.
In other words, the data contained inside the nodes of the list is treated as one resource that can be borrowed shared/mutably, and the links between the nodes are another resource that can be borrowed shared/mutably.
The idea to deal with such spatial partitioning is to introduce a different "key" for each partition; it's easy since they are static. This has been dubbed the PassKey pattern.
In the absence of brands, it will still require a run-time check: verifying that the elements-key is tied to this specific list instance is mandatory for safety. This is, however, a read-only comparison that will always be true, so the performance is about as good as it gets as far as run-time checks go.
The idea, in a nutshell:
let (handles, elements) = list.keys();
let h0 = handles.create(4);
handles.swap(h0, h1);
let e = elements.get(h0);
In your usecase:
It is always possible to change the links, so we will use internal mutability for this.
The borrow-check on elements inside handles will be carried out by borrowing elements.
The full implementation can be found here. It heavily uses unsafe, and I make no promise that it is fully safe, however it is hopefully sufficient for a demonstration.
In this implementation, I have opted for dumb handles and implemented the operations on the key types themselves. This limited the number of types who needed to borrow from the main list, and simplified borrowing.
The core idea, then:
struct LinkedList<T> {
head: *mut Node<T>,
tail: *mut Node<T>
}
struct Handles<'a, T> {
list: ptr::NonNull<LinkedList<T>>,
_marker: PhantomData<&'a mut LinkedList<T>>,
}
struct Elements<'a, T> {
list: ptr::NonNull<LinkedList<T>>,
_marker: PhantomData<&'a mut LinkedList<T>>,
}
LinkedList<T> will act as the storage, however will implement only 3 operations:
construction,
destruction,
handing out the keys.
The two keys Handles and Elements will both borrow the list mutably, guaranteeing that a single of (each of them) can exist simultaneously. Borrow-checking will prevent a new Handles or Elements from being created if any instance of them still lives for this list:
list: grants access to the list storage; Elements will only use it for checking (necessary) run-time invariants and never dereference it.
_marker: is the key to the borrow-checking actually guaranteeing exclusitivity.
Sounds cool so far? For completion, the last two structures then:
struct Handle<'a, T> {
node: ptr::NonNull<Node<T>>,
list: ptr::NonNull<LinkedList<T>>,
_marker: PhantomData<&'a LinkedList<T>>,
}
struct Node<T> {
data: T,
prev: *mut Node<T>,
next: *mut Node<T>,
}
Node is the most obvious representation of a doubly-linked list ever, so we're doing something right. The list in Handle<T> is there for the exact same purpose as the one in Elements: verifying that both Handle and Handles/Elements are talking about the same instance of list. It's critical for get_mut to be safe, and otherwise helps avoiding bugs.
There's a subtle reason for Handle<'a, T> having a lifetime tying to the LinkedList. I was tempted to remove it, however this would allow creating a handle from a list, destroying the list, then recreating a list at the same address... and handle.node would now be dangling!
And with, we only need to implement the methods we need on Handles and Elements. A few samples:
impl<'a, T> Handles<'a, T> {
pub fn push_front(&self, data: T) -> Handle<'a, T> {
let list = unsafe { &mut *self.list.as_ptr() };
let node = Box::into_raw(Box::new(Node { data, prev: ptr::null_mut(), next: list.head }));
unsafe { &mut *node }.set_neighbours();
list.head = node;
if list.tail.is_null() {
list.tail = node;
}
Handle {
node: unsafe { ptr::NonNull::new_unchecked(node) },
list: self.list, _marker: PhantomData,
}
}
pub fn prev(&self, handle: Handle<'a, T>) -> Option<Handle<'a, T>> {
unsafe { handle.node.as_ref() }.prev().map(|node| Handle {
node,
list: self.list,
_marker: PhantomData
})
}
}
And:
impl<'a, T> Elements<'a, T> {
pub fn get<'b>(&'b self, handle: Handle<'a, T>) -> &'b T {
assert_eq!(self.list, handle.list);
let node = unsafe { &*handle.node.as_ptr() };
&node.data
}
pub fn get_mut<'b>(&'b mut self, handle: Handle<'a, T>) -> &'b mut T {
assert_eq!(self.list, handle.list);
let node = unsafe { &mut *handle.node.as_ptr() };
&mut node.data
}
}
And this should be safe because:
Handles, after creating a new handle, only ever accesses its links.
Elements only ever returns references to data, and the links cannot be modified while it accesses them.
Example of usage:
fn main() {
let mut linked_list = LinkedList::default();
{
let (handles, mut elements) = linked_list.access();
let h0 = handles.push_front("Hello".to_string());
assert!(handles.prev(h0).is_none());
assert!(handles.next(h0).is_none());
println!("{}", elements.get(h0));
let h1 = {
let first = elements.get_mut(h0);
first.replace_range(.., "Hallo");
let h1 = handles.push_front("World".to_string());
assert!(handles.prev(h0).is_some());
first.replace_range(.., "Goodbye");
h1
};
println!("{} {}", elements.get(h0), elements.get(h1));
handles.swap(h0, h1);
println!("{} {}", elements.get(h0), elements.get(h1));
}
{
let (handles, elements) = linked_list.access();
let h0 = handles.front().unwrap();
let h1 = handles.back().unwrap();
let h2 = handles.push_back("And thanks for the fish!".to_string());
println!("{} {}! {}", elements.get(h0), elements.get(h1), elements.get(h2));
}
}

Hand-over-hand locking with Rust

I'm trying to write an implementation of union-find in Rust. This is famously very simple to implement in languages like C, while still having a complex run time analysis.
I'm having trouble getting Rust's mutex semantics to allow iterative hand-over-hand locking.
Here's how I got where I am now.
First, this is a very simple implementation of part of the structure I want in C:
#include <stdlib.h>
struct node {
struct node * parent;
};
struct node * create(struct node * parent) {
struct node * ans = malloc(sizeof(struct node));
ans->parent = parent;
return ans;
}
struct node * find_root(struct node * x) {
while (x->parent) {
x = x->parent;
}
return x;
}
int main() {
struct node * foo = create(NULL);
struct node * bar = create(foo);
struct node * baz = create(bar);
baz->parent = find_root(bar);
}
Note that the structure of the pointers is that of an inverted tree; multiple pointers may point at a single location, and there are no cycles.
At this point, there is no path compression.
Here is a Rust translation. I chose to use Rust's reference-counted pointer type to support the inverted tree type I referenced above.
Note that this implementation is much more verbose, possibly due to the increased safety that Rust offers, but possibly due to my inexperience with Rust.
use std::rc::Rc;
struct Node {
parent: Option<Rc<Node>>
}
fn create(parent: Option<Rc<Node>>) -> Node {
Node {parent: parent.clone()}
}
fn find_root(x: Rc<Node>) -> Rc<Node> {
let mut ans = x.clone();
while ans.parent.is_some() {
ans = ans.parent.clone().unwrap();
}
ans
}
fn main() {
let foo = Rc::new(create(None));
let bar = Rc::new(create(Some(foo.clone())));
let mut prebaz = create(Some(bar.clone()));
prebaz.parent = Some(find_root(bar.clone()));
}
Path compression re-parents each node along a path to the root every time find_root is called. To add this feature to the C code, only two new small functions are needed:
void change_root(struct node * x, struct node * root) {
while (x) {
struct node * tmp = x->parent;
x->parent = root;
x = tmp;
}
}
struct node * root(struct node * x) {
struct node * ans = find_root(x);
change_root(x, ans);
return ans;
}
The function change_root does all the re-parenting, while the function root is just a wrapper to use the results of find_root to re-parent the nodes on the path to the root.
In order to do this in Rust, I decided I would have to use a Mutex rather than just a reference counted pointer, since the Rc interface only allows mutable access by copy-on-write when more than one pointer to the item is live. As a result, all of the code would have to change. Before even getting to the path compression part, I got hung up on find_root:
use std::sync::{Mutex,Arc};
struct Node {
parent: Option<Arc<Mutex<Node>>>
}
fn create(parent: Option<Arc<Mutex<Node>>>) -> Node {
Node {parent: parent.clone()}
}
fn find_root(x: Arc<Mutex<Node>>) -> Arc<Mutex<Node>> {
let mut ans = x.clone();
let mut inner = ans.lock();
while inner.parent.is_some() {
ans = inner.parent.clone().unwrap();
inner = ans.lock();
}
ans.clone()
}
This produces the error (with 0.12.0)
error: cannot assign to `ans` because it is borrowed
ans = inner.parent.clone().unwrap();
note: borrow of `ans` occurs here
let mut inner = ans.lock();
What I think I need here is hand-over-hand locking. For the path A -> B -> C -> ..., I need to lock A, lock B, unlock A, lock C, unlock B, ... Of course, I could keep all of the locks open: lock A, lock B, lock C, ... unlock C, unlock B, unlock A, but this seems inefficient.
However, Mutex does not offer unlock, and uses RAII instead. How can I achieve hand-over-hand locking in Rust without being able to directly call unlock?
EDIT: As the comments noted, I could use Rc<RefCell<Node>> rather than Arc<Mutex<Node>>. Doing so leads to the same compiler error.
For clarity about what I'm trying to avoid by using hand-over-hand locking, here is a RefCell version that compiles but used space linear in the length of the path.
fn find_root(x: Rc<RefCell<Node>>) -> Rc<RefCell<Node>> {
let mut inner : RefMut<Node> = x.borrow_mut();
if inner.parent.is_some() {
find_root(inner.parent.clone().unwrap())
} else {
x.clone()
}
}
We can pretty easily do full hand-over-hand locking as we traverse this list using just a bit of unsafe, which is necessary to tell the borrow checker a small bit of insight that we are aware of, but that it can't know.
But first, let's clearly formulate the problem:
We want to traverse a linked list whose nodes are stored as Arc<Mutex<Node>> to get the last node in the list
We need to lock each node in the list as we go along the way such that another concurrent traversal has to follow strictly behind us and cannot muck with our progress.
Before we get into the nitty-gritty details, let's try to write the signature for this function:
fn find_root(node: Arc<Mutex<Node>>) -> Arc<Mutex<Node>>;
Now that we know our goal, we can start to get into the implementation - here's a first attempt:
fn find_root(incoming: Arc<Mutex<Node>>) -> Arc<Mutex<Node>> {
// We have to separate this from incoming since the lock must
// be borrowed from incoming, not this local node.
let mut node = incoming.clone();
let mut lock = incoming.lock();
// Could use while let but that leads to borrowing issues.
while lock.parent.is_some() {
node = lock.parent.as_ref().unwrap().clone(); // !! uh-oh !!
lock = node.lock();
}
node
}
If we try to compile this, rustc will error on the line marked !! uh-oh !!, telling us that we can't move out of node while lock still exists, since lock is borrowing node. This is not a spurious error! The data in lock might go away as soon as node does - it's only because we know that we can keep the data lock is pointing to valid and in the same memory location even if we move node that we can fix this.
The key insight here is that the lifetime of data contained within an Arc is dynamic, and it is hard for the borrow checker to make the inferences we can about exactly how long data inside an Arc is valid.
This happens every once in a while when writing rust; you have more knowledge about the lifetime and organization of your data than rustc, and you want to be able to express that knowledge to the compiler, effectively saying "trust me". Enter: unsafe - our way of telling the compiler that we know more than it, and it should allow us to inform it of the guarantees that we know but it doesn't.
In this case, the guarantee is pretty simple - we are going to replace node while lock still exists, but we are not going to ensure that the data inside lock continues to be valid even though node goes away. To express this guarantee we can use mem::transmute, a function which allows us to reinterpret the type of any variable, by just using it to change the lifetime of the lock returned by node to be slightly longer than it actually is.
To make sure we keep our promise, we are going to use another handoff variable to hold node while we reassign lock - even though this moves node (changing its address) and the borrow checker will be angry at us, we know it's ok since lock doesn't point at node, it points at data inside of node, whose address (in this case, since it's behind an Arc) will not change.
Before we get to the solution, it's important to note that the trick we are using here is only valid because we are using an Arc. The borrow checker is warning us of a possibly serious error - if the Mutex was held inline and not in an Arc, this error would be a correct prevention of a use-after-free, where the MutexGuard held in lock would attempt to unlock a Mutex which has already been dropped, or at least moved to another memory location.
use std::mem;
use std::sync::{Arc, Mutex};
fn find_root(incoming: Arc<Mutex<Node>>) -> Arc<Mutex<Node>> {
let mut node = incoming.clone();
let mut handoff_node;
let mut lock = incoming.lock().unwrap();
// Could use while let but that leads to borrowing issues.
while lock.parent.is_some() {
// Keep the data in node around by holding on to this `Arc`.
handoff_node = node;
node = lock.parent.as_ref().unwrap().clone();
// We are going to move out of node while this lock is still around,
// but since we kept the data around it's ok.
lock = unsafe { mem::transmute(node.lock().unwrap()) };
}
node
}
And, just like that, rustc is happy, and we have hand-over-hand locking, since the last lock is released only after we have acquired the new lock!
There is one unanswered question in this implementation which I have not yet received an answer too, which is whether the drop of the old value and assignment of a new value to a variable is a guaranteed to be atomic - if not, there is a race condition where the old lock is released before the new lock is acquired in the assignment of lock. It's pretty trivial to work around this by just having another holdover_lock variable and moving the old lock into it before reassigning, then dropping it after reassigning lock.
Hopefully this fully addresses your question and shows how unsafe can be used to work around "deficiencies" in the borrow checker when you really do know more. I would still like to want that the cases where you know more than the borrow checker are rare, and transmuting lifetimes is not "usual" behavior.
Using Mutex in this way, as you can see, is pretty complex and you have to deal with many, many, possible sources of a race condition and I may not even have caught all of them! Unless you really need this structure to be accessible from many threads, it would probably be best to just use Rc and RefCell, if you need it, as this makes things much easier.
I believe this to fit the criteria of hand-over-hand locking.
use std::sync::Mutex;
fn main() {
// Create a set of mutexes to lock hand-over-hand
let mutexes = Vec::from_fn(4, |_| Mutex::new(false));
// Lock the first one
let val_0 = mutexes[0].lock();
if !*val_0 {
// Lock the second one
let mut val_1 = mutexes[1].lock();
// Unlock the first one
drop(val_0);
// Do logic
*val_1 = true;
}
for mutex in mutexes.iter() {
println!("{}" , *mutex.lock());
}
}
Edit #1
Does it work when access to lock n+1 is guarded by lock n?
If you mean something that could be shaped like the following, then I think the answer is no.
struct Level {
data: bool,
child: Option<Mutex<Box<Level>>>,
}
However, it is sensible that this should not work. When you wrap an object in a mutex, then you are saying "The entire object is safe". You can't say both "the entire pie is safe" and "I'm eating the stuff below the crust" at the same time. Perhaps you jettison the safety by creating a Mutex<()> and lock that?
This is still not the answer your literal question of to how to do hand-over-hand locking, which should only be important in a concurrent setting (or if someone else forced you to use Mutex references to nodes). It is instead how to do this with Rc and RefCell, which you seem to be interested in.
RefCell only allows mutable writes when one mutable reference is held. Importantly, the Rc<RefCell<Node>> objects are not mutable references. The mutable references it is talking about are the results from calling borrow_mut() on the Rc<RefCell<Node>>object, and as long as you do that in a limited scope (e.g. the body of the while loop), you'll be fine.
The important thing happening in path compression is that the next Rc object will keep the rest of the chain alive while you swing the parent pointer for node to point at root. However, it is not a reference in the Rust sense of the word.
struct Node
{
parent: Option<Rc<RefCell<Node>>>
}
fn find_root(mut node: Rc<RefCell<Node>>) -> Rc<RefCell<Node>>
{
while let Some(parent) = node.borrow().parent.clone()
{
node = parent;
}
return node;
}
fn path_compress(mut node: Rc<RefCell<Node>>, root: Rc<RefCell<Node>>)
{
while node.borrow().parent.is_some()
{
let next = node.borrow().parent.clone().unwrap();
node.borrow_mut().parent = Some(root.clone());
node = next;
}
}
This runs fine for me with the test harness I used, though there may still be bugs. It certainly compiles and runs without a panic! due to trying to borrow_mut() something that is already borrowed. It may actually produce the right answer, that's up to you.
On IRC, Jonathan Reem pointed out that inner is borrowing until the end of its lexical scope, which is too far for what I was asking. Inlining it produces the following, which compiles without error:
fn find_root(x: Arc<Mutex<Node>>) -> Arc<Mutex<Node>> {
let mut ans = x.clone();
while ans.lock().parent.is_some() {
ans = ans.lock().parent.clone().unwrap();
}
ans
}
EDIT: As Francis Gagné points out, this has a race condition, since the lock doesn't extend long enough. Here's a modified version that only has one lock() call; perhaps it is not vulnerable to the same problem.
fn find_root(x: Arc<Mutex<Node>>) -> Arc<Mutex<Node>> {
let mut ans = x.clone();
loop {
ans = {
let tmp = ans.lock();
match tmp.parent.clone() {
None => break,
Some(z) => z
}
}
}
ans
}
EDIT 2: This only holds one lock at a time, and so is racey. I still don't know how to do hand-over-hand locking.
As pointed out by Frank Sherry and others, you shouldn't use Arc/Mutex when single threaded. But his code was outdated, so here is the new one (for version 1.0.0alpha2).
This does not take linear space either (like the recursive code given in the question).
struct Node {
parent: Option<Rc<RefCell<Node>>>
}
fn find_root(node: Rc<RefCell<Node>>) -> Rc<RefCell<Node>> {
let mut ans = node.clone(); // Rc<RefCell<Node>>
loop {
ans = {
let ans_ref = ans.borrow(); // std::cell::Ref<Node>
match ans_ref.parent.clone() {
None => break,
Some(z) => z
}
} // ans_ref goes out of scope, and ans becomes mutable
}
ans
}
fn path_compress(mut node: Rc<RefCell<Node>>, root: Rc<RefCell<Node>>) {
while node.borrow().parent.is_some() {
let next = {
let node_ref = node.borrow();
node_ref.parent.clone().unwrap()
};
node.borrow_mut().parent = Some(root.clone());
// RefMut<Node> from borrow_mut() is out of scope here...
node = next; // therefore we can mutate node
}
}
Note for beginners: Pointers are automatically dereferenced by dot operator. ans.borrow() actually means (*ans).borrow(). I intentionally used different styles for the two functions.
Although not the answer to your literal question (hand-over locking), union-find with weighted-union and path-compression can be very simple in Rust:
fn unionfind<I: Iterator<(uint, uint)>>(mut iterator: I, nodes: uint) -> Vec<uint>
{
let mut root = Vec::from_fn(nodes, |x| x);
let mut rank = Vec::from_elem(nodes, 0u8);
for (mut x, mut y) in iterator
{
// find roots for x and y; do path compression on look-ups
while (x != root[x]) { root[x] = root[root[x]]; x = root[x]; }
while (y != root[y]) { root[y] = root[root[y]]; y = root[y]; }
if x != y
{
// weighted union swings roots
match rank[x].cmp(&rank[y])
{
Less => root[x] = y,
Greater => root[y] = x,
Equal =>
{
root[y] = x;
rank[x] += 1
},
}
}
}
}
Maybe the meta-point is that the union-find algorithm may not be the best place to handle node ownership, and by using references to existing memory (in this case, by just using uint identifiers for the nodes) without affecting the lifecycle of the nodes makes for a much simpler implementation, if you can get away with it of course.

Resources