Introduction
"The code is more what you'd call 'guidelines' than actual rules." – Hector Barbossa
Scott Meyers' original Effective C++ book was phenomenally successful because it introduced a new style of programming book, focused on a collection of guidelines that had been learned from real world experience of creating software in C++. Significantly, those guidelines were explained in the context of the reasons why they were necessary – allowing the reader to decide for themselves whether their particular scenario warranted breaking the rules.
The first edition of Effective C++ was published in 1992, and at that time C++, although young, was already a subtle language that included many footguns; having a guide to the interactions of its different features was essential.
Rust is also a young language, but in contrast to C++ it is remarkably free of footguns. The strength and consistency of its type system means that if a Rust program compiles, there is already a decent chance that it will work – a phenomenon previously only observed with more academic, less accessible languages such as Haskell.
This safety – both type safety and memory safety – does come with a cost, though. Rust has a reputation for having a steep on-ramp, where newcomers have to go through the initiation rituals of fighting the borrow checker, redesigning their data structures and being befuddled by lifetimes. A Rust program that compiles may have a good chance of just working, but the struggle to get it to compile is real – even with the Rust compiler's remarkably helpful error diagnostics.
As a result, this book is aimed at a slightly different level than other Effective <Language> books; there are more Items that cover the concepts that are new with Rust, even though the official documentation already includes good introductions of these topics. These Items have titles like "Understand…" and "Familiarize yourself with…".
Rust's safety also leads to a complete absence of Items titled "Never…". If you really should never do something, the compiler will generally prevent you from doing it.
That said, the text still assumes an understanding of the basics of the language. It also assumes the 2018 edition of Rust, using the stable toolchain.
The specific rustc
version used for code fragments and error messages is 1.49. Rust is now stable enough
(and has sufficient back-compatibility guarantees) that the code fragments are unlikely to need changes for later
versions, but the error messages may vary with your particular compiler version.
The text also has a number of references to and comparisons with C++, as this is probably the closest equivalent language (particularly with C++11's move semantics), and the most likely previous language that newcomers to Rust will have encountered.
The Items that make up the book are divided into six sections:
- Types: Suggestions that revolve around Rust's core type system.
- Concepts: Core ideas that form the design of Rust.
- Dependencies: Advice for working with Rust's package ecosystem.
- Tooling: Suggestions on how to improve your codebase by going beyond just the Rust compiler.
- Asynchronous Rust: Advice for working with Rust's
async
mechanisms. - Beyond Standard Rust: Suggestions for when you have to work beyond Rust's standard, safe environment.
Although the "Concepts" section is arguably more fundamental than the "Types" section, it is deliberately placed second so that readers who are reading from beginning to end can build up some confidence first.
The following markers, borrowing Ferris from the Rust Book, are used to identify code that isn't right in some way.
Ferris | Meaning |
---|---|
This code does not compile! | |
This code panics! | |
This code block contains unsafe code. | |
This code does not produce the desired behaviour. |
Acknowledgments
My thanks go to:
- Tiziano Santoro, from whom I originally learned many things about Rust.
- Martin Disch, who pointed out potential improvements and inaccuracies in several Items.
- Clifford Matthews, Remo Senekowitsch, Kirill Zaborsky, and an anonymous ProtonMail user, who pointed out mistakes in the text.
Types
The first section of this book covers advice that revolves around Rust's type system. This type system is more expressive than that of other mainstream languages; it has more in common with "academic" languages such as OCaml or Haskell.
One core part of this is Rust's enum
type, which is considerably more expressive than the enumeration types in other
languages, and which allows for algebraic data types.
The other core pillar of Rust's type system is the trait
type. Traits are roughly equivalent to interface types in other
languages, but they are also tied to Rust's generics (Item 12), to allow interface re-use without runtime overhead.
Item 1: Use the type system to express your data structures
"who called them programers and not type writers" – thingskatedid@
The basics of Rust's type system are pretty familiar to anyone coming
from another statically typed programming language (such as C++, Go or Java).
There's a collection of integer types with specific sizes, both signed
(i8
,
i16
,
i32
,
i64
,
i128
)
and unsigned
(u8
,
u16
,
u32
,
u64
,
u128
).
There's also signed (isize
) and unsigned
(usize
) integers whose size matches the pointer size on the
target system. Rust isn't a language where you're going to be doing much in the way of converting between pointers and
integers, so that characterization isn't really relevant. However, standard collections return their size as a usize
(from .len()
), so collection indexing means that usize
values are quite common – which is obviously fine
from a capacity perspective, as there can't be more items in an in-memory collection than there are memory addresses on
the system.
The integral types do give us the first hint that Rust is a stricter world than C++ – attempting to
put a quart (i32
) into a pint pot (i16
) generates a compile-time error.
let x: i32 = 42;
let y: i16 = x;
error[E0308]: mismatched types
--> use-types/src/main.rs:14:22
|
14 | let y: i16 = x;
| --- ^ expected `i16`, found `i32`
| |
| expected due to this
|
help: you can convert an `i32` to an `i16` and panic if the converted value doesn't fit
|
14 | let y: i16 = x.try_into().unwrap();
| ^^^^^^^^^^^^^^^^^^^^^
This is reassuring: Rust is not going to sit there quietly while the programmer does things that are risky. It also
gives an early indication that while Rust has stronger rules, it also has helpful compiler messages that point the way
to how to comply with the rules. The suggested solution raises the question of how to handle situations where the
conversion would alter the value, and we'll have more to say on both error handling (Item 4) and using panic!
(Item 18) later.
Rust also doesn't allow some things that might appear "safe":
let x = 42i32; // Integer literal with type suffix
let y: i64 = x;
error[E0308]: mismatched types
--> use-types/src/main.rs:23:22
|
23 | let y: i64 = x;
| --- ^
| | |
| | expected `i64`, found `i32`
| | help: you can convert an `i32` to an `i64`: `x.into()`
| expected due to this
Here, the suggested solution doesn't raise the spectre of error handling, but the conversion does still need to be explicit. We'll discuss type conversions in more detail later (Item 6).
Continuing with the unsurprising primitive types, Rust has a
bool
type, floating point types
(f32
,
f64
)
and a unit type ()
(like C's void
).
More interesting is the char
character type, which holds a
Unicode value (similar to Go's rune
type). Although this is stored as 4 bytes internally, there are again no silent
conversions to or from a 32-bit integer.
This precision in the type system forces you to be explicit about what you're trying to express – a u32
value is
different than a char
, which in turn is different than a sequence of UTF-8 bytes, which in turn is different than a
sequence of arbitrary bytes, and it's up to you to specify exactly which you mean1. Joel Spolsky's famous blog
post can help you understand which you need.
Of course, there are helper methods that allow you to convert between these different types, but their signatures force
you to handle (or explicitly ignore) the possibility of failure. For example, a Unicode code point2 can always be represented in 32 bits, so 'a' as u32
is allowed, but the other direction is trickier (as there are u32
values that are not valid Unicode code points):
char::from_u32
returns anOption<char>
forcing the caller to handle the failure casechar::from_u32_unchecked
makes the assumption of validity, but is markedunsafe
as a result, forcing the caller to useunsafe
too (Item 16).
Moving on to aggregate types, Rust has:
- Arrays, which hold multiple instances of a single type, where
the number of instances is known at compile time. For example
[u32; 4]
is four 4-byte integers in a row. - Tuples, which hold instances of multiple heterogeneous types,
where the number of elements and their types are known at compile time, for example
(WidgetOffset, WidgetSize, WidgetColour)
. If the types in the tuple aren't distinctive – for example(i32, i32, &'static str, bool)
– it's better to give each element a name and use… - Structs, which also hold instances of heterogeneous types known at compile time, but which allows both the overall type and the individual fields to be referred to by name.
The tuple struct is a cross-breed of a struct
with a tuple: there's a name for the overall type, but no names
for the individual fields – they are referred to by number instead: s.0
, s.1
, etc.
struct TextMatch(usize, String);
let m = TextMatch(12, "needle".to_owned());
assert_eq!(m.0, 12);
This brings us to the jewel in the crown of Rust's type system, the enum
.
In its basic form, it's hard to see what there is to get excited about. As with other languages, the enum
allows you
to specify a set of mutually exclusive values, possibly with a numeric or string value attached.
enum HttpResultCode {
Ok = 200,
NotFound = 404,
Teapot = 418,
}
let code = HttpResultCode::NotFound;
assert_eq!(code as i32, 404);
Because each enum
definition creates a distinct type, this can be used to improve readability and maintainability of
functions that take bool
arguments. Instead of:
print(/* both_sides= */ true, /* colour= */ false);
a version that uses a pair of enum
s:
enum Sides {
Both,
Single,
}
enum Output {
BlackAndWhite,
Colour,
}
fn safe_print(sides: Sides, colour: Output) {
is more type-safe and easier to read at the point of invocation:
safe_print(Sides::Both, Output::BlackAndWhite);
Unlike the bool
version, if a library user were to accidentally flip the order of the arguments, the compiler would
immediately complain:
error[E0308]: mismatched types
--> use-types/src/main.rs:84:16
|
84 | safe_print(Output::BlackAndWhite, Sides::Single);
| ^^^^^^^^^^^^^^^^^^^^^ expected enum `Sides`, found enum `Output`
error[E0308]: mismatched types
--> use-types/src/main.rs:84:39
|
84 | safe_print(Output::BlackAndWhite, Sides::Single);
| ^^^^^^^^^^^^^ expected enum `Output`, found enum `Sides`
(Using the newtype pattern (Item 7) to wrap a bool
also achieves type safety and maintainability; it's generally
best to use that if the semantics will always be Boolean, and to use an enum
if there's a chance that a new
alternative (e.g. Sides::BothAlternateOrientation
) could arise in the future.)
The type safety of Rust's enum
s continues with the match
expression:
let msg = match code {
HttpResultCode::Ok => "Ok",
HttpResultCode::NotFound => "Not found",
// forgot to deal with the all-important "I'm a teapot" code
};
error[E0004]: non-exhaustive patterns: `Teapot` not covered
--> use-types/src/main.rs:65:25
|
51 | / enum HttpResultCode {
52 | | Ok = 200,
53 | | NotFound = 404,
54 | | Teapot = 418,
| | ------ not covered
55 | | }
| |_____- `HttpResultCode` defined here
...
65 | let msg = match code {
| ^^^^ pattern `Teapot` not covered
|
= help: ensure that all possible cases are being handled, possibly by adding wildcards or more match arms
= note: the matched value is of type `HttpResultCode`
The compiler forces the programmer to consider all of the possibilities3
that are represented by the enum
, even if the result is just to add a default arm _ => {}
. (Note that modern C++
compilers can and do warn about missing switch
arms for enum
s as well.)
The true power of Rust's enum
feature comes from the fact that each variant can have data that comes along with it,
making it into an algebraic data type (ADT). This is less
familiar to programmers of mainstream languages; in C/C++ terms it's like a combination of an enum
with a union
– only type-safe.
This means that the invariants of the program's data structures can be encoded into Rust's type system; states that
don't comply with those invariants won't even compile. A well-designed enum
makes the creator's intent clear to
humans as well as to the compiler:
pub enum SchedulerState {
Inert,
Pending(HashSet<Job>),
Running(HashMap<CpuId, Vec<Job>>),
}
Just from the type definition, it's reasonable to guess that Job
s get queued up in the Pending
state until the
scheduler is fully active, at which point they're assigned to some per-CPU pool.
This highlights the central theme of this Item, which is to use Rust's type system to express the concepts that are associated with the design of your software.
Returning to the power of the enum
, there are two concepts that are so common that Rust includes built-in enum
types
to express them.
The first is the concept of an Option
: either there's a value
of a particular type (Some(T)
), or there isn't (None
). Always use Option
for values that can be absent; never
fall back to using sentinel values (-1, nullptr
, …) to try to express the same concept in-band.
There is one subtle point to consider though. If you're dealing with a collection of things, you need to decide
whether having zero things in the collection is the same as not having a collection. For most situations, the
distinction doesn't arise and you can go ahead and use Vec<Thing>
: a count of zero things implies an absence of
things.
However, there are definitely other rare scenarios where the two cases need to be distinguished with
Option<Vec<Thing>>
– for example, a cryptographic system might need to distinguish between "payload transported
separately" and "empty payload provided". (This is related to the
debates around the NULL
marker columns in SQL.)
One common edge case that's in the middle is a String
which might be absent – does ""
or None
make more
sense to indicate the absence of a value? Either way works, but Option<String>
clearly communicates the possibility
that this value may be absent.
The second common concept arises from error processing: if a function fails, how should that failure be reported?
Historically, special sentinel values (e.g. -errno
return values from Linux system calls) or global variables (errno
for POSIX systems) were used. More recently, languages that support multiple or tuple return values (such as Go) from
functions may have a convention of returning a (result, error)
pair, assuming the existence of some suitable "zero"
value for the result
when the error
is non-"zero".
In Rust, always encode the result of an operation that might fail as a Result<T, E>
. The T
type holds the successful result, and the E
type holds error details on failure. Using the standard type makes the intent of the design clear, and allows the use
of standard transformations (Item 3) and error processing (Item 4); it also makes it possible to streamline error
processing with the ?
operator.
1: The situation gets muddier
still if the filesystem is involved, since filenames on popular platforms are somewhere in between arbitrary bytes and
UTF-8 sequences: see the std::ffi::OsString
documentation.
2: Technically, a Unicode scalar value rather than a code point
3: This also means that adding a new
variant to an existing enum
in a library is a breaking change (Item 21): clients of the library will need to change
their code to cope with the new variant. If an enum
is really just an old-style list of values, this behaviour can be
avoided by marking it as a non-exhaustive
enum
; see Item 21
Item 2: Use the type system to express common behaviour
Item 1 discussed how to express data structures in the type system; this Item moves on to discuss the encoding of behaviour in Rust's type system.
The first stage of this is to add methods to data structures: functions that act on an item of that type, identified
by self
. Methods can be added to struct
types, but can also be added to enum
types, in keeping with the pervasive
nature of Rust's enum
(Item 1). The name of a method gives a label for the behaviour that it encodes, and the method
signature gives type information for how to invoke it.
Code that needs to make use of behaviour associated with a type can accept an item of that type (or a reference to it), and invoke the methods needed. However, this tightly couples the two parts of the code; the code that invokes the method only accepts exactly one input type.
If greater flexibility is needed, the desired behaviour can be abstracted into the type system. The simplest such abstraction is the function pointer: a pointer to (just) some code, with a type that reflects the signature of the function. The type is checked at compile time, so by the time the program runs the value is just the size of a pointer.
fn sum(x: i32, y: i32) -> i32 {
x + y
}
// Explicit coercion to `fn` type is required...
let op: fn(i32, i32) -> i32 = sum;
Function pointers have no other data associated with them, so they can be treated as values in various ways:
// `fn` types implement `Copy`
let op1 = op;
let op2 = op;
// `fn` types implement `Eq`
assert!(op1 == op2);
// `fn` implements `std::fmt::Pointer`, used by the {:p} format specifier.
println!("op = {:p}", op);
// Example output: "op = 0x101e9aeb0"
One technical detail to watch out for: the explicit coercion to a fn
type is needed, because just using the name of a
function doesn't give you something of fn
type;
let op1 = sum;
let op2 = sum;
// Both op1 and op2 are of a type that cannot be named in user code,
// and this internal type does not implement `Eq`.
assert!(op1 == op2);
error[E0369]: binary operation `==` cannot be applied to type `fn(i32, i32) -> i32 {main::sum}`
--> use-types-behaviour/src/main.rs:53:21
|
53 | assert!(op1 == op2);
| --- ^^ --- fn(i32, i32) -> i32 {main::sum}
| |
| fn(i32, i32) -> i32 {main::sum}
|
help: you might have forgotten to call this function
|
53 | assert!(op1( /* arguments */ ) == op2);
| ^^^^^^^^^^^^^^^^^^^^^^
help: you might have forgotten to call this function
|
53 | assert!(op1 == op2( /* arguments */ ));
| ^^^^^^^^^^^^^^^^^^^^^^
Instead, the compiler error indicates that the type is something like fn(i32, i32) -> i32 {main::sum}
, a type that's
entirely internal to the compiler (i.e. could not be written in user code), and which identifies the specific function as
well as its signature. To put it another way, the type of sum
encodes both the function's signature and its
location (for optimization reasons); this
type can be automatically coerced (Item 6) to a fn
type.
Bare function pointers are very limiting, in two ways:
- The data provided when invoking a function pointer is limited to just what's held in its arguments (along with any global data).
- The only information encoded in the function pointer type is the signature of this particular function.
For the first of these, Rust supports closures: chunks of code defined by lambda expressions which can capture parts
of their environment. At runtime, Rust automatically converts a lambda together with its captured environment into a
closure that implements one of Rust's Fn*
traits, and this closure can in turn be invoked.
let amount_to_add = 2;
let closure = |y| y + amount_to_add;
assert_eq!(closure(5), 7);
The three different Fn*
traits express some nice distinctions that are needed because of this environment capturing
behaviour; the compiler automatically implements the appropriate subset of these Fn*
traits for any lambda
expression in the code (and it's not possible to manually implement any of these traits1, unlike C++'s operator()
overload).
FnOnce
describes a closure that can only be called once. If some part of its environment ismove
d into the closure, then thatmove
can only happen once – there's no other copy of the source item tomove
from – and so the closure can only be invoked once.FnMut
describes a closure that can be called repeatedly, and which can make changes to its environment because it mutably borrows from the environment.Fn
describes a closure that can be called repeatedly, and which only borrows values from the environment immutably.
The latter two traits in this list each has a trait bound of the preceding trait, which makes sense when you consider the things that use the closures.:
- If something only expects to call a closure once (expressed as a
FnOnce
), it's OK to pass it a closure that's capable of being repeatedly called (FnMut
). - If something expects to repeatedly call a closure that might mutate its environment (expressed as a
FnMut
), it's OK to pass it a closure that doesn't need to mutate its environment (Fn
).
The bare function pointer type fn
also notionally belongs at the end of this list; any
(not-unsafe
) fn
type automatically implements all of the Fn*
traits, because it borrows nothing from the
environment.
As a result, when writing code that accepts closures, use the most general Fn*
trait that works, to allow the
greatest flexibility for callers – for example, accept FnOnce
for closures that are only used once. The same
reasoning also leads to advice to prefer Fn*
trait bounds to bare function pointers (fn
).
The Fn*
traits are more flexible than a bare function pointer, but they can still only describe the behaviour of a
single function, and that only in terms of the function's signature. Continuing to generalize, collections of related
operations are described in the type system by a trait: a collection of related methods that some underlying item
makes publicly available. Each method in a trait also has a name, providing a label which allows the compiler to
disambiguate methods with the same signature, and more importantly which allows programmers to deduce the intent of the
method.
A Rust trait is roughly analogous to an "interface" in Go and Java, or to an "abstract class" (all virtual methods, no data members) in C++. Implementations of the trait must provide all the methods (but note that the trait definition can include a default implementation, Item 13), and can also have associated data that those implementations make use of. This means that code and data gets encapsulated together, in a somewhat object-oriented manner.
Returning to the original situation, code that accepts a struct
and calls methods on it is more flexible if instead
the struct
implements some trait, so that the calling code invokes trait methods rather than struct
methods. This
leads to the same kind of advice that turns up for other OO-influenced languages2: prefer accepting trait types to concrete types if future flexibility is anticipated.
Sometimes, there is some behaviour that you want to distinguish in the type system, but which cannot be expressed as
some specific method signature in a trait definition. For example, consider a trait for sorting collections; an
implementation might be stable (elements that compare the same will appear in the same order before and after the
sort) but there's no way to express this in the sort
method arguments.
In this case, it's still worth using the type system to track this requirement, using a marker trait.
pub trait Sort {
/// Re-arrange contents into sorted order.
fn sort(&mut self);
}
/// Marker trait to indicate that a [`Sortable`] sort stably.
pub trait StableSort: Sort {}
A marker trait has no methods, but an implementation still has to declare that it is implementing the trait –
which acts as a promise from the implementer: "I solemnly swear that my implementation sorts stably". Code that relies
on a stable sort can then specify the StableSort
trait bound, relying on the honour system to preserve its
invariants. Use marker traits to distinguish behaviours that cannot be expressed in the trait method signatures.
Once behaviour has been encapsulated into Rust's type system as a trait, there are two ways it can be used:
- as a trait bound, which constrains what types are acceptable for a generic data type or method at compile-time, or
- as a trait object. which constrains what types can be stored or passed to a method at run-time.
(Item 12 discusses which of the two you should prefer, where possible3.)
A trait bound indicates that generic code which is parameterized by some type T
can only be used when that type T
implements some specific trait. The presence of the trait bound means that the implementation of the generic can use the
methods from that trait, secure in the knowledge that the compiler will ensure that any T
that compiles does indeed
have those methods. This check happens at compile-time, when the generic is monomorphized (Rust's term for what C++
would call "template instantiation").
This restriction on the target type T
is explicit, encoded in the trait bounds: the trait can only be implemented by
types that satisfy the trait bounds. This is in contrast to the equivalent situation in C++, where the constraints on
the type T
used in a template<typename T>
are implicit 4: C++ template code still only compiles if all of the referenced methods are available at compile-time, but
the checks are purely based on method and signature. (This "duck typing"
leads to the chance of confusion; a C++ template that uses t.pop()
might compile for a T
type parameter of either
Stack
or Balloon
– which is unlikely to be desired behaviour.)
The need for explicit trait bounds also means that a large fraction of generics use trait bounds. To see why this is,
turn the observation around and consider what can be done with a struct Thing<T>
where there no trait bounds on
T
. Without a trait bound, the Thing
can only perform operations that apply to any type T
; this allows for
containers, collections and smart pointers, but not much else. Anything that uses the type T
is going to need
a trait bound.
pub fn dump_sorted<T>(mut collection: T)
where
T: Sort + IntoIterator,
T::Item: Debug,
{
// Next line requires `T: Sort` trait bound.
collection.sort();
// Next line requires `T: IntoIterator` trait bound.
for item in collection {
// Next line requires `T::Item : Debug` trait bound
println!("{:?}", item);
}
}
So the advice here is to use trait bounds to express requirements on the types used in generics, but it's easy advice to follow – the compiler will force you to comply with it regardless.
A trait object is the other way of making use of the encapsulation defined by a trait, but here different possible implementations of the trait are chosen at run-time rather than compile-time. This dynamic dispatch is analogous to the use of virtual functions in C++, and under the covers Rust has 'vtable' objects that are roughly analogous to those in C++.
This dynamic aspect of trait objects also means that they always have to be handled indirectly, via reference (&dyn Trait
) or a pointer (Box<dyn Trait>
). This is because the size of the object implementing the trait isn't known at
compile time – it could be a giant struct
or a tiny enum
– so there's no way to allocate the right
amount of space for a bare trait object.
A similar concern means that traits used as trait objects cannot have methods that return the Self
type, because the
compiled-in-advance code that uses the trait object would have no idea how big that Self
might be.
A trait that has a generic method fn method<T>(t:T)
allows for the possibility of an infinite number of implemented
methods, for all the different types T
that might exist. This is fine for a trait used as a trait bound, because the
infinite set of possibly invoked generic methods becomes a finite set of actually invoked generic methods at compile
time. The same is not true for a trait object: the code available at compile time has to cope with all possible T
s
that might arrive at run-time.
These two restrictions – no returning Self
and no generic methods – are combined into the concept of
object safety. Only object safe traits can be used as trait objects.
1: At least, not in
stable Rust at the time of writing. The
unboxed_closures
and
fn_traits
experimental features may
change this in future.
2: For example, Effective Java Item 64: Refer to objects by their interfaces
3: Spoiler: trait bounds.
4: The addition of concepts in C++20 allows explicit specification of constraints on template types, but the checks are still only performed when the template is instantiated, not when it is declared.
Item 3: Avoid match
ing Option
and Result
Item 1 expounded the virtues of enum
and showed how match
expressions force the programmer to take all
possibilities into account; this Item explores situations where you should prefer to avoid match
expressions –
explicitly at least.
Item 1 also introduced the two ubiquitous enum
s that are provided by the Rust standard library:
Option<T>
to express that a value (of typeT
) may not be presentResult<T, E>
, for when an operation to return a value (of typeT
) may not succeed, and may instead return an error (of typeE
).
For these particular enum
s, explicitly using match
often leads to code that is less compact than it needs to, and
which isn't idiomatic Rust.
The first situation where a match
is unnecessary is when only the value is relevant, and the absence of value (and any
associated error) can just be ignored.
struct S {
field: Option<i32>,
}
match &s.field {
Some(i) => println!("field is {}", i),
None => {}
}
For this situation, an if let
expression is one line shorter and, more importantly, clearer:
if let Some(i) = &s.field {
println!("field is {}", i);
}
However, most of the time the absence of a value, and an associated error, is going to be something that the programmer has to deal with. Designing software to cope with failure paths is hard, and most of that is essential complexity that no amount of syntactic support can help with – deciding what should happen if an operation fails.
In some situations, the right decision is to perform an ostrich manoeuvre and explicitly not cope with failure. Doing
this with an explicit match
would be needlessly verbose:
let result = std::fs::File::open("/etc/passwd");
let f = match result {
Ok(f) => f,
Err(_e) => panic!("Failed to open /etc/passwd!"),
};
Both Option
and Result
provide a pair of methods that extract their inner value and panic!
if it's absent:
unwrap
and
expect
. The latter allows the error message on
failure to be personalized, but in either case the resulting code is shorter and simpler – error handling is
delegated to the .unwrap()
suffix (but is still present).
let f = std::fs::File::open("/etc/passwd").unwrap();
Be clear, though: these helper functions still panic!
, so choosing to use them is the same as choosing to panic!
(Item 18).
However, in many situations, the right decision for error handling is to defer the decision to somebody else. This is
particularly true when writing a library, where the code may be used in all sorts of different environments that can't
be foreseen by the library author. To make that somebody else's job easier, prefer Result
to Option
, even
though this may involve conversions between different error types (Item 4); Result
has also a
[#must_use]
attribute to
nudge library users in the right direction.
Explicitly using a match
allows an error to propagate, but at the cost of some visible boilerplate (reminiscent of
Go):
pub fn find_user(username: &str) -> Result<UserId, std::io::Error> {
let f = match std::fs::File::open("/etc/passwd") {
Ok(f) => f,
Err(e) => return Err(e),
};
The key ingredient for reducing boilerplate is Rust's question mark operator
?
. This piece of
syntactic sugar takes care of matching the Err
arm and the return Err(...)
expression in a single character:
pub fn find_user(username: &str) -> Result<UserId, std::io::Error> {
let f = std::fs::File::open("/etc/passwd")?;
Newcomers to Rust sometimes find this disconcerting: the question mark can be hard to spot on first glance, leading to disquiet as to how the code can possibly work. However, even with a single character, the type system is still at work, ensuring that all of the possibilities expressed in the relevant types (Item 1) are covered – leaving the programmer to focus on the mainline code path without distractions.
What's more, there's generally no cost to these apparent method invocations: they are all generic functions marked as
[#inline]
, so the generated code
will typically compile to machine code that's identical to the manual version.
These two factors taken together mean that you should prefer Option
and Result
transforms to explicit match
expressions.
In the previous example, the error types lined up: both the inner and outer methods expressed errors as
std::io::Error
. That's often not the case; one function may
accumulate errors from a variety of different sub-libraries, each of which uses different error types. Error mapping in
general is discussed in Item 4; for now, just be aware that a manual mapping:
pub fn find_user(username: &str) -> Result<UserId, String> {
let f = match std::fs::File::open("/etc/passwd") {
Ok(f) => f,
Err(e) => {
return Err(format!("Failed to open password file: {:?}", e))
}
};
// ...
can be more succinctly and idiomatically expressed with the
.map_err()
transformation:
pub fn find_user(username: &str) -> Result<UserId, String> {
let f = std::fs::File::open("/etc/passwd")
.map_err(|e| format!("Failed to open password file: {:?}", e))?;
// ...
This approach generalizes more widely. The question mark operator is a big hammer; use transformation methods on
Option
and Result
types to manoeuvre them into a position where they can be a nail.
The standard library provides a wide variety of these transformation methods to make this possible, as shown in the following map. In line with Item 18, methods that can panic are highlighted in red.
(The online version of this diagram is clickable: each box links to the relevant documentation.)
One common situation isn't covered by the diagram is dealing with references. For example, consider a structure that optionally holds some data.
struct InputData {
payload: Option<Vec<u8>>,
}
A method on this struct
which tries to pass the payload to an encryption function with signature (&[u8]) -> Vec<u8>
fails if there's a naive attempt to take a reference:
impl InputData {
pub fn encrypted(&self) -> Vec<u8> {
encrypt(&self.payload.unwrap_or(vec![]))
}
}
error[E0507]: cannot move out of `self.payload` which is behind a shared reference
--> transform/src/main.rs:57:22
|
57 | encrypt(&self.payload.unwrap_or(vec![]))
| ^^^^^^^^^^^^
| |
| move occurs because `self.payload` has type `Option<Vec<u8>>`, which does not implement the `Copy` trait
| help: consider borrowing the `Option`'s content: `self.payload.as_ref()`
The error message describes exactly what's needed to make the code work, the
as_ref()
method1 on Option
. This method
converts a reference-to-an-Option
to be an Option
-of-a-reference:
pub fn encrypted(&self) -> Vec<u8> {
encrypt(self.payload.as_ref().unwrap_or(&vec![]))
}
To sum up:
- Get used to the transformations of
Option
andResult
, and preferResult
toOption
.- Use
.as_ref()
as needed when transformations involve references.
- Use
- Use them in preference to explicit
match
operations. - In particular, use them to transform result types into a form where the
?
operator applies.
1: Note that
this method is separate from the AsRef
trait, even though the method name is the same.
Item 4: Prefer idiomatic Error
variants
Item 3 described how to use the transformations that the standard library provides for the Option
and Result
types
to allow concise, idiomatic handling of result types using the ?
operator. It stopped short of discussing how best to
handle the variety of different error types E
that arise as the second type argument of a Result<T, E>
; that's the
subject of this Item.
This is only really relevant when there are a variety of different error types in play; if all of the different errors that a function encounters are already of the same type, it can just return that type. When there are errors of different types, there's a decision to be made about whether the sub-error type information should be preserved.
The Error
Trait
It's always good to understand what the standard traits (Item 5) involve, and the relevant trait here is
std::error::Error
. The E
type parameter for a Result
doesn't have to be a type that implements Error
, but it's a common convention that allows wrappers to express
appropriate trait bounds – so prefer to implement Error
for your error types. However, if you're writing
code for a no_std
environment (Item 37), this recommendation doesn't apply – the Error
trait is implemented
in std
, not core
, and so is not available.
The first thing to notice is that the only hard requirement for Error
types is the trait bounds: any type that
implements Error
also has to implement both:
- the
Display
trait, meaning that it can beformat!
ed with{}
, and - the
Debug
trait, meaning that it can beformat!
ed with{:?}
.
In other words, it should be possible to display Error
types to both the user and the programmer.
The only1 method in the trait is
source()
, which allows an Error
type to expose
an inner, nested error. This method is optional – it comes with a default implementation (Item 13) returning
None
, indicating that inner error information isn't available.
Minimal Errors
If nested error information isn't needed, then an implementation of the Error
type need not be much more than a
String
– one rare occasion where a "stringly-typed" variable might be appropriate. It does need to be a
little more than a String
though; while it's possible to use String
as the E
type parameter:
pub fn find_user(username: &str) -> Result<UserId, String> {
let f = std::fs::File::open("/etc/passwd")
.map_err(|e| format!("Failed to open password file: {:?}", e))?;
// ...
a String
doesn't implement Error
, which we'd prefer so that other areas of code can deal in Error
s. It's not
possible to impl Error
for String
, because neither the trait nor the type belong to us (the so-called orphan
rule):
impl std::error::Error for String {}
error[E0117]: only traits defined in the current crate can be implemented for arbitrary types
--> errors/src/main.rs:18:5
|
18 | impl std::error::Error for String {}
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^------
| | |
| | `String` is not defined in the current crate
| impl doesn't use only types from inside the current crate
|
= note: define and implement a trait or new type instead
A type alias doesn't help either, because it doesn't create a new type and so doesn't change the error message.
pub type MyError = String;
pub fn find_user(username: &str) -> Result<UserId, MyError> {
error[E0117]: only traits defined in the current crate can be implemented for arbitrary types
--> errors/src/main.rs:38:5
|
38 | impl std::error::Error for MyError {}
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^-------
| | |
| | `String` is not defined in the current crate
| impl doesn't use only types from inside the current crate
|
= note: define and implement a trait or new type instead
As usual, the compiler error message gives a hint of how to solve the problem. Defining a tuple struct that wraps the
String
type (the "newtype" pattern, Item 7) allows the Error
trait to be implemented, provided that Debug
and
Display
are implemented too:
#[derive(Debug)]
pub struct MyError(String);
impl std::fmt::Display for MyError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", self.0)
}
}
impl std::error::Error for MyError {}
pub fn find_user(username: &str) -> Result<UserId, MyError> {
let f = std::fs::File::open("/etc/passwd").map_err(|e| {
MyError(format!("Failed to open password file: {:?}", e))
})?;
// ...
For convenience, it may make sense to implement the From<String>
trait to allow string values to be easily
converted into MyError
instances (Item 6):
impl std::convert::From<String> for MyError {
fn from(msg: String) -> Self {
Self(msg)
}
}
When it encounters the question mark operator (?
), the compiler will automatically apply any relevant From
trait
implementations that are needed to reach the destination error return type. This allows further minimization:
pub fn find_user(username: &str) -> Result<UserId, MyError> {
let f = std::fs::File::open("/etc/passwd")
.map_err(|e| format!("Failed to open password file: {:?}", e))?;
// ...
For the error path here:
File::open
returns an error of typestd::io::Error
.format!
converts this to aString
, using theDebug
implementation ofstd::io::Error
.?
makes the compiler look for and use aFrom
implementation that can take it fromString
toMyError
.
Nested Errors
The alternative scenario is where the content of nested errors is important enough that it should be preserved and made available to the caller.
Consider a library function that attempts to return the first line of a file as a string, as long as it is not too long. A moment's thought reveals (at least) three distinct types of failure that could occur:
- The file might not exist, or might be inaccessible for reading.
- The file might contain data that isn't valid UTF-8, and so can't be converted into a
String
. - The file might have a first line that is too long.
In line with Item 1, you can and should use the type system to express and encompass all of these possibilities as an
enum
:
#[derive(Debug)]
pub enum MyError {
Io(std::io::Error),
Utf8(std::string::FromUtf8Error),
General(String),
}
The enum
definition includes a derive(Debug)
, but to satisfy the Error
trait a Display
implementation is also
needed.
impl std::fmt::Display for MyError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
MyError::Io(e) => write!(f, "IO error: {}", e),
MyError::Utf8(e) => write!(f, "UTF-8 error: {}", e),
MyError::General(s) => write!(f, "General error: {}", s),
}
}
}
It also makes sense to override the default source()
implementation for easy access to nested errors.
use std::error::Error;
impl Error for MyError {
fn source(&self) -> Option<&(dyn Error + 'static)> {
match self {
MyError::Io(e) => Some(e),
MyError::Utf8(e) => Some(e),
MyError::General(_) => None,
}
}
}
This allows the error handling to be concise while still preserving all of the type information across different classes of error:
/// Return the first line of the given file.
pub fn first_line(filename: &str) -> Result<String, MyError> {
let file = std::fs::File::open(filename).map_err(MyError::Io)?;
let mut reader = std::io::BufReader::new(file);
// (A real implementation could just use `reader.read_line()`)
let mut buf = vec![];
let len = reader.read_until(b'\n', &mut buf).map_err(MyError::Io)?;
let result = String::from_utf8(buf).map_err(MyError::Utf8)?;
if result.len() > MAX_LEN {
return Err(MyError::General(format!("Line too long: {}", len)));
}
Ok(result)
}
It's also a good idea to implement the From
trait for all of the sub-error types (Item 6):
impl From<std::io::Error> for MyError {
fn from(e: std::io::Error) -> Self {
Self::Io(e)
}
}
impl From<std::string::FromUtf8Error> for MyError {
fn from(e: std::string::FromUtf8Error) -> Self {
Self::Utf8(e)
}
}
This prevents library users from suffering under the orphan rules themselves: they aren't allowed to implement From
on
MyError
, because both the trait and the struct are external to them.
Better still, implementing From
allows for even more concision, because the question mark
operator will
automatically perform any necessary From
conversions:
/// Return the first line of the given file.
pub fn firstline(filename: &str) -> Result<String, MyError> {
let file = std::fs::File::open(filename)?; // via `From<std::io::Error>`
let mut reader = std::io::BufReader::new(file);
let mut buf = vec![];
let len = reader.read_until(b'\n', &mut buf)?; // via `From<std::io::Error>`
let result = String::from_utf8(buf)?; // via `From<std::string::FromUtf8Error>`
if result.len() > MAX_LEN {
return Err(MyError::General(format!("Line too long: {}", len)));
}
Ok(result)
}
Trait Objects
The first approach to nested errors threw away all of the sub-error detail, just preserving some string output
(format!("{:?}", err)
). The second approach preserved the full type information for all possible sub-errors, but
required a full enumeration of all possible types of sub-error.
This raises the question: is there a half-way house between these two approaches, preserving sub-error information without needing to manually include every possible error type?
Encoding the sub-error information as a trait object
avoids the need for an enum
variant for every possibility, but erases the details of the specific underlying error
types. The receiver of such an object would have access to the methods of the Error
trait – display()
,
debug()
and source()
in turn – but wouldn't know the original static type of the sub-error.
#[derive(Debug)]
pub enum WrappedError {
Wrapped(Box<dyn Error>),
General(String),
}
impl std::fmt::Display for WrappedError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Wrapped(e) => write!(f, "Inner error: {}", e),
Self::General(s) => write!(f, "{}", s),
}
}
}
It turns out that this is possible, but it's surprisingly subtle. Part of the difficulty comes from the constraints on trait objects (Item 12), but Rust's coherence rules also come into play, which (roughly) say that there can be at most one implementation of a trait for a type.
A putative WrappedError
would naively be expected to both implement the Error
trait, and also to implement the
From<Error>
trait to allow sub-errors to be easily wrapped. That means that a WrappedError
can be created from
an
inner WrappedError
, as WrapperError
implements Error
, and that clashes with the blanket reflexive implementation
of From
:
error[E0119]: conflicting implementations of trait `std::convert::From<WrappedError>` for type `WrappedError`:
--> errors/src/main.rs:239:1
|
239 | impl<E: 'static + Error> From<E> for WrappedError {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: conflicting implementation in crate `core`:
- impl<T> From<T> for T;
David Tolnay's anyhow
is a crate that has already solved these
problems, and which adds other helpful features (such as stack traces)
besides. As a result, it is rapidly becoming the standard recommendation for error handling – a recommendation
seconded here: consider using the anyhow
crate for error handling in applications.
Libraries versus Applications
The final advice of the previous section included the qualification "…for error handling in applications". That's because there's often a distinction between code that's written for re-use in a library, and code that forms a top-level application2.
Code that's written for a library can't predict the environment in which the code is used, so it's preferable to emit
concrete, detailed error information, and leave the caller to figure out how to use that information. This leans towards
the enum
-style nested errors described previously (and also avoids a dependency on anyhow
in the public API of the
library, cf. Item 24).
However, application code typically needs to concentrate more on how to present errors to the user. It also potentially
has to cope with all of the different error types emitted by all of the libraries that are present in its dependency
graph (Item 25). As such, a more dynamic error type (such as
anyhow::Error
) makes error handling simpler and more
consistent across the application.
Summary
This item has covered a lot of ground, so a summary is in order:
- The standard
Error
trait requires little of you, so prefer to implement it for your error types. - When dealing with heterogeneous underlying error types, decide whether preserving those types is needed.
- If not, use
anyhow
to wrap sub-errors. - If they are needed, encode them in an
enum
and provide conversions.
- If not, use
- Consider using the
anyhow
crate for convenient, idiomatic error handling.
It's your decision, but whatever you decide, encode it in the type system (Item 1).
1: Or at least the only non-deprecated, stable method.
2: This section is inspired by Nick Groenen's "Rust: Structuring and handling errors in 2020" article.
Item 5: Familiarize yourself with standard traits
Rust encodes key behavioural aspects of its type system in the type system itself, through a collection of fine-grained standard traits that describe those behaviours.
Many of these traits will seem familiar to programmers coming from C++, corresponding to concepts such as copy-constructors, destructors, equality and assignment operators, etc.
As in C++, it's usually a good idea to implement many of these traits for your own types; the Rust compiler will give you helpful error messages if some operation needs one of these traits for your type, and it isn't present.
Implementing such a large collection of traits may seem daunting, but most of the common ones can be automatically
applied to user-defined types, through use of derive
macros. This leads to type definitions like:
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
enum MyBooleanOption {
Off,
On,
}
This fine-grained specification of behaviour can be disconcerting at first, but it's important to be familiar with the most common of these standard crates so that the available behaviours of a type definition can be immediately understood.
A rough one-sentence summary each of the standard traits that this Item covers is:
Clone
: Items of this type can make a copy of themselves when asked.Copy
: If the compiler makes a bit-for-bit copy of this item's memory representation, the result is a valid new item.Default
: It's possible to make new instance of this type with sensible default values.PartialEq
: There's a partial equivalence relation for items of this type – any two items can be definitively compared, but it's not always true thatx==x
.Eq
. There's an equivalence relation for items of this type: any two items can be definitively compared.PartialOrd
: Some items of this type can be compared and ordered.Ord
: All items of this type can be compared and ordered.Hash
: Items of this type can produce a stable hash of their contents when asked.Debug
: Items of this type can be displayed to programmers.Display
: Items of this type can be displayed to users.
These traits can all be derive
d for user-defined types, with the exception of Display
(included here because of its
overlap with Debug
). However, there are occasions when a manual implementation – or no implementation –
is preferable.
Rust also allows various built-in unary and binary operators to be overloaded for user-defined types, by implementing
various traits from the std::ops
module. These traits are not
derivable, and are typically only needed for types that represent "algebraic" objects.
Other (non-derive
able) standard traits are covered in other Items, and so are not included here. These include:
Fn
,FnOnce
andFnMut
: Items of this type represent closures that can be invoked. See Item 2.Error
: Items of this type represent error information that can be displayed to users or programmers, and which may hold nested sub-error information. See Item 4.Drop
: Items of this type perform processing when they are destroyed, which is essential for RAII patterns. See Item 11.From
andTryFrom
: Items of this type can be automatically created from items of some other type, but with a possibility of failure in the latter case. See Item 6.Deref
andDerefMut
: Items of this type are pointer-like objects that can be dereferenced to get access to an inner item. See Item 9.Iterator
and friends: Items of this type can be iterated over. See Item 10.Send
andSync
: Items of this type are safe to transfer between, or be referenced by, multiple threads. See Item 17.
Clone
The Clone
trait indicates that it's possible to make a new copy of an item, by calling the
clone()
method. This is roughly equivalent to
C++'s copy-constructor, but more explicit: the compiler will never silently invoke this method on its own (read on to
the next section for that).
Clone
can be derive
d; the macro implementation clones an aggregate type by cloning each of its members in turn,
again, roughly equivalent to a default copy-constructor in C++. This makes the trait opt-in (by adding
#[derive(Clone)]
), in contrast to the opt-out behaviour in C++ (MyType(const MyType&) = delete;
).
This is such a common and useful operation that it's more interesting to investigate the situations where you shouldn't
or can't implement Clone
, or where the default derive
implementation isn't appropriate.
- You shouldn't implement
Clone
if the item embodies unique access to some resource (such as an RAII type, Item 11), or when there's another reason to restrict copies (e.g. if the item holds cryptographic key material). - You can't implement
Clone
if some component of your type is un-Clone
able in turn. Examples include:- Fields that are mutable references (
&mut T
), because the borrow checker (Item 14) only allows a single mutable reference at a time. - Standard library types that fall in to the previous category, such as
Mutex
orMutexGuard
.
- Fields that are mutable references (
- You should manually implement
Clone
if there is anything about your item that won't be captured by a (recursive) field-by-field copy, or if there is additional book-keeping associated with item lifetimes (for example: consider a type that tracks the number of extant items at runtime for metrics purposes)
Copy
The Copy
trait has a trivial declaration:
pub trait Copy: Clone { }
There are no methods in this trait, meaning that it is a marker trait – it's used to indicate some constraint on a type that's not directly expressed in the type system.
In the case of Copy
, the meaning of this marker is that not only can items of this type be copied (hence the Clone
trait bound), but also a bit-for-bit copy of the memory holding an item gives a correct new item. Effectively, this
trait is a marker that says that a type is a "plain old data" (POD)
type.
In contrast to user-defined marker traits (Item 1), Copy
has a special significance to the compiler1 over and above being available for trait bounds – it shifts
the compiler from move semantics to copy semantics.
With move semantics for the assignment operator, what the right hand giveth, the left hand taketh away:
#[derive(Debug)]
struct KeyId(u32);
let k = KeyId(42);
let k2 = k; // value moves out of k in to k2
println!("k={:?}", k);
error[E0382]: borrow of moved value: `k`
--> std-traits/src/main.rs:52:28
|
50 | let k = KeyId(42);
| - move occurs because `k` has type `main::KeyId`, which does not implement the `Copy` trait
51 | let k2 = k; // value moves out of k in to k2
| - value moved here
52 | println!("k={:?}", k);
| ^ value borrowed here after move
With copy semantics, the original item lives on:
#[derive(Debug, Clone, Copy)]
struct KeyId(u32);
let k = KeyId(42);
let k2 = k; // value bitwise copied from k to k2
println!("k={:?}", k);
This makes Copy
one of the most important traits to watch out for: it fundamentally changes the behaviour of
assignments – and this includes parameters for method invocations.
In this respect, there are again overlaps with C++'s copy-constructors, but it's worth emphasizing a key distinction: in
Rust there is no way to get the compiler to silently invoke user-defined code – it's either explicit (a call to
.clone()
), or it's not user-defined (a bitwise copy).
To finish this section, observe that because Copy
has a Clone
trait bound, it's possible to .clone()
any
Copy
-able item. However, it's not a good idea: a bitwise copy will always be faster than invoking a trait method.
Clippy (Item 29) will warn you about this:
let k3 = k.clone();
warning: using `clone` on a `Copy` type
--> std-traits/src/main.rs:68:18
|
68 | let k3 = k.clone();
| ^^^^^^^^^ help: try removing the `clone` call: `k`
|
= note: `#[warn(clippy::clone_on_copy)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
warning: 1 warning emitted
Default
The Default
trait defines a default constructor, via a
default()
) method. This trait can be
derive
d for user-defined types, provided that all of the sub-types involved have a Default
implementation of
their own; if they don't, you'll have to implement the trait manually. Continuing the comparison with C++, notice
that a default constructor has to be explicitly triggered; the compiler does not create one automatically.
The most useful aspect of the Default
trait is its combination with struct update
syntax. This syntax allows
struct
fields to be initialized by copying or moving their contents from an existing instance of the same struct
,
for any fields that aren't explicitly initialized. The template to copy from is given at the end of the initialization,
after ..
, and the Default
trait provides an ideal template to use:
#[derive(Default)]
struct Colour {
red: u8,
green: u8,
blue: u8,
alpha: u8,
}
let c = Colour {
red: 128,
..Default::default()
};
This makes it much easier to initialize structures with lots of fields, only some of which have non-default values. (The builder pattern, Item 8, may also be appropriate for these situations.)
PartialEq
and Eq
The PartialEq
and Eq
traits allow you to define equality for user-defined types. These traits have special significance
because if they're present, the compiler will automatically use them for equality (==
) checks, similarly to
operator==
in C++. The default derive
implementation does this with a recursive field-by-field comparison.
The Eq
version is just a marker trait extension of PartialEq
which adds the assumption of reflexivity: any type T
that claims to support Eq
should ensure that x == x
is true for any x: T
.
This is sufficiently odd to immediately raise the question: when wouldn't x == x
? The primary rationale behind this
split relates to floating point
numbers2, and specifically to the special "not a
number" value NaN (f32::NAN
/ f64::NAN
in Rust). The floating point specifications require that nothing
compares equal to NaN, including NaN itself; the PartialEq
trait is the knock-on effect of this.
For user-defined types that don't have any float-related peculiarities, you should implement Eq
whenever you
implement PartialEq
. The full Eq
trait is also required if you want to use the type as the key in a
HashMap
(as well as the Hash
trait).
You should implement PartialEq
manually if your type contains any fields that do not affect the item's identity,
such as internal caches and other performance optimizations.
PartialOrd
and Ord
The ordering traits PartialOrd
and Ord
allow comparisons between two items of a type, returning Less
, Greater
,
or Equal
. The traits require equivalent equality traits to be implemented (PartialOrd
requires PartialEq
, Ord
requires Eq
), and the two have to agree with each other (watch out for this with manual implementations in
particular).
As with the equality traits, the comparison traits have special significance because the compiler will automatically use
them for comparison operations (<
, >
, <=
, >=
).
The default implementation produced by derive
compares fields (or enum
variants) lexicographically in the order
they're defined, so if this isn't correct you'll need to implement the traits manually (or re-order the fields).
Unlike PartialEq
, the PartialOrd
trait does correspond to a variety of real situations. For example, it could be
used to express a subset relationship3 among collections: {1, 2}
is a
subset of {1, 2, 4}
, but {1, 3}
is not a subset of {2, 4}
nor vice versa.
However, even if a partial order does accurately model the behaviour of your type, be wary of implementing just
PartialOrd
(a rare occasion that contradicts the advice of Item 1) – it can lead to surprising results:
let x = Oddity(1);
let x2 = Oddity(1);
if x <= x2 {
println!("Never hit this!");
}
let y = Oddity(2);
if x <= y {
println!("y is bigger"); // Not hit
} else if y <= x {
// Programmers are likely to omit this arm
println!("x is bigger"); // Not hit
} else {
println!("neither is bigger"); // This one
}
Hash
The Hash
trait is used to produce a single value that has a high probability of being different for different items;
this value is used as the basis for hash-bucket based data structures like
HashMap
and
HashSet
.
Flipping this around, it's essential that the "same" items (as per Eq
) always produce the same hash; if x == y
(via Eq
) then it must always be true that hash(x) == hash(y)
. If you have a manual Eq
implementation, check
whether you also need a manual implementation of Hash
to comply with this requirement.
Debug
and Display
The Debug
and Display
traits allow a type to specify how it should be included in output, for either normal ({}
format argument) or debugging purposes ({:?}
format argument), roughly analogous to an iostream operator<<
overload
in C++.
The differences between the intents of the two traits go beyond which format specifier is needed, though:
Debug
can be automatically derived,Display
can only be manually implemented. This is related to…- The layout of
Debug
output may change between different Rust versions. If the output will ever be parsed by other code, useDisplay
. Debug
is programmer-oriented,Display
is user-oriented. A thought experiment that helps with this is to consider what would happen if the program was localized to a language that the authors don't speak;Display
is appropriate if the content should be translated,Debug
if not.
As a general rule, add an automatically generated Debug
implementation for your types unless they contain
sensitive information (personal details, cryptographic material etc.). A manual implementation of Debug
can be
appropriate when the automatically generated version would emit voluminous amounts of detail.
Implement Display
if your types are designed to be shown to end users in textual output.
Operator Overloads
Similarly to C++, Rust allows various arithmetic and bitwise operators to be overloaded for user-defined types. This
is useful for "algebraic" or bit-manipulation types (respectively) where there is a natural interpretation of these
operators. However, experience from C++ has shown that it's best to avoid overloading operators for unrelated types
as it often leads to code that is hard to maintain and has unexpected performance properties (e.g. x + y
silently
invokes an expensive O(N) method).
Continuing with the principle of least surprise, if you implement any operator overloads you should implement a
coherent set of operator overloads. For example, if x + y
has an overload
(Add
), and -y
(Neg
), then you should also implement x - y
(Sub
) and make sure it gives the same answer as x + (-y)
.
The items passed to the operator overload traits are moved, which means that non-Copy
types will be consumed by
default. Adding implementations for &'a MyType
can help with this, but requires more boilerplate to cover all of the
possibilities (e.g. 4 = 2 × 2 possibilities for combining reference/non-reference arguments to a binary operator).
Summary
This item has covered a lot of ground, so some tables that summarize the standard traits that have been touched on are
in order. First, the traits of this Item, all of which can be automatically derive
d except Display
.
Trait | Compiler Use | Bound | Methods |
---|---|---|---|
Clone | clone | ||
Copy | let y = x; | Clone | Marker trait |
Default | default | ||
PartialEq | x == y | eq | |
Eq | x == y | PartialEq | Marker trait |
PartialOrd | x < y , x <= y , | PartialEq | partial_cmp |
Ord | x < y , x <= y , | Eq + PartialOrd | cmp |
Hash | hash | ||
Debug | format!("{:?}", x) | fmt | |
Display | format!("{}", x) | fmt |
The operator overloads are in the next table. None of these can be derive
d.
Trait | Compiler Use | Bound | Methods |
---|---|---|---|
Add | x + y | add | |
AddAssign | x += y | add_assign | |
BitAnd | x & y | bitand | |
BitAndAssign | x &= y | bitand_assign | |
BitOr | x ⎮ y | bitor | |
BitOrAssign | x ⎮= y | bitor_assign | |
BitXor | x ^ y | bitxor | |
BitXorAssign | x ^= y | bitxor_assign | |
Div | x / y | div | |
DivAssign | x /= y | div_assign | |
Index | x[y] | index | |
IndexMut | x[y] | index_mut | |
Mul | x * y | mul | |
MulAssign | x *= y | mul_assign | |
Neg | -x | neg | |
Not | !x | not | |
Rem | x % y | rem | |
RemAssign | x %= y | rem_assign | |
Shl | x << y | shl | |
ShlAssign | x <<= y | shl_assign | |
Shr | x >> y | shr | |
ShrAssign | x >>= y | shr_assign | |
Sub | x - y | sub | |
SubAssign | x -= y | sub_assign |
For completeness, the standard traits that are covered in other items are included in the following table; none of these
traits are derive
able (but Send
and Sync
may be automatically implemented by the compiler).
Trait | Item | Compiler Use | Bound | Methods |
---|---|---|---|---|
Fn | Item 2 | x(a) | FnMut | call |
FnMut | Item 2 | x(a) | FnOnce | call_mut |
FnOnce | Item 2 | x(a) | call_once | |
Error | Item 4 | Display + Debug | [source ] | |
From | Item 6 | from | ||
TryFrom | Item 6 | try_from | ||
Into | Item 6 | into | ||
TryInto | Item 6 | try_into | ||
AsRef | Item 9 | as_ref | ||
AsMut | Item 9 | as_mut | ||
Borrow | Item 9 | borrow | ||
BorrowMut | Item 9 | Borrow | borrow_mut | |
ToOwned | Item 9 | to_owned | ||
Deref | Item 9 | *x , &x | deref | |
DerefMut | Item 9 | *x , &mut x | Deref | deref_mut |
Index | Item 9 | x[idx] | index | |
IndexMut | Item 9 | x[idx] = ... | Index | index_mut |
Pointer | Item 9 | format("{:p}", x) | fmt | |
Iterator | Item 10 | next | ||
IntoIterator | Item 10 | for y in x | into_iter | |
FromIterator | Item 10 | from_iter | ||
ExactSizeIterator | Item 10 | Iterator | (size_hint ) | |
DoubleEndedIterator | Item 10 | Iterator | next_back | |
Drop | Item 11 | } (end of scope) | drop | |
Sized | Item 14 | Marker trait | ||
Send | Item 17 | cross-thread transfer | Marker trait | |
Sync | Item 17 | cross-thread use | Marker trait |
1: As do
several of the other marker traits in std::marker
.
2: Of course, comparing floats for equality is always a dangerous game, as there is typically no guarantee that rounded calculations will produce a result that is bit-for-bit identical to the number you first thought of.
3: More generally, any lattice structure also has a partial order.
Item 6: Understand type conversions
In general, Rust does not perform automatic conversion between types. This includes integral types, even when the transformation is "safe":
let x: i32 = 42;
let y: i16 = x;
error[E0308]: mismatched types
--> use-types/src/main.rs:14:22
|
14 | let y: i16 = x;
| --- ^ expected `i16`, found `i32`
| |
| expected due to this
|
help: you can convert an `i32` to an `i16` and panic if the converted value doesn't fit
|
14 | let y: i16 = x.try_into().unwrap();
| ^^^^^^^^^^^^^^^^^^^^^
Rust type conversions fall into three categories:
- manual: user-defined type conversions provided by implementing the
From
andInto
traits - semi-automatic: explicit casts between values using the
as
keyword - automatic: implicit coercion into a new type
The latter two don't apply to conversions of user defined types (with a couple of exceptions), so the majority of this Item will focus on manual conversion. However, sections at the end will discuss casting and coercion – including the exceptions where they can apply to a user-defined type.
User-Defined Type Conversions
As with other features of the language (Item 5) the ability to perform conversions between values of different user-defined types is encapsulated as a trait – or rather, as a set of related generic traits.
The four relevant traits that express the ability to convert values of a type are:
From<T>
: Items of this type can be built from items of typeT
.TryFrom<T>
: Items of this type can sometimes be built from items of typeT
.Into<T>
: Items of this type can converted into items of typeT
.TryInto<T>
: Items of this type can sometimes be converted into items of typeT
.
Given the discussion in Item 1 about expressing things in the type system, it's no surprise to discover that the
difference with the Try...
variants is that the sole trait method returns a Result
rather than a guaranteed new
item; the trait definition also requires an associated type that provides the type of the error E
for failure
situations. You can choose to ignore the possibility of error (e.g. with .unwrap()
), but as usual it needs to be a
deliberate choice.
There's also some symmetry here: if a type T
can be transmuted into
a type U
, isn't that the same as it being
possible to create an item of type U
by transmutation from
an item of type T
?
This is indeed the case, and it leads to the first piece of advice: implement the From
trait for conversions. The
Rust standard library had to pick just one of the two possibilities (to prevent the system from spiralling around in
dizzy circles1), and came down on the side of
automatically providing Into
from a From
implementation.
If you're consuming one of these two traits, as a trait bound on a new trait of your own, then the advice is reversed:
use the Into
trait for trait bounds. That way, the bound will be satisfied both by things that directly implement
Into
, and by things that only directly implement From
.
This automatic conversion is highlighted by the documentation for From
and Into
, but it's worth reading the code
too:
impl<T, U> Into<U> for T
where
U: From<T>,
{
fn into(self) -> U {
U::from(self)
}
}
Translating a trait specification into words can help with understanding more complex trait bounds; in this case, it's
fairly simple: "I can implement Into<U>
for a type T
whenever U
already implements From<T>
".
It's also useful in general to look over the trait implementations for a standard library type. As you'd expect, there
are From
implementations for safe integral conversions (From<u32> for u64
) and TryFrom
implementations when
the conversion isn't safe (TryFrom<u64> from u32
).
There are also various blanket trait implementations. Into
just has the one shown above, but the From
trait has many
impl<T> From<T> for ...
clauses. These are almost all for smart pointer types, allowing the smart pointer to be
automatically constructed from an instance of the type that it holds, so that methods that accept smart pointer
parameters can also be called with plain old items; more on this below and in Item 9.
The TryFrom
trait also has a blanket implementation for any type that already implements the Into
trait in the
opposite direction – which automatically includes (as above) any type that implements From
in the same
direction. This conversion will always succeed, so the associated error type is 2 the helpfully named Infallible
.
There's also one very specific generic implementation of From
that sticks out, the reflexive implementation:
impl<T> From<T> for T {
fn from(t: T) -> T {
t
}
}
Translating into words, this just says that "given a T
I can get a T
". That's such an obvious "well, doh" that it's
worth stopping to understand why this is useful.
Consider a simple struct
and a function that operates on it (ignoring that this function would be better expressed as
a method):
/// Integer value from an IANA-controlled range.
#[derive(Clone, Copy, Debug)]
pub struct IanaAllocated(pub u64);
/// Indicate whether value is reserved.
pub fn is_iana_reserved(s: IanaAllocated) -> bool {
s.0 == 0 || s.0 == 65535
}
This function can be invoked with instances of the struct
let s = IanaAllocated(1);
println!("{:?} reserved? {}", s, is_iana_reserved(s));
but even if From<u64>
is implemented
impl From<u64> for IanaAllocated {
fn from(v: u64) -> Self {
Self(v)
}
}
it can't be directly invoked for u64
values
error[E0308]: mismatched types
--> casts/src/main.rs:75:25
|
75 | if is_iana_reserved(42) {
| ^^ expected struct `IanaAllocated`, found integer
However, a generic version of the function that accepts (and explicitly converts) anything satisfying
Into<IanaAllocated>
pub fn is_iana_reserved_anything<T>(s: T) -> bool
where
T: Into<IanaAllocated>,
{
let s = s.into();
s.0 == 0 || s.0 == 65535
}
allows this use:
if is_iana_reserved_anything(42) {
The reflexive trait implementation of From<T>
means that this generic function copes with items which are already
IanaAllocated
instances, no conversion needed.
This pattern also explains why (and how) Rust code sometimes appears to be doing implicit casts between types: the
combination of From<T>
implementations and Into<T>
trait bounds leads to code that appears to magically convert at
the call site (but which is still doing safe, explicit, conversions under the covers), This pattern becomes even more
powerful when combined with reference types and their related conversion traits; more in Item 9.
Casts
Rust includes the as
keyword to perform explicit
casts between some pairs
of types.
The pairs of types that can be converted in this way is a fairly limited set, and the only user-defined types it
includes are "C-like" enum
s (those that have an associated integer value). General integral conversions are included
though, giving an alternative to into()
:
let x: u32 = 9;
let y = x as u64;
let z: u64 = x.into();
The as
version also allows lossy conversions:
let x: u32 = 9;
let y = x as u16;
which would be rejected by the from
/ into
versions:
error[E0277]: the trait bound `u16: From<u32>` is not satisfied
--> casts/src/main.rs:113:20
|
113 | let y: u16 = x.into();
| ^^^^ the trait `From<u32>` is not implemented for `u16`
|
= help: the following implementations were found:
<u16 as From<NonZeroU16>>
<u16 as From<bool>>
<u16 as From<u8>>
= note: required because of the requirements on the impl of `Into<u16>` for `u32`
For consistency and safety you should prefer from
/ into
conversions to as
casts, unless you understand and
need the precise casting semantics (e.g
for C interoperability).
Coercion
The explicit as
casts described in the previous section are a superset of the implicit
coercions that the compiler will silently perform:
any coercion can be forced with an explicit as
, but the converse is not true. (In particular, the integral
conversions performed in the previous section are not coercions, and so will always require as
.)
Most of the coercions involve silent conversions of pointer and reference types in ways that are sensible and convenient for the programmer, such as:
- converting a mutable reference to a non-mutable references (so you can use a
&mut T
as the argument to a function that takes a&T
) - converting a reference to a raw pointer (this isn't
unsafe
– the unsafety happens at the point where you're foolish enough to use a raw pointer) - converting a closure that happens not to capture any variables into a bare function pointer (Item 2)
- converting an array to a slice
- converting a concrete item to a trait object, for a trait that the concrete item implements
- converting3 an item lifetime to a "shorter" one (Item 15).
There are only two coercions whose behaviour can be affected by user-defined types. The first of these is when a
user-defined type implements the Deref
or the
DerefMut
trait. These traits indicate that the user defined
type is acting as a smart pointer of some sort (Item 9), and in this case the compiler will coerce a reference to
the smart pointer item into being a reference to an item of the type that the smart pointer contains (indicated by
its Target
).
The second coercion of a user-defined type happens when a concrete item is converted to a trait object. This operation builds a fat pointer to the item; this pointer is fat because it also includes a pointer to the vtable for the concrete type's implementation of the trait – see Item 9.
1: More properly known as the trait coherence rules.
2: For now – this is
likely to be replaced with the !
"never" type in a future
version of Rust.
3: Rust refers to these conversions as "subtyping", but it's quite different that the definition of "subtyping" used in object-oriented languages.
Item 7: Embrace the newtype pattern
Item 1 described tuple structs, where the fields of a struct
have no names and are instead referred to by number
(self.0
). This Item focuses on tuple structs that have a single entry, which is a pattern that's sufficiently
pervasive in Rust that it deserves its own Item and has its own name: the newtype pattern.
The simplest use of the newtype pattern is to indicate additional semantics for a type, over and above its normal behaviour. To illustrate this, imagine a project that's going to send a satellite to Mars1. It's a big project, so different groups have built different parts of the project. One group has handled the code for the rocket engines:
/// Fire the thrusters. Returns generated force in Newton seconds.
pub fn thruster_impulse(direction: Direction) -> f64 {
// ...
return 42.0;
}
while a different group handles the inertial guidance system:
/// Update trajectory model for impulse, provided in pound force seconds.
pub fn update_trajectory(force: f64) {
// ...
}
Eventually these different parts eventually need to be joined together:
let thruster_force: f64 = thruster_impulse(direction);
let new_direction = update_trajectory(thruster_force);
Ruh-roh.
Rust includes a type alias feature, which allows the different groups to make their intentions clearer:
/// Units for force.
pub type NewtonSeconds = f64;
/// Fire the thrusters. Returns generated force.
pub fn thruster_impulse(direction: Direction) -> NewtonSeconds {
// ...
return 42.0;
}
/// Units for force.
pub type PoundForceSeconds = f64;
/// Update trajectory model for impulse.
pub fn update_trajectory(force: PoundForceSeconds) {
// ...
}
However, the type aliases are effectively just documentation; nothing stops a NewtonSeconds
value being used where a
PoundForceSeconds
value is expected:
let thruster_force: NewtonSeconds = thruster_impulse(direction);
let new_direction = update_trajectory(thruster_force);
Ruh-roh once more.
This is the point where the newtype pattern helps.
/// Units for force.
pub struct NewtonSeconds(pub f64);
/// Fire the thrusters. Returns generated force.
pub fn thruster_impulse(direction: Direction) -> NewtonSeconds {
// ...
return NewtonSeconds(42.0);
}
/// Units for force.
pub struct PoundForceSeconds(pub f64);
/// Update trajectory model for impulse.
pub fn update_trajectory(force: PoundForceSeconds) {
// ...
}
As the name implies, a newtype is a new type, and as such the compiler objects to type conversions:
let thruster_force: NewtonSeconds = thruster_impulse(direction);
let new_direction = update_trajectory(thruster_force);
error[E0308]: mismatched types
--> newtype/src/main.rs:76:43
|
76 | let new_direction = update_trajectory(thruster_force);
| ^^^^^^^^^^^^^^ expected struct `PoundForceSeconds`, found struct `NewtonSeconds`
The same pattern of using a newtype to mark additional "unit" semantics for a type can also help to make boolean arguments less ambiguous. Revisiting the example from Item 1, using newtypes makes the meaning of arguments clear:
struct DoubleSided(pub bool);
struct ColourOutput(pub bool);
fn safe_print(sides: DoubleSided, colour: ColourOutput) {
// ...
}
safe_print(DoubleSided(true), ColourOutput(false));
If size efficiency or binary compatibility is a concern, then the [repr(transparent)]
attribute
ensures that a newtype has the same representation in memory as the inner type.
That's the simple use of newtype, and it's a specific example of Item 1 – encoding semantics into the type system, so that the compiler takes care of policing those semantics.
Bypassing the orphan trait rule
The other common, but more subtle, scenario that requires the newtype pattern revolves around Rust's orphan trait rule. Roughly speaking, this says that you can only implement a trait for a type if:
- you own the trait, or
- you own the type.
Attempting to implement a foreign trait for a foreign type:
impl fmt::Display for rand::rngs::StdRng {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> Result<(), fmt::Error> {
write!(f, "<StdRng instance>")
}
}
leads to a compiler error (which in turn points the way back to newtypes).
error[E0117]: only traits defined in the current crate can be implemented for arbitrary types
--> newtype/src/main.rs:123:1
|
123 | impl fmt::Display for rand::rngs::StdRng {
| ^^^^^^^^^^^^^^^^^^^^^^------------------
| | |
| | `StdRng` is not defined in the current crate
| impl doesn't use only types from inside the current crate
|
= note: define and implement a trait or new type instead
The reason for this restriction is due to the risk of ambiguity: if two different crates in the dependency graph (Item 25) were both to (say) impl std::fmt::Display for rand::rngs::StdRng
, then the compiler/linker has no way to choose
between them.
This can frequently lead to frustration: for example, if you're trying to serialize data that includes a type from
another crate, the orphan crate rule prevents2 you from writing impl serde::Serialize for somecrate::SomeType
.
The newtype pattern means that you're creating a new type, which you own and so the second bullet above applies. Implementing a foreign trait is now possible:
struct MyRng(rand::rngs::StdRng);
impl fmt::Display for MyRng {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> Result<(), fmt::Error> {
write!(f, "<Rng instance>")
}
}
Newtype limitations
The newtype pattern solves these two classes of problems – preventing unit conversions and bypassing the orphan trait rule – but it does come with some awkwardness: every operation that involves the newtype needs to forward to the inner type.
On a trivial level that means that the code has thing.0
throughout, rather than just thing
, but that's easy and the
compiler will tell you where it's needed.
The more significant awkwardness is that any trait implementations on the inner type are lost: because the newtype is a new type, the existing inner implementation doesn't apply.
For derivable traits this just means that the newtype declaration ends up with lots of derive
s:
#[derive(Debug, Copy, Clone, Eq, PartialEq, Ord, PartialOrd)]
pub struct NewType(InnerType);
However, for more sophisticated traits some forwarding boilerplate is needed to recover the inner type's implemenation, for example:
use std::fmt;
impl fmt::Display for NewType {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> Result<(), fmt::Error> {
self.0.fmt(f)
}
}
1: Specifically, the Mars Climate Orbiter.
2: This is a sufficiently common problem for serde
that it
includes a mechanism to help.
Item 8: Use builders for complex types
Rust insists that all fields in a struct
must be filled in when a new instance of that struct
is created. This keeps
the code safe, but does lead to more verbose, boilerplate code than is ideal.
#[derive(Debug, Default)]
struct BaseDetails {
given_name: String,
preferred_name: Option<String>,
middle_name: Option<String>,
family_name: String,
mobile_phone_e164: Option<String>,
}
let dizzy = BaseDetails {
given_name: "Dizzy".to_owned(),
preferred_name: None,
middle_name: None,
family_name: "Mixer".to_owned(),
mobile_phone_e164: None,
};
This boilerplate code is also brittle: a future change that adds a new field to the struct
would require an update to
every place that builds the structure.
The boilerplate can be significantly reduced by implementing and using the
Default
trait, as described in Item 5:
let dizzy = BaseDetails {
given_name: "Dizzy".to_owned(),
family_name: "Mixer".to_owned(),
..Default::default()
};
This also helps when a new field is added, provided that the new field is itself of a type that implements Default
.
However, this is only straightforward if all of the field types also implement the Default
trait. If
there's a field that doesn't play along, an automatically derived implementation of Default
isn't possible:
#[derive(Debug, Default)]
struct Details {
given_name: String,
preferred_name: Option<String>,
middle_name: Option<String>,
family_name: String,
mobile_phone_e164: Option<String>,
dob: chrono::Date<chrono::Utc>,
last_seen: Option<chrono::DateTime<chrono::Utc>>,
}
error[E0277]: the trait bound `Date<Utc>: Default` is not satisfied
--> builders/src/main.rs:171:9
|
171 | dob: chrono::Date<chrono::Utc>,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Default` is not implemented for `Date<Utc>`
|
= note: required by `std::default::Default::default`
= note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)
As a result, all of the fields have to be filled out manually:
use chrono::TimeZone;
let bob = Details {
given_name: "Robert".to_owned(),
preferred_name: Some("Bob".to_owned()),
middle_name: Some("the".to_owned()),
family_name: "Builder".to_owned(),
mobile_phone_e164: None,
dob: chrono::Utc.ymd(1998, 11, 28),
last_seen: None,
};
These ergonomics can be improved if you implement the builder pattern for complex data structures.
The simplest variant of the builder pattern is a separate struct
that holds the information needed to construct the
item. For simplicity, the example will hold an instance of the item itself.
struct DetailsBuilder(Details);
impl DetailsBuilder {
/// Start building a new [`Details`] object.
fn new(
given_name: &str,
family_name: &str,
dob: chrono::Date<chrono::Utc>,
) -> Self {
DetailsBuilder(Details {
given_name: given_name.to_owned(),
preferred_name: None,
middle_name: None,
family_name: family_name.to_owned(),
mobile_phone_e164: None,
dob,
last_seen: None,
})
}
The builder type can then be equipped with helper methods that fill out the nascent item's fields. Each such method
consumes self
but emits a new Self
, allowing different construction methods to be chained.
/// Set the preferred name.
fn preferred_name(mut self, preferred_name: &str) -> Self {
self.0.preferred_name = Some(preferred_name.to_owned());
self
}
These helper methods can be more helpful than just simple setters:
/// Update the `last_seen` field to the current date/time.
fn just_seen(mut self) -> Self {
self.0.last_seen = Some(chrono::Utc::now());
self
}
The final method to be invoked for the builder consumes the builder and emits the built item.
/// Consume the builder object and return a fully built [`Details`] object.
fn build(self) -> Details {
self.0
}
Overall, this allows clients of the builder to have a more ergonomic building experience:
let also_bob =
DetailsBuilder::new("Robert", "Builder", chrono::Utc.ymd(1998, 11, 28))
.middle_name("the")
.preferred_name("Bob")
.just_seen()
.build();
The all-consuming nature of this style of builder leads to a couple of wrinkles. The first is that separating out stages of the build process can't be done on its own:
let builder = DetailsBuilder::new(
"Robert",
"Builder",
chrono::Utc.ymd(1998, 11, 28),
);
if informal {
builder.preferred_name("Bob");
}
let bob = builder.build();
error[E0382]: use of moved value: `builder`
--> builders/src/main.rs:238:19
|
230 | let builder = DetailsBuilder::new(
| ------- move occurs because `builder` has type `DetailsBuilder`, which does not implement the `Copy` trait
...
236 | builder.preferred_name("Bob");
| --------------------- `builder` moved due to this method call
237 | }
238 | let bob = builder.build();
| ^^^^^^^ value used here after move
|
note: this function consumes the receiver `self` by taking ownership of it, which moves `builder`
--> builders/src/main.rs:49:27
|
49 | fn preferred_name(mut self, preferred_name: &str) -> Self {
| ^^^^
This can be worked around by assigning the consumed builder back to the same variable:
let mut builder =
DetailsBuilder::new("Robert", "Builder", chrono::Utc.ymd(1998, 11, 28));
if informal {
builder = builder.preferred_name("Bob");
}
let bob = builder.build();
The other downside to the all-consuming nature of this builder is that only one item can be built; trying to
repeatedly build()
let smithy =
DetailsBuilder::new("Agent", "Smith", chrono::Utc.ymd(1999, 6, 11));
let clones = vec![smithy.build(), smithy.build(), smithy.build()];
falls foul of the borrow checker, as you'd expect:
error[E0382]: use of moved value: `smithy`
--> builders/src/main.rs:258:43
|
256 | let smithy =
| ------ move occurs because `smithy` has type `DetailsBuilder`, which does not implement the `Copy` trait
257 | DetailsBuilder::new("Agent", "Smith", chrono::Utc.ymd(1999, 6, 11));
258 | let clones = vec![smithy.build(), smithy.build(), smithy.build()];
| ------- ^^^^^^ value used here after move
| |
| `smithy` moved due to this method call
An alternative approach is for the builder's methods to take a &mut self
and emit a &mut Self
:
/// Update the `last_seen` field to the current date/time.
fn just_seen(&mut self) -> &mut Self {
self.0.last_seen = Some(chrono::Utc::now());
self
}
which removes the need for this self-assignment in separate build stages:
let mut builder = DetailsRefBuilder::new(
"Robert",
"Builder",
chrono::Utc.ymd(1998, 11, 28),
);
if informal {
builder.preferred_name("Bob"); // no `builder = ...`
}
let bob = builder.build();
However, this version makes it impossible to chain the construction of the builder together with invocation of its setter methods:
let builder = DetailsRefBuilder::new(
"Robert",
"Builder",
chrono::Utc.ymd(1998, 11, 28),
)
.middle_name("the")
.just_seen();
error[E0716]: temporary value dropped while borrowed
--> builders/src/main.rs:278:23
|
278 | let builder = DetailsRefBuilder::new(
| _______________________^
279 | | "Robert",
280 | | "Builder",
281 | | chrono::Utc.ymd(1998, 11, 28),
282 | | )
| |_________^ creates a temporary which is freed while still in use
283 | .middle_name("the")
284 | .just_seen();
| - temporary value is freed at the end of this statement
285 | // ANCHOR_END: refbuildchainfail
286 | let bob = builder.build();
| ------- borrow later used here
|
= note: consider using a `let` binding to create a longer lived value
As indicated by the compiler error, this can be worked around by let
ting the builder item have a name:
let mut builder = DetailsRefBuilder::new(
"Robert",
"Builder",
chrono::Utc.ymd(1998, 11, 28),
);
builder.middle_name("the").just_seen();
if informal {
builder.preferred_name("Bob");
}
let bob = builder.build();
This builder variant also allows for building multiple items. The signature of the build()
method has to not consume
self, and so must be:
/// Construct a fully built [`Details`] object.
fn build(&self) -> Details {
The implementation of this repeatable build()
method then has to construct a fresh item on each invocation. If the
underlying item implements Clone
, this is easy – the builder can hold a template and clone()
it for each
build. If the underlying item doesn't implement Clone
, then the builder needs to have enough state to be able to
manually construct an instance of the underlying item on each call to build()
.
With any style of builder pattern, the boilerplate code is now confined to one place – the builder – rather than being needed at every place that uses the underlying type.
The boilerplate that remains can potentially be reduced still further by use of a macro (Item 28), but if you go down
this road you should also check whether there's an existing crate (such as the
derive_builder
crate in particular) that provides what's needed – assuming
that you're happy to take a dependency on it (Item 25).
Item 9: Familiarize yourself with reference and pointer types
A pointer is just a number, whose value is the address in memory of some other object. In source code, the type of
the pointer encodes information about the type of the object being pointed to, so a program knows how to interpret the
contents of memory at that address. It's possible to play fast and loose with these constraints with raw pointers, but
they are very unsafe
(Item 16) and beyond the scope of this book.
Simple Pointer Types
The most ubiquitous pointer type in Rust is the reference &T
. Although this is a pointer value, the compiler ensures
that various rules are observed: it must always point to a valid, correctly-aligned instance of the relevant type, and
the borrow checking rules must be followed (Item 14). These additional constraints are roughly similar to the
constraints that C++ has when dealing with references rather than pointers; however, C++ allows footguns1 with dangling references:
// C++
const int& dangle() {
int x = 32; // on the stack, overwritten later
return x; // return reference to stack variable!
}
Rust's borrowing and lifetime checks make the equivalent code broken at compile time:
fn dangle() -> &'static i64 {
let x: i64 = 32; // on the stack
&x
}
error[E0515]: cannot return reference to local variable `x`
--> references/src/main.rs:384:5
|
384 | &x
| ^^ returns a reference to data owned by the current function
A Rust reference is a simple pointer, 8 bytes in size on a 64-bit platform (which this Item assumes throughout):
struct Point {
x: u32,
y: u32,
}
let pt = Point { x: 1, y: 2 };
let x = 0u64;
let ref_x = &x;
let ref_pt = &pt;
Rust allocates items on the stack by default; the Box<T>
pointer type (roughly equivalent to C++'s
std::unique_ptr<T>
) forces allocation to occur on the heap, which in turn means that the allocated item can outlive
the scope of the current block. Under the covers, Box<T>
is also a simple 8 byte pointer value.
let box_pt = Box::new(Point { x: 10, y: 20 });
Pointer Traits
A method that expects a reference argument like &Point
can also be fed a &Box<Point>
:
fn show(pt: &Point) {
println!("({}, {})", pt.x, pt.y);
}
show(ref_pt);
show(&box_pt);
(1, 2)
(10, 20)
This is possible because Box<T>
implements the Deref
trait,
with Target = T
. The Rust compiler looks for and uses implementations of this trait when it's dealing with
dereferences (*x
),
allowing coercion of types (Item 6).
There's also an equivalent DerefMut
for when a mutable
reference is involved.
The compiler has to deduce a unique type for an expression like *x
, which means that the Deref
traits can't be
generic (Deref<T>
): that would open up the possibility that a user-defined type could implement both Deref<TypeA>
and Deref<TypeB>
, leaving the compiler with a choice of TypeA
or TypeB
. Instead, the underlying type is an
associated type named Target
instead.
In contrast, the AsRef
and
AsMut
traits encode their destination type as a type
parameter, such as AsRef<Point>
, allowing a single container type to support multiple destinations. For example, the
String
type implements
Deref
withTarget = str
, meaning that an expression like&my_string
can be coerced to type&str
.AsRef<[u8]>
, allowing conversion to a byte slice&[u8]
.AsRef<OsStr>
, allowing conversion to an OS string.AsRef<Path>
, allowing conversion to a filesystem path.AsRef<str>
, as forDeref
A function that takes a reference can therefore be made even more general, by making the function generic over one of these traits. This means it accepts the widest range of reference-like types:
fn show_as_ref<T: AsRef<Point>>(pt: T) {
let pt = pt.as_ref();
println!("({}, {})", pt.x, pt.y);
}
Fat Pointer Types
Rust has two built-in fat pointer types: types that act as pointers, but which hold additional information about the thing they are pointing to.
The first such type is the slice: a reference to a subset of some contiguous collection of values. It's built from a
(non-owning) simple pointer, together with a length field, making it twice the size of a simple pointer (16 bytes on a
64-bit platform). The type of a slice is written as &[T]
– a reference to [T]
, which is the notional type for
a contiguous collection of values of type T
.
The notional type [T]
can't be instantiated, but there are two common containers that embody it. The first is the
array: a contiguous collection of values whose size is known at compile time. A slice can therefore refer to a subset
of an array:
let array = [0u64; 5];
let slice = &array[1..3];
The other common container for contiguous values is a Vec<T>
. This holds a contiguous collection of values whose size
can vary, and whose contents are held on the heap. A slice can therefore refer to a subset of a vector:
let mut vec = Vec::<u64>::with_capacity(8);
for i in 0..5 {
vec.push(i);
}
let slice = &vec[1..3];
There's quite a lot going on under the covers for the expression &vec[1..3]
:
- The
1..3
part is a range expression; the compiler converts this into an instance of theRange<usize>
type.- The
Range
type implements theSliceIndex<T>
trait, which describes indexing operations on slices of an arbitrary typeT
(so theOutput
type is[T]
).
- The
- The
vec[ ]
part is an indexing expression; the compiler converts this into an invocation of theIndex
trait'sindex
method onvec
, together with a dereference (i.e.*vec.index( )
). (The equivalent trait for mutable expressions isIndexMut
). vec[1..3]
therefore invokesVec<T>
's implementation ofIndex<I>
, which requiresI
to be an instance ofSliceIndex<[u64]>
. This works becauseRange<usize>
implementsSliceIndex<[T]>
for anyT
, includingu64
.&vec[1..3]
un-does the dereference, resulting in a final expression type of&[u64]
.
The second built-in fat pointer type is a trait object: a reference to some item that implements a particular trait. It's built from a simple pointer to the item, together with an internal pointer to the type's vtable, giving a size of 16 bytes (on a 64-bit platform). The vtable for a type's implementation of a trait holds function pointers for each of the method implementations, allowing dynamic dispatch at runtime (Item 12)2.
So a simple trait:
trait Calculate {
fn add(&self, l: u64, r: u64) -> u64;
fn mul(&self, l: u64, r: u64) -> u64;
}
with a struct
that implements it:
struct Modulo(pub u64);
impl Calculate for Modulo {
fn add(&self, l: u64, r: u64) -> u64 {
(l + r) % self.0
}
fn mul(&self, l: u64, r: u64) -> u64 {
(l * r) % self.0
}
}
let mod3 = Modulo(3);
can be converted to a trait object of type &dyn Trait
(where the dyn
keyword highlights the fact that dynamic dispatch is involved):
// Need an explicit type to force dynamic dispatch.
let tobj: &dyn Calculate = &mod3;
let result = tobj.add(2, 2);
assert_eq!(result, 1);
Code that holds a trait object can invoke the methods of the trait via the function pointers in the vtable, passing in
the item pointer as the &self
parameter; see Item 12 for more information and advice.
Other Pointer Traits
A previous section described several traits (Deref[Mut]
, AsRef[Mut]
and Index
) that are used when dealing with
reference and slice types. There are a few more that can also come into play when working with various pointer types,
whether from the standard library or user defined.
The simplest is the Pointer
trait, which formats a pointer
value for output. This can be helpful for low-level debugging, and the compiler will reach for this trait automatically
when it encounters the {:p}
format specifier.
The Borrow
and
BorrowMut
traits each have a single method
(borrow
and
borrow_mut
respectively) that has the
same signature as the equivalent AsRef
/ AsMut
trait methods.
However, the difference between them is still visible in the type system, because they have different blanket implementations for references to arbitrary types:
- For
&T
:impl<'_, T, U> AsRef<U> for &'_ T
impl<'_, T> Borrow<T> for &'_ T
- For
&mut T
:impl<'_, T, U> AsRef<U> for &'_ mut T
impl<'_, T> Borrow<T> for &'_ mut T
but Borrow
also has a blanket implementation for (non-reference) types:
impl<T> Borrow<T> for T
This means that a method accepting the Borrow
trait can cope equally with instances of T
as well as references-to-T
:
fn add_four<T: Borrow<i32>>(v: T) -> i32 {
v.borrow() + 4
}
assert_eq!(add_four(&2), 6);
assert_eq!(add_four(2), 6);
The standard library's container types have more realistic uses of Borrow
; for example,
HashMap::get
uses Borrow
to allow
convenient retrieval of entries whether keyed by value or by reference.
Finally, the ToOwned
trait builds on the Borrow
trait,
adding a to_owned()
method that produces
a new owned item of the underlying type, like Clone
. This means that:
- A function that accepts
Borrow
can receive either items or references-to-items, and can work with references in either case. - A function that accepts
ToOwned
can receive either items or references-to-items, and can build its own personal copies of those items in either case.
Smart Pointer Types
The Rust standard library includes a variety of types that act like pointers to some degree or another, mediated (as
usual, Item 5) by the standard traits described above. These smart pointer types each come with some particular
semantics and guarantees, which has the advantage that the right combination of them can give fine-grained control over
the pointer's behaviour, but has the disadvantage that the resulting types can seem overwhelming at first
(Rc<RefCell<Vec<T>>>
anyone?).
The first smart pointer type is Rc<T>
, which is a reference-counted
pointer to an item (roughly analogous to C++'s
std::shared_ptr<T>
). It implements all of the pointer-related
traits, so acts like a Box<T>
is many ways.
This is useful for data structures where the same item can be reached in different ways, but it removes one of Rust's
core rules around ownership – that each item has only one owner. Relaxing this rule means that it is now
possible to leak data: if item A has an Rc
pointer to item B, and item B has an Rc
pointer to A, then the pair
will never be dropped. To put it another way: you need Rc
to support cyclical data structures, but the downside is
that there are now cycles in your data structures.
The risk of leaks can be ameliorated in some cases by the related
Weak<T>
type, which holds a non-owning reference to the
underlying item (roughly analogous to C++'s std::weak_ptr<T>
).
Holding a weak reference doesn't prevent the underlying item being dropped (when all strong references are removed),
so making use of the Weak<T>
involves an upgrade to an Rc<T>
– which can fail.
Under the hood, Rc
is (currently) implemented as pair of reference counts together with the referenced items, all
stored on the heap.
let rc1: Rc<u64> = Rc::new(42);
let rc2 = rc1.clone();
let wk = Rc::downgrade(&rc1);
The next smart pointer type RefCell<T>
relaxes the rule
(Item 14) that an item can only be mutated by its owner or by code that holds the (only) mutable reference to the
item. This interior mutability allows for greater flexibility – for example, allowing trait implementations
that mutate internals even when the method signature only allows &self
. However, it also incurs costs: as well as the
extra storage overhead (an extra isize
to track current borrows), the normal borrow checks are moved from compile-time
to run-time.
let rc: RefCell<u64> = RefCell::new(42);
let b1 = rc.borrow();
let b2 = rc.borrow();
The run-time nature of these checks means that the RefCell
user has to choose between two options, neither pleasant:
- Accept that borrowing is an operation that might fail, and cope with
Result
values fromtry_borrow[_mut]
- Use the allegedly-infallible borrowing methods
borrow[_mut]
, and accept the risk of apanic!
at runtime if the borrow rules have not been complied with.
In either case, this run-time checking means that RefCell
itself implements none of the standard pointer traits;
instead, its access operations return a Ref<T>
or
RefMut
smart pointer type that does implement those traits.
If the underlying type T
implements the Copy
trait (indicating that a fast bit-for-bit copy produces a valid item,
see Item 5), then the Cell<T>
type allows interior mutation with less overhead – the get(&self)
method
copies out the current value, and the set(&self, val)
method copies in a new value. The Cell
type is used
internally by both the Rc
and RefCell
implementations, for shared tracking of counters that can be mutated without a
&mut self
.
The smart pointer types described so far are only suitable for single threaded use; their implementations assume that there is no concurrent access to their internals. If this is not the case, then different smart pointers are needed, which include the additional synchronization overhead.
The thread-safe equivalent of Rc<T>
is Arc<T>
, which uses
atomic counters to ensure that the reference counts remain accurate. Like Rc
, Arc
implements all of the various
pointer-related traits.
However, Arc
on its own does not allow any kind of mutable access to the underlying item. This is covered by the
Mutex
type, which ensures that only one thread has access
– whether mutably or immutably – to the underlying item. As with RefCell
, Mutex
itself does not
implement any pointer traits, but its lock()
operation returns an value that does
(MutexGuard
, which implements Deref[Mut]
).
If there are likely to be more readers than writers, the
RwLock
type is preferable, as it allows multiple readers
access to the underlying item in parallel, provided that there isn't currently a (single) writer.
In either case, Rust's borrowing and threading rules force the use of one of these synchronization containers in multi-threaded code (but this only guards against some of the problems of shared-state concurrency; see Item 17).
The same strategy – see what the compiler rejects, and what it suggests instead – can be sometimes be applied with the other smart pointer types; however, it's faster and less frustrating to understand what the behaviour of the different smart pointers implies. To borrow3 an example from the first edition of the Rust book,
Rc<RefCell<Vec<T>>>
holds a vector (Vec
) with shared ownership (Rc
), where the vector can be mutated – but only as a whole vector.Rc<Vec<RefCell<T>>>
also holds a vector with shared ownership, but here each individual entry in the vector can be mutated independently of the others.
The types involved precisely describe these behaviours.
1: Albeit with a warning from modern compilers.
2: This is somewhat simplified; a full vtable also includes information about the size and
alignment of the type, together with a drop()
function pointer so that the underlying object can be safely dropped.
3: Pun intended
Item 10: Consider using iterator transforms instead of explicit loops
The humble loop has had a long journey of increasing convenience and increasing abstraction.
The B language (the
precursor to C) just had while (condition) { }
, but with C the common scenario of iterating through indexes of
an array was made more convenient with the addition of the for
loop:
// C code
int i;
for (i = 0; i < len; i++) {
Item item = collection[i];
// body
}
The early versions of C++ improved convenience and scoping further by allowing the loop variable declaration to be embedded in
the for
statement (and this was also adopted by C in C99):
// C++98 code
for (int i = 0; i < len; i++) {
Item item = collection[i];
// ...
}
Most modern languages abstract the idea of the loop further: the core function of a loop is often to move to the next
item of some container, and tracking the logistics that are required to reach that item (index++
or ++it
) is mostly
an irrelevant detail. This realization produced two core concepts:
- Iterators: a type whose purpose is to repeatedly emit the next item of a container1, until exhausted.
- For-Each Loops: a compact loop expression for iterating over all of the items in a container, binding a loop variable to the item rather than to the details of reaching that item.
These concepts allow for loop code that's shorter, and (more importantly) clearer about what's intended:
// C++11 code
for (Item& item : collection) {
// ...
}
Once these concepts were available, they were so obviously powerful that they were quickly retrofitted to those languages that didn't already have them (e.g. for-each loops were added to Java 1.5 and C++11).
Rust includes iterators and for-each style loops, but it also includes the next step in abstraction: allowing the whole
loop to be expressed as an iterator transform. As with Item 3's discussion of Option
and Result
, this Item will
attempt to show how these iterator transforms can be used instead of explicit loops, and to give guidance as to when
it's a good idea.
By the end of this Item, a C-like explicit loop to sum the squares of the first five even items of a vector:
let mut even_sum_squares = 0;
let mut even_count = 0;
for i in 0..values.len() {
if values[i] % 2 != 0 {
continue;
}
even_sum_squares += values[i] * values[i];
even_count += 1;
if even_count == 5 {
break;
}
}
should start to feel more natural expressed as a functional style expression:
let even_sum_squares: u64 = values
.iter()
.filter(|x| *x % 2 == 0)
.take(5)
.map(|x| x * x)
.sum();
Iterator transformation expressions like this can roughly be broken down into three parts:
- An initial source iterator, from one of Rust's iterator traits.
- A sequence of iterator transforms.
- A final consumer method to combine the results of the iteration into a final value.
The first two of these effectively move functionality out of the loop body and into the for
expression; the last
removes the need for the for
statement altogether.
Iterator Traits
The core Iterator
trait has a very simple interface: a
single method next
that yields Some
items
until it doesn't (None
).
Collections that allow iteration over their contents – called iterables – implement the
IntoIterator
trait; the
into_iter
method of this trait
consumes Self
and emits an Iterator
in its stead. The compiler will automatically use this trait for
expressions of the form
for item in collection {
// body
}
effectively converting them to code roughly like:
let mut iter = collection.into_iter();
loop {
let item: Thing = match iter.next() {
Some(item) => item,
None => break,
};
// body
}
(To keep things running smoothly, there's also an implementation of IntoIterator
for any Iterator
, which just
returns self
; after all, it's easy to convert an Iterator
into an Iterator
!)
This initial form is a consuming iterator, using up the collection as it's created:
for item in collection {
println!("Consumed item {:?}", item);
}
Any attempt to use the collection after it's been iterated over fails:
println!("Collection = {:?}", collection);
error[E0382]: borrow of moved value: `collection`
--> iterators/src/main.rs:117:35
|
106 | let collection = vec![Thing(0), Thing(1), Thing(2), Thing(3)];
| ---------- move occurs because `collection` has type `Vec<Thing>`, which does not implement the `Copy` trait
...
110 | for item in collection {
| ----------
| |
| `collection` moved due to this implicit call to `.into_iter()`
| help: consider borrowing to avoid moving into the for loop: `&collection`
...
117 | println!("Collection = {:?}", collection);
| ^^^^^^^^^^ value borrowed here after move
|
note: this function consumes the receiver `self` by taking ownership of it, which moves `collection`
While simple to understand, this all-consuming behaviour is often undesired; some kind of borrow of the iterated items is needed.
To ensure that behaviour is clear, the examples here onwards use an Item
type that is not Copy
(Item 5), as
that would hide questions of ownership (Item 14) – the compiler would silently make copies everywhere.
// Deliberately not `Copy`
#[derive(Clone, Debug, Eq, PartialEq)]
struct Thing(u64);
let collection = vec![Thing(0), Thing(1), Thing(2), Thing(3)];
If the collection being iterated over is prefixed with &
:
for item in &collection {
println!("{}", item.0);
}
println!("collection still around {:?}", collection);
then the Rust compiler will look for an implementation of
IntoIterator
for the type &'a Collection
. Properly
designed collection types will provide such an implementation; this implementation will still consume Self
, but now
Self
is &Collection
rather than Collection
, and the associated Item
type will be a reference &'a Thing
.
This leaves the collection intact after iteration, and the equivalent expanded code is:
let mut iter = (&collection).into_iter();
loop {
let item: &Thing = match iter.next() {
Some(item) => item,
None => break,
};
println!("{}", item.0);
}
If it makes sense to provide iteration over mutable references2, then a
similar pattern applies for for item in &mut collection
: provide an implementation of IntoIterator
for &'a mut Collection
, with Item
of &'a mut Thing
.
By convention, standard containers also provide an iter()
method that returns an iterator over references to the
underlying item, and equivalent an iter_mut()
method if appropriate, with the same behaviour as just described. These
methods can be used in for
loops, but have a more obvious benefit when used as the start of an iterator
transformation:
let result: u64 = (&collection).into_iter().map(|thing| thing.0).sum();
becomes:
let result: u64 = collection.iter().map(|thing| thing.0).sum();
Iterator Transforms
The Iterator
trait has a single required method
(next
), but also provides default
implementations (Item 13) of a large number of other methods that perform transformations on an iterator.
Some of these tranformations affect the overall iteration process:
take(n)
restricts an iterator to emitting at mostn
items.skip(n)
skips over the firstn
elements of the iterator.step_by(n)
converts an iterator so it only emits every n-th item.chain(other)
glues together two iterators, to build a combined iterator that moves through one then the other.cycle()
converts an iterator that terminates into one that repeats forever, starting at the beginning again whenever it reaches the end. (The iterator must supportClone
to allow this.)rev()
reverses the direction of an iterator. (The iterator must implement theDoubleEndedIterator
trait, which has an additionalnext_back
required method.)
Other transformations affect the nature of the Item
that's the subject of the Iterator
:
map(|item| {...})
is the most general version, repeatedly applying a closure to transform each item in turn. Several of the following entries in this list are convenience variants that could be equivalently implemented as amap
.cloned()
produces a clone of all of the items in the original iterator; this is particularly useful with iterators over&Item
references. (This obviously requires the underlyingItem
type to implementClone
).copied()
produces a copy of all of the items in the original iterator; this is particularly useful with iterators over&Item
references. (This obviously requires the underlyingItem
type to implementCopy
).enumerate()
converts an iterator over items to be an iterator over(usize, Item)
pairs, providing an index to the items in the iterator.zip(it)
joins an iterator with a second iterator, to produce a combined iterator that emits pairs of items, one from each of the original iterators, until the shorter of the two iterators is finished.
Yet other transformations perform filtering on the Item
s being emitted by the Iterator
:
filter(|item| {...})
is the most general version, applying abool
-returning closure to each item reference to determine whether it should be passed through.take_while()
andskip_while()
are mirror images of each other, emitting either an initial subrange or a final subrange of the iterator, based on a predicate.
The flatten()
method deals with an iterator
whose items are themselves iterators, flattening the result. On its own, this doesn't seem that helpful, but it becomes
much more useful when combined with the observation that both
Option
and Result
act as iterators: they produce either
zero (for None
, Err(e)
) or one (for Some(v)
, Ok(v)
) items. This means that flatten
ing a stream of Option
/
Result
values is a simple way to extract just the valid values, ignoring the rest.
Taken as a whole, these methods allow iterators to be transformed so that they produce exactly the sequence of elements that are needed for most situations.
Iterator Consumers
The previous two sections described how to obtain an iterator, and how to transmute it into exactly the right form for precise iteration. This precisely-targetted iteration could happen as an explicit for-each loop:
let mut even_sum_squares = 0;
for value in values.iter().filter(|x| *x % 2 == 0).take(5) {
even_sum_squares += value * value;
}
However, the Iterator
documentation includes many additional
methods that allow an iteration to be consumed in a single method call, removing the need for an explicit for
loop.
The most general of these methods is for_each(|item| {...})
, which runs a closure for each item
produced by the Iterator
. This can do most of the things that an explicit for
loop could do (the exceptions are
described below), but for this general form an explicit loop is still preferable.
However, if the body of the for
loop matches one of a number of common patterns, the iterator-consuming method is
clearer. shorter and more idiomatic. These patterns include shortcuts for building a single value out of the
collection:
sum()
, for summing a collection of "algebraic" values (i.e. those that have the relevant operator overloads, Item 5).product()
, for multiplying together a collection of algebraic values.min()
andmax()
, for finding the extreme values of a collection, relative to theItem
'sPartialOrd
implementation (Item 5).min_by(f)
andmax_by(f)
, for finding the extreme values of a collection, relative to a user-specified comparison function.reduce(f)
is a more general operation that encompasses the previous methods, building an accumulated value of theItem
type by running a closure at each step that takes the value accumulated so far and the current item.fold(f)
is a generalization ofreduce
, allowing the "accumulated value" to be of an arbitrary type (not just theIterator::Item
type).scan(f)
generalizes in a slightly different way, giving the closure a mutable reference to some internal state at each step.
There are also methods for selecting a single value out of the collection:
find(p)
finds the first item that satisfies a predicate.position(p)
also finds the first item satisfying a predicate, but this time it returns the index of the item.nth(n)
returns the n-th element of the iterator, if available.
There are methods for testing against every item in the collection:
any(p)
indicates whether a predicate istrue
for any item in the collection.all(p)
indicates whether a predicate istrue
for all items in the collection.
There are methods that allow for the possibility of failure in the closures used with each items; in each case, if a closure returns a failure for an item, the iteration is terminated and the operation as a whole returns the first failure.
try_for_each(f)
is likefor_each
, but the closure can fail.try_fold(f)
is likefold
, but the closure can fail.try_find(f)
is likefind
, but the closure can fail.
Finally, there are methods that accumulate all of the iterated items into a new collection. The most important of these
is collect()
, which can be used to build a
new instance of any collection type that implements the
FromIterator
trait.
The FromIterator
trait is implemented for all of the standard library collection types
(Vec
,
HashMap
,
BTreeSet
etc.) but
this ubiquity also means that you often have to use explicit types, because otherwise the
compiler can't figure out whether you're trying to assemble (say) a Vec<i32>
or HashSet<i32>
.
// Build vector of even numbers: [0, 2, 4, 6, 8]
// Indicate collection type explicitly.
let v: Vec<i32> = (0..10).into_iter().filter(|x| x % 2 == 0).collect();
(As an aside, this example also illustrates the use of range expressions to generate the initial data to be iterated over.)
Other (more obscure) collection-producing methods include:
unzip()
, which divides an iterator of pairs into two collections.partition(p)
, which splits an iterator into two collections based on a predicate that is applied to each item.
This Item has touched on a wide selection of Iterator
methods, but this is only a subset of the methods available;
for more information, consult the iterator documentation or
read Chapter 15 of Programming Rust (2nd edition), which has extensive coverage of the possibilities.
This rich collection of iterator transformations is meant to be used, to produce code that is more idiomatic, more compact, and with clearer intent.
Building Collections From Result
Values
The previous section described the use of collect()
to build collections from iterators, but collect()
also has a
particularly helpful feature when dealing with Result
values.
Consider an attempt to convert a vector of i64
values into bytes, with the optimistic expectation that they will all
fit:
use std::convert::TryFrom;
let inputs: Vec<i64> = vec![0, 1, 2, 3, 4];
let result: Vec<u8> = inputs
.into_iter()
.map(|v| <u8>::try_from(v).unwrap())
.collect();
This works until some unexpected input comes along:
let inputs: Vec<i64> = vec![0, 1, 2, 3, 4, 512];
and causes a run-time failure:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: TryFromIntError(())', iterators/src/main.rs:206:36
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Following the advice of Item 3, we want to keep the Result
type in play, and use the ?
operator to make any
failure the problem of the calling code. The obvious modification to emit the Result
doesn't really help:
let result: Vec<Result<u8, _>> =
inputs.into_iter().map(|v| <u8>::try_from(v)).collect();
// Now what? Still need to iterate to extract results and detect errors.
However, there's an alternative version of collect()
, which can assemble a Result
holding a Vec
, instead of a
Vec
holding Result
s.
Forcing use of this version requires the turbofish (::<Result<Vec<_>, _>>
):
let result: Vec<u8> = inputs
.into_iter()
.map(|v| <u8>::try_from(v))
.collect::<Result<Vec<_>, _>>()?;
Combining this with the question mark operator gives useful behaviour:
- If the iteration encounters an error value, that error value is emitted to the caller and iteration stops.
- If no errors are encountered, the remainder of the code can deal with a sensible collection of values of the right type.
Loop Transformation
The aim of this Item is to convince you that many explicit loops can be regarded as something to be converted to iterator transformations. This can feel somewhat unnatural for programmers who aren't used to it, so let's walk through a transformation step by step.
Starting with a very C-like explicit loop to sum the squares of the first five even items of a vector:
let mut even_sum_squares = 0;
let mut even_count = 0;
for i in 0..values.len() {
if values[i] % 2 != 0 {
continue;
}
even_sum_squares += values[i] * values[i];
even_count += 1;
if even_count == 5 {
break;
}
}
The first step is to replace vector indexing with direct use of an iterator in a for-each loop:
let mut even_sum_squares = 0;
let mut even_count = 0;
for value in values.iter() {
if value % 2 != 0 {
continue;
}
even_sum_squares += value * value;
even_count += 1;
if even_count == 5 {
break;
}
}
An initial arm of the loop that uses continue
to skip over some items is naturally expressed as a filter()
:
let mut even_sum_squares = 0;
let mut even_count = 0;
for value in values.iter().filter(|x| *x % 2 == 0) {
even_sum_squares += value * value;
even_count += 1;
if even_count == 5 {
break;
}
}
Next, the early exit from the loop once 5 even items have been spotted maps to a take(5)
:
let mut even_sum_squares = 0;
for value in values.iter().filter(|x| *x % 2 == 0).take(5) {
even_sum_squares += value * value;
}
The value of the item is never used directly, only in the value * value
combination, which makes it an ideal target
for a map()
:
let mut even_sum_squares = 0;
for val_sqr in values.iter().filter(|x| *x % 2 == 0).take(5).map(|x| x * x)
{
even_sum_squares += val_sqr;
}
These refactorings of the original loop have resulting in a loop body that's the perfect nail to fit under the hammer of
the sum()
method:
let even_sum_squares: u64 = values
.iter()
.filter(|x| *x % 2 == 0)
.take(5)
.map(|x| x * x)
.sum();
When Explicit is Better
This Item has highlighted the advantages of iterator transformations, particularly with respect to concision and clarity. So when are iterator transformations not appropriate or idiomatic?
- If the loop body is large and/or multi-functional, it makes sense to keep it as an explicit body rather than squeezing it into a closure.
- If the loop body involves error conditions that result in early termination of the surrounding function, these are
often best kept explicit – the
try_..()
methods only help a little. However,collect()
's ability to convert a collection ofResult
values into aResult
holding a collection of values often allows error conditions to still be handled with the?
operator. - If performance is vital, an iterator transform that involves a closure should get optimized so that it is just
as fast as the equivalent explicit code. But if performance
of a core loop is that important, measure different variants and tune appropriately.
- Be careful to ensure that your measurements reflect real-world performance – the compiler's optimizer can give over-optimistic results on test data (as described in Item 30). The Godbolt compiler explorer is an amazing tool for exploring what the compiler spits out.
Most importantly, don't convert a loop into an iteration transformation if the conversion is forced or awkward. This is a matter of taste to be sure – but be aware that your taste is likely to change as you become more familiar with the functional style.
1: In fact, the iterator can be more general – the idea of emitting next items until done need not be associated with a container.
2: This method can't be provided if a mutation to
the item might invalidate the container's internal guarantees, for example the hash value used in a HashMap
.
Item 11: Implement the Drop
trait for RAII patterns
"Never send a human to do a machine's job." – Agent Smith
RAII stands for "Resource Acquisition Is Initialization"; this is a programming pattern where the lifetime of a value is exactly tied to the lifecycle of some additional resource. The RAII pattern was popularized by the C++ programming language, and is one of C++'s biggest contributions to programming.
With an RAII type,
- the type's constructor acquires access to some resource, and
- the type's destructor releases access to that resource.
The result of this is that the RAII type has an invariant: access to the underlying resource is available if and only if the item exists. Because the compiler ensures that local variables are destroyed when at scope exit, this in turn means that the underlying resources are also released at scope exit1.
This is particularly helpful for maintainability: if a subsequent change to the code alters the control flow, item
and resource lifetimes are still correct. To see this, consider some code that manually locks and unlocks a mutex;
this code is in C++, because Rust's Mutex
doesn't allow this kind of error-prone usage!
// C++ code
class ThreadSafeInt {
public:
ThreadSafeInt(int v) : value_(v) {}
void add(int delta) {
mu_.lock();
// ... more code here
value_ += delta;
// ... more code here
mu_.unlock();
}
A modification to catch an error condition with an early exit leaves the mutex locked:
// C++ code
void add_with_modification(int delta) {
mu_.lock();
// ... more code here
value_ += delta;
// Check for overflow.
if (value_ > MAX_INT) {
// Oops, forgot to unlock() before exit
return;
}
// ... more code here
mu_.unlock();
}
However, encapsulating the locking behaviour into an RAII class:
// C++ code
class MutexLock {
public:
MutexLock(Mutex* mu) : mu_(mu) { mu_->lock(); }
~MutexLock() { mu_->unlock(); }
private:
Mutex* mu_;
};
means the equivalent code is safe for this kind of modification:
// C++ code
void add_with_modification(int delta) {
MutexLock with_lock(&mu_);
// ... more code here
value_ += delta;
// Check for overflow.
if (value_ > MAX_INT) {
return; // Safe, with_lock unlocks on the way out
}
// ... more code here
}
In C++, RAII patterns were often originally used for memory management, to ensure that manual allocation (new
,
malloc()
) and deallocation (delete
, free()
) operations were kept in sync. A general version of this was added to
the C++ standard library in C++11: the std::unique_ptr<T>
type ensures that a single place has "ownership" of memory,
but which allows a pointer to the memory to be "borrowed" for ephemeral use (ptr.get()
).
In Rust, this behaviour for memory pointers is built into the language (Item 14), but the general principle of RAII is
still useful for other kinds of resources2. Implement Drop
for any types that hold
resources that must be released, such as:
- Access to operating system resources. For UNIX-derived systems, this usually means something that holds a file descriptor; failing to release these correctly will hold on to system resources (and will also eventually lead to the program hitting the per-process file descriptor limit).
- Access to synchronization resources. The standard library already includes memory synchronization primitives, but other resources (e.g. file locks, database locks, …) may need similar encapsulation.
- Access to raw memory, for
unsafe
types that deal with low-level memory management (e.g. for FFI).
The most obvious instance of RAII in the Rust standard library is the
MutexGuard
item returned by
Mutex::lock()
operations (should you choose to
ignore the advice of Item 17 and use shared-state parallelism). This is roughly analogous to the final C++ example
above, but here the MutexGuard
item acts as a proxy to the mutex-protected data in addition to being an RAII item
for the held lock:
use std::sync::Mutex;
struct ThreadSafeInt {
value: Mutex<i32>,
}
impl ThreadSafeInt {
fn new(val: i32) -> Self {
Self {
value: Mutex::new(val),
}
}
fn add(&self, delta: i32) {
let mut v = self.value.lock().unwrap();
*v += delta;
}
Item 17 advises against holding locks for large sections of code; to ensure this, use blocks to restrict the scope of RAII items. This leads to slightly odd indentation, but it's worth it for the added safety and lifetime precision.
fn add_with_extras(&self, delta: i32) {
// ... more code here that doesn't need the lock
{
let mut v = self.value.lock().unwrap();
*v += delta;
}
// ... more code here that doesn't need the lock
}
Having proselytized the uses of the RAII pattern, an explanation of how to implement it is in order. The
Drop
trait allows you to add user-defined behaviour to the
destruction of an item. This trait has a single method,
drop
, which the compiler runs just before the
memory holding the item is released.
#[derive(Debug)]
struct MyStruct(i32);
impl Drop for MyStruct {
fn drop(&mut self) {
println!("Dropping {:?}", self);
}
}
The drop
method is specially reserved to the compiler and can't be manually invoked, because the item would be
left in a potentially messed-up state afterwards:
x.drop();
error[E0040]: explicit use of destructor method
--> raii/src/main.rs:63:7
|
63 | x.drop();
| ^^^^
| |
| explicit destructor calls not allowed
| help: consider using `drop` function: `drop(x)`
(As suggested by the compile, just call drop(obj)
instead to
manually drop an item.)
The drop
method is therefore the key place for implementing RAII patterns, by ensuring that resources are released on
item destruction.
1: This also means that RAII as a technique
is mostly only available in languages that have a predictable time of destruction, which rules out most garbage
collected languages (although Go's defer
statement achieves some of
the same ends.)
2: RAII is also still useful for memory management in low-level
unsafe
code, but that is (mostly) beyond the scope of this book
Item 12: Prefer generics to trait objects
Item 2 described the use of traits to encapsulate behaviour in the type system, as a collection of related methods, and observed that there are two ways to make use of traits: as trait bounds for generics, or in trait objects. This Item explores the trade-offs between these two possibilities.
Rust's generics are roughly equivalent to C++'s templates: they allow the programmer to write code that works for some
arbitrary type T
, and specific uses of the generic code are generated at compile time – a process known as
monomorphization in Rust, and template instantiation in C++. Unlike C++, Rust explicitly encodes the expectations
for the type T
in the type system, in the form of trait bounds for the generic.
In comparison, trait objects are fat pointers (Item 9) that combine a pointer to the underlying concrete item with a pointer to a vtable that in turn holds function pointers for all of the trait implementation's methods.
let square = Square::new(1, 2, 2);
let draw: &dyn Drawable = □
These basic facts already allow some immediate comparisons between the two possibilities:
- Generics are likely to lead to bigger code sizes, because the compiler generates a fresh copy of the code
generic::<T>(t: &T)
for every typeT
that gets used; atraitobj(t: &dyn T)
method only needs a single instance. - Invoking a trait method from a generic will generally be ever-so-slightly faster than from code that uses a trait object, because the latter needs to perform two dereferences to find the location of the code (trait object to vtable, vtable to implementation location).
- Compile times for generics are likely to be longer, as the compiler is building more code and the linker has more work to do to fold duplicates.
In most situations, these aren't significant differences; you should only use optimization-related concerns as a primary decision driver if you've measured the impact and found that it has a genuine effect (a speed bottleneck or a problematic occupancy increase).
A more significant difference is that generic trait bounds can used to conditionally make methods available, depending on whether the type parameter implements multiple traits.
trait Drawable {
fn bounds(&self) -> Bounds;
}
struct Container<T>(T);
impl<T: Drawable> Container<T> {
// The `area` method is available for all `Drawable` containers.
fn area(&self) -> i64 {
let bounds = self.0.bounds();
(bounds.bottom_right.x - bounds.top_left.x)
* (bounds.bottom_right.y - bounds.top_left.y)
}
}
impl<T: Drawable + Debug> Container<T> {
// The `show` method is only available if `Debug` is also implemented.
fn show(&self) {
println!("{:?} has bounds {:?}", self.0, self.0.bounds());
}
}
let square = Container(Square::new(1, 2, 2)); // Square is not Debug
let circle = Container(Circle::new(3, 4, 1)); // Circle is Debug
println!("area(square) = {}", square.area());
println!("area(circle) = {}", circle.area());
circle.show();
// The following line would not compile.
// square.show();
A trait object only encodes the implementation vtable for a single trait, so doing something equivalent is much more
awkward. For example, a combination DebugDrawable
trait could be defined for the show()
case, together with some
conversion operations (Item 6) to make life easier. However, if there are multiple different combinations of distinct
crates, it's clear that the combinatorics of this approach rapidly become unwieldy.
Item 2 described the use of trait bounds to restrict what type parameters are acceptable for a generic function. Trait bounds can also be applied to trait definitions themselves:
trait Shape: Drawable {
fn render_in(&self, bounds: Bounds);
fn render(&self) {
self.render_in(overlap(SCREEN_BOUNDS, self.bounds()));
}
}
In this example, the render()
method's default implementation (Item 13) makes use of the trait bound, relying on the
availability of the bounds()
method from Drawable
.
Programmers coming from object-oriented languages often confuse trait bounds with inheritance, under the mistaken
impression that a trait bound like this means that a Shape
is-a Drawable
. That's not the case: the relationship
between the two types is better expressed as Shape
also-implements Drawable
.
Under the covers, trait objects for traits that have trait bounds
let square = Square::new(1, 2, 2);
let draw: &dyn Drawable = □
let shape: &dyn Shape = □
have a single combined vtable that includes the methods of the top-level trait, plus the methods of all of the trait bounds:
This means that there is no way to "upcast" from Shape
to Drawable
, because the (pure) Drawable
vtable can't be
recovered at runtime (see Item 19 for more on this). There is no way to convert between related trait objects,
which in turn means there is no Liskov substitution.
Repeating the same point in different words, a method that accepts a Shape
trait object
- can make use of methods from
Drawable
(becauseShape
also-implementsDrawable
, and because the relevant function pointers are present in theShape
vtable) - cannot pass the trait object on to another method that expects a
Drawable
trait object (becauseShape
is-notDrawable
, and because theDrawable
vtable isn't available).
In contrast, a generic method that accepts items that implement Shape
- can use methods from
Drawable
- can pass the item on to another generic method that has a
Drawable
trait bound, because the trait bound is monomorphized at compile time to use theDrawable
methods of the concrete type.
Another restriction on trait objects is the requirement for object safety: only traits that comply with the following two rules can be used as trait objects.
- The trait's methods must not be generic.
- The trait's methods must not return a type that includes
Self
.
The first restriction is easy to understand: a generic method f
is really an infinite set of methods, potentially
encompassing f::<i16>
, f::<i32>
, f::<i64>
, f::<u8>
, … The trait object's vtable, on the other, is very
much a finite collection of function pointers, and so it's not possible to fit an infinite quart into a finite pint
pot.
The second restriction is a little bit more subtle, but tends to be the restriction that's hit more often in practice
– traits that impose Copy
or Clone
trait bounds (Item 5) immediately fall under this rule. To see why it's
disallowed, consider code that has a trait object in its hands; what happens if that code calls (say) let y = x.clone()
? The calling code needs to reserve enough space for y
on the stack, but it has no idea of the size of y
because Self
is an arbitrary type. As a result, return types that mention1 Self
lead to a trait that is not
object safe.
There is an exception to this second restriction. A method returning some Self
-related type does not affect object
safety if Self
comes with an explicit restriction to types whose size is known at compile time: Self: Sized
. This
trait bound means that the method can't be used with trait objects anyway, because trait objects are explicitly of
unknown size (!Sized
), and so the method is irrelevant for object safety.
The balance of factors so far leads to the advice to prefer generics to trait objects, but there are situations where trait objects are the right tool for the job.
The first is a practical consideration: if generated code size or compilation time is a concern, then trait objects will perform better (as described at the start of this Item).
A more theoretical aspect that leads towards trait objects is that they fundamentally involve type erasure: information about the concrete type is lost in the conversion to a trait object. This can be a downside (see Item 19), but it can also be useful because it allows for collections of heterogeneous objects – because the code just relies on the methods of the trait, it can invoke and combine the methods of differently (concretely) typed items.
The traditional OO example of rendering a list of shapes would be one example of this: the same render()
method could
be used for squares, circles, ellipses and stars in the same loop.
let shapes: Vec<&dyn Shape> = vec![&square, &circle];
for shape in shapes {
shape.render()
}
A much more obscure potential advantage for trait objects is when the available types are not known at compile-time; if
new code is dynamically loaded at run-time (e.g via dlopen(3)
),
then items that implement traits in the new code can only be invoked via a trait object, because there's no source code
to monomorphize over.
1: At present, the restriction on
methods that return Self
includes types like Box<Self>
that could be safely stored on the stack; this restriction
might be relaxed in future.
Item 13: Use default implementations to minimize required trait methods
The designer of a trait has two different audiences to consider: the programmers who will be implementing the trait, and those who will be using the trait. These two audiences lead to a degree of tension in the trait design:
- To make the implementor's life easier, it's better for a trait to have the absolute minimum number of methods to achieve its purpose.
- To make the user's life more convenient, it's helpful to provide a range of variant methods that cover all of the common ways that the trait might be used.
This tension can be balanced by including the wider range of methods that makes the user's life easier, but with default implementations provided for any methods that can be built from other, more primitive, operations on the interface.
A simple example of this is the
is_empty()
method for an
ExactSizeIterator
; it has a default implementation
that relies on the len()
trait method:
fn is_empty(&self) -> bool {
self.len() == 0
}
The existence of a default implementation is just that: a default. If an implementation of the trait has a more optimal
way of determining whether the iterator is empty, it can replace the default is_empty()
with its own.
This approach leads to trait definitions that have a small number of required methods, plus a much larger number of default-implemented methods. An implementor for the trait only has to implement the former, and gets all of the latter for free.
It's also an approach that is widely followed by the Rust standard library; perhaps the best example there is the
Iterator
trait, which has a single required method
(next
) but which includes a panoply of
pre-provided methods (Item 10), over 50 at the time of writing.
Trait methods can impose trait bounds, indicating that a method is only available if the types involved implement
particular traits. The Iterator
trait also shows that this is useful in combination with default method
implementations. For example, the cloned()
iterator method has a trait bound and a default implementation:
fn cloned<'a, T: 'a>(self) -> Cloned<Self>
where
Self: Sized + Iterator<Item = &'a T>,
T: Clone,
{
Cloned::new(self)
}
In other words, the cloned()
method is only available if the underlying Item
type implements
Clone
; when it does, the implementation is automatically
available.
The final observation about trait methods with default implementations is that new ones can be safely added to a trait even after an initial version of the trait is released. An addition like this preserves backwards compatibility (see Item 21) for both users and implementors1 of the trait.
So follow the example of the standard library and provide a minimal API surface for implementors, but a convenient and comprehensive API for users, by adding methods with default implementations (and trait bounds as appropriate).
1: This is true even if an implementor was unlucky enough to have
already added a method of the same name in the concrete type, as the concrete method – known as an inherent
implementation – will
be used ahead of trait method. The trait method can be explicitly selected instead by casting: <Concrete as Trait>::method()
.
Concepts
The first section of this book covered Rust's type system, which helps provide the vocabulary needed to work with some of the concepts involved in writing Rust code.
The borrow checker and lifetime checks are central to what makes Rust unique; they are also a common stumbling block for newcomers to Rust.
However, it's a good idea to try to align your code with the consequences of these concepts. It's possible to re-create (some of) the behaviour of C/C++ in Rust, but why bother to use Rust if you do?
Item 14: Understand the borrow checker
Item 15: Understand lifetimes
"Eppur si muove" – Galileo Galilei
Item 16: Avoid writing unsafe
code
The memory safety guarantees of Rust are its unique selling point; it is the Rust language feature that is not found in any other mainstream language. These guarantees come at a cost; writing Rust requires you to re-organize your code to mollify the borrow checker (Item 14), and to precisely specify the pointer types that you use (Item 9).
Unsafe Rust weakens some of those guarantees, in particular by allowing the use of raw pointers that work more like old-style C pointers. These pointers are not subject to the borrowing rules, and the programmer is responsible for ensuring that they still point to valid memory whenever they're used.
So at a superficial level, the advice of this Item is trivial: why move to Rust if you're just going to write C code in
Rust? However, there are occasions where unsafe
code is absolutely required: for low-level library code, or
for when your Rust code has to interface with code in other languages (Item 38).
The wording of this Item is quite precise, though: avoid writing unsafe
code. The emphasis is on the "writing",
because much of the time the unsafe
code you're likely to need has already been written for you.
The Rust standard libraries contain a lot of unsafe
code; a quick search finds around 1000 uses of unsafe
in the
alloc
library, 1500 in core
and a further 2000 in std
. This code has been written by experts and is
battle-hardened by use in many thousands of Rust codebases.
Some of this unsafe
code happens under the covers in standard library features that we've already covered:
- The smart pointer types –
Rc
,RefCell
,Arc
and friends – described in Item 9 useunsafe
code internally in order to be able to present their particular semantics to their users. - The synchronization primitives –
Mutex
,RwLock
and associated guards – from Item 17 useunsafe
, OS-specific code internally.
The standard library1 also has other functionality covering more advanced features,
implemented with unsafe
internally:
std::pin::Pin
forces an item to not move in memory (Item 15). This allows self-referential data structures, often a bête noire for new arrivals to Rust.std::borrow::Cow
provides a clone-on-write smart pointer: the same pointer can be used for both reading and writing, and a clone of the underlying data only happens if and when a write occurs.- Various functions (
take
,swap
,replace
) instd::mem
allow items in memory to be manipulated without falling foul of the borrow checker.
These features may still need a little caution to be used correctly, but the unsafe
code has been encapsulated in a
way that removes whole classes of problems.
Moving beyond the standard library, the crates.io ecosystem also includes many crates that
encapsulate unsafe
code to provide a frequently-used feature. For example:
once_cell
provides a way to have something like global variables, initialized exactly once.rand
provides random number generation, making use of the lower-level underlying features provided by the operating system and CPU.byteorder
allows raw bytes of data to be converted to and from numbers.cxx
allows C++ code and Rust code to interoperate.
There are many other examples, but hopefully the general idea is clear. If you want to do something that doesn't
obviously fit with the constraints of Rust (especially Item 14 and Item 15) hunt through the standard library to see
if there's existing functionality that does what you need. If you don't find it, try also hunting through crates.io
;
after all, most of the time your problem will not be a unique one that no-one else has ever faced before.
Of course there will always be places where unsafe
is forced, for example when you need to interact with code written
in other languages via a foreign-function interface (FFI; see Item 38). But when it's necessary, consider writing a
wrapper layer that holds all the unsafe
code that's required, so that other programmers can then follow the advice
of this Item. This also helps to localize problems: when something goes wrong, the unsafe
wrapper can be the first
suspect.
Also, if you're forced to write unsafe
code, pay attention to the warning implied by the keyword itself: "Hic sunt
dracones".
- Write even more tests (Item 30) than usual.
- Run additional diagnostic tools (Item 31) over the code. In particular, run Miri over your
unsafe
code – Miri interprets the intermediate level output from the compiler, which allows it to detect classes of errors that are invisible to the Rust compiler. - Think about multi-threaded use, particularly if there's shared state (Item 17)
1: In practice, most of thisstd
functionality is actually provided by core
, and so is
available to no_std
code as described in Item 37.
Item 17: Be wary of shared-state parallelism
"Even the most daring forms of sharing are guaranteed safe in Rust." – Aaron Turon
The official documentation describes Rust as enabling "fearless concurrency" but this Item will explore why (sadly) there are still some reasons to be afraid.
This Item is specific to shared-state parallelism: where different threads of execution communicate with each other by sharing memory.
Data Races
Let's start with the good news, by exploring data races and Rust. The definition is roughly:
A data race is defined to occur when two distinct threads access the same memory location, where
- at least one of them is a write, and
- there is no synchronization mechanism that enforces an ordering on the accesses.
The basics of this are best illustrated with an example:
class BankAccount {
public:
BankAccount() : balance_(0) {}
int64_t balance() const {
return balance_;
}
void deposit(uint32_t amount) {
balance_ += amount;
}
bool withdraw(uint32_t amount) {
if (balance_ < amount) {
return false;
}
// What if another thread changes `balance_` at this point?
std::this_thread::sleep_for(std::chrono::milliseconds(500));
balance_ -= amount;
return true;
}
private:
int64_t balance_;
};
This example is in C++, not Rust, for reasons which will become clear shortly. However, the same general concepts apply in lots of other languages (that aren't Rust) – Java, or Go, or Python …
This class works fine in a single-threaded setting, but in a multi-threaded setting:
std::thread taker(take_out, &account, 100);
std::thread taker2(take_out, &account, 100);
std::thread taker3(take_out, &account, 100);
std::thread payer(pay_in, &account, 300);
where several threads are repeatedly trying to withdraw from the account:
int64_t check_balance(const BankAccount* account) {
int64_t bal = account->balance();
if (bal < 0) {
std::cerr << "** Oh no, gone overdrawn: " << bal << " **!\n";
abort();
}
std::cout << "Balance now " << bal << "\n";
return bal;
}
void take_out(BankAccount* account, int count) {
for (int ii = 0; ii < count; ii++) {
if (account->withdraw(100)) {
log("Withdrew 100");
} else {
log("Failed to withdraw 100");
}
check_balance(account);
std::this_thread::sleep_for(std::chrono::milliseconds(6));
}
}
then eventually things will go wrong.
** Oh no, gone overdrawn: -100 **!
The problem isn't hard to spot, particularly with the helpful comment in the withdraw()
method: when multiple threads
are involved, the value of the balance can change between the check and the modification. However, real-world bugs of
this sort are much harder to spot – particularly if the compiler is allowed to perform all kinds of tricks and
re-orderings of code under the covers (as is the case for C++).
The sleep
call also artificially raises the chances of this bug being hit and thus detected early; when these problems
are encountered in the wild they're likely to occur rarely and intermittently – making them very hard to debug.
The BankAccount
class is thread-compatible, which means that it can be used in a multithreaded environment as long
as the users of class ensure that access to it is governed by some kind of external synchronization mechanism.
The class can be converted to a thread-safe class1 – meaning that it is safe to use from multiple threads – by adding internal synchronization operations:
class BankAccount {
public:
BankAccount() : balance_(0) {}
int64_t balance() const {
const std::lock_guard<std::mutex> with_lock(mu_);
return balance_;
}
void deposit(uint32_t amount) {
const std::lock_guard<std::mutex> with_lock(mu_);
balance_ += amount;
}
bool withdraw(uint32_t amount) {
const std::lock_guard<std::mutex> with_lock(mu_);
if (balance_ < amount) {
return false;
}
// What if another thread changes `balance_` at this point?
balance_ -= amount;
return true;
}
private:
mutable std::mutex mu_;
int64_t balance_;
};
The internal balance_
field is now protected by a mutex mu_
: a synchronization object that ensures that only one
thread can successfully lock()
at a time. The second and subsequent callers will block until unlock()
is called
– and even then, only one of the blocked threads will unblock and proceed through lock()
.
All access to the balance now takes place with the mutex held, ensuring that its value is consistent between check and
modification. The std::lock_guard
is also worth highlighting:
it's an RAII class (cf. Item 11) that ensures that the mutex is unlocked when the scope exits, reducing the chances of
making a mistake around balancing manual lock()
and unlock()
calls.
However, the thread-safety here is still fragile; all it takes is one erroneous modification to the class:
bool transfer(BankAccount* destination, uint32_t amount) {
// oops, forgot about mu_
if (balance_ < amount) {
return false;
}
balance_ -= amount;
destination->balance_ += amount;
return true;
}
and the thread-safety has been destroyed2.
For a book about Rust, this Item has covered a lot of C++, so consider a straightforward translation of this class into Rust:
pub struct BankAccount {
balance: i64,
}
impl BankAccount {
pub fn new() -> Self {
BankAccount { balance: 0 }
}
pub fn balance(&self) -> i64 {
self.balance
}
pub fn deposit(&mut self, amount: i64) {
self.balance += amount
}
pub fn withdraw(&mut self, amount: i64) -> bool {
if self.balance < amount {
return false;
}
self.balance -= amount;
true
}
}
This works fine in a single-threaded context – even if that thread is not the main thread:
let mut account = BankAccount::new();
let payer = std::thread::spawn(move || pay_in(&mut account, 30));
// Wait for thread completion
payer.join().unwrap();
but a naïve attempt to use the BankAccount
across multiple threads:
let mut account = BankAccount::new();
let taker = std::thread::spawn(move || take_out(&mut account, 100));
let payer = std::thread::spawn(move || pay_in(&mut account, 300));
payer.join().unwrap();
taker.join().unwrap();
immediately falls foul of the compiler:
error[E0382]: use of moved value: `account`
--> deadlock/src/main.rs:76:40
|
74 | let mut account = BankAccount::new();
| ----------- move occurs because `account` has type `broken::BankAccount`, which does not implement the `Copy` trait
75 | let taker = std::thread::spawn(move || take_out(&mut account, 100));
| ------- ------- variable moved due to use in closure
| |
| value moved into closure here
76 | let payer = std::thread::spawn(move || pay_in(&mut account, 300));
| ^^^^^^^ ------- use occurs due to use in closure
| |
| value used here after move
With experience of the borrow checker (Item 14), the problem sticks out clearly: there are two mutable references to the same item, one more than is allowed. The rules of the borrow checker are that you can have a single mutable reference to an item, or multiple (immutable) references, but not both at the same time.
This has a curious resonance with the definition of a data race at the start of this Item: enforcing that there is a single writer, or multiple readers (but never both) means that there can be no data races. By enforcing memory safety, Rust gets thead safety "for free".
As with C++, some kind of synchronization is needed to make this struct
thread-safe. The most common mechanism is also
called Mutex
, but the Rust version "wraps" the protected data
rather than being a standalone object (as in C++):
pub struct BankAccount {
balance: std::sync::Mutex<i64>,
}
The lock()
method on this Mutex
generic returns
a MutexGuard
object with an RAII behaviour, like C++'s
std::lock_guard
: the mutex is automatically released at the end of the scope when the guard is drop
ped. (Rust's
Mutex
has no manual lock/unlock methods, as they would expose developers to the danger of forgetting to keep lock()
and unlock()
calls exactly in sync.)
The MutexGuard
object also acts as a proxy for the data that is enclosed by the Mutex
, by implementing the Deref
and DerefMut
traits (Item 9), allowing it to be used for both read operations:
pub fn balance(&self) -> i64 {
*self.balance.lock().unwrap()
}
and write operations:
// Note: no longer needs `&mut self`.
pub fn deposit(&self, amount: i64) {
*self.balance.lock().unwrap() += amount
}
There's an interesting detail lurking in the signature of this method: although it is modifying the balance of the
BankAccount
, this method now takes &self
rather than &mut self
. This is inevitable: if multiple threads are going
to hold references to the same BankAccount
, by the rules of the borrow checker those references had better not be
mutable. It's also another instance of the interior mutability pattern described in Item 9: borrow checks are
effectively moved from compile-time to run-time, but now with a specific concern for cross-thread synchronization.
Wrapping up shared state in a Mutex
mollifies the borrow checker, but there are still lifetime issues (Item 15) to fix.
{
let account = BankAccount::new();
let taker = std::thread::spawn(|| take_out(&account, 100));
let payer = std::thread::spawn(|| pay_in(&account, 300));
}
error[E0373]: closure may outlive the current function, but it borrows `account`, which is owned by the current function
--> deadlock/src/main.rs:190:44
|
190 | let taker = std::thread::spawn(|| take_out(&account, 100));
| ^^ ------- `account` is borrowed here
| |
| may outlive borrowed value `account`
|
note: function requires argument type to outlive `'static`
--> deadlock/src/main.rs:190:25
|
190 | let taker = std::thread::spawn(|| take_out(&account, 100));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: to force the closure to take ownership of `account` (and any other referenced variables), use the `move` keyword
|
190 | let taker = std::thread::spawn(move || take_out(&account, 100));
| ^^^^^^^
error[E0373]: closure may outlive the current function, but it borrows `account`, which is owned by the current function
--> deadlock/src/main.rs:191:44
|
191 | let payer = std::thread::spawn(|| pay_in(&account, 300));
| ^^ ------- `account` is borrowed here
| |
| may outlive borrowed value `account`
|
note: function requires argument type to outlive `'static`
--> deadlock/src/main.rs:191:25
|
191 | let payer = std::thread::spawn(|| pay_in(&account, 300));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: to force the closure to take ownership of `account` (and any other referenced variables), use the `move` keyword
|
191 | let payer = std::thread::spawn(move || pay_in(&account, 300));
| ^^^^^^^
The error message makes the problem clear: the BankAccount
is going to be drop
ped at the end of the block, but there
are two new threads which have a reference to it, and which will carry on running afterwards.
The standard tool for ensuring that an object remains active until all references to it are gone is a reference counted
pointer, and Rust's variant of this for multi-threaded use is
std::sync::Arc
:
let account = std::sync::Arc::new(BankAccount::new());
let account2 = account.clone();
let taker = std::thread::spawn(move || take_out(&account2, 100));
let account3 = account.clone();
let payer = std::thread::spawn(move || pay_in(&account3, 300));
Each thread gets its own (moved) copy of the reference counting pointer, and the underlying BankAccount
will only be
drop
ped when the refcount drops to zero. This combination of Arc<Mutex<T>>
is common in Rust programs that use
shared state parallelism.
Stepping back from the technical details, observe that Rust has entirely avoided the problem of data races that plagues
multi-threaded programming in other languages. Of course, this good news is restricted to safe Rust – unsafe
code (Item 16) and FFI boundaries in particular (Item 38) may not be data race free – but it's still a
remarkable phenomenom.
Standard Marker Traits
There are two standard traits that affect the use of Rust objects between threads. Both of these traits are marker traits (Item 5) that have no associated methods, but they have special significance to the compiler in multi-threaded scenarios.
- The
Send
trait indicates that items of a type are safe to transfer between threads; ownership of an item of this type can be passed from one thread to another. - The
Sync
trait indicates that items of a type can be safely accessed by multiple threads, subject to the rules of the borrow checker.
Both of these traits are
auto-traits: the compiler
automatically derives them for new types, as long as the constituent parts of the type are also Send
/Sync
.
The majority of safe types are Send
and Sync
, so much so that it's clearer to understand what types don't
implement these traits (written in the form impl !Sync for Type
).
A non-Send
type is one that can only be used in a single thread. The canonical example of this is the unsynchronized
reference counting pointer Rc<T>
(Item 9). The implementation of
this type explicitly assumes single-threaded use (for speed); there is no attempt at synchronizing the internal refcount
for multi-threaded use. As such, transferring an Rc<T>
between threads is not allowed; use Arc<T>
(with its
additional synchronization overhead) for this case.
A non-Sync
type is one that's not safe to use from multiple threads via non-mut
references (as the borrow
checker will ensure there are never multiple nut
references). The canonical examples of this are the types that
provide interior mutability in an unsynchronized way, such as
Cell<T>
and
RefCell<T>
. Use Mutex<T>
or
RwLock<T>
to provide interior mutability in a multi-threaded
environment.
(Raw pointer types like *const T
and *mut T
are also neither Send
nor Sync
; see Item 16 and Item 38.)
Deadlocks
Now for the bad news. Multi-threaded code comes with two terrible problems:
- data races, which can lead to corrupted data, and
- deadlocks, which can lead to your program grinding to a halt.
Both of these problems are terrible because they can be very hard to debug in practice: the failures occur non-deterministically and are often more likely to happen under load – which means that they don't show up in unit tests, integration tests, or any other sort of test (Item 30), but they do show up in production.
Rust has solved the problem of data races (as described above), but the problem of deadlocks applies to Rust as much as it does to any other language that supports multi-threading.
Consider a simplified multiple player game server, implemented as a multithreaded application in order to service many players in parallel. Two core data structures might be a collection of players, indexed by username:
players: Mutex<HashMap<String, Player>>,
and a collection of games in progress indexed by some unique identifier:
games: Mutex<HashMap<GameID, Game>>,
Both of these data structures are Mutex
-protected and so safe from data races; however, code that manipulates both
data structures opens up potential problems. A single interaction between the two might work fine:
fn add_and_join(&self, username: &str, info: Player) -> Option<GameID> {
// Add the new player.
let mut players = self.players.lock().unwrap();
players.insert(username.to_owned(), info);
// Find a game with available space for them to join.
let mut games = self.games.lock().unwrap();
for (id, game) in games.iter_mut() {
if game.add_player(username) {
return Some(*id);
}
}
None
}
However, a second interaction between the two independently locked data structures is where problems start:
fn ban_player(&self, username: &str) {
// Find all games that the user is in and remove them.
let mut games = self.games.lock().unwrap();
games
.iter_mut()
.filter(|(_id, g)| g.has_player(username))
.for_each(|(_id, g)| g.remove_player(username));
// Wipe them from the user list.
let mut players = self.players.lock().unwrap();
players.remove(username);
}
To understand the problem, imagine two separate threads using these two methods:
- Thread 1 enters
add_and_join()
and immediately acquires theplayers
lock. - Thread 2 enters
ban_player()
and immediately acquires thegames
lock. - Thread 1 now tries to acquire the
games
lock; this is held by thread 2, so thread 1 blocks. - Thread 2 tries to acquire the
players
lock; this is held by thread 1, so thread 2 blocks.
At this point the program is deadlocked: neither thread will ever progress, and nor will any other thread that does
anything with either of the two Mutex
-protected data structures.
The root cause of this is a lock inversion: one function acquires the locks in the order players
then games
,
whereas the other uses the opposite order (games
then players
). This is the simplest example of a more general
description of the problem:
- Build a directed graph where:
- Each
Mutex
instance is a vertex. - Each edge indicates a situation where one
Mutex
gets acquired while anotherMutex
is already held.
- Each
- If there are any cycles in the graph, a deadlock can occur.
A simplistic attempt to solve this problem involves reducing the scope of the locks, so there is no point where both locks are held at the same time:
fn add_and_join(&self, username: &str, info: Player) -> Option<GameID> {
// Add the new player.
{
let mut players = self.players.lock().unwrap();
players.insert(username.to_owned(), info);
}
// Find a game with available space for them to join.
{
let mut games = self.games.lock().unwrap();
for (id, game) in games.iter_mut() {
if game.add_player(username) {
return Some(*id);
}
}
}
None
}
fn ban_player(&self, username: &str) {
// Find all games that the user is in and remove them.
{
let mut games = self.games.lock().unwrap();
games
.iter_mut()
.filter(|(_id, g)| g.has_player(username))
.for_each(|(_id, g)| g.remove_player(username));
}
// Wipe them from the user list.
{
let mut players = self.players.lock().unwrap();
players.remove(username);
}
}
(A better version of this would be to encapsulate the manipulation of the players
data structure into add_player()
and remove_player()
helper methods, to reduce the chances of forgetting to close out a scope.)
This solves the deadlock problem, but leaves behind a data consistency problem: the players
and games
data
structures can get out of sync with each other.
- Thread 1 enters
add_and_join("Alice")
and adds Alice to theplayers
data structure (then releases theplayers
lock). - Thread 2 enters
ban_player("Alice")
and removes Alice from allgames
(then releases thegames
lock). - Thread 2 now removes Alice from the
players
data structure; thread 1 has already released the lock so this does not block. - Thread 1 now carries on and acquires the
games
lock (already released by thread 2). With the lock held, thread 1 adds "Alice" to a game in progress.
At this point, there is a game that includes a player that doesn't exist, according to the players
data structure!
The heart of the problem is that there are two data structures that need to be kept in sync with each other; the best way to do this is to have a single sychronization primitive that covers both of them.
struct GameState {
players: HashMap<String, Player>,
games: HashMap<GameID, Game>,
}
struct GameServer {
state: Mutex<GameState>,
// ...
}
Advice
The most obvious advice for how to avoid the problems that come with shared-state parallelism is simply to avoid shared-state parallelism. The Rust book quotes from the Go language documentation: "Do not communicate by sharing memory; instead, share memory by communicating".
The Go language has channels that are suitable for this built into the
language; for Rust, equivalent functionality is included in the standard library
in the std::sync::mpsc
module: the channel()
function returns a (Sender, Receiver)
pair that allows
values of a particular type to be communicated between threads.
If shared-state concurrency can't be avoided, then there are some ways to reduce the chances of writing deadlock-prone code:
- Put data structures that must be kept consistent with each other under a single lock.
- Keep lock scopes small and obvious; wherever possible, use helper methods that get and set things under the relevant lock.
- Avoid invoking closures with locks held; this puts the code at the mercy of whatever closure gets added to the codebase in the future.
- Similarly, avoid returning a
MutexGuard
to a caller: it's like handing out a loaded gun from a deadlock perspective. - Include deadlock detection tools in your CI system (Item 32), such as
no_deadlocks
, ThreadSanitizer, orparking_lot::deadlock
. - As a last resort: design, document, test and police a locking hierarchy that describes what lock orderings are allowed/required. This should be a last resort because any strategy that relies on engineers never making a mistake is obviously doomed to failure in the long term.
More abstractly, multi-threaded code is an ideal place to apply the general advice: prefer code that's so simple that it is obviously not wrong, rather than code that's so complex that it's not obviously wrong.
1: The third category of behaviour is thread-hostile: code that's dangerous in a multithreaded environment even if all access to it is externally synchronized.
2: The Clang C++ compiler includes a
-Wthread-safety
option, sometimes known as annotalysis,
that allows data to be annotated with information about which mutexes protect which data, and functions to be annotated
with information about the locks they acquire. This gives compile-time errors when these invariants are broken, like
Rust; however, there is nothing to enforce the use of these annotations in the first place – for example, when a
thread-compatible library is used in a multi-threaded environment for the first time.
Item 18: Don't panic
"It looked insanely complicated, and this was one of the reasons why the snug plastic cover it fitted into had the words DON’T PANIC printed on it in large friendly letters." – Douglas Adams
The title of this Item would be more accurately described as: prefer returning a Result
to using panic!
(but
don't panic is much catchier).
The first thing to understand about Rust's panic system is that it is not equivalent to an exception system (like the ones in Java or C++), even though there appears to be a mechanism for catching panics at a point further up the call stack.
Consider a function that panics on an invalid input:
fn divide(a: i64, b: i64) -> i64 {
if b == 0 {
panic!("Cowardly refusing to divide by zero!");
}
a / b
}
Trying to invoke this with an invalid input fails as expected:
// Attempt to discover what 0/0 is...
let result = divide(0, 0);
thread 'main' panicked at 'Cowardly refusing to divide by zero!', panic/src/main.rs:11:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
A wrapper that uses std::panic::catch_unwind
to catch the
panic
fn divide_recover(a: i64, b: i64, default: i64) -> i64 {
let result = std::panic::catch_unwind(|| divide(a, b));
match result {
Ok(x) => x,
Err(_) => default,
}
}
appears to work:
let result = divide_recover(0, 0, 42);
println!("result = {}", result);
result = 42
Appearances can be deceptive, however. The first problem with this approach is that panics don't always unwind; there
is a compiler option (which is also accessible via a
Cargo.toml
profile setting) that shifts panic
behaviour so that it immediately aborts the process.
thread 'main' panicked at 'Cowardly refusing to divide by zero!', panic/src/main.rs:11:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Abort trap: 6
This leaves any attempt to simulate exceptions entirely at the mercy of the wider project settings. It's also the case that some target platforms (for example WebAssembly) always abort on panic, regardless of any compiler or project settings.
A more subtle problem that's surfaced by panic handling is exception safety: if a panic occurs midway through an operation on a data structure, it removes any guarantees that the data structure has been left in a self-consistent state. Preserving internal invariants in the presence of exceptions has been known to be extremely difficult since the 1990s 1; this is one of the main reasons why Google (famously) bans the use of exceptions in its C++ code.
Finally, panic propagation also interacts poorly with FFI
(foreign function interface) boundaries (Item 38); use catch_unwind
to prevent panics in Rust code from propagating
to non-Rust calling code across an FFI boundary.
So what's the alternative to panic!
for dealing with error conditions? For library code, the best alternative is to
make the error someone else's problem, by returning a
Result
with an appropriate error type (Item 4). This allows the library user to make their own decisions about what
to do next – which may involve passing the problem on to the next caller in line, via the ?
operator.
The buck has to stop somewhere, and a useful rule of thumb is that it's OK to panic!
(or to unwrap()
, expect()
etc.) if you have control of main
; at that point, there's no further caller that the buck could be passed to.
Another sensible use of panic!
, even in library code, is in situations where it's very rare to encounter errors, and
you don't want users to have to litter their code with .unwrap()
calls.
If an error situation should only occur because (say) internal data is corrupted, rather than as a result of invalid
inputs, then triggering a panic!
is legitimate.
It can even be occasionally useful to allow panics that can be triggered by invalid input, but where such invalid inputs are out of the ordinary. This works best when the relevant entrypoints come in pairs:
- an "infallible" version whose signature implies it always succeeds (and which panics if it can't succeed),
- a "fallible" version that returns a
Result
.
For the former, Rust's API guidelines
suggest
that the panic!
should be documented in a specific section of the inline documentation (Item 27).
The String::from_utf8_unchecked
/ String::from_utf8
entrypoints in the
standard library are an example of the latter (although in this case, the panics are actually deferred to the point
where a String
constructed from invalid input gets used…).
Assuming that you are trying to comply with the advice of this item, there are a few things to bear in mind. The first
is that panic
s can appear in different guises; avoiding panic!
also involves avoiding:
Harder to spot are things like:
slice[index]
when the index is out of rangex / y
wheny
is zero.
The second observation around avoiding panic
s is that a plan that involves constant vigilance of humans is never a
good idea.
However, constant vigilance of machines is another matter: adding a check to your continuous integration (Item 32) system that spots new panics is much more reliable. A simple version could be a simple grep for the most common panicking entrypoints (as above); a more thorough check could involve additional tooling from the Rust ecosystem (Item 31), such as setting up a build variant that pulls in the no_panic crate.
1: Tom Cargill's 1994 article in the C++ Report explored just how difficult exception safety is for C++ template code, as did Herb Sutter's Guru of the Week #8 column.
Item 19: Avoid reflection
Programmers coming to Rust from other languages are often used to reaching for reflection as a tool in their toolbox. They can waste a lot of time trying to implement reflection-based designs in Rust, only to discover that what they're attempting can only be done poorly, if at all. This Item hopes to save that time wasted exploring dead-ends, by describing what Rust does and doesn't have in the way of reflection, and what can be used instead.
Reflection is the ability of a program to examine itself at run-time. Given an item at run-time, it covers:
- What information can be determined about the item's type?
- What can be done with that information?
Programming languages with full reflection support have extensive answers to these questions – as well as determining an item's type at run-time, its contents can be explored, its fields modified and its methods invoked. Languages that have this level of reflection support tend to be dynamically typed languages (e.g. Python, Ruby), but there are also some notable statically typed languages that also support this, particularly Java and Go.
Rust does not support this type of reflection, which makes the advice to avoid reflection easy to follow at this level – it's just not possible. For programmers coming from languages with support for full reflection, this absence may seem like a significant gap at first, but Rust's other features provide alternative ways of solving many of the same problems.
C++ has a more limited form of reflection, known as run-time type identification (RTTI). The
typeid
operator returns a unique identifier for every type, for
objects of polymorphic type (roughly: classes with virtual functions):
typeid
can recover the concrete class of an object referred to via a base class referencedynamic_cast<T>
allows base class references to be converted to derived classes, when it is safe and correct to do so.
Rust does not support this RTTI style of reflection either, continuing the theme that the advice of this Item is easy to follow.
Rust does support some features that provide similar functionality (in the
std::any
module), but they're limited (in ways explored below) and so
best avoided unless no other alternatives are possible.
The first reflection-like feature looks magic at first – a way of determining the name of an item's type:
let x = 42u32;
let y = Square::new(3, 4, 2);
println!("x: {} = {}", tname(&x), x);
println!("y: {} = {:?}", tname(&y), y);
x: u32 = 42
y: reflection::Square = Square { top_left: Point { x: 3, y: 4 }, size: 2 }
The implementation of tname()
reveals what's up the compiler's sleeve; the function is generic (as per Item 12) and
so each invocation of it is actually a different function (tname::<u32>
or tname::<Square>
):
fn tname<T: ?Sized>(_v: &T) -> &'static str {
std::any::type_name::<T>()
}
The std::any::type_name<T>
library function only has access to
compile-time information; nothing clever is happening at run-time.
The string returned by type_name
is only suitable for diagnostics – it's explicitly a "best-effort" helper whose
contents may change, and may not be unique – so don't attempt to parse type_name
results. If you need a
globally unique type identifier, use TypeId
instead:
use std::any::TypeId;
fn type_id<T: 'static + ?Sized>(_v: &T) -> TypeId {
TypeId::of::<T>()
}
println!("x has {:?}", type_id(&x));
println!("y has {:?}", type_id(&y));
x has TypeId { t: 12849923012446332737 }
y has TypeId { t: 7635675208524022980 }
The output is less helpful for humans, but the guarantee of uniqueness means that the result can be used in code.
However, it's usually best not to do so directly, but to use the
std::any::Any
trait1 instead.
This trait has a single method type_id()
, which
returns the TypeId
value for the type that implements the trait. You can't implement this trait yourself though,
because Any
already comes with a blanket implementation for every type T
:
impl<T: 'static + ?Sized> Any for T {
fn type_id(&self) -> TypeId {
TypeId::of::<T>()
}
}
Recall from Item 9 that a trait object is a fat pointer that holds a pointer to the underlying item, together with a
pointer to the trait implementation's vtable. For Any
, the vtable has a single entry, for a method that returns the
item's type.
let x_any: Box<dyn Any> = Box::new(42u64);
let y_any: Box<dyn Any> = Box::new(Square::new(3, 4, 3));
Modulo a couple of indirections, a dyn Any
trait object is effectively a combination of a raw pointer and a type
identifier. This means that Any
can offer some additional generic methods:
is<T>
to indicate whether the trait object's type is equal to some specific other typeT
.downcast_ref<T>
which returns a reference to the concrete typeT
, provided that the type matches.downcast_mut<T>
for the mutable variant ofdowncast_ref
.
Observe that the Any
trait is only approximating reflection functionality: the programmer chooses (at compile-time) to
explicitly build something (&dyn Any
) that keeps track of an item's compile-time type as well as its location. The
ability to (say) downcast back to the original type is only possible if the overhead of building an Any
trait object
has happened.
There are comparatively few scenarios where Rust has different compile-time and run-time types associated with an
item. Chief among these is trait objects: an item of a concrete type Square
can be coerced into a trait object
dyn Shape
for a trait that the type implements. This coercion builds a fat pointer (object+vtable) from a simple
pointer (object/item).
Recall also from Item 12 that Rust's trait objects are not really object-oriented. It's not the case that a Square
is-a Shape
, it's just that a Square
implements Shape
's interface. The same is true for trait bounds: a trait
bound Shape: Drawable
does not mean is-a, it just means also-implements; the vtable for Shape
includes the
entries for the methods of Drawable
.
For some simple trait bounds:
trait Drawable: Debug {
fn bounds(&self) -> Bounds;
}
trait Shape: Drawable {
fn render_in(&self, bounds: Bounds);
fn render(&self) {
self.render_in(overlap(SCREEN_BOUNDS, self.bounds()));
}
}
the equivalent trait objects:
let square = Square::new(1, 2, 2);
let draw: &dyn Drawable = □
let shape: &dyn Shape = □
have a layout whose arrows make the problem clear: given a dyn Shape
object, there's no way to build a dyn Drawable
trait object, because there's no way to get back to the vtable for impl Drawable for Square
– even though the
relevant parts of its contents (the address of the Square::bounds
method) is theoretically recoverable.
Comparing with the previous diagram, it's also clear that an explicitly constructed &dyn Any
trait object doesn't help.
Any
allows recovery of the original concrete type of the underlying item, but there is no run-time way to
see what traits it implement, nor to get access to the relevant vtable that might allow creation of a trait object.
So what's available instead?
The primary tool to reach for is trait definitions, and this is in line with advice for other languages – Effective Java Item 65 recommends "Prefer interfaces to reflection". If code needs to rely on certain behaviour being available for an item, encode that behaviour as a trait (Item 2). Even if the desired behaviour can't be expressed as a set of method signatures, use marker traits to indicate compliance with the desired behaviour – it's safer and more efficient than (say) introspecting the name of a class to check for particular prefix.
Code that expects trait objects can also be used with objects whose backing code was not available at program link time,
because it has been dynamically loaded at run-time (via dlopen(3)
or equivalent) – which means that
monomorphization of a generic (Item 12) isn't possible.
Relatedly, reflection is sometimes also used in other languages to allow multiple incompatible versions of the same
dependency library to be loaded into the program at once, bypassing linkage constraints that There Can Be Only
One. This is not needed in Rust, where cargo
already copes with multiple versions of the same library (Item 25).
Finally, macros – especially derive macros – can be used to auto-generate ancillary code that understands an item's type at compile-time, as a more efficient and more type-safe equivalent to code that parses an item's contents at run-time.
1: The C++ equivalent of Any
is
std::any
, and advice is to avoid it
too
Item 20: Avoid the temptation to over-optimize
"Just because Rust allows you to write super cool non-allocating zero-copy algorithms safely, doesn’t mean every algorithm you write should be super cool, zero-copy and non-allocating." – trentj
Dependencies
"When the Gods wish to punish us, they answer our prayers." – Oscar Wilde
For decades, the idea of code reuse was merely a dream. The idea that code could be written once, packaged into a library and re-used across many different applications was an ideal, only realized for a few standard libraries and for corporate in-house tools.
The growth of the Internet, and the rise of open-source software finally changed that. The first openly accessible repository that held a wide collection of useful libraries, tools and helpers, all packaged up for easy re-use, was CPAN: the Comprehensive Perl Archive Network, online since 1995. By the present day, almost every modern language1 has a comprehensive collection of open-source libraries available, housed in a package repository that makes the process of adding a new dependency easy and quick.
However, new problems come along with that ease, convenience and speed. It's usually still easier to re-use existing code than to write it yourself, but there are potential pitfalls and risks that come along with dependencies on someone else's code. This part of the book will help you be aware of these.
The focus is specifically on Rust, and with it use of the cargo
tool, but many of
the concerns, topics and issues covered apply equally well to other languages.
1: With the notable exception of C and C++, where package mangement remains somewhat fragmented.
Item 21: Understand what semantic versioning promises
"If we acknowledge that SemVer is a lossy estimate and represents only a subset of the possible scope of changes, we can begin to see it as a blunt instrument." – Titus Winters, "Software Engineering at Google"
Cargo, Rust's package manager, allows automatic selection of dependencies (Item 25) for Rust code according to
semantic versioning (semver). A Cargo.toml
stanza like
[dependencies]
serde = "1.0.*"
indicates to cargo
what ranges of semver versions are acceptable for this dependency (see the official
docs for more detail on specifying
precise ranges of acceptable versions).
Because semantic versioning is at the heart of the cargo
's dependency resolution process, this Item explores more
details about what that means.
The essentials of semantic versioning are given by its summary
Given a version number MAJOR.MINOR.PATCH, increment the:
- MAJOR version when you make incompatible API changes,
- MINOR version when you add functionality in a backwards compatible manner, and
- PATCH version when you make backwards compatible bug fixes.
An important detail lurks in the details:
- Once a versioned package has been released, the contents of that version MUST NOT be modified. Any modifications MUST be released as a new version.
Putting this in different words:
- Changing anything requires a new PATCH version.
- Adding things to the API in a way that means existing users of the crate still compile and work requires a MINOR version upgrade.
- Removing or changing things in the API requires a MAJOR version upgrade.
There is one more important codicil to the semver rules:
- Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.
Cargo adapts these rules slightly, "left-shifting" the rules so that changes in the left-most non-zero component indicate incompatible changes. This means that 0.2.3 to 0.3.0 can include an incompatible API change, as can 0.0.4 to 0.0.5.
Semver for Crate Authors
"In theory, theory is the same as practice. In practice, it's not."
As a crate author, the first of these rules is easy to comply with, in theory: if you touch anything, you need a new
release. Using Git tags to match releases can help with this – by default, a tag is fixed to a
particular commit and can only be moved with a manual --force
option. Crates published to
crates.io
also get automatic policing of this, as the registry will reject a second attempt to
publish the same crate version. The main danger for non-compliance is when you notice a mistake just after a release
has gone out, and you have to resist the temptation to just nip in a fix.
However, if your crate is widely depended on, then in practice you may need to be aware of Hyrum's Law: regardless of how minor a change you make to the code, someone out there is likely to depend on the old behaviour.
The difficult part for crate authors is the later rules, which require an accurate determination of whether a change is
back compatible or not. Some changes are obviously incompatible – removing public entrypoints or types, changing
method signatures – and some changes are obviously backwards compatible (e.g. adding a new method to a struct
,
or adding a new constant), but there's a lot of gray area left in between.
To help with this, the Cargo book goes into considerable detail as to what is and is not back-compatible. Most of these details are unsurprising, but there are a few areas worth highlighting.
- Adding new items is usually safe, but may cause clashes if code using the crate already makes use of something that
happens to have the same name as the new item.
- This is a particular danger if the user does a wildcard import from the crate, because all of the crates items are then automatically in the user's main namespace. Item 23 advises against doing this.
- Even without a wildcard import, a new trait method (with a default implementation, Item 13) or a new inherent method has a chance of clashing with an existing name.
- Rust's insistence on covering all possibilities means that changing the set of available possibilities can be a
breaking change.
- Performing a
match
on anenum
must cover all possibilities, so if a crate adds a newenum
variant, that's a breaking change (unless the enum is marked asnon_exhaustive
). - Explicitly creating an instance of a
struct
requires an initial value for all fields, so adding a field to a structure that can be publically instantiated is a breaking change. Structures that have private fields are OK, because crate users can't explicitly construct them anyway; astruct
can also be marked asnon_exhaustive
to prevent external users performing explicit construction.
- Performing a
- Changing a trait so it is no longer object safe (Item 2) is a breaking change; any users that build trait objects for the trait will stop being able to compile their code.
- Adding a new blanket implementation for a trait is a breaking change; any users that already implement the trait will now have two conflicting implementations.
- Changing library code so that it uses a new feature of Rust is an incompatible change: users of your crate who have not yet upgraded their compiler to a version that includes the feature will be broken by the change. Consider whether the minimum supported Rust version (MSRV) is considered to be part of your API1.
- Changing the license of an open-source crate is an incompatible change: users of your crate who have strict restrictions on what licenses are acceptable may be broken by the change. Consider the license to be part of your API.
- Changing the default features (Item 26) of a crate is potentially a breaking change. Removing a default feature is almost certain to break things (unless the feature was already a no-op); adding a default feature may break things depending on what it enables. Consider the default feature set to be part of your API.
An obvious corollary of the rules is this: the fewer public items a crate has, the fewer things there are that can induce an incompatible change (Item 22).
However, there's no escaping the fact that comparing all public API items for compatibility from one release to the next is a time-consuming process that is only likely to yield an approximate (major/minor/patch) assessment of the level of change, at best. Given that this comparison is a somewhat mechanical process, hopefully tooling (Item 31) will arrive to make the process easier2.
If you do need to make an incompatible MAJOR version change, it's nice to make life easier for your users by ensuring that the same overall functionality is available after the change, even the API has radically changed. If possible, the most helpful sequence for your crate users is to:
- Release a MINOR version update that includes the new version of the API, and which marks the older variant as
deprecated
, including an indication of how to migrate. - Subsequently release a MAJOR version update that removes the deprecated parts of the API.
A more subtle point is: make breaking changes breaking. If your crate is changing its behaviour in a way that's actually incompatible for existing users, but which could re-use the same API: don't. Force a change in types (and a MAJOR version bump) to ensure that users can't inadvertantly use the new version incorrectly.
For the less tangible parts of your API – such as the MSRV or
the license – consider setting up a continuous integration check (Item 32) that detects changes, using tooling
(e.g. cargo deny
, see Item 31) as needed.
Finally, don't be afraid of version 1.0.0 because it's a commitment that your API is now fixed. Lots of crates fall into the trap of staying at version 0.x forever, but that reduces the already-limited expressivity of semver from three categories (major/minor/patch) to two (effective-major/effective-minor).
Semver for Crate Users
As a user of a crate, the theoretical expectations for a new version of a dependency are:
- A new PATCH version of a dependency crate Should Just Work™.
- A new MINOR version of a dependency crate Should Just Work™, but the new parts of the API might be worth exploring to see if there are cleaner/better ways of using the crate now. However, if you do use the new parts you won't be able to revert the dependency back to the old version.
- All bets are off for a new MAJOR version of a dependency; chances are that your code will no longer compile and you'll need to re-write parts of your code to comply with the new API. Even if your code does still compile, you should check that your use of the API is still valid after a MAJOR version change, because the constraints and preconditions of the library may have changed.
In practice, even the first two types of change may cause unexpected behaviour changes, even in code that still compiles fine, due to Hyrum's Law.
As a consequence of these expectations, your dependency specifications will commonly take a form like "1.4.*"
or
"0.7.*"
; avoid specifying a completely wildcard dependency like "*"
or "0.*"
. A completely wildcard dependency
says that any version of the depdendency, with any API, can be used by your crate – which is unlikely to be
what you really want.
However, in the longer term it's not safe to just ignore major version changes in dependencies. Once a library has had
a major version change, the chances are that no further bug fixes – and more importantly, security updates –
will be made to the previous major version. A version specification like "1.4.*"
will then fall further and further
behind, with any security problems left unaddressed.
As a result, you either need to accept the risks of being stuck on an old version, or you need to eventually follow
major version upgrades to your dependencies. Tools such as cargo update
or Dependabot
(Item 31), can let you know when updates are available; you can then schedule the upgrade for a time that's convenient for
you.
Discussion
Semantic versioning has a cost: every change to a crate has to be assessed against its criteria, to decide the appropriate type of version bump. Semantic versioning is also a blunt tool: at best, it reflects a crate owner's guess as to which of three categories the current release falls into. Not everyone gets it right, not everything is clear-cut about exactly what "right" means, and even if you get it right, there's always a chance you may fall foul of Hyrum's Law.
However, semver is the only game in town for anyone who doesn't have the luxury of working in a highly-tested monorepo that contains all the code in the world. As such, understanding its concepts and limitations is necessary for managing dependencies.
1: However, opinions vary as to whether MSRV increases should be considered a breaking change.
2: rust-semverver
is a tool
that attempts to do something along these lines.
Item 22: Minimize visibility
Rust's basic unit of visibility is the module; by default, a module's items (types, methods, constants) are private and only accessible to code in the same module and its submodules.
Code that needs to be more widely available is marked with the pub
keyword, making it public to some other scope. A
bare pub
is the most common version, which makes the item visible to anything that's able to see the module it's in.
That last detail is important; if a somecrate::somemodule
module isn't visible to other code in the first place,
anything that's pub
inside it is still not visible.
The more-specific variants of pub
are as follows, in descending order of usefulness:
pub(crate)
is accessible anywhere within the owning crate. Another way of achieving the same effect is to have apub
item in a non-pub
module of the crate, butpub(crate)
allows the item to live near the code it is relevant for.pub(super)
is accessible to the parent module of the current module, which is occasionally useful for selectively increasing visibility in a crate that has a deep module structure.pub(in <path>)
is accesssible to code in<path>
, which has to be a description of some ancestor module of the current module. This is even more occasionally useful for selectively increasing visibility in a crate that has an even deeper module structure.pub(self)
is equivalent topub(in self)
which is equivalent to not beingpub
. Uses for this are very obscure, such as reducing the number of special cases needed in code generation macros.
The Rust compiler will warn you if you have a code item that is private to the module, but which is not used within that module (and its submodules):
// Private function that's been written but which is not yet used.
fn not_used_yet(x: i32) -> i32 {
x + 3
}
Although the warning mentions code that is "never used", it's often really a warning that code can't be used from outside the module.
warning: function is never used: `not_used_yet`
--> visibility/src/main.rs:47:8
|
47 | fn not_used_yet(x: i32) -> i32 {
| ^^^^^^^^^^^^
|
= note: `#[warn(dead_code)]` on by default
warning: 1 warning emitted
Separately from the question of how to increase visibility, is the question of when to do so. The answer: as little as possible, at least for code that's intended to be re-used as a self-contained crate (i.e. not a self-contained or experimental project that will never be re-used).
Once a crate item is public, it can't be made private again without breaking any code that uses the crate, thus
necessitating a major version bump (Item 21). The converse is not true: moving a private item to be public generally
only needs a minor version bump, and leaves crate users unaffected. (Read through the API compatibility
guidelines and notice how many are only
relevant if there are pub
items in play).
This advice is by no means unique to this Item, nor unique to Rust:
- The Rust API guideines includes advice that
- Effective Java (3rd edition) has:
- Item 15: Minimize the accessibility of classes and members
- Item 16: In public classes, use accessor methods, not public fields
- Effective C++ (2nd edition) has:
- Item 18: Strive for class interfaces that are complete and minimal (my italics)
- Item 20: Avoid data members in the public interface
Item 23: Avoid wildcard imports
Rust's use
statement pulls in a named item from another crate or module, and makes that name available for use in the
local module's code without qualification. A wildcard import (or glob import) of the form use somecrate::module::*
says that every public symbol from that module should be added to the local namespace.
As described in Item 21, an external crate may add new items to its API as part of a minor version upgrade; this is considered a backwards compatible change.
The combination of these two observations raises the worry that a non-breaking change to a dependency might break your code: what happens if the dependency adds a new symbol that clashes with a name you're already using?
At the simplest level, this turns out not to be a problem: the names in a wildcard import are treated as being lower priority, so any matching names that are in your code take precedence:
use bytes::*;
// Local `Bytes` type does not clash with `bytes::Bytes`.
struct Bytes(Vec<u8>);
Unfortunately, there are still cases where clashes can occur; for example, if the dependency adds a new trait and
implements it for some type T
, then if any method names from the new trait clash with existing method names in your
own code for T
trait BytesLeft {
// Method name clashes with wildcard-imported `bytes::Buf::remaining`...
fn remaining(&self) -> usize;
}
impl BytesLeft for &[u8] {
// ... and implementation for `&[u8]` clashes with `bytes::Buf` impl.
fn remaining(&self) -> usize {
self.len()
}
}
let arr = [1u8, 2u8, 3u8];
let v = &arr[1..];
assert_eq!(v.remaining(), 3);
then a compile-time error is the result:
error[E0034]: multiple applicable items in scope
--> wildcard/src/main.rs:31:22
|
31 | assert_eq!(v.remaining(), 3);
| ^^^^^^^^^ multiple `remaining` found
|
note: candidate #1 is defined in an impl of the trait `BytesLeft` for the type `&[u8]`
--> wildcard/src/main.rs:16:5
|
16 | fn remaining(&self) -> usize {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= note: candidate #2 is defined in an impl of the trait `bytes::Buf` for the type `&[u8]`
help: disambiguate the associated function for candidate #1
|
31 | assert_eq!(BytesLeft::remaining(&v), 3);
| ^^^^^^^^^^^^^^^^^^^^^^^^
help: disambiguate the associated function for candidate #2
|
31 | assert_eq!(bytes::Buf::remaining(&v), 3);
| ^^^^^^^^^^^^^^^^^^^^^^^^^
As a result, you should avoid wildcard imports from crates that you don't control. There are a couple of common exceptions to this advice, though.
Firstly, if you control the source of the wildcard import, then the concerns given above disappear. For example, it's
common for a test
module to do import super::*;
. It's also possible for crates that use modules primarily as a way
of dividing up code to have:
mod thing;
pub use thing::*;
Secondly, some crates have a convention that common items for the crate are re-exported from a prelude module, which is explicitly intended to be wildcard imported:
use thing::prelude::*;
Although in theory the same concerns apply in this case, in practice such a prelude module is likely to be carefully curated, and higher convenience may outweigh a small risk of future problems.
Finally, if you don't follow the advice of this Item, consider pinning dependencies that you wildcard import to a precise version, so that minor version upgrades of the dependency aren't automatically allowed.
Item 24: Re-export dependencies whose types appear in your API
The title of this Item is a little convoluted, but working through an example will make things clearer.
Item 25 describes how cargo
supports different versions of the same library crate being linked into a single binary,
in a transparent manner. Consider a binary that uses the rand
crate; more specifically, one
which uses some 0.8 version of the crate:
# Top-level binary crate
[dependencies]
dep-lib = "0.1.0"
rand = "0.8.*"
let mut rng = rand::thread_rng(); // rand 0.8
let max: usize = rng.gen_range(5..10);
let choice = dep_lib::pick_number(max);
The final line of code also uses a notional dep-lib
crate, and this crate internally uses1 a 0.7 version of the rand
crate:
# dep-lib library crate
[dependencies]
rand = "0.7.3"
use rand::Rng;
/// Pick a number between 0 and n (exclusive).
pub fn pick_number(n: usize) -> usize {
rand::thread_rng().gen_range(0, n)
}
An eagle-eyed reader might notice a difference between the two code examples:
- In version 0.7.x of
rand
(as used by thedep-lib
library crate), therand::gen_range()
method takes two non-self
parameters,low
andhigh
. - In version 0.8.x of
rand
(as used by the binary crate), therand::gen_range()
method takes a single non-self
parameterrange
.
This is a non-back-compatible change and so rand
has increased its leftmost version component accordingly, as required
by semantic versioning (Item 21). Nevertheless, the binary that combines the two incompatible versions works just
fine; cargo
sorts everything out2.
However, things get a lot more awkward if the library crate's API exposes a type from its dependency, making that
dependency a public
dependency.
In the example,
this involves an Rng
item – but specifically a version-0.7 Rng
item:
pub fn pick_number_with<R: Rng>(rng: &mut R, n: usize) -> usize {
rng.gen_range(0, n) // Method from the 0.7.x version of Rng
}
As an aside, think carefully before using another crate's types in your API: it intimately ties your crate to that of the dependency. For example, a major version bump for the dependency (Item 21) will automatically require a major version bump for your crate too.
In this case, rand
is a semi-standard crate that is high quality and widely used, and which only pulls in a small
number of dependencies of its own (Item 25), so including its types in the crate API is probably fine on balance.
Returning to the example, an attempt to use this entrypoint from the top-level binary fails:
let mut rng = rand::thread_rng();
let max: usize = rng.gen_range(5..10);
let choice = dep_lib::pick_number_with(&mut rng, max);
Unusually for Rust, the compiler error message isn't very helpful:
error[E0277]: the trait bound `ThreadRng: rand_core::RngCore` is not satisfied
--> re-export/src/main.rs:17:48
|
17 | let choice = dep_lib::pick_number_with(&mut rng, max);
| ^^^^^^^^ the trait `rand_core::RngCore` is not implemented for `ThreadRng`
|
::: /Users/dmd/src/effective-rust/examples/dep-lib/src/lib.rs:17:28
|
17 | pub fn pick_number_with<R: Rng>(rng: &mut R, n: usize) -> usize {
| --- required by this bound in `pick_number_with`
|
= note: required because of the requirements on the impl of `rand::Rng` for `ThreadRng`
Investigating the types involved leads to confusion because the relevant traits do appear to be implemented –
but the caller actually implements a RngCore_v0_8_3
and the library is expecting an implementation of RngCore_v0_7_3
.
Once you've finally deciphered the error message and realized that the version clash is the underlying cause, how can you fix it? The key observation is to realize that the while the binary can't directly use two different versions of the same crate, it can do so indirectly (as in the original example above).
From the perspective of the binary author, the problem can be worked around by adding an intermediate wrapper crate that
hides the naked use of rand
v0.7 types.
This is awkward, and a much better approach is available to the author of the library crate. It can make like easier for its users by explicitly re-exporting either:
- the types involved in the API
- the entire dependency crate.
For the example, the latter approach work best: as well as making the version 0.7 Rng
and RngCore
types available,
it also makes available the methods (like thread_rng
) that construct instances of the type:
// Re-export the version of `rand` used in this crate's API.
pub use rand;
The calling code now has a different way to directly refer to version 0.7 of rand
, as dep_lib::rand
:
let mut prev_rng = dep_lib::rand::thread_rng(); // v0.7 Rng instance
let choice = dep_lib::pick_number_with(&mut prev_rng, max);
With this example in mind, the advice of the title should now be a little less obscure: re-export dependencies whose types appear in your API.
1: This example (and indeed Item) is inspired by the approach used in the RustCrypto crates.
2: This is possible because the Rust toolchain handles linking, and does not have the constraint that C++ inherits from C of needing to support separate compilation.
Item 25: Manage your dependency graph
Like most modern programming languages, Rust makes it easy to pull in external libraries, in the form of crates. By default, Cargo will:
- download any crates named in the
[dependencies]
section of yourCargo.toml
file fromcrates.io
, - finding versions that match the preferences configured in
Cargo.toml
.
There are a few subtleties lurking underneath this simple statement. The first thing to notice is that crate names form a single flat namespace (and this global namespace also overlaps with the names of features in a crate, see Item 26). Names are generally allocated on a first-come, first-served basis, so you may find that your preferred name for a public crate is already taken. (However, name-squatting – reserving a crate name by pre-registering an empty crate – is frowned upon, unless you really are going to release in the near future.)
As a minor wrinkle, there's also a slight difference between what's allowed as a crate name in this namespace, and
what's allowed as an identifier in code: a crate can be named some-crate
but it will appear in code as some_crate
(with an underscore). To put it another way: if you see some_crate
in code, the corresponding crate name may be
either some-crate
or some_crate
.
The second aspect to be aware of is Cargo's version selection
algorithm. Each Cargo.toml
dependency line specifies an
acceptable range of versions, according to semver (Item 21) rules, and Cargo takes this into account when the same
crate appears in multiple places in the dependency graph. If the acceptable ranges overlap and are semver-compatible,
then Cargo will pick the most recent version of the crate within the overlap.
However, if there is no semver-compatible overlap, then Cargo will build multiple copies of the dependency at different versions. This can lead to confusion if the dependency is exposed in some way rather than just being used internally (Item 24) – the compiler will treat the two versions as being distinct crates, but its error messages won't necessarily make that clear.
Allowing multiple versions of a crate can also go wrong if the crate includes C/C++ code accessed via Rust's FFI
mechanisms (Item 38) . The Rust toolchain can internally disambiguate distinct versions of Rust code, but any included
C/C++ code is subject to the one definition rule: there can only
be a single version of any function, constant or global variable. This is most commonly encountered with the
*ring*
cryptographic library, because it includes parts of the BoringSSL
library (which is written in C and assembler).
The third subtlety of Cargo's resolution process to be aware of is feature unification: the features that get activated for a dependent crate are the union of the features selected by different places in the dependency graph; see Item 26 for more details.
Once Cargo has picked acceptable versions for all dependencies, its choices are recorded in the Cargo.lock
file.
Subsequent builds will then re-use the choices encoded in Cargo.lock
, so that the build is stable and no new downloads
are needed.
This leaves you with a choice: should you commit your Cargo.lock
files into version control or not?
The advice from the Cargo developers is that:
- Things that produce a final product, namely applications and binaries, should commit
Cargo.lock
to ensure a deterministic build. - Library crates should not commit a
Cargo.lock
file, because it's irrelevant to any downstream consumers of the library – they will have their ownCargo.lock
file; be aware that theCargo.lock
file for a library crate is ignored by library users.
Even for a library crate, it can be helpful to have a checked-in Cargo.lock
file to ensure that regular builds and
continuous integration (Item 32) don't have a moving target. Although the promises of semantic versioning (Item 21) should
prevent failures in theory, mistakes happen in practice and it's frustrating to have builds that fail because
someone somewhere recently changed a dependency of a dependency.
However, if you version control Cargo.lock
, set up a process to handle upgrades (such as GitHub's
dependabot). If you don't, your dependencies don't
stay pinned to versions that get older, outdated and potentially insecure.
Pinning versions with a checked-in Cargo.lock
file doesn't avoid the pain of handling dependency upgrades, but it does
mean that you can handle them at a time of your own choosing, rather than immediately when the upstream crate
changes. There's also some fraction of dependency upgrade problems that go away on their own: a crate that's released
with a problem often gets a second, fixed, version released in a short space of time, and a batched upgrade process
might only see the latter version.
Version Specification
The version specification clause for a dependency defines a range of allowed versions, according to the rules explained in the Cargo book.
- Avoid too-specific a version dependency: pinning to a specific version (
"=1.2.3"
) is usually a bad idea: you don't see newer versions (potentially including security fixes), and you dramatically narrow the potential overlap range with other crates in the graph that rely on the same dependency (recall that Cargo only allows a single version of a crate to be used within a semver-compatible range). - Avoid too-general a version dependency: it's possible to specify a version dependency (
"*"
) that allows non-semver compatible versions to be included, but it's a bad idea: do you really mean that the crate can completely change every aspect of its API and your code will still work? Thought not.
The most common Goldilocks specification is to allow semver-compatible versions ("1.*"
) of a crate, possibly with a
specific minimum version that includes a feature or fix that you require ("^1.4.23"
)
Solving Problems with Tooling
Item 31 recommends that you take advantage of the range of tools that are available within the Rust ecosystem; this section describes some dependency graph problems where tools can help.
The compiler will tell you pretty quickly if you use a dependency in your code, but don't include that dependency in
Cargo.toml
. But what about the other way around? If there's a dependency in Cargo.toml
that you don't use in
your code – or more likely, no longer use in your code – then Cargo will go on with its business. The
cargo-udeps
tool is designed to solve exactly this problem: it warns you when
your Cargo.toml
includes an unused dependency ("udep").
A more versatile tool is cargo deny
, which analyzes your dependency graph to
detect a variety of potential problems across the full set of transitive dependencies:
- Dependencies that have known security problems in the included version.
- Dependencies that are covered by an unacceptable license.
- Dependencies that are just unacceptable.
- Dependencies that are included in multiple different versions across the dependency tree.
Each of these features can be configured and can have exceptions specified; the latter inevitably becomes necessary for large projects, particularly for the multiple-version warning.
These tools can be run as a one-off, but it's better to ensure they're executed regularly and reliably by including them in your continuous integration system (Item 32). This helps to catch newly-introduced problems – including problems that may been introduced outside of your code, in an upstream dependency (for example, a newly reported vulnerability).
If one of these tools does report a problem, it can be difficult to figure out exactly where in the dependency graph the
problem arises. The cargo tree
command that's included
with cargo
helps here; it shows the dependency graph as a tree structure.
dep-graph v0.1.0
├── dep-lib v0.1.0
│ └── rand v0.7.3
│ ├── getrandom v0.1.16
│ │ ├── cfg-if v1.0.0
│ │ └── libc v0.2.94
│ ├── libc v0.2.94
│ ├── rand_chacha v0.2.2
│ │ ├── ppv-lite86 v0.2.10
│ │ └── rand_core v0.5.1
│ │ └── getrandom v0.1.16 (*)
│ └── rand_core v0.5.1 (*)
└── rand v0.8.3
├── libc v0.2.94
├── rand_chacha v0.3.0
│ ├── ppv-lite86 v0.2.10
│ └── rand_core v0.6.2
│ └── getrandom v0.2.3
│ ├── cfg-if v1.0.0
│ └── libc v0.2.94
└── rand_core v0.6.2 (*)
cargo tree
includes a variety of options that can help to solve specific problems, including:
--invert
shows what depends on a specific package, helping you to focus on a particular problematic dependency.--edges features
shows what crate features are activated by a dependency link, which helps you figure out what's going on with feature unification (Item 26).
What To Depend On
The previous sections have covered the more mechanical aspect of working with dependencies, but there's a more philosophical (and therefore harder to answer) question: when should you take on a dependency?
Most of the time, there's not much of a decision involved: if you need the functionality of a crate, you need that function and the only alternative would be to write it yourself1.
But every new dependency has a cost: partly in terms of longer builds and bigger binaries, but mostly in terms of the developer effort involved in fixing problems with dependencies when they arise.
The bigger your dependendency graph, the more likely you are to be exposed to these kinds of problems. The Rust crate ecosystem is just as vulnerable to accidental dependency problems as other package ecosystems, where history has shown that one developer yanking a package, or a team fixing the licensing for their package, can have widespread knock-on effects.
(More worrying still are supply chain attacks, where a malicious actor deliberately tries to subvert commonly-used dependencies, whether by typo-squatting or by more sophisticated attacks2.)
So for dependencies that are more "cosmetic", it's sometimes worth considering whether adding the dependency is worth the cost.
The answer is usually "yes", though – in the end, the amount of time spent dealing with dependency problems ends up being much less than the time it would take to write equivalent functionality from scratch.
1: If you are targetting a no_std
environment,
this choice may be made for you: many crates are not compatible with no_std
, particularly if alloc
is also
unavailable (Item 37).
2: Go's module transparency takes a step towards addressing this, by ensuring that all clients see the same code for a given version of a package.
Item 26: Be wary of feature
creep
Rust includes support for conditional compilation,
which is controlled by cfg
(and
cfg_attr
) attributes.
These attributes govern the thing – function, line, block etc. – that they are attached to (in contrast to
C/C++'s line-based preprocessor), based on configuration options that are either plain names (e.g. test
) or key-value pairs
(e.g. panic = "abort"
).
The toolchain populates a variety of config values that describe the target environment, including the OS (target_os
),
CPU architecture (target_arch
), pointer width (target_pointer_width
), and endianness (target_endian
); this allows
for code portability, where features that are specific to some particular target are only compiled in when building for
that target.
The Cargo package manager builds on this base cfg
mechanism to provide
the concept of features: specific named aspects of the
functionality of a crate that can be enabled when building the crate. Cargo ensures that the the feature
option is
populated with the configured values for each crate that it compiles, and the values are crate-specific. This is
Cargo-specific functionality; to the Rust compiler, feature
is just another configuration option.
At the time of writing, the most reliable way to determine what features are available for a crate is to examine the
crate's Cargo.toml
manifest file. For example, the
following chunk of a manifest file includes four features:
[features]
default = ["featureA"]
featureA = []
featureB = []
featureAB = ["featureA", "featureB"]
schema = []
[dependencies]
rand = { version = "^0.8", optional = true }
hex = "^0.4"
However, the four features are not just the four lines in the [features]
stanza; there are a couple of subtleties to
watch out for.
Firstly, the default
line in the [features]
stanza is a special feature name, used to indicate which of the features
should be enabled by default. These features can still be disabled by passing the --no-default-features
flag to the
build command, and a consumer of the crate can encode this in their Cargo.toml
file like so:
[dependencies]
somecrate = { version = "^0.3", default-features = false }
The second subtlety of feature definitions is hidden in the [dependencies]
section of the original Cargo.toml
example: the rand
crate is a dependency that is marked as optional = true
, and that effectively makes "rand"
into
the name of a feature. If the crate is compiled with --features rand
, then that dependency is activated (and the
crate will presumably include code that uses rand
and which is protected by #[cfg(feature = "rand")]
).
This also means that crate names and feature names share a namespace, even though one is global (governed by
crates.io
) and one is local to the crate in question. Consequently, chose feature names carefully to avoid
clashes with the names of any crates that might be relevant as potential dependencies.
So you can determine a crate's features by examining [features]
and optional
[dependencies]
in the crate's
Cargo.toml
file. To turn on a feature of a dependency, add the features
option to the relevant line in the
[dependencies]
stanza of your own manifest file:
[dependencies]
somecrate = { version = "^0.3", features = ["featureA", "rand" ] }
This line ensures that somecrate
will be built with both the featureA
and the rand
feature enabled. However, that
might not be the only features that are enabled, due to a phenomenon known as feature
unification. This means that a crate will
get built with the union of all of the features that are requested by anything in the build graph. In other words,
if some other dependency in the build graph also relies on somecrate
, but with just featureB
enabled, then the crate
will be built with all of featureA
, featureB
and rand
enabled, in order to satisfy everyone1. The same
consideration applies to default features: if your crate sets default-features = false
for a dependency, but some
other place in the build graph leaves the default features enabled, then enabled they will be.
Feature unification means that features should be additive; it's a bad idea to have mutually incompatible features because there's nothing to prevent the incompatible features being simultaneously enabled by different users.
A specific consequence of this applies to public traits, intended to be used outside the crate they're defined in. Consider a trait that includes a feature gate on one of its methods:
/// Trait for items that support CBOR serialization.
pub trait AsCbor: Sized {
/// Convert the item into CBOR-serialized data.
fn serialize(&self) -> Result<Vec<u8>, Error>;
/// Create an instance of the item from CBOR-serialized data.
fn deserialize(data: &[u8]) -> Result<Self, Error>;
/// Return the schema corresponding to this item.
#[cfg(feature = "schema")]
fn cddl(&self) -> String;
}
This leaves external trait implementors in a quandary: should they implement the cddl(&self)
method or not?
For code that doesn't use the schema
feature, the answer seems to obviously be "No" – the code won't compile if
you do. But if something else in the dependency graph does use the schema
feature, an implementation of this method
suddenly becomes required. The external code that tries to implement the trait doesn't know – and can't tell
– whether to implement the feature-gated method or not.
So the net is that you should avoid feature-gating methods on public traits. A trait method with a default implementation (Item 13) might be a partial exception to this – but only if it never makes sense for external code to override the default.
Feature unification also means that if your crate that has N independent2 features, then all of the 2N possible build combinations can occur in practice. To avoid unpleasant surprises, it's a good idea to ensure that your continuous integration system (Item 32) covers all of these 2N combinations, in all of the available test variants (Item 30).
Summing up the aspects of features to be wary of:
- Features should be additive.
- Feature names should be carefully chosen not to clash with potential dependency names.
- Having lots of independent features potentially leads to a combinatorial explosion of different build configurations.
However, the use of optional features is very helpful in controlling exposure to an expanded dependency graph (Item 25). This is particularly useful in low-level crates that are capable of being used in a no_std
environment (Item 37) – it's common to have a std
or alloc
feature that turns on functionality that relies on those libraries.
1: The cargo tree --edges features
command can help with determining which features are enabled for which crates, and why.
2: Features can force other features to be
enabled; in the original example the featureAB
feature forces both featureA
and featureB
to be enabled.
Tooling
Titus Winters (Google's C++ library lead) describes software engineering as programming integrated over time, or sometimes as programming integrated over time and people. Over longer timescales, and a wider team, there's more to a codebase than just the code held within it.
Modern languages, including Rust, are aware of this and come with an ecosystem of tooling that goes way beyond just converting the program into executable binary code (the compiler).
This section explores the Rust tooling ecosystem, with a general recommendation to make use of all of this infrastructure. Obviously, doing so needs to be proportionate – setting up CI, documentation builds, and six types of test would be overkill for a throwaway program that only gets run twice. But for most of the things described in this section, there's lots of "bang for the buck": a little bit of investment into tooling integration will yield worthwhile benefits.
Item 27: Document public interfaces
If your crate is going to be used by other programmers, then it's a good idea to add documentation for its contents, particularly its public API. If your crate is more than just ephemeral, throw-away code, then that "other programmer" includes the you-of-the-future, when you have forgotten the details of your current code.
This is not advice that's specific to Rust, nor is it new advice – for example, Effective Java 2nd edition (from 2008) has Item 44: "Write doc comments for all exposed API elements".
The particulars of Rust's documentation comment format – Markdown-based, delimited with ///
or //!
–
are covered in the Rust
book, but
there are some specific details worth highlighting.
/// Calculate the [`BoundingBox`] that exactly encompasses a pair
/// of [`BoundingBox`] objects.
pub fn union(a: &BoundingBox, b: &BoundingBox) -> BoundingBox {
- Use a code font for code: For anything that would be typed into source code as-is, surround it with back-quotes
to ensure that the resulting documentation is in a fixed-width font, making the distinction between
code
and text clear. - Add copious cross-references: Add a Markdown link for anything that might provide context for someone reading the
documentation. In particular, cross-reference identifiers with the convenient
[`SomeThing`]
syntax – ifSomeThing
is in scope, then the resulting documentation will hyperlink to the right place. - Consider including example code: If it's not trivially obvious how to use an entrypoint, adding an
# Examples
section with sample code can be helpful. - Document panics and
unsafe
constraints: If there are inputs that cause a function to panic, document (in a# Panics
section) the preconditions that are required to avoid thepanic!
. Similarly, document (in a# Safety
section) any requirements forunsafe
code.
The documentation for Rust's standard library provides an excellent example to emulate for all of these details.
Tooling
The Markdown format that's used for documentation comments results in elegant output, but this does also mean that there
is an explicit conversion step (cargo doc
). This in turn raises the possibility that something goes wrong along the
way.
The simplest advice for this is just to read the rendered documentation after writing it, by running cargo doc --open
.
You could also check that all the generated hyperlinks are valid, but that's a job more suited to a machine – via
the broken_intra_doc_links
crate attribute1:
#![deny(broken_intra_doc_links)]
/// The bounding box for a [`Polygone`].
#[derive(Clone, Debug)]
pub struct BoundingBox {
With this attribute enabled, cargo doc
will detect invalid links:
error: unresolved link to `Polygone`
--> docs/src/main.rs:4:29
|
4 | /// The bounding box for a [`Polygone`].
| ^^^^^^^^^^ no item named `Polygone` in scope
|
note: the lint level is defined here
--> docs/src/main.rs:2:9
|
2 | #![deny(broken_intra_doc_links)]
| ^^^^^^^^^^^^^^^^^^^^^^
= help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
error: aborting due to previous error
error: could not document `docs`
You can also require documentation, by enabling the #![warn(missing_docs)]
attribute for the crate. When this is
enabled, the compiler will emit a warning for every undocumented public item. However, there's a risk that enabling
this option will lead to poor quality documentation comments that are rushed out just to get the compiler to shut up
– more on this below.
As ever, any tooling that detects potential problems should form a part of your continuous integration system (Item 32), to catch any regressions that creep in.
Additional Documentation Locations
The output from cargo doc
is the primary place where your crate is documented, but it's not the only place –
other parts of a Cargo project can help users figure out how to use your code.
The examples/
subdirectory of a Cargo project can hold the code for standalone binaries that make use of your crate.
These programs are built and run very similarly to integration tests (Item 30), but are specifically intended to hold
example code that illustrates the correct use of your crate's interface.
On a related note, bear in mind that the the integration tests under the tests/
subdirectory can also serve as
examples for the confused user, even though their primary purpose is to test the crate's external interface.
For a published crate, any top-level README.md
file2 for the project is presented
as the main page content on the crates.io
listing for the crate (for example, see the rand crate
page). However, this content is not easily visible on the documentation pages, and so
serves a somewhat different purpose – it's aimed at people who are choosing what crate to use, rather than
figuring out how to use a crate they've already included (although there's obviously considerable overlap between the
two).
What Not to Document
When a project requires that documentation be included for all public items (as mentioned in the first section), it's very easy to fall into the trap of having documentation that's a pointless waste of valuable pixels. Having the compiler warn about missing doc comments is only a proxy for what you really want – useful documentation – and is likely to incentivize programmers to do the minimum needed to silence the warning.
Good doc comments are a boon that help users understand the code they're using; bad doc comments impose a maintenance burden and increase the chance of user confusion when they get out of sync with the code. So how to distinguish between the two?
The primary advice is to avoid repeating in text something that's clear from the code. Item 1 exhorted you to encode as much semantics as possible into Rust's type system; once you've done that, allow the type system to document those semantics. Assume that the reader is familiar with Rust – possibly because they've read a helpful collection of Items describing effective use of the language – and don't repeat things that are clear from the signatures and types involved.
Returning to the previous example, an overly-verbose documentation comment might be:
/// Return a new [`BoundingBox`] object that exactly encompasses a pair
/// of [`BoundingBox`] objects.
///
/// Parameters:
/// - `a`: an immutable reference to a `BoundingBox`
/// - `b`: an immutable reference to a `BoundingBox`
/// Returns: new `BoundingBox` object.
pub fn union(a: &BoundingBox, b: &BoundingBox) -> BoundingBox {
This comment repeats many details that are clear from the function signature, to no benefit.
Worse, consider what's likely to happen if the code gets refactored to store the result in one of the original arguments (which would be a breaking change, see Item 21). No compiler or tool complains that the comment isn't updated to match, so it's easy to end up with an out-of-sync comment:
/// Return a new [`BoundingBox`] object that exactly encompasses a pair
/// of [`BoundingBox`] objects.
///
/// Parameters:
/// - `a`: an immutable reference to a `BoundingBox`
/// - `b`: an immutable reference to a `BoundingBox`
/// Returns: new `BoundingBox` object.
pub fn union(a: &mut BoundingBox, b: &BoundingBox) {
In contrast, the original comment survives the refactoring unscathed, because its text describes behaviour not syntactic details:
/// Calculate the [`BoundingBox`] that exactly encompasses a pair
/// of [`BoundingBox`] objects.
pub fn union(a: &mut BoundingBox, b: &BoundingBox) {
The mirror image of the advice above also helps improve documentation: include in text anything that's not clear from the code. This includes preconditions, invariants, panics, error conditions and anything else that might surprise a user; if your code can't comply with the principle of least astonishment, make sure that the surprises are documented so you can at least say "I told you so".
Another common failure mode is when doc comments describe how some other code uses a method, rather than what the method does.
/// Return the intersection of two [`BoundingBox`] objects, returning `None`
/// if there is no intersection. The collision detection code in `hits.rs`
/// uses this to do an initial check whether two objects might overlap, before
/// perfoming the more expensive pixel-by-pixel check in `objects_overlap`.
pub fn intersection(
a: &BoundingBox,
b: &BoundingBox,
) -> Option<BoundingBox> {
Comments like this are almost guaranteed to get out of sync: when the using code (here hits.rs
) changes, the comment that
describes the behaviour is nowhere nearby.
Rewording the comment to focus more on the why makes it more robust to future changes:
/// Return the intersection of two [`BoundingBox`] objects, returning `None`
/// if there is no intersection. Note that intersection of bounding boxes
/// is necessary but not sufficient for object collision -- pixel-by-pixel
/// checks are still required on overlap.
pub fn intersection(
a: &BoundingBox,
b: &BoundingBox,
) -> Option<BoundingBox> {
When writing software, it's good advice to "program in the future tense"3: structure the code to accommodate future changes. The same is true for documentation (albeit not literally): focusing on the semantics, the whys and the why nots, gives text that is more likely to remain helpful in the long run.
1: Historically, this option used to be called
intra_doc_link_resolution_failure
.
2: The default behaviour of automatically including
README.md
can be overridden with the readme
field in
Cargo.toml
.
3: Scott Meyers "More Effective C++" Item 32.
Item 28: Use macros judiciously
"In some cases it's easy to decide to write a macro instead of a function, because only a macro can do what's needed" – Paul Graham, "On Lisp"
Item 29: Listen to Clippy
"It looks like you're writing a letter. Would you like help?" – Microsoft Clippit
Item 31 describes the ecosystem of helpful tools available in the Rust toolbox, but one tool is sufficiently helpful and important to get promoted to an Item of its very own: Clippy.
Clippy is an additional component for cargo that emits warnings about your Rust usage (cargo clippy ...
), across a
variety of categories:
- Correctness: warn about common programming errors.
- Idiom: warn about code constructs that aren't quite in standard Rust style.
- Concision: point out variations on the code that are more compact.
- Performance: suggest alternatives that avoid unnecessary processing or allocation.
- Readability: describe alterations to the code that would make it easier for humans to read and understand.
For example, the following code builds fine:
pub fn circle_area(radius: f64) -> f64 {
let pi = 3.14;
pi * radius * radius
}
but Clippy points out that the local approximation to π is unnecessary and inaccurate:
error: approximate value of `f{32, 64}::consts::PI` found. Consider using it directly
--> clippy/src/main.rs:5:18
|
5 | let pi = 3.14;
| ^^^^
|
= note: `#[deny(clippy::approx_constant)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#approx_constant
error: aborting due to previous error
error: could not compile `clippy`
The linked webpage explains the problem, and points the way to a suitable modification of the code:
pub fn circle_area(radius: f64) -> f64 {
std::f64::consts::PI * radius * radius
}
As shown above, each Clippy warning comes with a link to a webpage describing the error, which explains why the code is considered bad. This is vital, because it allows you to decide whether those reasons apply to your code, or whether there is some particular reason why the lint check isn't relevant. In some cases, the text also describes known problems with the lint, which might explain an otherwise confusing false positive.
If you decide that a lint warning isn't relevant for your code, you can disable it either for that particular item
(#[allow(clippy::some-lint)]
) or for the entire crate (#![allow(clippy::some-lint)]
). However, it's usually better
to take the cost of a minor refactoring of the code than to waste time and energy arguing about whether the warning is a
genuine false positive.
Whether you choose to fix or disable the warnings, you should make your code Clippy warning free.
That way, when new warnings appear – whether because the code has been changed, or because Clippy has been upgraded to include new checks – they will be obvious. Clippy should also be enabled in your continuous integration system (Item 32).
Clippy's warnings are particularly helpful when you're learning Rust, because they reveal gotchas you might not have noticed, and help you become familiar with Rust idiom.
Many of the Items in this book also have corresponding Clippy warnings, when it's possible to mechanically check the relevant concern:
- Item 1 suggests using more expressive types than plain
bool
s, and Clippy will also point out the use of multiplebool
s in function parameters and structures. - Item 3 covers manipulations of
Option
andResult
types, and Clippy points out a few possible redundancies, such as: - Item 3 also suggest that errors should be returned to the caller where possible; Clippy points out some missing opportunities to do that.
- Item 5 described Rust's standard traits, and included some implementation requirements that Clippy checks:
- Item 6 suggests implementing
From
rather thanInto
, which Clippy also suggests. - Item 6 also described casts, and Clippy can warn on:
- Item 9 describes fat pointer types, and various Clippy lints point out scenarios where there are unnecessary extra pointer indirection:
- Item 10 describes the panoply of different ways to manipulate iterators; Clippy includes a truly astonishing number of lints that point out combinations of iterator methods that could be simplified.
- Item 18 suggests limiting the use of
panic!
or related methods likeexpect
, which Clippy also detects. - Item 21 observes that importing a wildcard version of a crate isn't sensible; Clippy agrees.
- Item 23 suggests avoiding wildcard imports, as does Clippy.
- Item 24 and Item 25 touch on the fact that multiple versions of the same crate can appear in your dependency graph; Clippy can be configured to complain when this happens.
- Item 26 explains the additive nature of Cargo features, and Clippy includes a warning about "negative" feature
names (e.g.
"no-std"
) that are likely to indicate a feature that falls foul of this. - Item 26 also explains that a crate's optional dependencies form part of its feature set, and Clippy warns if there
are explicit feature names
(e.g.
"use-crate-x"
) that could just make use of this instead. - Item 27 describes conventions for documentation comments, and Clippy will also point out:
As the size of this list should make clear, it can be a valuable learning experience to read the list of Clippy lint warnings – including the pedantic or high-false-positive checks that are disabled by default. Even though you're unlikely to want to enable these warnings for your code, understanding the reasons why they were written in the first place will improve your understanding of Rust and its idiom.
Item 30: Write more than unit tests
"All companies have test environments.
The lucky ones have production environments separate from the test environment." – @FearlessSon
Like most other modern languages, Rust includes features that make it easy to write tests that live alongside your code, and which give confidence that the code is working correctly.
This isn't the place to expound on the importance of tests; suffice it to say that if code isn't tested, it probably doesn't work the way you think it does. So this Item assumes that you're already signed up to write tests for your code.
Unit tests and integration tests, described in the next two sections, are the key forms of test. However, the Rust toolchain and extensions to it allow for various other types of test; this Item describes their distinct logistics and rationales.
Unit Tests
The most common form of test for Rust code is a unit test, which might look something like:
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_nat_subtract() {
assert_eq!(nat_subtract(4, 3).unwrap(), 1);
assert_eq!(nat_subtract(4, 5), None);
}
#[should_panic]
#[test]
fn test_something_that_panics() {
nat_subtract_unchecked(4, 5);
}
}
Some aspects of this example will appear in every unit test:
- a collection of unit test functions, which are…
- marked with the
#[test]
attribute, and included within… - a
#[cfg(test)]
attribute, so the code only gets built in test configurations.
Other aspects of this example illustrate things that are optional, and may only be relevant for particular tests:
- The test code here is held in a separate module, conventionally called
tests
ortest
. This module may be inline (as here), or held in a separatetests.rs
file. - The test module may have a wildcard
use super::*
to pull in everything from the parent module under test. This makes it more convenient to add tests (and is an exception to the general advice of Item 23 to avoid wildcard imports). - A unit test has the ability to use anything from the parent module, whether it is
pub
or not. This allows for "whitebox" testing of the code, where the unit tests exercise internal features that aren't visible to normal users. - The test code makes use of
unwrap()
for its expected results; the advice of Item 18 isn't really relevant for test-only code, wherepanic!
is used to signal a failing test. Similarly, the test code also checks expected results withassert_eq!
, which will panic on failure. - The code under test includes a function that panics on some kinds of invalid input, and the tests exercise that in a
test that's marked with the
#[should_panic]
attribute. This might be a internal function that normally expects the rest of the code to respect its invariants and preconditions, or it might be a public function that has some reason to ignore the advice of Item 18. (Such a function should have a "Panics" section in its doc comment, as described in Item 27.)
Item 27 suggests not documenting things that are already expressed by the type system; similarly, there's no need to
test things that are guaranteed by the type system. If your enum
types start start holding values that aren't in the
list of allowed variants, you've got bigger problems than a failing unit test!
However, if your code relies on specific functionality from your dependencies, it can be helpful to include basic tests of that functionality. The aim here is not to repeat testing that's already done by the dependency itself, but instead to have an early warning system that indicates whether it's safe to include a new version of that dependency in practice – separately from whether the semantic version number (Item 21) indicates that the new version is safe in theory.
Integration Tests
The other common form of test included with a Rust project is integration tests, held under tests/
. Each file in
that directory is run as a separate test program that executes all of the functions marked with #[test]
.
Integration tests do not have access to crate internals, and so act as black-box tests that can only exercise the public API of the crate.
Doc Tests
Item 27 described the inclusion of short code samples in documentation comments, to illustrate the use of a particular
public API item. Each such chunk of code is enclosed in an implicit fn main() { ... }
and run as part of cargo test
, effectively making it an additional test case for your code, known as a doc test. Individual tests can also
be executed selectively by running cargo test --doc <item-name>
.
Assuming that you regularly run tests as part of your continuous integration environment (Item 32), this ensures that your code samples don't drift too far from the current reality of your API.
Examples
Item 27 also described the ability to provide example programs that exercise your public API. Each Rust file under
examples/
(or each subdirectory under examples/
that includes a main.rs
) can be run as a standalone binary
with cargo run --example <name>
or cargo test --example <name>
.
These programs only have access to the public API of your crate, and are intended to illustrate the use of your API as a
whole. Examples are not specifically designated as test code (no #[test]
, no #[cfg(test)]
), and they're a poor
place to put code that exercises obscure nooks and crannies of your crate –particularly as examples are not
run by cargo test
by default.
Nevertheless, it's a good idea to ensure that your continuous integration system (Item 32) builds and runs all the
associated examples for a crate (with cargo test --examples
), because it can act as a good early warning system for
regressions that are likely to affect lots of users. As noted above, if your examples demonstrate mainline use of your
API, then a failure in the examples implies that something significant is wrong.
- If it's a genuine bug, then it's likely to affect lots of users – the very nature of example code means that users are likely to have copied, pasted and adapted the example.
- If it's an intended change to the API, then the examples need to be updated to match. A change to the API also implies a backwards incompatibility, so if the crate is published then the semantic version number needs a corresponding update to indicate this (Item 21).
The likelihood of users copying and pasting example code means that it should have a different style than test code. In
line with Item 18, you should set a good example for your users by avoiding unwrap()
calls for Result
s. Instead,
make each example's main()
function return something like Result<(), Box<dyn Error>>
, and then use the question mark
operator throughout (Item 3).
Benchmarks
Item 20 attempts to persuade you that fully optimizing the performance of your code isn't always necessary. Nevertheless, there are definitely still times when performance is critical, and if that's the case then it's a good idea to measure and track that performance. Having benchmarks that are run regularly (e.g. as part of continuous integration, Item 32) allows you to detect when changes to the code or the toolchains adversely affect that performance.
The cargo bench
command1 runs special test cases that
repeatedly perform an operation, and emits average timing information for the operation.
However, there's a danger that compiler optimizations may give misleading results, particularly if you restrict the operation that's being performed to a small subset of the real code. Consider a simple arithmetic function:
pub fn factorial(n: u128) -> u128 {
match n {
0 => 1,
n => n * factorial(n - 1),
}
}
A naïve benchmark for this code:
#[bench]
fn bench_factorial(b: &mut Bencher) {
b.iter(|| {
let result = factorial(15);
assert_eq!(result, 1_307_674_368_000);
});
}
gives incredibly positive results:
test naive::bench_factorial ... bench: 0 ns/iter (+/- 0)
With fixed inputs and a small amount of code under test, the compiler is able to optimize away the iteration and directly emit the result, leading to an unrealistically optimistic result.
The (experimental) std::hint::black_box
function can help with
this; it's an identity function whose implementation the compiler is "encouraged, but not
required" (their italics) to pessimize.
Moving the code under test to use this hint:
#![feature(bench_black_box)] // nightly-only
pub fn factorial(n: u128) -> u128 {
match n {
0 => 1,
n => n * std::hint::black_box(factorial(n - 1)),
}
}
gives more realistic results:
test bench_factorial ... bench: 42 ns/iter (+/- 6)
The Godbolt compiler explorer can also help by showing the actual machine code emitted by the compiler, which may make it obvious when the compiler has performed optimizations that would be unrealistic for code running a real scenario.
Finally, if you are including benchmarks for your Rust code, the Criterion crate
may provide an alternative to the standard test::Bencher
functionality which is:
- more convenient (it runs with stable Rust)
- more fully-featured (it has support for statistics and graphs).
Fuzz Testing
Fuzz testing is the process of exposing code to randomized inputs in the hope of finding bugs, particularly crashes that result from those inputs. Although this can be a useful technique in general, it becomes much more important when your code is exposed to inputs that may be controlled by someone who is deliberately trying to attack the code – so you should run fuzz tests if your code is exposed to potential attackers.
Historically, the majority of defects in C/C++ code that have been exposed by fuzzers have been memory safety problems, typically found by combining fuzz testing with runtime instrumentation (e.g. AddressSanitizer or ThreadSanitizer) of memory access patterns.
Rust is immune to some (but not all) of these memory safety problems, particularly when there is no unsafe
code
involved (Item 16). However, Rust does not prevent bugs in general, and a code path that triggers a panic!
(cf. Item 18) can still result in a denial-of-service (DoS) attack on the codebase as a whole.
The most effective forms of fuzz testing are coverage-guided: the test infrastructure monitors which parts of the code
are executed, and favours random mutations of the inputs that explore new code paths. "American fuzzy lop"
(AFL) was the original heavyweight champion of this technique, but in more recent
years equivalent functionality has been included into the LLVM toolchain as
libFuzzer
.
The Rust compiler is built on LLVM, and so the cargo-fuzz
sub-command exposes
libFuzzer
functionality for Rust (albeit only for a limited number of platforms).
To set up a fuzz test, first identify an entrypoint of your code that takes (or can be adapted to take) arbitrary bytes of data as input:
/// Determine if the input starts with "FUZZ".
fn is_fuzz(data: &[u8]) -> bool {
if data.len() >= 3 /* oops */
&& data[0] == b'F'
&& data[1] == b'U'
&& data[2] == b'Z'
&& data[3] == b'Z'
{
true
} else {
false
}
}
Next, write a small driver that connects this entrypoint to the fuzzing infrastructure:
fuzz_target!(|data: &[u8]| {
let _ = is_fuzz(data);
});
Running cargo +nightly fuzz run target1
continuously executes the fuzz target with random data, only stopping if a
crash is found. In this case, a failure is found almost immediately:
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 74351546
INFO: Loaded 1 modules (2231 inline 8-bit counters): 2231 [0x10eaa2898, 0x10eaa314f),
INFO: Loaded 1 PC tables (2231 PCs): 2231 [0x10eaa3150,0x10eaabcc0),
INFO: 7 files found in /Users/dmd/src/effective-rust/examples/testing/fuzz/corpus/target1
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 7 min: 1b max: 8b total: 34b rss: 30Mb
#8 INITED cov: 22 ft: 22 corp: 5/18b exec/s: 0 rss: 30Mb
thread '<unnamed>' panicked at 'index out of bounds: the len is 3 but the index is 3', fuzz_targets/target1.rs:11:12
stack backtrace:
0: rust_begin_unwind
at /rustc/77652b9ef3fc98e2df0e260efedb80aa68c08c06/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/77652b9ef3fc98e2df0e260efedb80aa68c08c06/library/core/src/panicking.rs:142:14
2: core::panicking::panic_bounds_check
at /rustc/77652b9ef3fc98e2df0e260efedb80aa68c08c06/library/core/src/panicking.rs:84:5
3: _rust_fuzzer_test_input
4: ___rust_try
5: _LLVMFuzzerTestOneInput
6: __ZN6fuzzer6Fuzzer15ExecuteCallbackEPKhm
7: __ZN6fuzzer6Fuzzer6RunOneEPKhmbPNS_9InputInfoEbPb
8: __ZN6fuzzer6Fuzzer16MutateAndTestOneEv
9: __ZN6fuzzer6Fuzzer4LoopERNSt3__16vectorINS_9SizedFileENS_16fuzzer_allocatorIS3_EEEE
10: __ZN6fuzzer12FuzzerDriverEPiPPPcPFiPKhmE
11: _main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
and the input that triggered the failure is emitted.
Normally, fuzz testing does not find failures so quickly, and so it does not make sense to run fuzz tests as part of your continuous integration. The open-ended nature of the testing, and the consequent compute costs, mean that you need to consider how and when to run fuzz tests – perhaps only for new releases or major changes, or perhaps for a limited period of time2.
You can also make subsequent runs of the fuzzing infrastructure more efficient, by storing and re-using a corpus of previous inputs which the fuzzer found to explore new code paths; this helps subsequent runs of the fuzzer explore new ground, rather than re-testing code paths previously visited.
Testing Advice
An Item about testing wouldn't be complete without repeating some common advice (which is mostly not Rust-specific):
- As this Item has endlessly repeated, run all your tests in continuous integration on every change (with the exception of fuzz tests).
- When you're fixing a bug, write a test that exhibits the bug before fixing the bug. That way you can be sure that the bug is fixed, and that it won't be accidentally re-introduced in future.
- If your crate has features (Item 26), run tests over every possible combination of available features.
- More generally, if your crate includes any config-specific code (e.g.
#[cfg(target_os = "windows")]
), run tests for every platform that has distinct code.
Summary
This Item has covered a lot of different types of test, so a summary is in order:
- Write unit tests for comprehensive testing that includes testing of internal-only code; run with
cargo test
. - Write integration tests to exercise your public API; run with
cargo test
. - Write doc tests that exemplify how to use individual items in your public API; run with
cargo test
. - Write example programs that show how to use your public API as a whole; run with
cargo test --examples
orcargo run --example <name>
. - Write benchmarks if your code has significant performance requirements; run with
cargo bench
. - Write fuzz tests if your code is exposed to untrusted inputs; run (continuously) with
cargo fuzz
.
That's a lot of different types of test, so it's up to you much each of them is relevant and worthwhile for your project.
If you have a lot of test code and you are publishing your crate to crates.io,
then you might need to consider which of the tests make sense to include in the published crate. By default, cargo
will include unit tests, integration tests, benchmarks and examples (but not fuzz tests), which may be more than end
users need. If that's the case, you can either
exclude
some of the files,
or (for black-box tests) move the tests out of the crate and into a separate test crate.
1: Support for benchmarks
is not stable, so the command may need to be cargo +nightly bench
.
2: If your code is a widely-used open-source crate, the Google OSS-Fuzz program may be willing to run fuzzing on your behalf.
Item 31: Take advantage of the tooling ecosystem
Item 32: Set up a continuous integration (CI) system
Asynchronous Rust
Item 33: Distinguish between the async
and non-async
worlds
Item 34: Familiarize yourself with executors
Item 35: Use the type system to decipher problems
Item 36: Understand how Future
s are combined
Beyond Standard Rust
Item 37: Consider making library code no_std
compatible
Rust comes with a standard library called std
, which includes code for a wide variety of common tasks, from standard
data structures to networking, from multi-threading support to file I/O. For convenience, many of the items from std
are automatically imported into your program, via the prelude: a set
of common use
statements.
Rust also supports building code for environments where it's not possible to provide this full standard library, such as
bootloaders, firmware, or embedded platforms in general. Crates indicate that they should be built in this way by
including the #![no_std]
crate-level attribute at the top of src/lib.rs
.
This Item explores what's lost when building for no_std
, and what library functions you can still rely on –
which turns out to be quite a lot.
However, this Item is specifically about no_std
support in library code. The difficulties of making a no_std
binary are beyond this text, so the focus here is how to make sure that library code is available for those poor souls
who work in such a minimal environment.
core
Even when building for the most restricted of platforms, many of the fundamental types from the standard library are
still available. For example, Option
and
Result
are still available, albeit under a different name,
as are various flavours of Iterator
.
The different names for these fundamental types starts with core::
, indicating that they come from the core
library,
a standard library that's available even in the most no_std
of environments. These core::
types behave exactly the
same as the equivalent std::
types, because they're actually the same types – in each case, the std::
version
is just a re-export of the underlying core::
type.
This means that there's an easy way to tell if a std::
item is available in no_std
environment: visit the
doc.rust-lang.org
page for the std
item you're interested in, and follow the "source"
link (at the top-right). If that takes you to a src/core/...
location, then the item is available under no_std
via
core::
.
The types from core
are available for all Rust programs automatically; however, they typically need to be
explicitly use
d in a no_std
environment, because the std
prelude is absent.
In practice, relying purely on core
is too limiting for many environments, even no_std
ones. This is because a
core1 constraint of core
is that it performs no heap allocation.
Although Rust excels at putting items on the stack, and safely tracking the corresponding lifetimes (Item 15), this restriction still means that that standard data structures – vectors, maps, sets – can't be provided, because they need to allocate heap space for their contents. In turn, this also drastically reduces the number of available crates that work in this environment.
alloc
However, if a no_std
environment does support heap allocation, then many of the standard data structures from std
can
still be supported. These data structures, along with other allocation-using functionality, is grouped into Rust's
alloc
library.
As with core
, these alloc
variants are actually the same types under the covers; for example, following the source
link from the documentation for
std::vec::Vec
leads to a src/alloc/...
location.
A no_std
Rust crate needs to explicitly opt-in to the use of alloc
, by adding an extern crate alloc;
declaration2 to src/lib.rs
:
//! My `no_std` compatible crate.
#![no_std]
// Requires `alloc`.
extern crate alloc;
Functionality that's enabled by turning on alloc
includes many familiar friends, now addressed by their true names:
alloc::boxed::Box<T>
alloc::rc::Rc<T>
alloc::sync::Arc<T>
alloc::vec::Vec<T>
alloc::string::String
format!
alloc::collections::BTreeMap<K, V>
alloc::collections::BTreeSet<T>
With these things available, it becomes possible for many library crates to be no_std
compatible – e.g. for libraries
that don't involve I/O or networking.
There's a notable absence from the data structures that alloc
makes available, though – the
collections
HashMap
and
HashSet
are specific to std
, not alloc
.
That's because these hash-based containers rely on random seeds to protect against hash collision attacks, but safe random
number generation requires assistance from the operating system – which alloc
can't assume exists.
Another notable absence is synchronization functionality like
std::sync::Mutex
, which is required for multi-threaded code
(Item 17). These types are specific to std
because they rely on OS-specific synchronization primitives, which aren't
available without an OS. If you need to write code that is both no_std
and multi-threaded, third-party crates such as
spin
are probably your only option.
Writing code for no_std
The previous sections made it clear that for some library crates, making the code no_std
compatible just involves:
- Replacing
std::
types with identicalcore::
oralloc::
crates (which requiresuse
of the full type name, due to the absence of thestd
prelude). - Shifting from
HashMap
/HashSet
toBTreeMap
/BTreeSet
.
However, this only makes sense if all of the crates that you depend on (Item 25) are also no_std
compatible –
there's no point in becoming no_std
compatible if any user of your crate is forced to link in std
anyway.
There's also a catch here: the Rust compiler will not tell you if your no_std
crate depends on a std
-using
dependency. This means that it's easy for the work of making a crate no_std
-compatible to be undone – all it
takes is an added or updated dependency that pulls in std
.
To protect against this, add a CI check for a no_std
build, so that your CI system (Item 32) will warn you if
this happens. The Rust toolchain supports cross-compilation out of the box, so this can be as simple as performing a
cross-compile for a
target system (e.g. --target thumbv6m-none-eabi
) that does not support std
– any code that inadvertently
requires std
will then fail to compile for this target.
So: if your dependencies support it, and the simple transformations above are all that's needed, then consider making
library code no_std
compatible. When it is possible, it's not much additional work and it allows for the widest
re-use of the library.
If those transformations don't cover all of the code in your crate, but the parts that aren't covered are only a small or well-contained fraction of the code, then consider adding feature (Item 26) to your crate that turns on just those parts.
Such a feature is conventionally named either std
, if it enables use of std
-specific functionality:
#![cfg_attr(not(feature = "std"), no_std)]
or alloc
, if it turns on use of alloc
-derived function:
#[cfg(feature = "alloc")]
extern crate alloc;
As ever with feature-gated code (Item 26), make sure that your CI system builds all the relevant combinations –
including a build with the std
feature disabled on an explicitly no_std
platform.
Fallible Allocation
The earlier sections of this Item considered two different no_std
environments: a fully embedded environment with no
heap allocation whatsoever (core
), or an more generous environment where heap allocation is allowed (core
+
alloc
). However, there are some important environments that fall between these two camps.
In particular, Rust's standard alloc
library includes a pervasive assumption that heap allocations cannot fail, and
that's not always a valid assumption.
Even a simple use of alloc::vec::Vec
could potentially allocate on every line:
let mut v = Vec::new();
v.push(1); // might allocate
v.push(2); // might allocate
v.push(3); // might allocate
v.push(4); // might allocate
None of these operations returns a Result
, so what happens if those allocations fail?
The answer to this question depends on the toolchain, target and
configuration, but is likely to involve
panic!
and program termination. There is certainly no answer that allows an allocation failure on line 3 to be
handled in a way that allows the program to move on to line 4.
This assumption of infallible allocation gives good ergonomics for code that runs in a "normal" userspace, where there's effectively infinite memory (or at least where running out of memory indicates that the computer as a whole is likely to have bigger problems elsewhere).
However, infallible allocation is utterly unsuitable for code that needs to run in environments where memory is limited and programs are required to cope. This is a (rare) area where there's better support in older, less memory-safe, languages:
- C is sufficiently low-level that allocations are manual and so the return value from
malloc
can be checked forNULL
. - C++ can use its exception mechanism3 to catch allocation failures in the form
of
std::bad_alloc
exceptions.
At the time of writing, Rust's inability to cope with failed allocation has been flagged in some high-profile contexts (such as the Linux kernel, Android, and the Curl tool), and so work is on-going to fix the omission.
The first step is the "fallible collection allocation" changes, which
added fallible alternatives to many of the collection APIs that involve allocation. This generally adds a
try_<operation>
variant that results a Result<_, AllocError>
(although the try_...
variant is currently only
available with the nightly toolchain). For example:
Box::try_new
is available as an alternative toBox::new
Vec::try_reserve
is available as an alternative toVec::reserve
BTreeMap::try_insert
is available as an alternative toBTreeMap::insert
These fallible APIs only go so far; for example, there is (as yet) no fallible equivalent to
Vec::push
, so code that assembles a vector may need
to do careful calculations to ensure that allocation errors can't happen:
let mut v = Vec::new();
// Perform a careful calculation to figure out how much space is needed,
// here simplified to...
let required_size = 4;
v.try_reserve(required_size).map_err(|_e| {
MyError::new(format!("Failed to allocate {} items!", required_size))
})?;
// We now know that it's safe to do:
v.push(1);
v.push(2);
v.push(3);
v.push(4);
Fallible allocation is an area where work on Rust is on-going. The entrypoints described above will hopefully be stabilized and expanded, and there has also been a proposal to make infallible allocation operations controlled by a default-on feature – by explicitly disabling the feature, a programmer can then be sure that no use of infallible allocation has inadvertently crept into their program.
1: Pun intended.
2: Prior to Rust 2018, extern crate
declarations were used to pull in dependencies. This is now
entirely handled by Cargo.toml
, but the extern crate
mechanism is still used to pull in those parts of the Rust
standard library that are optional in no_std
environments.
3: It's also possible to add the
std::nothrow
overload to calls to new
and check for
nullptr
return values; however, there are still container methods like
vector<T>::push_back
that allocate under the covers,
and which can therefore only signal allocation failure via an exception.