#
std/io/error/
repr_bitpacked.rs

    1//! This is a densely packed error representation which is used on targets with
    2//! 64-bit pointers.
    3//!
    4//! (Note that `bitpacked` vs `unpacked` here has no relationship to
    5//! `#[repr(packed)]`, it just refers to attempting to use any available bits in
    6//! a more clever manner than `rustc`'s default layout algorithm would).
    7//!
    8//! Conceptually, it stores the same data as the "unpacked" equivalent we use on
    9//! other targets. Specifically, you can imagine it as an optimized version of
    10//! the following enum (which is roughly equivalent to what's stored by
    11//! `repr_unpacked::Repr`, e.g. `super::ErrorData<Box<Custom>>`):
    12//!
    13//! ```ignore (exposition-only)
    14//! enum ErrorData {
    15//!    Os(i32),
    16//!    Simple(ErrorKind),
    17//!    SimpleMessage(&'static SimpleMessage),
    18//!    Custom(Box<Custom>),
    19//! }
    20//! ```
    21//!
    22//! However, it packs this data into a 64bit non-zero value.
    23//!
    24//! This optimization not only allows `io::Error` to occupy a single pointer,
    25//! but improves `io::Result` as well, especially for situations like
    26//! `io::Result<()>` (which is now 64 bits) or `io::Result<u64>` (which is now
    27//! 128 bits), which are quite common.
    28//!
    29//! # Layout
    30//! Tagged values are 64 bits, with the 2 least significant bits used for the
    31//! tag. This means there are 4 "variants":
    32//!
    33//! - **Tag 0b00**: The first variant is equivalent to
    34//!   `ErrorData::SimpleMessage`, and holds a `&'static SimpleMessage` directly.
    35//!
    36//!   `SimpleMessage` has an alignment >= 4 (which is requested with
    37//!   `#[repr(align)]` and checked statically at the bottom of this file), which
    38//!   means every `&'static SimpleMessage` should have the both tag bits as 0,
    39//!   meaning its tagged and untagged representation are equivalent.
    40//!
    41//!   This means we can skip tagging it, which is necessary as this variant can
    42//!   be constructed from a `const fn`, which probably cannot tag pointers (or
    43//!   at least it would be difficult).
    44//!
    45//! - **Tag 0b01**: The other pointer variant holds the data for
    46//!   `ErrorData::Custom` and the remaining 62 bits are used to store a
    47//!   `Box<Custom>`. `Custom` also has alignment >= 4, so the bottom two bits
    48//!   are free to use for the tag.
    49//!
    50//!   The only important thing to note is that `ptr::wrapping_add` and
    51//!   `ptr::wrapping_sub` are used to tag the pointer, rather than bitwise
    52//!   operations. This should preserve the pointer's provenance, which would
    53//!   otherwise be lost.
    54//!
    55//! - **Tag 0b10**: Holds the data for `ErrorData::Os(i32)`. We store the `i32`
    56//!   in the pointer's most significant 32 bits, and don't use the bits `2..32`
    57//!   for anything. Using the top 32 bits is just to let us easily recover the
    58//!   `i32` code with the correct sign.
    59//!
    60//! - **Tag 0b11**: Holds the data for `ErrorData::Simple(ErrorKind)`. This
    61//!   stores the `ErrorKind` in the top 32 bits as well, although it doesn't
    62//!   occupy nearly that many. Most of the bits are unused here, but it's not
    63//!   like we need them for anything else yet.
    64//!
    65//! # Use of `NonNull<()>`
    66//!
    67//! Everything is stored in a `NonNull<()>`, which is odd, but actually serves a
    68//! purpose.
    69//!
    70//! Conceptually you might think of this more like:
    71//!
    72//! ```ignore (exposition-only)
    73//! union Repr {
    74//!     // holds integer (Simple/Os) variants, and
    75//!     // provides access to the tag bits.
    76//!     bits: NonZero<u64>,
    77//!     // Tag is 0, so this is stored untagged.
    78//!     msg: &'static SimpleMessage,
    79//!     // Tagged (offset) `Box<Custom>` pointer.
    80//!     tagged_custom: NonNull<()>,
    81//! }
    82//! ```
    83//!
    84//! But there are a few problems with this:
    85//!
    86//! 1. Union access is equivalent to a transmute, so this representation would
    87//!    require we transmute between integers and pointers in at least one
    88//!    direction, which may be UB (and even if not, it is likely harder for a
    89//!    compiler to reason about than explicit ptr->int operations).
    90//!
    91//! 2. Even if all fields of a union have a niche, the union itself doesn't,
    92//!    although this may change in the future. This would make things like
    93//!    `io::Result<()>` and `io::Result<usize>` larger, which defeats part of
    94//!    the motivation of this bitpacking.
    95//!
    96//! Storing everything in a `NonZero<usize>` (or some other integer) would be a
    97//! bit more traditional for pointer tagging, but it would lose provenance
    98//! information, couldn't be constructed from a `const fn`, and would probably
    99//! run into other issues as well.
    100//!
    101//! The `NonNull<()>` seems like the only alternative, even if it's fairly odd
    102//! to use a pointer type to store something that may hold an integer, some of
    103//! the time.
    104
    105use core::marker::PhantomData;
    106use core::num::NonZeroUsize;
    107use core::ptr::NonNull;
    108
    109use super::{Custom, ErrorData, ErrorKind, RawOsError, SimpleMessage};
    110
    111// The 2 least-significant bits are used as tag.
    112const TAG_MASK: usize = 0b11;
    113const TAG_SIMPLE_MESSAGE: usize = 0b00;
    114const TAG_CUSTOM: usize = 0b01;
    115const TAG_OS: usize = 0b10;
    116const TAG_SIMPLE: usize = 0b11;
    117
    118/// The internal representation.
    119///
    120/// See the module docs for more, this is just a way to hack in a check that we
    121/// indeed are not unwind-safe.
    122///
    123/// ```compile_fail,E0277
    124/// fn is_unwind_safe<T: core::panic::UnwindSafe>() {}
    125/// is_unwind_safe::<std::io::Error>();
    126/// ```
    127#[repr(transparent)]
    128#[rustc_insignificant_dtor]
    129pub(super) struct Repr(NonNull<()>, PhantomData<ErrorData<Box<Custom>>>);
    130
    131// All the types `Repr` stores internally are Send + Sync, and so is it.
    132unsafe impl Send for Repr {}
    133unsafe impl Sync for Repr {}
    134
    135impl Repr {
    136    pub(super) fn new(dat: ErrorData<Box<Custom>>) -> Self {
    137        match dat {
    138            ErrorData::Os(code) => Self::new_os(code),
    139            ErrorData::Simple(kind) => Self::new_simple(kind),
    140            ErrorData::SimpleMessage(simple_message) => Self::new_simple_message(simple_message),
    141            ErrorData::Custom(b) => Self::new_custom(b),
    142        }
    143    }
    144
    145    pub(super) fn new_custom(b: Box<Custom>) -> Self {
    146        let p = Box::into_raw(b).cast::<u8>();
    147        // Should only be possible if an allocator handed out a pointer with
    148        // wrong alignment.
    149        debug_assert_eq!(p.addr() & TAG_MASK, 0);
    150        // Note: We know `TAG_CUSTOM <= size_of::<Custom>()` (static_assert at
    151        // end of file), and both the start and end of the expression must be
    152        // valid without address space wraparound due to `Box`'s semantics.
    153        //
    154        // This means it would be correct to implement this using `ptr::add`
    155        // (rather than `ptr::wrapping_add`), but it's unclear this would give
    156        // any benefit, so we just use `wrapping_add` instead.
    157        let tagged = p.wrapping_add(TAG_CUSTOM).cast::<()>();
    158        // Safety: `TAG_CUSTOM + p` is the same as `TAG_CUSTOM | p`,
    159        // because `p`'s alignment means it isn't allowed to have any of the
    160        // `TAG_BITS` set (you can verify that addition and bitwise-or are the
    161        // same when the operands have no bits in common using a truth table).
    162        //
    163        // Then, `TAG_CUSTOM | p` is not zero, as that would require
    164        // `TAG_CUSTOM` and `p` both be zero, and neither is (as `p` came from a
    165        // box, and `TAG_CUSTOM` just... isn't zero -- it's `0b01`). Therefore,
    166        // `TAG_CUSTOM + p` isn't zero and so `tagged` can't be, and the
    167        // `new_unchecked` is safe.
    168        let res = Self(unsafe { NonNull::new_unchecked(tagged) }, PhantomData);
    169        // quickly smoke-check we encoded the right thing (This generally will
    170        // only run in std's tests, unless the user uses -Zbuild-std)
    171        debug_assert!(matches!(res.data(), ErrorData::Custom(_)), "repr(custom) encoding failed");
    172        res
    173    }
    174
    175    #[inline]
    176    pub(super) fn new_os(code: RawOsError) -> Self {
    177        let utagged = ((code as usize) << 32) | TAG_OS;
    178        // Safety: `TAG_OS` is not zero, so the result of the `|` is not 0.
    179        let res = Self(
    180            NonNull::without_provenance(unsafe { NonZeroUsize::new_unchecked(utagged) }),
    181            PhantomData,
    182        );
    183        // quickly smoke-check we encoded the right thing (This generally will
    184        // only run in std's tests, unless the user uses -Zbuild-std)
    185        debug_assert!(
    186            matches!(res.data(), ErrorData::Os(c) if c == code),
    187            "repr(os) encoding failed for {code}"
    188        );
    189        res
    190    }
    191
    192    #[inline]
    193    pub(super) fn new_simple(kind: ErrorKind) -> Self {
    194        let utagged = ((kind as usize) << 32) | TAG_SIMPLE;
    195        // Safety: `TAG_SIMPLE` is not zero, so the result of the `|` is not 0.
    196        let res = Self(
    197            NonNull::without_provenance(unsafe { NonZeroUsize::new_unchecked(utagged) }),
    198            PhantomData,
    199        );
    200        // quickly smoke-check we encoded the right thing (This generally will
    201        // only run in std's tests, unless the user uses -Zbuild-std)
    202        debug_assert!(
    203            matches!(res.data(), ErrorData::Simple(k) if k == kind),
    204            "repr(simple) encoding failed {:?}",
    205            kind,
    206        );
    207        res
    208    }
    209
    210    #[inline]
    211    pub(super) const fn new_simple_message(m: &'static SimpleMessage) -> Self {
    212        // Safety: References are never null.
    213        Self(unsafe { NonNull::new_unchecked(m as *const _ as *mut ()) }, PhantomData)
    214    }
    215
    216    #[inline]
    217    pub(super) fn data(&self) -> ErrorData<&Custom> {
    218        // Safety: We're a Repr, decode_repr is fine.
    219        unsafe { decode_repr(self.0, |c| &*c) }
    220    }
    221
    222    #[inline]
    223    pub(super) fn data_mut(&mut self) -> ErrorData<&mut Custom> {
    224        // Safety: We're a Repr, decode_repr is fine.
    225        unsafe { decode_repr(self.0, |c| &mut *c) }
    226    }
    227
    228    #[inline]
    229    pub(super) fn into_data(self) -> ErrorData<Box<Custom>> {
    230        let this = core::mem::ManuallyDrop::new(self);
    231        // Safety: We're a Repr, decode_repr is fine. The `Box::from_raw` is
    232        // safe because we prevent double-drop using `ManuallyDrop`.
    233        unsafe { decode_repr(this.0, |p| Box::from_raw(p)) }
    234    }
    235}
    236
    237impl Drop for Repr {
    238    #[inline]
    239    fn drop(&mut self) {
    240        // Safety: We're a Repr, decode_repr is fine. The `Box::from_raw` is
    241        // safe because we're being dropped.
    242        unsafe {
    243            let _ = decode_repr(self.0, |p| Box::<Custom>::from_raw(p));
    244        }
    245    }
    246}
    247
    248// Shared helper to decode a `Repr`'s internal pointer into an ErrorData.
    249//
    250// Safety: `ptr`'s bits should be encoded as described in the document at the
    251// top (it should `some_repr.0`)
    252#[inline]
    253unsafe fn decode_repr<C, F>(ptr: NonNull<()>, make_custom: F) -> ErrorData<C>
    254where
    255    F: FnOnce(*mut Custom) -> C,
    256{
    257    let bits = ptr.as_ptr().addr();
    258    match bits & TAG_MASK {
    259        TAG_OS => {
    260            let code = ((bits as i64) >> 32) as RawOsError;
    261            ErrorData::Os(code)
    262        }
    263        TAG_SIMPLE => {
    264            let kind_bits = (bits >> 32) as u32;
    265            let kind = kind_from_prim(kind_bits).unwrap_or_else(|| {
    266                debug_assert!(false, "Invalid io::error::Repr bits: `Repr({:#018x})`", bits);
    267                // This means the `ptr` passed in was not valid, which violates
    268                // the unsafe contract of `decode_repr`.
    269                //
    270                // Using this rather than unwrap meaningfully improves the code
    271                // for callers which only care about one variant (usually
    272                // `Custom`)
    273                unsafe { core::hint::unreachable_unchecked() };
    274            });
    275            ErrorData::Simple(kind)
    276        }
    277        TAG_SIMPLE_MESSAGE => {
    278            // SAFETY: per tag
    279            unsafe { ErrorData::SimpleMessage(&*ptr.cast::<SimpleMessage>().as_ptr()) }
    280        }
    281        TAG_CUSTOM => {
    282            // It would be correct for us to use `ptr::byte_sub` here (see the
    283            // comment above the `wrapping_add` call in `new_custom` for why),
    284            // but it isn't clear that it makes a difference, so we don't.
    285            let custom = ptr.as_ptr().wrapping_byte_sub(TAG_CUSTOM).cast::<Custom>();
    286            ErrorData::Custom(make_custom(custom))
    287        }
    288        _ => {
    289            // Can't happen, and compiler can tell
    290            unreachable!();
    291        }
    292    }
    293}
    294
    295// This compiles to the same code as the check+transmute, but doesn't require
    296// unsafe, or to hard-code max ErrorKind or its size in a way the compiler
    297// couldn't verify.
    298#[inline]
    299fn kind_from_prim(ek: u32) -> Option<ErrorKind> {
    300    macro_rules! from_prim {
    301        ($prim:expr => $Enum:ident { $($Variant:ident),* $(,)? }) => {{
    302            // Force a compile error if the list gets out of date.
    303            const _: fn(e: $Enum) = |e: $Enum| match e {
    304                $($Enum::$Variant => ()),*
    305            };
    306            match $prim {
    307                $(v if v == ($Enum::$Variant as _) => Some($Enum::$Variant),)*
    308                _ => None,
    309            }
    310        }}
    311    }
    312    from_prim!(ek => ErrorKind {
    313        NotFound,
    314        PermissionDenied,
    315        ConnectionRefused,
    316        ConnectionReset,
    317        HostUnreachable,
    318        NetworkUnreachable,
    319        ConnectionAborted,
    320        NotConnected,
    321        AddrInUse,
    322        AddrNotAvailable,
    323        NetworkDown,
    324        BrokenPipe,
    325        AlreadyExists,
    326        WouldBlock,
    327        NotADirectory,
    328        IsADirectory,
    329        DirectoryNotEmpty,
    330        ReadOnlyFilesystem,
    331        FilesystemLoop,
    332        StaleNetworkFileHandle,
    333        InvalidInput,
    334        InvalidData,
    335        TimedOut,
    336        WriteZero,
    337        StorageFull,
    338        NotSeekable,
    339        QuotaExceeded,
    340        FileTooLarge,
    341        ResourceBusy,
    342        ExecutableFileBusy,
    343        Deadlock,
    344        CrossesDevices,
    345        TooManyLinks,
    346        InvalidFilename,
    347        ArgumentListTooLong,
    348        Interrupted,
    349        Other,
    350        UnexpectedEof,
    351        Unsupported,
    352        OutOfMemory,
    353        InProgress,
    354        Uncategorized,
    355    })
    356}
    357
    358// Some static checking to alert us if a change breaks any of the assumptions
    359// that our encoding relies on for correctness and soundness. (Some of these are
    360// a bit overly thorough/cautious, admittedly)
    361//
    362// If any of these are hit on a platform that std supports, we should likely
    363// just use `repr_unpacked.rs` there instead (unless the fix is easy).
    364macro_rules! static_assert {
    365    ($condition:expr) => {
    366        const _: () = assert!($condition);
    367    };
    368    (@usize_eq: $lhs:expr, $rhs:expr) => {
    369        const _: [(); $lhs] = [(); $rhs];
    370    };
    371}
    372
    373// The bitpacking we use requires pointers be exactly 64 bits.
    374static_assert!(@usize_eq: size_of::<NonNull<()>>(), 8);
    375
    376// We also require pointers and usize be the same size.
    377static_assert!(@usize_eq: size_of::<NonNull<()>>(), size_of::<usize>());
    378
    379// `Custom` and `SimpleMessage` need to be thin pointers.
    380static_assert!(@usize_eq: size_of::<&'static SimpleMessage>(), 8);
    381static_assert!(@usize_eq: size_of::<Box<Custom>>(), 8);
    382
    383static_assert!((TAG_MASK + 1).is_power_of_two());
    384// And they must have sufficient alignment.
    385static_assert!(align_of::<SimpleMessage>() >= TAG_MASK + 1);
    386static_assert!(align_of::<Custom>() >= TAG_MASK + 1);
    387
    388static_assert!(@usize_eq: TAG_MASK & TAG_SIMPLE_MESSAGE, TAG_SIMPLE_MESSAGE);
    389static_assert!(@usize_eq: TAG_MASK & TAG_CUSTOM, TAG_CUSTOM);
    390static_assert!(@usize_eq: TAG_MASK & TAG_OS, TAG_OS);
    391static_assert!(@usize_eq: TAG_MASK & TAG_SIMPLE, TAG_SIMPLE);
    392
    393// This is obviously true (`TAG_CUSTOM` is `0b01`), but in `Repr::new_custom` we
    394// offset a pointer by this value, and expect it to both be within the same
    395// object, and to not wrap around the address space. See the comment in that
    396// function for further details.
    397//
    398// Actually, at the moment we use `ptr::wrapping_add`, not `ptr::add`, so this
    399// check isn't needed for that one, although the assertion that we don't
    400// actually wrap around in that wrapping_add does simplify the safety reasoning
    401// elsewhere considerably.
    402static_assert!(size_of::<Custom>() >= TAG_CUSTOM);
    403
    404// These two store a payload which is allowed to be zero, so they must be
    405// non-zero to preserve the `NonNull`'s range invariant.
    406static_assert!(TAG_OS != 0);
    407static_assert!(TAG_SIMPLE != 0);
    408// We can't tag `SimpleMessage`s, the tag must be 0.
    409static_assert!(@usize_eq: TAG_SIMPLE_MESSAGE, 0);
    410
    411// Check that the point of all of this still holds.
    412//
    413// We'd check against `io::Error`, but *technically* it's allowed to vary,
    414// as it's not `#[repr(transparent)]`/`#[repr(C)]`. We could add that, but
    415// the `#[repr()]` would show up in rustdoc, which might be seen as a stable
    416// commitment.
    417static_assert!(@usize_eq: size_of::<Repr>(), 8);
    418static_assert!(@usize_eq: size_of::<Option<Repr>>(), 8);
    419static_assert!(@usize_eq: size_of::<Result<(), Repr>>(), 8);
    420static_assert!(@usize_eq: size_of::<Result<usize, Repr>>(), 16);