Niche-filling layouts are picked over tagged ones even when detrimental.

[eddyb]

This prints the values in comments:

use std::num::NonZeroU8;

pub enum Foo { A(NonZeroU8, u32), B }
pub enum Bar { A(u8, u32), B }

fn main() {
    use std::mem::size_of;
    
    dbg!(size_of::<Foo>()); // = 8
    dbg!(size_of::<Option<Foo>>()); // = 12
    dbg!(size_of::<Option<Option<Foo>>>()); // = 12
    
    dbg!(size_of::<Bar>()); // = 8
    dbg!(size_of::<Option<Bar>>()); // = 8
    dbg!(size_of::<Option<Option<Bar>>>()); // = 8
}

Note that Foo contains strictly more opportunities for optimization, but it's optimized worse.

That's because it uses a "niche-filling" layout (using 0 in the NonZeroU8 field of A for B), but the "tag" layout wouldn't be any larger, because it has enough space in 8 bytes to fit the (byte-sized) tag, NonZeroU8 and u32.

Bar uses the tagged layout since it has no other choice, and ends up with 254 free values in the byte-sized tag, but Foo "jumps the gun" on using a pre-existing niche.