Fix off-by-one-error in stage numbering between rustc-guide and rustbuild

[dwijnand]

The rustc guide states that:

• Stage 0: the stage0 compiler is usually the current beta compiler
  (x.py will download it for you); you can configure x.py to use something
  else, though.
• Stage 1: the code in your clone (for new version) is then
  compiled with the stage0 compiler to produce the stage1 compiler.

However a run of rustbuild like so ./x.py build --stage 0 outputs:

Building stage0 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
Building stage0 test artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
Building stage0 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
[etc]

Personally I think the guide makes more sense. However fixing rustbuild would mean that --stage 1 would become --stage 2 (and so forth).

[dwijnand]

(I think this misunderstanding has been at the root of all my rustc feedback loop problems, so I'd be happy to resolve it or attempt to.)

[Mark-Simulacrum]

FWIW, the stages printed in rustc output aren't wrong -- the stage0 std artifacts there are going to be used for anything built by the stage0 compiler: for example, rustc, test artifacts. This pattern continues in latter stages.

Specifically, "Stage 0: the stage0 compiler is usually the current beta compiler" is true, but that's (mostly, modulo build scripts in stage0 std compilation) only true for the rustc binary itself (and associated dynamic libraries). The compiler we download is never used for anything beyond compiling rustbuild, std, test, and rustc, but when we're compiling test and rustc, that compiler is using the freshly produced std.

I'm not sure if any of that made sense -- I myself still struggle with this pretty much every time I confront any sort of staging issue or work with rustbuild. If you have suggestions, I'd love to hear them; I think the problem is that there are two concepts at play here: a compiler (with its set of dependencies) and it's "target" libraries (std, test, and rustc-ish). Both are staged, but in sort of a staggered manner. That makes talking about any of this quite hard.

[Centril]

I also found the situation quite confusing.

@Mark-Simulacrum Perhaps it would be easier to understand and give words to things if we had a sort of sequence/activity diagram or flowchart for stages and dependencies that exist. A picture says...

[ehuss]

I have found it to be confusing, too. I had some notes lying around about how it works. Perhaps one way to think of it is:

stage 0 uses stage 0 compiler to create stage 0 artifacts which will later be uplifted to stage1

That's a bit convoluted. I would expect it to be more along the lines of "stage 0 builds stage 1", but alas it is not.

It's also confusing because building HOST std and TARGET std are different based on the stage (notice below how stage2 only builds non-host std targets — I don't know why). And --keep-stage still seems a bit confusing to me.

Stage 0 Action	Output
beta extracted	build/HOST/stage0
stage0(beta) builds bootstrap	build/bootstrap
stage0(beta) builds libstd	build/HOST/stage0-std/TARGET
copy stage0-std (HOST only)	build/HOST/stage0-sysroot/lib/rustlib/HOST
stage0(beta) (sysroot stage0-sysroot) builds libtest	build/HOST/stage0-test/TARGET
copy stage0-test (HOST only)	build/HOST/stage0-sysroot/lib/rustlib/HOST
stage0(beta) (sysroot stage0-sysroot) builds rustc	build/HOST/stage0-rustc/HOST
copy stage0-rustc (except executable)	build/HOST/stage0-sysroot/lib/rustlib/HOST
build llvm	build/HOST/llvm
stage0(beta) (sysroot stage0-sysroot) builds codegen	build/HOST/stage0-codgen/HOST
stage0(beta) (sysroot stage0-sysroot) builds rustdoc	build/HOST/stage0-tools/HOST

--stage=0 stops here

Stage 1 Action	Output
copy (uplift) stage0-rustc executable	build/HOST/stage1/bin
copy (uplift) stage0-sysroot	build/HOST/stage1/lib
stage1 (sysroot stage1) builds libstd	build/HOST/stage1-std/TARGET
copy stage1-std (HOST only)	build/HOST/stage1/lib/rustlib/HOST
stage1 (sysroot stage1) builds libtest	build/HOST/stage1-test/TARGET
copy stage1-test (HOST only)	build/HOST/stage1/lib/rustlib/HOST
stage1 (sysroot stage1) builds rustc	build/HOST/stage1-rustc/HOST
copy stage1-rustc (except executable)	build/HOST/stage1/lib/rustlib/HOST
stage1 (sysroot stage1) builds codegen	build/HOST/stage1-codegen/HOST

--stage=1 stops here

Stage 2 Action	Output
copy (uplift) stage1-rustc executable	build/HOST/stage2/bin
copy (uplift) stage1-sysroot	build/HOST/stage2/lib and build/HOST/stage2/lib/rustlib/HOST
stage2 (sysroot stage2) builds libstd (except HOST?)	build/HOST/stage2-std/TARGET
copy stage2-std (not HOST targets)	build/HOST/stage2/lib/rustlib/TARGET
stage2 (sysroot stage2) builds libtest (except HOST?)	build/HOST/stage2-test/TARGET
copy stage2-test (not HOST targets)	build/HOST/stage2/lib/rustlib/TARGET
stage2 (sysroot stage2) builds rustdoc	build/HOST/stage2-tools/HOST
copy rustdoc	build/HOST/stage2/bin

--stage=2 stops here

Notes:

• Build scripts always use stage0(beta) and stage0(beta) sysroot (not stage0-sysroot).
• This does not include optional things like lld, stuff required for tests, etc.

Maybe adding a little extra detail to rustc-guide would help? Does any of that help, or is it more confusing now?

[dwijnand]

Thank you all for your comments.

I understand now that stage0 is a full-fledged stage and no one feels like that's wrong, which is a good thing because it means we just need to fix the guide not change the semantics of rustc guide.

I think there are several misleading things in that section of the rustc guide:

• "the stage0 compiler is usually the current beta compiler" is ambiguous about whether "the stage0 compiler" means "the compiler built at stage0" or "the compiler used to build stage0 [things]"
• "Stage 1: the code in your clone (for new version) is then compiled" implies (to me) that the code in my clone isn't compiled in stage 0

I'll come back to this use your info and insights to propose some changes to the rustc-guide repo.

[dwijnand]

Here's another part that has me doubting everything I think I know about how rustc is built, the guide says:

In particular, the newer version of the compiler, libstd, and other tooling may use some unstable features internally.

and in a few more places it makes reference to libstd being the compiler. Is that true?? I thought libstd was the standard library. Isn't src/rustc the new compiler? Or maybe for consumption by rustup or for rustc's test suite it's something a step down, like src/librustc_driver or so.

[dwijnand]

Also, in https://rust-lang.github.io/rustc-guide/how-to-build-and-run.html#workflow about the commend ./x.py build -i --stage 1 src/libstd --keep-stage 1 it says:

The effect of --keep-stage 1 is that we just assume that the old standard library can be re-used.

Which "old standard library" does it mean there, stage 0 or stage 1? Also, it's confusing to target stage 1 and keep stage 1. Assuming it's not equivalent to ./x.py build -i --stage 0 src/rustc, I think the guide should detail why it's not.

[jonas-schievink]

@dwijnand the first bit is just a bit confusingly worded. rustc is the compiler and libstd is the standard library (which includes libcore, liballoc, etc.). It just means "rustc, and libstd, and other tooling may use unstable features internally".

[dwijnand]

@jonas-schievink I'm not sure. You're right, that could be the interpretation, but in https://rust-lang.github.io/rustc-guide/how-to-build-and-run.html#build-flags it says from running ./x.py build -i --stage 1 src/libstd:

This final product (stage1 compiler + libs built using that compiler) is what you need to build other rust programs.

And again in https://rust-lang.github.io/rustc-guide/how-to-build-and-run.html#workflow:

• Initial build: ./x.py build -i --stage 1 src/libstd
  • As documented above, this will build a functional stage1 compiler