Attempt to run parallel frontend CI

[heinwol]

View all comments

WIP

Supersedes #157705.

The issue with the original change was arguments --parallel-frontend-threads=4 --iteration-count=2 being compiletest-only. They were being passed to other parts of the test harness (e.g. libtest) as is. These parts, however, have no clue of the arguments, hence the errors.

I've spent a lot of time on this and haven't found any reasonable way to fix this behavior. We could make bootstrap aware of these specific two arguments and have an additional internal logic for handling this. It's big of a hack, i reckon. The best decision i arrived at is to split testing into two parts: one for compiletest only and another for everything else. The issue is (AFAIK) we can't tell bootstrap (or x.py, at least) to "test the default stuff, but only for compiletest". When used like x test tests/ it runs all the tests in this directory, including non-default ones, and crashes as it can't find nodejs for doctests. --skip compiler/ --skip library/ --skip src/tools/ --skip tests/incremental ... is still not exhaustive list of exclusions.

I went on with a whitelist instead of a blacklist. But we, again, can't tell what tests are "default". There's a mechanism in bootstrap::core::builder::Builder for showing "dry-run" test suites, but it's not available from the cli. For now, my whitelist is quite small and i have no idea what should it be like.

Also, i added ENV RUST_TEST_THREADS=1 just because it was introduced by @ zetanumbers; i'm not sure it's necessary. It ran ok in the previous iteration.

Finally, i'm not sure it works for now, i'm still spending hours blindly guessing the maxumum working set of tests. I've decided to make the pr now just to have something done and seek advice.

r? @petrochenkov

---

try-job: optional-x86_64-gnu-parallel-frontend

[petrochenkov]

Also, i added ENV RUST_TEST_THREADS=1 just because it was introduced by @ zetanumbers; i'm not sure it's necessary. It ran ok in the previous iteration.

It's added to give threads inside the compiler opportunity to use multiple cores simultaneously, which gives better chance for bugs to occur.
Otherwise compiletest will just occupy the cores by running multiple tests (multiple rustc instances) at the same time.

[petrochenkov]

Only tests/ui tests need to be run in this mode.
For non-TestMode::Ui suites --parallel-frontend-threads=4 doesn't do anything anyway, and running ui-fulldeps is just not important enough.

The second run can then --exclude tests/ui.

View changes since the review

[petrochenkov]

The N in --iteration-count=N depends on the available CI resources.
The more the better, of course, but right now even one iteration would be acceptable, I think (see "Enabling parallel UI test suite on CI" in rust-lang/compiler-team#1005).
Note that RUST_TEST_THREADS=1 makes things really slow.
I'll leave this for the infra team to decide.

View changes since the review

[jieyouxu]

I will rerun the job again to check the time. Generally speaking, I would expect that this test job does not become the slowest job which is generally around 3h 15m.

[petrochenkov]

r? @jieyouxu

[heinwol]

It's added to give threads inside the compiler opportunity to use multiple cores simultaneously, which gives better chance for bugs to occur.

Oh, indeed.

Note that RUST_TEST_THREADS=1 makes things really slow.

The more the better, of course, but right now even one iteration would be acceptable, I think

I believe it can all be tweaked with a script though? For instance, we could use something like

$$\texttt{RUST\_TEST\_THREADS} = \texttt{available\_parallelism} / N$$

Which would improve performance drastically. This introduces machine-dependency, but maybe it's not big of an issue here

[heinwol]

@petrochenkov could you grant me bors capabilities, please?

[petrochenkov]

@bors delegate=try

[rust-bors]

✌️ @heinwol, you can now perform try builds on this pull request!

Warning

You used the legacy format of the delegate command.
The new format is @bors delegate[=<username>] [<try|review>].

You can now post @bors try to start a try build.

[heinwol]

@bors try jobs=optional-x86_64-gnu-parallel-frontend

[rust-bors]

☀️ Try build successful (CI)
Build commit: 92ec5ac (92ec5ac51762b85c9e9080ac9593442ab95cc150)
Base parent: f28ac76 (f28ac764c36004fa6a6e098d15b4016a838c13c6)

[jieyouxu]

Oh actually test failure could be genuine:

---- [ui] tests/rustdoc-ui/doctest/no-capture.rs stdout ----
Saved the actual stdout to `/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/doctest/no-capture/no-capture.stdout`
diff of stdout:
1	
2	running 1 test
-	hello!
-	test $DIR/no-capture.rs - Foo (line 6) ... ok
+	test $DIR/no-capture.rs - Foo (line 6) ... hello!
+	ok
5	
6	test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in $TIME
7	
Note: some mismatched output was normalized before being compared
-	test /checkout/tests/rustdoc-ui/doctest/no-capture.rs - Foo (line 6) ... hello!
+	test $DIR/no-capture.rs - Foo (line 6) ... hello!
+	ok
The actual stdout differed from the expected stdout
To update references, rerun the tests and pass the `--bless` flag
To only update this specific test, also pass `--test-args doctest/no-capture.rs`
error: 1 errors occurred comparing output.
status: exit status: 0

If we can fix this I can re-evaluate the CI time.

@rustbot author

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.

[jieyouxu]

Suggestion: like what the MCP (rust-lang/compiler-team#1005), can we start with --parallel-frontend-threads=4 and --iteration-count=1 to get this PR landed sooner? We can tune the specific M and N in a follow-up PR later based on more measurements.

View changes since the review

[heinwol]

It's really strange: I can reliably reproduce this test failure each time, even in completely non-parallel scenario:

...
ENV RUST_CONFIGURE_ARGS \
 --build=x86_64-unknown-linux-gnu \
 --enable-sanitizers \
 --enable-profiler \
 --enable-compiler-docs \
 --set llvm.libzstd=true
#  --set rust.parallel-frontend-threads=4

ENV RUST_TEST_THREADS=1
ENV SCRIPT python3 ../x.py --stage 2 test \
    tests/rustdoc-ui/doctest/no-capture.rs

I couldn't find any related issues in test suite. Maybe we could just ignore this test for now? Otherwise we could add some additional normalization rule (not sure if it doesn't diminish the purpose of the test though)

@petrochenkov

[petrochenkov]

For me tests/rustdoc-ui/doctest/no-capture.rs always fails when RUST_TEST_THREADS=1 is set, and always passes when it is not set.

Only tests/ui tests need to be run with RUST_TEST_THREADS=1, enabling it for other tests only slows down the run dramatically, without bringing any benefits.

[heinwol]

@bors try jobs=optional-x86_64-gnu-parallel-frontend

[rust-bors]

☀️ Try build successful (CI)
Build commit: d0ef6a5 (d0ef6a5adbb2a8855ba56c721f508e751f9a34fa)
Base parent: e2b71ad (e2b71ade2db4ea263ab0d561d889f3e3795a500d)

[jieyouxu]

That try job took 4h which is a bit concerning
@bors try

[rust-bors]

☀️ Try build successful (CI)
Build commit: 6f55b10 (6f55b1095f8273608d3f0300b995c1f2b5127a49)
Base parent: ea088e0 (ea088e0a3dd50c494b4589c3c121d7469dceec8f)

[jieyouxu]

@bors try

[rust-bors]

☀️ Try build successful (CI)
Build commit: 03525a9 (03525a9483204f984ec96b7c1a7ce38b8e7f6338)
Base parent: ea088e0 (ea088e0a3dd50c494b4589c3c121d7469dceec8f)

[jieyouxu]

• https://github.com/rust-lang/rust/actions/runs/28731953398 -> 3h 48m
• https://github.com/rust-lang/rust/actions/runs/28525065461 -> 4h 5m
• https://github.com/rust-lang/rust/actions/runs/28737553032 -> 3h 31m

🤔

[petrochenkov]

Things we can do to speedup the job:

• --iteration-count=1 instead of --iteration-count=2, as discussed above
• Replace x.py --stage 2 test --skip tests/ui with x.py --stage 2 build or with x.py --stage 2 test --test-args nonexistent-test-name. It is not very important to check that the stage 2 compiler (that is built using parallel frontend) executes all the tests correctly in single-threaded mode. In theory the produced compiler may be corrupted in some way, but that's highly unlikely (especially if all the parallel run-pass ui tests pass).
• RUST_TEST_THREADS=2