Custom test frameworks

[alexcrichton]

The compiler currently supports the --test compiler flag which will create a binary linked to the distributed test crate. The crate, however, is quite simplistic. There are a good number of possible extensions to the testing infrastructure which we'd love to have but the internal test crate may not be the best place for them.

It would be great if the compiler had an interface such that a custom test framework was supported. This could perhaps be a plugin, or it could perhaps be some other custom method. The resulting strategy should likely also have some Cargo integration to make using custom test frameworks as seamless as possible.

Some related topics (feel free to add more!)

• XML Test Report cargo#1273
• https://github.com/reem/stainless

[alexcrichton]

Another point to consider for this is a benchmarking infrastructure. Right now our benchmarking requires writing extern crate test but we don't necessarily want to stabilize this crate just yet. This would give us a nice valve to distribute the benchmarking infrastructure on crates.io.

[ekiwi]

I would like to add an additional use-case:

A deeply embedded target like a microcontroller has the following constraints:

• only libcore available
• only stack no heap memory allocation
• restricted code size (common flash sizes are 128kB)
• no support for stack unwinding, thus no panic
• because of size constraints, stack unwinding might never be feasible
• heap allocation is not too hard to make work, it would just make this more
  universal, if we did not require it

Thus a custom test harness is needed. It might have to following properties:

• does not allocate dynamic memory
• does not use panic
• returns compact error descriptions (maybe error numbers)
• makes it possible to split up tests into several binaries
  because one monolithic binary for all tests might not fit into program memory
• there might be a component running on the host pc, that loads the
  tests onto the microcontroller and collects test results e.g. via USB

The goals/observations here are:

• the current test syntax should be kept the same (I think that it's nice to
  have a common way of writing tests in the rust community)
• the test harness should be able to be made up of at least two different
  binaries: one that runs on the host pc and another one that runs on the
  target
• there needs to be a mechanism to be able to compile only a certain
  number of tests, so that we can make sure that they still fit
  on the target

Thus, for this use case to work, it does not suffice to be able to
use a custom libtest, but the mechanism (from what I could tell) hard
coded intorustc to create test runners needs to have some more flexibility.

[reem]

cc me

[ruuda]

One thing that I would love to see is support for test samples. I have a directory with a lot of files that a certain function should be able to process. At the moment, I test this in a single test by enumerating all files and calling the function, but this has several disadvantages:

• If the test fails, it is hard to see for which sample it failed; there is no individual ok/fail per test sample. It is possible to print this to stdout, but that is suppressed by default.
• It is not possible to see intermediate progress. It can be done with stdout but it is suppressed by default.
• It is not possible (within a single test) to test multiple samples in parallel.

A framework that can run the same test for multiple inputs, and report progress and ok/fail per input, would be especially useful for regression tests, similar to the compile-fail and compile-pass tests for Rust’s make check. Crates that could benefit from this include parsers, compilers, decoders (images, audio, video, …), deserialisers, etc.

I am not sure what the api would look like. Perhaps something like

#[test(files = "testsamples/*.dat")]
fn parse_pass(path: &Path) { … }

[ekiwi]

So I was a little bit tired yesterday. I rephrased my use case above to make it easier to read. If anyone is interested in this, I could devote some more time into working out, how exactly running tests on deeply embedded targets could work and what implications this would have for rustc.

[grissiom]

Hi @ekiwi , I'm interested in test framework in deeply embedded targets. But I'm a afraid that we are facing 2 questions here:

1, get Rust run on the board(with or without an RTOS as backend).
2, implement the test framework.

For the split tests, I think the test framework should define multiple "test targets" such as: "test1", "test2", "test_boot", "test_led" and so on and make different binaries for each target.