MonteCarlo writes the inputs and outputs of one simulation non-atomically

[thc1006]

Describe the bug

A simulation's inputs and outputs go to two files in two separate appends, with nothing tying them together:

with open(self.input_file, "a", encoding="utf-8") as f:
    f.write(inputs_json)
with open(self.output_file, "a", encoding="utf-8") as f:
    f.write(outputs_json)

If the second write fails, the first is already on disk. The run then has an inputs row with no matching outputs row, and the two files no longer line up by index.

Both run paths do this, monte_carlo.py:292-295 in serial and monte_carlo.py:434-437 in parallel, on develop at 235dc6e.

To Reproduce

The shape on its own, with the second write refused:

def record(inputs_json, outputs_json, fail_second=False):
    with open(inp, "a", encoding="utf-8") as f:
        f.write(inputs_json)
    if fail_second:
        raise OSError("no space left on device")
    with open(out, "a", encoding="utf-8") as f:
        f.write(outputs_json)

record('{"index": 0}\n', '{"index": 0}\n')
record('{"index": 1}\n', '{"index": 1}\n', fail_second=True)

inputs  2 lines
outputs 1 line

A disk filling up mid-run is the ordinary way to reach this. It does not need a crash.

Expected behavior

Either both rows land or neither does. A run that stops part way through should leave a checkpoint that append=True can continue from, rather than one that has to be trimmed by hand or thrown away.

Additional context

Found while reviewing #1054, and it predates that branch. Worth being precise about what does and does not change there, since the two are easy to confuse.

#1054 adds a completeness check that compares the two files after the run and refuses to report success when they disagree, so the damage is detected rather than silent. It does not make the pair atomic, and the checkpoint is still unusable afterwards. That branch is scoped to sampled-input reproducibility, so this was deliberately left out of it.

Two ways to close it, in increasing order of work:

• write to temporary files and os.replace() them into place once both are complete, which fits the append=False case;
• have workers send complete records to the parent and let one writer commit them, which also removes the shared lock the workers currently take around the pair.

The second is the one that also fixes a torn row from a worker killed mid-write, which the lock cannot prevent.

Signed-off-by: thc1006 84045975+thc1006@users.noreply.github.com