dump collection infra and python crypto dependency

[am11]

When linux arm32 test crashes in helix, the dump collector fails with:

/root/helix/scripts/cryptography/hazmat/bindings/_padding.abi3.so: wrong ELF class: ELFCLASS64
2021-11-27T15:41:26.934Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2021-11-27T15:41:26.934Z	INFO   	run.py	__init__(48)	get_log_files	Searching '/root/helix/work/workitem/e/..' for log files
2021-11-27T15:41:26.935Z	INFO   	run.py	__init__(66)	construct_log_list	Generated log list: 
2021-11-27T15:41:26.935Z	INFO   	run.py	__init__(90)	read_results	Searching '/root/helix/work/workitem/e' for test results files
2021-11-27T15:41:26.936Z	INFO   	run.py	__init__(90)	read_results	Searching '/root/helix/work/workitem/uploads' for test results files
2021-11-27T15:41:26.936Z	WARNING	run.py	__init__(103)	read_results	No results file found in any of the following formats: xunit, junit, trx
2021-11-27T15:41:26.937Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2021-11-27T15:41:26.937Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1710 bytes
+ python /root/helix/work/correlation/gen-debug-dump-docs.py -buildid 1487738 -workitem Invariant.Tests -jobid 2f68d8e7-c271-4e09-a101-d06ab0ecba67 -outdir /root/helix/work/workitem/uploads -templatedir /root/helix/work/correlation -dumpdir /home/helixbot/dotnetbuild/dumps -productver 7.0.0
read file: /root/helix/work/correlation/debug-dump-template.md
writing output file: /root/helix/work/workitem/uploads/how-to-debug-dump.md
done writing debug dump information
+ exit 139
+ export _commandExitCode=139
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 139

This is because the crypto package used by the tool does not support arm32 architecture and the request to port was rejected (pyca/cryptography#6286). Perhaps we should look into a different option for dump collection? e.g. tool based on .NET so it scales accordingly for all supported targets.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[danmoseley]

@MattGal should this go to core-eng

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

When linux arm32 test crashes in helix, the dump collector fails with:

/root/helix/scripts/cryptography/hazmat/bindings/_padding.abi3.so: wrong ELF class: ELFCLASS64
2021-11-27T15:41:26.934Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2021-11-27T15:41:26.934Z	INFO   	run.py	__init__(48)	get_log_files	Searching '/root/helix/work/workitem/e/..' for log files
2021-11-27T15:41:26.935Z	INFO   	run.py	__init__(66)	construct_log_list	Generated log list: 
2021-11-27T15:41:26.935Z	INFO   	run.py	__init__(90)	read_results	Searching '/root/helix/work/workitem/e' for test results files
2021-11-27T15:41:26.936Z	INFO   	run.py	__init__(90)	read_results	Searching '/root/helix/work/workitem/uploads' for test results files
2021-11-27T15:41:26.936Z	WARNING	run.py	__init__(103)	read_results	No results file found in any of the following formats: xunit, junit, trx
2021-11-27T15:41:26.937Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2021-11-27T15:41:26.937Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1710 bytes
+ python /root/helix/work/correlation/gen-debug-dump-docs.py -buildid 1487738 -workitem Invariant.Tests -jobid 2f68d8e7-c271-4e09-a101-d06ab0ecba67 -outdir /root/helix/work/workitem/uploads -templatedir /root/helix/work/correlation -dumpdir /home/helixbot/dotnetbuild/dumps -productver 7.0.0
read file: /root/helix/work/correlation/debug-dump-template.md
writing output file: /root/helix/work/workitem/uploads/how-to-debug-dump.md
done writing debug dump information
+ exit 139
+ export _commandExitCode=139
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 139

This is because the crypto package used by the tool does not support arm32 architecture and the request to port was rejected (pyca/cryptography#6286). Perhaps we should look into a different option for dump collection? e.g. tool based on .NET so it scales accordingly for all supported targets.

Author:	am11
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[MattGal]

@MattGal should this go to core-eng

Checking it out, will create an issue there if so...

[MattGal]

@am11 can you link the run you copied those logs from? Checking out other runs with crashes from ARM32 runs (example) I don't see any errors. Further, when using the packing test reporter, all the interesting uses of the crypto library on a Helix machine happen on the outer ARM64v8 host, so this is not likely blocking.

@danmoseley given this is all about https://github.com/dotnet/runtime/blob/main/eng/testing/gen-debug-dump-docs.py, and that nothing this script uses seems to import or use this whl, I don't think this is a core-eng issue. I'll definitely follow up with Adeel when he replies so I can get a sense of what happened on the unknown run these logs come from.

[am11]

@am11 can you link the run you copied those logs from?

I am working on updating libunwind dependency, and was trying to investigate the remaining arm32 crashes #62092.

linux arm: https://github.com/dotnet/runtime/pull/62092/checks?check_run_id=4341922004
linux-musl arm: https://github.com/dotnet/runtime/pull/62092/checks?check_run_id=4341931347

Checking out other runs with crashes from ARM32 runs (example) I don't see any errors.

Actually there is an error in that alpine run with a bit different wordings, search on page: _padding.abi3.so: Exec format error.
In glibc based distros, same error looks like _padding.abi3.so: wrong ELF class: ELFCLASS64.

[MattGal]

@am11 got it, I was looking for issues with the dump collection infra. This is actually being logged from the Azure Devops test reporter, and is thankfully benign.

We've been aware of these problems for some time, this is why we have to do the pip install inside the container. This issue, and other issues like "what happens when AzDO doesn't want to take our test results inside the work item?" were worked around by having reporting occur outside the work item.

In this case, since you now use the "packing" test reporter, nothing interesting is actually broken in this scenario. The reporter packs up any test results it finds to a common JSON format, then outside of the agent it uses a happier, ARM64 version of crypto (which, if it didn't work, the helix agent would die) to report. If you see something like this in the outer logs (example) then everything it wants to do gets done:

2021-11-29T18:41:28.553Z	INFO   	packing_test_reporter(47)	unpack_results	Unpacking 3635 bytes from '/ssd/work/B2C50977/w/B0C00966/e/__test_report.json'
2021-11-29T18:41:28.554Z	INFO   	packing_test_reporter(54)	unpack_results	Unpacked 8 test reports

@danmoseley now that I have these details I don't think this is interesting to track, but let me know if I'm missing something.

[danmoseley]

@MattGal sounds good thanks.

[safern]

Am I understanding correctly that there is no further action on our side on this issue and that we can close it?

[am11]

Yes, I think this is an external (Azure DevOps) issue and can be closed. I saw that azure-cli et al. are heavy consumers of hazmat crypto package https://github.com/search?q=org:Azure+hazmat+cryptography&type=code, but I don't know which repo in particular is the contributing factor to wrong ELF class: ELFCLASS64 message that we get in our logs.