Bad CPU type in executable causing iOS device test failure (exit code 21)

[MatousBot]

Suggested labels: Known Build Error, test-failure, os-ios, arch-arm64, untriaged

Note: This issue was automatically generated by AI-Test-Triage. Please review and triage accordingly.

Build Information

Build: 20260318.144, 20260318.5

Build error leg or test failing: iOS.Device.ParallelForEachAsync.Test.WorkItemExecution, iOS.CoreCLR.R2R.Test.WorkItemExecution, CoreMangLib.WorkItemExecution, Directed_2.WorkItemExecution, Exceptions.WorkItemExecution, GC-scenarios1.WorkItemExecution, IL_Conformance.WorkItemExecution, JIT.CodeGenBringUpTests_d.WorkItemExecution, JIT.CodeGenBringUpTests_do.WorkItemExecution, JIT.CodeGenBringUpTests_ro.WorkItemExecution, JIT.opt.WorkItemExecution, LoaderClassloaderGenerics.WorkItemExecution, Methodical_others.WorkItemExecution, Methodical_r1.WorkItemExecution, Methodical_ro.WorkItemExecution, Regression_2.WorkItemExecution, Regression_3.WorkItemExecution, Regression_4.WorkItemExecution, Regression_PdbOnly_r.WorkItemExecution, Regression_PdbOnly_r_3.WorkItemExecution, Regression_PdbOnly_r_4.WorkItemExecution, Regression_PdbOnly_ro.WorkItemExecution, Regression_d.WorkItemExecution, Regression_ro_1.WorkItemExecution, Regressions.WorkItemExecution, baseservices.WorkItemExecution, iOS.CoreCLR.Interpreter.Test.WorkItemExecution

Affected CI jobs: build_ios_arm64_Release_AllSubsets_Mono_Smoke, build_ios_arm64_Release_AllSubsets_NativeAOT_Smoke, build_ios_arm64_Release_AllSubsets_CoreCLR_Smoke, build_ios_arm64_Release_AllSubsets_Mono_RuntimeTests, build_ios_arm64_Release_AllSubsets_CoreCLR

Causing Commit Range Information

Commit range: {TODO: Insert commit range or link to changes}

Summary

• Process terminated with exit code 21 during xharness/apple build step
• Log shows error 'Bad CPU type in executable' when running /tmp/helix/working/B7200A35/p/dotnet-cli/dotnet
• Job: build_ios_arm64_Release_AllSubsets_Mono_Smoke; WorkItem: iOS.Device.ParallelForEachAsync.Test
• xharness invoked with --expected-exit-code 42 but script exited 21
• Build artifacts missing: AOTBuild.binlog move failed (No such file or directory)
• App signing failed: No embedded provisioning profile found; 'Failed to sign the app!'
• No xunit xml output found in uploads directory (ls: .../uploads/xunit-*.xml: No such file or directory)

Failure Statistics

Note: These statistics are based on the configured time range and might not reflect the full history of the failure.

• Occurrence Count: 29
• First Failure: 2026-03-19 01:57:09 UTC
• Last Failure: 2026-03-19 09:53:20 UTC

Work Items

Showing 10 most recent of 29 occurrences

Work Item	Console Log	Build	Job Name
Regression_PdbOnly_r Work Item	View Extract + sudo launchctl asuser 503 sudo -u helix-runner sh ./xharness-runner.apple.sh --target ios-device --command-timeout 11280 --timeout 03:00:00 --launch-timeout 00:05:00 --includes-test-runner --expected-exit-code 0 --app /tmp/helix/working/ACBC097A/w/B20909B4/e/Regression_PdbOnly_r.app --output-directory /tmp/helix/working/ACBC097A/w/B20909B4/uploads Device target detected but app not found, skipping signing.. build-apple-app.sh: line 5: /tmp/helix/working/ACBC097A/p/dotnet-cli/dotnet: Bad CPU type in executable mv: rename /private/tmp/helix/working/ACBC097A/w/B20909B4/e/xharness-output/AOTBuild.binlog to /tmp/helix/working/ACBC097A/w/B20909B4/uploads/AOTBuild.binlog: No such file or directory Signing bundle /tmp/helix/working/ACBC097A/w/B20909B4/e/Regression_PdbOnly_r.app No embedded provisioning profile found at /tmp/helix/working/ACBC097A/w/B20909B4/e/Regression_PdbOnly_r.app/embedded.mobileprovision! Failed to sign the app! Removing empty log files: ls: /tmp/helix/working/ACBC097A/w/B20909B4/uploads/xunit-*.xml: No such file or directory + exit_code=21 Full Log	20260318.5	build_ios_arm64_Release_AllSubsets_Mono_RuntimeTests
Regression_PdbOnly_r_3 Work Item	View Extract + sudo launchctl asuser 503 sudo -u helix-runner sh ./xharness-runner.apple.sh --target ios-device --command-timeout 11280 --timeout 03:00:00 --launch-timeout 00:05:00 --includes-test-runner --expected-exit-code 0 --app /tmp/helix/working/ACBC097A/w/9BCB0901/e/Regression_PdbOnly_r_3.app --output-directory /tmp/helix/working/ACBC097A/w/9BCB0901/uploads Device target detected but app not found, skipping signing.. build-apple-app.sh: line 5: /tmp/helix/working/ACBC097A/p/dotnet-cli/dotnet: Bad CPU type in executable mv: rename /private/tmp/helix/working/ACBC097A/w/9BCB0901/e/xharness-output/AOTBuild.binlog to /tmp/helix/working/ACBC097A/w/9BCB0901/uploads/AOTBuild.binlog: No such file or directory Signing bundle /tmp/helix/working/ACBC097A/w/9BCB0901/e/Regression_PdbOnly_r_3.app No embedded provisioning profile found at /tmp/helix/working/ACBC097A/w/9BCB0901/e/Regression_PdbOnly_r_3.app/embedded.mobileprovision! Failed to sign the app! Removing empty log files: ls: /tmp/helix/working/ACBC097A/w/9BCB0901/uploads/xunit-*.xml: No such file or directory + exit_code=21 Full Log	20260318.5	build_ios_arm64_Release_AllSubsets_Mono_RuntimeTests
Regression_PdbOnly_r_4 Work Item	View Extract + sudo launchctl asuser 503 sudo -u helix-runner sh ./xharness-runner.apple.sh --target ios-device --command-timeout 11280 --timeout 03:00:00 --launch-timeout 00:05:00 --includes-test-runner --expected-exit-code 0 --app /tmp/helix/working/ACBC097A/w/BB4509E6/e/Regression_PdbOnly_r_4.app --output-directory /tmp/helix/working/ACBC097A/w/BB4509E6/uploads Device target detected but app not found, skipping signing.. build-apple-app.sh: line 5: /tmp/helix/working/ACBC097A/p/dotnet-cli/dotnet: Bad CPU type in executable mv: rename /private/tmp/helix/working/ACBC097A/w/BB4509E6/e/xharness-output/AOTBuild.binlog to /tmp/helix/working/ACBC097A/w/BB4509E6/uploads/AOTBuild.binlog: No such file or directory Signing bundle /tmp/helix/working/ACBC097A/w/BB4509E6/e/Regression_PdbOnly_r_4.app No embedded provisioning profile found at /tmp/helix/working/ACBC097A/w/BB4509E6/e/Regression_PdbOnly_r_4.app/embedded.mobileprovision! Failed to sign the app! Removing empty log files: ls: /tmp/helix/working/ACBC097A/w/BB4509E6/uploads/xunit-*.xml: No such file or directory + exit_code=21 + '[' -f /tmp/helix/working/ACBC097A/w/BB4509E6/e/.retry ']' + '[' -f /tmp/helix/working/ACBC097A/w/BB4509E6/e/.reboot ']' Full Log	20260318.5	build_ios_arm64_Release_AllSubsets_Mono_RuntimeTests
Regression_PdbOnly_ro Work Item	View Extract + sudo launchctl asuser 503 sudo -u helix-runner sh ./xharness-runner.apple.sh --target ios-device --command-timeout 11280 --timeout 03:00:00 --launch-timeout 00:05:00 --includes-test-runner --expected-exit-code 0 --app /tmp/helix/working/ACBC097A/w/B1B10984/e/Regression_PdbOnly_ro.app --output-directory /tmp/helix/working/ACBC097A/w/B1B10984/uploads Device target detected but app not found, skipping signing.. build-apple-app.sh: line 5: /tmp/helix/working/ACBC097A/p/dotnet-cli/dotnet: Bad CPU type in executable mv: rename /private/tmp/helix/working/ACBC097A/w/B1B10984/e/xharness-output/AOTBuild.binlog to /tmp/helix/working/ACBC097A/w/B1B10984/uploads/AOTBuild.binlog: No such file or directory Signing bundle /tmp/helix/working/ACBC097A/w/B1B10984/e/Regression_PdbOnly_ro.app No embedded provisioning profile found at /tmp/helix/working/ACBC097A/w/B1B10984/e/Regression_PdbOnly_ro.app/embedded.mobileprovision! Failed to sign the app! Removing empty log files: ls: /tmp/helix/working/ACBC097A/w/B1B10984/uploads/xunit-*.xml: No such file or directory + exit_code=21 + '[' -f /tmp/helix/working/ACBC097A/w/B1B10984/e/.retry ']' + '[' -f /tmp/helix/working/ACBC097A/w/B1B10984/e/.reboot ']' Full Log	20260318.5	build_ios_arm64_Release_AllSubsets_Mono_RuntimeTests
Regression_d Work Item	View Extract + sudo launchctl asuser 503 sudo -u helix-runner sh ./xharness-runner.apple.sh --target ios-device --command-timeout 11280 --timeout 03:00:00 --launch-timeout 00:05:00 --includes-test-runner --expected-exit-code 0 --app /tmp/helix/working/ACBC097A/w/961608BE/e/Regression_d.app --output-directory /tmp/helix/working/ACBC097A/w/961608BE/uploads Device target detected but app not found, skipping signing.. build-apple-app.sh: line 5: /tmp/helix/working/ACBC097A/p/dotnet-cli/dotnet: Bad CPU type in executable mv: rename /private/tmp/helix/working/ACBC097A/w/961608BE/e/xharness-output/AOTBuild.binlog to /tmp/helix/working/ACBC097A/w/961608BE/uploads/AOTBuild.binlog: No such file or directory Signing bundle /tmp/helix/working/ACBC097A/w/961608BE/e/Regression_d.app No embedded provisioning profile found at /tmp/helix/working/ACBC097A/w/961608BE/e/Regression_d.app/embedded.mobileprovision! Failed to sign the app! Removing empty log files: ls: /tmp/helix/working/ACBC097A/w/961608BE/uploads/xunit-*.xml: No such file or directory + exit_code=21 Full Log	20260318.5	build_ios_arm64_Release_AllSubsets_Mono_RuntimeTests
Regression_ro_1 Work Item	View Extract + sudo launchctl asuser 503 sudo -u helix-runner sh ./xharness-runner.apple.sh --target ios-device --command-timeout 11280 --timeout 03:00:00 --launch-timeout 00:05:00 --includes-test-runner --expected-exit-code 0 --app /tmp/helix/working/ACBC097A/w/A1E808E3/e/Regression_ro_1.app --output-directory /tmp/helix/working/ACBC097A/w/A1E808E3/uploads Device target detected but app not found, skipping signing.. build-apple-app.sh: line 5: /tmp/helix/working/ACBC097A/p/dotnet-cli/dotnet: Bad CPU type in executable mv: rename /private/tmp/helix/working/ACBC097A/w/A1E808E3/e/xharness-output/AOTBuild.binlog to /tmp/helix/working/ACBC097A/w/A1E808E3/uploads/AOTBuild.binlog: No such file or directory Signing bundle /tmp/helix/working/ACBC097A/w/A1E808E3/e/Regression_ro_1.app No embedded provisioning profile found at /tmp/helix/working/ACBC097A/w/A1E808E3/e/Regression_ro_1.app/embedded.mobileprovision! Failed to sign the app! Removing empty log files: ls: /tmp/helix/working/ACBC097A/w/A1E808E3/uploads/xunit-*.xml: No such file or directory + exit_code=21 + '[' -f /tmp/helix/working/ACBC097A/w/A1E808E3/e/.retry ']' + '[' -f /tmp/helix/working/ACBC097A/w/A1E808E3/e/.reboot ']' Full Log	20260318.5	build_ios_arm64_Release_AllSubsets_Mono_RuntimeTests
Regressions Work Item	View Extract + sudo launchctl asuser 503 sudo -u helix-runner sh ./xharness-runner.apple.sh --target ios-device --command-timeout 11280 --timeout 03:00:00 --launch-timeout 00:05:00 --includes-test-runner --expected-exit-code 0 --app /tmp/helix/working/ACBC097A/w/B0D1099D/e/Regressions.app --output-directory /tmp/helix/working/ACBC097A/w/B0D1099D/uploads Device target detected but app not found, skipping signing.. build-apple-app.sh: line 5: /tmp/helix/working/ACBC097A/p/dotnet-cli/dotnet: Bad CPU type in executable mv: rename /private/tmp/helix/working/ACBC097A/w/B0D1099D/e/xharness-output/AOTBuild.binlog to /tmp/helix/working/ACBC097A/w/B0D1099D/uploads/AOTBuild.binlog: No such file or directory Signing bundle /tmp/helix/working/ACBC097A/w/B0D1099D/e/Regressions.app No embedded provisioning profile found at /tmp/helix/working/ACBC097A/w/B0D1099D/e/Regressions.app/embedded.mobileprovision! Failed to sign the app! Removing empty log files: ls: /tmp/helix/working/ACBC097A/w/B0D1099D/uploads/xunit-*.xml: No such file or directory + exit_code=21 + '[' -f /tmp/helix/working/ACBC097A/w/B0D1099D/e/.retry ']' + '[' -f /tmp/helix/working/ACBC097A/w/B0D1099D/e/.reboot ']' Full Log	20260318.5	build_ios_arm64_Release_AllSubsets_Mono_RuntimeTests
baseservices Work Item	View Extract + sudo launchctl asuser 503 sudo -u helix-runner sh ./xharness-runner.apple.sh --target ios-device --command-timeout 11280 --timeout 03:00:00 --launch-timeout 00:05:00 --includes-test-runner --expected-exit-code 0 --app /tmp/helix/working/ACBC097A/w/A69F0958/e/baseservices.app --output-directory /tmp/helix/working/ACBC097A/w/A69F0958/uploads Device target detected but app not found, skipping signing.. build-apple-app.sh: line 5: /tmp/helix/working/ACBC097A/p/dotnet-cli/dotnet: Bad CPU type in executable mv: rename /private/tmp/helix/working/ACBC097A/w/A69F0958/e/xharness-output/AOTBuild.binlog to /tmp/helix/working/ACBC097A/w/A69F0958/uploads/AOTBuild.binlog: No such file or directory Signing bundle /tmp/helix/working/ACBC097A/w/A69F0958/e/baseservices.app No embedded provisioning profile found at /tmp/helix/working/ACBC097A/w/A69F0958/e/baseservices.app/embedded.mobileprovision! Failed to sign the app! Removing empty log files: ls: /tmp/helix/working/ACBC097A/w/A69F0958/uploads/xunit-*.xml: No such file or directory + exit_code=21 + '[' -f /tmp/helix/working/ACBC097A/w/A69F0958/e/.retry ']' + '[' -f /tmp/helix/working/ACBC097A/w/A69F0958/e/.reboot ']' Full Log	20260318.5	build_ios_arm64_Release_AllSubsets_Mono_RuntimeTests
iOS.CoreCLR.Interpreter.Test Work Item	View Extract + sudo launchctl asuser 503 sudo -u helix-runner sh ./xharness-runner.apple.sh --target ios-device --command-timeout 1200 --timeout 00:12:00 --launch-timeout 00:05:00 --expected-exit-code 42 --app /tmp/helix/working/B1700944/w/B8340983/e/iOS.CoreCLR.Interpreter.Test.app --output-directory /tmp/helix/working/B1700944/w/B8340983/uploads Device target detected but app not found, skipping signing.. build-apple-app.sh: line 5: /tmp/helix/working/B1700944/p/dotnet-cli/dotnet: Bad CPU type in executable mv: rename /private/tmp/helix/working/B1700944/w/B8340983/e/xharness-output/AOTBuild.binlog to /tmp/helix/working/B1700944/w/B8340983/uploads/AOTBuild.binlog: No such file or directory Signing bundle /tmp/helix/working/B1700944/w/B8340983/e/iOS.CoreCLR.Interpreter.Test.app No embedded provisioning profile found at /tmp/helix/working/B1700944/w/B8340983/e/iOS.CoreCLR.Interpreter.Test.app/embedded.mobileprovision! Failed to sign the app! Removing empty log files: ls: /tmp/helix/working/B1700944/w/B8340983/uploads/xunit-*.xml: No such file or directory + exit_code=21 + '[' -f /tmp/helix/working/B1700944/w/B8340983/e/.retry ']' Full Log	20260318.5	build_ios_arm64_Release_AllSubsets_CoreCLR
iOS.CoreCLR.R2R.Test Work Item	View Extract + sudo launchctl asuser 503 sudo -u helix-runner sh ./xharness-runner.apple.sh --target ios-device --command-timeout 1200 --timeout 00:12:00 --launch-timeout 00:05:00 --expected-exit-code 42 --app /tmp/helix/working/B1700944/w/A4B20939/e/iOS.CoreCLR.R2R.Test.app --output-directory /tmp/helix/working/B1700944/w/A4B20939/uploads Device target detected but app not found, skipping signing.. build-apple-app.sh: line 5: /tmp/helix/working/B1700944/p/dotnet-cli/dotnet: Bad CPU type in executable mv: rename /private/tmp/helix/working/B1700944/w/A4B20939/e/xharness-output/AOTBuild.binlog to /tmp/helix/working/B1700944/w/A4B20939/uploads/AOTBuild.binlog: No such file or directory Signing bundle /tmp/helix/working/B1700944/w/A4B20939/e/iOS.CoreCLR.R2R.Test.app No embedded provisioning profile found at /tmp/helix/working/B1700944/w/A4B20939/e/iOS.CoreCLR.R2R.Test.app/embedded.mobileprovision! Failed to sign the app! Removing empty log files: ls: /tmp/helix/working/B1700944/w/A4B20939/uploads/xunit-*.xml: No such file or directory + exit_code=21 + '[' -f /tmp/helix/working/B1700944/w/A4B20939/e/.retry ']' + '[' -f /tmp/helix/working/B1700944/w/A4B20939/e/.reboot ']' Full Log	20260318.5	build_ios_arm64_Release_AllSubsets_CoreCLR

Error Message

{
  "ErrorMessage": [
    "Bad CPU type in executable",
    "No embedded provisioning profile found"
  ],
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Test	Pull Request
1348323	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#124328
1348301	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#125945
1348298	dotnet/runtime	iOS.Device.ParallelForEachAsync.Test.WorkItemExecution	#125151
1348249	dotnet/runtime	iOS.CoreCLR.R2R.Test.WorkItemExecution	#125441
1348236	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#125734
1348169	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#125955
1348113	dotnet/runtime	reflection.WorkItemExecution
1347880	dotnet/runtime	iOS.CoreCLR.R2R.Test.WorkItemExecution	#125950
1347667	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#125604
1347636	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#125945
1347473	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#125734
1345393	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#125755
1345239	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#125844
1344953	dotnet/runtime	iOS.Device.ParallelForEachAsync.Test.WorkItemExecution
1344853	dotnet/runtime	iOS.Device.ParallelForEachAsync.Test.WorkItemExecution	#125817
1344780	dotnet/runtime	iOS.Device.ParallelForEachAsync.Test.WorkItemExecution	#125829
1344794	dotnet/runtime	iOS.Device.ParallelForEachAsync.Test.WorkItemExecution
1344681	dotnet/runtime	iOS.Device.ParallelForEachAsync.Test.WorkItemExecution	#125441
1344650	dotnet/runtime	iOS.Device.ParallelForEachAsync.Test.WorkItemExecution	#124553
1344648	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#124629
1341883	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#125745

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	21

[dotnet-policy-service]

Tagging subscribers to this area: @akoeplinger, @matouskozak, @simonrozsival
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to 'os-ios': @vitek-karas, @kotlarmilos, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

Same as #125776?

[vitek-karas]

Note

This comment was AI/Copilot-generated.

Diagnostic run: PR #125897, runtime-ioslike build 1346650
Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1346650

Observed in Helix job d2bb5f40-cd12-4a57-89b0-2655b54b55fb:

• System.Runtime.Tests on DNCENGMAC174, macOS-15.7.4: arm64, proc_translated=0, rosetta_receipt=missing, rosetta_runtime=missing, x64_exec_works=false, x64 dotnet, then Bad CPU type in executable. Console: https://helix.dot.net/api/2019-06-17/jobs/d2bb5f40-cd12-4a57-89b0-2655b54b55fb/workitems/System.Runtime.Tests/console
• iOS.CoreCLR.Interpreter.Test on DNCENGMAC166, macOS-15.7.4: same result. Console: https://helix.dot.net/api/2019-06-17/jobs/d2bb5f40-cd12-4a57-89b0-2655b54b55fb/workitems/iOS.CoreCLR.Interpreter.Test/console
• iOS.CoreCLR.R2R.Test on DNCENGMAC124, macOS-15.7: x86_64, x64_exec_works=true, same x64 dotnet, and it got past the AOT build. Console: https://helix.dot.net/api/2019-06-17/jobs/d2bb5f40-cd12-4a57-89b0-2655b54b55fb/workitems/iOS.CoreCLR.R2R.Test/console

This run confirms the queue is mixed and the failure is on arm64 hosts without Rosetta trying to run the x64 Helix dotnet-cli.

[vitek-karas]

Validated against actual runs after 2026-03-26, not just the tracking issue. The pre-fix baseline (1346650) still shows the old arm64/no-Rosetta Bad CPU type in executable failure, while post-fix runs I checked (1353622, 1354852, 1357845, 1358791) no longer show that signature and appear to be running on x64 hosts, so this looks fixed for osx.15.amd64.iphone.open.

[vitek-karas]

The underlying issue (mixed hardware in the Helix queue) has been fixed. There don't seem to be any failures in the recent runs. So I'm closing this.