Problem deploying mons for `orch:cephadm:smoke-small` suite

[VallariAg]

I faced a problem with cephadm deploying mons with the macos branch (running with export TEUTHOLOGY_SUITE=orch:cephadm:smoke-small)
I tested it on centos9 smithi machine (It is persistent even after system prune).

Saw these in teuthology logs.

2024-06-06T12:57:09.843 INFO:teuthology.orchestra.run.4dd2579424b6.stdout:/usr/bin/ceph: stderr terminate called after throwing an instance of 'std::system_error'
2024-06-06T12:57:09.843 INFO:teuthology.orchestra.run.4dd2579424b6.stdout:/usr/bin/ceph: stderr   what():  No such file or directory
2024-06-06T12:57:09.843 INFO:teuthology.orchestra.run.4dd2579424b6.stdout:mon not available, waiting (14/15)...
2024-06-06T12:57:13.514 INFO:teuthology.orchestra.run.4dd2579424b6.stdout:Non-zero exit code 134 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:main -e NODE_NAME=4dd2579424b6 -v /var/lib/ceph/ec2f82c1-2403-11ef-be25-5d8b51cbbedc/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmp4rx7nehq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1fij3c01:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:main status
2024-06-06T12:57:13.514 INFO:teuthology.orchestra.run.4dd2579424b6.stdout:/usr/bin/ceph: stderr terminate called after throwing an instance of 'std::system_error'
2024-06-06T12:57:13.514 INFO:teuthology.orchestra.run.4dd2579424b6.stdout:/usr/bin/ceph: stderr   what():  No such file or directory
2024-06-06T12:57:13.514 INFO:teuthology.orchestra.run.4dd2579424b6.stdout:mon not available, waiting (15/15)...
2024-06-06T12:57:16.940 INFO:teuthology.orchestra.run.4dd2579424b6.stdout:Non-zero exit code 134 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:main -e NODE_NAME=4dd2579424b6 -v /var/lib/ceph/ec2f82c1-2403-11ef-be25-5d8b51cbbedc/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmp4rx7nehq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1fij3c01:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:main status
2024-06-06T12:57:16.940 INFO:teuthology.orchestra.run.4dd2579424b6.stdout:/usr/bin/ceph: stderr terminate called after throwing an instance of 'std::system_error'
2024-06-06T12:57:16.940 INFO:teuthology.orchestra.run.4dd2579424b6.stdout:/usr/bin/ceph: stderr   what():  No such file or directory
2024-06-06T12:57:16.940 INFO:teuthology.orchestra.run.4dd2579424b6.stderr:Error: mon not available after 15 tries

Here are the full teuthology logs:
teuthology.log

[VallariAg]

@zmc this was the macos branch issue I was talking about on call.