No Case: Fixed the PPO runtime error metrics.

[RuofanKong]

Why are these changes needed?

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[Edilmo]

LGTM

[dmlyubim]

Assuming this aggregation is tested, should be ok

[dmlyubim]

Be careful, though. I remember it was tricky for 0.8. There, the rllib only seemed to have captured partial metrics. I don't remember how it was deficient; but it was skipping over some evaluation -- either towers or SGD iterations, which resulted in missed first-chance errors. So better test with some degree of confidence there that deficient inputs do get captured as first-chance validation errors, not only some second chance errors when the weights are already garbage.

[RuofanKong]

Assuming this aggregation is tested, should be ok

Yep! Your PDP2 runtime error test is a great resource to test through, while they're working now.