Event Framework - Update Event Client SDK, Sidecar & existing logging for Async Audit capabilities

[mdebarros]

Goal:

As a Hub Operator I want to integrate Event Framework into Mojaloop components so that so that all events (errors, general, etc) can be audited and traced appropriately

Tasks:

• Event Framework
  • Event SDK for Clients [ ? ]
    • Update to enable "aysnc"-logging for when the audit events are published to the event stream.
  • Event SDK for Server [ ? ]
    • Update to enable "aysnc"-logging for when the audit events are published to the event stream.
  • Event Sidecar for Server [ ? ]
    • Update to enable "aysnc"-logging for when the audit events are published to the event stream.
• Existing Logging
  • Updated central-services-logger to use async logging via a file <-- ensure that the transport is configurable and can be changed from a file back to stdout <-- This is needed to ensure that logging is done asynchronously as logging to stdout is sync (example used in last perf work stream: https://github.com/mojaloop/central-services-shared/blob/release/v4.4.0-performance/src/logger.js)
  • Reconfigure docker containers to tail log file (example used in last perf work stream: https://github.com/mojaloop/central-ledger/blob/release/4.4.0-performance/Dockerfile)
    • Apply for:
      • account-lookup-service [PR]
      • bulk-api-adapter [PR]
      • central-event-processor
      • central-ledger
      • central-settlement [PR]
      • email-notifier
      • ml-api-adapter [PR]
      • quoting-service [PR]
        - [ ] mojaloop-simulator Not relevant as it still uses a custom logger

Acceptance Criteria:

• Designs are up-to-date
• Unit Tests pass
• Integration Tests pass
• Code Style & Coverage meets standards
• Changes made to config (default.json) are broadcast to team and follow-up tasks added to update helm charts and other deployment configs.
• Event SDK for clients:
  • Normal logs via Winston are done so via an "aysnc"-fashion to reduce IO

Pull Requests:

• Feature/846 async logging event-sdk#14

• Feature/846 async logging central-services-logger#7

• Fix/8.5.0 audit failure central-services-logger#8

• fix handlebars audit failure central-services-logger#9

• Fix/default transport central-services-logger#10 <-- fixes default output

• feature/846-async-logging email-notifier#84

• Feature/846 async logging central-event-processor#103

• Feature/846 async logging account-lookup-service#167

• Feature/846 async logging quoting-service#100

• Feature/846 async logging central-ledger#497

• Feature/846 async logging  ml-api-adapter#285

• feature/846 async logging bulk-api-adapter#17

• Feature/846 async logging central-settlement#199

• Feature/846 async logging 2 event-sdk#19

• feature/846 async logging helm#234 <-- Helm Changes

Follow-up:

• Update the following environment variables in helm:

  • EVENT_SDK_ASYNC_OVERRIDE_EVENTS -> A comma-separated list of events that should return immediately instead of waiting for the promises to resolve in the recorder.record() function. Any combination of: 'log,audit,trace'
  • CSL_LOG_TRANSPORT: 'file'
  • CSL_TRANSPORT_FILE_OPTIONS (optional dict, refer to central-services-logger readme)

• Create follow up stories for:

  • Ensuring a fully synchronous flow from publishing an event with the EventSDK client, and said event being received by the EventSDK Server (running in the sidecar) and published on the kafka stream
    • Currently, this uses Event.emit(), which is executed asynchronously.
    • As per discussion with @mdebarros, we essentially want a "stop the world" ability, to make sure that certain events (e.g. audit events) have guarantees that they are published to kafka.

Dependencies:

• N/A

Accountability:

• Owner: TBC
• QA/Review: TBC

[lewisdaly]

@mdebarros my first thought for how to implement the async vs sync event sdk is as follows:

If the async env variable is true, then don't wait for the recordMessage() function to return. For example:

/* event-sdk: src/span.ts*/
 /**
   * Logs INFO type message.
   * @param message if message is a string, the message is added to a message property of context of an event message.
   * If message is not following the event framework message format, the message is added as it is to the context of an event message.
   * If message follows the event framework message format, only the metadata is updated and if message lacks an UUID it is created. 
   * @param state optional parameter for state. Defaults to 'success'
   */
  async info(message: TypeOfMessage, state?: EventStateMetadata): Promise<any> {
    let { action, type } = new LogEventTypeAction(LogEventAction.info)

    //If ASYNC, then don't wait for this.record message to return
    if (Config.ASYNC) {
      this.recordMessage(message, type, action, state)
      return Promise.resolve(true)
    }

    await this.recordMessage(message, type, action, state)
  }

The only issue I can see of the top of my head is the potential for uncaught rejected promises.

Is this implementation similar to what you were thinking?

[lewisdaly]

@mdebarros re. the 2 points above:

Updated central-services-logger to use async logging via a file

Reconfigure docker containers to tail log file

Since we are going to be configuring the dockerfiles as in the example, I think we should make the file log the default option.

My only concern with this is that running these services locally for development will make viewing some of the logs more difficult to view. I guess we can obviously just set this back to using the console log for local development, but it might end up getting annoying.

Do you have any other thoughts on what should be our sensible default options?

[mdebarros]

@mdebarros re. the 2 points above:

Updated central-services-logger to use async logging via a file

Reconfigure docker containers to tail log file

Since we are going to be configuring the dockerfiles as in the example, I think we should make the file log the default option.

By this, you mean default in the Docker image, or default when you run the service natively?

My only concern with this is that running these services locally for development will make viewing some of the logs more difficult to view. I guess we can obviously just set this back to using the console log for local development, but it might end up getting annoying.

If possible, it would be good if we could output to console when running natively (i.e. for local development), and only in the docker image to a file by default.

Do you have any other thoughts on what should be our sensible default options?

I would say defaults should be based on use-case:

1. Native local (i.e. for development) --> console
2. Docker image --> log file which is redirected to stdout (as per example)

What are your thoughts?

[lewisdaly]

@mdebarros, yeah I think console for local logging, and a logfile for the dockerfile sounds like the best option, and should be do-able.

By this, you mean default in the Docker image, or default when you run the service natively?

Yeah I mean the default for when running inside of docker

[lewisdaly]

Waiting on the release of v8.5.1 of central-services-logger, which failed because of npm audit issues.

[lewisdaly]

As per conversation with @mdebarros today: we need the ability to control async vs non async per log level.

[lewisdaly]

Currently working on the helm updates to enable this functionality.