[qnap_nas] Add RFC 5424 syslog support for QTS 5.x access events. - #20734
[qnap_nas] Add RFC 5424 syslog support for QTS 5.x access events.#20734ie-ops wants to merge 2 commits into
Conversation
ie-ops
added
enhancement
|
Pinging @elastic/integration-experience (Team:Integration-Experience) |
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
|
Changelog link mismatch — expected
Tip If expected, add the |
💔 Build Failed
Failed CI StepsHistory |
![vera-review-bot[bot]](https://avatars.github.com/in/3297022?s=60&v=4){kind=small}
| SHARED: 'Users: (---|(%{DATA:user.domain}\\)?%{DATA:user.name}), Source IP: (---|127.0.0.1|%{IP:source.address}), Computer name: (---|%{HOSTNAME:source.domain})' | ||
| RESOURCE: '(\[%{DATA:qnap.nas.application}\] )?(---|%{FILE_PATH:qnap.nas.file.path}|%{DATA:qnap.nas.application})' | ||
| FILE_PATH: '[_%\(\)!$@:.,+~\-\s[:alnum:]]*(\/[_%\(\)!$@:.,+~\-\s[:alnum:]]*)+' | ||
| if: ctx._tmp?.sd_params == null |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/qnap_nas/data_stream/log/elasticsearch/ingest_pipeline/default.yml:170
The RFC 5424 MSG segment is captured into _tmp.message but never mapped anywhere, so the accessed-resource value is silently discarded; map it to qnap.nas.application for the structured-data branch.
Details
The new grok pattern (line 31) captures everything after the structured-data block into _tmp.message -- for the test fixtures that is Administration, the accessed resource. This new if: ctx._tmp?.sd_params == null guard disables grok__tmp_message_c75b80dc, which is the only processor that consumes _tmp.message, and remove_3f4f84fc then deletes the whole _tmp object, so the value is dropped.
This is a parity loss against the RFC-3164 path: the legacy access line ... Accessed resources: Administration, Action: Login Success produces qnap.nas.application: "Administration" (see test-access.log-expected.json lines 242 and 712). The equivalent RFC 5424 records for alice.johnson and bob.smith carry application="---" in the structured data, so after this change they end up with no qnap.nas.application and no message at all -- the accessed resource is not recorded anywhere on the document.
Recommendation:
Map the RFC 5424 MSG segment onto the same field the RFC-3164 path uses, only when the structured-data value did not already supply one:
- rename:
tag: rename_tmp_message_to_qnap_nas_application
field: _tmp.message
target_field: qnap.nas.application
ignore_missing: true
if: ctx._tmp?.sd_params != null && ctx.qnap?.nas?.application == nullPlace it after the structured-data rename processors and before remove_3f4f84fc, and regenerate test-access.log-expected.json.
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| field: event.original | ||
| patterns: | ||
| - '^(%{ECS_SYSLOG_PRI})?%{SYSLOGTIMESTAMP:_tmp.timestamp} %{NAS} %{SYSLOGPROG}: %{LOG_TYPE:event.provider}: %{GREEDYDATA:_tmp.message}' | ||
| - '^(%{ECS_SYSLOG_PRI})?1 %{RFC5424_TIMESTAMP:_tmp.timestamp_iso8601} %{NAS} %{PROG:process.name}: %{POSINT:process.pid:int} - \[QuLog@Access %{DATA:_tmp.sd_params}\] %{GREEDYDATA:_tmp.message}' |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/qnap_nas/data_stream/log/elasticsearch/ingest_pipeline/default.yml:31
The new RFC 5424 pattern only matches the QuLog@Access SD-ID, so event-log records sent in RFC 5424 hit no pattern and fail the grok; generalise the SD-ID capture so they do not raise pipeline errors.
Details
The log format is a device-wide QuLog Center setting, and this package already accepts both log types in the RFC-3164 path -- LOG_TYPE: '(event log|conn log)' on line 36 -- while the docs recommend enabling both Event Log and Access Log. Once a QTS 5.x device is switched to RFC 5424, its event-log records arrive in the same framing but with a different SD-ID, and neither the RFC-3164 pattern (no SYSLOGTIMESTAMP) nor this new pattern (hard-coded QuLog@Access) matches them.
grok_event_original_cad2ef7a has no ignore_failure, so every such record falls to the pipeline on_failure handler and is indexed with event.kind: pipeline_error, unparsed. The result is that enabling RFC 5424 to gain access-log support silently breaks event-log ingestion.
Recommendation:
Capture the SD-ID instead of hard-coding it, and branch on it, so non-access QuLog records still parse their header:
- '^(%{ECS_SYSLOG_PRI})?1 %{RFC5424_TIMESTAMP:_tmp.timestamp_iso8601} %{NAS} %{PROG:process.name}: %{POSINT:process.pid:int} - \[QuLog@%{WORD:_tmp.sd_id} %{DATA:_tmp.sd_params}\] %{GREEDYDATA:_tmp.message}'Then gate the QuLog@Access-specific processors on ctx._tmp?.sd_id == 'Access' and add a pipeline test fixture for at least one non-access RFC 5424 record.
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - rename: | ||
| tag: rename_tmp_sd_mac_to_qnap_nas_mac | ||
| field: _tmp.sd.mac | ||
| target_field: qnap.nas.mac |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/qnap_nas/data_stream/log/elasticsearch/ingest_pipeline/default.yml:126
The client MAC address is stored in a custom qnap.nas.mac field although it fits ECS source.mac exactly; rename the target to source.mac and drop the custom definition.
Details
The mac structured-data parameter sits alongside ip, which this PR already maps to source.address/source.ip, so it is the MAC of the connecting client. ECS defines source.mac for exactly that, and requires the RFC 7042 notation -- upper-case hex octets separated by hyphens -- which is the format the device emits (00-00-5E-00-53-23). No transformation is needed. Keeping it as a custom qnap.nas.mac field means detection rules and the Security app's host/network views will not see it.
Recommendation:
Retarget the rename to the ECS field:
- rename:
tag: rename_tmp_sd_mac_to_source_mac
field: _tmp.sd.mac
target_field: source.mac
ignore_missing: true
if: ctx._tmp?.sd_params != nullThen drop the mac entry from data_stream/log/fields/fields.yml and add the ECS reference:
- external: ecs
name: source.mac🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| <28>May 6 04:07:56 qnap-nas01 qulogd[1743]: conn log: Users: **test*ldap***testhost-smb-abc123**lower*test**.w.n, Source IP: 10.10.10.10, Computer name: Unknown, Connection type: SAMBA, Accessed resources: ---, Action: Login Fail | ||
| <30>1 2026-06-28T19:39:06.466+01:00 host-example qulogd: 19683 - [QuLog@Access mac="00-00-5E-00-53-23" ip="192.0.2.10" user="alice.johnson" source="example-source" computer="---" application="---" action="512" action_result="0" service="1024" extra_data="" client_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5b" client_app="Web Desktop" client_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 Edg/149.0.0.0"] Administration | ||
| <30>1 2026-06-28T19:39:07.123+01:00 host-example qulogd: 19683 - [QuLog@Access mac="00-00-5E-00-53-23" ip="192.0.2.11" user="bob.smith" source="example-source" computer="---" application="---" action="512" action_result="1" service="1024" extra_data="" client_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5c" client_app="Web Desktop" client_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 Edg/149.0.0.0"] Administration | ||
| <30>1 2026-06-28T19:39:08.000+01:00 host-example qulogd: 19683 - [QuLog@Access mac="00-00-5E-00-53-23" ip="192.0.2.12" user="carol.white" source="example-source" computer="---" application="File Station" action="512" action_result="0" service="1024" extra_data="" client_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5e" client_app="Web Desktop" client_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 Edg/149.0.0.0"] Administration |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/qnap_nas/data_stream/log/_dev/test/pipeline/test-access.log:16
All three new RFC 5424 fixtures use action="512" and computer="---", so the qnap.nas.action fallback and the source.domain rename are never exercised; add a fixture covering both.
Details
Two newly added branches have no test coverage:
- The
elsebranch ofscript_rfc5424_action_mapping(default.yml lines 154-158), which lazily buildsctx.qnap/ctx.qnap.nasand writes the raw code toqnap.nas.action. This is the most fragile part of the new script -- it constructs maps by hand -- and theqnap.nas.actionfield it populates is declared infields.ymland documented in the README, yet no fixture produces it. rename_tmp_sd_computer_to_source_domain, whoseifrequirescomputer != '---'. All three fixtures usecomputer="---", sosource.domain(and therelated.hostsappend that depends on it) is never populated from the structured data.
Recommendation:
Add one fixture line that carries a non-512 action code and a real computer name, then regenerate the expected file:
<30>1 2026-06-28T19:39:09.000+01:00 host-example qulogd: 19683 - [QuLog@Access mac="00-00-5E-00-53-24" ip="192.0.2.13" user="dave.brown" source="example-source" computer="pc-test-0002" application="File Station" action="768" action_result="0" service="1024" extra_data="" client_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5f" client_app="Web Desktop" client_agent="Mozilla/5.0"] Administration
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| changes: | ||
| - description: Add support for RFC 5424 syslog format emitted by QTS 5.x devices with QuLog@Access structured-data events. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/1 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/qnap_nas/changelog.yml:6
Changelog links a different PR number
Details
This changelog entry's link: points at pull/1, but it was added in PR #20734. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point each added changelog entry's link at this PR:
link: https://github.com/elastic/integrations/pull/20734🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
Review summaryIssues found across the latest commits efd7d9f — 1 high, 4 medium, 1 low
Package-level:
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
|
2 participants
Executive summary
This fix adds support for RFC 5424 syslog format emitted by QNAP QTS 5.x devices that send structured-data events using the QuLog@Access SD-ID. The ingest pipeline gains a new grok pattern to match the RFC 5424 header (version 1, ISO8601 timestamp, structured data block), a kv processor to parse the key-value pairs within the structured data, and a Painless script to map numeric action codes (e.g. 512) and action_result values to ECS event.action strings (login-success, login-fail). New fields are declared in fields.yml and ecs.yml, the README is updated, and three representative pipeline test fixtures are added.
Proposed commit message
Root cause
The first grok processor (
grok_event_original_cad2ef7a) uses only a%{SYSLOGTIMESTAMP}(RFC 3164 BSD timestamp) pattern; QTS 5.x devices emit RFC 5424 syslog with a version byte1and an ISO 8601 timestamp before the hostname, causing the pattern to fail immediately after the optional PRI field. No alternative pattern exists to match the RFC 5424 envelope or parse the[QuLog@Access key="val" ...]structured-data block.Approach
Add a second grok pattern to
grok_event_original_cad2ef7athat matches RFC 5424 format (<PRI>1 ISO8601-TIMESTAMP HOSTNAME APPNAME: PROCID - [QuLog@Access KV-PAIRS] MSG) alongside the existing RFC 3164 pattern. Extract structured-data key-value pairs via akvprocessor into_tmp.sd.*, then rename them to ECS/custom fields. Map numeric action codes (512 → login-success/login-fail based on action_result=0) toevent.actionvia a Painless script so the existing ECS categorization script sets correctevent.outcome,event.category, andevent.type. Guard RFC-3164-only processors (grok__tmp_message_c75b80dc, both date processors) withctx._tmp?.sd_params == nullorctx._tmp?.timestamp != nullconditions to prevent failure on RFC 5424 events.Implementation
packages/qnap_nas/data_stream/log/elasticsearch/ingest_pipeline/default.yml, add a second pattern togrok_event_original_cad2ef7a:'^(%{ECS_SYSLOG_PRI})?1 %{TIMESTAMP_ISO8601:_tmp.timestamp_iso8601} %{NAS} %{PROG:process.name}: %{POSINT:process.pid:int} - \[%{SD_ID} %{DATA:_tmp.sd_params}\] %{GREEDYDATA:_tmp.message}'. Add pattern definitionsTIMESTAMP_ISO8601: '%{YEAR}-%{MONTHNUM}-%{MONTHDAY}T%{HOUR}:%{MINUTE}:%{SECOND}(?:[.,]\d+)?(?:%{ISO8601_TIMEZONE:event.timezone})?',ISO8601_TIMEZONE: '(?:Z|[+-]%{HOUR}:?%{MINUTE})', andSD_ID: '%{WORD}@%{WORD}'.date__tmp_timestamp_to_@timestamp_e440143cifcondition fromctx.event?.timezone != nulltoctx.event?.timezone != null && ctx._tmp?.timestamp != null. Updatedate__tmp_timestamp_to_@timestamp_baf3310eiffromctx.event?.timezone == nulltoctx.event?.timezone == null && ctx._tmp?.timestamp != null. Insert a new date processor (tagdate_rfc5424_timestamp_to_@timestamp) withfield: _tmp.timestamp_iso8601,target_field: '@timestamp',formats: [ISO8601],if: ctx._tmp?.timestamp_iso8601 != nullimmediately after the two existing date processors and beforeset_event_created_e3f09e3b.kvprocessor block (tagkv_tmp_sd_params_to_tmp_sd) withfield: _tmp.sd_params,field_split: '" ',value_split: '="',trim_value: '"',target_field: _tmp.sd,ignore_missing: true,if: ctx._tmp?.sd_params != nullimmediately after theset_event_created_e3f09e3bprocessor.renameprocessors (each withignore_missing: true, guarded byctx._tmp?.sd_params != null) to map:_tmp.sd.ip→source.address;_tmp.sd.user→user.name;_tmp.sd.computer→source.domain;_tmp.sd.application→qnap.nas.application;_tmp.sd.client_agent→user_agent.original;_tmp.sd.client_app→qnap.nas.client_app;_tmp.sd.client_id→qnap.nas.client_id;_tmp.sd.mac→qnap.nas.mac;_tmp.sd.service→qnap.nas.service.script_rfc5424_action_mapping,if: ctx._tmp?.sd?.action != null) that maps action code'512'toevent.action = 'login-success'when_tmp.sd.action_result == '0'and to'login-fail'otherwise. For any unmapped action code, write the raw numeric value toqnap.nas.actionand leaveevent.actionunset so the existing ECS categorization script returns without overwriting. Explicitly handles fallback: all unrecognised codes produce noevent.action(the ECS script'sparams.get(ctx.event.action) == nullguard then returns early).if: ctx._tmp?.sd_params == nullto thegrok__tmp_message_c75b80dcprocessor so it is skipped for RFC 5424 events whose_tmp.messageis just the MSG word (e.g.Administration) not aUsers: …formatted string.packages/qnap_nas/data_stream/log/fields/fields.yml, add five new fields underqnap.nas:client_app(keyword),client_id(keyword),mac(keyword),action(keyword, for raw numeric code),service(keyword, for raw numeric service code), andsource(keyword, for the structured-datasourceattribute).packages/qnap_nas/data_stream/log/fields/ecs.yml, add- external: ecs\n name: user_agent.originalto expose the mapped client agent field.packages/qnap_nas/data_stream/log/_dev/test/pipeline/test-access.log, append the sanitized RFC 5424 event:<30>1 2026-06-28T19:39:06.466+01:00 host-example qulogd: 19683 - [QuLog@Access mac="00-00-5E-00-53-23" ip="192.0.2.10" user="alice.johnson" source="example-source" computer="---" application="---" action="512" action_result="0" service="1024" extra_data="" client_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5b" client_app="Web Desktop" client_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 Edg/149.0.0.0"] Administration. Add corresponding expected-output object totest-access.log-expected.jsonverifying@timestamp,user.name,source.ip,event.action=login-success,event.outcome=success,user_agent.original,qnap.nas.client_app,related.ip,related.user, and absence of---values.packages/qnap_nas/manifest.yml, bump version from1.25.3to1.26.0. Inpackages/qnap_nas/changelog.yml, prepend a new entry:version: '1.26.0',description: Add support for RFC 5424 syslog format emitted by QTS 5.x devices (QuLog@Access structured-data events),type: enhancement.Pipeline changes
<PRI>1 ISO8601 HOSTNAME PROG: PID - [QuLog@Access SD-PARAMS] MSG; add pattern definitions TIMESTAMP_ISO8601, ISO8601_TIMEZONE (capturing into event.timezone), SD_ID&& ctx._tmp?.timestamp != nullguard to prevent failure when RFC 5424 path leaves _tmp.timestamp unsetif: ctx._tmp?.sd_params == nullso RFC 5424 events skip the Users:/Connection-type parsing branchField / mapping changes
Sanitized error message
Processor 'grok' with tag 'grok_event_original_cad2ef7a' in pipeline 'logs-qnap_nas.log-default' failed with message '[on_failure_message]'Sanitized log (
event_sanitizedexcerpt)<30>1 2026-06-28T19:39:06.466+01:00 host-example qulogd: 19683 - [QuLog@Access mac="00-00-5E-00-53-23" ip="192.0.2.10" user="alice.johnson" source="example-source" computer="---" application="---" action="512" action_result="0" service="1024" extra_data="" client_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5b" client_app="Web Desktop" client_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 Edg/149.0.0.0"] AdministrationReviewer concerns
source.domainrename from_tmp.sd.computerfires only whencomputer != '---', but the condition checksctx._tmp?.sd?.computer != '---'before the kv processor has necessarily confirmed the field is non-null; this is safe becauseignore_missing: trueis set, but reviewers should confirm kv always populates_tmp.sd.computerbefore the rename runs.512; all other numeric action codes fall through toqnap.nas.actionas a raw string with no ECS mapping, which may surprise users who expect richer categorisation.event.typeis set to["start"]foraction_result=0(success) and["info"]for failure — usinginfofor an authentication failure is unusual; ECS recommends["start"]for successful auth and["end"]or["info"]for failures, so["info"]is defensible but worth a reviewer double-check.event.categoryandevent.kindfields are present in the expected output but the pipeline diff does not show explicitsetprocessors for them for the RFC 5424 path; reviewers should verify these are populated by a downstream processor already present in the pipeline (e.g. the existing event categorisation section not shown in the diff).changelog.ymlsetslink: https://github.com/elastic/integrations/pull/1which is a placeholder and must be updated to the real PR URL before merge.Self-review findings
Self-review invoked: no (1 cycle)
Final validation passed: yes
Risk and classification
Links
b57b0e02f567c44d