Skip to content

Managed ilasm: comprehensive parity fixes with native ilasm - #131507

Open
jkoritzinsky wants to merge 69 commits into
mainfrom
ilasm-fixups
Open

Managed ilasm: comprehensive parity fixes with native ilasm#131507
jkoritzinsky wants to merge 69 commits into
mainfrom
ilasm-fixups

Conversation

@jkoritzinsky

Copy link
Copy Markdown
Member

Managed ilasm parity fixes

This PR brings the managed ilasm (src/tools/ilasm) to near-parity with the native ilasm across 2,868 ilproj test files in the repo. Starting from ~2,128 managed ilasm failures, this work reduces real (non-cosmetic) differences to zero actionable items.

Summary of changes (53 commits, 23 files, +12,035 / -6,617 lines)

Parser & Grammar Fixes

  • Fixed ANTLR grammar for .language, .line (QSTRING support), value/instance keywords in identifiers, multiple ddItem without braces, &label references, atOpt with integer, empty pinvokeimpl(), security attribute blob type ordering, function pointer syntax
  • Added ANTLR parser error listener to DocumentCompiler for strict diagnostics
  • Fixed preprocessor #define macro expansion to re-lex multi-token values
  • Regenerated ANTLR parser after all grammar changes

Metadata Emission Fixes

  • MemberRef → MethodDef/FieldDef resolution: Resolve local method/field references to definition tokens (matching native ilasm behavior)
  • TypeRef → TypeDef resolution: Lazy TypeRef tracking with PseudoHandle for signature encoding. TypeRefs whose resolution scope matches the current assembly resolve to local TypeDef handles. Signature rewriter remaps PseudoHandle-based coded indices in all signature blobs and IL instruction tokens are backpatched.
  • Field attribute flags: HasDefault, HasFieldMarshal, HasFieldRVA, PinvokeImpl
  • Method attributes: Auto-instance calling convention, auto-RTSpecialName|SpecialName for .ctor/.cctor, PinvokeImpl flag
  • Param emission: Always emit Param rows for explicit parameters, auto-generate A_N names for unnamed parameters
  • ClassLayout: Emit for explicit layout types even without .pack/.size
  • GenericParamConstraint: Sort by Owner handle during emission
  • Custom attributes: Fix type-level and top-level handlers, blob prolog (WriteUInt16 not WriteInt32), module vs assembly ownership
  • Stackreserve: Pass directive value to PEHeaderBuilder
  • Locals: Build LOCAL_SIG standalone signature from parsed .locals declarations
  • Corelib TypeRef redirect: Normalize different corelib assembly names
  • Primitive type codes: Emit correct codes for well-known corelib types (System.StringString, etc.)
  • Leading-dot type names: Position 0 dot is part of the name, not a namespace separator
  • DebuggableAttribute: Deferred to BuildImage() so assembly refs are available
  • Vararg signatures: Fix parameter count (exclude sentinels) and parent resolution

Signature Rewriter (new)

  • Rewrites all signature blobs (field, method, standalone, property, TypeSpec, MethodSpec) from PseudoHandle-based TypeRef coded indices to resolved real handles
  • Fixed GetModifiedType to write modifier as raw coded index (not full type encoding)
  • Fixed GetArrayType to emit ELEMENT_TYPE_ARRAY prefix byte

IL Body Fixes

Test Coverage

  • 236 unit tests (up from ~25 at start), covering all major fix categories
  • Tests use CompileAndGetReader to verify metadata byte-level correctness

Comparison results (2,868 ilproj files)

Metric Before After
Matching (byte-identical ildasm) 0 9
Module-name-only diffs 1,862
Cascading-only diffs 676
Real primary diffs 2,128+ failures 420 (all cosmetic)
Managed ilasm failures 2,128 4 (2 TLS, 2 65K+ generics)

The remaining 420 files with "real" diffs are all non-actionable: custom attribute metadata ordering (246), PE header line ordering (44), field RVA section placement (21), assembly metadata cosmetic (19), corelib ref name cascading (12), ildasm typedef presentation (4), and override TypeRef formatting (3).

Note

This PR was authored with the assistance of GitHub Copilot.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 5 pipeline(s).
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR substantially expands managed ilasm parser/emitter parity with native ilasm, including multi-document compilation, grammar fixes, metadata/token resolution adjustments, and a large suite of new unit tests validating metadata/IL byte correctness.

Changes:

  • Extend the compiler pipeline (preprocessor + parser) to support multi-document input and stricter parser diagnostics.
  • Improve preprocessor macro expansion by re-lexing macro values into correct token streams.
  • Add broad managed ilasm regression/unit test coverage across directives, signatures, metadata emission, and IL encoding.

Reviewed changes

Copilot reviewed 47 out of 51 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs New tests for .vtfixup emission and .sdata layout validation.
src/tools/ilasm/tests/ILAssembler.Tests/TypeSignatureTests.cs New test for multidimensional array bounds parsing.
src/tools/ilasm/tests/ILAssembler.Tests/TypeReferenceTests.cs New tests for .this/.base/.nester diagnostics and TypeRef→TypeDef resolution/backpatching behavior.
src/tools/ilasm/tests/ILAssembler.Tests/TypedefTests.cs New tests for typedef alias resolution and missing-alias diagnostics.
src/tools/ilasm/tests/ILAssembler.Tests/SyntaxTests.cs New tests for string escape handling, numeric literal diagnostics, and parser error reporting.
src/tools/ilasm/tests/ILAssembler.Tests/SourceDirectiveTests.cs New tests for .language/.line directives and embedded PDB generation behavior.
src/tools/ilasm/tests/ILAssembler.Tests/SecurityTests.cs New tests for security directive diagnostics and DeclSecurity emission.
src/tools/ilasm/tests/ILAssembler.Tests/PropertyTests.cs New tests for property emission, initOpt constants, and property-owned custom attributes.
src/tools/ilasm/tests/ILAssembler.Tests/PreprocessedTokenSourceTests.cs Update preprocessor construction and add macro re-lexing tests.
src/tools/ilasm/tests/ILAssembler.Tests/ParameterTests.cs New tests for param constants, param row emission, signature rewrite correctness, and ldarg-by-name regression.
src/tools/ilasm/tests/ILAssembler.Tests/ModuleTests.cs New tests for module naming defaults and module-level field handling.
src/tools/ilasm/tests/ILAssembler.Tests/MethodTests.cs New tests for vararg, overrides/MethodImpl, calling convention inference, and modopt preservation.
src/tools/ilasm/tests/ILAssembler.Tests/MemberReferenceTests.cs Placeholder test file added for future coverage.
src/tools/ilasm/tests/ILAssembler.Tests/LocalTests.cs New tests for locals/name lookup and MemberRef→Def resolution behavior.
src/tools/ilasm/tests/ILAssembler.Tests/InteropTests.cs New tests for pinvoke parsing/diagnostics and PinvokeImpl emission.
src/tools/ilasm/tests/ILAssembler.Tests/InstructionTests.cs New tests for label fixups, prefix/opcode parsing, and FieldRVA emission.
src/tools/ilasm/tests/ILAssembler.Tests/GenericTests.cs New tests for generic parameter diagnostics and constraint emission correctness.
src/tools/ilasm/tests/ILAssembler.Tests/FunctionPointerTests.cs New tests for function pointer signature encoding.
src/tools/ilasm/tests/ILAssembler.Tests/ExportedTypeTests.cs New tests for exported type diagnostics, forwarding, and .export behavior.
src/tools/ilasm/tests/ILAssembler.Tests/ExceptionHandlingTests.cs New tests for EH blocks and label/offset handling.
src/tools/ilasm/tests/ILAssembler.Tests/EventTests.cs New test ensuring event-owned custom attributes are emitted.
src/tools/ilasm/tests/ILAssembler.Tests/DocumentCompilerTestHelpers.cs New helper utilities for compilation, diagnostics, and token operand decoding.
src/tools/ilasm/tests/ILAssembler.Tests/DataTests.cs New test for invalid metadata token diagnostics.
src/tools/ilasm/tests/ILAssembler.Tests/CustomAttributeTests.cs New tests for custom attribute blob parsing and emission correctness.
src/tools/ilasm/tests/ILAssembler.Tests/ILAssembler.Tests.csproj Link in shared ILOpcode metadata for tests that decode IL bodies.
src/tools/ilasm/src/ILAssembler/PreprocessedTokenSource.cs Add macro value re-lexing, exposed defined-variable state, and constructor updates.
src/tools/ilasm/src/ILAssembler/Options.cs Add OutputFileName option to drive default module naming.
src/tools/ilasm/src/ILAssembler/NameHelpers.cs Treat leading-dot type names as name (not namespace separator).
src/tools/ilasm/src/ILAssembler/NamedElementList.cs Allow duplicate names with first-wins lookup behavior.
src/tools/ilasm/src/ILAssembler/gen/ilasm-generator.csproj Improve ANTLR generated path rewrite robustness.
src/tools/ilasm/src/ILAssembler/gen/CILVisitor.cs Regenerated visitor surface to match grammar updates.
src/tools/ilasm/src/ILAssembler/gen/CILLexer.tokens Regenerated token map after grammar changes.
src/tools/ilasm/src/ILAssembler/gen/CILBaseVisitor.cs Regenerated base visitor after grammar changes.
src/tools/ilasm/src/ILAssembler/gen/CIL.g4 Major grammar adjustments (directives, tokens, keywords, vararg, byte blobs, etc.).
src/tools/ilasm/src/ILAssembler/DocumentCompiler.cs Add multi-document compile overload; add strict parser error listener; persist preprocessor defines across documents.
src/tools/ilasm/src/ILAssembler/Diagnostic.cs Add diagnostic ID/template for excessive generic parameter counts.
src/tools/ilasm/src/ILAssembler/BlobBuilderExtensions.cs Emit pseudo-handle coded indices for TypeRef entities in signatures.
src/tools/ilasm/src/ilasm/Program.cs Read/compile multiple input files; normalize native-style args; plumb OutputFileName into options.
src/tools/ilasm/src/ilasm/IlasmRootCommand.cs Add native-style option aliases (e.g., -OUTPUT, -DLL, etc.).
src/tools/ilasm/KNOWN-ISSUES.md Document current known limitation (TLS RVA statics).
Comments suppressed due to low confidence (3)

src/tools/ilasm/src/ILAssembler/NamedElementList.cs:85

  • NamedElementList now allows duplicate names (first-wins) via TryAdd, but Remove/RemoveAt still unconditionally remove the name mapping. If a non-mapped duplicate is removed, this drops the lookup entry for the still-existing first element; if the mapped element is removed, name lookup should fall back to the next remaining element with the same name. This can break name-based lookups after removals.
        public bool Remove(T item)
        {
            bool result = _elements.Remove(item);
            if (result)
            {
                _elementsByName.Remove(item.Name);
            }
            return result;
        }

        public void RemoveAt(int index)
        {
            T element = _elements[index];
            _elements.RemoveAt(index);
            _elementsByName.Remove(element.Name);
        }

src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs:119

  • Same endianness issue as above: these values are read from PE bytes (little-endian). Use BinaryPrimitives.Read*LittleEndian rather than BitConverter so the test behaves correctly on big-endian targets.
    src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs:168
  • Same endianness issue: slot count/RVA/tokens are read from PE bytes (little-endian). Using BitConverter makes the test endian-dependent; prefer BinaryPrimitives.Read*LittleEndian here too.

Comment thread src/tools/ilasm/src/ILAssembler/Options.cs
Comment thread src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Copilot AI review requested due to automatic review settings July 29, 2026 00:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@dotnet-policy-service dotnet-policy-service Bot added the linkable-framework Issues associated with delivering a linker friendly framework label Jul 29, 2026
Copilot AI review requested due to automatic review settings July 29, 2026 00:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 47 out of 51 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs:64

  • These reads interpret PE/CLR structures which are defined as little-endian. Using BitConverter makes the test endianness-dependent. Please use BinaryPrimitives.Read*LittleEndian here (and similarly for the other BitConverter reads later in this file).
    src/tools/ilasm/tests/ILAssembler.Tests/InstructionTests.cs:88
  • This test is reading a little-endian PE value. BitConverter makes the test endianness-dependent; use BinaryPrimitives.ReadInt32LittleEndian instead (and update the other BitConverter reads in this file as well).
    src/tools/ilasm/src/ILAssembler/Options.cs:139
  • The XML comment says this is "filename only, no directory", but the property doesn't enforce that. Either normalize to Path.GetFileName in the setter / consumer, or adjust the comment to avoid stating an invariant that isn't guaranteed.
        /// <summary>
        /// Output file name (filename only, no directory). Used as default module name when no .module directive is present.
        /// </summary>
        public string? OutputFileName { get; set; }

Copilot AI review requested due to automatic review settings July 29, 2026 00:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 47 out of 51 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/tools/ilasm/src/ILAssembler/Options.cs:139

  • Options.OutputFileName is new public API surface. With no linked api-approved issue in the PR metadata, this needs either API approval linkage or reducing the visibility (and adjusting tests accordingly) if it’s meant to be internal implementation detail.
        /// <summary>
        /// Output file name (filename only, no directory). Used as default module name when no .module directive is present.
        /// </summary>
        public string? OutputFileName { get; set; }

src/tools/ilasm/src/ILAssembler/NamedElementList.cs:48

  • Add/Insert now use TryAdd so the first duplicate name wins, but the rest of the type still assumes a 1:1 mapping between name and element (e.g., indexer setter overwrites, and Remove/RemoveAt always remove the name key). With duplicates present, the name map can become stale or lose the winning entry. Consider making the name-index semantics consistent across mutation operations (e.g., only remove the key when it points to the removed element, and re-point it to the next remaining element with the same name).
        public void Add(T item)
        {
            _elements.Add(item);
            // Use TryAdd to keep the first element for name lookup when duplicate names exist.
            // This matches native ilasm behavior where duplicate generic parameter names are allowed
            // and the first definition wins for name-based lookup.
            _elementsByName.TryAdd(item.Name, item);
        }

Comment thread src/tools/ilasm/src/ILAssembler/DocumentCompiler.cs
jkoritzinsky and others added 24 commits August 14, 2026 15:33
…rrectly ordered handles and matching emit order to assigned handles.
Add lazy TypeRef tracking with PseudoHandle for signature encoding.
During emission, TypeRefs whose resolution scope matches the current
assembly are resolved to local TypeDef handles. A signature rewriter
remaps PseudoHandle-based coded indices in all signature blobs before
they are written to the MetadataBuilder.

This eliminates TypeRef/MemberRef entries for self-referencing types,
matching native ilasm behavior where [self-assembly]Type references
resolve to TypeDef/MethodDef/FieldDef tokens.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix SignatureRewriter.GetModifiedType to write the modifier as a raw
TypeDefOrRefOrSpec coded index instead of a full type encoding (which
incorrectly added a CLASS/VALUETYPE prefix byte).

Emit ClassLayout table rows for types with explicit layout even when
.pack and .size are not specified, matching native ilasm behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TypeReferenceEntity now records blob locations for IL instruction
tokens (instr_type, instr_tok, instr_field mdtoken) and backpatches
them with the resolved handle when TypeRef→TypeDef resolution runs.
This fixes INVALID TOKEN errors in IL instructions like unbox.any,
castclass, and ldtoken that reference self-assembly types.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The InstructionEncoder's CodeBuilder used a default BlobBuilder with
256-byte chunks. ControlFlowBuilder label patching across chunk
boundaries caused IL byte corruption (wrong opcode/operand values).
Increase the initial capacity to 4096 bytes to avoid multi-chunk
issues for typical method bodies.

Also adds a regression test that verifies ldarg.s parameter name
resolution emits correct indices across the 512-byte boundary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The SignatureRewriter.GetArrayType method was missing the
ELEMENT_TYPE_ARRAY (0x14) prefix byte, causing multi-dimensional
array types to lose their array encoding during signature rewriting.
This affected ~120 files where parameters/fields with types like
int32[0...,0...] were emitted as plain int32.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The managed ilasm parsed .locals declarations and collected local
variable types in AllLocals, but never built a StandaloneSignature
from them. This caused methods with .locals to emit method bodies
without a locals signature, so ildasm did not show the .locals
directive in the disassembled output.

Build the LOCAL_SIG standalone signature after processing all method
declarations and connect it to the method's LocalsSignature property.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Exception handler catch type tokens were evaluated at parse time via
TypeReferenceEntity.Handle, which returned the PseudoHandle before
TypeRef→TypeDef resolution. The ControlFlowBuilder stored this stale
handle value and wrote it to the exception handler table during
serialization.

Fix by storing deferred ExceptionRegion records on the method entity
during parsing, then registering them with ControlFlowBuilder during
WriteContentTo after TypeRef resolution has set real handles.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…st, and NormalizeNativeArgs

- Fix ANTLR generated file headers to use relative grammar path
- Fix ilasm-generator.csproj to handle forward-slash paths from ANTLR
- Fix CIL.g4 comment about HEXBYTE to match actual grammar
- Fix ParserErrorListener SourceSpan length to be inclusive (+ 1)
- Add comments to NamedElementList TryAdd explaining first-wins intent
- Remove TODO prefixes from Options.cs remarks
- Replace NormalizeNativeArgs dictionary with pattern match (per @am11)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… token locations

- Fix method body fallback to emit proper header via AddMethodBody
  instead of raw WriteContentTo (per @copilot)
- Validate VisitHexbyte text is valid hex before parsing, gracefully
  handle non-hex ID tokens and values > 0xFF (per @copilot)
- Clone macro expansion tokens to inherit the original macro
  identifier's source location for stable diagnostics (per @copilot)
- Remove duplicate Replace in ilasm-generator.csproj (per @am11)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
VisitPropDecl and VisitEventDecl checked ChildCount != 2 to filter
non-accessor declarations, but this also filtered out customAttrDecl
entries (which have 1 child). Custom attributes inside property and
event blocks were silently dropped instead of being emitted.

Handle customAttrDecl in the property/event processing loop by
visiting the custom attribute and setting its Owner to the enclosing
property or event entity.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-RESOURCES error

- Fix MethodSpec rewriter to use correct signature header byte (0x0A)
  instead of SignatureAttributes.Generic (0x10) per ECMA-335.
- Fix array shape encoding to stop counting sizes/lower bounds at the
  first null dimension (contiguous from start), not the last non-null.
- Restore -o and -O short aliases for --output and --optimize to avoid
  breaking existing managed ilasm users/scripts.
- Throw ArgumentException on -RESOURCES= instead of silently dropping.
- Fix ilasm-generator.csproj: compute absolute grammar path in a
  separate MSBuild property to avoid nested single-quote issues.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The HasFieldMarshal flag was only set when MarshallingDescriptor.Count > 0,
but the FieldMarshal row was emitted whenever MarshallingDescriptor was
non-null (even if empty). Align both to use the same Count > 0 predicate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Addresses PR review comment r3610337318. Previously, a nested TypeRef
whose enclosing type resolved to a local TypeDef but which itself was
not found locally was emitted with the enclosing type's TypeDefinition
handle as its ResolutionScope, which is an invalid coded index and threw
ArgumentException at emission.

Record every TypeRef as a row in PseudoHandle (creation) order, matching
native ilasm which preserves all TypeRef rows, then resolve locals to
their TypeDef handles. Because PseudoHandle is the gapless 1-based index
into _typeReferences, the emitted row handle equals the PseudoHandle,
keeping rows aligned with the handles used during parsing/signature
encoding. Nested TypeRef resolution scopes now use the enclosing
TypeRef's PseudoHandle (a valid TypeReference coded index) instead of the
TypeDef it may have resolved to.

Add tests validating that resolved TypeRefs still emit rows in order and
that the nested-enclosing-local/nested-missing scenario emits a valid
ResolutionScope. Update the token-backpatching and self-assembly tests to
decode and validate the method IL operand tokens (via the shared ILOpcode
tables) instead of asserting TypeRef-row absence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab70f26b-7491-45bc-b4ce-6cb13c509fc2
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81eb86ba-1424-4d1e-8fed-ca5be7b511cb
… tests

Co-authored-by: jkoritzinsky <1571408+jkoritzinsky@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4307156b-37a7-495d-9af6-fe9174c05b0b
Copilot AI review requested due to automatic review settings August 14, 2026 22:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 51 out of 55 changed files in this pull request and generated 1 comment.

Comment on lines +4 to +18
using System;
using System.Buffers.Binary;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using System.Reflection;
using System.Reflection.Metadata;
using System.Reflection.Metadata.Ecma335;
using System.Reflection.PortableExecutable;
using System.Text;
using System.Threading.Tasks;
using Internal.IL;
using Xunit;
using DocumentCompilerTestHelpers = ILAssembler.Tests.DocumentCompilerTestHelpers;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-ILTools-coreclr linkable-framework Issues associated with delivering a linker friendly framework

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants