Managed ilasm: comprehensive parity fixes with native ilasm - #131507
Managed ilasm: comprehensive parity fixes with native ilasm#131507jkoritzinsky wants to merge 69 commits into
Conversation
|
Azure Pipelines: Successfully started running 5 pipeline(s). 11 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
This PR substantially expands managed ilasm parser/emitter parity with native ilasm, including multi-document compilation, grammar fixes, metadata/token resolution adjustments, and a large suite of new unit tests validating metadata/IL byte correctness.
Changes:
- Extend the compiler pipeline (preprocessor + parser) to support multi-document input and stricter parser diagnostics.
- Improve preprocessor macro expansion by re-lexing macro values into correct token streams.
- Add broad managed
ilasmregression/unit test coverage across directives, signatures, metadata emission, and IL encoding.
Reviewed changes
Copilot reviewed 47 out of 51 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs | New tests for .vtfixup emission and .sdata layout validation. |
| src/tools/ilasm/tests/ILAssembler.Tests/TypeSignatureTests.cs | New test for multidimensional array bounds parsing. |
| src/tools/ilasm/tests/ILAssembler.Tests/TypeReferenceTests.cs | New tests for .this/.base/.nester diagnostics and TypeRef→TypeDef resolution/backpatching behavior. |
| src/tools/ilasm/tests/ILAssembler.Tests/TypedefTests.cs | New tests for typedef alias resolution and missing-alias diagnostics. |
| src/tools/ilasm/tests/ILAssembler.Tests/SyntaxTests.cs | New tests for string escape handling, numeric literal diagnostics, and parser error reporting. |
| src/tools/ilasm/tests/ILAssembler.Tests/SourceDirectiveTests.cs | New tests for .language/.line directives and embedded PDB generation behavior. |
| src/tools/ilasm/tests/ILAssembler.Tests/SecurityTests.cs | New tests for security directive diagnostics and DeclSecurity emission. |
| src/tools/ilasm/tests/ILAssembler.Tests/PropertyTests.cs | New tests for property emission, initOpt constants, and property-owned custom attributes. |
| src/tools/ilasm/tests/ILAssembler.Tests/PreprocessedTokenSourceTests.cs | Update preprocessor construction and add macro re-lexing tests. |
| src/tools/ilasm/tests/ILAssembler.Tests/ParameterTests.cs | New tests for param constants, param row emission, signature rewrite correctness, and ldarg-by-name regression. |
| src/tools/ilasm/tests/ILAssembler.Tests/ModuleTests.cs | New tests for module naming defaults and module-level field handling. |
| src/tools/ilasm/tests/ILAssembler.Tests/MethodTests.cs | New tests for vararg, overrides/MethodImpl, calling convention inference, and modopt preservation. |
| src/tools/ilasm/tests/ILAssembler.Tests/MemberReferenceTests.cs | Placeholder test file added for future coverage. |
| src/tools/ilasm/tests/ILAssembler.Tests/LocalTests.cs | New tests for locals/name lookup and MemberRef→Def resolution behavior. |
| src/tools/ilasm/tests/ILAssembler.Tests/InteropTests.cs | New tests for pinvoke parsing/diagnostics and PinvokeImpl emission. |
| src/tools/ilasm/tests/ILAssembler.Tests/InstructionTests.cs | New tests for label fixups, prefix/opcode parsing, and FieldRVA emission. |
| src/tools/ilasm/tests/ILAssembler.Tests/GenericTests.cs | New tests for generic parameter diagnostics and constraint emission correctness. |
| src/tools/ilasm/tests/ILAssembler.Tests/FunctionPointerTests.cs | New tests for function pointer signature encoding. |
| src/tools/ilasm/tests/ILAssembler.Tests/ExportedTypeTests.cs | New tests for exported type diagnostics, forwarding, and .export behavior. |
| src/tools/ilasm/tests/ILAssembler.Tests/ExceptionHandlingTests.cs | New tests for EH blocks and label/offset handling. |
| src/tools/ilasm/tests/ILAssembler.Tests/EventTests.cs | New test ensuring event-owned custom attributes are emitted. |
| src/tools/ilasm/tests/ILAssembler.Tests/DocumentCompilerTestHelpers.cs | New helper utilities for compilation, diagnostics, and token operand decoding. |
| src/tools/ilasm/tests/ILAssembler.Tests/DataTests.cs | New test for invalid metadata token diagnostics. |
| src/tools/ilasm/tests/ILAssembler.Tests/CustomAttributeTests.cs | New tests for custom attribute blob parsing and emission correctness. |
| src/tools/ilasm/tests/ILAssembler.Tests/ILAssembler.Tests.csproj | Link in shared ILOpcode metadata for tests that decode IL bodies. |
| src/tools/ilasm/src/ILAssembler/PreprocessedTokenSource.cs | Add macro value re-lexing, exposed defined-variable state, and constructor updates. |
| src/tools/ilasm/src/ILAssembler/Options.cs | Add OutputFileName option to drive default module naming. |
| src/tools/ilasm/src/ILAssembler/NameHelpers.cs | Treat leading-dot type names as name (not namespace separator). |
| src/tools/ilasm/src/ILAssembler/NamedElementList.cs | Allow duplicate names with first-wins lookup behavior. |
| src/tools/ilasm/src/ILAssembler/gen/ilasm-generator.csproj | Improve ANTLR generated path rewrite robustness. |
| src/tools/ilasm/src/ILAssembler/gen/CILVisitor.cs | Regenerated visitor surface to match grammar updates. |
| src/tools/ilasm/src/ILAssembler/gen/CILLexer.tokens | Regenerated token map after grammar changes. |
| src/tools/ilasm/src/ILAssembler/gen/CILBaseVisitor.cs | Regenerated base visitor after grammar changes. |
| src/tools/ilasm/src/ILAssembler/gen/CIL.g4 | Major grammar adjustments (directives, tokens, keywords, vararg, byte blobs, etc.). |
| src/tools/ilasm/src/ILAssembler/DocumentCompiler.cs | Add multi-document compile overload; add strict parser error listener; persist preprocessor defines across documents. |
| src/tools/ilasm/src/ILAssembler/Diagnostic.cs | Add diagnostic ID/template for excessive generic parameter counts. |
| src/tools/ilasm/src/ILAssembler/BlobBuilderExtensions.cs | Emit pseudo-handle coded indices for TypeRef entities in signatures. |
| src/tools/ilasm/src/ilasm/Program.cs | Read/compile multiple input files; normalize native-style args; plumb OutputFileName into options. |
| src/tools/ilasm/src/ilasm/IlasmRootCommand.cs | Add native-style option aliases (e.g., -OUTPUT, -DLL, etc.). |
| src/tools/ilasm/KNOWN-ISSUES.md | Document current known limitation (TLS RVA statics). |
Comments suppressed due to low confidence (3)
src/tools/ilasm/src/ILAssembler/NamedElementList.cs:85
- NamedElementList now allows duplicate names (first-wins) via TryAdd, but Remove/RemoveAt still unconditionally remove the name mapping. If a non-mapped duplicate is removed, this drops the lookup entry for the still-existing first element; if the mapped element is removed, name lookup should fall back to the next remaining element with the same name. This can break name-based lookups after removals.
public bool Remove(T item)
{
bool result = _elements.Remove(item);
if (result)
{
_elementsByName.Remove(item.Name);
}
return result;
}
public void RemoveAt(int index)
{
T element = _elements[index];
_elements.RemoveAt(index);
_elementsByName.Remove(element.Name);
}
src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs:119
- Same endianness issue as above: these values are read from PE bytes (little-endian). Use BinaryPrimitives.Read*LittleEndian rather than BitConverter so the test behaves correctly on big-endian targets.
src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs:168 - Same endianness issue: slot count/RVA/tokens are read from PE bytes (little-endian). Using BitConverter makes the test endian-dependent; prefer BinaryPrimitives.Read*LittleEndian here too.
b5c5659 to
8b6090c
Compare
f5189d0 to
b5c5659
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 47 out of 51 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs:64
- These reads interpret PE/CLR structures which are defined as little-endian. Using BitConverter makes the test endianness-dependent. Please use BinaryPrimitives.Read*LittleEndian here (and similarly for the other BitConverter reads later in this file).
src/tools/ilasm/tests/ILAssembler.Tests/InstructionTests.cs:88 - This test is reading a little-endian PE value. BitConverter makes the test endianness-dependent; use BinaryPrimitives.ReadInt32LittleEndian instead (and update the other BitConverter reads in this file as well).
src/tools/ilasm/src/ILAssembler/Options.cs:139 - The XML comment says this is "filename only, no directory", but the property doesn't enforce that. Either normalize to Path.GetFileName in the setter / consumer, or adjust the comment to avoid stating an invariant that isn't guaranteed.
/// <summary>
/// Output file name (filename only, no directory). Used as default module name when no .module directive is present.
/// </summary>
public string? OutputFileName { get; set; }
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 47 out of 51 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
src/tools/ilasm/src/ILAssembler/Options.cs:139
Options.OutputFileNameis new public API surface. With no linkedapi-approvedissue in the PR metadata, this needs either API approval linkage or reducing the visibility (and adjusting tests accordingly) if it’s meant to be internal implementation detail.
/// <summary>
/// Output file name (filename only, no directory). Used as default module name when no .module directive is present.
/// </summary>
public string? OutputFileName { get; set; }
src/tools/ilasm/src/ILAssembler/NamedElementList.cs:48
Add/Insertnow useTryAddso the first duplicate name wins, but the rest of the type still assumes a 1:1 mapping between name and element (e.g., indexer setter overwrites, and Remove/RemoveAt always remove the name key). With duplicates present, the name map can become stale or lose the winning entry. Consider making the name-index semantics consistent across mutation operations (e.g., only remove the key when it points to the removed element, and re-point it to the next remaining element with the same name).
public void Add(T item)
{
_elements.Add(item);
// Use TryAdd to keep the first element for name lookup when duplicate names exist.
// This matches native ilasm behavior where duplicate generic parameter names are allowed
// and the first definition wins for name-based lookup.
_elementsByName.TryAdd(item.Name, item);
}
…nd link to issue).
…rrectly ordered handles and matching emit order to assigned handles.
Add lazy TypeRef tracking with PseudoHandle for signature encoding. During emission, TypeRefs whose resolution scope matches the current assembly are resolved to local TypeDef handles. A signature rewriter remaps PseudoHandle-based coded indices in all signature blobs before they are written to the MetadataBuilder. This eliminates TypeRef/MemberRef entries for self-referencing types, matching native ilasm behavior where [self-assembly]Type references resolve to TypeDef/MethodDef/FieldDef tokens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix SignatureRewriter.GetModifiedType to write the modifier as a raw TypeDefOrRefOrSpec coded index instead of a full type encoding (which incorrectly added a CLASS/VALUETYPE prefix byte). Emit ClassLayout table rows for types with explicit layout even when .pack and .size are not specified, matching native ilasm behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TypeReferenceEntity now records blob locations for IL instruction tokens (instr_type, instr_tok, instr_field mdtoken) and backpatches them with the resolved handle when TypeRef→TypeDef resolution runs. This fixes INVALID TOKEN errors in IL instructions like unbox.any, castclass, and ldtoken that reference self-assembly types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The InstructionEncoder's CodeBuilder used a default BlobBuilder with 256-byte chunks. ControlFlowBuilder label patching across chunk boundaries caused IL byte corruption (wrong opcode/operand values). Increase the initial capacity to 4096 bytes to avoid multi-chunk issues for typical method bodies. Also adds a regression test that verifies ldarg.s parameter name resolution emits correct indices across the 512-byte boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The SignatureRewriter.GetArrayType method was missing the ELEMENT_TYPE_ARRAY (0x14) prefix byte, causing multi-dimensional array types to lose their array encoding during signature rewriting. This affected ~120 files where parameters/fields with types like int32[0...,0...] were emitted as plain int32. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The managed ilasm parsed .locals declarations and collected local variable types in AllLocals, but never built a StandaloneSignature from them. This caused methods with .locals to emit method bodies without a locals signature, so ildasm did not show the .locals directive in the disassembled output. Build the LOCAL_SIG standalone signature after processing all method declarations and connect it to the method's LocalsSignature property. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Exception handler catch type tokens were evaluated at parse time via TypeReferenceEntity.Handle, which returned the PseudoHandle before TypeRef→TypeDef resolution. The ControlFlowBuilder stored this stale handle value and wrote it to the exception handler table during serialization. Fix by storing deferred ExceptionRegion records on the method entity during parsing, then registering them with ControlFlowBuilder during WriteContentTo after TypeRef resolution has set real handles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…st, and NormalizeNativeArgs - Fix ANTLR generated file headers to use relative grammar path - Fix ilasm-generator.csproj to handle forward-slash paths from ANTLR - Fix CIL.g4 comment about HEXBYTE to match actual grammar - Fix ParserErrorListener SourceSpan length to be inclusive (+ 1) - Add comments to NamedElementList TryAdd explaining first-wins intent - Remove TODO prefixes from Options.cs remarks - Replace NormalizeNativeArgs dictionary with pattern match (per @am11) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… token locations - Fix method body fallback to emit proper header via AddMethodBody instead of raw WriteContentTo (per @copilot) - Validate VisitHexbyte text is valid hex before parsing, gracefully handle non-hex ID tokens and values > 0xFF (per @copilot) - Clone macro expansion tokens to inherit the original macro identifier's source location for stable diagnostics (per @copilot) - Remove duplicate Replace in ilasm-generator.csproj (per @am11) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
VisitPropDecl and VisitEventDecl checked ChildCount != 2 to filter non-accessor declarations, but this also filtered out customAttrDecl entries (which have 1 child). Custom attributes inside property and event blocks were silently dropped instead of being emitted. Handle customAttrDecl in the property/event processing loop by visiting the custom attribute and setting its Owner to the enclosing property or event entity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-RESOURCES error - Fix MethodSpec rewriter to use correct signature header byte (0x0A) instead of SignatureAttributes.Generic (0x10) per ECMA-335. - Fix array shape encoding to stop counting sizes/lower bounds at the first null dimension (contiguous from start), not the last non-null. - Restore -o and -O short aliases for --output and --optimize to avoid breaking existing managed ilasm users/scripts. - Throw ArgumentException on -RESOURCES= instead of silently dropping. - Fix ilasm-generator.csproj: compute absolute grammar path in a separate MSBuild property to avoid nested single-quote issues. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The HasFieldMarshal flag was only set when MarshallingDescriptor.Count > 0, but the FieldMarshal row was emitted whenever MarshallingDescriptor was non-null (even if empty). Align both to use the same Count > 0 predicate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Addresses PR review comment r3610337318. Previously, a nested TypeRef whose enclosing type resolved to a local TypeDef but which itself was not found locally was emitted with the enclosing type's TypeDefinition handle as its ResolutionScope, which is an invalid coded index and threw ArgumentException at emission. Record every TypeRef as a row in PseudoHandle (creation) order, matching native ilasm which preserves all TypeRef rows, then resolve locals to their TypeDef handles. Because PseudoHandle is the gapless 1-based index into _typeReferences, the emitted row handle equals the PseudoHandle, keeping rows aligned with the handles used during parsing/signature encoding. Nested TypeRef resolution scopes now use the enclosing TypeRef's PseudoHandle (a valid TypeReference coded index) instead of the TypeDef it may have resolved to. Add tests validating that resolved TypeRefs still emit rows in order and that the nested-enclosing-local/nested-missing scenario emits a valid ResolutionScope. Update the token-backpatching and self-assembly tests to decode and validate the method IL operand tokens (via the shared ILOpcode tables) instead of asserting TypeRef-row absence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ab70f26b-7491-45bc-b4ce-6cb13c509fc2
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81eb86ba-1424-4d1e-8fed-ca5be7b511cb
… tests Co-authored-by: jkoritzinsky <1571408+jkoritzinsky@users.noreply.github.com>
…cenario that are not currently implemented.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4307156b-37a7-495d-9af6-fe9174c05b0b
3055fa8 to
4e03302
Compare
| using System; | ||
| using System.Buffers.Binary; | ||
| using System.Collections.Generic; | ||
| using System.Collections.Immutable; | ||
| using System.Linq; | ||
| using System.Reflection; | ||
| using System.Reflection.Metadata; | ||
| using System.Reflection.Metadata.Ecma335; | ||
| using System.Reflection.PortableExecutable; | ||
| using System.Text; | ||
| using System.Threading.Tasks; | ||
| using Internal.IL; | ||
| using Xunit; | ||
| using DocumentCompilerTestHelpers = ILAssembler.Tests.DocumentCompilerTestHelpers; | ||
|
|
Managed ilasm parity fixes
This PR brings the managed ilasm (
src/tools/ilasm) to near-parity with the native ilasm across 2,868 ilproj test files in the repo. Starting from ~2,128 managed ilasm failures, this work reduces real (non-cosmetic) differences to zero actionable items.Summary of changes (53 commits, 23 files, +12,035 / -6,617 lines)
Parser & Grammar Fixes
.language,.line(QSTRING support),value/instancekeywords in identifiers, multipleddItemwithout braces,&labelreferences,atOptwith integer, emptypinvokeimpl(), security attribute blob type ordering, function pointer syntaxDocumentCompilerfor strict diagnostics#definemacro expansion to re-lex multi-token valuesMetadata Emission Fixes
A_Nnames for unnamed parameters.pack/.sizeWriteUInt16notWriteInt32), module vs assembly ownershipPEHeaderBuilderLOCAL_SIGstandalone signature from parsed.localsdeclarationsSystem.String→String, etc.)BuildImage()so assembly refs are availableSignature Rewriter (new)
GetModifiedTypeto write modifier as raw coded index (not full type encoding)GetArrayTypeto emitELEMENT_TYPE_ARRAYprefix byteIL Body Fixes
instr_type,instr_tok, andinstr_fieldmdtoken pathsBlobBuilderchunk boundary corruption (InstructionEncoder produces corrupted IL when mixing OpCode with direct CodeBuilder writes across BlobBuilder chunk boundaries #127261) — increasedInstructionEncoderBlobBuilderinitial capacity to 4096Test Coverage
CompileAndGetReaderto verify metadata byte-level correctnessComparison results (2,868 ilproj files)
The remaining 420 files with "real" diffs are all non-actionable: custom attribute metadata ordering (246), PE header line ordering (44), field RVA section placement (21), assembly metadata cosmetic (19), corelib ref name cascading (12), ildasm typedef presentation (4), and override TypeRef formatting (3).
Note
This PR was authored with the assistance of GitHub Copilot.