Implement canonical interface names in wit-component

[chenyan2002]

Follow-up to #2556.

• Add resolve.use_canonical_names in wit-parser to merge interfaces with the same canonical version.
• Propagate versionsuffix to wit-component, so that we can link two interfaces via the canonical version name.
• Add a config flag semver_compat = none|merge|canonical to wasm-tools component new to control the merging behavior. Deprecate the old merge_imports_based_on_semver flag (equivalent to semver_compat=merge).

This is a breaking change. Notably,

• import a:b/c@0.1.1; import a:b/c@0.1.2; would fail under the new feature flag, because of duplicate imports. Users can use the implements feature to update the wit file if needed. See the merge-import-versions test.
• When printing the wit file, the minor version may change depending on which package version gets linked. For example import a:b/c@0.1.1 may become import a:b/c@0.1.2 if the wit package from deps/ is versioned at 0.1.2. See the canon-names-merge test.

Detailed changes

• Implements PackageName::canon_version_split to split canonical version and its version suffix.
• Defined a new wrapper type wit_parser::PackageKey, so that we can control whether to use the canonical version or not during construction time. Resolve will use PackageKey as the map key, instead of PackageName. Gradually, we can migrate all uses of PackageName to PackageKey.
• sort_unresolved_packages merges packages that share a canonical name, keeping the larger version in Resolve. This behavior needs to be made explicit in the spec.
• encode_interface() and encode_world() consistently use canonical names with version suffixes in the binary encoding.
• Tests are mostly generated by Claude with the following refactor:
  • Existing tests run in both modes (with and without canon-names feature)
  • Tests that produce different output under canon-names feature use *.canon-names.* alternate blessed files
  • In wit-component, test names with canon-names prefix are only run when canon-names feature is enabled.

Things left for future PRs

• Validate a given prefix/suffix combo is well-formed.
• In Resolve, when merging two canonical versions, check if the interfaces really conform to the subtyping relation.
• When both implement and versionsuffix are present, the versionsuffix should refer to the version from implement, instead of the main package name. Need to clarify this from the spec as well.
• Some ergonomic feature to auto-convert legacy wasm modules into the canon-names format.

[alexcrichton]

Thanks for the PR! Before diving too deep into review though I want to clarify a few things first. Primarily I don't think this'll work if it's a crate or a runtime flag feature unfortunately. We need a way to roll this out gradually which means that the previous implementation has to live side-by-side with the new implementation, and then eventually we can slowly transition everything over. Could you dig in a bit more to see if this is possible? If it's not possible that'd be somewhat surprising to me, so could you explain a bit more?

Second is that I'm a bit confused by the breaking change you mentioned here -- I would expect being able to import 0.0.1 and 0.0.2 at the same time to work out. This is similar to importing both WASIp2 and WASIp3 APIs which is intended to work.

[chenyan2002]

We need a way to roll this out gradually which means that the previous implementation has to live side-by-side with the new implementation, and then eventually we can slowly transition everything over.

You mean we need a runtime/CLI flag, instead of a feature flag? That's possible, but less cleaner. Primarily, we would like to define a custom Hash/Eq/Ord for PackageName like this: https://github.com/bytecodealliance/wasm-tools/pull/2602/changes#diff-5b7c6c5cfc5aaf75afd5c7532b27923eaabd7ef9a2a8a4df5636ef1796c927a0R261. Instead of a feature flag, I guess I can use a env var, or a global state that indicates which flag we pick?

If we opt to a runtime flag, like --enable-canon-names, we probably cannot define a custom Hash/Eq trait for PackageName. Then the merging logic would have to live inside wit-component and wit-parser somewhere. The down side is that the upstream libraries, like wac, would have to implement their own canonical version logic, instead of relying on the default Hash/Eq trait. Not sure how important is it to bake the canonical version logic into the PackageName natively in Hash/Eq .vs. let the upstream libraries do their own thing. My intuition is that the goal of canonical names is to make the upstream users' life easier, so that we want to implement all the canonicalization logic inside wit-parser and wit-component.

import 0.0.1 and 0.0.2

My bad. It should be 0.1.1 and 0.1.2, which has the same canonical version. Updated the description as well.

[chenyan2002]

I think the question comes down to this: after we fully switch over, what's the expected behavior of HashMap<PackageName, _>. Do we expect the key to be the canonicalized version or the full version?

During the transition period, we can probably wrap the PackageName struct and use a flag to decide its behavior. Something like this:

impl PackageKey {
  fn new(pkg: &PackageName, use_canonical: bool) -> Self
}

Update: this new function is actually better. It decides the behavior at construction time and we can rely on the regular Hash/Eq trait impl. I will go ahead and implement this.

[alexcrichton]

Personally what I'd expect is a configuration option on ComponentEncoder which would get reflected as a configuration flag on wasm-tools component new to use canonical names. This would be off-by-default because it's still an unstable feature, but the same wasm-tools binary or wit-component library would be capable of building both artifacts. This CLI flag would similarly then get reflected into wasm-component-ld to be available for languages like Rust/wasi-sdk.

Internally though I don't think that updating Hash/Eq on PackageName is the right way to go. We will still want the ability to handle a situation where the Rust standard library uses WASI 0.3.1 and the wasip3 crate manually used in an application uses WASI 0.3.2 or something like that. As-written where this rejects that in Resolve I don't think that's the way to go.

What I would roughly envision for this is that wit-component would get more complicated (I realize it's already quite complicated), but largely wit-parser wouldn't change. With wac I'm not 100% sure what the impact or best implementation route would be, however.

[chenyan2002]

I add a new CLI flag semver_compat = none|merge|canonical in wasm-tools component new reflect this. No more feature flags now.

I defined a new wrapper type PackageKey, so that we can control whether to use the canonical version or not during construction time. Resolve will use PackageKey as the map key, instead of PackageName. Gradually, we can migrate all uses of PackageName to PackageKey.

We will still want the ability to handle a situation where the Rust standard library uses WASI 0.3.1 and the wasip3 crate manually used in an application uses WASI 0.3.2 or something like that. As-written where this rejects that in Resolve I don't think that's the way to go.

In this case, Resolve will merge the version to the higher full version, which is 0.3.2. It won't reject the insertion. See here. The only place we reject it is when the user imports both 0.3.1 and 0.3.2 in a single world, which results in duplicate imports. Merging two worlds in Resolve will only unify the package to the larger version.

[alexcrichton]

Reading over this, how come there needs to be a flag in Resolve? That is semantically a very big change for Resolve since flipping that bool drastically changes loading/lookup behavior. What I would have otherwise expected is that wit-component gets a flag to emit canonical names, and perhaps encoding wit-to-wasm additionally gets a flag to emit canonical names, but I wouldn't expect Resolve to need to change all that much. The changes here seem like they're sort of half-reimplementing the merge-based-on-semver method and that seems ideal to keep that separate. Additionally it seems like that would enable loading two versions in one world, which seems like something we'll want to preserve to me.

[chenyan2002]

Perhaps it's easier to understand from bytecodealliance/wit-bindgen#1686, with the flag in Resolve, wit-bindgen automatically gets the canonical version support. I would expect we can do similar things for wac as well.

From the spec's point of view, canonical version essentially renames the package name to exclude the version suffix. This means that the Resolve behavior is going to change, because we are now loading a wit file with a different name, e.g., from a:b@1.0.0 to a:b@1, and a:b@1.0.1 to a:b@1. This necessarily means it's a new behavior in Resolve.

The changes here seem like they're sort of half-reimplementing the merge-based-on-semver method and that seems ideal to keep that separate.

With the changes in Resolve, merge-based-on-semver becomes a no-op, because there are no multiple packages with the same canonical version in Resolve, so there is nothing to merge, and this simplifies the logic in wit-component, as well as the upstream libraries like wit-bindgen and wac.

Additionally it seems like that would enable loading two versions in one world, which seems like something we'll want to preserve to me.

Not sure what you mean. We can do import a:b@1.0.0; import a:b@2.0.0; in one world. But we cannot do import 1.0.0 and 1.0.1 in one world. Do you mean we want to preserve that ability? If we follow the spec, I think 1.0.0 and 1.0.1 would be considered as the same package under the canonical version, so it's essentially duplicate imports?