Package vendor new feature

[memsharded]

Changelog: Feature: New vendor=True package creation and build isolation strategy
Docs: conan-io/docs#3756

Close #13171

[AbrilRBS]

Nice! Some minor super minor minor suggestions

[db4]

@memsharded I must be too late but the docs for these changes weren't available until recently. Looking into them I see that your approach is to remove all dependencies from the graph completely. I'm not sure it's a great idea. Even repackaged binaries may depend on each other; think of a dll that depends on some internal static libraries (which we would like to hide via repackaging) but requires another repackaged dll to run. Maybe a global vendor=True attribute is OK for someone, but I'm afraid there are too many cases where it won't fit.

[memsharded]

Of course. This is an advanced feature that must be used with lots of care.

But it is also the result of a large demand, see the ticket #13171, and how many tickets it has linked. A very popular use case is to distribute final products to other organizations, via Conan packages.

Organizations have the need to "repackage" or fully isolate dependencies sometimes. They are already doing it via a more manual process like using a deployer to collect all dependencies and then a conan export-pkg to repackage them. The risks these packages they are creating will have are exactly the same as the new vendor feature, but they have to do a large effort to implement it.

When we fully document it, we will make sure to clarify the risks and the cautions that users must have. Of course there are many cases that it won't be possible, this feature is intended for some particular use cases, and abuse is always possible, but we will try our best to warn against it. Still the users are responsibles, it is not great that many of them have to do a tedious and manual work for something that Conan can provide built-in.

[db4]

So it's mainly intended for packaging self-contained applications. It's understandable (due to relative simplicity), but personally I would also like to be able to package (exclude from the graph) specific requirements. Something like

def requirements(self):
    self.requires("dep/1.0", bundle=True)

[memsharded]

self.requires("dep/1.0", bundle=True)

I see what you mean, but what would be exactly the use case of this? Why vendoring only some dependencies, but not others?
The use case that we are focusing one is the demanded one, for example an organization that is creating an SDK that is built from several other Conan packages, and they want to share with another organization without them having to depend on the transitive packages used to create the SDK package.

[db4]

I'd like to bundle an application, but it in turn consists of some modules (dlls). These modules depend on each other. I need to keep these top-level dependencies but hide implementation details (static lib dependencies). I think I'm not alone but of course cannot predict how many users anticipate this feature.

[memsharded]

I'd like to bundle an application, but it in turn consists of some modules (dlls). These modules depend on each other. I need to keep these top-level dependencies but hide implementation details (static lib dependencies). I think I'm not alone but of course cannot predict how many users anticipate this feature.

But what if the DLLs have transitive dependencies themselves, would you also make the dlls fully vendor their transitive static libraries in turn, adding a vendor=True in every DLL package recipe? This sounds a bit difficult to maintain. The idea is that the dlls would also be vendored with the application, in a single package without any dependencies. If you are making visible and transitive the shared libraries that will also make visible and transitive all their transitive dependencies (headers, static libraries), but just vendor the static libraries that are direct dependencies of the application, then it doesn't sound like you are vendoring much, and vendoring just those static libraries will have almost no benefit? What would be the advantage of doing that?

[db4]

Actually, I would like to vendor everything except a small set of top-level dlls (that nonetheless depend on each other). The whole application is quite big, bundling everything together and making a new package on any change is impractical e.g. in terms of size (hundreds of megabytes in my case). Fine-grained vendoring would allow updating just one module for end users; I think it's much more convenient.

[memsharded]

Understood.

I think this is something that we can consider, lets wait to get more feedback from the feature, some real world usage from more users, and lets take this idea for the following iterations. I am not saying this is impossible, just that better go step by step, learn a bit more about the current proposal and iterate from there. Thanks for the feedback!