[RFC] Library Support and Boilerplate Comment Thread

[hansl]

As a follow up for #1692 (comment).

Continue the conversion here.

[achieverprince]

@hansl

Thanks for the official reply.
Hoping to see a update from the team soon :)

[robwormald]

Thoughts from me (I maintain ngrx as well as Angular devrel)

So long term, this will land in the devkit, but that's a fairly major architecture change. In the interim, lets start with a "seed" style repo people can just fork and start work with.

MVP should basically:

• use npm scripts for task running, with setup for build/test/publish: sample from ngrx/store: https://github.com/ngrx/store/blob/master/package.json#L7-L21
• use ngc + rollup to produce FESMs + metadata + d.ts (though this really only applies to single function things - this is not an ideal process for material-type megarepos.
• have a minimal test setup + integration spec to verify functionality + AoT friendlines

Anything beyond that is going to quickly get into the realm of bikeshedding best practices / fav code coverage tool of the week, so let's start minimally and go from there.

[mattem]

First off thanks @hansl for the update, it's very much appreciated.

I'd like to understand the future of ng cli, where this fits with the Angular Development Kit and the tools to generate custom templates.

Our company has put a lot of time and effort in to getting on board with Angular 4.x and using the ng cli, we have been though many refactors thought the libraries lifecycle. We are currently using a custom build process simular to what @robwormald describes for building and testing our component libraries, which we were hoping could be removed once this feature landed in cli.

I'm left worrying that we will have to change our non-library applications build process again once these new tools land, or will they just for library projects?

[robwormald]

At Google, we use the terms "binary" and "library" for this:

• a binary is a shippable-to-the-browser-artifact that's "self executing"
• a library is a designed to be consumed by binaries (or other libraries)

The current Angular CLI was, from the very beginning, built to be a tool for managing binaries. In the early days, when we used ember-cli + broccoli, outputting libraries would have been simpler to do, as it was less opinionated and more flexible. The Angular community demanded (rightly, IMO) that we switch over to Webpack. Webpack is designed to build binaries, and so we're kind of stuck.

@mattem the whole point of not giving you access to the internals of Webpack is that its our goal to allow seamless transition as we evolve the underlying tools. Long term our goal is to dramatically simplify the whole process, but that takes time. The point of this thread is to provide an interim solution.

[robwormald]

@mattem long term, we're working on eliminating the need to ship metadata and such to npm in favor of shipping precompiled code. We have this setup in place to allow us to be flexible with the compiler APIs (this is what allowed us to change everything in between v2 - v4) - once we're happy with the API we'll mark it as public, which opens up a number of really cool new possibilities. This is (currently) on the roadmap for about a year out, but the work to prepare for that is in progress.

[hansl]

@mattem Rob didn't answer this question, so I'll take a stab:

I'd like to understand the future of ng cli, where this fits with the Angular Development Kit and the tools to generate custom templates.

The DevKit will be a set of libraries and tooling that you can script or develop against. Parts of it will be to build, test and generate files. This sounds really generic because it is; it will be a lot of conventions enforced at the API level so that tools (such as IDEs, scripts or CIs) will work in a similar way.

The CLI in the future (version 2? 3?) will only be a tool that uses the DevKit in the background, abstracting it from users. If you use the CLI today, you should be very familiar to the CLI in the future. The real difference is that every pieces underneath will be able to be used independently by other tools. So you could see, e.g., a VSCode extension that knows how to build your app/library, which uses the same code path as the CLI.

[oliverjanik]

Pardon my French, but WTF? Has Angular jumped the shark?

One of the top priorities of Component framework is to facilitate re-usability of components. I don't know what Angular Devkit is and I don't understand what multi-application support means. I just want to package a measly combo-box and share it with the team and/or the world.

Lack of standardization and lack of guidance in this area is really hurting the ecosystem.

I realize that the underlying complexity is beyond understanding of any casual app developer who just uses NG but, frankly, put your hand on your heart and ask yourself this question:

"Have we dug ourselves into a complexity black hole?"

The promise of NG2 was to banish the complexity and unfamiliarity of NG1. Time for a sincere introspection is now.

Meanwhile for us, who have to explain to our managers what JIT is or AOT and why builds randomly break even though they're perfectly good Typescript and why it takes so long to extract common functionality that already works to be used in another project, well, we might be forced to jump the ship.

[yordis]

My 2 cents in the whole topic from the previous thread to the one I started.

First @oliverjanik I couldn't say it better.

I took the decision of do not move out of my Front end stack again, I can't be jumping between stacks while I want to create something mature that allow me to focus on the business rather than being "cool" or just change it for a minor improvement at the end of the day.

So I went back to Angular (I was in Angular 1 long time ago) because I saw the architecture changes and it looks fantastic pretty much, but then I hit wall that shouldn't be there for me #6466 , After a full week trying to get answer to my concern I realice that I will have to hack the setup and it will no be straight forward either.

Why should I hack something some simple as @oliverjanik explain, when all I want to do is to create the most common use case on any ecosystem: shared a reusable module.

CLI are meant to be use base on the ecosystem, I got that, but the same time there is common use case that should be fixed already, specially when there is a lot of people with the same concern.

And then even worse you read something like this #1692 (comment) that scare me a lot as something that wants to think in long term and base out a whole company on it.

If you (Angular Ecosystem main contributors) don't want to support your community (it doesn't mean you didn't do it at all), just be clear with us. A lot of people depends of you and sometimes it doesn't even take a week for implement something with a huge impact for the good of the community. No everything can be Google interests but at least be clear about it.

I am speaking from the side of a person that just started 2 weeks ago, a person that was happy in React ecosystem but always wanted to go back to Angular for many reasons, a person that wants to be part of your community and probably in the future could be a potential contributor, making your life easier.

Please think in your community, it will pay off in the long term and don't read this and feel bad about it and think in the negative mindset because I can't express how thankful I am that you did an amazing job on Angular but at the same time, don't let small things like this one opaque your greater.

[rjcorwin]

just be clear with us

@yordis Agreed. Before we talk about technical solutions, there should be a mention of this issue in the Docs. I created a PR angular/angular#17129

My 2 cents is that for most people, they just need to know there is some direction in creating reusable Angular Modules. If we could add @robwormald's solution to the docs, then count me in. It doesn't need to be the most technically optimal solution for now, there is a lot of value in rallying behind non-optimal solutions for the sake of having consensus. Until then, Angular is the framework that "Makes great decisions so you don't have to... but you can't reuse your code."

[robwormald]

@oliverjanik @yordis @rjsteinert appreciate the feedback.

I just want to package a measly combo-box and share it with the team and/or the world.

I don't want to in any way minimize what you're saying here, because there's a lot of good points in it - but we should be clear that this is absolutely possible today. Its absolutely more complicated than it needs to be, but there are some good examples out there you can use as a reference.

A great one to look it is ng-bootstrap: https://github.com/ng-bootstrap/ng-bootstrap/

Here's the packaging format guidelines Jason presented at ng-conf: https://goo.gl/AMOU5G - admittedly I had to ask around to track that down, so we'll see about getting that posted somewhere more reachable.

@filipesilva has also done great work on writing up a guide for this in the docs, and a starter repo you can clone, and unfortunately the pull request got caught behind our infrastructure changeover. I'm working on getting this merged ASAP.

You can see that PR here: angular/angular#16486
And the starter repo here: https://github.com/filipesilva/angular-quickstart-lib

Lack of standardization and lack of guidance in this area is really hurting the ecosystem.

It should be said that there's pretty much zero precedent for what we're trying to do here. Some of the challenges in this space alone:

• The JS ecosystem and NPM currently have no standard plan for shipping ES2015+ code to NPM.
• NodeJS and friends are still debating how node will load ES modules.
• Webpack, out of the box, doesn't transpile code coming out of NPM, but...
• ES5 code is effectively un-treeshakeable.
• Until about a month ago, no browser had any support for ES2015 Modules, static or dynamically loaded.
• NPM is built on a non-standard, non-statically analyzable module format that is not great for performance: https://nolanlawson.com/2016/08/15/the-cost-of-small-modules/
• Some developers want the easiest thing possible (drop the script in the page) and others want the absolute freedom to eek out every single ms of performance. This means UMD for the former, and ESM/ES2015/6/7 for the latter. See http://2ality.com/2017/04/setting-up-multi-platform-packages.html and note the meta there - it refers back to the above proposal from the Angular team.
• Historically, bundling all your code has been the "right" way of doing things, but in a world with http2(push) and low-power mobile devices, this may not be the correct path forward!

...and all this is before we even start talking about Angular! Again, we're totally with you on this, but this is an ecosystem we're all a little bit at the mercy of. One thing that became quite clear when joining the Angular team is that we really are working at the bleeding edge - a lot of this stuff we literally have to invent as we go. It's a big reason I love my job, but it means that sometimes, yeah, you're gonna have to try things that sometimes won't work.

Example: the FESM format mentioned in the above doc works great for projects like ngrx, but unfortunately it had an impact on libraries like Angular Material, and so we're tweaking those guidelines for cases like that. A lot of these things you just don't know until you try.

One thing I think is important to understand here is that we (the Angular team) are dealing with this problem twice - once internally, for all the projects at Google and Alphabet that use Angular, and once for the FOSS community. Why?

At Google, we use an entirely different set of tools than are used in the open source world:

• Blaze (or bazel.io) , our build system used to build everything at Google, from Angular to Maps to Gmail.
• Closure Compiler, which does bundling (replacing Webpack), optimization (replacing Uglify), module loading (replacing SystemJS or Webpack)
• A whole lot of strictly enforced conventions and rules, with the build infrastructure to enforce it.
• we have our own special module format (closure's goog.require)
• we don't use NPM or node_modules

This means @hansl, who runs the open-source facing CLI project, is doing work that can't be shared by the internal Angular infrastructure teams (because we're not using Webpack et al), and vice-versa.

If we're being perfectly honest, this situation, and the internals of CLI as it exists today are not sustainable. Having a great scaffold from the ember-CLI project and leveraging @TheLarkInn and the awesome Webpack community allowed us to get a functional tool that was usable ASAP - but its led to a fairly significant amount of technical debt.

This isn't unexpected, and again, it's the reason we've held fast on not exposing the Webpack internals - we knew at some point, we'd have to pay off that technical debt, and we want to do so without exposing developers to churn.

The idea of the DevKit project is to finally allow us to share tooling between our internal and external customers - meaning @hansl and co aren't duplicating work, and excitingly, developers of teams at Google can help improve the same tooling open-source Angular developers use. @alexeagle has written a bit about this - https://medium.com/@Jakeherringbone/what-angular-is-doing-with-bazel-and-closure-21f526f64a34 - We've requested and helped design features from the Typescript team (like the transformers pipeline) that will finally unlock further code sharing and streamline our TS tooling.

If you don't care about how it works, then you won't have to care about how it works. It will, however, allow developers who want to, to build and extend the toolchain to do all sorts of powerful things. As an off the cuff example - we speak to lots of teams who want a standard boilerplate for their internal packages. DevKit will enable that - and not in a patched-on-the-side fashion, but baked into the design from the beginning. Want to plug into the generation pipeline and add your own custom license to every generated file? Sure.

All software is about balancing tradeoffs - we made a decision early on to provide open-source developers who want to be productive building applications a tool they could use immediately. I think, with a few exceptions, we've delivered on that goal. We're trying to do this while also balancing long-term maintainability, and delivering for the teams at Google that pay our salaries (and those of our awesome contractors) and make the entire Angular project possible at all, and there's only so many hours in the day!

If there's anything else you'd like clarification on here, please don't hesitate to ask.

[pjpenast]

@robwormald I do not understand why you say the problem is Webpack when there are thousands of examples of libraries built with Webpack

I completely agree with @oliverjanik, it is not understandable that from the team of Angular is not given priority to this. You can't make a framework based on compononents that doesn'tallow reuse of components. It's completely absurd.

In our development team, the first thing we did when we started working with Angular was to create a library of reusable components for all our modules. It does not make sense that Google has not thought the same way.

We have also been able to create reusable components such as a combo box without effort. If you are interested in knowing our project is this:

https://github.com/Stratio/egeo

[yordis]

Two main concern in how to approach this because I am misunderstanding.

1. CLI force me to have bootstrapModule in my main module #6466 Based on that issue, why do the CLI force me to have a bootstrap application when I only want to have NgModule and use the CLI for run the testing in this case.
   I do not understand why is that a big deal right now, when it's pretty much run the karma testing but for some reason something is stopping before complaining about some code analysis I guess, no clue why.
   The use case is pretty straight forward. 3 applications. First application for the reusable core features. Second application for the github page release. Third application for development testing, like chicken sink for visual testing pretty much, exactly what material2 would use without needs of any gulp or any other setup.

What do we need to do for tackle that use case without any special case? Because so far there is only one issue with it

2. What happen if we just push the Angular projects as Typescript projects rather than Javascript? You mentioned that Webpack do not compile from node_modules?! The last time I check you actually have to put the exclude filter for node_modules for stop that to happening. Right?!

[dherges]

Please keep it simple.

Now we have that Angular Package Format, we need a projection: (entryFile: string = 'src/public_api.ts') => NgPackageFormat

interface NgPackageFormat {
  main: string,
  module: string,
  /* ... */
}

It will take more than just one input property ... we need ONE workflow for creating that Angular Package Format from source. I agree with @robwormald that unit testing support is a major part. Taking away the burden of configuring several build steps manually will be the NEXT step towards. I do not care about the second and third steps for now. You will fail and the ng cli has failed for almost a year in doing the big "Library developer mode". Please take one decision. Do one step.

-- edit: I excuse for the bold wording. Please let us just stay focused on improving step by step.