GitHub as a Provider

[bchavez]

As the title suggests, would be nice to set GitHub as a provider at a package level. Sometimes package maintainers do not immediately publish to NPM or bower, and they have fixed bugs sitting in their GitHub repo after a PR or commit. It would be nice to point LibMan to a repository on GitHub to retrive some code.

Allow me to specify:

• The GitHub release Tag.
• Or the commit hash of a particular branch.
• Or just the branch to pull the latest HEAD.

Thanks,
Brian

⏳ 🔍 "But I still haven't found what I'm for..."

[davidfowl]

This is a good idea.

[tb-mtg]

Can other providers be added too? (like jsdelivr or even npm)

[nickalbrecht]

Just branstorming some ideas I wanted to put here. To drive the intellisense, each package source implementation likely needs to be custom. Especially for the Library list. Intellisense for the files within a package is a little easier, provided the source provides a way to download the package in it's entirety, then you can build the intellisense from the contents after expanding it to the temp folder. I'm assuming you wouldn't want a GitHub plugin to depend on a full git client? If a developer is willing to sacrifice the intellisense aspect (especially for unique/obscure sources), it could be even simpler as theoretically you could simply provide an HTTP provider and pull files directly from a URL, resulting in a config similar to this example maybe? Could also allow for the library URL to be a zip file and simply have it download that and extract it to keep intellisense for choosing files.

Note: My examples below do NOT work, and are just brainstorming what they might look like. It'd be very easy to implement a low level HTTP provider that sacrificed intellisense in favor of wide spread support of disparate package sources.

{
    "libraries": [
        {
            //GitHub
            "provider": "http",
            //Maybe even a path to a release/zip?
            //"library": "https://github.com/twbs/bootstrap/releases/download/v4.1.0/bootstrap-4.1.0-dist.zip",
            "library": "https://raw.githubusercontent.com/twbs/bootstrap/v4-dev/dist/",
            "files": [
                "js/bootstrap.bundle.js",
                "css/bootstrap.css"
            ],
            "destination": "wwwroot/lib/bootstrap_from_GitHub"
        },
        {
            //Gist
            "provider": "http",
            "library": "https://gist.githubusercontent.com/user/commit/raw",
            "files": [
                "myFav.js",
                "popularstyles.css"
            ],
            "destination": "wwwroot/lib/Favourites_from_Gist"
        },
        {
            //Unpkg
            "provider": "http",
            "library": "https://unpkg.com/[email protected]/dist",
            "files": [
                "js/bootstrap.bundle.js",
                "css/bootstrap.css"
            ],
            "destination": "wwwroot/lib/bootstrap_from_Unpkg"
        }
    ],
    "version": "1.0"
}

And of course there's always the more structured approach where you have a full provider for something like GitHub, and have somethings like this

{
    "libraries": [
        {
            "provider": "github",
            "repository": "twbs/bootstrap",
            //Branch, or Commit, or Pull & Commit?
            "branch": "v4-dev",
            "commit": "commit_hash_here",
            "pull": "42",
            "flatten": ["dist"],
            "files": [
                "js/bootstrap.bundle.js",
                "css/bootstrap.css"
            ],
            "destination": "wwwroot/lib/bootstrap_from_GitHub_explicit"
        }
    ],
    "version": "1.0"
}

[justcla]

Regarding the GitHub provider:
Thanks for the suggestion @bchavez.
Cool design, @nickalbrecht. That looks good to me.

Regarding the http provider:
Thanks for the http provider examples, @nickalbrecht.
Yes, that's pretty much exactly how we see it working, too. We had a full group design session on this earlier in the week. We figured we'd call it the "URL" provider. (It might also support https)
It's likely this will be the very next provider we build. But we should move that chatter to another issue.

[jimmylewis]

@tb-mtg yes it is (or will be) extensible, though I'd like to see a better design around that. Right now you'd have to do both a build task package and a VSIX to make everything work on par with what's in VS now.

[nickalbrecht]

@jimmylewis What would a new provider need to implement on the VS side of things? I would have guessed that the existing contract interfaces would have been enough to allow for a single package in order to add support for a new provider to both VS and Build?

[jimmylewis]

@nickalbrecht Currently the Providers get hooked up to VS via the IProviderFactory interface, which requires that you register a MEF component in VS (easiest done with a VSIX). It's not ideal to have to install these to VS, but currently we don't have a good way to detect which providers to load from the project state and where to load them from.

(I'd love it if we can get to a point where we can detect/load from PackageReference or something.)

[nickalbrecht]

@jimmylewis By having an additional Nuget package installed (to support URLs for example), would it "just work" for the build process portion then? What would you loose out on the VS side of things, just the intellisense, and the trigger when the libman.json file is saved/touched?

[jimmylewis]

@nickalbrecht going off the top of my head, there needs to be a way to load the provider into the build process for the CI integration scenario. I don't know exactly how that would work offhand, but I imagine it could be done just by having a .targets in the package to hook into the restore step. It would be good to have an example of how this is meant to be done (@justcla, can you look into documenting how this can be extended?).

The VSIX would provide both IntelliSense and restore-on-save functionality in VS, at least until a different mechanism is available. So without a VSIX, the VS integration is currently not very extensible; you'd get a pretty vanilla editing experience and then rely on the restore-on-build package to get the files.

We were originally hoping to tackle the extensibility design after the initial release so we could get a better read on which scenarios need to be supported. That should still be on the roadmap AFAIK, though I haven't been as involved in planning recently.

[davidfowl]

I just realized we don't need to do this at all, we can just add support for https://www.jsdelivr.com/ which supports npm, and github

[bchavez]

Hey @davidfowl , great idea!

Just one concern, I don't immediately see a way to tell jsDelivr that I want a specific hash on a specific branch. For example, take a look at this repo:

https://github.com/google/diff-match-patch

There are no tags or releases on GitHub. Nor are there any (as far as I'm aware) official NPM packages by Google for this repo. So, attempting to get at any kind of source with jsdelivr.com:

https://cdn.jsdelivr.net/gh/google/diff-match-patch/
Couldn't find the requested release version latest.

https://cdn.jsdelivr.net/gh/google/diff-match-patch@HEAD/javascript/diff_match_patch.js
Couldn't find the requested release version HEAD.

https://cdn.jsdelivr.net/gh/google/diff-match-patch@9771b5f777e3abf97b842cc2a8c2bca296cde7b4/javascript/diff_match_patch.js
Couldn't find the requested release version 9771b5f777e3abf97b842cc2a8c2bca296cde7b4.

So, commit hashes don't seem to be supported. 😢

Little blocking edge cases like this that are going to either make or break this tool's usability for me. Death by a thousand cuts kind of thing.

Also, BTW great talk on Build 2018, SignalR and Bedrock! :)

💫 💥 Chaos Chaos - Do You Feel It?

[MartinKolarik]

@bchavez we might be able to add support for commit hashes to jsDelivr. Is there anything else that would be a problem?

EDIT: To clarify, this would make your 3rd example work but not the first two. The reason we originally decided to support releases (git tags) only is that using just a branch name leads to a) frequent updates, which can't be cached effectively (or complains from people when they are cached), and b) frequent breaking changes (code changes, renaming files, etc.). Specific commit hashes would prevent those problems but also work for projects that don't have any tags, so it seems to be a win-win.

[MartinKolarik]

https://cdn.jsdelivr.net/gh/google/diff-match-patch@9771b5f777e3abf97b842cc2a8c2bca296cde7b4/javascript/diff_match_patch.js works now.

[nickalbrecht]

The only thing that I can think of, is that with the GitHub through jsDelivr (or just GitHub in general really) there's no guarantee that a tag is a version number (or wholly parseable as one). This means there's a chance that LibMan will fail to get updates for a package. I'm guessing releases are more reliable for this? And there's definitely no way to check for updates if it's by commit hash. Unless there's some sort of date or something that's exposed in order to sort them?

What would happen if a command was issued to update a package (similar to #82) for a provider that does not provide versioning

[MartinKolarik]

The only thing that I can think of, is that with the GitHub through jsDelivr (or just GitHub in general really) there's no guarantee that a tag is a version number (or wholly parseable as one). This means there's a chance that LibMan will fail to get updates for a package.

I think it's a fair deal to say that update only works with semver versions (that's what jsDelivr does) but with a good version parsing algorithm you can make it work with almost any versioning scheme. Most packages provide either npm or GitHub releases so commits should be used only as a last resort.

[lbearl]

Is GitHub or jsDelivr support something that is still on the roadmap?

[MartinKolarik]

There's a PR with jsDelivr implementation (#338) and related discussion (#344).

[bchavez]

I tried using LibMan today and wanted to provide a quick update on this issue to drive home my point on the current state of affairs.

The Microsoft aspnet/jquery-ajax-unobtrusive project is not on CDNJS and the current version on NPMJS here is outdated.

So, The GitHub is probably the best place where you can get the latest 3.2.5 release of aspnet/jquery-ajax-unobtrusive. You can also get the latest from the "ajax.aspnetcdn.com" CDN run by The Microsoft. But, daayymm son... The Microsoft CDN don't work with LibMan.

Hopefully, by now ma point is clear. Whatever the reason, people are busy, lazy, forget, don't care, the physics of life & the universe, LibMan will ultimately need to deal with the ever increasing world of disorder rooted in the 2nd law of thermodynamics. LibMan, the odds are against you.

Lmao.

It is 2018 and it's still funny, at the end of the day, downloading the JS file (via right-click/save as...) to your PC is still the best way to get the latest JavaScript for your website. Ain't no half-baked tool gonna do it for you.

These are the real world scenarios that LibMan needs to deal with effectively if it wants to be taken seriously. Otherwise it will fall into the category of tools that everybody has but nobody uses.

(... looking at you defrag.exe 👀)

💫 💥 Chaos Chaos - Do You Feel It?

Re: #344

[MartinKolarik]

Since yesterday jsDelivr can also serve branches without a specific commit hash so all three links in @bchavez comment work now.

[jimmylewis]

Since the jsDelivr provider has been implemented, this functionality is now available (latest version of the NuGet packages, or Dev16 Preview 2 or newer).

[pabrams]

Doesn't seem very "implemented" to me. I'm constantly (but not consistently) getting errors like this:

2020-08-31T15:08:08.0456799Z Restore operation started...
2020-08-31T15:09:48.7753980Z libman.json : error LIB002: The "wet-boew/[email protected]" library could not be resolved by the "jsdelivr" provider [C:{my-folder}_work\r5\a{my-artifact}\drop\src{my-project}{my-project}.csproj]
2020-08-31T15:09:48.7755803Z
2020-08-31T15:09:48.7756176Z One or more libraries failed to restore
2020-08-31T15:09:48.7756310Z
2020-08-31T15:09:48.8736718Z ##[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1
2020-08-31T15:09:48.8743535Z ##[error]Dotnet command failed with non-zero exit code on the following projects : C:{my-folder}_work\r5\a{my-artifact}\drop\src{my-project}{my-project}.csproj

[nickalbrecht]

@pabrams That's a separate issue that is still in need of some troubleshooting. #370