Extend url2purl/purl2url coverage for Git-based source hosts by ziadhany · Pull Request #223 · package-url/packageurl-python

ziadhany · 2026-02-10T12:29:18Z

Currently, many fix-commit URLs are not converted to PURLs during collection. Adding support for these URLs would ensure more accurate url2purl and purl2url conversions, as well as generating commit and patch URLs from PURLs.

git.kernel.org cgit /repo/commit/?id=HASH
gcc.gnu.org Gitweb /?p=repo.git;a=commit;h=HASH
sourceforge.net Apache Allura /p/project/repo/ci/HASH/
googlesource.com Gitiles /repo/+/HASH
codeberg.org / gitea.com gitea / forgejo commit/{HASH}"
gitlab-sub gitlab.gnome.org
github
gitlab
bitbucket

Related issue:

Research and improve Url2Purl support aboutcode-org/vulnerablecode#2150

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add more tests Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add more test Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add a test Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add support for gitbox.apache.org Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

ziadhany · 2026-03-05T01:46:16Z

@TG1999 @keshav-space This PR is ready for review!. Take a look whenever you have a chance.

pombredanne · 2026-03-06T18:21:15Z

+    commit_match = re.search(gitea_commit_pattern, url)
+    if commit_match:
+        return PackageURL(
+            type="generic",


We need a Forgejo PURL! @johnmhoran

pombredanne · 2026-03-06T18:21:58Z

+        return None
+
+    return PackageURL(
+        type="generic",


There is in emerging git PURL for that from @darakian

pombredanne · 2026-03-06T18:22:59Z

+
+    if match := re.search(kernel_shorthand, url):
+        res = match.groupdict()
+        namespace = "git.kernel.org/pub/scm/linux/kernel/git/stable/"


This needs thinking and there is a emerging PURL registry that will cater to the kernel needs.

pombredanne · 2026-03-06T18:24:31Z

+    match = re.search(gitiles_project_pattern, url)
+    if match:
+        return PackageURL(
+            type="generic",


Likely also a candidate for the new git PURL type

pombredanne · 2026-03-06T18:25:26Z

+    commit_match = re.search(allura_pattern, url)
+    if commit_match:
+        return PackageURL(
+            type="generic",


We may have a sourceforge type? Or this is for a git type

pombredanne

Looking great ... we need some PURL type refinements.

…t repositories. Treat GitLab subdomains as GitLab when using the repository URL instead of a generic type. Add support for salsa.debian.org, gitlab.eclipse.org, forge.fedoraproject.org domains Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

ziadhany · 2026-04-13T12:34:51Z

@pombredanne, for undefined purl types (like Forgejo, sourceforge, or kernel), we should use generic for this, right?
the purl type registier take time , can we process this PR and once we have a new purl type we can just update the code.

mjherzog · 2026-04-13T16:54:57Z

We have an active PR for 'git' (package-url/purl-spec#823), but no PR for forejo or sourceforge.
The "kernel" case is pending a bigger decision for the new 'sid" PURL type.
We may want to wait for the 'git' PURL type to be registered, but we should not wait for the others.
BTW - my understanding is that 'git' is an option for SourceForge, but not the only VCS there - I suspect that there are still many projects using Subversion there.

pombredanne

Thanks! See my review comments. IMHO we can make the code simpler, more functions are OK.

pombredanne · 2026-04-30T09:17:39Z

+    return purl_data.qualifiers.get("commit_url", None)
+
+
+def get_patch_url(purl):


I am not sure it makes sense to put the patch and commit URLs as PURL qualifiers. Why would you want that? To me, parsing a URL to get a commit or patch out of it is not specifically tied to a PURL, this is a URL operation. Now we could also possibly get a commit or patch from a PURL, but this would not be qualifiers on the PURL either.

@pombredanne Ok, I thought it was similar to vcs_url in the purl-spec docs. The main goal is to get the patch text faster, since cloning a repo and using git takes a lot of time. I'm not sure if this is the right place for it in the qualifiers, though.

pombredanne · 2026-04-30T09:21:49Z

    return repo_url


+SUB_GITLAB_DOMAINS = [r"^git\.codelinaro\.org", r"^salsa\.debian\.org", r"^gitlab\.(?!com\b)[^/]+"]


What does this re mean?: (?!com\b)

There are likely a large number of gitlab instances, it feels not realistic to list them all here. Why would this need to be known if the repository_url is provided when parsing a PURL? IMHo we should not need that at all.

@pombredanne you are right this SUB_GITLAB_DOMAINS should be only part of url2purl

r"^gitlab\.(?!com\b)[^/]+" This is a hack based on my observation about how we can identify if this is a GitLab subdomain. I noticed that they always start with gitlab.*

Ex:

gitlab.freedesktop.org/xorg/xserver gitlab.xxx.com

The (?!com\b) this part is to filter out the main GitLab domain, gitlab.com. ( negative lookahead )

pombredanne · 2026-04-30T09:27:29Z

+    return get_generic_template_url(namespace, name, version, "patch_url")
+
+
+@commit_router.route("pkg:gitlab/.*", "pkg:bitbucket/.*", "pkg:github/.*")


Why not having three separate functions? the rules to build a URL from a PURL are not generic at all, so IMHO the code would be simpler and cleaner with 3 functions, one for each forge.

pombredanne · 2026-04-30T09:45:47Z

+
+@commit_router.route("pkg:generic/.*")
+def build_generic_commit_url(purl):
+    """


Can you give some examples as doctests?

pombredanne · 2026-04-30T09:51:07Z

+    if not (namespace and name):
+        return
+
+    return get_generic_template_url(namespace, name, None, "repo_url")


you should use kwargs, but in all cases this calls for using an object with separate methods for each url type

pombredanne · 2026-04-30T11:14:48Z

+    return get_generic_template_url(namespace, name, version, "commit_url")
+
+
+@patch_router.route("pkg:generic/.*")


The router is designed to route like a multidispatch, so you should use it for dispatching based on specific URL/PURL patterns

pombredanne · 2026-04-30T11:15:45Z

    )


+def build_route_regex(domain_patterns, path_suffix="/.*"):


this is too complex IMHO, just use instead a proper list of route patterns

pombredanne · 2026-04-30T11:17:14Z

+
+
+@purl_router.route(SUB_GITLAB_ROUTE_REGEX)
+def build_gitlab_sub_purl(url):


What you are missing also is a function that takes a URL and return details about a patch or commit, not even with PURL in the mix

Can you explain more about what you mean by getting patch/commit details? Do you mean fetching the data and parsing the text? I thought this will be a job for VulnerableCode or ScanCode.io.

ziadhany marked this pull request as ready for review February 10, 2026 12:34

ziadhany mentioned this pull request Feb 11, 2026

Proposal: Support for linux and android types package-url/purl-spec#810

Open

Extend purl2url and url2purl support for parsing more Git host providers

4c168a8

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

ziadhany force-pushed the url2purl branch from 031c8f8 to 4c168a8 Compare February 25, 2026 21:11

ziadhany added 7 commits February 25, 2026 23:14

Fix linting CI error

db9d6bc

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add purl2url support to get a vcs_url

7e87e09

Add more tests Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add more test and fix cgit regex pattern

24fdf1d

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Fix cgit regex pattern

7e24835

Add more test Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add support for kernel_shorthand pattern

0020818

Add a test Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Ensure the version is optional during purl creation

5909b64

Add a test Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Fix CI black format

c47acb5

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

ziadhany mentioned this pull request Mar 4, 2026

Add commit collection support to Apache Tomcat V2 importer aboutcode-org/vulnerablecode#2195

Open

7 tasks

ziadhany added 2 commits March 5, 2026 03:41

Update url2purl/purl2url and simplify the regex pattern

d02949b

Add support for gitbox.apache.org Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Fix CI black format

92da2f2

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

pombredanne reviewed Mar 6, 2026

View reviewed changes

Comment thread src/packageurl/contrib/purl2url.py Outdated

pombredanne reviewed Mar 6, 2026

View reviewed changes

Comment thread src/packageurl/contrib/purl2url.py

pombredanne reviewed Mar 6, 2026

View reviewed changes

Comment thread src/packageurl/contrib/url2purl.py Outdated

pombredanne reviewed Mar 6, 2026

View reviewed changes

pombredanne requested changes Mar 6, 2026

View reviewed changes

johnmhoran mentioned this pull request Mar 6, 2026

Add new PURL type: 'forgejo' package-url/purl-spec#828

Open

ziadhany added 2 commits March 11, 2026 15:03

No patch url for allura ( sourceforge.net , forge-allura.apache.org )

fbff2d3

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

pombredanne reviewed Mar 13, 2026

View reviewed changes

Comment thread tests/contrib/data/url2purl.json Outdated

ziadhany added 2 commits March 17, 2026 00:30

Fix invalid sub-gitlab domain

4929fc0

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Simplify build repo/commit/patch url from generic type template_url

205d048

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

ziadhany requested a review from pombredanne March 24, 2026 00:45

ziadhany mentioned this pull request Apr 9, 2026

fedcode-next: Extract fix commits from pull requests and issues body or comments in search for CVE-related messages aboutcode-org/vulnerablecode#2002

Open

ziadhany mentioned this pull request Apr 14, 2026

fedcode-next: Code pipeline and models to continuously automatically collect fix commits aboutcode-org/vulnerablecode#1721

Open

4 tasks

pombredanne reviewed Apr 30, 2026

View reviewed changes

		return purl_data.qualifiers.get("commit_url", None)


		def get_patch_url(purl):

		return repo_url


		SUB_GITLAB_DOMAINS = [r"^git\.codelinaro\.org", r"^salsa\.debian\.org", r"^gitlab\.(?!com\b)[^/]+"]

		return get_generic_template_url(namespace, name, version, "patch_url")


		@commit_router.route("pkg:gitlab/.", "pkg:bitbucket/.", "pkg:github/.*")

		return get_generic_template_url(namespace, name, version, "commit_url")


		@patch_router.route("pkg:generic/.*")



		@purl_router.route(SUB_GITLAB_ROUTE_REGEX)
		def build_gitlab_sub_purl(url):

Conversation

ziadhany commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ziadhany commented Mar 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ziadhany commented Apr 13, 2026

Uh oh!

mjherzog commented Apr 13, 2026

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ziadhany Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ziadhany commented Feb 10, 2026 •

edited

Loading

ziadhany Apr 30, 2026 •

edited

Loading