Skip to content

Make search for vulnerabilities faster#955

Merged
pombredanne merged 10 commits into
mainfrom
faster-vuln-search
Oct 20, 2022
Merged

Make search for vulnerabilities faster#955
pombredanne merged 10 commits into
mainfrom
faster-vuln-search

Conversation

@pombredanne

@pombredanne pombredanne commented Oct 18, 2022

Copy link
Copy Markdown
Member

This is a PR for #954 and attempts to make some small and critically needed adjustments for performance in the web UI.

Some approaches:

  1. Cascade queries from exact to approximate searches to avoid full table scans in all cases. This is a band-aid for now. The proper solution will likely require using full text search instead.
  2. Avoid iceberg queries with "prefetch related" to limit the number of queries that are needed in the UI
  3. Do not recreate querysets from scratch but instead allow these to be chained for simpler and correct code.
  4. Remove extra details from the vulnerability pacge: each package was further listing its related vulnerabilities creating an iceberg query.
  5. Enable the django-debug-toolbar with a setting to easily profile queries on demand.

Signed-off-by: Philippe Ombredanne pombredanne@nexb.com

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
pombredanne marked this pull request as draft October 18, 2022 14:50
tdruez and others added 8 commits October 18, 2022 21:36
Signed-off-by: Thomas Druez <tdruez@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
And chain querysets.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
And streamline template

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
pombredanne marked this pull request as ready for review October 20, 2022 11:47
@pombredanne
pombredanne requested a review from TG1999 October 20, 2022 11:48

@TG1999 TG1999 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@pombredanne
pombredanne merged commit 9db8b12 into main Oct 20, 2022
@pombredanne pombredanne changed the title [WIP] Make search for vulnerabilities faster Make search for vulnerabilities faster Oct 20, 2022
@pombredanne
pombredanne deleted the faster-vuln-search branch October 20, 2022 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants