Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: map artifacts to commits via repo tags #508

Merged
merged 44 commits into from
Dec 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
d85960a
feat: map artifacts to commits via repo tags
benmss Oct 10, 2023
e89c0d1
chore: reverted expected results file
benmss Oct 10, 2023
6e2b7db
chore: updated regex; added config option
benmss Oct 11, 2023
58b3e64
chore: minor optimisation of tag matching
benmss Oct 12, 2023
c54c2a6
chore: moved matched_tag length check out of for loop
benmss Oct 12, 2023
4d7adfc
chore: better handling of tag.commit
benmss Oct 12, 2023
56b3417
chore: fixed test value iteration method
benmss Oct 12, 2023
e3c30d3
chore: tried to clarify comment on tag matching
benmss Oct 12, 2023
7cf7942
chore: renamed commit_from_version function; added empty commit utili…
benmss Oct 13, 2023
fbda86d
chore: redesigned tag matching process
benmss Oct 13, 2023
8961935
chore: moved version pattern creation to function; updated tag.commit…
benmss Oct 19, 2023
0e6bc69
chore: escaped purl.name in regex string
benmss Oct 19, 2023
dddb89e
chore: do not fallback to default commit when commit finder fails
benmss Oct 25, 2023
e941592
chore: prevent commit finder false positive caused by accepting a mix…
benmss Oct 25, 2023
2499b01
chore: extend optional suffix matching to prefer tags that most close…
benmss Oct 26, 2023
bd8b8d6
chore: revert outdated logging change
benmss Nov 1, 2023
0373bc0
chore: attempt to update integration tests with commit finder functio…
benmss Nov 1, 2023
c8e7a90
chore: improve reporting for found commits with no branch
benmss Nov 1, 2023
7930d3d
chore: improve regex for tag matching; add regression test for tag ma…
benmss Nov 1, 2023
fdbf2a7
chore: removed shebang for repo / commit finder tests
benmss Nov 2, 2023
30e6a67
chore: remove extract tag function
benmss Nov 2, 2023
7dc72bc
chore: remove commit finder config option
benmss Nov 2, 2023
1304bfe
chore: addressed PR feedback
benmss Nov 2, 2023
cbf3e9a
chore: account for purls that are repositories
benmss Nov 3, 2023
154dfdc
chore: move has_trailing_zero check out of loop; remove string coerci…
benmss Nov 3, 2023
0a7ae77
chore: extract commit from tag retrieval to function for clarity; mov…
benmss Nov 3, 2023
0e66b8a
chore: improved regex related comments; moved zero digit extension ou…
benmss Nov 3, 2023
2482abc
chore: make regex patterns evaluate from end of string
benmss Nov 6, 2023
98225a2
chore: include purl and repo within commit finder test data, and some…
benmss Nov 7, 2023
c8bddf8
chore: update jackson-databind expected results file
benmss Nov 8, 2023
e3c8037
chore: make e2e repo/commit finder tests report correctly
benmss Nov 9, 2023
94aee85
chore: improve tag matching regex to handle case where the prefix con…
benmss Nov 9, 2023
39d4bc6
chore: fix name of defaults property for repo finder e2e test
benmss Nov 9, 2023
b616714
chore: add Hypothesis testing for commit finder; streamline existing …
benmss Nov 14, 2023
b3a5567
chore: improve Hypothesis text usage; separate pattern creation from …
benmss Nov 16, 2023
ab17a9c
chore: clarify behaviour of zero digit extension in commit finder pat…
benmss Nov 22, 2023
05b2e5f
chore: use enum for deps.dev supported types
benmss Nov 24, 2023
d350f22
chore: renamed main commit finder functions; clarified comments; rem…
benmss Nov 29, 2023
2ca40db
chore: rename comparison function; update commit finder failure messa…
benmss Nov 29, 2023
c27bd35
chore: add more units tests for commit finder; update commit finder l…
benmss Nov 29, 2023
4839352
chore: differentiate abstract purl type; move regex escape call to mo…
benmss Nov 30, 2023
f0bf672
chore: update api doc
benmss Nov 30, 2023
ebafb17
chore: fix altered usage of re.escape
benmss Dec 6, 2023
e9855b8
chore: update docs for commit finder
benmss Dec 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@ macaron.repo\_finder package
Submodules
----------

macaron.repo\_finder.commit\_finder module
------------------------------------------

.. automodule:: macaron.repo_finder.commit_finder
:members:
:undoc-members:
:show-inheritance:

macaron.repo\_finder.repo\_finder module
----------------------------------------

Expand Down
8 changes: 5 additions & 3 deletions docs/source/pages/using.rst
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ For more detailed information on converting a given artifact into a PURL, see `P

.. note:: If a repository is not also provided, Macaron will try to discover it based on the artifact purl. For this to work, ``find_repos`` in the configuration file **must be enabled**\. See `Analyzing more dependencies <#more-deps>`_ for more information about the configuration options of the Repository Finding feature.

.. note:: If no repository is provided, but the PURL contains a version (as with all of the above examples), Macaron will attempt to find the exact commit that matches the provided version. For this to work, the discovered repository must support and make use of tags to denote commits relating to released artifacts.



-------------------------------------------------
Verifying provenance expectations in CUE language
-------------------------------------------------
Expand Down Expand Up @@ -247,9 +251,7 @@ With ``path_to_sbom`` is the path to the SBOM you want to use.
Analyzing more dependencies
'''''''''''''''''''''''''''

In some cases the dependencies that Macaron discovers lack a direct connection to a repository for it to analyze. To improve results in these instances, the Repository Finding feature can be enabled. This feature makes use of a dependency's identifying information that can be found using the package registries located on the Internet.

.. note:: The Repository Finding feature currently only works for Java projects via SCM meta data found within artifact POM files.
In some cases the dependencies that Macaron discovers lack a direct connection to a repository for it to analyze. To improve results in these instances, Macaron is capable of automatically determining the repository and exact commit that matches the given dependency. For repositories, this is achieved through examination of SCM meta data found within artifact POM files (for Java), or use of Google's Open Source Insights API (for other languages). For commits, Macaron will attempt to match repository tags with the artifact version being sought, thereby requiring that the repository supports and uses tags on commits that were used for releases.

This feature is enabled by default. To disable, or configure its behaviour in other ways, a custom ``defaults.ini`` should be passed to Macaron during execution.

Expand Down
21 changes: 17 additions & 4 deletions scripts/dev_scripts/integration_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ RESOURCES=$WORKSPACE/src/macaron/resources
COMPARE_DEPS=$WORKSPACE/tests/dependency_analyzer/compare_dependencies.py
COMPARE_JSON_OUT=$WORKSPACE/tests/e2e/compare_e2e_result.py
TEST_REPO_FINDER=$WORKSPACE/tests/e2e/repo_finder/repo_finder.py
TEST_COMMIT_FINDER=$WORKSPACE/tests/e2e/repo_finder/commit_finder.py
RUN_MACARON="python -m macaron -o $WORKSPACE/output"
RESULT_CODE=0
UPDATE=0
Expand Down Expand Up @@ -277,7 +278,7 @@ declare -a COMPARE_FILES=(
"slf4j.json"
)

$RUN_MACARON analyze -c $WORKSPACE/tests/e2e/configurations/micronaut_test_config.yaml --skip-deps || log_fail
$RUN_MACARON analyze -purl pkg:maven/io.micronaut/[email protected] --skip-deps || log_fail

for i in "${COMPARE_FILES[@]}"
do
Expand Down Expand Up @@ -358,13 +359,14 @@ check_or_update_expected_output $COMPARE_JSON_OUT $JSON_RESULT $JSON_EXPECTED ||
echo -e "\n=================================================================================="
echo "Run integration tests with configurations for FasterXML/jackson-databind..."
echo -e "==================================================================================\n"
JSON_RESULT=$WORKSPACE/output/reports/github_com/FasterXML/jackson-databind/jackson-databind.json
JSON_RESULT=$WORKSPACE/output/reports/maven/com_fasterxml_jackson_core/jackson-databind/jackson-databind.json

echo -e "\n----------------------------------------------------------------------------------"
echo "FasterXML/jackson-databind: Check the e2e output JSON file with config and no dependency analyzing."
echo -e "----------------------------------------------------------------------------------\n"
JSON_EXPECTED=$WORKSPACE/tests/e2e/expected_results/jackson-databind/jackson-databind.json
$RUN_MACARON analyze -c $WORKSPACE/tests/e2e/configurations/jackson_databind_config.yaml --skip-deps || log_fail
$RUN_MACARON analyze -purl pkg:maven/com.fasterxml.jackson.core/[email protected] --skip-deps || log_fail
# Original commit f0af53d085eb2aa9f7f6199846cc526068e09977 seems to be first included in version tagged commit 2.14.0-rc1.

check_or_update_expected_output $COMPARE_JSON_OUT $JSON_RESULT $JSON_EXPECTED || log_fail

Expand All @@ -373,7 +375,7 @@ check_or_update_expected_output $COMPARE_JSON_OUT $JSON_RESULT $JSON_EXPECTED ||
# echo -e "----------------------------------------------------------------------------------\n"
# DEP_EXPECTED=$WORKSPACE/tests/dependency_analyzer/expected_results/cyclonedx_FasterXML_jackson-databind.json
# DEP_RESULT=$WORKSPACE/output/reports/github_com/FasterXML/jackson-databind/dependencies.json
# $RUN_MACARON analyze -c $WORKSPACE/tests/dependency_analyzer/configurations/jackson_databind_config.yaml || log_fail
# $RUN_MACARON analyze -purl pkg:maven/com.fasterxml.jackson.core/[email protected] || log_fail

# check_or_update_expected_output $COMPARE_DEPS $DEP_RESULT $DEP_EXPECTED || log_fail

Expand Down Expand Up @@ -654,3 +656,14 @@ then
echo -e "Expect zero status code but got $?."
log_fail
fi

# Testing the Commit Finder's tag matching functionality.
echo -e "\n----------------------------------------------------------------------------------"
echo "Testing Commit Finder tag matching functionality."
echo -e "----------------------------------------------------------------------------------\n"
python $TEST_COMMIT_FINDER || log_fail
if [ $? -ne 0 ];
then
echo -e "Expect zero status code but got $?."
log_fail
fi
Loading
Loading