feat: parse `requirements.txt` with locally referenced packages about cyclonedx-python HOT 20 CLOSED

Hey @mostafa - we've released cyclonedx-python 3.0.0 today. Would be awesome to get a PR to move to requirements-parser as you suggested.

Thanks!

from cyclonedx-python.

Hey @mostafa - thanks for getting involved.

This was absolutely my plan. However, since this ticket was created, the logic/classes that use pkg_resources now live in an upstream project cyclonedx-python - so I'll first relocate this issue (back) there.

Would be great to get a PR from you to make this change, but worth nothing we have a large(ish) release due to land probably next week now, so you might want to wait until that is merged before you branch.

from cyclonedx-python.

This issue was moved back to cyclonedx-python as since version 2.0.0 of cyclonedx-python-lib, Python specific parsers now reside in cyclonedx-python project.

from cyclonedx-python.

Perfect! Just give me a nod when you release next week. Alternatively, I can start working on the feature off the branch you want to merge and then rebase from the default branch after merging. Up to you! 🙂

from cyclonedx-python.

Tests passing here: https://github.com/CycloneDX/cyclonedx-python/runs/5277735507?check_suite_focus=true

from cyclonedx-python.

@madpah It's a dependency issue on my machine. So, my bad. 🤦

from cyclonedx-python.

We don't have locally referenced packages, but we do have a private pypi repo, and it appears that the pkg_resources library simply isn't up to the task of parsing a full pip requirements.txt file, including the --index-url directives.
#318

from cyclonedx-python.

just for the record: I do not have any opinion about a particular requirements-file-parser. Unlike @madpah I never dug deep into this topic.

regarding

Also, the pip-requirements-parser package has only one API, which is RequirementsFile.from_file, and if we want to parse requirements file as a string, we need to make our own API using internal APIs of the package or get rid of our string processing altogether.

For string input we could create a temp file and run RequirementsFile.parse/RequirementsFile.from_file on the temp file with include_nested=False, as all relative paths would be bare assumptions out of thin air based on current working dir and stuff.
For direct file input we would set include_nested=true, as relative paths from that file could be resolved.

from cyclonedx-python.

please open another issue, if you want to request another feature for the requirements parser.
this issue is about local packages.

I already did. See my post above and #318

from cyclonedx-python.

for discussions, feel free to switch to #319

from cyclonedx-python.

One more thing that's definitely off-topic:
I see lots of <file>.close() lines throughout the project inside context managers, which is redundant because the file itself is actually being opened inside the body of a with statement that is a context manager and will close the file automatically, even if an exception occurs inside the body.

from cyclonedx-python.

Root Cause: pkg_resources again does not support parsing requirements of this format (similar to requirements with hashes included) - see here.

Also - worth stating that given the example provided, we will not be able to determine the version for any locally referenced requirements through a requirements.txt method. Other methods such as EnvironmentParser would pick them up.

from cyclonedx-python.

@madpah i dont see an actual urge for this feature, and i dont see it as a bug.
relative paths are not to be expected to be output of pip freeze

but it is a convenient feature, no doubt.

from cyclonedx-python.

I agree @jkowalleck - will remove the bug label.

from cyclonedx-python.

@madpah
I was thinking about making a PR to replace pkg_resources with requirements-parser because the latter can parse all sorts of requirements. WDYT?

from cyclonedx-python.

@madpah I forked the master branch and tried to test the library locally but ran into an issue:
The tests fail because the method signature of BaseParser.get_components returns self._components which is a List[Component], but the code expects a set, so it can add to it; a method that is not supported on a list. There are 4 instances of this issue in various places. The following is an example of the tests that fail because of this error:

======================================================================
ERROR: test_conda_list_explicit_md5 (tests.test_parser_conda.TestCondaParser)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/somewhere/cyclonedx-python/tests/test_parser_conda.py", line 46, in test_conda_list_explicit_md5
    parser = CondaListExplicitParser(conda_data=conda_list_ouptut_fh.read())
  File "/somewhere/cyclonedx-python/cyclonedx_py/parser/conda.py", line 41, in __init__
    self._conda_packages_to_components()
  File "/somewhere/cyclonedx-python/cyclonedx_py/parser/conda.py", line 68, in _conda_packages_to_components
    c.external_references.add(ExternalReference(
AttributeError: 'list' object has no attribute 'add'

Is this something you're aware of? Or should I create a separate issue?

from cyclonedx-python.

@mostafa - not aware of this issue, and concerned as tests are passing in CI too. If you can raise an Issue for this, I’ll investigate later today.

Thanks again 🙏

from cyclonedx-python.

@madpah I saw the passing tests, which is why I am confused, but I'll create a separate issue for it.

from cyclonedx-python.

Tool might not supports all the features people use in their requirements.txt.
the capability is a file that is created by pip freeze. @llamahunter

please open another issue, if you want to request another feature for the requirements parser.
this issue is about local packages.

from cyclonedx-python.

@madpah @jkowalleck
I had some issues with a test (test_example_multiline_with_comments) not passing after using the requirements-parser project because it doesn't support either the hashes or multiline requirements (either or both) and I found that there's a more mature alternative, the pip-requirements-parser. WDYT? As you're the maintainer/developer, do you strictly want to use and develop the requirements-parser package?

Also, the pip-requirements-parser package has only one API, which is RequirementsFile.from_file, and if we want to parse requirements file as a string, we need to make our own API using internal APIs of the package or get rid of our string processing altogether.

from cyclonedx-python.