Simplify the dependency tracking about polylith HOT 9 CLOSED

Can you @seancorfield please run the tap command again (follow the instructions in the #274 issue) by using the latest commit from the issue-309 branch (d744ff69bc2761dc7f608c6cf4a5661763f6bde4) and post the result here?

Last time it looked like this:

"#read-workspace, elapsed time: 18.194704958 seconds"
"#with-changes, elapsed time: 0.281075637 seconds"
"#enrich-workspace, elapsed time: 14.747228914 seconds"
"#enrich-projects, elapsed time: 14.581228245 seconds"
"#workspace-from-disk, elapsed time: 3.16617327 seconds"
"#read-projects, elapsed time: 0.081658609 seconds"
"#read-bases, elapsed time: 1.321837468 seconds"
"#read-components, elapsed time: 1.448978695 seconds"

The dependency calculations (enrich-projects) took around 14.5 seconds, and my hope is that this version will be much faster (it is for the polylith codebase).

I also want you to export the workspace again (using both the code in the main branch and the issue-309 branch) and post the two files to me in e.g. Slack, so that I can verify that the dependency calculations works as before.

I have spent a lot of time optimising the code, and I have also prepared for some more performance improvements, which I will start with when this change seems to work.

from polylith.

Wow, I ran it twice because I couldn't believe how much faster it is:

Second run:

"#read-workspace, elapsed time: 4.080951104 seconds"
"#with-changes, elapsed time: 0.269324876 seconds"
"#enrich-workspace, elapsed time: 0.845941621 seconds"
"#enrich-projects, elapsed time: 0.723650273 seconds"
"#workspace-from-disk, elapsed time: 2.965395552 seconds"
"#read-projects, elapsed time: 0.043152994 seconds"
"#read-bases, elapsed time: 1.12040886 seconds"
"#read-components, elapsed time: 1.530630654 seconds"

First run:

"#read-workspace, elapsed time: 4.191250578 seconds"
"#with-changes, elapsed time: 0.271539524 seconds"
"#enrich-workspace, elapsed time: 0.889897239 seconds"
"#enrich-projects, elapsed time: 0.751817241 seconds"
"#workspace-from-disk, elapsed time: 3.029570897 seconds"
"#read-projects, elapsed time: 0.042178307 seconds"
"#read-bases, elapsed time: 1.152617275 seconds"
"#read-components, elapsed time: 1.548293194 seconds"

I sent you the exports via DM on Slack.

from polylith.

Hi again @seancorfield,

I found some minor issues, so please DM me a new export from the issue-309 branch (045b05353adb0441b53b480b877513d70aa3db13) and I will have another look.

I'm only interested in the internal workspace structure, so you don't have to post the performance result again this time.

from polylith.

There's basically no difference in the EDN file between the previous issue-309 version I sent you and this new one. Here's the diff:

Mon Jun 19 18:57:32
(/var/www/worldsingles)-(!2076)-> diff worldsingles-ws-309.edn worldsingles-ws-309-045b053.edn
6687c6687
<   :changed-files [""],
---
>   :changed-files ["deps.edn"],
11093d11092
<        "java.time"
21992c21991
<         {:sha "d744ff69bc2761dc7f608c6cf4a5661763f6bde4",
---
>         {:sha "045b05353adb0441b53b480b877513d70aa3db13",
80597c80596
<  {:args ["ws" "out:worldsingles-ws-309.edn"],
---
>  {:args ["ws" "out:worldsingles-ws-309-045b053.edn"],
80617c80616
<   :out "worldsingles-ws-309.edn",
---
>   :out "worldsingles-ws-309-045b053.edn",
80623c80622
<   {:name "0.2.18-issue309-02",
---
>   {:name "0.2.18-issue309-03",
80627,80628c80626,80627
<    :revision "issue309-02",
<    :date "2023-06-18"},
---
>    :revision "issue309-03",
>    :date "2023-06-20"},

Mon Jun 19 18:57:53
(/var/www/worldsingles)-(!2077)->

That single change for java.time is because this import block dropped the duplication:

      :imports
      ["expectations.clojure.test"
       "java.time"
       "java.time"
       "java.util"
       "ws.date-time.impl"]}]},

from polylith.

Please try again from the issue-309 branch (2c5351b383de1d0f6330e2fed825b79fc55c7636) @seancorfield .

from polylith.

The differences with what master produces are now very minimal:

• master has java.time duplicated in an :imports list -- that's fixed on issue-309
• issue-309 has two places where a brick shows up as both a direct and indirect dependency

So I'm wondering if there's a set/difference somewhere with the first argument not being a set? Both of the "extras" are in :test dependencies.

from polylith.

When a component is only used from the test context, we merge the :src and :test dependencies into :test, and in that case we have to exclude the :direct dependencies from the :indirect dependencies after the merge, which is now fixed and pushed.

You can try the latest commit in the issue-309 branch again (e471432cf9440cde8430b0a3cf368523a9644f46) and see if this fixes the previous problems.

from polylith.

This seems to address everything and is nice and fast -- thank you!

from polylith.

Happy to hear that its nice and fast! It's merged to master, so I will close the issue.

from polylith.