Hi, @phronmophobic.

I'm trying to understand the data that dewey provides. I noticed that there is a :lib key that - I presume - holds a unique identifier for each project, in a format like io.github.phronmophobic/dewey. Could you please explain the rationale for this format?

https://github.com/phronmophobic/dewey/blob/2318c3a0d08ea557e12278f2927e20b4b2fbe8a1/src/com/phronemophobic/dewey.clj#LL276C40-L276C75

Heya, I wanted to make a deps autoloader that would catch errors about missing ns and auto import them using clojure 1.12's clojure.repl.deps/add-libs.

I think that information is only on Dewey's larger release artifacts, and downloading all those files is a tough proposition.

Ideally I'd like to end up with something like

{clojure.data.csv {:git/url "https://github.com/clojure/data.json"
                   :git/sha "c323f899a06653af9d66a8e0212b65d0ac6f7b7f"
                   :git/tag "v1.1.0}
;; more ns -> dep mappings
}

Do you have any thoughts or suggestions on how to end up with this data? We talked briefly on slack and you already suggesting new build artifacts and an API.

The deps-libs data includes :lib key. Other data should as well.

Hi @phronmophobic,

Thank you for maintaining dewey. I'm trying to incorporate some of the provided data and retrieval pathways into my projects. Would you consider updating the docs to include the fields and their meanings included in each pre-computed dump? I think this would help make dewey more approachable and would remove the need to run the project and call com.phronemophobic.dewey.util/read-edn to see what's inside.

I know that the formatting of Dewey's data dumps is still a work in progress and open to change, so I wanted to flag some concerns and potential areas for improvement.

It looks like at the moment the formatting of the pre-retrieved data dump files is not consistent:

• analysis.edn.gz: first line is opening bracket [ followed by one project's analysis map {...} per line
• all-repos.edn: single line containing ({...} ...)
• deps-libs.edn: single line containing a map keyed by lib
• deps-tags.edn: single line containing a vector of 2-element vectors:
  • first element: map of repo information
  • second element: vector of maps, where each map represents a git tag of the repo

These differences create a bit of friction when consuming the data. It would be nice to either unify the format somehow, provide clear documentation about the differences, or add some helpers like read-edn to assist those trying to consume the data (or a combination of all three).

Related: read-edn does not currently work as documented in the README, because it consumes too much CPU and takes too long for certain files, like analysis.edn.gz.

As for the formatting, it would be nice if all of the provided data would be easy to join. Making the :lib key ubiquitous, as mentioned in #3 would be a step in the right direction. If all the data could be keyed by :lib, it would be better still, allowing lookups without iterating sequences to find the data to join. I don't know exactly how to achieve that without requiring the whole map to be loaded into memory first. Perhaps SQLite can be part of the solution?

We grab the latest commit of the default branch in default-branches.edn. We should add that info to deps-libs.edn since not every repo has a tagged commit to make a version from.

Hi @phronmophobic,

You've mentioned that some of the effort you are expanding on Dewey right now is around the repeatability of the analysis process. I'd like to suggest that when a Clojure repo is analyzed, the resulting analysis data should include the repo's commit hash on which the analysis was performed. I think the benefits of this one additional piece of data are fairly clear, but I could expand on my rationale if the use case is not evident. Do you have any thoughts on this idea?

Why is thgis in the list ?

It has no Clojure related keyword :
https://github.com/tonsky/FiraCode

The clj-kondo analysis dump can include one or more entries per repo, depending on how many bases (deps.edn, project.clj, etc.) the project has. The difference is represented through a top-level :basis key in each of the maps in the file.

Since the :basis key is not a part of clj-kondo's own output, I think it would be helpful to document it for Dewey users, and to explain its impact on the cardinality between repos and analysis maps in the pre-computed analysis.edn dumps. I initially assumed that each repo would have just a single analysis entry.

The current analysis data doesn't include https://github.com/clojure/clojure. We had a quick discussion about it on Slack, and you mentioned that it might be a good idea to add it. However, there were no specific commitments made regarding the timing. Just opening this issue as a reminder. Thanks, @phronmophobic!

Hi @phronmophobic

and thanks for many great libraries and code!

I was able to get 4431 repositories with 2 stars using this code:

(def base-request
  {:url          search-repos-url
   :method       :get
   :as           :json
   :query-params {:per_page 100
                  :sort     "updated"
                  :order    "desc"}})

and

(defn find-clojure-repos []
  (iteration
    (with-retries
      (fn [{:keys [url pushed_at last-response] :as k}]
        (prn (select-keys k [:url :pushed_at]))
        (let [req
              (cond
                ;; initial request
                (nil? k) (search-repos-request "language:clojure stars:2")
                ;; received next-url
                url (assoc base-request :url url)
                ;; received star number
                pushed_at (search-repos-request (str "language:clojure stars:2 " "pushed:<=" pushed_at))
                :else (throw (Exception. (str "Unexpected key type: " (pr-str k)))))]
          (rate-limit-sleep! last-response)
          (let [response (http/request (with-auth req))
                prev-items (into #{} (get-in last-response [:body :items] []))]
            (-> response
                (assoc ::key k
                       ::request req)
                (update-in [:body :items]
                           (fn [items]
                             (vec (remove (partial contains? prev-items) items)))))))))
    :kf
    (fn [response]
      (let [url (-> response :links :next :href)]
        (if url
          {:last-response response
           :url           url}
          (when-let [pushed_at (some-> response
                                       :body
                                       :items
                                       last
                                       ;; want to continue from where we left off
                                       :pushed_at)]
            {:pushed_at     pushed_at
             :last-response response}))))))

you'll notice that here pushed_at is used for searching and sort-by updated is used.

The search is done using pushed:<= ... so that identical updated timestamps is supported (this also introduces the need to remove items from the previous request, in order to not loop forever).

Is this something that you would like to integrate into dewey?

Thanks and kind regards!

Hi, @phronmophobic. Would you be open to expanding the config map which Dewey passes to clj-kondo?

My code is relying on {:var-definitions {:meta [:arglists]}} for handling some edge cases, particularly in Clojure core and the associated tests, because some of the definitions there are a little unusual. In some of the core definitions the arglist is defined only in meta and in some of the tests there are intentionally invalid definitions, and I check for empty :arglist-strs and empty :meta {:arglists ...} to remove those entries.

In general, I'm very much interested in delegating some of the data extraction requirements in my project to Dewey, but if there's a gap between my requirements and the provided data, I basically have to re-run the analysis on my end.

I guess this issue is two questions:

1. would you be open to adding {:var-definitions {:meta [:arglists]}} to the current analysis config?
2. do you have any thoughts about addressing such requirement gaps in general? perhaps it would be more pragmatic to ask clj-kondo for everything it is able to provide? I'm not sure if that would have other implications for performance or something else as you compile the pre-computed data sets