cljdoc-analyzer

Table of Contents

Rationale
History
Tips for library authors
- AOT Compilation
- Metadata Examples
Warning
Usage
Output format
- edn encoding
- map
Internal Workings
- Integration with cljdoc
Testing
Linting
Licenses

Rationale

A library to analyze then return public APIs for Clojure/ClojureScript projects recognizing that publics APIs can be altered programmatically at load time.

Cljdoc-analyzer takes care to:

Provide an isolated analysis environment to ensure that only project dependencies are included
Work around common issues with project dependencies that would otherwise prevent a project from being analyzed

The Internal Workings section provides more details.

Cljdoc-analyzer’s first use is for cljdoc but others might find value in this library.

History

Some Clojure/ClojureScript projects alter their APIs and related metadata at load time through libraries such as potemkin import-vars.

Codox was coded to understand these alterations for the Clojure projects it documents. Historically, cljdoc took advantage of these smarts via a branch on a fork of codox to get the API metadata it needed to document Clojure/ClojureScript project. This branch also had an unmerged PR that understood ClojureScript import-vars type metadata manipulation.

Cljdoc-analyzer picks up from the cljdoc codox unmerged PR in the internal metagetta sub-project and adds what was the cljdoc analysis-runner module under the main src.

See migration from analysis runner for more details.

Tips for library authors

AOT Compilation

AOT-compiled namespaces will lose their metadata, which means you’ll lose documentation for namespaces. Avoid having global :aot directives in your project; instead, place them in a specialized profile, such as :uberjar.

Metadata Examples

Cljdoc analyzer will recognize the following special metadata.

:no-doc

The :no-doc metadata key is a convention invented by codox to indicate that an element should not be included in API documentation.

Examples:

;; Documented
(defn square
"Squares the supplied number."
[x]
(* x x))

;; Not documented
(defn ^:no-doc hidden-square
"Squares the supplied number."
[x]
(* x x))

:no-doc can also be used at the namespace level:

For example:

(ns ^:no-doc hidden-ns)

For autodoc legacy reasons, codox considers :skip-wiki to be equivalent to :no-doc and cljdoc-analyzer does the same.

:added

To denote the library version the var was added in, use the :added metadata key:

(defn square
"Squares the supplied number."
{:added "1.0"}
[x]
(* x x))

:deprecated

Similar to :added, deprecated vars can be denoted with the :deprecated metadata key:

(defn square
"Squares the supplied number."
{:deprecated "2.0"}
[x]
(* x x))

Warning

Cljdoc-analyzer is not a static code analyzer. It is interested in the result of programmatic load time alterations. Use the same judgement you would when using any 3rd party library. If you don’t trust it, don’t cljdoc-analyze it.

Usage

cljdoc use

Cljdoc-analyzer’s first customer is cljdoc. This usage does not cater to general usability. Cljdoc started with conventional command line arguments but then switched to edn because it made more sense for its use case.

Example of analyzing cljfmt v0.6.4:

clojure -m cljdoc-analyzer.cljdoc-main \
  '{:project "cljfmt/cljfmt"
    :version "0.6.4"
    :jarpath "http://repo.clojars.org/cljfmt/cljfmt/0.6.4/cljfmt-0.6.4.jar"
    :pompath "http://repo.clojars.org/cljfmt/cljfmt/0.6.4/cljfmt-0.6.4.pom"
    :extra-repos {"clojars" {:url "https://repo.clojars.org/"}
                  "central" {:url "http://central.maven.org/maven2/"}}}'

The :extra-repos options is somewhat contrived as these repos already exist in cljdoc-analyzer’s default config, but this does reflect current cljdoc usage.

This will log to stdout and, if successful, write to a file in a predefined known spot, as indicated in the output logs:

2019-08-26 14:22:28,061 INFO  cljdoc-analyzer.runner - results file: /tmp/cljdoc/analysis-out/cljdoc-edn/cljfmt/cljfmt/0.6.4/cljdoc.edn

general use

You can use cljdoc-analyzer ad hoc to get data for a project published to a maven repo. For example:

clojure -m cljdoc-analyzer.main analyze \
  --project io.aviso/pretty --version "0.1.29" \
  --output-filename "io-aviso-pretty-0.1.29.edn"

On successful completion, you’ll find the output in the current directory in io.aviso-pretty-0.1.29.edn

When you are working on a local project, publish it to your local maven repo first, then run the cljdoc analyze command.

If you want to suppress items that have been marked to be excluded from documentation, use --exclude-with. To match cljdoc usage, you would exclude namespaces and publics tagged with :no-doc and/or :skip-wiki, and also use the --extra-repo option:

clojure -m cljdoc-analyzer.main analyze \
  --project io.aviso/pretty --version "0.1.29" \
  --output-filename "io-aviso-pretty-0.1.29.edn" \
  --exclude-with :no-doc \
  --exclude-with :skip-wiki \
  --extra-repo "clojars https://repo.clojars.org/" \
  --extra-repo "central http://central.maven.org/maven2/"

We can look at other features as we get a feel for what folks are interested in.

logging

If using cljdoc-analyzer as a library, provide your own logging config as appropriate for your app. A sample logback.xml config that logs to stdout can be found under resources.

Output format

The output is a map of namespaces and their publics.

edn encoding

The edn has a twist. Function arglists can sometimes contain regular expressions as desconstructed default values. Since edn does not support deserializing serialized regular expressions, we adapt by serializing regexes as #regex followed by the string version of regex. For example:

#".*booya.*"

is serialized as:

#regex ".*booya.*"

See cljdoc-analyzer.analysis-edn/serialize and cljdoc-analyzer.analysis-edn/deserialize.

map

The edn output is a map of:

:group-id project group-id
:artifact-id project artifact-id
:version project version
:analysis analysis for languages which can consist of a map with none, one or both of:
- clj list of namespaces (see below)
- cljs list of namespaces (see below)
:pom-str slurp of maven pom file

list of namespaces is a list of maps of:

:name namespace name
:doc namespace doc string
:author namespace author
:publics namespace publics which is a list of maps of:
- :name public element name
- :type one of: :macro :multimethod :protocol :var
- :doc doc string
- :file file relative to jar root
- :line line number
- :arglists list of vectors of arglists, omitted for def record and protocol elements
- :members only applicable when :type is :protocol, list of maps of:
  - :arglists list of vectors of arglists
  - :name name of protocol method
  - :type can this be only :var?

special metadata tags when present are included in publics:

:dynamic for dynamic defs

special metadata tags when present are included on namespaces and/or publics:

:added version an element was added
:deprecated version an element was deprecated
:no-doc author requests that this item be excluded from docs
:skip-wiki legacy synonym for :no-doc, please use :no-doc.

Internal Workings

We use clojure.tools.namespace and cljs.analyzer.api to load source and collect metadata. This requires the loading of a project’s dependencies. To avoid dependency conflicts and confusion, we keep dependencies at a minimum during metadata collection time by splitting the work into two distinct phases.

Prepare for analysis - the source for this work can be found under src. Here we do everything we can to prepare for metadata collection.
Collect metadata - the source for this work can be found under metagetta. A separate metagetta process is launched to collect metadata on sources prepared in step 1.

Here’s an overview diagram:

To understand how this fits in the bigger cljdoc picture see the the system overview in the cljdoc project.

Integration with cljdoc

Cljdoc passes the cljdoc-analyzer dependency to the CircleCI cljdoc analysis job.

This dependency is currently a :git/url dependency. To bring a new version of cljdoc-analyzer into production, we update the analyzer-version to the git :sha we’d like to use. At the time of this writing, this is specified in cljdoc.analysis.service.

Testing

We make use of kaocha for testing.

Metagetta is a separate sub-project with its own unit tests. To run metagetta unit tests:

cd modules/metagetta
clojure -A:test

Cljdoc-analyzer has integration and unit tests, to run them all, ensure you are in cljdoc-analyzer root dir and:

clojure -A:test

The integration tests can take a while to run, to run unit tests only:

clojure -A:test unit

To run integration tests only:

clojure -A:test integration

To automatically rerun tests on any changes, tack on a --watch to any of the above commands. See kaocha docs for other options.

Linting

We make use of clj-kondo for linting.

Our build server validates the code is lint free with script/lint and so can you.

Licenses

Otherwise EPL-2.0 see LICENSE

kingmob / cljdoc-analyzer Goto Github PK

cljdoc-analyzer's Introduction