dbuenzli / topkg Goto Github PK

This should be done quickly as it will change the build instructions of packages.

Currently to derive the build context we rely on the --installer and --vcs option (the latter being automatically discovered). Packages have --installer true in their OPAM build instructions which allows to detect pins when --vcs is also discovered to be true.

However this will not work well with the upcoming opam package build command which reads and execute the build: field of an OPAM file. If invoked in the development repository, since we have --installer true in those this will detect a pin build context and thus perform the watermarking process and mess the development repo.

We should deprecate --installer immediately, have a --pinned option that explicitely indicates that we are being pinned and combine with --vcs to decide if the watermarking process should be performed.

The build instructions then become:

build: [[
"ocaml" "pkg/pkg.ml" "build" 
        "--pinned" "%{pinned}%"
]]

Rather than :

build: [[
"ocaml" "pkg/pkg.ml" "build" 
        "--installer" "true"
]]

and opam package will simply build as a topkg build invocation would do.

Currently topkg build clean simply erases the build directory and PKG.install. This may however miss the symbolic links ocamlbuild generates. Maybe we should add a ?clean option to build description which by default would simply ocamlbuild -clean.

For now the full browser reload strategy is only supported on osx with chrome and safari. Help for extending support along the documentation is welcome.

• osx support for firefox and other browsers
• linux support beyond xdg-open
• windows everything.

those loops for ever
write_subst "/tmp/asd" [] "\"%%a%%\""
write_subst "/tmp/asd" [] "%%a"

quickly fixed (too quickly ??) with https://github.com/ocsigen/js_of_ocaml/pull/156/files#diff-2d8a8d7d55d223fe187efdf03f6ca050R82

Would be a nice command to have in the future.

It should basically work like topkg publish i.e. with either a delegate script or using GitHub's API (except here the target should simply be automatically selected according to the bug-report: field of the opam file).

The output should be similar to topkg status. Simply listing each issue with their number and title.

This issue tracks a few things that could be improved with opam v2.0.

• OPAM package description. Lookup in the new descr: field aswell.
• Archive distribution URI pattern, could be moved to an x-distrib-uri field in the OPAM file.
• In pin distribution watermarking replace the IPC to the topkg binary by calls to opam (if it includes an opam query-like functionality). This would avoid incorrect watermarks if topkg-care happens to be uninstalled.
• OPAM watermarks, support for watermarking with x- fields.
• Dep linting, remove the ocaml dependency.

A few Sys_error e need to be prefixed by the file path to avoid obscure error message.

/cc @samoht

Nice packaging system! I ran into a minor issue, though: The LGPL license is normally shipped as two separate files COPYING and COPYING.LESSER. I don't think this can be declared with the current interface. Some long running projects may also want to split up their change log into multiple files. How about adjusting the Pkg.doc field to handle these, as an alternative to the keyword arguments? Something like

type doc_field : ?role: [`Doc | `License | `Change_log] -> field
(* ... *)
val doc : doc_field

which of course defaults to ``Doc`.

Adding a ?licenses: std_file list parameter to describe would work too.

This commit 605fc11 introduced an issue that prevents compilation of some topkg packages under Cygwin on Windows. Specifically, the lines https://github.com/dbuenzli/topkg/blob/master/pkg/topkg.ml#L155-156 should test whether Sys.cygwin is true to select the right extention when compiling under Cygwin.

Tried installing topkg having just had homebrew upgrade my system compiler to 4.03.0. Seem to get conflicts -- isn't topkg-care supposed to be independent of topkg (I may have misread the mail you sent though)

: mort@greyjay:~$; opam install topkg-care

=-=- Synchronising pinned packages =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
[topkg-care] https://github.com/dbuenzli/topkg.git already up-to-date
The following actions will be performed:
  ∗  install topkg      *                     [required by topkg-care]
  ∗  install topkg-care *
===== ∗  2 =====
Do you want to continue ? [Y/n] y

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
[topkg-care] https://github.com/dbuenzli/topkg.git already up-to-date
[topkg] https://github.com/dbuenzli/topkg.git already up-to-date

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
[ERROR] The compilation of topkg failed at "ocaml pkg/pkg.ml build --pkg-name topkg --installer true".

#=== ERROR while installing topkg.~unknown ====================================#
# opam-version 1.2.2
# os           darwin
# command      ocaml pkg/pkg.ml build --pkg-name topkg --installer true
# path         /Users/mort/.opam/system/build/topkg.~unknown
# compiler     system (4.03.0)
# exit-code    2
# env-file     /Users/mort/.opam/system/build/topkg.~unknown/topkg-36614-d0f245.env
# stdout-file  /Users/mort/.opam/system/build/topkg.~unknown/topkg-36614-d0f245.out
# stderr-file  /Users/mort/.opam/system/build/topkg.~unknown/topkg-36614-d0f245.err
### stdout ###
# Cannot find file topfind.
### stderr ###
# Unknown directive `require'.



=-=- Error report -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
The following actions were aborted
  ∗  install topkg-care ~unknown
The following actions failed
  ∗  install topkg ~unknown
No changes have been performed

Installed and pinned packages are:

: mort@greyjay:~$; opam list
# Installed packages for system:
astring                    0.8.1  Alternative String module for OCaml
base-bytes                  base  Bytes library distributed with the OCaml compiler
base-unix                   base  Unix library distributed with the OCaml compiler
bos                     ~unknown (pinned)
camlp4               4.03+system  Camlp4 is a system for writing extensible parsers for programming languages
cmdliner                   0.9.8  Declarative definition of command line interfaces for OCaml
conf-m4                        1  Virtual package relying on m4
conf-ncurses            1+system  Virtual package relying on ncurses
conf-perl                      1  Virtual package relying on perl
cppo                       1.3.2  Equivalent of the C preprocessor for OCaml programs
cstruct                    2.1.0  access C structures via a camlp4 extension
cudf                         0.7  CUDF library (part of the Mancoosi tools)
dose                  3.2.2+opam  Dose library (part of Mancoosi tools)
extlib                     1.7.0  A complete yet small extension for OCaml standard library (reduced, recommended)
fmt                        0.7.1 (pinned)  OCaml Format pretty-printer combinators
fpath                   ~unknown (pinned)
functoria                  1.1.0  A DSL to organize functor applications.
io-page                    1.6.0  Allocate memory pages suitable for aligned I/O
ipaddr                     2.7.0  IP (and MAC) address representation library
jsonm                      0.9.1  Non-blocking streaming JSON codec for OCaml
logs                       0.5.0 (pinned)  Logging infrastructure for OCaml
lwt                        2.5.2  A cooperative threads library for OCaml
mirage                     2.9.0  The MirageOS library operating system
mirage-types               2.8.0  Module type definitions for Mirage-compatible applications
mirage-types-lwt           2.8.0  Lwt module type definitions for Mirage-compatible applications
ocamlbuild                 0.9.2  OCamlbuild is a build system with builtin rules to easily build most OCaml projects.
ocamlfind                  1.6.2  A library manager for OCaml
ocamlgraph                 1.8.5  A generic graph library for OCaml
ocplib-endian                0.8  Optimised functions to read and write int16/32/64 from strings and bigarrays, based on new primitives added in version 4
opam-lib                   1.2.2  The OPAM library
ppx_core          113.33.01+4.03  Standard library for ppx rewriters
ppx_deriving                 3.3  Type-driven code generation for OCaml >=4.02
ppx_driver        113.33.01+4.03  Feature-full driver for OCaml AST transformers
ppx_optcomp       113.33.00+4.03  Optional compilation for OCaml
ppx_sexp_conv     113.33.01+4.03  Generation of S-expression conversion functions from type definitions
ppx_tools             5.0+4.03.0  Tools for authors of ppx rewriters and other syntactic tools
ppx_type_conv     113.33.01+4.03  Support Library for type-driven code generators
re                         1.5.0  RE is a regular expression library for OCaml
result                       1.2  Compatibility Result module
rresult                    0.3.0 (pinned)  Result value combinators for OCaml
sexplib           113.33.00+4.03  Library for serializing OCaml values to and from S-expressions
uutf                       0.9.4  Non-blocking streaming Unicode codec for OCaml
: mort@greyjay:~$; opam pin
bos.~unknown                         git  https://github.com/dbuenzli/bos.git
fmt.0.7.1                            git  https://github.com/dbuenzli/fmt.git
fpath.~unknown                       git  https://github.com/dbuenzli/fpath.git
logs.0.5.0                           git  https://github.com/dbuenzli/logs.git
rresult.0.3.0                        git  https://github.com/dbuenzli/rresult.git
topkg.~unknown        (uninstalled)  git  https://github.com/dbuenzli/topkg.git
topkg-care.~unknown   (uninstalled)  git  https://github.com/dbuenzli/topkg.git

Depending on the directives use by the package description, errors may lead to IPC corruption:

topkg: [ERROR] Decode (package) v0: corrupted (package) v0 in
               "topkg" Input data: "Cannot find file topkg_huhu.ml.\n0\000topkg"

This is due to http://caml.inria.fr/mantis/view.php?id=7202 and nothing can be done about it.

A test between linux and osx shows discrepancies. The problem seems to stem from file permissions/umask issues.

Cloning the same repository, on osx, the files are not group writable while they are on linux. Currently we simply embed what we find on the disk.

Should we simply care about the executable bit and simply mark the files as rw for user and r for the group and the others ? Also, look into what git archive does.

Currently installing ppx binaries in lib leads to inacceptable relative hacks in .install files.

The function should also filter out directories since Topkg_vcs.tracked_files can return symlinks that may point to directories.

/cc @samoht

Currently a call to topkg build CONF..., does not remember the build configuration, this means that for example we cannot test if the tests are up-to-date when we do a topkg test since we no longer know the build system invocation. I suspect some frustration may come out of this in the future with people needing to do a topkg build && topkg test or even a topkg build CONF... && topkg test which means that we need again ad-hoc side shell-scripts and/or makefiles.

But this involves introducing STATE, iiiiik.

I get errors abount commands not found when running topkg, e.g.

$ topkg doc
topkg: [ERROR] ocamlbuild: no such command

The problem is that Topkg_os.Dir.test_cmd tries to test for the presence of commands by executing type, which is a shell built-in. The which command could be used instead.

One should not improve toys through hacks, but we could try to steal opam-publish's github token if it exists to improve the experience.

It shouldn't be too hard to support the simple cases which will cover 98% of the packages out there. However we'll need an opam with ocaml/opam#2254 to add some data to guide the process.

The problems is of course detecting them. In carcass this is done looking for a NULL byte. But this may not be totally fool proof. Maybe it's better to automatically exclude a few extensions in the default Pkg.files_to_watermark function.

Once ocaml-opam/opam-publish#42 is in.

See https://github.com/mirage/mirage-http/pull/25

$ topkg lint
[ ... ]
[FAIL] File opam and _tags dependency check:
       channel: opam present but _tags absent
       cohttp: opam present but _tags absent
       lwt: opam present but _tags absent
       mirage-conduit: opam present but _tags absent
       mirage-types-lwt: opam present but _tags absent

Some of them are false positive because name of ocamlfind library and opam package is different (and I don't really expect topkg to guess that) but the rest is just because I'm using _tags and src/_tags` files.

Seems that it's possible to specify boolean flags (with Env.bool) which is great. I'm just wondering what's your plans (if any) to ensure that these flags are set properly (no typo or none forgotten) in the corresponding opam file.

btw, I'm very happy to see this project, I'll give it a proper try soon.

Currently the build command description is:

?cmd:(Topkg.Conf.t -> Topkg.Conf.os -> Topkg.Cmd.t)

and the resulting command is given the paths to build. This however means that we are imposing an API and the build conventions of ocamlbuild on the command itself.

If we give the paths to build:

?cmd:(Topkg.Conf.t -> Topkg.Conf.os -> Topkg.fpath list -> Topkg.Cmd.t)

Other build system can adapt ocamlbuild's conventions to theirs without needing a shell-script wrapper to map to the idiom of their build system. And they can simply #require an adapter from a package without the need for shell script wrappers.

This should be designed in conjunction with clean (#52) and be done quickly AFAIK this wouldn't break any published topkg package at the moment.

E.g. see here.

To test I pinned topkg to my local checkout with opam pin add topkg .. The installed META still contains %%VERSION_NUM%%. I think the reason is that opam creates a copy of the source directory without the .git subdirectory, so vcs check in build_context fails.

There might be some issues with embedding the topkg.ml file into other repos which might have conflicting licenses (IANAL). Is it possible to somehow allow topkg to be an OPAM dependency?

The build, test and clean commands simply forward by invoking the corresponding commands in pkg/pkg.ml. One result of this is for example that we need to specify two -- tokens to be able to specify arguments to a test in topkg test.

It seems that defining ipcs for these and go through pkg/pkg.ml ipc could be cleaner.

Here like in #18 we need ocaml/opam#2254. Once this is in, the readme, license and change_log specifications of the std_files should deprecated in favour of appropriate opam x-fields since opam doesn't provide fields for this (ocaml/opam#733, ocaml/opam#1900).

Dunno if this is intentional :)

: mort@greyjay:~$; topkg help
...
BUGS
       Report them, see UNDEFINED for contact information.

See https://ci.appveyor.com/project/docker/datakit/build/1.0.112#L621

ocaml "pkg/pkg.ml" "build" "--pinned" "true" "--with-github" "disable"

Just went fine, but tried to build the github bindings instead of just failing to parse disable.

As noted in the man page:

    `P "$(b,WARNING.) The way this command works is at the moment
        moment very ad-hoc and ocamlbuild specific. It will
        change in the future.";

The future involves odoc. No plans to change this before.

From some other bugs on other trackers (I can't seem to find it...), I have a feeling you already know about this but I thought I'd leave this here so others may know:

#=== ERROR while installing topkg.0.7.1 =======================================#
# opam-version 1.2.2
# os           darwin
# command      ocaml pkg/pkg.ml build --pkg-name topkg --installer true
# path         /Users/dsheets/.opam/system/build/topkg.0.7.1
# compiler     system (4.02.3)
# exit-code    2
# env-file     /Users/dsheets/.opam/system/build/topkg.0.7.1/topkg-64213-e7966f.env
# stdout-file  /Users/dsheets/.opam/system/build/topkg.0.7.1/topkg-64213-e7966f.out
# stderr-file  /Users/dsheets/.opam/system/build/topkg.0.7.1/topkg-64213-e7966f.err
### stdout ###
# Cannot find file topfind.
### stderr ###
# Unknown directive `require'.

Or not, maybe we could leave this to the OPAM file.

One of the steps topkg takes to gradually introduce the idea of using OPAM files as the package metadata format (also outside OPAM managed installs) is to have them in the distribution and automatically install them in $(opam config var PKG:lib). One of the problem at the moment though is that these files lack the distribution version number.

Here are a few ways this could be solved:

1. Simply have version: "%%VERSION%%" in the OPAM files in the dev repo, this will be substituted appropriately by the watermarking process. This has the following problems:
   1. We currently simply use the opam file of the repo on topkg opam pkg so the field would need
      to be filtered out. I'm not very keen on having to butcher OPAM files, although the v2 API may help
      with this.
   2. I don't remember the details but I think this would affect the pinning process we'd get test.%%VERSION%% pins which would certainly break version comparison and be very unpleasant to the eye. If that's the case then this solution is ruled out.
2. Hack. We could simply append a proper version: field after the watermarking ran and before invoking the user distribution massage hook, to all OPAM files mentioned in Pkg.describe before running the watermarking process.

see ocsigen/reactiveData@e4913cf

Rather than rely on ocamlbuild's source copy rule and unconditionally generates install moves that source from the _build dir. For example it seems that ocamlbuild can't build source files that start with an _.

Windows support should be tested along two lines

1. Correctness of package installation using topkg. This is essential.
2. topkg the helper tool works correctly on Windows, this may uncover potential bugs in bos.

Discussion about policies whether debug should be enabled in releases always ended up inconclusively. But were leaning towards that they should not.

For a long time I have relied on not having true : debug in my _tags and invoke build commands through a shell script for development which would add -tag debug.

With topkg's command set I think I can get rid of these scripts in most of my packages --- and in carcass' topkg bodies. But this problem remain. Here's a way to solve it:

Add a --debug configuration key which, if unspecified, is determined according to build context (i.e. true if Dev and false on Pin and Release). The default build command will add the -tag debug according to whether the configuration key is enabled or not.

To determine what happens on topkg publish. Currently fixed to topkg publish doc distrib which may be suboptimal for certain project (or want to automate alt publication targets) and prevents to use topkg bistro which invokes topkg publish as such.

I just upgraded to topkg 0.7.4 to get shiny new testsuite-building abilities. I forgot to upgrade topkg-care, which crashed when topkg-care 0.7.3 tried to talk to topkg 0.7.4.

Should tokpg-care use an == dependency?

Competes with #18. Another source of error is wrong require field in META files. We could try to lint them the same way we do dependency linting betwen the _tags file and the opam file.

ocamlfind supports a scheme where executables are installed to lib/ and then invoked via ocamlfind pkg/executable. In particular this is the recommended (by Alain, etc) way to install ppx rewriters.

ocamlbuild provides rules to generate documentation directory from *.odoclfiles.
However, the target you must provide to ocamlbuild is not the directory itself:
*.odocl -> *.docdir/index.html
What are the options to install documentation with topkg + ocamlbuild ?