julialang / juleps Goto Github PK
View Code? Open in Web Editor NEWJulia Enhancement Proposals
License: Other
Julia Enhancement Proposals
License: Other
The /usr
directory cannot contain application-specific directories. Also, things under /usr
should generally be managed exclusively by distro package managers. So a better place (according to the FHS) would be /usr/local/share/julia
for architecture-independent files (like sources), and /usr/local/lib/julia
for binaries.
I have a private package kept with github, using their fee-enabled private setting. To use it locally, first I clone it and thereafter I use checkout. Checkout prompts me for the username and password. My username and password are stored using git's credentials manager.
Pkg3's clone and checkout should find them.
We should also fully document all the steps in making a release (e.g. https://github.com/JuliaLang/julia/blob/883c8a38920985e0c02df169ff8c379731d88fc6/Makefile#L112-L129)
Currently I'm organizing the packages which I develop by copying them to a folder with sym-links to .julia
, see https://discourse.julialang.org/t/recommended-setup-for-developing-packages/5725 for a discussion. I then started wondering how this would work with Pkg3 but did not find anything in the document, but got a bit confused how package-immutability and development would work together.
Maybe a section "Package development" should be added to:
https://github.com/JuliaLang/Juleps/blob/master/Pkg3.md#operations
It would be nice to have Transpiler from subset of Python (and special ~pythonjulia module named like "Juthon") to complete Julia lang.
And a dream looks like it should both be:
A good example of a transipiler from Python is Transcrypt that is a valid Python prior transpiling. But to achieve 1. this should be more like Julia in Python syntax (via IDE friendly mapping and juthon module with well-documented stubs) than attempt to implement Python's standard library.
The idea came from this discussion: Why Julia? Will Python/Numba and Python/Cython lose to Julia?
Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, [...] and takes a purely functional approach to package management
https://www.gnu.org/software/guix/
https://www.gnu.org/software/guix/manual/html_node/Package-Management.html
Guix is implemented in Scheme. Julia uses Scheme as its parser. All the ideas of Guix mentioned in the above quote appear in line with those of Julia.
I don't know if Pkg3 should be "like Guix", but I happen to be using, and interested in the development of, both Julia and Guix. I wanted to be sure that all Pkg3 designers are well aware of Guix/Nix and possibly can use ideas/features (and maybe code) from the Guix project, or even collaborate.
[sorry for wall of text - leaving town this weekend, want to write initial reactions down while fresh]
A package should not declare compatibility with a minor version series unless some version in that series has actually been published – this guarantees that compatibility can (and should) be tested. If a new compatible major or minor version of a package is released, this should be reflected by publishing a new patch that expands the compatibility claims.
So this is requiring upper bounds, and in a strict way where the bound must already exist (though unlike current upper bounds, these would be inclusive). While I like that idea in theory - only allowing package dependency versions to be installed that are known to work - it trades the "the new release of package A broke package B" problem for a smaller feasible set of allowed versions. Users of widely depended-on packages will be held back to old versions if they also want to use any dependent packages that update slowly. If both package B and package C depend on package A, package B hasn't tagged since A released version 1.4, and package C depends on a new feature in package A that was first released in version 1.8, you won't be able to use both package B and package C in the same environment until package B has tested and tagged a 1.8-compatible version.
Automatic testing of reverse dependencies and making an automatic set of new downstream tags with wider bounds if tests pass could help here, but we don't have that infrastructure yet (and requiring such infrastructure in order for a set of packages to progress cohesively may be a burden to place on organizations that want to run their own registry). For packages that can't be tested on CI, or start failing in the automatic test results, then you start needing to involve the authors of even sporadically-developed packages any time their dependencies put out new feature releases that people want to use.
We'd also need much better error messages and suggested fixes when dependency resolution fails to find a feasible set of versions. Resolution failure is luckily pretty rare right now, but can be very confusing when it happens. Downgrading or being held back to old versions does happen now with upper bounds, and being stricter about them would make that more common. If a set of versions that are known to work can be installed we should do that, but I fear the choice will often be between allowing untested versions to be installed or erroring when the user tries to do so. When those are the only choices, I think the former has a higher chance of allowing the user to get things done (or figure out how to fix the problems).
https://www.well-typed.com/blog/2014/09/how-we-might-abolish-cabal-hell-part-1/ is worth at least skimming, as Haskell's ecosystem has gone through many similar issues. By going from the current scheme of only applying upper bounds when problems are already known or the package author is choosing to be conservative, to a scheme where strict tested bounds are the only thing that's allowed to be released, we're moving from the Julia equivalent of "Type errors" or "Compile errors" to the equivalent of "solver failure."
Lastly, this makes package authors' adherence to semver (or lack thereof) much more consequential. We do want to encourage these processes and get people thinking more about them, but I think social expectations and more thorough documentation are safer ways to get there right now than baking them into the behavior of the package manager.
When the user wants to remove package or update packages from an environment, they will instead invoke an external package management mode (julia --pkg?), which makes it clear that changes will not affect any currently running Julia sessions. The impact on usability is a strict improvement:
- Adding packages and loading them is easier since one simply does using XYZ and answers interactive prompts.
- Removing and upgrading packages is no less difficult since it previously required restarting the current Julia process anyway, and is less confusing since the requirement to restart is explict since running a separate process clearly doesn't affect the current one.
Using the command line to manage packages sounds like a usability loss to me. This will particularly be the case for Windows users.
I think we should keep the current Pkg.*
functions. They should simply print a warning or ask to restart when modified packages are currently loaded.
Questions to answer in further revisions:
Will installed packages be read-only (at least until deletion is called for), or strictly checksummed, or immutable by convention only? What consequences, if any, would modifications have? Using code that doesn't match what the manifest specifies was installed isn't good for reproducibility, but what's the intended level of tracking and granularity of this? If packages aren't always git repositories any more, how are generated files and downloaded resources, which would be ignored from a version control perspective, dealt with?
This ties into the bigger separate question of where and how development happens if everything Pkg3 touches is immutable (by convention only, or strictly enforced). If you want to make a local change, does that require a separate installation mechanism and modifiable copy that lives outside of Pkg3 somewhere? Or if you make it locally do you then have untracked local modifications that never get recorded anywhere? (People will forget they've made this kind of change if packages aren't git repos.) Most other package systems work this way, but most other package systems have an unfriendly distinction between the way users work with packages and the way developers/contributors do. The low barrier to entry of contributing to Julia packages is a huge benefit to our ability to get users to become developers.
It would be nice to have uniform way to specify package configuration options. Currently, I'm using environment variables for this (e.g. PYTHON in PyCall), but that is somewhat non-Julian. More importantly, it requires some manual effort for me to save the option in a file so that it is "remembered" when you do Pkg.update
etcetera even if the environment variable is no longer set. Many packages will get this wrong, and even if they get it right it is a lot of duplication of effort.
My suggestion would be that the package TOML file should have something like:
[option.python]
value = "python3" # default
The user interface would be something like:
Pkg.option("PyCall", :python)
— return the current choice (saved in a file somewhere)Pkg.option("PyCall", python="python2")
— change the option, maybe rebuilding automatically unless this is called by the package during the buildPkg.add
and Pkg.build
will accept python="python2"
keyword arguments to set options when installing/building.The value of all options would be saved in an options.toml
file somewhere.
A related mechanism would be used for package alternatives, see #37.
Taken from JuliaLang/julia#16330:
It would be nice if it would be possible to have packages where the functionality is only available in precompiled form.
Originally posted as comment on commit, suggested to open issue instead.
cc @nalimilan
Extract from section Issues Beyond the Scope of This Julep
findfirst(x, v)
returns 0 if no value matching v
is found; however, if x
allows 0 as an index, the meaning of 0 is ambiguous. One could return typemin(Int)
or minimum(linearindices(x))-1
, but what if x
starts indexing at typemin(Int)
?It would be nice if supporting non-1-based indices was in the scope of this Julep, and it might be better to address non-1-based arrays before their use becomes more widespread.
Note that using first(linearindices(A))-1
as a sentinel value would be non-breaking for standard 1-based arrays.
If A starts indexing at typemin(Int)
, then returned sentinel value would wrap-around to typemax(Int)
(i.e. no error).
The calling code could then test for the sentinel value in the same way:
i = findfirst(!iszero, A)
if i != first(linearindices(A))-1
...
end
To simplify, could introduce new function sentinelindex
, and use this consistently instead:
sentinelindex(A) = first(linearindices(A))-1
i = findfirst(!iszero, A)
if i != sentinelindex(A)
...
end
Continuing the other half of the discussion on #3.
Continuing half of the discussion on #3.
currently,
Pkg.rm("MyPackage"); Pkg.rm("MyPackage");
Pkg.add("MyPackage")
# takes some time
Pkg.add("MyPackage")
# takes some time -- which is silly
Pkg.add(pkg) should return immediately with eg "$(pkg) already added" when pkg was add()ed already.
and
It would be useful to have a facility for applying mutable quality tags to a release.
e.g.
Maybe users could push releases into the registry without waiting for human review, but only official reviewers would be allowed to apply the "released" tag?
End users could configure their production environment to only run "stable" packages and their playing-with-new-stuff environment to run anything.
REQUIRES files could include quality tag qualifiers.
Perhaps in the future there could be a "certified" tag that indicates a quality review has been done by an approved body.
The current proposal doesn't mention namespaces anywhere. This issue is to discuss how namespaces could fit into the proposal and Julia's own namespacing in general.
Hi,
I saw that you are using TOML.
So Are you using TOML.jl package ? It seems that package is not maintained and doesn't support TOML V>0.2.
thanks
If environment manipulation gets moved to a command-line invocation, people are probably going to want to run it most often from shell mode or via run()
, which isn't any better than what we have now. Otherwise many Windows users aren't going to find it at all because the usual workflow there is often not via the command line. (IDE features can help, but shouldn't be required for this to be usable.)
As a thought, what if we make this tie into JuliaLang/julia#17997 instead? Instead of loading packages into Main
, load them into an environment-specific namespace. Enter and leave environments via Pkg3.enterenv("newenv")
, Pkg3.leaveenv()
etc. Make Pkg3.add
, Pkg3.rm
, and Pkg3.update
take an environment argument. If the user attempts to update or remove a loaded package from an environment, that environment needs to be reset via something similar to workspace()
.
Specifically with respect to
When started in interactive mode, Julia defaults to open; when started non-interactively, it defaults to closed.
So for code that depends on modules available via "open" mechanisms like LOAD_PATH, starting julia interactively and running include("script.jl")
will work but julia script.jl
will not?
This seems like it would be a major gotcha, I don't think interactivity is a good proxy for whether people want to be permissive about code loading or have a locked-down reproducible environment. (It is a good criteria for whether or not to enable user-prompting behavior though.)
So you can do,
Pkg.add("UTF64", "~/code/UTF64.jl")
edit:
i know you can just keep monkeying around with ~/.julia/v0.5/UTF64
, it just feels a little dirty
Right now,
as I understand it there is 2 levels of package updating:
I suggest perhaps it is worth going all out, and having one update/upgrade command for each level in the Semantic Versioning.
Perhaps:
So this is the addition of the middle stage.
Where you upgrade minor, without upgrading major.
I think if SemVer is being followed correctly then this is useful.
Since a Minor change, is supposed to be nonbreaking.
People don't really follow SemVer that closely, but perhaps the should.
In part the lack of following SemVer closely is because of a fear of declaring a package v1.0.0.
and packages in the 0.x.y stage do not have to obey most of SemVer, and can have breaking changes in their minor (or even patch) versions.
This itself suggests perhaps each there should be different behaviors for the update and upgrade commands depending on the package's version. Eg that any change on a 0.x.y package be considered a potentially breaking change, and thus require upgrade
More sanely, perhaps, it suggests that no package that is not tagged 1.0.0 or higher should be allowed in the official repo.
Since at the point one is asking to be added to the official repo, one really is almost by definition at least at 1.0.0. And so one should now be following SemVer properly.
I'm unclear as to how Pkg3 will handle unregistered packages. Is the expectation that all packages must be registered with at least a personal/private registry? Currently, it's very useful to run Pkg.clone
to try out someone's changes. Similarly, declaring dependencies with just a git url in a DECLARE file provides a convenient way of developing a collection of inter-dependent packages prior to registering them.
This is a really cool project, addressing one of the big issues I have with using Julia. I'm curious about when it'll be ready for testing. Would you consider opening a meta-issue with a few checkboxes that describe the current state of things, i.e. which pieces are missing and which are already working?
Apologies if it's too early for that.
We need an explicit policy for how deprecations will be handled post-1.0. Some options are:
--depwarn=no
by default.We can use this issue for preliminary discussion until the appropriate Julep is written.
Will one be able to set a separate environment for testing? I don't think that the current REQUIRE setup is sufficient for the current organization-by-organizations setup. Sometimes there are large changes that are happening in tandem, and it would be helpful to have a designated way to say "this package needs to be on master for tests".
One case where this shows up is in organizations where there's a change to a main package which cases changes in all dependents. Another case where this shows up is where you create features which are dependent on a large change in some other package, like getting your package ready for DataFrames' nullable-array interface and wanting the proper CI. Currently this is done with little hacks around it (calling an extra script to checkout packages only when the test is run on CI), but I think making it standard and explicit would be beneficial for both developers and those who are reviewing packages (@tkelman).
First of all, really nice work!
As noted in the DifferentialEquations plan for modularization (SciML/DifferentialEquations.jl#59), the only form of "conditional dependencies" that I really need is the ability to add/remove functionality at compile-time by knowing which packages are installed on the user's system. It seems like this can be done by the user specifying a flag in the Runtime Configuration file, but is there any way that this can be computed? As in, can this flag instead be determined a code which sees if packages X, Y, Z are installed, if so, allow me to use add functionality?
An example is that, at the top of the module, I'd want to:
if SUNDIALS_INSTALLED
using Sundials
end
and then inside some higher level API functions have a branch which is allowed if SUNDIALS_INSTALLED
(and error if not). If this is a possibility, then nothing would actually need to be changing at runtime, and thus the current conditional module problems that I have (which are not too difficult! I know there are more difficult conditional module problems which don't apply here) would be solved seamlessly to the user.
I wonder whether we want Pkg3 to contain something like Debian alternatives: a package A can depend on package B or package C or package D.
For example, the Plots.jl package might want to depend on having at least one plotting package installed (cc @tbreloff).
Or, now that FFTW.jl is being split off into its own package, and that package in turn will probably depend on an AbstractFFTs package (JuliaMath/FFTW.jl#2), other packages may want to depend on having some package installed that implements FFTs following the AbstractFFT interface (e.g. FFTW, or a native Julia FFT, or FFTPACK, or MKL, etcetera).
We need to figure out how we're going to handle branching going forward. For the immediate future we plan to keep master
as 1.1, but we will need to figure out how we want to handle breaking changes in the long term.
The key questions
Over in discourse it was discussed to distinguish between "runnable packages" (called projects in that thread) and "library packages" (called packages). The suggestion which gathered the most likes was not to distinguish between projects and packages, but instead to "standardize where to put runnable scripts into packages as we know them now. Say a folder run/
or scripts/
and the main program would be run/main.jl
. Pure "Projects" would have an empty src/
folder and full run/
folder and vice versa (most would have a bit of both). Similar to Pkg.test("SomePkg")
we could have a Pkg.run("SomePkg")
to run run/main.jl
." Also a command-line option could be good, say julia --run SomePkg
.
(I haven't followed this Julep too closely, please close this issue if this is in it already. Or let me know if this should be posted over in Julia itself.)
I'd like to see Pkg3 support some kind of anonymized opt-out telemetry.
For example, it could keep track of statistics like like how many times Pkg.add
has been called for a given package with a heuristic to not count CI testing. This would give us a more reliable guide of package popularity than github stars.
Hi folks,
The Pkg3 julep looks really good so far! I was a little confused about the language describing pkg precedence across the union of depots.
Some environment and/or Julia variable – DEPOT_PATH maybe? – will control the set of depots visible to a Julia process. The registries, libraries, packages, and environments visible to Julia are the union across all depots in the depot path
...
The set of installed library versions is the union across depots. If the same library version occurs multiple times in the depot path, the first occurance is used – different instances of the same library version may be different depending on how they are configured and installed. The set of installed package versions is the union across depots. If the same package version occurs multiple times in the depot path, the first occurance is used
Does this mean that the ordering of the depot paths in the DEPOT_PATH
environment variable determines pkg precedence? For example, if we have something like JULIA_DEPOT_PATH=/home/Julia/.julia:/usr/local/share/julia/system:/usr/local/share/julia/standard
then user packages would take precedence over system packages and so on?
Hi all
I saw the talk on Pkg3 and was a bit confused with the naming.
My personal expectation was something like this:
Manifest.toml
for the actual configurationManifest.lock
for the latest working setManifest.journal
for the log. This one I would expect to be a like firebird or another single file db with a lock.Of course Config.*
would be fine too. The mix of Config.toml
and Manifest.toml
was confusing me since many develop environments use them also.
Like android with Manifest.xml
or npm with packages.json
and packages-lock.json
or cargo with Cargo.toml
and Cargo.lock
.
Regards
I know this sounds like heresy but please hear out my argument.
Semantic versioning basically just tells me whether my code will explode or not. For example a library I'm depending on changes from 1.0.3
to 1.0.4
I don't care my code will work. If it changes to 1.1.4
I don't care because my code will work. If it changes to 2.0.0
I care because my code will explode. This is literally all the information I get out of semantic versioning.
Now if pkg3 were to use a system like the following: <number>:<timestamp>
for package versioning it would be much easier to implement and reason about than semantic versioning and it would give me, the end user, more information than semantic versioning.
Here's how this versioning system would work:
I create a package called foo
with a version 1:1494423107554
where the number 14944... is just a timestamp. Now everytime I want to release a non-breaking change of my package I just update the timestamp, so for instance I change some internal crap and then I publish 1:1494423232649
. Any consumer of my library now knows that they can use my library no problem, also its easy to always stay updated with the latest non-breaking change - you just pull the latest timestamp.
Finally, anytime I want to release a breaking change I just increment the initial number. For example, now I want to change some public api function so I do it and release a new version of my lib as 2:1494423377157
.
This seems a lot simpler to me (especially from an implementation and usage perspective) and it also increases the amount of information conveyed to the end user (now every user knows when a package was published which is something I do not know when looking at a version like 1.0.3
).
Lastly, in case anyone thinks it would be annoying to update the timestamp we all know that could be easily automated by a simple scripting command.
Thoughts?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.