gobolinux / compile Goto Github PK
View Code? Open in Web Editor NEWThe GoboLinux build tool
The GoboLinux build tool
When using overlayfs as unionfs implementation Compile may misbehave depending on the order of the dependencies listed in the Dependencies file. The problem goes down to the way in which overlayfs merges two given directories. For instance:
$ mount -t overlay overlay -o lowerdir=/Programs/LibXFCE4Util/4.14.0/include:/Programs/XFConf/4.14.3/include,upperdir=/tmp/upper_layer/include,workdir=/tmp/write_layer/include /Mount
$ ls /Mount/
xfce4
$ ls /Mount/xfce4
libxfce4util
When the arguments are swapped then the contents of both packages are shown under the mountpoint:
$ mount -t overlay overlay -o lowerdir=/Programs/XFConf/4.14.3/include:/Programs/LibXFCE4Util/4.14.0/include,upperdir=/tmp/upper_layer/include,workdir=/tmp/write_layer/include /Mount
$ ls /Mount/
xfce4
$ ls /Mount/xfce4
libxfce4util xfconf-0
This is with kernel 5.6.10-Gobo. We need to test with a recent version before contacting upstream or coming up with changes to our tools.
Here is the output
/Data/Compile/LocalRecipes/Glamor-EGL 29s
❯ Compile Glamor-EGL
Compile: Locating a recipe for Glamor-EGL ...
Compile: Found recipe for Glamor-EGL
GetRecipe: Trying to get /Data/Compile/LocalRecipes/Glamor-EGL/0.6.0
Compile: Recipe placed in /Data/Compile/LocalRecipes/Glamor-EGL/0.6.0
Compile: Runner is available but the sandbox can not be created.
Compile: Checking dependencies...
Compile: Compiling Glamor-EGL version 0.6.0.
Compile: Glamor-EGL will be installed as part of Xorg-Driver 7.7.
[...]
UnionSandbox: Cleaning up.
UnionSandbox: Moving entries to: /Programs/Xorg-Driver/7.7/.SandboxInstall_Root
SandboxInstall: Postprocessing Sandbox
Compile: Stripping executables...
SymlinkProgram: Directory /Programs/Glamor-EGL does not exist.
SymlinkProgram: Did you mean one of these? GParted
Gegl
Lame
tee: /Programs/Glamor-EGL/0.6.0/Resources/UseFlags: No such file or directory
Compile: Glamor-EGL 0.6.0 - Failed installing file.
If Runner is not found in the path hence not used it ends up in the correct place /Program/Xorg-Drive
.
I will try to fix the logic after Runner finish compilation.
Probably is related with this: #5 but not sure
It would be nice to have the option to tell Compile
to continue the installation a package after it has been built successfully. Too many times a package like Glibc
has too many file exists
errors due to symlink conflicts from its previous installation. It like playing a game of Cat and Mouse where each time a silly failure like that occurs, I have to manually remove the symlink and rebuild again, hoping it will succeed the next time until it actually does. I have to rebuild Glibc
over 10 times now because of this.
Hello lucas, hisham and everyone else,
How about giving Compile the ability to show which recipes are available at the gobo store?
Here is a syntax proposal BUT keep in mind, this is just a proposal, there can be other ways; I am more interested in the functionality and less in the syntax (though syntax of course also matters):
Compile htop --versions?
The above currently fails with a message such as:
Compile: Unknown option: --versions?
I propose that Compile may instead output this:
"The following recipes are available for htop (showing not more than the 5 latest)"
Htop 2.0.2-r1
Htop 2.0.1-r1
Htop 2.0.0-r1
Htop 1.0.3-r1
Htop 1.0.2-r1
I mention "5 latest" or "five latest" because it should not spam the user with like when there are 20 recipes.
Reasoning for this feature:
Compile Ruby
Compile: Locating a recipe for Ruby ...
Compile: Found recipe for Ruby
GetRecipe: Trying to get http://gobolinux.org/recipe-store/Ruby--2.4.2-r1--recipe.tar.bz2
GetRecipe: Downloading recipe from http://gobolinux.org/recipe-store/Ruby--2.4.2-r1--recipe.tar.bz2
Compile: Recipe placed in /Data/Compile/Recipes/Ruby/2.4.2-r1
Compile: Checking dependencies...
Compile: Compiling Ruby version 2.4.2, revision r1.
--2018-01-11 21:34:42-- https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.2.tar.gz
Resolving cache.ruby-lang.org... 2a04:4e42:1b::434, 151.101.113.178
Connecting to cache.ruby-lang.org|2a04:4e42:1b::434|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14187859 (14M) [application/octet-stream]
Saving to: 'ruby-2.4.2.tar.gz'
ruby-2.4.2.tar.gz 100%[=======================================================================================>] 13.53M 7.38MB/s in 1.8s
2018-01-11 21:34:52 (7.38 MB/s) - 'ruby-2.4.2.tar.gz' saved [14187859/14187859]
Compile: Unpacking file /Data/Compile/Archives/ruby-2.4.2.tar.gz...
PrepareProgram: Preparing...
PrepareProgram: Autoconf configure script detected.
checking for ruby... /System/Index/bin/ruby
config.guess already exists
config.sub already exists
checking build system type... mkdir: cannot create directory '/Depot/Temp/cg9251-32697': No such file or directory
mkdir: cannot create directory '/Depot/Temp/cg-9251': No such file or directory
config.guess: cannot create a temporary directory in /Depot/Temp
configure: error: cannot guess build type; you must specify one
PrepareProgram: configure failed.
Compile: Ruby 2.4.2 - Configuration failed.
^^^ the above is on a fresh install on a laptop; sorry for the typos, typing on
this thing is AWFUL ...
Anyway, I usually use /Depot/Temp for temp files
in the case above, I forgot to create it
My suggestion to fix this:
Compile has a long history -- and if we're being honest, it tends to show in the mishmash of styles in the scripts.
Since the plan is to stay with Bash for the most part and only port the more complex scripts to Python3 (per @lucasvr ), it seems like it would make sense to go through all scripts and update them to Bash-5.0 compatible syntax to the benefit of readability and uniformity.
If, during the cleanup process, it turns out that small changes can affect speedups or simplification, that'll just be a nice side benefit.
`foo`
to $(foo)
[ "$x" = "y" ]
to [[ "$x" == "y" ]])
and -a/-o
to &&/||
) and a smattering of `foo`
to $(foo)
There doesn't appear to be a rigidly enforced approach to variables -- I've seen all of $var
vs. "$var"
vs. ${var}
vs. "${var}"
so far. "$var"
and "${var}"
are apparently equivalent and I happen to like the latter more as it tends to stand out in code as a variable and works everywhere.
The same can be said about var=`foo`
vs. var="`foo`"
vs. var=$(foo)
vs. var="$(foo)"
. It seems that var=$(foo)
is safe, so that's what I'm going with until something breaks (at which point I'll do "$(foo)" instead).
I'm unsure if this should be posted under Recipes, Compile, or Scripts. if this is the wrong place for it feel free to move it or lmk.
Having an internal versioning system for recipes, as a way to check if the recipe you used has had any changes since you compiled the program.
Potentially through logging the recipe's source and commit to a Resources
file.
Say I install Bash 5.1
from Recipe, and then a few days later an issue is found+fixed in the recipe. Currently I'd have no way of knowing (w/o manual checking of dates) if I used the version before or after the Recipe's change.
This issue is found first hand with the installed version of Bzip2 1.0.8
provided in Gobo017, having used an outdated Recipe file, leading to some programs failing to find it's shared library.
Having Compile
(or an adjacent script) log some information about the Recipe, for exanple using git log [FOLDER_PATH] |head -1 > $target/Resources/RecipeCommit
to get the commit.
Then be able to check it against the current version in Recipes
as follows:
local name="$1" # program name
local version="$2" # program version
local recipesfolder="/Data/Compile/Recipes/${name}/${version}"
local installedfolder="/Programs/${name}/${version}"
local recipescommit=$(git log "${recipesfolder}" |head -1)
local installedcommit=$(cat "${installedfolder}/Resources/RecipeCommit")
if [[ "${recipescommit}" == "${installedcommit}" ]]
then
echo "up to date"
else
echo "out of date"
fi
Ideally, a bit more than just the commit would be logged, potentially the git repository/fork, branch, and commit; just something concrete, fast to generate, and easy to compare.
Hello lucas and hisham and everyone else,
I just had run Compile but python 3.x linked in. Compile complained but it also showed
the following:
Compile: Your Scripts package is too old. Please update it by running 'InstallPackage Scripts'.
My suggestion is:
(1) Perhaps Compile itself could also have a way to update (itself and/or also the Scripts collection)
I know from the ruby issue tracker that the ruby core team often wants explanations or even more importantly, a use case as to why, so here goes.
I am on a freshly-installed GoboLinux system, and I have never used GoboLinux before.
When I run, sudo Compile LXDM
, it responds with this:
Password:
Compile: Initializing recipes repository
Cloning into '/Data/Compile/Recipes'...
[...hangs for several minutes...]
fatal: unable to access 'https://github.com/gobolinux/Recipes.git/': getaddrinfo() thread failed to start
fatal: not a git repository (or any of the parent directories): .git
Compile: Could not find a recipe for LXDM
When I sudo git clone
the Recipes repository by hand, it simply says Cloning into 'Recipes'...
and hangs.
Even when I git clone
as a regular user into my home directory, it hangs.
I have also tried git clone
with a smaller repository, and that hangs as well.
I have found that when I curl
anything, that hangs too.
Perhaps this is an incompatibility with the WiFi service?
Thank you in advance!
When both meson
and ninja
are installed with Alien --install PIP3:<name>
, Compile fails to import them when building something with recipe_type=meson
.
This could be due to PYTHONPATH being reset when we launch Compile with Runner, but it needs more investigation.
I've been experimenting with building my own updated ISO of GoboLinux using GoboALFS and this bug has been inconvenient for when building meta recipes like Xorg-Lib
, where certain packages within the meta recipe depend on one and another and it requires that each individual package to be symlinked each time so that dependent programs/libs can detect the dependencies.
To temporarily get around this problem, I had to manually use Compile
for each recipe in the meta recipe and have it detect and build automatically for the meta recipe sub-directory in /Programs
.
The default behavior can't be to ignore the certificates when downloading files. It should be a flag that users enable by their own responsibility.
As reported by Stéphane Aulery on gobo-devel:
This code [1] accepts a lot of archive formats but this code is narrower [2]. Expected ?
[1] https://github.com/gobolinux/Compile/blob/master/Functions/Compile#L88
[2] https://github.com/gobolinux/Scripts/blob/master/Functions/Archive#L7
Certain packages fail to build if they are already installed under /Programs/*
.
It can be reproduced by installing LibBPF 0.2
and trying to reinstall it using Compile LibBPF 0.2
. In the presented scenario, the installation will fail with the following error:
real_install: cannot change permissions of ‘/usr/include/bpf’: No such file or directory
.
Doing a RemoveProgram LibBPF 0.2
prior to calling Compile LibBPF 0.2
will make it work.
Unless a user already knows this, this can be a very frustrating experience.
When one builds a recipe featuring a Resources/Defaults/Settings
directory, Compile will place those files under /Programs/Name/Version/Resources
, and SymlinkProgram will take care of copying them over to /Programs/Name/Settings
.
When a recipe does not include Resources/Defaults/Settings
, however, "make install" will overwrite settings files from previous versions of the same package at /Programs/Name/Settings
. It looks like the most sensible thing to do is to:
/Programs/Name/Version/Resources/Defaults/Settings
/P/N/V/Resources
with a recursive copy of the recipe's default settings, if that existsThis approach, at the very least, prevents problems with lost or missing settings files after the compilation of a frequently updated package.
I have found that some recipes contain multibyte characters and NewVersion is complaining and failing to generate a new version of the Recipe.
❯ NewVersion Wireshark 2.2.3 https://1.na.dl.wireshark.org/src/wireshark-2.2.3.tar.bz2
NewVersion: Locating a recipe for Wireshark...
NewVersion: Found recipe for Wireshark at /Data/Compile/LocalRecipes/Wireshark/2.2.0
GetRecipe: Trying to get /Data/Compile/LocalRecipes/Wireshark/2.2.0
NewVersion: Recipe placed in /Data/Compile/LocalRecipes/Wireshark/2.2.0
NewVersion: Creating recipe for Wireshark 2.2.3 based on 2.2.0.
Recipe: línea 2: Coincidencia: no se encontró la orden
Recipe: línea 2: Coincidencia: no se encontró la orden
NewVersion: Downloading source code...
/Data/Compile/LocalRecipes/Wireshark/2.2.3/Recipe: línea 2: Coincidencia: no se encontró la orden
I've been trying to compile the Ranger recipe on GL 016.01 in a VirtualBox VM (Win 7 host), but it fails with these messages:
"/usr/bin/env: 'python': No such file or directory
Compile: Your Scripts package is too old. Please update it by running 'InstallPackage Scripts'."
I've confirmed that I have Python 2.7 installed using the 'which' command, and I've also confirmed that I have Scripts 016.01 installed by looking in the /Programs directory. So I don't know why these errors are happening, but they are preventing the package from compiling.
Output from SystemInfo:
Kernel: Linux 4.9.16-Gobo x86_64
Compilation: #0 SMP PREEMPT Mon Mar 20 08:53:55 BRT 2017
Processor: Intel(R) Core(TM) i7-3940XM CPU @ 3.00 GHz
Memory: 16423312 kB
I'm not sure if this is the right place to bring this up, or if I should go to the recipe maker, or the Ranger devs themselves. If you need more information, please let me know. This is preventing me from installing the recipe, so any help you can give will be much appreciated. Thanks!
Would be good to know. Hopefully a permissive one for those of us who'd like to incorporate the GoboLinux approach to the filesystem under *BSD.
how to make it all work and make use of this can some body give me an simple example of use
When making a command like this:
NewVersion wifi-radar 2.0.s10 http://wifi-radar.tuxfamily.org/pub/wifi-radar-2.0.s10.tar.bz2
the link is altered to 'http://wifi-radar.tuxfamily.org/pub/wifi-radar-2.0.s10.s10.tar.bz2' and obviously the recipe fails, because the source is not found.
The problem is when there are letters in the program version numbering ( s10 , i586 , x64 ).
It will get more complicated now with Gobo-016, since there will be the need for 32bit and 64bit recipe versions, with 'i686' or 'x86_64' or 'x64' in the version numbering.
Another example:
NewVersion jre 8u112_i586 http://ftp.osuosl.org/pub/funtoo/distfiles/oracle-java/jre-8u112-linux-i586.tar.gz
the link will be changed to 'http://ftp.osuosl.org/pub/funtoo/distfiles/oracle-java/jre-8u112_i586-linux-i586.tar.gz'
Please see this gobolinux/Scripts#2
With the newest version of the Compile
tool supporting using Runner
to construct a sandbox during build (see this forum post [BACKUP]), focus now moves to verifying the various different Recipe types.
Each category should be populated with 2-3 test recipes for verification.
See the Recipe Format Specification documentation for the supported types.
Each test should be conducted with updated versions of /Data/Compile/Recipes
, /Programs/Compile/Current
, /Programs/Scripts/Current
on a system where all steps in the Known Issues and Fixes post-install guide have been completed. Note that the GrepReplace
step in particular is important, otherwise build failures WILL occur!
Please use the command Compile --debug --pure <Recipe>
and report any issues with a full build log.
Compile
Dependencies
The new Dependencies required during
Compile --pure
runs should actually belong to
Compile/Resources/Dependencies
.We already have GCC and Python. Lua and CMake should be there too.
Also, the way that I implemented
Compile --pure
will allow any hand-written dependency file which states a
specific > version (or a range of versions) of a program to get prioritized over dependencies that simply
list the program name.That is, if you add
CMake
toCompile/Resources/Dependencies
and addCMake >= x.y.z
to your Recipe,
the version requested by your Recipe will be honored.
First pass of the above was added in Compile PR #49.
Bison
and Flex
to BuildDependencies
, see Recipes PR 103)BuildDependencies
and some Dracut
changes)It seems that Compile, in the case of Pahole 1.18
, does not download the second archive specified in the urls section.
To reproduce try Compile Pahole 1.18
. It will fail with the message:
ls: cannot access '/Data/Compile/Archives/libbpf-0.2.tar.gz': No such file or directory
(If you are unable to reproduce, make sure libbpf-0.2.tar.gz
is not already present under /Data/Compile/Archives/
)
Looking at the output, It appears that Compile
does not even attempt to download the archive.
Consider updating Resources/Environment, as that file may hold a string with the installation path.
More and more projects seems to be moving to releases that are basically a version tag on services like github and gitlab, so it would be nice if Gobolinux would learn download them as a versioned recipe rather than a generic git recipe.
The docs docs=()
directive does not appear to do anything at all, unless I am mistaken.
It seems to be broken.
See package GLM
.
$ Compile XProto 7.0.31
...
Compile: Compiling XProto version 7.0.31.
Compile: XProto will be installed as part of Xorg-Proto 7.7.
...
SandboxInstall: Postprocessing Sandbox
Compile: Stripping executables...
SymlinkProgram: Directory /Programs/XProto does not exist.
Hi,
I write this down on behalf of ThePub from the IRC channel.
He commented that the scripts were moved to git + github, which is fine, but there was no "scripts upgrade path" to get git repository support in the current system available. (I think he referred to 015 default install).
The git-move happened after the 015 installation .iso, so it is understandable that it offers no git-support, but perhaps that could be added? Some way that people can make use of git-update of components (in the event that bugs were fixed etc...)
In the past I think this happened when the Compile-Recipe was updated, perhaps this could now happen via git (no idea if anything else changed too).
The idea would be to have the Gobolinux Scripts work reliably.
Perhaps something like Compile --sync or something, or compile --git - no idea about the specific command.
Hello,
I've encountered this issue updating the bullet recipe.
The saved log says:
Compile: Unpacking file /Data/Compile/Archives/bullet-2.87.tar.gz...
Compile: Directory /Data/Compile/Sources/bullet-2.87 not found.
The recipe is
# Recipe (MakeRecipe) for Bullet by Filipe Vieira, on Fri Jan 18 20:42:37 WET 2008
compile_version=1.10.0
url="https://github.com/bulletphysics/bullet3/archive/2.87.tar.gz"
file_size=56691047
file_md5=7566fc00d925a742e6f7ec7ba2d352de
file="bullet-2.87.tar.gz"
recipe_type=configure
autogen_before_configure=yes
pre_link()
{
# Enlightenment 18 will ask for the missing headers below
for i in ./src/BulletSoftBody/btSoftBodySolvers.h \
./src/BulletSoftBody/btDefaultSoftBodySolver.h \
./src/BulletSoftBody/btSoftBodySolverVertexBuffer.h \
do
cp -v $i $target/include/bullet/BulletSoftBody
done
cp -v ./src/BulletDynamics/ConstraintSolver/btFixedConstraint.h \
$target/include/bullet/BulletDynamics/ConstraintSolver
}
I just fresh install gobolinux and completed all instuctions in GoboLinux-017-Known-Issues-and-Fixes.
but i see Missing import: UnionFS if use Compile util...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.