jimporter / bfg9000 Goto Github PK
View Code? Open in Web Editor NEWbfg9000 - build file generator
Home Page: https://jimporter.github.io/bfg9000
License: BSD 3-Clause "New" or "Revised" License
bfg9000 - build file generator
Home Page: https://jimporter.github.io/bfg9000
License: BSD 3-Clause "New" or "Revised" License
There should be a function for build.bfg
files to ensure that bfg9000's version meets the requirements (it should probably also have an optional check for the python version).
It would be nice if people could add custom rules, languages, etc to bfg9000 so that they don't have to rely on me to support their use cases (especially important for really unusual use cases that I wouldn't want to add into bfg9000 proper).
The code for the builtin functions is already largely structured like this, with one file for each group of builtins, so it would just be a matter of letting users load their own modules at the same time. One way of handling this would be to look in my_project/bfg9000
and load any .py
files in there before executing the build.bfg
file.
We already support this for tests (under POSIX). However, we need a general way to set environment variables that get persisted correctly: local to each sub-command for test()
, but for all sub-commands for command()
. As usual, Windows is a pain in the ass here.
exit()
in a build.bfg file should just exit the script, not all of bfg9000. I'm not sure if this already works or not, but I'd guess not. At minimum, this should have tests.
Not all platforms have the install
utility. We should just issue a warning if it's not present. (This requires having a system to emit warnings.)
Before releasing 0.1, I'd like to at least make a best effort to fix all the OS X tests.
Running automated tests is a pretty important part of a build system! We should have some built-in way of managing this, even if it's just for relatively simple test frameworks for now. Eventually, I'll need to do some research on how various test frameworks expect to be run so that I can come up with a one-size-fits-most solution.
We really need some simple docs and a purdy homepage for bfg9000 before we release.
This would be especially helpful for Appveyor, since free accounts only get one concurrent build. Travis might benefit a small amount too.
Normally, the way you'd use a tool like ccache
or distcc
is to prepend it to your CC
env var like so: CC=ccache gcc
. This causes problems with bfg though. Command variables (like CC
) are converted into absolute paths so that builds are less dependent on the user's current environment. That is, CC=gcc
becomes CC=/usr/bin/gcc
.
ccache
and distcc
work? What if the user had a similar tool that we didn't recognize? What if there was a tool whose format looked like this, but the gcc
wasn't meant to refer to an actual file in the path? (For a simpler example of this problem, see #27.)which()
to get the absolute path. We can split the value into a list, but that doesn't solve everything.gcc
isn't installed, bfg will error out. This prevents users from starting a build, only to find out an hour in that they forgot to install something. However, we can obviously check for gcc
's existence some other way.ccache
and distcc
so that we know to treat the argument after it as a file that needs to be turned into an absolute path, e.g. ccache gcc
becomes /usr/bin/ccache /usr/bin/gcc
. This doesn't help us for commands we don't know about, though.bfg9000
is first run for the project) during the actual build process. This would prevent users from making any after-the-fact tweaks to the environment, which might be a bad thing. It would prevent users from tweaking env vars to, say, run unit tests in a slightly different configuration. Arguably, this should be handled internally in the build.bfg
file though.ccache
or distcc
. This has all the problems of (1), in that users won't be able to use tools bfg doesn't know about, and doesn't seem to have much benefit.So far, (1) and (3) seem like the best solutions. (2) is pretty complex and breaks probably-useful workflows, while (4) is strictly worse than (1). At the moment I'm leaning towards (1), but (3) is definitely simpler. It really comes down to a question of what's more important: simplicity or reproducibility.
One related question (see #27) is if there are commands that should not have their absolute paths resolved. Obviously, shell built-ins fall under this, but arguably, so does mkdir
and other commands that GNU Make assumes exist.
This is mainly necessary for propagating the source language from a static library to a whole-archive to a shared library. The hard part will be deciding on when functions like executable()
refer to a build and when they refer to a pre-existing file.
To link to external libraries, we should read from pkg-config files and use the appropriate options.
Once the API stabilizes some more, I should add docstrings to everything. I figure by the time 0.2 is out, things will be stable enough...
If an executable (or shared lib) links to a shared lib, said lib obviously needs to be present at runtime. To prevent errors, we should automatically mark runtime deps for installation if something requires it. This probably only applies to locally-built shared libs, but maybe Windows would want external shared libs to be copied over too? Windows is weird.
Currently, we use mkdir -p
for installing directories and for creating subdirs under Make. However, not all platforms support mkdir -p
(e.g. Windows, some POSIX systems). We should create a set of tools to pick the right program to use on all platforms.
MKDIR_P
environment variable as, e.g. mkdir -p
, and that doesn't really work with how we treat those variables right now: as a single word to be sent on to the which()
function for finding the absolute path (see #28). On the other hand, maybe this isn't something the user should be able to set.mkdir
to an absolute path like we do for other commands? Do we just want to do it sometimes (e.g. for gmkdir
on Windows)? "Always" or "never" are the simplest answers, but 1) it seems weird to reference mkdir
by abs path on Linux, and 2) it seems weird not to for gmkdir
on Windows.First, it should be noted that long-term, mkdir -p
will probably only apply to the Make backend. This means that we can probably assume the existence of other GNU tools. In that case, it's probably ok not to have a MKDIR_P
environment variable. Additionally, I'm leaning towards always using the abs path because that's more consistent with everything else.
VS2015 support already works, with both Ninja and MSBuild. However, I think the MSBuild files are broken for VS2013, etc. We should at least support VS2013 for 0.1.
I'm not 100% on this stuff, but the general idea is (on Linux) to append .1.2.3
or what-have-you to the filename, set the soname to libfoo.so.1, and set up some symlinks as needed. No idea how this works on OS X, and I'm sure it doesn't work on Windows at all.
Current plan:
VersionedSharedLibrary
to track the real .so and its aliasesldconfig
here?)Depends on #18.
To my knowledge, there's no existing meta-build system that supports globbing source files and regenerating the build files if the result of the glob changes. For instance, while CMake supports globbing and automatic build file regeneration, the two can't be used together [1].
Obviously, the hard part here is doing this fast: incremental builds should only regenerate the build files if a relevant file was added or deleted. Here's a potential solution:
.bfg_watch
file that lists these patterns and their results.bfg_watch
file, and go through each of the glob patterns to see if their results changed
The CC toolchain can generate Windows import libraries (e.g. with MinGW or cross-compiling). As an initial step towards supporting import libraries everywhere, let's add support for it in CC.
To make builds more reproducible, we shouldn't rely on commands being in the PATH
; instead, resolve them to absolute paths before writing them out in the build file. This will also help ameliorate #13.
Currently, variable expansion in paths can cause the string that gets sent to the shell to have spaces, but no quotes. Obviously, this is a problem. It doesn't matter in other contexts though, since 1) spaces in filenames are just a nightmare for Make, and 2) Ninja handles escaping for variables on its own.
One open question is how to handle shell globs (currently used by the install command). We could just pull the *
out of the path itself and call it a day. Note that this shouldn't matter in the long-term since the install command is due for replacement.
OS X has rpaths, although they seem pretty different from Linux. Still, we should try to support them. This may take a fair amount of preliminary work...
Static libraries don't allow you to pass link options or libs to them. However, we could allow this and just keep track of the options that were passed, and then forward them on to whatever links to our static lib. This probably requires #5 so that we can record these link options when installing our static lib.
To make portable executables (and DSOs) on Linux, we should use rpath. (LD_LIBRARY_PATH
is crap.) This presents some issues with how to create a rpath that works for both pre- and post-installation of the binary. Here are some possible solutions:
install
directive is separate from the build directives. This also wouldn't work if we support setting $(bindir)
and friends from the environment.LD_LIBRARY_PATH
for pre-installation execution. This could end up being extra management for the user if they want to run the binary by hand before installing it.chrpath
or patchelf
to modify the runpath when installing. This is probably the most-complete solution, but it imposes an additional dependency on bfg.I think this technically works, but it's kind of a pain, partly because zlib takes some effort to use under Windows (and has a funky library naming scheme). We could change the library used in the test (e.g. libogg?). We might also want to consult more of the Windows-specific environment variables to make things easier on people.
GCC (or clang with the CC interface) can be used on Windows. We should support this. We can do so by examining the CC
and CXX
environment variables and then picking the appropriate compiler type.
This way, if the tool (or backend) you wish to use isn't in the path, you can still use fancy new features.
It should be impossible to install something found via system_package
, system_executable
, boost_package
, etc. They're already installed! One way to do this would be to give all File
s an external
attribute to keep track of whether it's from the system (i.e. external to the project).
There should be a way to specify that there's a build.bfg file in a subdirectory that needs to be parsed. Remember that paths in the sub-build.bfg file should be relative to the current directory, not the root directory.
Tasks:
build.bfg
file
include
)?Currently, we autodetect the best build system and get version info by running it and grabbing the version. However, this fails if the name is something we don't expect (or it's not in the PATH). We should support checking standard env vars to get the build backend's name, e.g. $MAKE
. This may end up obviating #12.
This already kind of works, but we need to be sure that everything is actually correct on Windows (at least, correct enough that simple projects can be built).
To help make more reproducible builds, we could keep track of the exact results of package/executable lookups (e.g. system_package()
). This way, even if something about the environment changes, we'll still have consistent results (although sometimes this might just mean a build failure).
One open question is: where should this information be stored? Currently, I see .bfg_environ
as independent of the build.bfg
file, and so the results of lookups wouldn't go there. This implies that .bfg_environ
is immutable even when build.bfg
is edited, which is a potentially-nice property (keeps things simple). However, these results do technically come from the environment, so maybe they really do belong there.
Pretty self-explanatory. We should fix all the bits required to let people use Python 3. One of these days, I'm sure people will stop using Python 2... :) The most obvious thing to fix is all the uses of basestring
.
Currently, if I add a new rule for bfg9000 that needs special handling, I have to go and modify BuildInputs
too. This will make life hard for people extending bfg9000 (see #48), so we should fix it.
In order to support Java, we'll need to add a concept of "runtime" to bfg. Currently, all the code assumes native runtimes. We could add the JVM as either a new Platform
(that has the native platform as a member) or create a Runtime
type with both Native and JVM instances/subtypes.
It might be useful to add the ability to copy/link a directory, e.g. to make it easier to package up resource files. Obviously, you can already do this with command()
, but a platform-agnostic way could be useful. One question is what to call the link function. Symlinks and hardlinks are both nice, but "link" is too confusing: it sounds more like linking an executable.
I'm not even 100% sure I'd want these steps in the first place, so I'm leaving this open to let folks discuss it. If there's a consensus in favor of this, I'll add support.
Pretty self-explanatory. Something like global_options('-pthread', phase='link')
? Eventually, we'll probably want to add the ability to apply this only to a certain runtime (native, CLR, JVM, etc).
Currently, if there's an error in the build script, we just let the exception bubble up and kill the program. We should make the error output nicer.
Currently, the install command uses a variety of GNU tools (install
, cp -r
, and mkdir -p
). Write a new program that does this in a simpler way so that we can just use that. Then Windows users won't need GnuWin32 CoreUtils to install stuff.
Currently, bfg assumes for some file types that a file is a dependency iff it has a creator associated with it. This isn't quite right. A file is a dep iff it's internal to the project. Probably the best way to handle this is to treat includes
and libs
as sources of deps, but packages
as non-deps.
Fairly self-explanatory, I think.
If something you do (e.g. edit via emacs) creates a temp file in a folder that's been searched for files, that folder will be marked as changed, triggering regeneration of the backend build script. It would be nice if we could be smarter about this, but that'll probably require ninja-build/ninja#987 to be fixed first.
single-quoting the $@ causes the patsubst to not match, thus we just mkdir -p $targetdir/.dir and then touch the new directory $targetdir/.dir. I think the intent was to create a 0-sized file .dir in $targetdir.
Removing the quotes breaks spaced-things, we need to use double-quotes here, but I see no easy way to do so without significant changes. Not sure how important it is.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.