academysoftwarefoundation / rez Goto Github PK

rez-build will not complain when version value is missing from package.yaml and install a local package in ~/packages/package_name/None/Linux/

I've got a few improvements that I want to make to the variable expansion convention that I proposed earlier.

Current Incarnation

version.part(1) == 2
version.part(2) == 7
version.part(3) == 4

version.thru(1) == '2'
version.thru(2) == '2.7'
version.thru(3) == '2.7.4'

commands: |
  PYTHON_MAJOR_VERSION = '{version.part(1)}'
  PYTHON_MINOR_VERSION = '{version.part(2)}'

  CMAKE_PREFIX_PATH.prepend('$PYTHON_DIR')

  if machine.platform == 'Linux':
    PYTHON_DIR = '/usr/local/python-{version}'
    PATH.prepend('$PYTHON_DIR/bin')
  elif machine.platform == 'Darwin':
    PYTHON_DIR = '/usr/local/python-{version}'
    PATH.prepend('$PYTHON_DIR/Python.framework/Versions/{version.thru(2)}/bin')
  else:
    PYTHON_DIR = 'C:/Python{version.part(1)}{version.part(2)}'
    PATH.prepend('$PYTHON_DIR')
    PATH.prepend('$PYTHON_DIR/Scripts')

Inspiration

I want to switch over to using "new style" string formatting, which allows custom types to define their own formatting via a __format__ method. It also provides a way to pass custom "format specifiers" to those types (the format specifier is everything after the : in the following examples).

For example:

>>> '{:04d}'.format(1)
'0001'

>>> import datetime
>>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)
>>> '{:%Y-%m-%d %H:%M:%S}'.format(d)
'2010-07-04 12:15:58'

Proposal

Python's new-style formatters support attribute access {foo.myattr} and item access {foo[0]} as well as chaining the two {foo.myattr[0]}. However, unlike rez's current formatters, it does not support function execution {foo.myfunc()}. Thus, I'd like to remove the thru() and part() functions, and provide simply a parts attribute (0-indexed):

version.parts[0] == '2'
version.parts[1] == '7'
version.parts[2] == '4'

explicit formatting can be done like this:

"{version.parts[0]}.{version.parts[1]}.{version.parts[2]}" == '2.7.4'

but custom format specifiers for our ExactVersion class will provide a more abbreviated option:

"{version:#.#.#}" == '2.7.4'
"{version:##}" == '27'
"{version:#.#v#}" == '2.7v4'
"{version:#0.#1.#2}" == '2.7.4'
"{version:#1.#2}" == '7.4'
"{version:#000.#01.#2}" == '002.07.4'

and the yaml example would look like this:

commands: |
  PYTHON_MAJOR_VERSION = '{version.parts[0]}'
  PYTHON_MINOR_VERSION = '{version.parts[1]}'

  CMAKE_PREFIX_PATH.prepend('$PYTHON_DIR')

  if machine.platform == 'Linux':
    PYTHON_DIR = '/usr/local/python-{version}'
    PATH.prepend('$PYTHON_DIR/bin')
  elif machine.platform == 'Darwin':
    PYTHON_DIR = '/usr/local/python-{version}'
    PATH.prepend('$PYTHON_DIR/Python.framework/Versions/{version:#.#}/bin')
  else:
    PYTHON_DIR = 'C:/Python{version:##}'
    PATH.prepend('$PYTHON_DIR')
    PATH.prepend('$PYTHON_DIR/Scripts')

thoughts?

me:
detecting cpp compiler...
found cpp compiler: /usr/bin/gcc, id: 4.4.4
Couldn't detect compiler version, assuming 1.0.1...
cpp compiler version: 1.0.1

/usr/bin/gcc --version
gcc (GCC) 4.4.4 20100726 (Red Hat 4.4.4-13)

Sent at 8:34 PM on Tuesday
me: sorry, didn't mean to just dump this on you, but just so you're aware when you get to it yourself ;) I don't have time to fix the script, just hacked the version for now

Allan: cool thx. When I get a chance to look at this (next week) I might have to get you to try some code, as I probably won't be able to replicate the prob on a different system

Allan: oh can you try:
/usr/bin/gcc -dumpversion
maybe that doesn't work for gcc-4.4.4+... easy enough to fix, I'll make it try --version if that fails..

me: 11:04 jensj@teargas: ~/Documents 38 >/usr/bin/gcc -dumpversion
4.4.4

Allan: ah I see the prob... for some reason the 'compiler id' is 4.4.4, usually it's 'GNU'. Dunno why this behaviour
anyway should be a pretty simple fix

rez already has plenty of utilities for reading and validating data from multiple sources in rez.resources (see Issue #52 for a discussion on schemas). Instead of placing the schema definition within rezconfig we should use our existing metadata validators to do this work for us. I don't love the idea of having configuration files with huge "DO NOT EDIT" blocks.

It would make life a lot easier if it was possible to run the tests without having to reinstall rez prior to each run. I tried to jump start this by first prefixing all of the test modules with test_, since this is what unittest uses for discovery by default.

then, from the src directory, I ran python -m unittest2 discover -s rez/tests

i got a number of errors, but the ones that troubled me the most were these two:

======================================================================
ERROR: test_command (test_shells.TestShell)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/sv-dev01/devRepo/chad/shell/rez/src/rez/tests/test_shells.py", line 52, in test_command
    r = self._create_context(["hello_world"])
  File "/Volumes/sv-dev01/devRepo/chad/shell/rez/src/rez/tests/test_shells.py", line 32, in _create_context
    caching=False)
  File "rez/resolved_context.py", line 131, in __init__
    resolver.solve()
  File "rez/resolver.py", line 48, in solve
    verbose=self.verbose)
  File "rez/solver.py", line 1277, in __init__
    phase = _ResolvePhase(self.request_list.requirements, solver=self)
  File "rez/solver.py", line 729, in __init__
    scope = _PackageScope(package_request, solver=solver)
  File "rez/solver.py", line 537, in __init__
    package_request.range)
  File "rez/solver.py", line 1537, in _get_variant_slice
    slice = self.package_cache.get_variant_slice(package_name, range)
  File "rez/solver.py", line 510, in get_variant_slice
    building=self.building)
  File "rez/solver.py", line 265, in __init__
    "package family not found: %s" % package_name)
PackageFamilyNotFoundError: package family not found: hello_world

I believe that is using the hello_world package creating during installation. all the tests should be completely self contained and should only use packages created inside the data directory. several of the tests do this already.

This is the other issue that stuck out:

Traceback (most recent call last):
  File "/Volumes/sv-dev01/devRepo/chad/shell/rez/src/rez/tests/test_build.py", line 28, in setUpClass
    shutil.copytree(packages_path, cls.src_root)
  File "/usr/local/python-2.7.4/lib/python2.7/shutil.py", line 171, in copytree
    names = os.listdir(src)
OSError: [Errno 2] No such file or directory: '/Volumes/sv-dev01/devRepo/chad/shell/rez/src/rez/tests/data/build/packages'

It seems like there are packages missing for the build test? perhaps these weren't committed?

Another small one: ./examples/demo/run_demo display an error after an invocation of date. The date help is printed when this happens.

usage: date [-jnu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... 
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]

Here's a patch that fixes it, with lengthy explanation of what goes wrong.

diff --git a/bin/rez-context-info b/bin/rez-context-info
index ccf2f59..53e13c5 100755
--- a/bin/rez-context-info
+++ b/bin/rez-context-info
@@ -17,7 +17,19 @@ if [ ! "$REZ_CONTEXT_FILE" ]; then
        exit 1
 fi

-readable_time=`date -d "1970-01-01 $REZ_REQUEST_TIME sec GMT"`
+echo REZ_REQUEST_TIME=$REZ_REQUEST_TIME
+
+# Pre python port
+# readable_time=`date -d "1970-01-01 $REZ_REQUEST_TIME sec GMT"`
+# $ date -d "1970-01-01 $REZ_REQUEST_TIME sec GMT"
+# Thu Aug 15 03:30:29 UTC 2013
+# $ python
+# >>> import datetime
+# >>> print datetime.datetime.fromtimestamp(1376537429).strftime('%a %b %d %H:%M:%S UTC %Y')
+# Thu Aug 15 03:30:29 UTC 2013
+#
+readable_time=`python -c 'import sys, datetime ; print datetime.datetime.fromtimestamp(float(sys.argv[1])).strftime("%a %b %d %H:%M:%S UTC %Y")' $REZ_REQUEST_TIME`
+
 echo "requested packages (mode="$REZ_RESOLVE_MODE", time="$REZ_REQUEST_TIME: $readable_time"):"
 echo $REZ_REQUEST | tr ' ' '\n'
 echo

I've done a quick conversion of the README file and here's how it looks like.

https://github.com/bsergean/rez

Tell me what you think !

I've just finished converting rez-cmake to python, and the command strikes me as somewhat unnecessary. currently, rez-build creates a bash script build-env.sh which calls rez-cmake which in turn calls cmake.

here's what i don't like about rez-cmake

it doesn't do much
rez-cmake essentially just makes it a little easier to pass arguments to cmake, and thus doesn't do anything that rez-build couldn't easily bake into build-env.sh.

here's what it always does:

• force enables CMAKE_SKIP_RPATH
• sets -DCMAKE_MODULE_PATH=CMAKE_MODULE_PATH
• sets the default build system (eclipse)
• sets -C to CMAKE_INITIAL_CACHE if already set

here's what it optionally does:

• deletes the cache (CMakeLists.txt)
• toggles central vs local install (-DCENTRAL=1)
• toggles coverage (-DCOVERAGE=1)
• sets build system based on user input

it adds another layer of confusion
rez-build simply passes all args after the first -- along to rez-cmake untouched. and i assume it is by design that one could cram rez-cmake, cmake, and make args this because rez-cmake passes along all args after --- to cmake.

e.g.

rez-build -v 0 -- -b eclipse --- -DFOO=1 -- 

having 3 sets of arguments for rez-build is tolerable. but when you throw in a 4th set of arguments for a cmake wrapper, which might actually conflict with arguments passed directly to cmake, then i think it's crossed a line of convolutedness.

how would you feel about moving the rez-cmake arguments and behavior to rez-build and call cmake directly in build-env.sh? since rez-cmake has now been converted to python, it would be easy to create a function shared by both rez-build and rez-cmake for performing the tasks therein, and we could keep rez-cmake around as a cli utility, but not actually have rez-build use it.

I want to be able to do the following:

rez-env '(foo ~~bar)'

where the package 'bar' will be added to the wrapper shell but some special syntax (~~ in this case) will prevent its tools to be exposed to the main / master shell.

It would be nice if rez's package files supported the concept of "optional requirements" (i'm sure we can come up with a less oxymoronic name). These packages would be loaded if found successfully, but otherwise would not result in an error.

my current scenario is that I'm using rez to build MtoA (the Maya to Arnold translator), which has an optional plugin for shave and a haircut. We have shave for maya 2012 installed, but we don't yet have it for maya 2014. it would be nice if in the mtoa package.yaml file I could effectively say "load shave if you can", then in the build script (in this case it's scons) handle the logic if it is or is not there.

This feature is a nice analog to cmake's find_package, without the REQUIRED flag.

From Allan: For 2.0 I would also add: setup.py. I think this is achievable, and the installation code right now is ugly and non-standard.

Howdy,

Here's a small diff to make it so the init.csh that gets produced by install.sh is "sour cable". There was 2 straights if that should have been endif, one extra double quote in there; and the PATH settings is closer to the one for bash now. I'm still pretty rusty with git so here's a straight git diff on that file; I'll try to make a proper pull request.

diff --git a/init.csh b/init.csh
index 5a36c5d..fff5231 100644
--- a/init.csh
+++ b/init.csh
@@ -30,20 +30,18 @@ else
     # where rez will publish local packages to (ie those installed with rez-build -- -- install)
     if (! $?REZ_LOCAL_PACKAGES_PATH ) then
         setenv REZ_LOCAL_PACKAGES_PATH !REZ_LOCAL_PKGS_PATH!
-    fi
+    endif


     # where rez-egg-install will install python egg packages to
-    if (! $?REZ_EGG_PACKAGES_PATH" ) then
+    if (! $?REZ_EGG_PACKAGES_PATH ) then
         setenv REZ_EGG_PACKAGES_PATH !REZ_PACKAGES_PATH!
-    fi
+    endif


     # expose rez binaries, replacing existing rez paths if they have been set already
     set clean_path = `echo "$PATH" | /usr/bin/tr ':' '\n' | grep -v '^$' | grep -v '!REZ_BASE_PATH!' | /usr/bin/tr '\n' ':'`
-    set rez_path   = ":${REZ_PATH}/bin"
-    set full_path  = "$clean_path$rez_path"
-    setenv PATH $full_path
+    setenv PATH ${clean_path}:${REZ_PATH}/bin

     if (! $?REZ_RELEASE_EDITOR ) then
         setenv REZ_RELEASE_EDITOR !REZ_RELEASE_EDITOR!

Now when I try to run the rez run hello_world, I get an error about the -p option of mkdir, which is probably only available with GNU mkdir and not the BSD one that I'm running on my Mac.

Cheers,

• Benjamin

also ther may be toolchain-specific stuff in rez-run still? Remove if so

some still exist in docs, one_liners.txt at least

How should internal package variants be modelled in Rez?

The current variant system caters well for the cases where a variant is an external dependency (such as the gcc version, or the target operating system). But what about cases where the variant is internal to the package?

For example, the packages foo and boo both depend on the same version of bah. However they depend on different variants of bah; foo wants it compiled with option A, boo with option B. The options are mutually exclusive, and are build time options for bah.

I could make foo and bar variants of bah, allowing me to create the different builds, but this reverses the dependency which is incorrect.

I could make a meta-package for the two options so they can be listed as variants of bah, but this will become messy as the number of options increases.

Or I could make two packages; bah_A and bah_B. But this again will become messy as the number of options increases.

Is there a better way of modelling this in Rez?

The default build process isn't releasing from a fresh export of the repo, this should be changed. It will also have to create package-release.yaml and variant-release.yaml files, and this will take into account per-variant timestamps as well.

I'm getting errors when I'm running configure.sh, I've fixed it so I get some relevant info on a Mac. This is what with the fix:

detecting OS distribution...
OS distribution is: Apple

detecting OS distribution version...
OS distribution version is: 10.8.3

Patch:

diff --git a/configure.sh b/configure.sh
index 29d2148..c6f2e6c 100755
--- a/configure.sh
+++ b/configure.sh
@@ -246,6 +246,9 @@ if [ "$distro" == "" ]; then
         distro=`lsb_release -i | awk '{print $NF}'`
         if [ $? -ne 0 ]; then distro=; fi
     fi
+    if [ "$osname" == "Darwin" ]; then
+        distro=Apple
+    fi
     if [ "$distro" == "" ]; then
         echo $echoerr"Could not identify OS distribution - $or_set""REZCONFIG_DISTRO" 1>&2
         echo "Setting to DISTRO_UNKNOWN-0!"$echoreset 1>&2
@@ -263,6 +266,10 @@ if [ "$distro_version" == "" ]; then
         distro_version=`lsb_release -r | awk '{print $NF}'`
         if [ $? -ne 0 ]; then distro_version=; fi
     fi
+    if [ "$osname" == "Darwin" ]; then
+        distro_version=`sw_vers -productVersion`
+        if [ $? -ne 0 ]; then distro_version=; fi
+    fi
     if [ "$distro_version" == "" ]; then
         echo $echoerr"Could not identify OS distribution version - $or_set""REZCONFIG_DISTRO_VERSION" 1>&2
         echo "Setting to 0!"$echoreset 1>&2

Last week I started experimenting with using rez on linux and osx, and quickly discovered that OS packages are not mutually exclusive (I think I may be the first person actually using rez on two operating systems at the same site). There is nothing stopping a resolve on osx from pulling in packages that require Linux. Making Linux and Darwin anti-packages of each other only protects against bad resolves by aborting the resolution.

My plan is to create an "os" package, with versions per os: os-darwin, os-linux, os-windows. Versions are inherently mutually exclusive, so it will prevent os-darwin and os-linux from being loaded simultaneously, and thus allow packages that require or vary on these to resolve properly.

To achieve this, we will first need to implement a new type of Version subclass that supports a string label.

name: os
version: linux

name: os
version: darwin

name: os
version: windows

similarly, architectures can be implemented with arch-i386 and arch-x86_64 packages.

name: arch
version: x86_64

name: arch
version: i386

lastly, specific versions of an operating system can be implemented as their own packages. e.g. osx-10.8.1, fedora-19, etc. Here are some complete examples from windows, with numeric versions taken from wikipedia.

name : windows
version: 7  # technically, this is version 6.1
requires:
- os-windows
variants:
- [arch-x86_64]
- [arch-i386]

name : windows
version: 6.0
version_alias: Vista
requires:
- os-windows
variants:
- [arch-x86_64]
- [arch-i386]

name : windows
version: 5.1
version_alias: XP
requires:
- os-windows
variants:
- [arch-x86_64]
- [arch-i386]

name : windows
version: 5.0
version_alias: 2000
requires:
- os-windows
- arch-i386

One feature that I'd really like to see in rez is the ability to manage external applications using a single multi-version package file.

A quick clarification of terms:

• built package: a package that is built and installed to a central location using rez's build tools; typically contains all the resources for the packages (libraries, executables, includes) under the package's root directory.
• external package: a package which is not built by rez, and whose resources typically exists outside of the central rez install path.

I've added this multi-version package feature in my own fork and used it for awhile, and it makes managing applications like maya and nuke much easier. Previously I had a directory structure like this:

maya/
├── 2012.17
│   │── package.yaml
│   │── platform-darwin
│   └── platform-linux
├── 2013.0
│   │── package.yaml
│   │── platform-darwin
│   └── platform-linux
├── 2014.0
│   │── package.yaml
│   └── platform-linux
├── 2014.2
│   │── package.yaml
│   └── platform-linux
├── 2014.4
│   │── package.yaml
│   └── platform-linux
├── 2014.50
│   │── package.yaml
│   └── platform-linux
├── 2014.51
│   │── package.yaml
│   └── platform-linux
└── 2014.59
    │── package.yaml
    └── platform-linux

(Note that all of those variant sub-directories are empty, they exist because rez errors if they don't. Also, I stopped using the ext symlink + '!ROOT!' because I found it hurt more than helped)

Now, with my fork, I have a single file called maya.yaml that looks like this:

name : maya

config_version : 0

versions :
- "2012.17"
- "2014.0"
- "2014.2"
- "2014.4"
- "2014.50"
- "2014.51"
- "2014.59"

# -- global settings:

variants:
- [ platform-linux ]

help: google-chrome http://download.autodesk.com/global/docs/maya2014/en_us/index.html

commands: |
  # TODO: support padding in rez
  env.MAYA_VERSION_NUM = version.major
  env.CMAKE_MODULE_PATH.append('{root}/cmake')

  if machine.platform == 'linux':
    env.MAYA_LOCATION = '/usr/autodesk/maya{version:#0.#01}-x64'
  elif machine.platform == 'darwin':
    env.MAYA_LOCATION = '/Applications/Autodesk/maya{version:#0.#01}/Maya.app/Contents'
  elif machine.platform == 'windows':
    env.MAYA_LOCATION = 'C:/Autodesk/maya{version.major}-x64'
  env.PATH.prepend('$MAYA_LOCATION/bin')
  env.MAYA_DEBUG_ENABLE_CRASH_REPORTING = 1

tools:
- maya
- mayapy

---

# -- version overrides:

version : 2012|2013

requires:
- python-2.6

# local variant override for 2012/2013:
# moving forward we will only support linux
variants:
- [ platform-linux ]
- [ platform-darwin ]

---

version : 2014

requires:
- python-2.7

The --- separates yaml sub-documents. The first sub-doc provides the global values that apply to all versions, and the remaining sub-docs provide version-specific overrides.

On the one hand, I know that package files are meant to be immutable once released, and while I try to stick to that rule for built packages, for external packages I find it very difficult. Adjustments sometimes need to be made that affect the package.yaml of every version of a package, or, even worse in terms of automation, most of them but not all. This single file gives a comprehensible birds-eye-view of the differences between versions. I put these external package definitions under version control and manage new releases and modifications by editing and committing these files.

Another perk is that when editing many rez package files it is much easier to keep track of what package you're editing, since they're not all named package.yaml.

One issue that I still need to address is time-stamping, as I'm still a novice with rez's timestamp mechanics. I'm hoping that these can be handled either automatically in an external side-car file (e.g. .maya-release.yaml) or by manually, by editing a timestamps metadata field in the package file. I'm open to suggestions.

To me this is a killer feature and I'd like for it to be included in rez, but I'd like to hear what people think.

There are a couple of different docstring formats being used in rez right now, we should decide on a convention and stick to it.

First, we need to decide what documentation generator we will use as that will dictate the available styles. I think that sphinx is the obvious choice, because it is the only actively maintained documentation generator for python out there. I've used epydoc, and it was easy to use, but it was already pretty outdated 5 years ago, so I would not bet on that horse now. Doxygen was designed for C++ and its python support is just a hack on top of that. IIRC, it could not do python introspection on live python objects like sphinx and epydoc, it could just parse python code.

So, if sphinx is the choice we have 3 styles to choose from: native, numpy, and google. here's a sphinx extension that adds supports for the latter 2: http://sphinx-doc.org/latest/ext/napoleon.html

Here they are for your simple viewing pleasure:

sphinx

def func(arg1, arg2):
    """Summary line.

    Extended description of function.

    :param arg1: Description of arg1
    :type arg1: int
    :param arg2: Description of arg2
    :type arg2: str
    :returns: Description of return value
    :rtype: bool

    """
    return True

numpy

def func(arg1, arg2):
    """Summary line.

    Extended description of function.

    Parameters
    ----------
    arg1 : int
        Description of arg1
    arg2 : str
        Description of arg2

    Returns
    -------
    bool
        Description of return value

    """
    return True

google

def func(arg1, arg2):
    """Summary line.

    Extended description of function.

    Args:
        arg1 (int): Description of arg1
        arg2 (str): Description of arg2

    Returns:
        bool: Description of return value

    """
    return True

A note on sphinx:

If you have not used sphinx before, but you have used other documentation generators like epydoc, it might be confusing at first. It is important to know that sphinx is designed in layers: the primary layer is a pure restructuredText parser, which translates an rst file into rich output like html, usually one html file per rst file. Hence, most of the sphinx documentation covers writing restructuredText documents from scratch and has nothing to say about generating API documentation from source code. That is handled by a set of extensions (namely, the autodoc extension) which generate rst documents from python modules and their docstrings, which are then converted into rich output as a second pass.

dynamic shells, caching, rez-egg-install etc

md yay, pdf boo.

Currently rez release releases all variants in a single pass, which is generally a good thing.

However, we are finding that in some cases we have to go back and release an additional variant to an existing package (for example, an additional boost variant). We are unable to do this with the current implementation of rez-release.

One solution would be to increment the sub-patch version number (so 1.0.0 would become 1.0.0.1) however this means releasing all variants again. For some tools this takes many hours. However for external packages (which we are currently releasing using the ExternalProjects macro in CMake we want the version number to match the external number.

Another solution would be to add a -v argument to rez release which behaves in a similar way to rez build. This creates problems with tagging and if misused could result in a package being partially released. That said this might become more necessary if we consider distributing variant builds across a CI system such as Jenkins.

Has anyone else encountered this problem? Does anyone have any suggestions?

I'd like to look into using schema.py to define and validate our metadata schemas in rez.resources. Its Schema class is pretty similar to my MetadataSchema class (formerly MetadataValidator), but is more fully-featured. I think the additional features could come in handy in the future, and could also help consolidate some concepts within rez.resources: namely the idea of 'path patterns'.

Currently we use schemas to describe the contents of files, and we use the
concept of a 'path pattern' to define where we will find that file in our directory tree. I would like to investigate treating the directory structure itself as a schema, with file metadata schemas nested within it. After reading over the docs, I think that schema.py will be flexible enough to do this. I think the hardest part will be modifying the ResourceIterator to work based on Schema instances.

Two other schema validation libraries I looked at are jsonschema and pykwalify. Despite its name, jsonschema is generic enough to validate yaml data, but might be a stretch to validate data coming from python files (callables, for example). The same goes for pykwalify, but to use it we'd also have to port it from python 3.x to 2.x.

schema.py is designed to validate python structures, so that applies equally to data from from json, yaml, or a python file. I think that makes it a good choice.

Timestamps are currently associated with packages, not variants. This needs to be changed, otherwise we cannot facilitate adding of variants to a package after it's been installed - something we're going to need to do.

Hey there,

There is a lot of excitement at my company with adopting rez-config for software configuration ; it feels like it's gonna fix a bunch of headaches for us here. Big thanks for making rez-config.

One thing that I'd love to have thought is not forcing a shell on users ; currently it looks like bash is the only option for working with rez. If you look at virtualenv for python, which also is a system that change the environment (to run a different version of python), it can happily work with a different shell (zsh or tcsh).

It looks like what you'd need is to move to subcommands; this is what git, mercurial, subversion etc ... do; Instead of typing git-add, you just go git add (git SPACE add), and here you go. So this is the only CLI user interface change that people would have to do. git would be rez in your case, and rez would be a python script. In order to not dump the whole source code in it you can still split the code base per sub-command and just import the subcommand.

So you'd go I took a quick glance at your source code and you guys are using optparse in python. If you were to switch to argparse (standard in python 2.7, but it can be backpatched to 2.5, I did that), you would have subcommand support for free.

=> Thoughts ? Would a small throw-away fork on github make sense to see what I have in mind ?

Thanks !

• Benjamin

ps:

1. Great choice on yaml, it's a very readable configuration format.
2. It feels to me that if you'd switch the license to BSD or something like that, it might be easier for company to contribute code.

And ensure that resetenv is working correctly wrt this. Relevant test cases should be added to the commands unittest.

This is not a rez issue, but since there's not a mailing list that I know of yet, so go vote for this ticket: https://bitbucket.org/xi/pyyaml/issue/4/facilitate-modification-of-serialized-yaml

.metadata/info.txt needs a major face lift and should probably happen now while we're introducing so many changes

info.txt stores release info, but why is the release_time in a different file than the release info? Is release_time meaningfully different than ACTUAL_BUILD_TIME and BUILD_TIME from info.txt?

It seems like this file predates use of yaml file format, so maybe now that we have yaml we can consider condensing this all into .metadata/release.yaml or even just .release.yaml.

Also, we need to address version-control-specific data, and no longer assume SVN.

python -V reports 2.7 so when striping the last ., python version appears as 2 and the test version < 25 fails. That should be easy to fix. If rez does not support python 3 it would also be good to make a test to fail configure if python version > 3

From Chad:

• should be achievable by both yaml or python, though python will obviously be more powerful.

• the install() function should make use of rex, and should use the same idioms that we are designing for the commands section. e.g. testing if a package is set:

  if pkgs.mypackage:
      blah = pkgs.mypackage.root
  

• new install-specific commands should also be provided through rex, such as patch()

• we need to write pure python routines for finding libraries, include directories, etc. rez_install_cmake will no longer be necessary for this. finding libraries should support static vs dynamic vs no preference.

  pkgs.mypackage.libraries.foo
  pkgs.mypackage.static_libraries.foo
  pkgs.mypackage.include_directories
  

• the end goal is to remove cmake as a middle-man for building "external" package (e.g. packages which have their own build system which we did not write ourselves).

It should write appropriate format matching given filename extension. Specifically, including .dot.

Hum, not sure what to do about that one ...

bash-3.2$ file /usr/bin/rez
/usr/bin/rez: Mach-O executable i386

https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/Rez.1.html

Currently I am maintaining a private branch of Rez with some studio specific changes.

An example of these changes are a different tag name (we use name-version instead of just version) and popping an event onto ActiveMQ in rez release.

Currently this works ok, but I can envisage problems when it comes to merging Chad's pure python changes. And on an ongoing basis we'll have to do some merging/patching as Rez continues to evolve.

Would there be a benefit to considering plugins for the Rez architecture? The repository strategy is already a plugin of sorts with Git/Svn/Mercurial being chosen at runtime, but perhaps this idea could be extended? Some of the issues (such as tag naming) would be resolved with improved configuration, however sending events is very studio specific and most likely only configurable in real python.

This would be easier to implement with Chad's pure python changes too. What do you think?

The inexact versioning mechanism means that the package descriptor 'foo-5' is supposed to map to foo-5[.x.x.x...]. However if an exact match ('foo-5') exists in the repository, rez will always pick that, and ignore other foo.5.x[.x.x...] packages.

The -i is not supported on the BSD/MAC sed. So I'm just creating a temp file instead.

diff --git a/examples/demo/run_demo b/examples/demo/run_demo
index 9b5861a..704ede5 100755
--- a/examples/demo/run_demo
+++ b/examples/demo/run_demo
@@ -92,10 +92,11 @@ cp -rf ./projects ./build/projects
 find ./build -type d -name '.svn' | xargs rm -rf
 for yaml in `find ./build/projects -name 'package.yaml'`
 do
-       sed -i -e "s/!OS!/$osname/g" \
+       tmpf=`python -c 'import tempfile, sys ; print tempfile.mktemp()'`
+       sed -e "s/!OS!/$osname/g" \
                -e "s/!CXX!/$_REZ_CPP_COMPILER_NAME-$_REZ_CPP_COMPILER_VER/g" \
-               -e "s/!PYVER!/$_REZ_PYTHON_VER/g" \
-               $yaml
+               -e "s/!PYVER!/$_REZ_PYTHON_VER/g" < $yaml > $tmpf
+       mv $tmpf $yaml
 done

eg patching an env containing 'foo-1.5|3' fails - looks like a simple string parsing issue.

I would love to see the rez codebase made more PEP8 compliant: http://www.python.org/dev/peps/pep-0008/

current problems:

• some files use tabs instead of space tabs
• use of type(foo) == type(bar) instead of isinstance
• use of _g_variable for global constants (_ prefix denotes private variables)
• general whitespace issues like foo=3 instead of foo = 3 and foo(1+2) instead of foo(1 +2)
• importing all: from module import *
• importing modules from inside a package without the module prefix: e.g. import rez_config instead of import rez.rez_config.

non-pep8 style issues:

• the rez_ prefix in front of some of the modules is redundant and non-standard. I get the impression that rez used to be a bunch of separate modules that were not under a package, but now that it is a package, there's no longer a need for the prefix.

now is the time to fix these things since the code is in such flux that other changes are highly unlikely to merge anyway.

If you give me the go ahead, I can automatically fix all whitespace issues with autopep8. I use these settings:

autopep8 -i -r -v --pep8-passes=300 --ignore=E501,E302

Hi,

I'd be interested in providing support for Windows.

Having briefly looked through your manual and source-dived part of your 2.0 release, I've gathered that much of Rez is built upon bash, which I can imagine being a challenge if ported to cmd.exe.

Is there anything else you think I should be aware of? Which part of your codebase would you recommend I get started with?

Edit

In an attempt to make this issue referable to newcomers and have them get up to speed more quickly, I'll collect some relevant links here.

• Windows Fork
• Rez Usergroup

Best,
Marcus

So with the init.csh and the mktemp fix I'm a little further, but now that I'm doing real stuff it fails. I can now see the GNU sed errors, I'll try to see how to debug them, and see if something easy can be done to detect the distribution

panzani rez$ uname -a
Darwin panzani.local 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan  6 22:37:10 PST     2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64

panzani rez$ uname -a | awk '{ print $3 }'
12.3.0

Errors below.

[venv] panzani rez$ ./examples/demo/run_demo

INSTALLING REZ LOCALLY...


Detecting operating system...
Operating system is: Darwin

detecting shell...
Shell is: tcsh

detecting OS distribution...
Could not identify OS distribution - either specify manually in configure.sh, or set $REZCONFIG_DISTRO
Setting to DISTRO_UNKNOWN-0!

detecting cmake...
found cmake binary: /usr/bin/cmake

detecting cpp compiler...
found cpp compiler: /usr/bin/c++, id: GNU
cpp compiler version: 4.2.1

detecting python...
found python binary: /Users/bsergean/src/rez/venv/bin/python
python version: 2.7.2

detecting pyyaml...
found pyyaml at /Users/bsergean/src/rez/venv/lib/python2.7/site-packages

detecting pydot...
found pydot at /Users/bsergean/src/rez/venv/lib/python2.7/site-packages

detecting pyparsing...
found pyparsing at /Users/bsergean/src/rez/venv/lib/python2.7/site-packages

There were 1 issues, please review above. Note that installation has still succeeded.
rez.configured written.
Now run ./install.sh


rez 1.6.9 installed successfully.


READYING TEST PACKAGES...

sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory

BUILDING TEST PACKAGES...


BUILDING: hello_world_py-1.0.0

rez-build: invoking rez-config with args:
   --time=1376457195
requested packages: python-!PYVER!
package search paths: /Users/bsergean/src/rez/examples/demo/build/published_packages
Invalid version: Invalid version '!PYVER!'
rez-build failed - an environment failed to resolve.

the merge_osx branch tries to solve some problems with regard to which browser is used to open the help url, but i think ultimately the changes are lipstick on a pig, as they say in the south (or so i'm lead to believe by television).

i think it is generally a bad idea to attempt to specify the application that should open the file / url within the package.yaml. using an environment variable for $BROWSER is at least somewhat portable, but, even assuming everyone has this variable set, it still only solves the problem for webpages.

i think a better solution is to use the OS-specific command for opening a file: e.g. open on osx, and xdg-open on linux (gnome, kde, xfce). this behavior would occur if the help is in the form of an absolute path or url only (i.e. no proceeding application name).

It seems like the order you specify your packages to a rez-env or rez-config call affects the resolution. I experienced this where one of my requested packages was unversioned and the other package had a specific version.

Suppose my package "foo" has two versions -- 1.0 and 2.0. Now suppose I have another package "bar-1.0" that requires "foo-1.0". This call will succeed: "rez-env bar-1.0 foo" where I will have an environment with bar-1.0 and foo-1.0. However, if I switch the order and run "rez-env foo bar-1.0" I will get a conflict saying that foo-1.0 conflicts with foo-2.0.

The problem is that in the latter case where the unversioned package is specified first, rez will first resolve it to the latest version ("foo-2.0" in this case) and then try to resolve the other requests around that. But that seems to go against the general idea of rez where an unversioned request is supposed to be "any compatible version of foo."

This is problematic for end users if they need to specify packages in a particular order to get the right rez environment.

It would be nice if rez allowed 0-padding of version numbers. We name our maya installations after the Maya API version (e.g. 201217 becomes 2012.17), so for initial releases, that means our version is 2014.00 (201400 becomes 2014.00). This is somewhat of a special case, but in general, it would be nice if rez understood a few more common versioning conventions so that we could stick more closely with the actual version used by the vendor. Another example is the 7.0v2 convention used by The Foundry.

..to list info on wrappers currently in the env

the problem is simple:

$ echo $MAYA_LOCATION

$ rez-env maya
> $ echo $MAYA_LOCATION
/usr/autodesk/maya2014.51-x64
> $ rez-env nuke
>> $ echo $MAYA_LOCATION
/usr/autodesk/maya2014.51-x64

rez makes no effort whatsoever to remove old variables.

This is a pretty big problem. what if you drop into a shell with a resolve that sets LD_LIBRARY_PATH, then drop into another shell below that which does not. LD_LIBRARY_PATH is still set, potentially along with dozens of other variables which are not relevant and potentially harmful to the current resolve.

I believe if we can keep track of which variables rez sets, then we can unset all of those before the next resolve. This is one of the reasons that I'm not a fan of the source command, since it can wreak total havok on the environment in ways that rez cannot easily track (aside from the fact that it breaks promise of parity between various shells).

I can't figure out that one ... I tried on a Linux box and it works perfectly; I installed a non-system cmake in /usr/local/bin, I've installed py-parsing + pydot to the system python. Still no luck, not sure what to try now, and I'm not familiar with cmake unfortunately.

INSTALLING REZ LOCALLY...


Detecting operating system...
Operating system is: Darwin

detecting shell...
Shell is: tcsh

detecting OS distribution...
OS distribution is: Apple

detecting OS distribution version...
OS distribution version is: 10.8.3

detecting cmake...
found cmake binary: /usr/local/bin/cmake

detecting cpp compiler...
found cpp compiler: /usr/bin/c++, id: Clang
Couldn't detect compiler version, assuming 1.0.1...
cpp compiler version: 1.0.1

detecting python...
found python binary: /usr/bin/python
python version: 2.7.2

detecting pyyaml...
found pyyaml at /Library/Python/2.7/site-packages

detecting pydot...
found pydot at /Library/Python/2.7/site-packages

detecting pyparsing...
found pyparsing at /Library/Python/2.7/site-packages

There were 1 issues, please review above. Note that installation has still succeeded.
rez.configured written.
Now run ./install.sh


rez 1.6.9 installed successfully.


READYING TEST PACKAGES...


BUILDING TEST PACKAGES...


BUILDING: hello_world_py-1.0.0

rez-build: invoking rez-config with args:
   --time=1376545154
requested packages: python-2.7.2
package search paths: /Users/bsergean/src/rez/examples/demo/build/published_packages

rez-build: in new env:

running rez-config v1.6.9

REZ_REQUEST_TIME=1376545154
requested packages (mode=latest, time=1376545154: Wed Aug 14 22:39:14 UTC 2013):
Darwin
python-2.7.2
cmake-2.8.9

resolved packages:
Darwin              /Users/bsergean/src/rez/examples/demo/build/published_packages/Darwin               (local)
cmake-2.8.9         /Users/bsergean/src/rez/examples/demo/build/published_packages/cmake/2.8.9/Darwin   (local)
python-2.7.2        /Users/bsergean/src/rez/examples/demo/build/published_packages/python/2.7.2/Darwin  (local)

number of failed attempts: 0

context file:
/Users/bsergean/src/rez/examples/demo/build/projects/hello_world_py/1.0.0/build/build-env.context


rez-build: calling rez-cmake -d ../
rez-cmake: calling cmake with the following arguments:  -DCMAKE_MODULE_PATH=/Users/bsergean/src/rez/examples/demo/build/rez_local_install/1.6.9/cmake;/Users/bsergean/src/rez/examples/demo/build/published_packages/Darwin/cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_SKIP_RPATH=1 ../
-- The C compiler identification is GNU 4.2.1
-- The CXX compiler identification is Clang 4.0.0
-- Could not determine Eclipse version, assuming at least 3.6 (Helios). Adjust CMAKE_ECLIPSE_VERSION if this is wrong.
-- Checking whether C compiler has -isysroot
-- Checking whether C compiler has -isysroot - yes
-- Checking whether C compiler supports OSX deployment target flag
-- Checking whether C compiler supports OSX deployment target flag - yes
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.27.1") 
CMake Error at /Users/bsergean/src/rez/examples/demo/build/rez_local_install/1.6.9/cmake/RezInstallPython.cmake:31 (message):
  a version of python must be listed as a requirement when using the
  'rez_install_python' macro.  Packages for this build are: python-2.7.2;
Call Stack (most recent call first):
  CMakeLists.txt:7 (rez_install_python)

Trying to build a package with a version number of '0' breaks, .e.g, in package.yaml

version: 0

This will cause rez-build to fail.

We are starting to build packages with both ICC and GCC variants. As ICC requires GCC at run time (and I think build time), my icc package looks something like this:

name: icc
version: 14.0.0
variants:
    - [ CentOS-6.2, gcc-4.1.2 ]
    - [ CentOS-6.2, gcc-4.4.6 ]
commands:
    - CXX=!ROOT!/bin/icpc
    - CC=!ROOT!/bin/icc

To build all ICC and GCC variants of a higher level package, we must do something like this:

name: foo
version: 1.0.0
variants:
    - [ CentOS-6.2, gcc-4.1.2 ]
    - [ CentOS-6.2, gcc-4.4.6 ]
    - [ CentOS-6.2, icc-14.0.0, gcc-4.1.2 ]
    - [ CentOS-6.2, icc-14.0.0, gcc-4.4.6 ]

This creates three problems:

1. The icc package redefines the CXX and CC environment variables, which are also defined in the gcc package. Rez does not allow this (it's a conflict) and so temporarily I've commented this restriction out. I understand the restriction, but maybe we can loosen it a little?
2. The foo package now has unbalanced variants. This doesn't appear to be a problem (it works) but is maybe not considered very neat. This would perhaps benefit from naming the variant folders as we discussed in another thread.
3. In theory we are able to successfully mix and match compilers are runtime - they are not mutually exclusive. Assuming another package bah with the same variants as foo we can build an environment using the gcc-4.4.6 variant of foo and the icc-14.0.0, gcc-4.4.6 variant of bah.

Currently (I believe) a call rez-env foo bah will pick the first variant that creates a resolution (both GCC), rez-env foo bah icc-14.0.0 gcc-4.4.6 will resolve for both packages to use the ICC variant. Is there currently some syntax I can use to be more specific, e.g. rez-env foo:gcc-4.4.6 bah:icc-14.0.0?

Perhaps this is an extension of what was discussed in issue 21.

To extend point 1) a little further with another use case:

We want to define a PYTHON_EXE environment variable which points to the current python interpreter for the resolved environment. For example:

name: maya
requires:
    -python-2.6
commands:
    - export PYTHON_EXE=mayapy

name: python
version: 2.6.6
commands:
    - export PYTHON_EXE=python2.6

As both the python and maya package define PYTHON_EXE we get a conflict. The way around this I see would be to define the python-2.6 requirement of maya as a separate mayapy package.

name: maya
requires:
    -mayapy-2.6

name: mayapy
version: 2.6.6
commands:
    - export PYTHON_EXE=mayapy

But our python applications have three variants for python 2.5, 2.6 and 2.7 and for applications which are pure python, they work in a normal python interpreter, as well as Maya/Nuke/Houdini's. To follow this approach we'd have to build many more variants of essentially the same code which seems wasteful:

name: foo
variants:
    - [ python-2.5 ]
    - [ python-2.6 ]
    - [ python-2.7 ]
    - [ mayapy-2.6 ]
    - [ hython-2.6 ]
    - [ nukepy-2.6 ]

I don't like this approach. Any suggestions?

If a package.yaml was modified after it was cached 'rez-env' will only return the (outdated) cached entry. 'rez-env --no-cache' returns the expected updates.

Workaround: restart memcached server