sclorg / container-common-scripts Goto Github PK

goldmann/docker-squash#145 and goldmann/docker-squash#144 -> we should fix our scripting so it works for both versions, or use the library.

I'm using Vagrant for testing and it rejects to sync common scripts to the virtual machine (using rsync) because there is an infinite loop of symlinks: common/tests/failures/check contains common which is a symlink back to ../../.. which causes this loop:

common/tests/failures/check/common/tests/failures/check/common/tests/failures/check/common/tests/failures/check/common…

Is this something we need to have here?

I am looking into the new generate/generate-all makefile rules and I realized there is a DG_CONF and conf value hardcoded in common.mk and generate.sh file as well. It is not even possible to change it using some environment variable. These scripts support only Fedora 27, s2i-python-container, postgresql-container and maybe some other projects contains only Fedora 26 Dockerfile, which means these files can't be properly generated using generate rule from common scripts.

The RHSCL CI is blocked when OpenShift 3.X does not start up.

We have to investigate how to start OpenShift 3. properly so tests are not blocked.

See.

Hidden files .image-id and .image-id.raw are not removed after build.

So in scenario:

1. make test TARGET=centos7
2. make test TARgET=fedora

Fedora is not supported for all versions. But the second point uses the hidden files from previous build and for some versions it tests centos based images.

Also for some images these hidden files are in .gitignore, so user can't be even aware that some old files needs to be cleaned.

Brain dump of some ideas how to make the debugging easier:

• print docker/podman version at the beginning of the test (plus some other related packages)
• do some performance check at the beginning (how quickly network works by pulling some small image and installing a package; how quickly a container starts) -- seeing bigger numbers might indicate that the issues are caused by infra performance
• add possibility to not dispose a machine immediately -- some message like [keep-alive: 2h] in the github comment could make the machine be kept a little longer after the test is done (see https://plugins.jenkins.io/ghprb/ for how to work with the comments)
• similarly as the point above [debug] could turn on the debug (more verbose) output for the tests

Sometimes, we got an infrastructure error like this:

https://ci.centos.org/job/SCLo-container-python-rh-fedora/281/console

16:44:11 VERSION="3.7" SKIP_SQUASH=1 UPDATE_BASE=1 OS=fedora CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="" CUSTOM_REPO="" /usr/bin/env bash common/build.sh
16:44:11 -> Version 3.7: building image from 'Dockerfile.fedora' ...
16:44:11 Sending build context to Docker daemon 94.72 kB

16:44:11 Step 1/14 : FROM registry.fedoraproject.org/f31/s2i-base:latest
16:44:11 Trying to pull repository registry.fedoraproject.org/f31/s2i-base ... 
16:44:12 latest: Pulling from registry.fedoraproject.org/f31/s2i-base
16:44:12 82422234ceca: Pulling fs layer
16:44:12 64f48977383f: Pulling fs layer
16:44:12 c8ba9d4c9a9a: Pulling fs layer
16:44:12 error pulling image configuration: received unexpected HTTP status: 503 Service Unavailable
16:44:12 make[1]: Leaving directory `/root/sources'
16:44:12 make[1]: *** [3.7] Error 1
16:44:12 make: *** [build-serial] Error 2
16:44:12 Build step 'Execute shell' marked build as failure
16:44:12 [PostBuildScript] - Executing post build scripts.
16:44:12 [SCLo-container-python-rh-fedora] $ /bin/bash /tmp/jenkins2251871663693525052.sh
16:44:12 Warning: Permanently added 'n37.pufty.ci.centos.org,172.19.3.101' (ECDSA) to the list of known hosts.
16:44:12 [PostBuildScript] - Executing post build scripts.

This can be solved in this code: https://github.com/sclorg/container-common-scripts/blob/master/build.sh#L111
I would say, try it two times, not more.

Up until now we did not have an issue with different versions for the same Fedora release as the version was tied to the release (we could not have had eg. both nodejs 10 and 12 on Fedora XY).
This has however changed with modularity and we need to update the image tags to reflect the possibility.

Recently hit in: sclorg/s2i-nodejs-container#246

This issue is going to be used as a tracker for testing SCL containers in OpenShift 4 environment.
Please reference all PRs in container-common-scripts into this issue.

Tests are run by PR comment [test-openshift-4].

What should be done:

• set path to oc binary to version 4 ct_os_set_path_oc
• set docker registry by Image Registry Operator (https://docs.openshift.com/container-platform/4.3/registry/configuring-registry-operator.html)
• ???

PR:

• #145
• sclorg/rhscl-container-ci#35

Proof of Concept: sclorg/s2i-php-container#286

We miss tests for OpenShift.

It needs to be enhanced at least for one image.

We should have a simple test that would be called for images where we expect nodejs and npm to be available. Something like npm install react, npm --version, ...

I'm not sure if this can be caused by mine setup. It started to behave like this only recently.

$ basename "`pwd`"
s2i-ruby-container

I have pulled most recent common scripts(a8d2885) and I'm on subscribed RHEL7 VM.

$ PATH="$PATH:~/.local/bin" make build TARGET=rhel7 VERSIONS=2.4 TEST_MODE=yes
 . . .
2017-09-20 16:19:59,739 root         INFO     New squashed image ID is c92ecb5b50cb3337dd7256d28823bc2a134571cdd575546ec0d40dd8ab5cf60a
2017-09-20 16:20:17,347 root         INFO     Done
ruby 2.4 => c92ecb5b50cb3337dd7256d28823bc2a134571cdd575546ec0d40dd8ab5cf60a
make[1]: Leaving directory `/home/vagrant/Work/RH/centos/containers/s2i-ruby-container'

No tagged image is present, but it's found by ID (found in output above).

$ docker images -a | grep -E '(s2i|ruby|c92ecb5b50cb)'
<none>                                                      <none>              c92ecb5b50cb        About an hour ago   1.05 GB
registry.access.redhat.com/rhscl/s2i-base-rhel7             1                   d46c583c0e87        3 weeks ago         459.8 MB

Is this expected behavior?

https://google.github.io/styleguide/shell.xml#Function_Names suggests to use :: separator for libraries separation, so we can end up with something like:

ct::os::print_logs "some"

instead of

ct_os_print_logs "some"

During the testing in CVP this issue was caught:

Testing in CVP environment. No need to login to OpenShift cluster. This is already done by CVP pipeline.
In project cvp-5c79a57dd70cc51dd4c37825-jlgfj (CVP sandbox) (cvp-5c79a57dd70cc51dd4c37825-jlgfj) on server https://172.27.0.1:443

You have no services, deployment configs, or build configs.
Run 'oc new-app' to create an application.
Testing in CVP environment. No need to create OpenShift project. This is done by CVP pipeline
Testing image nodejs-14 in CVP pipeline.
error: only a partial match was found for "nodejs-14:14": "cvp-5c79a57dd70cc51dd4c37825-jlgfj/nodejs-14:latest"

The argument "nodejs-14:14" only partially matched the following Docker image, OpenShift image stream, or template:

* Image stream "nodejs-14" (tag "latest") in project "cvp-5c79a57dd70cc51dd4c37825-jlgfj"
  Use --image-stream="cvp-5c79a57dd70cc51dd4c37825-jlgfj/nodejs-14:latest" to specify this image or template

Waiting for nodejs-14-testing pod becoming ready .................................................................................................... FAIL
NAME                                       IMAGE REPOSITORY                                                                                TAGS      UPDATED
imagestream.image.openshift.io/nodejs-14   image-registry.openshift-image-registry.svc:5000/cvp-5c79a57dd70cc51dd4c37825-jlgfj/nodejs-14   latest    5 minutes ago
In project cvp-5c79a57dd70cc51dd4c37825-jlgfj (CVP sandbox) (cvp-5c79a57dd70cc51dd4c37825-jlgfj) on server https://172.27.0.1:443

You have no services, deployment configs, or build configs.
Run 'oc new-app' to create an application.
Build config bc/ does not exist for some reason.
Import probably failed.
OpenShift tests for nodejs-14 failed.

There are two possibilities:

1. In case of CVP, we have to use tag latest.
2. During the import into OpenShift 4, we can tag it to nodejs-14:14.
   The taging can be done in file: https://github.com/sclorg/container-common-scripts/blob/master/test-openshift.yaml
   especially after this ansible command:
   https://github.com/sclorg/container-common-scripts/blob/master/test-openshift.yaml#L53

In my opinion, the first option is better.

WDYT?

By changing this repo we should not directly influence other images, since they contain specific references to this repo and updating those references should be tested on each-image basis. It might still make sense to run builds of other images (not necessarily tests for those, since this repo contains mostly build-time only things) -- just to see how much harm a particular PR can cause in the future, which should help creating the PR without requiring to change code in depended repositories.

E.g. #65, #62. Basically, OpenShift isn't providing enough API for avoiding such hacks, but that should at least be discussed with openshift upstream so we know how to do proper scripting correctly. Reading the openshift testing library (edit: i mean test-lib-openshift.sh), I more and more think that the library should be more than just testing library, but rather convenient library for daily work with openshift...

Tests in openshift are failing in python container with the latest version of common module.

16:44:12 Uploading image centos/python-27-centos7:2.7 as python-27-centos7:2.7:2.7
16:44:17 Error response from daemon: Get http://172.30.1.1:5000/v1/users/: dial tcp 172.30.1.1:5000: connect: connection refused
16:44:22 Login Succeeded
16:44:22 Error parsing reference: "172.30.1.1:5000/sclorg-test-31971/python-27-centos7:2.7:2.7" is not a valid repository/tag

See the image name with two tags python-27-centos7:2.7:2.7. The full log is here.

The commit 24a8d12 seems suspicious to me. Is it possible that the IMAGE_NAME variable already contains the tag and the ct_os_test_s2i_app_func function adds another one in image_tagged?

I've checkouted the submodule in sclorg/s2i-python-container#394 from 57d2c55 back to e65a5f5 and we'll see. The non-os tests there already failed but for a different reason.

There are already some Python test which has to be checked by each PR.

It would be nice to at there some pre-commit checker like https://github.com/sclorg/betka/blob/master/.pre-commit-config.yaml

ct_os_install_in_centos function is not used in:

However, instead of removing it, it might be good to have a binary (e.g. ct-centos-prepare-machine, similarly can be then ct-fedora-prepare-machine) in common scripts that would call this function. But this must be consulted before, since I'm personally not sure whether this will be useful.

I've broken test_from_dockerfile by the implementation of the branch name support in b3d5a0a

It seems that the systems our CI is running on are too old and the readarray command there doesn't provide -d option. From my testing, the option is supported in bash version 5 but not in version 4.

I'll try to fix it asap. Please do not update in the meantime.

If the docker registry returns HTTP 503, I would prefer to run docker pull once more.
Sometimes, a second run works properly.

What do you think? @pkubatrh @hhorak @praiskup.

See the error:

make[1]: Entering directory `/tmp/daily_scl_tests/s2i-nodejs-container'
mkdir -p 10/root
go-md2man -in "10/README.md" -out "10/root/help.1"
chmod a+r "10/root/help.1"
VERSION="10" SKIP_SQUASH=1 UPDATE_BASE= OS=fedora CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="" CUSTOM_REPO="" /usr/bin/env bash common/build.sh
-> Version 10: building image from 'Dockerfile.fedora' ...
Sending build context to Docker daemon 50.69 kB

Step 1/13 : FROM registry.fedoraproject.org/f29/s2i-base:latest
Trying to pull repository registry.fedoraproject.org/f29/s2i-base ... 
latest: Pulling from registry.fedoraproject.org/f29/s2i-base
7692efc5f81c: Pulling fs layer
aaf5ad2e1aa3: Pulling fs layer
05ba4d604a56: Pulling fs layer
error pulling image configuration: received unexpected HTTP status: 503 Service Temporarily Unavailable
make[1]: *** [10] Error 1
make[1]: Leaving directory `/tmp/daily_scl_tests/s2i-nodejs-container'
make: *** [build-serial] Error 2

So anyone can hook some custom target into 'make clean', like:

include common/common.mk 
clean-hook += my_clean

(naming by automake)

It would be nice to be able to selectively pick particular VERSIONs for remote images, e.g. And later even the test-cases.

We could employ ShellCheck or some other static analysis tools (or even plain sed calls...)

Moved from sclorg/postgresql-container#210.

.. and simply expect that OpenShift is installed on the box, or fail (a bit related to #40).

It would be really suspicious to require our end-users/contributors to run the openshift test-scripts "locally" ATM, since the tasks require root account. We could have some script for user's convenience (doing the oc installation), but the images in CI should be as much prepared as possible, same as user's boxes -- and we should only provide the login info into the openshift instance.

Reported in sclorg/s2i-python-container#220. So moving there.

on mac osx 10.12.6
$make build TARGET=centos7 VERSIONS=2.7
VERSION="2.7" SKIP_SQUASH=1 UPDATE_BASE= OS=centos7 CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="3.3" common/build.sh
common/build.sh: line 34: syntax error near unexpected token `;;'
make[1]: *** [2.7] Error 2

.. so it looks like the rc was scaled up to 2:
https://ci.centos.org/job/SCLo-container-postgresql-rh-openshift/43/consoleText

We can see errors during build like this:

https://ci.centos.org/job/SCLo-container-container-common-scripts-test/202/console

22:24:55 + cd sources
22:24:55 + make check
22:24:55   RUN TEST 'path_foreach'
22:24:55   RUN TEST 'random_string'
22:24:55   RUN TEST 'test_npm'
22:24:55 + . test-lib.sh
22:24:55 ++ EXPECTED_EXIT_CODE=0
22:24:55 + NPM_REGISTRY=
22:24:55 ++ ct_build_s2i_npm_variables
22:24:55 ++ npm_variables=
22:24:55 ++ '[' -n '' ']'
22:24:55 ++ echo ''
22:24:55 + output=
22:24:55 + test x == x
22:24:55 + ca_file=/etc/pki/ca-trust/source/anchors/RH-IT-Root-CA.crt
22:24:55 + NPM_REGISTRY=https://foobar.registry.org
22:24:55 + '[' -f /etc/pki/ca-trust/source/anchors/RH-IT-Root-CA.crt ']'
22:24:55   RUN TEST 'image_availability'
22:25:50 quay.io/centos7/ruby-24-centos7 could not be downloaded via 'docker'
22:25:50 image_availability test completed successfully.
22:25:50   RUN TEST 'public_image_name'
22:25:50 public_image_name test completed successfully.
22:25:50 cd tests/failures/check && make tag && ! make check && make clean
22:25:50 make[1]: Entering directory `/root/sources/tests/failures/check'
22:25:50 make[2]: Entering directory `/root/sources/tests/failures/check'
22:25:50 mkdir -p v0/root
22:25:50 go-md2man -in "v0/README.md" -out "v0/root/help.1"
22:25:50 chmod a+r "v0/root/help.1"
22:25:50 VERSION="v0" SKIP_SQUASH=1 UPDATE_BASE= OS=centos7 CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="" CUSTOM_REPO="" REGISTRY=""quay.io/"" /usr/bin/env bash common/build.sh
22:25:50 -> Version v0: building image from 'Dockerfile' ...
22:25:50 -> Pulling image centos/s2i-core-centos7 before building image from Dockerfile.
22:25:50 Using default tag: latest
22:25:50 Trying to pull repository docker.io/centos/s2i-core-centos7 ... 
22:25:51 toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
22:25:51 Pulling image centos/s2i-core-centos7 failed.
22:25:51 Let's wait 5 seconds and try again.
22:25:56 Using default tag: latest
22:25:56 Trying to pull repository docker.io/centos/s2i-core-centos7 ... 
22:25:56 toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

It looks like command not found in the test suite.

https://gist.github.com/rhscl-automation/a424c41937ec2b077b74bbab1c7b56e3#file-s2i-ruby-container-231-sh-L12574

The aim of the tool, in Python, would be to do the following things.

• Automatically update upstream SCL GitHub repository to the new
• Copy / create a directory structure like is mentioned here https://github.com/sclorg/welcome/blob/master/contribution.md#adding-a-new-version
• It would be nice to run also tests on all architectures, like Fedora, CentOS, RHEL
• Store output into the JSON file. What failed, what was successful
• TBD

.. and also on not subscribed RHEL boxes (so far, docker build for RHEL images works
only on successfully subscribed RHEL7 box).

Per @hhorak report, it is possible to (optionally) provide -v /host.repo:/etc/yum.repos.d/host.repo option for docker build . command.

IIUC In order to maintain compatibility of c++ software with centos6, it is necessary to build it on centos6.

I've got patches for creating devtoolset images for centos6 but they require having support for that on this project.

Would it be possible to add this support so that adding a Dockerfile.centos6 file creates centos6 versions?

At your disposal for any suggestions/critiques/ideas

@omron93 proposed this idea in #19, and I'm +1, thoughts?

Sometimes it happens that we have only RHEL packages available and CentOS are not yet build, but we want to submit the PR with new version. Or we might only have Fedora variant available, but no CentOS or RHEL Dockerfiles. So, the testing (building) script should work the way that the testing would not be run (and test whould not fail) in cases the particular Dockerfile variant is missing (e.g. only Dockerfile.rhel7 is available and not Dockerfile for CentOS).

This is quite suspicious, that calling distgen few times in a row results in different output every time. I smell timestamps on files might be the cause here.

(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  DG     9.6/cccp.yml
  DG     9.6/root/usr/share/container-scripts/postgresql/README.md
  DG     9.6/root/usr/share/container-scripts/postgresql/common.sh
  DG     9.6/root/usr/bin/run-postgresql-slave
  DG     9.6/root/usr/share/container-scripts/postgresql/openshift-custom-recovery.conf.template
  DG     10/cccp.yml
  DG     10/root/usr/share/container-scripts/postgresql/README.md
  DG     10/root/usr/share/container-scripts/postgresql/common.sh
  DG     10/root/usr/bin/run-postgresql-slave
  DG     10/root/usr/share/container-scripts/postgresql/openshift-custom-recovery.conf.template
  DG     11/cccp.yml
  DG     11/root/usr/share/container-scripts/postgresql/README.md
  DG     11/root/usr/share/container-scripts/postgresql/common.sh
  DG     11/root/usr/bin/run-postgresql-slave
  DG     11/root/usr/share/container-scripts/postgresql/openshift-custom-recovery.conf.template
  CP     9.6/root/usr/libexec/fix-permissions
  CP     9.6/content_sets.yml
  CP     9.6/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql-replication.conf.template
  CP     9.6/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql.conf.template
  CP     9.6/root/usr/share/container-scripts/postgresql/scl_enable
  CP     9.6/root/usr/bin/run-postgresql
  CP     9.6/root/usr/bin/run-postgresql-master
  CP     9.6/root/usr/bin/container-entrypoint
  CP     9.6/root/usr/bin/usage
  CP     9.6/root/usr/libexec/check-container
  CP     9.6/root/usr/share/container-scripts/postgresql/start/set_passwords.sh
  CP     9.6/s2i/bin/assemble
  CP     9.6/s2i/bin/usage
  CP     10/root/usr/libexec/fix-permissions
  CP     10/content_sets.yml
  CP     10/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql-replication.conf.template
  CP     10/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql.conf.template
  CP     10/root/usr/share/container-scripts/postgresql/scl_enable
  CP     10/root/usr/bin/run-postgresql
  CP     10/root/usr/bin/run-postgresql-master
  CP     10/root/usr/bin/container-entrypoint
  CP     10/root/usr/bin/usage
  CP     10/root/usr/libexec/check-container
  CP     10/root/usr/share/container-scripts/postgresql/start/set_passwords.sh
  CP     10/s2i/bin/assemble
  CP     10/s2i/bin/usage
  CP     11/root/usr/libexec/fix-permissions
  CP     11/content_sets.yml
  CP     11/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql-replication.conf.template
  CP     11/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql.conf.template
  CP     11/root/usr/share/container-scripts/postgresql/scl_enable
  CP     11/root/usr/bin/run-postgresql
  CP     11/root/usr/bin/run-postgresql-master
  CP     11/root/usr/bin/container-entrypoint
  CP     11/root/usr/bin/usage
  CP     11/root/usr/libexec/check-container
  CP     11/root/usr/share/container-scripts/postgresql/start/set_passwords.sh
  CP     11/s2i/bin/assemble
  CP     11/s2i/bin/usage
  LN     9.6/README.md
  LN     9.6/test
  LN     9.6/s2i/bin/run
  LN     10/README.md
  LN     10/test
  LN     10/s2i/bin/run
  LN     11/README.md
  LN     11/test
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk

The test for the PostgreSQL container includes a nice function add_cleanup_command that allows to extend the default cleanup action. It would be good to have this function in the test-lib.sh so we can use it in other container images tests as well:
https://github.com/sclorg/postgresql-container/blob/1cbbb075/test/run_test#L48-L58

... until the IP is available?

It is really ugly that we loose the git history all the time with new versions of our containers, speaking in examples - consider PostgreSQL container:

1. working with 9.2 directory, we iterated a lot (git blame and git history gives us useful info
2. but then we needed to work on 9.4, so we copied and pasted whole 9.2 directory into 9.4, which means that the whole 9.4 directory lost all the useful commit history
3. moving to 9.5 made this even worse, and upcoming 9.6 won't help obviously

Well, e.g. in Fedora dist-git, we have master branch which guarantees that all the necessary info for git blame persists. It is fair also to keep the "authorship" of contributions.

Perhaps we should have e.g. latest directory now (instead of 9.6) and make the 9.6 be a symbolic link to latest?

The squash.py script fails on RHEL8 as it is using docker_squash but we are using podman on RHEL8.

Not sure if we even need to squash the images during the tests in this repository, so disabling the squashing phase would be an easy way to work around this issue.

There are ugly hacks in the distgen logic related to *.rhel7 lsuffix (e.g. #75 fights with that), but there's no easy way to drop the old layout because it is required by:

• non distgen-enabled images
• common script scripts
• rebuild helper
• CI (probably)

So we would have to either (a) move all containers towards the new layout, or (b) support two layouts. IMO, (b) is less intrusive (using generators in source repo should stay opt-in) and I would prefer that. But anyways, take this issue as a tracker and place to discuss issues.

Additional note to #75, it would be much more natural if we generated (distgen layout) just:

postgresql-container/
└── output/
    ├── Dockerfile
    └── root/

Or, on demand (e.g. to file the generated branch):

postgresql-container/
├── 10
│   ├── centos-7-x86_64
│   │   ├── Dockerfile
│   │   └── root
│   └── rhel-7-x86_64
│       ├── Dockerfile
│       └── root/
...

As suggested in sclorg/httpd-container#94, there could be a generic ct_run_cmd function similar to what https://github.com/sclorg/httpd-container/ and some other images use -- such a function might check the output and exit code, might also print some standardized logging etc...

A note: we might get some more inspiration in beakerlib where rlRun has a similar function.

With umask set to 077, the image build and test might fail. While we might not fix this properly, it would be good to find out that the umask is the cause soon by having explicit check for umask during:

• make build
• a new function in test-lib.sh to be called in every test/run

It might be included in one of the general functions that is used in all containers (or supposed to be used that way). For example the ct_show_results.

Users should be able to append to:
CLEAN_FILES += file

So when make clean is hit, files from $CLEAN_FILES are removed.

##Description

When running make, I get an syntax error in common/build.sh

Versions

Using OSX 10.13.2
Tried both ZSH & Bash
GNU bash, version 4.4.12(1)-release (x86_64-apple-darwin17.0.0)
GNU Make 4.2.1 Built for x86_64-apple-darwin17.3.0

$ make build TARGET=rhel7 VERSIONS=7.1

VERSION="7.1" SKIP_SQUASH=0 UPDATE_BASE= OS=rhel7 CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="5.5" common/build.sh
common/build.sh: line 55: syntax error near unexpected token `{stdout_fd}'
make[1]: *** [common/common.mk:53: 7.1] Error 2
make[1]: Leaving directory '/Users/bobby/code/s2i-php-container'
make: *** [common/common.mk:42: build-serial] Error 2

$ make check

VERSION="5.6" SKIP_SQUASH=1 UPDATE_BASE= OS=centos7 CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="5.5" common/build.sh
common/build.sh: line 55: syntax error near unexpected token `{stdout_fd}'
make[1]: *** [common/common.mk:53: 5.6] Error 2
make[1]: Leaving directory '/Users/bobby/code/s2i-php-container'
make: *** [common/common.mk:42: build-serial] Error 2

Hmm

I've tried every syntax checker there is, and it all looks fine - as it probably is given others are using it. But don't know why this is happening?

parse_output ()
{
  local command=$1 filter=$2 var=$3 stream=$4
  local raw_output= rc=0
  {
      raw_output=$(_parse_output_inner)
  } {stdout_fd}>&1 {stderr_fd}>&2
  rc=$?
  eval "$var=\$raw_output"
  (exit $rc)
}

Any clues? :/

Seems like this is result of some work-around from 5fe4cbb commit. But explicitly installing v1.10.6 fails for me with:

$ easy_install -q --user docker_py==1.10.6 docker-squash==1.0.5
error: Setup script exited with error in docker-py setup command: Invalid environment marker: python_version < "3.5"

OTOH, easy_install -q --user docker-squash==1.0.5 works just fine. Do we still need explicit version of docker_py?

• git doesn't track file permissions, thus
• git clone initiates the permissions according the user's umask value
• docker build just copies the files as-is into container
• especial problems are with root/usr directory created by ADD root / command, this affects the whole image (/usr directory might be unreadable)

In RPMs' specfiles, we explicitly set the permissions in %install phase (or by %attr in %files), though there's no such way in Dockerfile.

Many tests share similar or same parts, like:

• cleaning the containers after running the test
• running the container while storing the ID into a cidfile
• checking the man help.1 file
• ...

So we should eventually have at least set of functions shared in more containers to make the tests maintenance easier.

We should not only test whether the file exists but also if it is a symlink in case it fails the -e test.

https://github.com/sclorg/container-common-scripts/blob/master/test.sh#L23