plus3it / amigen7 Goto Github PK

Problem Description:

GetAmznLx.sh fails on RHEL 7.6.

Expected Behavior:

Script should detect RHEL version as el7 and get the proper rpms.

Actual Behavior:

ELVERSION returns "el7.6". This causes the case evaluation on line 29 to fail since it is looking for "el7".

(Detailed) Steps to reproduce:

Run the script on RHEL 7.6.

(Optional) Fix recommendation:

Insert a "cut" at the end of the command to strip off the .6.

Problem Description:

Once new EBS Volume is mounted as boot volume for ec2 instance, unable to login via ec2-user or any other user. Existing Private Key refused.

Expected Behavior:

Existing private key for ec2-user should be replicated to newly built LVM EBS volume

Actual Behavior:

Unable to login to instance with new LVM EBS volume

(Detailed) Steps to reproduce:

Perform the public network processes.
Swap boot EBS with LVM ebs, so that LVM EBS is new boot device (/dev/sda1)
Power on instance
Attempt to login via ec2-user with initial private key or any other user.

(Optional) Fix recommendation:

Ensure existing users and ec2-user keys are replicated on new EBS volume.

Nature of documentation error

• Typographical or Grammar Errors
• Vague
• Misleading/Counter-intuitive
• Wholly Incorrect
• Missing Document (e.g., broken link-text)

Document Location

Most of the contents in the Docs folder is YEARS old and no longer accurately reflects the content they're supposed to document the usage of.

Relate: the disk setup script probably needs to be fully-documented to make usage of the -p/--partitioning option more obvious.

Proposed Fix

Problem Description:

Due to NVMe devices using different naming conventions for disks/partition (/dev/nvmeXnYpN vice /dev/xvdXN), script fails when using ${CHROOTDEV}N as targeting-method. See:

• Line 56
• Line 84
• Line 93
• Lines 95-96
• Lines 110-111

Expected Behavior:

The DiskSetup.sh script should work on both NVMe- and non NVME-enabled instance-types

Fix recommendation:

Change targeting method used for the above-enumerated lines.

Is your feature request related to a problem? Please describe.

When creating non-AWS build-types – particularly those that don't typically include execution of userData, it will be necessary for the root account to be enabled for direct-login to VMs spawned from resultant templates

Describe the solution you'd like

1. Add ability to easily set a root password into the resultant templates
2. Add ability to SSH directly into the root user account with the set password

Is your feature request related to a problem? Please describe.

AMIGen7's current configuration of cloud-init cause the default-user to be created with the SELinux user-context, unconfined_u. With newer security guidelines applied, this context will make it so that the default-user cannot actually escalate privileges, as needed, due to a bad, combined SELinux label – seen when doing sudo -i as:

$ sudo -i
sudo: unconfined_u:sysadm_r:sysadm_t:s0-s0:c0.c1023 is not a valid context

Describe the solution you'd like

Change the default-user's SELinux user-context to staff_u

Describe alternatives you've considered

Additional context

Describe the bug
AMIgen will (effectively) silently fail if the @Core RPM group is not available in the source yum repository.

Severity
FATAL: script appears to work as expected but the error is hidden. This can result in what appear on the face to be successful AMI-creation runs that, instead, produce broken AMIs

Fix Suggestions
Update ChrootBuild.sh script to add an error-exit immediately after Line 170 if the value of INCLUDE_PKGS is null.

Describe the bug
When using to generate VMware and other image-types, /var/run does not get created as a symlink to /run

Severity

• Completely Broken (No work-around evident)
• Severely Broken (Work-around possible but difficult)
• Moderately Broken (Trivial work-around)
• Nuisance (Functions but untrapped errors can slip through)

To Reproduce

1. Use scripts to create image
2. Launch image as VM
3. Attempt to execute systemd-enabled tasks (busctl, timedatectl, etc.)

Expected behavior

All commands that interact with systemd should function properly

Deviance Description

Resulting-images do not have /var/run as a symlink to /run

Describe the bug
AMIgen assumes that resultant instances are relatively short-lived, such that the kernel will either never be updated (absent a rebuild) or will only live long enough to be updated a couple times. Thus, no change is made to the value of installonly_limit in the /etc/yum.conf file's default value of 5. If a system-owner persists a system long enough to install more than 2-4 kernel updates, subsequent updates will fail due to lack of free space on the /boot partition.

Severity

• Completely Broken (No work-around evident)
• Severely Broken (Work-around possible but difficult)
• Moderately Broken (Trivial work-around)
• Nuisance (Functions but untrapped errors can slip through)

To Reproduce
Steps to reproduce the behavior:

1. Build a instance with a kernel that's -5 of current
2. Iteratively yum udate kernel-<VERSION> (where <VERSION> is some version between what's installed and whatever latest available is)
3. Eventually, one of the iteration will fail due to an out-of-space error condition
4. Run rpm -q kernel and find that there are 3+ kernels already installed and /boot has less free space than the size of the to-be-installed kernel RPM

Expected behavior
The kernel update RPMs should never fail to install due to space-limitations.

Fix Suggestions

Update AMIgen's ChrootCfg.sh (or later) script to modify the installonly_limit value in the /etc/yum.conf file to 2

• For environments where privilege-delegation separates Cfn privileges from IAM privileges, add a template for creating the validation instance's instance-role/policy
• For environments where where Cfn users/roles also have IAM privileges, add a composite-template for creating the validation instance's instance-role/policy along with performing the EC2-based validation.

Ensure that resultant AMIs have FIPS mode available and enabled.

Note: will be necessary to ensure that boot= kernel parameter must be defined to ensure that kernel will boot.

Problem Description:

Currently, AMIs are all created with EXT4 filesystems. Some users want XFS "since that's what the vendor default is"

Expected Behavior:

Selectable fstype for root-filesystems ...with XFS as an option

Actual Behavior:

EXT4 is hard-coded

Fix recommendation:

Add new flag to appropriate script(s) to select fstype for root filesystems. Default to prior behavior so as not to break dependant projects.

Looks like resultant AMI has some packages from the plus repo rather than from base/updates (or even the still-acceptable extras):

The post-launch active repo list is also too broad:

Fixed primary functionality associated to #7, need to extend the package-selection functionality to make the fix for #7 complete.

Enablement of FIPS mode can break a number of things - which can be quite deleterious if the breakage impacts launch-time tasks. Need to make generation of FIPS-enabled images selectable. Due to hardening requirements, FIPS should default to enabled but be overridable.

Given that more than one build-module handles FIPS configuration, might want to make the override happen in the form of an environment variable (FIPSDISABLE?)

Describe the bug
When running the ChrootBuild.sh script the /mnt/ec2-user/bin folder/contents and any associated libraries are not available. This is causing failure for anything trying to execute from the /bin folder in the chroot'd environment. For example, /bin/yum-config-manage fails to run.

Severity

• Completely Broken (No work-around evident)
• Severely Broken (Work-around possible but difficult)
• Moderately Broken (Trivial work-around)
• Nuisance (Functions but untrapped errors can slip through)

To Reproduce
Steps to reproduce the behavior:

1. Create a new instance based on the RHEL 7.6 AMI (in GovCloud this is Red Hat Enterprise Linux 7.6 (HVM), SSD Volume Type - ami-5a740e3b)
2. Follow the instructions in "Docs/README_PublicRun.md"
3. See the errors when you run ./ChrootBuild.sh
   A. "warning: %post(rh-amazon-rhui-client-3.0.22-1.el7.noarch) scriptlet failed, exit status 127"
   B. "There are no enabled repos."
   C. "chroot: failed to run command ‘/bin/yum-config-manager’: No such file or directory"
   D. List of all the packages not installed
   "package Red_Hat_Enterprise_Linux-Release_Notes-7-en-US is not installed
   "package audit is not installed..." and more.

Expected behavior
Significantly less errors. Preferably none. All packages to be installed. /mnt/ec2-user/bin to exist as well as required libraries for yum-config-manager to run successfully.

Deviance Description
Errors cause failure to create new AMI.

Additional context

Fix Suggestions

Lines 82-83 have a silent failure if named EPEL repository is already present in CHROOT. Probably want to implement a pre-flight so that anything invoking script with stop-on-error doesn't abort.

Some install-defaults for sshd service have UsePrivilegeSeparation enabled. Either:

• Ensure that build scripts create/configure the appropriate config items
  • Ensure that sshd user exists
  • Ensure that sshd group exists
  • Ensure that ssh_keys group exists
• Ensure that build scripts disable this functionality
  • Set UsePrivilegeSeparation to no

Describe the bug
Systemd-managed services (identified in spel Issue plus3it/spel/issues/317) are not enabled/started in instances created from this project's automation.

Severity

• Completely Broken (No work-around evident)
• Severely Broken (Work-around possible but difficult)
• Moderately Broken (Trivial work-around)
• Nuisance (Functions but untrapped errors can slip through)

To Reproduce
Steps to reproduce the behavior:

1. Generate AMI from AMIgen7-based automation (see spel project.
2. Launch instance from newly-produced AMI
3. Check enable/run status or named services - find them to be preset-disabled

Expected behavior

Each of the following services should be preset-enabled and running:

• ec2sys-autotune (autotune.service)
• amazon-ssm-agent (amazon-ssm-agent.service)
• ec2-hibinit-agent (hibinit-agent.service)
• ec2-instance-connect (ec2-instance-connect.service)

Fix Suggestions

Update the AWScliSetup.sh script to perform a chroot()ed enablement of the impacted services.

Is your feature request related to a problem? Please describe.
By default, cloud-init creates the provisioning-user with the first available uid value higher than the SYS_UID_MAX value declared in /etc/login.defs. Further, this account (rightly) gets created with no password-aging attributes. Security scanners treat this as an interactive user and will alert on any tests around password-aging.

Describe the solution you'd like
Have default-user created with a uid value less than the SYS_UID_MAX value declared in /etc/login.defs.

Describe alternatives you've considered
Could force password-aging attributes onto account, but that will risk the "break glass" use-case for the account once expiry has been reached.

Could also use the uid parameter to hard-code a uidvalue less than theSYS_UID_MAXvalue declared in/etc/login.defs`, but don't know if that could create other problems, later, for applications using the resultant AMI.

Additional context
cloud-init provides the system: true attribute that results in creating a user with a uid value less than the SYS_UID_MAX value declared in /etc/login.defs. However, such users get created with no ${HOME} which means no ability to create a ${HOME}/.ssh/authorized_keys file. While cloud-init also provides the notional ability to override this behavior by explicitly declaring the no_create_home parameter and setting its value to false, trying to use this as an override-attempt seems to be ignored.

EC2 validator-template goes to CREATE_COMPLETE long before S3 contents have actually been updated. Add signaling so that impatient test-user(s) has better idea of when/whether the S3 artifact-write has actually completed.

/etc/systemd/journald.conf currently has Storage="persistent" value set. Need to remove quotation marks around persistent.

Problem Description:

Graceful shutdown via AWS GUI/CLI not currently possible

Expected Behavior:

Issue an instance-shutdown via the AWS software and have the OS undertake a graceful shutdown procedure.

Actual Behavior:

(Nitro HVM) Instance OSes have no "hook" to receive the graceful shutdown requests. Ultimately, the cloud-layer has to initiate a forcible shutdown.

(Detailed) Steps to reproduce:

Issue a shutdown request via AWS cloud-layer tools

(Optional) Fix recommendation:

Add the ACPID package to the AMI. Per the AWS FAQ:

Q. How are instance reboot and termination EC2 API requests implemented by the Nitro Hypervisor?

The Nitro Hypervisor signals the operating system running in the instance that it should shut down cleanly by industry standard ACPI methods. For Linux instances, this requires that acpid be installed and functioning correctly. If acpid is not functioning in the instance, termination events will be delayed by multiple minutes and will then execute as a hard reset or power off.

Describe the bug
Some groupnames passable via the -g flag are actually a group of groups + individual RPMs. Failure to expand the sub-groups can cause build failures.

Severity

May be worked around using the -m flag and a sufficiently fully-specified manifest if needing other than the core package-group.

Expected behavior
Specifying any given yum group via -g should act the same as doing yum install <GROUPNAME>.

Fix Suggestions
If optional-group is not core, look for and iterate over/expand subgroups as part of the current package-enumeration routine.

Problem Description:

On instances launched RHEL 7 AMIs /var/run is not a link to /run. This breaks DBUS upgrades

Expected Behavior:

/var/run is a link to /run in all created AMIs

Actual Behavior:

/var/run is not a link to /run in all created AMIs

(Detailed) Steps to reproduce:

Build new RHEL 7 AMIs from current scripts

(Optional) Fix recommendation:

Add an "end-of-build" type of task to ensure that /var/run is a link to /run.

Looks like EL7's disk geometry related dracut modules don't grok LVM. Need to locate and patch requisite scripts to be more like the AMIgen6-patched versions.

Is your feature request related to a problem? Please describe.

Make the GRUB_TIMEOUT value settable

Describe the solution you'd like

Currently, the GRUB_TIMEOUT value is set to ONE second. As AMIgen gains use on more platforms – and as CSPs begin to provide access to virtual consoles – the default (1s) value will be too short for users seeking to be able to access the GRUB menu for diagnostic and other reasons.

This is fixable by changing line-37 of the GrubSetup.sh script to make use of a parameter-set value.

Describe alternatives you've considered

Continue with static value but set to a much higher value: however, this will too-broadly slow boot processes and change expected behaviors

Additional context

• Virtual-console support in Azure
• Virtual-console support in AWS

Beta AMIs don't have swapVol configured into the swap system

Is your feature request related to a problem? Please describe.
Some users of AMIs created from this automation find the static partitioning-scheme problematic on long-lived systems or systems with specialty monitoring and/or tooling needs

Describe the solution you'd like
Add a flag that allows the passing of a partitioning-map to DiskSetup.sh (replacing or overriding the static values in its CarvLVM function. Probably want to pass the partitioning scheme as JSON-type data or possibly delimited triplets (MountLocation, VolumeName, VolumeSize).

Describe alternatives you've considered
Continue leaving storage relayout as a launch-time task for AMI owners...

Additional context
Other tooling that inherits this will need to be updated to take advantage and flag/option will need to be wholly optional so as not to break reliant tooling.

Is your feature request related to a problem? Please describe.

After some security profiles are applied to a spel AMI, the provisioning user may no longer be able to execute sudo.

Describe the solution you'd like

Update CleanChroot.sh's default-user setup section to add:

selinux_user: unconfined_u

Further, also ensure that:

sudoers:    files

Is present in the /etc/nsswitch.conf file.

The above will, effectively, implement the fixes described in the Red Hat Solution #3916301 prior to any problematic hardening.

Describe the bug
When using build-scripts to facilitate a chroot()ed build for VMware, current logic can result in / and/or /boot being overlay-mounted on top of the desired / and /boot in the chroot() target

Severity
Manual edit of MkChrootTree.sh required to surmount

To Reproduce
Steps to reproduce the behavior:

1. Launch work-host under VMware
2. Start hand-executing AMIgen7 scripts
3. Run MkChrootTree.sh
4. Note that the mounts in the chroot() target are "missing" (not present in df output but present in /proc/mounts)

Expected behavior

Neither the / nor the /boot from the boot-OS should be mounted into the chroot()

Fix Suggestions
Add /\/ /d and /\/boot /d to partition-setup sed block for the MkChrootTree.sh script

Problem Description:

When generating new "recovery" AMIs, the GrubSetup.sh currently sets the root= token to the chroot-built device's real devnode rather than properly referencing its LABEL value

Expected Behavior:

The root= tokens in the chroot()ed /boot/grub2/grub.cfg should be set to root=LABEL=<LABEL>

Actual Behavior:

The root= tokens in the chroot()ed /boot/grub2/grub.cfg are set to root=/chroot/devnode

(Detailed) Steps to reproduce:

Run GrubSetup.sh to create non-LVM'ed AMI

Fix as necessary...

Is your feature request related to a problem? Please describe.

With the merging of fixes for #120 via PR #121, sudo behavior for the default-user becomes sub-optimal: users currently have to specifically request an SELinux role and type to get a useful permissions-set when elevating privileges. Casual sudo users won't understand the need for this and will complain about permission-errors when elevating privileges.

Describe the solution you'd like

Ensure that the default-user's sudoers mapping contains an appropriate privilege-elevation rights-mapping. Something like:

TYPE=sysadm_t ROLE=sysadm_r 

Or (less optimally):

TYPE=unconfined_t ROLE=unconfined_r 

Added to the cloud.cfg stanza for the default-user is recommended.

Describe alternatives you've considered

Additional context

Problem Description:

Yum exits zero if any of the specified packages install successfully. This leads to false successes, when a specified package is unavailable.

If all of the specified packages are unavailable, then yum exits non-zero.

Expected Behavior:

Expected the build to fail if any specified packages did not get installed into the AMI.

Actual Behavior:

Build "succeeds" even when specified packages do not get installed.

(Detailed) Steps to reproduce:

Specify an available package and an unavailable package in the EXTRARPMS options.

(Optional) Fix options:

1. Loop over all packages, to install them individually. This will cause yum to exit non-zero if any of the packages are unavailable.

2. Test that all the specified packages actually are all installed after each yum install.

3. Parse the yum history.

4. Tee the yum output and parse that.

Also, would like to apply the fix to both AMIgen7 and AMIgen6...

The grub2 RPM-manifest lists /etc/sysconfig/grub as a component-file. However, this file has proven to be absent in the 7.2.1511 RPMs. Previous workaround was to copy build-host's file to chroot. However, this has unreliable results. Need to change method.

Is your feature request related to a problem? Please describe.
Two scenarios:

1. Some yum repository-maintainers fail to reliably-ensure that the groups metadata currently relied upon by AMIgen exists
2. Some AMIgen users might like to implement build-profiles beyond what's found in @Core

Describe the solution you'd like
Add a switch-argument to the ChrootBuild.sh script to allow the passing of an RPM manifest. Ideally, the mainifest can be readable from a variety of locations (e.g., local file, HTTP URL or cloud-storage URL(s))

Hi, just found this repo and giving it a try. I was prompted to run GetAmznLx.sh, but it wasn't present. I pulled this from your AMIgen6 repo and seemed to work fine though. Thanks

Need to validate that the chroot-based generation-process isn't causing /var/lock to fail to be a symlink of ../run/lock:

• Identify if an AMIgen7 or spel issue
• Remediate if an AMIgen7 issue
• Investigate spel if not an AMIgen7 issue

@lorengordon one to add to (link into) the task-board

After applying recommended hardening, the users created by cloud-init will no longer be able to use sudo if they've not been granted an appropriate SEL login user context:

$ sudo -i
sudo: PERM_SUDOERS: setresuid(-1, 1, -1): Operation not permitted

Ensure that the default provisioning user be given an appropriate SEL login user context to prevent the above.

Describe the bug

Severity

• Completely Broken (No work-around evident)
• Severely Broken (Work-around possible but difficult)
• Moderately Broken (Trivial work-around)
• Nuisance (Functions but untrapped errors can slip through)

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Deviance Description

Screenshots

Additional context

Fix Suggestions

Appears that when instances launch on 10Gbps-enabled nodes, networking isn't getting properly set up. Fix for EL6 doesn't seem to translate to EL7