elliotweiser / ansible-osx-command-line-tools Goto Github PK

Hiya, thank you for publishing Vagrant boxes for macOS! Would it be possible to offer HyperV provided boxes as well? Unfortunately, Docker in Windows relies on HyperV these days, and HyperV is unable to run concurrently with the VirtualBox hypervisor. In other words, developers working with Windows hosts must choose to boot with either HyperV enabled to run Docker, or else HyperV disabled to run VirtualBox and Vagrant. But if the macOS Vagrant boxes were integrated with the HyperV provider, then Windows users would be able to hack on Docker and Vagrant projects at the same time, without having to constantly reboot. What do you think?

Hi guys,

I'm trying to run this role in the new macOS Mojave 10.14.5 but the task Check for Command Line Tools in Software Update list keeps failing. Also, its preceding tasks Is the C++ compiler useable? and Check the Command Line Tools package metadata are failing too.

Do you guys have any clue on this?

Thanks

Hi there! Great role - thanks for providing this for the Mac community!

That said, I created this over the weekend because I had no idea your role existed already:

https://github.com/lhazlewood/ansible-role-xcode-clt

(I think the reason being is that I've always know this stuff as 'Xcode Command Line Tools' not 'OS X command line tools, so when I did a search for 'xcode' on galaxy.ansible.com I don't think your role came up. I've seen more than a few dotfiles-with-ansible projects and most always said "install Xcode command line tools first", so I thought I just had to create a role).

It wasn't a wasted effort for me since I'm new to ansible and that was my first role and I needed the learning experience.

Anyway, one of the techniques I picked up from perusing other authors' roles is some use include and only include when on the target OS. For example:

---
- include: macos.yml  # actually does idempotent xcode clt install
  when: ansible_os_family == 'Darwin'

This makes a lot of sense to me (with my still new ansible knowledge) because I'm attempting to make a 'take this machine and install everything a dev needs' role that works simultaneously on both mac and linux since I use both.

This allows the task to skip innocuously when not on mac os, but the same role can be referenced in the same 'dev machine' role and/or playbook.

Does this make sense? Or might there be another ansible best practice for targeting the same role for both mac and linux? If not, what are your thoughts on adding this technique to this role? Thanks again!

Apparently the current check for testing whether command-line-tools have already been installed is not working.

- name: Check that command line tools are installed
  command: /usr/bin/xcode-select -print-path
  register: xcode_select_res
  ignore_errors: true
  changed_when: false

...
when: xcode_select_res.stdout|length == 0

Currently the check passes if XCode has been installed, but the CLTools haven't.

Proper way for defining whether the command-line-tools have actually been installed would be running the following command:
pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
which fails if the CLTools are not available.

I don't know the purpose of gcc check but it fails and brings up a pop-up in the target Mac system. Is it expected behavior?

In one of my playbook's builds on Travis CI, I started getting build errors due to timeouts:

TASK [elliotweiser.osx-command-line-tools : Install Command Line Tools] ********


No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
903Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

This only happens in the xcode11.4 Travis CI build environment (10.15), and doesn't seem to happen on the 10.14 environment.

Could this be a Travis CI misconfiguration, or could it really be taking that long?

Builds are hanging because Travis is unable to provide a confirmation to homebrew uninstall script:

Are you sure you want to uninstall Homebrew? This will remove your installed packages! [y/N] 

Affected OSX versions in Travis
osx_image: xcode10.3
osx_image: xcode10.1
osx_image: xcode9.2

Solution
Pipe the uninstall script execution with the command line tool yes.

yes | /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall.sh)"

Hi,

I was trying to run this role on a fresh install of macOS 10.15.3. This fails with the following log:

TASK [elliotweiser.osx-command-line-tools : Is the C++ compiler useable?] ******************************************************************************************************************************************
fatal: [xxx.foo.com]: FAILED! => {"changed": false, "cmd": ["g++", "--version"], "delta": "0:00:00.011904", "end": "2021-06-11 08:04:35.781916", "msg": "non-zero return code", "rc": 1, "start": "2021-06-11 08:04:35.770012", "stderr": "xcode-select: error: no developer tools were found at '/Applications/Xcode.app', and no install could be requested (perhaps no UI is present), please install manually from 'developer.apple.com'.", "stderr_lines": ["xcode-select: error: no developer tools were found at '/Applications/Xcode.app', and no install could be requested (perhaps no UI is present), please install manually from 'developer.apple.com'."], "stdout": "", "stdout_lines": []}
...ignoring

TASK [elliotweiser.osx-command-line-tools : Check the Command Line Tools package metadata] *************************************************************************************************************************
fatal: [xxx.foo.com]: FAILED! => {"changed": false, "cmd": ["pkgutil", "--pkg-info=com.apple.pkg.CLTools_Executables"], "delta": "0:00:00.029960", "end": "2021-06-11 08:04:37.625898", "msg": "non-zero return code", "rc": 1, "start": "2021-06-11 08:04:37.595938", "stderr": "No receipt for 'com.apple.pkg.CLTools_Executables' found at '/'.", "stderr_lines": ["No receipt for 'com.apple.pkg.CLTools_Executables' found at '/'."], "stdout": "", "stdout_lines": []}
...ignoring

TASK [elliotweiser.osx-command-line-tools : Prepare to install Command Line Tools] *********************************************************************************************************************************
changed: [xxx.foo.com] => {"changed": true, "dest": "/tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress", "gid": 0, "group": "wheel", "mode": "0644", "owner": "admin", "size": 0, "state": "file", "uid": 501}

TASK [elliotweiser.osx-command-line-tools : Check for Command Line Tools in Software Update list (MacOS < 10.15).] *************************************************************************************************
skipping: [xxx.foo.com] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [elliotweiser.osx-command-line-tools : Check for Command Line Tools in Software Update list (MacOS >= 10.15).] ************************************************************************************************
fatal: [xxx.foo.com]: FAILED! => {"changed": false, "cmd": "set -o pipefail; softwareupdate -l | grep -B 1 -E 'Command Line Tools' | awk -F'*' '/^\\*/ {print $2}' | sed 's/^ Label: //' | grep -iE '[0-9|.]' | sort | tail -n1\n", "delta": "0:00:06.431488", "end": "2021-06-11 08:04:48.428174", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2021-06-11 08:04:41.996686", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

The problem seems to be that the command-line tool is not available from the softwareupdate -l command:

softwareupdate --list
Software Update Tool

Finding available software
Software Update found the following new or updated software:
* Label: macOS Catalina 10.15.7 Update-
	Title: macOS Catalina 10.15.7 Update, Version:  , Size: 4667570K, Recommended: YES, Action: restart,

I have another VM on 10.15.4 where this works fine though.

Hi, looks like the command lines tool check is failing at the moment:

TASK [elliotweiser.osx-command-line-tools : Check for Command Line Tools in Software Update list (MacOS >= 10.15).] ************************************************************
fatal: [a.b.com.au]: FAILED! => {"changed": false, "cmd": "set -o pipefail; softwareupdate -l | grep -B 1 -E 'Command Line Tools' | awk -F'*' '/^\\*/ {print $2}' | sed 's/^ Label: //' | grep -iE '[0-9|.]' | sort | tail -n1\n", "delta": "0:00:06.299439", "end": "2021-04-27 20:44:09.861464", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2021-04-27 20:44:03.562025", "stderr": "No new software available.", "stderr_lines": ["No new software available."], "stdout": "", "stdout_lines": []}

This is on a mac mini, happening on a variety, all recently updated:

ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H1030

Command now returns an exit status of 1 even though the result is 'No new software available':

Running this role on a macOS 10.13 High Sierra installs old command line tools version.

Context

Running this role on a macOS 10.13.4 Vagrant box installs correctly the command line tools but not the latest version. The version installed is the one for Xcode 8.2.

Ansible version is 2.5.0

Process

Run this role with a simple playbook on macOS 10.13:

---

- hosts: all
  roles:
    - { role: elliotweiser.osx-command-line-tools }

Expected result

Latest version should be installed.

Current result

Latest version is not installed, CLTools for Xcode 8.2 are installed instead.

vag-macos13:~ vagrant$ pkgutil --pkgs | grep CLTools
com.apple.pkg.CLTools_Executables
com.apple.pkg.CLTools_SDK_OSX1012

vag-macos13:~ vagrant$ softwareupdate --history
Display Name                                       Version    Date
------------                                       -------    ----
Command Line Tools (macOS El Capitan version 10.11) for Xcode 8.2        09/05/2018 at 17:21:32

vag-macos13:~ vagrant$ softwareupdate -l
Software Update Tool

Finding available software
Software Update found the following new or updated software:
   * Command Line Tools (macOS High Sierra version 10.13) for Xcode-9.3
        Command Line Tools (macOS High Sierra version 10.13) for Xcode (9.3), 187312K [recommended]

On a freshly provisioned Mac OS X host, I see the following message when the "Check for Command Line Tools in Software Update list" task runs:

fatal: [mac10.12-staging]: FAILED! => {"changed": false, "cmd": "softwareupdate -l | grep -B 1 -E 'Command Line Tools' | awk -F'' '/^ +\/ {print $2}' | sed 's/^ *//' | head -n1", "delta": "0:00:32.352443", "end": "2017-09-28 12:02:36.534806", "failed": true, "failed_when_result": true, "rc": 0, "start": "2017-09-28 12:02:04.182363", "stderr": "No new software available.", "stderr_lines": ["No new software available."], "stdout": "", "stdout_lines": []}

When I re-run the playbook, the task succeeds the second time (very strange).

elliotweiser/ansible-osx-command-line-tools: 73acd94056c5b2305d59d3f8708a56740a959c61
ansible: 2.9.4
  config file = /Users/d/.macos-provisioning/ansible.cfg
  configured module search path = ['/Users/d/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/d/.macos-provisioning/.venv/python-venv-3.7.7/lib/python3.7/site-packages/ansible
  executable location = /Users/d/.macos-provisioning/.venv/python-venv-3.7.7/bin/ansible
  python version = 3.7.7 (default, Mar 10 2020, 15:43:33) [Clang 11.0.0 (clang-1100.0.33.17)]
MacOS: 10.15.6 (19G2021)
Command Line Tools: v11,5 (29.05.20, 06:02)

A playbook run errored out at task 'Install Command Line Tools' with message 'su_list' is undefined (tasks/main.yaml:89). After digging deeper into task 'Check for Command Line Tools in Software Update list (MacOS >= 10.15).' i found out that

set -o pipefail; softwareupdate -l | grep -B 1 -E 'Command Line Tools'

returned with an unexpected non zero exit code.

Solution one Approach one: Silence the breaking error in task 'Install Command Line Tools'

when: (su_list is defined) and (pkg_info.rc != 0 or compiler.rc != 0 or not clt.stat.exists)

But then i somehow realised that something is wrong with my current macos 10.15 installation. (System was created from scratch but the current user is imported from macos 10.14 Time Machine backup)
An inspection of /Library/Developer/CommandLineTools revealed that the SDK 10.14 was used as default instead of 10.15 which led to the described error.

Final solution: Clean up CommandLineTools installation

1. Uninstall all Command Line Tools
2. Reset default Command Line Tools
3. Reboot
4. Install Command Line Tools from scratch

sudo -rm /Library/Developer/CommandLineTools
sudo xcode-select -r
sudo shutdown -r now
# after reboot
git # in order to trigger ui installation of Command Line Tools

I noticed Travis now includes an xcode11.4 environment, running 10.15, so I'm working on incorporating it into my upstream builds which use this role.

Apple is deprecating Python 2.7 in macOS 12.3.

https://developer.apple.com/documentation/macos-release-notes/macos-12_3-release-notes

I believe this means that no version of Python will be included with stock macOS going forward. Python 3 can be installed via Xcode or Homebrew (which of course itself depends on Xcode's CLI tools), but this creates a circular dependency in this Ansible role. Since it requires Python to be available and Python now requires CLI tools.

Afaik, the only way to address this is to refactor the entire role using the raw and/or script modules and avoid relying on any data from gather_facts/setup.

Do you have any desire to undertake this? I'm afraid the role will have no feasible use beyond macOS 12.3 otherwise (please correct me if I'm missing something here).

I'm a Github/Ansible newb so apologies if I'm doing something wrong here.

When I run your CLT install playbook agains macOS 12 it forces a user dialogue to appear on the screen and the playbook fails until the user actions it.

I altered a bash script I found elsewhere to work with macOS 12 but I have no idea how to amend your code to use it. Here's the code I altered that works fine over SSH.

#!/bin/bash
# Only run if the tools are not installed yet
xcode-select -p &> /dev/null
if [ $? -ne 0 ]; then
  touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress;
  PROD=$(softwareupdate -l |
    grep "\*.*Command Line" |
    tail -n 1 | sed 's/^[^C]* //')
  softwareupdate -i "$PROD" --verbose;
else
  echo "Xcode CLI tools already installed"
fi

It seems that the step Check for Command Line Tools in Software Update list occasionally hangs and therefore breaks the build.

Could be something to do with this? https://apple.stackexchange.com/questions/82551/mac-app-store-update-stuck-installing-calculating

Is there anyway I can work around this, perhaps by adding a retry after x time?

Linked from geerlingguy/ansible-role-homebrew#86

I have been you your role for a while now and it seems to run into issues at this task, it will timeout and fail.

1. if I ssh into the same unit and run softwareupdate -l
   I get e list of available updates
2. if I run softwareupdate -i -a to install the updates the task will then continue

all systems are 10.15.7 and unless a manually intervein the task will always

What fails:

TASK [elliotweiser.osx-command-line-tools : Check for Command Line Tools in Software Update list] ******************************************************************************************************************************************************************************
fatal: [vm-mac]: FAILED! => {"changed": false, "cmd": "set -o pipefail; softwareupdate -l | grep -B 1 -E 'Command Line Tools' | awk -F'*' '/^ +\\*/ {print $2}' | sed 's/^ *//' | grep -iE '[0-9|.]' | sort | tail -n1\n", "delta": "0:00:10.361220", "end": "2019-10-01 12:38:50.808701", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2019-10-01 12:38:40.447481", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

Why:
softwareupdate --list seems to have changed output for mac OS catalina

Others do also suffer from this change:
microsoft/macos-cookbook#202

(moved from geerlingguy/ansible-role-homebrew#126)

This error recectly occured:

elliotweiser.osx-command-line-tools was NOT installed successfully: - sorry, elliotweiser.osx-command-line-tools was not found on https://galaxy.ansible.com.

Using elliotweiser.osx_command_line_tools works fine. Was this renamed on purpose?

Other projects, like geerlingguy.mas fail as well, because they use this as a dependency.

Builds are failing because there is a conflict with six 1.4.1 that comes with Python 2.7 and a newer version required by some of the dependencies. Pip attempts to uninstall six 1.4.1 but it fails because it doesn't have permission to remove files and directories in the path where six is installed by default: /Library/Python/2.7/site-packages/

OSX versions affected: 10.13 and 10.14 (potentially others)
pip version: 20.0.2

Solution

Add a flag to ignore six when installing pip modules:
sudo -H pip install -U virtualenv --ignore-installed six