a keyboard cover, in case of nearby cats that may scamper around and corrupt sensitive boot commands
Note: Windows hosts are affected by a packer bug where attempts to kill a packer process by sending a Control+C signal, result in a half-dead packer that often awakes during subsequent builds, corrupting them. Task Manager is your friend.
General advice
Recommend 16 GB or more of guest disk space and 4 GB or more of guest RAM, to account for the girth of modern software development environments.
Providers/Hypervisors
Some packer templates support multiple hypervisor options. By default, Packer will attempt to target all configured hypervisors. Or, if you are interested in merely a subset of the hypervisors, ensure that the -only <comma,separated,providers> flag is specified to the packer build... command.
Regardless of provider, be sure to change directory to the guest OS desired (e.g. debian/), as Packer builds are relative to the current working directory, rather than relative to the packer JSON directory.
Note that many packer hypervisors deliver build keystrokes via host-timed keyup, keydown pairs. This means that if your host is overloaded (CPU, RAM, HD), then packer will deliver spurious keyboard activity to the VM during a build. In particular, this often looks like repeatttttttttttted keystrokes, as the keydown event is sent too slowly to faithfully execute the boot_command. So whichever machine is packing VMs should have sufficient spare capacity to build: few running applications, and plenty of CPU speed, RAM, and available HD space.
VirtualBox
One cleanup tip: As with all Vagrant hypervisors, VirtualBox sometimes leaves virtual machine data around when vagrant destroy [-f], or a signal interrupted vagrant up should have deleted these artifacts. When this happens, the user can launch the VirtualBox application and delete these files manually. VirtualBox will likely complain with multiple error prompts, but these can largely be ignored.
VMware
VMware boxes can be packed without a Vagrant plugin, but running the boxes to test them requires a paid license, even for users who have already paid for VMware. Go figure.
qemu/libvirt
qemu AKA libvirt boxes are fragile, requiring more care than VirtualBox or VMware providers. libvirt support for macOS hosts is nascent, so packing and running libvirt boxes is best performed from Linux hosts such as Debian, Ubuntu, or RHEL derivatives. qemu is slower than other hypervisors, especially when KVM is unavailable. This dramatically increases the time required for both packing and running qemu/libvirt boxes. Read: vagrant up --provider libvirt && vagrant ssh -c 'uname -a' for mcandre/debian takes several minutes, and packer build -only qemu debian.json takes over 3 hours. Speed demon!
The process for properly installing the vagrant-libvirt plugin is rather involved, requiring multiple separate packages to be setup. See the vagrant-libvirt README for more detail.
Once vagrant-libvirt is fully installed with native extensions, the host should be configured to avoid hibernation for at least 4 hours, in order to ensure that the packer build completes without network interruption.
In addition, libvirt requires additional manual configuration in order to correctly integrate with Vagrant via vagrant-libvirt:
The libvirt-bin and libvirt-guests services should be running. Consult your host operating system's init system.
The user running Vagrant must have sufficient permission to access the libvirt socket, such as adding the user to the libvirtd UNIX group.
Guest operating systems must name their network adapters according to the legacy Linux scheme in order to integrate with vagrant-libvirt and obtain an IP address. See fix-libvirt-networking.debian.sh in debian/ for an example GRUB configuration to enforce this policy in the guest OS at packing time.
libvirt may come preconfigured with extraneous networks and volumes that conflict with vagrant-libvirt. See virsh net-list and virsh vol-list --pool default to examine these resources.
Finally, some libvirt guests may do a poor job persisting file changes across vagrant package boundaries. To work around this limitation, ensure that the file system is explicitly synchronized at the end of provisioning scripts, e.g. sync in GNU/Linux.
TESTING
These boxes are designed as minimal bases for constructing build bot virtual machines, so that mcandre/tonixxx can use the boxes to conveniently cross-compile applications for many different kernels. The boxes are expected to feature:
working package manager, for installation of devopment tools like gcc, curl, lua, etc.
bidirectional-capable host->guest and guest->host synced folders, for copying source code to the box and copying artifacts back to the host.vagrant-rsync-back
The best way to ensure that the boxes are suitable for this development workflow is to attempt to install some package, and to check that files can be copied from the host and guest and back again. This workflow is automated in a testing script. Example:
vmware-iso: Password:rm: /Users/vagrant/payload: is a directory2017/12/16 10:00:28 ui: vmware-iso: Password:rm: /Users/vagrant/payload: is a directory2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.safaridavclient.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.safaridavclient.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariBookmarksSyncAgent.plist: Could not find specified service vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariBookmarksSyncAgent.plist: Could not find specified service vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariHistoryServiceAgent.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariHistoryServiceAgent.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.Safari.SafeBrowsing.Service.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.Safari.SafeBrowsing.Service.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.notificationcenterui.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.notificationcenterui.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.IMLoggingAgent.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.IMLoggingAgent.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.CalendarAgent.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.CalendarAgent.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.AddressBook.AssistantService.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.AddressBook.AssistantService.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariLaunchAgent.plist: Could not find specified service vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariLaunchAgent.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.AddressBook.ContactsAccountsService.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.AddressBook.ContactsAccountsService.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.icloud.fmfd.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.icloud.fmfd.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.cloudpaird.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.cloudpaird.plist: Could not find specified service vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariCloudHistoryPushAgent.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariCloudHistoryPushAgent.plist: Could not find specified service vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariPlugInUpdateNotifier.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariPlugInUpdateNotifier.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.imagent.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.imagent.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.AddressBook.abd.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.AddressBook.abd.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.Siri.agent.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.Siri.agent.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.icloud.findmydeviced.findmydevice-user-agent.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.icloud.findmydeviced.findmydevice-user-agent.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.cloudphotosd.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.cloudphotosd.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.touristd.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.touristd.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.iCloudUserNotifications.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.iCloudUserNotifications.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.cloudd.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.cloudd.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariNotificationAgent.plist: Could not find specified service vmware-iso: /System/Library/CoreServices/SafariSupport.bundle/Contents/Resources/com.apple.SafariNotificationAgent.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchDaemons/com.apple.findmymacmessenger.plist: Could not find specified service vmware-iso: /System/Library/LaunchDaemons/com.apple.findmymacmessenger.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.AddressBook.SourceSync.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.AddressBook.SourceSync.plist: Could not find specified service2017/12/16 10:00:28 ui: vmware-iso: /System/Library/LaunchAgents/com.apple.siriknowledged.plist: Could not find specified service vmware-iso: /System/Library/LaunchAgents/com.apple.siriknowledged.plist: Could not find specified service vmware-iso: Started erase on disk1s1 Macintosh HD2017/12/16 10:01:01 ui: vmware-iso: Started erase on disk1s1 Macintosh HD2017/12/16 10:01:01 ui: vmware-iso: Creating a temporary file vmware-iso: Creating a temporary file2017/12/16 10:01:02 ui: vmware-iso: Securely erasing a file vmware-iso: Securely erasing a file vmware-iso: Creating a secondary temporary file2017/12/16 10:02:16 ui: vmware-iso: Creating a secondary temporary file vmware-iso: Mounting disk2017/12/16 10:02:30 ui: vmware-iso: Mounting disk2017/12/16 10:02:30 ui: vmware-iso: Finished erase on disk1s1 Macintosh HD vmware-iso: Finished erase on disk1s1 Macintosh HD vmware-iso: Partition / is not shrinkable2017/12/16 10:02:30 ui: vmware-iso: Partition / is not shrinkable
Credit Tim Sutton's repo for prior art.
Obtain an ISO and packer config that successfully boots into the graphical installer
Automate installation via text, by supplying PKG files into the ISO that preempt the graphical installer.
Fix or ignore launchd messages on boot. (Ignoring.)
Resolve blank gray screen boot. (By using the official createinstallmedia, configuring VirtualBox virtual hardware for a supported Apple device, and by waiting 30 seconds for SetupAssistant to load.)
Resolve UEFI boot menu. (By ensuring at least 5500MB for createinstallmedia disk.)
Working around boot error with macOS 10.13 High Sierra by targeting macOS 10.12 Sierra instead.
Test entire boot_command
Test all provisioning scripts
Purchase VMware plugin for Vagrant.
Test Vagrant file upload, shell script provisioning.
Test shared folders.
Try navigating macOS installation via keyboard with packer boot_command. (Seems to work fairly well!)
Setup destination disk so that Disk Utility can see it. (Once in Disk Utility app, press Command+2 to view all devices.
Figure out how to select the View All Devices menu option in Disk Utility, and how to quit Disk Utility. (Use Control+F2, arrow keys, and spacebar to navigate application menus.)
Work around packer assumption of IBM scancodes, whereas macOS VM's use Apple keyboard scancodes. (Using VMware builder to work around scancode mismatch by sending key codes via VNC.)
Figure out how to select the install volume from SetupAssistant purely by keyboard. (Use Control+F2 keyboard navigation to access Terminal.app, enter launchctl kickstart system/com.apple.VoiceOver to enable VoiceOver, use Control+F2 again to quit Terminal.app, use arrow keys to launch macOS installer, spacebar and tab to navigate most of the UI, and then use the VoiceOver Control+Option+(arrow key) to select install target volume.
Fix boot loop after running macOS installer.
Fix boot loop in Windows 10. (Patch VMware with unlocker, specify smc.version = "0" in VMX, and specify cpuid.1.eax = "0000:0000:0000:0001:0000:0110:1010:0101" in VMX.)
Rerun packer build and ensure that the VoiceOver Option/Alt hotkeys are correctly being processed.
Fix Fusion boot loop halfway through installation. (vmware-iso builder was defaulting to SCSI interface for virtual hard drive. vmware-iso does not offer a standard way to specify SATA, but was able to disable the SCSI drive and enable a SATA connection to disk.vmdk via vmx_data.
Fix country selection. (Patched packer to recognize Super/Windows/Command keys.)
Fix "Express Setup" section of boot_command. (Use VoiceOver via packer patched with Super key support.)
Fix VoiceOver not fully progressing initial install (Appears to be a timing issue, so adding more <wait>s to delay before submitting keys.)
Fix shutdown command (echo password into the sudo command, authorizing a shutdown).
Fix shutdown command some more (specify -S flag to sudo so that the password is read from STDIN rather than a live user terminal connection).
Fix ssh service not launching at boot. (Reboot into Recovery Mode. Reconfigure the Disabled key in /Volumes/Macintosh HD/System/Library/LaunchDaemons/ssh.plist to false.)
Fix VoiceOver triggering the VoiceOver menu on VO-<arrow key> instead of navigating UI elements.
Fix Rescue Mode trigger. (Repeatedly send Command+R key down events with no key up for several minutes.)
Fix <rOn> missing during boot_command. (Evidently the key is being sent; the packer patch simply forgot to log this. The logging has been corrected.)
Fix Recovery Mode not rebooting into normal mode. (Reexamined the packer log, scrolled up. Realized that macOS was rebooting into normal mode and launching SSH. Apparently the provisioning scripts completed quickly enough that the output was overshadowed by the rest of the packer log completing the build.)
Fix sshd not launching (It was, though public key authentication was misconfigured.)
Fix ssh public key authentication. (Running sed -i '' 's/#PubkeyAuthentication yes/PubkeyAuthentication yes/' /etc/ssh/sshd_config during packing.)
Fix ssh timeout when connecting to the built box. (Working around public key misconfiguration by using password authentication method temporarily.)
Fix Vagrantfile customizations apparently not being applied to box on vagrant up. (They are.)
Fix sudo networksetup -detectnewhardware failing during darwin guest plugin setup. (Configured /etc/sudoers to not require a password for vagrant user sudo.)
Fix Terminal treated as damaged and unable to be opened. (Reran packer build.)
Theories for fixing Recovery Mode not rebooting into normal mode:
Use menubar to reboot.
Theories for fixing missing <rOn> during boot_command:
Temporarily insert <leftSuperOn><rOn><wait5><rOff><leftSuperOff> at top of boot_command, check output of PACKER_LOG=1 packer build... for <rOn>, <rOff> presence.
Theories for fixing Rescue Mode trigger:
Test <rOn> again quickly with Ubuntu. Specificaly, test <leftSuperOn><rOn>.
Create a snapshot just before submitting sudo shutdown -r now, and time how long it takes to get to the pale VMware boot screen. Take several times to evaluate consistency (if any).
Try syncing to make reboot times more consistent.
Try preemptively killing certain processes to make reboot times more consistent.
Once shutdown is initiated, send repeated Command+R key down signals with no key up, for five minutes.
Theories for fixing VoiceOver menu:
Try different keyboard profiles.
Revert to old launchctl kickstart system/com.apple.VoiceOver method of enabling VoiceOver.
Revert packer A-Z hold patch.
Theories for fixing ssh service not launching at boot.
Patch packer to support holding down Super and A-Z keys, in order to trigger a boot into Recovery Mode during boot_command, so that /Volumes/Macintosh HD/System/Library/LaunchDaemons/ssh.plist can be edited without SIP interfering.
Practice timing the Command+R hotkey with plain VMware outside of packer.
Fix Vagrant SSH public key missing (wget unavailable from fresh macOS install).
Theories for obtaining Vagrant SSH public key:
Manually echo the key contents
Install wget via Homebrew
Try using fetch (unavailable from standard macOS install)
Try using curl
Try using Ruby
Try using Python
Try using Perl
Theories for fixing boot loop in Fusion:
Compare packer effective VMX to known working manual Fusion VMX.
Theories for fixing boot loop in VMware Workstation Player in Windows on Apple hardware:
Compare VMX to the VMX on Gist.
Try using USB 2.0 virtual interface.
Unfortunately, the macOS 10.12 Sierra installation's timing is so erratic, either requiring ~30 minutes to complete, or else several hours, that there is little chance of boot_command succeeding.
Manually install macOS 10.12 Sierra.
Export OVF.
Use packer to complete Vagrant box.
Theories for fixing boot issues upon macOS installation:
Rerun packer build.
Rerun ISO generation.
Have diskutil repair /Volumes/Macintosh HD before installing macOS.
Use journaled HFS format instead of APFS.
Try configuring "smc.present": "FALSE" in VMX. (Causes macOS to stall during boot.)
Configure "guestCPUID.0": "00000001 000106e5 00100800 0098e3fd bfebfbff" in VMX.
Configure "cpuid.brandstring": "ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" in VMX.
Find the BaseSystem, InstallESD, and AppleDiagnostics DMG's.
Use installer to apply these DMG's to the target volume '/Volumes/Macintosh HD'
Theories for enabling VoiceOver via Terminal:
/System/Library/CoreServices/VoiceOver.app/Contents/MacOS/VoiceOver (Currently reports an error Someone has already checked in my running label. Exiting.
Hack the install ISO to include osascript. Then use osascript to activate VoiceOver.
Theories for fixing VoiceOver label:
Find and delete a *.pid or similar file.
Keyboard and mouse profile name candidates:
"windowsProfile"
"windows8Profile"
"windows10Profile"
"52559783-3576-1836-f8f7-d7d6c26c"
"528012fe-5e19-5990-f152-fea4aa11"
"520d3bb1-7ff1-2cc1-2d13-b4aeb9ed"
"5277983a-933b-b280-03db-d4137af3"
"52d1d410-5846-7f89-c532-2dc8a012"
"529767a2-56c6-806d-2b75-54536ebd"
"52bfd239-ee02-d3f9-bd90-9423c03c"
Theories for sending Command modifier:
Install macOS graphically, export as OVF. Use packer to continue provisioning OVF into a Vagrant box.
Modify install ISO so that VoiceOver is enabled by default.
Send raw Command+F5 scancode alongside packer in a carefully timed shell script.
Configure a PC keyboard and mouse profile with Left Alt (host) mapped to Command (guest).
Use the global menu bar to launch Terminal.app. At this point, hopefully VoiceOver can be enabled, or the installer can be launched from command line!
Write a polyglot boot_command whose IBM scancodes operate as Apple scancodes. For example, instead of using <down>, which packer maps to IBM scancode [0x50 0xd0], use b, which maps to IBM scancode [0x30 0xb0], which macOS would interpret as [0x30] => Down arrow, hopefully ignoring the 0xb0 as junk input and allowing for further keyboard keys. Not sure if the rest of the boot_command can be written in this way, and it's a pain to debug. If by some magic all the necessary keys for packing do have reasonably nice IBM equivalent scancodes, then a scancode mapping file could be written, and then a preprocessing step before packer build would involve applying github.com/hashicorp/packer/blob/master/builder/virtualbox/common's scancodes() over each command in the JSON boot_command array, emitting a polyglot, really screwed up looking packer.json to be generated that emits the desired scancodes to the macOS guest.
Find some way of setting up VirtualBox so that IBM scancodes are intercepted and translated to appropriate Apple scancodes.
Abandon boot_command and use VBoxManage controlvm "$VM" keyboardputscancode <hexpair> [<hexpair> [<hexpair> ...]]] to send raw scancodes to the macOS guest VM. Will need to find a way to space these out, simulating the behavior of <wait> in a normal packer boot_command string.
Replace the boot_command section with a script that runs parallel to packer -build, that sends appropriate keycodes, with sleep timers, to the VM.
Theories for intercepting and translating scancodes sent to VBoxManage:
File a ticket to configure scancode mappings, so that VBoxManage modifyvm <VM> --fromscancodes ibm --toscancodes apple or similar flags configure VirtualBox to automatically perform scancode translation between different scancode sets.
Write a bash script virtualbox-remap-scancodes-from-ibm-to-apple.sh with a function VBoxManage that intercepts IBM scancodes in VBoxManage controlvm "$VM" keyboardputscancode commands, maps them to Apple scancodes, and calls the real VBoxManage with the appropriate arguments.
Theories for spacing out VBoxManage controlvm "$VM" keyboardputscancode's:
Find some sort of VBoxManage sleep sub-command
Write VM's whose sole purpose is to sleep 1 second, 5 seconds, and 10 seconds, like boot_command's <wait>, <wait5>, <wait10>, and use VBoxManage commands to spin these up, wait for them to halt, and again.
Create an OVF with manual VBoxManage controlvm "$VM" keyboardputscancode and an export, and then use packer to complete the Vagrant box via the virtualbox-ovf builder.
Receives a Vagrant Cloud access token as an environment variable.
Receives a namespaced public box name (e.g. mcandre/minix) as a CLI argument.
Receives a box filename (e.g. minix-virtualbox.box) as a CLI argument.
Obtains a second token for the upload.
Uploads the box.
Confirms the upload using that second token.
Or, use jq to inject a Vagrant Cloud post-processor https://www.packer.io/docs/post-processors/vagrant-cloud.html for some packer builds, but not by default, so that users without permission to upload these boxes can still reliably run packer build 100% sucessfully.
Try using FreeBSD in VirtualBox without the virtualbox-ose-additions package. We're running headless, so any way to reduce clutter like X11, LLVM, Perl, and Python would be solid for saving disk space.
Have export DEBIAN_FRONTEND=noninteractive defined prior to all apt-get commands, including install, remove, and purge, in each individual shell script that runs any apt-get commands.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent