on Windows, Firefox installer insists on installing to C:\Program Files

This isn't an issue for Aurora & Nightly, but General Release and Beta get installed to the same path causing contention if you want to install both. Ideally, we'd like a way to force the Win installer to allow us to install to a specific path: i.e.: C:\Program Files\Mozilla Firefox Beta

Cygwin on windows server determines the OS as "cygwin" and, hence, tragically fails when it can't find a configs/cygwin.ini. However, we also know that Win10 cygwin determins the OS as "windows".
🙀

Currently we get annoying notices like this because of the way that I naively load the OS-specific config in download/install/profile/etc. It'd be nice to move this into init.py and load it once and just import the config in the modules that need it. It should also clean up some boilerplate.

$ ff -c aurora

----------------------------------------
LOADING configs/darwin.ini
----------------------------------------


----------------------------------------
LOADING configs/darwin.ini
----------------------------------------


----------------------------------------
LOADING configs/darwin.ini
----------------------------------------

Downloading aurora to _temp/browsers/FirefoxDevEdition.dmg
[localhost] local: which hdiutil
[localhost] local: /usr/bin/hdiutil attach _temp/browsers/FirefoxDevEdition.dmg -mountpoint _temp/browsers/_dmg_temp
Moving _temp/browsers/_dmg_temp/FirefoxDeveloperEdition.app to _temp/browsers/FirefoxDevEdition.app
[localhost] local: /usr/bin/hdiutil detach _temp/browsers/_dmg_temp
[localhost] local: _temp/browsers/FirefoxDevEdition.app/Contents/MacOS/firefox --version

Currently we use a cloud-services centric way to specify a path to prefs.
-a loop-server -t e2e-test would tell ff-tool to look for prefs in the following dir:
./loop-server/e2e-test/prefs.ini

However, we could make the tool more universal with something like:
-d loop-server/e2e-test
which would stipulate a relative path to your prefs file. there is nothing else in ff-tool specifically tied to either test-type or application.

This would also preclude users from having to clone services-test (or use exact same file structure) in order to load their own custom browser prefs

python performs install then launch synchronously. Launching browser leaves script in wait state so downloading next browser isn't possible til that activity is terminated.

We could perhaps spawn multiple threads?? @pdehaan

See: https://github.com/mozilla/mozdownload
We'd pass in param here:
https://github.com/rpappalax/ff-tool/blob/dev/fftool/firefox_download.py#L55

The version currently gets set via a config var, but we'd probably want to override that with a version arg here:
https://github.com/rpappalax/ff-tool/blob/dev/fftool/firefox_download.py#L55

Need to find out how we can auto-mount a DMG and silent install.

I assume doing uninstall for Darwin is as simple as deleting the folder with the *.app.

i like the simplicity of .md files except when they become too scrolly.
it might be nice to have the docs on the readthedocs so we can have a section to click to that talks about the special use-case for cloud services testing.
just a thought.

$ ff --version

----------------------------------------
LOADING configs/darwin.ini
----------------------------------------

usage: ff [-h] [-c {release,beta,aurora,nightly,ALL}] [-p PROFILE] [-e ENV]
          [-t TEST_TYPE] [-a APP] [--no-launch] [--no-profile]
          [--install-only] [--clean-profiles]
ff: error: unrecognized arguments: --version

Probably just return the __version__ string from ./fftool/init.py:7 or somethin'.

First pass. Not as lean as I'd hoped, but there were a few sharp edges once I tested locally on Firefox Release. I'll PR this tomorrow once I give it one more pass.

import os
import shutil
from fabric.api import local


cmd_hdiutil = None

output = local("which hdiutil", capture=True)
for line in output.splitlines():
    cmd_hdiutil = line


def replace_ext(filename, ext):
    """
    Takes a filename, and changes it's extension.
    """
    basename = os.path.splitext(filename)[0]
    args = {"basename": basename, "ext": ext}
    return "{basename}.{ext}".format(**args)


def attach(dmg_path, mountpoint):
    args = {
        "hdiutil": cmd_hdiutil,
        "dmg_path": dmg_path,
        "mountpoint": mountpoint
    }
    cmd = "{hdiutil} attach {dmg_path} -mountpoint {mountpoint}".format(**args)
    local(cmd, capture=True)


def detach(mountpoint):
    args = {
        "hdiutil": cmd_hdiutil,
        "mountpoint": mountpoint
    }
    cmd = "{hdiutil} detach {mountpoint}".format(**args)
    local(cmd, capture=True)


def move_app(src, dest):
    """
    Loop over all the files in the `src` file's directory looking for any *.app
    file and copy it into the `dest` directory.
    """
    tmp_dir = os.path.dirname(src)
    # Loop over each file in the `temp_dir` looking for any *.app files.
    for file in os.listdir(tmp_dir):
        if file.endswith(".app"):
            # Once we match our *.app file, delete any existing file in the
            # `dest` folder and copy the new *.app over.
            src_path = os.path.join(tmp_dir, file)

            if os.path.exists(dest):
                print("Deleting existing {0} file".format(dest))
                shutil.rmtree(dest)

            print("Moving {0} to {1}".format(src_path, dest))
            shutil.copytree(src_path, dest)


def extract_dmg(file):
    """
    Mount the *.dmg, copy the *.app file, and unmount the *.dmg file.
    """
    dmg_dirname = os.path.dirname(file)
    dmg_filename = os.path.basename(file)
    app_filename = replace_ext(dmg_filename, "app")
    tmp_dir = os.path.join(dmg_dirname, "_dmg_temp")
    app_src_path = os.path.join(tmp_dir, app_filename)
    app_dest_path = os.path.join(dmg_dirname, app_filename)

    attach(file, mountpoint=tmp_dir)
    move_app(app_src_path, app_dest_path)
    detach(mountpoint=tmp_dir)
    ver = get_firefox_version(app_dest_path)
    print("Installed {0}".format(ver))


def get_firefox_version(app):
    bin = os.path.join(app, "Contents", "MacOS", "firefox")
    if not os.path.exists(bin):
        print("{0} not found. Aborting.".format(bin))
        return

    cmd = "{0} --version".format(bin)
    output = local(cmd, capture=True)
    for line in output.splitlines():
        return line


extract_dmg("_temp/browsers/FirefoxRelease.dmg")
extract_dmg("_temp/browsers/FirefoxBeta.dmg")
extract_dmg("_temp/browsers/FirefoxDevEdition.dmg")
extract_dmg("_temp/browsers/FirefoxNightly.dmg")

logic should already be in place. We just need to add an arg for it to main.py.

something like:

-u

Saw mozinfo earlier and figured it's probably better to use that, instead of our homegrown regex based OS detection.

Will need a bit of Cygwin testing to make sure everything is still kosher (and it self-identifies as "windows").

Sample usage and output:

import mozinfo

print("bits: {}".format(mozinfo.bits)) # 64
print("os: {}".format(mozinfo.os)) # mac
print("os_version: {}".format(mozinfo.os_version)) # 10.11
print("processor: {}".format(mozinfo.processor)) # x86_64
print("version: {}".format(mozinfo.version)) # OS X 10.11.4

DOWNLOAD FIREFOX

Downloading nightly to _temp/browsers/FirefoxNightly.dmg
Traceback (most recent call last):
File "/Users/rpappalardo/git/tmp/venv/bin/ff", line 11, in
sys.exit(main())
File "/Users/rpappalardo/git/tmp/venv/lib/python2.7/site-packages/fftool/main.py", line 29, in main
download(options.channel)
File "/Users/rpappalardo/git/tmp/venv/lib/python2.7/site-packages/fftool/firefox_download.py", line 58, in download
platform=ch_platform
File "/Users/rpappalardo/git/tmp/venv/lib/python2.7/site-packages/mozdownload/factory.py", line 125, in init
scraper_types[scraper_type].init(self, *_kwargs)
File "/Users/rpappalardo/git/tmp/venv/lib/python2.7/site-packages/mozdownload/scraper.py", line 344, in init
Scraper.init(self, *args, *_kwargs)
File "/Users/rpappalardo/git/tmp/venv/lib/python2.7/site-packages/mozdownload/scraper.py", line 133, in init
self._retry_check_404(self.get_build_info)
File "/Users/rpappalardo/git/tmp/venv/lib/python2.7/site-packages/mozdownload/scraper.py", line 148, in _retry_check_404
self._retry(func, *_retry_kwargs)
File "/Users/rpappalardo/git/tmp/venv/lib/python2.7/site-packages/mozdownload/scraper.py", line 139, in _retry
return redo.retry(func, *_retry_kwargs)
File "/Users/rpappalardo/git/tmp/venv/lib/python2.7/site-packages/redo/init.py", line 152, in retry
return action(_args, *_kwargs)
File "/Users/rpappalardo/git/tmp/venv/lib/python2.7/site-packages/mozdownload/scraper.py", line 401, in get_build_info
self.date, self.build_index)
File "/Users/rpappalardo/git/tmp/venv/lib/python2.7/site-packages/mozdownload/scraper.py", line 482, in get_build_info_for_date
raise errors.NotFoundError(message, url)
mozdownload.errors.NotFoundError: Folder for builds on 2016-05-27-12-50-00 has not been found: https://archive.mozilla.org/pub/firefox/nightly/2016/05/

Currently it looks like hot garbage and is all OOP class based.

Possibly just make it a regular file which exposes a few methods.

Possibly split it into two separate files; one for "FirefoxEnvHandler" and one for "IniHandler". They are pretty separate concepts and probably have no reason to be living under the same roof.

While we're at it, clean_folder() is weird, and possibly doesn't belong anywhere since I don't think it is widely used in the codebase. Maybe just inline the shutil call.

Currently I believe that mozprofile requires Python 2.7 and fails under 3.x.
Need to verify and make sure we clearly outline requirements in the README.md.

$ export PREFS_ROOT_DIR = '../services-test'

should be something like
$ export PATH_PREFS_ROOT=$HOME/my-git-dir/services-test

this would be very handy for gofaster testing.
🏃 ⏳

OK, the way I see it, we have 4 major commands (with profile having 2 main options):

$ ff download [-c {channel}]
$ ff install [-c {channel}]
$ ff uninstall [-c {channel}]
$ ff profile --create {name}
$ ff profile --delete {name}

I'm not 100% sold on how ff profile -c/--create should work though. Currently when creating a profile, we need the following bits of information:

def create_mozprofile(application, test_type, env, profile_dir):

Where, the args are as follows:

• application: name of an application, ie: "loop-server"
• test_type: directory of the test type we want to run, ie: "stack-check"
• env: testing environment we plan on using, ie: "dev", "stage", or "prod" (or something else)
• profile_dir: name of the directory to save the profile to (which will have a prefs.js and user.js -- or something close to that).

We have those first 3 args, so we can do something like load the following config files:

• "prefs.ini"
• "{application}/prefs.ini:{env}"
• "{application}/{test_type}/prefs.ini:{env}"

I don't know how we want to pass those 4 args.

Not sure if we need to add one more nested depth of commands for ff profile so it's something like:

$ ff profile create --application loop-server --test_type stack-check --env stage --name rpapa9000

otherwise, we may be able to do something like this:

$ ff profile --create rpapa9000 --application loop-server --test_type stack-check --env stage
# ...
$ ff profile --delete rpapa9000

Not sure if that is awkward since we're treating --create and --delete as sub-subparsers, but binding the profile name/directory to it.

If we go with the latter option of ff profile --delete {name}, not sure how awkward it may be if ff profile --help gives you a bunch of non-relevant CLI options for --delete when we really only want the name of the profile to delete.

DISCUSS!

A common use case for testing is to launch a 'fresh' profile, but not necessarily one with any particular name other than it be unique.

In that case, we should add a function to generate a random profile name.

Currently we do a neat trick of adding globals in __init__.py, so we should remove any references from _temp/browsers/ and _temp/profiles/ in any *.py or *.ini files and just isolate all references to one place.

Currently this will implode if the _temp/profiles/ subdirectory doesn't exist. This gets created if you run make clean (or maybe make build), but it seems like we've hit this issue on all 3 major browsers, so I could probably be handling this a bit better.

Cygwin on Windows server and Windows 10 interprets paths differently.
Not sure what the cause is...

Win10 will construct a path with backslashes, whereas WinSrv will constract a path with forward slashes. Also, changing to forward slashes on Win10 yields "cannot find path specified error"

This may be an issue of path interpretation on Popen w/ Cygwin.
We'll go ahead and support WinSrv by default (since our automated tests run with that OS).

introduced with last PR

Currently we're using both (for technical reasons I now forget).

It'd be nice to standardize on one, if possible. And since we started with Fabric and switched to Popen, I'm guessing we should just try and stick w/ Popen since obviously there was something that we were having issues doing w/ Fabric.

Currently we have a half-broken way of specifying an --app, --test-type, and --env for recreating configs to test things like loop-server e2e tests for staging.

This won't work (currently) because our configs are stored in mozilla-services/services-test repo and not in the ff-tool (which would be absurd).

We'll need a minor refactorage where we can specify the services-test repo as an ENV var (or something).

Initial thoughts are:

$SERVICES_TEST_REPO=/Users/pdehaan/dev/github/services-test/ ff -c aurora --app loop-server --test-type e2e-test --env stage

That'd be great for testing and verifying, but can be verbose. But thankfully you can also export it as an env var so you never need to specify it on every run.

Because reasons: https://bitbucket.org/pypa/pypi/issues/148/support-markdown-for-readmes

https://testpypi.python.org/pypi/ff-tool/

seems like mozprofile may also be limited to python3 atm:
(venv) : $ python --version
Python 3.4.4
(venv) : $ pip install -r requirements.txt
Collecting mozdownload==1.20.2 (from -r requirements.txt (line 1))
Downloading mozdownload-1.20.2.tar.gz
Collecting mozprofile==0.28 (from -r requirements.txt (line 2))
Using cached mozprofile-0.28.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/private/var/folders/1n/znmq6xc95wq38tm9qh93bz7r0000gn/T/pip-build-1r7ljues/mozprofile/setup.py", line 12, in
assert sys.version_info[0] == 2
AssertionError

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/1n/znmq6xc95wq38tm9qh93bz7r0000gn/T/pip-build-1r7ljues/mozprofile/

There are a few projects (like shavar) that have multiple pref sets for 1 test environment (i.e. STAGE). Currently our tool is designed around a section header being a test env (like [stage]).

We'll need to figure out a way to adapt this so you could have multiple "named" pref sets for one test env.

https://wiki.mozilla.org/Services/TrackingProtection/Shavar_Server_-_Testing#SET_CUSTOM_PREFS

The following args probably need a hollywood makeover

no-launch
no-profile
install-only

Per conversation, we'll drop no-profile altogether.

no-launch vs. install-only - aren't these really the same? (we prob only need one)
@pdehaan?

ok, i've added a linux.ini file (but only because my version of ubuntu detects os as "linux" and not "linux-gnu". using the same file verbatim that was in services-test/_utils

Here's how far i get:

env)rpappalax@vimdiesel:~/git/ff-tool$ ff -c nightly

LOADING configs/linux.ini

LOADING configs/linux.ini

LOADING configs/linux.ini

Downloading nightly to temp/browsers/firefox-nightly.tar.bz2
/home/rpappalax/git/ff-tool/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/home/rpappalax/git/ff-tool/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Traceback (most recent call last):
File "/home/rpappalax/git/ff-tool/venv/bin/ff", line 9, in
load_entry_point('ff-tool==0.0.1', 'console_scripts', 'ff')()
File "/home/rpappalax/git/ff-tool/fftool/main.py", line 14, in main
download(options.channel)
File "/home/rpappalax/git/ff-tool/fftool/firefox_download.py", line 69, in download
install(channel)
File "/home/rpappalax/git/ff-tool/fftool/firefox_install.py", line 23, in install
local('tar -jxf {0} && mv firefox {1}'.format(installer, install_dir)) # NOQA
NameError: global name 'installer' is not defined

this can fill a hard disk if left unchecked. be sure to leave a clear warning

Windows bin path has spaces: C:\Program Files
which causes the script to choke in various places. This can be largely alleviated by using quotes, but in some places quotes must be escaped with backslashes. While this works in some places, in other places the backslashes cause problems and in others the quotes do. Need a more robust solution all-around to handling the default Windows path (with spaces).

Example error:

CREATING ENV FILE (.env)

export PATH_SIKULIX_BIN='/cygdrive/c/sikulix'
export PATH_FIREFOX_PROFILES='/cygdrive/c/Users/Administrator/AppData/Roaming/Mozilla/Firefox'
export PATH_FIREFOX_APP_NIGHTLY='/cygdrive/c/"Program Files"/Nightly/firefox.exe'
export PATH_FIREFOX_APP_AURORA='/cygdrive/c/"Program Files"/"Firefox Developer Edition"/firefox.exe'
export PATH_FIREFOX_APP_BETA='/cygdrive/c/"Program Files"/"Mozilla Firefox Beta"/firefox.exe'
export PATH_FIREFOX_APP_RELEASE='/cygdrive/c/"Program Files"/"Mozilla Firefox"/firefox.exe'

• source ./.env
  ++ export PATH_SIKULIX_BIN=/cygdrive/c/sikulix
  ++ PATH_SIKULIX_BIN=/cygdrive/c/sikulix
  ++ export PATH_FIREFOX_PROFILES=/cygdrive/c/Users/Administrator/AppData/Roaming/Mozilla/Firefox
  ++ PATH_FIREFOX_PROFILES=/cygdrive/c/Users/Administrator/AppData/Roaming/Mozilla/Firefox
  ++ export 'PATH_FIREFOX_APP_NIGHTLY=/cygdrive/c/"Program Files"/Nightly/firefox.exe'
  ++ PATH_FIREFOX_APP_NIGHTLY='/cygdrive/c/"Program Files"/Nightly/firefox.exe'
  ++ export 'PATH_FIREFOX_APP_AURORA=/cygdrive/c/"Program Files"/"Firefox Developer Edition"/firefox.exe'
  ++ PATH_FIREFOX_APP_AURORA='/cygdrive/c/"Program Files"/"Firefox Developer Edition"/firefox.exe'
  ++ export 'PATH_FIREFOX_APP_BETA=/cygdrive/c/"Program Files"/"Mozilla Firefox Beta"/firefox.exe'
  ++ PATH_FIREFOX_APP_BETA='/cygdrive/c/"Program Files"/"Mozilla Firefox Beta"/firefox.exe'
  ++ export 'PATH_FIREFOX_APP_RELEASE=/cygdrive/c/"Program Files"/"Mozilla Firefox"/firefox.exe'
  ++ PATH_FIREFOX_APP_RELEASE='/cygdrive/c/"Program Files"/"Mozilla Firefox"/firefox.exe'
• ff -c nightly
  /bin/sh: /cygdrive/c/Program: No such file or directory

Not sure if we can just transfer this over to the mozilla-services org: https://github.com/organizations/mozilla-services/

this is true (at least for configs/darwin.ini)

Currently we're using Fabric to set chmod +x on the installer *.exe on Windows (see ./fftool/firefox_install.py:27)
That seems like a bit of overkill and a blunt tool, when we could just do this directly with a bit of Python magic.

See http://stackoverflow.com/questions/12791997/how-do-you-do-a-simple-chmod-x-from-within-python for details on doing this in Python (and the | trick to add some bits).

Also, add outlawg statements to warm users of defaults when user leaves options unspecified

Windows profile path should behave no differently than OSX/Linux, for consistency.
Only the binary for Windows needs to be installed globally in C:/Program Files/

pip install only copies the module directory (fftool) into site-packages, so running ff-tool after pip install, tool will be unable to find config dir

https://github.com/rpappalax/test-out

Currently get_os() returns platform.system().lower() (and then sprinkles on some magic). This is "funny" because on Windows+Cygwin, it could return "windows" or "cygwin", meaning it will try and load configs/windows.ini (on our Win10 laptop), or configs/cygwin.ini (on our Win2012 server). I was seeing similar jank before for Linux as well.

So we should probably tweak the is_windows() to determine if it is running Windows or Cygwin and then get_os() will always return "windows".

    @staticmethod
    def get_os():
        """
        Do our best to determine the user's current operating system.
        """
        system = platform.system().lower()
        return re.split('[-_]', system, maxsplit=1).pop(0)

Add a $ ff --clean-profiles flag which deletes the _temp/profiles directory to avoid lingering profiles.

During investigation it appears that EACH profile takes up ~15MB of space, which is bad.

We should provide users with the ability to see firefox logging from stdout.

Add a new param -l LOGGING

this can be easily fixed by replacing read with readline and printing each line

should be ready to go, but we'll need to verify that pip install is working properly.
Also, since mozbase locks us into python2.x, I'd like to add a graceful failure msg which references the bug for mozbase supporting python 3