Run this cron job daily to remove any files owned by the deploy user that are older than 7 days. Otherwise /tmp fills up with imagemagick temp files.

sudo find /tmp -type f -mtime +7 -user deploy -execdir rm -- '{}' \;

ISSUE
For security purposes our servers should drop ssh sessions after a specified amount of inactivity. We want to be resilient on poor network connections though.

PROPOSED SETTINGS
ClientAliveInterval 120
ClientAliveCountMax 5

In the proposed settings, the client would have to fail at five consecutive 2-minute checks (i.e. 10 minutes), but would remain alive if the system provides a successful response in any one of the tries. In the original, the session dies immediately on a single check at 10 minutes with no retries.

This needs to go in /etc/ssh/sshd_config
See sample at https://github.com/curationexperts/ansible-hydra/blob/master/roles/system_setup/templates/ssh_config

I'm seeing this in Tufts and then trying to run from-cm as it's currently checked in. I've tried against both ubuntu 16.04 and 16.10 and get the same error.

Testing from the command line I get the following:

fatal: [54.197.209.70]: FAILED! => {"changed": false, "cmd": "/usr/local/bin/gem install --no-user-install --no-document bundler", "failed": true, "msg": "ERROR:  Error installing bundler:\n\t\"bundle\" from bundler conflicts with /usr/local/bin/bundle", "rc": 1, "stderr": "ERROR:  Error installing bundler:\n\t\"bundle\" from bundler conflicts with /usr/local/bin/bundle\n", "stderr_lines": ["ERROR:  Error installing bundler:", "\t\"bundle\" from bundler conflicts with /usr/local/bin/bundle"], "stdout": "", "stdout_lines": []}
	to retry, use: --limit @/Users/mark/Documents/workspace/_no_backup_/deploying_hydra/frbm-cm/build_cypripedium_server.retry

PLAY RECAP ***************************************************************************************************
54.197.209.70              : ok=27   changed=20   unreachable=0    failed=1   

MARKs-MacBook-Pro-2:frbm-cm mark$ ssh [email protected]
Welcome to Ubuntu 16.10 (GNU/Linux 4.8.0-56-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

Your Ubuntu release is not supported anymore.
For upgrade information, please visit:
http://www.ubuntu.com/releaseendoflife

New release '17.04' available.
Run 'do-release-upgrade' to upgrade to it.


*** System restart required ***
Last login: Fri Nov  3 00:32:40 2017 from 67.220.22.48
ubuntu@demo:~$ sudo gem install bundler
bundler's executable "bundle" conflicts with /usr/local/bin/bundle
Overwrite the executable? [yN]  n
ERROR:  Error installing bundler:
	"bundle" from bundler conflicts with /usr/local/bin/bundle
ubuntu@demo:~$ bundle --version
Bundler version 1.16.0
ubuntu@demo:~$ gem install bundler
Fetching: bundler-1.16.0.gem (100%)
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /usr/local/lib/ruby/gems/2.4.0 directory.
ubuntu@demo:~$ 

Currently, Val uses Ed25519 ssh keys (and they're more secure, so maybe all of us should be doing this). As-is, our sshd config does not support these keys. We should fix this. See https://linux-audit.com/using-ed25519-openssh-keys-instead-of-dsa-rsa-ecdsa/ for some pointers.

One of the most time-consuming maintenance tasks on ansible-samvera built systems is looking up the postgres password. Instead, let's record it in a secure way and make the ubuntu user not have to type it.

• Follow the guide at https://wiki.postgresql.org/wiki/Pgpass
• The postgres password should be recorded in a known place on the system
• The ubuntu user shouldn't need to type the password in order to connect to postgres

Today the fits download site went offline. I hope it comes back soon, and in the meantime I've been able to copy the .zip file from previously installed machines, but we should cache important installers somewhere for these kinds of outages.

ISSUE
We'd like to keep the install user's home directory as clean as possible, right now we have

vagrant@camper:~$ ls
imagemagick_sources  install  nodesource_setup.sh

Is there any reason not to include RAILS_ENV=production in the environment instead of having to prefix every command with it?

See ffmpeg vs. fits. Fits will inherit project-wide {{ install_path }}. ffmpeg assumes a ~/install exists. Pick one pattern or the other and make sure all roles ensure that their expected install directory exists at the beginning of the role.

Talk to Jamie about whether we could use the nailgun pattern for imagemagick and ffmpeg as well as fits

ISSUE
We'd like to keep the home directory for the install as clean as possible.

Right now we have something like

vagrant@camper:~$ ls
imagemagick_sources  install  nodesource_setup.sh

I've created a Packer build using these roles and it creates a VM output. In the process, I found there were slight differences between the VM source image I used and the source Ubuntu image/AMI you all must be using. I'm debating submitting PRs for these but for the moment have just handled them via a shell script in Packer.

I thought I'd document them here in case others use these roles and run into them.

Missing: nokogiri and nodejs (or any JavaScript runtime, but NodeJS was the one I chose to install)

The environment also needed to be set to UTF-8 for the Postgres databases to be created using that rather than LATIN1: update-locale LANG="en_US.UTF-8" LANGUAGE="en_US" (Having the databases created as LATIN1 caused the Fedora environment fail to start -- though this wasn't noticed in the Fedora role but in the last step of the first_deploy role where the admin set is created).

Fwiw, I also had to set the perms on /var/www to deploy:deploy (though for some reason you all aren't needing to do this, I guess). But this https://stackoverflow.com/questions/24470520/capistrano-mkdir-permission-denied solved a perm problem for me.

If you think it's worth splitting some of these into smaller tickets and getting rid of this all-in-one ticket, that's fine with me.

They should be configured with the name of the solr index we created:

deploy@mira-fc4:/opt/epigaea/current/config$ cat solr.yml
production:
url: http://127.0.0.1:8983/solr/epigaea

deploy@mira-fc4:/opt/epigaea/current/config$ cat blacklight.yml
production:
adapter: solr
url: http://127.0.0.1:8983/solr/epigaea

set up appropriate configurations for all content typically stored in $RAILS_ROOT/tmp in development environments

https://github.com/curationexperts/epigaea/blob/80b78ded44ee21ee3ead3c7b19497f99f37c6fb6/config/deploy.rb#L37-L49
append :linked_dirs, "public/assets"
append :linked_dirs, "tmp/pids"
append :linked_dirs, "tmp/cache"
append :linked_dirs, "tmp/sockets"
append :linked_dirs, "log"

link the draft dir specified in config/environments/produciton.rb config.drafts_storage_dir

append :linked_dirs, "tmp/drafts"

link the draft dir specified in config/environments/production.rb config.exports_storage_dir

append :linked_dirs, "tmp/exports"

link the template dir specified in config/environments/produciton.rb config.templates_storage_dir

append :linked_dirs, "tmp/templates"

We've hardcoded the ssh timeout into our config and may have cases (like building VMs or training EC2 instances for Camps) where we either want longer or no timeouts.

Possible options would be to make the timeouts settings configurable variables, or to pull out the timeout setting and make it a separate role:

  roles:
    ...
    - { role: sshd_config,  ClientAliveInterval=0 }
    ...

OR

  roles:
    ...
    - { role: sshd_config }
    - { role: sshd_timeout }
    ...

Compare this repo's usage https://github.com/curationexperts/ansible-samvera/search?utf8=✓&q=deploy&type=
to ansible-hydra usage https://github.com/curationexperts/ansible-hydra/search?utf8=✓&q=capistrano_user&type=

If for any reason we use a different user name to run the rails application - like when building a Vagrant VM for Samvera Camp, you can't use any of the scripts that assume there's a user named deploy. For my specific case, I was going to try to leverage the sidekiq role to get sidekiq running as a system service, but the role is hardcoded to use the deploy user which doesn't exist in the VM environment. Ideally I could use the role, but with the vagrant user.

ACCEPTANCE

• Notify potential users of deprecation
  • Schedule call with D. Sanford
• Merge into ansible-samvera as deprecated release (or find other means to support builds based on ansible-hydra until all DCE systems rebuilt)
• Delete ansible-hydra repo

It looks like after some packages are installed via apt get upgrade, apt get update, the server needs a restart. Rather than adding this to the packages role, however, let's make it a separate role that can be placed at the end of a playbook. That ensures it captures any other changes made during the configuration and proves that the web server, solr, and fedora restart correctly after a reboot.

• htop should install by default

Hyrax's default admin set contains a slash ("admin_set/default"), which leads to a bug where it can't be displayed unless passenger is configured to allow encoded slashes. For more context, see:

• samvera-labs/nurax-pre2023#32
• samvera/hyrax#2446 (comment)
• curationexperts/epigaea#808

The fix is to add this line to conf-enabled/passenger.conf:

  PassengerAllowEncodedSlashes on

• Add this to ansible so it's set by default for future builds
• Check existing projects (e.g., demo.curationexperts.com, FRBM) and ensure it's added there

See https://github.com/curationexperts/ansible-hydra/blob/master/roles/sufia_dependencies/install/handlers/main.yml#L36-L38

• In the solr role, change log4j.properties to set log level to WARN. Otherwise we swamp our splunk instance.

I'm getting

TASK [first_deploy : ensure default admin set with capistrano] ********************************************************************************
fatal: [52.206.81.96]: FAILED! => {"changed": true, "cmd": "cap localhost hyrax:ensure_default_admin_set", "delta": "0:00:00.287908", "end": "2017-11-03 20:28:26.710093", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-11-03 20:28:26.422185", "stderr": "(Backtrace restricted to imported tasks)\ncap aborted!\nDon't know how to build task 'hyrax:ensure_default_admin_set' (see --tasks)\n\n(See full trace by running task with --trace)", "stderr_lines": ["(Backtrace restricted to imported tasks)", "cap aborted!", "Don't know how to build task 'hyrax:ensure_default_admin_set' (see --tasks)", "", "(See full trace by running task with --trace)"], "stdout": "", "stdout_lines": []}
	to retry, use: --limit @/Users/mark/Documents/workspace/_no_backup_/deploying_hydra/tufts-cm/tufts-ubuntu.retry

Because of https://github.com/curationexperts/ansible-samvera/blob/v1.1.0/roles/first_deploy/tasks/main.yml#L80-L83

- name: ensure default admin set with capistrano
  shell: cap localhost hyrax:ensure_default_admin_set
  args:
    chdir: /home/{{ ansible_ssh_user }}/{{ project_name }}

But the hyrax:ensure_default_admin_set Capistrano task isn't defined in all of our projects. Could we just call the rake task directly?
execute :rake, 'hyrax:default_admin_set:create'