davidalger / capistrano-magento2 Goto Github PK

There are a couple of instances where it would be useful to be able to apply patches after "composer install" is run: patches provided by Magento Enterprise support and patches to resolve small issues that have yet to be rolled into a new release.

The patching functionality needs to be able to handle applying traditional .patch files as well as patches contained within shell scripts provided by Magento Enterprise support. My idea would be to have a patch directory at the root of the Magento install. Any .patch files contained in this folder would be automatically applied. .sh or files with no extension (the patch files provided by Magento don't always have extensions) would be executed by source.

Any failure in applying patches or shell scripts would need cause the deployment to fail. I just reviewed a patch script provided Magento Enterprise and any failure in applying the patch returns a non-zero exit code, so this should be simple. patch also returns a non-zero exit code

The other thing to consider would be how developers would apply the patch files on their local development machines.

@davidalger I'd be curious to hear your thoughts on how this should be implemented. Also, do you think you'd have any time to implement this over the few weeks?

I'm currently using magento 2.1.1 and set the magento_deploy_themes to ['Vendor/theme', 'Magento/backend'] but during static content compilation the --themes option is still missing, see result:

/usr/bin/env php -f bin/magento -- setup:static-content:deploy en_US en_GB de_DE| stdbuf -o0 tr -d .

Hi,

This is just an idea, not even sure it could work.
My question is why not copying vendor folder from previous version and then composer update instead of composer install? If works could be faster :)

Best regards,
Razvan

Hi,

due to my server configuration I need to run this static-content:deploy command:

HTTPS=on php bin/magento setup:static-content:deploy en_US

This creates static files in "secure" directory. To deploy this with capistrano I created this task

desc "Deploy static files"
task :static_deploy do
    on roles(:app) do
        within release_path do
            execute "cd '#{release_path}' && HTTPS=on php bin/magento setup:static-content:deploy en_US"
        end
    end
end

Is it possible to pass "HTTPS=on" in your script?

Would be good have entire access to Command Cli Magento adding something like this in file magento.rake :

desc 'Run Magento Cli'
  task :cli, :command do |t,args|
    on primary do
      within release_path do
        execute :magento, args[:command]
      end
    end
  end

Usage:
cap stage magento:cli['command']
E.g:

cap production magento:cli['indexer:status']
cap production magento:cli['cache:clean full_page']

What do you think?

I just upgraded one of my projects to Capistrano 3.7.1. After updating the lock '3.6.1' line in my tools/cap/config/deploy.rb to lock '3.7.1' and running bundle exec cap stage deploy, I got this deprecation warning:

[Deprecation Notice] Future versions of Capistrano will not load the Git SCM
plugin by default. To silence this deprecation warning, add the following to
your Capfile:

    require "capistrano/scm/git"
    install_plugin Capistrano::SCM::Git

The deployment continued without a problem, but this project's readme should be updated to include those two configuration lines in the Capfile.

The current deployment process uses bin/magento setup:upgrade for the purpose of upgrading the schema and data in a Magento 2 installation if needed. This command also re-generates the app/etc/config.php file. Due to the errors Magento 2 will throw up if a site has modules requiring schema and/or data updates installed, the site must be put into maintenance mode before linking the new release and running setup:upgrade. This results in a downtime between 5 and 8 seconds on average (by my rather unscientific estimation based on real-life experience… ;)).

Stated goal of this task: eliminate all downtime for deployments which do not trigger schema or data upgrades.

Additionally, as the app/etc/config.php file is regenerated by the setup:ugprade command, this causes a potential fringe situation on a multi-node cluster deployment (as of 0.5.3) where setup:upgrade updates this file on one (but not all) nodes in the deployment. This is only an issue if the config.php file re-created when this runs is different (for some reason or other) than the one committed to the repository. Eliminating the use of setup:upgrade will have the effect of eliminating this potential issue as well.

At the suggestion of @hostep over on issue #33 I went looking into the use of setup:db:status and found the following:

1. The built-in setup:db:status command will report on whether or not any installed module(s) require schema and/or data upgrade. Example output of this command:

   All modules are up to date.
   

   or

   The module code base doesn't match the DB schema and data.
            Magento_Cms     schema:       2.0.0  ->  2.0.1      
         Magento_Bundle       data:       2.0.0  ->  2.0.2      
   Run 'setup:upgrade' to update your DB schema and data.
   

The command returns the same exit code of 0 regardless of the db status.

1. There are also two other interesting built-in console commands, setup:db-schema:upgrade and setup:db-data:upgrade. Upon my investigation into the code, the difference between these commands and setup:upgrade is the following:

   setup:upgrade calls updateModulesSequence, installSchema, and installDataFixtures whereas the other two commands only call installSchema and installDataFixtures respectively.

2. These additional commands are available in versions 2.0 and 2.1 making no special version checks necessary.

The result of this task will be to cease use of setup:upgrade and transition to only calling the schema/data upgrade specific tasks on deploy, eliminating the potential fringe issue with a deploy creating unexpected changes to config.php as well as eliminate all downtime on deploys which do not introduce module (schema and/or data) version changes.

setup:static-content:deploy takes the languages you want to generate as arguments. If no arguments are provided it deploys en_US.

I am working on a multilanguage website and I need to generate files for all configured languages.
AFAIK there are no way to tell setup:static-content:deploy to compile all languages, so I need a way to add argument to the command in the deploy phase.

Can we have a variable to put in deploy.yml containig an array of languages that are then passed to setup:static-content:deploy ?

Hello,

First of all thank you for this great gem, I wanted to share my experience so far. I had some problem after successfully running the setup:di:compile-multi-tenant task, it seams like it was expecting to find a current symlink. This is the error i got:

 DEBUG [5e3b1234]       Magento\AdminNotification\Controller\Adminhtml\Notification\MassRemove\Interceptor
 DEBUG [5e3b1234]       Magento\AdminNotification\Controller\Adminhtml\Notification\MarkAsRead\Interceptor
 DEBUG [5e3b1234]       Magento\AdminNotification\Controller\Adminhtml\Notification\MassMarkAsRead\Interceptor
 DEBUG [5e3b1234]   On *nix systems, verify the Magento application has permissions to modify files created by the compiler in the "var" directory. For instance, if you run the Magento application using Apache, the owner of the files in the "var" directory should be the Apache user (example command: "chown -R www-data:www-data <MAGENTO_ROOT>/var" where MAGENTO_ROOT is the Magento root directory).
  INFO [5e3b1234] Finished in 54.483 seconds with exit status 0 (successful).
  INFO [3889191b] Running /usr/bin/env rm -f var/di/relations.ser as [email protected]
 DEBUG [3889191b] Command: cd /home/user/example.com/releases/20160507060723 && /usr/bin/env rm -f var/di/relations.ser
  INFO [3889191b] Finished in 0.321 seconds with exit status 0 (successful).
 DEBUG [fd1c5a48] Running if test ! -d /home/user/example.com/current; then echo "Directory does not exist '/home/user/example.com/current'" 1>&2; false; fi as [email protected]
 DEBUG [fd1c5a48] Command: if test ! -d /home/user/example.com/current; then echo "Directory does not exist '/home/user/example.com/current'" 1>&2; false; fi
 DEBUG [fd1c5a48]   Directory does not exist '/home/user/example.com/current'

After i manually created a current symlink, i was able to finish the deployment on the next run.

cheers,

capistrano-pending is a really useful Gem that can be used to see what changes are about to be deployed to an environment.

I'd like to see usage of this Gem incorporated into the capistrano-magento2, similar to how you've included terminal-notifier.

My proposal of how it would work:

Whenever you run cap <STAGE> deploy, you would see a list of all of the pending changes as the first output of the deploy command:

If there are no pending changes, the command would warn you about this and ask you if you wanted to continue deploying:

While a developer could just include the capistrano-pending Gem into their project and run cap <STAGE> deploy:pending, the benefit of my suggested approach is that they would automatically see pending changes AND wouldn't accidentally deploy "no changes", which I've done more than once.

If I were to submit a PR that implemented this change, would you accept it?

Available commands:
help
list
admin
admin:user:create
i18n
i18n:collect-phrases
i18n:pack
info
info:adminuri
info:backups:list
info:currency:list
info:dependencies:show-framework
info:dependencies:show-modules
info:dependencies:show-modules-circular
info:language:list
info:timezone:list
maintenance
maintenance:allow-ips
maintenance:disable
maintenance:enable
maintenance:status
module
module:disable
module:enable
module:status
module:uninstall
sampledata
sampledata:deploy
sampledata:remove
setup
setup:backup
setup:config:set
setup:cron:run
setup:db-data:upgrade
setup:db-schema:upgrade
setup:db:status
setup:di:compile
setup:di:compile-multi-tenant
setup:install
setup:performance:generate-fixtures
setup:rollback
setup:store-config:set
setup:uninstall
setup:upgrade

It would be nice to have a way to skip static_content_deploy and di_compile when deploying to a development environment.

When a db role is not defined, the capistrano-pending functionality doesn't work. For example, if you're using a role configuration like this:

role :app, %w{
  [email protected]
}

You'll get output like this:

However if you setup both the app and db roles like this:

server 'example.com', user: 'www-data', roles: %w{app db}

You'll get output like expected:

Forcing users to define a db role doesn't seem like the right approach, so I'll let you take it from here to come up with the proper approach to this bug as I don't have any more time to look into this.

This seems to be a bug caused by the Capistrano :revision variable (see here) not being populated unless a db role is defined. If you agree that this seems to be a core Capistrano bug, not anything related to this Gem, then you're free to close this issue. We may just want to update the relevant section of this Gem's readme to alert users of this bug and open an issue on the Capistrano issues list.

Unable to complete the deployment due to this error:

ArgumentError: Malformed version number string 2.1.5

Tracing it back to magento2.rb and that maybe the capture :php portion of the code isn't returning a string as expected?

SSHKit.config.command_map[:magento] = "/usr/bin/env php -f bin/magento --"

module Capistrano
  module Magento2
    module Helpers
      def magento_version
        return Gem::Version::new((capture :php, "-f #{release_path}/bin/magento -- -V").split(' ').pop)
      end

      # ...more methods...
  end
end

Looking through the various forks I found minhnd55/capistrano-magento2 has a solution that works

I'll have a pull request ready soon to patch the issue.

Our hosting provider has PHP 7.0 installed to a custom path in /opt, with a base version of PHP 5.4 in /usr/bin. Unfortunately, /usr/bin/env php is hard-coded in magento2.rb, which picks up PHP 5.4.

Ideally, we would be able to override the PHP path with a setting in deploy.rb or [stage].rb. To that end, I tried adding SSHKit.config.command_map[:magento] with the correct path to production.rb. That worked fine for the first occurrence, but not the second one. I also tried to add the setting myself in a fork, but I could get it to output the path correctly. (My Ruby skills are not strong enough to figure why it did not output correctly.)

P.S. Thank you for creating this wonderful tool. Other than the path issue, it works great!

Hi,

I have tried both "en_GB" and 'en_GB"

set :magento_deploy_languages, 'en_GB'

(Backtrace restricted to imported tasks)
cap aborted!
NoMethodError: undefined method `join' for "en_GB":String

Tasks: TOP => magento:setup:static-content:deploy
(See full trace by running task with --trace)
The deploy has failed with an error: undefined method `join' for "en_GB":String


** DEPLOY FAILED
** Refer to log/capistrano.log for details. Here are the last 20 lines:

If a deploy is made when magento is in maintenance mode then is set back online. It would be useful to detect the current application maintenance status and disable maintenance mode only if it was already disabled at the beginning of the deploy process.

Hi,

I'm not sure if this is directly an issue for deployment, however it poses a problem.

Currently using the recommended .gitignore from here:
https://github.com/magento/magento2-community-edition/blob/master/.gitignore

The 'lib/' directory is excluded. Some extensions however are placing directories in here, for example Magestore and Magebirdpopup.

Currently I'm forcing a git add on these directories to add them in. Is this the best approach or should it be catered for in deployment?

Kind regards,
Kieron

I use jenkins to do my deployments and it already has downloaded all composer modules before it goes off to deploy to my sites.

So it's not necessary to run invoke magento:composer:install.

I've tried copying deploy.rake to lib/capistrano/tasks/deploy.rake and commented out the above call, however it still seems to run the magento:composer:install

This issue is a similar problem to the one reported in #25. It is also somewhat related to #35.

Here is an example screenshot demonstrating the problem:

I'm using Magento EE 2.1.2. Errors generated during setup:di:compile are not being reported when deploying. The errors are essentially being ignored and the deployment continues as if nothing happened.

I think the problem is that we assumed that Magento 2.1.x fixed the issue where setup:di:compile and setup:static-content:deploy were not returning non-zero exit codes upon failure, but this is not always the case. While the magento/magento2#3189 PR did fix the return code issue in many situations, it looks there are certain "soft" errors that don't result in error codes: https://github.com/magento/magento2/blob/1ade3b769a937f19682bbbe4e51b6c956147952f/setup/src/Magento/Setup/Module/Di/Compiler/Log/Writer/Console.php#L72

Preconditions

1. Install composer 1.3.0 or higher.
2. Check out Magento 2.1.3.
3. Install capistrano-magento2 5.9.

Steps to reproduce

1. configure and run cap production deploy.

Expected result

1. Deploy runs fine.

Actual result

1. Part of the log ( I added -vv for debugging purpose to static content generation command, so the output has more details, not only errors count )

02 Processing file 'requirejs/requirejs' for area 'frontend', theme 'NAMESPACE/nec', locale 'en_US'
02 	Deploying the file to 'frontend/NAMESPACE/nec/en_US/requirejs/requireminjs'
02 Processing file 'requirejs/domReadyjs' for area 'frontend', theme 'NAMESPACE/nec', locale 'en_US'
02 	Deploying the file to 'frontend/NAMESPACE/nec/en_US/requirejs/domReadyminjs'
02 Processing file 'js-translationjson' for area 'frontend', theme 'NAMESPACE/nec', locale 'en_US'
02 	Deploying the file to 'frontend/NAMESPACE/nec/en_US/js-translationjson'
02 #0 /var/www/magento/releases/20170221151910/vendor/colinmollenhour/cache-backend-redis/Cm/Cache/Backend/Redisphp(313): Credis_Client->__call('hget', Array)
02 #1 /var/www/magento/releases/20170221151910/vendor/magento/zendframework1/library/Zend/Cache/Corephp(390): Cm_Cache_Backend_Redis->save('a:105:{i:123;s:', '537_TRANSLATE_E', Array, false)
02 #2 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Corephp(74): Zend_Cache_Core->save('a:105:{i:123;s:', '537_TRANSLATE_E', Array, false, 8)
02 #3 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Adapter/Zendphp(47): Magento\Framework\Cache\Core->save('a:105:{i:123;s:', 'TRANSLATE_EN_US', Array, false)
02 #4 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/Barephp(75): Magento\Framework\Cache\Frontend\Adapter\Zend->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
02 #5 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/TagScopephp(49): Magento\Framework\Cache\Frontend\Decorator\Bare->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
02 #6 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/Barephp(75): Magento\Framework\Cache\Frontend\Decorator\TagScope->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
02 #7 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/Barephp(75): Magento\Framework\Cache\Frontend\Decorator\Bare->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
02 #8 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/Cache/Type/AccessProxyphp(85): Magento\Framework\Cache\Frontend\Decorator\Bare->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
02 #9 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/Barephp(75): Magento\Framework\App\Cache\Type\AccessProxy->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
02 #10 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/TagScopephp(49): Magento\Framework\Cache\Frontend\Decorator\Bare->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
02 #11 /var/www/magento/releases/20170221151910/vendor/magento/framework/Translatephp(489): Magento\Framework\Cache\Frontend\Decorator\TagScope->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
02 #12 /var/www/magento/releases/20170221151910/vendor/magento/framework/Translatephp(185): Magento\Framework\Translate->_saveCache()
02 #13 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/Areaphp(240): Magento\Framework\Translate->loadData(NULL, false)
02 #14 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/Areaphp(211): Magento\Framework\App\Area->_initTranslate()
02 #15 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/Areaphp(138): Magento\Framework\App\Area->_loadPart('translate')
02 #16 /var/www/magento/releases/20170221151910/vendor/magento/module-translation/Model/Json/PreProcessorphp(84): Magento\Framework\App\Area->load('translate')
02 #17 /var/www/magento/releases/20170221151910/vendor/magento/framework/View/Asset/PreProcessor/Poolphp(74): Magento\Translation\Model\Json\PreProcessor->process(Object(Magento\Framework\View\Asset\PreProcessor\Chain))
02 #18 /var/www/magento/releases/20170221151910/vendor/magento/framework/View/Asset/Sourcephp(152): Magento\Framework\View\Asset\PreProcessor\Pool->process(Object(Magento\Framework\View\Asset\PreProcessor\Chain))
02 #19 /var/www/magento/releases/20170221151910/vendor/magento/framework/View/Asset/Sourcephp(105): Magento\Framework\View\Asset\Source->preProcess(Object(Magento\Framework\View\Asset\File))
02 #20 /var/www/magento/releases/20170221151910/vendor/magento/framework/View/Asset/Filephp(150): Magento\Framework\View\Asset\Source->getFile(Object(Magento\Framework\View\Asset\File))
02 #21 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/View/Asset/Publisherphp(73): Magento\Framework\View\Asset\File->getSourceFile()
02 #22 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/View/Asset/Publisherphp(61): Magento\Framework\App\View\Asset\Publisher->publishAsset(Object(Magento\Framework\View\Asset\File))
02 #23 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/Deploy/LocaleDeployphp(398): Magento\Framework\App\View\Asset\Publisher->publish(Object(Magento\Framework\View\Asset\File))
02 #24 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/Deploy/LocaleDeployphp(235): Magento\Deploy\Model\Deploy\LocaleDeploy->deployFile('js-translation', 'frontend', 'Magento/blank', 'en_US', NULL)
02 #25 [internal function]: Magento\Deploy\Model\Deploy\LocaleDeploy->deploy('frontend', 'Magento/blank', 'en_US')
02 #26 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/Statephp(171): call_user_func_array(Array, Array)
02 #27 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/DeployManagerphp(175): Magento\Framework\App\State->emulateAreaCode('frontend', Array, Array)
02 #28 [internal function]: Magento\Deploy\Model\DeployManager->Magento\Deploy\Model\{closure}(Object(Magento\Deploy\Model\Process))
02 #29 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/Processphp(53): call_user_func(Object(Closure), Object(Magento\Deploy\Model\Process))
02 #30 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/ProcessManagerphp(55): Magento\Deploy\Model\Process->run()
02 #31 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/ProcessQueueManagerphp(152): Magento\Deploy\Model\ProcessManager->fork(Object(Closure))
02 #32 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/ProcessQueueManagerphp(130): Magento\Deploy\Model\ProcessQueueManager->fork(Object(Magento\Deploy\Model\ProcessTask))
02 #33 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/ProcessQueueManagerphp(88): Magento\Deploy\Model\ProcessQueueManager->internalQueueProcess(Array, Array)
02 #34 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/DeployManagerphp(187): Magento\Deploy\Model\ProcessQueueManager->process()
02 #35 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/DeployManagerphp(122): Magento\Deploy\Model\DeployManager->runInParallel(Object(Magento\Deploy\Model\DeployStrategyProvider))
02 #36 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Console/Command/DeployStaticContentCommandphp(386): Magento\Deploy\Model\DeployManager->deploy()
02 #37 /var/www/magento/releases/20170221151910/vendor/symfony/console/Symfony/Component/Console/Command/Commandphp(257): Magento\Deploy\Console\Command\DeployStaticContentCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
02 #38 /var/www/magento/releases/20170221151910/vendor/symfony/console/Symfony/Component/Console/Applicationphp(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
02 #39 /var/www/magento/releases/20170221151910/vendor/symfony/console/Symfony/Component/Console/Applicationphp(195): Symfony\Component\Console\Application->doRunCommand(Object(Magento\Deploy\Console\Command\DeployStaticContentCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
02 #40 /var/www/magento/releases/20170221151910/vendor/magento/framework/Console/Cliphp(96): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
02 #41 /var/www/magento/releases/20170221151910/vendor/symfony/console/Symfony/Component/Console/Applicationphp(126): Magento\Framework\Console\Cli->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
02 #42 /var/www/magento/releases/20170221151910/bin/magento(23): Symfony\Component\Console\Application->run()
02 #43 {main}
02 
02 Successful: 2249 files; errors: 1

What could possibly be wrong, because while running setup:static-content:deploy from magento itself, it works, with no errors?! Please advice!

I also opened a question in magento repository: magento/magento2#8643.

Thanks in advanced!

I'm getting this error when the deploy:reset_permissions task is run:

sudo: unable to execute /bin/chmod: Argument list too long

The pub/media directory contains 6492 files:

See this article for context on the error: http://msyk.es/blog/linux-commands-argument-list-too-long/

We're running a site over multiple servers and when running a deployment that includes any changes to the DB, we were getting errors due to the setup:upgrade running on all the servers and trying to make the the same DB change multiple times.

It would be great if setup:upgrade only ran on the servers with a role of "db", rather than how it is currently configured to run for 'all' roles.

I'm trying to purge varnish during a deployment, i've seen in magento.rake the code that purges varnish, but i'm getting an error when it runs.

I've added the following to my production.rb:

set :ban_pools, ['default']
set :varnish_cache_hosts, [first_ip_address', 'second_ip_address']

During a deployment I get an error because the Magento vanish.vcl expect the X-Magento-Tags-Pattern header which doesn't appear to be set during the purge request.

Is the curl command that attempts to flush varnish passing the appropriate header?

@davidalger would it be possible to make this a setting to skip magento:composer:install.

Also is it possible to skip maintenance mode? This causes an exception for me due to a bug in data migration tool

Magento 2.1.1 added a number of new flags to the bin/magento setup:static-content:deploy command:

Since it is now possible to exclude certain themes from the static compilation, I'd recommend adding a :magento_deploy_exclude_themes argument that will accept an array of themes to exclude.

For example, this:

set :magento_deploy_exclude_themes, ['Magento\blank', 'Magento\luma', 'ClassyLlama\basetheme']

Would result in this command being run:

bin/magento setup:static-content:deploy --exclude-theme="Magento/blank" --exclude-theme="Magento/luma" --exclude-theme="ClassyLlama/basetheme"

Adding this feature will reduce the deployment time significantly. As a point of reference, on a recent deployment of a Magento 2.1.0 site with 6 themes (only 3 of which needed to be compiled), the static asset compilation to 8:50m out of the 15:15m deploy time. If three of those themes were not compiled, the total compilation would have been reduced to around 11 minutes.

When deploying a new theme, it fails because setup:static-content:deploy runs before setup:upgrade and there is no such theme found in theme table in database.

Error thrown is:

Unable to load theme by specified key: '<vendor>/<theme>' 

Wonder if static content should not be moved after setup upgrade.

Hello,

I'm having issue during static content deploy. See below:

DEBUG [023dc095] Command: cd /var/www/html/project/releases/20161123062111 && set -o pipefail; magento setup:static-content:deploy en_US -t Smartwave/porto -t Smartwave/porto_rtl -t Kartcent/default -t Magento/backend | stdbuf -o0 tr -d .

DEBUG [023dc095] bash: magento: command not found

There is a long-standing bug that prevents new themes from being registered when Magento is in production mode. This means that when Capistrano tries to deploy a new theme to the server, an error like [LogicException] Unable to load theme by specified key: 'ClassyLlama/example' will occur.

This Gem is not recognizing this output as an error, so the deployment proceeds even after error is triggered:

I would think the https://github.com/davidalger/capistrano-magento2/blob/master/lib/capistrano/magento2.rb#L44..L48 section should be expanded to check for the presence of [.*Exception] in the output. Doing this search will catch the different types of exceptions that can be thrown during asset compilation. Look at the two exceptions in this method for an example: https://github.com/magento/magento2/blob/1ade3b769a937f19682bbbe4e51b6c956147952f/lib/internal/Magento/Framework/View/Design/Theme/FlyweightFactory.php#L53

If the :ban_pools and :varnish_cache_hosts values are set in the configuration files, the magento:cache:varnish:ban command should fail if Varnish doesn't return a "200 Purged" message. This will ensure that a user won't mistakenly think that Varnish purging is working.

Composer dev dependencies should not be installed on production servers #22 fix this problem.

Magento 2 deployment is broken in this release with the addition of 'pipefail'.

Static content deployment now fails as it is not using 'php' i.e.

01:13 magento:setup:static-content:deploy
01 touch /home/mage2lexeldevser/public_html/releases/20161128161238/pub/static/deployed_version.txt
01 stdin: is not a tty
✔ 01 [email protected] 0.088s
02 set -o pipefail; magento setup:static-content:deploy en_GB | stdbuf -o0 tr -d .
02 stdin: is not a tty
02 bash: magento: command not found
(Backtrace restricted to imported tasks)
cap aborted!

This use to run as follows:

00:51 magento:setup:static-content:deploy
01 touch /home/mage2lexeldevser/public_html/releases/20161124160928/pub/static/deployed_version.txt
01 stdin: is not a tty
✔ 01 [email protected] 0.076s
02 php -f bin/magento -- setup:static-content:deploy en_GB -t Magento/luma -t Magento/backend | stdbuf -o0 tr -d .
02 stdin: is not a tty
02 Requested languages: en_GB

I've removed my capistrano-magento2-0.5.7 directory and rolled back to capistrano-magento2-0.5.6 to get this back to a working version.

Thanks.

@davidalger was wondering if you had any plans to incorporate module enabling/disabling into this amazing project? apologize if i'm just not looking in the right place, but i didn't notice any reference to the CLI commands for modules, only printouts of disabled modules in the setup:upgrade task (+1 to #34 by the way)

from my experience, there'd basically be two types of module installations...

1. static-content-less modules - all you'd have to do i believe is run module:enable before setup:upgrade and that would be it.
2. static-content-ful modules - this is where it could get tricky and #34 may come in handy. If static content needs to be generated for the module, then module:enable needs to be run before static content, but then that would mean maintenance mode would be on for the duration of static content. not good.

currently what we are doing is just running the deploy without enabling the module, then going directly into the release folder and running the module-related commands again. very clunky and totally defeats the purpose of the awesomeness of capistrano. any thoughts?

I'm using Magento EE 2.1.1 with Capistrano 0.5.1. Capistrano is ignoring static asset compilation errors. Screenshot demonstrating problem (I changed command_output to true to see all output for the sake of debugging):

In this commit, the explicit check for the "Compilation from source" error message was removed. In order to fix this issue, I propose that this check gets added back (and possibly ALSO check for the presence of the "New version of deployed files" success message).

If you'd like me to submit this as a PR, let me know.

If the bin/magento setup:upgrade command fails, the site will remain in maintenance mode even after Capistrano is finished running. This results in a non-atomic deployment when there are errors with setup:upgrade. Example screenshot:

It seems like it would make sense for the capistrano-magento2 Gem to be smart enough to take this scenario into account and take the site out of maintenance mode if setup:upgrade fails.

I was able to use cap stage deploy:rollback to successfully rollback to a previous working release.

Presumably the second run with https: 'on' is failing every time after magento update.

Successful: 0 files; errors: 7

Successful: 6 files; errors: 1

Please, help, how can I override this task to exclude this behavior.

I ran a deployment to prod (bundle exec cap prod deploy) and it did not report any errors during the di:compile step.

00:46 magento:setup:static-content:deploy
01 php -f bin/magento -- setup:static-content:deploy | stdbuf -o0 tr -d .
✔ 01 [email protected] 62.177s
01:49 magento:setup:di:compile
01 php -f bin/magento -- setup:di:compile-multi-tenant
✔ 01 [email protected] 38.825s
02:28 magento:maintenance:enable
01 php -f bin/magento -- maintenance:enable
✔ 01 [email protected] 0.706s

I only attempted this because I saw the same error when trying to recreate an issue on my local dev machine by running my dev environment in production mode which caused me to run the di:compile step by hand on my dev environment.

When I ran di:compile manually on the production environment it produced an error at the end:

$ bin/magento setup:di:compile-multi-tenant
Errors during compilation:
    ClientNamespace\GetRewardsPoints\Block\Product\Rewards
        Incorrect dependency in class ClientNamespace\GetRewardsPoints\Block\Product\Rewards in /var/www/prod/releases/20160831142430/app/code/ClientNamespace/GetRewardsPoints/Block/Product/Rewards.php
\Magento\Store\Model\StoreManagerInterface already exists in context object
Total Errors Count: 1

report originally from @mttjohnson

Do you know why this is the case? When I do a deployment without checking this directory into version control, the deployment fails

In Capistrano 3.6, tasks can't be run twice without explicitly setting "reenable" (see capistrano/capistrano#1686).

While I haven't experienced any problems caused by the second call to magento:setup:permissions being skipped, the behavior of this Gem needs to be changed to take into consideration the Capistrano 3.6 behavior. So either explicitly set "reenable" or change magento:setup:permissions to only be called once.

Hello!

In the deploy, we do cache:flush, then we set maintenance:disable.

In practice this can take quite some time and varnish can cache again from a client browser request, while we still point to a maintenance:enable status. For example it toked more than 5 seconds to disable: Finished in 5.229 seconds with exit status 0 (successful)..

This cause site to show disabled mode, after changing the current folder, when deploy is complete.

Suggesting a quick fix, would be to set maintenance:disable first, before cache:flush in here:

cc @davidalger

Thanks!

If I freshly clone a repository ending up with a master branch checked out, then deploy to a target which is configured to deploy from say develop, a branch I have not yet checked out, the following will turn up in the output even when the deploy is going forward:

Changes pending deployment on web1 (remotes/origin/develop~5 -> ):
Warning: It appears you may be going backwards in time on web1 with this deployment!

This turns up in the local deploy log:

 DEBUG [3de4df86] Running /usr/bin/env git name-rev --always --name-only 9c64223a2a0080653c5f4164e3c8ed410b24afca as dalger@localhost
 DEBUG [3de4df86] Command: /usr/bin/env git name-rev --always --name-only 9c64223a2a0080653c5f4164e3c8ed410b24afca
 DEBUG [3de4df86] 	remotes/origin/develop~5
 DEBUG [3de4df86] Finished in 0.007 seconds with exit status 0 (successful).
 DEBUG [c1b1a750] Running /usr/bin/env git rev-parse --abbrev-ref --symbolic-full-name develop@{u} as dalger@localhost
 DEBUG [c1b1a750] Command: /usr/bin/env git rev-parse --abbrev-ref --symbolic-full-name develop@{u}
 DEBUG [c1b1a750] 	fatal: no such branch: 'develop'
 DEBUG [c1b1a750] Finished in 0.008 seconds with exit status 32768 (failed).
 DEBUG [30c9b477] Running /usr/bin/env git name-rev --always --name-only develop as dalger@localhost
 DEBUG [30c9b477] Command: /usr/bin/env git name-rev --always --name-only develop
 DEBUG [30c9b477] 	Could not get sha1 for develop. Skipping.
 DEBUG [30c9b477] Finished in 0.006 seconds with exit status 0 (successful).

After a try I faced that this just supose that magento is installed in the root folder of the repository cloned but in my case magento is installed in the src folder of my repository.

Do you now if there is a light way e.g. setting a variable to customize where the magento is located?

Thanks you.

nginx is running as www-data, deployment is via another use account

2016/09/27 13:23:41 [crit] 20330#20330: *2085 stat() "/var/www/vhosts/xx/current/pub/" failed (13: Permission denied), client: 127.0.0.1, server: www.xxx.com, request: "GET / HTTP/1.1", host: "www.xxx.com"
2016/09/27 13:23:41 [crit] 20330#20330: *2085 stat() "/var/www/vhosts/xxx/current/pub/" failed (13: Permission denied), client: 127.0.0.1, server: www.xxx.com, request: "GET / HTTP/1.1", host: "www.xxx.com"
2016/09/27 13:23:41 [crit] 20330#20330: *2085 stat() "/var/www/vhosts/xxx/current/pub/index.php" failed (13: Permission denied), client: 127.0.0.1, server: www.xxx.com, request: "GET / HTTP/1.1", host: "www.xxx.com"

Is it possible to make the permissions being set an option also? Thank you

If not if you have any idea how to fix this! :)

We noticed an instance of upgrading a magento site from 2.0 to 2.1 caused a lot more work for the db upgrade process and that step in the deployment tasks failed because we hit a PHP memory limit. We were able to restore a database backup and run the db upgrade manually on the server passing in an option on the command line to increase the memory limit so setup:upgrade would complete successfully.

Most deployments have never had any problems, but the update between 2.0 and 2.1 was more extensive than most on the db upgrade.

Command line execution of php on the server had a memory_limit of 1G
php -i | grep memory_limit
memory_limit => 1G => 1G

Some people have suggested removing the memory limit on the setup:upgrade call like:
php -d memory_limit=-1 bin/magento setup:upgrade

Is that something that should be in the capistrano module's call to setup:upgrade (or related db upgrade calls), or do you think that's something better left to changing the system default to be above 1G to avoid this in the future?

Feature suggestion:

1. Add ability to specify a different branch to be checked out using a command like this:

   BRANCH=feature/custom-feature bundle exec cap stage deploy 
   

2. Add a check to prevent different branches from being checked out by different users.

See this article for reference implementation for both of the above items: https://firmhouse.com/blogdeploying-branches-with-capistrano/

Hello,

I'm using your gem in linux (with zsh shell) and at the end of a deploy terminal notifier spits out some garbage text (see screenshot).

Also, terminal notifier outputs the same even in other unrelated commands I call (eg, vagrant or ansible). Since it is not essential can the dependency be removed? I tried to google around to find some solutions but can't find anything.

Hello,

I have a problem when I deploy my Magento2 application, during this command.
/usr/bin/env php -f bin/magento -- setup:static-content:deploy en_US

I have following error :
SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

I have a database server separated from my web server, and I see that symlink to env.php file is not done when setup:static-content:deploy is launched.

Is there a way to do symlinks before launching magento commands ?

Thank you for your help.

Current permissions implementation (as of v0.5.4) looks like the following:

set :magento_deploy_chmod_d, fetch(:magento_deploy_chmod_d, '2770')
set :magento_deploy_chmod_f, fetch(:magento_deploy_chmod_f, '0660')
set :magento_deploy_chmod_x, fetch(:magento_deploy_chmod_x, ['bin/magento'])

execute :find, release_path, "-type d ! -perm #{fetch(:magento_deploy_chmod_d).to_i} -exec chmod #{fetch(:magento_deploy_chmod_d).to_i} {} +"
execute :find, release_path, "-type f ! -perm #{fetch(:magento_deploy_chmod_f).to_i} -exec chmod #{fetch(:magento_deploy_chmod_f).to_i} {} +"

fetch(:magento_deploy_chmod_x).each() do |file|
   execute :chmod, "+x #{release_path}/#{file}"
end

ref: https://github.com/davidalger/capistrano-magento2/blob/master/lib/capistrano/tasks/magento.rake#L306-L311

My long-term goal would be to see a better set of default permissions applied by default. It could incorporate things such as read-only access to all php files while allowing write to directories which M2 needs write access to in order to function (think var and pub/media).

I'm hesitant to do this without putting a lot of research effort into this for a few reasons:

1. The permissions that work well for me might not work for others.

2. Tighter security on the files means less flexibility.

3. Anyone is free to implement their own customized permissions via a custom rake task which executes after the one which runs here:

   after 'magento:setup:permissions', :my_custom_permissions_task do
     # your custom permissions code
   end

Any thoughts on how to achieve this and/or make permissions handling more flexible while tightening security of files on the app server(s) is welcome!

Hi,

The capistrano tasks does not set the production mode. Is it possible to add a step to set the deployment to the production mode automatically?

When running the deployment it fails on version 2.1.0 as specific content flags such as --no-javascript or --no-css are not valid until v2.1.1. Same issue if you use the :magento_deploy_themes option as it is also not available until 2.1.1. Both documentation and code should reflect this version. Created pull request: #46