craftcms / aws-s3 Goto Github PK

Used the {id} variable to place the uploaded assets into a folder titled with the entry ID, but it still uploads to the root of the parent folder.

This may already be in your queue since its a new release but it looks like one of the Volume changes with beta 24 is creating an issue here. I'm running 1.0.5 on the plugin. Thanks.

Class craft\awss3\Volume contains 14 abstract methods and must therefore be declared abstract or implement the remaining methods (craft\base\VolumeInterface::getFileList, craft\base\VolumeInterface::getFileMetadata, craft\base\VolumeInterface::createFileByStream, ...)

Current Behaviour
After installing and following the plugin instructions on the README.md, setting up a new volume via the admin forces users to re-enter API Key and Secret (even if these are set in the .env).

Ideal Behaviour
If the S3_API_KEY and S3_SECRET are set in .env these should overwrite any settings in the admin panel, otherwise what is the point of storing these values in the .env

If a CloudFront id is set when trying to delete a file it shows this message:
"Failed to invalidate the CDN path for myimage.jpg"

IMO this header is called Cache-Control not CacheControl
https://github.com/craftcms/aws-s3/blob/develop/src/Volume.php#L241

same here:
https://github.com/craftcms/google-cloud/blob/master/src/Volume.php#L205

Hey Andris!

I've been running with the dev-craft-3.1 branch without issue for the last few weeks, but just FYI, the recent commit you made (e0636f1) is causing a migration error for me now..

After running a composer update and getting the latest commit..

  - Updating craftcms/cms (3.1.6.1 => 3.1.7): Loading from cache
  - Updating craftcms/server-check (1.1.6 => 1.1.7): Loading from cache
  - Updating league/flysystem-aws-s3-v3 (1.0.21 => 1.0.22): Loading from cache
  - Updating craftcms/aws-s3 dev-craft-3.1 (0c2b23b => 3cb4190):  Checking out 3cb4190265

I get this error running the migration...

Exception: Undefined variable: settings (/Users/michael.roper/Sites/website-v3/vendor/craftcms/aws-s3/src/migrations/m180929_165000_remove_storageclass_setting.php:30)
#0 /Users/michael.roper/Sites/website-v3/vendor/craftcms/aws-s3/src/migrations/m180929_165000_remove_storageclass_setting.php(30): yii\base\ErrorHandler->handleError(8, 'Undefined varia...', '/Users/michael....', 30, Array)
#1 /Users/michael.roper/Sites/website-v3/vendor/craftcms/cms/src/db/Migration.php(56): craft\awss3\migrations\m180929_165000_remove_storageclass_setting->safeUp()
#2 /Users/michael.roper/Sites/website-v3/vendor/craftcms/cms/src/db/MigrationManager.php(243): craft\db\Migration->up(true)
#3 /Users/michael.roper/Sites/website-v3/vendor/craftcms/cms/src/db/MigrationManager.php(163): craft\db\MigrationManager->migrateUp(Object(craft\awss3\migrations\m180929_165000_remove_storageclass_setting))
#4 /Users/michael.roper/Sites/website-v3/vendor/craftcms/cms/src/services/Updates.php(223): craft\db\MigrationManager->up()
#5 /Users/michael.roper/Sites/website-v3/vendor/craftcms/cms/src/console/controllers/MigrateController.php(252): craft\services\Updates->runMigrations(Array)

I'm using Environment variables for my secret settings, and have non-secret settings in config/volumes.php, if that makes a difference.

I'm struggling to find any information on this alert error (Class 'Aws\Credentials\Credentials' not found) I'm getting when trying to "refresh" the Bucket selection dropdown(no buckets are showing). All credentials for the AWS connection have been tested and they work on local machine. I'm seeing this error on the offsite development server install of craft v3.1.24. Thanks for your help.

I wrote a plugin that creates Entries that contain image assets. After around 1k entires, it dies with error:

Exception 'Aws\S3\Exception\S3Exception' with message 'Error executing "ListObjects" on "***"; AWS HTTP error: Client error: 'GET ***' resulted in a '400 Bad Request' response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message (truncated...)
 ExpiredToken (client): The provided token has expired. - <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message>

I added a try/catch in order to retry the save after a sleep, but it was not helpful. Is there a better way to handle this or is this a bug?

INFO:
CraftCMS: 3.0
PHP: 7.0

I have a single S3 bucket for all my sites, with an assets directory that contains subdirectories for each site, so it looks like this:

• https://mybucket.s3.amazonaws.com/assets/site1/asset.png
• https://mybucket.s3.amazonaws.com/assets/site2/asset.png
• etc.

I set up CloudFront on one of the sites, with the following origin settings:

As you can see, I have set an "origin path" so that my URLs look like this:

https://xxxxxxxxx.cloudfront.net/asset.png

instead of this:

https://xxxxxxxxx.cloudfront.net/assets/site1/asset.png

It works, but the Craft CMS's S3 plugin doesn't seem to support such a configuration. Here are my settings:

The problem is that I need to set "Subfolder" to the proper path relative to the bucket because that's where I need my assets for that site to be uploaded, but it results in Craft thinking my assets have URLs like this:

https://xxxxxxxxx.cloudfront.net/assets/site1/asset.png

instead of this:

https://xxxxxxxxx.cloudfront.net/asset.png

Would it be possible to add an extra setting called "Include subfolder to public URLs" or something like that? It would be enabled by default of course because I realize my use case is a minority, but I still think it should be supported.

Thanks!

I'd love to be able to set the CF Distribution Id in the volumes php along with the s3 secrets, etc.

I tried adding the cfDistributionId key in my volume config but it is not showing up in the craft admin backend.

We have noticed that when a volume has a URL, the assets in the S3 bucket are automatically set to "public-read" on upload. But this is not wanted in all cases. For example, if you have a restricted Cloudfront distribution. Then the objects in the bucket should not be public and the bucket policy allows access to the objects for the Cloudfront distribution.

It would be helpful if you could overwrite this dependency. A parameter (preferably only in volumes.php) that allows you to set the ACL yourself?

Someone has a pretty good answer for the correct IAM settings for an Assets S3 bucket on the Stack Exchange:
https://craftcms.stackexchange.com/questions/13815/correct-iam-settings-for-amazon-s3-bucket

Might be worth documenting these settings with this plugin?

When installing aws-s3 plugin 1.1.0 via Craft installer with Project Config and an existing project.yaml I get the following error.

    > installing aws-s3 ... Exception: Invalid argument supplied for foreach() (/app/vendor/craftcms/aws-s3/src/migrations/Install.php:60)
#0 /app/vendor/craftcms/aws-s3/src/migrations/Install.php(60): yii\base\ErrorHandler->handleError(2, 'Invalid argumen...', '/app/vendor/cra...', 60, Array)
#1 /app/vendor/craftcms/aws-s3/src/migrations/Install.php(34): craft\awss3\migrations\Install->_convertVolumes()
#2 /app/vendor/craftcms/cms/src/db/Migration.php(56): craft\awss3\migrations\Install->safeUp()
#3 /app/vendor/craftcms/cms/src/db/MigrationManager.php(243): craft\db\Migration->up(true)
#4 /app/vendor/craftcms/cms/src/base/Plugin.php(152): craft\db\MigrationManager->migrateUp(Object(craft\awss3\migrations\Install))
#5 /app/vendor/craftcms/cms/src/services/Plugins.php(513): craft\base\Plugin->install()
#6 /app/vendor/craftcms/cms/src/migrations/Install.php(1112): craft\services\Plugins->installPlugin('aws-s3')
#7 /app/vendor/craftcms/cms/src/migrations/Install.php(1006): craft\migrations\Install->_installPlugins()
#8 /app/vendor/craftcms/cms/src/migrations/Install.php(70): craft\migrations\Install->insertDefaultData()
#9 /app/vendor/craftcms/cms/src/db/Migration.php(56): craft\migrations\Install->safeUp()
#10 /app/vendor/craftcms/cms/src/db/MigrationManager.php(243): craft\db\Migration->up(true)
#11 /app/vendor/craftcms/cms/src/console/controllers/InstallController.php(161): craft\db\MigrationManager->migrateUp(Object(craft\migrations\Install))
#12 [internal function]: craft\console\controllers\InstallController->actionCraft()
#13 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#14 /app/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#15 /app/vendor/yiisoft/yii2/console/Controller.php(148): yii\base\Controller->runAction('', Array)
#16 /app/vendor/yiisoft/yii2/base/Module.php(528): yii\console\Controller->runAction('', Array)
#17 /app/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('install', Array)
#18 /app/vendor/craftcms/cms/src/console/Application.php(93): yii\console\Application->runAction('install', Array)
#19 /app/vendor/yiisoft/yii2/console/Application.php(147): craft\console\Application->runAction('install', Array)
#20 /app/vendor/yiisoft/yii2/base/Application.php(386): yii\console\Application->handleRequest(Object(craft\console\Request))
#21 /app/craft(23): yii\base\Application->run()
#22 {main}

Hi gang. Hitting a Twig bug with this version after migrating from Craft 2. After performing an upgrade, then installing the S3 plugin, when I go to any of my S3 Asset volume's settings pages in the CP, I get the following Twig error:

Twig Runtime Error – Twig_Error_Runtime
Key "1" for array with keys "0" does not exist.

    1. in /vendor/craftcms/aws-s3/src/templates/volumeSettings.html at line 125

                                  id: 'expiresAmount',
                    value: expires[0],
                    size: 2,
                    class: 's3-expires-amount'
                }) }}
            </div>
            {{ forms.select({
                id: 'expiresPeriod',
                options: periods,
                value: expires[1],
                class: 's3-expires-period'
            }) }}
        </div>
        {{ forms.hidden({
            name: "expires",
            value: volume.expires,
            class: "expires-combined"
        }) }}
    {% endset %}

Steps to reproduce

1. Deploy updated files/templates/configs for Craft 3
2. Run composer install
3. Perform Craft 3 upgrade (everything works)
4. Enable S3 plugin from Plugins page in CP
5. Click on any S3 volume in the Assets section of the CP (non-S3 volumes load fine)

Versions

Craft: 2.7.5 - 3.1.14
S3: 1.2.0

I have an S3 bucket set up with a folder structure like this:

root
— craft-assets
— — blog

I'm able to set up a volume using "craft-assets" as the "path to subfolder," and when I'm viewing that volume I get "blog" as a subfolder under a dropdown. However, I'd like to connect directly to the blog folder. I tried creating a similar volume with "craft-assets/blog" as the path to subfolder, but I always get a 403 Forbidden error when I try to upload to it.

Can the path to subfolder only be one folder deep?

I try to understand what's happening in the Install migration.
Is volumes.type = 'craft\volumes\AwsS3' a legacy thing? I can't find any reference to that class.

https://github.com/craftcms/aws-s3/blob/master/src/migrations/Install.php#L62

Similar things happen in Rackspace and GoogleCloud plugin.

Sometimes we need to use an API key locally to authenticate with a bucket; however, upon deploying to our AWS servers, we prefer to use an IAM EC2 policy to authorize the bucket; however, if I set the AWS Key ID setting or AWS Secret setting to an environment variable such as $AWS_KEY_ID, which is set locally but not on our staging/production environment, the volume will still try to authenticate with the key.

Ideally, if the environment variable doesn't exist, it should not try to authenticate with a key but rely on the hosts configuration.

By default, it generates a base URL using the virtual-hosted–style url:: http://mybucketname.s3.amazonaws.com/

The default should be https.

Furthermore, by defaulting to the path-based URL, SSL will work no matter what the bucket name is (even if it contains a period "."): https://s3.us-east-1.amazonaws.com/dev.mybucketname/

https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro

I could not get Craft 3 to index an S3 bucket, despite the same credentials working in Craft 2.

The culprit ended up being thousands S3 log files causing a 500 error, with PHP running out of memory. I'm not sure if there is a way for the Craft S3 adapter to ignore and short-circuit past gz and txt files, but wanted to open an issue in case there is a simple fix.

If anybody else has this problem, the fix was to enable a bucket lifecycle rule to retroactively nuke the 500K logfiles, and to disable logging going forward.

Hi,

I have ran into this issue updating a C2 site to craft 3.

Craft version: latest stable
PHP version: 7.2

Here is the volumes table in the db:

Thanks,
Mark

Using Craft 3.1.7, project config enabled and dev-craft-3.1 version of plugin following error occurs when accessing Control panel:
Setting unknown property: craft\awss3\Volume::storageClass

In loadBucketList should it default to protocol-relative rather than http? Or possibly include a preferred protocol env config?

Hello!

Running into an issue that has been a little difficult to track down. The attached error message appears whenever I try to upload a file via the Admin. I have tried using admin level API Key/Secret so I do not believe it's an IAM permissions issue with S3 as this still results in the error.

##Admin Configuration

# The AWS API key with read/write access to S3
S3_API_KEY="xxx"

# The AWS API key secret
S3_SECRET="xxx"

# The name of the S3 bucket
S3_BUCKET="assets.thewellresource.org.au"

# The region the S3 bucket is in
S3_REGION="ap-southeast-2"

<?php

return [
    's3Uploads' => [
        'hasUrls' => true,
        'url' => 'https://'.getenv('S3_BUCKET').'.s3.amazonaws.com/',
        'keyId' => getenv('S3_API_KEY'),
        'secret' => getenv('S3_SECRET'),
        'bucket' => getenv('S3_BUCKET'),
        'region' => getenv('S3_REGION'),
    ],
];

It’s recommended to not use volume.php for configuration any longer, and environmental settings should be used instead. craftcms/cms#3655

With this change it’s not possible to set bucket and region to an environment var any longer.

Hello! Saw that—from #46—you can set the path prefix to allow Craft to invalidate CloudFront assets. Super neat!

Right now I have the distribution ID configured, but it looks like that isn't enough to get Craft to invalidate relevant assets within CloudFront.

Could you provide an example of what this might look like?

After my initial migration, I log into the CP and this plugin is in an uninstalled state (as it is not present in C2 I guess). When I then attempt to install, it gives the error and the CP is then unreachable:

yii\base\UnknownPropertyException: Setting unknown property: craft\awss3\Volume::urlPrefix in /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/base/Component.php:209
Stack trace:
#0 /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/BaseYii.php(546): yii\base\Component->__set('urlPrefix', 'https://s3-ap-s...')
#1 /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/base/BaseObject.php(107): yii\BaseYii::configure(Object(craft\awss3\Volume), Array)
#2 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/helpers/Component.php(80): yii\base\BaseObject->__construct(Array)
#3 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/services/Volumes.php(499): craft\helpers\Component::createComponent(Array, 'craft\\base\\Volu...')
#4 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/services/Volumes.php(222): craft\services\Volumes->createVolume(Array)
#5 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/services/Volumes.php(236): craft\services\Volumes->getAllVolumes()
#6 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/models/VolumeFolder.php(93): craft\services\Volumes->getVolumeById(4)
#7 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/fields/Assets.php(805): craft\models\VolumeFolder->getVolume()
#8 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/fields/Assets.php(124): craft\fields\Assets->_folderSourceToVolumeSource('folder:df4ad037...')
#9 /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/base/BaseObject.php(109): craft\fields\Assets->init()
#10 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/fields/BaseRelationField.php(173): yii\base\BaseObject->__construct(Array)
#11 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/helpers/Component.php(80): craft\fields\BaseRelationField->__construct(Array)
#12 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/services/Fields.php(548): craft\helpers\Component::createComponent(Array, 'craft\\base\\Fiel...')
#13 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/services/Fields.php(573): craft\services\Fields->createField(Array)
#14 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/elements/db/ElementQuery.php(1536): craft\services\Fields->getAllFields()
#15 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/elements/db/ElementQuery.php(1073): craft\elements\db\ElementQuery->customFields()
#16 /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/db/QueryBuilder.php(227): craft\elements\db\ElementQuery->prepare(Object(craft\db\mysql\QueryBuilder))
#17 /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/db/Query.php(146): yii\db\QueryBuilder->build(Object(craft\elements\db\UserQuery))
#18 /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/db/Query.php(274): yii\db\Query->createCommand(Object(craft\db\Connection))
#19 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/db/Query.php(177): yii\db\Query->one(NULL)
#20 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/elements/db/ElementQuery.php(1215): craft\db\Query->one(NULL)
#21 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/elements/User.php(367): craft\elements\db\ElementQuery->one()
#22 /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/web/User.php(690): craft\elements\User::findIdentity('1')
#23 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/web/User.php(474): yii\web\User->renewAuthStatus()
#24 /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/web/User.php(192): craft\web\User->renewAuthStatus()
#25 /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/web/User.php(360): yii\web\User->getIdentity()
#26 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/web/User.php(189): yii\web\User->getIsGuest()
#27 /var/www/vhosts/c3-dev/vendor/craftcms/cms/src/web/Application.php(259): craft\web\User->getIsGuest()
#28 /var/www/vhosts/c3-dev/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#29 /var/www/vhosts/c3-dev/web/index.php(21): yii\base\Application->run()
#30 {main}

This occurs even though I have my settings for the volumes in .env and config/volumes.php

I'm loading the S3 settings via a volumes.php config with references to ENV variables form the .ENV file. When adding this S3 asset source to Craft 3.1 the AWS credentials are being stored in the project.yaml file. This also puts them into version control and is a potential security issue.
I know this is related to issue #35, but is there a way to prevent them from being added to the yaml?

Hi.

Just installed craftcms/aws-s3 with composer on my local MAMP stack, when finish the installation i get a few migration errors, "Unknown Property: Setting unknown property: craft\services\ProjectConfig::muteEvents"

Looking at the migrations, it seems to be this line that's responsible:

# m180929_165000_remove_storageclass_setting.php, line 23
$projectConfig->muteEvents = true;

Full error:

Unknown Property: Setting unknown property: craft\services\ProjectConfig::muteEvents

Migration: craft\awss3\migrations\m180929_165000_remove_storageclass_setting

Output:

Exception: Setting unknown property: craft\services\ProjectConfig::muteEvents (/Users/kristoffer/Documents/dev/craft3orango/vendor/yiisoft/yii2/base/Component.php:209)
#0 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/aws-s3/src/migrations/m180929_165000_remove_storageclass_setting.php(23): yii\base\Component->__set('muteEvents', true)
#1 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/cms/src/db/Migration.php(56): craft\awss3\migrations\m180929_165000_remove_storageclass_setting->safeUp()
#2 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/cms/src/db/MigrationManager.php(243): craft\db\Migration->up(true)
#3 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/cms/src/db/MigrationManager.php(163): craft\db\MigrationManager->migrateUp(Object(craft\awss3\migrations\m180929_165000_remove_storageclass_setting))
#4 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/cms/src/services/Updates.php(223): craft\db\MigrationManager->up()
#5 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/cms/src/controllers/BaseUpdaterController.php(445): craft\services\Updates->runMigrations(Array)
#6 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/cms/src/controllers/UpdaterController.php(208): craft\controllers\BaseUpdaterController->runMigrations(Array, 'restore-db')
#7 [internal function]: craft\controllers\UpdaterController->actionMigrate()
#8 /Users/kristoffer/Documents/dev/craft3orango/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#9 /Users/kristoffer/Documents/dev/craft3orango/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#10 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/cms/src/web/Controller.php(109): yii\base\Controller->runAction('migrate', Array)
#11 /Users/kristoffer/Documents/dev/craft3orango/vendor/yiisoft/yii2/base/Module.php(528): craft\web\Controller->runAction('migrate', Array)
#12 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/cms/src/web/Application.php(297): yii\base\Module->runAction('updater/migrate', Array)
#13 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/cms/src/web/Application.php(683): craft\web\Application->runAction('updater/migrate')
#14 /Users/kristoffer/Documents/dev/craft3orango/vendor/craftcms/cms/src/web/Application.php(243): craft\web\Application->_processUpdateLogic(Object(craft\web\Request))
#15 /Users/kristoffer/Documents/dev/craft3orango/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#16 /Users/kristoffer/Documents/dev/craft3orango/web/index.php(21): yii\base\Application->run()
#17 {main}

Craft version: 3.1.0
AWS S3-version: 1.1.1

Am I missing something?

Thanks.

Error message (shown below) is trying to use the following host name:
'brigade-website.s3.s3-us-west-2.amazonaws.com'

config/volumes.php
'url' => 'url' => 'https://'.getenv('S3_REGION').'.amazonaws.com/'.getenv('S3_BUCKET').'/',

/admin/utilities/php-info
S3_BUCKET= brigade-website
S3_REGION= s3-us-west-2

When uploading an image the following error is given:
Upload failed. The error message was: “Error executing "ListObjects" on "https://brigade-website.s3.s3-us-west-2.amazonaws.com/?prefix=craftcms_assets%2Fblocks%2Fimage-copy%2F&delimiter=%2F&encoding-type=url"; AWS HTTP error: cURL error 51: SSL: no alternative certificate subject name matches target host name 'brigade-website.s3.s3-us-west-2.amazonaws.com' (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)”

In the AS3 settings ClodFront distribution ID label is misspelled.

The "Bucket" dropdown does not work when using a Heroku add-on such as Bucketeer as we don't have sufficient privileges to list all buckets in the root.
Having the option to manually type in a bucket name in the UI would resolve this.
Currently we have to hack it by setting up a volume referencing our own S3 bucket and then updating the corresponding volumes row in the database.

I've been trying to use Minio with CraftCMS, but I keep getting the following error despite having configured Minio URL as url in the volume's configuration.

Here is my volume's configuration for my local Minio instance (Credentials are temporary and local 😛 ):

Description

I have useProjectConfigFile set to true.

I updated Craft CMS from 3.1.5 to 3.1.7 and the Amazon S3 plugin from 1.0.8 to 1.1.2 locally, and then pushed all that to the production server, which has allowAdminChanges set to false. I then went to the CP on the production site, and sure enough there were some migrations to run, so I did. But then I got this error message: "One of Amazon S3’s migrations failed."

Not Supported: Changes to the project config are not possible while in read-only mode.

Migration: craft\awss3\migrations\m180929_165000_remove_storageclass_setting

Output:

> update in {{%volumes}} ... done (time: 0.004s)
Exception: Changes to the project config are not possible while in read-only mode. (/home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/services/ProjectConfig.php:330)
#0 /home/colloquesims/colloquesims.com/vendor/craftcms/aws-s3/src/migrations/m180929_165000_remove_storageclass_setting.php(35): craft\services\ProjectConfig->set('volumes.f394388...', Array)
#1 /home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/db/Migration.php(56): craft\awss3\migrations\m180929_165000_remove_storageclass_setting->safeUp()
#2 /home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/db/MigrationManager.php(243): craft\db\Migration->up(true)
#3 /home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/db/MigrationManager.php(163): craft\db\MigrationManager->migrateUp(Object(craft\awss3\migrations\m180929_165000_remove_storageclass_setting))
#4 /home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/services/Updates.php(223): craft\db\MigrationManager->up()
#5 /home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/controllers/BaseUpdaterController.php(521): craft\services\Updates->runMigrations(Array)
#6 /home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/controllers/UpdaterController.php(209): craft\controllers\BaseUpdaterController->runMigrations(Array, 'restore-db')
#7 [internal function]: craft\controllers\UpdaterController->actionMigrate()
#8 /home/colloquesims/colloquesims.com/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#9 /home/colloquesims/colloquesims.com/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#10 /home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/web/Controller.php(109): yii\base\Controller->runAction('migrate', Array)
#11 /home/colloquesims/colloquesims.com/vendor/yiisoft/yii2/base/Module.php(528): craft\web\Controller->runAction('migrate', Array)
#12 /home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/web/Application.php(297): yii\base\Module->runAction('updater/migrate', Array)
#13 /home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/web/Application.php(683): craft\web\Application->runAction('updater/migrate')
#14 /home/colloquesims/colloquesims.com/vendor/craftcms/cms/src/web/Application.php(243): craft\web\Application->_processUpdateLogic(Object(craft\web\Request))
#15 /home/colloquesims/colloquesims.com/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#16 /home/colloquesims/colloquesims.com/public/index.php(21): yii\base\Application->run()
#17 {main}

I had to temporarily set allowAdminChanges to true in order to run the migration successfully, and then set it back to false. Is this something that Craft could do automatically?

Additional info

• Craft version: 3.1.7
• PHP version: 7.1.26
• Database driver & version: MySQL 5.5.5

When reindexing it takes a long time and most of the time seems to be due to Craft downloading all assets to get their size (both file size and image size).

File size should be directly accessible from S3 as is, but image size could be stored as a custom metadata and efficiently (hopefully) remove the need to download files just to reindex them.

As far as i can tell there is no hook or event short of extending the volume and overriding the addFileMetadataToConfig method that will allow me to remove the must-revalidate header that is being added to all my S3 uploaded files.

When i run a Chrome audit on my site I am getting a "Uses inefficient cache policy" due to the must-revalidate header. Please make this option (whether to include must-revalidate and ideally any other headers) configurable in a setting somewhere.

The POST data types[craft\awss3\Volume][expires] is empty while saving the Asset Volume.

It would be great if this plugin could also support DigitalOcean Spaces. Currently it's not possible to configure the bucket.

This is a feature request.

When you have the Access Key ID and Secret Access Key set to an env variable but that env variable is not defined it would be great if that defaulted to the EC2 IAM.

Purpose:
This would allow you to set those to an env that can be used when working locally but when deploying to EC2 servers it would default to the EC2 IAM when those envs are not set. This way if you are pulling down the database from the server you don't have to change those for each volume in the CMS.

Is it possible to have the .env file to also support the name of the subfolders?

Hi,

in field settings creating custom path using {slug} or even simple folder doesn't creates that folder in S3, the image gets uploaded just fine but in the configured bucket. This could be potentially be permissions issue set in S3 policy, but either way, would like to see if someone is using this and can test on their side.

Version: Craft CMS 3.0.3.1

This might be a an issue for https://github.com/craftcms/cms but after I run composer require craftcms/aws-s3, I'm unable to activate the plugin in the control panel. The AWS plugin itself is marked with a check but when I open the plugin to activate it nothing happens.

When I upload animated gif using webUI, the uploaded file does not show animation.

When I use a third party tool to upload the same file, animation works.

Here are the examples of the same file, uploaded using Craft and other tool respectively.
https://d1ybdyvhkcqsml.cloudfront.net/static_img/webimg/screnshot52.gif
https://d1ybdyvhkcqsml.cloudfront.net/static_img/webimg/screnshot54.gif

I am at loss of words on how this can happen. Craft plugin and the other tool I am using must be utilizing the same AWS API calls. How is it possible is beyond my understanding

When I want to replace an asset through the 'upload files button' in the S3 volume (with an asset of the same filename),

I get a 'doesn't have the original file extension' message (even though the new asset has).

Then a new asset with a new filename is created.

I'm working on upgrading our site from Craft2 to Craft3, and have been experiencing trouble getting S3 buckets to work, which we use heavily.

Upgrading Craft and installing the aws-s3 plugin all went fine, and our settings were migrated, or so it seemed. The site was erroring at first (I didn't think to save that, unfortunately), which was remedied by viewing and re-saving all of the s3 asset volumes.

However, now the site doesn't actually display any of the images, and trying to preview (or download) it spinwaits for a long time and then displays "an unknown error occurred". No logs seem to be emitted in storage/logs.

I tried to make a new asset volume to recreate one, but it doesn't work properly, either (no files indexed). I'm able to list all my buckets, but after creating it, there are no files indexed - and I've tried both with and without public URLs, since some of the other issues implied there might be something to that.

This led me to think that maybe it was an issue with indexing, and when I try to refresh the asset index in the CP manually, I'm rewarded with an interesting error:

Error executing "ListObjects" on "https://s3.us.amazonaws.com/assets.clever.com?prefix=website%2Ffiles%2F&encoding-type=url"; AWS HTTP error: cURL error 6: Could not resolve host: s3.us.amazonaws.com (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

Where website_files is the subfolder under the assets.clever.com s3 bucket.
Strangely, I can find no references to the s3.us.amazonaws.com url anywhere in the code, so I'm not clear where it's coming from, or how best to fix it.

Any suggestions would be greatly appreciated!

When I try to upload a CSS using craft webUI using Amazon S3 plugin, I get an error that file type not allowed

If I use a third party tool to upload a css file on my S3 bucket, craft refuses to index that file (manual indexing)

Why? This was never an issue on Craft 2

BTW, this might be an issue with other types of files too (like js). I have not checked yet.

If you already know the bucket name, there is no need to add the ListAllMyBuckets setting to your IAM User permissions. This method is just a convenience for the UI.

It would be nice to have a plain text option to specify the bucket name if it's already known so bucket permissions can be more restrictive. In agency situations, the ListAllMyBuckets permission exposes all other bucket names which an agency may not want a client to see.

Once active - trying to create a volumn /settings/assets/volumes/new or view existing /settings/assets/volumes/1 throws

Internal Server Error
An error occurred when rendering a template.

stack track on single call - http://pastebin.com/n666PVuQ

It would be great if there was the option to store the AWS ID, secret and bucket name in a config file (so it could be made with environment variables) to make this plugin more 12 factor-y.

craftcms/cms#3219 (comment)

The goal is to be able to use env vars for things that would change per-env…which is pretty much everything (bucket, region, keyId, secret, url)

Hi,
why don't I see m3u8 Files in Data-Manager?