AwsS3V3Adapter::visibility throws Unable to retrieve the visibility for file at location UnableToRetrieveMetadata exception about flysystem HOT 7 CLOSED

@terrafrost that's very weird. The visibility functions are not really intended for directories though, any particular reason why you're checking the visibility of a dir?

from flysystem.

I'm more specifically trying to make it so that the folders can be moved (eg. $filesystem->move('RandomFolder', 'RandomFolder2', $config)), however, from that, I'm getting a Unable to copy file UnableToMoveFile exception. move() calls $this->copy($source, $destination, $config) which, in turn, does this:

        try {
            $visibility = $config->get(Config::OPTION_VISIBILITY);

            if ($visibility === null && $config->get(Config::OPTION_RETAIN_VISIBILITY, true)) {
                $visibility = $this->visibility($source)->visibility();
            }
        } catch (Throwable $exception) {
            throw UnableToCopyFile::fromLocationTo(
                $source,
                $destination,
                $exception
            );
        }

So that's where the error is ultimately coming from. I did $filesystem->visibility('RandomFolder') vs $filesystem->move('RandomFolder', 'RandomFolder2', $config) just because that was the depth of how far into the problem I dug.

from flysystem.

Digging into it some more I see I'm getting a Aws\S3\Exception\S3Exception with this as the message:

Error executing "GetObjectAcl" on "https://whatever.s3.us-east-2.amazonaws.com/RandomFolder?acl"; AWS HTTP error: Client error: `GET https://whatever.s3.us-east-2.amazonaws.com/RandomFolder?acl` resulted in a `404 Not Found` response: NoSuchKeyThe specified key does not exist. (truncated...) NoSuchKey (client): The specified key does not exist. - NoSuchKeyThe specified key does not exist.

The folder definitely exists on S3 (as $filesystem->directoryExists('RandomFolder') is testament to). Maybe it'd be good not to check the visibility in copy() if it's trying to copy a folder?

from flysystem.

@terrafrost can you try setting the global option retain_visibility to false?

from flysystem.

Just for clarity, it's the options that you pass into the Filesystem constructor:

$filesystem = new Filesystem($adapter, ['retain_visibility' => false]);

from flysystem.

That got me past the visibility error! Altho now it's failing here:

        try {
            $this->client->copy(
                $this->bucket,
                $this->prefixer->prefixPath($source),
                $this->bucket,
                $this->prefixer->prefixPath($destination),
                $this->visibility->visibilityToAcl($visibility ?: 'private'),
                $this->createOptionsFromConfig($config)['params']
            );
        } catch (Throwable $exception) {
            throw UnableToCopyFile::fromLocationTo($source, $destination, $exception);
        }

If I do exit($exception->getMessage()) I get this back:

Error executing "HeadObject" on "https://whatever.s3.us-east-2.amazonaws.com/RandomFolder"; AWS HTTP error: Client error: `HEAD https://whatever.s3.us-east-2.amazonaws.com/RandomFolder` resulted in a `404 Not Found` response NotFound (client): 404 Not Found (Request-ID: ...) -

It's just weird because $filesystem->directoryExists('RandomFolder') returns true. If I do $filesystem->directoryExists('FakeFolder') it returns false.

My current code:

$client = new \Aws\S3\S3Client([
    'version' => 'latest',
    'region' => 'us-east-2',
    'credentials' => [
        'key' => env('AWS_ACCESS_KEY_ID'),
        'secret' => env('AWS_SECRET_ACCESS_KEY'),
    ],
]);

// The internal adapter
$adapter = new \League\Flysystem\AwsS3V3\AwsS3V3Adapter(
    // S3Client
    $client,
    // Bucket name
    'mybucket'
);

//$filesystem = $adapter;
$filesystem = new \League\Flysystem\Filesystem($adapter, ['retain_visibility' => false]);

//$config = new \League\Flysystem\Config();
$config = [];

//echo $filesystem->directoryExists('FakeFolder') ? 't' : 'f'; exit;

try {
    $filesystem->move('RandomFolder', 'RandomFolder2', $config);
} catch (\Exception $e) {
    echo $e->getMessage(); exit;
}

from flysystem.

So I was able to isolate this down to an issue with the S3 PHP SDK:

$client = new \Aws\S3\S3Client([
    'version' => 'latest',
    'region' => 'us-east-2',
    'credentials' => [
        'key' => env('AWS_ACCESS_KEY_ID'),
        'secret' => env('AWS_SECRET_ACCESS_KEY'),
    ],
]);

$result = $client->headObject([
	'Bucket' => 'mybucket',
	'Key' => 'NewFolder' // an empty folder
]);

echo '<pre>';
var_dump($result);

That reproduced the issue so I created an issue with AWS to get to the bottom of the issue.

Here was their response:

As I understand from the case notes, you have a PHP code where you are performing a head object API call on a folder that exists but you are witnessing 404 object not found error. Thus, you want us to investigate the same. Please do correct me If I have misunderstood your query.

To begin with, I would like to elaborate on the nature of prefixes/folders. In S3, buckets and objects are the primary resources. S3 has a flat structure instead of a hierarchy like you would see in a file system. However, for the sake of organizational simplicity, the S3 console supports the folder concept as a means of grouping objects.

Now, when you create a folder from S3 console, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named 'newfolder' in your bucket, the Amazon S3 console creates a 0-byte object with the key newfolder/. The console creates this object to support the idea of folders as informed in the below attached document. Hence, here newfolder has its own existence as a 0 byte object.

[+] https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html

Based on that I decided to try this instead:

$result = $client->headObject([
	'Bucket' => 'mybucket',
	'Key' => 'NewFolder/' // an empty folder
]);

And that worked!

from flysystem.