Comments (10)
jamhall
commented on May 23, 2024
Hi there! What do you mean by list buckets? Is this an amazon s3 feature? Do you want a command or a URL to list the buckets? Please give me more info :)
One of the main reasons I ported it to nodejs is because fakes3 objects weren't persistent, i.e. every time you restarted the server, you lost access to the objects! :(
from s3rver.
Lochemage
commented on May 23, 2024
Yes, I'm talking about s3's ability to retrieve a listing of all buckets or object keys within a bucket.
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listBuckets-property
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjects-property
Also, I believe this is not the case any longer with fakes3. At least I am able to list and fetch objects that I put there from before a restart.
from s3rver.
jamhall
commented on May 23, 2024
This functionality definitely exists in s3rver. Please have a look in the test directory for examples :)
from s3rver.
Lochemage
commented on May 23, 2024
Hmm, then I will have to investigate further, because all of my unit tests that test bucket listing all fail with a result size of 0. It may just be that the data I get is not aws sdk compatible.
from s3rver.
jamhall
commented on May 23, 2024
What happens when you go to the bucket url directly? http://localhost:yourChosenPort/yourBucketName . That should return an Xml list of bucket objects.
from s3rver.
Lochemage
commented on May 23, 2024
Okay, after some investigating it seems like I am getting a full listing from the Contents list only if I use an empty Prefix value. Any prefix value causes the result list to come up empty.
Just to double check, I have run the unit tests on your repo and those same tests fail as well.
This may be related to me using Node.js version 4.0.0, I know there have been a lot of changes in it.
from s3rver.
Lochemage
commented on May 23, 2024
Here is the output I am getting.
S3rver Tests
√ should fetch fetch five buckets
√ should fail to create a bucket because of invalid name
√ should fail to create a bucket because name is too long
√ should fail to create a bucket because name is too short
√ should delete a bucket
√ should not fetch the deleted bucket
√ should list no objects for a bucket
√ should store a text object in a bucket
√ should store a text object with some custom metadata
√ should return a text object with some custom metadata
√ should store an image in a bucket
√ should store a gzip encoded file in bucket
√ should copy an image object into another bucket
√ should fail to copy an image object because the object does not exist
√ should fail to copy an image object because the source bucket does not exist
√ should store a large buffer in a bucket (5138ms)
√ should get an image from a bucket
√ should get image metadata from a bucket using HEAD method
√ should store a different image and update the previous image (1047ms)
√ should get an objects acl from a bucket
√ should delete an image from a bucket
√ should not find an image from a bucket
√ should fail to delete a bucket because it is not empty
√ should upload a text file to a multi directory path
√ should find a text file in a multi directory path
√ should list objects in a bucket (62ms)
1) should list objects in a bucket filtered by a prefix
2) should list objects in a bucket filtered by a marker
3) should list objects in a bucket filtered by a marker and prefix
√ should list no objects because of invalid prefix
√ should list no objects because of invalid marker
√ should generate a few thousand small objects (9300ms)
√ should return one thousand small objects (1384ms)
√ should return 500 small objects (1138ms)
√ should delete 500 small objects (2022ms)
S3rver Tests with Static Web Hosting
√ should create a site bucket
√ should upload a html page to / path
√ should upload a html page to a directory path
√ should get an index page at / path
√ should get an index page at /page/ path
√ should get a 404 error page
38 passing (22s)
3 failing
1) S3rver Tests should list objects in a bucket filtered by a prefix:
Uncaught AssertionError: expected 0 to be 4
+ expected - actual
-0
+4
at Assertion.fail (D:\Programming\Outward\s3rver\node_modules\should\lib\assertion.js:228:17)
at Assertion.prop.(anonymous function) (D:\Programming\Outward\s3rver\node_modules\should\lib\assertion.js:69:14)
at Response.<anonymous> (D:\Programming\Outward\s3rver\test\test.js:459:39)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:353:18)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:105:20)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at callNextListener (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:95:12)
at IncomingMessage.onEnd (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\event_listeners.js:207:11)
at endReadableNT (_stream_readable.js:893:12)
2) S3rver Tests should list objects in a bucket filtered by a marker:
Uncaught AssertionError: expected 0 to be 5
+ expected - actual
-0
+5
at Assertion.fail (D:\Programming\Outward\s3rver\node_modules\should\lib\assertion.js:228:17)
at Assertion.prop.(anonymous function) (D:\Programming\Outward\s3rver\node_modules\should\lib\assertion.js:69:14)
at Response.<anonymous> (D:\Programming\Outward\s3rver\test\test.js:475:39)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:353:18)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:105:20)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at callNextListener (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:95:12)
at IncomingMessage.onEnd (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\event_listeners.js:207:11)
at endReadableNT (_stream_readable.js:893:12)
3) S3rver Tests should list objects in a bucket filtered by a marker and prefix:
Uncaught AssertionError: expected 0 to be 2
+ expected - actual
-0
+2
at Assertion.fail (D:\Programming\Outward\s3rver\node_modules\should\lib\assertion.js:228:17)
at Assertion.prop.(anonymous function) (D:\Programming\Outward\s3rver\node_modules\should\lib\assertion.js:69:14)
at Response.<anonymous> (D:\Programming\Outward\s3rver\test\test.js:486:39)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:353:18)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:105:20)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:595:14)
at Request.transition (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:21:10)
at AcceptorStateMachine.runTo (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:14:12)
at D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:37:9)
at Request.<anonymous> (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\request.js:597:12)
at Request.callListeners (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at callNextListener (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\sequential_executor.js:95:12)
at IncomingMessage.onEnd (D:\Programming\Outward\s3rver\node_modules\aws-sdk\lib\event_listeners.js:207:11)
at endReadableNT (_stream_readable.js:893:12)
npm ERR! Test failed. See above for more details.
from s3rver.
Lochemage
commented on May 23, 2024
I've done a little more investigating into your s3rver code and have narrowed down the problem point.
In file-store.js, getObjects() has a line where it tests the prefix var key = file.replace(bucketPath + '/', '');. However, the list of keys retrieved by var keys = utils.walk(bucketPath); seems to format each key using backslashes. I was able to fix it by changing each reference of '/' with '' instead.
from s3rver.
jamhall
commented on May 23, 2024
ah, you're on Windows. Sorry, I haven't tested on Windows. Thanks for the head up! I'll commit a patch to solve this.
from s3rver.
Lochemage
commented on May 23, 2024
Works now!
from s3rver.
Related Issues (20)
- replace moment.js with maintaned date library HOT 2
- vhostBuckets Documentation Confusing HOT 2
- Close Stale Issues/PRs HOT 5
- Error = A parameter you provided functionality that in not implemented HOT 6
- Error: fopen(httpseek://): failed to open stream ... HOT 4
- Storage Class not being respected HOT 5
- Question: Is s3rver a s3 "private" / "public" or both? HOT 1
- key should be Key, capitalized, following the AWS SDK documentation HOT 4
- Content Range header not returned when requested range is 'bytes=0-' HOT 1
- Dependency Dashboard
- Failed to upload large file (53Mb)
- Apparent race condition deleting objects results in ENOENT HOT 1
- Custom Store
- attention on files contains url ecoding putcopy function need decodeURIComponent
- Security Issue in Dependency - CVE-2022-24434 HOT 4
- Feature request - storing/retrieving checksum metadata (x-amz-checksum) inside object attributes
- Simulate slow fetch
- fast-xml-parser vulnerable to prototype pollution SNYK-JS-FASTXMLPARSER-3325616 sonatype-2023-0998
- if i use aws java v2 sdk putobject fail HOT 2
- ListObjectsV2 MaxKeys parameter is broken
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from s3rver.