sine-io / cosbench-sineio Goto Github PK

e.msg is:
com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty; Request ID: xxx; S3 Extended Request ID: xxx; Proxy: null), S3 Extended Request ID: xxx

if this, never return true

actual result:

solution:
add other ops to runningchart.ftl and finalchart.ftl files

org.eclipse.osgi_3.18.100.v20220817-1601.jar

org.eclipse.equinox.launcher_1.6.400.v20210924-0641.jar

I'm able to run a simple test workload that creates and deletes a bucket with the s3v2 driver. When I added a "prepare" stage, this error was seen in the system.log from each driver:

2021-12-17 18:41:13,221 [ERROR] [AbstractCommandTasklet] - driver report error: HTTP 500 - Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@17dbac8: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@87e916e: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@768f57d5: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@f67f625: Unable to contact EC2 metadata service.]

A workaround I found is to start the driver like this:
export AWS_REGION=us-east-1; ./start-driver.sh

workloads like these:

try to add async s3client.

Currently it's not possible to run a mixed workload of two Range Read operations.

For example the following work stage doesn't work. It is complaining that: "cannot parse the xml configuration from , due to element "storage" occurs more than once. (parent class: com.intel.cosbench.config.Work)
location: /work/work/work/work/storage "

16MB, error is:
software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: tx0000000000000000008da-00620f1939-db6d16-zone-1643100400)

16KB, error is:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Read after end of file

xml:

<storage type="siov2" config="endpoint=http://ip:7480;accesskey=xxx;secretkey=xxx;path_style_access=true;timeout=10000000" />
<workstage name="test mwrite">
    <work name="w_1" workers="1" totalOps="1" driver="driver1">
        <operation type="mwrite" ratio="100" config="cprefix=thisistestbucket-0;containers=c(4);oprefix=w1-testmwrite-;objects=s(1,10);sizes=c(16)MiB" />
    </work>
</workstage>

wrong:

correct:

screenshot:

Stop time: add date and time, not noly time

error screenshot:

xml:

<workstage name="cleanup objects">
      <work type="cleanup" workers="5" config="cprefix=thisistestbucket-0;containers=s(1,1);oprefix=w1-testmwrite-;objects=r(1,10)" />
</workstage>

<workstage name="cleanup objects">
      <work type="cleanup" workers="5" config="cprefix=thisistestbucket-0;containers=r(1,1);oprefix=w1-testmwrite-;objects=s(1,10)" />
</workstage>

<workstage name="test dispose">
      <work type="dispose" workers="5" config="cprefix=thisistestbucket-0;containers=s(1,1)" />
</workstage>

handbook:

error e.g:

<workstage name="test write">
    <work name="w_1" workers="50" totalOps="5000000" driver="driver1">
        <operation type="write" ratio="100" config="cprefix=feiteng-0;containers=c(2);oprefix=w1-1kb-;objects=s(1,20000000);sizes=c(1)KiB" />
    </work>
</workstage>

result:

write e.g:

<workstage name="test-write">
    <work name="w_1" workers="50" totalOps="5000000" driver="driver1">
        <operation type="write" ratio="100" config="cprefix=feiteng-0;containers=c(2);oprefix=w1-1kb-;objects=s(1,20000000);sizes=c(1)KiB" />
    </work>
</workstage>

result:

doDispose method:

getList method:

TOOL changed from ncat to telnet

1. load only 1 driver
   
   

2. got an error when Op-info is 'write'
   

run-history.csv

storage's type is not sio/siov2/gdas, but use mwrite/mprepare/head/mfilewriter/restore.

mission not return an error.

References:
intel-cloud/cosbench#148
intel-cloud/cosbench#31
intel-cloud/cosbench#64

There's serious brokenness in the sequential and range iterators, when used in a "normal" workstage.
I've been testing the new mwrite mode, intending to use that to create new multipart objects rather than the r() iterator and the "prepare" stage (which doesn't at least currently support multipart).

s() is said to be thread-safe. However, I can't get that working much at all. It seems to ignore the division strategy (other than "none") for one. The r() selector sort of works, yet only with worker counts relatively low, below 10 or so. Above, that some objects are overwritten when they shouldn't and others not created at all - guessing because of race conditions. The "object" division strategy must be used unless the worker count is >= containers (then the "container" strategy is ok) - not itself a bug.

The original developers of COSbench appear to have tested primarily using the u() random iterator - and that does work for the purpose, yet there are times when I want to ensure a certain set of objects are written or deleted, within a normal workstage.

Just getting this original issue out there, since it sure could use some help. I'll next have a look at the code myself to see what might be happening, then add a comment here if I figure it out. One of the original developers says this in issue #64 and did a commit, though that didn't seem to fix it. I believe it was just not really fixed before development stopped at 0.4.2: "using atomic structure in Range selector is insufficient to avoid race condition, synchronization primitive will be added in object picker to coordinate container and object generators."

mission log is split when size is larger than 11MB.

[root@localhost mission]# ls M5DF9B8401D.log*
M5DF9B8401D.log  M5DF9B8401D.log.1  M5DF9B8401D.log.2

and workload.log only gather the newest one.

There's another fork of COSbench adding object prefetch and more importantly to me, range reads. The implementation of the latter is limited in that for each workstage the size of the range is fixed and offset into the object is random. That's good enough for me, for now.

Would you please add this functionality - or something like it, to your fork, from this one?
https://github.com/bissenbay/cosbench

I very much appreciate all the improvements you've made. Great to finally see v4 signatures (default) with the s3v2 driver, multi-part uploads, and a simple way to disable SSL cert checking!

csv files do not have this column. 🤔 😢

When benchmarking 4k s3 objects read, we can find a big difference between response time and proc time.
The code is here.
https://github.com/sine-io/cosbench-sineio/blob/main/dev/cosbench-driver/src/com/intel/cosbench/driver/operator/Reader.java#L88-L95

copyLarge(in, cout); takes too long when object size is small.Maybe you can consider adjusting the implementation of this part.

OS is Ubuntu 20.10, with netcat-openbsd 1.217-2ubuntu1. cosbench-ehualu 0.4.7.5 from the .tar.gz binaries in the tree.

The loop in cosbench-start.sh that checks for module active fails because the osgi service listener seems to need a leading CR to accept the ss command. There's also some cruft in the boot log displayed at the end, so I filtered that a bit. Proposed patch:

83c83
<                 echo -e "ss -s ACTIVE cosbench\ndisconnect\n" | $TOOL $TOOL_PARAMS 0.0.0.0 $OSGI_CONSOLE_PORT | grep $module >> /dev/null
---
>                 echo -e "\nss -s ACTIVE cosbench\ndisconnect\n" | $TOOL $TOOL_PARAMS 0.0.0.0 $OSGI_CONSOLE_PORT | grep $module >> /dev/null
114c114
< cat $BOOT_LOG
---
> cat $BOOT_LOG | fgrep -v '<>'

<workstage name="get 4kb obs">
	  <work name="w-1" workers="50" totalOps="10" driver="driver1">
        <operation type="read" ratio="100" config="cprefix=test0;containers=c(2);oprefix=w1-4kib-;objects=s(1,10)" />
      </work>
	  
	  <work name="w-2" workers="50" totalOps="10" driver="driver2">
        <operation type="read" ratio="100" config="cprefix=test0;containers=c(2);oprefix=w2-4kib-;objects=s(1,10)" />
      </work>
    </workstage>

mission never end.