zrlio / jnvmf Goto Github PK
View Code? Open in Web Editor NEWA NVMf library for Java
License: Apache License 2.0
A NVMf library for Java
License: Apache License 2.0
I like one simple read/write application to start integrate with my application. can you write one for us?
benchmark is a little complex, and cannot very easily figure out the things. can we wrapper just one nvmehost class that provide read and write and trim API?
Hello,
I have already build jNVMf and run benchmark success on HW. It is important for me to test the Latency and performance . Followed your tips in https://github.com/zrlio/jNVMf/issues/10. The fio command observed:
fio -filename=/dev/ nvme7n1p1 -direct=1 -rw read -m SEQUENTIAL -ioengine=libaio -qd 16 -s 8192 -numjobs=1 -i 3 -n 10 -group_reporting -name=dum
It remind me that engine not loaded, but I really install the libaio1-0.3.109-17.15.x86_64 and libaio-devel-0.3.109-17.15.x86_64 .
Would you tell me whether the fio command is right (if no, please correct me) or any problem about running benchmark.
t[ns] 3000010853, #ops 146204, iops 48734, mean[ns] 20308, tp[MB/s] 190.37, min[ns] 18221, max[ns] 1837523, p1.0% 18928, p10.0% 19134, p20.0% 19226, p30.0% 19278, p40.0% 19315, p50.0% 19351, p75.0% 19509, p90.0% 19762, p95.0% 21468, p98.0% 23864, p99.0% 34480, p99.9% 139207, p99.99% 336263
t[ns] 3000018876, #ops 149431, iops 49810, mean[ns] 19951, tp[MB/s] 194.57, min[ns] 18176, max[ns] 343425, p1.0% 18960, p10.0% 19100, p20.0% 19191, p30.0% 19251, p40.0% 19292, p50.0% 19327, p75.0% 19459, p90.0% 19624, p95.0% 19898, p98.0% 22009, p99.0% 26741, p99.9% 119892, p99.99% 192324
t[ns] 3000012927, #ops 148848, iops 49615, mean[ns] 20033, tp[MB/s] 193.81, min[ns] 18208, max[ns] 882391, p1.0% 18991, p10.0% 19160, p20.0% 19233, p30.0% 19280, p40.0% 19314, p50.0% 19343, p75.0% 19477, p90.0% 19624, p95.0% 19913, p98.0% 22116, p99.0% 27807, p99.9% 119912, p99.99% 191930
t[ns] 3000009064, #ops 148966, iops 49655, mean[ns] 20013, tp[MB/s] 193.96, min[ns] 18193, max[ns] 349483, p1.0% 18964, p10.0% 19118, p20.0% 19209, p30.0% 19265, p40.0% 19303, p50.0% 19335, p75.0% 19463, p90.0% 19610, p95.0% 19860, p98.0% 22077, p99.0% 27729, p99.9% 119893, p99.99% 191591
t[ns] 3000008017, #ops 149035, iops 49678, mean[ns] 20010, tp[MB/s] 194.05, min[ns] 18208, max[ns] 796195, p1.0% 18981, p10.0% 19132, p20.0% 19219, p30.0% 19272, p40.0% 19308, p50.0% 19340, p75.0% 19469, p90.0% 19615, p95.0% 19888, p98.0% 21939, p99.0% 27598, p99.9% 119802, p99.99% 192175
t[ns] 3000002473, #ops 148447, iops 49482, mean[ns] 20086, tp[MB/s] 193.29, min[ns] 18113, max[ns] 2266093, p1.0% 18979, p10.0% 19147, p20.0% 19232, p30.0% 19281, p40.0% 19317, p50.0% 19349, p75.0% 19499, p90.0% 19660, p95.0% 20357, p98.0% 22456, p99.0% 28993, p99.9% 119761, p99.99% 191566
t[ns] 3000004605, #ops 149284, iops 49761, mean[ns] 19963, tp[MB/s] 194.38, min[ns] 18193, max[ns] 811031, p1.0% 18944, p10.0% 19083, p20.0% 19179, p30.0% 19239, p40.0% 19280, p50.0% 19315, p75.0% 19443, p90.0% 19602, p95.0% 20139, p98.0% 22321, p99.0% 27661, p99.9% 119587, p99.99% 191189
t[ns] 3000017266, #ops 149155, iops 49718, mean[ns] 19992, tp[MB/s] 194.21, min[ns] 18180, max[ns] 791290, p1.0% 18971, p10.0% 19125, p20.0% 19214, p30.0% 19269, p40.0% 19307, p50.0% 19340, p75.0% 19473, p90.0% 19620, p95.0% 19930, p98.0% 22137, p99.0% 27667, p99.9% 119476, p99.99% 190673
t[ns] 3000015082, #ops 149280, iops 49759, mean[ns] 19979, tp[MB/s] 194.37, min[ns] 18069, max[ns] 417457, p1.0% 18975, p10.0% 19121, p20.0% 19212, p30.0% 19267, p40.0% 19305, p50.0% 19339, p75.0% 19472, p90.0% 19618, p95.0% 19920, p98.0% 21942, p99.0% 27511, p99.9% 119657, p99.99% 190424
t[ns] 3000017630, #ops 149112, iops 49703, mean[ns] 19999, tp[MB/s] 194.15, min[ns] 18214, max[ns] 801005, p1.0% 18978, p10.0% 19120, p20.0% 19210, p30.0% 19264, p40.0% 19301, p50.0% 19335, p75.0% 19469, p90.0% 19618, p95.0% 19974, p98.0% 22322, p99.0% 27803, p99.9% 119764, p99.99% 190266
Thanks.
do you like it?
Hello, there is issue when I run benchmark. And I follow your example, It failed with : Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
dums@machine:~/jNVMf$ java -cp target/jnvmf-1.6-jar-with-dependencies.jar:target/jnvmf-1.6-tests.jar -Djnvmf.legacy=true com.ibm.jnvmf.benchmark.NvmfClientBenchmark -a 192.168.215.143 -p 4420 -g 4096 -i 3 -m RANDOM -n 10 -nqn nqn.2014-08.org.nvmexpress:uuid:8048289a-acf2-4b84-b113-e71024f24740 -qd 1 -rw read -s 4096 -qs 64 -H -I
read 4096bytes with QD = 1, time[s] = 3, pattern = RANDOM, runs = 10
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
at com.ibm.disni.util.NetUtils.getIntIPFromInetAddress(NetUtils.java:51)
at com.ibm.disni.verbs.impl.RdmaCmNat.resolveAddr(RdmaCmNat.java:157)
at com.ibm.disni.verbs.RdmaCmId.resolveAddr(RdmaCmId.java:143)
at com.ibm.disni.RdmaEndpoint.connect(RdmaEndpoint.java:100)
at com.ibm.jnvmf.NvmfRdmaEndpoint.connect(NvmfRdmaEndpoint.java:59)
at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:79)
at com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
at com.ibm.jnvmf.QueuePair.(QueuePair.java:195)
at com.ibm.jnvmf.QueuePair.(QueuePair.java:134)
at com.ibm.jnvmf.AdminQueuePair.(AdminQueuePair.java:31)
at com.ibm.jnvmf.Controller.(Controller.java:66)
at com.ibm.jnvmf.Nvme.connect(Nvme.java:50)
at com.ibm.jnvmf.Nvme.connect(Nvme.java:44)
at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.connect(NvmfClientBenchmark.java:216)
at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.(NvmfClientBenchmark.java:203)
at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.main(NvmfClientBenchmark.java:551)
Is there some problems about classpath? Could you please specify me detail?
Thanks
or only inline send model?
I feel RDMA read /write with SGL support is better for large IO.
can you point to me where to change code, to make jNVMf can connect to this kind of target.
I know for now, jNVMf only connect standard nqn subsystem name with correct format. how can I work around this? can you change the code or point to me the code?
hello buddy,
===this part cannot compile well even after I put done into my class member. I am just using one simple example. I will paste you all the code.
volatile boolean done = false;
sqe.setNumberOfLogicalBlocks(1);
sqe.setNamespaceIdentifier(new NamespaceIdentifier(1));
volatile boolean done = false;
/* set callback */
command.setCallback(new OperationCallback() {
@OverRide
public void onStart() {
}
@Override
public void onComplete() {
System.out.println("command completed");
/* check response */
NvmCompletionQueueEntry cqe = response.getResponseCapsule().getCompletionQueueEntry();
StatusCode.Value statusCode = cqe.getStatusCode();
if (!statusCode.equals(GenericStatusCode.getInstance().SUCCESS)) {
System.out.println("Command not successful");
}
done = true;
}
@Override
public void onFailure(RdmaException e) {
System.out.println("There was an RDMA error when executing the command");
done = true;
}
});
command.execute(response);
/* wait until command is done */
while (!done);
hello buddy,
SPDK support couples of NVMe request type. how about jNVMf, can you point to me the code?
kernel target server will have different process for different cmd parameter.
req->cmd.psdt = SPDK_NVME_PSDT_SGL_MPTR_CONTIG;
req->cmd.dptr.sgl1.keyed.type = SPDK_NVME_SGL_TYPE_KEYED_DATA_BLOCK;
req->cmd.dptr.sgl1.keyed.subtype = SPDK_NVME_SGL_SUBTYPE_ADDRESS;
req->cmd.dptr.sgl1.keyed.length = req->payload_size;
req->cmd.dptr.sgl1.address = (uint64_t)payload;
static int
nvme_rdma_build_contig_request(struct nvme_rdma_qpair *rqpair,
struct spdk_nvme_rdma_req *rdma_req)
{
struct nvme_request *req = rdma_req->req;
void *payload = req->payload.contig_or_cb_arg + req->payload_offset;
struct ibv_mr *mr;
uint64_t requested_size;
assert(req->payload_size != 0);
assert(nvme_payload_type(&req->payload) == NVME_PAYLOAD_TYPE_CONTIG);
requested_size = req->payload_size;
if (!g_nvme_hooks.get_rkey) {
mr = (struct ibv_mr *)spdk_mem_map_translate(rqpair->mr_map->map, (uint64_t)payload,
&requested_size);
if (mr == NULL) {
return -1;
}
req->cmd.dptr.sgl1.keyed.key = mr->rkey;
} else {
req->cmd.dptr.sgl1.keyed.key = spdk_mem_map_translate(rqpair->mr_map->map,
(uint64_t)payload,
&requested_size);
}
if (requested_size < req->payload_size) {
SPDK_ERRLOG("Data buffer split over multiple RDMA Memory Regions\n");
return -1;
}
/* The first element of this SGL is pointing at an
* spdk_nvmf_cmd object. For this particular command,
* we only need the first 64 bytes corresponding to
* the NVMe command. */
rdma_req->send_sgl[0].length = sizeof(struct spdk_nvme_cmd);
/* The RDMA SGL needs one element describing the NVMe command. */
rdma_req->send_wr.num_sge = 1;
req->cmd.psdt = SPDK_NVME_PSDT_SGL_MPTR_CONTIG;
req->cmd.dptr.sgl1.keyed.type = SPDK_NVME_SGL_TYPE_KEYED_DATA_BLOCK;
req->cmd.dptr.sgl1.keyed.subtype = SPDK_NVME_SGL_SUBTYPE_ADDRESS;
req->cmd.dptr.sgl1.keyed.length = req->payload_size;
req->cmd.dptr.sgl1.address = (uint64_t)payload;
return 0;
}
then the application will hang at below code: at the endpoint.connect part. it will hang, never return.
FabricsConnectResponseCqe connect(long timeout, TimeUnit timeoutUnit) throws IOException {
NativeBuffer buffer = new NativeByteBuffer(
ByteBuffer.allocateDirect(RdmaCmRequestPrivateData.SIZE));
RdmaCmRequestPrivateData privateData = new RdmaCmRequestPrivateData(buffer);
privateData.setQueueId(queueId);
privateData.setRdmaQpReceiveQueueSize(submissionQueueSize);
privateData.setRdmaQpSendQueueSize((short) (submissionQueueSize - 1));
this.endpoint.getConnParam().setPrivate_data(buffer.getAddress());
this.endpoint.getConnParam().setPrivate_data_len((byte) RdmaCmRequestPrivateData.SIZE);
InetSocketAddress socketAddress = controller.getTransportId().getAddress();
try {
//FIXME: rejected requests are not handled by DiSNI
//FIXME: check for overflow
endpoint.connect(socketAddress, (int)TimeUnit.MILLISECONDS.convert(timeout, timeoutUnit));
} catch (Exception exception) {
throw new IOException(exception);
}
Cadmin@ogden-dirt:/tmp/wayne/jNVMf> java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8089 -cp target/jnvmf-1.7-jar-with-dependencies.jar:target/jnvm1.7-tests.jar -Djnvmf.legacy=true com.ibm.jnvmf.utils.NvmfClientBenchmark -a 192.168.219.3 -p 4420 -g 4096 -i 3 -m RANDOM -n 10 -nqn nqn.2019-12.com.wayne:test -qd 1 -rw read -s 4096 -qs 64 -H -I
Listening for transport dt_socket at address: 8089
Name:abc
abc
read 4096bytes with QD = 1, time[s] = 3, pattern = RANDOM, runs = 10
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
hello buddy, Kernel will have below error code to do automatically recover from fatal RDMA error, make sure no resource leak at all. can you add this into this product?
since I contribute couple of ideas. :) can you also add me as co-owner? :)
===kernel way to handle RDMA error=====
Wayne: kernel will detect RDMA error, then trigger error recovery, mark ctrl is reconnecting, and doing reset and reconnection work in one kernel work item.
static int __nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc, int tag)
{
if (unlikely(wc->status != IB_WC_SUCCESS)) {
nvme_rdma_wr_error(cq, wc, "RECV");
return 0;
}
static void nvme_rdma_wr_error(struct ib_cq *cq, struct ib_wc *wc,
const char *op)
{
struct nvme_rdma_queue *queue = cq->cq_context;
struct nvme_rdma_ctrl *ctrl = queue->ctrl;
if (ctrl->ctrl.state == NVME_CTRL_LIVE)
dev_info(ctrl->ctrl.device,
"%s for CQE 0x%p failed with status %s (%d)\n",
op, wc->wr_cqe,
ib_wc_status_msg(wc->status), wc->status);
nvme_rdma_error_recovery(ctrl);
}
static void nvme_rdma_error_recovery(struct nvme_rdma_ctrl *ctrl)
{
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RECONNECTING))
return;
queue_work(nvme_rdma_wq, &ctrl->err_work);
}
I like this idea, host use this java lib, and target use kernel nvme-of target. can we work it together to make it happen?
Hello
I plan to play the software on HW, but there is some problem about target configuration when used nvmetcli. it occurs errors:
/ports/1/subsystems> create nqn.2019-08.com.dumnet:test
Could not symlink nqn.2019-08.com.dumnet:test in configFS: [Errno 2] No such file or directory
Would you please help me ?
sudo nvmetcli
/ports/1/subsystems> cd /
/> ls
o- / ..................................... [...]
o- hosts ............................... [...]
o- ports ............................... [...]
| o- 1 ................................. [...]
| o- referrals ....................... [...]
| o- subsystems ...................... [...]
o- subsystems .......................... [...]
o- nqn.2019-08.com.dumnet:test ....... [...]
o- allowed_hosts ................... [...]
o- namespaces ...................... [...]
o- 1 ............................. [...]
/> cd /ports/subsystems
No such path /ports/subsystems
/> cd /ports/1/subsystems
/ports/1/subsystems> ls
o- subsystems ............................ [...]
/ports/1/subsystems> create nqn.2019-08.com.dumnet:test
Could not symlink nqn.2019-08.com.dumnet:test in configFS: [Errno 2] No such file or directory
hello buddy,
I fixed the typo issue. but I still suffer error. it told me invalid namespace. but I copy the string from the dmesg. Could you please help me?
wayne@ubuntu:~/jNVMf$ java -cp target/jnvmf-1.5-jar-with-dependencies.jar:target/jnvmf-1.5-tests.jar -Djnvmf.legacy=true com.ibm.jnvmf.benchmark.NvmfClientBenchmark -a 192.168.147.130 -p 4420 -g 4096 -i 3 -m RANDOM -n 10 -nqn nqn.2014-08.org.nvmexpress:uuid:ea997850-3aa5-47e0-b1eb-bd11046df1e5 -qd 1 -rw read -s 4096 -qs 64 -H -I
read 4096bytes with QD = 1, time[s] = 3, pattern = RANDOM, runs = 10
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" com.ibm.jnvmf.UnsuccessfulComandException: Command was not successful. {StatusCodeType: 1 - Command Specific, SatusCode: 130 - One or more of the parameters (Host NQN, Subsystem NQN, Host Identifier, Controller ID, Queue ID) specified are not valid., CID: 0, Do_not_retry: true, More: false, SQHD: 0}
at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
at com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
at com.ibm.jnvmf.QueuePair.(QueuePair.java:195)
at com.ibm.jnvmf.QueuePair.(QueuePair.java:134)
at com.ibm.jnvmf.AdminQueuePair.(AdminQueuePair.java:31)
at com.ibm.jnvmf.Controller.(Controller.java:66)
at com.ibm.jnvmf.Nvme.connect(Nvme.java:50)
at com.ibm.jnvmf.Nvme.connect(Nvme.java:44)
at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.connect(NvmfClientBenchmark.java:216)
at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.(NvmfClientBenchmark.java:203)
at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.main(NvmfClientBenchmark.java:551)
==below is the dmesg that I get the nqn name to put into argument list above==
[34926.170782] RPC: Registered rdma transport module.
[34926.170783] RPC: Registered rdma backchannel transport module.
[34993.373669] nvmet_rdma: enabling port 1 (192.168.147.130:4420)
[35059.810576] nvmet: creating controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:ea997850-3aa5-47e0-b1eb-bd11046df1e5.
[35059.810765] nvme nvme1: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.147.130:4420
[35059.810932] nvme nvme1: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[35161.418867] nvmet: connect request for invalid subsystem nqn.2014-08.org.nvmexpress:uuid:ea997850-3aa5-47e0-b1eb-bd11046df1e5!
[35474.273405] nvmet: creating controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:ea997850-3aa5-47e0-b1eb-bd11046df1e5.
[35474.273544] nvme nvme1: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.147.130:4420
[35474.275321] nvme nvme1: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[35523.940618] nvmet: connect request for invalid subsystem nqn.2014-08.org.nvmexpress:uuid:ea997850-3aa5-47e0-b1eb-bd11046df1e5!
wayne@ubuntu:~/jNVMf$
struct rdma_conn_param param = {};
struct spdk_nvmf_rdma_request_private_data request_data = {};
struct spdk_nvmf_rdma_accept_private_data *accept_data;
struct ibv_device_attr attr;
int ret;
struct rdma_cm_event *event;
struct spdk_nvme_ctrlr *ctrlr;
ret = ibv_query_device(rqpair->cm_id->verbs, &attr);
if (ret != 0) {
SPDK_ERRLOG("Failed to query RDMA device attributes.\n");
return ret;
}
param.responder_resources = spdk_min(rqpair->num_entries, attr.max_qp_rd_atom);
ctrlr = rqpair->qpair.ctrlr;
if (!ctrlr) {
return -1;
}
request_data.qid = rqpair->qpair.id;
request_data.hrqsize = rqpair->num_entries;
request_data.hsqsize = rqpair->num_entries - 1;
request_data.cntlid = ctrlr->cntlid;
param.private_data = &request_data;
param.private_data_len = sizeof(request_data);
param.retry_count = 7;
param.rnr_retry_count = 7;
ret = rdma_connect(rqpair->cm_id, ¶m);
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.