NVMe over fabric 内核SPDK实现比较

NVMe over fabric 内核SPDK实现比较

目录

  1、SPDK启动器主流程

1.1 spdk_nvme_connect 返回了一个spdk_nvme_ctrlr

1.1.1 spdk_nvme_connect_async 

1.1.2 nvme_init_controllers，初始化完成后状态为NVME_CTRLR_STATE_READY

1.2 create_ctrlr

1.3  nvme_ctrlr_populate_namespaces

2、几个重要的概念（poller，channel，group概念）

3、分支流程

3.1 kernel心跳

3.1.1 host心跳机制

3.1.2 kernel--target心跳机制

3.2 SPDK 心跳

3.2.1 spdk-- ini

3.2.2 spdk-- tgt

3.3 SPDK异步事件管理

3.3.1 spdk_ini

3.3.2 spdk_tgt

---

  1、SPDK启动器主流程

bdev_modules_initbdev_nvme_library_initspdk_nvme_connectcreate_ctrlrnvme_ctrlr_populate_namespacesspdk_nvme_probespdk_bdev_nvme_set_hotplug

1.1 spdk_nvme_connect 返回了一个spdk_nvme_ctrlr

spdk_nvme_connectspdk_nvme_connect_async   返回 probe_ctx （里面有probe_cb, attach_cb, remove_cb, trid）nvme_init_controllers

1.1.1 spdk_nvme_connect_async 

spdk_nvme_connect_asyncnvme_driver_init     #暂时没看出来怎么使用的？？？spdk_nvme_probe_ctx_init 初始化 probe_ctx， probe_cb  attach_cb remove_cb赋值给probe_ctxspdk_nvme_probe_internalnvme_transport_ctrlr_scan ：nvme_fabric_ctrlr_scan (tcp_ops)nvme_ctrlr_probenvme_transport_ctrlr_construct :  nvme_tcp_ctrlr_constructnvme_ctrlr_construct     spdk_nvme_ctrlr状态为 NVME_CTRLR_STATE_INITnvme_tcp_ctrlr_create_qpairnvme_ctrlr_get_capnvme_ctrlr_get_vsctrlr->adminq的state置为 NVME_QPAIR_ENABLED

nvme_tcp_ctrlr_construct    nvme_ctrlr_construct     spdk_nvme_ctrlr状态设置为 NVME_CTRLR_STATE_INITnvme_tcp_ctrlr_create_qpairnvme_qpair_initnvme_tcp_alloc_reqsnvme_transport_ctrlr_connect_qpair 设置qp_state为 NVME_QPAIR_CONNECTINGnvme_tcp_ctrlr_connect_qpairspdk_sock_connect    发起tcp连接请求并设置tcp层的qp recv_state为  NVME_TCP_PDU_RECV_STATE_AWAIT_PDU_READYnvme_tcp_qpair_icreq_send 发起nof连接请求nvme_tcp_qpair_write_pdu   icreq 发给对端nvme_tcp_qpair_process_completionsspdk_sock_flush        至此，sock缓冲区才被清空，请求才被真正发出nvme_tcp_read_pdutqpair->recv_state变为 NVME_TCP_PDU_RECV_STATE_AWAIT_PDU_CHnvme_tcp_read_datanvme_tcp_pdu_ch_handle***状态变为 NVME_TCP_PDU_RECV_STATE_AWAIT_PDU_PSHnvme_tcp_pdu_psh_handlenvme_tcp_icresp_handle****tqp的state变为 NVME_TCP_QPAIR_STATE_RUNNINGqp的recv状态NVME_TCP_PDU_RECV_STATE_AWAIT_PDU_READY

nvme_fabric_qpair_connectspdk_nvme_ctrlr_cmd_io_raw  （nvme_completion_poll_cb这个东西改了 status->done = true）nvme_qpair_submit_request*****spdk_nvme_wait_for_completion

1.1.2 nvme_init_controllers，初始化完成后状态为NVME_CTRLR_STATE_READY

ctrlr管理与整个流程息息相关，一堆操作设置控制器***
nvme_init_controllersspdk_nvme_probe_poll_async    nvme_ctrlr_poll_internalnvme_ctrlr_process_init

1.2 create_ctrlr

create_ctrlrspdk_nvme_ctrlr_get_num_nsspdk_io_device_register    : nvme_bdev_ctrlrspdk_poller_register       ：bdev_nvme_poll_adminq

1.3  nvme_ctrlr_populate_namespaces

nvme_ctrlr_populate_namespacesnvme_ctrlr_populate_standard_namespace     ： g_populate_namespace_fn的成员spdk_bdev_register        ： nvmelib_fn_table        nvme_ifbdev_start**examine***spdk_bdev_get_io_channelspdk_get_io_channelbdev_channel_createbdev_nvme_get_io_channelspdk_get_io_channelbdev_nvme_create_cbspdk_nvme_ctrlr_alloc_io_qpair并为该io_channel注册一个定时poller     ： bdev_nvme_poll定时poller回调函数由线程定时自己进行调度执行，非定时poller插入线程的active_pollers中nvme_bdev_attach_bdev_to_nsnvme_ctrlr_populate_namespace_done 

nvme_ctrlr_populate_standard_namespacespdk_bdev_registerbdev_startvbdev_gpt_examine****spdk_bdev_readspdk_bdev_read_blocksbdev_read_blocks_with_mdbdev_io_submit_bdev_io_submitbdev_io_do_submitbdev_nvme_submit_request_bdev_nvme_submit_requestspdk_bdev_io_get_bufbdev_nvme_get_buf_cbbdev_nvme_readvspdk_nvme_ns_cmd_readv_with_mdnvme_qpair_submit_request     调到传输层，进行对应处理

2、几个重要的概念（poller，channel，group概念）

poller：

bdev_nvme_poll_adminq

bdev_nvme_poll

bdev_nvme_hotplug

3、分支流程

3.1 kernel心跳

3.1.1 host心跳机制

nvme_start_ctrlnvme_start_ctrlnvme_start_keep_aliveschedule_delayed_work(&ctrl->ka_work, ctrl->kato * HZ);  :nvme_keep_alive_worknvme_keep_alive   发送一个心跳报文，看是否能回超时即 nvme_reset_ctrlasync_event_work    : nvme_async_event_work

3.1.2 kernel--target心跳机制

target 心跳机制
nvmet_alloc_ctrlnvmet_start_keep_alive_timernvmet_keep_alive_timer   ：INIT_DELAYED_WORK(&ctrl->ka_work, nvmet_keep_alive_timer)schedule_delayed_work需要在一定时间收到心跳包，然后重新修改timer时间。mod_delayed_work(system_wq, &ctrl->ka_work, ctrl->kato * HZ);

3.2 SPDK 心跳

3.2.1 spdk-- ini

keep_alive只是发了一个心跳报文，没做任何处理
create_ctrlrspdk_nvme_ctrlr_register_timeout_callback   ： timeout_cbnvme_tcp_qpair_process_completionsnvme_tcp_qpair_check_timeoutnvme_request_check_timeouttimeout_cbbdev_nvme_reset

3.2.2 spdk-- tgt

每次收到host的心跳报文时更新ctrl的last_keep_alive_tick，看是否超时
_spdk_nvmf_subsystem_add_ctrlr_spdk_nvmf_ctrlr_add_admin_qpairspdk_nvmf_ctrlr_start_keep_alive_timerkeep_alive_poller    ：spdk_nvmf_ctrlr_keep_alive_pollspdk_nvmf_ctrlr_disconnect_qpairs_on_pgspdk_nvmf_qpair_disconnectspdk_nvmf_ctrlr_disconnect_qpairs_done

3.3 SPDK异步事件管理

3.3.1 spdk_ini

nvme_ctrlr_process_initNVME_CTRLR_STATE_CONFIGURE_AER :  nvme_ctrlr_configure_aernvme_ctrlr_configure_aernvme_ctrlr_cmd_set_async_event_config --- set_feature设置异步事件nvme_ctrlr_configure_aer_done  当setfeature产生cqe时触发回调nvme_ctrlr_construct_and_submit_aer  --- 提交异步事件请求分配异步事件空间时绑定对应回调nvme_ctrlr_async_event_cbnvme_ctrlr_async_event_cb       --- tgt回了cqe后回调，根据不同异步事件类型处理 active_proc->aer_cb_fn(active_proc->aer_cb_arg, cpl) ： 即aer_cb如果是ns_change，host再发一个identify命令，更新ns.
nvme_tcp_qpair_process_completionsnvme_tcp_read_pdunvme_tcp_pdu_psh_handlenvme_tcp_capsule_resp_hdr_handlenvme_tcp_req_completenvme_complete_requestcb_fn(cb_arg, cpl);     调用nvme_ctrlr_async_event_cbaer_cb在创建ctrlr时已经注册
create_ctrlrspdk_nvme_ctrlr_register_aer_callback(ctrlr, aer_cb, nvme_bdev_ctrlr);aer_cbnvme_ctrlr_populate_namespaces

3.3.2 spdk_tgt

spdk_nvmf_tcp_poll_group_pollspdk_nvmf_tcp_req_processspdk_nvmf_request_execspdk_nvmf_ctrlr_async_event_requestspdk_nvmf_ctrlr_process_admin_cmdspdk_nvmf_ctrlr_set_featuresspdk_nvmf_ctrlr_set_features_async_event_configuration第二次poll，spdk_nvmf_ctrlr_async_event_request  (host请求异步事件，压根没做任何处理，只是填了一下状态码)ctrlr->aer_req = req    并将req赋值给了ctrlr->aer_req.,当有异步事件的时候，tgt就会通过这个req_rsp回过去############### SPDK TGT
spdk_nvmf_subsystem_remove_ns
spdk_nvmf_ns_resize  -- _spdk_nvmf_ns_resize
spdk_nvmf_subsystem_add_ns   -- spdk_nvmf_subsystem_pause  --spdk_rpc_nvmf_subsystem_add_ns三种都会引起 spdk_nvmf_subsystem_ns_changed其实tgt是可以感知ns_change变化的，只是好像啥也没做。
spdk_rpc_bdev_lvol_resizevbdev_lvol_resize_vbdev_lvol_resize_cb spdk_bdev_notify_blockcnt_change_resize_notify             或者  _remove_notify************
spdk_bdev_notify_blockcnt_change    :   _resize_notify_resize_notify						或者  _remove_notifyspdk_nvmf_ns_event					SPDK_BDEV_EVENT_RESIZE || SPDK_BDEV_EVENT_REMOVEspdk_nvmf_ns_resize_spdk_nvmf_ns_resizespdk_nvmf_subsystem_ns_changed         只是将ctrlr中的ns_list进行更新spdk_nvmf_subsystem_resume spdk_nvmf_subsystem_state_change		  subsystem_state_change_on_pgspdk_nvmf_poll_group_resume_subsystempoll_group_update_subsystemspdk_nvmf_ctrlr_async_event_ns_noticespdk_nvmf_request_complete        :   填充 resp并回给host####### 下面流程仅供参考
nvmf_tgt_parse_conf_startspdk_nvmf_parse_confspdk_nvmf_parse_nvmf_tgtspdk_nvmf_tgt_createnvmf_target_advance_statenvmf_create_nvmf_tgtspdk_nvmf_tgt_createspdk_rpc_nvmf_create_targetspdk_nvmf_tgt_createspdk_nvmf_tgt_create_poll_groupspdk_nvmf_poll_group_add_subsystempoll_group_update_subsystemspdk_nvmf_ctrlr_async_event_ns_notice将事件填充到aer_req里面回给host####这里应该是理解错了，这个是另外一个流程，reservation request的流程
spdk_nvmf_tcp_req_processspdk_nvmf_request_execspdk_nvmf_ctrlr_process_io_cmdspdk_nvmf_ns_reservation_requestspdk_nvmf_subsystem_update_nssubsystem_update_ns_on_pgspdk_nvmf_poll_group_update_subsystempoll_group_update_subsystemspdk_nvmf_ctrlr_async_event_ns_noticespdk_nvmf_request_completespdk_nvmf_tcp_req_completespdk_nvmf_tcp_req_processrequest_transfer_outspdk_nvmf_tcp_send_capsule_resp_pdu