很优秀的工作！谢谢您的贡献，我想知道如果我以问题未输入的话，代码中是有一个参数的设置还是可以直接改输入的呢？例如：c直接改成cq？？

this issue is for testing

Thank you for providing this useful collection of models! DKVMN seems to perform best on my dataset. Yet, I am having trouble to display the knowledge states as described in their paper. Are you aware of any source where the code for creating knowledge states for DKVMN or SKVMN are available?

Thank you very much in advance!

您好，感谢贵团队对代码进行开源，能够让我学习到有价值的知识。目前项目中没有一题多知识点的数据集嘛。就比如说在qikt模型当中，需要对一个问题对应的多个知识点进行平均嵌入操作。

您好，我在用assist2015数据集跑akt、atdkt、simplekt和sparsekt等模型时都报了下述图片上的错误，求助如何

需要在具体代码后加to(device)。

我尝试在examples文件夹下执行python -u data_preprocess.py --dataset_name ednet。

最终报错
train+valid original interactions num: 178558352, select num: 0, qs: 12277, cs: 188, seqnum: 588152

dropnum: 1848
train+valid sequences interactions num: 178556504, select num: 178556504, qs: 12277, cs: 188, seqnum: 1340548
dropnum: 438
original test interactions num: 45675101, select num: 0, qs: 12267, cs: 188, seqnum: 147038
test sequences interactions num: 45674663, select num: 45674663, qs: 12267, cs: 188, seqnum: 340129

Killed

在killed之前我观察到系统内存不断升高，直至killed。
我个人的机器配置有192G内存。请问是处理这个数据集需要更大的内存还是说预处理代码存在内存泄露？

求助大佬~
我是把整个包以zip的方式下载然后作为project在pycharm中打开。我按照pykt的quick start中的提示，先将路径定位到我电脑中这个包里面examples文件夹的位置，然后运行python data_preprocess.py --dataset_name=assist2009， 就会出现下述报错：

请问我要跑其他公共数据是都需要自行先把相关的数据集下载好吗~

我尝试从网上下载提到的这份assist2009的数据，但是又出现的另一个错：

我下载的数据是这样的：

我从网上找上述错误的解决方法，我将utf-8 改成 gbk 又报了其他的错：

我看网上解决方案都是加个float，但是加了这个又会报错，然后有评论说编码转成utf-8.
已经彻底晕了，求助热心的大佬！！！

Similarly to my previous issue, it is not clear how I should lay out the files after downloading the various parts of the ednet dataset.
I currently have:
data/ednet/contents
data/ednet/KT1
data/ednet/KT2
data/ednet/KT3
data/ednet/KT4

I am getting a "ValueError, No objects to concatenate" when I run "python data_preprocess.py -d ednet", and am not enjoying reverse engineering the data processing scripts.

juyi的acc比auc高很多，

ednet在dkt的效果为92%以上，ednet，我用的是KT1

2023-09-19 00:37:50 - main - said: init_model
model_name:sparsekt
The wrong model name was used...
model is None
Traceback (most recent call last):
File "wandb_sparsekt_train.py", line 39, in
main(params)
File "/root/pykt-toolkit-main/examples/wandb_train.py", line 123, in main
opt = Adam(model.parameters(), learning_rate)
AttributeError: 'NoneType' object has no attribute 'parameters'

train_model in train_model.py

the function doesn't evaluate model with test dataset when save_model==True

however, in some cases we need to save model and evaluate with test dataset

            if save_model:
                torch.save(model.state_dict(), os.path.join(ckpt_path, model.emb_type+"_model.ckpt"))
            max_auc = auc
            best_epoch = i
            testauc, testacc = -1, -1
            window_testauc, window_testacc = -1, -1
            if not save_model:
                if test_loader != None:
                    save_test_path = os.path.join(ckpt_path, model.emb_type+"_test_predictions.txt")
                    testauc, testacc = evaluate(model, test_loader, model.model_name, save_test_path)
                if test_window_loader != None:
                    save_test_path = os.path.join(ckpt_path, model.emb_type+"_test_window_predictions.txt")
                    window_testauc, window_testacc = evaluate(model, test_window_loader, model.model_name, save_test_path)
            validauc, validacc = auc, acc
        print(f"Epoch: {i}, validauc: {validauc:.4}, validacc: {validacc:.4}, best epoch: {best_epoch}, best auc: {max_auc:.4}, train loss: {loss_mean}, emb_type: {model.emb_type}, model: {model.model_name}, save_dir: {ckpt_path}")
        print(f"            testauc: {round(testauc,4)}, testacc: {round(testacc,4)}, window_testauc: {round(window_testauc,4)}, window_testacc: {round(window_testacc,4)}")

I will wait for other opinions.

代码修改：y,loss = model.train_one_step(data, device)后续一路添加

Hi, can I use the multi-skill exercises of myself dataset?
For example：i have a exercises list $E={e_1,e_2,...,e_n}$，$e_ i$ have skill ${s_j，s_k，s_h}$ (j!=k&&k!=h&&j!=h)
whether support my dataset from other source(sush as :codeforces.com,luogu.com.cn)?

尊敬的作者团队，你们好：
我在尝试参加AAAI竞赛，在使用akt模型预测测试集时报出如下错误：

Namespace(atkt_pad=0, save_dir='saved_model/peiyou_akt_qid_saved_model_3407_0_0.2_256_512_8_4_0.0001_0_1', test_filename='pykt_test.csv', train_ratio=0.5, use_pred=0, use_wandb=0)
Start predicting model: akt, embtype: qid, save_dir: saved_model/peiyou_akt_qid_saved_model_3407_0_0.2_256_512_8_4_0.0001_0_1, dataset_name: peiyou
model_config: {'dropout': 0.2, 'd_model': 256, 'd_ff': 512, 'num_attn_heads': 8, 'n_blocks': 4}
data_config: {'dpath': '../data/peiyou', 'num_q': 7652, 'num_c': 865, 'input_type': ['questions', 'concepts'], 'max_concepts': 6, 'min_seq_len': 3, 'maxlen': 200, 'emb_path': '', 'train_valid_file': 'train_valid_sequences.csv', 'folds': [0, 1, 2, 3, 4]}
Start predict use_pred: False, ratio: 0.5...
Traceback (most recent call last):
  File "wandb_eval.py", line 74, in <module>
    main(params)
  File "wandb_eval.py", line 55, in main
    dfinal = evaluate_splitpred_question(model, data_config, testf, model_name, save_test_path, use_pred, ratio, atkt_pad)
  File "/media/sda2/yueying/pykt-toolkit/pykt/models/evaluate_model.py", line 611, in evaluate_splitpred_question
    qidxs, ctrues, cpreds = predict_each_group2(dtotal, dcur, dforget, curdforget, is_repeat, qidx, uid, idx, model_name, model, t, end, fout, atkt_pad)
  File "/media/sda2/yueying/pykt-toolkit/pykt/models/evaluate_model.py", line 1034, in predict_each_group2
    y, reg_loss = model(ccc.long(), ccr.long(), ccq.long())
  File "/media/sda1/yueying/miniconda3/pkgs/envs/pykt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/sda2/yueying/pykt-toolkit/pykt/models/akt.py", line 84, in forward
    q_embed_data, qa_embed_data = self.base_emb(q_data, target)
  File "/media/sda2/yueying/pykt-toolkit/pykt/models/akt.py", line 77, in base_emb
    qa_embed_data = self.qa_embed(target)+q_embed_data
  File "/media/sda1/yueying/miniconda3/pkgs/envs/pykt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/sda1/yueying/miniconda3/pkgs/envs/pykt/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 160, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/media/sda1/yueying/miniconda3/pkgs/envs/pykt/lib/python3.7/site-packages/torch/nn/functional.py", line 2199, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

希望作者能够提供一些错误产生的线索，感谢！

首先感谢项目组的开源，目前我在调试qikt模型时出现些问题，不知道是不是有些包的版本不一样导致的，所以希望项目组可以考虑出一个基础环境安装的包

Hello, you've done a great job in knowledge tracking, but I have some questions about your open-source code. I noticed that there are two files in your code for evaluating models, eval.py and predict.py. As far as I understand, predict.py might not be using the fusion method, which means it could potentially lead to label leakage. Is my understanding correct, or could you please explain the difference between these two files?

Hi, your pykt is a very nice project. But I got the following traceback on my machine :

  File "**/pykt-toolkit/examples/wandb_train.py", line 71, in main
    (train_loader, valid_loader) = init_dataset4train(dataset_name, model_name, data_config, fold, batch_size)
ValueError: too many values to unpack (expected 2)

after my troubleshooted,I find we have a package named pykt,and a module named pykt too. The paths for both of them are as follows：

• module pykt : **/pykt-toolkit/pykt
• package pykt ：**/envs/pykt/lib/python3.7/site-packages/pykt

According to the description of the Modules section of the python website (https://docs.python.org/3/tutorial/modules.html)，`from-import` will choise package pykt frist.

So, have you encountered the same problem, or can you give me some suggestions to solve it?
thank you very much!

请问dkt_forget模型中，num_rgap、num_sgap、num_pcount三个参数代表什么呢？

How long does it takes for GKT training?

Sharing other datasets training time of GKT might be helpful!

Thank you!

I modified the Saint model structure in the saint.py file. However, when I call the same command 'python wandb_saint_train.py --dataset_name=assist2009 --seed=3407 --dropout=0.3 --use_wandb=0 --add_uuid=0' before and after the modification, the accuracy and AUC remain exactly the same. Why is that?

def train_one_step(self,data,process=True,return_all=False):
    outputs,data_new = self.predict_one_step(data,return_details=True,process=process)
    # all 
    loss_q_all = self.get_loss(outputs['y_question_all'],data_new['rshft'],data_new['sm'])
    loss_c_all = self.get_loss(outputs['y_concept_all'],data_new['rshft'],data_new['sm'])

作者你好，我看outputs中的y_question_all和y_concept_all好像是t时刻的预测值，而rshft是t+1时刻的真实值。 为什么要用这两个值做损失？请问是我哪里理解错误了吗

The que_dkt_demo file of the example folder had import the DKTQue method, but I don't find the relation file.
The raw code was " from pykt.models.dkt_que import DKTQue "

How to get other data sets besides ASSIST？

您好，请问Hyper parameter tunning results给出的每个模型每一折的参数配置，对应了当前模型在该折的最优性能吗

您好！
非常感谢您详实的工作，我相信这将对知识追踪领域产生长远的影响。
请问是否有计划支持 multi GPU training，这对更大的batch和更复杂的网络实现而言是必要的。

Hi, I'm following the quick start guide, it is unclear how you are meant to access python data_preprocess.py [parameter] from the previous instructions on the page.

It would be great if this could be clarified

Hi , I am currently working with DKT-F and encountered the following error: "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!"

FileNotFoundError: [Errno 2] No such file or directory: '../data/ednet\train_valid_sequences.csv'
when running wandb_saint_train.py --dataset_name=ednet

I have already ran python data_preprocess.py --dataset_name=ednet

感谢pykt为知识追踪领域做出的巨大贡献。
在阅读论文后，关于如何复现Question Level和KC Level仍存有疑问。我运行example中wandb_xxxx_train.py再运行wandb_predict.py后，所示结果包含One-by-One的结果等。请问其他结果中对应的是Question Level还是KC Level的呢？

{'testauc': 0.9188002221943278, 'testacc': 0.8678735316393662, 'window_testauc': 0.9201199409709553, 'window_testacc': 0.868663798838086, 'oriaucconcepts': 0.807274877304899, 'oriauclate_mean': 0.807413805143088, 'oriauclate_vote': 0.8049668720325175, 'oriauclate_all': 0.8058672455891169, 'oriaccconcepts': 0.7851768050790059, 'oriacclate_mean': 0.807144009898327, 'oriacclate_vote': 0.8066383344988972, 'oriacclate_all': 0.8018075205766851, 'windowaucconcepts': 0.8093884777107377, 'windowauclate_mean': 0.8101170671331457, 'windowauclate_vote': 0.8078638216548034, 'windowauclate_all': 0.8085259538001138, 'windowaccconcepts': 0.7864040341303994, 'windowacclate_mean': 0.808367475743734, 'windowacclate_vote': 0.8078753971395257, 'windowacclate_all': 0.8028583348487928}

期待您的回答，感谢

The assistment2017 dataset (anonymized_full_release_competition_dataset.csv) does not include concepts. Why does the data set generated after "data_preprocess.py" have concepts?

Hello, sorry to bother you again. May I ask if all models have a training Epoch of 200？

Hello, I encountered this error while running the code. How can I resolve it? Looking forward to your reply very much

Thank you for sharing a masterpiece framework.

I ran various models on the nips_task34 dataset, and the results were well reproduced.

However, on the assistment2009 dataset, I get about 5% better performance on average than reported in the pyKT and QIKT papers, etc. (e.g., DKT and SparseKT-topk achieve about 0.82 and 0.83 val auc, respectively.)

Is there something I'm missing?

Hello, I want to ask your opinion on the AKT model with regard to the reason why that model performs best in your delicate framework. (https://arxiv.org/abs/2206.11460)

the image above is the figure of AKT model represented in the paper

qa_embed_diff_data = self.qa_embed_diff(
target) # f_(ct,rt) or #h_rt (qt, rt)差异向量
if self.separate_qa:
qa_embed_data = qa_embed_data + pid_embed_data *
qa_embed_diff_data # uq* f_(ct,rt) + e_(ct,rt)
else:
qa_embed_data = qa_embed_data + pid_embed_data *
(qa_embed_diff_data+q_embed_diff_data) # + uq *(h_rt+d_ct) # （q-response emb diff + question emb diff）

and the code above is what you implemented at pykt/models/akt.py.

I think you followed the right way as the paper's author described.
The point is that I think AKT model has the best performance because it has a chance to know the target answers with "f(c_t, r_t) variation vector" (at the paper), which is "qa_embed_diff_data" (at your code).

As a result, in my opinion, AKT has the best performance because of its already-known target issue.

To resolve the issue, I suggest modifying Architecture forward function as the following code:

        else:  # dont peek current response
            pad_zero = torch.zeros(batch_size, 1, x.size(-1)).to(self.device)
            q = x
            k = torch.cat([pad_zero, x[:, :-1, :]], dim=1)
            v = torch.cat([pad_zero, y[:, :-1, :]], dim=1)
            x = block(mask=0, query=q, key=k, values=v, apply_pos=True) # True: +FFN+残差+laynorm 非第一层与0~t-1的的q的attention, 对应图中Knowledge Retriever
            # mask=0，不能看到当前的r
            flag_first = True

thank you for your attention :)

作者你好，非常感谢你们提供的非常有用的kt库，但是有个疑问咨询一下，在pykt论文中你们也提到一题多知识点存在信息泄露的问题，通常可以利用知识点聚合方式来避免，但是我看你们实现的代码里面关于模型输出然后loss计算那一块，也还是输入扩展后的知识点序列，然后one-by-one的预测？可能是我对代码的理解偏差，望指正，期待您的回复！

There is only an entry for ednet5w passing the dataset_name in. Without passing in the dataset_name, start_i is never defined and it errors out

Can you share the benchmark results of your experiments with the ednet dataset?
�Still need to develop the data processing part for ednet experiments?

Thank you very much for open sourcing this framework, which I think greatly facilitates the development of knowledge tracing. I would like to ask if it is possible to publish the optimal hyperparameters for each model on each dataset.

您好，想问一下这个地方rel是如何得到的？

请问一下大佬，我们选用数据时候可以用验证集AUC吗 Epoch: 8, validauc: 0.7784, validacc: 0.7796, best epoch: 8, best auc: 0.7784, train loss: 10.460885047912598, emb_type: qid, model: iekt, save_dir: 还是必须用test呢？

Hello, Thanks for the nice framework for knowledge tracing.
I'm also impressed with the delicate experiments you've done on the paper(https://arxiv.org/abs/2206.11460)

Now I'm working on reproducing your Table 2 results shown on the paper, with the assist2015 dataset you've also shared.
And, I got confused about getting the best hyperparameters as you mentioned at A.3 Hyperparameter Search Details of Representative DLKT Baselines.

So, there are two different settings on your code, which is,

1. values in configs/kt_config.json
2. default values of every argument in wandb/wandb_{model_name}_train.py
   Which values do I have to follow to get the high result close to table 2?

Thanks.

Thank you very much for the great contribution of pykt library in the field of knowledge tracing. In the reproduction process, the default value of the parameter "emb_type" is "qid", I would like to know what are the values of the parameter "emb_type"?

Discussed in #104

我有个问题 就是我用pykt库里面的 iekt模型跑 然后auc是0.8318。然后我需要atdkt模型数据集里面的 历史正确性这个数据。就把iekt的数据loader换成了atdkt的loader 数据集不变.然后啥也没改 iekt的auc就到了0.923。
修改代码如下：

elif model_name in que_type_models:

#     curvalid = KTQueDataset(os.path.join(data_config["dpath"], data_config["train_valid_file_quelevel"]),
#                     input_type=data_config["input_type"], folds={i}, 
#                     concept_num=data_config['num_c'], max_concepts=data_config['max_concepts'])
#     curtrain = KTQueDataset(os.path.join(data_config["dpath"], data_config["train_valid_file_quelevel"]),
#                     input_type=data_config["input_type"], folds=all_folds - {i}, 
#                     concept_num=data_config['num_c'], max_concepts=data_config['max_concepts'])
elif model_name in ["iekt"]:
    curvalid = ATDKTDataset(os.path.join(data_config["dpath"], data_config["train_valid_file"]), data_config["input_type"], {i})
    curtrain = ATDKTDataset(os.path.join(data_config["dpath"], data_config["train_valid_file"]), data_config["input_type"], all_folds - {i})

请问哪里出错了啊？

In the pykt paper, it seems to evaluate model with KC level.

How can we evalutate model in question level?

It is different with early fusion because, in the peper, DKT/GKT didn't evaluate with early fusion setting.

Do you have any plan to add question level evaluation metrics?

Great work, I managed to run examples and prepare my own "mock" - it works just fine. I have some difficulties though when analyzing the results - I'd like to check individual users' predictions to analyze how it changes when sequences change, go through some examples, etc. Where should I look for stuff like that in the repository?

Excuse me, can the model run only in the GPU environment that supports cuda?