petersh6 / paper-notes Goto Github PK

PDF: [Memory-Efficient Pipeline-Parallel DNN Training](Memory-Efficient Pipeline-Parallel DNN Training)

https://assured-cloud-computing.illinois.edu/files/2014/03/Zorro-Zero-Cost-Reactive-Failure-Recovery-in-Distributed-Graph-Processing.pdf

http://proceedings.mlr.press/v97/qiao19a.html
Eric Xing组的，从理论上证明self-correcting behavior of iterative-convergent algorithms. 但实际中可能无法做到完美的self-correcting

PDF: http://www.cs.cornell.edu/courses/cs717/2001fa/papers/p204-strom.pdf

PDF & Talk: https://www.usenix.org/conference/nsdi12/technical-sessions/presentation/zaharia
这一篇信息量比较大，先理解整体逻辑和思路，最后记录一下没有读的部分，以后回来再看

• CTDG: Continous-time dynamic graphs; Timed lists of events
• two types of events:
  • node-wise event: 更新节点attr如果没有该节点就新增一个
  • interaction event: temporal edge (multigraph )
  • 实验中暂不支持删除，不过附录中似乎讨论了删除操作

Core Moudels

Memory Module

• 新加入的结点memory初始化为zero vector
• $S_i(t)$是对于节点i在时间t的memory state
• 当出现一个event之后memory state会更新

Message Function

• Msg是用来udpate memory state的
• Function:
  • $m_i(t)=msg_s(s_i(t^-),s_j(t^-), \delta{t},e_{ij}(t))$
  • $m_j(t)=msg_d(s_j(t^-),s_i(t^-), \delta{t},e_{ij}(t))$
  • $m_i(t)=msg_n(s_i(t^-),t,v_{i}(t))$
  • $msg$均为learnable function，例如MLPs，也有可能是identity

Message Aggregator

• Batch processing events -> 多个不同时间的event(msg)针对同一个node
• $\overline{m_i}(t) = agg(m_i(t_1), ..., m_i(t_b))$
• $agg$可以是RNN或者attention，也可以是最简单的most recent msg和mean msg

Mem Updater

• 把$m_i(t)$（message）作为输入，然后node i之前memory作为hidden state，送到LSTM（或GRU），更新hidden state，即得到新的memory.
• $s_i(t) = mem(\overline{m_i}(t), s_i(t^-))$

Embeddings

• h函数有多种实现方式
  • identity：emb(i, t) = si(t) 直接用memory
  • time projection：emb(i, t) = (1 + $\delta t w$) *si(t)， w可学习（JODIE）
  • **Temporal Graph Attention
    
• Temporal Graph Sum

https://www.usenix.org/conference/nsdi21/presentation/iyer

PDF: https://shivaram.org/publications/chi-vldb18.pdf

PDF: http://proceedings.mlr.press/v97/qiao19a.html

Websites: [CheckFreq: Frequent, Fine-Grained DNN Checkpointing](CheckFreq: Frequent, Fine-Grained DNN Checkpointing)
感觉是做的很简洁的一篇工作

PDF
Talk

PDF: https://static.googleusercontent.com/media/research.google.com/zh-CN//archive/mapreduce-osdi04.pdf
Talk: https://www.bilibili.com/video/BV1Vb411m7go?from=search&seid=2081351075929972797

https://www.cs.sjtu.edu.cn/~shen-yy/p437-shen.pdf

PDF: https://arxiv.org/abs/1506.08603
不鸽了 开始把暑假读的论文都总结一遍
以后读完一篇要马上总结，不然要点都忘了

PDF: https://research.fb.com/wp-content/uploads/2021/04/CPR-Understanding-and-Improving-Failure-Tolerant-Training-for-Deep-Learning-Recommendation-with-Partial-Recovery.pdf
facebook的一篇工作，关于推荐系统

PDF: https://dl.acm.org/doi/10.1145/3341301.3359646
Video: https://www.youtube.com/watch?v=ILBPCi6Il1U
#18

PDF: https://lamport.azurewebsites.net/pubs/chandy.pdf
最经典的分布式快照算法，Spark中有实现

PDF: https://www.cs.rice.edu/~dbj/pubs/ftcs87-sbml.pdf

PDF: https://cs-people.bu.edu/liagos/material/sosp19.pdf
Talk: https://www.youtube.com/watch?v=ANSxtucbMzg

PDF https://www.cs.utexas.edu/~lorenzo/papers/SurveyFinal.pdf
综述论文 看一点总结一点