Comments (3)
xiaoyao3302
commented on August 16, 2024
hello 感谢您的关注，我是这么理解这个问题的，我觉得其实可以把label的数量和网络塌陷分开理解，就是假设极端情况没有label，这时候很容易导致平凡解，有label的时候会避免这个现象的出现，同时label越多，对语义的学历能力越强，但是这对模型的塌陷没有太大的影响，因为两只网络的输入和预训练模型都是一样的（如果不是随机初始化的话），再加上监督信号很有可能也是一样的，所以两只网络很容易塌陷。我们的cosine loss只是为了防止两只网络塌陷，强行使两只网络不一致，这样随着训练的进行会因为可以从不同角度去理解所以导致对语义的理解慢慢变好，在有的情况下其实单用这么一个loss的话甚至可能会影响性能（所以才需要CPL），所以我们的这个loss的设计初衷也没有想从提高性能的角度出发。我个人的理解是把性能和塌陷两件事情分开理解，不知道您是否认同？
from ccvc.
Inev777
commented on August 16, 2024
我明白了，协同训练这种框架能带来性能的提升，但也有缺点，cps
采用的不同的网络初始化，并不能有效避免坍塌，网络初始化权重比较接近的情况下，仍然会导致网络坍塌，loss_cosine是为了避免这种情况的发生，即使有时候会影响一点点性能，但也是必要的，保证了训练的稳定性。我能理解了，非常感谢您的工作，给了我很大的启发，祝大佬科研顺利！
Edmond Skywalker ***@***.***>于2024年3月6日 周三19:30写道：
… hello
感谢您的关注，我是这么理解这个问题的，我觉得其实可以把label的数量和网络塌陷分开理解，就是假设极端情况没有label，这时候很容易导致平凡解，有label的时候会避免这个现象的出现，同时label越多，对语义的学历能力越强，但是这对模型的塌陷没有太大的影响，因为两只网络的输入和预训练模型都是一样的（如果不是随机初始化的话），再加上监督信号很有可能也是一样的，所以两只网络很容易塌陷。我们的cosine
loss只是为了防止两只网络塌陷，强行使两只网络不一致，这样随着训练的进行会因为可以从不同角度去理解所以导致对语义的理解慢慢变好，在有的情况下其实单用这么一个loss的话甚至可能会影响性能（所以才需要CPL），所以我们的这个loss的设计初衷也没有想从提高性能的角度出发。我个人的理解是把性能和塌陷两件事情分开理解，不知道您是否认同？
—
Reply to this email directly, view it on GitHub
<#12 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BGXKONPL752FUWLEZQ4TBHLYW344FAVCNFSM6AAAAABEIZ6IWGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBQGY3DOMJRGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
from ccvc.
xiaoyao3302
commented on August 16, 2024
大佬加油！
from ccvc.
Related Issues (14)
- Results HOT 2
- About unlabeled.txt HOT 3
- Question about the CPL loss HOT 2
- uestion about the setting of "mode_confident"
- L2 normalization和CosineSimilarity
- Confusion about Equations HOT 3
- Questions about retraining HOT 4
- Question about the setting of "mode_confident" HOT 4
- Question about the inplanes of ResNet HOT 1
- Question about the setting in your experiment HOT 4
- Question about the GPU HOT 1
- About Supervised loss and consistency loss HOT 11
- Question about the evaluation method HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ccvc.