Comments (7)
chrismre
commented on August 29, 2024
This a great issue, Sen!
A naive guess is that the symmetry isn't broken consistently, there is a always a symmetry e.g., p -> 1-p for labeling function accuracies. This is where the theory elects a particular winner @ajratner @fredsala any thoughts here?
Can you add in the LF function accuracies to confirm or refute that guess? Thanks!
from metal.
ajratner
commented on August 29, 2024
Hi Sen,
Thanks for posting- keep them coming! At the end of the training we learn
the model up to symmetry- and then break symmetry based on the fact the LFs
are non-adversarial. But apparently this isn’t happening correctly here!
Either way it’s an easy thing to resolve- I will do so a little bit later
today
Thanks,
Alex
…On Sat, Sep 1, 2018 at 10:56 AM SenWu ***@***.***> wrote:
Hi,
Sorry for bothering you again.
I found some weird behavior in the label model where I run the label model
twice with the same data and I got the totally opposite result. Here you
can find a snapshot of my code.
[image: screen shot 2018-09-01 at 10 54 35 am]
<https://user-images.githubusercontent.com/5580008/44948599-7b1df400-add5-11e8-8334-1e09becd9edb.png>
Data: basics_tutorial.pkl
Code: snorkel-metal 0.1.1
Please let me know if I am doing something wrong.
Thanks,
Sen
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#36>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABgw_ftyKs7sGFM_ZHNk1bDqdKGrbHskks5uWsoygaJpZM4WWT_4>
.
from metal.
ajratner
commented on August 29, 2024
@senwu @chrismre Yes this is definitely what's going on- in transit now but then I'll resolve in a bit! First try is just by changing the defaults to put centered L2 on by default (not currently on by default) in addition to initialization to resolve this symmetry- should take care of it
from metal.
ajratner
commented on August 29, 2024
Hi @senwu I'm having a hard time reproducing the behavior you have above- have you pulled the latest version of master? Any other differences that might explain this?
I still think this must be a symmetry-breaking issue, just confused that I can't replicate. We can talk on slack abt too!
from metal.
ajratner
commented on August 29, 2024
Looks like this is fixed on master, need to propagate to the pip installable version
from metal.
lukehsiao
commented on August 29, 2024
If you have the latest code on master and are tagging releases to pypi, I'd highly recommend incrementing the version number in master after a release (e.g. to v0.1.6). This avoids the issue where if a user installs from pypi, they get v0.1.5, and if a user installs from GitHub, they get v0.1.5, but they are not actually the same versions.
from metal.
ajratner
commented on August 29, 2024
@lukehsiao thanks, will do!
@senwu note I added in centered L2 reg (see updated Basics tutorial) which might also generally be useful for these types of symmetry-breaking issues, if they come up
Also, unless any strong objections, I'm going to start closing issues when they are fixed on master; then I will increase the frequency of pushing new versions to pip. Happy to hear pushback on this... but since I still view this repo as in 'dev-heavy mode' (research!!), I think this will be cleaner
from metal.
Related Issues (20)
- Using BERT with MeTaL HOT 6
- Generative model snorkel vs Generative model metal HOT 3
- KeyError: 'METALHOME' error from MMTL_Basics tutorial HOT 1
- [QUESTION] Mixing textual and numerical inputs with multi-branch models HOT 2
- MMTL training accuracies swinging between two datasets HOT 3
- issue keras or Tensorflow in the future? HOT 1
- multiple
- when applying metal, how to estimate noisy-label probability for multilabel problem of a single task HOT 2
- Keep track of changes as CHANGELOG HOT 2
- I think there are some errors in the mt_end_model.py HOT 1
- Potential bug for the predict_proba function in label_model.py
- How to use weak supervised learning with MMTL HOT 1
- links to basic snorkel tutorials are broken. HOT 1
- How does an end model improve upon Snorkel? HOT 1
- End model seems to always do better when assuming a 50/50 class balance HOT 1
- Supporting "default" class HOT 1
- Question on LF influence on Snorkel predictions based on coverage HOT 1
- Change LF stat "empirical accuracy" to "precision" HOT 1
- Problems with networkx when using LabelFunction HOT 1
- Singular error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metal.