Giter VIP home page Giter VIP logo

Comments (1)

mmaaz60 avatar mmaaz60 commented on September 14, 2024

Hi @Glupapa,

Thank you for your interest in our work. The main reason of this type of fluctuation could be attributed to the dynamic nature of LLM. I would recommend reporting standard deviation by performing inference multiple times in this case.

I tried running RefSeg benchmarks multiple times on RefCOCO and following are the resutls,

# RUN 1
[
  {
    "model": "results/refseg",
    "dataset": "refcoco|val",
    "giou": "0.80515635",
    "ciou": "0.7929469"
  },
  {
    "model": "results/refseg",
    "dataset": "refcoco|testA",
    "giou": "0.83403975",
    "ciou": "0.83168846"
  },
  {
    "model": "results/refseg",
    "dataset": "refcoco|testB",
    "giou": "0.776042",
    "ciou": "0.7664252"
  }
]

# RUN 2
[
  {
    "model": "results/refseg",
    "dataset": "refcoco|val",
    "giou": "0.8066321",
    "ciou": "0.7977847"
  },
  {
    "model": "results/refseg",
    "dataset": "refcoco|testA",
    "giou": "0.8239293",
    "ciou": "0.81979144"
  },
  {
    "model": "results/refseg",
    "dataset": "refcoco|testB",
    "giou": "0.7788188",
    "ciou": "0.761931"
  }
]

# RUN 3
[
  {
    "model": "results/refseg",
    "dataset": "refcoco|val",
    "giou": "0.8075647",
    "ciou": "0.80325586"
  },
  {
    "model": "results/refseg",
    "dataset": "refcoco|testA",
    "giou": "0.83058774",
    "ciou": "0.82961744"
  },
  {
    "model": "results/refseg",
    "dataset": "refcoco|testB",
    "giou": "0.7687606",
    "ciou": "0.75880194"
  }
]

In summary,

ciou (val) - 79.79 with std of 0.419
ciou (testA) - 82.70 with std of 0.519
ciou (testB) - 76.23 with std of 0.312

Note that the standard deviations are ranging from 0.3 to 0.6 approx. which is quite reasonable. In some runs (for example Run # 1 & Run # 2 for testA), you may notice a fluctuation of more than 1.0 points, however running the experiment again normalize it with an overall fluctuation of around 0.6 points.

I hope this will help, please let me know if you have any questions. Thank You and Good Luck!

from groundinglmm.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.