audiolabs / webmushra Goto Github PK
View Code? Open in Web Editor NEWa MUSHRA compliant web audio API based experiment software
Home Page: https://audiolabs.github.io/webMUSHRA/
License: Other
a MUSHRA compliant web audio API based experiment software
Home Page: https://audiolabs.github.io/webMUSHRA/
License: Other
I need to evaluate 4 different speech synthesis systems.
I have 100 sentences and I have synthesized audio of each sentence from each TTS system.
The users should evaluate a sentence generated by all systems with mushra method, but I do not want a single user to have to go through 100 tests. Is it possible to randomly select eg. 10 pages that would be presented to the user?
The script service/write.php uses a function called get_magic_quotes()
that has been removed in PHP 8: https://www.php.net/manual/en/function.get-magic-quotes-gpc.php
Using a fresh XAMPP install (which ships with PHP 8), the results were not written because of that.
Since magic quotes are also not supported in PHP 8, I removed the conditional sanitizing and now everything seems to work fine.
It seems that all the wave played are mono, and my headset is ringing only to the left.
I'm trying to have a little questionnaire after each page of mushra test, but it seems that it can be done only for the end page. Is there a way to do that ?
lets update the dependencies one by one:
First of all, thanks for releasing this cool tool for listening test.
webMUSHRA works so great locally and I believe it works for a crowd-sourced listening test. But the current README doesn't explain it. Can you please elaborate it? Thanks!
And there are more questions: where does it save the ratings from multiple users? How does it differ the ratings of different users? Using different csv files?
Thanks a lot and best regards,
Jun
PHP Warning: Invalid argument supplied for foreach() in webMUSHRA/service/write.php
I think #4 is fixed, we can make a public release and add more things later.
Shall we reset the version to 1.0.0
(it's on 1.3.0
currently)? I would keep it like this because it reflects the difference to our internal version.
In the future I would suggest that we increase the version number when we change the api, and write release notes so that users can better rely on a stable webmushra
Hi, thanks for the awesome tool! I am doing some listening experiment that uses audio stimuli of ~30s length, and received error "Duration of stimulus (reference) must not exceed 12s." when running. Is there any way to increase the limit?
Is there any setting to ensure participants rate all the audio samples before continuing to the next page?
When you click "Next", the page's scroll position stays the same. Participants might miss instructions.
Currently when I run the command
docker-compose -f docker-compose.yml up
It launches webMUSHRA, however, this is locally hosted on my laptop and I cannot access the link from another computer. How do I make it publicly available? I tried techniques like port forwarding but it did not work for docker-compose. Can you please recommend a way to do this. Thank you.
Finally, the audio worklets are close to get into stable browsers. Lets retire the script processor asap. Changing the code looks be rather simple...
https://developers.google.com/web/updates/2017/12/audio-worklet
I have to say this is a great project. Thanks for the developers.
One small question, this project seems not support play stereo wav file. How to make it happen? thanks.
This would allow to host the site easily on services such as glitch.com or zeit.now
When I visit https://audiolabs.github.io/webMUSHRA/ and play the reference audio, I can only hear the audio in the left speaker. I find this behavior in both Chrome and Firefox. I want to have the audio in both speakers.
Hello,
If you want to customize some words in the interface (for example, in BS. 1116, changing the describing word "annoying" to "different"), where can I do it?
Thanks for your help in advance!
Hi, I am so grateful for this webMUSHRA.
I was wondering whether I can have a training phase and an evaluation phase by using a single YAML configuration.
Thanks!
It seems that errors are not thrown, even if stopOnErrors
is set to true
@mschoeffler does this have to do with these commented code? if this code is active the error handling does work as expected. This was commented by Marlene Rösch one year ago, so I would expect this was commented out by accident.
Can you please check this and fix?
Thanks for the webMUSHRA!
I deployed webMUSHRA on an aws ec2 server. Php is also installed.
The problem happens after I finished the test and clicked "send results". No error reported, however, I can't see any folder generated in my webMUSHRA folder on the EC2 server?
This has been tested locally, which successfully created the result folder.
not sure if this is an issue you can even fix, so close/ditch if that's the case, but the config files are very unforgiving of non LF line breaks.......it took me quite a while to troubleshoot a students config to figure this out (they edited in Windows Notepad).
Perhaps mention this in the docs (i.e. not to use notepad ;-) )
p.s. WebMushra is an awesome resource, thank you!
Trying to run docker-compose -f docker-compose.yml build
I am getting the following error:
AttributeError: module 'platform' has no attribute 'linux_distribution'
According to some googling, this is because the function linux_distribution() was removed in Python 3.8.
Are you aware of this and/or are there any plans to support 3.8 anytime soon?
Hello. I want to give a bulletted instruction set to the listeners in the first page. In the 'content :' key, I am able to add a single, long line. But, not able to break it into multiple lines. I tried the yaml clipping keys '>', '|', '>-'. They didn't work. '\n' also didn't work. Is there way to add a multiline instruction?
Thanks.
Currently when the reference and the stimuli have a different number of samples an error is thrown and if stopOnErrors: true
is set, experiments will not be started.
If experiments are started nonetheless webMUSHRA behaves as followed:
The expected behaviour in this case would be that the stimuli are filled up/padded with silence. Even though we decided earlier to force users to make sure that references and stimuli have the same length; user feedback shows that this fix would increase the usability for many users.
Is it possible to have a feedback box for each test condition so that the participants can explain and justify their grading for each signal. I set up a 'finish' page after each 'mushra' page in the .yaml file to try and get feedback that way but the listening test terminates as soon as the participant hits 'Send Results' after writing their comments. As far as I am aware, there is no obvious way to get comments for each grade for each test condition. Please let me know if this is a possibility.
Hello,
i have a (simply) question.
Is it possible to change the mushra scale labels (excellent - good - fair - poor - bad) ?
thanks in advance!
best regards
simon
It would be great to have the chance to create for instance 100 pages (mushra type) but limit the test randomly to only 10 pages for each user
Lets have this in mind and check if we are affected:
https://bugs.chromium.org/p/chromium/issues/detail?id=807017
I was testing the use of webmushra today for an experiment of mine and I accidentally clicked on one of the logos at the bottom of the screen losing my progress and interrupting my experiment.
Those logos definitely should not be clickable as it might lead to the loss of data and time.
Ideally, I'd rather see the logos at the beginning and the end in a bigger format as I believe that having the logos during the test conditions could distract the user during experimenting.
How can I add a logo at the bottom of each page ? Or at least of the first page of the test ?
Thanks you
Hi,
thanks a lot for this awesome tool and the provision of a docker specification. To further ease use of the tool, would it be possible to use Github-CI for building the docker image and pushing the image at e.g. Docker Hub? This would very much increase usability as only the docker-compose.yml
needs to be configured and the image would simply be downloaded.
Many thanks in advance.
Max
Both for the default project and when entering my own audio files, they are only played in the left ear when using webMUSHRA. Is this a bug or is it something I missed in the configuration?
Hi,
The algorithm currently used for randomly ordering the experiment pages involves providing function() { return 0.5 - Math.random(); }
to array.sort
.
This method is biased: it is more probable for the first page in the YAML config file to be presented in the first pages. (see https://stackoverflow.com/a/18650169)
If possible, I'd recommend using the Fisher-Yates (aka Knuth) Shuffle as recommended here: https://stackoverflow.com/a/2450976.
Regards,
Krlln
Lines 73 to 75 in c929877
Hey guys,
play/pause gets triggered when pressing and when releasing the spacebar on Firefox 88.0.1
I didn't notice the issue on other browsers.
Hey, I want to run a MUSHRA test where the sound files can't be normalized. Is there a way to disable this?
Hey,
I will like to play the audio only once when the "Play button" is pressed. Currently, in type mushra, it continues to play the audio in a loop. Is there a way to stop continuous loop playback? I will appreciate the help.
Add support for pointing method that has been published in https://scholar.google.de/citations?view_op=view_citation&hl=de&user=7Pd9rlwAAAAJ&citation_for_view=7Pd9rlwAAAAJ:IjCSPb-OGe4C
Hi, I noticed there is a questionnaire field in the finish
page of complete.yaml
, which allows collecting some information like age from the participants. However, I don't think these information was submitted to the backend as they don't show up in the results csv, and there are no documentation of the questionnaires.
Is there a known way to use webMUSHRA, so as to take dynamic annotations by the users (for example assessing quality each half a second for an audio track)?
Or maybye if you could provide me a starting point for me to alter the provided code?
Thanks in advace
Hi, thank you for this amazing repository. :)
I've set up webMUSHRA using Docker as written in README.md.
After that, I tried complete.yaml.
The test worked well, but when I pressed the "send result" button on the last page, an error occurred as follows.
"An error occured while sending your data to the server! Please contact the. experimenter."
Also, there was no new folder in the results folder.
Could you please check on this issue?
Hi again! (sorry for many questions....)
My mono sound files only plays back through left channel.
Is there any way to play through both channels?
loading the mushra_showresults.yaml
yields in an error. The problem is that it cannot find the MushraResults
page (see here. Maybe this is a leftover from an older branch?
to make it easier for users and experimenters and developers to see which version of webMUSHRA is used, it would help to have an info button in the user interface that shows the version of webMUSRA when clicked.
Implementation wise this could easily be done by parsing the package.json
grunt.file.readJSON('package.json')
Hey guys,
I'm setting up webMUSHRA on a https server and ran into the following problem:
On most browsers (Firefox, Chrome, Brave, Edge) the sound stimuli get duplicated, triggering the MUSHRA validator if the duplicated stimuli exceed 12s. On Safari on mac everything works fine.
When running the same setup on my local machine (On MacOS with php -S localhost:8000
) it works as intended on all browsers.
Any ideas where the issue is coming from?
Here's my config file
The rest of the code is unmodified apart from changing the rating scale labels of the MUSHRA page.
Thanks, Tobias
Hi,
Is there a simple way to reset the starting time of each condition whenever the corresponding Play button is pressed? Meaning that I don't want the audio to resume from the last index of the previously played condition.
Thanks!
Hi to everyone,
I am trying to play an audio file of 6 channels (5.1) (96000 KHz, 24 bit) on a MUSHRA page via MOTU 828x. Although all of the 6 speakers reproduce the signal, it is highly distorted. However, when I use 16 bits per sample the audio reproduction is fine. The sound card supports both 16 and 24 bits per sample at 96000 kHz.
Is this a problem of the webMUSHRA code and is there any idea on how it can be solved?
Thanks in advance!
The 1st page of the test doesn't have working audio. Adding a 'welcome' page, or navigating to another page and back again fixes it, though.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.