google / synthmark Goto Github PK
View Code? Open in Web Editor NEWAudio performance benchmark for jitter, theoretical latency, etc.
License: Apache License 2.0
Audio performance benchmark for jitter, theoretical latency, etc.
License: Apache License 2.0
This will prepare us for #72
so we can always reach the "Cancel" button.
About a third of the time. SynthMark would hang when the test was finished. It would then never print the test results.
This was caused by returning AAUDIO_CALLBACK_RESULT_STOP from the callback and then immediately closing the stream. This caused a race condition.
The fix involves returning AAUDIO_CALLBACK_RESULT_CONTINUE and then preventing the callbacks from executing the thread procedure multiple times.
Just leave the int main() in the apps folder and have it call a library function SynthMarkCommand(argc,argv).
CDD says:
The SynthMark app needs to be run using the "Automated Test" option and achieve the following results:
voicemark.90 >= 32 voices
latencymark.fixed.little <= 15 msec
latencymark.dynamic.little <= 50 msec
The corresponding SynthMark results are:
voice.mark.little = 73.4724
voice.mark.big = 276.446
audio.latency.msec.2.18.18 = 4
audio.latency.msec.2.18.54 = 8
LatencyMark does a linear search, which can be very tedious.
Go up by powers of two then do binary search once bracketed.
LatencyMark: #8, 10 seconds with bursts = 128 -----------
LatencyMark: detected glitch at 2.97 seconds
ERROR - at maximum buffer size and still glitching
Got LatencyMark = 0.0 msec
CDD_SUMMARY_BEGIN -------
voicemark.90 = 67.6868
latencymark.fixed.little = 2
latencymark.dynamic.little = 0
CDD_SUMMARY_END --------
It measures "latency.light.big = 96", which is confusing.
The 0 result for "latencymark.dynamic.little = 0" is because it hit the maximum latency.
Originally posted by @philburk in #86 (comment)
It should report some high maximum value like 512 msec.
The GUI app allows us to run SynthMark on unrooted phones.
On https://github.com/google/synthmark/blob/master/docs/README.md the link titled "Synthesizer Architecture" links to: https://github.com/google/synthmark/blob/master/docs/Synthesizer.md which does not exist.
This would involve storing all important timing information in a compact format.
It can be dumped to a VCD format file and viewed using a VCD waveform viewer.
Store:
task awake/asleep
buffer depth
cpu index
Dump:
synthesized ideal burst timing so we can see when a task is late
Some results appear in command line but not on app.
Too many ways to log in SynthMark
mLogTool->log() writes to an std::ostream or printf if ostream not specified. Prefixed with "[log]".
Prefix did not appear in app because of a bug.
SynthMarkResult has a std::string that gets appended to by various tests and then printed at the end. Appears in log after test is complete.
mLogTool->log(mResult.getResultMessage().c_str());
printf used for errors and some logs
printf("allNotesOn(%d) exceeded maxVoices of %d\n", numVoices, mMaxVoices);
Open /proc file for each CPU at start of benchmark.
Read frequency periodically, accumulate in time slots.
Display "heat map" ASCII chart that includes CPU index.
The new code that uses strtol() is causing good command line arguments to fail.
REPRODUCE
On S9 running Android OC_MR1, enter:
adb shell synthmark -tv -p90
It will reject the "90" as invalid!
This works fine on Android Pie..
Here is the commit for the code that is failing.
a93ca11
We should test on Pixel running OC_MR1 to see if it is specific to Samsung.
Use Monsoon power meter
Measure VoiceMark 95 then divide that range by 10 steps and run UtilizationMark.
So we can refer users to a specific version.
Add app version to the reports for Share.
Steps to repro:
android
subdirectory in Android Studio (tried both 3.4 canary 4 and 3.2.1)Expected result:
Actual result:
This will make it easier to see when there is a big gap, for example when it jumps to 50 msec.
CPUs may be split 2+2, 2+4, 4+4 or 2+6, etc.
Benchmark each CPU individually.
https://en.wikipedia.org/wiki/ARM_big.LITTLE
On a 4 core device, run SynthMark and click Settings.
Fader allows one to select CPU 0-7. Expect only CPU 0-3.
Selecting CPU 4-7 returns error -22 from setCpuAffinity().
This is so that touch boost will not affect the performance result by raising the CPU clock frequency.
Ideally the test should be run after the touch boost has timed out.
On some Android systems, some cores are reserved for the foreground process.
Background processes should avoid them and foreground processes can use them.
We should call Process.getExclusiveCores() and then display the resulting int array.
https://developer.android.com/reference/android/os/Process?hl=ml#getExclusiveCores()
If you enter -nx then it will just set n to zero with no error message.
It should print an error message and abort.
Maybe -od for optimize using debug
-on for no optimization (default)
-xd for experimental Deadline
After a glitch happens in LatencyMark, the experiment restarts, but the experiment is not taken back to a clean status.
HostCpuManagerBase, for example, keeps the last workUnits and CustomCpuManager keeps some time measurements that may be screwed.
Does it make sense to restore the system to an initial state when the experiment is restarted?
cat /dev/stune/rt/schedtune.boost
I ran the SynthMark App latency test on Android device with 6/74 voices. I expected a high latency because it was a changing workload. But I measured a latency of only 6 msec!
But then I built SynthMark using the NDK and ran it as a command line executable.
adb shell synthmark -tl -n5 -N75 -c1
That measured a latency of 112 msec!
I think the difference between the app and the command line may be because:
Run these two commands
synthmark -tj -c3 -n2
synthmark -tj -c3 -n2 -N60
Expect very different result.
Actually they are about the same.
Return error or implement -N option.
Model it after the way that Rick ORang Loopback works.
Python code to drive Rick ORang is in google3 "android/media/performance/audio_loopback_usb_test.py"
Currently it can only be run from the command line using -ta
Create a scripts folder
Getting the following linker error when building for Android:
undefined reference to `HostCpuManager::mWorkloadHintsEnabled'
Add a new test that gives a clear measure of the time to recover when the work load increases. LatencyMark restarts the test with a bigger buffer when there is a glitch. Add a test that just measures how long it takes to render in real-time again after a workload increase.
Could be the number of wakeups that would cause an underrun with a single buffer.... or...
Running SynthMark from the commend line is often not an option.
So we need to control the APK by sending an Intent.
Please add some more steps to the instructions here:
https://github.com/google/synthmark/blob/master/scripts/android/autopower/README.md
Let's make it very easy.
Please describe connection of each cable, including power and USB.
Please also describe commands for applying voltage.
Every step from start to finish.
Oboe has a tool called StabilizedCallback that forces a consistent CPU utilization by running a fake load. We could measure various fake load techniques in SynthMark.
REPRODUCE
adb shell synthmark -tj -n16 -w0
adb shell synthmark -tj -n16 -w1
EXPECT
Similar render times because load is not changing.
ACTUAL
Render times with -n16 are very low indicating high utilization.
Also this is printed:
workUnits=16 requestDLBandwidth=0.900000 [###########################___]
SynthMark initially sets the utilization very high (90%) so that it can calibrate the workload. But it does not lower the utilization until the workload changes. After the initial calibration, it could check to see if the utilization is set too high and lower it.
It would be nice to see the progress of the test.
But we do not want to print from the real-time thread because
it would interfere with the timing.
So we should write the logs to a FIFO and pull from it and print in a background thread.
Threads
The Q CDD may recommend specific values. These should be made more obvious.
Allow SynthMark app to run in background by not stopping it. Use a Fragment?
When the app is launched from an Intent, that should be enabled so we can run other apps in front of SynthMark.
In some cases the slower CPU is identified as "big".
And vice versa.
What is the best way to do this?
I noticed that because of
synthmark/source/tools/LatencyMarkHarness.h
Line 109 in 14859cb
It doesn't really matter the result of this this condition
We will always use the STATE_RAMP_UP approach.
So we can see how the CPU governor behaves
Please add the following, or something similar, taken from ag/7201451/2..3
---- cut here ---
Explanation of benchmarks:
The voicemark.90 is a measure of how much computation the device is
capable of when running at 90% of its maximum CPU load.
A “voice” is a unit of computation based on a two oscillator
subtractive synthesizer that is common in plugins and sound formats
such as Downloadable Sounds (DLS) and Sound Font (SF2). It is roughly
equivalent to a “Mini Moog” synthesizer.
The number of voices was chosen because that is enough polyphony for
a decent keyboard app with voices that release slowly. It allows for
poly-timbral apps that combines multiple types of sound, such as drums
and pads. Note that many phones can play > 200 voices so this number
should be easily achieved.
The latencymark.fixed.little value gives us the minimum latency of the
top level buffer closest to the app for a constant workload. The DSP
and other lower level components might add 4-6 msec of latency when
using EXCLUSIVE MMAP mode in AAudio. This would give us a total output
latency of around 20 msec.
The latencymark.dynamic.little measures latency for a changing workload,
which is common in interactive applications. Higher latency is required
because the scheduler may not increase the clock rate rapidly enough to
meet the changing demand.
(Note that the latency values are for the top level of a virtual audio
device with 2 msec buffers. So SynthMark can achieve lower latency values
than the actual audio framework with multiple levels and a DSP.)
Google is disabling Travis today.
When launched from an Intent, it did a linear search instead of the modified binary search.
So it is taking longer to run the test.
We just need this in the app. We can always control-C the command line version.
It should call its superclass.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.