Comments (11)
While waiting for an answer, I've checked the code better and managed to answer some of my questions. Plus, I got new ones.
This is what I've learned:
- Using a
[0.25, 0.5, 0.25]
window for downsampling from 48k to 24k is indeed cheap, but it leads to a low pass filter with no steep decay; however, this should be ok for pitch estimation. - The autocorrelation with a maximum lag of 4 is computed since the pitch is estimated not on the audio frames directly, but on their LP residuals.
- The estimated inverse filter coefficients used to compute the LP residuals are corrected assuming -40 dBFS noise floor;
ac[0] *= 1.0001f
accounts for the reduction done inac[i] -= ac[i]*(.008f*i)*(.008f*i)
; the correction factors are modeled as such assuming white noise, the correlation of which follows the(0.008/i)^2
law. pitch_downsample()
performs both the 2x decimation and the LP residual extraction, whereaspitch_search()
finds the best and the second best pitch candidates - which are further refined inremove_doubling()
.
My remaining questions are the following:
- The pitch estimation is performed at 12 kHz and 24 kHz; is it truly needed? Could one use a lower sample rate?
- Why is
pitch_search()
called passingPITCH_MAX_PERIOD-3*PITCH_MIN_PERIOD
instead ofPITCH_MAX_PERIOD
for themax_pitch
argument? - In
find_best_pitch()
the numerators arexcorr[i]^2
, why not justxcorr[i]
? remove_doubling()
usessecond_check[16] = {0, 0, 3, 2, 3, 2, 5, 2, 3, 2, 3, 2, 5, 2, 3, 2}
, it looks like these are multipliers used to look for specific (sub)harmonics; where do those values come from?remove_doubling()
looks at higher harmonics of the initial estimated pitch period, hence removing pitch period doubling errors; why there's no removal for halving errors?
Alessio
from rnnoise.
@alebzk the pitch estimation code is mostly copied from Opus, so it's quite possible everything in there isn't optimal for rnnoise. The idea as you seem to have guessed is to compute the auto-correlation on the residual. The (.008fi)(.008f*i) term here is means to approximate a Gaussian for lag windowing (stabilizes the LPC analysis). As for the c1 term, it's meant to convolve the LPC filter with a slight low-pass filter to make the analysis better. I'm sure it would be possible to run the pitch even lower than 12 kHz, but that seemed fine for the use I had.
Why is pitch_search() called passing PITCH_MAX_PERIOD-3*PITCH_MIN_PERIOD instead of PITCH_MAX_PERIOD for the max_pitch argument?
This avoids searching for very short periods, which can sometimes cause false detection due to formants. The very short periods are searched through remove_doubling()
In find_best_pitch() the numerators are xcorr[i]^2, why not just xcorr[i]
Yes, because we want to maximize xy/sqrt(xx*yy) but rather than compute a sqrt(), we just square everything and since the xx term is constant, we only need to maximize xy^2/yy
remove_doubling() uses second_check[16] = {0, 0, 3, 2, 3, 2, 5, 2, 3, 2, 3, 2, 5, 2, 3, 2}, it looks like these are multipliers used to look for specific (sub)harmonics; where do those values come from?
They were hand-tuned to look for expected peaks that wouldn't be expected for a different pitch period. For example, if there's a peak at T/6, then we can expect one at 5*T/6, which is a position you wouldn't expect to find a peak if the period was T/3.
remove_doubling() looks at higher harmonics of the initial estimated pitch period, hence removing pitch period doubling errors; why there's no removal for halving errors?
Period doubling (or tripling) is a common error for auto-correlation-based pitch estimators since if there's a periodicity at T, then there's also going to be a periodicity at 2T and 3T, ... OTOH, there won't be a periodicity at T/2.
from rnnoise.
Hi again,
It looks like the pitch estimation method you implemented is SIFT [1]. Right?
If there are other relevant details on the exact method, could you explain and/or share a reference?
Alessio
[1] Markel, J. (1972). The SIFT algorithm for fundamental frequency estimation. IEEE Transactions on Audio and Electroacoustics, 20(5), 367-377.
from rnnoise.
Thanks so much for the detailed answer!
from rnnoise.
hi,@alebzk:
from your issue,I learned a lot,I am new to pitch detection.If my training sample's sample rate is 16000,is the macro define PITCH_BUF_SIZE need to change?What is the relation between the sample rate and the PITCH_BUF_SIZE?
expecting your reply,Thanks!
from rnnoise.
Hi @zhly0,
@jmvalin can surely say more, but I'm happy to give a first answer.
Yes. PITCH_BUF_SIZE unit is number of samples at 48 kHz; hence, it has to be adapted if the sample rate changes. However, note that the whole pitch estimation algorithm is kind of specific for the 48k case (there are 2 downsampling steps and the estimated pitch is used to compute spectral the cross-correlation features). If you want to avoid resampling from 16k to 48k, then part of the code must be adapted.
Alessio
from rnnoise.
@alebzk thanks for your write up, very helpful (altough im just half way through). (and thanks to jmvalin ofcourse!)
I dont understand your comment: "The autocorrelation with a maximum lag of 4 is computed since the pitch is estimated not on the audio frames directly, but on their LP residuals." I dont understand where the residual is calculated (residual as in difference between x_lp and lpcestimated).
Are you referring to:
opus_val32 sum = SHL32(EXTEND32(x[i]), SIG_SHIFT)
?
Thanks again
from rnnoise.
See this call:
celt_fir5(x_lp, lpc2, x_lp, len>>1, mem);
in pitch_downsample.c
from rnnoise.
@jmvalin thanks! I have gone through that function plenty of times but double checked now when you wrote, and I think my problem comes from the fact that lpc coefficients are defined with opposite sign compared to how I thought they were.
I now assume that you define them as e.g. Matlab do: https://se.mathworks.com/help/signal/ref/lpc.html.
Thanks.
from rnnoise.
@jmvalin @alebzk I do not understand the pitch estimation method,especially, the "pitch_search" function in pitch.c. can you share me some details?
from rnnoise.
@shakingWaves @zhly0 @jmvalin @alebzk
I also do not understand the pitch estimation method, If my training sample's sample rate is 16000, how to change pitch_downsample,pitch_search,remove_doubling those funtions?
Thanks!
from rnnoise.
Related Issues (20)
- Noise data
- About the result
- Questions About Training
- Potential underflow / overflow when converting from float to int16 HOT 1
- What is the range of the input data?
- About memory HOT 2
- rnnoise not filtering this specific noise
- file generated unreadable HOT 3
- Data imbalance problem for Voice Activity Detection output block
- About rnnoise_demo output HOT 1
- New model and default branch has changed from "master" to "main" HOT 4
- Releasing RNNoise version 0.2 HOT 15
- RNNoise v0.2 does not compile on Mac and Linux HOT 8
- Possible to support basic (non-VSX) Altivec?
- Why is a biquad filter needed before denoise? HOT 1
- reference values of training loss
- Control the size of model HOT 1
- Enable Discussions feature and convert "issues" that are not issues to a discussion
- Comparison of lite vs full model in 0.2.0
- Training with already Clean and Noisy Data HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rnnoise.