krakenrf / heimdall_daq_fw Goto Github PK
View Code? Open in Web Editor NEWCoherent data acquisition signal processing chain for multichannel SDRs
License: GNU General Public License v3.0
Coherent data acquisition signal processing chain for multichannel SDRs
License: GNU General Public License v3.0
Short description
If rtl_daq.c
is not able to push data downstream fast enough, data reads and writes on the same memory sections can occur simultaneously, leading to undefined behaviour and incoherent data between the receiver channels
Steps to reproduce
Unknown, as I observed this on my own fork of the software, running on a RPi 4 and with added sample export (which will have an impact on performance). However, this should always happen if the performance is not high enough for rtl_daq.c to process data in real time. While I cannot say if anybody else will encounter this issue, I still suggest implementing at least my workaround as to avoid silent undefined behaviour.
Explanation
rtl_daq.c
maintains the circular buffer rtl_rec
, which contains NUM_BUFF
(8) slots to which rtlsdrCallback can write, and from which the infinite while-loop in main
reads. The index of the "slot" to which is written is determined by calculating
300 int wr_buff_ind = rtl_rec->buff_ind % NUM_BUFF; // Calculate current buffer index in the circular buffer
and the index of the slot from which is read is calculated by calculating
671 rd_buff_ind = read_buff_ind % NUM_BUFF;
where rtl_rec->buff_ind and read_buff_ind are two counters, that count how many packets are read from the device and how many are sent downstream, respectively.
There are checks (ll. 600-605) which aim to ensure that data is not read from the same slot by enforcing rtl_rec->buff_ind <= read_buff_ind
, however if rtl_rec->buff_ind
is an integer multiple of NUM_BUFF
larger than read_buff_ind
, this check will not fail, even though the modulo function will refer to the same slot in the circular buffer. In this case, rtlsdrCallback
may write data while main
is reading.
Proposed solution
I believe there is no "clean" solution to this issue that does not cause data loss. Increasing NUM_BUFF
(potentially dynamically to use most of the available resources?) will decrease the likelyhood of this occuring, however at some point data must be dropped if some part of the toolchain is not able to keep up.
Instead, I propose to ensure that no simultaneous read and write occures and issue a warning in this case: Add a second check to the for-loop in lines 600-605:
if(rtl_rec->buff_ind % NUM_BUFF == read_buff_ind % NUM_BUFF) {
log_warn("Likely race condition. Skipping data aquicision. RTL Buff index: %d, read_buff_ind: %d", rtl_rec->buff_ind, read_buff_ind);
data_ready = 0;
break;
}
This will wait until the buffer "slots" are unequal. It might also be preferable to "fast-forward" read_buff_ind
(i.e. set it to rtl_rec->buff_ind
?) to indicate a data loss downstream due to the jump. However, I have no clue if this might cause other issues.
21:30:04 WARN rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3089, read_buff_ind: 3047
21:30:04 WARN rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3089, read_buff_ind: 3048
21:30:04 WARN rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3090, read_buff_ind: 3048
21:30:04 WARN rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3090, read_buff_ind: 3048
21:30:04 WARN rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3090, read_buff_ind: 3048
21:30:04 WARN rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3089, read_buff_ind: 3048
21:30:04 WARN rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3090, read_buff_ind: 3049
I'm trying to diagnose an issue where the software crashes when making adjustments to my KrakenSDR and set the log level to 2. rtl_daq.log keeps filling up with these warnings. I'm not sure if that's normal or related to my problems. Running the latest pi image on a 2GB RPi 4.
Description:
The three Shell scripts daq_start_sm.sh daq_synthetic_start.sh unit_test.sh
redirect some error messages to /dev/NULL. This file does not exist in Ubuntu, and in no other operating system I know of. Windows uses NUL but that's not meant here.
The null device in Ubuntu is /dev/null
.
Current Behaviour:
A redirection to /dev/NULL
will fail with an error message unless called with root-privileges. If called as root, a new text file will be created (or an existing one overwritten), which, while not causing huge issues, is not desired.
Resolution:
Change all redirections to /dev/null
.
Would you have examples for using GNU Radio to build a radio astronomy interferometer using just two of the KrakenSDR RTL-SDR coherent antennas? I've started working in GNU Radio but from my understanding the KrakenSDR isn't naturally coherent like the USRP B210 SDR module.
Any help would or examples would help. Thank you!
I'm interested in doing some work to clean up the KrakenSDR tools to make them less dependant on running on a single-function system and more palatable for distro packaging (or just running on a regular laptop without stomping over distro-provided components). Would you be receptive to PRs making these kinds of changes?
My rough plan is to start with heimdall_daq:
Do you plan to get the librtlsdr changes upstreamed to remove the need for a patched version? I'm not sure about the performance cost of making whether to use zero-copy or not runtime configurable, off the top of my head that's the only bit that looks problematic.
When running this install script in process, I get the output below the script. Ubuntu 22.04 LTS install. (Looking for alternatives to Raspi4.) Output files attached as well.
cd
git clone https://github.com/krakenrf/kfr
cd kfr
mkdir build
cd build
cmake -DENABLE_CAPI_BUILD=ON -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Release ..
make
The CXX compiler identification is Clang 14.0.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - failed
-- Check for working CXX compiler: /usr/bin/clang++
-- Check for working CXX compiler: /usr/bin/clang++ - broken
CMake Error at /usr/share/cmake-3.22/Modules/CMakeTestCXXCompiler.cmake:62 (message):
The C++ compiler
"/usr/bin/clang++"
is not able to compile a simple test program.
It fails with the following output:
Change Dir: /home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeTmp
Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_8e768/fast && /usr/bin/gmake -f CMakeFiles/cmTC_8e768.dir/build.make CMakeFiles/cmTC_8e768.dir/build
gmake[1]: Entering directory '/home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_8e768.dir/testCXXCompiler.cxx.o
/usr/bin/clang++ -MD -MT CMakeFiles/cmTC_8e768.dir/testCXXCompiler.cxx.o -MF CMakeFiles/cmTC_8e768.dir/testCXXCompiler.cxx.o.d -o CMakeFiles/cmTC_8e768.dir/testCXXCompiler.cxx.o -c /home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
Linking CXX executable cmTC_8e768
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_8e768.dir/link.txt --verbose=1
/usr/bin/clang++ CMakeFiles/cmTC_8e768.dir/testCXXCompiler.cxx.o -o cmTC_8e768
/usr/bin/ld: cannot find -lstdc++: No such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[1]: *** [CMakeFiles/cmTC_8e768.dir/build.make:100: cmTC_8e768] Error 1
gmake[1]: Leaving directory '/home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeTmp'
gmake: *** [Makefile:127: cmTC_8e768/fast] Error 2
CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
CMakeLists.txt:19 (project)
-- Configuring incomplete, errors occurred!
See also "/home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeOutput.log".
See also "/home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeError.log".
Receiving a numerous amount of these when running the example below:
Example 3 - Cross Correlation Sample and Phase Coherent Check
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
Sending control message
Control interface communication finished
Reconfiguration succesfully finished
Sending control message
Control interface communication finished
Reconfiguration succesfully finished
Sending control message
Control interface communication finished
Reconfiguration succesfully finished
/usr/local/lib/python3.10/dist-packages/gnuradio/krakensdr/krakensdr_correlator.py:46: RuntimeWarning: divide by zero encountered in log10
x_corr_plot = 10*np.log10(np.abs(x_corr))
Failed to put IQ Samples into the Queue
Exception:
I encountered a slight error while using the make
command, and I hope to receive some advice.
The following is the output after using make
:
gcc -Wall -std=gnu99 -march=native -O2 -I. -c -o ini.o ini.c
gcc -Wall -std=gnu99 -march=native -O2 -I. -c -o log.o log.c
gcc -Wall -std=gnu99 -march=native -O2 -I. -c -o iq_header.o iq_header.c
gcc -Wall -std=gnu99 -march=native -O2 -I. -c -o sh_mem_util.o sh_mem_util.c
gcc -Wall -std=gnu99 -march=native -O2 -I. log.o ini.o iq_header.o -o rtl_daq.out rtl_daq.c -lpthread -lzmq -L. -lrtlsdr -lusb-1.0
gcc -Wall -std=gnu99 -march=native -O2 -I. sh_mem_util.o log.o ini.o iq_header.o -o rebuffer.out rebuffer.c -lrt -lm
gcc -Wall -std=gnu99 -march=native -O2 -I. sh_mem_util.o log.o ini.o iq_header.o -o iq_server.out iq_server.c -lrt
gcc -Wall -std=gnu99 -march=native -O2 -I. -c fir_decimate.c -o fir_decimate.o
gcc -Wall -std=gnu99 -march=native -O2 -I. fir_decimate.o sh_mem_util.o log.o ini.o iq_header.o -o decimate.out -lrt -lkfr_capi
/usr/bin/ld: error: cannot find -lkfr_capi
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_allocate'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_allocate'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_allocate'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_allocate'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_filter_create_fir_plan_f32'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_filter_process_f32'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_filter_process_f32'
collect2: error: ld returned 1 exit status
make: *** [Makefile:38: decimate_x86] Error 1
The error seems to indicate that it is related to KFR, but I have successfully installed the KFR project.
Any suggestion will be appreciated.
Running sudo ./Firmware/daq_start_sm.sh (as per the instructions) fails because fir_filter_designer.py does not exist:
sudo ./Firmware/daq_start_sm.sh
Config file check bypassed [ WARNING ]
kernel.sched_rt_runtime_us = -1
awk: cannot open daq_chain_config.ini (No such file or directory)
mkfifo: cannot create fifo '_data_control/fw_decimator_in': No such file or directory
mkfifo: cannot create fifo '_data_control/bw_decimator_in': No such file or directory
mkfifo: cannot create fifo '_data_control/fw_decimator_out': No such file or directory
mkfifo: cannot create fifo '_data_control/bw_decimator_out': No such file or directory
mkfifo: cannot create fifo '_data_control/fw_delay_sync_iq': No such file or directory
mkfifo: cannot create fifo '_data_control/bw_delay_sync_iq': No such file or directory
mkfifo: cannot create fifo '_data_control/fw_delay_sync_hwc': No such file or directory
mkfifo: cannot create fifo '_data_control/bw_delay_sync_hwc': No such file or directory
python3: can't open file '/home/rich/krakensdr_pr/heimdall_daq_fw/fir_filter_designer.py': [Errno 2] No such file or directory
DAQ chain not started!
I use docker image https://github.com/godsic/krakensdr-containers/ . In container I run $ ./daq_synthetic_start.sh
and get the error:
Traceback (most recent call last):
File "/root/krakensdr/heimdall_daq_fw/Firmware/_testing/test_data_synthesizer.py", line 246, in <module>
FIFO_rd_thread_inst0 = FIFO_rd_thread(M)
File "/root/krakensdr/heimdall_daq_fw/Firmware/_testing/test_data_synthesizer.py", line 66, in __init__
self.ctr_fifo_descriptor = open("_data_control/rec_control_fifo", 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '_data_control/rec_control_fifo'
Can you help to solve this problem? Thanks.
Hello all,
works great on Pi4, trying to install on a NUC Celeron 2 core with installation script
wget https://raw.githubusercontent.com/krakenrf/krakensdr_docs/main/install_scripts/krakensdr_x86_install_doa_pr.sh
sudo chmod +x krakensdr_x86_install_doa_pr.sh
./krakensdr_x86_install_doa_pr.sh
Goes to the end (after 3h), but launching gives
root@debian:/krakensdr_doa# ./krakensdr_doa_start.sh/krakensdr_doa# ./kraken_doa_start.sh
-bash: ./krakensdr_doa_start.sh: No such file or directory
root@debian:
./kraken_doa_start.sh: line 4: conda: command not found
./kraken_doa_start.sh: line 5: conda: command not found
Shut down DAQ chain ..
Config file check bypassed [ WARNING ]
kernel.sched_rt_runtime_us = -1
Traceback (most recent call last):
File "/root/krakensdr_doa/heimdall_daq_fw/Firmware/fir_filter_designer.py", line 16, in
from scipy import signal
ModuleNotFoundError: No module named 'scipy'
DAQ chain not started!
Starting KrakenSDR Direction Finder
Web Interface Running at 0.0.0.0:8080
PHP Data Out Server Running at 0.0.0.0:8081
No web page reachable at port 8080 ...
Why are shown some py stuff ?
Thank you for your great job,
Regards,
François
split from prior issue
from decimator.log
double free or corruption (out)
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007eff478e7c73 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2 0x00007eff47897986 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x00007eff478817f4 in __GI_abort () at abort.c:79
#4 0x00007eff478dbd5e in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7eff47a1387c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#5 0x00007eff478f195c in malloc_printerr (str=str@entry=0x7eff47a16450 "double free or corruption (out)") at malloc.c:5659
#6 0x00007eff478f3a80 in _int_free (av=0x7eff47a51c80 <main_arena>, p=0x7eff1a374030, have_lock=<optimized out>) at malloc.c:4583
#7 0x00007eff478f6103 in __GI___libc_free (mem=<optimized out>) at malloc.c:3386
#8 0x00000000004016c0 in main ()
recompiling with -g:
#8 0x00000000004016c0 in main (argc=<optimized out>, argv=<optimized out>) at fir_decimate.c:347
corresponding to near end of main(): free(fir_input_buffer_i);
Line number likely not exact to your repo due to additional log_info()
calls added for debug while trying to diagnose.
Earlier #if ARM_NEON
switches between using malloc()
in the true path and kfr_allocate()
in the non ARM_NEON path.
Wrapping the free()'s in similar #if ARM_NEON directives seems to resolve the issue.
#ifdef ARM_NEON
free(fir_input_buffer_i);
free(fir_input_buffer_q);
free(fir_output_buffer_i);
free(fir_output_buffer_q);
#else // X86
kfr_release( fir_input_buffer_i);
kfr_release( fir_input_buffer_q);
kfr_release( fir_output_buffer_i);
kfr_release( fir_output_buffer_q);
#endif
Seems like an easy fix, only shows up as the application is exiting though so mostly no impact other than potential red herring when dealing with an unexpected termination.
Following the manual build instructions sets up a conda activate kraken
environment on my user account. That seems to get through the compile and then sudo daq_start_sm.sh looks like it runs, but once I realize the _logs directory is where the output logs are I see that much of that script failed. The sudo removes the conda environment and
-Wl,-rpath='$$ORIGIN'
to CFLAGS in Makefilecorruption trap and fix split to seperate issue noted in comment below
16:21:47 INFO fir_decimate.c:121: Decimation ratio: 1
16:21:47 INFO fir_decimate.c:122: CPI size: 1048576
16:21:47 INFO fir_decimate.c:123: Calibration sample size : 65536
16:21:47 INFO fir_decimate.c:147: Input shared memory interface succesfully ini
tialized
16:21:48 INFO fir_decimate.c:161: Output shared memory interface succesfully in
itialized
16:21:48 INFO fir_decimate.c:212: FIR filter coefficienrs are initialized, tap
size: 1
double free or corruption (out)
Maybe I'll manage to catch it under gdb later, but some of this weekend salvaged for other activities.
When setting the decimation ratio to anything > 1, the log file of the decimator module will flood messages such as this one:
WARN fir_decimate.c:281: Frame index missmatch. Expected 511 <--> 504 Received
where the supposed mismatch (here 511-504) is exactly (decimation - 1). I'm pretty sure this is a bug in fir_decimate.c, l. 241:
expected_frame_index += dec;
Which should be += 1 (or completely removed).
The reason I believe this is an incorrect line is that as per my understanding, the decimation will be done in the fir_decimate.c file - checking that the INCOMING index increased by steps of [decimation] does not make any sense to me - as the upstream is not "aware" of any decimation going on downstream. In addition, apart from writing a warning message, this check does not seem to do anything else.
This check might be useful if within the rebuffer-module packets are repacked, however this is not controlled by the decimation ratio and if such a test was intended, then probably some check based on the relationship between daq_block_size and cpi_size*decimation_ratio should be used?
This returns value with leading space:
https://github.com/krakenrf/heimdall_daq_fw/blob/main/Firmware/daq_start_sm.sh#L24
So this check will not pass:
https://github.com/krakenrf/heimdall_daq_fw/blob/main/Firmware/daq_start_sm.sh#L116
This seem to do the job:
out_data_iface_type=$(awk -F'=' '/out_data_iface_type/ {gsub (" ", "", $0); print $2}' daq_chain_config.ini)
Can i use this firmware in other sdr like hackrf or pluto and what the modification that i need?
Thanks
Hi,
I'm using the KrakenSDR with gr-krakensdr. I have set the gains in the GNURadio block, and also in the daq_chain_config.ini.
However, in the below DAB radio recording, during the null symbol (no signal), the signal is automatically amplified. Is there a way to disable this behaviour, as it is affecting my signal processing?
Thank you.
The expected output with a normal RTL-SDR is below.
[meta]
ini_version = 7
config_name = kraken_default
[hw]
name = kraken5
unit_id = 0
ioo_type = 0
num_ch = 5
en_bias_tee = 0,0,0,0,0
[daq]
log_level = 5
daq_buffer_size = 262144
center_freq = 700000000
sample_rate = 2048000
gain = 496
en_noise_source_ctr = 1
ctr_channel_serial_no = 1000
[pre_processing]
cpi_size = 1048576
decimation_ratio = 1
fir_relative_bandwidth = 1.0
fir_tap_size = 1
fir_window = hann
en_filter_reset = 0
[calibration]
corr_size = 65536
std_ch_ind = 0
en_iq_cal = 1
amplitude_cal_mode = channel_power
en_gain_tune_init = 0
gain_lock_interval = 0
unified_gain_control = 0
require_track_lock_intervention = 0
cal_track_mode = 2
cal_frame_interval = 687
cal_frame_burst_size = 10
amplitude_tolerance = 2
phase_tolerance = 1
maximum_sync_fails = 10
iq_adjust_source = explicit-time-delay
iq_adjust_amplitude = 0,0,0,0
iq_adjust_time_delay_ns = 0, 0, 0, 0
[adpis]
en_adpis = 0
adpis_proc_size = 8192
adpis_gains_init = 496,496,496,496,496
[data_interface]
out_data_iface_type = eth
Are there any plans for x86_64 flatpak package? Or would y'all be up for a pull request with a flatpak build scripts?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.