krakenrf / heimdall_daq_fw Goto Github PK

Short description

If rtl_daq.c is not able to push data downstream fast enough, data reads and writes on the same memory sections can occur simultaneously, leading to undefined behaviour and incoherent data between the receiver channels

Steps to reproduce
Unknown, as I observed this on my own fork of the software, running on a RPi 4 and with added sample export (which will have an impact on performance). However, this should always happen if the performance is not high enough for rtl_daq.c to process data in real time. While I cannot say if anybody else will encounter this issue, I still suggest implementing at least my workaround as to avoid silent undefined behaviour.

Explanation

rtl_daq.c maintains the circular buffer rtl_rec, which contains NUM_BUFF (8) slots to which rtlsdrCallback can write, and from which the infinite while-loop in main reads. The index of the "slot" to which is written is determined by calculating

300 int wr_buff_ind = rtl_rec->buff_ind % NUM_BUFF; // Calculate current buffer index in the circular buffer

and the index of the slot from which is read is calculated by calculating

671 rd_buff_ind = read_buff_ind % NUM_BUFF;

where rtl_rec->buff_ind and read_buff_ind are two counters, that count how many packets are read from the device and how many are sent downstream, respectively.

There are checks (ll. 600-605) which aim to ensure that data is not read from the same slot by enforcing rtl_rec->buff_ind <= read_buff_ind, however if rtl_rec->buff_ind is an integer multiple of NUM_BUFF larger than read_buff_ind, this check will not fail, even though the modulo function will refer to the same slot in the circular buffer. In this case, rtlsdrCallback may write data while main is reading.

Proposed solution
I believe there is no "clean" solution to this issue that does not cause data loss. Increasing NUM_BUFF (potentially dynamically to use most of the available resources?) will decrease the likelyhood of this occuring, however at some point data must be dropped if some part of the toolchain is not able to keep up.

Instead, I propose to ensure that no simultaneous read and write occures and issue a warning in this case: Add a second check to the for-loop in lines 600-605:

if(rtl_rec->buff_ind % NUM_BUFF == read_buff_ind % NUM_BUFF) { 
    log_warn("Likely race condition. Skipping data aquicision. RTL Buff index: %d, read_buff_ind: %d", rtl_rec->buff_ind, read_buff_ind);           
    data_ready = 0;
    break;
}                                                                  

This will wait until the buffer "slots" are unequal. It might also be preferable to "fast-forward" read_buff_ind (i.e. set it to rtl_rec->buff_ind?) to indicate a data loss downstream due to the jump. However, I have no clue if this might cause other issues.

21:30:04 WARN  rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3089, read_buff_ind: 3047
21:30:04 WARN  rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3089, read_buff_ind: 3048
21:30:04 WARN  rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3090, read_buff_ind: 3048
21:30:04 WARN  rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3090, read_buff_ind: 3048
21:30:04 WARN  rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3090, read_buff_ind: 3048
21:30:04 WARN  rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3089, read_buff_ind: 3048
21:30:04 WARN  rtl_daq.c:608: Circular buffer may overrun. Consider increasing the number of buffers. RTL Buff index: 3090, read_buff_ind: 3049

I'm trying to diagnose an issue where the software crashes when making adjustments to my KrakenSDR and set the log level to 2. rtl_daq.log keeps filling up with these warnings. I'm not sure if that's normal or related to my problems. Running the latest pi image on a 2GB RPi 4.

Description:

The three Shell scripts daq_start_sm.sh daq_synthetic_start.sh unit_test.sh redirect some error messages to /dev/NULL. This file does not exist in Ubuntu, and in no other operating system I know of. Windows uses NUL but that's not meant here.

The null device in Ubuntu is /dev/null.

Current Behaviour:

A redirection to /dev/NULL will fail with an error message unless called with root-privileges. If called as root, a new text file will be created (or an existing one overwritten), which, while not causing huge issues, is not desired.

Resolution:

Change all redirections to /dev/null.

Would you have examples for using GNU Radio to build a radio astronomy interferometer using just two of the KrakenSDR RTL-SDR coherent antennas? I've started working in GNU Radio but from my understanding the KrakenSDR isn't naturally coherent like the USRP B210 SDR module.

Any help would or examples would help. Thank you!

I'm interested in doing some work to clean up the KrakenSDR tools to make them less dependant on running on a single-function system and more palatable for distro packaging (or just running on a regular laptop without stomping over distro-provided components). Would you be receptive to PRs making these kinds of changes?

My rough plan is to start with heimdall_daq:

• Clean up build process
• Vendor in the patched support libraries librtlsdr, Ne10, KFR (don't install to system library locations)
• Patch the various processes used in DAQ to use proper service management under systemd instead of kraken_doa_start.sh just forking them off to the background of a running shell

Do you plan to get the librtlsdr changes upstreamed to remove the need for a patched version? I'm not sure about the performance cost of making whether to use zero-copy or not runtime configurable, off the top of my head that's the only bit that looks problematic.

Hi In krakenrf board between Individual tuner on/off DIP switched and USB Type-C DATA there are 3 identical ic,I want to know here name and what it do do and here datasheet if there

Thanks

When running this install script in process, I get the output below the script. Ubuntu 22.04 LTS install. (Looking for alternatives to Raspi4.) Output files attached as well.
cd
git clone https://github.com/krakenrf/kfr
cd kfr
mkdir build
cd build
cmake -DENABLE_CAPI_BUILD=ON -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Release ..
make

• The CXX compiler identification is Clang 14.0.0
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - failed
  -- Check for working CXX compiler: /usr/bin/clang++
  -- Check for working CXX compiler: /usr/bin/clang++ - broken
  CMake Error at /usr/share/cmake-3.22/Modules/CMakeTestCXXCompiler.cmake:62 (message):
  The C++ compiler

  "/usr/bin/clang++"

  is not able to compile a simple test program.

  It fails with the following output:

  Change Dir: /home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeTmp

  Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_8e768/fast && /usr/bin/gmake -f CMakeFiles/cmTC_8e768.dir/build.make CMakeFiles/cmTC_8e768.dir/build
  gmake[1]: Entering directory '/home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeTmp'
  Building CXX object CMakeFiles/cmTC_8e768.dir/testCXXCompiler.cxx.o
  /usr/bin/clang++ -MD -MT CMakeFiles/cmTC_8e768.dir/testCXXCompiler.cxx.o -MF CMakeFiles/cmTC_8e768.dir/testCXXCompiler.cxx.o.d -o CMakeFiles/cmTC_8e768.dir/testCXXCompiler.cxx.o -c /home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
  Linking CXX executable cmTC_8e768
  /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_8e768.dir/link.txt --verbose=1
  /usr/bin/clang++ CMakeFiles/cmTC_8e768.dir/testCXXCompiler.cxx.o -o cmTC_8e768
  /usr/bin/ld: cannot find -lstdc++: No such file or directory
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  gmake[1]: *** [CMakeFiles/cmTC_8e768.dir/build.make:100: cmTC_8e768] Error 1
  gmake[1]: Leaving directory '/home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeTmp'
  gmake: *** [Makefile:127: cmTC_8e768/fast] Error 2

  CMake will not be able to correctly generate this project.
  Call Stack (most recent call first):
  CMakeLists.txt:19 (project)

-- Configuring incomplete, errors occurred!
See also "/home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeOutput.log".
See also "/home/kd7cao/kfr/build/kfr/build/CMakeFiles/CMakeError.log".

CMakeError.log

CMakeOutput.log

Receiving a numerous amount of these when running the example below:
Example 3 - Cross Correlation Sample and Phase Coherent Check

Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
Sending control message
Control interface communication finished
Reconfiguration succesfully finished
Sending control message
Control interface communication finished
Reconfiguration succesfully finished
Sending control message
Control interface communication finished
Reconfiguration succesfully finished
/usr/local/lib/python3.10/dist-packages/gnuradio/krakensdr/krakensdr_correlator.py:46: RuntimeWarning: divide by zero encountered in log10
x_corr_plot = 10*np.log10(np.abs(x_corr))
Failed to put IQ Samples into the Queue
Exception:

I encountered a slight error while using the make command, and I hope to receive some advice.

The following is the output after using make:

gcc -Wall -std=gnu99 -march=native -O2 -I. -c -o ini.o ini.c
gcc -Wall -std=gnu99 -march=native -O2 -I. -c -o log.o log.c
gcc -Wall -std=gnu99 -march=native -O2 -I. -c -o iq_header.o iq_header.c
gcc -Wall -std=gnu99 -march=native -O2 -I. -c -o sh_mem_util.o sh_mem_util.c
gcc -Wall -std=gnu99 -march=native -O2 -I. log.o ini.o iq_header.o -o rtl_daq.out rtl_daq.c -lpthread -lzmq -L. -lrtlsdr -lusb-1.0
gcc -Wall -std=gnu99 -march=native -O2 -I. sh_mem_util.o log.o ini.o iq_header.o -o rebuffer.out rebuffer.c -lrt -lm
gcc -Wall -std=gnu99 -march=native -O2 -I. sh_mem_util.o log.o ini.o iq_header.o -o iq_server.out iq_server.c -lrt
gcc -Wall -std=gnu99 -march=native -O2 -I. -c fir_decimate.c -o fir_decimate.o
gcc -Wall -std=gnu99 -march=native -O2 -I. fir_decimate.o sh_mem_util.o log.o ini.o iq_header.o -o decimate.out -lrt -lkfr_capi
/usr/bin/ld: error: cannot find -lkfr_capi
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_allocate'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_allocate'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_allocate'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_allocate'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_filter_create_fir_plan_f32'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_filter_process_f32'
fir_decimate.o:fir_decimate.c:function main: error: undefined reference to 'kfr_filter_process_f32'
collect2: error: ld returned 1 exit status
make: *** [Makefile:38: decimate_x86] Error 1

The error seems to indicate that it is related to KFR, but I have successfully installed the KFR project.

Any suggestion will be appreciated.

Running sudo ./Firmware/daq_start_sm.sh (as per the instructions) fails because fir_filter_designer.py does not exist:

sudo ./Firmware/daq_start_sm.sh
Config file check bypassed [ WARNING ]
kernel.sched_rt_runtime_us = -1
awk: cannot open daq_chain_config.ini (No such file or directory)
mkfifo: cannot create fifo '_data_control/fw_decimator_in': No such file or directory
mkfifo: cannot create fifo '_data_control/bw_decimator_in': No such file or directory
mkfifo: cannot create fifo '_data_control/fw_decimator_out': No such file or directory
mkfifo: cannot create fifo '_data_control/bw_decimator_out': No such file or directory
mkfifo: cannot create fifo '_data_control/fw_delay_sync_iq': No such file or directory
mkfifo: cannot create fifo '_data_control/bw_delay_sync_iq': No such file or directory
mkfifo: cannot create fifo '_data_control/fw_delay_sync_hwc': No such file or directory
mkfifo: cannot create fifo '_data_control/bw_delay_sync_hwc': No such file or directory
python3: can't open file '/home/rich/krakensdr_pr/heimdall_daq_fw/fir_filter_designer.py': [Errno 2] No such file or directory
DAQ chain not started!

I use docker image https://github.com/godsic/krakensdr-containers/ . In container I run $ ./daq_synthetic_start.sh and get the error:

Traceback (most recent call last):
  File "/root/krakensdr/heimdall_daq_fw/Firmware/_testing/test_data_synthesizer.py", line 246, in <module>
    FIFO_rd_thread_inst0 = FIFO_rd_thread(M)
  File "/root/krakensdr/heimdall_daq_fw/Firmware/_testing/test_data_synthesizer.py", line 66, in __init__
    self.ctr_fifo_descriptor = open("_data_control/rec_control_fifo", 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '_data_control/rec_control_fifo'

Can you help to solve this problem? Thanks.

Hello all,
works great on Pi4, trying to install on a NUC Celeron 2 core with installation script

wget https://raw.githubusercontent.com/krakenrf/krakensdr_docs/main/install_scripts/krakensdr_x86_install_doa_pr.sh
sudo chmod +x krakensdr_x86_install_doa_pr.sh
./krakensdr_x86_install_doa_pr.sh

Goes to the end (after 3h), but launching gives

root@debian:/krakensdr_doa# ./krakensdr_doa_start.sh
-bash: ./krakensdr_doa_start.sh: No such file or directory
root@debian:/krakensdr_doa# ./kraken_doa_start.sh
./kraken_doa_start.sh: line 4: conda: command not found
./kraken_doa_start.sh: line 5: conda: command not found
Shut down DAQ chain ..
Config file check bypassed [ WARNING ]
kernel.sched_rt_runtime_us = -1
Traceback (most recent call last):
File "/root/krakensdr_doa/heimdall_daq_fw/Firmware/fir_filter_designer.py", line 16, in
from scipy import signal
ModuleNotFoundError: No module named 'scipy'
DAQ chain not started!
Starting KrakenSDR Direction Finder
Web Interface Running at 0.0.0.0:8080
PHP Data Out Server Running at 0.0.0.0:8081

No web page reachable at port 8080 ...

Why are shown some py stuff ?

Thank you for your great job,

Regards,
François

split from prior issue

from decimator.log

double free or corruption (out)

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007eff478e7c73 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007eff47897986 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007eff478817f4 in __GI_abort () at abort.c:79
#4  0x00007eff478dbd5e in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7eff47a1387c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007eff478f195c in malloc_printerr (str=str@entry=0x7eff47a16450 "double free or corruption (out)") at malloc.c:5659
#6  0x00007eff478f3a80 in _int_free (av=0x7eff47a51c80 <main_arena>, p=0x7eff1a374030, have_lock=<optimized out>) at malloc.c:4583
#7  0x00007eff478f6103 in __GI___libc_free (mem=<optimized out>) at malloc.c:3386
#8  0x00000000004016c0 in main ()

recompiling with -g:

#8  0x00000000004016c0 in main (argc=<optimized out>, argv=<optimized out>) at fir_decimate.c:347

corresponding to near end of main(): free(fir_input_buffer_i); Line number likely not exact to your repo due to additional log_info() calls added for debug while trying to diagnose.

Earlier #if ARM_NEON switches between using malloc() in the true path and kfr_allocate() in the non ARM_NEON path.

Wrapping the free()'s in similar #if ARM_NEON directives seems to resolve the issue.

#ifdef ARM_NEON
    free(fir_input_buffer_i);
    free(fir_input_buffer_q);
    free(fir_output_buffer_i);
    free(fir_output_buffer_q);
#else // X86
    kfr_release( fir_input_buffer_i);
    kfr_release( fir_input_buffer_q);
    kfr_release( fir_output_buffer_i);
    kfr_release( fir_output_buffer_q);
#endif

Seems like an easy fix, only shows up as the application is exiting though so mostly no impact other than potential red herring when dealing with an unexpected termination.

Following the manual build instructions sets up a conda activate kraken environment on my user account. That seems to get through the compile and then sudo daq_start_sm.sh looks like it runs, but once I realize the _logs directory is where the output logs are I see that much of that script failed. The sudo removes the conda environment and

• the python scripts are failing due to missing packages.
  • fixed by various sudo dnf install and sudo pip install commands to make packages available in root environment.
• decimator.out fails to find libkfr_capi.so, maybe I missed a step on
  installing that library? (review of installation step problem added in comment blelow)
  • fixed by adding -Wl,-rpath='$$ORIGIN' to CFLAGS in Makefile
  • rebuilding
  • copying the libkfr_capi.so file to the same directory as decimator.out

corruption trap and fix split to seperate issue noted in comment below

16:21:47 INFO  fir_decimate.c:121: Decimation ratio: 1
16:21:47 INFO  fir_decimate.c:122: CPI size: 1048576
16:21:47 INFO  fir_decimate.c:123: Calibration sample size : 65536
16:21:47 INFO  fir_decimate.c:147: Input shared memory interface succesfully ini
tialized
16:21:48 INFO  fir_decimate.c:161: Output shared memory interface succesfully in
itialized
16:21:48 INFO  fir_decimate.c:212: FIR filter coefficienrs are initialized, tap 
size: 1
double free or corruption (out)

When setting the decimation ratio to anything > 1, the log file of the decimator module will flood messages such as this one:

WARN fir_decimate.c:281: Frame index missmatch. Expected 511 <--> 504 Received

where the supposed mismatch (here 511-504) is exactly (decimation - 1). I'm pretty sure this is a bug in fir_decimate.c, l. 241:

expected_frame_index += dec;

Which should be += 1 (or completely removed).

The reason I believe this is an incorrect line is that as per my understanding, the decimation will be done in the fir_decimate.c file - checking that the INCOMING index increased by steps of [decimation] does not make any sense to me - as the upstream is not "aware" of any decimation going on downstream. In addition, apart from writing a warning message, this check does not seem to do anything else.

This check might be useful if within the rebuffer-module packets are repacked, however this is not controlled by the decimation ratio and if such a test was intended, then probably some check based on the relationship between daq_block_size and cpi_size*decimation_ratio should be used?

This returns value with leading space:
https://github.com/krakenrf/heimdall_daq_fw/blob/main/Firmware/daq_start_sm.sh#L24

So this check will not pass:
https://github.com/krakenrf/heimdall_daq_fw/blob/main/Firmware/daq_start_sm.sh#L116

This seem to do the job:
out_data_iface_type=$(awk -F'=' '/out_data_iface_type/ {gsub (" ", "", $0); print $2}' daq_chain_config.ini)

Can i use this firmware in other sdr like hackrf or pluto and what the modification that i need?
Thanks

Hi,
I'm using the KrakenSDR with gr-krakensdr. I have set the gains in the GNURadio block, and also in the daq_chain_config.ini.
However, in the below DAB radio recording, during the null symbol (no signal), the signal is automatically amplified. Is there a way to disable this behaviour, as it is affecting my signal processing?
Thank you.

The expected output with a normal RTL-SDR is below.

[meta]
ini_version = 7
config_name = kraken_default

[hw]
name = kraken5
unit_id = 0
ioo_type = 0
num_ch = 5
en_bias_tee = 0,0,0,0,0

[daq]
log_level = 5
daq_buffer_size = 262144
center_freq = 700000000
sample_rate = 2048000
gain = 496
en_noise_source_ctr = 1
ctr_channel_serial_no = 1000

[pre_processing]
cpi_size = 1048576
decimation_ratio = 1
fir_relative_bandwidth = 1.0
fir_tap_size = 1
fir_window = hann
en_filter_reset = 0

[calibration]
corr_size = 65536
std_ch_ind = 0
en_iq_cal = 1
amplitude_cal_mode = channel_power
en_gain_tune_init = 0
gain_lock_interval = 0
unified_gain_control = 0
require_track_lock_intervention = 0
cal_track_mode = 2
cal_frame_interval = 687
cal_frame_burst_size = 10
amplitude_tolerance = 2
phase_tolerance = 1
maximum_sync_fails = 10
iq_adjust_source = explicit-time-delay
iq_adjust_amplitude = 0,0,0,0
iq_adjust_time_delay_ns = 0, 0, 0, 0

[adpis]
en_adpis = 0
adpis_proc_size = 8192
adpis_gains_init = 496,496,496,496,496

[data_interface]
out_data_iface_type = eth

Are there any plans for x86_64 flatpak package? Or would y'all be up for a pull request with a flatpak build scripts?