Giter VIP home page Giter VIP logo

Comments (15)

Algunenano avatar Algunenano commented on September 24, 2024 1

Some alternatives:

  • #67005: Alert during cmake run. Forces disabling ASLR in the whole system and it only works during build, you will still have issues if you try to run a binary you haven't built yourself or that you build with ASLR disabled but then it was enabled again.
  • #67073: Remove custom getauxval. It works, but I requires increasing the dependency on glibc to 2.16 (from 2.4) which is bad.
  • #67081: Hack to catch the re-exec call to avoid it crashing.
  • Ditch dynamic dependency on glibc and fully statically link musl. Long term stale project. Not sure how static libc + sanitizer works.

from clickhouse.

Algunenano avatar Algunenano commented on September 24, 2024 1

BTW why it pops up only now? AFAIR the compiler had been updated long time ago - #60469

It's not now. It has been failing for months but different developers have addressed it differently, and every couple of weeks somebody internally asks why the sanitizer builds are crashing or why they can't build with sanitizer, so @rschu1ze decided to do something

from clickhouse.

azat avatar azat commented on September 24, 2024

And there are also SIGSEGV during installation of TSan/MSan builds from time to time:

Stateless tests (tsan) [1/5] — Invalid check_status.tsv

Setting up clickhouse-server (24.5.1.1318+tsan) ...
Segmentation fault (core dumped)
dpkg: error processing package clickhouse-server (--install):
 installed clickhouse-server package post-installation script subprocess returned error exit status 139

But I cannot reproduce the issue:

root@5acc492f7e7e:/# dpkg -i /root/*.deb  |& cat
(Reading database ... 4437 files and directories currently installed.)
Preparing to unpack .../clickhouse-common-static_24.5.1.1318+tsan_amd64.deb ...
Unpacking clickhouse-common-static (24.5.1.1318+tsan) over (24.5.1.1318+tsan) ...
Preparing to unpack .../clickhouse-server_24.5.1.1318+tsan_amd64.deb ...
Unpacking clickhouse-server (24.5.1.1318+tsan) over (24.5.1.1318+tsan) ...
Setting up clickhouse-common-static (24.5.1.1318+tsan) ...
Setting up clickhouse-server (24.5.1.1318+tsan) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.34.0 /usr/local/share/perl/5.34.0 /usr/lib/x86_64-linux-gnu/perl5/5.34 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.34 /usr/share/perl/5.34 /usr/local/lib/site_perl) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
groupadd: group 'clickhouse' already exists
useradd: user 'clickhouse' already exists
Cannot set 'net_admin' or 'ipc_lock' or 'sys_nice' or 'net_bind_service' capability for clickhouse binary. This is optional. Taskstats accounting will be disabled. To enable taskstats accounting you may add the required capability later manually.
ClickHouse binary is already located at /usr/bin/clickhouse
Symlink /usr/bin/clickhouse-server already exists but it points to /clickhouse. Will replace the old symlink to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-server to /usr/bin/clickhouse.
Symlink /usr/bin/clickhouse-extract-from-config already exists but it points to /clickhouse. Will replace the old symlink to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-extract-from-config to /usr/bin/clickhouse.
Symlink /usr/bin/clickhouse-keeper already exists but it points to /clickhouse. Will replace the old symlink to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-keeper to /usr/bin/clickhouse.
Symlink /usr/bin/clickhouse-keeper-converter already exists but it points to /clickhouse. Will replace the old symlink to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-keeper-converter to /usr/bin/clickhouse.
Symlink /usr/bin/ch already exists. Will keep it.
Symlink /usr/bin/chl already exists. Will keep it.
Symlink /usr/bin/chc already exists. Will keep it.
Creating clickhouse group if it does not exist.
 groupadd -r clickhouse
Creating clickhouse user if it does not exist.
 useradd -r --shell /bin/false --home-dir /nonexistent -g clickhouse clickhouse
Will set ulimits for clickhouse user in /etc/security/limits.d/clickhouse.conf.
Config file /etc/clickhouse-server/config.xml already exists, will keep it and extract path info from it.
/etc/clickhouse-server/config.xml has /var/lib/clickhouse/ as data path.
/etc/clickhouse-server/config.xml has /var/log/clickhouse-server/ as log path.
Users config file /etc/clickhouse-server/users.xml already exists, will keep it and extract users info from it.
Log directory /var/log/clickhouse-server/ already exists.
Data directory /var/lib/clickhouse/ already exists.
Pid directory /var/run/clickhouse-server already exists.
 chown -R clickhouse:clickhouse '/var/log/clickhouse-server/'
 chown -R clickhouse:clickhouse '/var/run/clickhouse-server'
 chown  clickhouse:clickhouse '/var/lib/clickhouse/'
Password for the default user is an empty string. See /etc/clickhouse-server/users.xml and /etc/clickhouse-server/users.d to change it.
Setting capabilities for clickhouse binary. This is optional.
 chown -R clickhouse:clickhouse '/etc/clickhouse-server'

ClickHouse has been successfully installed.

Start clickhouse-server with:
 sudo clickhouse start

Start clickhouse-client with:
 clickhouse-client

Play: https://play.clickhouse.com/play?user=play#U0VMRUNUIGNoZWNrX2R1cmF0aW9uX21zLCBjaGVja19zdGF0dXMsIHJlcG9ydF91cmwKRlJPTSBjaGVja3MKV0hFUkUgcmVwb3J0X3VybCBMSUtFICclc3RhdGVsZXNzX3Rlc3RzX190c2FuX18lJwogICAgQU5EIGNoZWNrX3N0YXJ0X3RpbWUgPj0gbm93KCkgLSBJTlRFUlZBTCAzIERBWQogICAgYW5kIGNoZWNrX2R1cmF0aW9uX21zIDwgMzAwZTMKICAgIGFuZCBjaGVja19zdGF0dXMgPSAnZXJyb3InCk9SREVSIEJZIGNoZWNrX3N0YXJ0X3RpbWU=

from clickhouse.

azat avatar azat commented on September 24, 2024

Looks related

from clickhouse.

azat avatar azat commented on September 24, 2024

We could try sudo sysctl vm.mmap_rnd_bits=28 (if there is such sysctl in ubuntus on CI)

from clickhouse.

azat avatar azat commented on September 24, 2024

I've tried #64090 since I saw that in google/sanitizers#856 there were issues reported for the 6.5.0-28-generic kernel, while CI has 6.5.0-1014-aws, so more or less the same, but it did not help 100%.
By some reason it helps for builds - where protoc is used, but not for tests where clickhouse binary is used

from clickhouse.

azat avatar azat commented on September 24, 2024

And it will be hard to built protoc without sanitizers -

ClickHouse/contrib/google-protobuf-cmake/CMakeLists.txt

Lines 341 to 349 in 3878155

if (ENABLE_FUZZING)
# `protoc` will be built with sanitizer and it could fail during ClickHouse build
# It easily reproduces in oss-fuzz building pipeline
# To avoid this we can try to build `protoc` without any sanitizer with option `-fno-sanitize=all`, but
# it this case we will face with linker errors, because libcxx still will be built with sanitizer
# So, we can simply suppress all of these failures with a combination this flag and an environment variable
# export MSAN_OPTIONS=exit_code=0
target_compile_options(protoc PRIVATE "-fsanitize-recover=all")
endif()

from clickhouse.

azat avatar azat commented on September 24, 2024

So kernel.randomize_va_space=0 helps for MSan/TSan

MSan has some strange issues with getauxval override with ASLR, but turning it OFF helps anyway:

# cd /src/clickhouse/.cmake-msan/contrib/arrow-cmake && /src/clickhouse/.cmake-msan/contrib/google-protobuf-cmake/protoc -I /src/clickhouse/contrib/orc/c++/../proto --cpp_out="/src/clickhouse/.cmake-msan/contrib/arrow-cmake" /src/clickhouse/contrib/orc/c++/../proto/orc_proto.proto
MemorySanitizer:DEADLYSIGNAL
==41141==ERROR: MemorySanitizer: SEGV on unknown address 0x2ffec5be94a8 (pc 0x630a1c8069c6 bp 0x000000000002 sp 0x7ffec5be94a0 T41141)
==41141==The signal is caused by a WRITE memory access.
    #0 0x630a1c8069c6 in __auxv_init_procfs .cmake-msan/./base/glibc-compatibility/musl/getauxval.c:78
    #1 0x630a1c807049 in getauxval .cmake-msan/./base/glibc-compatibility/musl/getauxval.c:200:12
    #2 0x630a1b963851 in __sanitizer::ReExec() crtstuff.c
    #3 0x630a1b9df4ca in __msan::InitShadowWithReExec(bool) crtstuff.c
    #4 0x630a1b97536b in __msan_init (/src/clickhouse/.cmake-msan/contrib/google-protobuf-cmake/protoc+0x24136b) (BuildId: 95c3c051aef70edc003ed3a9cd175c6b301391e5)
    #5 0x630a1b9e6978 in msan.module_ctor main.cc
    #6 0x630a1c807e8c in __libc_csu_init (/src/clickhouse/.cmake-msan/contrib/google-protobuf-cmake/protoc+0x10d3e8c) (BuildId: 95c3c051aef70edc003ed3a9cd175c6b301391e5)
    #7 0x71919742a263 in __libc_start_main csu/../csu/libc-start.c:343:6
    #8 0x630a1b957f0d in _start (/src/clickhouse/.cmake-msan/contrib/google-protobuf-cmake/protoc+0x223f0d) (BuildId: 95c3c051aef70edc003ed3a9cd175c6b301391e5)

from clickhouse.

azat avatar azat commented on September 24, 2024

May 17 18:34:12 ThreadSanitizer: CHECK failed: tsan_platform_linux.cpp:282 "((personality(old_personality | ADDR_NO_RANDOMIZE))) != ((-1))" (0xffffffffffffffff, 0xffffffffffffffff) (tid=123304)

This is likely due to docker forbids this via seccomp

from clickhouse.

azat avatar azat commented on September 24, 2024

BTW initial issue is likely due to ubuntu decided to adjust mmap_rnd_bits - https://bugs.launchpad.net/ubuntu/+source/llvm-toolchain-14/+bug/2048768/comments/8

from clickhouse.

Algunenano avatar Algunenano commented on September 24, 2024

After updating the compilers it seems that the problem is in our custom getauxval method, which apparently is not happy with how the sanitizers deal with high ASLR bits. Removing the custom getauxval solves the issue, but introduces a dependency on glibc 2.16

from clickhouse.

Algunenano avatar Algunenano commented on September 24, 2024

Another option:

  • Remove the custom getauxval only when building with sanitizers, where the glibc dependency is not important. In fact my local builds right now depend on (GLIBC_2.23) __signgam and depending on 2.16 just for the sanitizers (not for release) might be acceptable.

from clickhouse.

azat avatar azat commented on September 24, 2024

@Algunenano seems that it is OK to remove custom getauxval for sanitizers case if it is already depends on newer glibc.

BTW why it pops up only now? AFAIR the compiler had been updated long time ago - #60469

from clickhouse.

Algunenano avatar Algunenano commented on September 24, 2024

It seems official builds of the sanitizer only depended on 2.4 until now:

/tmp $ wget https://s3.amazonaws.com/clickhouse-builds/PRs/66976/009da21694539f4c35983898fb34d4a492c5a692/package_msan/clickhouse
--2024-07-25 11:44:20--  https://s3.amazonaws.com/clickhouse-builds/PRs/66976/009da21694539f4c35983898fb34d4a492c5a692/package_msan/clickhouse
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.8.77, 52.217.44.102, 52.217.225.64, ...
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.8.77|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1122496123 (1.0G) [binary/octet-stream]
Saving to: ‘clickhouse’

clickhouse                                                             100%[=========================================================================================================================================================================>]   1.04G  15.8MB/s    in 2m 13s  

2024-07-25 11:46:33 (8.08 MB/s) - ‘clickhouse’ saved [1122496123/1122496123]

/tmp $ chmod +x clickhouse
/tmp $ objdump -T clickhouse | grep GLIBC | sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -u
2.2.5
2.3
2.3.2
/tmp $ ./clickhouse local
MemorySanitizer:DEADLYSIGNAL
==61494==ERROR: MemorySanitizer: SEGV on unknown address 0x2ffe6243a498 (pc 0x652cee7b8304 bp 0x000000000003 sp 0x7ffe6243a480 T61494)
==61494==The signal is caused by a WRITE memory access.
==61494==WARNING: invalid path to external symbolizer!
==61494==WARNING: Failed to use and restart external symbolizer!
    #0 0x652cee7b8304  (/tmp/clickhouse+0x3f5304) (BuildId: 93d52fe5368deb1f40feb365384a96192cd9abc8)
    #1 0x652cee7b8289  (/tmp/clickhouse+0x3f5289) (BuildId: 93d52fe5368deb1f40feb365384a96192cd9abc8)
    #2 0x652cee47582d  (/tmp/clickhouse+0xb282d) (BuildId: 93d52fe5368deb1f40feb365384a96192cd9abc8)
    #3 0x652cee4efef6  (/tmp/clickhouse+0x12cef6) (BuildId: 93d52fe5368deb1f40feb365384a96192cd9abc8)
    #4 0x652cee486d07  (/tmp/clickhouse+0xc3d07) (BuildId: 93d52fe5368deb1f40feb365384a96192cd9abc8)
    #5 0x652cee4fb098  (/tmp/clickhouse+0x138098) (BuildId: 93d52fe5368deb1f40feb365384a96192cd9abc8)
    #6 0x652cee7b8efc  (/tmp/clickhouse+0x3f5efc) (BuildId: 93d52fe5368deb1f40feb365384a96192cd9abc8)
    #7 0x743a47c37ea3  (/usr/lib/libc.so.6+0x25ea3) (BuildId: 3de7fd3e8d993406afdcb908e63b88f2f4effea9)
    #8 0x652cee46a26d  (/tmp/clickhouse+0xa726d) (BuildId: 93d52fe5368deb1f40feb365384a96192cd9abc8)

MemorySanitizer can not provide additional info.
SUMMARY: MemorySanitizer: SEGV (/tmp/clickhouse+0x3f5304) (BuildId: 93d52fe5368deb1f40feb365384a96192cd9abc8) 
==61494==ABORTING
/tmp $ sudo sysctl kernel.randomize_va_space=0
kernel.randomize_va_space = 0
/tmp $ ./clickhouse local 'Select 1'
Decompressing the binary................................
1
/tmp $ objdump -T clickhouse | grep GLIBC | sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -u
2.2.5
2.3
2.3.2
2.3.3
2.3.4
2.4

@alexey-milovidov WDYT, would it be acceptable to have sanitizer builds depending on GLIBC 2.16? It's been available for 12 years at this point and we would keep 2.4 for release/debug x86

from clickhouse.

Algunenano avatar Algunenano commented on September 24, 2024

We decided that keeping compatibility was important so you can take binaries almost anywhere and use them without worries, so we are patching getauxval to support msan and tsan re-exec.

from clickhouse.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.