Hi, I tried compiling the techempower benchmarks with the following

Wow thanks for this investigation <a class="user-mention notranslate" data-hovercard-t

The techempower example crashes for longer messages about haywire HOT 4 CLOSED

haywire commented on July 4, 2024

The techempower example crashes for longer messages

from haywire.

Comments (4)

rzezeski commented on July 4, 2024

I was able to replicate on SmartOS. The problem is a buffer
overrun. The create_response_buffer() function creates a
hardwired 1024 byte buffer for the response.

Here is output from MDB showing overrun detection logged by
libumem. The first stack is the stack that allocated the buffer.
The second stack is the stack that freed the buffer and
subsequently where libumem caught the overrun.

> ::umem_status
Status:         ready and active
Concurrency:    32
Logs:           (inactive)
Message buffer:
umem allocator: redzone violation: write past end of buffer
buffer=543000  bufctl=5280c0  cache: umem_alloc_1152
previous transaction on buffer 543000:
thread=29  time=T-0.000050987  slab=4d17c0  cache: umem_alloc_1152
libumem.so.1'umem_cache_alloc_debug+0xfd
libumem.so.1'umem_cache_alloc+0xb3
libumem.so.1'umem_alloc+0x64
libumem.so.1'umem_malloc+0x3f
libumem.so.1'calloc+0x60
techempower_benchmark'create_response_buffer+0x4c
techempower_benchmark'hw_http_response_send+0x36
techempower_benchmark'get_plaintext+0x115
techempower_benchmark'http_request_on_message_complete+0x6d
techempower_benchmark'http_parser_execute+0x360a
techempower_benchmark'http_stream_on_read+0x4e
techempower_benchmark'uv__read+0x41a
techempower_benchmark'uv__stream_io+0xdd
techempower_benchmark'uv__io_poll+0x4ec
techempower_benchmark'uv_run+0xb1
umem: heap corruption detected
stack trace:
libumem.so.1'umem_err_recoverable+0xcd
libumem.so.1'umem_error+0x1aa
libumem.so.1'umem_free+0xa8
libumem.so.1'process_free+0x145
libumem.so.1'umem_malloc_free+0x1d
techempower_benchmark'http_server_after_write+0x8d
techempower_benchmark'uv__write_callbacks+0x147
techempower_benchmark'uv__stream_io+0x17c
techempower_benchmark'uv__run_pending+0xa8
techempower_benchmark'uv_run+0x5d
techempower_benchmark'connection_consumer_start+0x138
techempower_benchmark'uv__thread_start+0x40
libc.so.1'_thrp_setup+0x8a
libc.so.1'_lwp_start+0x0

Here is a dump of the 1024 bytes returned by calloc(), notice
that the string is cutoff:

> 0x543000,0t1024::dump
         \/ 1 2 3  4 5 6 7  8 9 a b  c d e f  v123456789abcdef
543000:  00000000 00000000 10040000 efbeadde  ................
543010:  48545450 2f312e31 20323030 204f4b00  HTTP/1.1 200 OK.
543020:  0d0a5365 72766572 3a204861 79776972  ..Server: Haywir
543030:  652f6d61 73746572 0d0a4461 74653a20  e/master..Date:
543040:  4d6f6e20 41707220 32302032 313a3137  Mon Apr 20 21:17
543050:  3a323920 32303135 0a436f6e 74656e74  :29 2015.Content
543060:  2d547970 653a2074 6578742f 706c6169  -Type: text/plai
543070:  6e0d0a43 6f6e6e65 6374696f 6e3a204b  n..Connection: K
543080:  6565702d 416c6976 650d0a43 6f6e7465  eep-Alive..Conte
543090:  6e742d4c 656e6774 683a2032 3035360d  nt-Length: 2056.
5430a0:  0a0d0a43 48415054 45522049 2e20446f  ...CHAPTER I. Do
5430b0:  776e2074 68652052 61626269 742d486f  wn the Rabbit-Ho
5430c0:  6c652020 416c6963 65207761 73206265  le  Alice was be
5430d0:  67696e6e 696e6720 746f2067 65742076  ginning to get v
5430e0:  65727920 74697265 64206f66 20736974  ery tired of sit
5430f0:  74696e67 20627920 68657220 73697374  ting by her sist
543100:  6572206f 6e207468 65206261 6e6b2c20  er on the bank,
543110:  616e6420 6f662068 6176696e 67206e6f  and of having no
543120:  7468696e 6720746f 20646f3a 206f6e63  thing to do: onc
543130:  65206f72 20747769 63652073 68652068  e or twice she h
543140:  61642070 65657065 6420696e 746f2074  ad peeped into t
543150:  68652062 6f6f6b20 68657220 73697374  he book her sist
543160:  65722077 61732072 65616469 6e672c20  er was reading,
543170:  62757420 69742068 6164206e 6f207069  but it had no pi
543180:  63747572 6573206f 7220636f 6e766572  ctures or conver
543190:  73617469 6f6e7320 696e2069 742c203c  sations in it, <
5431a0:  616e6420 77686174 20697320 74686520  and what is the
5431b0:  75736520 6f662061 20626f6f 6b2c3e20  use of a book,>
5431c0:  74686f75 67687420 416c6963 65203c77  thought Alice <w
5431d0:  6974686f 75742070 69637475 72657320  ithout pictures
5431e0:  6f722063 6f6e7665 72736174 696f6e73  or conversations
5431f0:  3f3e2053 6f207368 65207761 7320636f  ?> So she was co
543200:  6e736964 6572696e 6720696e 20686572  nsidering in her
543210:  206f776e 206d696e 64202861 73207765   own mind (as we
543220:  6c6c2061 73207368 6520636f 756c642c  ll as she could,
543230:  20666f72 20746865 20686f74 20646179   for the hot day
543240:  206d6164 65206865 72206665 656c2076   made her feel v
543250:  65727920 736c6565 70792061 6e642073  ery sleepy and s
543260:  74757069 64292c20 77686574 68657220  tupid), whether
543270:  74686520 706c6561 73757265 206f6620  the pleasure of
543280:  6d616b69 6e672061 20646169 73792d63  making a daisy-c
543290:  6861696e 20776f75 6c642062 6520776f  hain would be wo
5432a0:  72746820 74686520 74726f75 626c6520  rth the trouble
5432b0:  6f662067 65747469 6e672075 7020616e  of getting up an
5432c0:  64207069 636b696e 67207468 65206461  d picking the da
5432d0:  69736965 732c2077 68656e20 73756464  isies, when sudd
5432e0:  656e6c79 20612057 68697465 20526162  enly a White Rab
5432f0:  62697420 77697468 2070696e 6b206579  bit with pink ey
543300:  65732072 616e2063 6c6f7365 20627920  es ran close by
543310:  6865722e 20546865 72652077 6173206e  her. There was n
543320:  6f746869 6e672073 6f207665 72792072  othing so very r
543330:  656d6172 6b61626c 6520696e 20746861  emarkable in tha
543340:  743b206e 6f722064 69642041 6c696365  t; nor did Alice
543350:  20746869 6e6b2069 7420736f 20766572   think it so ver
543360:  79206d75 6368206f 7574206f 66207468  y much out of th
543370:  65207761 7920746f 20686561 72207468  e way to hear th
543380:  65205261 62626974 20736179 20746f20  e Rabbit say to
543390:  69747365 6c662c20 3c4f6820 64656172  itself, <Oh dear
5433a0:  21204f68 20646561 72212049 20736861  ! Oh dear! I sha
5433b0:  6c6c2062 65206c61 7465213e 20287768  ll be late!> (wh
5433c0:  656e2073 68652074 686f7567 68742069  en she thought i
5433d0:  74206f76 65722061 66746572 77617264  t over afterward
5433e0:  732c2069 74206f63 63757272 65642074  s, it occurred t
5433f0:  6f206865 72207468 61742073 6865206f  o her that she o

from haywire.

rzezeski commented on July 4, 2024

In case it wasn't clear, the large string continues to run well
past the end of the 1024 byte buffer, as shown by dumping the
next 1024 bytes.

> 0x5433f0,0t1024::dump
         \/ 1 2 3  4 5 6 7  8 9 a b  c d e f  v123456789abcdef
5433f0:  6f206865 72207468 61742073 6865206f  o her that she o
543400:  75676874 20746f20 68617665 20776f6e  ught to have won
543410:  64657265 64206174 20746869 732c2062  dered at this, b
543420:  75742061 74207468 65207469 6d652069  ut at the time i
543430:  7420616c 6c207365 656d6564 20717569  t all seemed qui
543440:  7465206e 61747572 616c293b 20627574  te natural); but
543450:  20776865 6e207468 65205261 62626974   when the Rabbit
543460:  20616374 75616c6c 7920746f 6f6b2061   actually took a
543470:  20776174 6368206f 7574206f 66206974   watch out of it
543480:  73207761 69737463 6f61742d 706f636b  s waistcoat-pock
543490:  65742c20 616e6420 6c6f6f6b 65642061  et, and looked a
5434a0:  74206974 2c20616e 64207468 656e2068  t it, and then h
5434b0:  75727269 6564206f 6e2c2041 6c696365  urried on, Alice
5434c0:  20737461 72746564 20746f20 68657220   started to her
5434d0:  66656574 2c20666f 72206974 20666c61  feet, for it fla
5434e0:  73686564 20616372 6f737320 68657220  shed across her
5434f0:  6d696e64 20746861 74207368 65206861  mind that she ha
543500:  64206e65 76657220 6265666f 72652073  d never before s
543510:  65656e20 61207261 62626974 20776974  een a rabbit wit
543520:  68206569 74686572 20612077 61697374  h either a waist
543530:  636f6174 2d706f63 6b65742c 206f7220  coat-pocket, or
543540:  61207761 74636820 746f2074 616b6520  a watch to take
543550:  6f757420 6f662069 742c2061 6e642062  out of it, and b
543560:  75726e69 6e672077 69746820 63757269  urning with curi
543570:  6f736974 792c2073 68652072 616e2061  osity, she ran a
543580:  63726f73 73207468 65206669 656c6420  cross the field
543590:  61667465 72206974 2c20616e 6420666f  after it, and fo
5435a0:  7274756e 6174656c 79207761 73206a75  rtunately was ju
5435b0:  73742069 6e207469 6d652074 6f207365  st in time to se
5435c0:  65206974 20706f70 20646f77 6e206120  e it pop down a
5435d0:  6c617267 65207261 62626974 2d686f6c  large rabbit-hol
5435e0:  6520756e 64657220 74686520 68656467  e under the hedg
5435f0:  652e2049 6e20616e 6f746865 72206d6f  e. In another mo
543600:  6d656e74 20646f77 6e207765 6e742041  ment down went A
543610:  6c696365 20616674 65722069 742c206e  lice after it, n
543620:  65766572 206f6e63 6520636f 6e736964  ever once consid
543630:  6572696e 6720686f 7720696e 20746865  ering how in the
543640:  20776f72 6c642073 68652077 61732074   world she was t
543650:  6f206765 74206f75 74206167 61696e2e  o get out again.
543660:  20546865 20726162 6269742d 686f6c65   The rabbit-hole
543670:  2077656e 74207374 72616967 6874206f   went straight o
543680:  6e206c69 6b652061 2074756e 6e656c20  n like a tunnel
543690:  666f7220 736f6d65 20776179 2c20616e  for some way, an
5436a0:  64207468 656e2064 69707065 64207375  d then dipped su
5436b0:  6464656e 6c792064 6f776e2c 20736f20  ddenly down, so
5436c0:  73756464 656e6c79 20746861 7420416c  suddenly that Al
5436d0:  69636520 68616420 6e6f7420 61206d6f  ice had not a mo
5436e0:  6d656e74 20746f20 7468696e 6b206162  ment to think ab
5436f0:  6f757420 73746f70 70696e67 20686572  out stopping her
543700:  73656c66 20626566 6f726520 73686520  self before she
543710:  666f756e 64206865 7273656c 66206661  found herself fa
543720:  6c6c696e 6720646f 776e2061 20766572  lling down a ver
543730:  79206465 65702077 656c6c2e 20456974  y deep well. Eit
543740:  68657220 74686520 77656c6c 20776173  her the well was
543750:  20766572 79206465 65702c20 6f722073   very deep, or s
543760:  68652066 656c6c20 76657279 20736c6f  he fell very slo
543770:  776c792c 20666f72 20736865 20686164  wly, for she had
543780:  20706c65 6e747920 6f662074 696d6520   plenty of time
543790:  61732073 68652077 656e7420 646f776e  as she went down
5437a0:  20746f20 6c6f6f6b 2061626f 75742068   to look about h
5437b0:  65722061 6e642074 6f20776f 6e646572  er and to wonder
5437c0:  20776861 74207761 7320676f 696e6720   what was going
5437d0:  746f2068 61707065 6e206e65 78742e20  to happen next.
5437e0:  46697273 742c2073 68652074 72696564  First, she tried

Here is the hardwired size:

https://github.com/kellabyte/Haywire/blob/master/src/haywire/http_response.c#L69

from haywire.

jlouis commented on July 4, 2024

Thank you so much for the digging. I've some envy of mdb/libumem now, though I think I can get some of it from the thread/address sanitizer and llvm.

I somewhat suspected there to be a buffer, but I had stared myself blind on the hw_string's and not the output buffer. I'll just bump it up to contain the message then :)

from haywire.

kellabyte commented on July 4, 2024

Wow thanks for this investigation @rzezeski :) Yeah Haywire needs some dynamic memory allocation here but in a way that performs well.

from haywire.

The techempower example crashes for longer messages about haywire HOT 4 CLOSED

Comments (4)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent