zebrafishlabs / nginx-statsd Goto Github PK
View Code? Open in Web Editor NEWAn nginx module for sending stats to statsd.
Home Page: http://www.zebrafishlabs.com
License: Other
An nginx module for sending stats to statsd.
Home Page: http://www.zebrafishlabs.com
License: Other
I want to count incoming/outgoing traffic through statsd. This works:
statsd_count "mysite.outgoing_body_traffic" $body_bytes_sent;
However, $body_bytes_sent does not include HTTP headers. When I try one of the following, nginx won't start:
statsd_count "mysite.outgoing_traffic" $bytes_sent;
statsd_count "mysite.incoming_traffic" $request_length;
The error messages I get:
Restarting nginx: nginx: [emerg] unknown "bytes_sent" variable
Restarting nginx: nginx: [emerg] unknown "request_length" variable
Any ideas why $body_bytes_sent works perfectly but $bytes_sent and $request_length are unknown variables? Logging them to a file (via log_format) works.
Thanks!
objs/addon/nginx-statsd-master/ngx_http_statsd.o: In function ngx_http_statsd_udp_send': /tmp/soft/nginx-statsd-master/ngx_http_statsd.c:367: undefined reference to
ngx_udp_connect'
collect2: ld returned 1 exit status
My statsd server is not receiving any packets from nginx, and as far as I can tell, the system is not sending any UDP requests.
I've only monitored:
netstat -npuc
to check for outgoing UDP transactions. Nothing shows up.
I'm running nginx 1.7.9, with spdy, ssl, and this module. Any ideas you have to help me figure out what's going on here?
Does it (nginx-statsd) support POST requests. I configure as:
statsd_count "myapp.request" 1;
but it's only sends GET requests to statsd server. Please give me some information. Thanks
Author: Eran Kornblau [email protected]
Date: Thu Mar 2 08:46:00 2017 -0500
Added missing static specifiers.
As of the above commit, the ngx_udp_connect function is static (not exported) and so cannot be used by this library.
I don't know of a good fix, but if some suggests one to me I will attempt to implement it.
Currently I'm applying this patch to my nginx build.
diff -ru nginx-1.18.0/src/core/ngx_resolver.c nginx-1.18.0-jrw/src/core/ngx_resolver.c
--- nginx-1.18.0/src/core/ngx_resolver.c 2020-04-21 14:09:01.000000000 +0000
+++ nginx-1.18.0-jrw/src/core/ngx_resolver.c 2021-10-21 15:42:19.808367999 +0000
@@ -56,7 +56,7 @@
((u_char *) (n) - offsetof(ngx_resolver_node_t, node))
-static ngx_int_t ngx_udp_connect(ngx_resolver_connection_t *rec);
+ngx_int_t ngx_udp_connect(ngx_resolver_connection_t *rec);
static ngx_int_t ngx_tcp_connect(ngx_resolver_connection_t *rec);
@@ -4397,7 +4397,7 @@
}
-static ngx_int_t
+ngx_int_t
ngx_udp_connect(ngx_resolver_connection_t *rec)
{
int rc;
I'm receiving wrong targets in graphite from statsd. All the dashes are replaced by underscores. is this because of this function
https://github.com/zebrafishlabs/nginx-statsd/blob/master/ngx_http_statsd.c#L712
And if so then why are you escaping the targets like that?
Hello,
after upgrading some packages via apt-get upgrade
in ubuntu 10.04, nginx with the nginx-statsd module compiled works very unstable (i.e. about every 2nd request fails with no data sent to the client). The syslog contains a lot of these errors:
Dec 12 01:25:56 server1 kernel: [ 5066.291028] nginx[18247]: segfault at 0 ip 000000000040c161 sp 00007fff26117c60 error 4 in nginx[400000+fd000]
And the nginx error.log has many of these:
2012/12/12 01:25:56 [alert] 10491#0: worker process 18247 exited on signal 11
I recompiled, downloaded the newest versions of nginx, pcre and updated everything but the only thing that seems to help is to disable nginx-statsd by either not compiling it (i.e. do not add --add-module=/path/to/nginx-statsd
) or to just not use any statsd-related configuration directives.
I use nginx 1.2.6 in ubuntu 10.04 server. Linux server1 2.6.32-40-server #87-Ubuntu SMP Tue Mar 6 02:10:02 UTC 2012 x86_64 GNU/Linux
These are packages I have upgraded with aptitude:
libc-bin (2.11.1-0ubuntu7.10, 2.11.1-0ubuntu7.12),
vim-common (7.2.330-1ubuntu3, 7.2.330-1ubuntu3.1),
libcupsimage2 (1.4.3-1ubuntu1.6, 1.4.3-1ubuntu1.9),
libxslt1-dev (1.1.26-1ubuntu1, 1.1.26-1ubuntu1.1),
mysql-server (5.1.63-0ubuntu0.10.04.1, 5.1.66-0ubuntu0.10.04.3),
libdbus-1-3 (1.2.16-2ubuntu4.5, 1.2.16-2ubuntu4.7),
perl (5.10.1-8ubuntu2.1, 5.10.1-8ubuntu2.2),
libpython2.6 (2.6.5-1ubuntu6, 2.6.5-1ubuntu6.1),
libcap2 (2.17-2ubuntu1, 2.17-2ubuntu1.1),
libxslt1.1 (1.1.26-1ubuntu1, 1.1.26-1ubuntu1.1),
perl-base (5.10.1-8ubuntu2.1, 5.10.1-8ubuntu2.2),
libmysqlclient-dev (5.1.63-0ubuntu0.10.04.1, 5.1.66-0ubuntu0.10.04.3),
perl-modules (5.10.1-8ubuntu2.1, 5.10.1-8ubuntu2.2),
mysql-server-core-5.1 (5.1.63-0ubuntu0.10.04.1, 5.1.66-0ubuntu0.10.04.3),
python2.6 (2.6.5-1ubuntu6, 2.6.5-1ubuntu6.1),
libglib2.0-0 (2.24.1-0ubuntu1, 2.24.1-0ubuntu2),
libmysqlclient16 (5.1.63-0ubuntu0.10.04.1, 5.1.66-0ubuntu0.10.04.3),
vim-tiny (7.2.330-1ubuntu3, 7.2.330-1ubuntu3.1),
python2.6-minimal (2.6.5-1ubuntu6, 2.6.5-1ubuntu6.1),
libcups2 (1.4.3-1ubuntu1.6, 1.4.3-1ubuntu1.9),
mysql-client-core-5.1 (5.1.63-0ubuntu0.10.04.1, 5.1.66-0ubuntu0.10.04.3),
libxml2 (2.7.6.dfsg-1ubuntu1.5, 2.7.6.dfsg-1ubuntu1.7),
libxml2-dev (2.7.6.dfsg-1ubuntu1.5, 2.7.6.dfsg-1ubuntu1.7),
lsb-base (4.0-0ubuntu8, 4.0-0ubuntu8.1),
libtiff4 (3.9.2-2ubuntu0.10, 3.9.2-2ubuntu0.12),
libc6-dev (2.11.1-0ubuntu7.10, 2.11.1-0ubuntu7.12),
linux-libc-dev (2.6.32-43.97, 2.6.32-45.101),
libc-dev-bin (2.11.1-0ubuntu7.10, 2.11.1-0ubuntu7.12),
libc6 (2.11.1-0ubuntu7.10, 2.11.1-0ubuntu7.12),
mysql-common (5.1.63-0ubuntu0.10.04.1, 5.1.66-0ubuntu0.10.04.3),
mysql-server-5.1 (5.1.63-0ubuntu0.10.04.1, 5.1.66-0ubuntu0.10.04.3),
lsb-release (4.0-0ubuntu8, 4.0-0ubuntu8.1),
mysql-client-5.1 (5.1.63-0ubuntu0.10.04.1, 5.1.66-0ubuntu0.10.04.3)
Any ideas or known incompatibilities?
Regards
groe
I am forking this to UKHomeOffice so I can tag a release there and have definitely stable known code, but may be nice for others to have a tagged version here.
according to https://graphite.readthedocs.io/en/latest/tags.html
tag format is my.series;tag1=value1;tag2=value2
however nginx-statsd escapes too many kinds of characters ,
to get tag compatibility
need to replace the escape mask from line:
nginx-statsd/ngx_http_statsd.c
Line 701 in 07dcefd
/* ?>=< ;:98 7654 3210 /.-, +*)( '&%$ #"! */
// 0xfc00bfff, /* 1111 1100 0000 0000 1011 1111 1111 1111 */
0xD400BFFF, /* 1101 0100 0000 0000 1011 1111 1111 1111 */
this repo looks out of maintenance for a long time, posting the patch here so that everyone can see it
ngx_http_statsd_udp_send calls uc->connection and when uc is null it generates segfault.
Here is the trace:
Program received signal SIGSEGV, Segmentation fault. 0x00000000004c1301 in ngx_http_statsd_udp_send (l=0xf54590, buf=0xfb98a8 "retail.pricing.fk_proxy.ip_10_98_169_234.80.status.200:1|c\003\243FB\a\356\001", len=58) at ../nginx-statsd/ngx_http_statsd.c:382 382 ../nginx-statsd/ngx_http_statsd.c: No such file or directory. in ../nginx-statsd/ngx_http_statsd.c (gdb) bt #0 0x00000000004c1301 in ngx_http_statsd_udp_send (l=0xf54590, buf=0xfb98a8 "retail.pricing.fk_proxy.ip_10_98_169_234.80.status.200:1|c\003\243FB\a\356\001", len=58) at ../nginx-statsd/ngx_http_statsd.c:382 #1 0x00000000004c10a2 in ngx_http_statsd_handler (r=0xf45960) at ../nginx-statsd/ngx_http_statsd.c:313 #2 0x000000000045c465 in ngx_http_log_request (r=0xf45960) at src/http/ngx_http_request.c:3455 #3 0x000000000045c2d6 in ngx_http_free_request (r=0xf45960, rc=0) at src/http/ngx_http_request.c:3402 #4 0x000000000045af7b in ngx_http_set_keepalive (r=0xf45960) at src/http/ngx_http_request.c:2840 #5 0x000000000045a3f5 in ngx_http_finalize_connection (r=0xf45960) at src/http/ngx_http_request.c:2483 #6 0x000000000045a058 in ngx_http_finalize_request (r=0xf45960, rc=0) at src/http/ngx_http_request.c:2384 #7 0x0000000000477feb in ngx_http_upstream_finalize_request (r=0xf45960, u=0xfb66f0, rc=0) at src/http/ngx_http_upstream.c:3560 #8 0x00000000004770eb in ngx_http_upstream_process_request (r=0xf45960) at src/http/ngx_http_upstream.c:3161 #9 0x0000000000476e08 in ngx_http_upstream_process_upstream (r=0xf45960, u=0xfb66f0) at src/http/ngx_http_upstream.c:3092 #10 0x00000000004721db in ngx_http_upstream_handler (ev=0x1005230) at src/http/ngx_http_upstream.c:977 #11 0x000000000043b78e in ngx_epoll_process_events (cycle=0xf38620, timer=60000, flags=1) at src/event/modules/ngx_epoll_module.c:691 #12 0x000000000042bfd4 in ngx_process_events_and_timers (cycle=0xf38620) at src/event/ngx_event.c:248 #13 0x0000000000439696 in ngx_worker_process_cycle (cycle=0xf38620, data=0x0) at src/os/unix/ngx_process_cycle.c:816 #14 0x0000000000435ef6 in ngx_spawn_process (cycle=0xf38620, proc=0x439490 , data=0x0, name=0x4c69c3 "worker process", respawn=0) at src/os/unix/ngx_process.c:198 #15 0x0000000000439089 in ngx_reap_children (cycle=0xf38620) at src/os/unix/ngx_process_cycle.c:627 #16 0x0000000000437d19 in ngx_master_process_cycle (cycle=0xf38620) at src/os/unix/ngx_process_cycle.c:180 #17 0x000000000040727f in main (argc=1, argv=0x7fffd7f4cee8) at src/core/nginx.c:407
sudo yum install wget curl unzip gcc-c++ pcre-devel zlib-devel
mkdir ~/custom-nginx
cd ~/custom-nginx
# tried 1.8.0 also
sudo wget http://nginx.org/download/nginx-1.6.2.tar.gz
sudo tar zxvf nginx-1.6.2.tar.gz
ls ~/custom-nginx
cd nginx-1.8.0/src/http/modules/
sudo wget https://github.com/zebrafishlabs/nginx-statsd/archive/master.tar.gz
sudo tar xvf master.tar.gz
sudo rm master.tar.gz
sudo ./configure --add-module=/home/manjeshpv/custom-nginx/nginx-1.8.0/src/http/modules/ngx_pagespeed/ --user=nobody --group=nobody --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
sudo make
I got error while
https://gist.github.com/manjeshpv/c1e39ecf83ad9b6fc478c16d1184d2e5
Hello.
If you make nginx with "--with-openssl=..." then shared (system) openssl lib should not be used. But nginx-statsd "config" adds "-lssl".
The mod_statsd project for Apache provides very detailed statistics by logging every hit and it's response code.
By default, they have them by individual page types, but I roll up to the top level domain to get the following entries, which I find extremely useful.
www_hostname_com.GET.200.count
www_hostname_com.GET.403.count
www_hostname_com.GET.404.count
www_hostname_com.POST.200.count
Is it possible to duplicate this functionality?
This repo is dead, the addon no longer compiles with current nginx, and the link to www.zebrafishlabs.com is dead. Is there a fork that is being maintained/other people are using? I checked a few and didn't see any with stars.
As I have written in details here http://serverfault.com/questions/597195/sending-data-to-statsd-does-not-work-in-some-locations-with-content-by-lua-file I have a problem with this module not pushing data into statsd when I exit my lua script with return ngx.redirect .
I don't want to use https://github.com/lonelyplanet/openresty-statsd as it would require me to add statsd pushing code to all my scripts..
Could you help me please resolve this issue then?
Tengine/nginx: https://github.com/alibaba/tengine
Tengine version: Tengine/2.0.3 (nginx/1.4.7)
nginx: [emerg] can't locate symbol in module "ngx_http_statsd" in /etc/tengine/tengine.conf:16
configuration file /etc/tengine/tengine.conf test failed
Error by : load ngx_http_statsd.so;
$ cat /tmp/build.log
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.7knwYD
/data/hukai/rpmbuild/BUILDROOT/tengine-statsd-14.08-20.el6.x86_64/usr' mkdir: created directory
/data/hukai/rpmbuild/BUILDROOT/tengine-statsd-14.08-20.el6.x86_64/usr/lib64'/data/hukai/rpmbuild/BUILDROOT/tengine-statsd-14.08-20.el6.x86_64/usr/lib64/tengine' mkdir: created directory
/data/hukai/rpmbuild/BUILDROOT/tengine-statsd-14.08-20.el6.x86_64/usr/lib64/tengine/modules'linux-vdso.so.1 => (0x00007fff6f314000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f651a666000)
libc.so.6 => /lib64/libc.so.6 (0x00007f651a2d4000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f651a091000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f6519db2000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f6519bae000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f6519981000)
libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f65195a1000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f651939d000)
libz.so.1 => /lib64/libz.so.1 (0x00007f6519186000)
/lib64/ld-linux-x86-64.so.2 (0x0000003169e00000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f6518f7b000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6518d78000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6518b5d000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6518940000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f6518720000)
Other infomations:
open("/usr/lib64/tengine/modules/ngx_http_statsd.so", O_RDONLY) = 5
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\r\0\0\0\0\0\0"..., 832) = 832
fstat(5, {st_mode=S_IFREG|0755, st_size=99101, ...}) = 0
mmap(NULL, 2106664, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7f826cea2000
mprotect(0x7f826cea4000, 2097152, PROT_NONE) = 0
mmap(0x7f826d0a4000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x2000) = 0x7f826d0a4000
close(5) = 0
write(3, "2014/08/21 00:35:34 [emerg] 1857"..., 117) = 117
write(2, "nginx: [emerg] can't locate symb"..., 95nginx: [emerg] can't locate symbol in module "ngx_http_statsd" in /etc/tengine/tengine.conf:16
) = 95
close(4)
26831: /usr/lib64/tengine/modules/ngx_http_statsd.so: error: symbol lookup error: undefined symbol: ngx_http_core_module (continued)
undefined symbol: ngx_http_core_module (/usr/lib64/tengine/modules/ngx_http_statsd.so)
26831: /usr/lib64/tengine/modules/ngx_http_statsd.so: error: symbol lookup error: undefined symbol: ngx_io (continued)
undefined symbol: ngx_io (/usr/lib64/tengine/modules/ngx_http_statsd.so)
26831: /usr/lib64/tengine/modules/ngx_http_statsd.so: error: symbol lookup error: undefined symbol: ngx_conf_set_num_slot (continued)
undefined symbol: ngx_conf_set_num_slot (/usr/lib64/tengine/modules/ngx_http_statsd.so)
Unused direct dependencies:
/usr/lib64/libssl.so.10
While looking for a quick and easy way to send some nginx data to statsd I came upon this module and while it's doing its job properly, there is one thing that was bothering me: for each configured metric it would generate a separate UDP packet. This would mean that when used in a relatively large farm with multiple vhosts, each with multiple metrics defined, it would generate a flurry of packets, even when not under heavy load, that would hammer the statsd service. One obvious solution to this was to combine multiple metrics in a single packet, and the following patch tries to do just that.
--- ngx_http_statsd.c.orig 2015-11-02 22:07:22.736000288 +0000
+++ ngx_http_statsd.c 2015-11-02 22:20:44.615986241 +0000
@@ -18,7 +18,7 @@
#define STATSD_TYPE_COUNTER 0x0001
#define STATSD_TYPE_TIMING 0x0002
-#define STATSD_MAX_STR 256
+#define STATSD_MAX_STR 1432
#define ngx_conf_merge_ptr_value(conf, prev, default) \
if (conf == NGX_CONF_UNSET_PTR) { \
@@ -235,7 +235,8 @@
ngx_int_t
ngx_http_statsd_handler(ngx_http_request_t *r)
{
- u_char line[STATSD_MAX_STR], *p;
+ u_char startline[STATSD_MAX_STR], *p, *line;
+ size_t togo;
const char * metric_type;
ngx_http_statsd_conf_t *ulcf;
ngx_statsd_stat_t *stats;
@@ -262,6 +263,8 @@
}
stats = ulcf->stats->elts;
+ line = startline;
+ togo = STATSD_MAX_STR;
for (c = 0; c < ulcf->stats->nelts; c++) {
stat = stats[c];
@@ -287,13 +290,26 @@
if (metric_type) {
if (ulcf->sample_rate < 100) {
- p = ngx_snprintf(line, STATSD_MAX_STR, "%V:%d|%s|@0.%02d", &s, n, metric_type, ulcf->sample_rate);
+ p = ngx_snprintf(line, togo, "%V:%d|%s|@0.%02d\n", &s, n, metric_type, ulcf->sample_rate);
} else {
- p = ngx_snprintf(line, STATSD_MAX_STR, "%V:%d|%s", &s, n, metric_type);
+ p = ngx_snprintf(line, togo, "%V:%d|%s\n", &s, n, metric_type);
+ }
+ if (p - line >= togo) {
+ if (line != startline) {
+ ngx_http_statsd_udp_send(ulcf->endpoint, startline, line - startline - sizeof(char));
+ c--;
+ }
+ line = startline;
+ togo = STATSD_MAX_STR;
+ } else {
+ togo -= p - line;
+ line = p;
}
- ngx_http_statsd_udp_send(ulcf->endpoint, line, p - line);
}
}
+ if (togo < STATSD_MAX_STR) {
+ ngx_http_statsd_udp_send(ulcf->endpoint, startline, line - startline - sizeof(char));
+ }
return NGX_OK;
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.