gamenet / redis-memory-analyzer Goto Github PK
View Code? Open in Web Editor NEWRedis memory profiler to find the RAM bottlenecks throw scaning key space in real time and aggregate RAM usage statistic by patterns.
License: MIT License
Redis memory profiler to find the RAM bottlenecks throw scaning key space in real time and aggregate RAM usage statistic by patterns.
License: MIT License
Can you please add min and max count of elements in the list statistics?
Hi all
I've got another crash yesterday:
`
Traceback (most recent call last):
File "/usr/bin/rma", line 11, in
sys.exit(main())
File "/usr/lib/python3.5/site-packages/rma/cli/rma_cli.py", line 60, in main
app.run()
File "/usr/lib/python3.5/site-packages/rma/application.py", line 113, in run
str_res += self.do_ram(res)
File "/usr/lib/python3.5/site-packages/rma/application.py", line 147, in do_ram
ret += (rule.analyze(aggregate_patterns))
File "/usr/lib/python3.5/site-packages/rma/rule/List.py", line 74, in analyze
agg = ListAggreegator((ListStatEntry(x, self.redis) for x in data), len(data))
File "/usr/lib/python3.5/site-packages/rma/rule/List.py", line 42, in init
self.encoding = pref_encoding([obj.encoding for obj in g00])
File "/usr/lib/python3.5/site-packages/rma/rule/List.py", line 42, in
self.encoding = pref_encoding([obj.encoding for obj in g00])
File "/usr/lib/python3.5/site-packages/rma/rule/List.py", line 74, in
agg = ListAggreegator((ListStatEntry(x, self.redis) for x in data), len(data))
File "/usr/lib/python3.5/site-packages/rma/rule/List.py", line 16, in init
self.encoding = redis.object('encoding', key_name).decode('utf8')
AttributeError: 'NoneType' object has no attribute 'decode'
'
Using Python 3.8 I'm getting this error ...
Traceback (most recent call last):
File "/usr/local/bin/rma", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/rma/cli/rma_cli.py", line 96, in main
start_time = time.clock()
AttributeError: module 'time' has no attribute 'clock'
Likely cause is this:
The function time.clock() has been removed, after having been deprecated since Python 3.3: use time.perf_counter() or time.process_time() instead, depending on your requirements, to have well-defined behavior. (Contributed by Matthias Bussonnier in bpo-36895.)
Hi! I'm deserializing the output from rma and it seems that stat is not outputting a snake case in the json response:
{ "Avg": 0, "Count": 1, "Encoding": "", "Free": 0, "Match": "item:*", "Max": 0, "Min": 0, "Ratio": 0.0, "Real": 0, "TTL Avg": -1, "TTL Max": -1, "TTL Min": -1, "Useful": 0 }
Can't we make the field naming policy consistent to make it easier to deserialize? Thank you!
Now we scanning of large database can demand up to 3-4 hours. Besides perfomance issues we should add some progress bars to easier to understand what's going on.
rma -s 10.0.0.3
Match *: 0%| | 0/34 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/usr/local/bin/rma", line 9, in
load_entry_point('rma==0.1.16', 'console_scripts', 'rma')()
File "/usr/local/lib/python3.5/dist-packages/rma-0.1.16-py3.5.egg/rma/cli/rma_cli.py", line 97, in main
File "/usr/local/lib/python3.5/dist-packages/rma-0.1.16-py3.5.egg/rma/application.py", line 127, in run
File "/usr/local/lib/python3.5/dist-packages/rma-0.1.16-py3.5.egg/rma/scanner.py", line 85, in scan
File "/usr/local/lib/python3.5/dist-packages/rma-0.1.16-py3.5.egg/rma/scanner.py", line 53, in batch_scan
File "/usr/local/lib/python3.5/dist-packages/rma-0.1.16-py3.5.egg/rma/scanner.py", line 61, in resolve_types
File "/usr/local/lib/python3.5/dist-packages/rma-0.1.16-py3.5.egg/rma/scanner.py", line 58, in resolve_types
File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 3498, in call
return client.evalsha(self.sha, len(keys), *args)
File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 2704, in evalsha
return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 755, in execute_command
return self.parse_response(connection, command_name, **options)
File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 768, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 638, in read_response
raise response
redis.exceptions.ResponseError: Error running script (call to f_b89215e5d8808de13e1c715aaedbb744ce4536e7): @user_script:5: @user_script: 5: Unknown Redis command called from Lua script
Any ideas ?
When trying to run against a Redis Cluster node:
Traceback (most recent call last):
File "/usr/local/bin/rma", line 9, in <module>
load_entry_point('rma==0.1.6', 'console_scripts', 'rma')()
File "/usr/local/lib/python3.4/dist-packages/rma/cli/rma_cli.py", line 95, in main
app.run()
File "/usr/local/lib/python3.4/dist-packages/rma/application.py", line 109, in run
for v in scanner.scan(limit=self.limit):
File "/usr/local/lib/python3.4/dist-packages/rma/scanner.py", line 61, in scan
for key_tuple in self.batch_scan():
File "/usr/local/lib/python3.4/dist-packages/rma/scanner.py", line 44, in batch_scan
yield from self.resolve_types(ret)
File "/usr/local/lib/python3.4/dist-packages/rma/scanner.py", line 50, in resolve_types
key_with_types = msgpack.unpackb(self.resolve_types_script(ret))
File "/usr/local/lib/python3.4/dist-packages/redis/client.py", line 2694, in __call__
return client.evalsha(self.sha, len(keys), *args)
File "/usr/local/lib/python3.4/dist-packages/redis/client.py", line 1944, in evalsha
return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
File "/usr/local/lib/python3.4/dist-packages/redis/client.py", line 573, in execute_command
return self.parse_response(connection, command_name, **options)
File "/usr/local/lib/python3.4/dist-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.4/dist-packages/redis/connection.py", line 582, in read_response
raise response
redis.exceptions.ResponseError: CROSSSLOT Keys in request don't hash to the same slot
This looks similar to #23 and #29 but in a different location.
Python version 3.9.2
Redis version 6.0.8
/ # rma -b ram -s redis-master.production [12/57307]
Match *: 100%|██████████████▉| 55430/55447 [00:01<00:00, 34761.63it/s]
Aggregating keys by pattern and type Apply rules
Processing Set patterns: 0%| | 0/6 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/usr/local/bin/rma", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/rma/cli/rma_cli.py", line 105, in main
app.run()
File "/usr/local/lib/python3.9/site-packages/rma/application.py", line 143, in run
str_res.append(self.do_ram(keys))
File "/usr/local/lib/python3.9/site-packages/rma/application.py", line 176, in do_ram
ret[redis_type] = rule.analyze(keys=aggregate_patterns, total=total_keys)
File "/usr/local/lib/python3.9/site-packages/rma/rule/Set.py", line 90, in analyze
agg = SetAggregator(progress_iterator((SetStatEntry(x, self.redis) for x in data), progress), len(data))
File "/usr/local/lib/python3.9/site-packages/rma/rule/Set.py", line 59, in __init__
self.ttlMin = min(ttls)
ValueError: min() arg is an empty sequence
Aggregating keys by pattern and type
Traceback (most recent call last):
File "/home/ubuntu/.local/bin/rma", line 11, in <module>
sys.exit(main())
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/cli/rma_cli.py", line 97, in main
app.run()
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/application.py", line 133, in run
keys = {k: self.get_pattern_aggregated_data(v) for k, v in keys.items()}
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/application.py", line 133, in <dictcomp>
keys = {k: self.get_pattern_aggregated_data(v) for k, v in keys.items()}
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/application.py", line 181, in get_pattern_aggregated_data
split_patterns = self.splitter.split((ptransform(obj["name"]) for obj in data))
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/splitter.py", line 32, in split
return self.unfold_to_list(pass2, separator)
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/splitter.py", line 82, in unfold_to_list
for compound_key in dict_build(sub_tree):
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/splitter.py", line 10, in dict_build
for d in dict_build(value, pre=pre+[key]):
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/splitter.py", line 10, in dict_build
for d in dict_build(value, pre=pre+[key]):
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/splitter.py", line 10, in dict_build
for d in dict_build(value, pre=pre+[key]):
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/splitter.py", line 10, in dict_build
for d in dict_build(value, pre=pre+[key]):
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/splitter.py", line 10, in dict_build
for d in dict_build(value, pre=pre+[key]):
File "/home/ubuntu/.local/lib/python3.5/site-packages/rma/splitter.py", line 10, in dict_build
Hello, I don’t know much about the result parameters of memory analysis. Can you tell me something about it?
Match | Count | Avg field count | Key mem | Real | Ratio | Value mem | Real | Ratio | System | Encoding | Total mem | Total aligned |
---|---|---|---|---|---|---|---|---|---|---|---|---|
job:* | 5254 | 9.00 | 299485 | 619988 | 2.07 | 685451 | 942984 | 1.38 | 1345024 | ziplist [100.0%] | 984936 | 2907996 |
LIKE:* | 1890 | 1.02 | 5744 | 30262 | 5.27 | 1932 | 15432 | 7.99 | 91344 | ziplist [100.0%] | 7676 | 137038 |
game::count: | 1231 | 1.00 | 7386 | 19696 | 2.67 | 1234 | 9848 | 7.98 | 59088 | ziplist [100.0%] | 8620 | 88632 |
LIKE:game:like:* | 1207 | 1.00 | 3621 | 19312 | 5.33 | 1210 | 9656 | 7.98 | 57936 | ziplist [100.0%] | 4831 | 86904 |
integration:privacy:* | 530 | 3.00 | 20140 | 33920 | 1.68 | 0 | 25440 | 25440.00 | 42400 | ziplist [100.0%] | 20140 | 101760 |
I am also very interested in how the 10% of the info memory is achieved? Can you explain it to me? Thank you very much.
Hello,
Thanks for excellent tool. I was wondering how can I get it to ignore namespace, "production" as every key in my redis is prefixed with "production". right now my report looks like below:
name | count | type | percent |
---|---|---|---|
production:* | 7848517 | hash | 56.37% |
production:* | 3350289 | list | 24.06% |
production:* | 2038396 | set | 14.64% |
production:* | 664480 | string | 4.77% |
production:: | 86739 | hash | 0.62% |
Can we somehow tell RMA to skip this and group on 2nd level instead?
redis-memory-analyzer/rma/application.py
Line 163 in dd1e749
App crashes if db0 is empty. That's because the total number of keys is hard-coded to db0.
Match *: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 4995001/5000000 [01:27<00:00, 53299.06it/s]INFO:root:
Limit 5000000 reached
Aggregating keys by pattern and type
Apply rules
Processing keys: 0%| | 0/355 [00:00<?, ?it/s]Traceback (most recent call last):
File "/usr/local/bin/rma", line 9, in <module>
load_entry_point('rma==0.1.13', 'console_scripts', 'rma')()
File "/usr/local/lib/python3.5/site-packages/rma-0.1.13-py3.5.egg/rma/cli/rma_cli.py", line 92, in main
File "/usr/local/lib/python3.5/site-packages/rma-0.1.13-py3.5.egg/rma/application.py", line 118, in run
File "/usr/local/lib/python3.5/site-packages/rma-0.1.13-py3.5.egg/rma/application.py", line 153, in do_ram
File "/usr/local/lib/python3.5/site-packages/rma-0.1.13-py3.5.egg/rma/rule/KeyString.py", line 57, in analyze
ValueError: min() arg is an empty sequence
After trying @misterion's fix:
total_elements = len(data)
if total_elements == 0:
continue
the problem looks like
Traceback (most recent call last):
File "/usr/local/bin/rma", line 9, in <module>
load_entry_point('rma==0.1.13', 'console_scripts', 'rma')()
File "/usr/local/lib/python3.5/site-packages/rma-0.1.13-py3.5.egg/rma/cli/rma_cli.py", line 92, in main
File "/usr/local/lib/python3.5/site-packages/rma-0.1.13-py3.5.egg/rma/application.py", line 118, in run
File "/usr/local/lib/python3.5/site-packages/rma-0.1.13-py3.5.egg/rma/application.py", line 153, in do_ram
File "/usr/local/lib/python3.5/site-packages/rma-0.1.13-py3.5.egg/rma/rule/Hash.py", line 96, in analyze
File "/usr/local/lib/python3.5/site-packages/rma-0.1.13-py3.5.egg/rma/rule/Hash.py", line 63, in __init__
ValueError: min() arg is an empty sequence
I install python3.5 and use pip to install rma. And run
rma -l 100
and get error
Match *: 0%| | 0/100 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/usr/local/bin/rma", line 9, in <module>
load_entry_point('rma==0.1.14', 'console_scripts', 'rma')()
File "/usr/local/lib/python3.5/site-packages/rma/cli/rma_cli.py", line 92, in main
app.run()
File "/usr/local/lib/python3.5/site-packages/rma/application.py", line 106, in run
for v in scanner.scan(limit=self.limit):
File "/usr/local/lib/python3.5/site-packages/rma/scanner.py", line 96, in scan
'name': key_name.decode("utf-8"),
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xac in position 0: invalid start byte
It seems rma try to decode key name, but the key name can be byte stream.
[root@ip-XX ec2-user]# pip install rma
Collecting rma
Downloading rma-0.1.16.zip (41kB)
100% |████████████████████████████████| 51kB 7.5MB/s
Collecting redis (from rma)
Downloading redis-2.10.6-py2.py3-none-any.whl (64kB)
100% |████████████████████████████████| 71kB 9.1MB/s
Collecting tabulate (from rma)
Downloading tabulate-0.8.2.tar.gz (45kB)
100% |████████████████████████████████| 51kB 10.6MB/s
Collecting tqdm (from rma)
Downloading tqdm-4.19.4-py2.py3-none-any.whl (50kB)
100% |████████████████████████████████| 51kB 9.5MB/s
Collecting msgpack-python (from rma)
Downloading msgpack-python-0.4.8.tar.gz (113kB)
100% |████████████████████████████████| 122kB 9.3MB/s
Installing collected packages: redis, tabulate, tqdm, msgpack-python, rma
Running setup.py install for tabulate ... done
Running setup.py install for msgpack-python ... done
Running setup.py install for rma ... done
Successfully installed msgpack-python-0.4.8 redis-2.10.6 rma-0.1.16 tabulate-0.8.2 tqdm-4.19.4
[root@ip-XX ec2-user]# rma
[ec2-user@ip-XX ~]$ rma --help
Traceback (most recent call last):
File "/usr/local/bin/rma", line 11, in
load_entry_point('rma==0.1.16', 'console_scripts', 'rma')()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 564, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2662, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2316, in load
return self.resolve()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2322, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/usr/local/lib/python2.7/site-packages/rma/init.py", line 1, in
from rma.application import RmaApplication
File "/usr/local/lib/python2.7/site-packages/rma/application.py", line 145
self.reporter.print(str_res)
^
SyntaxError: invalid syntax
[ec2-user@ip-XX ~]$ cat /etc/issue
Amazon Linux AMI release 2017.09
Kernel \r on an \m
When trying to run on RedisCloud, I get the below error.
The error makes not of using EVAL instead of EVALSHA.
I'm not familiar with the lua capabilities, nor the internals of RMA, so I'm not sure if it is feasible to fall back to using EVAL if EVALSHA fails?
Any suggestions?
rma -s <host> -p <port> -d 0 -a <password>
Match *: 0%| | 0/331 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/redis/client.py", line 2694, in __call__
return client.evalsha(self.sha, len(keys), *args)
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/redis/client.py", line 1944, in evalsha
return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/redis/client.py", line 573, in execute_command
return self.parse_response(connection, command_name, **options)
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
raise response
redis.exceptions.NoScriptError: No matching script. Please use EVAL.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/adamgriffiths/.anaconda/envs/systema3/bin/rma", line 11, in <module>
sys.exit(main())
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/rma/cli/rma_cli.py", line 92, in main
app.run()
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/rma/application.py", line 106, in run
for v in scanner.scan(limit=self.limit):
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/rma/scanner.py", line 85, in scan
for key_tuple in self.batch_scan():
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/rma/scanner.py", line 53, in batch_scan
yield from self.resolve_types(ret)
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/rma/scanner.py", line 61, in resolve_types
raise e
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/rma/scanner.py", line 58, in resolve_types
key_with_types = msgpack.unpackb(self.resolve_types_script(ret))
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/redis/client.py", line 2699, in __call__
return client.evalsha(self.sha, len(keys), *args)
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/redis/client.py", line 1944, in evalsha
return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/redis/client.py", line 573, in execute_command
return self.parse_response(connection, command_name, **options)
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/home/adamgriffiths/.anaconda/envs/systema3/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
raise response
redis.exceptions.ResponseError: Error running script (call to f_b89215e5d8808de13e1c715aaedbb744ce4536e7): @user_script:5: @user_script: 5: This Redis command is not allowed from scripts
This is the same error as issue #23 but in different location. This is happening when using the ram behavior. All other behaviors are working. Using the latest version 0.1.14.
Match *: 39452it [00:00, 54207.86it/s]
Aggregating keys by pattern and type
Apply rules
Processing String patterns: 0%| | 0/1 [00:00<?, ?it/s]WARNING:rma.rule.ValueString:ResponseError("unknown command 'DEBUG'",)
Traceback (most recent call last):
File "/usr/local/bin/rma", line 9, in <module>
load_entry_point('rma==0.1.14', 'console_scripts', 'rma')()
File "/usr/local/lib/python3.4/dist-packages/rma/cli/rma_cli.py", line 92, in main
app.run()
File "/usr/local/lib/python3.4/dist-packages/rma/application.py", line 118, in run
str_res += self.do_ram(keys)
File "/usr/local/lib/python3.4/dist-packages/rma/application.py", line 153, in do_ram
ret += (rule.analyze(keys=aggregate_patterns, total=total_keys))
File "/usr/local/lib/python3.4/dist-packages/rma/rule/ValueString.py", line 103, in analyze
min_bytes = min(used_bytes)
ValueError: min() arg is an empty sequence
subj, make also varsion sharing in right way (more about this http://stackoverflow.com/questions/17583443/what-is-the-correct-way-to-share-package-version-with-setup-py-and-the-package/17626524#17626524). From solution in #16
I'm running on python 3.5 and I installed RMA from this source code directly. If I run something like
rma -s <instance> -d 6 -b scanner
I see proper results, but when specifying other 'behaviors' or even not specifying that flag all yield the same error message:
Aggregating keys by pattern and type
Apply rules
WARNING:rma.rule.GlobalKeySpace:*max* option skipped: ResponseError("unknown command 'CONFIG'",)
Processing String patterns: 0%| | 0/164933 [00:00<?, ?it/s]WARNING:rma.rule.ValueString:ResponseError("unknown command 'DEBUG'",)
WARNING:rma.rule.ValueString:ResponseError("unknown command 'DEBUG'",)
....
And that warning repeats itself (maybe once for ever key?).
I'm using this on AWS's ElastiCache running redis 2.8.22 on a debug instance I created from a snapshot. Is amazon preventing this debug command from being executed somehow?? I checked the parameter group settings and couldn't find an option that looked like it matched. Any advise?
Thanks!
James Flowers
I run rma with limit setting
rma -s host -l 100000
got the results and see that percents in the table display % of all redis keys instead % of limit
Keys by types
| Match | Count | Type | % |
|:--------------------------------------------|--------:|:-------|:------|
| event:user:* | 46232 | list | 0.14% |
| event:data:* | 42982 | string | 0.13% |
| notification:data:* | 5552 | string | 0.01% |
| LIKE:LIST:* | 2370 | zset | 0.00% |
| LIKE:* | 2349 | hash | 0.00% |
| event:game:* | 158 | list | 0.00% |
| user:event_personal:* | 143 | zset | 0.00% |
| jabber:userRosterGroup:* | 108 | string | 0.00% |
| user:*:info | 27 | string | 0.00% |
| user:ban:target:log:* | 22 | list | 0.00% |
| LIKE:LIST:beauty:* | 16 | zset | 0.00% |
| user:privacy:* | 13 | hash | 0.00% |
| event:users:* | 12 | list | 0.00% |
| LIKE:beauty:* | 9 | hash | 0.00% |
| user:*:blocklog | 8 | string | 0.00% |
| Syncopate\Features\BeautyContest\Feed:*:*:* | 8 | set | 0.00% |
| LIKE:guild:* | 7 | hash | 0.00% |
| user:*:unblockkey | 6 | string | 0.00% |
| LIKE:LIST:guild:* | 5 | zset | 0.00% |
| beauty:ip:* | 3 | set | 0.00% |
| slide:info:*:* | 2 | hash | 0.00% |
| game:event_guides:* | 2 | zset | 0.00% |
| mm:urllist:* | 1 | string | 0.00% |
| game:*:*:categoryIndex | 1 | string | 0.00% |
| user:spamblocked:dialy:comment:* | 1 | string | 0.00% |
Can you add support for connecting to servers with SSL?
I have a lot of keys with invalid namespace ('_' separator instead of ':'), this lead to huge report full of these keys.
I'd love to have '--exclude' parameter, ideally regexp, that would allow me to put there all problematic keys, like group of keys with non ':' separator and other like this.
This way I can run report and update my exclude rule until I get only keys with normal namespace. Then after fixing namespace separator for all the problematic keys, I'd be able to re-run the test without the exclude rule and see only normal grouped keys.
If it's decided to have '-exclude ' but accept matcher string with wildcards, instead of regexp, then it should allow having multiple '--exclude' parameters.
after installing with
pip install rma
executing rma --help ends with
I am not familiar with python but it seems I have a missing dependency but I cannot figure out what it is ?!
Traceback (most recent call last):
File "/usr/bin/rma", line 9, in <module>
load_entry_point('rma==0.1.16', 'console_scripts', 'rma')()
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 378, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2566, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "build/bdist.linux-x86_64/egg/rma/__init__.py", line 1, in <module>
File "/usr/lib/python2.7/site-packages/rma-0.1.16-py2.7.egg/rma/application.py", line 145
self.reporter.print(str_res)
Hi
Current output format seems human-readable-friendly, but hard to process/use on scripts.
Have you considered doing a JSON format version of this tool?
Regards
I have python 3.7 installed. I am using a mac. The default installation of python 3 for mac users leaves python 2 installed because python 3 is not backwards compatible: https://stackoverflow.com/questions/5846167/how-to-change-default-python-version
However rma does not look for the python 3 libraries it needs, instead it uses the default ones and fails as follows:
rma --help
Traceback (most recent call last):
File "/usr/local/bin/rma", line 6, in <module>
from rma.cli.rma_cli import main
File "/usr/local/lib/python2.7/site-packages/rma/__init__.py", line 1, in <module>
from rma.application import RmaApplication
File "/usr/local/lib/python2.7/site-packages/rma/application.py", line 145
self.reporter.print(str_res)
^
SyntaxError: invalid syntax
I do not use python regularly as a programming language, so I am unfamiliar with how the python community typically resolves this issue. I am trying to use rma because it provides a unique feature set. It would be helpful if rma either did the right thing in this situation or there was a paragraph in the README explaining how to resolve this.
Hi
I'm trying to run rma on our redis instance, but it crashed after few hours of work. Here is the stacktrace:
WARNING:rma.rule.ValueString:Invalid encoding from server for key
event:data:1HDcLhTraceback (most recent call last): File "/usr/bin/rma", line 11, in <module> sys.exit(main()) File "/usr/lib/python3.5/site-packages/rma/cli/rma_cli.py", line 60, in main app.run() File "/usr/lib/python3.5/site-packages/rma/application.py", line 113, in run str_res += self.do_ram(res) File "/usr/lib/python3.5/site-packages/rma/application.py", line 147, in do_ram ret += (rule.analyze(aggregate_patterns)) File "/usr/lib/python3.5/site-packages/rma/rule/ValueString.py", line 67, in analyze with RealStringEntry(key_name=x, redis=self.redis) as stat: File "/usr/lib/python3.5/site-packages/rma/rule/ValueString.py", line 37, in __init__ sdslen_response = redis.debug_sdslen(key_name) File "/usr/lib/python3.5/site-packages/rma/redis.py", line 225, in debug_sdslen return parse_debug(self.execute_command("DEBUG SDSLEN", key)) File "/usr/lib/python3.5/site-packages/redis/client.py", line 573, in execute_command return self.parse_response(connection, command_name, **options) File "/usr/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response response = connection.read_response() File "/usr/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response raise response redis.exceptions.ResponseError: no such key
rma -s bg-redis.x.x.x.x -p 6379 -a password
Match *: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋| 378067/378654 [04:42<00:00, 1337.09it/s]
Aggregating keys by pattern and type
Traceback (most recent call last):
File "/Users/syedahmed/anaconda3/bin/rma", line 11, in
sys.exit(main())
File "/Users/syedahmed/anaconda3/lib/python3.7/site-packages/rma/cli/rma_cli.py", line 97, in main
app.run()
File "/Users/syedahmed/anaconda3/lib/python3.7/site-packages/rma/application.py", line 133, in run
keys = {k: self.get_pattern_aggregated_data(v) for k, v in keys.items()}
File "/Users/syedahmed/anaconda3/lib/python3.7/site-packages/rma/application.py", line 133, in
keys = {k: self.get_pattern_aggregated_data(v) for k, v in keys.items()}
File "/Users/syedahmed/anaconda3/lib/python3.7/site-packages/rma/application.py", line 186, in get_pattern_aggregated_data
aggregate_patterns[pattern] = list(filter(lambda obj: fnmatch.fnmatch(ptransform(obj["name"]), pattern), data))
File "/Users/syedahmed/anaconda3/lib/python3.7/site-packages/rma/application.py", line 186, in
aggregate_patterns[pattern] = list(filter(lambda obj: fnmatch.fnmatch(ptransform(obj["name"]), pattern), data))
File "/Users/syedahmed/anaconda3/lib/python3.7/fnmatch.py", line 36, in fnmatch
return fnmatchcase(name, pat)
File "/Users/syedahmed/anaconda3/lib/python3.7/fnmatch.py", line 70, in fnmatchcase
match = _compile_pattern(pat)
File "/Users/syedahmed/anaconda3/lib/python3.7/fnmatch.py", line 46, in _compile_pattern
return re.compile(res).match
File "/Users/syedahmed/anaconda3/lib/python3.7/re.py", line 234, in compile
return _compile(pattern, flags)
File "/Users/syedahmed/anaconda3/lib/python3.7/re.py", line 286, in _compile
p = sre_compile.compile(pattern, flags)
File "/Users/syedahmed/anaconda3/lib/python3.7/sre_compile.py", line 764, in compile
p = sre_parse.parse(p, flags)
File "/Users/syedahmed/anaconda3/lib/python3.7/sre_parse.py", line 930, in parse
p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
File "/Users/syedahmed/anaconda3/lib/python3.7/sre_parse.py", line 426, in _parse_sub
not nested and not items))
File "/Users/syedahmed/anaconda3/lib/python3.7/sre_parse.py", line 816, in _parse
p = _parse_sub(source, state, sub_verbose, nested + 1)
File "/Users/syedahmed/anaconda3/lib/python3.7/sre_parse.py", line 426, in _parse_sub
not nested and not items))
File "/Users/syedahmed/anaconda3/lib/python3.7/sre_parse.py", line 580, in _parse
raise source.error(msg, len(this) + 1 + len(that))
re.error: bad character range y-m at position 167
I installed rma via pip install rma
but then getting the error when trying to run:
[06:48] # rma --help
Traceback (most recent call last):
File "/usr/local/bin/rma", line 9, in <module>
load_entry_point('rma==0.1.16', 'console_scripts', 'rma')()
File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 521, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2632, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2312, in load
return self.resolve()
File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2318, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/local/lib/python2.7/site-packages/rma/__init__.py", line 1, in <module>
from rma.application import RmaApplication
File "/usr/local/lib/python2.7/site-packages/rma/application.py", line 145
self.reporter.print(str_res)
^
SyntaxError: invalid syntax
I have Python 2.7.10 installed
Apply rules
Processing String patterns: 18%|███████████████████████▊ | 856/4819 [01:09<05:37, 11.74it/s]WARNING:rma.rule.ValueString:ResponseError('no such key',)
Processing String patterns: 19%|█████████████████████████▊ | 927/4819 [01:14<04:56, 13.12it/s]WARNING:rma.rule.ValueString:ResponseError('no such key',)
Processing String patterns: 20%|██████████████████████████▉ | 970/4819 [01:17<04:47, 13.37it/s]WARNING:rma.rule.ValueString:ResponseError('no such key',)
WARNING:rma.rule.ValueString:ResponseError('no such key',)
Processing String patterns: 23%|██████████████████████████████▌ | 1107/4819 [01:27<04:21, 14.18it/s]WARNING:rma.rule.ValueString:ResponseError('no such key',)
Processing String patterns: 23%|██████████████████████████████▉ | 1123/4819 [01:28<04:16, 14.43it/s]WARNING:rma.rule.ValueString:ResponseError('no such key',)
Processing String patterns: 33%|███████████████████████████████████████████▊
Processing Set patterns step failed on version 0.1.16 with unsupported operand type error.
RMA installed with pip. Python 3.4.
Processing Set patterns: 43%|████████████████████████████████████▍ | 3/7 [00:10<00:14, 3.58s/it]
Traceback (most recent call last):
File "/home/user/.local/bin/rma", line 9, in <module>
load_entry_point('rma==0.1.16', 'console_scripts', 'rma')()
File "/home/user/.local/lib/python3.4/site-packages/rma/cli/rma_cli.py", line 97, in main
app.run()
File "/home/user/.local/lib/python3.4/site-packages/rma/application.py", line 143, in run
str_res.append(self.do_ram(keys))
File "/home/user/.local/lib/python3.4/site-packages/rma/application.py", line 176, in do_ram
ret[redis_type] = rule.analyze(keys=aggregate_patterns, total=total_keys)
File "/home/user/.local/lib/python3.4/site-packages/rma/rule/Set.py", line 84, in analyze
agg = SetAggregator(progress_iterator((SetStatEntry(x, self.redis) for x in data), progress), len(data))
File "/home/user/.local/lib/python3.4/site-packages/rma/rule/Set.py", line 43, in __init__
self.encoding = pref_encoding([obj.encoding for obj in g00], redis_encoding_id_to_str)
File "/home/user/.local/lib/python3.4/site-packages/rma/rule/Set.py", line 43, in <listcomp>
self.encoding = pref_encoding([obj.encoding for obj in g00], redis_encoding_id_to_str)
File "/home/user/.local/lib/python3.4/site-packages/rma/helpers/__init__.py", line 6, in progress_iterator
for i in x:
File "/home/user/.local/lib/python3.4/site-packages/rma/rule/Set.py", line 84, in <genexpr>
agg = SetAggregator(progress_iterator((SetStatEntry(x, self.redis) for x in data), progress), len(data))
File "/home/user/.local/lib/python3.4/site-packages/rma/rule/Set.py", line 26, in __init__
self.total = self.valueAlignedBytes + self.system
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
Error occurs only while processing single database. With others one works fine.
First of all, thank you for this tool.
I've encountered an issue while I was trying to analyze what's taking the memory. While there's a guard against some cases when keys were removed after the scan, rma
crashes when it encounters an integer-containing key that's already gone when ValueString
gets to it.
Here's an example traceback:
Traceback (most recent call last):
File ".../rma", line 9, in <module>
load_entry_point('rma==0.1.11', 'console_scripts', 'rma')()
File ".../rma/cli/rma_cli.py", line 92, in main
app.run()
File ".../rma/application.py", line 118, in run
str_res += self.do_ram(keys)
File ".../rma/application.py", line 153, in do_ram
ret += (rule.analyze(keys=aggregate_patterns, total=total_keys))
File ".../rma/rule/ValueString.py", line 81, in analyze
with RealStringEntry(redis=self.redis, info=key_info, use_debug=use_debug_command) as stat:
File ".../rma/rule/ValueString.py", line 34, in __init__
self.useful_bytes = self.get_int_encoded_bytes(redis, key_name)
File ".../rma/rule/ValueString.py", line 13, in get_int_encoded_bytes
num_value = int(redis.get(key_name))
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
The solution seems to handle TypeError
as well as ValueError
in RealStringEntry.get_int_encoded_bytes
. E.g. replacing except ValueError:
with except (ValueError, TypeError):
. Tried this on my local copy and it worked for me - I got a sensible-looking report. I'm not really sure if that's completely correct, though - maybe manually raising ResponseError("no such key")
would be a better approach there.
Thanks!
Got another crash today
Traceback (most recent call last):
File "/usr/bin/rma", line 9, in <module>
load_entry_point('rma==0.1.5', 'console_scripts', 'rma')()
File "/usr/lib/python3.5/site-packages/rma/cli/rma_cli.py", line 95, in main
app.run()
File "/usr/lib/python3.5/site-packages/rma/application.py", line 121, in run
str_res += self.do_ram(keys)
File "/usr/lib/python3.5/site-packages/rma/application.py", line 155, in do_ram
ret += (rule.analyze(aggregate_patterns))
File "/usr/lib/python3.5/site-packages/rma/rule/List.py", line 75, in analyze
agg = ListAggregator((ListStatEntry(x, self.redis) for x in data), len(data))
File "/usr/lib/python3.5/site-packages/rma/rule/List.py", line 43, in __init__
self.encoding = pref_encoding([obj.encoding for obj in encode_iter], redis_encoding_id_to_str)
File "/usr/lib/python3.5/site-packages/rma/rule/List.py", line 43, in <listcomp>
self.encoding = pref_encoding([obj.encoding for obj in encode_iter], redis_encoding_id_to_str)
File "/usr/lib/python3.5/site-packages/rma/rule/List.py", line 75, in <genexpr>
agg = ListAggregator((ListStatEntry(x, self.redis) for x in data), len(data))
File "/usr/lib/python3.5/site-packages/rma/rule/List.py", line 33, in __init__
self.valueMin = min(min_iter)
ValueError: min() arg is an empty sequence
Hi,
Trying to install on ubuntu 14.04:
Downloading/unpacking rma
Downloading rma-0.1.5.tar.gz
Running setup.py (path:/tmp/pip_build_root/rma/setup.py) egg_info for package rma
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_root/rma/setup.py", line 47
setup(**sdict, install_requires=['redis', 'tabulate', 'tqdm', 'msgpack-python'])
^
SyntaxError: invalid syntax
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_root/rma/setup.py", line 47
setup(**sdict, install_requires=['redis', 'tabulate', 'tqdm', 'msgpack-python'])
^
SyntaxError: invalid syntax
----------------------------------------
-t option of rma doesn't work. I've run rma like this:
ma -s 172.29.7.13 -t string -l 100000
and got statistics about all types of keys
redis.exceptions.ResponseError: unknown command 'CONFIG'
I think this is because ElasticCache comes with come commands disabled. Is this step critical?
Now most of rules use one Redis encoding request per key and this is one perfomance breakdown issue in this tool. Scanner output should be rewrited to return tuple with type as key and value as list of tuples with key name field, encoding field and specific output (for example sdslen for strings).
if msgpack package (that is a newer version of msgpack-python) is already installed in venv, rma have the following backtrace
Traceback (most recent call last):
File "/Users/xxx/.pyenv/versions/venv-py37/bin/rma", line 11, in <module>
load_entry_point('rma', 'console_scripts', 'rma')()
File "/Users/xxx/Documents/Workspace/redis-memory-analyzer/rma/cli/rma_cli.py", line 109, in main
app.run()
File "/Users/xxx/Documents/Workspace/redis-memory-analyzer/rma/application.py", line 127, in run
for v in scanner.scan(limit=self.limit):
File "/Users/xxx/Documents/Workspace/redis-memory-analyzer/rma/scanner.py", line 100, in scan
'encoding': redis_encoding_str_to_id(key_encoding),
File "/Users/xxx/Documents/Workspace/redis-memory-analyzer/rma/redis_types.py", line 58, in redis_encoding_str_to_id
raise ValueError("Invalid encoding `%s` given" % key_encoding)
ValueError: Invalid encoding `raw` given
This is because of msgpack seems to return str instead of bytes
Adding bytes conversion could resolve that
def redis_encoding_str_to_id(key_encoding):
key_encoding = key_encoding.encode("utf8")
if key_encoding in REDIS_ENCODING_STR_TO_ID_LIB:
return REDIS_ENCODING_STR_TO_ID_LIB[key_encoding]
raise ValueError("Invalid encoding `%s` given" % key_encoding)
I'm getting the following trace. Any thoughts on the fix?
Match *: 99%|################################################################################################################################################################ | 494131/500000 [00:08<00:00, 62306.83it/s]INFO:root:
Limit 500000 reached
Match *: 500001it [00:08, 58686.04it/s]
Aggregating keys by pattern and type
Apply rules
WARNING:rma.rule.GlobalKeySpace:*max* option skipped: ResponseError("unknown command 'CONFIG'",)
Processing String patterns: 24%|##################################7 | 119871/496878 [00:20<01:02, 6006.85it/s]WARNING:rma.rule.ValueString:ResponseError("unknown command 'DEBUG'",)
Processing Hash patterns: 0%| | 0/73 [00:00<?, ?it/s]Traceback (most recent call last):
File "/usr/local/bin/rma", line 9, in <module>
load_entry_point('rma==0.1.14', 'console_scripts', 'rma')()
File "/usr/local/lib/python3.4/dist-packages/rma/cli/rma_cli.py", line 92, in main
app.run()
File "/usr/local/lib/python3.4/dist-packages/rma/application.py", line 118, in run
str_res += self.do_ram(keys)
File "/usr/local/lib/python3.4/dist-packages/rma/application.py", line 153, in do_ram
ret += (rule.analyze(keys=aggregate_patterns, total=total_keys))
File "/usr/local/lib/python3.4/dist-packages/rma/rule/Hash.py", line 98, in analyze
agg = HashAggregator(progress_iterator((HashStatEntry(x, self.redis) for x in data), progress), len(data))
File "/usr/local/lib/python3.4/dist-packages/rma/rule/Hash.py", line 55, in __init__
self.encoding = pref_encoding([obj.encoding for obj in g00], redis_encoding_id_to_str)
File "/usr/local/lib/python3.4/dist-packages/rma/rule/Hash.py", line 55, in <listcomp>
self.encoding = pref_encoding([obj.encoding for obj in g00], redis_encoding_id_to_str)
File "/usr/local/lib/python3.4/dist-packages/rma/helpers/__init__.py", line 6, in progress_iterator
for i in x:
File "/usr/local/lib/python3.4/dist-packages/rma/rule/Hash.py", line 98, in <genexpr>
agg = HashAggregator(progress_iterator((HashStatEntry(x, self.redis) for x in data), progress), len(data))
File "/usr/local/lib/python3.4/dist-packages/rma/rule/Hash.py", line 43, in __init__
self.fieldMin = min(args2)
ValueError: min() arg is an empty sequence
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.