Comments (6)
Can't we just expand a field name that contains dots – e.g. "permission.user.read"
– to a nested object or would this be breaking any conventions?
from logstash-filter-grok.
I also have replicated this problem with Logstash 1.5.6 and Logstash 2.10. Grok Filter and Nested Objects/Fields Ambiguity, I only just found this issue and seems like it's in a more relevant location with respect to the Grok filter plugin.
What I find strange is that if you launch Logstash with --debug
you can see the semantic matches like %{PATTERN:[object][field]}
being expanded similar to a the custom pattern matches.
from logstash-filter-grok.
So, turns out that the grok filter uses Oniguruma syntax. Looking into the Oniguruma syntax, you can see in L209 of the syntax guide, how the name match reflects that in the custom pattern match of the grok filter: Ruby Regex Syntax
[(?<name>subexp)] define named group
(All characters of the name must be a word character.)
There you have it, it explains the invalid char in group name
error in more detail by defining what characters are legal, of which the square brackets are not included. This is an upstream change and would be hard to justify to push it that far up. There may also be some regex related reasons, such as square brackets usually referencing a regex character class.
I think the only current option is to create the custom pattern file and reference it in the grok match:
Pattern File
PERMS [r-]
Grok Filter
grok {
patterns_dir => "/etc/logstash/conf.d/patterns"
match => [ "source_field", "%{PERMS:[permission][user][read]}" ]
}
Due to the requirement of no period separated fields in Elasticsearch 2.0, which is how nested fields used to be referenced, I think it might be good to disambiguate the documentation to cover this senario in detail. Additionally, the Logstsah Configuration test should probably pick this up and concisely inform you of the reason.
from logstash-filter-grok.
Any update on this issue? I am having difficulty with the same problem (logstash 5.6.4)
from logstash-filter-grok.
from logstash-filter-grok.
Hey, Just found this issue.. had some troubles doing a regex capture, see #66
from logstash-filter-grok.
Related Issues (20)
- Allow grok pattern to be passed in as a parameter HOT 3
- Memory leak on 5.x
- Ability to auto-sort match options by frequency of match HOT 1
- getting values as an array HOT 1
- remove_field not working HOT 2
- "Prefix" functionality for grok HOT 4
- Track number of failed matches when using multiple pattern per field
- Warn when patterns don't have anchors HOT 1
- Significant Performance Regression using Jruby Timeouts HOT 8
- Implement ECS-Compatibility Mode HOT 4
- Logstash pipeline to remove passwords from log data HOT 1
- Using event fileds in configuration options HOT 1
- pure ruby regexp <capture:int> coercion does not work
- captures with same name won't coerce properly
- Logstash for iis SMTP
- Incoherent behavior of field references with overwrite
- [Test Failure] Syslog grok tests are failing on Logstash 8.x
- Behaviour when pattern writes to same input field (without "overwrite" option) HOT 1
- Add property to limit backtracking
- Regression on grok is case of match failure HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logstash-filter-grok.