document how to remove port from the instance label about docs HOT 8 OPEN

Maybe the above regexps could be improved a bit further - not from a functional PoV but in terms of understanding:

AFAIU, in Prometheus regexps are always anchored (which doesn't matter above), but one could either remove all my anchors (which I think makes it less readable as in all other places where regexps are used they're typically not anchored)... or fully anchor them explicitly on both sides.

But again, as all the unanchored sides have (.+) it doesn’t really matter.

from docs.

I believe you're over-thinking this. Just pick some character for the separator which you know is never going to appear in the __address__, given that you know it has to contain either an IP address or a DNS name.

    relabel_configs:
      - source_labels: [instance, __address__]
        separator: '!'
        regex: '!([^!]+):(\d+)'
        target_label: instance
        replacement: '${1}'   # this is implicit default anyway

Space works fine, as indeed should the default semicolon (how many DNS names do you see that contain semicolons?). Otherwise there are plenty of non-printing and unicode symbols you could use.

However, if I were documenting this as best practice, I wouldn't even do that. I would simply show:

    relabel_configs:
      - source_labels: [__address__]
        regex: '(.+):(\d+)'
        target_label: instance
        replacement: '${1}'   # this is implicit default anyway

This is because IMO, best practice is that the instance label consistently comes either from the rewriting rules, or from per-target labels in service discovery. Therefore, if you're doing it in rewriting, it's OK to ignore any value set in service discovery.

In any case, setting the instance label in SD only works in limited cases (static and file SD); as soon as you upgrade to a more sophisticated SD mechanism, they won't directly set the 'instance' label so you have to do it in rewriting rules.

from docs.

Prometheus regexps are always anchored (which doesn't matter above), but one could either remove all my anchors (which I think makes it less readable as in all other places where regexps are used they're typically not anchored)... or fully anchor them explicitly on both sides.

In Prometheus the anchoring of regexps is pervasive, and in my opinion it can and should be relied on. For example, in PromQL you would idiomatically write

foo{instance=~"bar|baz"}

and not

foo{instance=~"^(bar|baz)$"}

from docs.

I believe you're over-thinking this. Just pick some character for the separator which you know is never going to appear in the address, given that you know it has to contain either an IP address or a DNS name.

I don't think that this would really cover all thinkable cases, though.
instance, AFAIU, is only initialised with __address__ if not set, and only then one would have the guarantee that e.g. space isn't used as "literal" character.

But some awkward setups may simply always manually set instance to whichever value they like.

As for the anchoring,... as said, I'd be open to change that... though personally I still find it less easily readable without the anchors.

from docs.

instance, AFAIU, is only initialised with __address__ if not set

That's correct - and only after the target rewriting rules have run. So if instance was not set in the SD, and not set it target rewriting, then it's set to a copy of __address__.

and only then one would have the guarantee that e.g. space isn't used as "literal" character

Sorry, I don't understand that. __address__ as it comes from the service discovery should consist of either ipaddress:port or hostname:port, to be usable directly.

If it comes in any other form, then it needs rewriting to be applied to it anyway. For example, if SD returns hostname<space>ipaddress, then you can have rewriting rules to set "instance" to the part before the space, and "address" to the part after the space with ":port" appended.

But some awkward setups may simply always manually set instance to whichever value they like

They could, and it should work with the rules I gave. Can you give a concrete example where it would not?

from docs.

Sorry, I don't understand that. address as it comes from the service discovery should consist of either ipaddress:port or hostname:port, to be usable directly.

Yes.

If it comes in any other form, then it needs rewriting to be applied to it anyway. For example, if SD returns hostnameipaddress, then you can have rewriting rules to set "instance" to the part before the space, and "address" to the part after the space with ":port" appended.

If anything would return this as __address__ I'd rather consider it even to be a bug.

They could, and it should work with the rules I gave. Can you give a concrete example where it would not?

As far as I understand it, you 2nd rule, will simply always take __address__, try to split it up and if that works, it would set instance to that part before the final colon, right?
So it would override instance even if it had already set some value, e.g. intentionally by other label rewriting or target labels.

IMO that shouldn't be done, because if a user intentionally set that (or intentionally accepts foreign labels) then these are likely the values that are truly wanted.

And as far as I understand your 1st rule, if some awkward setup intentionally set instance to !foo it would result in something like !foo!hostname:port, which, when matched against the regex !([^!]+):(\d+) wouldn't match at all as the first subexpression can only match foo but then it expects a : but finds the 2nd !.

from docs.

If anything would return this as __address__ I'd rather consider it even to be a bug.

static_configs and file_sd_configs can return whatever strings you like for the __address__. (http_sd_configs too, now I think of it). Example here of how it can be useful.

As far as I understand it, you 2nd rule, will simply always take __address__, try to split it up and if that works, it would set instance to that part before the final colon, right?
So it would override instance even if it had already set some value, e.g. intentionally by other label rewriting or target labels.

That's correct. As I said before, that case was for the case where instance is always being set by rewriting rules. The first, and slightly more complex rule, is if you want any instance label set by static_configs etc to take precedence. I consider it bad practice to "mix and match" like that: I think you should be consistent, and either set it in the SD data, or set it in rewriting.

And as far as I understand your 1st rule, if some awkward setup intentionally set instance to !foo it would result in something like !foo!hostname:port, which, when matched against the regex !([^!]+):(\d+) wouldn't match at all as the first subexpression can only match foo but then it expects a : but finds the 2nd !.

Well that's true, but it provides the exactly behaviour you wanted: if any instance label is present, with any value, then the regex does not match and the instance label is not overwritten.

It doesn't make any difference whether the existing value is instance="foo" or instance="!foo". Neither of these will match the regex, because of the inserted '!' separator

The value being matched is <instance>!<__address__> and it only matches if ! is in the first position and nowhere else, which can only be true if instance is unset.

from docs.

i wish this was in the official example/config file, so it was more obvious that it is necessary if you dont want port numbers in your other consuming interfaces. cheers, and thanks for the thread, it helped me set a relabel stanza on every job in my config file.

from docs.