hmm, in prometheus, is there a way to take a variable containing a value like `host:9121` and get `host:9100` back to use in a metrics query?

I can see label_replace but I don't think that's what I'm wanting?

@thisismissem Depending on what you need label_replace could help. e.g. given 2 metrics:

up{instance="demo-service-0:10000"} = 1
demo_num_cpus{instance="demo-service-1:10001"} = 4

you can do:

label_replace(up{instance="demo-service-0:10000"}, "instance", "$1-1:10001", "instance", "(.*)-0:10000") * on (instance) demo_num_cpus

and get out:

{instance="demo-service-1:10001"} = 4

Example on promlabs demo site:

https://demo.promlabs.com/query?g0.expr=label_replace%28up%7Binstance%3D%22demo-service-0%3A10000%22%7D%2C+%22instance%22%2C+%22%241-1%3A10001%22%2C+%22instance%22%2C+%22%28.*%29-0%3A10000%22%29+*+on+%28instance%29+demo_num_cpus&g0.show_tree=0&g0.tab=table&g0.range_input=1h&g0.res_type=auto&g0.res_density=medium&g0.display_mode=lines&g0.show_exemplars=0

Prometheus Time Series Collection and Processing Server

@thisismissem I adapter the use-case a little bit to the data that's available in the demo.promlabs.com site. In your case the host part would be the same, but label names might be different. Additionally, in this example that value of the up metric is 1 so I could multiply it with the demo_num_cpus to get just the second metric's value. In real world case this might be different too.
@antolius ended up using a variable with regex pattern to set $server, and then a second "hidden" variable for setting $instance (could've also gone the opposite way too, I think?)