AMG cannot reach on-premise Prometheus/Loki on AKS Arc (Azure Local) as Datasource

Question

AMG cannot reach on-premise Prometheus/Loki on AKS Arc (Azure Local) as Datasource

Schultz, Michael 20

Problem:

AMG uses internal DNS 127.0.0.11 which ignores vNet DNS settings. Cannot reach private endpoints on HCI cluster.

Environment:

AMG: p-ccc-amg, West Europe, Standard tier
HCI: AKS Arc on Azure Local
MOC Cloud Provider
MOC_LOAD_BALANCER_ENABLED: false (MetalLB)
vWAN connectivity exists and works from VM

Error:

dial tcp: lookup <fqdn> on 127.0.0.11:53: no such host (standard)

Question:

What is the officially supported way to connect AMG to on-premise Prometheus/Loki on AKS Arc with MetalLB/MOC Cloud Provider?

Is Azure Private Link Service possible in this scenario? 🙌

0 comments

Answer accepted by question author

2 additional answers

Your answer

Answer 1

Hello Schultz, Michael,

Thank you for reaching out to the Microsoft Q&A forum.

When investigated you’re hitting the fact that Azure Managed Grafana (AMG) in Azure Local (MOC/Azure Stack HCI) can’t “see” your on-prem Prometheus/Loki because:

AMG resolves names via its internal kube-DNS (127.0.0.11), not your vNet-configured DNS
Azure Private Link Service (PLS) isn’t available on Azure Stack HCI (so you can’t front your on-prem Prometheus service with a PLS and then create a “managed private endpoint” from AMG)

Officially, Microsoft supports two main patterns for getting on-prem Prometheus metrics into AMG:

• Push into Azure Monitor Managed Prometheus – Deploy the Azure Monitor for containers extension (with Prometheus scraping) on your AKS Arc cluster – Metrics flow into an Azure Monitor workspace in regular Azure – In AMG, add the Prometheus data source and choose Azure Auth → Managed Identity to point at your Azure Monitor workspace endpoint – Docs: • https://learn.microsoft.com/azure/azure-monitor/essentials/prometheus-grafana?tabs=azure-managed-grafana

• https://learn.microsoft.com/azure/managed-grafana/troubleshoot-managed-grafana#connect-on-prem-prometheus

• Use a self-hosted Grafana inside your network – Stand up Grafana on a VM or on-prem k8s and point it at your Prometheus/Loki via MetalLB – Then surface that Grafana UI via your own networking (VPN, ExpressRoute, Application Gateway, etc.)

Right now, you cannot directly use Azure Private Link Service from AMG in Azure Local/MOC because PLS isn’t supported there. If you need a fully managed solution in the public cloud, you’d have to host your Prometheus in a public-Azure AKS or in another subscription that supports PLS, expose it via a Private Link Service, and then use AMG’s “Managed Private Endpoint” feature:

• Tutorial: connect to a self-hosted Prometheus service on an AKS cluster using a managed private endpoint https://learn.microsoft.com/azure/managed-grafana/tutorial-mpe-oss-prometheus

Let me know which direction makes sense for you, or if you need more details on:

• Setting up Azure Monitor container insights with Prometheus scraping • Configuring the Prometheus data source in AMG for Azure Monitor • Deploying a self-hosted Grafana in your on-prem network

Reference list

Troubleshoot Azure Managed Grafana Connection to On-Premises Prometheus Server https://learn.microsoft.com/azure/managed-grafana/troubleshoot-managed-grafana#connect-on-prem-prometheus
Connect Grafana to Azure Monitor managed service for Prometheus https://learn.microsoft.com/azure/azure-monitor/essentials/prometheus-grafana?tabs=azure-managed-grafana
Tutorial: connect to a self-hosted Prometheus service on an AKS cluster using a managed private endpoint https://learn.microsoft.com/azure/managed-grafana/tutorial-mpe-oss-prometheus

If the answer was helpful, kindly User's image & Up-vote this can be beneficial to other community members.

Siva shunmugam Nadessin 9,625 Reputation points Microsoft External Staff Moderator

2026-04-22T17:50:25.3366667+00:00

Schultz, Michael

Just checking in to see if the solution shared above help you to resolve your issue. please reach out to us If you have any further questions.

Thanks
Schultz, Michael 20 Reputation points

2026-04-23T13:28:43.0466667+00:00

Hi,

Thank you very much for taking the time to look into this and for the clear, well-reasoned confirmation. The clarification around AMG traffic routing through the Microsoft-managed VNet, and the fact that MOC/MetalLB on Azure Stack HCI cannot back a Private Link Service, was exactly the certainty we needed.

Based on your answer, we've decided not to pursue AMG as the central platform for this scenario. Instead, we'll expand our existing self-hosted Grafana stack on HCI and integrate Azure Monitor data via Workload Identity directly from there.

A clear "not supported" with the underlying reasoning is honestly just as valuable as a "yes" — it saves us from chasing an unsupported path. Really appreciate the thorough and professional response.

Best regards,

Michael

Answer 2

Jose Benjamin Solis Nolasco 7,996 Volunteer Moderator

Hello Michael Schultz,

The 127.0.0.11 IP address is the internal DNS server inside Microsoft's hidden Grafana containers. Because it is a managed service, that internal DNS doesn't know how to talk to your company's on-premises DNS servers over the vWAN to find your Prometheus server's name.

Just use the IP address Since your vWAN is already working, go into your Grafana data source settings and swap out the domain name for the direct MetalLB IP address. If there is no name to look up, the DNS error should go away

If you want to understand the enterprise approach, follow these steps:

Create an internal Azure Load Balancer in your Azure VNet. Point its backend directly to your on-premises MetalLB IP addresses.
Create an Azure Private Link Service and attach it to that Load Balancer.
Go to your Grafana settings, click "Managed Private Endpoints," and connect it to your new Private Link Service.

https://learn.microsoft.com/en-us/azure/managed-grafana/how-to-connect-to-data-source-privately

😊 If my answer helped you resolve your issue, please consider marking it as the correct answer. This helps others in the community find solutions more easily. Thanks!

Schultz, Michael 20 Reputation points

2026-04-22T07:00:22.5066667+00:00

Hi Jose Benjamin, thank you for the quick response!

Unfortunately, both approaches don't apply to our setup. Let me clarify the constraints we're working with:

Regarding the IP-only approach:

The MetalLB IPs (e.g. 10.x.x.x) are private addresses inside the HCI network. Since AMG routes its outbound traffic through the Microsoft Managed VNet – not through our p-xxx-vnet – these IPs are simply not reachable from AMG, regardless of whether we use a FQDN or a direct IP.

Regarding the Private Link Service approach:

This is exactly what we'd love to use, but it requires an Azure Standard Load Balancer as the PLS frontend. Our HCI cluster (AKS Arc on Azure Local) uses the MOC Cloud Provider (not the Azure Cloud Provider), with MetalLB as the load balancer and MOC_LOAD_BALANCER_ENABLED: false. There is no Azure Standard Load Balancer in this setup, so attaching a Private Link Service is not possible.

So our core question remains: What is the officially supported or recommended way to connect Azure Managed Grafana to Prometheus/Loki running on AKS Arc (Azure Local) with MOC/MetalLB, where Azure Private Link Service is not an option?

Any guidance from the product team would be greatly appreciated!

Answer 3

Azure Managed Grafana (AMG) is not covered in the provided material, and there is no documented, supported pattern in the context for connecting AMG to on-premises Prometheus/Loki running on AKS Arc on Azure Local via MetalLB or the MOC cloud provider, nor any statement that Azure Private Link Service is supported for this AMG → AKS Arc on Azure Local scenario.

The context only documents:

Supported private endpoint scenarios for Azure Local/AKS Arc nodes and workloads (for PaaS services such as Storage, SQL, Key Vault, ACR) and their routing/DNS requirements.
That Azure Local and the Arc resource bridge must always use public Azure Arc endpoints (Azure Arc Private Link is not supported for Azure Local nodes/Arc resource bridge).
How private endpoints are routed (ExpressRoute/S2S VPN) and how DNS and proxy bypass lists must be configured for AKS workloads and Arc resource bridge.

From this, the only supported and documented connectivity patterns are:

Azure Local/AKS Arc workloads using private endpoints to reach Azure PaaS services.
Outbound-only connectivity from Azure Local infrastructure to Azure/Internet via firewall, proxy, or Arc gateway.

There is no documentation in the context that:

Describes AMG’s DNS behavior or how to override its 127.0.0.11 resolver.
Describes AMG using Azure Private Link Service to reach on-premises services.
Describes a supported AMG → on-premises AKS Arc integration pattern.

Therefore, an officially supported way to connect AMG to on-premises Prometheus/Loki on AKS Arc with MetalLB/MOC Cloud Provider, or the use of Azure Private Link Service for this specific scenario, is not documented in the provided context.

References:

Share via

AMG cannot reach on-premise Prometheus/Loki on AKS Arc (Azure Local) as Datasource

2 additional answers

Your answer