Sovereign Cloud - MetalLB and AKS additions

03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/Readme.md

@@ -49,7 +49,18 @@ Use **GitHub Codespaces with the Azure / Infra / Sovereign Cloud devcontainer**
 ### Set up before Challenge 1
 
 1. Sign in with a **personal GitHub account**. Optionally set **Settings > Codespaces > Default idle timeout** at [github.com/settings/codespaces](https://github.com/settings/codespaces) **before creating the Codespace**. The default is 30 minutes; you can choose 5-240 minutes, subject to organization policy. The setting applies to new Codespaces.
-2. Open [microsoft/MicroHack](https://github.com/microsoft/MicroHack), then select **Code > Codespaces > ... > New with options**.
+2. Open the **repository home page**, [microsoft/MicroHack](https://github.com/microsoft/MicroHack), while signed in to GitHub:
+
+   - Above the file list, select the green **Code** button, then the **Codespaces** tab. This is the repository's Code menu, not a menu inside VS Code.
+
+     <a href="https://docs.github.com/assets/images/help/codespaces/who-will-pay.png"><img src="https://docs.github.com/assets/images/help/codespaces/who-will-pay.png" alt="GitHub documentation example of the Code menu with the Codespaces tab selected" width="420"></a>
+
+   - In the **top-right corner of the Codespaces tab**, select **...**, then **New with options**. Do not use the quick-create button: this repository has multiple devcontainers, and you need to choose the Sovereign Cloud one.
+
+     <a href="https://docs.github.com/assets/images/help/codespaces/default-machine-type.png"><img src="https://docs.github.com/assets/images/help/codespaces/default-machine-type.png" alt="GitHub documentation example showing the Codespaces three-dot menu and New with options" width="420"></a>
+
+   These two navigation screenshots are examples from [GitHub's Codespaces documentation](https://docs.github.com/en/codespaces/developing-in-a-codespace/creating-a-codespace-for-a-repository#creating-a-codespace-for-a-repository); repository names and existing Codespaces may differ. If the **Codespaces** tab is missing, check that you are signed in. Alternatively, open [Create a codespace](https://github.com/codespaces/new), select **microsoft/MicroHack**, and continue with the options below.
+
 3. Select **Azure / Infra / Sovereign Cloud** and **2-core**, then **Create codespace**. Wait for setup to finish before using the terminal.
 
    ![Codespace creation options showing the Sovereign Cloud devcontainer and a 2-core machine](./img/codespaces-create.png)

03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/challenges/challenge-01.md

@@ -10,7 +10,7 @@ The goal of this exercise is to establish foundational sovereign cloud governanc
 
 ## Actions
 
-- Create and assign Azure Policy controls to restrict deployments to EU sovereign regions (Norway East, Germany North, North Europe).
+- Create and assign Azure Policy controls to restrict deployments to the lab-approved European regions (Norway East, Germany North, North Europe, West Europe). West Europe accommodates Azure Local management resources when LocalBox is registered there.
 - Enforce resource tagging requirements for data classification and compliance tracking.
 - Block public IP resource creation and evaluate storage public-network-access restrictions. Disabling public network access does not create or verify a private endpoint.
 - Assign least-privilege RBAC roles for the SovereignOps team.

03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/challenges/challenge-06.md

@@ -2,27 +2,46 @@
 
 ## Goal
 
-The goal of this challenge is to operate a sovereign hybrid cloud environment by combining Microsoft Sovereign Public Cloud and Sovereign Private Cloud components. You will work with Azure Local, simulated via Azure Arc Jumpstart LocalBox, and provision your own VM. You will use Azure Arc to manage that VM through Azure, review its security posture with Microsoft Defender for Cloud, and assess its OS updates with Azure Update Manager.
+The goal of this challenge is to operate a sovereign hybrid cloud environment by combining Microsoft Sovereign Public Cloud and Sovereign Private Cloud components. You will work with Azure Local, simulated via Azure Arc Jumpstart LocalBox, and provision your own VM. You will use Azure Arc to manage that VM through Azure, review its security posture with Microsoft Defender for Cloud, and assess its OS updates with Azure Update Manager. You will also deploy a container application in your team's namespace on the shared AKS cluster and access it privately from your Sovereign Cloud Codespace.
 
 ## Scenario
 
 Your organization must run workloads in a sovereign cloud while still leveraging Azure's management and governance capabilities. Azure Local represents your sovereign on-premises infrastructure, and Azure Arc enables you to apply consistent governance across your hybrid estate.
 
 ## Actions
 
+Before deploying, confirm that your Challenge 1 exercise policies are back in
+**DoNotEnforce** and ask the facilitator to confirm that the shared LocalBox custom
+location's Azure region is permitted in your assigned resource group. This region
+can differ from the resource group's location. If a policy blocks creation, follow
+the [location-policy troubleshooting steps](../walkthrough/challenge-06/solution-06.md#if-validation-or-deployment-is-blocked-by-a-location-policy);
+do not disable organizer-managed policies or change allowlists yourself.
+
 * Explore the LocalBox hybrid infrastructure in the Azure Portal
-* Deploy a sample application to AKS on Azure Local
 * Deploy your own VM on Azure Local using Azure Arc VM management and verify that guest management is connected
 * Verify Microsoft Defender for Cloud coverage and review security recommendations for the VM you provisioned
 * Use Azure Update Manager to assess OS updates on the VM you provisioned
+* Verify your Console-provided Microsoft Entra administrator-group membership and the existing Azure Arc Enabled Kubernetes Cluster User Role with the facilitator; portal workload visibility alone does not prove group membership
+* Inspect the organizer-prepared `arcnetworking` extension (`microsoft.arcnetworking`) on `localbox-aks` in `rg-localbox-shared`: confirm **Succeeded**, healthy MetalLB workloads, and the existing **`aks-pool`** with **ARP** advertisement and organizer-reserved service VIPs. Do not install or configure shared networking
+* Follow [Task 5 of the walkthrough](../walkthrough/challenge-06/solution-06.md#task-5-deploy-a-container-to-the-aks-cluster-deployed-on-azure-local) in **Bash**: run `az connectedk8s proxy` with Microsoft Entra authentication and an isolated kubeconfig, derive a unique namespace from your assigned resource group, and deploy `aks-local-sample-app.yaml` only in that namespace
+* Verify successful rollout and the Service's assigned IP, then keep the proxy and namespace-scoped `kubectl port-forward service/aks-container-1 8080:80` running in separate terminals. Open port **8080** through the **Private** Codespaces Ports view (or `localhost:8080` when running directly on a local workstation). Never expose the API proxy port or use service-account/admin tokens; no RDP or static routes are needed
+* Clean up only your team's application resources and, when no longer needed by teammates, your exercise namespace. Leave shared infrastructure unchanged
+
+The organizer prepares MetalLB with `resources/prepare-localbox.ps1`; the Console's LocalBox deployment hook alone is not sufficient. If extension health, the pool, or access prerequisites are missing, contact the facilitator rather than installing extensions, creating pools, or granting additional permissions.
+
+The default service VIP range is **`10.10.0.10-10.10.0.100`**, excluding nodes **`10.10.0.101-10.10.0.199`**, control-plane IP **`10.10.0.5`**, and gateway **`10.10.0.1`**. Confirm any organizer customization. Inspect the pool's **IPAddressPool** and **L2Advertisement** in `kube-system` read-only; participants do not create them.
 
 ## Success criteria
 
 * You can navigate and understand the LocalBox hybrid environment in the Azure Portal
 * You have deployed your own VM on Azure Local via the Azure Portal and verified that guest management is Enabled (Connected)
 * You have verified Defender for Servers coverage for your VM and reviewed its available recommendations, or identified that its assessment is still pending
 * You have completed an Azure Update Manager assessment for your VM and reviewed the results, including when no updates are pending
-* You have deployed a sample application to AKS on Azure Local
+* You have verified the shared MetalLB extension, workloads, and reserved ARP IP pool without changing shared infrastructure
+* You have deployed the sample application to your team's unique AKS namespace, verified rollout, and recorded a Service IP from the reserved pool
+* You have opened the application through a Microsoft Entra Arc proxy and a private application port-forward, with the API proxy never exposed
+* You understand that production clients use the MetalLB VIP through configured network routing; proxy plus port-forward access does **not** validate that load-balancer network path
+* You have cleaned up only your team's application resources
 * You understand how Azure Arc provides a unified control plane for sovereign hybrid scenarios
 
 ## Learning resources
@@ -34,3 +53,4 @@ Your organization must run workloads in a sovereign cloud while still leveraging
 * [Microsoft Defender for Cloud with Arc-enabled servers](https://learn.microsoft.com/azure/defender-for-cloud/quickstart-onboard-machines)
 * [Azure Update Manager overview](https://learn.microsoft.com/azure/update-manager/overview)
 * [Azure Arc Jumpstart - LocalBox](https://jumpstart.azure.com/azure_jumpstart_localbox)
+* [Azure CLI Cluster Connect proxy reference](https://learn.microsoft.com/cli/azure/connectedk8s#az-connectedk8s-proxy)

03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/deploy-localbox.ps1

@@ -24,6 +24,9 @@ The azure_arc branch, tag, or commit used for both Bicep and runtime artifacts.
 .PARAMETER AzureLocalResourceProviderObjectId
 Optional tenant-specific object ID of the Microsoft.AzureStackHCI enterprise
 application. The script resolves it through Microsoft Graph when omitted.
+.PARAMETER AzureLocalInstanceLocation
+Azure Local registration region, separate from the Azure host region.
+Defaults to West Europe to align with the Challenge 1 location allowlist.
 .PARAMETER NoWait
 Submit the deployment without waiting for ARM completion.
 #>
@@ -54,7 +57,7 @@ param(
     [string]$AzureLocalResourceProviderObjectId,
 
     [ValidateSet('australiaeast', 'southcentralus', 'eastus', 'westeurope', 'southeastasia', 'canadacentral', 'japaneast', 'centralindia')]
-    [string]$AzureLocalInstanceLocation = 'australiaeast',
+    [string]$AzureLocalInstanceLocation = 'westeurope',
 
     [switch]$NoWait
 )

03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/resource-providers.ps1

@@ -85,6 +85,7 @@ $providers = @(
     # Kubernetes (for AKS Arc)
     "Microsoft.Kubernetes",
     "Microsoft.KubernetesConfiguration",
+    "Microsoft.KubernetesRuntime",
     "Microsoft.ContainerService",
     "Microsoft.ContainerInstance",
     "Microsoft.ContainerRegistry",

03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/labautomation/shared-deploy-lab.ps1

@@ -213,7 +213,7 @@ else {
                 -ResourceGroupName $localBoxResourceGroupName `
                 -Location $localBoxLocation `
                 -AzureLocalResourceProviderObjectId $azureLocalResourceProviderObjectIds[0] `
-                -AzureLocalInstanceLocation 'australiaeast' `
+                -AzureLocalInstanceLocation 'westeurope' `
                 -UseConsoleCredentials `
                 -NoWait
 
@@ -246,6 +246,25 @@ else {
 
 
 $localBoxScope = "/subscriptions/$SubscriptionId/resourceGroups/$localBoxResourceGroupName"
+for ($attempt = 1; $attempt -le 60; $attempt++) {
+    Update-MhhToken | Out-Null
+    $runtimeProvider = Get-AzResourceProvider -ProviderNamespace Microsoft.KubernetesRuntime -ErrorAction Stop
+    if ($runtimeProvider.RegistrationState -eq 'Registered') { break }
+    Start-Sleep -Seconds 10
+}
+if ($runtimeProvider.RegistrationState -ne 'Registered') {
+    throw 'Microsoft.KubernetesRuntime registration did not complete. LocalBox MetalLB preparation is blocked.'
+}
+$runtimeObjectIds = @(& az ad sp list --filter "appId eq '087fca6e-4606-4d41-b3f6-5ebdf75b8b4c'" --query '[].id' --output tsv --only-show-errors)
+if ($LASTEXITCODE -ne 0) { throw 'Unable to resolve the Microsoft.KubernetesRuntime service principal for MetalLB.' }
+$runtimeObjectIds = @($runtimeObjectIds | Where-Object { -not [string]::IsNullOrWhiteSpace($_) })
+$runtimeObjectId = [guid]::Empty
+if ($runtimeObjectIds.Count -ne 1 -or -not [guid]::TryParse($runtimeObjectIds[0], [ref]$runtimeObjectId) -or $runtimeObjectId -eq [guid]::Empty) {
+    throw 'Expected exactly one valid Microsoft.KubernetesRuntime service-principal object ID. Have the tenant administrator verify provider registration.'
+}
+Update-AzTag -ResourceId $localBoxScope -Operation Merge `
+    -Tag @{ 'microhack-k8s-runtime-object-id' = $runtimeObjectId.ToString() } -ErrorAction Stop | Out-Null
+
 foreach ($participantObjectId in $AllowedEntraUserIds) {
     foreach ($roleName in @('Reader', 'Azure Stack HCI VM Contributor')) {
         $roleAssignment = Get-AzRoleAssignment `

03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/hosted-events/readme.md

@@ -21,9 +21,59 @@ The shared hook submits `localbox-*` in `rg-localbox-shared`, then waits up to s
 1. Inspect the Azure deployment, Client bootstrap logs and Azure Local/Arc status. Resolve failures before proceeding.
 2. If the Client password is unknown, an authorized event lead/coach uses the Azure VM's **Help > Reset password** to set a known password for the `arcdemo` host account, then connects to `LocalBox-Client` through Bastion. Do not expose this password through participant credentials or job logs. A host password reset does not change the nested-node Windows credentials. See [LocalBox credentials](#localbox-credentials).
 3. Copy **Lab Group ObjectId** from the Console's **Credentials** tab for the preparation prompt or `-AksAdminGroupObjectId`. Confirm that intended Azure lab identities/coaches are members; see the group dependency below.
-4. Follow [LocalBox preparation](../localbox/readme.md), once per shared subscription, in elevated PowerShell 7. Azure authentication uses the Client VM's managed identity; preparation constructs the nested Windows credential locally from the installed configuration without printing it. An explicit `-NodeCredential` overrides that value after a nested-account rotation.
+4. Follow [LocalBox preparation](../localbox/readme.md), once per shared subscription, in elevated PowerShell 7. It creates AKS and then installs/verifies the shared MetalLB extension and reserved ARP pool. Azure authentication uses the Client VM's managed identity; preparation constructs the nested Windows credential locally from the installed configuration without printing it. An explicit `-NodeCredential` overrides that value after a nested-account rotation.
 5. Run [Pester health checks](../tests/readme.md) for LocalBox and every selected participant lab. A control-plane-only pass does not establish full readiness.
 6. Complete the [participant VM readiness exercise](../localbox/manual-preparation.md#step-6-test-the-environment), including guest management, Defender and Update Manager. Subscription-wide paid-plan changes require the authorized owner and approved budget.
+7. Verify an intended participant can use Entra-authenticated `az connectedk8s proxy` and a private application port-forward from their Codespace. Participants inspect MetalLB, not install it, and never need LocalBox-Client access. This validates the lab access path, not routing to the MetalLB service VIP.
+
+Shared setup registers `Microsoft.KubernetesRuntime`, waits for registration,
+resolves its tenant service-principal object ID using the Console's existing
+Graph-capable identity, and records it in the shared RG tag
+`microhack-k8s-runtime-object-id`. Missing registration or identity resolution
+fails setup explicitly. Organizer preparation reads that nonsecret tag; it does
+not require Graph permissions on the Client managed identity. This does not move
+AKS or MetalLB installation into the Console hook: they remain part of the
+organizer's post-provisioning preparation.
+
+### LocalBox registration region and location policies
+
+There are three separate locations to check: the participant resource group's
+metadata location, the Azure region hosting the LocalBox simulator, and the
+Azure Local/custom-location registration region. The hosted
+[shared hook](../../labautomation/shared-deploy-lab.ps1) explicitly passes
+`AzureLocalInstanceLocation = westeurope`; the
+[manual deployment](../manual-setup/localbox/deploy-localbox.ps1) defaults to
+`westeurope`. Host-region fallback does not change that registration parameter.
+The hosted deployer's default also matches West Europe. This changes registration
+for fresh deployments, not the Azure host-region selection. Inspect the deployed
+custom location's **JSON View** for its actual `location`.
+
+Azure Local VM management resources use the custom location's region even when
+participants create them in a resource group whose location differs. Policy
+permission on the shared LocalBox group therefore does not establish permission
+to create the participant resources. Before the event, validate the
+[Challenge 6 VM creation flow](../../walkthrough/challenge-06/solution-06.md#23-review-and-create)
+in the designated test participant group, not only in the shared group.
+
+For a policy denial, collect the exact assignment/definition IDs and rejected
+resource type/location. First check whether a participant left their Challenge 1
+location assignment or bonus initiative enforcing; those exercise assignments
+should be **DoNotEnforce**. For an inherited or organizer-owned denial, ask the
+policy owner to review the approved locations or an appropriately scoped
+exception. Do not disable unrelated policies or automatically broaden the
+subscription allowlist. The hosted control tags below do not override arbitrary
+location-deny policies.
+
+[Microsoft's current Azure Local region list](https://learn.microsoft.com/azure/azure-local/concepts/system-requirements-23h2#azure-requirements)
+includes West Europe as its only European region for hyperconverged deployments.
+The current Challenge 1 exercise allowlists include West Europe, but older
+assignments may still use the original three-region list. This does not prove
+that West Europe is allowed by the event subscription's inherited policies, and
+it does not permit Australia East. Existing Azure assignments are not updated
+automatically. Earlier hosted test environments registered in Australia East
+are not relocated by this change; validate the next event using a fresh deployment
+from the updated content. Existing-environment cleanup is a separate organizer
+action, not part of the registration-region change.
 
 ### Hosted MCAPS control-tag initiative
 

03-Azure/01-03-Infrastructure/01_Sovereign_Cloud/resources/localbox/manual-preparation.md

@@ -97,6 +97,7 @@ Before participants begin, follow the [Challenge 6 walkthrough](../../walkthroug
 3. Confirm that the same VM appears in Defender for Cloud **Inventory**, verify Defender for Servers coverage and onboarding, and review its assessment status. Allow time for recommendations to populate; an empty list is not proof of completed assessment
 4. Locate that VM in Azure Update Manager **Resources -> Machines**, run **Check for updates**, and verify a successful assessment and its timestamp. Zero pending updates is a valid result; a pending or failed assessment is not
 5. Confirm that neither exercise requires selecting shared cluster nodes, the LocalBox host, Arc Resource Bridge, or another participant's VM, or changing subscription-level settings
+6. Complete the organizer [AKS and MetalLB preparation](readme.md#metallb-preparation), then use the same participant identity to inspect MetalLB, deploy the sample into a unique team namespace, and access it through `az connectedk8s proxy` plus a private `kubectl port-forward`. Follow the current Challenge 6 walkthrough; no Client VM credentials or Windows routes should be needed by the attendee. Verify the Service receives a reserved VIP, but record that port-forwarding does not validate routing through that VIP.
 
 Resolve connectivity, extension provisioning, or permission failures before the event. Keep any permission changes limited to the operation and resource scope actually required. Do not install patches or restart shared resources during this readiness test.