Azure Stack Log File Locations

I have been trying to find several of these so I was glad to see this list published today.  I simply copied this from MSDN and cleaned it up to make it readable.

Deployment Logs

Can’t get Azure Stack to install correctly?

On the host, run Invoke-DeploymentLogCollection.ps1 from <mounted drive letter>:\AzureStackInstaller\PoCDeployment\

The logs will be collected to C:\LogCollection\<GUID>

Note: Running the script in verbose mode (with the -verbose parameter) provides details about what is being saved from each machine (localhost or VM).

For the host and for the VMs, this script collects the following data:

Event logs (from event viewer) “$env:SystemRoot\System32\Winevt\Logs\*”
Tracing directory (includes the SLB Host Agent logs) “$env:SystemRoot\tracing\*”},
Azure Stack POC Deployment Logs “$env:ProgramData\Microsoft\AzureStack\Logs”},
Azure Consistent Storage   (ACS) Logs “$env:ProgramData\Woss\wsmprovhost
Compute Resource Provider (CRP) Logs “C:\CrpServices\Diagnostics”}

Azure Stack Deployment failure logs:

On the Physical Host, these are under C:\ProgramData\Microsoft\AzureStack\Logs

Fabric Detailed logs:

C:\ProgramData\Microsoft\AzureStack\Logs\AzureStackFabricInstaller

Services/Extension logs:

C:\ProgramData\Microsoft\AzureStack\Logs\AzureStackServicesInstallerf60e6d-95c2-458e-baa5-32f3e9b6ab77

Log Collection for ACS issues

  1. Capture Blob Service logs
    1. Zip the following folder on C:\programdata\woss on the physical host machine.
    2. Capture Event Viewer logs on physical host machine
      1. Capture Admin, Debug and Operational channels event logs under “Application and Services Logs\Microsoft\AzureStack\ACS-BlobService” (use “Save All Events As” on the right pane)
      2. Also Capture the Application and System logs in Event viewer (Windows Logs)
  2. Capture blob service crash dumps [if any] on the physical machine. To find the crash dump location, use the following command.
C:\>reg query "HKLM\SOFTWARE\Microsoft\Windows\Windows ErrorReporting\LocalDumps\BlobSvc.exe" /v DumpFolder

In this TP1 release, the automatic log collection process does not include log collection for SRP.  To collect the SRP service log, please follow the instruction as below. The AddUnitOfTime(value) should be changed accordingly based on when the error happens.

  1. Login to the xRPVM virtual machine as domain admin (AzureStack\administrator)
  2. Open a PowerShell window, and run the cmdlets below:
cd “C:\Porgramdata\Windows Fabric\xRPVM\Fabric\work\Applications\SrpServiceAppType_App1\SrpService.Code.10.105521000.151210.0200”
Import-Module .\WossLogCollector.ps1
$cred = get-credential
$end = get-date
$start = $end.AddMinutes(-10)
Invoke-WossLogCollector -StartTime $start -EndTime $end -Credential $cred -TargetFolderPath \\sofs.azurestack.local\share -SettingsStoreLiteralPath file:\\sofs.azurestack.local\Share\ObjectStorageService\Settings -WindowsFabricEndpoint ACSVM.AzureStack.local:19000
  1. Collected logs are under \\sofs.azurestack.local\share. The logs are packed as a ZIP file tagged with timestamp.

Manual Log Collection for the Multi-Compartment Network Proxy Service

Multi-Compartment Network Proxy enables the agent inside the VM to reach the metadata server and report back status.

To get proxy logs:

Check if the proxy service is running (sc query or powershell’s get-service):

Wpr –start tracer.wprp –logmode

<restart proxy service>

<repro>

Wpr –stop c:\tracefile.etl

Network Resource Provider (NRP) logs

They are not enabled by default. To gather logs in a specific situation:

  1. As the Service Admin, start the trace on the xRPVM:
cmd /c "logman start mysession -p {7978B48C-76E9-449D-9B7A-6EFA497480D4} -o .\etwtrace.etl -ets"
  1. Reproduce the error scenario
  2. Stop the trace:
    cmd /c "logman stop mysession -ets"
    cmd /c "tracerpt etwtrace.etl -y"

Summary/Additional Logs Locations on the Hyper-V host

C:\ProgramData\Microsoft\AzureStack Directory “Logs” includes Azure Stack  POC deployment/TiP deployment logs
C:\ProgramData\WOSS Azure Consistent Storage   (ACS) Logs
C:\ProgramData\WossBuildout\<GUID>\Deployment Azure Consistent Storage   (ACS) deployment logs
C:\ProgramData\Microsoft\ServerManager\Events File server events
C:\ProgramData\Windows Fabric\Log\Traces WinFabric traces
C:\Windows\System32\Configuration Desired State Configuration (DSC) files
C:\ClusterStorage\Volume1\Share\Certificates Certificates
C:\Windows\tracing\ Directory “SlbHostAgentLogs” includes SI   Host Plugin logs
C:\Windows\Cluster\Reports Cluster creation & Storage validation
C:\Windows\Logs\DISM\dism.log   and C:\Windows\Logs\CBS\cbs.log Role additions (e.g. Hyper-V)

ARM Logs

On the PortalVM, Portal /ARM logs can be found in Event Viewer – Applications and Services Logs – Microsoft – AzureStack.

Note: ARM logs are in “Frontdoor”.

Control Plane Logs

In Event Viewer, under Applications and Services Logs -> Microsoft -> AzureStack.  Some key logs to look at:

–        Azure resource manager: Microsoft-AzureStack-Frontdoor/Operational

–        Subscriptions service: Microsoft-AzureStack-Subscriptions/Operational

–        Portal Shell site: Microsoft-AzureStack-ShellSite/Operational

Compute Resource Provider (CRP) diagnostics logs

On the xRPVM machine, these are located under

C:\CrpServices\Diagnostics\\AzureStack.compute.log

Note: the logs are rolled and backed up every day

Logs for Azure provisioning agent

The log produced by the Azure provisioning agent (PA) is inside a tenant virtual machine at

C:\windows\panther\wasetup.xml

Logs for VM DSC extensions

In addition to any ARM messages in the portal, these logs are currently stored in the tenant-deployed VM:

C:\Packages\Plugins

For DSC extension, the DscExtensionHandler log is in

C:\WindowsAzure\Logs\Plugins\Microsoft.Powershell.DSC\<version>\DscExtensionHandler.<index>.<date>-<time>.log

And

C:\Packages\Plugins\Microsoft.Powershell.DSC\<version>\Status
C:\Packages\Plugins\Microsoft.Powershell.DSC\<version>\Status\0.status
.status
C:\WindowsAzure\Logs\WaAppAgent.log

Getting the status for VM extensions

You can also use a script such as the following one, targeting the corresponding Resource Group:

Get-AzureRmResourceGroupDeployment -ResourceGroupName $tenantRGName 
 
       $vms =  Get-AzureRmVM -ResourceGroupName $tenantRGName 
        foreach($vmname in $vms.OSProfile.ComputerName) 
        { 
            $status = Get-AzureRmVM -ResourceGroupName $tenantRGName -Status -Name $vmname 
            Write-Host $status.Name 
            Write-Host $status.ExtensionsText 
            Write-Host $status.StatusesText 
            Write-Host $status.BootDiagnosticsText 
            Write-Host $status.DisksText 
            Write-Host $status.VMAgentText 
        }

Information about Resource Deployment Status:

In the Portal, under Resource Explorer/Subscriptions/<yourSubscription>/<your resource group>/Deployments/Microsoft.Template/Operations/

Examples:

Error :

“properties”: {

“provisioningState”: “Failed”,

“timestamp”: “2016-02-04T23:32:21.3841604Z”,

“duration”: “PT3.343173S”,

“trackingId”: “0af85051-1350-493c-bc95-313dd6c0a874”,

“statusCode”: “BadRequest”,

“statusMessage”: {

“error”: {

“code”: “OperationNotSupported”,

“message”: “PUT operation on resource /subscriptions/922819A8-30D6-49E5-9D15-010A66FD6B39/resourcegroups/newme1/providers/Microsoft.Network/loadBalancers/mesos-master-lb-01234567/inboundNatRules/SSH-mesos-master-01234567-0 is not supported.”,

“details”: [],

“innerError”: “Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.ValidationException: PUT operation on resource /subscriptions/922819A8-30D6-49E5-9D15-010A66FD6B39/resourcegroups/newme1/providers/Microsoft.Network/loadBalancers/mesos-master-lb-01234567/inboundNatRules/SSH-mesos-master-01234567-0 is not supported.”

},

“targetResource”: {

“id”: “/subscriptions/922819A8-30D6-49E5-9D15-010A66FD6B39/resourceGroups/newme1/providers/Microsoft.Network/loadBalancers/mesos-master-lb-01234567/inboundNatRules/SSH-mesos-master-01234567-0”,

“resourceType”: “Microsoft.Network/loadBalancers/inboundNatRules”,

“resourceName”: “mesos-master-lb-01234567/SSH-mesos-master-01234567-0”

}

Succeeded:

“properties”: {

“provisioningState”: “Succeeded”,

“timestamp”: “2016-02-04T23:32:30.9349745Z”,

“duration”: “PT6.7912649S”,

“trackingId”: “8b9a3e65-6a64-4fba-a69e-ff64631e8a92”,

“statusCode”: “Created”,

“targetResource”: {

“id”: “/subscriptions/922819A8-30D6-49E5-9D15-010A66FD6B39/resourceGroups/newme1/providers/Microsoft.Network/networkInterfaces/jb-01234567-nic”,

“resourceType”: “Microsoft.Network/networkInterfaces”,

“resourceName”: “jb-01234567-nic”

}

Azure Stack Home Lab Build

In the beginning there was…nothing

Microsoft Azure Stack is not something I could build and run on my desktop like my Configmgr lab, which runs on my W10 desktop while I use it for work as well.  I had to decide if I wanted to spend the money on a server or wait for the hardware to fall into my hands.  Because I saw this a real opportunity to learn something that would be valuable to my customers and myself, and because my wife…I mean CFO, approved the request I purchased the individual parts to build my own server for my home lab.

Having had a server in my home lab previously I knew of the drawbacks.  Noise and heat.  Lots of both.  Doing a customer demo when you are sweating and they cannot hear you makes doing a demo difficult.  That meant I had to meet the hardware requirements of Azure Stack, and it had to be quiet and cool.

And then there was…

After lots of research, I ordered all of my hardware.  Things have not changed much since I got my A+ back in 1995, thankfully.  And yes it did boot the first time I powered it on.  I won’t get into the hardware details but you can take a stab at it based on the images.

Empty Case
The case is a little larger than I expected. It has plenty of room to add additional drives later on though. It is behind my desk between the desk and the haze-grey wall you see here. It would actually fit under my desk but until I decide to move my Dell desktop it will stay back there.

 

IMG_0870
MSI motherboard. While a gaming motherboard it supports my needs, which don’t include gaming.
IMG_0874
Motherboard installed, i7 CPU installed, H20 powered CPU cooler installed and connected. The memory is installed but I later moved it for quad channel
Hard Drives
This shows the five drives mounted. It is a bit blurry but my Pearl Izumu shoes are in focus.
First boot
This shows the system running. Memory moved for quad channel, video card mounted, power supply, so on.
CPU Temp
Closer shot of the system running. You can see the “40” indicating the CPU temp.
If I were in charge of the world
And now Windows Server 2016 TP4 installing from USB. With a shot of my whiteboard cleaned off for Azure Stack testing. And also my daughters “If I Were In Charge Of The World” worksheet from first grade.
MAS Extracting
Azure Stack extracting in Server 2016 TP4

IMG_0887

You can download the bits for Azure Stack here, and this is a link to the instructions on how to setup Azure Stack.  The instrcutions assume you have met the list of hardware requirements listed here and other requirements listed here.

Since I had a few hours while it installed, I went for a run out at Goldmine trail, which is about 20 minutes from me.

IMG_0890

The install threw an error and I had to go visit a customer to upgrade them to Configmgr current branch, so it would have to wait until I got back.  At least, my server was nice and warm at home.

DCCX7560

Back at home.  My install on the base OS did not work.  It complained that my time was off.  There is no UTC/local time setting in my BIOS and here in AZ we don’t do DST, so today we are equal to MST.  All times and time zones matched so after talking to a couple Azure TSP’s I decided to install using the VHD method, which worked, not sure why but it did.  It only took about two hours to install instead of the estimated four.

After returning from the northwest…

I completed the setup in my lab and it was pretty painless to get running.  I have since reinstalled it on the same system, starting from scratch.

When setup is complete and you run Failover Cluster Manager, or Hyper-V Manager, you will see the list of VM’s running.  If the PowerShell script throws an error during setup, fix the issue and then run the script again and it will pick up where it left off.  At the end you should have something like this.

ClusterMGR-01

Azure Stack VMs

Here is a breakdown of each VM you have running once the installation of Azure Stack has completed:

ADVM Virtual machine that hosts Active Directory, DNS, and DHCP services for Microsoft Azure Stack. These infrastructure foundational services are required to bring up the Azure Stack as well as the ongoing maintenance.

ACSVM Virtual machine that hosts the Azure Consistent Storage services. These services run on the Service Fabric on a dedicated virtual machine.

MuxVM Virtual machine that hosts the Microsoft software load balancer component and network multiplexing services.

NCVM Virtual machine that hosts the Microsoft network controller component, which is a key component of the Microsoft software-defined networking technology. These services run on the Service Fabric on this dedicated virtual machine.

NATVM Virtual machine that hosts the Microsoft network address translation component. This enables outbound network connectivity from Microsoft Azure Stack.

xRPVM Virtual machine that hosts the core resource providers of Microsoft Azure Stack, including the Compute, Network, and Storage resource providers.

SQLVM Virtual machine that hosts SQL Servers which is used by various fabric services (ACS and xRP services).

PortalVM Virtual machine that hosts the Control Plane (Azure Resource Manager) and Azure portal services and various experiences (including services supporting admin experiences and tenant experiences).

ClientVM Virtual machine that is available to developers for installing PowerShell, Visuall Studio, and other tools.

And…Storage services in the operating system on the physical host include:

ACS Blob Service Azure Consistent Storage Blob service, which provides blob and table storage services. SoFS Scale-out File Server. ReFS CSV Resilient File System Cluster Shared Volume.Virtual Disk, Storage Space, and Storage Spaces Direct are the respective underlying storage technology in Windows Server to enable the Microsoft Azure Stack core storage resource provider.

 

Here is a graphic representation of the VM’s and services.

Azure Stack TP1 POC logical architecture

 Manually Shutting Down Azure Stack

If you need to manually shutdown and startup the VM’s this is the recommended order:

ClientVM, PortalVM, xRPVM, ACSVM, MuxVM, NCVM, SQLVM

Hyper-V will shut down the other three for you as you shutdown your server.  The start order is reverse.  If you need to manually shut down, after you start the VM’s ensure that all the services on xRPVM are running and happy, if it isn’t you will have issues since it hosts the core of the Azure Stack resources.

Azure Stack Portal

This is a set of images on the ClientVM.  Showing the portal view, then the ARM templates resources imported from GitHub, then as it appears in the marketplace.  Then a custom built VM template imported into the marketplace, using PowerShell to add a SQL 2014 custom VM into the platform image repository (PIR), and finally the custom SQL 2014 VM listed in the marketplace.

Image 001

Image 002

Image 003

Image 004

Image 005

Image 006

Image 008