# Internal Server Error: An Azure App Service tale

## Previously on…

When you read my posts, you do know by now I am working with Rest APIs and OAuth2, with some DevOps in between. In another post, I talk about branching etiquette. There you can read that the customer is also using an environment-based branching strategy.

## Context

At my customer, we are at the point we start to redeploy our existing APIs using Infrastructure-as-code. A new way of branching and deployment has been put in place. However, when surfing to the endpoint of the application, you get an Internal Server Error.

The setup described below, has been setup on a Continuous Deployment pipeline using ARM templates. There is a convention that the DevOps team follows: each application is deployed in its virtual directory, using the application name

# How it begins…

Below I will describe first how to get in a situation similar to the one that I debugged. Afterwards, I will explain how I debugged that situation.

## Create the Azure App Service

First, create the Azure Application Service. Secondly, using the Azure Portal, go to the configuration page. Then setup a virtual directory with the following parameters: `/app_name` to `\site\wwwroot\appname`

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676832503052/18fbdc95-b3e6-48e5-b8a5-0f0b5db8374d.png align="center")

## Deploy the Azure App Service

Using a Powershell terminal, I published the application and compressed the output folder. That zip file was deployed using the AzCli to the Azure App Service.

`Set-Location C:\git\SomeRepo\src\WeatherApp`

`dotnet publish -o appname`

`cd .\appname`

`Compress-Archive .` [`appname.zip`](http://appname.zip)

`az login`

`az webapp deployment source config-zip --src .\`[`appname.zip`](http://appname.zip) `-n "webapp-230219103841" -g "rg-230219103841"`

# Debugging the situation

My colleague told me that the deployment had succeeded. I navigated to the Azure App Service. The browser reported an Internal Server Error.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676832436841/0bfd28af-60a7-475b-a4a7-576ff08239c1.png align="center")

## Use Kestrel

The first step I took, was to check Application Insights but found that no error was logged. After that, Using the Azure Portal, I checked the Logs. I got no connection with the streaming logs due to the enterprise proxy server.

Next, using the console in the Azure Portal, I started the application. First, I got an error that the [Asp.net core Development certificate](https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-dev-certs) was not available. So I created and configured a generated self-signed certificate. That error was mitigated and now I got no error at all.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676832413601/1b8a1b10-d9c6-4fba-b453-ee4a450be53a.png align="center")

After waiting a while, the console let me know that the process was killed due to no CPU activity.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676832396030/94591dee-bb88-4734-93ea-e26dc3224d8b.png align="center")

To exclude the idea there was something wrong with the HTTPS endpoint, I edited the appsettings.json and removed the HTTPS endpoint. Next, I removed the self-signed certificated. The result was the same.

## Use Advanced Tools (SCM)

On the "[troubleshoot page of Microsoft](https://learn.microsoft.com/en-us/aspnet/core/test/troubleshoot-azure-iis?view=aspnetcore-7.0#app-startup-errors)", I read that I should enable the stdout-logging to troubleshoot the problem.

At my customer, access to Kudu does not seem possible, due to Proxy restrictions. [I looked into the possibility of uploading my web.config instead of the generated one](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/web-config?view=aspnetcore-7.0). This, however, was not feasible, because I had to wait 15min each time I made a change.

## Logging and Advanced Tools

After searching for the procedure to make the proxy server work with Azure functionality, I finally got access to the Log Streaming service and the Advanced Tools.

### Application Insights Logging and Logstream

I could find nothing in Application Insights. Maybe I had some logging using the LogStream in Azure Portal.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676832312714/64857ab9-1d18-4ebc-8335-fdc09ebd8f98.png align="center")

After enabling the options, I navigated to the Azure App Service. Afterwards in the LogStream, I could find more information.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676832304561/0e99ea3c-4a4a-470f-98c8-1282ed7a029e.png align="center")

Sadly, it did not help me much. Except something seemed very wrong?

### Advanced Tools: Kudu

The advanced tools will help me debug the application. [I can edit the web.config and have immediate feedback](https://learn.microsoft.com/en-us/aspnet/core/test/troubleshoot-azure-iis?view=aspnetcore-7.0#aspnet-core-module-stdout-log-azure-app-service). I had heard of the ANCM module before, but never really understood it. [It is a native IIS module that plugs into the IIS pipeline. This way ASP.NET Core applications can work with IIS](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/aspnet-core-module?view=aspnetcore-7.0#log-creation-and-redirection).

I had to enable the stdout logging. To do this I set the `stdoutLogEnabled` to `true`.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676832342412/3cd8dae4-3aab-4ea9-9ce5-2fa60b3ef1b9.png align="center")

After pressing save, I went back to Kudu and looked for the output. I navigated again to the Azure App Service and nothing changed. It was still the same Internal Server Error.

I searched for the logging, but found nothing. It seemed that the application module was just not started at all.

At this point, I was searching for any clue... So using Kudu, I browsed around.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676831955166/1e2e2a4e-1221-4a32-9e2a-2d64ed1a9497.png align="center")

I found a file named `eventlog.xml`.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676831973816/12ad285b-7fe3-4597-8d2f-400ed9c83700.png align="center")

I noticed that the system complained that the `processpath` is required. That meant the system did not find the web.config. The `processpath` has the value `dotnet.` I put the `web.config` in the folder `wwwroot`.

I surfed to Azure App Service and I got a different error page. I got an error stating: ANCM Failed to Find Native Dependencies:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676831884761/27e4e936-88aa-4baf-8869-495bd45199b5.png align="center")

However, I did not find stdout logging. I checked the `eventlog.xml` again.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676831846099/b3705cab-f9b6-4977-a58d-4ee38cd792fc.png align="center")

It complained about a missing `aspnetcorev2_inprocess.dll`. Which I found strange. I changed the `.\wwwroot\web.config` that the `WeatherApp.dll` is found in the `appname` folder.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676831816317/00c52e35-f2d2-4aba-83cb-c9a1d24eaf6f.png align="center")

Surfing again to the Azure AppService, I got a webpage stating: Page Not Found. However, I found the stdout logging file. But the file was empty.

[I read documentation about the web.config and the stdout logging](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/logging-and-diagnostics?view=aspnetcore-7.0). I need to add `handlersettings` to state the level of logging.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676831768365/073c5656-a48a-42c8-b4e2-b1c7a8b56e88.png align="center")

FInally, I got logging.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676831751932/aa70ad5a-13da-4c69-aeaf-fbc117f9ed12.png align="center")

## Solution

Everything fell into place. From the days that I deployed applications in IIS on a PaaS, I know that a `web.config` should be in every application that you define in IIS. The `web.config` was not read by IIS the `appname`. It was read in the folder `wwwroot`. That meant that the configuration around Virtual Paths and Directory needed to be investigated. Using the configuration page in the Azure Portal, I saw that the appname defined in the virtual path did not follow the company's convention.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676831691507/087da227-5b5a-445f-9dd8-f35c07550ba5.png align="center")

I changed the value of the virtual path from `app_name` to `appname.`

Finally, I surfed to `/WeatherForeCast` of the app and got a nice Not Authenticated Error. The app is working.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676831728757/899d88ef-2c6b-4fe7-a42d-c7f5dace39d9.png align="center")

# Outro

Although the solution is straightforward, I learned what ANCM is and how to log at that level.

# Sources

The following are the sources I used to mimic the situation:

* [Deploying multiple virtual directories to a single Azure Website | Microsoft Learn](https://learn.microsoft.com/en-us/archive/blogs/tomholl/deploying-multiple-virtual-directories-to-a-single-azure-website)
    
* [dotnet publish command - .NET CLI | Microsoft Learn](https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-publish)
    
* [Visual Studio publish profiles (.pubxml) for](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/visual-studio-publish-profiles?view=aspnetcore-7.0) [ASP.NET](http://ASP.NET) [Core app deployment | Microsoft Learn](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/visual-studio-publish-profiles?view=aspnetcore-7.0)
    
* [Deploy a .NET 6 Web Application to an Azure App Service by the command line - DEV Community 👩‍💻👨‍💻](https://dev.to/kasuken/deploy-a-net-6-web-application-to-an-azure-app-service-by-the-command-line-533j)
    

The following sources I have used to help me debug the situation.

* [Troubleshoot](https://learn.microsoft.com/en-us/aspnet/core/test/troubleshoot-azure-iis?view=aspnetcore-7.0) [ASP.NET](http://ASP.NET) [Core on Azure App Service and IIS | Microsoft Learn](https://learn.microsoft.com/en-us/aspnet/core/test/troubleshoot-azure-iis?view=aspnetcore-7.0)
    
* [ASP.NET](http://ASP.NET) [Core Module (ANCM) for IIS | Microsoft Learn](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/aspnet-core-module?view=aspnetcore-7.0#log-creation-and-redirection)
    
* [Configure endpoints for the](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel/endpoints?view=aspnetcore-7.0#listenoptionsusehttps) [ASP.NET](http://ASP.NET) [Core Kestrel web server | Microsoft Learn](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel/endpoints?view=aspnetcore-7.0#listenoptionsusehttps)
    
* [Kudu service overview - Azure App Service | Microsoft Learn](https://learn.microsoft.com/en-us/azure/app-service/resources-kudu)
    
* [https://learn.microsoft.com/en-us/azure/app-service/configure-common?tabs=portal](https://learn.microsoft.com/en-us/azure/app-service/configure-common?tabs=portal)