startup exception for aspnet core worker docker container attempting to set secret from env variable in Kubernetes

Problem

I'm trying to set secrets in an aspnet core (3.0 SDK) worker application that runs in a container in Kubernetes. I am able to get the application to consume environment variable secrets when i run it using Visual Studio 2019 in a local docker container and in a container on local kubernetes instance on Windows. It is a Linux container.

However, when i deploy it to AKS, on startup the application fails to startup. It crashes with the error:

Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused process_linux.go:449: container init caused setenv: invalid argument: unknown

I'm pretty sure the secrets are deployed correctly as below.

I carry this out by defining my secrets with yaml similar to that below. Deployment is being carried out by Azure Devops. The __ token values are replaced out by one of the pipeline tasks. I can check the values on the AKS instance secrets page in the dashboard and can see that they have been deployed correctly.

As inferred below in the names of the variables, Kubernetes required base64 encoded strings for secrets

apiVersion: v1 kind: Secret metadata: name: paymentservicesettings type: Opaque data: dbconnectionstring: __dbconnectionstringbase64__ storagebaseurl: __storagebaseurlbase64__ storageconnectionstring: __storageconnectionstringbase64__ baseuri: __baseuribase64__ subscriptionkey: __subscriptionkeybase64__ userid: __useridbase64__ usersecret: __usersecretbase64__ where the valuesbase64 are base 64 encoded strings.

i run this via kubectl apply -f appsettings.yml command. I can see the settings are defined correct when I check them via the dashboard.

I then define the deployment like this:

apiVersion: apps/v1
kind: Deployment
metadata:
annotations: deployment.kubernetes.io/revision: "1" generation: 1 labels: run: payment-service name: payment-service namespace: default spec:
progressDeadlineSeconds: 600 replicas: 3 revisionHistoryLimit: 10 selector: matchLabels: run: payment-service strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: run: payment-service spec: containers: - image: org.azurecr.io/payment-service:__tagBuildId__ // **this tag is auto replaced for every deployment** imagePullPolicy: IfNotPresent name: payment-service env: - name: "ASPNETCORE_ENVIRONMENT" value: "Kubernetes" - name: "dbconnectionstring" valueFrom: secretKeyRef: name: paymentservicesettings key: dbconnectionstring - name: "storagebaseurl" valueFrom: secretKeyRef: name: paymentservicesettings key: storagebaseurl - name: "storageconnectionstring" valueFrom: secretKeyRef: name: paymentservicesettings key: storageconnectionstring - name: "baseuri" valueFrom: secretKeyRef: name: paymentservicesettings key: baseuri - name: "subscriptionkey" valueFrom: secretKeyRef: name: paymentservicesettings key: subscriptionkey - name: "userid" valueFrom: secretKeyRef: name: paymentservicesettings key: userid - name: "usersecret" valueFrom: secretKeyRef: name: paymentservicesettings key: usersecret dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 imagePullSecrets: - name: acr-auth
When I first deploy the service, it pulls correctly and then attempts to start up. Unfortunately it then seems that it can't find the env variables and crashes. Because the pods crash, I am unable to debug to find out why they are doing this.

I have had a look at trying to debug via Azure Dev Spaces, but it looks like this is currently unsupported for worker services.

I've tried to ensure I've got as much exception handling code as possible in the C# side to prevent crashes on startup, but this hasn't made any difference

Update

I've discovered the reason this happened is due to text encoding.

This post pointed me in the right direction: Powershell base64 encoding required

It turns on that in Windows, UTF-16 is used when encoding a string to base64.

So, this script:

Param (
[Parameter(Mandatory=$true)][string] $Value, [Parameter(Mandatory=$true)][string] $VariableName ) $bytes = [System.Text.Encoding]::Unicode.GetBytes($value); $base64Value = [System.Convert]::ToBase64String($bytes); echo "##vso[task.setvariable variable=$VariableName]$base64Value";

Encoding the value:

https://www.google.com
produces an encoded value of:

aAB0AHQAcABzADoALwAvAHcAdwB3AC4AZwBvAG8AZwBsAGUALgBjAG8AbQA=

Decoding this value with populate base64 decoding sites produces:

This is incorrect. It is obvious that something is being inserted between the characters (it looks like spaces).

I checked this string on: A popular base64 encode/decoding site

The string with the incorrect string encoding can be seen more clearer on this site:

It turns out UTF-16 is being used by PowerShell to create the encoded base64 string. To fix this, I updated the script to force PowerShell to use UTF-8.

$value = "https://www.google.com"
$enc = [System.Text.Encoding]::UTF8 $bytes = $enc.GetBytes($value) $base64Value = [System.Convert]::ToBase64String($bytes); echo $base64Value;
This produces a shorter base64 encoded string.

aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbQ==

which looks like this:

So, bottom line. Be sure which encoding is being used when converting a string to base64 (in either PowerShell or .NET languages). Check whether UTF-16 or some other encoding is being used unexpectedly.