I am going to learn about ethical hacking, which means using hacking skills to protect systems and data. The importance of cybersecurity continues to grow as technology becomes more essential in our lives. Understanding cybersecurity will help me to grow as a software architect in an enterprise size companies. Fifteen years ago, there was a gap between software developers and system engineers. These days, that gap has been closed due to native cloud software development. To spike my career further, I want to be the bridge between the software developers and the cybersecurity experts. Security is difficult and, in my humble opinion, you tend to implement practises when you understand them.

I will learn a 5-day course named Certified Ethical Hacker (CEH) in my time. My goal is getting a better understanding. I will take the CEH exam. But my real goals are to have a more in-depth understanding of the whole world of cybersecurity and be the bridge between Security and Development.

Overview of the Course

The CEH program covers many topics. After seeing the video Module 00, It seems taht I will explore areas like:

• Foot printing: Gathering information about a target.
• Vulnerability Analysis: Finding weaknesses in systems.
• Malware Threats: Learning about different types of harmful software.

This course should help me think like a hacker while also learning how to defend against cyber threats.

Learning Approach

As I take the CEH course, it is essential for me to build a strong foundation first. This means learning the basics before moving on to more complex topics. The course is structured with various modules, and I will have both lectures and labs to practice what I learn.

• Practice in Labs: Creating a Virtual environment where I can practise, so I can engage with the given labs. That should give me hands-on experience and a chance to use different tools. However, they do supply me with a remote lab environment where I can practise. Maybe I will start using there environment and not losing time setting up my own.
• Review and Revisit: I will not hesitate to go back to previous lessons and labs.
• Books: They do supply me with a book that consists out of 2000 pages :| . Do I fully understand why I am going to do?

Exams in the CEH Program

There will be two exams in the CEH program:

• Multiple Choice Exam: This is a multiple-choice test with 125 questions. I hear that I have four hours to complete it, and the passing score ranges from 69% to 84%...

• Practical Exam: This exam includes 20 challenges based on lab exercises. Passing both exams will earn me the title of CEH Master, which is a significant achievement in the field of cybersecurity. This will be the more difficult one, I think.

What can you do with such an exam?

Completing the CEH course could open doors for me in cybersecurity and having a CEH certification is a start at least for roles like:

• Security Analyst
• Penetration Tester
• Forensics Investigator

However, as I stated in the beginning, my goal is to be the bridge between software development and cybersecurity to become a better consultant in development teams in big enterprise, or maybe, who know, a security analyst? ... time will tell.

In this article we explore data integrity and authenticity when dealing with sensitive information.

This section outlines the fundamental importance of making HTTP request bodies tamperproof, particularly when handling personally identifiable information (PII). We introduce digital signatures as a key mechanism for data integrity.

Why should you care?

When extending the user registration process, using Auth0 (an OIDC Provider), it is important to ensure data integrity and authenticity. Particularly when dealing with sensitive information that could lead to personal identity identification (PII) like the national register numbers.

In this blog post, I will explore difficult ways how to achieve this. Making the HTTP request body is tamper-proof is a step in protecting data as it moves from one server to another. The use of signatures for HTTP request bodies provides a robust mechanism for ensuring data integrity and security in this context. Frameworks like JOSE (JSON Object Signing and Encryption) can help in this matter.

A lot of the internet is already protected by using the HTTPS protocol, right? Well, HTTPS only encrypts the data; it does not guarantee that the content is not tampered with during transit. HTTPS provides a secure channel but does not protect against all types of tampering. For instance, if data is decrypted and re-encoded, the integrity may be compromised without detection. In scenarios where sensitive data is transferred – such as through webhooks – signing the request body ensures the data's integrity independently of the transport layer's security.

HTTP request body signatures add a layer of security by allowing the recipient to verify that the body has not been altered in transit. They act as a digital signature, much like those used in emails, ensuring the integrity of the sent content.

I will also investigate if using an access or ID token could help in this matter.

Previously on ...

Here, we recap earlier discussions on signatures and OpenID Connect, setting the stage for a deeper exploration into data security practices in HTTP requests.

I wrote about OpenID Connect before and how to mock your provider, and thus creating and faking tokens...

• https://dotnet.kriebbels.me/oauth2-open-id-connect-accessidtoken
• https://dotnet.kriebbels.me/signature-validation-required-microsoft-says-no
• https://dotnet.kriebbels.me/how-to-read-a-claim-from-a-token-in-dotnet-6
• https://xebia.com/blog/openid-connect-mocking-you/
• https://xebia.com/blog/openid-connect-mocking-you/

I did a talk as well on how to mock your OIDC Provider:

• https://www.visug.be/Events/92
• https://www.updateconference.net/en/2023/session/mock-your-openid-connect-provider

This article aims to further expand upon previously covered concepts and enhance understanding.

Understanding JWT, JOSE and Access/ID/... Tokens

Before we explore individual token types, this section provides an overview of JWT and its role within the broader JOSE ecosystem. That is important for safeguarding data communicated via OpenID Connect and OAuth2 protocols.

OpenID Connect and OAuth2 protocols often leverage JWT to communicate authorization and identification details.

Discussions with colleagues and customers highlight common misuses of JWT as information repositories, necessitating a balance between security and data sensitivity.

There are a lot of tokens in the OpenID Connect world, but because the goal of this document is about making something tamperproof, we will focus on the access token as an example.

Access Tokens Explained

In this section, gain insights into access tokens, particularly within the JWT format, and how they serve as credentials for accessing protected resources, highlighting their role in ensuring authenticity and confidentiality.

An access token is a credential used to access protected resources. When used in a JWT format, it typically contains claims about the user and any permissions granted. These tokens can be JSON Web Signature (JWS) or JSON Web Encryption (JWE) encoded to ensure authenticity and confidentiality, respectively.

An ID token contains information about the user when access is granted. This is always in a JWT format.

These tokens can (should!) have a JSON Web Signature (JWS). This ensures authenticity.

When the information is really sensitive, they can be encrypted using the JSON Web Encryption (JWE) to ensure confidentiality.

Making the tokens tamperproof

Learn how the JOSE framework enables tamper proofing of access tokens. This section describes the use of JSON Web Signatures and JSON Web Encryption for securing token integrity and confidentiality.

By making the access token tamperproof, the JOSE framework is used to Secure the Payload.

The JSON Object Signing and Encryption (JOSE) framework offers a suite of technologies to secure communications:

• JWT: JSON-based claims as a token.
• JWS: JSON Web Signatures to secure and protect against tampering.
• JWE: JSON Web Encryption to protect sensitive data.
• JWK: JSON Web Key defines cryptographic keys in JSON.

JOSE enables you to sign and encrypt HTTP request bodies, ensuring integrity and confidentiality.

Understanding the Versatility of JSON Web Tokens

JSON Web Tokens (JWT) are often thought of as mere access tokens, but their potential extends far beyond this basic use case. A JWT is a compact, URL-safe means of representing claims between two parties. These tokens are particularly useful in scenarios that require verification and integrity. In this documentation, we will explore howJWT`s can be utilized beyond traditional access tokens and how they can ensure the integrity and non-tamperability of HTTP request bodies.

The Concept of 'Super Tokens'!

Overloading access tokens or ID tokens with too much information can lead to what is known as "super tokens". These are risky because they could get logged in various places (e.g., proxies, logs), potentially exposing sensitive data unnecessarily. Instead, balancing token content with just enough claim data ensures better security and usability.

Including too much in a single token can lead to technical problems as well. When a token gets to big, it can not be saved into a cookie, or used in the URL of a browser.

Storing large amounts of data in a token can lead to performance issues. Tokens should remain lightweight to travel over networks efficiently.

Super tokens present a single point of failure. If a super token is compromised, it can grant extensive access, leading to potentially severe security breaches. Managing permissions using super tokens ensure that only the right users have access to certain parts of a system can become complex and error-prone.

What should not be stored in a token?

• Highly sensitive data such as passwords, credit card details, or private keys should never be stored in tokens. This is because tokens are often stored in less secure environments, such as cookies or local storage, which could lead to exposure if an attacker gains access.

• Information that is subject to frequent change should not be stored in tokens, as updating a token across a system can be cumbersome. A token is valid for the specified time. This means that, in that time window, that data is valid.

HTTP Header vs HTTP Body?

Tokens are commonly attached to HTTP headers for authentication purposes. For instance, an access token in the header can authenticate API requests.

Headers should only contain information necessary for message transmission like authentication tokens and content type identifiers. Headers should be lightweight and used to convey metadata.

The body is used for the main content of the HTTP message. Sensitive data that is sensitive, dynamic or is not used for authorization should be safely passed here...

However, it becomes essential to ensure the tamper-proofing of the request body.

And thus, we have the reason why I did some digging into this topic: Making the body tamperproof can be done by

• Levering the JOSE framework
• Using a Signature in the HTTP Header

JWT Structure

Above, we discussed that the JOSE framework has two formats:

• JWT
• JWE

A JWT consists of three parts:

1. Header: Contains the type of token (JWT) and the hashing algorithm used.
2. Payload: A set of claims, which can be any information you want to share.
3. Signature: Used to verify the integrity and authenticity of the token.

Example of a JWT signed with a shared secret (HS256):

// Header
{
  "alg": "HS256",
  "typ": "JWT"
}

// Payload
{
  "sub": "1234567890",
  "name": "John Doe",
  "scope": "openid profile email",
  "country": "belgium",
  "email": "john@doe.com"
}

// Signature
HMACSHA256(
  base64UrlEncode(header) + "." +
  base64UrlEncode(payload),
  secret
)

Body and JWT in HTTP Requests

In this section, read how JWT can enhance HTTP request body security, ensuring data sent is from an authentic source. We cover encoding sensitive data within JWTs for tamper proofing.

While JWT are commonly used for authentication, they are not limited to being access tokens. They can also serve as secure message carriers or embedded within HTTP request bodies to ensure data integrity.

The following steps outline the process:

• By applying JWT encoding on the body of an HTTP request, any alteration to the body would be detected through signature verification. This ensures that the data is indeed from an authentic source and has not been tampered with on its way to the server.

• You can encode sensitive parts of the HTTP message (for example, headers or the body itself) into the JWT payload. Only an entity with the correct secret or public key can validate the JWT and trust the data.

• Using JWT not in an authorization context simplifies the process. There's no need for a centralized authority to verify token signatures. Just an agreement between two parties and where to find a shared secret or the location of the public key.

Example HTTP Request

An example use case could be a secure data submission where JSON payload (representing a transaction) is signed using JWT technology before it is sent to prevent unauthorized changes. The National Register Number (nrn) is the real identification of the user.

POST /transaction HTTP/1.1
Date: Fri, 11 Oct 2024 09:38:52 GMT
Host: api.dummy.com
Content-Type: application/json

{
  "amount": "1500",
  "currency": "USD",
  "recipient": "Alice",
  "jwt": "eyJhbGciOiJSUzI1NiJ9.ewogICJucm4iOiI4MzEyMDcwMjQ1MCIKfQ.ezfyvYmmQ4Cq8nXS1Hy0Duo9i-4yooV3BHk3NLbF2dKMnBgpymsigq951AQ1d_sXEfVOGlOqXSGynV8PL2vR52lO1vOXHU8Cm4yoCgCAwyG5k9XWCi9DSiBj8SXT0xLxXI94p0Oy7COhNUeEnoVrRRJ4XeTLB0bBPuFVjFgJT4435ZnkRMsY7oHe7czSdg1lL-ZgFjirq2YVAbx5wI1DyJ0QPiIVN126pZ-Tsx06I9pCukQARPDqLMESszmd7rx1e4OcaW4GnFicGSkQ8pqOYIXzfSpT-NNlHSlygSioCrF9VgwPbPZDyzqJNcrAQovMrLCyIliD92GPeVHinQdrog"
}

After decoding the field jwt, it should resemble the following request

POST /transaction HTTP/1.1
Date: Fri, 11 Oct 2024 09:38:52 GMT
Host: api.dummy.com
Content-Type: application/json

{
  "amount": "1500",
  "currency": "USD",
  "recipient": "Alice",
  "jwt": {
        "nrn": "83120702450"
    }
}

Or you can sign the entire body with the needed change on the Content-Type header

POST /transaction HTTP/1.1
Date: Fri, 11 Oct 2024 09:38:52 GMT
Host: api.dummy.com
Content-Type: application/jwt

eyJhbGciOiJSUzI1NiJ9.ewogICJhbW91bnQiOiAiMTUwMCIsCiAgImN1cnJlbmN5IjogIlVTRCIsCiAgInJlY2lwaWVudCI6ICJBbGljZSIsCiAgImp3dCI6IHsKICAgICAgICAibnJuIjogIjgzMTIwNzAyNDUwIgogICAgfQp9.QCC1KQn9lTAaPXqokVySRdMlLRoC4ALiQjagyyZbSPkicFe5HyEUZoEB9PtXLMM8LCc9-vt4ygiRhitImT_WLHNmBSBkubrbUwO8FxZ_GSZFuPrA5vnk4F63Ro6N86krvE46cHOX3-q-2AByl82leRlv_LywTfQIw-TnOvwfNaL5bCGDa9rv7bbZd2LD7FusV0t7_oU0ZCPpOHq-FolKQCLOy9RUzXzkxWDlYNw5AlTfkQ_RYV9QjwmfNYMwh2BnmpxfBSDc0aqSfPmcvD2c_J27hznGjkCORLYjNoprhJw2_F8JDuVF38JxD-8Kvr-G0_x5E0oLRF5aBLdc1vp-tQ

JSON Web Encryption (JWE)

This section explains the structure and utility of JWE tokens, detailing their multiple encryption layers to provide comprehensive data security.

JWE tokens encapsulate multiple encryption layers. Here's a breakdown of its structure:

• JWE Encrypted Key: This is the encrypted version of the Content Encryption Key (CEK). The CEK is essential for decrypting the actual payload.

• JWE Initialization Vector: This element maximizes security by ensuring that identical data never results in the same encryption output.

• JWE Authentication Tag: This ensures that the data integrity is maintained, verifying that the data has not been tampered with.

• JWE Ciphertext: This is the encrypted data payload that you aim to secure.

Example: Deciphering the JWE Token

To decrypt a JWE token, follow these steps:

1. Understand the JWE Header:
   • Identify the encryption algorithm used from the JWE header.
2. Request Decryption for JWE Encrypted Key:
   • Send a request to the server to obtain the Content Encryption Key (CEK).
3. Decrypt the Payload:
   • Using the CEK, decrypt the JWE ciphertext to extract the original payload.

JWS Verification

Understand the process of using JWS verification to confirm the integrity of data post-decryption, ensuring its trustworthiness.

After decryption, if the data includes a JWS token, its integrity can be confirmed through verification. This step ensures that the data has not been altered and is indeed from a trusted source.

Using Signature in Header

We discuss the technique of signing HTTP requests by attaching a digital signature to the header, enabling servers to verify request authenticity and integrity.

The concept of signing involves creating a digital signature for the body of a HTTP request. This digital signature is then included in the request headers, allowing the server to validate the request's authenticity and integrity.

• The Request Body: The sensitive data you want to protect.
• RSA-SHA256 Signature: A cryptographic hash of the request body, encrypted with a private key. This hash is sent to the server for verification.

Steps

• Generate RSA Key Pair: You need a RSA private key for signing and a corresponding public key for signature verification

• For each request

  • sign the Request: Calculate the SHA256 hash of the request body, and encrypt the hash using your private key

  • Attach Signature to Header: Include your signature e.g. with a header name Signature or X-Payload-Signature header when you send your HTTP request.

  • Server-side Verification: The server uses the public key to verify the received signature and the integrity of the request body.

POST /transaction HTTP/1.1
Date: Fri, 11 Oct 2024 09:38:52 GMT
Host: api.dummy.com
Digest: SHA-512=tVL/etooLtB3qNVs693Ka1xpbsjgJvf8Xb/gprvktjYRnNPac9eEWw7XVrXxbfuLh45eLiGBLMEeepHvZ2cuvA==
Signature: keyId="aa2bd832-191e-4d2b-98d4-b0d4778844f9",created=1728639532,algorithm="hs2019",headers="(request-target) host digest (created)",signature="aAaII33mSJsfuXuTRJet5tJF6cQuYL77+b0gU1UZQMVjrz6jf1uNm5ixjN/CiId8Q5lTf9N308MH9ktSGWvzrtq67v73hG9PxEtCWuUR5tEiBdUk4Jt8xYszAuJYfGCqkFeoIiB8dbWNcTRRrZSAJAUPbSoY4eVyl0w91zvsr2wv8Gu3vo18XxHqtccOA0yHV4HVdJdbX1l0XLicTILTC2U27gxbpL8qsHZ1jgGByk/Wmm3KHmFtgEVGLVj3kbc712r5l54se4cSoQGst+rAy3xk//07x3icjLynObflLtpy3qqZgAR1L01xH1ISK7WCsy7G8e8ueQuGU5hAbir0lA=="
Content-Length: 91
Content-Type: application/json

{
  "amount": "1500",
  "currency": "USD",
  "recipient": "Alice",
  "nrn": "83120702450"
}

Conclusion

Reflecting on the information presented, this conclusion underscores the role of JWT and related technologies in securing sensitive information across web communications.

I have come to an understanding that a JWT is a format, a simple carrier format and can be used broader than only a token in the header.

JWT and JWE are part of the JOSE (JSON Object Signing and Encryption) framework. JWT is popular now, because Oauth2 and OpenID Connect uses the JWT for formatting the access and ID tokens. And using that framework, helps us safeguards sensitive information

When an access or ID token is uses the JWT format, it should not be used as an information repository. The JWT format can benefit your application architecture, by formatting the HTTP Request body.

However, there are other possibilities that keep the HTTP Request Body human-readable, by only adding a signature HTTP Headers.

Sources

A list of tools, references, and resources consulted while writing this article provides you with additional context and avenues for further exploration into this topic.

Tools to play with

• JOSE Generator
• Private and Public Key Generator
• Tool that helps you gain insight in creating a signature for a HTTP Request Body
  • UUID4 generator, to be used in the tool above

Sources for this article

• belgianmobileid aka ITSME information about JOSE
• https://medium.com/@idenisko/understanding-the-jwe-token-a-practical-guide-942224f6a9b6
• https://developer.visa.com/pages/encryption_guide/jwe-jws
• https://auth0.com/docs/get-started/apis/configure-json-web-encryption
• https://openid.net/developers/certified-openid-connect-implementations/
• https://openid.net/specs/draft-jones-json-web-encryption-02.html
• https://jwt.io/introduction/
• https://oauth.net/2/
• https://dev.to/ariefwara/rest-api-body-payload-signature-2277
• https://hookdeck.com/webhooks/guides/how-to-implement-sha256-webhook-signature-verification
• Content-Type: application/jwt
• Content-Type: application/json
• A really nice read on how protecting data is used in Open Banking in Europe. Even design principles are listed

TLDR; Microsoft's AddMicrosoftAccount() method does not do signature validation for tokens, which poses potential security risks. This article explores the implications of this, using an analogy of concert ticket validation, to explain the importance of signature validation in OpenID Connect (OIDC) systems. It also delves into Microsoft's approach to OIDC, highlighting some integration challenges and the risks of not properly validating signatures, especially in scenarios where secure communication channels cannot be guaranteed.

NuGet Packages mentioned:

• NuGet Gallery | Microsoft.AspNetCore.Authentication.MicrosoftAccount 8.0.4

• NuGet Gallery | Microsoft.AspNetCore.Authentication.OpenIdConnect 8.0.4

Previously on

In a previous piece for XPRT Magazine, I wrote an article about mocking your OpenID Connect Provider. For the latest magazine, I rewrote the application that allowed me to demonstrate how to mock the authorization code flow with PKCE. This modification is important for a live demo at Techorama, which is held annually. I will be attending as a speaker on this subject. Moreover, during the event of Dotnet Friday, a question about mocking OIDC (OpenID Connect) for Microsoft Entra was raised, prompting an exploration of OIDC's capabilities with Microsoft accounts. However, without any experience, I cannot say where the difficulties lay.

Context

Lately, my fascination with web development and identity security has taken a new turn. While prepping for a tech demo, I stumbled upon complexities that many developers face—using OAuth and OpenID Connect for securing apps in a testing context, using a browser. It's compelling how these systems are vital yet somewhat obscured in daily coding routines.

The tech scene itself is evolving, with an increasing emphasis on robust security protocols. Recent breaches and escalated security mandates make it imperative for applications to be fortified yet user-friendly. Hence, the dive into the authenticity mechanisms of OIDC with a focus on e.g. Microsoft Entra was not merely by chance but a necessity. As I worked through integrating Microsoft accounts into the authentication flow of my demo app, it revealed facets and pitfalls of current authentication packages.

Introduction to OIDC and Its Necessity

If you've ever logged into a service or app using your Facebook, Google, or Microsoft account, then you've encountered OpenID Connect (OIDC) without perhaps realizing it. This protocol is extensively used because it streamlines the authentication process for millions of users across various platforms by allowing them to sign in with existing identities. This is a better scenario than providing another company with your credentials. Those companies are mentioned a lot in the news. According to EU-law, companies have an obligation to inform the user about what information is stolen.

What is OIDC? — An Analogy

Imagine you're attending a concert where the ticket purchase is done through an app using Google Pay, which doesn't expose your bank account details. Here, the concert venue represents the application you want access to.

Google acts as the OIDC service — and validates your purchase without sharing your sensitive bank details—and the ticket represents the authorization given to you by OIDC to access the venue.

In this analogy, your ticket is not just a piece of paper but contains a sophisticated hologram (representing a digital certificate).

This hologram shows that the ticket is genuine when inspected under a specific light (the public key) matching a sticker (the private key) you were given when purchasing the ticket. This matching process ensures the ticket’s authenticity privately without exposing sensitive data—a core principle in OIDC using Authorization Code Flow with PKCE (Proof Key for Code Exchange). PKCE adds an extra layer of security, ensuring that even if the criminals grab your ticket, they can’t misuse it because they cannot provide the purchase proof that the concert venue will ask on entrance.

Why OIDC?

The necessity of OIDC lies in its ability to standardize secure and scalable user authentication across diverse digital systems. Using familiar accounts like Facebook, Google, or Microsoft for authentication doesn’t just reduce the burden of managing multiple usernames and passwords for users; it also outsources the security responsibilities to entities that specialize in security, thereby enhancing overall safety. By using OIDC, businesses can offload much of the cybersecurity risks associated with user data management and password protection.

Experiencing OIDC with Microsoft Entra

While my experiences are limited to working with Auth0 by Okta, other providers that provide OpenID as well, e.g., Entra. However, I do know Microsoft from the past. That Microsoft does use a standard but creates a dialect from that standard.

Let us find out if they now abide by the standard.

Integration Challenges

For a working source code, to mock out the OIDC using the Microsoft Account package.

This link is the commit that gives you a working example using the OIDC package.

Conflicting and outdated sources

When you download from Entry a sample application for Asp.NET Core, you get a deprecated one.

It lets me download the following:

active-directory-aspnetcore-webapp-openidconnect-v2-aspnetcore3-1.zip

While their documentation seems up-to-date on what library you should use, I find not matching information in the code samples.

For example:

The following is found on the quick-start: https://learn.microsoft.com/en-gb/entra/identity-platform/quickstart-web-app-dotnet-core-sign-in

However, because I wanted to showcase that when you use the Microsoft Account NuGet package, I wanted to try that out. Just going forward without looking too much into it.

Microsoft.AspNetCore.Authentication.MicrosoftAccount is available for long-term support! (Currently, that is version 8.0.4, and on the NuGet site, it links to the main page as a project source: https://github.com/dotnet/aspnetcore/tree/v8.0.4) This repository contains the generic OIDC NuGet package source code, as well as the Microsoft Account one. https://github.com/dotnet/aspnetcore/blob/main/src/Security/Authentication/

sub is not provided

When mocking out Entra, I noticed some small changes between the sample I used for the Microsoft Account and the one from the OpenIdConnect Generic package.

Notice in the example below, there is no sub claim to be found, and you get by default more claims than in the generic OIDC Library.

Get a user - Microsoft Graph v1.0 | Microsoft Learn

public sealed record UserInfoEndpointResponseBody(
    [property: JsonPropertyName("@odata.context")] string ODataContext,
    [property: JsonPropertyName("businessPhones")] string[] BusinessPhones,
    [property: JsonPropertyName("displayName")] string DisplayName,
    [property: JsonPropertyName("givenName")] string GivenName,
    [property: JsonPropertyName("jobTitle")] string? JobTitle,
    [property: JsonPropertyName("mail")] string Mail,
    [property: JsonPropertyName("mobilePhone")] string? MobilePhone,
    [property: JsonPropertyName("officeLocation")] string? OfficeLocation,
    [property: JsonPropertyName("preferredLanguage")] string? PreferredLanguage,
    [property: JsonPropertyName("surname")] string Surname,
    [property: JsonPropertyName("userPrincipalName")] string UserPrincipalName,
    [property: JsonPropertyName("id")] string Id
);

This is the refactored sample I used for the OIDC Library according to this link:

Microsoft identity platform UserInfo endpoint - Microsoft identity platform | Microsoft Learn

public sealed record UserInfoEndpointResponseBody(
    [property: JsonPropertyName("@odata.context")] string ODataContext,
    [property: JsonPropertyName("sub")] string Id,
    [property: JsonPropertyName("name")] string DisplayName,
    [property: JsonPropertyName("given_name")] string GivenName,
    [property: JsonPropertyName("family_name")] string Surname,
    [property: JsonPropertyName("preferred_username")] string UserPrincipalName,
    [property: JsonPropertyName("email")] string Mail,
    [property: JsonPropertyName("phone_number")] string? MobilePhone,
    [property: JsonPropertyName("locale")] string? PreferredLanguage
);

token signature is not validated

In the Microsoft Account library, there is no code that gives me the option to validate the signature. It just does not validate. Using the given commit, you will notice that the JWKS endpoint is not consulted: https://github.com/dotnet/aspnetcore/blob/main/src/Security/Authentication/MicrosoftAccount/src/MicrosoftAccountHandler.cs

Signature Validation Analogy: Concert Tickets

Imagine you've purchased a ticket to see your favourite band perform live. This ticket allows you entry into the concert venue, but it’s not just any ticket. It’s embedded with a unique holographic seal that proves it's genuine. This is akin to the cryptographic signature on a token in the OIDC process.

Now, suppose a scalper tries to sell counterfeit tickets outside the venue. The venue security (acting as the client in the OIDC process) uses a special light (public key) to check the authenticity of the holographic seal (signature) on your ticket (the token). If the holographic seal lights up correctly, it confirms the ticket wasn't tampered with and is valid, allowing you entrance to the concert. If the ticket fails this test, it means it might have been tampered with, similar to a potential token modification in a cyberattack.

Importance of Signature Validation

In the context of the concert:

— Prevention of Fraud: Just as the holographic seal prevents counterfeit or altered tickets from passing as genuine, signature validation in OIDC prevents tampered tokens from being used to falsely claim an identity or permissions.

— Security Protocol Integrity: Should the holographic seal be poorly crafted, allowing fake tickets to appear valid, the entire security protocol of the venue would be undermined, just as weak or absent signature validation can expose a system to breaches.

Potential Risks Without Proper Validation

During a concert:

1. Unauthorized Access: If someone with a counterfeited or tampered ticket gets past security, similar to a cyberattacker using a manipulated token, they gain unauthorized access. This could lead to overcrowding, an unpleasant experience for genuine ticket holders, or other security risks.

2. Horizontal and Vertical Attacks: In concert terms, a horizontal attack might involve attendees accessing areas reserved for other ticket holders of the same level (e.g., general admission), whereas a vertical attack would be akin to someone with a general admission ticket gaining entry to a VIP area. Without signature validation, similar breaches in software systems can lead to unauthorized data or functionality access.

Let me dive into the source code of the OIDC Library

When using the OIDC Library, it is stated that the default signature is not validated by default. https://github.com/dotnet/aspnetcore/blob/main/src/Security/Authentication/OpenIdConnect/src/OpenIdConnectHandler.cs

Notice the line in the code: https://openid.net/specs/openid-connect-core-1_0.html#IDTokenValidation

The decision to allow TLS validation in place of explicit signature checking in direct communication scenarios acknowledges the security provided by modern TLS configurations. This means that the token should not go over any proxies or any other systems that can deep inspect packages and thus give the possibility to intercept and temper. A token is always fetched using the supplied Backchannel of the library. That means that the Web Application will fetch the token. The end user is never involved in this process. When you deploy in Azure, you should be able to trust Azure about the inspection that they do or not do with your traffic. Note that everything with tokens is fascinating to hackers:

• Token Theft Playbook: Token theft occurs when threat actors compromise and replay tokens issued to a user, bypassing multifactor authentication and gaining access to organizational resources. This page includes prerequisites like access to Microsoft Entra ID sign-in and audit logs, recommendations such as enabling advanced hunting features, and requirements for configuring a SIEM (Security Information and Event Management) tool for centralized log visibility and threat detection.

• Compromised Microsoft Key: The incident involved the threat actor acquiring a private encryption key (MSA key) and using it to forge access tokens for Outlook Web Access (OWA) and Outlook.com. The compromised key could have allowed the threat actor to forge access tokens for various Azure Active Directory applications.

However, having not to validate the signature means that it simplifies implementations by reducing overhead where secure communication channels can be guaranteed. Remember that in any scenario where the communication may involve indirect or less secure channels, signature validation becomes a MUST for ensuring that the ID Token has not been tampered with or falsified in transit.

Thus, do not use the Microsoft Account library because you cannot configure the library to validate the signature in case there is no direct communication. I fail to see why you want to use the Microsoft Account library.

The links below indicate that AddMicrosoftAccount() is deprecated. But the source code in the package does not have any of the annotations that it will/is deprecated or obsolete.

• The Closed issue on https://github.com/dotnet/AspNetCore.Docs/issues/14455 references the still open issue https://github.com/dotnet/aspnetcore/issues/10037 from May 7, 2019.

Outro

Reflecting on this journey of navigating through OIDC and Microsoft Entra, I am both enlightened and a bit overwhelmed. The nuances of modern identity management systems like OIDC are vast. Yet, understanding them is not just about building secure applications, but also about empowering them to be robust in the face of evolving digital threats. My goal has been to bring to light the often overlooked aspects of app security — a commitment I will continue to explore and share insights on.

In a previous blog post, I described LibMan or Npm for a new project. I explored how to manage the required scripts.

I find it important to understand the history of things. This helps me to learn why we use this technology, and what were the shortcomings of the previous tech. Understanding history helps build the future.

Context

During my education at BINF at Hogeschool Gent, I was primarily taught ASP.NET WebForms as the go-to framework for web development. I focused more on backend development in my career. A new project starts and I realized that I missed out on the advancements in web development. Creating a scaffolded Asp.net core MVC Project led me to understand the history a bit better of ASP.NET MVC, AJAX, and jQuery.

ASP.NET WebForms drawbacks

ASP.NET WebForms is a framework introduced by Microsoft for building web applications using the .NET platform. It was similar to Windows Forms development. That allowed us to create web pages that closely resembled desktop user interfaces.

However, using ASP.NET WebForms has several drawbacks. I will not go into the details of what it was in detail. I will address the issues that web forms had.

• The tight integration of server controls with HTML and JavaScript made it difficult to test and maintain applications.

• The complex view state management mechanism resulted in large and complex view states, negatively impacting performance.

• The postback model used in WebForms caused full page reloads for each user interaction. That leads to slower user experiences.

Introduction of AJAX and ASP.NET AJAX

In 2005, the concept of AJAX (Asynchronous JavaScript and XML) emerged as a way to enable server communication but without a full page reload! Microsoft recognized the potential of AJAX and integrated it into the ASP.NET framework with the release of ASP.NET AJAX in 2007.

Using Ajax in web forms provided developers with server controls, client libraries, and toolkits. Features such as partial page rendering, update panels, and rich client-side controls improved the interactivity of web applications.

ASP.NET MVC and jQuery integration

Despite the advancements brought by AJAX, the tight coupling, view state, and postback model concerns persisted in WebForms. In response, Microsoft introduced ASP.NET MVC in 2007 as a flexible framework that followed the Model-View-Controller architectural pattern. This pattern promotes a clear separation of concerns and facilitated the separation of UI elements from their underlying logic.

To simplify client-side development within ASP.NET MVC, jQuery was integrated. jQuery provides a user-friendly API for DOM manipulation, event handling, adhoc backend calls and animations.

HtmlHelpers or TagHelpers

I want to mention the way that Asp.net core MVC Razor pages are built. In the case of client validation, it does not seem to matter much what approach is used: TagHelpers or HtmlHelpers. jQuery Validate searches for data attributes on the generated HTML.

Tag Helpers in ASP.NET Core are components that enable server-side code to participate in generating and processing HTML elements. They provide a more HTML-centric way of working with server-side code.

<!-- Tag Helper: <a> element with an `asp-controller` attribute -->
<a asp-controller="Home" asp-action="Index">Home</a>

In the above code snippet, the <a>-element is a Tag Helper. The asp-controller and asp-action attributes are Tag Helper attributes that help generate the appropriate href attribute for the anchor element.

HTML Helpers are methods in ASP.NET Core that assist in generating HTML markup. They are invoked within Razor views and help create HTML elements with the necessary attribute values.

<!-- HTML Helper: <a> element with controller and action parameters -->
@Html.ActionLink("Home", "Index", "Home")

Html.ActionLink is an HTML Helper method. It generates an anchor element with the text "Home" and sets the href attribute based on the specified controller and action values.

To quote the docs from Microsoft

The Visual Studio editor helps you write all of the markup in the Tag Helper approach of the register form, while Visual Studio provides no help for most of the code in the HTML Helpers approach. IntelliSense support for Tag Helpers goes into detail on working with Tag Helpers in the Visual Studio editor.

Client-side validation in ASP.NET MVC

In WebForms, server-side validation was the primary method employed. However, we developers added the client-side validation to enhance the user experience. The postbacks in web forms were too slow to have to wait on the validation. We had to code the validation rules twice: in javascript and on the backend. Clientside validation was there for the user experience and the serverside validation was there for the same basic validation to ensure no user can circumvent the rules. The backend could also do more advanced validation.

Asp.net core MVC has a different approach. jQuery Validation was introduced into the framework. This plugin allows us to define validation rules and messages for form fields. Rules can be specified using HTML attributes, enabling customization and client-side validation setup.

To streamline the integration of client-side validation into ASP.NET MVC, the jQuery Validate Unobtrusive plugin was introduced. This plugin utilizes HTML5 data attributes to specify validation rules.

I come to the point where I understand why the following scripts are here.

But now I got the question, how are the Html 5 data attributes used in the scaffolded project and do I still need to code my validation rules twice, once on the backend and now using HTML 5 data attributes? I will explore this by creating a Registration form and working my way through this until I understand it.

How ASP.NET core MVC validation works

In this example, I show how C# data attributes are used in ASP.NET MVC to generate validation. The client-side validation scripts use the HTML5 data-val-* attributes to perform client-side validation.

Step 1: Create a Model Class

Create a model class that represents the data you want to capture from the user. For this example, let's say we have a simple Registration model with two properties: Name and Email.

public record Registration
{
    [Required(ErrorMessage = "Name is required.")]
    public string Name { get; set; }

    [Required(ErrorMessage = "Email is required.")]
    [EmailAddress(ErrorMessage = "Invalid email address.")]
    public string Email { get; set; }
}

Here, we are using C# validation attributes like [Required] and [EmailAddress] to define the validation rules for the Name and Email properties.

Step 2: Create a View

Create a view that is strongly typed to the Registration model. I do this by right-clicking and selecting Creating a RazorPage based on the model Registration.

@page
@model ScaffoldedProject.Models.Registration

@{
    ViewData["Title"] = "Registration";
}

<h1>Registration</h1>

<h4>Registration</h4>
<hr />
<div class="row">
    <div class="col-md-4">
        <form asp-action="Registration">
            <div asp-validation-summary="ModelOnly" class="text-danger"></div>
            <div class="form-group">
                <label asp-for="Name" class="control-label"></label>
                <input asp-for="Name" class="form-control" />
                <span asp-validation-for="Name" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="Email" class="control-label"></label>
                <input asp-for="Email" class="form-control" />
                <span asp-validation-for="Email" class="text-danger"></span>
            </div>
            <div class="form-group">
                <input type="submit" value="Create" class="btn btn-primary" />
            </div>
        </form>
    </div>
</div>

<div>
    <a asp-action="Index">Back to List</a>
</div>

@section Scripts {
    @{await Html.RenderPartialAsync("_ValidationScriptsPartial");}
}

In the example VS2022 provided, I can see that tag helper such as asp-for, asp-validation-summary, asp-validation-for, and asp-action is used to generate the HTML form elements, display validation summary and field-specific validation messages, and define the form submit action.

The asp-validation-* tag helpers are responsible for rendering the data attributes needed for client-side validation. These data attributes are then utilized by the jquery.validate.unobtrusive.js script to perform client-side validation based on the rules defined in the C# validation attributes.

What about Intellisense and type safety?

When working with HTML, I got intellisense:

When I choose a property that does not exist in the model, I do get a build error as well.

I got the Registration in Red, because VS2022 does not find a controller

I fixed this error by creating a RegistrationController in a folder called Controllers

Step 3: Render the View

When you run the application and browse to the URL associated with this view, the HTML form elements will be rendered based on the model and the validation attributes. The generated HTML will include data attributes based on the validation attributes:

• the Required attribute applied to the Name property of the RegistrationModel

        [Required(ErrorMessage = "Name is required.")]
        public string Name { get; set; }
  

  using the following snippet of the razor from above

               <div class="form-group">
                    <label asp-for="Name" class="control-label"></label>
                    <input asp-for="Name" class="form-control" />
                    <span asp-validation-for="Name" class="text-danger"></span>
                </div>
  

  Results in the following generated HTML.

    <div class="form-group">
                    <label class="control-label" for="Name">Name</label>
                    <input class="form-control input-validation-error" type="text" data-val="true" data-val-required="Name is required." id="Name" name="Name" value="" aria-describedby="Name-error" aria-invalid="true">
                    <span class="text-danger field-validation-error" data-valmsg-for="Name" data-valmsg-replace="true"><span id="Name-error" class="">Name is required.</span></span>
    </div>
  

  the same story applies for the EmailAddress attribute on the Email property

    <div class="form-group">
                    <label class="control-label" for="Email">Email</label>
                    <input class="form-control input-validation-error" type="email" data-val="true" data-val-email="Invalid email address." data-val-required="Email is required." id="Email" name="Email" value="" aria-describedby="Email-error" aria-invalid="true">
                    <span class="text-danger field-validation-error" data-valmsg-for="Email" data-valmsg-replace="true"><span id="Email-error" class="">Email is required.</span></span>
    </div>
  

These data attributes are used by jQuery Validate and jQuery Validate Unobtrusive. In Summary, These scripts perform client-side validation, defined in the C# data attributes.

Step 4: Client-side Validation

The data-val-required attribute and the data-val-email attribute will be used by jQuery Validate to validate the required and email fields respectively. jQuery Validate will be able to do this because jQuery Validate Unobtrusive defined the rules and added support to those rules in the jQuery Validate framework.

The above text displays after the inputbox loses focus.

Difference between jQuery Validate and jQuery Validate Unobtrusive

The difference between jQuery Validate and jQuery Validate Unobtrusive lies in how they handle client-side validation based on the data attributes generated from C# validation attributes.

The two scripts, jquery.validate.js and jquery.validate.unobtrusive.js are included in the ValidationscriptsPartial.cshtml in an ASP.NET core MVC project.

jQuery Validate

This script allows you to define validation rules and error messages for form fields using a flexible API. I can customize the validation rules and messages for each field. It supports custom validation methods and remote validation.

jQuery Validate will scan the HTML form elements for the data attributes and applies the specified validation rules based on those attributes. When the form is submitted, jQuery Validate intercepts the submission and validates the fields based on the defined rules. If any validation errors occur, it displays error messages and prevents form submission until all fields are valid.

Example

In the code of jquery.validate I found the following code for required and email.

// https://jqueryvalidation.org/jQuery.validator.methods/
    methods: {

        // https://jqueryvalidation.org/required-method/
        required: function( value, element, param ) {

            // Check if dependency is met
            if ( !this.depend( param, element ) ) {
                return "dependency-mismatch";
            }
            if ( element.nodeName.toLowerCase() === "select" ) {

                // Could be an array for select-multiple or a string, both are fine this way
                var val = $( element ).val();
                return val && val.length > 0;
            }
            if ( this.checkable( element ) ) {
                return this.getLength( value, element ) > 0;
            }
            return value !== undefined && value !== null && value.length > 0;
        },

        // https://jqueryvalidation.org/email-method/
        email: function( value, element ) {

            // From https://html.spec.whatwg.org/multipage/forms.html#valid-e-mail-address
            // Retrieved 2014-01-14
            // If you have a problem with this implementation, report a bug against the above spec
            // Or use custom methods to implement your own email validation
            return this.optional( element ) || /^[a-zA-Z0-9.!#$%&'*+\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/.test( value );
        },

So I can see that jQuery Validate offers the rules that I need.

jQuery Validate Unobtrusive

This script is an additional script that works in conjunction with jQuery Validate and ASP.NET MVC. It helps bridge the gap between C# validation attributes and jQuery Validate by applying the validation rules defined in C# to the client-side validation. Note that only the attributes in the jQuery Validate Ubobtrusive can be used. When I want to use custom attributes, I will need to do some extra work.

When I include the jquery.validate.unobtrusive.js file, it reads the data attributes and wires up jQuery Validate to use those rules for client-side validation. It takes care of mapping the C# validation attributes (such as Required, StringLength, etc.) to the appropriate jQuery Validate rules.

jquery.validate.unobtrusive.js depends on jquery.validate.js. It does not override or replace the core functionality provided by jquery.validate.js.

Example

In the code of jquery.validate.unobtrusive.js I found the following code for email attribute.

    adapters....addBool("email")....;

   adapters.addBool = function (adapterName, ruleName) {
        /// <summary>Adds a new adapter to convert unobtrusive HTML into a jQuery Validate validation, where
        /// the jQuery Validate validation rule has no parameter values.</summary>
        /// <param name="adapterName" type="String">The name of the adapter to be added. This matches the name used
        /// in the data-val-nnnn HTML attribute (where nnnn is the adapter name).</param>
        /// <param name="ruleName" type="String" optional="true">[Optional] The name of the jQuery Validate rule. If not provided, the value
        /// of adapterName will be used instead.</param>
        /// <returns type="jQuery.validator.unobtrusive.adapters" />
        return this.add(adapterName, function (options) {
            setValidationValues(options, ruleName || adapterName, true);
        });
    };

And of course for the required attribute.

 adapters.add("required", function (options) {
        // jQuery Validate equates "required" with "mandatory" for checkbox elements
        if (options.element.tagName.toUpperCase() !== "INPUT" || options.element.type.toUpperCase() !== "CHECKBOX") {
            setValidationValues(options, "required", true);
        }
    });

     function setValidationValues(options, ruleName, value) {
        options.rules[ruleName] = value;
        if (options.message) {
            options.messages[ruleName] = options.message;
        }
    }

Can I update jQuery, jQuery Validate and jQuery Validate Unobtrusive?

Now I understand the link between everything, and after writing my previous post about LibMan and npm, I can answer this question.

The generated data-val-* attributes are the interface for the validation rules. When using the tag helpers, that happens using the asp-validation-for attribute.

I can confidently update to a more recent version of jQuery Validate Unobtrusive. This is the framework I need to manage. It depends on jQuery Validate and jQuery. When updating to a newer script Unobtrusive script, I can refer to the release notes of this framework.

I will not use LibMan but instead, opt for npm. If I have other dependencies that utilize jQuery and jQuery Validate, I can easily identify conflicts when updating.

Outro

I've shared the history and evolution I learned of ASP.NET MVC, AJAX, and jQuery. With the release of ASP.NET MVC, the integration of jQuery provides a user-friendly API for client-side development. This integration, combined with the use of data attributes and validation attributes in ASP.NET MVC, allows for seamless client-side validation. There is also an evolution towards tag helpers instead of Html helpers.

By utilizing C# validation attributes, I can easily define validation rules for our model properties which are automatically translated into HTML form elements with corresponding data attributes.

Understanding this process has given me the confidence to update these scripts using tools like npm to manage dependencies and avoid conflicts within my application.

In one of my next posts, I will figure out how to add support for FluentValidation and how to support custom attributes on my models.

Sources

Tag Helpers in ASP.NET Core | Microsoft Learn

https://learn.microsoft.com/en-us/aspnet/core/tutorials/razor-pages/razor-pages-start?view=aspnetcore-7.0&tabs=visual-studio

Releases · aspnet/jquery-validation-unobtrusive (github.com)

ASP.NET Web Forms - Wikipedia

The History of ASP.NET – Part I | DotNetCurry

Upgrade Your Client-Side Script Approach in a Basic Dotnet Asp.Net Core MVC App (kriebbels.me)

Does anyone know the history of ASP.NET Webforms? - Stack Overflow

In my previous articles, I discussed DevOps, DotNet and Security. This article builds upon those three components and explores the role of package managers in managing server-side and client-side libraries in a .NET MVC project.

Context

I find myself working on a new Asp.Net Core project. In the previous projects, the pen-testers and the SAST scan informed me that I use old versions of javascript libraries like jQuery. Managing and updating these libraries are not provided by default it seems. This led me to explore how to manage those files.

At the beginning of time, we used linked files and created scripts. We as developers manually copied and included libraries in our projects. Over time, the development community has witnessed an evolution in the way libraries are managed.

All this was prone to errors and made updates difficult. For the backend developers, GAC was the first solution to try to manage the DLL hell. The introduction of package managers like NuGet was a successor. It offered a centralized repository for clientside and serverside libraries. The goal was to simplify management and the updating process. Javascript developers needed something less coupled to NuGet and npm was born.

JavaScript has been the go-to language for some backend applications but more for web applications. To manage client-side libraries, we developers have relied on tools like npm and Yarn. These package managers allow us to integrate libraries and frameworks such as jQuery, AngularJS, and React.

JavaScript package managers like npm and yarn are popular. Npm has become the choice. It has compatibility with server-side JavaScript frameworks like Node.js.

In the .NET world, a newly scaffolded ASP.NET Core project does not include clientside package management for the already given client libraries.

First thing first. I am talking about package managers. What is the purpose of a package manager so I can understand it better? What do I need to do because it is not included in a default scaffolded ASP.Net Core project?

Why a package manager?

Package managers serve an important role in software development:

• Simplifying Dependency Management

Managing dependencies manually can quickly become complex and error-prone. Package managers automate the process of library installation. No more copy and paste.

• Ensuring Version Control

Each library update may introduce bug fixes, security patches, or new features. Package managers help enforce version control by providing a centralized repository of libraries and their corresponding versions. This allows me to select the most suitable version and traceability.

• Streamlining interdependencies

Package managers enable reproducible builds, ensuring that the same set of libraries and versions are used across different machines or deployments. That will reduce the risk of compatibility issues because the package manager will try to figure out what common packages are compatible with each other.

• Discoverability

Package managers provide a centralized repository or registry where I can discover new libraries, frameworks, and tools.

• Security and Maintenance

By regularly updating packages through package managers, I can address security concerns and reduce the risk of bugs and attacks.

Show me something already

I will scaffold an asp.net core MVC project. I will investigate the javascript files. Discover what package managers there are. I will try to manage them using a package manager and convert them to use another one.

An Asp.Net Core MVC Fresh Project

Let us look at what I received after I created an Asp.Net Core MVC Application using VS 2022.

I notice a lot of given client javascript files.

• site.js: This file is a custom JavaScript file where I can write my client-side javascript code.

• bootstrap.js: This file is the JavaScript component of the Bootstrap framework. It adds dynamic behaviour to the Bootstrap UI components.

•   /*!
      * Bootstrap v5.1.0 (https://getbootstrap.com/)
      * Copyright 2011-2021 The Bootstrap Authors (https://github.com/twbs/bootstrap/graphs/contributors)
      * Licensed under MIT (https://github.com/twbs/bootstrap/blob/main/LICENSE)
      */
  

• jquery.js: This is the jQuery library file. It provides an API for manipulating and interacting with HTML elements.

    /*!
     * jQuery JavaScript Library v3.5.1
     * https://jquery.com/
     *
     * Includes Sizzle.js
     * https://sizzlejs.com/
     *
     * Copyright JS Foundation and other contributors
     * Released under the MIT license
     * https://jquery.org/license
     *
     * Date: 2020-05-04T22:49Z
     */
  

• jquery.validate.js: This is a popular jQuery called jQuery Validation plugin that provides extensive form validation capabilities.

     * jQuery Validation Plugin v1.17.0
     *
     * https://jqueryvalidation.org/
  

• jquery.validate.unobtrusive.js: This file is a jQuery plugin that works in conjunction with jQuery Validation and the ASP.NET Core MVC framework. The purpose of jQuery Validation Unobtrusive is to validate user input on the client side before submitting the form to the server. More on that in a later blog post.

    // Unobtrusive validation support library for jQuery and jQuery Validate
    // Copyright (c) .NET Foundation. All rights reserved.
    // Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.
    // @version v3.2.11
  

I can safely say that four of the five files are part of a package. The downside is that Microsoft delivers us library as files. That gives us the possibility to add or change code. From the moment that I or somebody else does this, hell can break loose later on in the development cycle. Imagine a bugfix or I want to use new features. But for that, I need a new file. Updating the file will translate into losing the functionality of the changes I made. So I need to compare the sources... Those practices are not anymore in this day and age.

Let us try to add some package management.

What Package Manager do we have?

As a dotnet backend developer, I use NuGet. When I take on the role of a FrontEnd developer, I seem to have these choices:

1. npm (Node Package Manager) is the default package manager for the JavaScript/Typescript ecosystem. It is widely used for managing dependencies.

2. Yarn is a fast and reliable package manager developed by Facebook. It aims to address some performance and security issues found in npm. Yarn uses the same package.json format as npm and is compatible with existing npm packages.

3. Pnpm is a fast and space-efficient package manager that reduces duplication of dependencies across projects. It uses a unique approach called "shared dependencies" to save disk space and improve installation speed.

But what does Microsoft offer? Microsoft has something that is called LibMan.

Enter LibMan

LibMan (short for Library Manager) is a client-side library acquisition tool and not a traditional package manager. It is part of the Microsoft ecosystem and is primarily used in combination with ASP.NET Core projects to manage client-side library assets.

LibMan simplifies acquiring client-side libraries. It is designed to integrate with ASP.NET Core projects. It allows me to specify the libraries I require. It will download the files directly from external sources like Content Delivery Networks (CDNs), GitHub repositories or other locations.

It does not provide the same level of dependency management and version control as npm or Yarn.

Let me try out LibMan.

Creating a LibMan.json

Since VS 2017 there is support for LibMan. I use Visual Studio 2022 Community 17.6.4. When I right-click on my project I get the possibility to manage ClientSide Libraries.

However, In the Menu Project, I only have a user interface for NuGet Manager.

After I click on Manage Client-Side Libraries, a libman.json file is created.

There I can define library location. This file includes information about the libraries, versions, target paths and source. The target paths are already used in the HTML and JavaScript files that I receive when I create an asp.net core MVC project from scratch. Because I do not change the target path, all should still work as intended.

For this project, I reversed-engineered it to the following libman.json file:

{
  "version": "1.0",
  "defaultProvider": "cdnjs",
  "libraries": [
    {
      "library": "bootstrap@5.1.0",
      "destination": "wwwroot/lib/bootstrap/dist",
      "files": [
        "js/bootstrap.js",
        "js/bootstrap.min.js"
      ]
    },
    {
      "library": "jquery@3.5.1",
      "destination": "wwwroot/lib/jquery/dist",
      "files": [
        "jquery.js",
        "jquery.min.js"
      ]
    },
    {
      "library": "jquery-validate@1.17.0",
      "destination": "wwwroot/lib/jquery-validation/dist",
      "files": [
        "jquery.validate.js",
        "additional-methods.js",
        "jquery.validate.min.js",
        "additional-methods.min.js"
      ]
    },
    {
      "library": "jquery-validation-unobtrusive@3.2.11",
      "destination": "wwwroot/lib/jquery-validation-unobtrusive/",
      "files": [
        "jquery.validate.unobtrusive.js",
        "jquery.validate.unobtrusive.min.js"

      ]
    }
  ]
}

When saving the file, Visual Studio goes to work and I got the following output:

Restore operation started
Restoring libraries for project ScaffoldedProject
Restoring library bootstrap@5.1.0... (ScaffoldedProject)
Downloading file https://cdnjs.cloudflare.com/ajax/libs/bootstrap/5.1.0/js/bootstrap.js... (ScaffoldedProject)
Downloading file https://cdnjs.cloudflare.com/ajax/libs/bootstrap/5.1.0/js/bootstrap.min.js... (ScaffoldedProject)
wwwroot/lib/bootstrap/dist/js/bootstrap.js written to destination (ScaffoldedProject)
wwwroot/lib/bootstrap/dist/js/bootstrap.min.js written to destination (ScaffoldedProject)
Restoring library jquery@3.5.1... (ScaffoldedProject)
Downloading file https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.js... (ScaffoldedProject)
Downloading file https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js... (ScaffoldedProject)
wwwroot/lib/jquery/dist/jquery.js written to destination (ScaffoldedProject)
wwwroot/lib/jquery/dist/jquery.min.js written to destination (ScaffoldedProject)
Restoring library jquery-validate@1.17.0... (ScaffoldedProject)
Downloading file https://cdnjs.cloudflare.com/ajax/libs/jquery-validate/1.17.0/jquery.validate.js... (ScaffoldedProject)
Downloading file https://cdnjs.cloudflare.com/ajax/libs/jquery-validate/1.17.0/additional-methods.js... (ScaffoldedProject)
Downloading file https://cdnjs.cloudflare.com/ajax/libs/jquery-validate/1.17.0/jquery.validate.min.js... (ScaffoldedProject)
Downloading file https://cdnjs.cloudflare.com/ajax/libs/jquery-validate/1.17.0/additional-methods.min.js... (ScaffoldedProject)
wwwroot/lib/jquery-validation/dist/jquery.validate.js written to destination (ScaffoldedProject)
wwwroot/lib/jquery-validation/dist/additional-methods.js written to destination (ScaffoldedProject)
wwwroot/lib/jquery-validation/dist/jquery.validate.min.js written to destination (ScaffoldedProject)
wwwroot/lib/jquery-validation/dist/additional-methods.min.js written to destination (ScaffoldedProject)
Restoring library jquery-validation-unobtrusive@3.2.11... (ScaffoldedProject)
Downloading file https://cdnjs.cloudflare.com/ajax/libs/jquery-validation-unobtrusive/3.2.11/jquery.validate.unobtrusive.js... (ScaffoldedProject)
Downloading file https://cdnjs.cloudflare.com/ajax/libs/jquery-validation-unobtrusive/3.2.11/jquery.validate.unobtrusive.min.js... (ScaffoldedProject)
wwwroot/lib/jquery-validation-unobtrusive/jquery.validate.unobtrusive.js written to destination (ScaffoldedProject)
wwwroot/lib/jquery-validation-unobtrusive/jquery.validate.unobtrusive.min.js written to destination (ScaffoldedProject)
Restore operation completed

When I do not have access to Visual Studio, I can use the CLI with the following command: libman restore.

Updating packages

Can we now update safely to the latest minor? Because the package uses SemVer, I should be able to update it without problems. However, updating a Major version will need some extra investigation. I quote SemVer.org

Given a version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when you make incompatible API changes

2. MINOR version when you add functionality in a backward compatible manner

3. PATCH version when you make backward compatible bug fixes

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

In the libman.json, Visual Studio offers a tooltip with more information about a package:

In the image above I notice that there is a newer version. Security-wise, I should always use the latest stable version that is most compatible with what I use or need.

The update libman.json looks like this (redacted):

**redacted**
    {
      "library": "bootstrap@5.3.0",
**redacted**
    },
    {
      "library": "jquery@3.7.0",
**redacted**
    },
    {
      "library": "jquery-validate@1.19.5",
**redacted**
    },
    {
      "library": "jquery-validation-unobtrusive@4.0.0",
**redacted**

Bootstrap, JQuery and JQuery-Validate is updated to the latest Minor. That ensures backward compatibility with new features. However, the jQuery-Validation-unobstrusive got a new Major version. This means that breaking changes can occur. Not only in the library itself but in what it uses as a dependency. When I look at the GitHub repository, I notice that the package manager NuGet is used and for jQuery NuGet Version 1.8 is used. So I already start to feel the pain of manually working out the dependencies between all those files.

So... why not npm or yarn or ...

LibMan is not a full-fledged package manager. I will use npm and yarn as a comparison.

npm and Yarn have:

• a lot of packages from the official package registry. They excel in managing complex dependency trees and versioning control. This all makes it easier to ensure compatibility between different packages.

• large and active developer communities, which means there is extensive community support, documentation, and tutorials available. LibMan is only an acquiring tool.

• build configuration options. This allows me to define custom scripts, and automate tasks. This flexibility can be beneficial for large-scale projects or when you require more fine-grained control over build processes.

• good compatibility with build tools like Webpack, Babel, and ESLint.

• support for plugins and extensions that further extend their capabilities.

In this project, there are dependencies between them. It makes more sense to ensure compatibility between all those files.

What would that look like? To mimic the functionality of the libman.json file using npm, I can leverage npm's package.json file. I can manage the dependencies as regular npm packages. The following is package.json file that replicates the libman.json configurations:

{
  "name": "@Scaffold-1.0.0",
   "version": "1.0.0",
  "scripts": {
    "update-libs": "npm update bootstrap jquery jquery-validation jquery-validation-unobtrusive"
  },
  "dependencies": {
    "bootstrap": "^5.3.0",
    "jquery": "^3.7.0",
    "jquery-validation": "^1.19.5",
    "jquery-validation-unobtrusive": "^4.0.0"
  },
  "devDependencies": {  }
}

In this package.json file:

• All the required libraries (bootstrap, jquery, jquery-validation, jquery-validation-unobtrusive) are defined as regular dependencies under the "dependencies" section.

• The script allows me to update the libraries later using the npm run update-libs command if newer versions become available.

To install the dependencies and set up the required folder structure, I can run npm install or I can do a right click in Visual Studio and choose Restore Packages. This will install the defined packages, and create the necessary directories under node_modules.

PATH=.\node_modules\.bin;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VisualStudio\NodeJs\win-x64;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VisualStudio\NodeJs;C:\Program Files\Microsoft Visual Studio\2022\Community\Web\External;%PATH%;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\cmd
"C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VisualStudio\NodeJs\npm.CMD" install
added 6 packages, and audited 7 packages in 6s
2 packages are looking for funding
  run `npm fund` for details

With this setup, I can handle the installation, updating, and version control of the specified libraries in a similar way to libman.json. Visual Studio visualises this on the project level, under dependencies.

The files are downloaded where npm handles them, in node_modules.

And now the pain begins. I need to change all the html files that have referenced the lib folder. I also need to ensure that only the needed files from the node_modules are available when the project is published. I include the following in my csproj file.

<ItemGroup>
  <None Include="node_modules\**\*"
        CopyToPublishDirectory="PreserveNewest"
        CopyToPublishDirectoryExclude="**/*.ts; **/*.map"
        CopyToPublishDirectoryAlways="false" />

  <Content Include="node_modules\**\*.min.js"
           CopyToPublishDirectory="PreserveNewest"
           CopyToPublishDirectoryAlways="false" />
</ItemGroup>

The funny thing is that when I add the node_modules folder, it disappears from the Dependencies in the Solution Explorer... Down the rabbit hole, I go. I notice that there are other dependencies, that I did not specify using LibMan.json

However, here I will end the blog post. I think I made it clear what the difference is between LibMan and e.g. npm and why I should address this from the start before I do my coding to ensure I have a working version. I should at least use LibMan, to protect the project from people that want to manually changes the javascript files.

Outro

I have explored the importance and purpose of package managers. I find package managers to be indispensable tools in my workflow. I do not understand why the scaffolded project does not come with preconfigured libman.json or with a package.json

LibMan offers a direct link to a source and will make it harder for us developers to change the javascript files to change something. Having such a libman.json, makes it easier to migrate to a later version. It gives a sense of urgency to try to update these files to the latest (minor) version, ensuring we have the latest secure versions.

However, the fun begins when you want to use more packages and have common dependencies. From then on, npm will help you out.

What do you think? What did I miss? Do you have any other insights? Let me know! Let us engage.

Sources

Below I give an overview of the sources I used to write this.

javascript client package managers - Google Search

JavaScript package managers compared: npm, Yarn, or pnpm? - LogRocket Blog

Which package manager to choose for your next javascript project? | by Himanshu Garadiya | Jun, 2023 | Medium

Package Managers in JavaScript (c-sharpcorner.com)

Bower is dead, long live npm. And Yarn. And webpack. | Snyk

Semantic Versioning 2.0.0 | Semantic Versioning (semver.org)

Add nuspec and build file (#77) · aspnet/jquery-validation-unobtrusive@6c2113e (github.com)

https://jspm.org/

Client-side library acquisition in ASP.NET Core with LibMan | Microsoft Learn

Use LibMan with ASP.NET Core in Visual Studio | Microsoft Learn

aspnet/LibraryManager (github.com)

npm (npmjs.com)

An alternative way to use Visual Studio with npm | by EasyLOB | Medium

In my previous articles, I discussed the creation of a mocked OpenID Connect provider and the evolution of the HttpClient. Today, I want to discuss how user attributes (claims) are presented to the application using tokens.

Context

When I explored a codebase, I noticed that the access tokens and ID tokens are retrieved from the HttpRequest. This happened using custom code to access the headervalue of id_token and access_token. However, Asp.net core does give us the possibility to access those claims using HttpContext.User.Claims property.

In my code, I typically use the AddJwtBearer methods for backend services. When working on an MVC application, I tend to use the AddOpenIdConnect method. I find it a bit unclear when to use what. I want to clarify the differences between the two. It is not the scope of this article to state clearly the functional purpose of all the types of tokens.

Before we dive into the mechanics of accessing and utilizing claims, let's discuss what claims are and their purpose.

Claims

A claim represents a piece of information about a user. An example of a claim is a user's name, email address, and role,...

Tokens

A token is a string that serves as proof of identity or permission. Tokens are used to authenticate and authorize users.

Claims are stored in a token by e.g. Saml, Oauth2, OpenIdConnect or other authorization providers. The application will use the tokens for authentication and authorization purposes. The application is in Oauth2/OpenIdConnect jargon called a resource server. I will slightly mention 3 types of tokens: JWT, SAML and Opaque.

JWT (JSON Web Token):

JWT is a compact, URL-safe, and self-contained token format that consists of three base64-encoded parts separated by dots.

The three parts are the header, the payload, and the signature.

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

When decoded using jwt.ms, I get the following content

{
  "alg": "HS256",
  "typ": "JWT"
}.{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022
}.[Signature]

In this example, sub, name and iat are claims.

SAML (Security Assertion Markup Language) Token

SAML tokens are XML-based tokens used for exchanging authentication and authorization data between identity providers and service providers ( the application = resource server).

<Assertion xmlns="urn:oasis:names:tc:SAML:2.0:assertion" ID="_934873948nc4" IssueInstant="2022-03-01T12:34:56Z" Version="2.0">
  <Issuer>https://identityprovider.com</Issuer>
  <Subject>
    <NameID>1234567890</NameID>
    <SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
      <SubjectConfirmationData NotOnOrAfter="2022-03-01T12:40:00Z" Recipient="https://serviceprovider.com" />
    </SubjectConfirmation>
  </Subject>
  <Conditions NotBefore="2022-03-01T12:34:56Z" NotOnOrAfter="2022-03-01T12:37:56Z">
    <AudienceRestriction>
      <Audience>https://serviceprovider.com</Audience>
    </AudienceRestriction>
  </Conditions>
  <AttributeStatement>
    <Attribute Name="email" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified">
      <AttributeValue>john.doe@example.com</AttributeValue>
    </Attribute>
    <Attribute Name="role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified">
      <AttributeValue>User</AttributeValue>
    </Attribute>
  </AttributeStatement>
  <AuthnStatement AuthnInstant="2022-03-01T12:34:56Z">
    <AuthnContext>
      <AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:Password</AuthnContextClassRef>
    </AuthnContext>
  </AuthnStatement>
</Assertion>
```

In the example above, the creator of the token is https://identityprovider.com . The token is for the application ( = resource server ) called https://serviceprovider.com.

The tags Issuer and Subject, together with the attributes email and role are claims.

Opaque Token

An opaque token is a token that is not self-contained. That means the token does not contain readable information. I need the IDP server to know what claims this token represents. An opaque token is typically a long and random string with no useful information embedded within the token itself.

Here's an example of an opaque token:

JP67LLI73Y2DF3428IYV888PZFGTXZVH5H9S

In this example, due to the nature of an opaque token, there are no claims defined. To have that information, I need to make a call to an introspective endpoint.

Tokens purposes

OAuth2 has introduced the access_token and OpenIDConnect has introduced an id_token. The OpenIDConnect protocol is built on top of the OAuth2 protocol. When there is an OpenIDConnect server involved, the server can send an access_token and an id_token. It is important to know when to use what type of token. Most of the time but not always, an access_token can be decoded and the claims can be extracted. An id_token can always be decoded. An ID Token is always a JWT.

What if an access_token is an opaque token? The Oauth2 protocol defines an introspection endpoint. That allows the resource server to retrieve claims based on the supplied token. This is more secure. Everyone can decode a JWT token. It is not advisable to send over sensitive data using a JWT token.

Id Token

An ID token contains specific information about the user and is for authentication/authorization purposes.

E.g. Upon logging into the web application, the user is redirected to the identity provider for authentication (e.g. Google/Auth0/Microsoft). Once authenticated, the IdP generates an ID token, such as a JWT, containing the user's ID, name (e.g., John Doe), and email address (e.g., john.doe@example.com). The web application receives and validates the ID token. This means that we consider the user's identity is verified.

Access Token

An access token is primarily used to access protected resources. resources delivered by our resource server ( = application ). When the request has a valid access_token defined in the header, the application ( resource server) allows the request to be processed further in the pipeline.

E.g. After the user has successfully authenticated themselves and obtained the ID token, they want to place an order, which requires proper authorization. It is not because the application knows who you are, that you are allowed to place an order! The client application requests an access token from the identity provider. The identity provider verifies the user's credentials and checks their permissions. When the Idp provider can validate that the user may place an order, the Idp will issue an access token. What also can be, is that the IDP server will provide a token, but without mentioning the access_rights. The resource needs to be sure that all needed claims are there and valid, so that the user may place an order.

Middleware with Tokens and Claims

From here on, I will focus only on the JWT tokens and the OAuth2 and OpenIDConnect protocols.

Overview

To understand how claims are processed within an ASP.NET Core application, let's take a helicopter overview of the journey of an HTTP request. Read more about Middleware: ASP.NET Core Middleware | Microsoft Learn

Imagine a scenario where you have a WeatherForecastController in your application. Upon receiving an HTTP request, the middleware pipeline kicks in and performs tasks, such as authentication and authorization. The middleware responsible for handling authentication will extract and validate the ID token and access token from the request's headers or cookies.

If the access token is opaque, meaning it cannot be decoded locally, the authentication middleware will need to communicate with the token introspection endpoint provided by the OpenID Connect provider to retrieve relevant claims.

Once the tokens have been validated and decoded, the claims are extracted and stored in the HttpContext's claims collection. These claims are then available throughout the request lifecycle.

It's important to note that if e.g. the audience, issuer, and time validity of the tokens don't match the expected values, then the middleware will discard the tokens. The action of the controller will not be called!

How to manipulate the Claims mapping process?

In some cases, I want to filter or map the claims received from the tokens to fit your domain-specific requirements or align with a different naming convention. ASP.NET Core provides a mechanism to achieve this.

Oauth2

Asp.net core does a lot for us. It is all about using the method AddAuthentication and AddJwtBearer when we just want to work with the OAuth2 protocol. Essentially it comes down to the following code.

//Program.cs

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
        .AddJwtBearer(options =>
        {
            //Ensuring the token comes from the IDP we expect!
            options.Authority = "https://identityprovider.com";
            //Ensure the token we want to process is the one that is for us
            options.Audience = "https://api.myapp.com";
            options.TokenValidationParameters = new TokenValidationParameters
            {
                ValidateIssuer = true,
                ValidateAudience = true
                //...Other settings for validation of the token
            };
            Events = new JwtBearerEvents
            {
                OnTokenValidated = async ctx =>
                {
                    var claims = new List<Claim>
                    {
                        new Claim("ExtraClaim", "SomeValue")
                    };
                var extraIdentity = new ClaimsIdentity(claims);   
                ctx.Principal.AddIdentity(extraIdentity );
                }
            }
        });
//...

IApp app = builder.Build();
    app.UseRouting();
    app.UseAuthentication(); // Enable authentication middleware
    app.UseAuthorization(); // Enable authorization middleware

    app.UseEndpoints(endpoints =>
    {
        endpoints.MapControllers();
    });

Read here for more information.

OpenID connect

OpenID Connect allows us to work differently. Unlike the AddJwtBearer method, we now have a ClaimsActions dictionary.

The ClaimActions dictionary is an option available when configuring the OpenID Connect authentication middleware using the AddOpenIdConnect() method. It allows me to customize the claim-handling behaviour of the middleware during the authentication process.

The ClaimActions dictionary provides a way to manipulate and control how the claims are mapped to the ClaimsIdentity in the authentication middleware. The dictionary maps the claim types to specific actions that should be performed on the claims during the mapping process. By using the ClaimActions.Clear(), the default mapping that Asp.net core does, will not occur.

Here is an example of setting up the ClaimActions dictionary:

services.AddAuthentication()
    .AddOpenIdConnect("OpenID", options =>
    {
        // Other configuration options
        //...
        //disable dotnet claimactions
        options.ClaimActions.Clear(); 

        //only map the claims I want!
        options.ClaimActions.MapUniqueJsonKey("role", "role");
        options.ClaimActions.MapUniqueJsonKey("sub", "userId");
        options.ClaimActions.MapUniqueJsonKey("email", "emailAddress");
        options.ClaimActions.MapUniqueJsonKey("custom_claim", "customClaimName");

        // Additional configuration
    });

AddJwtBearer or AddOpenIdConnect?

I can use the AddOpenIdConnect method instead of the AddJwtBearer method for validating access tokens and performing claims mapping.

The AddJwtBearer method is typically used when you have a separate authentication server that issues JWT tokens and you want to validate these tokens in my application. It is a lightweight middleware that focuses solely on token validation and does not perform any claims mapping or user information retrieval.

The AddOpenIdConnect method is used when you want to use OpenID Connect, which is an extension of OAuth 2.0 for authentication. It provides additional features like identity provider discovery, user authentication, claims-based authorization, and more. The AddOpenIdConnect middleware not only validates the tokens but also handles user information retrieval and claims mapping automatically.

ClaimsTransformation

For more advanced scenarios, I can implement a custom ClaimsTransformer. This allows me to intercept the claims before they are stored in the HttpContext and manipulate them as needed. This approach allows you to filter out unnecessary claims or transform their names to match your application's terminology.

In the following example, I want to parse the provider and userid from the sub-claim defined by the IdpProvider Auth0.

public class Auth0ProviderClaimsTransformation : IClaimsTransformation
{
    // Regular expression pattern to parse Auth0 user ID
    private readonly string auth0UserIdPattern = @"^(.*)\|(.*)$";

    public Task<ClaimsPrincipal> TransformAsync(ClaimsPrincipal principal)
    {
        var identity = (ClaimsIdentity)principal.Identity;

        // Get the sub claim (Auth0 user ID)
        var subClaim = identity.FindFirst("sub");

        if (subClaim != null)
        {
            // Extract provider and user ID from Auth0 user ID
            var matches = Regex.Match(subClaim.Value, auth0UserIdPattern);
            if (matches.Success)
            {
                var provider = matches.Groups[0].Value;;
                var userId = matches.Groups[1].Value;

                // Add provider and userId claims
                identity.AddClaim(new Claim("provider", provider));
                identity.AddClaim(new Claim("userid", userId));
            }
        }

        return Task.FromResult(principal);
    }
}
```

I have to register the class as a scoped service on the service collection and it works automagically.

builder.Services.AddTransient<IClaimsTransformation, Auth0ProviderClaimsTransformation >();

More on that here.

Accessing Claims in ASP.NET Core 6

When it comes to accessing claims in ASP.NET Core 6, I have several options depending on the context in which you need to utilize them. It comes down to tracking down where to access the HttpContext and use the User property. On the User property, I have access to the Claims property.

Authentication and Authorization Middleware

In the authentication middleware and authorization middleware, I can access claims directly from the HttpContext, ensuring that the necessary authentication and authorization checks are in place before proceeding.

AddJwtBearer method

I repeat the example used above but with a focus on where I can access the claims using the HttpContext. Read more about JwtBearerEvents here.

services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
        .AddJwtBearer(options =>
        {
            // Configure JWT Bearer options
            // ...

            options.Events = new JwtBearerEvents
            {
                OnAuthenticationFailed = async context =>
                {
                    var httpContext = context.HttpContext;
                    // Access HttpContext
                    // ...

                    await Task.CompletedTask;
                },
                OnTokenValidated = async context =>
                {
                    var httpContext = context.HttpContext;
                    // Access HttpContext
                    // ...

                    await Task.CompletedTask;
                }
            };
        });

AddOpenIdConnect method

The same way of working with the AddJwtBearer method applies when working with the AddOpenIdConnect method.

services.AddAuthentication(options =>
{
    options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme;
    options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme;
})
.AddCookie()
.AddOpenIdConnect(options =>
{
    // Configure OpenID Connect options
    // ...

    options.Events = new OpenIdConnectEvents
    {
        OnRedirectToIdentityProvider = async context =>
        {
            var httpContext = context.HttpContext;
            // Access HttpContext
            // ...

            await Task.CompletedTask;
        },
        OnTokenValidated = async context =>
        {
            var httpContext = context.HttpContext;
            // Access HttpContext
            // ...

            await Task.CompletedTask;
        }
    };
});

Controller

When working in a controller action or service layer, it is as easy to access the HttpContext.User property.

Why is it easy? Let me copy and paste the following code from this page from Microsoft Learn.

ClaimsPrincipal principal = HttpContext.User as ClaimsPrincipal;  
if (null != principal)  
{  
   foreach (Claim claim in principal.Claims)  
   {  
      Console.WriteLine("CLAIM TYPE: " + claim.Type + "; CLAIM VALUE: " + claim.Value + "</br>");  
   }  
}

How would I access it in a controller then? Below you find some code that shows the access to the claim, using the HttpContext.User property. Read here for more information.

  [ApiController]
    [Route("[controller]")]
    public class WeatherForecastController : ControllerBase
    {
        private static readonly string[] Summaries = new[]
        {
            "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
        };

        [HttpGet]
        [Authorize]
        public IEnumerable<WeatherForecast> Get()
        {
            var email = HttpContext.User.FindFirstValue(ClaimTypes.Email);

            if (email == null)
            {
                throw new Exception("Email claim not found.");
            }

I strongly recommend passing the claims as parameters or including them as part of a Data Transfer Object (DTO) when using a service in your controller. Do not use the IHttpContextAccessor in your domain or business logic layer. This ensures loose coupling between the service layer and the HttpContext. This allows for better testability and maintainability.

From my experience, all the tokens from the access_token and id_token are available. According to ChatGPT, the id_token should have precedence over the access_token when a claim is defined in both tokens but have a different value. I want to validate what it said, however, I do not find it in the sources on GitHub.

Other sources

I do like to mention the following sources that I may not explicitly mention in a link:

User claims in ASP.NET Core using OpenID Connect Authentication | Software Engineering (damienbod.com)

Introducing the OpenIddict client | Kévin Chalet's blog (kevinchalet.com)

Adding custom claims to a user during authentication with ASP.NET Core 2.0 - Joonas W's blog

aspnetcore/src/Components at main · dotnet/aspnetcore · GitHub

Outro

Claims play a good role in an application's authentication and authorization process and make the whole token process abstract for a dot net developer. Claims encapsulate user-specific information and are stored and transferred through tokens.

By understanding the tokens and claims, navigating the middleware pipeline, customizing claims, and accessing them securely, I can unlock claims-based authentication and authorization.

Shame about my research on the id_token and access_token. But maybe that is something for a later post.

Let us connect! What did you find about this post? What would like to read more about or less?

Change is inevitable! I often find myself diving into new codebases and projects, each with its own set of challenges. Previously I explored the IConfiguration of DotNet with references to Net Framework. I will explore now a scenario that I am tasked with modernizing an application's HttpClient usage.

Context

I arrive at the customer's site, ready to dive into the codebase. As I explore the application, I notice that the HttpClient usage is outdated and relies on manual and repetitive configuration. Each request is created and managed individually, with setting BaseUri, OAuth2 Headers and APIM Subscription keys and disposing of the HttpClient. That repetitiveness leads to code duplication. Manual disposing of the HTTP client can lead to performance issues.

However that the codebase is in DotNet 6, why are developers still using the same way of working as in .Net Framework?

There are numerous reasons. Some people are working with .NET Framework for a long time! While the syntax is the same, DotNet is built from the ground up. Working with the same syntax gives a false sense of familiarity.

Another possibility is that developers may not be fully aware of the benefits and advantages of using HttpClient how it is intended to use.

Code sometimes seems just to simple too, which can feel like a hassle for the untrained. For simple use cases where only a few HTTP requests are made, using HttpClient with a using statement just seems easy to do.

Migrating an entire codebase to adopt a new way of working, without coaching and guidance, can be a time-consuming and resource-intensive process. Developers might prioritize other tasks and will take shortcuts, wich will lead to technical dept in the future.

Problems with DotNet Framework HttpClient

The .Net Framework HttpClient had its problems. The HttpClient had limited control over connection management. It did not handle socket exceptions or connection failures gracefully. That leads to issues when dealing with unreliable networks or high-traffic scenarios.

It also has limited timeout-handling capabilities. I had to rely on workarounds or custom code to implement timeouts for HTTP requests.

In the old days, the HttpClient was not designed with dependency injection in mind. This made it challenging to mock or replace HttpClient instances during unit testing. It required creating a wrapper or using complex techniques to inject a mock implementation. The tightly coupled nature of HttpClient made it difficult to isolate and mock its behaviour.

The HttpClient had limited flexibility in handling request and response content. It lacked built-in support for content negotiation, deserialization, or handling different media types.

Creating a new instance of HttpClient for each request in the .NET Framework had performance implications. It resulted in the overhead: DNS resolution and TCP connections

DotNet HttpClient

The HttpClient has a lot of improvements:

• in making HTTP requests:

  • performance,

  • efficiency

  • flexibility

• with features:

  • connection pooling,

  • automatic decompression,

  • timeout handling,

  • HTTP/2 support

DotNet & Framework

In the .NET Framework, the HttpClient creates HTTP requests. Behind the scenes, HttpClient used the HttpWebRequest class to perform the actual network communication. This meant that each HttpClient instance created a new underlying HttpWebRequest.

Let us take a look at the sequence diagram:

The HttpClient has been completely reworked to address the limitations. The new HttpClient is a cross-platform implementation. It uses the new HttpMessageHandler. That HttpMessageHandler provides an efficient and performant way to make HTTP requests.

Let's take a look at the updated sequence diagram to understand the changes.

In the image above, the HttpClient does not rely on the class HttpWebRequest. I The sequencediagram shows that there is a difference between setting up the Httpclient and using the HttpClient. The HttpClient uses the HttpMessageHandler to handle the http requests. There is no more overhead of creating a new instance for each request. This results in improved performance!

DotNet HttpClient Features

Now that I have explored the changes in HttpClient, let's take a closer look at some of the new features and improvements.

The new HttpClient introduces connection pooling. This allows multiple requests to reuse the same underlying TCP connection. This significantly improves performance by reducing the overhead of establishing new connections for each request. The HttpClient handles the decompression like gzip or deflate! Besides connection pooling and auto decompression, the HTTP client provides better control over request timeouts. I can now set individual timeouts for different stages of the request. Those stages are:

• establishing a connection,

• sending the request,

• and receiving the response.

This will help my application handle timeouts and prevent bottlenecks.

To keep up with modern standards, the HttpClient supports HTTP/2. That is the latest version of the HTTP protocol. HTTP/2 offers improved performance by allowing multiple requests to be multiplexed over a single TCP connection. This results in faster web applications. Read more about it on Cloudflare.

How to use it?

• Create your own HttpClient or... ( do not please)

• Use the HttpExtension methods and thus use the IHttpClientFactory

The HttpClientFactory is a feature introduced in .NET Core 2.1 and continued in .NET 5/6. That provides a centralized and efficient way to create and manage HttpClient instances.

Microsoft does lever us official guidance on what to use. .AddHttpClient and/or the HttpClientFactory.

HttpClient instances created by IHttpClientFactory are intended to be short-lived.

• Recycling and recreating HttpMessageHandler's when their lifetime expires is essential for IHttpClientFactory to ensure the handlers react to DNS changes. HttpClient is tied to a specific handler instance upon its creation, so new HttpClient instances should be requested in a timely manner to ensure the client will get the updated handler.

• Disposing of such HttpClient instances created by the factory will not lead to socket exhaustion, as its disposal will not trigger disposal of the HttpMessageHandler. IHttpClientFactory tracks and disposes of resources used to create HttpClient instances, specifically the HttpMessageHandler instances, as soon their lifetime expires and there's no HttpClient using them anymore.

Approaches

There are multiple ways of working with the HTTP client. I made a summary from Use the IHttpClientFactory - .NET | Microsoft Learn into an overview table.

Using direct HttpClient injection involves manually configuring an instance of HttpClient with the handlers into the services. Do not use this approach.

The IHttpClientFactory provides a centralized way of managing and reusing HttpClient instances. This approach helps address issues related to long-lived HttpClient instances, including DNS changes, connection pooling, and proper resource disposal. By calling the AddHttpClient method on the service collection, there is a HttpClient is ready to be injected into the services.

This approach is suitable for smaller or simpler projects where direct injection is sufficient. To quote Microsoft:

Lifetime management of HttpClient instances created by IHttpClientFactory is completely different from instances created manually. The strategies are to use either short-lived clients created by IHttpClientFactory or long-lived clients with PooledConnectionLifetime set up. For more information, see the HttpClient lifetime management section and Guidelines for using HTTP clients.

I need to define multiple HttpClient configurations to access multiple endpoints. I have to use a key that uniquely identifies a HttpClient configuration. This is done by using AddHttpClient("key") on the service collection.

To use a specific configuration of the HttpClient, I inject the IHttpClientFactory and use the key that corresponds to that configuration. The HttpClient should be disposed of when using the IHttpClientFactory.

Instead of working with named HttpClient's creations on the IHttpClientFactory, Typed clients are a convenient and type-safe way to work with. Register it with the dependency injection container using the AddHttpClient<IServiceInterface,MyHttpClientConsumerService> method. The service is registered in the container as transient! I do not need to register it again! Typed clients make HTTP calls more structured and maintainable. It is important that the HttpClient that is not injected into a singleton service! I quote Microsoft on this one:

Typed clients are expected to be short-lived in the same sense as HttpClient instances created by IHttpClientFactory (for more information, see HttpClient lifetime management). As soon as a typed client instance is created, IHttpClientFactory has no control over it. If a typed client instance is captured in a singleton, it may prevent it from reacting to DNS changes, defeating one of the purposes of IHttpClientFactory.

When using typed clients, I have two options for creating them. The default approach uses the IHttpClientFactory behind the scenes, using .AddHttpClient<,> method. The HttpClient will be created and managed by the DI container. It simplifies the creation of HttpClient instances.

// Define the typed client interface
public interface IMyTypedClient
{
    Task<MyModel> GetData();
}
public class MyTypedClient
{
    HttpClient _httpClient...
    public MyTypedClient(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<MyModel> GetData()
    {
        return _httpClient.GetAsync(...)
    }
}
// Register the typed client with the default IHttpClientFactory
services.AddHttpClient<IMyTypedClient, MyTypedClient>();

// Inject the typed client into a class or controller
private readonly IMyTypedClient _myTypedClient;

public MyClass(IMyTypedClient myTypedClient)
{
    _myTypedClient = myTypedClient;
}

// Usage
var data = await _myTypedClient.GetData();

I also can use the Named HttpClient instances. Named clients are useful for more advanced scenarios where I need fine-grained control over HttpClient settings. However, I need to use the IHttpClientFactory to create my Named HttpClient instance.

// Register a named HttpClient instance with additional configurations
services.AddHttpClient("MyNamedClient", client =>
{
    client.BaseAddress = new Uri("https://api.example.com");
    // Other HttpClient configurations
});

// Inject the named client into a class or controller
private readonly HttpClient _myNamedClient;

public MyClass(IHttpClientFactory httpClientFactory)
{
    _myNamedClient = httpClientFactory.Create("MyNamedClient");
}

// Usage
var data = await _myNamedClient.GetAsync(".../Data");

Polly integrates!

Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, Rate-limiting and Fallback in a fluent and thread-safe manner.

Polly can be used with the IHttpClientFactory. The integration allows me to apply policies to the HttpClient instances created by the IHttpClientFactory.

services.AddHttpClient("MyClient")
    .AddTransientHttpErrorPolicy(policy => policy.RetryAsync(3))
    .AddTransientHttpErrorPolicy(policy => policy.CircuitBreakerAsync(5, TimeSpan.FromSeconds(30)));

In this example, two Polly policies are applied to the HttpClient with the name "MyClient":

• RetryAsync: Retry failed requests up to 3 times.

• CircuitBreakerAsync: Break the circuit if 5 consecutive requests fail within a 30-second window.

It is not needed to have try-catch scenarios in the code itself to react on e.g. HTTP status codes and invent the backoff interval.

Explore the Polly documentation for more advanced usage and customization options!

Remember to configure Polly policies according to your application's specific needs! There is a potential impact on the underlying services and resources being called. E.g. if a request is retried multiple times, it can result in duplicate data creation, unintended modifications, or unintended deletion of resources.

Mocking DotNet HttpClient

Because the HttpClient is just a facade, I do not mock the HttpClient, but rather the HttpMessageHandler that the HttpClient uses. By using a mocked HttpMessageHandler, I can control the behaviour of the HttpClient. This allows me to test my code without making actual network requests.

// Create a mocked HttpMessageHandler
var mockHandler = new Mock<HttpMessageHandler>();
mockHandler
    .Protected()
    .Setup<Task<HttpResponseMessage>>(
        "SendAsync",
        ItExpr.IsAny<HttpRequestMessage>(),
        ItExpr.IsAny<CancellationToken>()
    )
    .ReturnsAsync(new HttpResponseMessage(HttpStatusCode.OK));

var httpClient = new HttpClient(mockHandler.Object);

// Use the HttpClient in your unit tests
var response = await httpClient.GetAsync("https://api.example.com/users");

// Assert the expected behavior
Assert.Equal(HttpStatusCode.OK, response.StatusCode);

In the code above, I use a mocked HttpMessageHandler. Notice that I do not mock the HttpClient. The HttpClient is just a facade. The actual work happens with the HttpMessageHandlers.

When I want to test more integrated with multiple classes, I override my registrations in my service collection. This enables me to mock the HttpClient and control its behaviour in the grand scheme of things. The HttpClient's HttpMessageHandler can be replaced with a mocked version without modifying the code under test.

// Create a mocked HttpMessageHandler
var mockHandler = new Mock<HttpMessageHandler>();
mockHandler
    .Protected()
    .Setup<Task<HttpResponseMessage>>(
        "SendAsync",
        ItExpr.IsAny<HttpRequestMessage>(),
        ItExpr.IsAny<CancellationToken>()
    )
    .ReturnsAsync(new HttpResponseMessage(HttpStatusCode.OK));

// Create a new ServiceCollection
var services = new ServiceCollection();

// Register the HttpClient with the mocked HttpMessageHandler
services.AddHttpClient("MyHttpClient")
    .ConfigurePrimaryHttpMessageHandler(() => mockHandler.Object);

// Build the ServiceProvider
var serviceProvider = services.BuildServiceProvider();

// Resolve the HttpClient from the ServiceProvider
var httpClient = serviceProvider.GetRequiredService<IHttpClientFactory>()
    .CreateClient("MyHttpClient");

// Use the HttpClient in your unit tests
var response = await httpClient.GetAsync("https://api.example.com/users");

// Assert the expected behavior
Assert.Equal(HttpStatusCode.OK, response.StatusCode);

In the above example, I create a mocked HttpMessageHandler using Moq. The test is set up to return an HTTP 200 OK response for any request. I register the HttpClient with the mocked HttpMessageHandler in the ServiceCollection. I do that by using the AddHttpClient method and configure the primary HttpMessageHandler. Finally, I resolve the HttpClient from the ServiceProvider and use it in the unit test.

Outro

I did have some difficulties writing this post. There is enough documentation to find, but all the different ways of creating an HttpClient and the lifetime of the service and instances threw me off.

However, I think I managed to explain what the evolution of the HttpClient went trough and how to use the HttpClient in Dotnet 6.

In my previous blog posts, I wrote about the modularity of the configuration that Dotnet 6 offers. About sources and sections and how to register them. I mentioned how to manually build the configuration object and how to use the default sources that Dotnet 6 offers. I also mentioned security-related information: using secrets.json, AppConfiguration Service and Azure key vault.

Context

When coaching individuals, I've observed that when transitioning from the DotNet Framework to Dotnet Core, people tend to instinctively want to work by what they know. That is not strange at all. In the article Why Do Your Employees Resist New Tech? (hbr.org)" are some reasons mentioned like lack of skills, costs, complexity, and infrastructure challenges. By writing these blog posts, I want to help inspire fellow developers to embrace the new way of working and actually... show how cool is the stuff that Microsoft has realised! Microsoft also promotes some good techniques on how to create architectures and code.

I notice that there is no little awareness about using the strongly typed classes that represent (part of) sections and how to validate the settings on the startup of an application. A big difference in design is the possibility to monitor changes and ensure that the application reacts to those changes, without the need to reboot the application.

In this post, I mention how to consume configuration, and the logic behind the different ways by showing sequence diagrams made in Mermaid Live. I also show how to debug a startup error on an Azure app service when a configuration setting does not validate.

Consume IConfiguration in Program.cs

I will use this way of working when setting up my services or consuming settings in the Program.cs. However, I suggest binding sections to objects mentioned in my previous post. This ensures type safety and not working with strings.

To read from IConfiguration or ConfigurationManager I can use the GetValue method and specify the configuration key, which can be composed of multiple sections separated by colons (:). That colon will be there when there is a hierarchy defined. In the case of the configuration key --another--secret--value, I need to read another--secret--value. This is explained in the section above named "Prefix Dashes".

 _config["my:secret:value"]
 _config.GetValue<bool>("my:secret:value")
 _config["another--secret--value"]
 _config.GetValue<bool>("another--secret--value")

Another important thing to note is that when you use an IConfiguration object or custom binding to a class, the validation of settings will not be triggered. The validation specified by ValidateOnStart will be activated, as you might guess, at the start of the application. This means I cannot rely on validation because it will not be triggered before calling app.Run() , app.Start() or app.StartAsync. After all, I am using configuration to setup my application.

Consume IOptions***<> in the services

I may use the IConfiguration and ConfigurationManager objects to configure my services. The idea is that I use the IOptions<MySettings> object in my services. DotNet will make the mapping between the sources and the IOptions objects for me. This just works nicely!

When you want to cringe again, at how we need to do it in Net Framework, go read this old documentation: How to: Create Custom Configuration Sections Using ConfigurationSection | Microsoft Learn

Inject in a service IOptions<>, IOptionsMonitor<> or IOptionsSnapshot

Let me register MySettings object using the .AddOptions method.

services.AddOptions<MySettings>(Configuration.GetSection("MySettings"));

In a service or any other class where I want to use the configuration, I can inject IOptions<MySettings> in the constructor:

public class MyController : Controller
{
    private readonly IOptions<MySettings> _mySettings;

    public MyController(IOptions<MySettings> mySettings)
    {
        _mySettings = mySettings;
    }

    public IActionResult Index()
    {

        return View();
    }
}

That's it! Now I can use MySettings configuration object in my code without worrying about parsing or loading configuration data myself.

Difference between IOptions, IOptionsSnapshot and IOptionsMonitor

IOptions is the base interface for accessing configuration options and it is injected as a singleton service and cached for the lifetime of the application. It returns a single snapshot of the configured options at the time of its initialization.

using Microsoft.Extensions.Options;

public class MyService
{
    private readonly IOptions<MySettings> _options;

    public MyService(IOptions<MySettings> options)
    {
        _options = options;
    }

    public string GetOptionValue()
    {
        return _options.Value.MyOptionValue;
    }
}

IOptionsSnapshot, on the other hand, is a variation of IOptions that updates its values dynamically as it listens for changes in the underlying configuration system. This makes it more suitable for use in long-lived scopes such as request-scope services.

using Microsoft.Extensions.Options;

public class MyService
{
    private readonly IOptionsSnapshot<MySettings> _options;

    public MyService(IOptionsSnapshot<MySettings> options)
    {
        _options = options;
    }

    public string GetOptionValue()
    {
        return _options.Value.MyOptionValue;
    }
}

IOptionsMonitor is similar to IOptionsSnapshot, but with the added capability of allowing registration for change notifications. It is useful when I want to monitor changes to the configuration data and perform actions based on the changes.

using Microsoft.Extensions.Options;

public class MyService: IDisposable
{
    private readonly IOptionsMonitor<MySettings> _options;
    private readonly IDisposable _optionsChangeToken;

    public MyService(IOptionsMonitor<MySettings> options)
    {
        _options = options;
        _optionsChangeToken = _options.OnChange((newOptions) => {
            // Perform actions based on the new options
        });
    }

    public string GetOptionValue()
    {
        return _options.CurrentValue.MyOptionValue;
    }

    public void Dispose()
    {
        _optionsChangeToken.Dispose();
    }
}

Let me illustrate the above using some real use cases.

Suppose I have an ASP.NET Core application that uses IOptions to access configurations related to the database connection string, cache settings, and other application settings. Since IOptions is cached for the lifetime of the application, it is useful when I need to access these configuration settings multiple times throughout the lifetime of the application. The mermaid diagram below illustrates this.

When I have a long-lived scoped service such as a request-scope service, I would need to use IOptionsSnapshot. For instance, suppose I have a request-scope service that provides caching for API responses, and I configure the cache duration using IOptionsSnapshot. In this case, IOptionsSnapshot ensures that the cache expiration time is updated in real-time as soon as the configuration value gets updated, making it more suitable for long-lived scoped services like request-scope services. The mermaid diagram below illustrates this.

Suppose I have an application where I need to monitor changes to the configuration data and perform actions based on the changes. In this case, I would use IOptionsMonitor. For example, imagine that I have an email notification service in my application, and I have set the MySettings section in the Azure AppService AppSettings. If at some point I need to change the values of the MySettings section, it triggers change notifications, which the IOptionsMonitor listens to and subsequently performs necessary actions like reconnecting to the new server. The mermaid diagram below illustrates this and shows the difference with the IOptionSnapshot as well.

IOptions StartupValidation Checkup in Azure Portal

Remember the following line of code:

serviceCollecton.AddOptions<MyViewOnSettings>("mysettings").ValidateOnStart().ValdiateDataAnnotations();

When a service encounters validation errors during startup, it will return an HTTP 500(.xx) response. When developing locally, I can run in debug mode to identify issues. To gain insight into what's happening in the cloud, Azure provides access to the Application Event Log, where I can find startup errors caused by the validation of options.

In the Azure App Service page, select the Diagnose and solve problems tab, then click on the Diagnostic Tools option. Within the Diagnostic Tools tab, choose Application Event Logs to view the list of events.

Read more about this on the following page: How to view Azure App Service Event Logs? - Bernard Lim | Azure | .NET | SharePoint | M365 (thebernardlim.com)

Do you want more?

While creating this post, I accessed a lot of sources. A couple of posts stood out.

• Configuration in ASP.NET Core | Microsoft Learn

• Validated Strongly Typed IOptions (kaylumah.nl)

• Configuration (andrewlock.net)

• Understanding IOptions, IOptionsMonitor, and IOptionsSnapshot in .NET 7 | Code4IT

• How to view Azure App Service Event Logs? - Bernard Lim | Azure | .NET | SharePoint | M365 (thebernardlim.com)

• Options pattern in ASP.NET Core | Microsoft Learn

Outro

This concludes the theory about Configuration. I mentioned in my first post I will make a drawing. That is underway. I am trying out some doodling techniques on my tablet. I will post it, with a summary and links to all these blog posts, later this year. For now, it is time to write about different topics

Did you like this three-part series? What would you change? What information are you missing? Let us connect and engage!

Previously, in my blog post, I discussed that Dotnet 6 Configuration allowes dynamic reloading of configuration data without application restarts and supported multiple sources such as environment variables, appsettings.json, user secrets, Azure KeyVault, and command line arguments.

I emphasized the significance of proper configuration management and introduced concepts like layered configuration, sections, and different configuration sources like file-based storage, Azure Key Vault for sensitive data, and the Azure App Configuration service for centralized management.

I also mentioned the usage of the secrets.json file for sensitive values and gave a glimpse of upcoming topics such as configuration defaults, consuming configuration data, and debugging in Azure deployments.

Context

By reading my previous post on Dotnet 6 Configuration, I can now tell how to map the theory on the practical side.

This time, the context of this post is rather short. You can read more on the how and the why in my previous post. I explained the IOptions Pattern multiple times and to help myself to give others a better understanding, I write this post.

To work with this code, I strongly suggest checking out a tool called LINQPad. That is a paid tool, but it is useful to try out snippets of code, like the ones that I offer here in my posts. My go-to tool is the open-source variant of LINQPad called RoslynPad.

Let us dive into it.

Creating an IConfiguration object

Before we continue to discuss what configurations are default loaded, it is interesting to see how I create an IConfiguration object.

The code below creates an instance of the ConfigurationBuilder class.

var builder = new ConfigurationBuilder()
    .SetBasePath(Directory.GetCurrentDirectory())
    .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
    .AddJsonFile($"appsettings.{environment}.json", optional: true, reloadOnChange: true)
    .AddEnvironmentVariables()
    .AddCommandLine(args);

var configuration = builder.Build();

The ConfigurationBuilder instance loads an application configuration from various sources: appsettings.json, appsettings.{environment}.json, environment variables, and command-line arguments.

• The SetBasePath method sets the base path for configuration files to the current directory.

• The AddJsonFile method loads the appsettings.json file, with the optional flag set to true (so it will not throw an exception if the file is missing), and reloadOnChange set to true (so the configuration values will be reloaded when the file changes).

• The AddJsonFile method also loads an environment-specific appsettings.{environment}.json file, if it exists. The {environment} value is interpolated from the ASPNETCORE_ENVIRONMENT or DOTNET_ENVIRONMENT environment variable.

• The AddEnvironmentVariables method loads any configuration values that are specified as environment variables.

• The AddCommandLine method loads configuration values from command-line arguments.

It's worth noting that the code I provide, sets up configuration manually using ConfigurationBuilder. However, both HostBuilder (for console apps) and WebApplicationBuilder (for ASP.NET Core apps) automatically set up different configuration sources behind the scenes. I don't necessarily need to use ConfigurationBuilder directly in my code when using these builders.

Now we got a hands-on feeling of how I can do it manually, I am ready to explore what the HostBuilder already provides.

Default configuration sources

Only for a typical old-school Console Application, I still need to choose my configuration sources. When I build a web application or a backend service, Dotnet offers us predefined configuration sources and providers.

For a backend service: HostBuilder

The class HostBuilder automatically sets the environment to Production.This behaviour can be altered by setting the environment variable DOTNET_ENVIRONMENT. This is important to read environment-specific files e.g. appsettings.<dotnet_environment>.json.

The HostBuilder also sets the content root of the application to the base directory of the assembly.

The default configuration sources that the HostBuilder provides are in this order:

• appsettings.json,

• appsettings.{Environment}.json,

• environment variables.

For a WebApplication

Like creating a backend service, when I create an ASP.NET Core web application, the application is configured with preconfigured configuration sources.

This is achieved through the WebApplicationBuilder. That class extends the HostBuilder to provide additional web-specific features and configurations.

The WebApplicationBuilder provides a default configuration for the app in the following order, from highest to lowest priority:

• Command-line arguments,

• Non-prefixed environment variables using the Non-prefixed environment variables configuration provider.

• User secrets when the app runs in the Development environment.

• appsettings.{Aspnetcore_Environment}.json. E.g.: appsettings.Development.json.

• appsettings.json.

For Testing

Testing and files do not have a good relationship. I can work with different kinds of sources for testing purposes. There is an alternative: [the InMemoryCollection provider](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.configuration.memoryconfigurationbuilderextensions.addinmemorycollection?view=dotnet-plat-ext-7.0#microsoft-extensions-configuration-memoryconfigurationbuilderextensions-addinmemorycollection(microsoft-extensions-configuration-iconfigurationbuilder-system-collections-generic-ienumerable((system-collections-generic-keyvaluepair((system-string-system-string)))))). By reusing the configurationbuilder, I can add one more configuration source. This allows me to add or override configuration values using the: syntax. Because the highest layer always wins, I can override the configuration for my (integration) tests.

Let us explore more on how the syntax works when consuming the configuration.

Non-default configuration source

By default, the Azure KeyVault and the Azure AppConfiguration Service are not available. Microsoft has Nuget packages and tutorials on how to add those. More on that later.

Adding a Keyvault

The code shows the addition of the Azure Key vault as a source to a backend service when the DOTNET_ENVIRONMENT environment variable equals the value Production. Values specified in the appsettings.json, appsettings.Production.json, and environment variables will be overwritten by the values provided by the Azure Keyvault instance.

            Host.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((context, configBuilder) =>
            {
            if (context.HostingEnvironment.IsProduction())
                {
                    var builtConfig = configBuilder.Build();
                    var vaultName = builtConfig["KeyVault:Name"];
                    var clientId = builtConfig["KeyVault:ClientId"];
                    var clientSecret = builtConfig["KeyVault:ClientSecret"];

                    configBuilder.AddAzureKeyVault(
                        $"https://{vaultName}.vault.azure.net/",
                        clientId,
                        clientSecret);
                }
            })

Adding an App Configuration Service

In this tutorial, Microsoft will provide a comprehensive guide for implementing dynamic configuration updates in an ASP.NET Core app using the App Configuration provider library. The guide emphasizes the importance of setting up the app to respond to changes in the App Configuration store and provides step-by-step instructions to achieve this. From adding a sentinel key for configuration consistency to reloading data and implementing request-driven configuration refresh,they cover all the essential aspects.

var builder = WebApplication.CreateBuilder(args);

// Load configuration from Azure App Configuration
builder.Configuration.AddAzureAppConfiguration(options =>
{
    options.Connect(connectionString)
           .Select("WebApp:MySettings:*", LabelFilter.Null)
           .ConfigureRefresh(refreshOptions =>
                refreshOptions.Register("WebApp:AzureAppConfiguration:Sentinel", refreshAll: true));
});

var app = builder.Build();

// Use Azure App Configuration middleware for dynamic configuration refresh.
app.UseAzureAppConfiguration();

IConfiguration vs ConfigurationManager

Let me show the example with the Azure KeyVault. Before DotNet 6, when configuring the Azure Key Vault provider, it is necessary to have the URL to the key vault. That is typically something I store in the app settings. However, to use the value to connect to the key vault, I have to have access to the IConfiguration object. Note that the application is still configured, so it can be that configuration of the application is not yet complete.

I recommend watching the complete example on this link. Below I show the needed statements that illustrate the problem.

    Host.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration((context, config) =>
        {
                var builtConfig = config.Build();

                config.AddAzureKeyVault(new ... builtConfig["AzureKeyVault:Url"] ...
            }
        })

To access the setting AzureKeyVault:Url, I need to build the partial configuration result by calling IConfigurationBuilder.Build. From there, the required configuration values can be retrieved and used to add the remaining configuration sources.

When the application is finished setting up and ready to start, the framework will call the IConfigurationBuilder.Build implicitly to generate the final IConfigurationRoot and using that for the final app configuration.

However, the downside of this approach is that Build() has to be called twice - once to build the IConfigurationRoot uses the first sources and then again to build the IConfigurationRoot uses all the sources, including the Azure Key Vault source. This can result in messiness.

.Net 6 solved this by introducing the ConfigurationManager, which acts as a ConfigurationBuilder and implements the IConfiguration interface. Using the ConfigurationManager comes at a cost tough. Use it when needed, otherwise, use the built IConfiguration object in the program.cs

Andrew Look wrote a good post on this: Looking inside ConfigurationManager in .NET 6 (andrewlock.net)

Register the configuration

I will explain how to register the settings using the AddIOptions and the PostConfigure<> methods. Off course, I mention how to validate your settings using DataAnnotations.

The difference between the AddIOptions and the Configure<> method will be covered as well.

Dependency Injection

Dotnet 6 has support from the ground up to work with Dependency Inversion. To ensure that is enforced, Dotnet enforces you to work with a dependency container. I need to register all my settings and services in the IServiceCollection. When the application is run, I can inject the services that are registered into the container.

In my previous blog, I wrote that configuration is split into sections for easier management of the applications modules. Using the IServiceCollection and sections, I am inspired to group my configuration sections functional.

Below you will read how I can access those separate sections.

Segregate the configuration into sections

Below I show a configuration section called MySettings in the appsettings.json file.

{
  "MySettings": {
    "Username": "myuser",
    "Password": "mypassword",
    "MaxConnections": 10
  }
}

Create a sealed record to hold the configuration data. The record properties must match the names of the configuration keys.

public class MySettings
{
    public string Username { get; set; }
    public string Password { get; set; }
    public int MaxConnections { get; set; }
}

In the code below, I use the IConfiguration.GetSection() method to get a reference to the configuration section, and then use the Bind() method to bind the configuration data to an instance of the MySettings class:

var section = Configuration.GetSection("MySettings");
var mySettings = new MySettings();
section.Bind(mySettings);

After this code executes, the mySettings object will contain the values from the "MySettings" configuration section.

The Bind() method recursively binds nested configuration subsections to nested object classes.

To access that object in a class, I can opt to register it in the dependency container. However, I rather use the IOptions-pattern.

Configure a section with .AddOptions<>()

The .AddOptions<>() method combined with the IOptions<T> interface provides flexibility and features. AddOptions is an extension method on the IServiceCollection interface.

The IOptions-patterns allow me to configure individual options independently:

• I can register multiple options classes for the same section and combine them as needed. While I can do this, I do not have a valid use case at the moment of writing.

serviceCollecton.AddOptions<MyViewOnSettings>("mysettings");
serviceCollecton.AddOptions<AnotherView>("mysettings");

• Validation rules and defining default values for options are also a possibility. I use the validation to ensure that my application is started in a valid state on startup. I can achieve this by adding validation to my class by using the DataAnnotations namespace and adding validation attributes to the properties of my class.

serviceCollecton.AddOptions<MyViewOnSettings>("mysettings").ValidateOnStart().ValdiateDataAnnotations();

internal sealed record MyViewOnSettings
{
    [Required] 
    public string Something {get;set;}
}

Override your setting using .PostConfigure<>

• I can execute post-configuration actions on the options object. That enables me dynamic modifications or custom logic based on the configured options. I use this in my Integration Tests. I noticed that people are putting their Validations inside the PostConfigure method. That has to be from the era before the IOptions pattern existed. Use the DataAnnotations or Use Validation-method with a custom delegate handler.

public sealed class WeatherForecastServerSetupFixture : WebApplicationFactory<Program>
{
    private Func<ITestOutputHelper?> _testoutputhelper = () => null;

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureTestServices(services =>
            {

                services.PostConfigure<JwtBearerOptions>(JwtBearerDefaults.AuthenticationScheme,
                    options =>
                    {

Difference between .Configure<> and .AddOptions<>

Behind the scenes, both Configure<TOptions>() and OptionsBuilder<TOptions>.Configure() methods end up doing the same thing by registering an instance of ConfigureNamedOptions<TOptions>.

The OptionsBuilder<TOptions>.Bind() method is equivalent to the Configure<TOptions>(IConfiguration config) method. While registering multiple implementations for the same service type provides a single instance of the latest registered dependency for that type, the options framework can work with an IEnumerable<IConfigureOptions<TOptions>> to provide all registered implementations.

The Configure<> method was available in an earlier version of DotNet Core. The AddOptions<> way of working came in a later version of DotNet. The AddOptions<>-method offers additional methods using the builder pattern, like ValidateOnStart() and ValidateDataAnnotations()

Read here for more information: Question: AddOptions<T>() vs. Multiple Configure<T>(…) · Issue #514 · dotnet/extensions (github.com)

What`s next?

In one of my upcoming posts, I will discuss how to consume the registered configuration from the dependency container.

Outro

As I made progress at the beginning of my journey with .NET Core 2.1, I came to appreciate the importance of IConfigurationBuilder. Through this, I was able to learn how to register and inject configurations into my services, allowing seamless access to settings throughout my application. I naturally divided settings into sections and found it quite effortless to configure them using .AddOptions<>. Additionally, I discovered that .PostConfigure<> enables me to override settings as needed, making my application flexible and adaptable for my tests!

However, I encountered my fair share of challenges along the way. The differences between .Configure<> and .AddOptions<> took some time to understand, and I had to understand when to use the DotNet 6 ConfigurationManager to embrace the power of instant access to settings instead of usage of IConfiguration that needs to be built first...

I performed the all-important IOptions startup validation checkup, ensuring the integrity of my configurations! This checkup gave me peace of mind, protecting my application against potential misconfigurations.

In my previous blogs, you may have noticed my growing interest in security and privacy topics. Of course, there is my already existing passion for DevOps. However, in my latest blog post, I outlined how DotNet 6 offers possibilities to store sensitive data outside the committable configuration: e.g. appsettings.json. In this post, I will combine all three topics and discuss possibilities for cleaning the Git history of sensitive information.

Context

Let me summarize what I read on the blogpost of GitGuardian and the storage of credentials in the git repository:

Secrets may be stored intentionally for convenience, but they can also be hidden in text files, Slack messages, and debug application logs, or may simply be forgotten. Git's design promotes the free distribution of code and that makes it easy for secrets to be leaked.

Attackers can exploit those leaks to gain access to sensitive information. By this personal services can be targeted. Code reviews will not detect secrets buried in Git's history.

I want to give a real-world scenario where storing a token leads to data leakage, discussed in the following article: 7 Real-Life Data Breaches Caused by Insider Threats | Ekran System. Let me take the case of Slack.

In December 2022, Slack's security team detected suspicious activity on the company's GitHub account, revealing that a malicious actor had gained unauthorized access to the company's resources by stealing Slack employees' tokens.

The investigation of this cybersecurity incident showed the perpetrators stole Slack's private code repositories which often contain sensitive information. Slack did not disclose the type of information stolen or disclosed further information on who the vendor was or what services or products they provided. This decision may be due to several factors, such as preserving the integrity of ongoing investigations, preventing additional harm, or maintaining the confidentiality of affected parties. Publicly revealing certain details could inadvertently aid the perpetrators or heighten the risk of subsequent attacks.

Protect the remote GIT repository

These days a lot of tooling exists to help you protect your GIT repositories from containing sensitive data. I mention Windows Defender for Azure DevOps, GitLeaks for AWS/Azure and Advanced Security on Github.

Windows Defender for Azure DevOps

Microsoft has a product called Windows defender that offers a plugin for Azure Devops

Defender for DevOps empowers developers prioritize critical code fixes with Pull Request annotations and assign developer ownership by triggering custom workflows.

Enabling it is quite easy. Look at the following page to know more.

Configure the Microsoft Security DevOps Azure DevOps extension | Microsoft Learn

GitLeaks

I will not discuss the AWS part. I am Azure focussed, so that is what I discuss in this post.

Here you find a summary of the video

As a developer, I do not always have the rights or the power to change the environment to leverage an enterprise tool like Windows Defender.

However, there is a public repository available on GitHub called GitLeaks. Let me copy and paste their definitions:

Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.

I encourage you to read my blog post where I position SAST in the development lifecycle.

The following blog is a good read as a Developer's Guide to Using Gitleaks to Detect Hardcoded Secrets.

That blog will discuss Gitleaks. The blog explains that the tool is ISO-27001 compliant and works on public, private, remote, or local repositories.

Gitleaks has two main commands: detect and protect:

• The detect command scans the repository for potential vulnerabilities and generates a report with a list of potential vulnerabilities.

• The protect command creates a hook to prevent commits that introduce security vulnerabilities.

In a development environment

I can set up GitLeaks on my developer computer using Docker, Go or use Make to create my build. The beauty of GitLeaks is that I can set it up as a precommit-hook.

In one of my previous blog posts, I wrote about Branch Hygiene to ensure I use a good naming strategy.

After configuring my pre-commit hook, I can try to commit a client secret and the following will be the result.

➜ git commit -m "this commit contains a secret"
Detect hardcoded secrets.................................................Failed

In an Azure Pipeline

By using GitLeaks with Azure DevOps, regularly scanning code using GitLeaks and integrating it into the development pipeline can prevent code merges with potential vulnerabilities and prevent them from moving into production, protecting organizations' valuable data, and reputations.

I can use Gitleaks as a plugin on Visual Studio Marketplace. When reading that page, I noticed that the author JoostVoskuil gives credit to a colleague of mine: Jesse Houwing - Xebia | Xpirit.

Mark Patton gives a good view on how to use GitLeaks in Azure DevOps.

The post provides instructions for adding the task and configuration with the appropriate parameters, including the path to the repository, report output location, and thresholds for failing a build.

This is a good demonstration of this matter:

GitHub Advanced Security: Secret Scanning

GitHub offers what they call Advanced security. One of those features is called "Secret Scanning": GitHub scans every public repository for known types of secrets to avoid any misuse of accidentally committed secrets. Secret scanning searches through all Git history on all branches present in your GitHub repository for secrets. Any strings that match patterns provided by secret scanning partners, other service providers, or defined by organizations or users are reported as alerts in the Security tab of repositories.

The secret scanning feature offers alerts for partners that run automatically on public repositories and public npm packages to alert service providers about leaked secrets on GitHub.com. Another part of this feature is the encryption of the identified secrets using symmetric encryption during transmission and rest.

Secret scanning alerts for users are available for free on all public repositories. Organizations using GitHub Enterprise Cloud can enable secret scanning alerts for users on their private as well as internal repositories and public repositories for free with a license for GitHub Advanced Security. An alert is also sent to contributors who committed the secret if they haven't ignored the repository.

When you want to know more about GitHub AdvancedSecurity, Go visit this blogpost about Advanced Security. It is written by my Xebia | Xpirit college Rob Bos and he is known as an authority when it comes to Github.

How to manually undo my mistake

There are multiple strategies to undo a mistake after doing a git push. I can opt for removing the file history or replacing text. I listed filter-branch, filter-repo and BFG as possible options. GitHub offers a page about what to do when you committed a secret and you want to remove it.

First thing first: Retract your secret

If I have leaked a secret and pushed it to a remote repository like GitHub or Azure DevOps, it is imperative that I consider it leaked. I should revoke it immediately.

GitGuardian has a blog post on this matter. I want to give credit to Rob to bring this to my attention.

If I remove the secret from GIT's history, it may still be accessible by bad actors who may have obtained it earlier. The window of abuse should be as small as possible! I can still rewrite the history and do the cleanup, but security-wise, I need to revoke it first. GitGuardian offers a GitHub page (link) where I can a good overview of where and what I can do to revoke a secret.

Gihub offers a functionality called: "push protection". GitHub will not accept the content and will thus not leak it. This functionality will be available on Azure DevOps as well. I can enable push protection on GitHub for your public repos for free.

Remove file history

If I have accidentally committed and pushed an appsettings.json file containing a ClientSecret to a Git repository, it is essential to remove or revoke the ClientSecret immediately.

1. First, remove the ClientSecret from the appsettings.json file of the local repository.

2. Commit the changes and push the changes to the remote Git Repository

git add .
git commit -m "Removed ClientSecret from appsettings"
git push

However, the ClientSecret still exists in the repository history, and it is accessible through Git commands like git log or git checkout.

To remove the ClientSecret from the Git repository history:

1. Identify the hash of the Git commit that added the appsettings.json file with the ClientSecret. I can do this by running the git log command and finding the commit that added the file.

2. Run the following command to remove the file from the commit history:

git filter-branch --tree-filter 'rm -f appsettings.json' --prune-empty HEAD

This command will remove the file from the commit history for all branches.

1. push --force will overwrite the remote repository's history

git push --force

The command overwrites the existing commit history for that file. This can lead to loss of data and confusion for other team members.

Replace text

Luckily, there are other options available, such as replacing text. Read it on the site of GitGuardian or read the following summary

1. Download and install git-filter-repo

2. Create replacements.txt with on the left of ==> , what I want to replace and on the right side of ==> the text that I want to replace with: toreplace==>replacewidth. With a concrete example: '123abc'==>ENV[‘AUTH_TOKEN’].

    ‘eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c’==>ENV[‘AUTH_TOKEN’]
   

3. Use git filter-repo --replace-text ../replacements.txt --force to remove selected lines of code containing sensitive information

4. Force Push:

   1. Use git push --all --tags --force for remote repositories

BFG Repo Cleaner

There is an alternative for the filter-branch and filter-repo commands that seems quite popular.

The BFG is a simpler and faster alternative to git-filter-branch for cleansing unwanted data from your Git repository history, such as removing large files or sensitive information like passwords and credentials. While git-filter-branch is a powerful tool with capabilities beyond what BFG can offer. BFG excels in the tasks due to its speed, and simplicity.

The following post Removing sensitive data from your Git history with BFG - DEV Community

I will give a summary on how to clean sensitive information using the BFG.

Use --replace-text to clean strings from your repo history. Each string will be rewritten as "***REMOVED***" by default. This is a two-step process.

1. Create a file passwords.txt. Add a line for each secret that needs to be removed.

fooPassword1
barPassword2
ey...

1. Execute bfg --replace-text

Execute the command with a reference to passwords.txt and the repository in question, here called foobar.git

$ bfg --replace-text passwords.txt foobar.git

Outro

I had initially used filter-branch for my Dotnet 6 Configuration post to erase sensitive info from my history. But, the more I delved into the topic, the more I realised there's a lot more to it than meets the eye. My next move is to put my research to the test and show what commands I used by attaching some before and after screenshots to illustrate the effects.

In my previous post, I described how I regained focus on my blogging. This time, let us focus on how we can configure our settings in the DotNet "core" world.

Context

It's not uncommon to come across projects with poorly managed configurations. When I first started building .NET Framework applications, the configuration process seems easy. However, over time, the <appsettings> section became more chaotic. There is a possibility to cut this up into multiple sections, but that was writing a lot of boilerplate. Storing secrets in your configuration was not safe at all. While there is a way to encrypt your settings, I do not know a lot of people of embracing this way of working. Developing microservices and managing all settings in multiple web.config ensures losing the will to live.

The .NET Core configuration is the new cool kid around the block. It provides a more flexible, efficient, and extensible way of handling application configuration.

The new configuration system in .NET Core replaces the app.config and web.config files used in previous versions of DotNET. That makes it easier to configure applications across multiple environments and platforms.

In .NET Framework web applications, making changes to the web.config file would trigger the recycling of the application pool, which would cause all sessions to be lost and all user connections to be dropped. The .NET Core configuration has support for reloading configuration data dynamically without restarting the application, which is a huge benefit for applications running in production that need to be updated or configured on the fly.

Finally, The .NET Core Configuration Framework comes with several providers that support a wide range of sources. The system is designed to be extensible. Developers can create custom configuration providers, handlers, and parsers to support their configuration formats or sources. Because of the extensibility the framework offers, adding secure sources for the storage of sensitive data is just a breeze using the Azure KeyVault!

Despite how much better configuration management has gotten in .NET, I can't help but notice that many developers are still working with outdated and painful techniques. That's why I'm writing this blog post - to talk about the importance of and benefits of properly managing configuration settings in .NET applications. I want to share my knowledge about the capabilities and show my fellow developers colleagues around the world, that it's never too late to start doing things the right way.

In this post, I will explore the layered configuration, grouping of configuration settings and different sources.

Let's discuss some theoretical concepts.

Layered configuration

Before we try to define a configuration object in code, I want to have an understanding of how settings are defined. A common way to define my settings is to put them in the appsettings.json and use the Azure Keyvault. These days, on top of using the appsettings.json, there is an option to use the Azure App Configuration service as well.

When I have multiple sources, creating a configuration object is based on the combination of those sources. Some sources could be:

• environment variables,

• appsettings.json file

• appsettings.{env}.json file

• user secrets,

• app configuration service

• key vault,

• command line arguments

A configuration object can be built by a lot of sources. When asking for a configuration value, the system uses the newest value for the asked configuration key. First, it checks environment variables, then app settings, ..., and finally, command line arguments.

The above image comes from the site of Andrew Lock.

When a key name already exists in other sources, the value will be overridden. I cannot remove a key's value by adding another layer. However, I can override it and set it to null.

With that understanding, let's delve into the sections.

Sections

In .NET Framework, configuration data is stored in the App.config file for console/Windows applications and in the Web.config file for web applications. The configuration data is stored as XML and contains settings for the application, such as database connection strings, app settings, and more.

While App.config and Web.config worked well in the past in that period, but it had limitations to be used now.

In .NET 6, sections provide a much more flexible and organized way to work with configuration data:

• Organize configuration data into logical groups instead of grouping it by type of configuration like ConnectionStrings, AppSettings,... without having to write boilerplate code.

• Selectively override or replace specific configuration values

• Useful for large projects with different environments

• Add, remove, and modify configuration sections without affecting other sections

• Define strongly typed classes and validation with ease

Sources

Having covered the way configuration is layered and grouped, we can now examine the sources

Store settings in a file, not code!

There are a lot of different default configuration providers. The one that is used most is the JSON provider. It loads configuration settings from e.g. appsettings.json. There are possibilities to use different formats like INI or XML. There are possibilities to use different types of sources: databases and REST-APIs,... There is an option to write a custom configuration source and provider, which can be useful during migration from .NET Framework to .NET 6. Another use case that I can think of is, are needed transformations to be done before consuming the values.

Store sensitive data in a vault

Azure Key Vault is a critical component for securing sensitive data in cloud-based applications. It provides a centralized, highly secure storage environment for managing cryptographic keys, secrets, and certificates. Key Vault helps prevent unauthorized access, offers key management functionalities, and ensures compliance with regulations. Its integration with Azure services simplifies secure secret retrieval, making it an essential tool for protecting valuable information in the cloud.

Dashing

To store a setting in the key vault, use double dashes (--) before the value. This ensures that the value is treated as a single entity instead of being split into separate key-value pairs, as double dashes are used to separate these pairs in the configuration.

For example, if I had a secret that had the value my--secret--value, and I tried to store it without using double dashes, it would be split into three separate key-value pairs: my, secret, and value.

If we were to store this secret in a configuration file using double dashes, it would look like this:

{
  "my": {
    "secret": {
      "value": true
    }
  }
}

Prefix Dashes

When we add double dashes before the value --another--secret--value, it will be saved as a single value.

{
  "another--secret--value": true
}

This is important because it ensures that our key-value pairs are correctly interpreted and that the value we are trying to save is treated as a single value and not as a separate key-value pair.

Let us explore the centralization of settings of multiple services that belong logically together, to make life just a lot easier.

Store on-the-fly-needed settings using a central service!

The following video was helpful as it provided insight into the Azure App Configuration service. However, I will give a summary as well for the comfort of the reader.

Azure App Configuration service makes it possible to keep the application configurations in a single location. I will be able to store settings for different environments like development, testing, and production. I can change on the fly my configuration without redeploying it.

To top it off, the Azure App Configuration service allows applications to retrieve configuration data from Azure Key Vault. The application does not need to have any connection anymore with the Azure Key Vault.

Having the configuration read by the application is as easy as adding a package. Next is to configure an additional source to the configuration. It is possible to read configuration values from Azure App Configuration before falling back to appsettings.json.

Visit the following link: Quickstart: Create an Azure App Configuration store | Microsoft Learn.

Do not commit sensitive data: secrets.json

The secrets.json file is used to store sensitive values. Those values could be ClientSecrets, Passwords, API keys,... That kind of information should not be committed to my code repository.

I can manage the user secrets in Visual Studio by right-clicking on a project

JetBrains Rider does not offer a user interface. I need to download and install a plugin for that.

However, I like to manage it myself. The secrets.json is stored at the following location: %APPDATA%\Microsoft\UserSecrets\<user_secrets_id>\secrets.json. In the preceding file path, the <user_secrets_id> is mentioned. When configured, I can look into my .csproj file and look for the the UserSecretsId value. The following is an example:

<PropertyGroup>
  <TargetFramework>net6</TargetFramework>
  <UserSecretsId>79a3edd0-2092-40a2-a04d-dcb46d5ca9ed</UserSecretsId>
</PropertyGroup>

The content of a secrets.json looks like following json.

{
  "my:secret:value": "foobar"
}

For a web application, this will be automatically loaded and override or add the value for

{
  "my": {
    "secret": {
      "value": true
    }
  }
}

What`s next?

In one of my upcoming posts, I will first do a small recap with a drawing. Then I discuss the configuration defaults the HostBuilder class and ASP.NET Core provide. I will explain how to consume configuration data in Program.cs using the IConfiguration interface and how it differs from ConfigurationManager. Plus, I will explore sections in my configuration using the .AddOptions<> method. Furthermore, I will demystify the difference between .Configure<> and .AddOptions<> and the difference between IOptions<>, IOptionsMonitor<>, or IOptionsSnapshot<>. Finally, I will give an insight moment on how I debug the startup validation of my configuration when deployed my service is deployed in Azure.

Outro

This is a new format that I am playing with. I am in doubt if I want to have multiple posts that tackle a section or use the format that I am used to when writing an article for Xprt Magazine. I aim for a reading time of between four and thirteen minutes.

I am always happy to get feedback. Let`s engage!

In my previous post, I shared my journey towards obtaining the AZ-204 Azure Developer Associate certification. I discussed the challenges I faced and shared my learning strategies.

Context

After obtaining my AZ-204 certification, I decided to take a week-long break from blogging. Discovering the right learning strategy had been difficult for me, as trying to learn facts without what seems any context or doubt about what the questions will be like, made me anxious.

Taking a break meant interrupting a habit, and not resuming it after a short pause could lead to the formation of a new habit. I noticed that after a week of not blogging, I made another decision to stop blogging for another week. That process was repeated four times.

I felt conflicted. I had promised myself to continue blogging this year, but the process of learning and pushing myself towards the certification had reignited an old struggle. Everything seemed dull, and getting back into blogging felt challenging. My motivation had disappeared.

Fortunately, I had a scheduled appointment with my coach. During that session, we used Transactional Analysis to address the disruption I was experiencing. We found that the active state was not the adult ego state, but rather the rebellious child's ego state.

Identifying the root cause is one thing, but altering one's state of mind requires more than just realization. I devised plans to ensure that I would study for my next certification using the approach I described in my previous post. I committed to refining the learning process I had employed and vowed to continue refining it. After making this promise, my state of mind shifted back to the adult ego state.

Transactional Analysis

Let's start by taking a closer look at the concept of Transactional Analysis. According to Eric Berne, the founder of Transactional Analysis, we all have a Child, Parent, and Adult ego state.

The Child's ego state is emotional, impulsive, and creative. The Parent ego state is nurturing, caring, and controlling. The Adult ego state is rational, analytical, and problem-solving. These ego states work together in different situations.

For me, Transactional Analysis was a useful tool to help me connect with my rebellious child. It helped me explore what the problem was.

When I neglect my emotions and blog only on pure willpower, then it will eventually hurt me. I will get a bad feeling. When I do that for too long, I will associate blogging with a bad feeling.

I encourage you to read the following article, written on the site of Forbes about how to use Transactional Analysis to discover your authentic self.

Atomic Habits

At the start of the year 2023, I started setting small, achievable goals for my writing and made sure to stick to them consistently. Stolen from my habit of making software design decisions, is making the implicit explicit. By making them explicit, telling my intentions is helping me enforce my new habit.

James Clear, the author of Atomic Habits, wrote: "When you do something once, it's not a habit. When you miss it twice, it's not a habit either. But when you do something twice in a row easily, you start to create a new habit".

However, it is still up to me, to protect my state of mind and health, and to ensure I have the will and energy to follow through with creating new habits.

Self-care or self-sabotage?

When we are under stress, it is common to enter the critical child state, where we view the world from a negative and self-punishing perspective. In this state, it is easy to cancel and not pursue my goals. Is that self-care then? In that emotional state, my main goal is to feel safe and secure and to ensure to keep away from the chaos.

Self-care is essential during stressful times. Self-care does not mean stopping. It's about finding a balance between taking care of yourself and taking action to achieve your goals.

Self-care can include a range of activities and practices that help you feel renewed, refreshed, and restored. Exercise, healthy eating, and rest, as well as emotional practices like mindfulness, meditation, and seeking support, are things that are working for me best.

I am glad that I have a new tool to practise to support my further endeavours.

Outro

Transactional Analysis is an important tool that I highly recommend mastering. I find it essential for staying true to myself, my vision, and my values. This has been a significant discovery and can be quite intimidating: making decisions based not only on what I feel and think is best but most importantly, aligning them with my mission and values: I want to stay focused on the road towards fulfilment.

Learning and growing is a journey, and it's okay to go at my own pace and make mistakes along the way. I will keep searching for the path that works best for me.

At the beginning of the year, I challenged myself to pass a tech exam called AZ-204. In this blog post, I'll talk about my experience that leads up to taking the exam and share my process of studying for it.

Context

As a consultant, obtaining certifications demonstrates to potential customers that I have gone beyond the minimum expectations, giving them more confidence when working with me. However, it's crucial to remember that while certification may show a certain level of knowledge, it doesn't necessarily reflect a complete understanding of the subject.

After I began working for Xebia | Xpirit, one of my tasks was to earn Azure certifications such as the AZ-900 (Azure Fundamentals) and AZ-204 (Azure Developer Associate). I decided to start on the AZ-204 because I was advised to wait on the AZ-900 until Microsoft offered a free course with a seat for taking the exam. I eventually attended one of these free sessions. Those sessions are called Virtual Training Days. During that session the moderator informed us that there was no free way to attempt the certification.

With this in mind, I was curious about the process of earning certifications. After all, it's Microsoft - they must have excellent learning material and documentation of Microsoft, as well as learning paths on Azure and Pluralsight. Pluralsight even provides a practice exam. I thought I was prepared... so, how challenging could earning a certification be?

Move heaven and earth to get myself started

Getting started with something can be quite difficult. I know what I want to achieve. Most of the time I have that epic vision in my head that represents the result. If I want to do something new, it requires a lot of will- and mental power to get started on that. Once I am invested, I can dial back and work on a path towards that goal. The next difficult step is being consistent towards achieving that goal.

I make it easier to achieve my goals by breaking them down into smaller, achievable steps that allow me to take gradual progress towards the final objective. This may lead me to work more than just focusing on the end goal alone. However, if getting started seems like a challenge, I take the initial smaller steps to invest myself in the larger goal.

Getting a certificate is challenging.

When I began studying for the AZ-204 exam, I assumed it would be similar to studying in high school or university. To begin, I focused on theoretical knowledge and started with a Pluralsight video, followed by reading and note-taking on the Microsoft learning path. To evaluate my understanding, I took a free test exam provided by Cybervista and Pluralsight.

Unfortunately, I scored only 22%, indicating my strategy had failed. I felt lost and disillusioned, but I realized I had gained more knowledge on Azure and its ecosystem. Despite my efforts, I still had a long way to go and struggled to pass the first module after investing 16 hours in studying.

There are 4 steps to master something. Below I have an image, borrowed from a website called "ZenTools" that shows the different kinds of learning stages.

According to Wikipedia, the process of learning a new skill can be divided into four stages. Let me assess my situation using this learning model.

• Unconscious incompetence: An individual may not understand or know how to do something and may not recognize their deficit.

  • I was not at this stage.

• Conscious incompetence: That individual becomes conscious of their incompetence and recognizes the value of the new skill.

  • According to the test, I was definitely in that stage. I became consicious of my flaws.

• In the conscious competence stage, the individual comes to understand how to perform the skill but still requires concentration to execute.

  • According to that first test, I had no competence. However, I work daily with a subset of Azure Services. It is difficult for me to admit that according to this test, then, I was not consciously competent.

• In the unconscious competence stage, the skill has been practised so much that it can be performed easily and second nature. The individual may even be able to teach it to others.

  • Just not applicable at that point.

One of the first steps to learning something is: Know what to do. The test on CyberVista confirmed that I did not know what to do. At this point, I reached out to Bas van de Sande. I think he knows a thing or two about getting a certification... He recommend that I learned each day for one hour. For sources, he recommended me ExamTopics and the learning path that Microsoft provided.

ExamTopics offers me to learn question by question. When learning a subject that is represented by a question, I should use the discussion that is there in the answer section. When the discussion is not clear on a subject, I should visit the given links for that question. By reading the given links, I should know the answer.

Seek out a learning strategy.

I wanted that achievement as soon as possible. However, I want to brute force my way through. I did not want to wait and have a long process. I wanted instant gratification. I threw out the advice about studying each day for one hour. I did follow up on the advice about studying with the questions provided by ExamTopics.

I have to say I had my brighter moments because I felt horrible. One of the upsides that ExamTopics has, is that I can extract a PDF of the questions provided. So I started the export to PDF. That was over 800 pages with discussions included.

Starting to go through the questions, I noticed that there is a lot of doubt. There is a provided answer and I have the community's answer as well! With that, I have a lot of discussions and source references. This was hell for me. I "just" want to learn and have clear answers, put on a platter. After some retries and brute forcing, I noticed that it gave me a lot of negative energy.

After some extra brute forcing myself into this, I decided to do it differently. I will learn each day for one hour as Bas suggested. Kristof Van Hees has a method that is similar to that of Bas. He recommends me Whizlabs. Whizlabs offers a lot of questions. Whizlabs does not work with PDF but offers an interactive way of working. I got direct feedback if I get it right or wrong. There is a reason why the answer I picked was good or wrong. For each question, Whizlabs provides a source as well. In this phase, I just tried out the questions and my focus was on getting good answers. I verified why I picked the good answer. This way of learning was a better fit for me, but I did not get a feeling I deepened my knowledge of the subjects. I redid the questionnaires multiple times.

I talked to my coach Kristien about this topic. I wanted to postpone my deadline, however, she urged me to try the exam. When I take an exam, I can have a free do-over. What have I got to lose then? At this point, I have accumulated 60 hours worth of studying.

Let`s go! Setting a deadline...

With a lot of self-doubts, I bought a seat to take an exam at home. Microsoft offered me to buy access to the MeasureUp test exam as well. I can recommend everybody to do this. Those test exams are more difficult than the ones on Whizlabs. I just took the test exam and score 55%. To pass the Microsoft exams I need 70%. MeasureUp works in the same way as Whizlabs. The difference is that Whizlabs offers more videos and education around the topic. MeasureUp explains why an answer is good and why it is bad. I visited the links they provide for each answer. I started doing that because I did not understand some answers and felt insecure about some topics. Then it hits me, I think I figured out how to study for the exam.

How to study for the exam.

Remember Bas, that good guy that told me how to study for the exam? Well, it was only ExamTopics that was not working for me. Understand that I will go through a process that is different from the one I know. Expect to fail... hard.

After I understand what kind of sources I got at my disposal, I figured out the following process:

• Ensure that I go through the learning path that Microsoft presents. This is important to understand the concepts and the dependency between all the subjects presented. It is very important for me, to get a good understanding of how all those concepts are linked. Without that, question-based studying offers me no reference points. During the process of finding out what works for me to study, I thought this was not needed. But looking back, I can say that it is not wasted time.

• Go watch the videos as a summary.

• Go and find my source for questions. I will use Whizlabs again.

• For each question:

  • Understand the question, do a guess, understand the answers, and visit the source.

  • Read the whole page and then go to the next question.

• Do this every day for several questions if I want to quantify my progress or work for some time. I will start with ten questions or one hour. I can adjust accordingly after witnessing my progression.

• Once I get 70% on the test exams, order my seat to take the exam use MeasureUp and take the exam with confidence. Remember, I got a free do-over.

Summary certification preparation

Below I have made a list of sources:

• Reddit: r/AzureCertificiaton: az 204: I used this source to understand that I am not alone in this.

• Thomas Maurer (MVP): AZ-204 Study Guide: Developing Solutions for Microsoft Azure: A recommended source on certifications. Read it before I start. I wish I knew this before I got my certification.

• Microsoft Learn: Exam AZ-204: Developing Solutions for Microsoft Azure - Certifications. useful to buy my seat for the exam and to buy my access to the MeasureUp test exams.

• Exampro: Az 204 full video: A good explanation of the concepts given with taken the examination in mind.

• Pluralsight: Developing Solutions for Microsoft Azure (AZ-204) Learning Path: The videos are good, structured and to the point. However, it feels like there is missing some information at the time of writing. The access to CyberVista is useful as well for having another set of questions to learn from.

Show off your hard work

I recommend adding your certifications to your LinkedIn profile. It's a great way to showcase your skills and achievements to potential employers or potential customers.

Don't be afraid to let others know that you're certified and proud of your accomplishments.

Sometimes people might feel hesitant to share their certifications. Some people fear it might come across as bragging. Others might feel like they don't want to be perceived as arrogant.

However, as long as you are humble and gracious in the way you share your achievements, there's nothing wrong with letting others know about your certifications.

If you want to check out how that is visualised, check mine out here: Microsoft Certified: Azure Developer Associate - Credly

Outro

Initially, I had a lot of different points of view when writing this post. Should I just write about how to achieve the Az-204? Should I write about my journey in achieving that certificate? Should I write about the aftereffects of obtaining that certificate? I ended up with a mixture of telling where I failed, what my pitfalls were and how I am going to study for other certifications. I can only hope this can help others as well.

At Techorama 2023, I was surprised when some people asked me when I was going to write a new blog post. I like that people are holding me accountable. At the start of my blogging journey, I set a goal for myself, which was to have a weekly blog post in 2023 to build out my authority. I am humbled by this and feel inspired to write more.

In my next post, I will talk about why I stopped blogging for a bit and what helped me regain focus again.

Furthermore, I am still on the path of protecting my privacy. I listed what I want to blog about and I intend to keep that promise.

In my previous blog post, I wrote about JSON RCE attacks. The possibility of such an attack exists, was brought to my attention in a report by CheckMarx. I heard about SAST scans from the product team that I worked with. Our team was also briefed about application security by Wesley Cabus on a knowledge-sharing moment at a Xebia | Xpirit Tuesday.

Context

I am trying to gain a better understanding of how I can code with a security mindset. Although I have heard of Checkmarx, I did not got any insight into the process how security reports came to be. I still have many questions about the process. For example, why does it take two days before results are available? Why is this information only available at the end of the process? Additionally, I am confused as to why security is only available at the end of the development lifecycle. What about the packages that are used in the project? In a previous post, I wrote about the RCE (Remote Code Execution) that could be done with version 11 of Newtonsoft, but in the project, versions 12 and 13 were used. However, the SAST still flagged it.

Recently, I came across a company called Contrast Security on LinkedIn. I was able to schedule a meeting with them. During this meeting, I received an introduction to application security. Next followed was a demo of their product Contrast Assess. They stressed that they offered a community edition. The community edition is free to play with. I want to stress that I am not affiliated, but I cannot hide that I am an enthusiast. I still need to play with the community edition.

Before I can show off how their community edition works, I believe it is important to understand application security. As developers, it is important to know what to use and when.

Contrast Security mentioned that other colleagues of mine at Xebia, have taken interest in their products: Carlo Klerk and Maarten Plat from the Xebia Security team. Funny how small the world is. Xebia is off course a big company, it is hard to know all my colleagues.

Maarten Plat is busy with a 5-part blog series about this subject. Part one tackles the subject SAST. Part two tackles the part DAST. Other blog posts will tackle IAST and SCA. The last part will be about integrating those tools into a Continuous Integration (CI) pipeline. Maarten also made a tutorial about one of the products of Contrast Security, so be sure to follow him! Let us read below what all those abbreviations mean...

Application Security

It is crucial to understand what tools are used during the development and product lifecycle. Writing secure code is essential. When you add a package with vulnerabilities or use an insecure way of working, you want to be notified as soon as possible. The same mindset applies to writing unit tests, automate your deployment to allow (automated) integration testing. This allows a better understanding of business, infrastructure and security requirements.

At the beginning of a lifecycle

Even before you have runnable code, your work can already be insecure. The longer you wait to adjust this, the harder it is going to be. Read more about Mitigating technical debt with developer-driven security (securecodewarrior.com)

On the website of contrast security, you can learn how to code securely.

Static Code Scanning

Static Application Security Testing (SAST) is a process that analyzes the source code to find security vulnerabilities. Some popular SAST tools are Checkmarx, SonarQube, and Fortify. SAST is very useful at the beginning of a greenfield development project. There are no real deliverables yet. It can take a lot of time to scan the entire repository.

However, there seem to be SAST tools out there that can also integrate into the CI pipeline. The downside of using the SAST tool is that you as a developer need to investigate false positives and negative negatives.

A great comparison of SAST tools is defined in the following blog post. There you can read that an effective SAST tool should support the shift-left philosophy. That means it allows security testing to be done earlier in the software development lifecycle. It should scan entire repositories, integrate seamlessly with the CI/CD pipeline, and offer fast scanning speeds. A SAST tool should minimize false positives and promote developer productivity by providing useful suggestions and resources for code fixes and library updates. The OWASP Foundation also states this. They also specify a list of vendors of SAST tools...

As a developer, I want to have great feedback with not much of false positives or negative negatives. My experience is that CheckMarx will choke itself when a developer writes loosely coupled code, using dependency injection. Because my colleague did not write its code loosely, he got a bunch of reported problems and I did not. However, there are cases mentioned in his report that I should apply as well.

As mentioned before, these tools will scan your repositories and make an in-memory model of your code. This way, they can try to see where possible problems exist. If you add code that is not used, the tool will know that and give you remarks on it. Even if that codepath is not used. But we all delete our dead code, right?

Software Composition Analysis

Software Composition Analysis (SCA) is done by tools that manage the packages you use in your application. You want to make sure you use safe packages. JFrog, Mend, Snyk, Contrast SCA, and Dependency-Check are some tools that are used for that.

JFrog has a good blog post that tells you about the dangers of using third-party packages. I created a summary of how they suggest you can protect yourself from an attack like that. JFrog recommends:

• establishing a vetting process for your supply chain,

• shift security left in the SDLC,

• verify binary code integrity,

• require an SBOM for all software,

• protect your SDLC with multi-factor authentication,

• avoid dependency confusion

• avoid typosquatting attacks,

• and use version pinning for packages.

A lot of these recommendations need an investigation that goes beyond the scope of this blog post... I linked corresponding definitions and other blog posts that are worth reading.

Below you find an example of the JFrog Artifactory.

I loved the way that Contrast Security presented their SCA tool to me: Contrast SCA | Software Supply Chain Security and Risk Management | Contrast Security. Contrast Security does not only list the packages with known vulnerabilities but also identifies what code the developer uses from the package. E.g. when there is a vulnerable unused package in your repository, Contrast shows you are not using that package, thus you should not care about it. Another use case would be identifying where a vulnerable package is used in the organisation: remember Log4J, a lot of companies needed to identify manually if they were using that vulnerable package.

During the cycle of development and testing

Interactive Application Security Testing (IAST) is more dynamic and interactive than SAST. From what I understand is that IAST is useful for integration tests or local tests where you get direct feedback.

When I did my research for this blog post, I noticed that Contrast Security has a vision of what to use in the CI/CD pipeline. In summary, they find that traditional scanning tools are slow, compute-intensive, and expensive, hindering development pipelines and CI/CD processes. Their product uses instrumentation technology to attach to an application and observe how it handles requests during runtime. Instrumentation is also used by e.g. New Relic and Azure Application Insights.

Visit the blog site mytechramblings. There you find a real-life example of how to instrument code. Explaining instrumentation is not part of the scope of this blog post.

In the images below, you can find an example request, where the email address kristof@xebia.com is not properly handled by the application.

Their tool observes that

• there isn’t sufficient validation and sanitization of the user input in this request. That is a confirmation that there is an SQL injection flaw.

• the unsanitized input, which is the email address, ends up being used directly in the SQL Query. That is displayed in red in the image below.

Contrast tries to help the developer by giving a similar overview in the form of a stack trace. That overview shows the problem, where to fix that problem and gives guidance on how to remediate it.

The following video that I found gives a demo about Contrast Security. However, that one is 4 years old. Although the UI is outdated, you can get a good grasp on how such a tool can work.

When you have a deliverable, before production

Dynamic Application Security Testing (DAST) is a process that analyzes the application in its running state to find security vulnerabilities. Examples of DAST tools are OWASP ZAP, Burp Suite, and Acunetix. I already downloaded YSOSerial.Net for my previous blog post. However, when I cloned that repository, compiled that project and ran that, Windows Defender did not let me start that executable... You can find a great example of the usage of YSOSerial.Net in the following video.

Another popular tool is Burp Suite. I recommend watching the next video on it.

DAST is useful to test your application before you go into production. It is more difficult to find and understand errors. Unlike SAST, DAST is using your application as a black box. DAST does not know a thing about the application under test. Such a tool allows you to see if any known attacks are going to work on your application. My blog post about JSON RCE Attack was a way of DAST.

After deployment

Also after deployment, there is a need for protection. I will discuss RASP (Runtime Application Self-Protection) and WAF (Web Application Firewalls).

Runtime Application Self-Protection (RASP) tools help you detect errors in real-time. Some popular RASP tools are Sqreen, Aqua, and Wallarm. Contrast Security also has a tool that does this.

Web Application Firewalls or WAFs can be configured with specific rules to identify and block known attack patterns such as SQL injection and cross-site scripting (XSS) based on signatures.

The difference with Runtime Application Self-Protection, or RASP, is a security technology deployed within the application itself. RASP provides real-time protection against attacks by monitoring the application's behaviour. It responds to threats as they arise. RASP is also capable of detecting and blocking attacks that were not previously known by looking at the behaviour. E.g. When a user tries to use a SQL injection attack, the tool will be able to detect that and throws an exception that results in an Internal Server Error. To see it in action, Contrast has a video to show how their product defends against the Log4J exploit.

Outro

I did my best to not let this post be a Contrast Security advertisement. I want to give them credit because they did take the time to explain to me the basics of application security. With that demo and the information is given, I did my research and learned a lot. I hope I can sparkle your interest in Application Security as well, so we can all code with a security mindset.

I want to test out the community edition that Contrast Security offers. I will share my experiences as well. That can be something to play with on an innovation day that is organized at Xpirit | Xebia. I do not know at this point where to find the time... But I promise I will add this to my to-do list.

In the previous blog post in this series, I discuss the evolution of email identities. I state the history of using creative and unique nicknames in the early days of the internet to the increasing need for real names to establish credibility and trust. I explored various solutions to establish trust in email communication: SPF, DKIM, DMARC, and sub-addressing.

In this series, I am hoping to regain control of my data. I hop into the world of unique credentials for each website. I will use email forwarding service SimpleLogin.io to protect privacy and reduce spam.

Context

I'm writing this because I'm fed up with spam and the constant mistrust in companies' security measures. Let me give an example of why it is important to protect your data and even using a unique login is not a luxury anymore...

OpenAI announced that ChatGPT has been breached and data is leaked. The vulnerability is known as CVE-2023-28859. In a later blog post, I will write about how this could potentially be solved. I appreciate how OpenAI addresses this issue. They demonstrate a high level of transparency regarding the events that transpired, the information that was exposed, and their efforts to resolve the matter. As time progresses, it will become evident whether this was simply an unfortunate incident or if their organizational structure is conducive to managing such unforeseen challenges.

I want to stress that as long as humans are involved, things will go wrong eventually. We need to find a decent balance between security, privacy and convenience.

Security is difficult. Look at the US Department of Defense... Documents about e.g. the war in Ukraine are found on social media. The American Congress seems pretty mad about it...

Unscrambling the Email Privacy Process

Let me explain how the system works by breaking it down into smaller, manageable pieces.

Gather all the eggs

What simple login does, is create relationships between an alias and your email inbox. SimpleLogin.io is not a place to read your mail, nor is it an SMTP server. SimpleLogin.io uses the word 'mailbox' which I found confusing in the beginning. I expected to read my emails in a 'mailbox'. However, it is just a concept to let the user know to what mailbox the alias will be linked. A mailbox is e.g. john.doe@gmail.com.

As a user, you can have multiple mailboxes. When you first signup, a mailbox will be created. That will also be your default mailbox. I do not have multiple ones at the time of writing.

When you want to use aliases on an e.g. commercial website, you can create subdomains for your alias. It can look like 'commercialwebsite.1234@johndoe.simplelogin.com'. The part 'commercialwebsite.1234' can be whatever you want it can be!

The use of a subdomain is essential because it allows the entire system to identify the default mailbox designated to receive emails. However, I opted to have my custom domain name. To get started, you do not need to spend money to get started with SimpleLogin.io.

Receiving Emails in Your Basket

Once you have your forwarding email setup you can provide any alias email directly to the commercial website. The alias does not need to exist. SimpleLogin will create one, link it to you default mailbox and will forward you the mailbox. It replaces the sender with the alias.

Bunny-Approved Email Deliveries

I had difficulties understanding how to send an email using the alias from SimpleLogin.

However, it is really that simple. Before you can send an email, create a contact on SimpleLogin. Navigate to SimpleLogin website.

Press on Create Custom Alias

I create an alias with the name: test.perjurer933@simplelogin.com

Now you need to define where that SimpleLogin needs to forward the email. Click on contacts and fill in the name of your contact: info@commercialcompany.com.

To get an overview, this is what you need to do.

In your email, you can now send a mail to: test.perjurer933.simplelogin.com. Everything will be done for you.

Hunting for Missing Functionality

The current setup lacks an SMTP server. I just want to set up a mail server that does the SimpleLogin magic instead of working with alias in my to/from. That would be an error-proof experience. Using rules to autogenerate alias and throwing errors when the alias is not defined. This would stop human error of forgetting to create an alias if needed.

It is on their roadmap. If you cannot wait, somebody already created a SimpleLogin SMTP relay server. However, do you really want to go down that road?

My Egg-sperience Implementing It

Below I will go more in-depth on how I implemented SimpleLogin.

Prepping the System

I set this up using an existing domain and a subdomain. I needed to make the following steps:

1. Domain Ownership Verification: Prove you own the domain by adding a special code (TXT record) given to you.

2. MX Record: Add two mailbox (MX) records to help emails go to the right place.

3. SPF (Optional): Add a special note (TXT record) to let others know your emails are real and not spam.

4. DKIM (Optional): Add some name labels (CNAME records) to make your emails more secure and trusted.

5. DMARC (Optional): Add a rule (TXT record) to tell others what to do if your email doesn't pass security checks.

When you are not familiar with DNS settings, this can overwhelm you. I love the functionality that SimpleLogin offers to easily verify each step that you undertake.

I do not own johndoe.com, so I cannot verify this step.

Custom Domain Configuration

Configure the DNS records and check out the SimpleLogin page for a smooth experience.

Once you follow the Custom Domain configuration, your dns records looks like the following.

Bitwarden Integration

I'll show you how to integrate Bitwarden.

It was difficult to find the functionality of API Key's. However, a simple Google search lead me to the forum of SimpleLogin.

Defining a naming strategy

I already discussed this in a previous blog post. But it is important to have a naming strategy. A consistent naming strategy helps you easily identify the purpose of each alias and the service it is associated with. This makes it easier to manage your aliases and keep your inbox organized. A structured naming strategy makes it easier to set up filters and rules in your email client. That allows you to automatically sort incoming emails based on their alias.

Here are some naming strategies:

1. Name + Random Number: use your name followed by a random number, e.g., john.12345@domain.com.

2. Name + Date: Combine your name with the current date, e.g., john.20230409@domain.com.

3. Name + Service: Use your name and the name of the service the alias is for, e.g., john.netflix@domain.com.

4. Random Characters: Create an alias using completely random characters, e.g., x4tYz1@domain.com.

5. Acronyms + Numbers: Use acronyms or abbreviations related to the service or your interests, followed by numbers, e.g., moviebuff2023@domain.com.

6. Name + Purpose: Combine your name with the purpose of the alias, e.g., john.shopping@domain.com.

I am opting for the way that Bitwarden suggests a name. You can ask Bitwarden to generate an alias and it will generate the alias for you on the site SimpleLogin

When the generation occurs, I added the name of the commercial website in the prefix as well.

In this case, that would be commercialwebsite.scramblermurmering087@domain.com

Catch-all Strategy Conundrum

Is it wise to enable catch-all functionality indefinitely? What about the potential for recurring spam and automatic alias creation? Wikipedia is quite sure that this can be abused.

SimpleLogin provides an option to turn off the catch-all functionality. It then works with a regex rule system. I will quote them.

For a greater control than a simple catch-all, you can define a set of rules to auto create aliases. A rule is based on a regular expression (regex): if an alias fully matches the expression, it'll be automatically created.

The following image is a print screen of the existing system. You can define a regex prefix rule. That regex rule is coupled with a mailbox. There is a debug zone as well.

Into the Easter Bunny's Burrow

It is difficult to work with structure. So sometimes... you just need to do. The more you go down the burrow, the more knowledge you gather.

Updating Egg-isting Accounts

With the existing setup, I can already start updating my emails. I do not need to specify naming strategies. I want to get a feeling of how this work in practice. I need to update my emails. The problem is, this can take a while. So where do I start?

I have a premium Bitwarden account. That allows me to get reporting on what login is leaked and what passwords have leaked. There is an entire post about it. I will not repeat it here. That helped me to get started with those accounts first.

Receiving an Email

Discover how an alias email is forwarded to your mailbox. The following is a censored and anonymised extract of an email header.

Delivered-To: john.doe@gmail.com
Received: Sat, 8 Apr 2023 22:48:25 -0700 (PDT)
...
header.from=simplelogin.co
Return-Path: <s1.lmycyibrge3abcdefheydglbagy3dombwhboq.zos4xr6@simplelogin.co>
Received: from mail-2061.simplelogin.co (mail-2061.simplelogin.co. [***])
...
        for <john.doe@gmail.com>
...
Received-SPF: pass (google.com: domain of s1.lmycyibrge3abcdefheydglbagy3dombwhboq.zos4xr6@simplelogin.co designates **** as permitted sender) client-ip=***;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@simplelogin.co header.s=dkim header.b=XronwIgb;
       spf=pass***
       dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=simplelogin.co
DKIM-Signature: ***;
Subject: Waiting on confirmation
X-SimpleLogin-Type: Forward
X-SimpleLogin-Envelope-To: commercialwebsite.1234.dummy007@simplelogin.com
From: "CommercialWebsite - info at commercialwebsite.com" <info_at_commercialwebsite_com_ypnadtx@simplelogin.co>
To: commercialwebsite.1234.dummy007@simplelogin.com
...

It will tell you that the email journey starts from the Commercial Website, passes through SimpleLogin, and finally is delivered to John Doe's Gmail account. Security checks (DKIM, SPF, DMARC) are performed by SimpleLogin and Gmail during the process.

Sending an email

Let us forget that we received the email from the commercial website. We do not have any aliases yet defined for this website. I'll explain how I initiated an email and how the reply system works, using a SimpleLogin example.

In this example, I wanted my account deleted. I could not initiate the delete sequence on the website. I needed to send an email to that company. I did not want that the company knows what my original email address is.

I went over to my simplelogin.com account. (note that there are browser extensions as well).

I pressed the New Custom Alias button.

First, create an alias prefix. I would use commercialwebsite.1234.override311@aleeas.com

You select your mailbox, in this story, it would be john.doe@gmail.com.

After I send the email, I can look at the headers. This is what it looks like.

Date: Sat, 8 Apr 2023 09:55:42 +0200
Subject: Delete my account
From: John Doe <john.doe@gmail.com>
To: "Commercial Company | info at commercialcompany.com" <info_at_commercialcompany_com_yblraaaaa@simplelogin.com>

I got a reply and that looks the same as the receiving email. Let us take a look.

Delivered-To: john.doe@gmail.com
ARC-Authentication-Results: i=1; mx.google.com;
       dkim=pass header.i=@simplelogin.co header.s=dkim header.b=Qay02Ky8;
       spf=pass 
       dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=simplelogin.co
Return-Path: <s1.lmycyibraalbagy3dkojshboq.mtf6b4aaaneq@simplelogin.co>
Received: from mail-200*.simplelogin.co (mail-*.simplelogin.co. [*.*.*.*])
Received-SPF: pass;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@simplelogin.co header.s=dkim header.b=Qay**Ky8;
       spf=pass 
       dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=simplelogin.co
Subject: Re: Delete my account
Date: Sat, 08 Apr 2023 10:48:09 +0000 (UTC)
X-SimpleLogin-Type: Forward
X-SimpleLogin-EmailLog-ID: ***
X-SimpleLogin-Envelope-To: info_at_commercialcompany_com_yblraaaaa@simplelogin.com 
From: "Company Website - info at companywebsite.com" <info_at_companywebsite_com_yblrlaaa@simplelogin.co>
To: John D <info_at_commercialcompany_com_yblraaaaa@simplelogin.com>
X-SimpleLogin-Want-Signing: yes

You can see that sending and receiving an email works flawlessly.

Undiscussed functionalities.

There is so much more to investigate. I will list the functionalities here for your convenience. I was missing an easy overview, so let me give that to you.

1. Secure your account with Two-Factor Authentication (2FA) and FIDO-supported security keys.

2. Receive updates on new features via newsletters.

3. Use custom domains and suffix generators for your aliases.

4. Transform sender email addresses into different formats to protect your privacy.

5. Enable reverse-alias replacement to hide your contact email when replying.

6. Choose sender address inclusion options for reverse-aliases.

7. Automatically expand alias info and include website addresses in aliases created via browser extensions.

8. Set one-click unsubscribe preferences, quarantine and bounce management.

9. Decide how disabled aliases and blocked contacts should be handled.

10. Include original sender information in email headers.

11. Import and export aliases from other platforms or in CSV format.

12. Request a copy of your data according to GDPR regulations.

13. Delete your account if you no longer want to use the service.

Outro

I want to reflect on my experiences and the challenges faced. Some websites do not offer account deletion and allow email changes. Sometimes I just want to delete my account and that is not always possible. I also notice that not everyone answers my requests to delete my data or change my email. I will find out if GDPR is useful for me as a person. I am curious if I can execute my rights. How that will go when I file a complaint, will be another story I suppose.

I was surprised by the number of password and username leaks uncovered through Bitwarden reports. Bitwarden and SimpleLogin come at a cost, but it's worth it. I'm still using Gmail but first things first. I am working on decoupling from known websites.

I do want to have a better workflow to create an alias for contact when I need to initiate the email tough. The first time is overwhelming, but now it is okay. I will look into the extensions for the browsers I use.

I need to search for a better way of defining my naming strategy. But at this moment, I have used the catch-all strategy. This means that anybody can mail at that subdomain and it will be delivered that the default mailbox. Off course, by the time this is published, I have disabled that catch-all way of working.

Have a happy easter all!

As mentioned in the initial article of this series, I've grown increasingly frustrated with phishing emails and fraudulent text messages. My objective is to regain control of my data, and writing these blog posts serves as a commitment to taking action. I will share my past experiences, current efforts, and the knowledge I acquire along the way.

In the previous blog post in this series, we discussed the importance of password managers, specifically KeePass and Bitwarden, and how they can help keep your online accounts secure.

Context

So far, we've determined that it's crucial to safeguard oneself against cybercriminals who exploit leaked data. Your information is out there on the internet, but by implementing two-factor authentication (2FA) and using unique passwords, you can protect your accounts on various websites from conventional unauthorized access methods.

Managing unique login names for each website isn't too challenging, and it makes the combination harder to guess. This is a relatively simple task when using a password manager. However, the issue I want to tackle goes beyond that—I want to avoid phishing emails and scams. To achieve this, I need a unique email address for logins and registrations, or even better, a separate email address for each website I sign up for.

The concept of using a unique email address allows you to pinpoint the source of data leaks or sales. Additionally, you can switch the email address you provided to a different one and disregard any messages sent to the original account.

The Phantom Nickname: Evolution of Email Identities.

In the early days of the internet, email addresses often featured nicknames or pseudonyms. Creative and unique nicknames allowed us to express ourselves. For me, it was a reference to the Kilrathi: A race of evolved cats in space. However, it became increasingly difficult to be able to register those popular nicknames. Many people used those nicknames in forums, chat rooms and online communities. It was natural for users to continue using these nicknames for their email addresses as well.

Over time, the need for using real names in email addresses emerged. We (Gen X / Millenials) became adults. Thus we needed an email address with real names. It was essential to use real names to establish credibility, trust, and professionalism.

Using your real name can help verify the other party the legitimacy of an email, reducing the risk of phishing and other scams.

Order 66? Establishing Trust in the Email Wars.

In the early days of the internet, home computers and poorly secured email servers were vulnerable to exploitation by botnets and cyber criminals. The systems were compromised and used to send out large volumes of spam or fake emails. The origin of the email seemed genuine because of the true origin of the message. This led to a significant decrease in trust and reliability of email communications, as well as an increased risk of cyber threats like phishing and malware distribution. ISPs started blocking outbound connections on Port 25 for residential customers to prevent their computers from being used as spam relays by botnets or cyber criminals. This action forced users to rely on their ISP's mail server or use an alternative, authenticated SMTP port (such as 587) provided by their email service.

The issue with using nicknames or arbitrary names in email addresses is that it can undermine the social and visible trust between parties. A solution is needed to ensure the recipient can verify the email's legitimacy. Fortunately, some ingenious technical solutions have been developed:

• SPF (Sender Policy Framework): This protocol helps defend against email spoofing and phishing attacks by allowing recipient mail servers to confirm that an email claiming to be from a particular domain was indeed sent from an authorized server.

• DKIM (DomainKeys Identified Mail): This method helps guarantee that the email hasn't been tampered with and verifies the sender's identity.

• DMARC (Domain-based Message Authentication, Reporting, and Conformance): This approach enables domain owners to dictate how receiving mail servers should manage emails failing SPF or DKIM checks. It also allows domain owners to oversee email delivery and possible misuse of their domain.

The issue with the solutions mentioned earlier is that they don't consider the average person's understanding of email technology. Many of us are unaware of the processes that occur when an email is sent. Most of us understand the basics.

But, these methods don't appear to be very effective, because spammers and phishers can make new inboxes with custom origins way quicker than email providers can figure out they're not legit.

Exploring the alias: A New Hope.

People are starting to experiment again with their email names. Some tested strategies are used by creating a unique email address for each site. When the alias is too obvious, others can guess your "master" email or other aliases. So you will need a random string in that email address as well. Does it ring a bell to password creation?

On the following Reddit forum, you can find discussions about the consequences of using random prefixes in an email address when registering on a website. This website also discusses the uses of an alias. Others are discussing other strategies to avoid as well. I am not alone in this. Others have blogged about this as well.

Some companies may even struggle to understand when an email address is created specifically for their organization, leading them to believe that the origin is being falsified.

Not everyone has a technical background, so misunderstandings can arise. Tools can also misidentify your legit random alias as a fake email.

I have not chosen why a strategy that I will implement. That I need to do. I do however explore multiple options that I could implement.

The Catch-All Menace

The catch-all inbox approach is suitable if you have a personal domain for creating your email addresses.

The way it functions is by activating an option that allows all emails sent to any address ending with your domain to be received in a "catch-all" inbox. Congratulations, you can now use an alias on a website. However, there are some downsides to utilizing this method.

1. Catch-all inboxes can receive more spam as they accept emails sent to any address using your domain suffix, even if the address doesn't exist.

2. Managing and organizing emails in a catch-all inbox can be more challenging, as emails from various aliases are combined in a single location.

To hide your true inbox, this seems a valid option for me. Easy to set up and use. I need to think about the naming strategy I would use.

However, the downsides outweigh the benefits. The biggest one for me is I can't reply without revealing my true email address.

Divide to Rule the Galaxy

In the spirit of the hit and run and divide and conquer tactic, consider creating a limited number of real email addresses that you designate for non-important matters and can close whenever you want. Managing too many at once would be overwhelming, so it's best to keep the count reasonable.

Alongside these disposable addresses, maintain a couple of email accounts for official communications. It's also a good idea to switch up your official email address from time to time for added security.

However, there is a serious downside to using this. It is time-consuming, you have a potential loss of information, and it gets complex very fast. Not to forget you get a reputation risk. Some service providers may view frequently changing email addresses as a sign of suspicious activity, potentially leading to account restrictions or limitations.

I won't pursue this option: I do not want to be seen as a criminal and most of all, it just takes too much time and management.

Force Amplified: Enhancing Powers with a Galactic Plus

For your convenience, I quote Wikipedia on this one:

Some mail services support a tag included in the local-part, such that the address is an alias to a prefix of the local part. For example, the address joeuser+tag@example.com denotes the same delivery address as joeuser@example.com. RFC 5233,<sup>[15]</sup> refers to this convention as subaddressing, but it is also known as plus addressing, tagged addressing or mail extensions.

So how can you use this?

• For Gmail, Hotmail, Outlook, this is supported!

• For your company, when using Exchange, you can use it as well. e.g. john.doe+1234@company.com

Great, you can now leave a valid personal email address at that company! While this is a great feature to help you filter your email inbox, not all companies are keen on this.

Email addresses with a plus sign are sometimes mistakenly deemed invalid. A company might quietly remove the part after the plus sign and use your actual email address instead. It's not surprising at all. When companies offer promotions like "Sign up and get a 5% discount when subscribing to our newsletter," they might want to verify whether the provided email address is associated with an existing customer. This allows them to better track and understand their customer base, as well as tailor their marketing efforts to target the right audience.

The act of "sub-addressing an email" will not help you when your data is leaked as well. A malicious actor could extract your real email address and use it for spam/phishing campaigns or link it to other data breaches. The regex is fairly easy.

string pattern = @"^([\w\.\-]+)(?:\+[\w\.\-]+)?(@\w+(?:\.[\w\-]+)+)$";
string substitution = @"$1$2";
string input = @"john.doe+1234@company.be";
RegexOptions options = RegexOptions.Multiline;
Regex regex = new Regex(pattern, options);
string result = regex.Replace(input, substitution);

Officially, you won't be able to reply from the subaddress. When you respond to an email sent to name+newsletter@company.com, the reply will be sent from your main email address, name@company.com. However, some email providers like Gmail have an unsupported work-a-round. Read here for more information. The downside of this is that you need to do a manual action for each subaddress you want to send from.

Because it is fairly easy for criminals to still reach me when my data is leaked, I do not want to pursue this option. However, it is a good start.

Harnessing the Power of Temporary Email

Disposable email addresses are temporary, short-lived email accounts that provide a convenient solution for situations where you need a temporary or throwaway email address. These addresses are particularly useful for signing up for newsletters, online services, or websites that require email verification but may result in unwanted email traffic or spam.

They offer a quick and easy way to create a temporary email address with a short lifespan, ranging from a few minutes to several hours. Once the designated time has passed, the email address is automatically deleted, along with any emails that were received.

How does it work? Fairly easy, you surf to a provider like temp-mail.org and you got an inbox. You have an email assigned to you.

There are apps as well to use on your smartphones. How that can come into play, I do not know yet.

Some websites and services may block disposable email addresses, as they are often associated with spam or fraudulent activities as well. Temp-mail.org has a solution tough to help you create short-lived email addresses using your domain name.

However, due to verification that can be happening to through what hoops the mail has been sent, you do not want your good own personal domain to be blacklisted by others. But this can give you an insight into how criminals do this tough.

I will not pursue this option completely for important activities.

The Galactic Relay: Mastering Alias Email Forwarding

Some services aim to protect users' email privacy by masking their real email addresses. This is done by giving an alias.

I will mention two services: SimplyLogin and Apple's "Hide My Email". These services offer privacy-focused features designed to protect users' email addresses and reduce spam.

1. SimplyLogin: I quote their site: SimpleLogin is an open source email aliasing service that allows you to receive and send emails anonymously. Based in France, SimpleLogin has helped 50,000+ people protect their mailboxes against spams, phishing and data breaches.

2. Hide My Email by Apple): I quote their site: With Hide My Email, you can generate unique, random email addresses that forward to your personal email account, so you don’t have to share your real email address when filling out a form on the web or signing up for a newsletter. You can choose to forward emails to your iCloud Mail address or any email address associated with your Apple ID.

For the customers of Apple: you have a built-in way. I am not using that ecosystem.

There are other services as well, like Firefox relay.

SimplyLogin integrates with Gmail, Outlook and others... while Firefox Relay, on the other hand, is a standalone service that requires a separate browser extension to use. The upside is that Firefox Relay is completely free to use. SimplyLogin has a free plan, but does not seem to fit my needs. For me, the most important integration at this time is that Bitwarden also integrates with SimplyLogin for generating email aliases.

What I do like about the features that SimplyLogin offers is custom domain support, catch-all inboxes, and email forwarding. Firefox Relay seem to focus primarily on the core functionality of generating aliases.

SimplyLogin offers mobile apps for both Android and iOS. Firefox Relay currently only supports desktop browsers.

Both Firefox and SimplyLogin have a great reputation when it comes down to privacy and security because Firefox is created by Mozilla and SimplyLogin is bought by Proton AG.

Outro

If you want to read more about Email Privacy, I encourage you to read EmailPrivacy on Reddit.

In my upcoming blog post, I'll be sharing my journey with SimplyLogin and a custom domain. The quest for online privacy can be overwhelming, and sometimes it feels like we're battling against an empire of companies. But like a Jedi, we must never give up our search for the right tools to protect ourselves. With SimplyLogin, I hope to have found a way to strike a blow against the dark side of data collection and have a quieter life in the digital galaxy.

In my previous blog post, I wrote about deserialization attacks and how to prevent them. I ended the post with a section called Hunger. There I stated I still doubted the link between JSON inside a string property and when the validation attributes are triggered.

Context

The SAST (Static Application Security Testing)-squad informed us that the problem was not resolved. The property validation using data annotations didn't silence the SAST scan. The software informed us that the property still flowed from one end to the other without any sanitization occurring. Both my colleague and I were genuinely frustrated. We added the data validation attributes...

The security team gave us the OWASP's cheat sheet. Our code followed everything it mentioned, except for whitelisting objects before conversion. I was flabbergasted. I hadn't done this in the other projects, so what was the distinction?

We went on to implement a custom JSON converter using System.Text.Json, which first checks if the model to be deserialized is known.

Since the tool kept complaining about the property flow, we opted for AutoMapper instead of some code that maps one object to another. Additionally, we employed dependency injection to inject validators and utilized data annotations on the model. This approach aligned more closely with the existing code of the other projects

In all honesty, I believe the static tool scan just can't handle validation attributes, an AutoMapper and dependency injection.

The security team planned to examine our whitelisting usage and close the issue.

Naturally, being me, I couldn't just be content and let it be. I felt compelled to share my findings in this post.

Did you know...

• The word "serialization" originated from the Latin word "seriēs," which means "a sequence or succession." It is used in computer science to refer to the process of converting an object into a sequence of bytes or characters so that it can be stored, transmitted or reconstructed later. The term "serialization" is used because the process involves converting an object's internal state into a serial sequence of data that can be read or written in sequence.

• The term "deserialization" is a logical extension of "serialization," which refers to the process of converting a programming object or structure into a format that can be transmitted over a network or stored in a file

• SAST stands for Static Application Security Testing. It is a security testing methodology used to analyze the source code, bytecode, or binary code of an application to identify potential security vulnerabilities. SAST tools scan the code without executing the application, providing an efficient way to detect issues early in the development process.

The attack of the JSONs.

While conducting my research, I felt compelled to test out this kind of attack on a personal web API project I had created for demonstration purposes.

I stumbled upon the following sources as I delved into researching deserialization attacks targeting both the body and the authorization header.

• Reddit - Dive into anything: In this source, you'll find a link to a presentation by hackers, where they discuss how they discovered the vulnerabilities.

• Hacking JWT Tokens: Bruteforcing Weak Signing Key (JWT-Cracker) | by Shivam Bathla | Pentester Academy Blog: An excellent read that demonstrates the importance of employing an RSA method for validating your tokens.

• Critical vulnerabilities in JSON Web Token libraries (auth0.com: A valuable read from Auth0, covering various vulnerabilities associated with JSON Web Tokens.

The head and body by a Mermaid.

In the section titled "AuthorizationHeader," you'll discover a brief overview highlighting the significance of signature validation. Meanwhile, in the "Body" section, you'll learn about the process after JSON has been deserialized, and how data annotations won't be of assistance in that scenario.

AuthorizationHeader

Since JWT is also in JSON format, it's crucial to ensure that your configuration is set up correctly. If not, the same attack could be applied to a JWT. Having a robust method for signing your token is of paramount importance.

If you permit an embedded certificate or certificate location, use a weak symmetric key, or have no key at all, you must exercise caution, as attackers can exploit these vulnerabilities. Attackers may impersonate another identity, or, if an outdated serializer is active, they could even carry out an RCE (Remote Code Execution) attack.

Keep in mind that once the signature is verified, the payload can be considered trustworthy.

From Body

After conducting some research, I struggled to find a reliable source that explains the process of transitioning from an HTTP request to an HTTP response. However, by setting breakpoints, I managed to create the following sequence diagram to illustrate the process.

When you create a custom validation attribute and examine the stack trace, you'll see the validation context upon opening it. You'll notice that an object instance is already present. This serves as clear evidence that attribute validation takes place on the object model. That means that the deserialization process takes place first.

To safeguard your server from a JSON attack, this point in the process is too late.

Whitelist your models

To protect yourself from unexpected models, whitelist your models in a custom json serialser.

public class Program
{
...   
 builder.Services.AddControllers()
        .AddMvcOptions(options =>
        {
          //Only allowd types for the input can be allowed
            var jsonInputFormatter =
                (SystemTextJsonInputFormatter)options.InputFormatters.First(f =>
                    f.GetType() == typeof(SystemTextJsonInputFormatter));
            options.InputFormatters.Clear();
            options.InputFormatters.Add(jsonInputFormatter);

            jsonInputFormatter.SerializerOptions.Converters.Add(
                new WhitelistJsonConverter(AllowedTypesForDeserialisation.AllowedTypes));
        });

using System;
using System.Linq;
using System.Text.Json;
using System.Text.Json.Serialization;

public class WhitelistJsonConverter : JsonConverterFactory
{
    private readonly Type[] _allowedTypes;
    public WhitelistJsonConverter(params Type[] allowedTypes){
        _allowedTypes = allowedTypes;
    }

    public override bool CanConvert(Type typeToConvert) {
        return Array.Exists(_allowedTypes, type => type == typeToConvert);
    }

    public override JsonConverter CreateConverter(Type typeToConvert, JsonSerializerOptions originalOptions)
    {
        var optionsWithoutThisConverter = new JsonSerializerOptions(originalOptions);
        optionsWithoutThisConverter.Converters.Remove(this);

        var converterType = typeof(WhitelistJsonConverterInner<>).MakeGenericType(typeToConvert);
        return (JsonConverter)Activator.CreateInstance(converterType, new object[] { optionsWithoutThisConverter });
    }

    private class WhitelistJsonConverterInner<T> : JsonConverter<T>{
        private readonly JsonSerializerOptions _options;

        public WhitelistJsonConverterInner(JsonSerializerOptions originalOptions){
            _options = new JsonSerializerOptions(originalOptions);
        }

        public override T Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options){
            return JsonSerializer.Deserialize<T>(ref reader, _options);
        }

        public override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options){
            JsonSerializer.Serialize(writer, value, _options);
        }
    }
}

However, just whitelisting your models for the JsonSerialiser is not enough. Sanitizing input in your REST API can protect it from some attacks. That may not be enough. A string property containing a malicious JSON object could be passed to another module or server in the chain, resulting in bad things happening downstream. This is similar to a SQL injection attack, which doesn't happen on the API, but on the database. It's important to always sanitize your input to prevent attacks at any point in the chain.

Do or do not. There is no try...

I experimented with three different approaches in a custom project, but none seemed to work. I'll outline the JSONs I used. The System.Text.Json package doesn't offer a feature like Newtonsoft.Json, which uses the $type property to inform the code about the object that should be constructed. This is why I used Newtonsoft.Json in my attempt.

Authorization Header

The following JSONs show a possibility of how an attacker can try to execute a remote process. I converted this to a signed JWT.

{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022,
  "exp": 1616239022,
  "process": {
    "$type": "System.Diagnostics.Process, System.Diagnostics.Process, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a",
    "StartInfo": {
      "FileName": "cmd.exe",
      "Arguments": "/c calc.exe",
      "CreateNoWindow": true,
      "UseShellExecute": false
    }
  }
}

The above translates into a JWT. I censored the JWT, but the payload is the same.

eyJh***.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJleHAiOjE2MTYyMzkwMjIsInByb2Nlc3MiOnsiJHR5cGUiOiJTeXN0ZW0uRGlhZ25vc3RpY3MuUHJvY2VzcywgU3lzdGVtLkRpYWdub3N0aWNzLlByb2Nlc3MsIFZlcnNpb249NC4wLjAuMCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNLZXlUb2tlbj1iMDNmNWY3ZjExZDUwYTNhIiwiU3RhcnRJbmZvIjp7IkZpbGVOYW1lIjoiY21kLmV4ZSIsIkFyZ3VtZW50cyI6Ii9jIGNhbGMuZXhlIiwiQ3JlYXRlTm9XaW5kb3ciOnRydWUsIlVzZVNoZWxsRXhlY3V0ZSI6ZmFsc2V9fX0.jDT3u***

Body

The following JSON is the Json I used to try to create a file test.txt with the content test.

{ "minMaxTempC": { "min": -1, "max": 2 }, "name": "crazy", "$type": "System.Diagnostics.Process, System.Diagnostics", "StartInfo": { "FileName": "cmd.exe", "Arguments": "test > test.txt", "RedirectStandardOutput": true } }

Because the model was not the one that the server expected, I got bad requests. So I ensured that it had the same properties as the model.

public record TemperatureCelciusBucket(TempCRange minMaxTempC, string Name);
public record TempCRange(double Min, double Max);

In the JsonSerialiserSettingsProvider, you can notice the following comment about the type handling issue.

You need to set up your program to set the TypeNameHandling to All to ensure the attack could be done. Note, you want to set it to None when you try to protect yourself against an attack.

builder.Services.AddControllers().AddNewtonsoftJson(options =>
{
    options.SerializerSettings.TypeNameHandling = TypeNameHandling.All;
    options.SerializerSettings.ContractResolver = new DefaultContractResolver();
});

On the website of JSON.NET, for that property and option, you do not get that warning. Check it out for yourself:

TypeNameHandling Enumeration (newtonsoft.com)

They do mention that is a good practice to whitelist the objects that you want to use, which is also recommended by OWASP.

Outro

In my earlier post, I suggested using some sort of middleware to assist with JSON deserialization; however, this is an inefficient approach. Instead, it's better to add a custom converter and whitelist it from there. This way, you'll utilize the existing middleware and adhere to the Open/Closed principle from the SOLID pattern.

All my approaches seem to have failed to execute such an attack. I want to stress though that I am not a pentester. It seemed fun to try out this attack because it seemed simple enough. I sat on my peak of "Mount Stupid".

The JSON.NET nuget-package that I have consumed, seems to have scars. Such an attack can now be mitigated. It is a whole new world that opens and I am liking it.

I recently came across a fitting T-shirt, invented by Ricki Burke on LinkedIn that encapsulates the importance of security.

Needless to say, I want one of those :)

I will take a brief pause from my series on securing and reclaiming control of personal data. To give a little spoiler, it is about using the tool SimpleLogin | Open source anonymous email service.

In previous posts, I mentioned that a white-hat hacker demonstrated their methods of gaining unauthorized access. This was on Azure Cloudbrew. That person states that hackers behave much like little bunnies, hopping between computers and servers while accumulating tiny bits of crucial information. They discreetly gather details such as client IDs, technologies in use (e.g., JSON, ASP.NET Core), and the versions of installed DLLs. Those hackers gradually acquire a better understanding of what they can accomplish. Once armed with enough knowledge, they can orchestrate a full-scale attack.

Let us zoom in on what a hacker can do with a JSON deserialization attack.

Context

I recently helped a colleague tackle a problem where input validation was missing. There was no distinct separation between layers when processing an HTTP request and initiating a follow-up request.

After reviewing the initial security report and researching the vulnerability online, we have taken the follow actions:

• implemented layer separation,

• updated Swagger's JSON serializer to have no type handling,

• and added some validation checks.

I thought we were making progress, but due to ambiguous requirements, only a "string.IsNullOrEmpty" check was conducted.

The second security report still indicated that the property was transferred from input to output without any validation checks. I did not understand the problem. I researched the vulnerabilities but did not came across this problem. The security team firmly declared that such code would not be permitted in production.

When I sought more information about the security scan and its purpose, the team responded vaguely, saying it was simple and referring to a particular line in the report. However, when I requested specific guidance on addressing the issue and pinpointing the vulnerabilities detected by the tool, the conversation became tense and defensive. I had to remind people that I also find security important. I do not want to discuss if the report is wrong or not. I admitted I didn't fully understand the attack mentioned concerning possible JSON in a string property. This has nothing to do with the importance of validation of the input.

Ultimately, the developers in that meeting determined that using RegEx validation on the properties of the Models that ASP.NET Core processes and deserializes was the way to go. At least, we will discover that after the security scan runs again. That will take two days before we know if that will fix the result.

After that meeting, I talked with another coworker. We agreed that simulating the attack was the optimal strategy for better understanding the issue.

The journey starts

After googling a bit a round, I came to a very intresting magazine called Pentestmag.

There I have a step-by-step example of how I can recreate the problem. However, they make use of tools that I am not familiar with yet. Let us take a step back. First I need to know what the vulnerability is all about. OWASP seems a good source for that. The vulnerability is listed here: OWASP Top Ten 2017 | A8:2017-Insecure Deserialization | OWASP Foundation to stress the problem, I make a copy from that website for your easy reading:

The impact of deserialization flaws cannot be overstated. These flaws can lead to remote code execution attacks, one of the most serious attacks possible. The business impact depends on the protection needs of the application and data.

OWASP

OWASP have a sheet cheat for all kind of technologies and vulnerabilities. You can find that on this location: Deserialization - OWASP Cheat Sheet Series.

After reading the sheet cheat, I can not help but figure out that I should just not use JSON at all. The plot thickens when they link to YsoSerial.net. YsoSerial.Net is a tool designed to generate and analyze payloads for .NET applications that are vulnerable to deserialization attacks.

YsoSerial.Net

For the reader, I will make copy and paste the introduction on what YsoSerial.Net is:

ysoserial.net is a collection of utilities and property-oriented programming "gadget chains" discovered in common .NET libraries that can, under the right conditions, exploit .NET applications performing unsafe deserialization of objects. The main driver program takes a user-specified command and wraps it in the user-specified gadget chain, then serializes these objects to stdout. When an application with the required gadgets on the classpath unsafely deserializes this data, the chain will automatically be invoked and cause the command to be executed on the application host.

For a person that is just trying to create a sample project about deserialisation hacks, I need to understand first what the text above really means. They talk about gadgets. What have small physical devices have to do with JSON?

What are the gadgets they talk about?

The term "gadget" is confusing for someone like me. I am not familiar with computer security on that level yet. The word "gadget" is often used to describe small, innovative devices or tools in everyday life. However, the underlying concept of a "gadget" is a versatile, reusable, and adaptable component without injecting any code into memory. A gadget is a small piece of reusable code that can be leveraged to create an exploit. Gadget chaining is then reusing a lot of small pieces, executing after each other. That is used in ROP (Return-Oriented Programming)

Down the rabbit hole, we go... What is ROP now?

Return-Oriented Programming (ROP) is an advanced exploitation technique used by attackers to bypass security mechanisms, such as non-executable memory protections (e.g., DEP or Data Execution Prevention). ROP allows an attacker to execute arbitrary code without injecting any new code into the target process or system. Instead, ROP relies on reusing existing code sequences, called "gadgets," that are already present in the memory of the target program or system libraries. Hence the word "gadget chaining".

Getting out of the rabbit hole and into the next.

Let us go back to the magazine and after some good keyword searching using Google, I came to a blogpost that gives us a beautiful showcase of how dangerous JSON can be. Let me give you a summary before you hop over to see what is happening.

First of all, the author describes insecure deserialization as a critical vulnerability in the OWASP Top 10 list, which can lead to issues like denial-of-service attacks, authentication bypasses, and arbitrary code execution.

In the blog post by Nairuz Abulhul, Nairuz demonstrates exploiting insecure deserialization in a .NET application using JSON. Nairuz uses Burp Suite. That can be used for manual fuzzing. This means generating error messages to see how the application will react. I learned about Fuzzing on an innovation day at Xebia | Xpirit. Big shout out to (6) Michael Contento | LinkedIn who introduced me to this topic.

Through testing, Nairuzidentifies that the Bearer token, which is part of the OAuth 2.0 authorization framework, is the vulnerable parameter. To exploit this vulnerability, ysoserial.net is used. As a result, she can do a remote code execution attack, compromising the target machine. To prevent such attacks, the author recommends avoiding serialization if possible, using digital signatures like HMAC to ensure data integrity, and always validating and sanitizing user input before serialization.

But my code runs in Azure...

One of my thoughts was that Azure is secured in a manner, that Fort Knox would be jealous of... After a little bit of searching, I came across a site called https://cloud.hacktricks.xyz/. Once your browse a bit through the site, you notice a section dedicated to Azure.

Searching for a bit more on this topic, you find a blog post that explains how to create a file on the Azure WebApp by having code executed by the JsonDeserialiser.

Easy fix?

1. Choose libraries that have built-in security mechanisms to prevent deserialization attacks. For example, the System.Text.Json library in .NET Core is considered safer than Newtonsoft.Json, as it doesn't support vulnerable deserialization features by default.

2. Binary formatters, like BinaryFormatter and NetDataContractSerializer, can introduce security risks. Instead, use safer alternatives such as DataContractJsonSerializer, XmlSerializer, or System.Text.Json.

3. Restrict deserialization to specific types: If you have to use a library that supports type information during deserialization, restrict the deserialization process to a predefined set of known and safe types. For example, in Newtonsoft.Json, you can use the TypeNameHandling and SerializationBinder settings to control the deserialization process.

Show me code

I deleted the code that was in here. There is a better solution. I mentioned it on the follow up blogpost.

Hunger

I am still not satisfied with my search for this vulnerability. It was stated by the security team that it is very important to validate your properties as well. I am not discussing that. But I want to understand what mechanisms are in play to abuse that. One thing that comes to mind is using JsonConvert.DeserializeObject method on a string property of the input object.

I am also not convinced yet, that adding validation on your properties using attributes will mitigate the problem. A good read for this would be the following article by Microsoft: runtime/ThreatModel.md at main · dotnet/runtime · GitHub

Outro

I still need to search for time to play with this. I did not end up playing with creating the source code myself. However, I gained insight into how hackers can misuse JSON and even a security framework like OAuth2 to execute remote code. I have found some examples and sources that are clear enough to set my next baby steps.

I started this series when I got fed up with all those scam emails and sms. It is time to retake my digital data. In my previous post, I talked about 2FA and the weak link named "the user".

Context

Many websites require us to create a login that includes various personal information such as a username, email, phone number, and sometimes even a national registry number. Initially, a lot of people used the same login credentials for multiple websites. As a result, the next step in the evolution of online security was to require passwords with more characters, making it more difficult for hackers to guess them through brute force methods. However, this was not enough, as many websites did not store passwords securely, resulting in data breaches where login credentials were exposed. Criminals could then use these compromised credentials on other websites until they gained access. Therefore, it has become increasingly important to use unique passwords for each website where you have an account. Password managers are software that helps to achieve this.

I suggest watching Michael McIntyre's humorous video on passwords.

Hackers were interested only in our credentials?

Before personal information became a valuable commodity, gaining access to email accounts, online banking systems, and other platforms through stolen login credentials was a valuable goal for hackers.

By obtaining login credentials, hackers can access sensitive information such as financial data, personal communications, and confidential business information. In addition, login credentials can be used to carry out fraudulent activities such as identity theft, bank fraud, and phishing scams.

Moreover, with the rise of e-commerce and other online transactions, the value of personal information has also increased over time. Login credentials, combined with personal information like names, addresses, and social security numbers can be used for more advanced and sophisticated cyber attacks like spear phishing, social engineering attacks, and more.

Password managers

A password manager is a software application or tool that helps users generate, store, and manage their passwords securely. This eliminates the need for users to remember multiple complex passwords, making it easier to maintain good password hygiene and protect their online accounts from security breaches and hacking attempts. Password managers should encrypt your passwords and store them securely, making it much harder for hackers to access them.

However, there is a downside to this as well. Since your master password is the key to all of your stored passwords, if it is compromised, then all of your passwords are compromised. This is why it is crucial to choose a strong master password and to keep it secure. Because companies will do everything to protect your data... right? Oh, wait... we all remember how Facebook uses and leaked our data... Erm... Still, a password manager has its uses. It still protects us from criminals that could gain access to other websites and use that sensitive information to do more harm.

There are a lot of sites that give an overview of "password manager"-related data breaches. I started with this article: Which Password Managers Have Been Hacked? – Best Reviews

Dedicated or browser?

Even if you do not explicitly choose a password manager, modern browsers like Firefox, Edge and Chrome offer to save your passwords. Browser-based password managers often have limited features compared to dedicated password manager apps. For example, they may not offer advanced security features such as two-factor authentication or password sharing.

1. Cross-device syncing: While some browsers may offer cross-device syncing of passwords, this feature may not work as well as dedicated password managers. For example, you may have trouble syncing passwords across multiple devices or browsers.

2. Security risks: Browser-based password managers are still vulnerable to security risks, such as phishing attacks and data breaches. In addition, browser extensions can be a target for attackers who want to compromise your passwords.

So what is out there?

Other sites are better equipped to give you answers to the question of what good password managers are. Go visit passwordmanager.com. But while you are here...

I'd like to draw attention to two open-source password managers: Bitwarden and Keepass. Both of these options are open source, with Bitwarden providing a unified solution for both Windows and mobile devices, while Keepass offers a more customizable experience, allowing you to pick and choose the features you desire from a wide array of available plugins.

Bitwarden can be self-hosted if desired using the new unified deployment that requires just one docker image. It adds a layer of protection in the event of a breach on their servers. Go ahead an play around with our free Azure credits each month. For the ones that want to make use of their Synology NAS, there are good reads to find as well. Do make sure that you take every precaution when you want to host a service and expose it to the internet. Another downside is that you need to update the software as well with the updates they provide.

Though it's worth noting that the service has yet to experience any data breaches and appears to have strong security measures in place. They had two vulnerabilities that have been addressed. The pull requests are an interesting read: CVE-2020-15879 and CVE-2019-19766 . CVE-2020-15879 was discovered by HackerOne.

It is cool to note that Bitwarden also has a public API. If you want to integrate Bitwarden's password management and security features into your application or website. Just make sure your integration does not have any security risks...

While there have been some vulnerabilities identified in Keepass, the tool's offline nature makes it a less attractive target for hackers seeking large amounts of valuable data to sell to malicious parties.

Form filling

While it's important to use unique passwords to limit the scope of a potential cyber attack, it's also important to remember that if personal data is stolen, it can be difficult to regain control of that information. Therefore, if possible, it's best to avoid storing personal data with the company or website in question. Instead, consider using the form-filling feature and opting for guest access when possible, to limit the amount of personal data that needs to be shared. There will be another post on the dangers of leaking data...

Outro

I did some research for this blog post and I am happy with my choice of using Bitwarden. However, the way that Chrome and Edge offer to fill in login prompts, gives a superior user experience than the way that Bitwarden offers. Not that Bitwarden's offers are bad, not at all. I am still in doubt so I use both of them. That is bad practice because my passwords are now out of sync and in two places. I need to make work of that.

The problem is not that I do not use a password manager... The problem is the leak at the companies where my data is stored. By using a password manager I can limit the risk that criminals can do something with my credentials. However, the criminals are not interested in only those credentials... Personal information is also leaked. That is the reason why I get scam emails and sms. I need to do more research on how to protect myself further.

For the persons that are still sceptical about what leaking personal information can do, listen to this flemish podcast (start at minute 20). To quote this source: It's the beginning of a con that involves tricking parents into thinking they're speaking to their children.

In my previous post, I stated that I will begin a series that will give the reader insight into my journey of protecting my data. I summed up multiple subjects I want to explore. I already put my IoT devices on my guest network. But that is for another post.

Context

This week, it came to our ears in Tweakers.net, TikTok and other sources that a data breach happened at the company LastPass. LastPass is a company that is owned by LogMeIn. LastPass has a product that is a password manager. A password manager should keep all of your passwords safe in a vault. In a follow-up post, I will write about the importance and dangers of password managers.

A PCMag investigation found that the senior DevOps programmer of LastPass installed malware, allowing hackers to record the victim's keystrokes and obtain the master password. LastPass recently revealed that hackers accessed its cloud backups through this programmer, gaining access to customer data.

This could have been avoided when the system was protected with two-factor authentication, right... right?

What is two-factor authentication?

Two-factor authentication (2FA) is a security process that requires a user to provide two forms of identification to access a system or application. The two factors of identification typically include something the user knows (such as a password or PIN) and something the user has (such as a physical token or a mobile device). By requiring two forms of identification, 2FA makes it more difficult for an attacker to gain unauthorized access to a system or application, even if they have obtained the user's password.

Types of two-factor authentication

By now you figured out that by enabling 2FA, you protect your login against password breaches. Even if a hacker knows your password, they won't be able to log in without the second factor of authentication. You have multiple forms of two-factor authentication.

1. SMS Authentication: This involves sending a unique code to your phone via SMS, which you then enter into the website or app you're trying to access. While this method is convenient, it is not the most secure option, as hackers can intercept text messages, trick you into giving up the code, or even clone your phone number. Therefore, SMS authentication is not recommended for high-security scenarios.

   

2. Authenticator Apps: Authenticator apps like Google Authenticator, Authy, and Microsoft Authenticator generate time-based one-time passwords (TOTP) that can be used as the second factor for authentication. These apps are more secure than SMS authentication since they don't rely on SMS, and the generated codes are unique and expire after a short time. However, if someone gains access to your phone or the authentication app itself, they can generate the codes and bypass the authentication. So, it's still important to protect your phone and authentication app with a strong password or PIN.

   

3. Hardware Tokens: Hardware tokens are physical devices that generate one-time passwords or use public key cryptography to authenticate the user. They are more secure than SMS and authenticator apps because they are not vulnerable to online attacks or phishing. Even reverse engineering them is hard. However, hardware tokens can be lost or stolen, and they may not be as convenient to carry around as a phone. In 2011, there was a data breach in one of those companies. The information was leaked on how those numbers could be generated. Every hardware token was vulnerable back then...

   

4. Biometric Authentication: Biometric authentication uses physical characteristics like fingerprints, facial recognition, or iris scans to verify your identity. This method is more secure than the others since biometric features are unique to each individual and can't be easily replicated or stolen. However, biometric authentication is not foolproof, as there have been instances of hackers bypassing fingerprint sensors or using deep fake technology to trick facial recognition systems.

   

Multiple culprits... but 2FA should protect you, right?

Well, it all comes down to the user. LastPass has protected its vaults with 2FA. Only four seniors within the company have access to that vault. The hackers attempted to log in to the server using a keylogger. The 2FA protection they used, asked for approval. The senior developer approved the login, so the hackers were able to access the vault.

Outro

Security is only as strong as the people who use it. I have worked at some companies where usernames and passwords were shared among developers. One developer wants to access the vault. It is the responsible guard of the vault who approves the login. I suspect this was happening as well at LastPass. Otherwise, I cannot understand why the senior developer approved the login if he did not initiate it.

The whole internet is focused on the vulnerability of the software that was not patched, and there was no separation of home devices and work devices. If everything fails, we should be able to rely on our 2FA. Otherwise, what is the point of setting it up?

I do have issues with the strength of security sometimes. It holds you back and that is frustrating. The internet is a crowded place with a lot of people who want to do good. However, it's also filled with malicious actors who are constantly looking for ways to exploit vulnerabilities and gain unauthorized access to sensitive information. This is why companies need to prioritize security measures and implement the latest technology to protect themselves and their users. It may seem like an inconvenience, but in the end, it's better to err on the side of caution and take every possible step to ensure the safety of our online identities and data.