Conditional Access policies are a powerful security tool in Azure, allowing administrators to control access to resources based on conditions like user location, device state, or app sensitivity. However, implementing these policies without proper testing can cause disruptions for users. That’s where ReportOnly mode comes in handy.

By using ReportOnly mode, you can monitor how your policies would behave without enforcing them, giving you insight into their potential impact on users. This post will show you how to use PowerShell and the Microsoft Graph API to gather sign-in logs for policies that are in ReportOnly mode. With this script, you can test the effect of your policies and refine them before rolling them out.

Prerequisites

Before running the script, make sure you have the following:

• Microsoft Graph PowerShell Module: You need to install the Microsoft.Graph.Reports module if you don’t have it installed already.
  • To install it, use:

Install-Module Microsoft.Graph -Scope CurrentUser

• Azure AD Audit Permissions: You’ll need permissions to read audit logs in Azure AD, such as AuditLog.Read.All within Microsoft Graph.

PowerShell Script for Gathering ReportOnly Conditional Access Sign-In Logs

Here’s the full PowerShell script that will help you retrieve all sign-in logs related to Conditional Access policies with a ReportOnly result:

# Import the Microsoft Graph Reports module
Import-Module Microsoft.Graph.Reports

# Connect to Microsoft Graph with appropriate scope
Connect-MGGraph -Scope AuditLog.Read.All

# Fetch all sign-in logs
$SignIns = Get-MGAuditLogSignIn

# Initialize an array to store the reports
$Reports = @()

# Loop through each sign-in log entry
ForEach ($SignIn in $SignIns) {

    # Get the list of applied conditional access policies
    $Policies = $SignIn.AppliedConditionalAccessPolicies

    # Loop through each policy
    ForEach ($Policy in $Policies) {

        # Check if the policy result is in ReportOnly mode (and not 'Not Applied')
        If (($Policy.Result -like "reportOnly*") -and ($Policy.Result -ne "reportOnlyNotApplied")) {

            # Create a structured report object for each matching log entry
            $Report = [ordered]@{
                'Login Time'   = $SignIn.CreatedDateTime
                'Policy Name'  = $Policy.DisplayName
                'Username'     = $SignIn.UserDisplayName
                'Result'       = $Policy.Result
                'Device Name'  = $SignIn.DeviceDetail.DisplayName
                'OS Version'   = $SignIn.DeviceDetail.OperatingSystem
                'Browser'      = $SignIn.DeviceDetail.Browser
                'IP Address'   = $SignIn.IPAddress
            }

            # Add the report to the collection
            $Reports += @(New-Object -TypeName psObject -Property $Report)
        }
    }
}

# Output the report in table format
$Reports | Format-Table

Script Breakdown

Here’s a breakdown of how this script works:

1. Import the Microsoft Graph Module:
   • The script starts by importing the Microsoft.Graph.Reports module, which is required to access audit logs through the Microsoft Graph API.
2. Authentication:
   • The Connect-MGGraph command is used to authenticate with the Microsoft Graph API, allowing access to audit logs. The AuditLog.Read.All scope provides the necessary permission to read sign-in logs.
3. Fetch Sign-In Logs:
   • The Get-MGAuditLogSignIn command retrieves all available sign-in logs, which contain information about applied Conditional Access policies.
4. Filter by ReportOnly Policies:
   • The script loops through each sign-in log and then checks the result of each applied policy. It filters out policies that are not in ReportOnly mode or that have the result reportOnlyNotApplied.
5. Create a Report Object:
   • For each relevant log entry, the script creates a report with key details such as the policy name, username, result, device information, and IP address.
6. Output:
   • Finally, the script outputs the gathered data in a table format using Format-Table, which allows you to review the sign-in logs in a structured way.

Why Use This Script?

This script is useful for administrators who want to:

• Test Conditional Access policies in ReportOnly mode and understand their impact without enforcing them.
• Gather insights on how policies would affect user sign-ins.
• Prepare reports on Conditional Access activity to share with teams or adjust policies before enforcement.

By running this script, you can ensure that your Conditional Access policies are functioning as intended and minimize any potential disruptions when you roll them out.

Running the Script

1. Open PowerShell as an administrator.
2. Copy and paste the script into the PowerShell window.
3. Press Enter to run the script.
4. Review the output, which will display sign-ins affected by Conditional Access policies in ReportOnly mode.

This approach gives you a clear understanding of how your Conditional Access policies are behaving, allowing you to make informed decisions before enabling them across your environment.