How to Gather ReportOnly Conditional Access Sign-In Logs

Conditional Access policies are a powerful security tool in Azure, allowing administrators to control access to resources based on conditions like user location, device state, or app sensitivity. However, implementing these policies without proper testing can cause disruptions for users. That’s where ReportOnly mode comes in handy.

By using ReportOnly mode, you can monitor how your policies would behave without enforcing them, giving you insight into their potential impact on users. This post will show you how to use PowerShell and the Microsoft Graph API to gather sign-in logs for policies that are in ReportOnly mode. With this script, you can test the effect of your policies and refine them before rolling them out.

Prerequisites

Before running the script, make sure you have the following:

• Microsoft Graph PowerShell Module
  Install it using:

  powershell

  Install-Module Microsoft.Graph -Scope CurrentUser

• Azure AD Audit Permissions
  You’ll need AuditLog.Read.All permissions within Microsoft Graph to access audit logs.

PowerShell Script for Gathering ReportOnly Conditional Access Sign-In Logs

# Import the Microsoft Graph Reports module
Import-Module Microsoft.Graph.Reports

# Connect to Microsoft Graph with appropriate scope
Connect-MGGraph -Scope AuditLog.Read.All

# Fetch all sign-in logs
$SignIns = Get-MGAuditLogSignIn

# Initialize an array to store the reports
$Reports = @()

# Loop through each sign-in log entry
ForEach ($SignIn in $SignIns) {
    $Policies = $SignIn.AppliedConditionalAccessPolicies
    ForEach ($Policy in $Policies) {
        If (($Policy.Result -like "reportOnly*") -and ($Policy.Result -ne "reportOnlyNotApplied")) {
            $Report = [ordered]@{
                'Login Time'   = $SignIn.CreatedDateTime
                'Policy Name'  = $Policy.DisplayName
                'Username'     = $SignIn.UserDisplayName
                'Result'       = $Policy.Result
                'Device Name'  = $SignIn.DeviceDetail.DisplayName
                'OS Version'   = $SignIn.DeviceDetail.OperatingSystem
                'Browser'      = $SignIn.DeviceDetail.Browser
                'IP Address'   = $SignIn.IPAddress
            }
            $Reports += @(New-Object -TypeName psObject -Property $Report)
        }
    }
}

# Output the report in table format
$Reports | Format-Table

Script Breakdown

• Import the Microsoft Graph Module: Enables access to audit logs via Microsoft Graph
• Authentication: Connect-MGGraph authenticates with the required scope
• Fetch Sign-In Logs: Get-MGAuditLogSignIn retrieves all sign-in events
• Filter by ReportOnly Policies: Loops through applied policies and filters for those in ReportOnly mode
• Create a Report Object: Captures key details like policy name, user, device, and result
• Output: Displays the data in a structured table format

Why Use This Script?

This script is useful for administrators who want to:

• Test Conditional Access policies in ReportOnly mode without enforcing them
• Understand how policies would affect user sign-ins
• Prepare reports to share with teams or refine policies before rollout

Running the Script

1. Open PowerShell as an administrator
2. Copy and paste the script into the PowerShell window
3. Press Enter to run the script
4. Review the output to see sign-ins affected by ReportOnly policies

This approach gives you a clear understanding of how your Conditional Access policies are behaving, allowing you to make informed decisions before enabling them across your environment.