ASP.NET, Coding, Sitecore

Investingation application pool crash due to StackOverflow exception in Sitecore

A web application crashing with StackOverflow exception is quite the pain to resolve. We had a customer, experiencing random app pool crashed, and had to figure out what caused it. When dealing with a StackOverflow exception, nothing in logged in the Sitecore log files, and the only thing showing the error, is the Windows Event log (c00000fd). The solution was built on a Sitecore CMS 8.2, with Active directory module installed.

Active Directory Module in Sitecore

We previously had crash issues within the AD module, when the AD contained nested circular roles. When Sitecore access a user with nested circular roles, a StackOverflow exception is thrown. Therefore, we tested their AD for circular roles. To do this we used a PowerShell script querying the AD for these roles. This version is slightly modified from Richard Muellers version from 2011:

Function CheckNesting ($Group, $Parents)
{
 # Recursive function to enumerate group members of a group.
 # $Group is the group whose membership is being evaluated.
 # $Parents is an array of all parent groups of $Group.
 # $Count is the number of groups involved in circular nesting.
 # $GroupMembers is the hash table of all groups and their group members.
 # $Count and $GroupMembers must have script scope.
 # If any group member matches any of the parents, we have
 # detected an instance of circular nesting.

# Enumerate all group members of $Group.
 ForEach ($Member In $Script:GroupMembers[$Group])
 {
 Write-Host "Testing Group: $Member"
 # Check if this group matches any parent group.
 ForEach ($Parent In $Parents)
 {
 If ($Member -eq $Parent)
 {
 Write-Host "Circular Nested Group: $Parent"
 $Script:Count = $Script:Count + 1
 # Avoid infinite loop.
 Return
 }
 }
 # Check all group members for group membership.
 If ($Script:GroupMembers.ContainsKey($Member))
 {
 # Add this member to array of parent groups.
 # However, this is not a parent for siblings.
 # Recursively call function to find nested groups.
 $Temp = $Parents
 CheckNesting $Member ($Temp += $Member)
 }
 }
}

# Hash table of groups and their direct group members.
$GroupMembers = @{}

# Search entire domain.
$Domain = [System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain()
$Root = $Domain.GetDirectoryEntry()
$Searcher = [System.DirectoryServices.DirectorySearcher]$Root

$Searcher.PageSize = 200
$Searcher.SearchScope = "subtree"
$Searcher.PropertiesToLoad.Add("distinguishedName") > $Null
$Searcher.PropertiesToLoad.Add("member") > $Null

# Filter on all group objects.
$Searcher.Filter = "(objectCategory=group)"
$Results = $Searcher.FindAll()

# Enumerate groups and populate Hash table. The key value will be
# the Distinguished Name of the group. The item value will be an array
# of the Distinguished Names of all members of the group that are groups.
# The item value starts out as an empty array, since we don't know yet
# which members are groups.
ForEach ($Group In $Results)
{
 $DN = [string]$Group.properties.Item("distinguishedName")
 $Script:GroupMembers.Add($DN, @())
}

# Enumerate the groups again to populate the item value arrays.
# Now we can check each member to see if it is a group.
ForEach ($Group In $Results)
{
 $DN = [string]$Group.properties.Item("distinguishedName")
 $Members = @($Group.properties.Item("member"))
 # Enumerate the members of the group.
 ForEach ($Member In $Members)
 {
 # Check if the member is a group.
 If ($Script:GroupMembers.ContainsKey($Member))
 {
 # Add the Distinguished Name of this member to the item value array.
 $Script:GroupMembers[$DN] += $Member
 }
 }
}

# Count the number of circular nested groups found.
$Script:Count = 0
# Retrieve array of all groups in the domain.
$Groups = $Script:GroupMembers.Keys
# Enumerate all groups and check group membership of each.
Write-Host "Domain: $Domain"
Write-Host "Root: $Root"
Write-Host "Searcher: $Searcher"

ForEach ($Group In $Groups)
{
 # Check group membership for circular nesting.
 CheckNesting $Group @($Group)
}

Write-Host "Number of circular nested groups found = $Script:Count"

After running this script, we could rule out the Active Directory Module and circular nested roles.

Number of circular nested groups found = 0

Using recursive metodes in custom code.

Another thing that could cause a StackOverflow exception, is recursive calls in custom code. However the solution was quite large in terms of code and classes. Therefore, I used a C# script to identify recursive calls and LinkPad to run it. The Script need Mono.Cecil.dll to run, and can be found here: Mono.Cecil

const string AssemblyFilePath = @"{AssemblyFilePath}";

void Main(){
 var assembly = ModuleDefinition.ReadModule(AssemblyFilePath);
 var calls =
     (from type in assembly.Types 
     from caller in type.Methods
     where caller != null && caller.Body != null
     from instruction in caller.Body.Instructions
     where instruction.OpCode == OpCodes.Call
     let callee = instruction.Operand as MethodReference
     select new { type, caller, callee }).Distinct();

var directRecursiveCalls =
     from call in calls
     where call.callee == call.caller
     select call.caller;

foreach (var method in directRecursiveCalls)
     Debug.WriteLine(method.DeclaringType.Namespace + "." + method.DeclaringType.Name + "." + method.Name + " calls itself");
}

This script iterates through assembly files and locate recursive calls:

Results for recursive calls in Assembly.Kernel.dll: 6
Assembly.Kernel.Providers.ActivityStream.SingleActivityFetcher.GetActivityRootItem
Assembly.Kernel.Pipelines.Events.Development.SerializeEvent.SerializeItemAndParent 
Assembly.Kernel.Localisation.Client.ResourceManager.GenerateAppFile
Assembly.Kernel.HttpHandlers.MenuService.GetParent
Assembly.Kernel.HttpHandlers.MenuService.GetParentOfSelf 
Assembly.Kernel.Helpers.SitecoreHelper.GetDescendants

This will output all direct recursive calls in the assembly, and make it a lot faster to check if any of them could be causing a StackOverflow exception. However, in our case, this wasn’t the issue.

Creating a memory dump on the time of crash

Out last resort was to attach a tool to the w3wp.exe process and create a dump file when the application crashed on a StackOverflow exception. This can cause the application to run slow and might not be possible in a production environment.

Sitecore Support suggested we use a lightweight tool called ProcDump. Using this tool was a lot less performance consuming than other tools we tried.  We attached it and set it up to collect a memory dump on first chance of a StackOverflow exception:

procdump -e 1 -f "System.StackOverflowException" -ma [Name or PID]

Take a look at the link above to see how to use ProcDump.

Looking through the memory dump, it was clear a save event in Sitecore was resaving and kept firing the save event. In a save event we were adjusting ItemSecurity, with this:

item.Editing.BeginEdit();
item.Security.SetAccessRules(accessRules);
item.Editing.EndEdit(false, true);

This has worked for a long time in the solution, but we recently upgraded it to Sitecore 8.2, and apparently the method SetAccessRules() now saves the item itself.

To fix it Sitecore support suggested we wrapped it in a event disabler, because we are modifying more fields than security:

using(new EventDisabler()){
    item.Editing.BeginEdit();
    item.Security.SetAccessRules(accessRules);
    item.Editing.EndEdit(false, true);
}

After being through all these differnt approches, the memory dump showed us exactly where to look. After identifying where the issue was it didn’t take long to fix and prevent it from happening again. The memory dump is the easiest way to locate a StackOverflow exception, however, it might not always be possible in a production environment.

The other approaches might be useful in other scenarios as well. The AD circular nested roles were nearly impossible to find the first time we encountered it. This script would have helped us a lot back then.

 

Advertisements
ASP.NET, Coding

Automatic AssemblyInfo creation on build!

I am currently developing a Sitecore module for a client, who needed a versioning based on builddate. They wanted the module to display an message to the user if it was out of date. We therefore needed to update the assembly files versions on each release. To ensure we remember to to update the version with every release, I looked into making this automatic. This can be usefull for Sitecore modules and all other applications.

To achieve this, I used AssemblyInfo for versioning. First we need to retrive the version form the assembly file:

var version = FileVersionInfo.GetVersionInfo(Assembly.GetExecutingAssembly().Location).FileVersion;

We use it to check against a webservice, where the client provided the newest version number in XML.

<ApplicationVersions>
   <SitecoreConnector>2017.05.31.1003</SitecoreConnector>
</ApplicationVersions>

There is a lot of modules or plugins which can auto increment the version in assemblyinfo. However we needed it to be based on date and time. Therefore I decided to use T4 TextTemplates, to create a AssemblyVersion.cs file:

<#@ template language="C#" #>
<#@ output extension=".cs" #>  
using System;
using System.Reflection;
 
[assembly: AssemblyVersion("<#=DateTime.Now.ToString("yyyy.MM.dd.HHmm")#>")]
[assembly: AssemblyFileVersion("<#=DateTime.Now.ToString("yyyy.MM.dd.HHmm")#>")]
[assembly: AssemblyCopyright("Copyright © Customer <#=DateTime.Now.Year#>")]

T4 is used by developers as part of an application or tool framework to automate the creation of text files with a variety of parameters. These text files can ultimately be any text format, such as code (for example C#), XML, HTML or XAML. T4 Text transformation (Wiki)

I use C# to get the date and time, and format it to my versioning needs. It can also be used to set the Copyright year dynamically. When this file is saved, Visual Studio automatically runs the template and create a AssemblyVersion.cs file:

using System;
using System.Reflection;
 
[assembly: AssemblyVersion("2017.05.31.1003")]
[assembly: AssemblyFileVersion("2017.05.31.1003")]
[assembly: AssemblyCopyright("Copyright © Customer 2017")]

To automate this creation, I have created a pre-build event for the project. This event uses TextTransform.exe to transform the T4 template before the build.

set textTemplatingPath="%CommonProgramFiles(x86)%\Microsoft Shared\TextTemplating\$(VisualStudioVersion)\texttransform.exe"
if %textTemplatingPath%=="\Microsoft Shared\TextTemplating\$(VisualStudioVersion)\texttransform.exe" set textTemplatingPath="%CommonProgramFiles%\Microsoft Shared\TextTemplating\$(VisualStudioVersion)\texttransform.exe"
%textTemplatingPath% "$(ProjectDir)AssemblyVersion.tt"

And thats it! Automatic created AssemblyVersion.cs which runs on every build.