Hi everyone, Buz Brodin here. Recently I worked on a problem where a customer reported that sometimes App-V applications (shortcuts on desktops) that were deployed to users logged in to a Citrix server never appeared for some of the users, or they did appear but showed up rather slowly (within two hours or so). For other users everything seemed to work perfectly fine.
When this happens, it’s usually because one of the required applications is failing, which can cause a delay of up to 90 minutes before the other applications are deployed. If you see a scenario such as this one, here is a method you can use to find what is targeted to a user SID and what application is having the problem.
1. First, have one of the users who is experiencing the problem log in to the system. Once you confirm that the user did not receive the required applications, collect the CIAGENT and the DCMAGENT logs and use CMTrace to combine them into one file.
2. Using the SID of the user (in this example we’ll use S-1-5-21-2414399092-292161696-3476272628-181234), search for InitiateCIAgentJob in the combined log and find the first instance that mentions this SID. Here you will find the CIAGENT and DCMAGENT Job ID:
DCMAgentJob({2CAFAAAF-E4DE-46BD-8D8B-6488E21C6F61}): CDCMAgent::InitiateCIAgentJob – Starting CI Agent Job {B4B0C5E5-C9B0-4D87-9058-F8B03C820A78} for target: S-1-5-21-2414399092-292161696-3476272628-181234. Refer to this CI agent job ID in ciagent.log for more details DCMAgent 3/25/2016 4:21:47 PM 5244 (0x147C)
DCMAgentJob ({2CAFAAAF-E4DE-46BD-8D8B-6488E21C6F61})
CIAGENTLOG – {B4B0C5E5-C9B0-4D87-9058-F8B03C820A78}
3. Filter for the CIAGENT JOB ID first: This shows all of the Application IDs targeted for that user as well as the status at the job. Here’s an example showing some of the required applications:
CIAgentJob({B4B0C5E5-C9B0-4D87-9058-F8B03C820A78}): AddCI called for ScopeId_AA0BA009-8B99-4F9F-A54D-4E5F3F43F0FA/RequiredApplication_5f588354-71d3-4dce-b366-5fd7db6331cf:6 CIAgent 3/25/2016 4:21:47 PM 5244 (0x147C)
CIAgentJob({B4B0C5E5-C9B0-4D87-9058-F8B03C820A78}): AddCI called for ScopeId_AA0BA009-8B99-4F9F-A54D-4E5F3F43F0FA/RequiredApplication_0ef22753-ceab-4941-bafe-8fe937addbf6:10 CIAgent 3/25/2016 4:21:47 PM 5244 (0x147C)
CIAgentJob({B4B0C5E5-C9B0-4D87-9058-F8B03C820A78}): AddCI called for ScopeId_AA0BA009-8B99-4F9F-A54D-4E5F3F43F0FA/RequiredApplication_2166f012-1d6e-4b5a-84a9-20b0379b2fed:9 CIAgent 3/25/2016 4:21:47 PM 5244 (0x147C)
4. This is the important part: Notice the status of the job noting a failure:
CIAgentJob({B4B0C5E5-C9B0-4D87-9058-F8B03C820A78}): QueueDebug – Executing Event. CIAgent 3/25/2016 4:21:47 PM 9576 (0x2568)
5. Now look at the time between this line and the previous line – it’s exactly 90 minutes, which is the default timeout for DCMAGENT.
CIAgentJob({B4B0C5E5-C9B0-4D87-9058-F8B03C820A78}): CAgentJob::VersionInfoTimedOut for ModelName ScopeId_AA0BA009-8B99-4F9F-A54D-4E5F3F43F0FA/RequiredApplication_4ff04363-a52b-4fb1-a2f0-3328d9b346e3, version 8 not available. CIAgent 3/25/2016 5:51:47 PM 9448 (0x24E8)
Now we know something about what failed; it was this required application: RequiredApplication_4ff04363-a52b-4fb1-a2f0-3328d9b346e3, and it failed with this message version 8 not available.
6. Next, filter your log for that RequiredApplicationID: 4ff04363-a52b-4fb1-a2f0-3328d9b346e3. You’ll probably see entries similar to the following:
CIAgentJob({5E261F3F-8BB7-4E9F-9FE9-57DC253D6AD2}): CAgentJob::VersionInfoTimedOut for ModelName ScopeId_AA0BA009-8B99-4F9F-A54D-4E5F3F43F0FA/RequiredApplication_4ff04363-a52b-4fb1-a2f0-3328d9b346e3, version 8 not available. CIAgent 3/25/2016 5:32:05 PM 12312 (0x3018)
VersionInfo ModelName=ScopeId_AA0BA009-8B99-4F9F-A54D-4E5F3F43F0FA/RequiredApplication_4ff04363-a52b-4fb1-a2f0-3328d9b346e3, Version=8, IsVersionSpecific=TRUE timed out CIAgent 3/25/2016 5:32:05 PM 5172 (0x1434)
5. For the final step, unfilter your log and search for this required application string (4ff04363-a52b-4fb1-a2f0-3328d9b346e3:8). You are looking for an entry that starts with PopulateCIsFromAssignment:
DCMAgentJob({6D2C8935-2B30-468E-8901-739238DE3D32}): CDCMAgentJob::PopulateCIsFromAssignment – [SKIPPING] CI policy Id :ScopeId_AA0BA009-8B99-4F9F-A54D-4E5F3F43F0FA/RequiredApplication_4ff04363-a52b-4fb1-a2f0-3328d9b346e3 version:8 – Assignment:{4B634692-9479-4904-8BAA-42B3BCD9C7C0}. DCMAgent 3/25/2016 4:08:22 PM 8416 (0x20E0)
From there, go up a few lines to the first instance of CDCMAgentJob that comes before PopulateCIsFromAssignment. This gives you the friendly name of the deployment that failed:
DCMAgentJob({6D2C8935-2B30-468E-8901-739238DE3D32}): CDCMAgentJob::SetupJob – Add assignment Contoso_Application_73003101084_r1_APPL-D-ContosoApplication-7300-3-10-1084_Install({4B634692-9479-4904-8BAA-42B3BCD9C7C0}) DCMAgent 3/25/2016 4:08:22 PM 8416 (0x20E0)
Contoso_Application_73003101084_r1_APPL-D-ContosoApplication-7300-3-10-1084_Install
Now that you know which deployment is failing, simply remove this deployment and the problem with the 90 minute delay should be resolved.
Hopefully this method will make troubleshooting these issues a bit easier moving forward. Keep in mind that other things also need to happen in this scenario, such as CIAGENT download and policy requests, but at its core DCMAGENT is doing the work here.
Buz Brodin, Senior Support Escalation Engineer
Microsoft Enterprise Cloud Group