Server Side Sync with Forward mailbox and SMTP

Server Side Sync with Forward mailbox and SMTP

Most often when setting up server side sync, especially in Online environments, Dynamics 365 CE/CDS is connected to Exchange Online. This is rather straight forward. Even in less straight forward cases like On-prem to On-prem or the so called hybrid cases of on-line to on-prem where the involved parties are Microsoft Dynamics 365 and Exchange in some manner are all rather well documented and might be a bit tricky but generally there are some good instructions for how to get it working. Like for instance this. However, if your setups require more complex email management then the documentation and blogs around the Internet start getting a lot sparser. This article will detail a complex setup with Server Side Sync using Forward Mailbox to a proxy O365 exchange account and outgoing email using SMTP using the SMTP service SMTP2GO.
I recently migrated a customer with a complex setup from Email router to Server Side Sync as the Email router has been long deprecated and the indications I am getting from Microsoft are that it is hight time to start transitioning away from it to Server side Sync or to some other solution that can solve it, like for instance Riva. I will not go into the advantages of using that in this post, but generally it can be said that it has a lot more configuration options and logging options available, but at a price of course. Most of my customers try to get the Server Side Sync (SSS) to work and if this cannot be done, then other venues, like Riva or custom code are evaluated.
The background to why this complex setup was required was that my customer had their main Exchange server managed by their parent’s parent company in Germany. If you are not aware of the requirements for setting up Server Side Synchronization (SSS) from Dynamics 365 to an Exchange, it requires the use of an account using “Application Impersonation”. Asking the Exchange admins for this permission, although I have heard (I am no Exchange guru though) that is can be limited to specific users, was perceived to be practically impossible. We could, however, setup Forwarding rules with “Forward as attachment” on the public folders where the incoming emails were received.
As for outgoing email, the story was more or less the same. We could not connect to the Exchange server in Germany to send any emails. However, my customer were allowed controll over the DNS entries of the domains they worked with hence they could add SPF records to other email sending servers. When using the Email router, we had installed this on a VM hosted in Amazon Web Services and then sent email using the Amazon Simple Email Service (SES) on port 25. When we tried this with Server Side Sync, however, we noticed that we were now “outside traffic”, no longer coming from inside Amazons networks, and were hence throttled on Port 25. We tried all other possible SMTP ports for SES but nothing seemed to work with SSS.

Incoming – Forward mailbox

Ok, so how to solve it? Let’s start with the incoming email. As I have hinted the best method for this is to use the “Forward Mailbox” technique. This means that you set up a special mailbox, you actually create a new mailbox in Dynamics of the type Forward mailbox. This is then linked to a normal Exchange mailbox user account. It cannot be a public folder or something else. It has to be a user. You might be able to use a O365 Exchange Kiosk but be aware of the size limitations if you decide to keep the email on the server. Otherwise an E1 is probably recommended. This email address is never seen by any customer, so can be rather obscure, like forwardmailbox@contoso.onmicrosoft.com.

Rules are then set up on the onprem server to forward emails coming in to all relevant email addresses (be it users, public folders, groups or otherwise) as attachments to this forward mailbox. Why forward as attachment you might ask? The reason is that it keeps the entire header of the email intact which allows Dynamics to parse it and connect it correctly. Below is an figure discribing this incoming email flow with two queues. Typically you would have many more queues. My customer has 100+ queues.

Incoming email using forward mailbox proxied via O365

Setting up forwarding rules for “forward as attachment” is done in the following steps:

Using Outlook Web Access, click on the Settings cog (1) and then Mail (2).
Go to Inbox and Sweep rules and press (+) to create a new.

Apply it to all email (if that is what you want) and then select the action – “Forward the message as an attachment to” and select the Forward mailbox contact that either created before or do it in the next step.

This is how you create a contact (this is rather straight forward)

After this, you should have a forwarding rule which is activated and you should be able to start seeing emails landing in the forward mailbox shortly after they have arrived in the normal inbox, and with the orginal email as an attachment.

When creating the forward mailbox you have to remember to get a global admin to approve the email address (1) even if you switched this off for users or queues. You also need to “Test & Enable Mailbox” (2)

Outgoing – SMTP

Outgoing email we solved by finding an SMTP service that did support Dynamics 365. As mentioned above we first tried working with Amazon Simple Email Service but found that there was no combination of settings that would make this work. The best possible option was port 25 using TLS, but as we were “outside” traffic (as compared to when we were using the Email router and running on a AWS VM) we got throttled rather quickly and I couldn’t even get the 100+ queues through the Test & Enable until things started breaking.
It is important here, to understand that there is a difference between SMTP using TLS (more modern way of securing SMTP) and using SMTP with SSL. The former seems to be what Dynamics 365 is using though I havn’t found any really good definition saying this is so.

It turned out that my customer was using SMTP2GO for another service so we tried it out and it worked fine using port 587 using TLS. SMTP2GO, it seems, has a load of different ports and variations of security setup that can be used. According to my customers operations people, they also like it better than Amazon SES as it gives better feedback on bounces and such which is good (data which would be nice to get into Dynamics of course – good ISV opportunity there!).

Outgoing settings are not that complicated – each mailbox has the Server Profile “SMTP2GO” which uses the SMTP protocol to send

To understand some of the details of how the SMTP2GO Server profile is setup, look at this picture.

The Server Profile for SMTP2GO – note that the Incoming Server location is not used/is relevant.

One of the problems I found was that I wasn’t able to set the credentials centrally, on the server profile. It just seems this isn’t supported for SMTP, I do not know why. Hence we had to add the credential (the same) to each and every single queue. I found this was rather easily done in bulk using SSIS with Kingswaysoft or your other favorite tool for this like Scribe or CozyRoc (I havn’t tried them but I guess you could). Or you can of course write a small program. I did try to do it using workflows or bulk edit but that didn’t work. Maybe with some shoehorning you could get that to work. Maybe a Flow could work too. The fields that had to be set can be seen in the picture below:

The queue mailbox record – Note the three fields marked that you need to set as the credentials are set on the queue level. Also note that as the Incoming is “Forward Mailbox” it will not be expected to be tested in the testrun – hence Incoming Email Status :”Not Run”

Testing SMTP Server/Service
When working with this, and testing out different SMTP providers it is sometimes hard to know where the problem is. Hence it is good to have a good tool to test the SMTP email server to see that it works, that your credential for it work and so on. I got a good tip from one of the operations technicians at my customer, which was the service SMTPer as seen below:

SMTPer – www.smtper.net – great tool for testing a SMTP server

Limitations

What are the limitations of using a technique like this for server side synchronization?

First of all I would say it is that you will not be able to get Appointment, Contacts and Task (ACT) synchronized. Hence it is mostly useful in applications where the Dynamics/CDS is set up to work for Customer Service or in other non-personal uses. If you want ACT synchronization I would recommend trying to get a Dynamics-Exchange synchronization working somehow. Talk to some Exchange experts to see if they have some interesting views on how to solve your issue.

The second drawback of this is complexity. This solution has many moving parts and it can go wrong in many places. There are many different accounts and password that it depends on, thankfully everything will not break if just one password is invalidated, but you will see issues. It is also dependent on different technologies like Dynamics email handling, Exchange rules, SMTP services and so on. This requires rather a broad skillset or several people being involved. Especially if something breaks or just doesn’t work supergood.

Conclusions

Hence, this a setup that I would only recommend if you do not have the option of using Exchange. If you have the option of using Exchange, but cannot get it to work for some reason, try harder or ask for help. Using this kind of solution will limit the end users functionality of Dynamics and is hence more of a “last resort”.

Warnings mean something – don’t just click them away

Warnings mean something – don’t just click them away

Warnings like the one you see above, I often see in my customers systems when rudely looking over their shoulders. And what do they do? Call support and tell them that there is some weird error about email coming up and that there must be some error in the CRM-Exchange inter-server email synchronization mechanism? No. They don’t. They ignore it. Totally. Every time. They typically get a bit annoyed every time and think the system is a bit crappy but they ignore it. And we as their CRM-partner might never know that there is anything wrong unless we have very deep monitoring of their system (has anyone built this? Would be a great tool! Tanguy to the rescue perhaps?)

Well, as you guessed I don’t agree that you should just click away these error messages or actually any error messages. They usually mean something. In the case above, there is something wrong with the synchronization of emails, why emails that have be written and sent in CRM have not been delivered. Once fixed, these WILL be sent, which might be a bit problematic. Once, maybe 6 or 7 years ago, we had a customer who had had problems with synchronization of emails and hadn’t told us. We got to know during our summer vacation (don’t you always), and when we fixed it, the CRM system sent out loads of old emails to their customers. Which was not really appreciated. Well, we all do misstakes, but not all learn from them. So, do check which emails have Status Reason”Pending Send” as you can see in the picture below.

Having emails with this status for more than 24 hours is also what causes the alert at the top of this post. So, before fixing Server Side Sync, the Email Router or installing Outlook clients, do check this list to make sure that there are not a lot of old emails that will be sent to your customers and contacts. Advanced Find is very useful if you want to find them all, and not use the built in view that I am showing above.

Gustaf Westerlund
MVP, Founder and CTO at CRM-konsulterna AB
www.crmkonsulterna.se

Queue question and the difference between community work and putting food on the table

Queue question and the difference between community work and putting food on the table

I received a question per email today and as I Work as a Consultant, Selling my time by the hour, while at the same time sharing my knowledge and experience on this blog and in Other places, I thought I might share my general view on how I separate the Community Work from the Work that puts food on my table.

I am a strong believer in the notion that we need to share knowledge with each other to be able to all grow and be stronger. This is why I started the blog, started CRM User Group Sweden, run Techie After Work (with Allan Varcoe and Peter Björkmarker), present at Conferences where I do not get paid for speaking. If we all do this in the Dynamics CRM Community, it will grow large, new people will quickly learn from the more experienced, and our CRM system, will prosper to all our benefit.

We all have to put food on the table

But, I also have to put food on the table, so when people ask me to help them by contacting me by
email or phone, they do actual ask me as a Consultant and I can and will charge for the time I help out. Typically an agreement, including NDA, is signed so that I can help out in a more direct manner, which of course, is different than somebody asking something general in the comment on a blog post.

So, if you have a question to me, regarding a post, I recommend leaving a comment. Sadly, there are a lot of comment spammers out there, looking for some links to their sites, why I need to moderate all comments, as I respect you as readers and I do not want to waste your time with crap comments and irrelevant links to companies trying to sell training or otherwise.

So, on to the question I got, as it was clearly directed as a comment, will answer it in the form of a blog post instead of an email reply, so that others in the Community can also benefit from it.

“I was reading your blog post on sending emails from a queue (http://gustafwesterlund.blogspot.com/2013/02/sending-emails-from-queue-with-email.html) and had a question.  I’m using CRM 2015 and trying to set up a queue for sending and I was wondering if the generic address needs to point to a mailbox on the server or if it can be a distribution list or mail-enabled public folder.  I’ve tried setting up an existing distribution list address for the email address, but I get errors because it can’t log on to a mailbox, so I didn’t know if it was possible or if there’s something else I need to do instead.”

It is a good question. It is not obvious why you cannot use one type of email address or Another, but if you refer to the implementation guide (yes, I know, it is a very large document(s) and hard to navigate, but try to find your way around it) you will see that it says that the email address that you indicate in a queue has to be a user mailbox. It cannot be anything else. There are several different ways of logging in to this, the simplest is to set the credentials in the queue, but this will require you to change the password in the queue every time you change it in the AD. I wrote the Synchronization overview chapter in the CRM Field Guide (http://gustafwesterlund.blogspot.se/2015/04/the-crm-2013-field-guide-is-out.html) and I would strongly recommend that you read this or the similar parts in the implementation guide for more detailed information.

Gustaf Westerlund
MVP, Founder and CTO at CRM-konsulterna AB
www.crmkonsulterna.se

Be aware of Synchronous Workflows on incoming emails and Queue items

Be aware of Synchronous Workflows on incoming emails and Queue items

How synchronous workflows that fail, make emails disappear.

During the last few Days I have been having the not so pleasant experience of trying to find some really nasty errors in incoming emails to a queue in CRM. We are using CRM 2013 SP1 (on prem) and hence are using the Server Side Sync. Everything had been working great and we had developed some neat workflows that did some magic to convert emails to cases (as those features were not available, and not good enough either) when we develop our stuff.

Well, thing stopped working, I tried sending emails to the CRM server, but none were received in CRM. All I got in CRM was a not so descriptive general alert saying:

“An error occurred while creating incoming email in Microsoft Dynamics CRM for the mailbox XYZ. To view the email that failed, see the alerts in the mailbox record.”

and in the mailbox:
“An error occurred while creating the incoming email “Test 1512″ in Microsoft Dynamics CRM for the mailbox XYZ.”

As I am working on my telepathic abilities, I have not yet reached a level where I can understand what is wrong so I had a look in the trace, and I was very happy I was running an on-prem server. It is a bit long, don’t worry if you don’t understand it all, nobody (I think) does.

[2015-05-15 15:21:13.459] Process:CrmAsyncService |Organization:673c5996-a1fa-e311-93ed-00155d0a831d |Thread:    4 |Category: Platform.Metadata |User: 00000000-0000-0000-0000-000000000000 |Level: Error |ReqId:  | <>c__DisplayClass1.<LoadMetadataContainerFromDatabase>b__0  ilOffset = 0x14
>Multi-org sharable cache loading system and non-system metadata with build number 6.1.0.581 and language 1033
[2015-05-15 15:33:45.132] Process:CrmAsyncService |Organization:00000000-0000-0000-0000-000000000000 |Thread:   26 |Category: Exception |User: 00000000-0000-0000-0000-000000000000 |Level: Error |ReqId: cf367175-12a2-4719-a5af-f53f7f254399 | CrmException..ctor  ilOffset = 0x7
 at CrmException..ctor(String message, Exception innerException, Int32 errorCode, Boolean isFlowControlException)  ilOffset = 0x7
 at CrmException..ctor(String message, Exception innerException, Int32 errorCode)  ilOffset = 0x5
 at Exceptions.ThrowIfGuidEmpty(Guid parameter, String name)  ilOffset = 0x18
 at OrganizationSdkServiceInternal.Retrieve(String entityName, Guid id, ColumnSet columnSet, CorrelationToken correlationToken, CallerOriginToken callerOriginToken, WebServiceType serviceType, Boolean checkAdminMode)  ilOffset = 0x16
 at InprocessServiceProxy.RetrieveCore(String entityName, Guid id, ColumnSet columnSet)  ilOffset = 0x28
 at OrganizationServiceProxy.Retrieve(String entityName, Guid id, ColumnSet columnSet)  ilOffset = 0x4
 at EntityDictionary.TryRetrieveEntity(String key)  ilOffset = 0xF8
 at EntityDictionary.System.Collections.Generic.IDictionary<System.String,Microsoft.Xrm.Sdk.Entity>.get_Item(String key)  ilOffset = 0x1B
 at VisualBasicValue`1.Execute(CodeActivityContext context)  ilOffset = 0x3A
 at ActivityExecutor.ExecuteInResolutionContext(ActivityInstance parentInstance, Activity`1 expressionActivity)  ilOffset = 0x35
 at InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)  ilOffset = 0x2F
 at ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)  ilOffset = 0x16
 at ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)  ilOffset = 0x9C
 at ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)  ilOffset = 0x1C
 at ActivityExecutor.OnExecuteWorkItem(WorkItem workItem)  ilOffset = 0x53
 at Callbacks.ExecuteWorkItem(WorkItem workItem)  ilOffset = 0xD
 at Scheduler.OnScheduledWork(Object state)  ilOffset = 0xC5
 at SendOrPostThunk.UnhandledExceptionFrame(Object state)  ilOffset = 0x0
 at PumpBasedSynchronizationContext.DoPump()  ilOffset = 0x2E
 at WorkflowApplication.Invoke(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout)  ilOffset = 0x23
 at WorkflowInvoker.Invoke(Activity workflow, IDictionary`2 inputs, TimeSpan timeout, WorkflowInstanceExtensionManager extensions)  ilOffset = 0x36
 at SynchronousWorkflowActivityHost.ExecuteWorkflowUsingInvoker(Activity workflow, ICommonWorkflowContext context)  ilOffset = 0xD0
 at SynchronousWorkflowActivityHost.StartWorkflow(WorkflowActivationData activationData, ICommonWorkflowContext context)  ilOffset = 0x73
 at SynchronousWorkflowActivityHost.StartWorkflow(ICommonWorkflowContext context)  ilOffset = 0xBA
 at WorkflowProcessServiceInternalHandler`1.ExecuteSyncWorkflow(Guid workflowId, PipelineExecutionContext pipelineContext, IGenericEventData workflowInstanceData, Boolean isTriggered)  ilOffset = 0x59
 at WorkflowProcessServiceInternalHandler`1.ExecuteTriggeredSyncWorkflow(Guid workflowId, PipelineExecutionContext pipelineContext)  ilOffset = 0x21
 at SyncWorkflowExecutionPlugin.Execute(IServiceProvider provider)  ilOffset = 0x2C
 at V5PluginProxyStep.ExecuteInternal(PipelineExecutionContext context)  ilOffset = 0x2A3
 at VersionedPluginProxyStepBase.Execute(PipelineExecutionContext context)  ilOffset = 0x65
 at Pipeline.Execute(PipelineExecutionContext context)  ilOffset = 0x65
 at MessageProcessor.Execute(PipelineExecutionContext context)  ilOffset = 0x1FB
 at InternalMessageDispatcher.Execute(PipelineExecutionContext context)  ilOffset = 0xE4
 at ExtensiblePlatformMessageDispatcher.Execute(PipelineExecutionContext pluginContext)  ilOffset = 0x0
 at ExtensiblePlatformMessageDispatcher.CreateWithInvocationSource(BusinessEntity entity, Int32 invocationSource, ExecutionContext context)  ilOffset = 0xB4
 at BusinessProcessObject.Create(IBusinessEntity entity, ExecutionContext context)  ilOffset = 0x54
 at GenericActivityServiceBase.Create(IBusinessEntity entity, ExecutionContext context)  ilOffset = 0xA1
 at CommunicationActivityServiceBase.Create(IBusinessEntity entity, ExecutionContext context)  ilOffset = 0x1A
 at EmailService.SetRecipientsAddAdditionalAttributeAndCreate(AddressEntry[][] allResolvedAddressEntries, Email email, Entity emailDeltaEntity, String traceSubject, ExecutionContext context)  ilOffset = 0x1DA
 at EmailService.Deliver(Boolean userPromote, Guid emailId, String messageId, String subject, String from, String to, String cc, String bcc, DateTime receivedOn, String submittedBy, String importance, String body, BusinessEntityCollection attachments, Guid campaignResponseId, Entity emailDeltaEntity, ExecutionContext context, Boolean validateBeforeDeliver)  ilOffset = 0x524
 at EmailService.DeliverIncoming(String messageId, String subject, String from, String to, String cc, String bcc, DateTime receivedOn, String submittedBy, String importance, String body, BusinessEntityCollection attachments, Entity extraProperties, Boolean validateBeforeCreate, ExecutionContext context)  ilOffset = 0xB8
 at RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)  ilOffset = 0xFFFFFFFF
 at RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)  ilOffset = 0x25
 at RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)  ilOffset = 0xCF
 at LogicalMethodInfo.Invoke(Object target, Object[] values)  ilOffset = 0x4F
 at InternalOperationPlugin.Execute(IServiceProvider serviceProvider)  ilOffset = 0x57
 at V5PluginProxyStep.ExecuteInternal(PipelineExecutionContext context)  ilOffset = 0x200
 at VersionedPluginProxyStepBase.Execute(PipelineExecutionContext context)  ilOffset = 0x65
 at Pipeline.Execute(PipelineExecutionContext context)  ilOffset = 0x65
 at MessageProcessor.Execute(PipelineExecutionContext context)  ilOffset = 0x1C5
 at InternalMessageDispatcher.Execute(PipelineExecutionContext context)  ilOffset = 0xE4
 at ExternalMessageDispatcher.ExecuteInternal(IInProcessOrganizationServiceFactory serviceFactory, IPlatformMessageDispatcherFactory dispatcherFactory, String messageName, String requestName, Int32 primaryObjectTypeCode, Int32 secondaryObjectTypeCode, ParameterCollection fields, CorrelationToken correlationToken, CallerOriginToken originToken, UserAuth userAuth, Guid callerId, Guid transactionContextId, Int32 invocationSource, Nullable`1 requestId, Version endpointVersion)  ilOffset = 0x16E
 at OrganizationSdkServiceInternal.ExecuteRequest(OrganizationRequest request, CorrelationToken correlationToken, CallerOriginToken callerOriginToken, WebServiceType serviceType, UserAuth userAuth, Guid targetUserId, OrganizationContext context, Boolean returnResponse, Boolean checkAdminMode)  ilOffset = 0x1F1
 at OrganizationSdkServiceInternal.ExecuteRequest(OrganizationRequest request, CorrelationToken correlationToken, CallerOriginToken callerOriginToken, WebServiceType serviceType, Boolean checkAdminMode)  ilOffset = 0x2D
 at OrganizationSdkServiceInternal.Execute(OrganizationRequest request, CorrelationToken correlationToken, CallerOriginToken callerOriginToken, WebServiceType serviceType, Boolean checkAdminMode)  ilOffset = 0x26
 at InprocessServiceProxy.ExecuteCore(OrganizationRequest request)  ilOffset = 0x34
 at IncomingEmailProviderBase.DeliverMessageInternal(EmailMessage emailMessage, Boolean validateBeforeCreate)  ilOffset = 0x1B7
 at IncomingEmailProviderBase.DeliverMessage(EmailMessage emailMessage, Boolean validateBeforeCreate)  ilOffset = 0x19
 at GetItemsStep.ProcessResponse()  ilOffset = 0x5AB
 at ExchangeIncomingEmailProviderStep.EndOperation()  ilOffset = 0xFC
 at ExchangeIncomingEmailProvider.ReceiveEmails()  ilOffset = 0x92
 at IncomingActivityProviderBase.Run()  ilOffset = 0x42
 at MailboxEmailOperation.PerformOperation()  ilOffset = 0x2C
 at MailboxOperationBase`1.Execute()  ilOffset = 0xAA
 at MailboxProcessingOperation.PerformOperation()  ilOffset = 0x1A
 at MailboxOperationBase`1.Execute()  ilOffset = 0xAA
 at MailboxOperationCommand.InternalExecute(MailboxAsyncEvent asyncEvent)  ilOffset = 0x64
 at AsyncCallbackHandler`2.ResumeExecution(IAsyncEvent asyncEvent)  ilOffset = 0x56
 at AsyncEventExecutionManager`2.ExecuteHandler(IAsyncEventHandlerFactory handlerFactory)  ilOffset = 0x8A
 at PoolHandler.ProcessAsyncEvent(IAsyncEventExecutionManager asyncEventExecutionManager)  ilOffset = 0x144
 at AsyncEventOperation.<.ctor>b__2(IServiceOperation operation)  ilOffset = 0x0
 at FaultToleranceBehavior.Execute(ServiceOperationAction operation, IServiceOperation operationParameter)  ilOffset = 0x18
 at MonitoredOperation.Execute()  ilOffset = 0xD
 at ThreadPoolQueueExecutionEngine.InvokeNextOperationInThreadPool(Object state)  ilOffset = 0xC
 at ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)  ilOffset = 0x70
 at ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)  ilOffset = 0x4
 at QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()  ilOffset = 0x30
 at ThreadPoolWorkQueue.Dispatch()  ilOffset = 0xA3
>Crm Exception: Message: Expected non-empty Guid., ErrorCode: -2147220989, InnerException: System.ArgumentException: Expected non-empty Guid.
Parameter name: id
[2015-05-15 15:33:45.304] Process:CrmAsyncService |Organization:00000000-0000-0000-0000-000000000000 |Thread:   26 |Category: Platform |User: 00000000-0000-0000-0000-000000000000 |Level: Error |ReqId: cf367175-12a2-4719-a5af-f53f7f254399 | ExceptionConverter.ConvertToFault  ilOffset = 0x69
>UNEXPECTED: no fault?

My first thought was that the stack trace was really long… and then there was some mentioning of User with Guid 00000000-0000-0000-0000-000000000000.

Another kind of queue ticket

As the Server Side Sync has something of a rep for not being the most reliable guy in Town, I wasn’t really sure if this error really was an error that we had caused or if there was something wrong with the server side sync. As my perception was that we hadn’t really changed any thing, I started with the recreating the mailbox, recreated the server profile, ran test and enable and that worked, but when I tried sending my own emails, from different email accounts, as you never know if there is something spooking it with a specific email address, I still never got the queue to read any emails, not even the test emails created by CRM. I was starting to get a bit short on hair by this time, so I resorted to my classical troubleshooting method of backing off as far as possible and seeing if things still were broken, and I hence deactivated all workflows and plugins that were associated to email and Queue item, and gave the user full access and I was once again awarded with this method, emails started arriving in the queue in CRM.

I then revoked admin access and started re-enabling each single workflow/plugin until thing started
breaking again. This turned out to be a painstaking process as the only way I found to test this was to send an email to CRM and then wait for about 5 minutes until it was/or was not pulled into the queue.
It took me more or less a days work just to find the two workflows that were causing the problems.

The actual problems in them selves, arn’t really that interesting, it turned out that the error message above was actually quite correct, If you try to set a null-user to an owner field, CRM will respond like this.

What I did notice however was that the workflows that we were using on “queue item” in this case were synchronous (yes, yes, we do have good reasons for it, I hope you do to, Otherwise, please choose asynchronous/background workflows) and as they broke, CRM did two interesting facts that are not very desirable and noteworthy:

  • The mailbox was shut down for further email processing. It had to be “Test and Enabled” to start receiving emails again. I have yet to see a good way to do this in an automated fashion and to be able to monitor the mailboxes to see if they are working as expected.
  • The email that was the cause of the breaking of the workflow, got lost. Even after the mailbox got reactivated, it had marked this email as read and did not read it into CRM despite the fact that it was neither in the queue nor as a lone activity which might have been the case if the Creation of the queue item broke. But obviously the entire process is handled atomically.

Based on this I have the following recommendations:

  1. If possible try to avoid an synchronous logic on receiving email or Creation of queue items. Try to move logic to asynchronous as this will cause errors to be non-fatal, like the two types of effects that are hard to handle above.
  2. If you do need to have synchronous logic on the receiving of emails or Creation of queue items try to add conditions in the beginning that will ensure that the logic only runs in those very specific cases where they are supposed to. Be aware of the fact that queue items can be associated to many different entities, so don’t rely on the fact that they should be only connected to something specific. At least take into consideration the fact that they will be connected to emails at some Point in time.
  3. I read an interesting fact today, that the activation of business rules Controls in which order they will be executed. If you are using CRM 2015 with server side (Entity) scope, this could probably cause similar issues if the activation is done haphazardly and causes the execution to be done in a way that can cause a bug and break processing of the queue item or email. I havn’t checked if these are run pre- or poststate but I think that depends on the action.
  4. If you get errors, especially when using CRM Online, try to replicate the error in the sandbox Environment and then either move backwards to a Point where it starts working again, or deactive Everything related to emails and queue items, make sure it works, and move forward. When you know where the error is, finding what the error is, is so much easier.

We also found a related bug in CRM today, if you try to create an asynchronous workflow that triggers on a field which has field based rights activated on it, it will break, with some weird error message about not being able to change domain for the user. The simple fix for that is to make the workflow synchronous and to make it run as the user that activates it, that worked. We didn’t try to run it as the user who created the workflow, as that is the same logic that is used in asynchronous workflows, I wasn’t too keen on trying something that was more likely to fail. If you have tried it, please leave a comment below.

I hope to see a lot of people tomorrow on the 19:th at the Swedish CRM UG Meeting!

Gustaf Westerlund
MVP, Founder and CTO at CRM-konsulterna AB
www.crmkonsulterna.se