by gustaf | Feb 25, 2018
Related to my last post, on working with the API quickly, Microsoft have now released official
documentation that they will, effectivly March the 19:th start limiting the number of API-calls per instance that is allowed to stop what is called “noisy neighbour” problems.
First of all, read the full article here: https://docs.microsoft.com/en-us/dynamics365/customer-engagement/developer/api-limits
Let’s break this down a bit, 60 000 calls per 5 minutes, translates to about 200 calls per second. If you break this, you will start getting exceptions, until the 5 minute period has ended. You are expected to back off, and essentialy handle this. That is the short version. Read the full article for more details.
Update: George Doubinski, a friend of mine and one of the brains of CRM Tip of the Day made me aware of the fact that the limit is per user. I will update the article below on what this means.
What does this mean? Is this a problem?
For most organizations, no, at least that I work with, I not even close to breaking this. If they are using some integration tools like Kingswaysoft or other tools which enable multithreaded integrations, but generally do not need that kind of data throughput then you might temporarily be shut down, but it should self heal after some time, as after each 5 minute time span, you will get another 60 000 requests. That could probably quite easily be fixed by checking the settings of the integration tool. Update: Also, if you integrate each system using a separate accont, you do not risk one system temporarily blocking many other systems from integrating to Dynamics 365. If you are using normal users, this will of course entail a certain license cost, why I generally recommend using app users for integrations, if possible. And after this, you should have one app user for each integrating system.
However, there are some organizations where I forsee issues, and these are organizations which have combinations of any of the following criteria:
- Third party products which, like Marketing Automation, (ClickDimensions, FreshRelevance, SalesForce Marketing Cloud) which have not had time, or got this in their scope yet, and have large amounts of data that they integrate into Dynamics 365. Update: Especially if the user they are using to integrate, the service user, is a normal user, either used by a normal user, or shared with integrations with other systems.
- Legacy Code that has been upgrade to new SDK but uses inefficient architecture – can for example have issues with using ExecuteMultiple which in the article above is described as the recommended best practice. Typically for the reason that the architecure of the code, would require major rewriting to allow for ExecuteMultiple. Update: In this case I strongly recommend looking at using a dedicated user for this specific integration, to isolate any limiations set on that user.
- Organizations with multiple heavy integrations to Dynamics 365. Will be hard to control that the sum total does not exceed 60k per second, and handle back-off in a controlled way. The only reasonable way would probably be to rewrite the integrations to use a proxy or queue instead like Azure Service Bus Queues to integrate and have a single integration interface. Probably a lot easier to write in a blog article than to do in real life. Update: This was an incorrection deduction from my part, as it is not based on the sum total, but on the sum per user, this is not a risk unless many integrations use the same user for integration which I do not recommend.
- Organizations with complex heavy integrations with thousands of lines of integration code that need to be redesigned, rewritten, tested and deployed before March 19:th. And there is no way to test it as there is no TAP/Beta program for this “Feature”. Update: This is still very relevant. Even such a small change as changing the integrating user for an external system should be thoroughly tested and for larger implementations that can be hard to do before March 19.
Example
I see is a typical B2C organization running Dynamics 365 with a marketing automation addon with email tracking and webtracking. They also have a very time critical integration of orders to be able to handle any incidents. Even if the order integration in itself does not reach the limits, it is not unforseeable that a mailblast, especially a good mail blast, to which many customers read the emails click the links, go their site, check their offers and start ordering, would cause a surge of traffic on the Marketing Automation integration – Dynamics 365 API. This of course depends on the settings of this, but perhaps it is critical that all events be tracked to Dynamics. With a mailblast to let’s say 1 Million recipients, quickly hitting the 60 k/5 min limit would happen. When this happens, this would also block all orders from going to Dynamics, causing an effective stop for working with any new incidents in the system.
Update: This is, of course, only relevant if both systems are integrating using the same user. Don’t. However, the marketing automation system above, would hit the limit fast anyway and if the supplier of this system didn’t have time to update their product/service then it would handle this incorrectly. I recommend checking integrating systems and try to turn down the verbosity of what they are writing to Dynamics 365. Then after March 18 when we see how this falls out in detail, you can test a more verbose setting in a test environment, and then see how that falls out.
Summary
For small and medium companies with low complexity working mainly with B2B. I don’t see that much of a problem. Larger companies with complex integrations, large databases, integrations to webtracking, email tracking which often will be B2C companies which have higher levels of automation and larger databases of customers, will probably have larger problems with this and need to start think about this right now.
We need to come back to this subject post March 19, to see how this will really work. But I think the real problem will be for the larger orgs with many and heavy integrations.
I would be really glad to hear your views on this like I got Georges’.
Gustaf Westerlund
MVP, Founder and Principal Consultant at CRM-konsulterna AB
www.crmkonsulterna.se
by Gustaf Westerlund | Oct 2, 2016
As many of you probably have heard Dynamics CRM will as of Nov 1 2016 be part of Dynamics 365 in a commendable push from Microsoft to bring the Dynamics products closer together. This is partly a branding thing as the products themselves, as Dynamics CRM, will still be the same product from a technical perspective, at least in the Enterprise Edition, (little is known of the Business Edition) and partly a technical concept as they are introducing something called the Common Data Model which is canonical data model (unified data model) to which all “Apps” are integrated to automatically. This is what I wanted to discuss some.
This all sounds great, and I will admit that I havn’t had time to fiddle with it yet, but I have had the pleasure of working with quite a few integration projects between CRM and ERP. And that is not easy, even if you do have a ready made point-to-point integration. So I would just like to make a few points that I hope you do consider before switching it on and hoping it will solve all your issues.
1. Addresses in ERP and CRM are typically not the same. In ERP the addresses that are needed can typically be invoicing address and delivery address, while in CRM the most important addresses are visiting address and postal address. If you naivly presume these to be the same the effects can be dramatic and sometime even catastrophic. I friend of mine, Peter Björkmarker, told me a story of a company integrated just like this, and as CRM was set as the customer data master, it overwrote all invoice addresses in the ERP system with visiting addresses. Next month, all invoices which were sent out were automatically, without anyone noticing sent out to the wrong address, hence nobody paid them. The company got into an accute cashflow problem and almost filed for bankcrupcy. So this is no joke.
2. Ready built integration are usually on a technical level, but you expect it to work on a business level. Integration technology is usually about moving data, but just having the data in the other system doesn’t always DO anything. An example is if you have a boolean field on the customer in the ERP where the financial people can block the customer from further business if they havn’t payed their invoices. So you integrate this field to CRM and can now see it on the account form. But without any additional logic in CRM it will still be possible to create opportunities, quotes and orders. Maybe not what you would like.
3. Data structures are different. My colleague Rickard Norström, whose blog you can find in the list to the right, was part of a CRM project which integrated to Dyn AX. One of their issues was the AX address data structure. An address record in AX can be used by both an account and a contact, And I think even several accounts. When this address is changed, of course this is seen in all affected places. As this is very different from the customeraddress built in logic in CRM they had to create their own new address entity to solve this. Other typical areas where there are large differences are in the logic of setting prices on opps/quotes/order. As you can expect, a system like AX with MPC and many other deep links into costs can of course use that as a base for pricing, something that is very hard for CRM. It also has more complex or just different ways of handling pricelists. I was working with an iScala integration and iScala for instance can have a current price in a pricelist and a comming price with a specified date on which the new price will be enforced. However, no event in the system will trigger at that time. Customer specific pricelists are also something that occur, not advisable but existing especially for larger customer accounts.
4. Centralized integration architecture. The Common Data Model sounds great but it only handles two of the components in the Business system infrastructure. If you for instance are a Telco the amount of business systems will be a lot more, billing systems, provisioning system, logistic systems, product configurators, etc. Banks are also complex worlds. Many of these have tried to consolidate their integrations to integration hub technologies like WebSphere or BizTalk and if done properly they will of course have their company defined canonical data model. It would be interesting to see the story of how the Common Data Model works together with this. It probably can by shuffeling data using Logic Apps to and from the CDM, but in essence you will have two hubs to orcestrate. Another option is of course to use the CDM as the central hub for all information, as long as that is extendable and doable. So, from this perspective, the main issue is probably, if we have 8 systems connected to our existing integration hub, is it plausible to use the CDM or do we manually integrate anyway directly to each application?
To conclude, I think the CDM will be a good tool but I will keep my expectation to a reasonable level and I recommend you do this too. Do not think it will make your highly customized CRM and AX automatically integrate all data and make it work from a business perspective, that would simply be too increadible. If they manage that, I will buy the entire team building CDM a beer (or similar).
Gustaf Westerlund
MVP, Founder and CTO at CRM-konsulterna AB
www.crmkonsulterna.se
by Gustaf Westerlund | Oct 20, 2009
The NDA of the new version of SharePoint 2010 has been lifted and hence a lot of new functionality is out!
You can read a lot about it on the SharePoint Team Blog here: http://blogs.msdn.com/sharepoint/archive/2009/10/19/sharepoint-2010.aspx
From a CRM perspective, this is very interesting. The new version of CRM is rumored to have a much tighter integration with SharePoint, so from that perspective it is almost essential to learn a bit more about SharePoint.
There are some areas of SharePoint that are of special interest from a CRM perspective:
- Enhance BI functionality including performance Point server now included in the new SharePoint
- Business Data Catalog enhanced – an excellent tool to work with in relations to CRM
- Enhanced Document Management – one of the weaker parts of Dynamics CRM 4 is the built in document management so integration with SharePoint in regards to document management is often used.
- Enhanced API and development capabilities. When integration with SharePoint development is often required to create the necessary functionality. The SharePoint API is currently ok but compared to the excellent API of Dynamics CRM, it can do with an upgrade. Hopefully to the mark set by Dynamics CRM.
So that is all good news.
However, there is still one major issue from a CRM – SharePoint integration perspective that Microsoft REALLY need to address and that is the licensing issues of SharePoint users working in a very slight way with CRM data by, for instance, looking at data from a data warehouse in SharePoint with drill-down functionality. Today, all users with access to this need to have a full read only CRM license. Similarly, if there is a webpart used in SharePoint to create leads in CRM, but with no other need for access, a full CRM CAL is required for all users of the Intranet since they are employees and hence cannot use the external connector license. This is preposterous! Please Microsoft, we really like CRM, and we want to pay our dues, just make it fair!
Gustaf Westerlund
Microsoft Dynamics CRM Chief Architect
Logica
www.logica.com
by Gustaf Westerlund | Jan 24, 2008
As I have been writing, I am currently involved in a rather large integration project. For those of you who don’t have experience of working with larger integration projects I would like to just share my view of how to set up the architecture connecting Microsoft Dynamcis CRM with the integration engine (BizTalk, WebSphere, Sonic and more). Bellow, I will refer to the Integration Engine, simply as “the Bus”
First of all, communication with the bus can be divided into two different classes:
- CRM Initiated Communication
- Bus initiated communication
In the picture to the right, these two are represented with a connection each (connection 1 = CRM Initiated Communication, connection 2 = bus initiated communication)
CRM initiated communication might be considered the simplest, since it mainly controlled from CRM. It can be code that runs from aspx-pages, callouts/plug-ins or workflow addons. It simply consumes the webservice at the CRM adapter on the bus.
Bus initiated communication is set up by the bus consuming a customly created web service. This “proxy” webservice translates from integrationmessages and datastructures to CRM native communication.
Integration is then definied by defining messages that uses either connection 1 or 2. Each message has two parts, Request and Response, and the response is the syncronous answer to the request. Assyncronous messages should be set up as two individual messages one for each adapter.
So, why create a proxy web service? Why cannot the bus connect directly to the CRM web service? The reason for this is usually that the communication is defined using messages (as described above) and the person responsible for the CRM Adapter on the bus cannot be expected to have any CRM knowledge in general or specificly CRM Webservice knowledge, hence that is the CRM developers job.
These are some of my experiences and reflections on integration with a centralized integration architecture. If you have any comments or you have some other method that you use, please leave a comment, so that we can discuss it further.
Gustaf Westerlund
Microsoft Dynamics CRM Consultant
WM-Data/Logica CMG
www.logicacmg.com
by Gustaf Westerlund | Jun 17, 2007
Integrating a map service into MS CRM with mesh-up technology is not very hard and can add very strong functionality to the users/customer. I havn’t worked in a project yet that wanted this functionality which is the reason why I havn’t blogged about it.
However, today I had a look at the CRM Team blog and found that Ayaz Ahmad, a CRM MVP has written a blog entry on how to make a simple CRM integration with Microsoft Mappoint.
Please have a look here if you are interested:
http://blogs.msdn.com/crm/archive/2007/06/14/microsoft-dynamics-crm-and-mappoint-integration-a-picture-is-worth-a-thousand-words.aspx
I am not sure how well MapPoint works in Europe/Northern Europe but I don’t think integration with Google Earth (sorry MS, I know you hate them 🙂 or other similar services (www.hitta.se, www.eniro.se) is very much harder as long as the do have a wsdl-webservice that can easily be consumed, or can easily be configured using URL-parameters.
Anyway, start off with MapPoint, if it doesn’t go all the way for you, try working with some other Map Service supplier. There might also be some nice ActiveX-based map-service if a better map GUI is needed.
Gustaf Westerlund
CRM and SharePoint Consultant
Humandata AB
www.humandata.se
Recent Comments