API per user limits – The good, the bad and the ugly

API per user limits – The good, the bad and the ugly

(Updated) Microsoft recently released some throttling that have been causing some stir in the community, especially since the latest throttle, the concurrency throttling, was not very openly announced, some partners and customers were hit rather hard by it as it affected their abilities to manage large dataloads in the system.

Now Microsoft have announce another API based limitation which is based on the users and the type of licenses the have. You can read some about it here if you like. This article will discuss what this means and my personal view of the good, the bad and the ugly of it.

First of all we need to understand what it is. It is a API limit that will be set per user and based on the type of license that the user is allocated. The highest is if you have a Dynamics 365 App user license, like Sales, Customer Service or similar, which will give you 20 000 requests per 24 hours. The lowest is a Power App – Per App license which will give you 1 000 requests per 24 hours. Note that these are connected to the user and not summed/aggregated to the instance level (allthough I would think that would be a good idea). Well, really, the lowest of them all are Application, Non-interactive or admin-users that don’t use a license as these will be allocated 0.

I have not seen any UI for this yet, so I don’t know how this will look, but what the page is saying is that API-calls can be reallocated from normal users to application users/non-interactive users. (UPDATE – See update at the bottom regarding this, thank you observant readers!) Not sure if it will also be possible to reallocate API-calls between normal user and another normal user.

There will also be an additional SKU for buying 10 000 additional API calls per day that can be allocated to a user.

 

The Good

What is good about this then you might ask? Well, I think it is fair. Large customers pay a lot of money for their instances and usually use it a lot with a lot of integrations. It is only fair that they are allowed to use the API:s more than a small customer who has created some super duper application that blasts Dynamics with massive amounts of calls. The small customer can still do this, but they just have to pay a bit extra for those API-calls if they arn’t covering that with their users.
I also hope that this might enable Microsoft to relax the currently rather tight throttling on the API:s a bit.

According the the licensing documentation in general, existing customers will not be hit by this until October 2020, in other words, more than a year from now. This will hence probably only now affect new customers.

The bad

This implementation certainly has some bad parts. The most obvious is the too stringent connection to users which makes it weird. I don’t know how this will be managed in the UI but let’s say we have an instance with 500 users mixed Sales Enterprise, Customer Service Professional and Team Member. We also have 10 application users that are used for Portals, Forms Pro and custom integrations to many other systems. Each integration using a separate integration user to reduce the attack area in the unlikely event of a hacker attack. So what we will need to do is to first figure out how much API-usage we are using for all the normal users (for instance via PCF:s, Flows, Plugins, Workflows etc) and all the integration application users. Currently the https://admin.powerplatform.microsoft.com does not give us this granularity. There are indications but in this case one would need deep granualar data, preferably with trend analysis.

Another part of this that could be done better is the “buying addional API-calls”. Why not just adapt the method used in Azure? In other words, you pay as you go. With this current method, you have to know beforehand how much a particular user will use and if you overshoot the user will be shut down causing unnecessary support costs for customers, partners and Microsoft.

I also wonder how this practically is going to be handled? Are admins going to go into each of the 500 user records, reduce the API-calls allocated and move to Application users? If the admin moves all calls, which effectivly will stop plugins, workflows, javascripts with server calls etc how will the error handling of that look?

The Ugly

What is really the difference between something bad and something ugly? I would say that something bad is a design decision that we might dislike or might be disadvantage to the customers, it requires some sort of conscious perspective. Ugly on the other hand is the parts where where, in this case, Microsoft just have forgotten to think about something or neglected perspectives which causes issues for partners or customers. Based on this, I would say that the following are the bad aspects of this;

Timing

Again Microsoft are rolling out a change with a rather short timeframe. They probably feel that a month or two of notice by publishing the article above is notice enough, but they have to realize that many customers cannot act that fast. If you are a small customer with extensive use of Dynamics, for instance if you are using Dynamics 365 in a B2C aspect with a Marketing Automation integration and you are targeting millions of customers with sendouts and hits on your webpage being mirrored to your Dynamics all the time, this will cause some hefty API traffic. And your org might not be very big if you are totally e-commerce oriented.

Maybe only new customers, for now

Lastly I really hope that it is true that the API limitation will not affect current customers, it is not very clear and hence we are left in the dark again. If there is a problem with application users etc not being able to log in, I hope Microsoft support will be ready for the storm that will hit them.

On the other hand, new customers might have tested the system, evaluated the costs and are now faced with this. Not sure that will be optimal either, there is risk of loosing a customer or two there.

Communication

As this is a rather drastic change and may be viewed as a “breaking change” if not the one year grace period mentioned in the licensing in general applies to this. No matter, this should have been communicated very clearly months ahead to remove any kind of doubt from partners and customers. Both via blogs, emails to admins of organizations using Application users/non-interactive users as this should be easy to figure out via telemetry. Currently no one knows exactly when this will hit them/their customers or how they are to manage it.

 

This is generally very unclear. I shouldn’t have to write an article like this, speculating about what is or isn’t going to happen. If I have problems figuring this out, being an MVP, customers are probably very much in the dark, both existing and new.

 

Conclusion

In conclusion I think this is a good idea that got rushed. It should have been passed through a couple of more hoops before being launched to get the right feedback. The main things that I think Microsoft should change before rolling this out that, from my perspective, still give the same effect, are:

  1. Aggregate all API-Calls that are counted to a per instance level. It will make it easier to manage, stop the breaking change and make it easier to understand.
  2. Enable admins to add a per-use, after the fact, payment option, (like Azure) for any additional API-calls.

     

    If this is going to be useful or not also is very dependent on the fact that we can reallocate a lot of the API-calls from users to the integration users. For instance, I have a B2C customer with 1M+ API calls per 24/h and if it will not be possible to take the sum of hundreds of users and allocate those to the application users we are using, then this will be a very hurtful change.

    In the meantime, I do recommend that you keep a close eye to what is going on within this area as it will most likely affect you if you are running any application accounts, which you probably are, like Dynamics Portal, Forms Pro, Voice of the Customer and many more. If you go into the list of users and change view to “Application users” (or whatever it might be called in your language) you will see the list. I think Micrsoft will make some changes, or some announcements to this before October 1. Let’s see what.

Update 2019-09-04

There has been some chatter going around regarding this and do note the comments below which include interesting links and good thoughts. There are some additional points that need to be pointed out. Instead of changing the original article I will continue to add updates like these.

Normal UI usage will count

Initially I did not think that normal UI usage would count towards the API request calls. With “normal” in this case, as an old Dynamics 365/CRM geek, I of course mean a model driven App, but the same also goes for canvas Apps or actually any use of the CDS, what so ever. What this will mean when a user runs out of API requests, will be interesting to see. How many requests are used when the application is used, of course depends a lot on what you do. If you switch on F12 in Chrome you can check the network traffic and see for yourself.

Batching will be your friend

Using batching will from now on not only be a general best practice but also make you save money. If you use tools like Kingswaysoft this is easy to configure, to make sure that you have large batches when for instance doing CUD calls. When writing code directly, you will need to understand how to do this directly. Note that sometimes this will require entire rewrites of the code. I have seen programs off the shore of Orion that you wouldn’t believe with tons of single queries instead of one single call. Most often written by devs who have no or very little experience of writing code towards Dynamics 365/CDS.

Unclear if possible to move API-calls

As several people here and on Twitter have commented, it is probably incorrect to interpret that API:s can be moved from normal users to application users and non-interactive users. This will cause major headaches for some customers which will be struck with lots of additonal costs. Costs that are not very welcome as the per GB cost recently increased 800% hurting especially the larger customers with massive integrations and extensive use of the system. I do, for instance, have a customer that exceeds 1M requests per day 365 days a year. This would require them to buy over 100 addon 10k API requests SKU:s, despite the fact that their 500 users gives them a total of over 5M requests per day, something they will not be using through the UI unless someone is drinking very large amounts of coffee. – NEW Update: This was an incorrect interpretation. You cannot reallocate API calls from normal users. 

The price is here

The price for the 10k/24h SKU will be $50/month. This means that for a customer like mine having major integrations causing around 1M API-calls per day, this would cost an additional per month $5 000 or yearly $60 000. I sincerely hope they will relax the throttling to make it worth it. If/when they do, I will read my Macciavelli again.

 

Update 2019-09-05

First of all I will write a new blog article on this, when the dust settles and we know what is going on. Currently there are quite a lot of unknowns and I wouldn’t be surprised if Microsoft announced a thing or two soon. I have been told that the FAQ will be updated in a couple of days.

Batching – again

There were some discussions on if batching actually were going to be useful in this case or not. I have now gotten confirmed that a batched request will be considered as one (1) call. This is both for batched Creates/Updates/Deletes and Queries of multiple records (that would be very strange if it wasn’t one record, but I had to ask).

Data Export Service etc.

Data Export Service and other services that run under the system account will not count towards the API request. This is good news as this opens up for many users to be able to use this method to offload the API:s for reads.

What is the competition up to

I checked to see how SFDC are handling this and as far as I can see they have a similar setup as can be read here:

https://developer.salesforce.com/docs/atlas.en-us.salesforce_app_limits_cheatsheet.meta/salesforce_app_limits_cheatsheet/salesforce_app_limits_platform_api.htm

and here

https://support.geckoboard.com/hc/en-us/articles/216804218-I-ve-hit-my-Salesforce-API-request-limit

I am no expert on their licensing model, but I think it is good to know that this isn’t just a PowerPlatform thing. However, there are some distinct differences:

  1. The API calls are not counted for normal browser/client usage. Only “real” API calls.
  2. They have real enforcement blocking an entire instance/org if they overshoot
  3. All API:s per user license are summed up to the org level

Microsoft Addon apps will include request

If you buy Dynamics Portals, this will include some additional licenses. The same goes for Forms Pro. Hence there should be some default API request assignment to those application users that are installed. I do wonder if it would be financially beneficial to piggyback on those application users? There is also no current method for ISV:s to bundle API-requests into their product if they install an application user upon installation.

CSP / Distributor silence

We have still heard nothing of the 10k addon SKU from any distrubutor, EA or CSP. It will be interesting to see if it will reach the entire distribution chain by October 1 when customers will start being notified that they are in violation (new customers). 

Uninstalling Employee Self Service Portal – step by step

Uninstalling Employee Self Service Portal – step by step

A customer of involontarily gotten a Dynamics 365 Employee portal installed. I had to remove it. It has 21 (!?) solutions which are interdependent. This is how I fixed it.

As always, I start off by trying to find if anyone else has run into the same issue, almost. This guy had with another portal type:
https://community.dynamics.com/crm/b/dynamicscrmbestpractices/archive/2017/10/14/dynamics-365-uninstalling-microsoft-portals-steps 

And then my Portal buddy Nick Doelman did some heavy lifting (he actually does!) and sent me this link. It did turn out to be a bit old though. Still useful, but the solutions described in it are not entirely accurate.

https://community.dynamics.com/365/b/dynamics365portalssupport/archive/2017/02/27/portal-troubleshooting-part-three-uninstalling-portal-solutions 

Sitemap

First of all, before you do anything else, start by removing the following things from the sitemap (unless you havn’t added more things pointing to Portal components)

1. The entire Portal “Area”
2. The SubArea and Group under Settings for Portal Settings.

Solution removal

So which solutions are we to uninstall? I will take it from the top, in the order I uninstalled them.

1 ESSPortal
2 BaseHtmlEditor_portal
3 MicrosoftForumsWorkflows
4 MicrosoftForums
5 Feedback
6 KnowledgeManagement

When removing KnowledgeManagement I got this error:
The entity with ObjectTypeCode = 10460 was not found in the MetadataCache
Tried different way, after some time, just refreshed and it was gone. Maybe this should be removed earlier. Not entirely sure which entity this was, as it was removed and I didn’t save a metadata reference before starting.

7 MicrosoftAzureStorage
8 MicrosoftBingMapsHelper
9 CustomerService
10 WebNotification
11 MicrosoftGetRecordIDWokrflowHelper
12 MicrosoftIdentity
There are two dialogs that 
Process/Dialog – Change Password, Removed all Steps. Save & Close
Reset Security Stamp – Removed all steps. Save & Close

13 MicrosoftIdentityWorkflows
14 MicrosoftIdentitySystemWorkflows
15 MicrosoftCrmPortalBaseWorkflows
16 MicrosoftCrmPortalBaseSystemWorkflows
17 Portal Timeline
18 MicrosoftWebForms

When removing this solution I also got the same weird error: The entity with ObjectTypeCode = 10439 was not found in the MetadataCache. 

What I found was that it was actually the sitemap that I hadn’t cleaned up from Portal things. This is why I recommend removing all portal related entries from all Sitemaps before starting to remove any solutions.

19 MicrosoftCrmPortalBase
20 MicrosoftCrmPortalDependencies
21 Portal Privacy Extensions

Admin portal for managing your tickets

Admin portal for managing your tickets

In case you haven’t seen it, the Dynamics 365 Admin Portal it is a great place to create and manage you Dynamics 365 tickets.

You get to it using this URL: https://admin.dynamics.com/

I haven’t found a link to the admin portal from the instance manger or any other place in O365 or Dyn365 yet so I have just created a bookmark for it, so I suggest you do the same.

Also, as I am a frequent Microsoft Support user, and often have customers with many instances, I have tried to consolidate many instances into one ticket. They don’t want that. Better to create one ticket for each instance. (probably gets someone a higher salary too :)) However, I did notice that there seems to be some duplicate detection going on, so try to avoid using the same “Issue Summary” for several tickets – as that will just make the later ones disappear without any error message (hint Microsoft, please fix this! – at least give a decent error message).

In order to create tickets, you need to be either O365 Global Admin or Dynamics 365 Service Administrator. There might be some other admin as well that has the right to create tickets, but I don’t think so.

Gustaf Westerlund
MVP, Founder and Principal Consultant at CRM-konsulterna AB
www.crmkonsulterna.se

Dynamics 365 Support – your best friends when you have Dynamics 365 Online

Dynamics 365 Support – your best friends when you have Dynamics 365 Online

If you are using Dynamics 365 Online, Dynamics 365 Support is your IT-department for your Dynamics system and you need to know or start learning how to working with them in a good way. There are several things you should consider.

Registering a support ticket

First of all, registering a support ticket is of course important. This might not be as trivial as it seems, as it depends on which type of license you have. If you have bought your licenses using CSP (Cloud Service Provider) or Select/Volume Licensing then you will most probably be directed to your CSP/Select partner for any support issues. You can of course buy premier support from Microsoft but that might not always be an option for smaller companies. There are some tricks as well if you are stuck with CSP/Select or similar, and will be happy to tell you about these in person if I meet you, but I won’t write about them, as I am afraid it will be “fixed” which would be sad.

If you have bought your licenses directly through the O365 portal, you can also create a ticket using the portal. You need to be a Global Admin or a Dynamics 365 Service Admin to place a ticket. Currently you cannot indicate that the ticket is for Dynamics so you simply write so in the text and the O365 support team will create a new Dynamics 365 ticket for you. I was amazed at how fast the Office 365 support called me after registering a ticket, I think it took less than 1 minute. So I would suggest you choose the option “call me”.

If you have a partner, I strongly suggest that you register them in the Office 365 portal for the Dynamics 365 subscription as Partner of Record (PoR). This will make your company show up in their Microsoft partner portal where they can register a support ticket for you.

Checklist
When registering the ticket with Microsoft, there are some things you have to note:

  1. Be polite. Even if your system is slower than sirup in January (a Swedish saying) or nobody can log in, you will not be getting any help faster by screaming or being rude. I like to quote my grandfather who had a good saying: “Better to be strong in argument and withheld in words, than the other way round.” I have actually gotten written compliments for being nice by the support people, do you think they will go the extra mile for me?
  2. Be very exact with what is you think is wrong. The best way is to formulate it like this: “I did X and expected Y but I got Z”. Also be clear on when you define the case as closed. This is particularly important, and hard when you have performance issues or intermittent errors. But try to be as exact as you can. For example: “When users run Chrome open a Contact (after the contact form already has been loaded for another record) it takes 10 seconds. We expect it to take no more than 2 seconds.”
  3. Only have one issue per ticket. Better to have many tickets open. I have had 4 tickets open for one single customer at the same point in time.
  4. If you think the issue is important, describe it in business impact terms. For example “The bug X affects all sales people in the organization effectivly stopping them from using Dynamics 365. It severly risks our entire Dynamics 365 investment as users look at other solutions and the user adoption drops dramatically and the trust in the system is drastically lowered for every hour this bug is not fixed. We have 4354 sales people in our global sales organization spread out over 34 countries. It also affects managements abilities to manage the company as no pipleline and forecast numbers are produced which could potentially cause sever business damages that cannot easily be fixed within the coming 4-5 years.” – This was just made up 🙂 but it is important to not hold back when doing this, especially if you are Swedish or from other cultures which value humbleness.
  5. Be clear on limits of the error. 
    1. Which Operating system?
    2. Which browsers?
    3. What times? Can you see any clear patterns on when it happens?
    4. Geos?
    5. Security roles?
    6. Types of users affected, AD users federated to O365 or Cloud users?
    7. How long?
    8. Which versions of Dynamics 365 are you running?
    9. Which instances have the issue (the url:s)?
Types of tickets
What types of tickets can be registered? Should you only register a ticket when you want help with something?
Well, no, you can and should register a ticket for many different reasons. As I mentioned at the top of the article it is important to think about Microsoft Support as your Dynamics 365 IT-department. They will help you with a lot of things that your IT-department would have helped you with had they hosted your Dynamics system. The main difference is probably that they do nothing else than host Dynamics 365, every day, year after year… not many IT-departments that can compare with that.
Typical issues they can help out with are:
  • Something seems to have stopped working that used to work, and you didn’t change anything. Like you cannot create an Application User anymore.
  • Something clearly is a bug in Dynamics 365 and you want Microsoft to know about it (they might not know) and you want to let them know how important it is for you (they will surely not know this).
  • You need some information from the Server, like the “Top 10 table Usage”  report from the SQL database.
  • You have performance issues.
  • You need to change the planned upgrades because your team won’t make the set time.
  • You want to set some non-clustered indexes on the database.
  • You are doing a migration of large amounts of data and you will be a bad neighbour for a few days and if possible you would like a little more than the 2 allocated threads in the API.
  • You want a database copy prepared for installation into an on-prem equivalent.
  • You need some detailed information on how the service is managed, like
    • How backups are stored
    • How to restore backups from the 90 day tape backup
    • Where an instance is hosted
  • You would like to move an instance to another datacenter
  • You need to remove a managed solution but it has dependencies that shouldn’t be there…
  • You need to do some task with an instance that you cannot do yourself with the instance manager.
In other words, they will be your superfriend when working with Dynamics 365 and you want to be really nice to your super friend!

Gustaf Westerlund
MVP, Founder and Principal Consultant at CRM-konsulterna AB
www.crmkonsulterna.se

Federated user unable to configure CRM for Outlook

Federated user unable to configure CRM for Outlook

When using federated users in a CRM Online, we encountered an error where accessing the CRM worked fine, but accessing the system using the Outlook client (configuring it) or accessing it via the tablet client did not work. It seems that some additional settings were required on the AD FS server.

We we working with a customer who had set up a CRM Online with federated users to their AD via their AD FS. When accessing this via the web client, it all worked well, but when trying to configure CRM for Outlook we just got an error “An error has occured. Please contact the system administrator”. Our customer’s IT manager started a premier support case as I felt this was most probably a AD FS related issue (there were no problems with the non-federated users).

After a couple of screen sharing sessions we got some great help from Pedro R at Microsoft Support and I learned something new which I thought I’d share with you and that is this;

When using federated users in CRM Online and when using IFD CRM with AD FS you have to enable the Outlook and Tablet authentication to pass through the AD FS, otherwise it is blocked.

There is a Technet article on this which can be found here:  https://technet.microsoft.com/en-us/library/hh699726.aspx#BKMK_WS2012R2

I think this is often something that might be overlooked why I thought sharing it with you would be a good idea. Maybe you’ll find it and save yourself some time!

Gustaf Westerlund
MVP, Founder and CTO at CRM-konsulterna AB
www.crmkonsulterna.se