CI-D Measure dependency on Dataverse

CI-D Measure dependency on Dataverse

When measures are created in CI-D to be used in CI-J there are dependencies created which will stop you from removing the measure. CI-D is not very helpful in telling you how to remove it. I will try to be a bit more helpful.

When trying to remove the Measure this is the error I got:

Error: Detected DataVerse dependencies in Measure: ListMemberships. Please delete these dependencies and merge again.
Request ID: 14056846-b9c1-4ec8-98c2-88778e518b88
Time: 2024-12-05, 11:12:10

My investigation below with the help of Microsoft Support lead me to the understanding according to the image above. As you can see dependency is actually sort of circular, or at least one part of the dependency (dependent component) is at the entityanalytics on the record which points to msdynci_listmemberships, on the other hand (required component) on the table msdynci_listmemberships.

The fix, is to remove the record in entityanalytics. Easiest way in this case, is to just use a browser console where you are logged into the right Dynamics.

To remove the entityanalytics record that is blocking this:

Xrm.WebApi.deleteRecord('entityanalyticsconfig', 'fc105928-3170-ef11-a671-000d3a64e5b5')

After this I was able to remove the measure in CI-D.

Below I will detail the steps I did to understand this.

The first step is to look in the table “entityanalyticsconfig” searching for the name of the Measure. Its name is “msdynci_” + name of the measure. Check out the metadata if you are unsure.

<fetch top="50">
  <entity name="entityanalyticsconfig">
    <filter>
      <condition attribute="parententitylogicalname" operator="eq" value="msdynci_listmemberships" />
    </filter>
  </entity>
</fetch>
{
  "value": [
    {
      "isenabledforadls": true,
      "parententityid": "dd7613fe-806e-ef11-a670-000d3ad82e35",
      "modifiedon": "2024-09-11T11:29:54Z",
      "componentidunique": "807e7e6d-71f5-446a-8009-437cbf5a922a",
      "isenabledfortimeseries": false,
      "versionnumber": 15487568487,
      "parententitylogicalname": "msdynci_listmemberships",
      "entityanalyticsconfigid": "fc105928-3170-ef11-a671-000d3a64e5b5",
      "componentstate": 0,
      "organizationid": "ba0a9fa5-462b-45cc-b603-e9b16acd63c4",
      "solutionid": "fd140aae-4df4-11dd-bd17-0019b9312238"
    }
  ]
}

As you can see above the id is fc105928-3170-ef11-a671-000d3a64e5b5. This is the record I did the removal on above.

To further understand the dependency record for this, let’s dig into that table, searching for a record with this id.

<fetch top="50">
  <entity name="dependency">
    <attribute name="dependencyid" />
    <attribute name="dependencytype" />
    <filter>
      <condition entityname="dcn" attribute="objectid" operator="eq" value="fc105928-3170-ef11-a671-000d3a64e5b5" />
    </filter>
    <link-entity name="dependencynode" from="dependencynodeid" to="dependentcomponentnodeid" link-type="inner" alias="dcn">
      <attribute name="componenttype" />
      <attribute name="objectid" />
      <attribute name="parentid" />
    </link-entity>
    <link-entity name="dependencynode" from="dependencynodeid" to="requiredcomponentnodeid" link-type="inner" alias="rcn">
      <attribute name="componenttype" />
      <attribute name="objectid" />
      <attribute name="parentid" />
    </link-entity>
  </entity>
</fetch>
{
  "value": [
    {
      "rcn.componenttype": 1,
      "rcn.parentid": "00000000-0000-0000-0000-000000000000",
      "dcn.parentid": "00000000-0000-0000-0000-000000000000",
      "dcn.objectid": "fc105928-3170-ef11-a671-000d3a64e5b5",
      "dependencyid": "e46e52db-46a7-4585-8a6a-6ba888a5bd1f",
      "dcn.componenttype": 430,
      "rcn.objectid": "dd7613fe-806e-ef11-a670-000d3ad82e35",
      "dependencytype": 2
    }
  ]
}

The result shows us that the required component id is: dd7613fe-806e-ef11-a670-000d3ad82e35 which is of type: 1.

By checking this table in Microsoft Learn, we can see what this is: (both link and table)

By searching the table called “entity” for 430, we can find what table this represents:

https://learn.microsoft.com/en-us/power-apps/developer/data-platform/webapi/reference/dependency?view=dataverse-latest

ValuesLabel
1Entity
2Attribute
3Relationship
4Attribute Picklist Value
5Attribute Lookup Value
6View Attribute
7Localized Label
8Relationship Extra Condition
9Option Set
10Entity Relationship
11Entity Relationship Role
12Entity Relationship Relationships
13Managed Property
14Entity Key
16Privilege
17PrivilegeObjectTypeCode
20Role
21Role Privilege
22Display String
23Display String Map
24Form
25Organization
26Saved Query
29Workflow
31Report
32Report Entity
33Report Category
34Report Visibility
35Attachment
36Email Template
37Contract Template
38KB Article Template
39Mail Merge Template
44Duplicate Rule
45Duplicate Rule Condition
46Entity Map
47Attribute Map
48Ribbon Command
49Ribbon Context Group
50Ribbon Customization
52Ribbon Rule
53Ribbon Tab To Command Map
55Ribbon Diff
59Saved Query Visualization
60System Form
61Web Resource
62Site Map
63Connection Role
64Complex Control
70Field Security Profile
71Field Permission
90Plugin Type
91Plugin Assembly
92SDK Message Processing Step
93SDK Message Processing Step Image
95Service Endpoint
150Routing Rule
151Routing Rule Item
152SLA
153SLA Item
154Convert Rule
155Convert Rule Item
65Hierarchy Rule
161Mobile Offline Profile
162Mobile Offline Profile Item
165Similarity Rule
66Custom Control
68Custom Control Default Config
166Data Source Mapping
201SDKMessage
202SDKMessageFilter
203SdkMessagePair
204SdkMessageRequest
205SdkMessageRequestField
206SdkMessageResponse
207SdkMessageResponseField
210WebWizard
18Index
208Import Map
300Canvas App
371Connector
372Connector
380Environment Variable Definition
381Environment Variable Value
400AI Project Type
401AI Project
402AI Configuration
430Entity Analytics Configuration
431Attribute Image Configuration
432Entity Image Configuration
<fetch top="50">
  <entity name="entity">
    <attribute name="logicalname" />
    <filter>
      <condition attribute="entityid" operator="eq" value="dd7613fe-806e-ef11-a670-000d3ad82e35" />
    </filter>
  </entity>
</fetch>
{
  "value": [
    {
      "logicalname": "msdynci_listmemberships",
      "entityid": "dd7613fe-806e-ef11-a670-000d3ad82e35"
    }
  ]
}

Hence, from this I generated the image above. Since it is something of a circular reference, Microsoft Support suggested that I remove the entity analytics record with id: e46e52db-46a7-4585-8a6a-6ba888a5bd1f. However, not entirely sure what this is and hence I tried the one I had gotten in the first steps above, and that worked.

Generally I think it is a good idea to investigate what the dependencies point to, before removing something. In this case the dependency record in itself wasn’t removed, only the dependent part.

Thanks to Microsoft support for helping out with this and I hope it might shed some insights if you are having similar issues. I also think breaking down the dependency table in a tool in for instance XrmToolBox would be a great idea. I have a bit of a bad conscience for BulkDeleteManager that I own and I am not giving enough love, so feel free to build it based on this and I guess some more stuff. I will be happy to help out the the investigations if you are willing to build it.

Bulk Delete failing

Bulk Delete failing

I have recently been working with a customer with a large (500+ GB dataverse db) production instance and we are attempting to cut this down quite a lot. The natural way of doing this is bulk delete but it seems that if the underlying SQL isn’t up to speed, your jobs might end without actually being done.

My customer has a production instance of over 500GB which is costing them some substantial money and hence we started looking at removing some of the less necessary data. In this case it was the order products (salesorderdetail) which we have perceived that we do not need more than one year after delivery. Hence we set up an advanced find, agreed on the exact filters, saved the view and then tried to remove the records (well over 5M) using bulk delete. I restarted it several times. You can see the chronological results in the screenshot below:

As you can see, sometimes it actually deleted a few records, but most times, it didn’t. My professional analysis of this (=guess) is that this is caused by bulk delete not handling exceptions like SQL Timeout properly or that it has a limit on the number of times it will retry.

I also, in parallell tried to remove the same records with the same FetchXML using SSIS/Kingswaysoft and here I have several times gotten this error. I have had to turn down the knobs to a very low setting to get it to work, but the error message I did get was:

{“error”:{“code”:”0x80044151″,”message”:” Sql error: SQL timeout expired. CRM ErrorCode: -2147204783 Sql ErrorCode: -2146232060 Sql Number: -2″,”@Microsoft.PowerApps.CDS.ErrorDetails.ApiExceptionSourceKey”:”Plugin/Microsoft.Crm.ObjectModel.CustomBusinessEntityService”,”@Microsoft.PowerApps.CDS.ErrorDetails.ApiStepKey”:”60ccbb1b-ea3e-db11-86a7-000a3a5473e8″,”@Microsoft.PowerApps.CDS.ErrorDetails.ApiDepthKey”:”1″,”@Microsoft.PowerApps.CDS.ErrorDetails.ApiActivityIdKey”:”4edd4e59-4b1e-4d0b-8d13-79ea53ab10ff”,”@Microsoft.PowerApps.CDS.ErrorDetails.ApiPluginSolutionNameKey”:”System”,”@Microsoft.PowerApps.CDS.ErrorDetails.ApiStepSolutionNameKey”:”System”,”@Microsoft.PowerApps.CDS.ErrorDetails.ApiExceptionCategory”:”SystemFailure”,”@Microsoft.PowerApps.CDS.ErrorDetails.ApiExceptionMessageName”:”unManagedidssqltimeouterror”,”@Microsoft.PowerApps.CDS.ErrorDetails.ApiExceptionHttpStatusCode”:”503″,”@Microsoft.PowerApps.CDS.HelpLink”:”http://go.microsoft.com/fwlink/?LinkID=398563&error=Microsoft.Crm.CrmException%3a80044151&client=platform”,”@Microsoft.PowerApps.CDS.InnerError.Message”:” Sql error: SQL timeout expired. CRM ErrorCode: -2147204783 Sql ErrorCode: -2146232060 Sql Number: -2″}}
–batchresponse_5165dc8d-efdb-4237-b62f-267ed01f73b8–
) (SSIS Integration Toolkit for Microsoft Dynamics 365, v23.2.0.25101 – DtsDebugHost, v16.0.1000.6)System.Net.WebException

(Status Reason: ServiceUnavailable): The remote server returned an error: (503) Server Unavailable.”.

As you can see, there is a lot of noise but it clearly, in the highlighted part, say that there is a SQL Timeout.

Hence my takeaway from this is that you need to be a bit wary of bulk delete in large instances or in general as it might indicate that it has completed successfully but in fact it stopped due to SQL Timeout (or some other platform related issue).

On a personal note, I really hope the bulk delete functionality gets a modernized revamp soon. It is really old and is becoming more and more relevant.

Calculated columns + Azure Synapse Link != true

Calculated columns + Azure Synapse Link != true

I was recently helping my colleague Ebba Linnea Nilsson with a support ticket with data not being propagated correctly from dataverse to a datalake via Azure Synapse Link. It turned out that this was all by design. A design that might not be what normal users would expect.

Calculated columns and now recently the formula columns are both very useful way of being able to calculate data in a field that is based on other fields. Common scenarios are calculations like “Weighted revenue” which is the probability multiplied by the estimated revenue for an opportunity. However, there are scenarios where you need to be aware of how these fields actually work or you might get an unwanted or unexpected behaviour.

The first thing that needs to be understood is that these column types are calculated “on-the-fly” everytime dataverse attemts to access these columns. It might seem like the data is “in the columns” but it really isn’t, it is calculated. This is a big difference from for instance rollup-columns is that those columns are calculated on a regular interval by the system, and the result is stored in the record.

What does this mean for Azure Synapse Link? Well, let’s say we have a simple calculation, that sets the value “A” into all records for this calculated column. We then enable the Azure Synapse Link which will make an initial sync and set the column in the datalake to “A”. Now we change the calculation of the rule to output “B” instead. As no records are actually changed, this will not cause any records in the datalake to be updated, hence they will all still have the value “A”. From a user perspective comparing Dynamics 365 to the datalake without any underlying understanding of how this functions, it will look like an error. Same column has different values comparing what is in dataverse with what is in the datalake.

As soon as a record is actually changed, all columns for that record will then be sent to the datalake, and hence the calculated column will be set to “B” at that time. It is hence possible, to manually or semimanually force a resync, but it would require some bulk like for instance SSIS with Kingswaysoft especially for implementations with large amounts of records.

An important question to ask, is why would you want to calculate the data in dataverse and then use it in in the datalake. If you have a propper datalake architecture it should be easier to make calculated columns/fields in the datalake/datalakehouse. If the data is calculated only for use in the datalake, I would suggest moving the calculation to the datalake.

There are, of course, scenarios when it is preferrable to have calculations in one place and reuse the output in many places. However, this understanding of what can reasonably be expected is then essential.

As for product improvements, I have added an idea on the subject, if you agree with me, please vote! Microsoft Idea (dynamics.com)

A final note is that this type of unexpected behaviour is not limited to just Azure Synapse Link but really to any integrations based on either “modified on” or change tracking without doing periodic synchronizations. Hence I would also like to give a general warning about this.

Top Table Usage in PPAC

Top Table Usage in PPAC

Top Ten Table usage is back from being lost when Organizational Insights was discontinued. A bit tricky to find so check out the video. Awesome tool when trying to reduce the size of large instances, especially important now that the prices per GB are going up to $40/GB (subject to you license agreement).

How to reduce SubscriptionTrackingDeletedObject table in CDS/Dynamics 365 – Updated!

How to reduce SubscriptionTrackingDeletedObject table in CDS/Dynamics 365 – Updated!

One of my customers is a B2C customer with a very large online database exceeding 500 GB. With a very active Marketing automation tool interated, we generate a lot of data in Dynamics 365 CE which after defined retention periods needs to be removed. This has caused some side effects, that a table called SubscriptionTrackingDeletedObject has become very large. This article will describe how to set a configuration to reduce its size. UPDATED – Based on some new learning and information from Microsoft this article has now been updated!

We often monitor the Organizational Insights, and now lately the brand new capacity feature that can be found in the left hand menu in https://admin.powerplatform.microsoft.com, if you have a CDS/Dynamics 365 CE instance.

An interesting table started growing rapidly and we had no clue what this was, and I had during my now 15 years of working with Dynamics 365 never seen it. It was called SubscriptionTrackingDeletedObject. When I came back from my Swedish summer vaccation, it had grow to over an amazing 181M records. Time to fix this.

First thing, as usual is of course to google it (yes, it is a verb, get used to it). All I found was this somewhat informative post by my good friend Chris Cognetta who is an ace with infrastructure issues.

http://cognettacloud.net/2016/06/21/crm-database-log-growth-issue/ 

However, it seemed that they just truncated the table, and we were online so that was a bit tricky, to say the least. I was at this time a bit upset that Microsoft were taking up around 50GB of space for my customer without giving me any way of managing that, or having any direct use of it. I counted to ten and called Microsoft Support.

After a few emails back and forth, the excellent support technician at Microsoft informed me that there is actually a setting in the infamous super secret setting tool with the Star Trek-sounding name OrgDBOrg (it is pronounced “Org-D-Borg” in case you ever get stuck in Dynamics trivia). The setting is called ExpireSubscriptionsInDays. I will quote the support technician in what this table is used for and if anyone has any more information, please leave a comment.

“The SubscriptionTrackingDeletedObject table is the table that logs records for number of days before deleting inactive subscriptions as well as timed out deletion services.”

I am not sure for which purpose. If it is in regards to GDPR or some restore mechanism. I would like to know though. Default value for this i 90, which means that these logs will be stored for 90 days. The minimum they can be set to is 1. As I am currently not entirely sure what these logs are used for, I would not recommend you set them to 1, but I did set my customers to 5, hoping that this is not going to come back with a vengance.

We have during the day seen a dramatic drop in the amount of records in this table, with about 30M but and it is still ongoing, but hard to measure as there is a delay in the capacity measurement the Powerplatform admin portal.

Update! The size of the table fell to 160M rows but never below this so after some further discussions with Microsoft support they did some more investigation into this subject and came back with the following recommendation:

1. Reduce the value gradually from 90 to 60 and then on
2. Never go below 15

There is however, another related setting called ExpireChangeTrackingInDays which is located just next to the ExpireSubscriptionsInDays. This is defaulted to 30. We reduced this to 15. 

Based on these recommendations we tried 60 days and this resulted in a most dramatic drop to around 20M rows. – End of update

 

So, how do you do this? First, download the OrgDbOrgSettings tool and install it in the instance where you are having issues. Check out these links below for that:

https://github.com/seanmcne/OrgDbOrgSettings/blob/master/readme.md

https://github.com/seanmcne/OrgDbOrgSettings/releases

A word of advice regarding OrgDbOrg; don’t think that you are Captain Kirk and go flying off into the Beta Quadrant and beam every single setting just because you can. It won’t make your system better, rather the opposite. Make really sure on what you are doing and don’t even trust a blog article like this, read the KB-article linked in the tool and make up your own mind. It is a powerful tool, like a jackhammer.

After you have installed the OrgDbOrgSettings tool, you can see it and open it by clicking the display name.

Then just find the “ExpireSubscriptionsInDays” – Press “Edit” and change to whatever you would like it to be. You will typically have to confirm to save it to Dynamics 365 CE/CDS

With that done you should just have to wait for the magic to be done.

As far as I have understood these two settings and this table is used to indicate how long changes and deletes are stored in this table and related tables for integrating systems to be able to read. This can, for example be Data Export Service, the old Dynamics for Outlook client etc. Hence reducing the numbers to, for instance 15 (the lowest recommended number by Microsoft) can result in some changes not being propageted to these integrated systems in the case that the integrations or just an offline client being offline for more than 15 days. And I also got the feeling that setting it to 5 was below some internal threashold and hence wasn’t really supported despite the fact that it says in OrgDBOrg that the lowest value is 1.