Jonas, FetchXml Builder and Kingswaysoft

Jonas, FetchXml Builder and Kingswaysoft

As you probably will have heard by now, my colleague and friend, Jonas Rapp suffered a stroke about two weeks ago. Daryl LaBar, Carina Claesson, Benedikt Bergmann and I talked about this on the XrmToolCast which you can listen to here: https://xrmtoolcast.libsyn.com/coming-together-to-support-jonas

I would, with this post, like to give my personal perspective on how I use FetchXml Builder as an integral part of making scripts with SSIS and Kingswaysoft.

And before I begin, I would just like to urge everyone to donate a dollar or two to Jonas as he is having tough time. Or just share your story of why you love to use his tools. Everything matters. And if you want to donate, just use this address: https://fetchxmlbuilder.com/donate. If he has helped you, as he has helped me, why not help him back. Why not leave a comment with how much you donated, and when we have 10 comments, I will share how much I have donated.

With one of my customers, I work a lot with trying to manage their rather larger database, around 500 GB, and doing what I can to make sure that it doesn’t grow out of proportion. This also entails looking into GDPR rules and making sure that personal information is removed, and kept according to specific rulesets that become rather complex. So complex that one advanced find query isn’t enough to define them. And the removal of data is not seldom in the millions of records, why I need a tool that can do deletes quickly. Hence, I have found that SSIS and Kingswaysoft with Azure Data Factory for deploying these packages to be run continously, is a good method.

However, there is a very important piece of the puzzle that is missing when you want to do this, and that is being able to assemble the FetchXmls to the nitty gritty level that they are as efficient as possible and for that job, there really is only one tool, FetchXml Builder. Hence I often set up my general idea of how I would like the rules to be executed and then start assembling the data often into different Cache transforms, and then filter the dataflows using lookups based on this.

So, for me, FetchXml Builder is essential when working with this and this is my story of how critical it is for me.

Can we expect a new API?

Can we expect a new API?

Dataverse, and new terms for picklists, fields and more. This has been described in detail by many others in the community like Jukka Niiranen. However Microsoft are saying that there will be no new API, that all API:s will stay the same so this article is just going to discuss this subject briefly.

First of all, they will, of course do exactly what they are saying. The current WebAPI will of course be available for many years to come as we have seen the old 2011 SOAP API still be available and used a lot, still today, often for the reason that there isn’t full functional parity between the API:s.

Some tools allow you to seamlessly choose between the API:s, like Kingswaysoft, and I have asked Daniel Cai, the owner and head developer/architect at Kingswaysoft which API I should use. I don’t know if he has changed his recommendation, but last year, he still recommended, if my memory hasn’t failed me, to use the old SOAP API due to stability and functionality issues. I will let him, on their blog describe their thoughts on this, if they want to.

However, I think this is rather telling of how Microsoft act and will act regarding an older API, that it will be maintained and kept alive for many years.

That brings us to the interesting question if there will be a new API based on these new terms, I frankly do not know, but my deduction would say a definite yes.

It think the easiest way of seeing this is falsifying this and extrapolating it. In other words, what would happen if they didn’t create a new API and we looked at the platform as a whole in 5-7 years without knowing the history? It would look totally crazy. A platform with one set of terms in the UI and another set of terms in the API:s. It would make documentation really complex as there would be need for translations between these two concepts everywhere and the bar for getting developers into the platform would be higher than necessary.

Hence the only reasonable deduction is that they will make a new API with the new terms. It will probably be release in 6 months or so, but the current WebAPI will still continue to work for many years to come.

These are my personal thoughts and are not based on any information from Microsoft, despite the fact that I am an MVP. I urge you to make your own deductions and I think you will find that this seems reasonable. Please share your thoughts disagreements or otherwise in the comment section below.

Why I love Kingswaysoft

Why I love Kingswaysoft

The reason I often recommend Kingswaysoft over other methods of data migration or even sometime data integration is rather simple, it can do what others can’t. Not even Microsoft. And it can do it fast.

So have I been payed by Kingswaysoft to write this? No, not a penny, I havn’t even been given a free license even though they might if I asked. I think it is just fair that I explain why I am such a strong proponent for this product and if anyone disagrees, please feel free to drop a comment below.

API knowledge

Kingswaysoft both know how the API:s of the Power Platform/CDS/Dataf??x (called CDS below) work, sometimes even better than Microsoft themselves as I have seen feedbacks given in such detail from them on what is missing from specific API:s to reach feature parity, that it is scary. Kingswaysoft also has a blog which has details recommendations and built in recommendations in the product on how to maximize performance with the Dataflex API. They have also built in handling for throttling, multi-threading, batching and more to just make it transparent. When looking at other integration products and connectors. I have never seen anything that is close to this depth.

Datamodel and Dynamics knowledge

Dynamics 365 Sales, Customer Service and other first party apps have some very peculiar oddities that need to be taken into consideration. These oddities range from for example:

  • How to delete Marketing List members – by using the party member and list as key
  • How audit logs are handled
  • How Activities work, with activity pointer and activity parties
  • Setting CreatedOn/CreatedBy/ModifiedOn/ModifiedBy
  • Reading/Writing personal views (saved advanced finds)
  • Setting statusvalues of some fringe entities

Even though many tools, like Power Automate, Data Flows, Azure Data Factory, etc. all seem to handle common tasks like creating or reading a contact or account rather well, it soon becomes a problem when you start looking at some of the areas above. And you often need to in a migration.

It can use the power of SSIS

SQL Server Integration Services is a very powerful framework once you start getting your head around it. Yes, it has some quirks to it but in general it can do some rather cool things and is good when you want to sequence many different tasks in order to reduce overall runtime. Using built in features like Cache Transforms with memory storage, you can make filters using memory based lookups with millions of records super fast, once everything is loaded.

…and extend it

And of course Kingswaysoft didn’t settle for just building connectors to Dynamics 365. They have a pack called “Productivity Pack” that adds a lot of nice features to SSIS that makes your day a lot easier.

And if you have problems – great support

And if you ever run into problems, their support is great. We have identified buggs and they have fixed the bug and sent us a special deploy just a day or two after.

Flip side

There is always a flip side. I think one is that SSIS isn’t always your most stable product. For this I don’t think Kingswaysoft are to blame but it affects their product experience none the less.

Another obvious flip side is that this requires a skillset that is a bit “off the tracks” even though it isn’t super hard to learn the basics, becoming really proficient with SSIS takes time.

Finally

So, a product as good as this is probably super expensive, right? No. It isn’t. If you are running things from within Visual Studio, then you don’t even need a license, but if you plan to run a migration or so, I certainly recommend one anyway, to get access to support. And buy the Ultimate license to get access to the productivity pack and all the other connectors while you’re at it. It is worth it.

Just to be clear, I am not saying that the other products are bad. I am just saying that once you choose a product, you will start to decend the rabitts hole. If the product isn’t up to speed you will have a couple of options:

  • Back up and redo with another technology
  • Patch with another technology
  • Create some kind of workaround

Neither of these solutions is rather palatable. Hence it is often tempting to choose a product that you know can do the job. And apart from coding, SSIS with Kingswaysoft will very seldom have any issues.

Team member licensing – hammer coming down!

Team member licensing – hammer coming down!

The team member licensing option is something that has been a subject of debate for quite some time. What can it actually be used for and what not? I have heard “experts” suggesting that it be used for integrations but if your read the Dynamics 365 Licensing Guide, appendix A, you can read all about what Team member licensing is and isn’t. Typical scenarios for Team members are:

  • Read only users
  • Users that use only slim parts of the system, and not the “cool” First-Party-App features in Sales, Customer Service, Field Service etc.
  • Users that just track activities

There are some updates to the details of what a team member can and cannot do, and I think the most important is that a team member cannot CUD (Create, Update or Delete) accounts any more.

How urgent is this? Well, for new instances being created, this will be enforced as of April 1 2020. But for existing instances, it will be enforced on July 1 2020. Hence, you still have some time if you have an exiting org.  

You can opt-in to early access updates. Goes without saying that you shouldn’t do in your production system, and probably not even in you dev/test environments since as soon as you have you won’t be able to depoy changes. So, having an out-of-ALM-environment where you can test this might be a good idea. 

So, the BIG question is, “Are we compliant?” 

Well, there is actually a report that you can generate from the Power Platform Admin Center and I have recorded a video below that shows how you can use that and collate the data with pivoting in Excel.

D365UG Summit – will be awesome!

D365UG Summit – will be awesome!

The yearly CRMUG or D365UG Summit is here! Nowdays it is colocated with eXtreme365 which is a partner event. I was planned to present 4 sessions (panels and presentations) for D365UG but as I am writing this on the second day of eXtreme365 I happend to cover for a speaker yesterday that couldn’t come. Tomorrow is the big day where D365UG starts.

 

Top Table Usage in PPAC

Top Table Usage in PPAC

Top Ten Table usage is back from being lost when Organizational Insights was discontinued. A bit tricky to find so check out the video. Awesome tool when trying to reduce the size of large instances, especially important now that the prices per GB are going up to $40/GB (subject to you license agreement).