by Gustaf Westerlund | Jun 25, 2012
Migrations are often complicated and not very sexy, all the work you put into it, configuration, scripts and code will be more or less useless after the migration has been done. However, choosing the right strategy is essential for a good CRM implementation. I will go through some of the strategies that are common and discuss some of the advantages and disadvantages.
Greenfield
Greenfield migrations are not really migrations, but rather the lack of migration. This means that the system will be set up without any migrated data, no accounts, no contacts etc. The idea is then that all the users will create data in the new system as the feel the need for it.
This is often used for start-ups when there is no or very little data. It can also be used when the data quality is very bad and trying to filter out the “good” data is just to complicated. Normal problems can be large numbers of duplicates, errors in the data itself, like “.” and “,” being inserted instead of real data just to make the forms savable.
Advantages of Greenfield migrations is that the system will be up and running in no time. Migrations might otherwise take considerable amounts of time. This also mean that the cost of Greenfield migrations are very low. Do note however, that there is need for some general system configuration, like creating users, setting up queues.
The disadvantage is that the users do not feel that the new system is helping them out and they have to reinsert a lot of data which can create considerable amounts of annoyance with the new system. This can in the long term increase the bad-will towards the system, lowering usage which can jeopardize the entire CRM investment.
Standard import
The standard import functionality of Dynamics CRM is reasonably powerful and can usually be used for importing simpler data. It is based on data being stored in csv-files or excel as xml-files. The first of these being a bit problematic as it is rather old and dependant on the regional setting. For instance, the value separator in Sweden is semicolon “;” but in USA comma “,”. It is also common that the csv-files contain data that risk messing up the file syntax, breaking the imports. In general the excel-as-xml format is to be preferred as it also contains field names and proper matching.
This functionality has greatly been enhanced in CRM 2011 compared to CRM 4. For instance, it now support uploading of zip-files containing xml-data files with relations between the files. It can even handle multiple relations that otherwise would take three import runs, all in one go. For instance if you have data with accounts containing primary contact and all contacts also contain parent customer.
The limitations of standard import is that it cannot really handle large amounts of data as the maximum upload size is 8 MB (can be changed in an onpremise installation). Complicated data structures are also hard to handle and it is rather time consuming to run it as there is no scripting capabilities. The lack of scripting also makes the ordering of the different imports in a zip more or less impossible. There is also no support for logic like “If the contacts email adress exists, update the data of that contact, otherwise create a new contact” which is often very useful when importing data from multiple sources. It is also hard to do testruns of the migration without a lot of manual steps involved. There is some error handling but it is very rudimentary and cannot handle more complex logic.
The advantages of standard import is mainly that it requires no extra software and it is relativly easy as the options are limited.
Third party products
One of the more powerful options when doing migrations, is using third party products, like Scribe, Import Manager. These include lots of options that the standard import doesn’t like:
- ODBC/OLE DB connections to a data source
- Logic to handle update or create
- Scripting – custom logic to handle some of the data conversions
- Run as service – ability to be run in the background as a service
- Reuse for integrations – these tools can also be used for integrations why the effort invested can at least partly be reused.
- Support for custom addons
- More powerful – has support for multi threading and large data sources
- Easier to do test migrations as all steps can be set up in a run-script, less manual labour.
- Can handle more complex data
- Better error handling
Not all products support all of these features.
The main drawbacks are the licensing fees required. Do note that some of these companies have special migration licenses which are not the same as the full license used for integrations. These products are also a bit more complicated due to the larger sets of options available. Hence it also takes some time to learn the product why my recommendation is to choose one and stick with it.
Custom migration program
The most advanced for of migrations need to be done by using custom code. There are really very few limitations to what can be done when writing custom programs that migrated data, it is more up to the skills of the developer and the time available.
Some of the advantages:
- Limitations only in skills and available time and money
- Easy to do test runs
- Complicated error handling logic can be created
The main disadvantages are:
- Demands developers
- Time consuming – from around 100 hours to several thousand hours
General recommendations
Migrations are often complex and it is not until you have worked with the data a bit that you start to get a feel for how problematic it is going to be. Hence I usually never give fixed prices on migrations as it would either be inflated or put me at risk. It is also often the case that customers do not realize the complexity of migrations, especially smaller customers it is therefor essential to involve them a lot in the work.
De-scoping the migration is also very important. Usually not all data is required, especially if the old system can be maintained with read-only access. Try to de-scope in width (which entities are really required) and in depth (how old data is required).
Data quality can also be a problem so try to evaluate this early on. Examples are names being stored both as “First name Last name” and “Last name, First name” or fill-in data like “.”, “,” or “-” that has been entered into required fields to make the forms savable.
Large migrations might required delta migrations, when migrations are run in two steps, first the major migration, then after it has been completed, a smaller to migrate the data that was changed during the main migration. This will put additional demands on migration scripts and/or code so try to avoid it if possible.
Sometimes the new system required data that doesn’t exist and hence has to be created. I usually refer to this as migration of non-existent data. It is easy to forget if your perspective is to map the data from the old system without looking at which data is required in the new.
Migrations between two systems that with very different data models is theoretically complicated. There might, for instance exist data in several places that need to fit into one. Data might need to be restructured in complex ways. In depth knowledge of source and target data model is essential and the proper skills required to understand complex data modelling is also a strong recommendation.
I hope I have shed some light on this subject which can be discussed at length.
Gustaf Westerlund
CEO, Chief Architect and co-Founder at CRM-konsulterna AB
www.crmkonsulterna.se
by Gustaf Westerlund | Jun 21, 2012
Unsupported customizations are certain customizations that fall outside of what is supported by Microsoft when developing and customizing for Dynamics CRM. Several aspects of this subject can be discussed in depth and I will try to touch base on some of them.
When creating Microsoft Dynamics CRM, Microsoft realized something very important, the system will need to be customized by partners to fully fit the needs of the customers. However, customizations need to be upgradable and support update rollups and hotfixes. Previously, all upgrades would require rather large efforts in re-writing lot of the code that was custom built for the customer. However, the definition of supported and unsupported customizations and the promise from Microsoft that all supported customizations will be upgradable and updatable at least to the next version, is something that is very good and will ensure a better return of investment of the custom development made for customer and also improve partner-customer relations since paying for upgrades of already made customizations seldom improves the partner-customer relationship.
We have worked with many systems where other consultants and developers have created solutions and here are some of the more common unsupported customizations, why they will cause problems and alterntive solutions:
HTML DOM addressing and modifications
Addressing HTML DOM components directly using document.getElementById and replacing parts of the normal HTML DOM with custom components. For instance, addressing an isv-button in CRM 4.0 and disabling or removing it with javascript. Another example is replacing lookups with dropdows, for instance letting the user select contact for an opportunity.
This is very common i CRM 4.0 but is also quite common in CRM 2011. The problem with this unsupported customization is that when upgrading the system the names of the elements in the HTML DOM will probably change. This is obvious in the case of isv button addressing in CRM 4.0 since the ribbon HTML DOM element names are different. The code will hence break after an upgrade.
Developers doing these kinds of unsupported customizations are usually web developers that are used to being able to modify webpages with javascripts anyway they like and they are not familiar with the SDK and do not forsee the consequences of these unsupported customizations.
Alternative solutions are to strictly use the object model described in the SDK and only use the documented methods. Note that the SDK in CRM 2011 has been improved a lot in this perspective. It is also possible to enable and disable buttons in the ribbon with ribbon customizations. It is also possible to use small iframes with aspx-pages behind (or virtual pages created with javascript) that show custom logic and work with the CRM form using cross-frame scripting. Do note that the hosting application and the cross-framed page need to be in the same security zone in IE for it to allow this. Preferably place them on the same server.
Database modifications
Modifications to the CRM database are also some of the more common unsupported customizations that I have experienced. It can be anything from custom views to new stored procedures or even modifications to the existing procedures. Do note that creating indexes in the database is supported but not upgradable.
Developers making these kinds of customizations usually do not realize the effects that they have and might be used to and very familiar with writing T-SQL. Modifications to the existing tables and views are very hazardous since CRM is a metadata driven application meaning that the database contains metainformation on how the database looks. If the metainformation gets out of sync with the real database, you will have a very nasty problem on your hands.
Additions to the database, without any modifications, will usually work with updates to the system. However, when upgrading or redeploying the system these usually cause quite a lot of problems. It is also a general recommendation to try to keep all business logic in the same layer of the application and to not mix them between the database and higher layers (like plugins). This can sometimes not be avoided however.
So, if you do need to create some custom views or similar, create a database next to the CRM database with hosts these and use cross database addressing to work with the data.
Modifications to CRM files or additions to the CRM installation directory
As the aspx-files of CRM can be found in the CRMweb directory (or inetpub for port 80 installation) it can be tempting to modify these. In CRM 4.0, when there was no support for javascript includes, modification to the file global.js was also quite common. I have also seen systems where entire IIS virtual directories have been set up in the assembly directory of CRM. This is also unsupported.
The reason these customizations should be avoided is that any update rollup that is installed might overwrite these files and you customization might be lost. With upgrades the customization will most certainly be lost or stop working.
There are no good alternatives to these kind of modifications. Try working with Iframed custom pages using cross-frame scripting if possible. Modifications to the global.js are very hazardous since it is not very uncommon for it to be updated in normal update rollups, making any customizations to it rather hard to maintain. This is not necessary in CRM 2011 as the web resource-functionality allows for better reuse of javascript code.
Unsupported customizations are necessary anyway
If the requirement still demands unsupported customizations, and there is no good way around it, I recommned having a discussion with the customer on the effects of creating unsupported customizations in regards to upgrades and increased maintainance costs. If the still feel it is required, I recommend you getting their formal signature on a document describing it as an unsupported customzation and that the customer understands that the function might not be upgradable and that upgradecosts will be fully charged to the customer. It might feel a bit formal, but trust me, when the customer wants to upgrade, you will be happy you have the document.
Another important aspect that you need to remember when creating unsupported customizations, is that you need to take this fact into account. For instance, if addressing an HTML DOM element, make sure to handle the fact that it might not exist, and have some resonable fallback from this.
It is also essential to document each unsupported customization with extra care. I recommend a special document for this. It should include some of the following topics:
- Description of requirement
- Technical desciption of the solution
- Why it has to be made with unsupported customizations – This is important as newer versions of CRM might enable you to rewrite this in a supported manner.
- Possible effects on an installation of an update rollup –
- Possible effects after an upgrade
- Checklist to verify the functionality.
Gustaf Westerlund
CEO, Chief Architect and co-Founder at CRM-konsulterna AB
www.crmkonsulterna.se
by Gustaf Westerlund | Jun 5, 2012
My colleague Rickard and I are working with one of our customers who have gotten managed to get themselves into a rather big mess with huge numbers of attributes on the account entity that should not really be there. If any of you ever wondered what the limitation for number of attributes in Dynamics CRM is, it is around 700 and it is not recommended to use since the system becomes very hard to work with. Proper data modelling with experienced architects is usually a good idea that pays off tenfold later, so that you do not get your self into this situation.
However, the production system at hand has several hundred attributes that were erroneous in a managed solution, and something had to be done.
The first thing you might think of trying is just to remove the attributes from the development environment, packaging a new version of the managed solution and overwriting the production environments managed solution with this new reduced solution. This will, however, not work, as solutions cannot remove attributes (they contain the state of the CRM-system, not the transactions).
After a few more ideas had been tried, we finally registered a support case with Microsoft where a very helpful technician assisted us with a solution to this, based on this very good blog posting by Gonzalo Ruiz. The latest news on this is that it seems to be working, although neither Rickard nor myself is doing the actual work.
This has also caused us to have a very in-depth discussion on if managed solutions are suitable for normal project production systems or not. I had a discussion with former Microsoft employee Peter Björkmarker who is a very skilled architect, and we came to the conclusion that managed solutions are really best suited for ISV-packages, and not so much for normal customer projects where unmanaged solution are easier to work with. This is, however, something I would be more than happy to discuss, so if you have any opinion on the subject, please leave a comment!
Gustaf Westerlund
CEO, Chief Architect and co-Founder at CRM-konsulterna AB
www.crmkonsulterna.se
The standard import functionality of Dynamics CRM is reasonably powerful and can usually be used for importing simpler data. It is based on data being stored in csv-files or excel as xml-files. The first of these being a bit problematic as it is rather old and dependant on the regional setting. For instance, the value separator in Sweden is semicolon “;” but in USA comma “,”. It is also common that the csv-files contain data that risk messing up the file syntax, breaking the imports. In general the excel-as-xml format is to be preferred as it also contains field names and proper matching.
Recent Comments