Top Ten Table usage is back from being lost when Organizational Insights was discontinued. A bit tricky to find so check out the video. Awesome tool when trying to reduce the size of large instances, especially important now that the prices per GB are going up to $40/GB (subject to you license agreement).
The Excel import functionality in Dynamics CRM is quite good. Version 7.1 (2015 Spring Wave) introduced some new features, like finally being able to work with xlsx-files which helps out a lot.
Generally we recommend using ETL-tools like our favorite KingswaySoft in SSIS, and sometimes the only way to migrate really complex data is to write your own code, but it usually isn’t necessary (if you disagree, please leave a comment below!).
However, both ETL and especially code, takes some training to get going with and some of our customers and others I have spoke to have expressed a will to handle migration themselves which makes the Excel import the best candidate to work with as most people have experience with Excel.
First of all, my recommendation is to complete the customizations in the target system. That way, when you download the Import templates, you will get all the fields that are shown on the “standard” (fallback) form.
|Excel Account Import Template (Swedish) from CRM 2016|
As of CRM 2015 Spring wave (7.1) the Excel import template will be formatted as a excel table, which makes it easier to work with, I think.
During the work with migration of data, you can correct data in one of three places.
1. In the source system.
2. In the Excel files while transfering it.
3. In the target system/after migration
Typically the time when the live migration needs to be done, is as short as possible. As everyone needs to stop working in the old system, migration needs to be done, and then everyone can work in the new system. Hence it is typically done during evenings, nights or weekends, depending on which type of business you are in.
Due to this, you need to make sure that when you are doing the real live migration, you need to make sure that it will be as controlled and as smooth as possible. This requires you to first set up the migration and then test it properly. Dry runs are therefore a must unless you are CRM Rambo.
If you plan steps in the migration in the above described stage (2) – during the transfer, you will need to re-do these every time you do a dry run and when you do the live migration. If possible to change the data in the source system instead, for instance remove duplicates, you reduce risk and speed up the migration process.
Generally it is a really good idea to try to clean up you data before migrating it. Many are the CRM system owners who have understood quite a lot about how their users actually use the system, when the are to migrate the data and really start digging their head into it.
Some of the common problems when migrating with Excel are:
- Complicated to handle GUIDs – ETL tools often have ways to migrate the GUID from a source system, very useful if moving from CRM Onprem to CRM Online.
- Cannot handle updates/upserts easily. Can however upload several files at once instead. For instance, contacts have Parent accounts, and Accounts have Primary Contacts. Just make a zip with both xlsx-files and upload. CRM will figure out that they are interdependant.
- Duplicates. Duplicates. Duplicates. So you switched off the duplicate detection for accounts and thought you’d be fine? Well, not so much, lookup fields to account, for instance, by default, will use the primary field, which is “name” and if you have several accounts with the same name, the import function will not know which to select and you will get an error. Typical fields where this is a problem are:
- Contact – Parent Customer / Company (pointing to account)
- Opportunity – Potential Customer / Account (pointing to account)
- Opportunity – Contact
- Account – Primary Contact
- Account – Originating Lead
- Contact – Originating Lead
- Opportunity – Originating Lead
- Only active? Are you only going to import active records? Will that work? Do active records have relationships to inactive records? For instance, you might have an active account that has an inactive primary contact. Or active accounts with inactive parent accounts. If only moving the active accounts and active contacts, the import will fail as the target system will not find the primary contact.
- Activities. The activity parties, in other words the “to”, “cc”, “bcc”, “required”, “optional” etc. fields on activities are handled by a rather complex mapping entity called activity party. There is some support for migrating this with Excel but it is not very easy, so if you do have a lot of activities to migrate, I would strongly recommend using an ETL tool with support for activities.
- If you add a column to the CRM 2016 Excel Template, despite having the Display name in CRM, it will not be automatically mapped like the other columns as there is underlying data to handle it. It is no big issue as the import mechanism usually interprets this correctly anyway. Do note that the Display name mapping is case sensitive and CRM does allow for duplicates in fields having the same Display name, but that doesn’t make importing easier.
Some tricks that I use
- Generate an import template for the entity at hand, let’s say account from the target system, then remove all the columns you don’t want to import. Remove the entire column. Save this. Verify the template by filling in one or two rows by hand and importing. Make sure that the lookups match data in the system.
- Export data from the source system in the same order to match the import template, if possible, into Excel. Then, verify that all columns are identical (ie. picklist are the same etc.) and then copy-as-values from the source excel sheet to the target excel sheet without the header columns.
- Verify that the data in the target excel sheet is correct, that for instance there are cities in the city column, not street names.
- Then import the data into CRM. The first couple of times you will get errors. If you don’t you are very very lucky. Migrating data is complicated, as data is seldom clean and well ordered. So, when it is done, go to the imports section in CRM and go through the errors and try to understand what it is trying to say. Typically a specific lookup cannot be found or it has multiple values. My suggestion is to fix it in the source data (1 in the list above) and not try to do it in the Excel sheet (2). Also plan for making several test runs of migration. Write down the steps you need to do when doing it live, so you have your plan set.
- When importing data with lookups to data where there are lot’s of duplicates in the primary field, do note that you can change the mapping of the lookup field so that it mapps not on the primary field but on some other field instead. For instance, if there are several accounts with the same name but all accounts have different account numbers, you can use account number to indicate parent customer on a contact instead and then change the mapping when importing.
- Make sure that if you do change data in Excel and use formulas, copy and paste-as-values back to the cells/columns as the import does not support formulas. (not sure if the new CRM 2015 Spring Wave/2016 does, havn’t tested, but I wouldn’t provoke it.)
- All picklist values need to exist. Using the import templates is a good tool here as they include the possible values. But you can paste incorrect data into the cells anyway and then import it where you will get errors on the rows with the incorrect data.
- Decide what is a good enough result before. Is 99% good enough? That would mean that if you have 100 000 accounts, 1000 accounts are not imported… so maybe that is not good enough. Remember Pareto’s law, that the last 20% take 80% of the work. So fixing the last 1% can be a real pain, why it is sometimes easier just do fix it manually after import.
- Zip files with inter-dependencies. Like account-contact.
- You can increase the max file upload size. but the max size for a file after uncompression is still 20 MB, if I am not mistaken. This means that you need to think about how you upload your data if you have large amounts of data. ETL tools might be a better option here as well.
- After getting errors and getting an import of let’s say 50%, instead of trying fix the last 50% and importing those, I recommend deleting all and going back from the start. As you need to plan for doing many trial runs before doing a live migration, this is the only way to make sure that you get the source data better and better.
If you have done your own imports and migrations with Excel, I am sure you have some tips of your own, please share them below! I do moderate all comments to avoid spam.
MVP, Founder and CTO at CRM-konsulterna AB
The dynamic Excel export functionality of Dynamics CRM is a real killer functionality. I often get standing ovations from users, especially managers, when showing dynamic pivot table functions of CRM, partly due to the fact that many users don’t actually know what pivot tables are and find them rahter magical.
I was working with a demo environment today and had some issues getting the dynamic excel to load data from CRM properly. I got the error message: “Connection failed”. This was a lab environment with a separate Windows 7 virtual machine in an on-premise installation. This is important, because in IFD or CRM-online, dynamic excel data is tunneled through the outlook addon, but not when using the on-premise installation when it communicates directly with the server.
So I started looking for answers on the internet and found the following thread on the CRM forum which helped out a bit: http://social.microsoft.com/Forums/en/crmdeployment/thread/c2d3029c-5487-435b-90e7-ba030aeba8ac
I started by trying to do the obvious thing by opening the TCP port 1433, the SQL communication port in the firewall. This did not help. According to some of the participants of the thread, this KB article by Microsoft might help: http://support.microsoft.com/kb/968872 – it does more or less the same thing, opens 1433 and some additional ports in the firewall all related to SQL. However, it did not help either.
One of the participants of the thread mentioned that he had opened the TCP port 53021, something I found very strange if it was to work, since ports this high are not supposed to be used as incoming ports but are ephemeral ports. I tried opening this port as well, but it didn’t work either. The port number seemd sort of random so it might be that my deployment just used some other high port number.
A colleague of mine working at Hermelin IT-Partner, Anders Jildestrand, then joined me in the hunt for the mysterious TCP port. With his assistance and netstat we got the following result:
Proto Local Address Foreign Address State
TCP 192.168.75.15:3389 192.168.75.131:29837 ESTABLISHED
TCP 192.168.75.15:52422 192.168.75.13:5555 ESTABLISHED
TCP 192.168.75.15:52426 WIN-LGEKH2VQ6FI:epmap TIME_WAIT
TCP 192.168.75.15:52427 WIN-LGEKH2VQ6FI:49155 TIME_WAIT
TCP 192.168.75.15:52428 192.168.75.13:63831 SYN_SENT
And if you are sharp eyed, you might spot it, as we had the SQL-server on the CRM server, and the TCP-SYN was sent but never answered, it is of course the last row, hence the TCP port 63831.
After opening this port in the firewall, the dynamic Excel worked just like it should.
The question now remains. Why this port?
We also had a discussion on which program, CRM or SQL Server that is actually answering this call, and it seems that it is actually SQL Server since the ODBC connectionstring in the Excel-file is directed to the SQL-server, not the CRM-server. But I think we could probably investigate this further with fiddler. If you have any interesting input, please leave a comment!
CEO, Chief Architect and co-Founder at CRM-konsulterna AB