innofactor / innofactor.crm.ci Goto Github PK

Modifications are needed to able to see the logs generated from our cmdlets.

Add option to only update existing records if any of the included attributes have been changed.
This to avoid unnecessary updates and changes of modifiedon/by.

Hi Guys,

I just installed the build and release tasks in TFS Build 2018 (on-premise), I got this error when I tried to use the ShuffleExport task:
The specified module 'C:\Agent_work_tasks\ShuffleExport_c0da7e50-d3a6-417e-9c8b-11da841352fc\9.0.5\ps_modules\CI\Innofactor.Crm.CI.dll' was not loaded because no valid module file was found in any module directory.

I got it working by cloning the github repository and building the Innofactor.Crm.CI solution and then copying this dll and all others from this location: "Innofactor.Crm.CI\Cmdlets\Shuffle\bin\Debug" to "C:\Agent_work_tasks\ShuffleExport_c0da7e50-d3a6-417e-9c8b-11da841352fc\9.0.5\ps_modules\CI"

Just wondering if I missed something when I installed the VSTS Build tasks or maybe this is just an issue with an on-premise version? Anyway it is working now that I did the following copy of DLL's.

Thanks,
Tim

When trying to import records, I get the the following log entry: "Cannot find resource Innofactor.Crm.Shuffle.Resources.ShuffleDefinition.xsd" Then nothing seems to happen, no records are imported or failed.

I'm using the latest plugin from the release section and the latest version of XRM Toolbox and V9.0 Dynamics.

Did I miss something to install the plugin?

JanFebContentSnippet.zip

I've been wondering if we can make something to enable users to easily clean up log files that accumulate over time. An non-technical user wouldn't really go into %appdata usually and the log files can accumulate and become quite big over time.

This needs to be fixed. I've noticed such statements in logs after the recent update

##[debug]  *** Logger: Invalid section stack index: -1
##[debug][END of  -unknown section-]

When I press 'Do the shuffle' in the Shuffle Builder, I get the following error: Cannot find resource Innofactor.Crm.Shuffle.Resource.ShuffleDefinition.xsd.

XML:
<DataBlock Name="Contacts" Entity="contact" Type="Entity"> <Export> <Attributes> <Attribute /> </Attributes> </Export> </DataBlock>

When using the VSTS export task a connection can not be made and the following error is thrown:

2018-01-17T15:39:29.3492395Z ##[error]Connection not established. Last CRM error message: Unable to Login to Dynamics CRMOrganizationServiceProxy is null

I used the same Connection String for both this task as well as Wael's Ping CRM Task:
AuthType=Office365;Username=$(SourceUsername);Url=https://$(SourceDomain).crm.dynamics.com;Password=$(SourcePassword)

Using Hosted VS2017 agent.

First let me say I'm so happy that you decided to release this! I've been looking for quite some time for an improved CI flow for our dynamics development at work. And after reading your blog posts last fall I was on the verge of building something similar on my own until I realized a week ago that you actually released it! Good job! :D

However, there are some config options that I believe are a hustle IMHO. For webresources you are required to add a prefix. Which makes sense I guess since a "publisher" in the crm has a prefix used for anything it creates. However, the prefix is then appended with an underscore (if not already present) and also a forward slash. This seems a bit odd to be honest.

E.g., we use webpack to build our css and js code and end up with the following files (names obscured)
ourprefix_library.js -> webresource: ourprefix_library
ourprefix_style.css -> webresource: ourprefix_style
ourprefix_ourcustomapp_js.js -> webresource: ourprefix_ourcustomapp_js
ourprefix_ourcustomapp_html.html -> webresource: ourprefix_ourcustomapp_html

But trying to configuring the webresource update build task using "ourprefix_" as prefix setting does not work since it also appends a slash meaning it will look for webresources named "ourprefix_/library" etc.
Moreover, it should probably strip file extensions from the local file names before looking for the webresource.

As far as I can tell the issue is here.

I can definitely make a pull request if you guys think this request makes sense!

From here : https://marketplace.visualstudio.com/items?itemName=InnofactorSE.cinteros-devutils-ci-build-tasks

Replace the DevUtils nuget with the new Xrm.Utils (https://github.com/Innofactor/Xrm.Utils) project to make it fully open source. Work is in progress already in the features/utils branch

During import, when there is no Match node in Import node, exception is thrown:

Nullable object must have a value.

   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at System.Nullable`1.get_Value()
   at Cinteros.Crm.Utils.Shuffle.Shuffler.ImportDataBlock(DataBlock block, CintDynEntityCollection cEntities)
   at Cinteros.Crm.Utils.Shuffle.Shuffler.ImportToCRM(ShuffleBlocks blocks)
   at Cinteros.Crm.Utils.Shuffle.Shuffler.QuickImport(IContainable container, XmlDocument Definition, XmlDocument Data, EventHandler`1 ShuffleEventHandler, String defpath, Boolean clearRemainingShuffleVars)
   at Cinteros.Crm.Utils.Shuffle.Shuffler.QuickImport(IContainable container, XmlDocument Definition, XmlDocument Data, EventHandler`1 ShuffleEventHandler, String defpath)
   at Innofactor.Crm.Shuffle.Runner.ShuffleRunner.<>c__DisplayClass10_0.<btnShuffle_Click>b__0(DoWorkEventArgs eventargs)
   at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
   at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)

Consider that you want to use your data file as a base for version control. However that would be easier to do if this tool would split the data file into entity folders and then into one file per record.

One could consider ids as filenames or use the definition import match attribute as a filename.

Make it easier to build Shuffle Definitions by adding metadata awareness to the Shuffle Builder.
Similar functionality as in FetchXML Builder would be nice, to be able to select entities, attributes and relationships.

As the attributes on M:M-entity are a Guid type, code looking for EntityReference and fails.

• Deployment: Online
• DB Version: 9.2.21113.132
• Connection Controls Version: 1.2021.9.46
• XrmToolBox Version: 1.2021.12.53
• Tool Version: 1.2021.3.1

Hi,

when I try to use the shuffle import/export tasks using a ClientSecret connectionstring, I am unable to connect to power platform

can It be the included crm sdk dll's need to be updated?

regards

Hi,

Is it a known issue or limitation that you cannot export more than 5000 records in a DataBlock?

If yes, are there any plans to change this if possible?

We currently have a regular entity with ~6600 records, but only 5000 are in the data file.
Unless I am looking over a certain option for this?

Kind regards,

If you include a datetime attribute, it will say "not identical" when they are really identical.
Has to compare local or utc for both source and target, now they are different.

See code: https://github.com/Innofactor/Innofactor.Crm.CI/blob/master/shared/Innofactor.Crm.Shuffle/ShuffleDataImport.cs#L21-L43

This should be "identical"... see:

• Deployment: Online
• DB Version: 9.2.21113.132
• Connection Controls Version: 1.2021.9.46
• XrmToolBox Version: 1.2021.12.53
• Tool Version: 1.2021.12.1

Hi,

Our Shuffle Export steps all of a sudden stopped working. They now complain about the Export Datablock being an invalid child of the DataBlock element.
Did something break in the 9.0.5 release for the VSTS Build Steps?
Validating the definition file manually using XrmToolbox works.

Please find our shuffle definition attached.
shufflewebformsxml.txt

Below is the error log returned from TFS:

2018-05-09T10:55:00.5912131Z ##[section]Starting: Shuffle Export
2018-05-09T10:55:00.5916879Z ==============================================================================
2018-05-09T10:55:00.5917335Z Task : Innofactor DevOps: Shuffle Export
2018-05-09T10:55:00.5917719Z Description : Export data and solutions using Innofactor Shuffle.
2018-05-09T10:55:00.5918087Z Version : 9.0.5
2018-05-09T10:55:00.5918449Z Author : Innofactor Sweden
2018-05-09T10:55:00.5918760Z Help :
2018-05-09T10:55:00.5919139Z ==============================================================================
2018-05-09T10:55:02.2316216Z Loading definition from C:\vstsagent\Build_work\12\s\shuffle_webforms.xml
2018-05-09T10:55:02.3735678Z Starting export
2018-05-09T10:55:03.4167656Z ##[error]The element 'DataBlock' has invalid child element 'Export'. List of possible elements expected: 'Relation'.
2018-05-09T10:55:03.4435277Z ##[section]Finishing: Shuffle Export

When defining and saving definitions with filters containing non-ASCII chars in Builder (Innofactor.Crm.XTB.Shuffle) - filters are builded with incorrect chars. Eg. filter name eq 'ółęć' are interpreted with bad charset:
[INFO] Adding filter: name Equal ĂłĹ'ÄtÄ┼
after adding BOM to definition it works fine:
[INFO] Adding filter: name Equal ółęć

A new feature to att importsequencenumber with a unique number.
Same number for all import, the number can be random or manually set by user.
If random, it should be a new number for every "start import".

Now easier to track new imported records.

• Deployment: Online
• DB Version: 9.2.21113.132
• Connection Controls Version: 1.2021.9.46
• XrmToolBox Version: 1.2021.12.53
• Tool Version: 1.2021.3.1

Noticed the shuffle tasks failing when upgraded dependencies and published an update. The error was Export-CrmShuffle: Could not load file or assembly 'Microsoft.IdentityModel.Clients.ActiveDirectory, Version=3.19.8.16603, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

Hi,

I'm having issues importing N-N relations.
when I try to import one of these intersects I'm getting the following error

invalid attribute count for intersect objectParameter name: AttribuesActual value was 3.

I've tried different configurations in the xml, all with the same result.

<DataBlock Name="Entity Permission Web Role" Entity="adx_entitypermission_webrole" Type="Intersect"> <Export> <Attributes> <Attribute Name="adx_entitypermissionid" IncludeNull="true" /> <Attribute Name="adx_webroleid" IncludeNull="true" /> </Attributes> </Export> <Import CreateWithId="true"> <Match> <Attribute Name="adx_entitypermission_webroleid" /> </Match> </Import> <Relation Block="Entity Permission" Attribute="adx_entitypermissionid" /> <Relation Block="Web Role" Attribute="adx_webroleid" /> </DataBlock>

<DataBlock Name="Entity Permission Web Role" Entity="adx_entitypermission_webrole" Type="Intersect"> <Export> <Attributes> <Attribute Name="adx_entitypermission_webroleid" /> <Attribute Name="adx_entitypermissionid" IncludeNull="true" /> <Attribute Name="adx_webroleid" IncludeNull="true" /> </Attributes> </Export> <Import CreateWithId="true"> <Match> <Attribute Name="adx_entitypermission_webroleid" /> </Match> </Import> <Relation Block="Entity Permission" Attribute="adx_entitypermissionid" /> <Relation Block="Web Role" Attribute="adx_webroleid" /> </DataBlock>

<DataBlock Name="Entity Permission Web Role" Entity="adx_entitypermission_webrole" Type="Intersect" > <Export> <Attributes> <Attribute Name="adx_entitypermissionid" IncludeNull="true" /> <Attribute Name="adx_webroleid" IncludeNull="true" /> </Attributes> </Export> <Import CreateWithId="true" > <Match> <Attribute Name="adx_entitypermissionid" /> <Attribute Name="adx_webroleid" /> </Match> </Import> <Relation Block="Entity Permission" Attribute="adx_entitypermissionid" /> <Relation Block="Web Role" Attribute="adx_webroleid" /> </DataBlock> </Blocks> </ShuffleDefinition>

We are getting the following error on an import of an intersect entity

*** Error record: <null> ***
	Unable to cast object of type 'System.Guid' to type 'Microsoft.Xrm.Sdk.EntityReference'.
4	      
*** Exception:
System.InvalidCastException: Unable to cast object of type 'System.Guid' to type 'Microsoft.Xrm.Sdk.EntityReference'.
   at Cinteros.Crm.Utils.Shuffle.Shuffler.ImportDataBlock(IExecutionContainer container, DataBlock block, EntityCollection cEntities)
---------------------------------------------------------
      System.InvalidCastException: Unable to cast object of type 'System.Guid' to type 'Microsoft.Xrm.Sdk.EntityReference'.
   at Cinteros.Crm.Utils.Shuffle.Shuffler.ImportDataBlock(IExecutionContainer container, DataBlock block, EntityCollection cEntities)
      Unable to cast object of type 'System.Guid' to type 'Microsoft.Xrm.Sdk.EntityReference'.
      Innofactor.Crm.XTB.Shuffle
         at Cinteros.Crm.Utils.Shuffle.Shuffler.ImportDataBlock(IExecutionContainer container, DataBlock block, EntityCollection cEntities)
---------------------------------------------------------

Sample of the data file:

<Block Name="nrq_nrq_assettype_nrq_assettype" Count="19">
    <Entities>
      <Entity name="nrq_nrq_assettype_nrq_assettype" id="b7338001-8b03-eb11-a813-000d3a2c0643">
        <Attribute name="nrq_assettypeidone" type="Guid">8e1a0f97-2903-eb11-a813-000d3aae65d4</Attribute>
        <Attribute name="nrq_assettypeidtwo" type="Guid">505cb840-f5f8-ea11-a815-000d3aae65d4</Attribute>
      </Entity>
      <Entity name="nrq_nrq_assettype_nrq_assettype" id="d719a107-8b03-eb11-a813-000d3a2c0643">
        <Attribute name="nrq_assettypeidone" type="Guid">c7f03f3c-2a03-eb11-a813-000d3a2e9b6c</Attribute>
        <Attribute name="nrq_assettypeidtwo" type="Guid">505cb840-f5f8-ea11-a815-000d3aae65d4</Attribute>
      </Entity>

Cannot validate Solution Import block with Overwrite customizations or Publish after import set to false, as these attributes are required by the schema but the default values for these properties in the tool are False which removes them from the XML when unchecked.

Data files contain a timestamp ExportTime.
This makes version control think that every export contains changed data.
It should be possible to instruct Shuffle to exclude ExportTime.

The Configuration Migration Tool from Microsoft uses an XML based schema to define data to transport, quite similar to a simple Shuffle Definition file.
Sample schemas for moving Portal configuration can be found here.

It would be great to be able to import these schema files and convert to Shuffle Definitions, for projects wanting to take their data transportation to the next level.

For records duplicaterule SetStatus doesn't work. PublishDuplicateRuleRequest/UnpublishDuplicateRuleRequest should be used instead.