kevinjump / usync-legacy Goto Github PK
View Code? Open in Web Editor NEWuSync for Umbraco 4/6 - bits of the database to disk, and stuff like that
Home Page: https://jumoo.co.uk/usync/
License: Mozilla Public License 2.0
uSync for Umbraco 4/6 - bits of the database to disk, and stuff like that
Home Page: https://jumoo.co.uk/usync/
License: Mozilla Public License 2.0
Migrated issue https://github.com/KevinJump/jumps.umbraco.usync/issues/61 from old repo:
Hi,
I'm having a lot of problems with the images importing with uSync. I'm doing what Kevin Jump said me, it means put the media importing to false and leaving the images in the root media folder (in the uSync>Data>Media I just have the configs). The problem what I'm having now is that I'm doing a full import and the images are created in the media tab in Umbraco but they are not visible, is very strange, I can see the item in the media tab but when I access the image doesn't appear and the website doesn't charge them neither.
Any thoughts?
Thanks!
I can see as well that when uSync imports the images a part of the problem that it's importing them in wrong way is changing all the config files from each media in uSync folder to remove the src href to the media folder and leave it empty :( it's frustrating.
I'm trying to use uSync to migrate content alongside its media from one umbraco installation to another.
I have installed uSync.BackOffice.3.0.2 and uSync.ContentEdition.3.0.0 on an Umbraco 7.3 on both installations.
I can manual export from one installation and then manual full import on another by copying the files over to the "\uSync\data" folder.
Everything is copied over as expected, except the media references in the content nodes. The media is there in the media library and the files are there but where those media files have been referenced in the content nodes, they do not appear (media picker and rich text area).
I think it's worth to note that I am also heavily using Archetype for my content nodes so the media pickers and the rich text content all sit within an archetype data type which is also being exported/imported without any problems.
Does uSync create new id's for imported media items?
Thanks
The order in which compositions are written out to the xml are inconsistent, and this can cause unnecessary change detection when running usync.
if you have two or more compositions in a doctype they will be written out to the xml on export
<Compositions>
<Composition Key="3ebac230-ef1f-4f00-8c5b-9958a90b923f">sEOData</Composition>
<Composition Key="3936644d-a5b2-4d1e-b70c-df4613bac6d2">pageVisability</Composition>
</Compositions>
however the order is dependent on internal umbraco factors, so on a compare the order might be reveresed, this causes the file to be diffrent and usync to fire a change. in the dashboard you get an apparent change of composistion
This is not critical as the sync will work, but its confusing and should be fixed
When a Document Type has the same name as a folder in the document types the import fails with an error: Sequence contains more than one matching element
this is because we are looking for children of a folder, and finding more than one - because we are not looking for only folders.
to fix this the call for children needs to ask for only container items
var children = _entityService.GetChildren(parent.Id, UmbracoObjectTypes.DocumentTypeContainer).ToArray();
If you delete a content node from Umbraco (v7.3), then restart Umbraco (by modifying the web.config), on reload of the Umbraco backoffice or Front Office, we receive an exception which states: "Cannot insert duplicate key row in object 'dbo.umbracoNode' with unique index 'IX_umbracoNode_uniqueID'. "
This seems to be caused by uSync (ContentEdition v3.0.0) trying to insert the content node even though I deleted it. Since the content node is still in the database, but is marked as "trashed".
The way to fix this is to delete the folder in uSync\data\content that matches the content node I deleted.
On those shockingly rare occasions when you get an error deep inside uSync on startup, the resulting YSOD doesn't show the full stack trace.
just need to make sure we are throwing things properly.
I am running into an issue where after performing NuGet package restore (from a new pull of our branch) I am getting the following error from the developer section:
Request error: The URL returned a 404 (not found): backoffice/uSync/uSyncApi/GetSettings
It works fine on my machine (where the project was initially created) - but does not work on other machines I restore from. This is the folder for USync as it stands (on all machines):
and here is the NuGet Screen:
I am not sure how to fix this problem 100%, so I wanted to report it.
Update the Dashboard from a nasty old ascx file to jazzy angular.
I've noticed that the uSync projects are targeting a mix of .NET framework 4.5.1 and 4.5.2. Would it be possible to just target 4.5 across all the uSync projects? It doesn't appear there is any specific need to target a later version.
I ask because we have use uSync in our own custom sync controller that toggle handlers based on the sites' needs and we're still using framework 4.5. I'm currently compiling from the Git source but would much rather use the nuget package!
When uSync looks for the goodwill licence, it logs a info line to the log file.
this is apparently annoying :)
I've been playing around with media types and folders and there appears to be some conflict with Master and Folder when exporting/importing media types.
Given the way the back office behaves, I don't think Master is really necessary or relevant for media types. The back office never really supported inheritance of media items well, and now it looks like it doesn't even show the expansion arrows in such cases.
The problem, as it stands, is that uSync exports media items with their first composition as their master and skips any folder processing. When these items are imported, they're pathed under the first component, and "disappear" in the back office.
It seems to me that, on import, media types' Master can be shoved in a composition and not have any bearing on the path of the item. The Folder property can then be used to generate the appropriate path and parent id.
Content Mapper for Vorto.
On its own vorto usually wouldn't need a mapper, but if you embed datatypes that do need a mapper into a vorto then the vorto needs to trigger the mapping for each of its languages.
I am getting value can not be null exception at startup. As a result usync sync fails.
Following is the stack trace:
[ArgumentNullException: Value cannot be null.
Parameter name: value]
System.Xml.Linq.XAttribute..ctor(XName name, Object value) +90525
Umbraco.Core.Services.EntityXmlSerializer.Serialize(ILanguage language) +271
Umbraco.Core.Services.PackagingService.Export(ILanguage language, Boolean raiseEvents) +156
Jumoo.uSync.Core.Serializers.LanguageSerializer.SerializeCore(ILanguage item) +43
Jumoo.uSync.Core.Serializers.LanguageSerializer.IsUpdate(XElement node) +214
Jumoo.uSync.Core.Serializers.SyncBaseSerializer1.DeSerialize(XElement node, Boolean forceUpdate) +50 Jumoo.uSync.BackOffice.Handlers.LanguageHandler.Import(String filePath, Boolean force) +58 Jumoo.uSync.BackOffice.Handlers.uSyncBaseHandler
1.ImportFolder(String folder, Boolean force, Dictionary2 updates) +160 Jumoo.uSync.BackOffice.Handlers.uSyncBaseHandler
1.ImportAll(String folder, Boolean force) +284
Jumoo.uSync.BackOffice.uSyncBackOfficeContext.ImportAll(String folder, Boolean force) +451
Jumoo.uSync.BackOffice.uSyncApplicationEventHandler.Setup() +222
Jumoo.uSync.BackOffice.uSyncApplicationEventHandler.ApplicationStarted(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext) +141
Umbraco.Core.ApplicationEventHandler.OnApplicationStarted(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext) +37
Umbraco.Core.CoreBootManager.b__8(IApplicationEventHandler x) +133
Umbraco.Core.EnumerableExtensions.ForEach(IEnumerable1 items, Action
1 action) +141
Umbraco.Core.CoreBootManager.Complete(Action1 afterComplete) +293 Umbraco.Web.WebBootManager.Complete(Action
1 afterComplete) +60
Umbraco.Core.UmbracoApplicationBase.StartApplication(Object sender, EventArgs e) +234
Umbraco.Core.UmbracoApplicationBase.Application_Start(Object sender, EventArgs e) +9
[HttpException (0x80004005): Value cannot be null.
Parameter name: value]
System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app) +4467101
System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers) +118
System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context) +172
System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) +364
System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext) +290
[HttpException (0x80004005): Value cannot be null.
Parameter name: value]
System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +4531288
System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +94
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +191
Content Edition (or any addon) should report its version to the dashboard.
a recent change added the ability to have a "SecondPass" handler that was called at the end of the import process so that you could do things once all imports had being performed.
for v7.4 I am looking at changing this so you can extend existing handlers to be called post import to do anything that may need doing.
The basics of the flow are:
ISyncHandlers
) are loaded and ran as now
RequiresPostProcessing
ISyncPostImportHandler
) are loaded,
RequiresPostProcessing
this is 'hopefully' nicer as
Basic changed for this feature are checked in KevinJump/uSync@31f8207
It appears that the code that maps integer Ids to Guids is being ran indiscriminately on export. This is particularly apparent when you have a property of type TextString where any 4 to 10 character number can be converted to a Guid if it matches a node in the CMS.
e.g. ExecutableName property with value of "PR1051.exe" has a value of "PR3D08C9E2-D369-426A-875C-19685F7FABBC.exe" after export.
Should TextString properties be mapped at all?
At the moment each bit of uSync that needs it gets its own dashboard tab.
also it's hard to tell when ContentEdition is installed.
Merge the dashboards. into one.
Mapper for the LeBlender Grid,
this one is a bit of a pain, because when we are exporting the grid we look for the editor alias, to direct us to the right mapper, however this is fluid for le blender (you can set it to what ever you want).
so out of the box, the mapper will need to be configured to find the right control after the user creates the le blender grid.
uSync doesn't have a nestedContent Mapper, but adding one shouldn't be to hard (infact @madsoulswe has most of one written already).
Hi we are using Dictionary to localize labels in our application and uSync to push entries between environments.
Our workflow workflow is as follows:
Those translated dictionary entries never "get back" to the development environment, meaning that the source of dictionary entries is an incomplete (missing translations) dataset of the live data.
With this approach we have a problem with the out-of-the-box feature of uSync of updating the dictionary entry, as when new features are pushed to production, translations done by business users are overwritten and, thus, lost.
Is there a way to enable only creation (and eventually deletions) of dictionary entries, without updating them?
I would not mind using a customized handler based on the existing one if I knew how to "register" it so that the customized version is run instead of the out-of-the-box handler.
Thanks for such an awesome and useful product.
The File watcher import - (import when files change on the disk) isn't currently working.
setting <WatchForFileChanges>true</WatchForFileChanges>
in uSyncBackOffice.Config, should cause uSync back office to perform an import shortly (8.128 seconds) after an *.config files are copied into the usync folder.
at the moment this fails:
Error during import: System.NullReferenceException: Object reference not set to an instance of an object.
at Umbraco.Web.RequestLifespanMessagesFactory.Get()
at Umbraco.Core.Services.MediaService.Umbraco.Core.Services.IMediaServiceOperations.Save(IMedia media, Int32 userId, Boolean raiseEvents)
at Umbraco.Core.Services.MediaService.Save(IMedia media, Int32 userId, Boolean raiseEvents)
at Jumoo.uSync.Core.Serializers.MediaSerializer.DeserializeCore(XElement node, Int32 parentId, Boolean forceUpdate)
at Jumoo.uSync.Core.Serializers.ContentBaseSerializer`1.Deserialize(XElement node, Int32 parentId, Boolean forceUpdate)
at Jumoo.uSync.Content.MediaHandler.Import(String file, Int32 parentId, Boolean force)
at Jumoo.uSync.Content.BaseContentHandler`1.ImportFolder(String folder, Int32 parentId, Boolean force, Dictionary`2 updates)
at Jumoo.uSync.Content.BaseContentHandler`1.ImportFolder(String folder, Int32 parentId, Boolean force, Dictionary`2 updates)
at Jumoo.uSync.Content.BaseContentHandler`1.ImportFolder(String folder, Int32 parentId, Boolean force, Dictionary`2 updates)
at Jumoo.uSync.Content.BaseContentHandler`1.ImportAll(String folder, Boolean force)
at Jumoo.uSync.BackOffice.uSyncBackOfficeContext.ImportAll(String folder, Boolean force)
at Jumoo.uSync.BackOffice.uSyncFileWatcher.ChangeTimerElapsed(Object sender, ElapsedEventArgs e)
this is likely caused by issues with context from within the file events.
Affects: Content Edition.
The NameChecker that cleans up folders after a rename, isn't finding the GUID key values in content files so isn't doing the tidy up.
Just need to update the nameChecker GetKey to also do this
Annoyingly the actionlogger can fail and thow an exception when writing out the changes to the usync log. this is annoying because the actual export/import will have worked fine.
Umbraco 7.4 Introduces folders in the Document types, media types and data types trees, support is needed to handle these EntityFolder
items in exports/imports
We had converted a property on a document type (umbNaviHide) into a composition document type which was then added back to the document type that had originally had it.
Since it is a umbraco reserved property name we could not alter the property name on the composition (which is normally how we would handle this). On our dev environment this worked flawlessly because the sequence of actions was very clear:
remove property from doctype A
create new doc type B
add property to doc type B
add doc type B as a composition to doc type A
However, when these changes were synced to the next environment we encountered a runtime error about duplicate property names (umbNaviHide) on doctype A. This runtime error prevented access to {siteurl}/umbraco and as such was not resolvable via the GUI.We fixed the issue by removing the old property from cmsPropertyData and then removing the property from cmsPropertyType.
However, we were concerned about future changes that involve property names and usync and are not that comfortable with such a specific recovery plan as we may not be the developers who maintain the system.
Is there any current plan to help mitigate issues like this?
Assume you have a DataType that represents a MultiNodeTreePicker set to pick a particular document type beneath a particular start node in the content tree:
<?xml version="1.0" encoding="utf-8"?>
<DataType Name="App Picker" Key="285b4e36-ad32-46ac-9866-1992af9d28ca" Id="Umbraco.MultiNodeTreePicker" DatabaseType="Nvarchar">
<Nodes>
<Node Id="1073" Value="Catalogue\Apps" Type="content" MapGuid="1" />
</Nodes>
<PreValues>
<PreValue Id="58" Value="{
 "type": "content",
 "id": 1073
}" Alias="startNode" MapGuid="1" />
<PreValue Id="59" Value="App" Alias="filter" />
<PreValue Id="60" Value="1" Alias="minNumber" />
<PreValue Id="61" Value="" Alias="maxNumber" />
<PreValue Id="62" Value="0" Alias="showEditButton" />
</PreValues>
</DataType>
Note that MapGuid="1" is correctly set on the PreValue containing the id of the startNode.
If importing content into a new Umbraco instance we can't guarantee that NodeIds will be the same so we cannot use the Id in the startNode xml (it needs to be mapped). However, if we try to create the DataType before we have created the Content then mapping will fail (as content doesn't exist). If we instead try to create the Content before the DataType it will fail (as the property can't be created as no DataType).
DataTypes should be parsed in two phases:
When a content type is saved Allowed Templates is not being serialized
AllowedTemplates should contain all the non-default templates for a datatype. it doesn't
the de-serialization code appears to be there . will need to check it
Steps:
Suggestions (needs tweeking)
ContentSerializer:91
IContentType contentType = _contentService.FindContentTypeByAlias(type);
if (item.ContentTypeId != contentType.Id)
{
item.ChangeContentType(contentType);
}
(the type variable is only used in the creation of the content)
The problem here is that FindContentTypeByAlias is a private function and not accessible. To get this to work, we would need Umbraco to change the interface of IContentService (how close are you working with the Umbraco team?).
The other way is to extend Umbraco source code with a function ChangeContentType accepting a string(TypeAlias).
What do you think about this?
Inside content image cropper images will contain path to a media item.
<bannerImage><![CDATA[{
"focalPoint": {
"left": 0.45748987854251011,
"top": 0.40070921985815605
},
"src": "/media/1015/site-banner.jpeg"
}]]></bannerImage>
*This should not be a problem if you use "MoveMedia = true" as you copy the media folder between sites, and the paths stay the same. *
but for movemedia=false During export and subsequent import this media location should be mapped, we need a ImageCropper Mapper to map these values.
previous versions also had a catch all errors and carry on setting. so that you never got a ysod from usync.
<usync
dontThrowErrors="false">
The dashboard buttons should be disabled when running an import just to stop anything firing off.
There are quite a lot of 3rd party mappers in the core now, and every time a mapper gets added it changes the version of the core, which has a knock on effect for the version of back office, when there is no discernable difference to the core behavior of usync.
I am going to move the 3rd party mappers out of the usync.core into their own assembly - this can then be referenced, by usync.content, and if needed by people just using the core. without it causing increments in all the versions of everything.
We hade an issue where 2016 (and other dates) was replace all over the site
Example page, som these were replaced with id's
History page
Hola
Umbraco 7.4.1
uSync 3.1.0.740
Adding additional child document types through the permissions section in the doctype editor does not persist the updated collection to the config files.
No errors that I can see are logged. If you need more info let me know and I'll dig around.
Content Serialization uses the update date for checking if content is new, the time written to disk can occasionally be out by a few milliseconds compared to that in umbraco, this means updates locally are flagged when they are not.
update checking to only apply if time is 1 second or more out.
As I mentioned in my comment to #59, I have a use case where I need to leave properties that in are Umbraco (but not uSync configs) in tact. I'd like to have a way to specify, on the handler, whether the deserialize operation should remove properties or not. This really only makes sense on the ContentType and MediaType serializers.
I've been trying to come up with clean methods to do this and I think the cleanest way to do this would be to pass the uSyncBackOfficeConfig to the Core Init() method (Or at least a bool pulled from the BackOfficeConfig). Everything else looks like it'll just tangle up all the other serializers with extra fields that don't necessarily make sense. I'm not sure that having Core depend on the BackOfficeConfig is appropriate coupling, either.
Any thoughts?
the delayed save is back ๐ข
when datatypes are saved Umbraco Fires the Saved event idependently of the properties getting saved, this means you can end up with missing settings in your datatype.
this use to be aproblem which it turns out hasn't gone away.
usync needs to re-add it's 4 second wait after save (uses timers and queues)
new version is missing uSync.Once.
usync looks for the presence of a usync.once file in the root, if it exists it runs through the import and then creates a usync.locked file.
when a usync.locked file exists no usync runs will occur.
not complex :) but needs to be added back.
for datatypes where there is a mapped id, usync will have a go at finding the ID on the target system, if the target content/template etc doesn't exist - uSync will blank the field.
https://our.umbraco.org/projects/developer-tools/usync/usync/73681-usync-resets-contentpickers
change behavior to just leave in the value whatever you found?
Assume you have a media item that has an umbracoFile property with a value of "/media/1084/image.png" and an associated _usyncMedia folder containing image.png.
Because the umbracoFile property is not loaded prior to moving the image, every full import always uploads every file to new folders resulting in duplicate, orphaned media items and an ever-growing media folder.
The Third Party mapper library will need its own nuget package and it will need to be included as part of the usync.content edition package.
the only complexity is applying xslt transforms to the usynccore.config on install to add all the thirdparty config lines.
In my Umbraco 7.3.0 site I am getting an error when I delete an item from the content tree (moves into recycle bin) and do a full import.
The error:
Cannot insert duplicate key row in object 'dbo.umbracoNode' with unique index 'IX_umbracoNode_uniqueID'. The duplicate key value is {guid}.
The statement has been terminated.
The nuget packages I have installed are
-uSunc v3.0.3
-uSync.Core v5.1.0
-uSync.BackOffice v3.0.3
-uSync.ContentEdition v3.0.0
Steps to reproduce:
An upgrade of umbraco from v7.3 to v7.4 can cause a YSOD when working with member types.
pre v7.4 membertypes had a guid value but it wasn't getting set so it might be . 00000000-0000-0000-0000-000000000000.
subsequently usyncs lookup by guid fails.
Since the refactor in f5e24d9 the ability to import content with a prevalue datatype has regressed.
It appears that the mapping code is only running when importing DataTypes and not when importing the content itself. Note that the export works correctly.
For example, a DropDownList datatype with options { Alias | Name } { 0 | Option 0 }, { 1 | Option 1 }, { 2 | Option 2}. If the content has selected Option 0 it correctly writes the alias of 0 in the .config file. However, on import it is supposed to look up the PreValue and store the Id of the prevalue in the property. Instead, it is simply storing 0 in the property value (and therefore not mapping to the correct prevalue).
Currently content doesn't get mapped if it's inside a grid,
you can just throw the standard mapper at a grid and it will most likely work, but to be sure we should build a grid mapper
Better Control over how handlers behave on a import / export
I know some people have a create only directory handler, so it would be nice to be able to do that type of thing via the config.
Just while i am working this out any comments are welcome.
Admittedly, this is on a site with a lot of doctypes and templates.
On a 8GB PC and the site running in debug mode, I managed to get an OutOfMemoryException while trying to create a snapshot. The snapshot itself should have consisted of a composition doctype consisting of one tab and 5 properties, and changes to 2 doctypes adding the composition to its makeup.
The error appears as a simple message within the uSync snapshot admin page.
I imagine this might come as a bit of a surprise, but if there's any diagnostic or logging steps you wish me to take.
Running Umbraco 7.4.1:
packages.config:
<package id="uSync" version="3.1.1.740" targetFramework="net452" />
<package id="uSync.Core" version="5.3.1.740" targetFramework="net452" />
<package id="uSync.Snapshots" version="1.0.2.740" targetFramework="net452" />
Enhanced change details, so you can looking side an item and see what is changing.
I am getting an error that is a little hard to reproduce but I think I can explain what is going on. Basically uSync is trying to delete a property type but is unable to because there are items in cmsPropertyData that are constraining it. These items have the gridview data for a page we started early in the project; later we removed the gridview but the entries in the database remained. uSync does not have a reference to this property on the content item anymore and must not be looking to remove the items in cmsPropertyData and running into the issue I have below.
Here is more information on the issue. On Import, uSync is throwing this error:
The DELETE statement conflicted with the REFERENCE constraint "FK_cmsPropertyData_cmsPropertyType_id". The conflict occurred in database "MyWebsite", table "dbo.cmsPropertyData", column 'propertytypeid'.
The statement has been terminated.
Running a sql trace I found this statement being executed:
exec sp_executesql N'SELECT *
FROM [cmsContentType]
INNER JOIN [umbracoNode]
ON [cmsContentType].[nodeId] = [umbracoNode].[id]
LEFT JOIN [cmsDocumentType]
ON [cmsDocumentType].[contentTypeNodeId] = [cmsContentType].[nodeId]
WHERE ([umbracoNode].[nodeObjectType] = @0)
AND ([umbracoNode].[parentID] = @1)
ORDER BY ([umbracoNode].[text])',N'@0 nvarchar(40),@1 int',@0=N'a2cb7800-f571-4787-9638-bc48539a0efb',@1=2601
There are 8 delete statements after this using sp_executesql. The delete statements remove all entries from the following tables where nodeId (or reference columns to) = 2601
umbracoUser2NodeNotify
umbracoUser2NodePermission
cmsTagRelationship
cmsContentTypeAllowedContentType
cmsContentTypeAllowedContentType
cmsContentType2ContentType
cmsContentType2ContentType
cmsPropertyType
It is this command that throws the error
exec sp_executesql N'DELETE FROM cmsPropertyType WHERE contentTypeId = @0',N'@0 int',@0=2601
Looking at the database, I can see the entry that the statement is trying to delete in [cmsPropertyType], and there are 4 entries in [cmsPropertyData] that use the [cmsPropertyType].[id] in their propertytypeid columns.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.