marklogic-attic / xqsync Goto Github PK

XQSync is a command-line, Java-based tool, useful for synchronizing MarkLogic databases to and from other databases, filesystems, and zip-files.

License: Other

Shell 0.53% Java 99.26% XQuery 0.22%

java xqsync marklogic xcc

xqsync's Introduction

XQSync: a Wheelbarrow for Content

MarkLogic Server includes built-in support for online, transactional backup and restore of both databases and forests. However, the on-disk format of these backups is platform-specific.

XQSync is an application-level synchronization tool that can copy documents and their metadata between MarkLogic databases. It can also package documents and their metadata as zip archives, or write them directly to a filesystem. XQSync can synchronize an entire database, a collection, a directory, or the results of evaluating an XQuery expression. Finally, XQSync can make some simple changes along the way: it can add a prefix or append a suffix to every document URI, and it can add new read permissions to every document.

Use XQSync when:

You want to copy a database between platforms.
You want to back up a portion of a database.

Note: Starting with MarkLogic 6, Marklogic Content Pump (mlcp), is a fully-supported tool that covers the same ground as this long-standing open source project. Content Pump is not supported on older versions of MarkLogic Server. Stick with XQSync if you are running earlier versions of MarkLogic.

Running XQSync

XQSync is a Java command-line tool. The entry point is the main method in the com.marklogic.ps.xqsync.XQSync class. This class takes zero or more property files as its arguments. Any specified system properties will override file-based properties, and properties found in later files may override properties specified in earlier files on the command line. See src/xqsync.sh for a sample shell script.

java -cp xqsync.jar:xcc.jar:xstream.jar:xpp3.jar com.marklogic.ps.xqsync.XQSync

Note: XQSync needs a lot of heap space for large synchronization tasks. Be prepared to increase the Java VM heap space limit, using -Xmx. Depending on the version of Java used, -Xincgc may also help.

Required properties:

one of: INPUT_PACKAGE, INPUT_CONNECTION_STRING
one of: OUTPUT_PACKAGE, OUTPUT_CONNECTION_STRING

Note that these requirements can be overriden by a subclass of com.marklogic.ps.xqsync.Configuration. See Customization for details.

A full listing of available properties.

Getting Help

If you have a question, have an issue, or have a feature request:

Post a question to Stack Overflow with the markogic and xqsync tags.
Submit issues or feature requests at https://github.com/marklogic-community/xqsync/issues

xqsync's People

Contributors

Stargazers

Watchers

Forkers

ukcrpb6 mblakele alexthaler coveros-archive vladistan igor-bartashevich pebsconsulting vijayeluri

xqsync's Issues

skip retries on XDMP-SPECIALPROP

The retries are not needed since XDMP-SPECIALPROP is not a transient error. We could also use some docs on why XDMP-SPECIALPROP happens.

2013-12-16 10:00:13.286 WARNING: error writing document (/tests/229), will retry 4 more times.
2013-12-16 10:00:13.549 WARNING: error writing document (/tests/229), will retry 3 more times.
2013-12-16 10:00:14.059 WARNING: error writing document (/tests/229), will retry 2 more times.
2013-12-16 10:00:15.070 WARNING: error writing document (/tests/229), will retry 1 more times.
2013-12-16 10:00:17.083 SEVERE: sync failed for: /tests/229
2013-12-16 10:00:17.083 WARNING: error in input package /Users/mblakele/backup/tests/tests-000.zip
2013-12-16 10:00:17.084 SEVERE: non-fatal
java.util.concurrent.ExecutionException: com.marklogic.ps.xqsync.SyncException: write failed, all retries exhausted for /tests/229
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at com.marklogic.ps.xqsync.Monitor.monitor(Monitor.java:154)
    at com.marklogic.ps.xqsync.Monitor.run(Monitor.java:87)
Caused by: com.marklogic.ps.xqsync.SyncException: write failed, all retries exhausted for /tests/229
    at com.marklogic.ps.xqsync.SessionWriter.write(SessionWriter.java:305)
    at com.marklogic.ps.xqsync.XQSyncDocument.write(XQSyncDocument.java:164)
    at com.marklogic.ps.xqsync.XQSyncDocument.sync(XQSyncDocument.java:128)
    at com.marklogic.ps.xqsync.CallableSync.call(CallableSync.java:87)
    at com.marklogic.ps.xqsync.CallableSync.call(CallableSync.java:32)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:695)
Caused by: com.marklogic.xcc.exceptions.XQueryException: XDMP-SPECIALPROP: xdmp:document-set-properties("/tests/229", <prop:last-modified xmlns:prop="http://marklogic.com/xdmp/property">2013-12-08T15:40:38-08:00</prop:last-modified>) -- Cannot update server-maintained properties
 [Session: user=q, cb=test [ContentSource: user=q, cb=test [provider: address=localhost/127.0.0.1:9000, pool=0/64]]]
 [Client: XCC/6.0-1, Server: XDBC/7.0-1]
on line 4

XPP3 links are broken

The XQSync README file points to jar files for XPP3 at http://www.extreme.indiana.edu/dist/java-repository/xpp3/jars/ but that link isn't working.

Ditto the links on the XPP3 page at http://www.extreme.indiana.edu/xgws/xsoap/xpp/ .

Is this becoming orphanware?

XCC 9.0.1 Session.commit() now returns boolean instead of void

XCC Session.commit() has changed in 9.0.1 to now return boolean instead of being void.

XQSync Session implements the Session interface and must be updated in order to be compatible with XCC 9.0.1 (and later).

com.marklogic.ps.Session is not abstract and does not override abstract method commit() in 
com.marklogic.xcc.Session
public class Session implements com.marklogic.xcc.Session {
       ^
xqsync/src/main/java/com/marklogic/ps/Session.java:99: error: commit() 
in com.marklogic.ps.Session cannot implement commit() in 
com.marklogic.xcc.Session
    public void commit() throws RequestException{
                ^
  return type void is not compatible with boolean

tutorial is out of date for MarkLogic 5

It refers to port 8000 for the use cases.

Java 7 compatibility

http://marklogic.github.com/xqsync/tutorial.html states that XQSync does not work with Java 1.7.

I am not sure if it is still the case as xqsync seems to work with Java 7.

What kind of problems one can encounter when running xqsync on java 7?
If there are still problems it would be good to make the library compatible with java 7 and if it is already compatible tutorial needs to be updated.

Invalid zip files being generated

For some reason, our xqsync zip file created by an export xqsync process can't be opened by the import xqsync process. Linux unzip can open it, however.

Initially we thought this was because of double-slashes ("//") in some URIs creating invalid filenames inside the zip, but after excluding them the zip is still not working. Error reads:

2012-02-23 13:04:56.494 SEVERE: fatal error
java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(Unknown Source)
at java.util.zip.ZipFile.(Unknown Source)
at com.marklogic.ps.xqsync.InputPackage.(InputPackage.java:73)

Additional diagnostics or tips here would be appreciated.

connection string doesn't like ^ as a character in password

(mle-1.b) ==> bin/xqsync.sh '-DINPUT_CONNECTION_STRING=xcc://foo:bar^@localhost:8010/MarkMail' -DOUTPUT_PACKAGE=export.zip '-DINPUT_COLLECTION_URI=export-doc-attachments'

added system properties
logging to CONSOLE
logging to file simplelogger-%u-%g.log
2013-06-21 18:24:39.747 INFO: setting up com.marklogic.ps.SimpleLogger@3597a37c for: com.marklogic.ps
2013-06-21 18:24:39.750 INFO: XQSync starting: version 2012-08-27.0 on 1.6.0_31 (Java(TM) SE Runtime Environment)
2013-06-21 18:24:39.753 INFO: XCC version = 6.0-3
2013-06-21 18:24:39.758 INFO: setting up com.marklogic.ps.SimpleLogger@3597a37c for: com.marklogic.ps
2013-06-21 18:24:39.759 INFO: Configuration is com.marklogic.ps.xqsync.Configuration
2013-06-21 18:24:39.759 INFO: setting up com.marklogic.ps.SimpleLogger@3597a37c for: com.marklogic.ps
2013-06-21 18:24:39.759 INFO: first-time setup
2013-06-21 18:24:39.760 INFO: setting up com.marklogic.ps.SimpleLogger@3597a37c for: com.marklogic.ps
2013-06-21 18:24:39.763 INFO: using OUTPUT_PACKAGE=export.zip
2013-06-21 18:24:39.771 INFO: using INPUT_CONNECTION_STRING=xcc://foo:bar^@localhost:8010/MarkMail
2013-06-21 18:24:39.772 INFO: using INPUT_COLLECTION_URI=export-doc-attachments
2013-06-21 18:24:39.775 INFO: input from connection:
Exception in thread "main" java.net.URISyntaxException: Illegal character in authority at index 6: xcc://foo:bar^@localhost:8010/MarkMail
at java.net.URI$Parser.fail(URI.java:2810)
at java.net.URI$Parser.parseAuthority(URI.java:3148)
at java.net.URI$Parser.parseHierarchical(URI.java:3059)
at java.net.URI$Parser.parse(URI.java:3015)
at java.net.URI.(URI.java:577)
at com.marklogic.ps.xqsync.Configuration.configureInput(Configuration.java:412)
at com.marklogic.ps.xqsync.Configuration.configure(Configuration.java:322)
at com.marklogic.ps.xqsync.XQSync.initConfiguration(XQSync.java:128)
at com.marklogic.ps.xqsync.XQSync.main(XQSync.java:71)

Workaround is to URL encode the character as %5E, like

xcc://foo:bar%5E@localhots:8010/MarkMail