Dremio Cloner is a python-based utility for Dremio Enterprise. It supports the following commands: get, put, cascade-acl, report-acl, report-reflections.

Dremio Cloner can be utilized for:

• Migrating entire Dremio environments, for example, from community edition to enterprise edition
• CI/CD processes
• Disaster Recovery scenarios
• Partial backup/restore
• Security Audit reporting
• Reflection reporting

Dremio Cloner is executed with the following command:

python dremio_cloner.py [config_file.json]`

Dremio Migration Tool helps to migrate spaces and folders to new paths. This is done by reading in a Dremio Cloner Export, modifying it and writing it into a new directory or file. It also rewrites and reformats the SQL queries, probably SQL comments can be lost. The output file or directory can be read by Dremio Cloner and written into a destination system.

The Migration Tool is executed with the following command:

python dremio_migration.py [config_migration_file.json]

Dremio Cloner requires Python 3 and requires some additional Python libraries, please install:

$ pip install mo-sql-parsing requests

If you are using Dremio Migration Tool, you additionally need to install sqlparse:

$ pip install sqlparse

Older versions of Dremio Cloner used the Python package moz-sql-parser, which is now deprecated and got replaced by mo-sql-parsing. If you ran an older version of Dremio Cloner before, you need to uninstall these packages before installing mo-sql-parsing.

$ pip list
...
mo-dots            4.22.21108
mo-future          3.147.20327
mo-imports         3.149.20327
mo-kwargs          4.22.21108
mo-logs            4.23.21108
moz-sql-parser     4.40.21126

$ pip uninstall -y moz-sql-parser mo-dots mo-future mo-imports mo-kwargs mo-logs 
$ pip install mo-sql-parsing requests

Command "get" selectively saves definitions for objects such as Source, Space, Folder, PDS, VDS, ACLs, Reflections, Queues, Rules, Tags, Wikis, and Votes from a Dremio environment into a JSON file.

The command is configured with a JSON file with configuration attributes listed below. For detailed description of the configuration JSON attributes, see Reference section below in Appendix 1.

• "command":"get"
• "source": defines source Dremio Environment with
  • "endpoint"
  • "username"
  • "password"
  • "verify_ssl"
  • "is_community_edition"
  • "graph_api_support"
  • "is_dremio_cloud"
  • "dremio_cloud_org_id"
  • "dremio_cloud_project_id"
• "target": defines an output filename or a directory with
  • "filename"
  • "directory"
  • "overwrite"
• "options":
  • logging options
    • "logging.level"
    • "logging.format"
    • "logging.filename"
    • "logging.verbose"
  • miscellaneous options
    • "max_errors"
    • "http_timeout"
  • scope of Space processing
    • "space.process_mode"
    • "folder.process_mode"
    • "space.filter"
    • "space.filter.names"
    • "space.exclude.filter"
    • "space.folder.filter"
    • "space.folder.filter.paths"
    • "space.folder.exclude.filter"
  • scope of Source processing
    • "source.process_mode"
    • "source.filter"
    • "source.filter.names"
    • "source.filter.types"
    • "source.exclude.filter"
    • "source.folder.filter"
    • "source.folder.filter.paths"
    • "source.folder.exclude.filter"
  • scope of PDS processing
    • "pds.process_mode"
    • "pds.filter"
    • "pds.filter.names"
    • "pds.exclude.filter
    • "pds.list.useapi"
  • scope of VDS processing
    • "vds.process_mode"
    • "vds.filter"
    • "vds.filter.names"
    • "vds.exclude.filter"
    • "vds.dependencies.process_mode"
  • scope of Reflection processing
    • "reflection.process_mode"
    • "reflection.id_include_list"
    • "reflection.only_for_matching_vds&quot
  • scope of Workload Management processing
    • "wlm.queue.process_mode"
    • "wlm.rule.process_mode"
  • scope of processing other objects
    • "user.process_mode"
    • "group.process_mode"
    • "wiki.process_mode"
    • "tag.process_mode"
    • "home.process_mode"
    • "vote.process_mode"

Please see a sample JSON configuration file in the config folder of this repository.

Command "put" selectively updates an existing Dremio Environment from a JSON file previously generated by "get" command.

Command "put" can also process ACL transformation. For example, it can transform ACLs to use LDAP_GROUP_PROD instead of LDAP_GROUP_DEV.

Command "put" can also process Source transformations. For example it can transform paths and references in objects to use SOURCE_PROD instead of SOURCE_DEV. PLEASE NOTE: Use of the source transformation feature is against best practices. As a best practice it is recommended that sources are named the same in all environments. In addition, for the source transformation to succeed as expected you must ensure that no VDS, PDS, Column or Folder in the system contains the same name (nor will it contain an exact substring match) as the original\source data source name.

The command is configured with a JSON file with configuration attributes listed below. For detailed description of the configuration JSON attributes, see Reference section below in Appendix 1.

• "command":"put"
• "source": defines an output filename with
  • "filename"
  • "directory"
• "target": defines target Dremio Environment with
  • "endpoint"
  • "username"
  • "password"
  • "verify_ssl"
  • "is_community_edition"
  • "is_dremio_cloud"
  • "dremio_cloud_org_id"
  • "dremio_cloud_project_id"
• "options":
  • logging options
    • "logging.level"
    • "logging.format"
    • "logging.filename
    • "logging.verbose"
  • miscellaneous options
    • "max_errors"
    • "http_timeout"
    • "source.retry_timedout"
    • "dry_run"
  • processing of User and Group objects missing in the target environemnt
    • "space.ignore_missing_acl_user"
    • "space.ignore_missing_acl_group"
    • "folder.ignore_missing_acl_user"
    • "folder.ignore_missing_acl_group"
    • "source.ignore_missing_acl_user"
    • "source.ignore_missing_acl_group"
    • "pds.ignore_missing_acl_user"
    • "pds.ignore_missing_acl_group"
    • "vds.ignore_missing_acl_user"
    • "vds.ignore_missing_acl_group"
  • scope of Space processing
    • "space.process_mode"
    • "space.filter"
    • "space.filter.names"
    • "space.exclude.filter"
    • "folder.process_mode"
    • "space.folder.filter"
    • "space.folder.filter.paths"
    • "space.folder.exclude.filter"
  • scope of Source processing
    • "source.process_mode"
    • "source.filter"
    • "source.filter.names"
    • "source.filter.types"
    • "source.exclude.filter"
    • "source.folder.filter"
    • "source.folder.filter.paths"
    • "source.folder.exclude.filter"
  • scope of PDS processing
    • "pds.process_mode"
    • "pds.filter"
    • "pds.filter.names"
    • "pds.exclude.filter
    • "pds.list.useapi"
  • scope of VDS processing
    • "vds.process_mode"
    • "vds.filter"
    • "vds.filter.names"
    • "vds.exclude.filter"
    • "vds.max_hierarchy_depth"
  • scope of Reflection processing
    • "reflection.process_mode"
    • "pds.reflection_refresh_mode"
    • "reflection.id_include_list"
  • scope of processing other objects
    • "user.process_mode"
    • "group.process_mode"
    • "wiki.process_mode"
    • "tag.process_mode"
    • "home.process_mode"
    • "vote.process_mode"
  • acl transformation processing
    • "transformation"
      • "acl"
        • "file"
  • source transformation processing
    • "transformation"
      • "source"
        • "file"

Please see a sample JSON configuration file in the config folder of this repository.

Command "cascade-acl" selectively propagates ACLs in an object hierarchy.

The command is configured with a JSON file with configuration attributes listed below. For detailed description of the configuration JSON attributes, see Reference section below in Appendix 1.

• "command":"cascade-acl"
• "target": defines Dremio Environment to be processed with
  • "endpoint"
  • "username"
  • "password"
  • "verify_ssl"
• "options":
  • logging options
    • "logging.level"
    • "logging.format"
    • "logging.filename
    • "logging.verbose"
  • miscellaneous options
    • "max_errors"
    • "http_timeout"
    • "source.retry_timedout"
    • "dry_run"
  • scope of Space processing
    • "space.filter"
    • "space.exclude.filter"
    • "space.cascade-acl-origin.override-object"
    • "space.folder.filter"
    • "space.folder.exclude.filter"
    • "space.folder.cascade-acl-origin.filter"
  • scope of Source processing
    • "source.filter"
    • "source.exclude.filter"
    • "source.cascade-acl-origin.override-object"
    • "source.folder.filter"
    • "source.folder.exclude.filter"
  • scope of PDS processing
    • "pds.filter"
    • "pds.exclude.filter
    • "pds.list.useapi"
  • scope of VDS processing
    • "vds.filter"
    • "vds.exclude.filter"

Note, if none of space.cascade-acl-origin.override-object, space.folder.cascade-acl-origin.filter, and source.cascade-acl-origin.override-object specified:

• each Space ACL will be propagated through its hierarchy and applied to Folders and VDSs as per filter configuration
  • To cascade ACLs for all spaces, specify {"space.filter": "*"}
  • To omit cascading any space ACLs, specify {"space.filter": ""}
  • To cascade ACLs for a specific named space, specify {"space.filter": "spacename"} where spacename should be replaced with the actual name of the space
• each Source ACL will be propagated through its hierarchy and applied to PDSs as per filter configuration
  • To cascade ACLs for all sources, specify {"source.filter": "*"}
  • To omit cascading any source ACLs, specify {"source.filter": ""}
  • To cascade ACLs for a specific named source, specify {"source.filter": "sourcename"} where sourcename should be replaced with the actual name of the source

Please see a sample JSON configuration file in the config folder of this repository.

Command "report-acl" produces a selective security report on all objects with ACL in a Dremio environment.

The command is configured with a JSON file with configuration attributes listed below. For detailed description of the configuration JSON attributes, see Reference section below in Appendix 1.

• "command":"report-acl"
• "source": defines Dremio Environment with
  • "endpoint"
  • "username"
  • "password"
  • "verify_ssl"
  • "is_rbac_version"
• "target": defines an output filename with
  • "filename"
• "options":
  • logging options
    • "logging.level"
    • "logging.format"
    • "logging.filename
    • "logging.verbose"
  • miscellaneous options
    • "max_errors"
    • "http_timeout"
    • "source.retry_timedout"
  • report format
    • "report.csv.delimiter"
    • "report.csv.newline"
  • scope of Space processing
    • "space.filter"
    • "space.exclude.filter"
    • "space.folder.filter"
    • "space.folder.exclude.filter"
  • scope of Source processing
    • "source.filter"
    • "source.exclude.filter"
    • "source.folder.filter"
    • "source.folder.exclude.filter"
  • scope of PDS processing
    • "pds.filter"
    • "pds.exclude.filter
    • "pds.list.useapi"
  • scope of VDS processing
    • "vds.filter"
    • "vds.exclude.filter"

Please see a sample JSON configuration file in the config folder of this repository.

Command "report-reflections" produces a reflection report with reflection usage information and ranking on potentially duplicate reflections.

The command is configured with a JSON file with configuration attributes listed below. For detailed description of the configuration JSON attributes, see Reference section below in Appendix 1.

• "command":"report-acl"
• "source": defines Dremio Environment with
  • "endpoint"
  • "username"
  • "password"
  • "verify_ssl"
• "target": defines an output filename with
  • "filename"
• "options":
  • logging options
    • "logging.level"
    • "logging.format"
    • "logging.filename
    • "logging.verbose"
  • miscellaneous options
    • "max_errors"
    • "http_timeout"
    • "source.retry_timedout"
  • report format
    • "report.csv.delimiter"
    • "report.csv.newline"

Note, that this command does not provide any option for Scope definition. Please see a sample JSON configuration file in the config folder of this repository.