Giter VIP home page Giter VIP logo

coyote's Introduction

Coyote DX

Build Status codecov Code Climate

This is a lightweight toolkit for performing basic data exchange (integration) tasks.

The goal has evolved into creating a data exchange tool along the line of build tools like Maven, Gradle and Ant, where a configuration file (pom.xml, build.gradle and build.xml respectively) is written and run (mvn, gradle and ant respectively). The result being data read from one system and written to another.

Using CoyoteDX, it is possible to craft an "exchange" file (e.g. newcontacts.json) and calling Coyote DX to run that exchange (e.g. cdx newcontacts.json). No coding necessary; all the components are either contained in Coyote DX or a library dropped into its path.

The primary use case involves running this "exchange" as required, most likely in cron, a scheduler, or a dedicated service running exchanges on the host. An exchange is run every 15 minutes for example. This models the well known batch integration pattern.

A related use case involves the exchange running continually as a background process executing when some event occurs. The exchange blocks until the event is detected at which time the exchange processes the event. For example, the exchange job may wait until new record has been added to the source system and when that record is detected, it is read in and passed through to the destination system(s). In this manner, it operates less like a batch exchange and more like a real time exchange handling time sensitive data as it becomes available.

So far, it has been useful in integrating applictions, connecting field devices to the cloud, performing load, performance, and integration testing, modeling new data exchanges, keeping systems in-synch during migrations, testing service APIs and connecting hardware prototypes to test systems. This tool is helpful in more cases than integration; is can be used to exchange data between any network connected system actor. One application involved monitoring systems, regularly polling system metrics and writing events when metrics exceeded threshholds.

New components are being added regularly and its design supports third-party contributions without recompilation. Just add your library to the path and your "exchange" file can reference the new components in that library.

Documentation

There is a project wiki with is constantly updated with the latest information about the toolkit, how to use it, examples and best practices. It is the primary source for infromation on the toolkit.

Development Status

This library is currently past prototyping in initial development and well into testing. Documentation is being generated and the code is in use by integration teams uncovering issues for resolution and new use cases for toolkit expansion.

No broken builds are checked-in, so what is in this repository should build and provide value. This project is indirectly supporting integration efforts and is being tested using real world scenarios. As new use cases are discovered the toolkit is updated to support them.

Feel free to copy whatever you find useful and please consider contributing so others may benefit as you may have.

Project Goals

This project has a simple goal: make executing data exchange jobs quick and simple.

  • Configuration file based, for easy operation of many tasks (i.e "exchange" file),
  • Support command-line operations with no coding (just an "exchange" file),
  • Do not require complicated frameworks or facilities (e.g. containers)
  • Enable integrations prototyping, and development operations
  • Provide utilities to assist in the reading, writing and transformation of data,
  • Simple Configuration; if we need a GUI to configure the tools, we are not simple enough.

coyote's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

coyote's Issues

Vault is caches statically in Template - Service Jobs may overwrite

Jobs often specify secrets in a vault. A single vault instance is statically cached in the Template cache. Since services contain multiple Jobs, the last Job specifying a vault will win.

Services need to have a separate Vault configuration and Jobs within those services should share that vault.

Runtime Template Delimiters

Look into setting up templates that use different delimiters specifying when they are evaluated. For example
{# token #} - evaluated during configuration
[# token #] - evaluated during runtime

The idea is to clearer resolution of symbols and to allow tokens to remain in templates when they cannot possibly be resolved.

Feature Template

This is a feature

As a I so that

Acceptance Criteria

Scenario-given-when-then

Scenario-Given-When-Then

Test Strategy

This will require unit tests, system tests, and E2E tests.

Security

This will not change the security disposition
There are no risks identified.

Tasks

Here are some tasks

  • [] Task 1
  • {] Task 2
  • [] Task 3

Installer - Console Mode Line Endings

When using the installer in console mode, it has been observed that the line endings of the cdx and csv script files were converted to DOS line endings.

It could be a setting in IzPack, or the product was built on a Windows workstation with Git line endings configured incorrectly.

This issue needs to be confirmed.

Uniform Dependency Versions

In creating a base Docker image, it was noticed that we have multiple versions of the same library. For example:

  • slf4j-api-1.7.21.jar & slf4j-api-1.7.7.jar
  • commons-net-3.3.jar& commons-net-3.6.jar

There are also several unnecessary files (like Selenium APIs) that do not belong in the base image and may not be necessary for the project.

Go through all the JARs in the base image and find duplicates and remove them.
Remove Selenium (and related) libraries.

JdbcWriter AutoAdjust

The AutoAdjust feature does not check size correctly and will attempt to adjust the sizes of all columns in an Oracle table.

Size comparisons should be fixed, and probably only used on text/character fields.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.