Write the Oracle Tools Book

We need to update to support the latest github API - without this we can't publish documentation

Upgrade to use Coherence 3.7.1.10

The Notified implementation can never be guaranteed to operate correctly when we consider the possibility of Operating System "notification". Furthermore the behavior and timing of notification can differ significantly across platforms, especially between Windows and Linux, worse when these platforms are virtualized.

Consequently the safest thing to do is remove this support - it's not used anywhere in the Coherence Incubator.

Oracle Tools fails to work on Windows due to timing and class-path issues

When running Oracle Tools on JDK8 (b91), Coherence (any version using XSD) won't start due to improved/tightened JDK8 XSD loading permissions.

To work around this we should add the following default system property to the Coherence-based Schemas, at least until this is either fixed in JDK8 or Coherence (with a change of XSD loading security).

schema.setSystemProperty("javax.xml.accessExternalSchema", "file");

Consider a test that wants to set numerous System properties, and/or establish local JMX instrumentation for a test.

Instead of having @before test methods do this via calls to System.setProperty(...) etc, it would be nice if we could simply leverage an Application Schema.

eg: In the case of a Coherence-based test we could do this.

@Before
public void setup()
{
ClusterMemberSchema schema = new ClusterMemberSchema().setClusterPort(...);

LocalApplicationBuilder builder = new LocalApplicationBuilder();
LocalApplication application = builder.build(schema);   //<-- will set the local application properties etc, from the schema. }

For platforms that don't correctly isolate Coherence applications, and/or those that don't respect standard network settings, we should provide a top-level Abstract class that we can base all our tests (especially for the Coherence Incubator).

This will ensure we always have the correct isolation.

By introducing a specialized Junit Test Runner for Containers, we can do a lot more isolation and dramatically simplify test setup/cleanup.

eg: we can support annotations like
@SystemIsolation(NONE)
@SystemIsolation(FOR_EACH_TEST_METHOD)
@SystemIsolation(FOR_TEST_CLASS)

This could then be extended for Coherence as well.

NOTE: When we do this we can remove the requirement to inherit from the AbstractTest class (we could theoretically deprecate it)

We noticed that due to earlier cleanup / refactoring of the Container streams I/O is no longer output from JUnit tests when running in Container Mode.

This is due to the "default" scope using the new ContainerScope implementation, that of which captures and pipes output. This is incorrect behavior for the "default" scope, which should instead send I/O directly to the Platform Scope.

When running the VirtualizationClassLoaderTests on Windows with Java 7 and 8, the order of the test method execution becomes significant. This is because some methods cause side-effects which then later methods either implicitly depend upon or don't expect. As the order of method execution for Junit (based on Java 7 and 8 method reflection) is somewhat random on Windows, this can cause tests to fail some what randomly.

We need to ensure that there's no implicit ordering or dependencies or side-effects in our test methods, particularly those testing ClassLoaders.

We need to clean up the missing javadocs and ensure that we have clean builds on Java 6, 7 and 8

Currently the application names produced by an application group builder start at 0. This is confusing, especially when being used with Coherence (and other cluster-based systems), where member names start at 1.

Fixing this will be a BREAKING CHANGE for anyone that assumes the name of an application starts at 0.

Document Coherence Member Schema and Coherence Member Builder

Upgrade to use the latest version of Coherence (or a version pom)

Once we have a custom Container runner we could introduce a specific Coherence-based runner that can setup/establish clusters using annotations.

This will make testing way simpler and more isolated through declarative configuration.

eg:

@cluster(name=RANDOM, port=RANDOM, ttl=0)

We can also extend this to pre-load cache data, record cache data after runs/exceptions and compare cluster state to that which is desired. ie: assert that the state of a cluster after running a test is as expected.

NOTE: When we do this we can remove the requirement to inherit from the AbstractCoherenceTest class (we could theoretically deprecate it)

Add Average to oracle-tools for monitoring/JMX use

When running on some platforms (like Windows) copying all of the System Properties, including those defined at runtime by the Java Platform, from one process and passing them to another can be problematic. In some instances attempting to re-define the Java Platform defined system properties can cause applications (ie: Java) not to start. Hence we need to avoid providing this facility!

To solve this issue we're going to refactor the PropertiesBuilder.fromCurrentSystemProperties method to instead become PropertiesBuilder.fromCurrentNonStandardSystemProperties so that the PropertyBuilder that is created only contains those defined by the end-user.

It turns out we don't actually use this method anywhere anyway - apart from in tests. Anyone using this method outside of the tests are essentially in trouble anyway - as their applications won't be cross platform.

An ApplicationGroupController allows programmatic control of ApplicationGroups at runtime. Specific examples include;

a). the ability to perform automatic "Rolling Restarts" of (specific) Applications in an ApplicationGroup.

b). the ability to introduce Chaos Monkeys (see: http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html) into an Application Group.

c). systematically perform custom actions against Applications in an ApplicationGroup (eg: running a report).

Currently the Deferred#get() method does not permit returning null object references as null has the implied meaning "object not yet available". This is unfortunately as it doesn't let null be returned - when it is a valid object.

To resolve this we need to refactor the semantics so that:

1. We throw an UnresolvableInstanceException - when the Object can never be resolved (thus replacing ObjectNotAvailableException which is misleading)

2. We throw an InstanceUnavailableException - when the Object resolution failed, but may be retried in the future (thus replacing the return value of "null").

I noticed when running test on Windows, specifically using a Jenkins installation that was in "C:\Program Files\jenkins" that the application builders may not work as expected and in some circumstances, actually fail.

The proposed solution is to introduce the concept of a ClassPath class that can be used to represent and manipulate ClassPaths in a platform independent manner, instead of having class path parsing code all over the place.

We really need the ability to kill an application in addition to simply destroying it. This is especially required for failure-based testing.

Note: we may need to use some operating system level calls to achieve this.

Currently we use the term "virtualization" to represent the concept of running a Java application "virtually" with in another application. Though correct in that we actually "virtualize" System properties, I/O, MBean servers etc, this however can be confusing when running such applications in a virtual appliance.

We should rename the internal classes to be a "Container" instead of "Virtualized". While this may cause confusion when discussing using applications in a container, it is closer to what is happening in a traditional sense (with the additional characteristic of isolating I/O streams and MBean servers which containers dont' traditionally do).

We should introduce a Coherence-based StorageMemberSchema to capture the standard settings for a Storage Enabled Members.

Once we've done this we should make the existing CoherenceMemberSchema inherit from the StorageMemberSchema and deprecate it.

When attempting to run Coherence in WKA mode (using system properties), both the WKA port ("tangosol.coherence.wka.port") and the Local Address Port ("tangosol.coherence.localport") need to be set to the same value.

Unfortunately the ClusterMemberSchema doesn't provide an out-of-the-box method to set the LocalPort which is somewhat annoying (as you have to use setSystemProperty).

We should hence;
i). add a "setLocalHostPort" method, and
ii). when a call to "setWellKnownAddressPort" is made, it should also call this method (to make it easy for the developer).

We should provide a simplification of this:

getMBeanAttribute(new ObjectName("Coherence:type=Service,name=Federated[DistributedCache],nodeId=1"),
"StatusHA", String.class)

Often we would like the ability to remotely execute Callables/Runnables in Java Applications (whether they are executing in a container, native or remotely). While we can currently execute Callables when Java Applications are running in a container, we don't yet have the ability or infrastructure to do this in Natively (or eventually Remotely).

Now that we can remotely execute Callables on Applications we can introduce the ability to use this with deferred asserts. ie: assert that eventually executing some remove callable arrives in a required state.

We should introduce an abstract a Coherence-based MemberSchema as the parent of all MemberSchemas. This will allow us to share common settings easily between MemberSchema.

Currently the processes built by the NativeApplicationBuilder(s) will create (child) processes that are independent of the creating (parent) process. Should the parent process terminate (for what ever reason), the child processes will be let running (ie: orphaned). Often this is undesirable (and not the expected default behavior).

Leveraging the work in #51, it should be possible to introduce the concept of orphan-protection for native application builders.

We've already fixed white space issues in the class path, but the system properties that are added to the command must also be sanitized if they have white spaces.

When creating external/native Java processes, it would be nice if we could control/debug them.

To enable this we should add the ability to specify debug properties when using External/Native Java Application builders (they don't apply to Virtual/Container-based builders).

Currently we only allow capturing stdout for an ApplicationConsole. This is problematic when we need to test with applications specifically use other streams, like stderr. Furthermore we currently provide no facility to control or provide stdin for an application, thus again limiting the ability to write fully interactive tests.

To resolve this I propose we redesign/refactor how we use ApplicationConsoles. In particular I suggest the following interface:

interface ApplicationConsole
{
/**
* Obtains the BufferedReader that can be used to read content an Application sent to stdout.
*/
public BufferedReader getStdOutReader();

/**
* Obtains the BufferedReader that can be used to read content an Application sent to stderr.
*/
public BufferedReader getStdErrReader();

/**
* Obtains the PrintWriter that can be used to send input to the stdin of an Application.
*/
public PrintWriter getStdInWriter();

/**
* Closes the ApplicationConsole.
*/
public void close();
}

Document Application Group and Application Group Builders

When using an Iterator there's often the requirement to capture an individual value and then use it later.

For example: When using an AvailablePortIterator to configure a cluster, the first port chosen should be used for all other members in the cluster (so that a cluster is formed).

Instead of choosing the port "early", we often want to delay the choice until the last minute. To do this we need a way of "capturing" a choice and then using the choice in other places.

Capture<Integer> clusterPort = new Capture<Integer>(availablePortIterator);

The first time the clusterPort is used it won't have a value, in which case the availablePortIterator is used to determine a value. After that the first "captured" value will always be used.

This allows a single jar to be generated, thus making adoption easier

Randy suggested that we should move to version 4 from version 3

http://mvnrepository.com/artifact/net.java/jvnet-parent/4

We should introduce a Coherence-based ManagementMemberSchema to capture the standard settings for a Management Node.

When creating application processes we often would like to see diagnostic information about how the process. For example, the environment, settings, classpath etc, that the Oracle Tools established to establish an application.

To provide this capability, on a schema-by-schema basis we should add the ability to do this:

schema.setDiagnosticsEnabled(true);

Additionally, we should allow:

1. this on a builder-by-builder basis.
2. provide a system property (like oracle.tools.runtime.diagnostics=true).

The license headers in all files contain the text:

• You can obtain a copy of the License by consulting the LICENSE.txt file
• distributed with this file, or by consulting
• or https://oss.oracle.com/licenses/CDDL

It should be:

• You can obtain a copy of the License by consulting the LICENSE.txt file
• distributed with this file, or by consulting https://oss.oracle.com/licenses/CDDL

We need to fix this in:

1. JIndent Templates.
2. Every File.
3. The template documents.

Currently we rely on JMX being enabled to provide cluster/member information that for the most part is used for integration testing (with assertThat).

Now that we have the ability to submit arbitrary remote executables, there's little reason to use/require JMX. Changing to use remote executables will reduce/eliminate the requirement for JMX setup / configuration / installation / virtualization, especially in a Container-based environment.

We have a setLogLevel method but should also have a setLog method.

Document Distributed Testing Best Practises

Currently when we use a virtualized system (of System resources), the StdOut and StdErr streams are prefixed with [(Default Virtual System) XX:].

This is not really required and actually makes the output look weird. We can easily remove this (and line numbering).

To indicate that virtualization is occurring we should simply log a message when virtualization starts and stops (via logging).

When running a Java application internally when it is closed its JMX server is left running. This leaves the port in use but can also on rare occasions stop the JVM from shutting down. The port in use is not a big issue if using an AvailablePortIterator to assign port numbers but could be for repeated use of a schema in a test suite with a fixed JMX port. The intermittent problem with the JVM came to light occasionally when running tests for the .Net wrapper of Oracle Tools. Occasionally the test runner process in Visual Studio would not stop as the internal JVM still had live non-daemon threads associated to JMX running. Closing the JMX servers seemed to make this issue go away - although it was very intermittent to start with.

Closing the JMX server also allows test code to wait for an internal process to be closed. It is possible to register a listener on a JMX connection to a virtual process and be notified when that connection closes.
E.G you can do something like this...

JMXConnector jmxConnector = member.getDeferredJMXConnector().get();
final CountDownLatch memberDestroyedLatch = new CountDownLatch(1);
jmxConnector.addConnectionNotificationListener(new NotificationListener()
{
@Override
public void handleNotification(Notification notification, Object handback)
{
memberDestroyedLatch.countDown();
}
},

new NotificationFilter()
{
@Override
public boolean isNotificationEnabled(Notification notification)
{
return false == JMXConnectionNotification.OPENED.equals(notification.getType());
}
},
null);

member.destroy();

memberDestroyedLatch.await();

This is useful where your test needs to know the process has really stopped before it does something

Paul Makin found that when running Native / External Application-based Tests on Windows NT / XP, destroying the application may lead to tests hanging.

This appears to be caused by the order in which we close the i/o streams. To resolve the problem we must close the streams before interrupting the i/o threads and before destroying the underlying process.

We've seen numerous times that failing to set a schema to inherit environment properties can cause application start and shutdown issues. In every case it seems that failing to inherit environment properties is the underlying cause (as without it new processes have no environment variables - which can make locating the Java executable/binary problematic).

Consequently we should make it inheriting the default, not clearing (as it is now).

NOTE: all existing functional tests are set to inherit, so it's a good reason to make it the default!

The current approach to "ensuring" a Deferred is to simply poll at regular intervals until a maximum timeout is reached.

While useful, this could be improved as continuous polling is often a waste of time. Instead we should allow different strategies to be specified, including:

1. Polling (as we do now)
2. Exponential Back-Off (both random and uniform)
3. Fibonacci Back-Off (both random and uniform)

By default we should use a Random Fibonacci Back-Off approach as this has proven to be more effective than polling and exponential back-off algorithms.

To implement this approach we should modify the Ensured class to resolve and use an Iterator that provides suitable times to wait for each attempt at resolving the underlying resource. Once this refactoring has occurred, we can define several Iterator implementations that provides Ensured's with the appropriate values. Those would include:

1. ConstantIterator that returns a constant value - forever.
2. FibonacciIterator that returns a number from the Fibonacci sequence.
3. ExponentialIterator that returns a number from an Exponential sequence.
4. RandomIterator that returns a random number between 0 and some other number, that of which comes from an Iterator.

We should introduce a Coherence-based ProxyMemberSchema to capture the standard settings for a Proxy Node.

In the latest release of Java (on OSX), applications appear to run with awt enabled. This causes background testing applications to gain UI focus and thus stop developers working in the foreground on other applications. To resolve this we need to introduce the headless mode switch and set it "on" by default.

eg: -Djava.awt.headless=true