See NETTY-447

At the moment the user must take care of removing Channel instances from ChannelLocal when the Channel is closed. If he forget this he may see some kind of leak. I think we should register a ChannelListener to an added Channel which takes care of removing the Channel from the ChannelLocal once the Channel was closed. Doing this in an automatic way would also be more in line with what we do in other "util" classes like DefaultChannelGroup.

The idea is to make it configurabel via a constructor. Using false (not automatic remove) in 3.2 branch and true (automatic remove) in master.

We currently simply throw the exception raised by codec implementation as-is. However, it makes distinguishing other business logic exceptions from the codec exceptions. For example, HttpMessageDecoder raises an IllegalArgumentException when decoding fails. We should:

• define CodecException, EncoderException, DecoderException
• make the exception types under the codec packages extend them
• modify all current codec impls to wrap non-CodecExceptions properly

Many times, I have to deal with binary packets which start with a length byte(s) and always end with a delimiter byte(s) to signify end of frame. I have found this pattern useful to sync with the client again, when partial frames or corrupted frames are transmitted.

If this is a normal usecase, can we have an implementation which will

1. Do normal length based decoding.
2. If last byte is the expected delimiter, provide the frame to the next decoder/handler in pipeline.
3. If not, then search for delimiter byte(s) within the incoming bytes and discard the portion till the delimiter byte(s).
4. Redo step 1 onwards.

ChannelFutureAggregator.addFuture(..) is not thread-safe and can so have some side-effects. We need to make sure its usable in multi-threaded systems

In SslHandler we use an regex to see if an Exception should be surpressed or not. This may break for various reasons in the future. Like OS Locale, JDK etc.

So we should review this:

See for example:
https://github.com/netty/netty/blob/master/src/main/java/org/jboss/netty/handler/ssl/SslHandler.java#L494

ChannelPipeline is basically a list of ChannelHandlers. An event is executed sequentially. However, this is not enough for the applications that require more omplex event flow such as Web Server.

For example, a user might want different pipelines handle an HttpRequest for different URI patterns.

Common Pipeline ---- if URL matches /websocket ---- WebSocket pipeline
                ---- If URL matches /pages ---- Static page pipeline
                ---- Otherwise --- Redirect-to-home pipeline

There are a couple changes required to make this work:

• ChannelPipeline implements ChannelSink so that a downstream event is propagated to the previous pipeline
• Move Channel.write() and other method that triggers a downstream event to `ChannelHandlerContext' because Netty cannot determine from which pipeline it is firing a downstream event.
  • If we remove such methods from Channel, a user must have a reference to ChannelHandlerContext to perform I/O, and it's not going to be very user-friendly.
  • Alternatively, we can retain such methods in Channel but document them enough so that a user is well-aware that the events triggered by them are evaulated only by the common pipeline.

Feed back appreciated.

QueryStringDecoder currently does not parse
field1=value1;field2=value2
correctly.

http://en.wikipedia.org/wiki/Query_string says
This convention is a W3C recommendation.[3] W3C recommends that all web servers support semicolon separators in the place of ampersand separators.

When a new channel goes into service or out of service, we see the following events:

• channelOpen
• channelBound
• channelConnected
• channelDisconnected
• channelUnbound
• channelClosed

Do we really need all these 3 events per (dis)connection?

Why don't we just simplify like this:

• Make Channel implementations public so that people can extend it and perform additional action in the constructor - this replaces channelOpen
• Replace channelBound and channelConnected with channelAttached.
• Replace channelDisconnected, channelUnbound, and channelClosed with channelDetached.

However, channelAttached and channelDetached is not exactly same with the replaced events because a user can attach an unconnected channel to an event loop. I think a user can simply add a listener to the returned ChannelFuture.

Related issues: #66, #67

Hi,
I just downloaded netty , 3.2.7-Final , recently and started slamming a home made server based on it. I began getting clients hanging
which suggested that accept was throwing without closing the serversocket. Indeed, that is what I found,

java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:249)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

I also was getting the logging call in the exception handler to throw an Error due to fact that I don't have logging setup.
This leaves the server socket bound but with no one to accept connections. I added some code to run() method in

org/jboss/netty/channel/socket/nio/NioServerSocketPipelineSink.java

to catch Throwable around the logging request and close the server socket in finally block but I'm sure there
is a better way to approach this for those familiar with the netty code. In any case, this would seem to create a
bad failure mode if left as it currently is and it was not obvious to me how to deal with this in higher level code without
changing the catch and finally blocks in this code. When I saw this in my own code, I closed the serversocket and bound a new
one but again there is probably a better way to handle this.

Thanks.

  } finally {
            channel.shutdownLock.unlock();

// mjm
try { channel.socket.close(); } catch (Exception eee) {}
closeSelector();
}

We need to split the current ChannelBuffer interface into multiple smaller ones:

• SequentialReadBuffer
• SequentialWriteBuffer
• SequentialIoBuffer extends the two above
• RandomReadBuffer extends SequentialReadBuffer
• RandomWriteBuffer extends SequentialWriteBuffer
• RandomIoBuffer extrends the two above

Users will usually use RandomIoBuffer in most cases (i.e. ChannelBuffer is renamed to RandomIoBuffer)

Do you have an idea for better names? Let me know!

Blah!

We don't need to support JBoss Logging anymore in Netty 4. It means we can simply using widely adopted logging framework and get rid of our thin logging layer. There are two candidates:

• java.util.logging
• SLF4J

SLF4J has much nicer API, but it's somewhat difficult to configure for a novice. java.util.logging's API is pretty ugly, but it will let us continue being a zero-dependency library. What do you prefer and why? I'm leaning toward java.util.logging because of zero dependency is so attractive and we do not log very much.

Let's make all event (I/O) loop implement ScheduledExecutorService so that people can run their task in the I/O worker thread. Because it's going to be provided by all transports, we should introduce a transport-agnostic abstract event loop implementation.

Here's the current idea:

SocketChannel ch = ...;
NioEventLoop el = ...;
el.attach(ch);
...
el.schedule*(...); // from any thread, maybe from your handler
...
ch.detach(); // when you want to suspend I/O completely.

This also opens the possibility of letting user use a java.nio.channels.SocketChannel instance not created by Netty.

I believe there is an issue with CookieEncoder. It can only set one cookie at a time. See http://stackoverflow.com/questions/8312229/netty-cannot-set-multiple-cookies-with-one-cookieencoder for more detailed information.

Let me know if you require further analysis of me.

See @2a9c5efe8fc7eb3a18a2891dc4505a8cdf6f54c5

In WebSocketServerHandshaker10:

ChannelPipeline p = channel.getPipeline();
p.remove("aggregator");
p.replace("decoder", "wsdecoder", new WebSocket08FrameDecoder(true, this.allowExtensions));
p.replace("encoder", "wsencoder", new WebSocket08FrameEncoder(false));

I think the above code doesn't work if the handlers are not named "aggregator", "decoder", and "encoder".

I think these methods should be used instead:

remove(Class<T> handlerType)
replace(Class<T> oldHandlerType, String newName, ChannelHandler newHandler)

Am I correct?

We need a high performance direct / heap buffer pool that can be used by users. So far, we were reluctant about exposing direct buffers to users, but with proper pooling with good documentation, it should be OK in many cases. Heap buffer pool might also be useful when a user creates heap buffers too often because creating a byte array consumes memory bandwidth because JVM always memset the array with NUL.

However, we still need to allow people to use non-pooled heap buffers because it's often very convenient. We can ditch non-pooled direct buffers which make all kind of troubles.

We also should give people the option to shrink or limit the pool size. We will also have to provide some basic stats for diagnosis so that a user can determine the optimal pool size.

Since JVM does not deallocate a direct buffer timely, I'm going to allocate and deallocate it directly using Unsafe. If Unsafe is unavailable (probably non-Sun JVM), we can always fall back to NIO's direct ByteBuffer and call System.gc() explicitly on OutOfMemoryError.

Most users are looking for a way to assign some attachment to a Channel. At the moment they have two choices (out of the box).

1. Use a shared ChannelLocal instance for this
2. Use ChannelHandlerContext for this (the attachment is only visible in the ChannelHandler which belongs the the context here).

ChannelLocal is often not easy to spot for new-comers and also add some extra overhead because of the needed CAS semantic.

We should add methods to the Channel interface to set and get an Channel related attachment + @deprecate ChannelLocal. This will make it easier for users and also eliminate the overhead which ChannelLocal introduce.

It's not compatible with Java 5.

The code below is in Scala.
Please see the last 2 lines.

Although the doc says that "array" returns the backing byte array of this buffer,
the returned array is useless if it can be anything.
The code below shows that its size is not the same as buffer.readableBytes.

Is this a bug?

import java.nio.charset.Charset
import org.jboss.netty.buffer.ChannelBuffers
import org.jboss.netty.handler.codec.base64.{Base64, Base64Dialect}

val buffer = Base64.decode(
  ChannelBuffers.copiedBuffer("aGk=", Charset.forName("UTF-8")),
  Base64Dialect.URL_SAFE)

buffer.readableBytes  // => 2: correct because "aGk=" is "hi" encoded in base 64
buffer.buffer.hasArray  // => true
buffer.array  // => Array(104, 105, 0): is this correct???

See stackoverflow:

http://stackoverflow.com/questions/8317766/about-handlers-in-netty-and-concurrency

When using SslHandler in "client mode" I often find myself to write code to issue the handshake once the connection event is fired. I think its common enough to make it part of SslHandler and so make it easier to use for newbies.

Just installed Eclipse Indigo Service Release 1

It comes with M2E - Maven Integration for Eclipse 1.1

When I load up the Netty project, I get the following error:

Plugin execution not covered by lifecycle configuration: org.apache.maven.plugins:maven-antrun-plugin:1.7:run (execution: write-version, phase: validate)   pom.xml /netty  line 320    Maven Project Build Lifecycle Mapping Problem

      <plugin>
        <artifactId>maven-antrun-plugin</artifactId>
        <version>1.7</version>
        <executions>
          <execution>
            <id>write-version</id>
            <phase>validate</phase>
            <goals>
              <goal>run</goal>
            </goals>
         ...
        </execution>
      </plugin>

Seems like maven-antrun-plugin does not like the the validate phase.

The error goes away if I set the phase to package - but I am not sure if this is what is required.

At the moment all ChannelHandlers are execute in a recursive way. This can cause all kind of problems and is also sometimes not what would give you the best "expierence". It would make sense to make this kind of pluggable, or at least support other execution logics.

For example we could also provide some execution logic which is more similar on what node.js does etc

ChannelFuture holds a list of ChannelFutureListeners. A user can add or remove one or more ChannelFutureListeners from any thread until the future is complete. Once complete, only a single thread will retrieve the list of the listeners exclusively (i.e. no other threads will ever read or write it.)

Currently, we just use a synchronized block, but it's not optimal and we need to experiment with various data structures.

To make this happen:

1. Write a performance test application dedicated to simulate the real world ChannelFuture usage.
2. Design and implement the optimal data structure.

If the user writes a FileRegion via Channel.write(..) it will be directly passed to the Channel. This is even true if a SslHandler or a ZlibEncoder is in place. So the encoding / encrypting will be skipped silently. This makes sense if you know how the FileRegion is used but is not so simple to guess for most users.

We should provide a ChannelDownStreamHandler which will encoder the FileRegion to ChannelBuffers if a SslHandler or a ZlibEncoder is in the ChannelPipeline to make it more easy to use

If one tries to replace a handler in the middle of a pipeline preserving handler's name, the current implementation does not work completely correctly. The handler object is replaced in the pipeline, but calls to get() return the old handler object.

The following test cases demonstrates the behavior:

package org.jboss.netty.channel;

import static org.junit.Assert.*;

import org.junit.Test;

public class TestDefaultChannelPipeline {
    @Test
    public void testReplaceChannelHandler() {
        DefaultChannelPipeline pipeline = new DefaultChannelPipeline();

        SimpleChannelHandler handler1 = new SimpleChannelHandler();
        pipeline.addLast("handler1", handler1);
        pipeline.addLast("handler2", handler1);
        pipeline.addLast("handler3", handler1);
        assertTrue(pipeline.get("handler1") == handler1);
        assertTrue(pipeline.get("handler2") == handler1);
        assertTrue(pipeline.get("handler3") == handler1);

        SimpleChannelHandler newHandler1 = new SimpleChannelHandler();
        pipeline.replace("handler1", "handler1", newHandler1);
        assertTrue(pipeline.get("handler1") == newHandler1);

        SimpleChannelHandler newHandler3 = new SimpleChannelHandler();
        pipeline.replace("handler3", "handler3", newHandler3);
        assertTrue(pipeline.get("handler3") == newHandler3);

        SimpleChannelHandler newHandler2 = new SimpleChannelHandler();
        pipeline.replace("handler2", "handler2", newHandler2);
        assertTrue(pipeline.get("handler2") == newHandler2);
    }
}

The last assert fails. The above behavior has been verified against the latest code in the 3.2 branch.

I know this feature is not strictly required for most of use cases. But I have encountered a situation where I need to shutdown only output portion of a socket. A server implementation that I can't make any change on relies on end-of-stream on its reading side. Though I'll find out some hacks for the time being, it would be good to have such a feature exposed as a public API.

The new code located in io.netty.channel.socket.sctp package does only compile with on linux (maybe also solaris).

At least on windows and osx it fails the build because "com.sun.nio.scpt" package is not present.

Check this out: https://github.com/ning/jvm-compressor-benchmark/wiki

See NETTY-443

Netty project is growing. We need to split into multiple submodules.

A service may need to add service-specific headers in addition to the WebSocket handshake headers (such as additional application-specific authentication tokens). It'd be nice to be able to add those custom headers before the connect method is called on the WebSocketClient.

I think it would be useful to provide an ChannelHandler which adds SEDA support. So upstream and downstream events are passed to a different ThreadPool

There is an issue raised against webbit, but I think potentially this functionality belongs within netty rather than webbit?

I wanted to confirm where the code should be written, and then hopefully contribute a patch with tests and implementation.

Both 3.2 and master should change their groupId to io.netty. From the next release, we are going to deploy to the Sonatype OSS repository.

https://groups.google.com/group/netty/browse_thread/thread/671561e7615a1e50?hl=de

The new license header should look like:

/*
 * Copyright 2011 The Netty Project
 *
 * The Netty Project licenses this file to you under the Apache License,
 * version 2.0 (the "License"); you may not use this file except in compliance
 * with the License. You may obtain a copy of the License at:
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 * License for the specific language governing permissions and limitations
 * under the License.
 */

Something similar applies to all outputs such as Javadoc, Xref, User guide, etc.

A user schedules a lot of timeouts with HashedWheelTimer, there is a chance some slot has way too many items comparing to other slots. It would be very useful if HashedWheelTimer exposes the detailed stats about the distribution of the timeouts in the wheel, a user will be able to increase the size of the wheel or choose to use different timer implementation.

We should remove @version Javadoc tags because it's not relevant anymore since we moved to Git.

Initial discussion for this request is here:

http://groups.google.com/group/netty/browse_thread/thread/55e267a48aedb2ab#

Basically, it would be nice if ChannelBuffers could (somehow) validate that received messages are compliant with a specified Charset (at least UTF-8).

This would be useful for text based protocols or protocols which have text based frames (such as web sockets :)).
Currently the only way to do the validation is to actually do the String/CharBuffer conversion (which requires a copy). It seems to be a fit for the ChannelBuffers as I assume a goal of ChannelBuffers is to minimize memory copies.

It would be nice if this could work with a dynamic buffer, verifying that the data is compliant with a specified charset as new data is added...and allowing for partial characters at the end of the buffer (assuming there would be additional data entered to complete any partial character).

In AbstractChannel we call:

    protected AbstractChannel(
            Channel parent, ChannelFactory factory,
            ChannelPipeline pipeline, ChannelSink sink) {

        this.parent = parent;
        this.factory = factory;
        this.pipeline = pipeline;

        id = allocateId(this);

        pipeline.attach(this, sink);
    }

Thats not safe as allocateId and pipeline.attach(..) could see a not fully initialized channel instance

It would be nice and useful to make ChannelFuture implement java.util.concurrent.Future. We don't need to define an additional type that represents the result of the future but we can just use Void. On failure, ExecutionException should be raised. On success, null is returned.

DynamicChannelBuffer currently simply doubles its capacity when it needs more room. This is fine when the size of the buffer is not large, but doubling the capacity of a 128MiB buffer doesn't sound right. DynamicChannelBuffer should be rewritten as a composite buffer of smaller chunks (e.g. 2MiB) by taking advantage of the buffer pool proposed in #62.

In FrameDecoder, the call to ChannelBuffer cumulation = cumulation(ctx); is always executed and a buffer is created, even though we might be able to read all from the input buffer. It should be lazily created.

org.jboss.netty.example.sctp.SctpClient fails to make an sctp connection in Open Solaris 10 (guest VM in Virtual Box 4.1.6)

java.net.SocketException: Option not supported by protocol
at sun.nio.ch.SctpChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SctpChannelImpl.finishConnect(SctpChannelImpl.java:471)
...

OS Version: SunOS 5.11 snv_111b i86pc i386 i86pc Solaris

Java Version: "1.7.0"
Java(TM) SE Runtime Environment (build 1.7.0-b147)
Java HotSpot(TM) Client VM (build 21.0-b17, mixed mode)

SctpServer and SctpClient are in same guest VM and using loop-back interface.

ReplayingDecoder use Atomic* fields to represent some states. This is not needed, we should remove them