sgg / brokaw Goto Github PK

• implement HEAD, BODY and STAT
• implement overviews (bonus points for capabilities based fallback)
• implement LIST
• warn about potential foot-guns w/r/t to messing w/ underlying connections
• expose interface for sending raw commands

All (?) of Brokaw's iterators are over fixed collections so they could ExactSizeIterator.

• Basic article retrieval using NntpClient
• Basic article retrieval using NntpConnection
• Demonstrate usage of NntpConnection w/ typed Responses
• Bonus: REPL-like Newsreader CLI ----> #18

Implement the compression as described in RFC 8054. Humorously none of the servers I use even seem to use this.

Building, testing, rustfmt, etc via GH Actions.

Create a containerized news server that we can use for integration testing. This is useful for #21 and #28

The client half of the types provided by Brokaw more or less boils down to two things:

1. Command encoding (implemented via ToString Encode)
2. Response decoding (implemented via TryFrom<RawResponse>)

One could extend Brokaw to support server-side connections by implementing (1) command decoding and (2) response encoding.

It might be worthwhile to reconsider the use of ToString as a trait for command encoding as it prevents Brokaw from supporting non UTF-8 messages (tracked in #26)

Command serialization is currently built on Display. This implicitly requires that commands be entirely UTF-8. To support i18n and binary payloads we should decouple serialization from Display.

Capabilities are currently string based. The common ones could be typed allowing for feature fallback/selection in NntpClient without a bunch of String look ups in the hashmap.

For better isolation we could spin up innd in a container as reaching out to public servers is brittle (and potentially annoying for the owners)

Giganews (and decedents) use compression via an XFEATURES extension. This extension does not seem to be documented publically but some old newsposts indicate some folks Giganews might have a spec.

I have done some reverse engineering and will eventually put the notes here.

Much of the commands module is boilerplate code and couple be replaced w/ a couple proc-macros

Add the ability to post articles. Now would probably be a good time to introduce some integration tests!

POST is particularly tricky because (like IHAVE) it is a two stage command. If we model this incorrectly we could end up with an inconsistent stream state. The workflow is as follows:

1. Client sends POST
2. Server replies with status
3. If status is 340, send post and check response. Otherwise stop.

• Implement POST command(s)
• Provide builder(s)
• Provide first class support in the NntpClient so that users do not have to deal with the stateful nature themselves
• Provide doc examples and worked example

Brokaw supports NNTP over TLS (also known as "NNTPS") but it does not support opportunistic TLS (STARTTLS) as described in RFC 4642.

1. Find a server that actually supports this.
2. Implement STARTTLS

We allocate a buffer JIT if NntpConnection encounters a multiline response. We should preallocate this and store it with the connection.

Implement Astraweb-style compression. This could be provided as part of overviews in the NntpClient and as separate deserialization functions.

Astraweb-style header compression leverages two custom commands, XZVER and XZHDR.

The closes thing I've found to documentatoin is an old post from their helpdesk (pasted below).

============================================================

We are proposing two new commands as an extension to the NNTP protocol: XZVER and XZHDR as counterparts to XOVER and XHDR respectively.
These two new optional commands will appear in the list returned from the "HELP" command.


Request: XZVER [range]
Response:

224 compressed data follows
=ybegin line=128 size=-1

=yend crc=FFFFFFFF
.


Request: XZHDR header [range]
Response

221
compressed data follows
=ybegin line=128 size=-1

=yend crc=FFFFFFFF
.


Where [range] is in the standard article number range as per XOVER.
Notes:
- The multi-line responses conform to standard NNTP protocol with a single period "." on a line by itself.
- Line terminals are \r\n
- The actual XOVER / XHDR response is compressed with zlib, then yEnc encoded.
- The compression conforms with zlib's deflate() method and there is NO gzip done on it. ie. Just pure gzip'ed data with no extra padding info to make it gzip-file compliant.
- The yEnc spec requires that the "size" parameter in "=ybegin line=123 size=456" be present, but for efficiency and performance sake, the total size is not known ahead of time so it has been set to some arbitrary value that should be ignored. The yEnc CRC32 value should be sufficient to validate the payload. On top of that, zlib has builtin adler32 which is an alternate form of CRC32.


It is assumed that the implementation for XZVER be duplicated from XOVER, with two additional steps at the end, that being: yEnc decode the data, then zlib inflate the data.
============================================================

A REPL-like Newsreader CLI would make for a cute example