Giter VIP home page Giter VIP logo

postsack's Introduction

license Rust CI

Postsack

A high level visual overview of swaths of email

TLDR! A web demo that shows how Postsack clusters a set of 10.000 fake emails

Do you have many emails? I have a lot of emails. I'm not a inbox zero person. I recently realized that my Gmail account contains roughly 650.000 emails. I looked at that and I began to wonder.. Why?.. Sure, I've been using Gmail since 2004 but still, that's 38.000 Emails per year which strikes me as a bit on the crazy side of things. I wanted to know where these mails came from.

Gmail did not offer an easy way of visualizing all my emails, I also couldn't find a tool for it. Hence I build my own. It parses all your mails and shows configurable clusters of mails in a nice visualization.

It looks like this

Example

Features

  • Written in Rust: Very fast email parsing / import. My 650k mails are imported in ~1 Minute on a Macbook M1 Pro Max and ~ 2 Minutes on a Intel Core i7-8700B 3.2 Ghz.
  • Import all your local mails (currently, only Maildir, MBox, Apple Mail and Gmail Backups are supported)
  • Build up clustered visualizations of your mails to see and understand what kind of emails you have
  • Cluster the emails by sender domain / name, month, day, year, name, and some more
  • Additional filters for seen mails or tags / labels
  • See all the mails for the current set of filters / current cluster
  • Save the generated database as a SQLite file so you can do additional queries yourself (or open it again)
  • Cross platform (macOS (from 10.12 on), Windows, Linux and a Web Demo)
  • The app is 13MB big and consumes ~150MB of memory on macOS

The look is similar on all platforms as it uses the Rust egui GUI library.

Videos

Here's a video showing the UI in action (e.g. me selecting some mail clusters)

postsack_video.mp4

Here's another video where you can see the importer importer 650k mails (it is a bit boring but.)

postsack_importer_video.mp4

Web Demo

In addition to that, you can also play around with some fake data in this Postsack Web Demo

Using It

Currently, Postsack supports three different types of mail storage:

There're open issues for other formats such as maildir, notmuch or Outlook but if you use one of these formats your best bet would be to export your emails as MBox which seems to be something most mail apps support. Alternatively, I'd be more than happy for PR's implementing additional Mail Storage Formats.

If you have mails in any of the archives above, you can start Postsack select the folder with the emails and it will do the rest.

Current State

I've created issues for some of the missing functionality. Most importantly more email formats (as explained above). Beyond that, there're probably bugs, there's a certain lack of tests and documentation, the windows build is on shaky grounds, the light theme is wonky, some parts need a healthy refactoring to be useful beyond Postsack, and it would be great if the binaries could be generated from the Github actions.

Installation

For macOS, Linux or Windows install instructions, please refer to the Readme in the postsack-native folder

Deploying to crates.io for cargo install

One of the issues I ran into was that many of the emails I had received over the years were not properly standards compliant. I forked email-parser, email parser that Postsack is using in order to support all the weird issues I encountered. However, this PR still needs a couple of enhancements before it will be merged into email-parser. Therefore, Postsack is currently dependent on a fork of a crates.io crate. This means that I can't deploy this to crates yet. Once The aforementioned pull request has been improved and merged, I will subsequently draft a crates.io release.

Overview

Here's an overview of the different crates in the Postsack Workspace:

Why Egui?

I had build an app in Druid last year and I liked the experience. This time I wanted to try out a different Rust gui library. Between Iced and Egui I went with the latter because the terse code examples were tempting. Also, I had heard good things about it.

I might want to try to re-implement the postsack ui in another UI library. However something I really dig about egui is how quickly it allows building a simple UI for a specific task (say you want to automate a certain bash script). The main downside was that it is currently very limited in what it can do (e.g. available widgets, configuration, layout options, etc).

What does Postsack mean?

Postsack (or Postbeutel) is German for a bag full of mail

Can I also delete the selected clusters?

Currently you can't. I wanted to add this as a feature but it is quite involved. I might start working on this next.

postsack's People

Contributors

terhechte avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

postsack's Issues

Discontinue use of the dedicated GPU on Intel MacBook Pros

Postsack causes MacBook Pros with two GPUs to switch to the dedicated GPU, causing increased battery drain.

There's no reason I can see why it would need to do that; I presume it's a consequence of the Egui library usage.

Please investigate and fix, so battery life usage can decrease. Thanks!

Pre-compiled binaries for older macOS versions

Very cool project, and I just wanted to try it out (without pulling the repo and compiling myself). Sadly, the pre-compiled macOS binary only seems to work for macOS 12 (Monterey) and later.

It would be great if you can provide pre-compiled binaries that work on older versions too.

I see you already set the osx_minimum_system_version to 10.14, but it seems that didn't have the desired effect or was it maybe added after the initial release was made?

Consider Refactoring link.rs

The link is a bit of a terrible abstraction to handle background threads via egui. The way it works is that when an action should be performed in the background, it will send the action over a crossbeam channel to a separate thread which continuously listens for input from the channel. The thread processes the action (e.g. performs a query) and sends it back.

Egui, in the default event loop, will check for any response in the receiver crossbeam channel (e.g. 30 times per second on 30fps). If a response is available it will be applied.

There're a couple of downsides to this:

  • Request and Response are separated which makes it very difficult to use the Engine via Link in a non-async context.
  • WASM doesn't support threads, so there is a wonky abstraction to allow sync access (with way too many cfg(target_arch = wasm32)
  • This solution requires multiple additional abstractions which make it all a bit confusing to understand.

Egui apparently would work from separate threads but this might not work for other UI libraries (if we ever port it). So maybe a solution would be to have a SyncLink and AsyncLink with a closure that makes sure the request matches the response.

Action when clicking an email

If the users clicks an email in the mail panel, the email is currently only printed out to the console. There're a couple of possible behaviours here:

  • Use the importer email parsers and display the email in a window
  • Open the email in Nautilus / Finder / Explorer
  • Open the email in the installed mail app for reading.

Maybe this can be made configurable.

Better MBOX parser

Postsack currently uses mbox-reader for MBOX parsing, but it doesn't properly implement the standard. It only checks for the FROM string at the beginning of a line which means any email containing a newline with a FROM somewhere in the body is regarded as two different emails. The correct way to detect a new email in MBOX according to the RFC 4155 is:

Each message in the mbox database MUST be immediately preceded
by a single separator line, which MUST conform to the following
syntax:

  • The exact character sequence of "From";

  • a single Space character (0x20);

  • the email address of the message sender (as obtained from the message envelope or other authoritative source), conformant with the "addr-spec" syntax from RFC 2822;

  • a single Space character;

  • a timestamp indicating the UTC date and time when the message was originally received, conformant with the syntax of the traditional UNIX 'ctime' output sans timezone (note that the use of UTC precludes the need for a timezone indicator);

  • an end-of-line marker.

Add abstractions for generic way of displaying *groupable* information

Right now, Postsack only displays emails. However, most of the code is written in a way that it could easily be used on any kind of groupable information.

  • The importer is abstracted away from the dabs
  • The database is abstracted away from the guy
  • The gui only uses a couple of currently hardcoded information from ps_core which can be extracted into a trait
    • The Field::all_cases and the fields that are available for grouping
    • Everything in the FilterPanel and the MailPanel. Those would need to be more generic
    • The importer flow (select mail folder)
    • The copy texts

It would still be some work, but in the end Postsack could be more generic and be applied to most groupable data.

Windows Version: Apple Importer Fails

The Apple Mail importer currently fails on Windows. No, I doubt many people would want to import Apple Mail folders on Windows, but I think the issue is just something with the \n / \r\n line endings.

Partnership with Himalaya

Hi there,

I'm the author of the Pimalaya project, which aims to provide Rust tools to manage your personal information. One of the subproject we have is a library to manage emails, and few clients built at the top of it (a CLI, a Vim plugin and an Emacs plugin). We plan to implement a GUI using egui. Then I found your project (I love the concept!). It could be so great to work together, here some ideas I had in mind:

  • We plan to implement the mbox backend, and I see that you have issues with your actual parser, we could work together on implementing our own.
  • You could use pimalaya-email to fetch emails and do your reports instead of reimplement standards (IMAP, Maildir, mbox, archives etc).
  • If your project can be exported as a component then it could be integrated inside our email GUI.

Let me know!

Light Theme

Currently the basics for supporting a light theme are in place, but it is not working yet. While Egui does support light themes, the platform specific theme customisation is not finished for light themes yet (in ps-gui/src/platform)

Proper Windows Build

Copy-Pasting from postsack-native:

Building for Windows works but doesn't create a proper application (e.g. with an icon) and also doesn't create a proper installer.

Add native Gmail backup functionality?

Currently, Postsack only works if the emails are already downloaded to the users system, which means for Gmail to use Gmvault. However, this function could be added to be available out of the box via gbackup-rs. This would make it easier for a potential user to use the app without having to resort to third party tools before being able to actually use it.

Core dump on wayland (swaywm)

I'm using the Archlinux aur package with wayland (sway wm) but it crashes with the following error:

[wayland-client error] Attempted to dispatch unknown opcode 0 for wl_shm, aborting.
[1]    3407478 IOT instruction (core dumped)  postsack

I can launch it with WINIT_UNIX_BACKEND=x11 postsack but that's not ideal.

Improvements to filters to help finding huge emails

When receiving emails with attachments, the mailbox may get bloated.

Especially in conjunction with eventually deleting emails as in #16 , it would be useful to add filters specific to the size of an email.

Possible filters

Items marked with MVP seem desirable to have in the first incarnation.

  • min size in Kb (MVP)
    • a slider to filter emails smaller than the minimum size.
    • Probably '0' as value would mean the filter is disabled, but there might be better ways to indicate this from a UX perspective.
  • attachment
    • a checkbox to require emails to have an attachment
    • file extension multi-selection
      • Lists all actually used extensions and allows to focus on emails which have at least one of them.

Possible GUI Improvements

Extended statistics about displayed emails

Along with knowing how many emails are currently displayed, it would certainly be useful how much storage space they require. Maybe this information could be added to the top of the GUI.

Screen Shot 2022-01-04 at 10 02 22 AM

Real-time updates when changing the filter

This would allow to remove the 'apply' button and would probably make it fun to play with sliders. It might also be an interesting exercise in async programming to make it deal with possibly longer filtering times gracefully. If filtering through 650k emails gets sluggish, one might consider throwing multiple threads into the mix.

Play around with other UI Toolkis

As Postsack has a good separation between the individual parts, it would be possible to re-implement the ui nee ps-gui in another Rust UI toolkit such as Druid or Iced. This could be interesting in order to have a comparison how these different toolkits fare for a slightly more complex desktop app (also, to have a nice overview).

Generate Mail Deletion Rules

One of the initial features I wanted to implement was being able to drill down to a specific set of emails (say everything from @groupon.com that was send in December 2016) and generate a deletion rule for my mail app (or gmail) so that I could safely remove those emails.

However, there is no standard around this, different apps have different rules, and most of them are not open / documented:

  • The gmail rule system is very simple and the only way to generate rules from the queries in Postsack would be to generate a rule for each email with the email identifier which would be a huge rule list
  • Apple Mail has a ruling system, but there is no way to import or export them, they're saved in a Plist in the V9 Mail folder and not documented.
  • I didn't look into other apps

The better solution would be to write a Pop3 / IMAP client which understands these rules and uses them to delete emails. I started on something like that but gave up quickly because the focus was to finish Postsack first.

Improve the deployment GitHub Action

The yaml currently contains the hard coded version for the generated deb package. This can probably be read from the environment. Apart from that, it might also be useful to generate Fedora packages and maybe even ARM64 builds?

Killed

Hi there! I tried to run this on my single maildir. After about 30 minutes the process dies:

Basic Structure
swappy-20220127_183440

The setup:
Screenshot from 2022-01-27 18-36-02

When it's processing:

Screenshot from 2022-01-27 18-36-32

When it hangs, it only adds one additional line that says killed

P.S. Thanks for the Nix shell file. That rocks!!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.