I guess that I would be good to review all those binaries and see if they should remain as part of the libappimage project or moved elsewere. Opening an issue for it.

Originally posted by @azubieta in #77 (comment)

When I built an large (5GB) x86 AppImage on my x86_64 openSUSE linux I get the following error when running it:

Cannot open /proc/self/exe: Value too large for defined data type
Failed to get fs offset for /proc/self/exe

It seems my image is too big. A friend of mine is using Ubuntu and for him the generated AppImage works fine.
Since it's a video game with packed size of 5GB I can't get it smaller.
I googled a lot and found out the following. Maybe I need to rebuild some of my software stack (maybe coreutils) with the following gcc option

 -D_FILE_OFFSET_BITS=64

But I think this is already activated on my whole software stack, because this is stuff which changed in like 2006. But of course I'm not sure.

I successfully built x86 appimages (also games) which are smaller than 2GB which work flawlessly.

Strace prints the following line when executing the binary

[...]
open("/proc/self/exe", O_RDONLY)        = -1 EOVERFLOW (Value too large for defined data type)
write(2, "Cannot open /proc/self/exe: Valu"..., 66Cannot open /proc/self/exe: Value too large for defined data type
[...]

I'm using openSUSE 42.3 x86_64 with the default kernel.

If you need any more info, just ask.
Thank you in advance

https://www.freebsd.org/cgi/man.cgi?crypt(3)

Currently, libappimage logging is controlled by the "verbose" boolean parameter on some of the public interface functions. This method doesn't allow specifying verbosity levels or pretty printing errors.

We should try to avoid any form of (slow) virtual binding for such low-level stuff by using templating and type traits instead of abstract base classes and implementations thereof.

When trying to compile libappimage, it dies during building of squashfuse, unless libfuse development files are installed, but a cmake check for that is missing

The resources extractor currently trust on the read method to resolve links while reading the files. As both, libarchive and squashfuse, don't support to move backward or directly accessing an entry while iterating resolving a link implies to re-iterate the payload file system until a regular file is found. This could be improved by building a cache of the existent links in a first iteration so the regular files can be properly extracted in the second.

This may be also a good opportunity to improve the constructor using an enum to describe the different resources to be extracted.

Detecting loops while extracting/reading files from the AppImage Payload is a common procedure in several parts. It would be nice to group all the implementations into a single class.

When I compile libappimage and try to use it in KDE's KFileMetaData (https://phabricator.kde.org/D18450), cmake configuration fails with:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
LIBAPPIMAGE_LIBRARIES
linked by target "kfilemetadata_appimageextractor" in directory /path_to_folder/kfilemetadata/src/extractors
linked by target "appimageextractortest" in directory /path_to_folder/kfilemetadata/autotests

Configuring incomplete, errors occurred!
See also "/path_to_folder/build-kfilemetadata-Desktop-Default/CMakeFiles/CMakeOutput.log".
CMake Project parsing failed.

This is on Kubuntu 18.10.

I'm not super experienced with this, so this may also be a problem on my side.

Desktop Entries parsing and edition is a required functionality in libappimage. The old implementation uses glib, new implementation uses xdg-utils. But there are other alternatives which should be reviewed.
Our candidates so far are:

• glib-2.0
• linuxdeploy-desktopfile
• xdg-utils

How do deal with distributions?

By the time libappimage lands in debian stable or the oldest still-supported Ubuntu LTS, we will long have moved on.

Can we somehow enforce that every user of this library bundles their own private copy, ensuring that once this is in debian stable or the oldest still-supported Ubuntu LTS, people don't use that outdated version but the most recent one?

While looking into TheAssassin/AppImageLauncher#97, I found that if an AppImage uses a symlink for the root desktop file (IIRC linuxdeploy uses that method, too), appimage_type2_get_desktop_filename_and_key_file will report success and returns the correct path, however g_key_file_load_from_squash doesn't support reading from symlinks, and fails to create the GKeyFile instance. For some reason, however, that function doesn't report an error. It makes the assumption that if a file is found in the root directory, it's a regular file. That is not acceptable.

We must:

• change the function to return an error if loading of the key file fails
• make that function aware of symlinks

CC @azubieta would appreciate if you could look into that ASAP.

There are cases in which the desktop integration process can fail. The current implementation may leave trash files in the user home in such cases. We should clean up them.

Should we be using the C or the C++ version now? The reason is that I don't want to have to make changes in 2 places.

LibAppImage is the foundation of all AppImage tools, therefore, it should provide a consistent and functional API for all of its clients. Currently, its state is not the best among the most prominent issues we found:

• duplicated code
• unnecessary (utilities) entries at the public interface
• mixed concerns: file examination and desktop integration
• missing file examination functionalities (list files, extract files, appimage files traversal)
• dependencies included as submodules. Not all libs are made to be used as submodules but every lib is made to be used in the traditional way (installed in the system). Using submodules also temp to the developer to use the inner sections of the lib which are not expected to be consistent between version, which will produce build fails and increase the maintenance work.

As a major modification of the public interface and the inner implementation is required it would be a good idea to use this opportunity to improve the code-base and ease maintenance by rewriting it in modern C++ .

It's purposed to split the current libappimage.so in two different dynamic shared objects (DSOs): libappimage and libappimage-desktop-integration. Both will be produced from the same code base (the libappimage project). Besides the DSO version, there will be a static version with will have zero external dependencies in order to allow its usage by third-party projects that don't want to depend on the whole libstd++. The public headers files will be kept in bare C initially also to increase compatibility, an additional C++ interface could be published later.

The libappimage module will be responsible for: reading AppImage files information and contents extraction.

The libappimage-desktop-integration module will be responsible for: performing the desktop integration and disintegration of applications packed into AppImages.

The current implementation of the traversal over type 1 AppImages ignores all kind of files but regular and symlinks.

Replace sqfs traversal by efficient search for finding the files that need to be integrated.

Ideally do this upstream in https://github.com/vasi/squashfuse.

Currently libappimage directly extracts e.g., desktop files directly to $HOME/.local/share (or deletes them from) on each invocation of

• appimage_type1_register_in_system
• appimage_type2_register_in_system
• appimage_unregister_in_system

Each change in /.local/share can trigger local caches such as Sycoca to be re-generated, which results in a lot of CPU activity.

So we should also apply the same logic that @rszibele used in AppImageCommunity/appimaged#50 to all files (desktop files, icon files, mime type files and such) that get copied to HOME/.local/share for system integration.

We should only atomically (=everything in one go) move files to (from a temporary location to which they get extracted and modified in) or delete files from $HOME/.local/share after a certain timer of inactivity has passed.

We should do all editing/rewriting of desktop files outside of HOME/.local/share in a separate directory.

This will solve:

• AppImageCommunity/appimaged#12
• AppImageCommunity/appimaged#19

Please make sure that when the desktop integration functions are called, then e.g., if the desktop file in the top-level directory is a symlink, it is handled correctly.

Reference:
AppImageCommunity/appimaged#54

This code should not have to be in here, your library shall specification details like "interpretation of boolean values". ld-desktopfile can do it (it's conformance tested even).

Originally posted by @TheAssassin in https://github.com/AppImage/libappimage/timeline

Shotcut-181118.glibc2.14-x86_64.AppImage from https://github.com/mltframework/shotcut/releases

While working on #9, I added a new function, which lacked symlink handling in the type 2 code at first. Therefore, I refactored the code to be more useful, fixed a few annoyances, and implemented some actual error handling (which is missing in the majority of the legacy code of libappimage right now).

Since I didn't encounter issues for type 1 yet, I didn't work on that. However, also that code base should be reviewed, and perhaps refactored soon.

Currently, only the main icon is being deployed we should also look for and deploy the "Desktop Actions" icons and the mime-type packages icons.

AppImageCommunity/appimaged#50 from @rszibele needs to be ported over here.

It checks if the following programs are available in the users $PATH:

• update-desktop-database
• update-mime-database
• gtk-update-icon-cache
• kbuildsycoca5

Based on the availability of each program, it executes each of them within a thread if more than 3 seconds have passed since the last AppImage was added. If another AppImage is dropped into one of the watched directories while still within the 3 seconds, then the timer is reset. So if suddenly a user drops 20 AppImages into the directory, the update will only be executed once.

The icons get extracted (in a pretty naive but apparently working way, to .local/share/icons/hicolor/32x32/apps, ignoring the format and the resolution, which works fine). Pretty sure it's been done like that forever (I doubt @azubieta changed the basic algorithm), however, since the rewriting in #14, this extraction doesn't add the file extension to the icon any more, producing bugs like TheAssassin/AppImageLauncher#105. appimaged is only broken because we didn't update libappimage in there for a while.

The breaking commit is IMO 7f390f6.

TODO:

• add unit test that breaks because of this bug
• fix bug in an elegant way

When diagnosing why my main system started to take a long time to boot, I was shocked to discover that there were approximately more than 30K directories all starting with libappimage- This is totaling to about 130+ GB of data. They all seem to have sizes of 4.0 KB each. Unfortunately, I can't give more exact details about this since I have deleted all of them already. Is this normal behavior or not?

Based on the results of AppImageCommunity/appimaged#53, regular expressions seem to be really CPU intensive.

Hence, we should replace them with something more efficient. Possibly fnmatch(3) would be more efficient (to be verified),

After having replaced regular expressions with something else, please test that

• --appimage-extract still works with * wildcards
• appimaged consumes significantly less CPU

The current CMake files are made from snippets from different parts and they don't follow a given style. In order to improve the overall project readability, we must define and apply a code style to all cmake files.

One of the functionalities that are still missing at libappimage_0.2 is the thumbnails generation. A correct implementation of the standard will require using some image manipulation library (which are known to be huge). The current implementation is really naive as it ignores the Images size and format.

I consider that a correct implementation can easily be achieved by a third party tool without size or linking dependencies restrictions like "appimaged" or "AppImageLaunher". So I would like to leave it out of our public interface.

@probonopd @TheAssassin comments are welcome

This issue is related to #33

Until the end of March, I'd like to release libappimage 1.0.0. Being the first stable release we ever provide, this marks a milestone in our development history. This library has been largely rewritten, and now the codebase is ready for more public usage.

When integrating, we should rewrite desktop files to use an absolute Icon= path. This will

• Prevent from rendering issues when the icon size does not match the size of the icon theme directory (at least on some desktops, see below)
• Remove the need to log out and log in again after having installed appimaged/AppImageLauncher and similar tools

In order to have a reliable code base and to not jump blindly into a release, libappimage 0.2 must have an overall code coverage indicator of 90%. This issue is to keep @probonopd and @TheAssassin aware of the progress.

To reproduce the results run:

cmake -DBUILD_COVERAGE_TESTS=On .
make -j`nproc`
make tests_code_coverage

Continuation of #33 (comment).

There should be an equivalent of appimage_extract_file_following_symlinks that returns the file's contents (as uchar* or something) so it can be fed directly into e.g. QImage for thumbnail generation without having to save it to a temporary location first.

At the moment the functions exposed by appimage.h only allow to do very high level things with the appimage, such as directly installing it, directly creating a thumbnail.

However, being it a generic library to handle appimages I think it would be more useful if it exposed in addition to the functions already present, which are very useful already, a few lower level functions and the main handler struct. Which would allow a developer using the library to have more control especially on the registration of the image in the system, to customise the process.

For example, i am trying to develop a thumbnailer for any appimage, that could be registered along with the other system thumbnailers in /usr/share/thumbnailers (in Arch Linux at least), however the appimage_create_thumbnail function does not allow me to choose where to create it.

Another example is theming, as I do believe, although I haven't verified it yet, that placing an icon in the .local/share/icons folder overrides automatically any icon theme, and furthermore no theme provides icons named app_6a8124552b098217d5f7c428fe2a1ab8, so may like to be able to place them somewhere else...

It seems to me that as it is at the moment, this library is more of a libappimagetools than a libappimage, if this makes it clearer :)

Edit: /usr/share/thumbnailers only applies to Nautilus and Nemo

As part of the libappimage rewrite on C++, it's proposed to replace our own ELF files reading functions by a more widely used and tested ones. Those will be libelf and LIEF. The positive aspects of this change are that we will reduce our codebase and the need for maintaining that part. The negative: we will have another dependency.

1. libelf

• size: ~200 kb
• license: LGPL

2. LIEF

• size: ~ 2Mb
• license: Apache

Is it viable to have it? Does it imply any licensing issues?

Probably the value after Exec= in the generated .desktop file needs to be quoted to prevent from this.

The dynamic library handling, specifically the handler code for image processing libraries, should be moved into a proper separate module and receive proper unit tests.

Continuation of #33 (comment).

At the moment, we have type-specific functions called appimage_handler::traverse (if it were C++), which call a callback on every path that matches a given pattern.

The callbacks usually perform name comparisons, for instance:

if (strcmp(params->file_path, filename) == 0)
    // do something meaningful

As you can see, in this common case, the callback is only supposed to be run once when the right file has been found, as the comparison checks for equal strings. However, even if the file is found, the loop will continue to call the callback on all the files left in the AppImage.

This is highly inefficient, but luckily, we're not the first ones to solve this problem. There is an elegant way to allow for some "feedback" from the callback to the caller: we simply change the callbacks' return values from void to bool. Whenever the callback returns true, the loop in traverse can be aborted.

Depending on the location of the given path, this will greatly speed up use cases such as appimage_extract_file_following_symlinks. Also, our dummy functions which we use for testing can become no-ops (O(1) instead of O(n)).

CC @azubieta @probonopd if you have any points why this is a bad idea, please let me know.

A new and good logging system should be created after #51 Therefore the "verbose" parameter present in many of the C public interface will no longer be functional. In order to ease the transition to the newly written code, we should create aliases for such functions without the mentioned parameter.

The static linked binaries of libappimage are not being installed and its cmake target is not properly configured which makes it unusable by third parties.

`Index: libappimage/src/libappimage_shared/elf.c

--- libappimage.orig/src/libappimage_shared/elf.c 2019-01-09 05:42:49.721466452 -0700
+++ libappimage/src/libappimage_shared/elf.c 2019-01-11 09:07:48.259236674 -0700
@@ -136,7 +136,7 @@
}
if ((ehdr.e_ident[EI_DATA] != ELFDATA2LSB) &&
(ehdr.e_ident[EI_DATA] != ELFDATA2MSB)) {

•   fprintf(stderr, "Unkown ELF data order %u\n",
  

•   fprintf(stderr, "Unknown ELF data order %u\n",
    	ehdr.e_ident[EI_DATA]);
    return -1;
  

  }`

As @kossebau show at #17 the CMake config file is incomplete. The libappimage has issues with it's dependencies. Which are mostly public and client projects fail at linking against them.

This linking error is produced because of the name of the targets which are not recreated in the config file.

Running the latest appimaged from downloads, I'm observing the following behavior on Ubuntu 14.04 LTS:

1. Placing <project>.AppImage in ~/Downloads successfully triggers the appimage_register_in_system.
2. During registration, appimaged copies <project>.xml to $HOME/.local/share/mime/packages and names it appimage_<hash>_<project>.xml.
3. During registration, appimaged copies <mime-icon>.svg to $HOME/.local/icons/hicolor/scalable/mimetypes and names it appimage_<hash>_<mime-icon>.svg.

What seems to be missing is, the xml still references the original .svg by it's original name, so the icon never shows up in Nautilus, regardless of how many times you restart nautilus or logout of the Desktop.

Here's an example xml. For demonstration purposes, I'll call this project "starbright".

Mime Package

• mime/packages/starbright.xml:

  <?xml version="1.0" encoding="UTF-8"?>
  <mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
    <mime-type type="application/x-starbright-file">
      <!---          ^-- EXPECTS icon to be named e.g. application-x-starbright-file.svg  -->
      <sub-class-of type="application/xml"/>
      <comment>Starbright File</comment>
      <glob pattern="*.starb"/>
    </mime-type>
  </mime-info>

Mime Icon

• icons/hicolor/scalable/mimetypes/application-x-starbright-file.svg

What happens is appimaged does the following:

1. Copies
   <AppImage>/share/mime/packages/starbright.xml -->
   
   $HOME/.local/share/mime/packages/appimaged_d41d8cd98f00b204e9800998ecf8427e_starbright.xml
   ✅ Good!
2. Copies
   <AppImage>/share/icons/hicolor/scalable/mimetypes/application-x-starbright-file.svg -->
   
   $HOME/.local/share/icons/hicolor/scalable/mimetypes/appimaged_d41d8cd98f00b204e9800998ecf8427e_application-x-starbright-file.svg
   ⛔️ This causes problem

Now the mime type value inside the xml file is mismatched from its respective icon. Is this a limitation of the mimetype support for this OS, or is there a gap in functionality with the way the icon is currently being installed?

<?xml version="1.0" encoding="UTF-8"?>
   <mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
-     <mime-type type="application/x-starbright-file">
+     <!-- Help! How to reference after rename? -->
+     <mime-type type="appimaged_d41d8cd98f00b204e9800998ecf8427e_application/x-starbright-file">
         <sub-class-of type="application/xml"/>
         <comment>Starbright File</comment>
         <glob pattern="*.starb"/>
      </mime-type>
   </mime-info>
</xml>

If the svg name is copied to hicolor/scalable/mimetypes/application-x-starbright-file.svg (without the appimaged_d41d8cd98f00b204e9800998ecf8427e_ prefix), the icon will show up.

Take a deep look at the warnings in the code base.

#25 (comment)

https://github.com/AppImage/libappimage/blob/8b3969667004659f0033c71556a161bca3d60690/src/libappimage/libappimage.c#L396 and https://github.com/AppImage/libappimage/blob/8b3969667004659f0033c71556a161bca3d60690/src/libappimage/type2.c#L68

Ultimaker/cura-build#136 (comment)

have a version number in my launcher

We should rewrite the version appending (and any other such logic) independently from the underlying image format, so that it will work with all current (and future) image formats.

Different image types should not behave differently.

The PayloadIterator provides an SLT friendly interface to traverse the files contained in the AppImage payload. It directly provides access to the entry path by means of operator*() and to the rest of entry properties by means of other function in the same class. This design mostly influenced by the limitations of libarchive and squashfuse.

Creating PayloadEntry objects seems to be a wrong approach as our back-end doesn't allow to keep references to a single entry while traversing.

This issue was opened to give another review to the design.

I need a function that returns the highest GLIBC_ version needed of a file or directory.

The equivalent of

https://github.com/AppImage/pkg2appimage/blob/bb64b4389123a1e0c0bb10b030140c27b59010fb/functions.sh#L134-L139