crac / openjdk-builds Goto Github PK

Executive summary: I've read the manual, but it's not clear to me why checkpointing a Java app running inside a docker container, should I checkpoint the docker container itself too; I just wanted to "snapshot" my running Java app? Could you kindly clarify why docker checkpoint is needed when performing a CRaC/criu for a java app running inside a docker container, even if I just collect the files in a persistent way, please?

Details

Hi,
I've been experimenting with this project following this video from Devoxx and this great tutorial.

Since I'm on Mac OSX (and not linux) I operate inside docker container.

My goal is to "snapshot" a running Java app, using CRaC/criu at a point in time and restore it, following the tutorials mentioned and the documentation I could find here on github.

Since I operate the CRaC inside a container because I'm on Mac OSX, I make sure the files are collected on a mounted volume, so I can mount them across container restarts.

I have created a banal Java app to test this, here: https://github.com/tarilabs/demo20230223-counting-on-crac

# create directory to host crac files dump
mkdir crac-files
# prepare a docker container as the lab environment to operate within
docker build -f src/main/docker/Dockerfile.jvm -t demo20230223-counting-on-crac .
docker run -it --privileged -v $(pwd)/crac-files:/opt/crac-files --rm --name demo20230223-counting-on-crac demo20230223-counting-on-crac
java -XX:CRaCCheckpointTo=/opt/crac-files $JAVA_OPTS -jar $JAVA_APP_JAR

In another shell I perfom:

docker exec -it -u root demo20230223-counting-on-crac /bin/bash
ps -u root
# typically java is PID 9, used below
jcmd 9 JDK.checkpoint

Up to here, everything works as expected, the app is checkpointed and dump files are created.

Now I want to restore, using the command:

java -XX:CRaCRestoreFrom=/opt/crac-files

I have tried 3 use-cases

Case A

If in the first shell, as the docker container is still running, I execute the restorefrom, it works.

Case B

If in the second shell, I capture a docker checkpoint with something ~like:

exit
docker ps -a
docker commit CONTAINER_ID demo20230223-counting-on-crac:checkpoint

then in the first shell, I restart from the checkpoint with something ~like:

exit
docker run -it --privileged -v $(pwd)/crac-files:/opt/crac-files -p 8080:8080 --rm --name demo20230223-counting-on-crac demo20230223-counting-on-crac:checkpoint
java -XX:CRaCRestoreFrom=/opt/crac-files

it works.

Case C

In the second shell, I just exit.
In the first shell, I just exit.
No container is running and no docker-checkpoint was taken.
In the first shell I go with:

docker run -it --privileged -v $(pwd)/crac-files:/opt/crac-files --rm --name demo20230223-counting-on-crac demo20230223-counting-on-crac
java -XX:CRaCRestoreFrom=/opt/crac-files

I get:

Error (criu/cr-restore.c:1506): Can't fork for 9: File exists
Error (criu/cr-restore.c:2593): Restoring FAILED.

I don't get why I cannot just restart the Java app from the dumped files (which are available across container restart as they are on the host disk), somehow additional status of the docker container must also be captured (with the docker checkpoint) ?

Is this a limitation of the system I'm using Mac OSX, and if I was Linux I could have turned off and turned on the linux computer across Java app checkpoint and restore?

Thanks!

Hi again!

I'm trying build project with maven in Dockerfile and when I try create a checkpoint, this exception comes:

An exception during a checkpoint operation:
jdk.crac.CheckpointException
        at java.base/jdk.crac.Core.checkpointRestore1(Core.java:141)
        at java.base/jdk.crac.Core.checkpointRestore(Core.java:246)
        at java.base/jdk.crac.Core.checkpointRestoreInternal(Core.java:262)
        Suppressed: jdk.crac.impl.CheckpointOpenSocketException: tcp localAddr 0.0.0.0 localPort 5000 remoteAddr 0.0.0.0 remotePort 0
                at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:91)
                at java.base/jdk.crac.Core.checkpointRestore1(Core.java:145)
                ... 2 more
        Suppressed: jdk.crac.impl.CheckpointOpenSocketException: tcp localAddr 0.0.0.0 localPort 9090 remoteAddr 0.0.0.0 remotePort 0
                at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:91)
                at java.base/jdk.crac.Core.checkpointRestore1(Core.java:145)

My Dockerfile

FROM ubuntu:20.04
RUN mkdir /src
COPY . /src
WORKDIR /src
RUN apt-get update -y
RUN apt-get install maven -y
RUN mvn --settings /src/settings.xml clean install

RUN mkdir /app
COPY /src/target/*.jar /app/app.jar

RUN mkdir /opt/crac-files

ENV JAVA_HOME /opt/jdk
ENV PATH $JAVA_HOME/bin:$PATH
ADD "https://github.com/CRaC/openjdk-builds/releases/download/17-crac%2B5/openjdk-17-crac+5_linux-x64.tar.gz" $JAVA_HOME/openjdk.tar.gz
RUN tar --extract --file $JAVA_HOME/openjdk.tar.gz --directory "$JAVA_HOME" --strip-components 1; rm $JAVA_HOME/openjdk.tar.gz;

EXPOSE 5000 9090

When I built the project on host and copy jar to container in Dockerfile, the checkpoint works.
Thanks!

Hi, I'm trying to create a checkpoint and got no error at command line but on application's log and dump4.log show errors:

Command line message:
root@fcb2c0b1a9bc:/app# jcmd app.jar JDK.checkpoint 7: Command executed successfully

Application's log:
CR: Checkpoint ... JVM: invalid info for restore provided (may be failed checkpoint)

And the dump4.log

(00.001102) Version: 3.17.1-crac (gitid 8926431+1)
(00.001156) Running on fcb2c0b1a9bc Linux 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64
(00.001168) File /run/criu.kdat does not exist
(00.001182) sockets: Probing sock diag modules
(00.002733) sockets: Done probing
(00.003475) Pagemap is fully functional
(00.003618) Found anon-shmem device at 1
(00.004979) Hugetlb size 2 Mb is supported but cannot get dev's number
(00.005042) Hugetlb size 1024 Mb is supported but cannot get dev's number
(00.005050) Reset 99's dirty tracking
(00.005067) ... done
(00.005125) Dirty tracking support is OFF
(00.005605) Found task size of 7ffffffff000
(00.032784) Restoring netdev veth idx 10
(00.033637) Dumping netns links
(00.033728) LD: Got link 1, type 772
(00.033733) LD: Got link 2, type 768
(00.033735) LD: Got link 3, type 776
(00.033737) LD: Got link 10, type 1
(00.034686) vdso: Parsing at 7ffca1fab000 7ffca1fac000
(00.034699) vdso: PT_LOAD p_vaddr: 0
(00.034702) vdso: DT_HASH: 120
(00.034703) vdso: DT_STRTAB: 2d8
(00.034705) vdso: DT_SYMTAB: 1b8
(00.034706) vdso: DT_STRSZ: 72
(00.034707) vdso: DT_SYMENT: 18
(00.034709) vdso: nbucket 3 nchain c bucket 7ffca1fab128 chain 7ffca1fab134
(00.034714) vdso: rt [vdso] 7ffca1fab000-7ffca1fac000 [vvar] 7ffca1fa7000-7ffca1fab000
(00.035244) vdso: Parsing at 7f68df8c9000 7f68df8cb000
(00.035252) vdso: PT_LOAD p_vaddr: 0
(00.035254) vdso: DT_HASH: b4
(00.035256) vdso: DT_STRTAB: 1f0
(00.035257) vdso: DT_SYMTAB: 140
(00.035258) vdso: DT_STRSZ: c0
(00.035259) vdso: DT_SYMENT: 10
(00.035261) vdso: nbucket 3 nchain b bucket 7f68df8c90bc chain 7f68df8c90c8
(00.035263) vdso: compat [vdso] 7ffca1faf000-7ffca1fb1000 [vvar] 7ffca1fab000-7ffca1faf000
(00.035523) cpu: x86_family 6 x86_vendor_id GenuineIntel x86_model_id Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
(00.035531) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 832
(00.035540) cpu: fpu: x87 floating point registers xstate_offsets 0 / 0 xstate_sizes 160 / 160
(00.035543) cpu: fpu: AVX registers xstate_offsets 576 / 576 xstate_sizes 256 / 256
(00.036437) Warn (criu/kerndat.c:1470): CRIU was built without libnftables support
(00.037314) No MOVE_MOUNT_SET_GROUP kernel feature
(00.037508) ptrace(PTRACE_GET_RSEQ_CONFIGURATION) is not supported
(00.037608) Found mmap_min_addr 0x10000
(00.037626) files stat: fs/nr_open 1048576
(00.037628) skip kerndat_save_cache
(00.037641) ========================================
(00.037642) Dumping processes (pid: 7)
(00.037644) ========================================
(00.037650) rlimit: RLIMIT_NOFILE unlimited for self
(00.037655) Running pre-dump scripts
(00.037714) irmap: Searching irmap cache in work dir
(00.037730) No irmap-cache image
(00.037732) irmap: Searching irmap cache in parent
(00.037997) No parent images directory provided
(00.038002) irmap: No irmap cache
(00.038005) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:1 xgetbv1:1 xsaves:1
(00.038044) cg-prop: Parsing controller "cpu"
(00.038048) cg-prop: Strategy "replace"
(00.038052) cg-prop: Property "cpu.shares"
(00.038054) cg-prop: Property "cpu.cfs_period_us"
(00.038056) cg-prop: Property "cpu.cfs_quota_us"
(00.038057) cg-prop: Property "cpu.rt_period_us"
(00.038058) cg-prop: Property "cpu.rt_runtime_us"
(00.038059) cg-prop: Parsing controller "memory"
(00.038061) cg-prop: Strategy "replace"
(00.038062) cg-prop: Property "memory.limit_in_bytes"
(00.038063) cg-prop: Property "memory.memsw.limit_in_bytes"
(00.038065) cg-prop: Property "memory.swappiness"
(00.038066) cg-prop: Property "memory.soft_limit_in_bytes"
(00.038067) cg-prop: Property "memory.move_charge_at_immigrate"
(00.038068) cg-prop: Property "memory.oom_control"
(00.038069) cg-prop: Property "memory.use_hierarchy"
(00.038071) cg-prop: Property "memory.kmem.limit_in_bytes"
(00.038072) cg-prop: Property "memory.kmem.tcp.limit_in_bytes"
(00.038073) cg-prop: Parsing controller "cpuset"
(00.038074) cg-prop: Strategy "replace"
(00.038076) cg-prop: Property "cpuset.cpus"
(00.038077) cg-prop: Property "cpuset.mems"
(00.038083) cg-prop: Property "cpuset.memory_migrate"
(00.038086) cg-prop: Property "cpuset.cpu_exclusive"
(00.038087) cg-prop: Property "cpuset.mem_exclusive"
(00.038089) cg-prop: Property "cpuset.mem_hardwall"
(00.038090) cg-prop: Property "cpuset.memory_spread_page"
(00.038091) cg-prop: Property "cpuset.memory_spread_slab"
(00.038092) cg-prop: Property "cpuset.sched_load_balance"
(00.038094) cg-prop: Property "cpuset.sched_relax_domain_level"
(00.038095) cg-prop: Parsing controller "blkio"
(00.038096) cg-prop: Strategy "replace"
(00.038098) cg-prop: Property "blkio.weight"
(00.038099) cg-prop: Parsing controller "freezer"
(00.038100) cg-prop: Strategy "replace"
(00.038102) cg-prop: Parsing controller "perf_event"
(00.038103) cg-prop: Strategy "replace"
(00.038104) cg-prop: Parsing controller "net_cls"
(00.038106) cg-prop: Strategy "replace"
(00.038107) cg-prop: Property "net_cls.classid"
(00.038108) cg-prop: Parsing controller "net_prio"
(00.038110) cg-prop: Strategy "replace"
(00.038111) cg-prop: Property "net_prio.ifpriomap"
(00.038112) cg-prop: Parsing controller "pids"
(00.038113) cg-prop: Strategy "replace"
(00.038114) cg-prop: Property "pids.max"
(00.038116) cg-prop: Parsing controller "devices"
(00.038117) cg-prop: Strategy "replace"
(00.038118) cg-prop: Property "devices.list"
(00.038151) Preparing image inventory (version 1)
(00.038174) Add pid ns 1 pid 99
(00.038284) Add net ns 2 pid 99
(00.038309) Add ipc ns 3 pid 99
(00.038316) Add uts ns 4 pid 99
(00.038320) Add time ns 5 pid 99
(00.038326) Add mnt ns 6 pid 99
(00.038330) Add user ns 7 pid 99
(00.038350) Add cgroup ns 8 pid 99
(00.038355) cg: Dumping cgroups for 99
(00.038373) cg: - New css ID 1 (00.038375) cg: - [] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0]
(00.038377) cg: - [blkio] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0] (00.038378) cg: - [cpu] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0]
(00.038379) cg: - [cpuacct] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0] (00.038381) cg: - [cpuset] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0]
(00.038382) cg: - [devices] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0] (00.038383) cg: - [freezer] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0]
(00.038384) cg: - [hugetlb] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0] (00.038386) cg: - [memory] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0]
(00.038387) cg: - [name=systemd] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0] (00.038388) cg: - [net_cls] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0]
(00.038389) cg: - [net_prio] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0] (00.038390) cg: - [perf_event] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0]
(00.038392) cg: - [pids] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0] (00.038393) cg: - [rdma] -> [/docker/fcb2c0b1a9bc4ee2e6c8c7e61ccd1eecbd60dd58a368463bdd72069f527c4203] [0]
(00.038394) cg: Set 1 is criu one
(00.038404) Detected cgroup V1 freezer
(00.038493) Error (compel/src/lib/ptrace.c:27): suspending seccomp failed: Operation not permitted
(00.038516) Unlock network
(00.038525) Unfreezing tasks into 1
(00.038527) Unseizing 7 into 1
(00.038529) Error (compel/src/lib/infect.c:356): Unable to detach from 7: No such process
(00.038538) Error (criu/cr-dump.c:2053): Dumping FAILED.

I tried with all releases version and only openjdk-17-crac+3 got no exception on command line and I'm running docker on Win 11 with image docker from maven:3.8.6-eclipse-temurin-11 and install the openJDK manually like tutorial:

Java version:
root@fcb2c0b1a9bc:/opt/crac-files# java -version openjdk version "17-crac" 2021-09-14 OpenJDK Runtime Environment (build 17-crac+3-15) OpenJDK 64-Bit Server VM (build 17-crac+3-15, mixed mode, sharing)

Tutorial: https://foojay.io/today/how-to-run-a-java-application-with-crac-in-a-docker-container/

Thanks!

Hi, I trying to run spring boot from your examples with forked tomcat inside docker with latest build 17-crac+5, getting error on the attempt to checkout app, can you give any advice?

root@bacc339c8eaf:/# jcmd 9 JDK.checkpoint
9:
CR: Checkpoint ...
--------------------
023-05-03T07:55:34.079Z  INFO 9 --- [Attach Listener] jdk.crac                                 : /opt/app/spring-boot-0.0.1-SNAPSHOT.jar is recorded as always available on restore
JVM: invalid info for restore provided: queued code -1
An exception during a checkpoint operation:
jdk.crac.CheckpointException
        at java.base/jdk.crac.Core.checkpointRestore1(Core.java:141)
        at java.base/jdk.crac.Core.checkpointRestore(Core.java:246)
        at java.base/jdk.crac.Core.checkpointRestoreInternal(Core.java:262)

Also output of criu check --all contains a lot of not supported and denied lines maybe it is the reason?

 ./criu check --all
Warn  (criu/kerndat.c:1470): CRIU was built without libnftables support
Error (criu/cr-check.c:750): couldn't suspend seccomp: Operation not permitted
Error (criu/cr-check.c:793): Dumping seccomp filters not supported: Permission denied
Warn  (criu/cr-check.c:813): Dirty tracking is OFF. Memory snapshot will not work.
Error (criu/cr-check.c:1208): UFFD is not supported
Error (criu/cr-check.c:1208): UFFD is not supported
Error (criu/cr-check.c:1273): Time namespaces are not supported
Warn  (criu/cr-check.c:1334): Nftables based locking requires libnftables and set concatenations support
Error (criu/cr-check.c:996): failed to mount autofs: No such device
Looks good but some kernel features are missing
which, depending on your process tree, may cause
dump or restore failure.

Hi,

I'm getting jdk.crac.CheckpointException exception when trying create a checkpoint on my simple project.

Since I'm on Mac OS Ventura (M1 architecture) and I cannot run CraC stable in the container I created an Ubuntu instance(Ubuntu Server 22.04 LTS) on Amazon.

Then I took the following steps.

wget https://github.com/CRaC/openjdk-builds/releases/download/17-crac%2B5/openjdk-17-crac+5_linux-x64.tar.gz
tar xfv openjdk-17-crac+5_linux-x64.tar.gz
sudo vi ~/.bash_profile

Then typed

export JAVA_HOME="$HOME/openjdk-17-crac+5_linux-x64"
export PATH=$JAVA_HOME/bin:$PATH

source ~/.bash_profile
java -version
openjdk version "17-crac" 2021-09-14
OpenJDK Runtime Environment (build 17-crac+5-19)
OpenJDK 64-Bit Server VM (build 17-crac+5-19, mixed mode, sharing)

Then I created manifest.txt file into greetings folder and Greetings.java file into greetings/org/jugistanbul/crac folder.

greetings
|_manifest.txt
|_org
|____jugistanbul
|______________crac
|__________________Greetings.java

Greetings.java

package org.jugistanbul.crac;

public class Greetings 
{
    private static int counter;
    
    public static void main(String[] args) throws InterruptedException {

        while (true){
            System.out.println(String.format("%sth greetings from İstanbul", ++counter));
            Thread.sleep(1000);
        }
    }
}

manifest.txt

Manifest-Version: 1.0
Main-Class: org.jugistanbul.crac.Greetings

Then I execute the following commands separately in greetings folder.

javac org/jugistanbul/crac/Greetings.java
jar -cfm greetings.jar manifest.txt org/jugistanbul/crac/Greetings.class
java -jar greetings.jar

The output I get is as I expected

1th greetings from İstanbul
2th greetings from İstanbul
3th greetings from İstanbul

Then I run the jar again with -XX:CRaCCheckpointTo parameter as follows

java -XX:CRaCCheckpointTo=image -jar greetings.jar

And run the following command on another shell window

jcmd greetings.jar JDK.checkpoint

Result is:

May 24, 2023 3:06:42 PM jdk.internal.util.jar.PersistentJarFile beforeCheckpoint
INFO: /home/ubuntu/greetings/greetings.jar is recorded as always available on restore
JVM: invalid info for restore provided: queued code -1
An exception during a checkpoint operation:
5th greetings from İstanbul
jdk.crac.CheckpointException
at java.base/jdk.crac.Core.checkpointRestore1(Core.java:141)
at java.base/jdk.crac.Core.checkpointRestore(Core.java:246)
at java.base/jdk.crac.Core.checkpointRestoreInternal(Core.java:262)
6th greetings from İstanbul
7th greetings from İstanbul

The image directory contains

ls -ls
total 44
 4 -rw-r--r-- 1 ubuntu ubuntu    28 May 24 15:06 cppath
 8 -rw------- 1 ubuntu ubuntu  6653 May 24 15:06 dump4.log
32 -rw------- 1 ubuntu ubuntu 32768 May 24 15:06 perfdata