dynup / kpatch Goto Github PK

View Code? Open in Web Editor NEW

1.4K 1.4K 297.0 2.43 MB

kpatch - live kernel patching

License: GNU General Public License v2.0

Makefile 1.90% Shell 21.72% C 70.49% C++ 5.58% Assembly 0.31%

kpatch's People

Contributors

Stargazers

Watchers

Forkers

jpoimboe turbosree puiterwijk pavlinux gordongong unixbhaskar sparderwee iomato bowei rousya mhiramat again4you ngeorgieff ryanmiao useidel mhiramathitachi beauhoyt madper lulinqing glongwave raceli linuxqiao terrywang jstancek flaming-toast lovelylich goldyfruit gnehzuil olshanov colinianking gukai toutiao husttsq 0xf1sh safe3 gooops llsdimple devonbessemer cyclefusion nkwilson kragniz rafaelicobr theojulienne vvoody grail cyber43 lzufalcon hustcat archer239915 cloudxtreme euspectre diggold libin2015 arges londer15 bhassani dwdm shekkbuilder panmin2014 daveti ben-romer nonerkao rosslagerwall fehraraujo yinghaoxie finallyjustice lihas lamproae nan619 tomzhic han-ian bearieq kamalesh-babulal slavaforks zhouchengming1 joe-lawrence martinbj2008 ieswxia hitstar chenlong12580 jiaoyk claud-ling guihecheng iredko lingfennan im-bill royaflash playsand knightku yankunsam xuyue00249845 lsai bryant1410 lavalake macrosheep vincentbernat taeung huawei-nos asicer arusv81

kpatch's Issues

kpatch-build option --vmlinux not recognized

Using --vmlinux option currently fails with:

getopt: unrecognized option '--vmlinux'
ERROR: getopt failed

This option is not included in getopt arguments:

diff --git a/kpatch-build/kpatch-build b/kpatch-build/kpatch-build
index 3cb05ca..6d1f5bb 100755
--- a/kpatch-build/kpatch-build
+++ b/kpatch-build/kpatch-build
@@ -92,7 +92,7 @@ usage() {
        echo "          -d, --debug     Keep scratch files in /tmp" >&2
 }

-options=$(getopt -o hr:s:c:d -l "help,sourcerpm:,sourcedir:,config:,debug" -- "$@") || die "getopt failed"
+options=$(getopt -o hr:s:c:v:d -l "help,sourcerpm:,sourcedir:,config:,vmlinux:,debug" -- "$@") || die "getopt failed"

 eval set -- "$options"

Use of WARN family of macros resulting in changed section false positives

In order create a trivial testcase for a modified __bug_table section, I created the following patch:

diff --git a/mm/ksm.c b/mm/ksm.c
index 175fff7..35da89f 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -14,6 +14,8 @@
  * This work is licensed under the terms of the GNU GPL, version 2.
  */

+/* New comment */
+
 #include <linux/errno.h>
 #include <linux/mm.h>
 #include <linux/fs.h>

However, I discovered another gotcha, as create-diff-object improperly detects a change to remove_stable_node(). The WARN family of macros also apparently does weird things:

$ objdump -d ksm.o | grep -A 45 .text.remove_stable_node > before
$ objdump -d ksm.o.patched | grep -A 45 .text.remove_stable_node > after
$ diff before after
42c42
<   70: be cb 02 00 00          mov    $0x2cb,%esi

---
>   70: be cd 02 00 00          mov    $0x2cd,%esi

The operand address changes by 2, the same line offset introduced by the patch. Further study yields that 0x2cb and 0x2cd are 715 and 717 in decimal, the respective line numbers of the only WARN_ON_ONCE() in ksm.o.

Looking at the macro, WARN_ON_ONCE -> WARN_ON -> __WARN. __WARN calls warn_slowpath_null() with __FILE__ and __LINE__ as arguments. __FILE__ is relocated and points to the filename in .rodata.str1.1 but __LINE__ is hardcoded into the instructions:

  0x0000000000000078  X86_64_32S      000000000000000000    +110 .rodata.str1.1
  0x000000000000007d  X86_64_PC32     000000000000000000      -4 warn_slowpath_null

kpatch_create_dynamic_rela_sections: 1583: expected bundled symbol for section __jump_table for dynrela src __tracepoint_netif_receive_skb

With the patch from #219 and the fix from #220, I get the following error:

dev.o: changed function: rollback_registered_many
dev.o: changed function: __netdev_upper_dev_link
dev.o: changed function: net_rx_action
dev.o: changed function: __netif_receive_skb_core
dev.o: changed function: napi_gro_complete
dev.o: changed function: __dev_set_promiscuity
dev.o: changed function: __dev_set_allmulti
dev.o: changed function: netdev_has_upper_dev
dev.o: changed function: netdev_master_upper_dev_get
dev.o: changed function: netdev_upper_dev_unlink
dev.o: changed function: unregister_netdevice_queue
dev.o: changed function: dev_change_net_namespace
dev.o: changed function: __dev_change_flags
dev.o: changed function: __netdev_update_features
dev.o: changed function: register_netdevice
dev.o: changed function: netdev_run_todo
/home/jpoimboe/git/kpatch/kpatch-build/create-diff-object: kpatch_create_dynamic_rela_sections: 1583: expected bundled symbol for section __jump_table for dynrela src __tracepoint_netif_receive_skb

Unpatching must check new functions in the backtrace

Currently kpatch_backtrace_address_verify() is reused in unpatching path but it just checks old functions in the backtrace. However, unpatching the old functions no more exists on the stack because it already replaced by the new one. Thus we need to check the new function body on it.
This is not only for kmod/core issue, but also kpatch-build(add-patches-section.c?), because this requires recording new_size in kpatch_patch and kpatch_func.

Make kpatch-build works on Debian/Ubuntu

Debian/Ubuntu is another important Linux distro. Their package management is based on .deb, not .rpm.

If we support those distros, we could get more people involved.

insmod: ERROR ... Unknown symbol in module

Tested and hit the same issue on two fc20 machines:

$ uname -r
3.13.6-200.fc20.x86_64

$ sudo insmod kpatch-foo.ko 
insmod: ERROR: could not insert module kpatch-foo.ko: Unknown symbol in module
$ dmesg | tail
...
[59874.745996] kpatch_foo: module verification failed: signature and/or  required key missing - tainting kernel
[59874.746086] kpatch_foo: Unknown symbol kpatch_register (err 0)
[59874.746119] kpatch_foo: Unknown symbol kpatch_unregister (err 0)

$ sudo insmod kpatch-foo.ko 
insmod: ERROR: could not insert module kpatch-foo.ko: Unknown symbol in module
$ dmesg | tail
...
[59874.745996] kpatch_foo: module verification failed: signature and/or  required key missing - tainting kernel
[59874.746086] kpatch_foo: Unknown symbol kpatch_register (err 0)
[59874.746119] kpatch_foo: Unknown symbol kpatch_unregister (err 0)
[59924.752963] kpatch_foo: Unknown symbol kpatch_register (err 0)
[59924.752982] kpatch_foo: Unknown symbol kpatch_unregister (err 0)

kmod/core: Introduce Mutex lock for protecting kpatch

Current kpatch_register and kpatch_unregister seems not to be protected from each other. Since those APIs are open for other modules, it can be called concurrently. We should have a mutex to protect the patching/unpatching process.

changes to .kprobes.text not detected

With the following patch, kpatch-build doesn't detect any changes:

~ $ cat ~/nmi.patch 
Index: src/arch/x86/kernel/nmi.c
===================================================================
--- src.orig/arch/x86/kernel/nmi.c
+++ src/arch/x86/kernel/nmi.c
@@ -518,6 +518,8 @@ do_nmi(struct pt_regs *regs, long error_

    inc_irq_stat(__nmi_count);

+   printk("nmi\n");
+
    if (!ignore_nmis)
        default_do_nmi(regs);

~ $ kpatch-build -d nmi.patch 
DEBUG mode enabled
Using cache at /home/jpoimboe/.kpatch/3.13.10-200.fc20.x86_64/src
Testing patch file
patching file arch/x86/kernel/nmi.c
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
nmi.o: no changed functions were found
ERROR: kpatch build failed. Check /tmp/kpatch-build-1398776924.log for more details.

I think it's because do_nmi() is annotated with __kprobes which places it in the .kprobes.text section. But I'd expect the inclusion logic to catch the fact that .kprobes.text changed and report an error.

kpatch-build: lookup_local_symbol sysctl_print_dir.isra.2 (proc_sysctl.c)

$ cat smp-2.patch 
Index: src/fs/proc/proc_sysctl.c
===================================================================
--- src.orig/fs/proc/proc_sysctl.c
+++ src/fs/proc/proc_sysctl.c
@@ -24,6 +24,7 @@ void proc_sys_poll_notify(struct ctl_tab
    if (!poll)
        return;

+   printk("kpatch-test: testing .smp_locks section changes\n");
    atomic_inc(&poll->event);
    wake_up_interruptible(&poll->wait);
 }
$ kpatch-build -d smp-2.patch 
DEBUG mode enabled
Using cache at /home/jpoimboe/.kpatch/3.14.3-200.fc20.x86_64/src
Testing patch file
checking file fs/proc/proc_sysctl.c
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
proc_sysctl.o: changed function: put_links
proc_sysctl.o: changed function: insert_header
proc_sysctl.o: changed function: proc_sys_poll_notify
proc_sysctl.o: changed function: __register_sysctl_table
proc_sysctl.o: changed function: retire_sysctl_set
/usr/local/libexec/kpatch/create-diff-object: kpatch_create_patches_sections: 1421: lookup_local_symbol sysctl_print_dir.isra.2 (proc_sysctl.c)
ERROR: kpatch build failed. Check /tmp/kpatch-build-1400189994.log for more details.

rename replacement function from "foo" to "foo__patched"

This would be useful to e.g. clarify that kpatch is involved in an oops backtrace.

unsupported section change in do_nanosleep()

I'm hitting an error with following patch applied to RHEL7 GA kernel (3.10.0-123.el7):

# cat hrtimer.patch 
diff --git a/kernel/hrtimer.c b/kernel/hrtimer.c
index e268e26..3cb4aef 100644
--- a/kernel/hrtimer.c
+++ b/kernel/hrtimer.c
@@ -1552,6 +1552,8 @@ static int __sched do_nanosleep(struct hrtimer_sleeper *t, enum hrtimer_mode mod
    } while (t->task && !signal_pending(current));

    __set_current_state(TASK_RUNNING);
+   if (jiffies == 0)
+       printk("what?\n");

    return t->task == NULL;
 }

kpatch-build -s rpmbuild/BUILD/kernel-3.10.0-123.el7/linux-3.10.0-123.el7.x86_64 -d -v /usr/lib/debug/lib/modules/3.10.0-123.el7.x86_64/vmlinux -t vmlinux hrtimer.patch
...
hrtimer.o: hrtimer.o: changed section .sched.text not selected for inclusion
hrtimer.o: hrtimer.o: changed section .rela.sched.text not selected for inclusion
hrtimer.o: hrtimer.o: changed section .rodata.str1.1 not selected for inclusion
hrtimer.o: hrtimer.o: changed section .rela__mcount_loc not selected for inclusion
hrtimer.o: 4 unsupported section change(s)
/usr/local/libexec/kpatch/create-diff-object: unreconcilable difference

Forwarding github messages to maillist

It could better to record github issues, and make maillist noisy, :-).

BTW, what's the IRC channel?

apt-get source does not always retrieve the source of the currently running kernel

Hello!
I was skimming over the kpatch-build source when I noticed these few lines:

echo "Downloading and unpacking kernel source for $ARCHVERSION"
apt-get source linux || die "'apt-get source linux' failed. you may need to run 'apt-get install dpkg-dev'"

So this doesn't always grab the source for $ARCHVERSION, it could be a few revisions behind, as apt-get source linux always grabs the most recent revision. For example, on my ubuntu vm running 3.13.0-24 kernel will grab the sources for 3.13.0-29, a few revisions ahead. I thought this might be important since there is the possibility that the fetched source contains patches that the currently running kernel doesn't have.

The Ubuntu wiki suggests using apt-get source linux-image-$(uname -r) to fetch the source of a specific kernel version, but this doesn't work unfortunately (it will ignore the version you give it and default to apt-get source linux, always) This is apparently a problem other people have experienced: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/642749

A solution on the ubuntu forums suggests directly cloning the ubuntu kernel repository (git clone git://kernel.ubuntu.com/ubuntu/ubuntu-$(lsb_release -sc).git) and checking out a particular tag that corresponds to the version you want.

I thought the intention was to grab the source of the currently running kernel, so I thought I'd bring this up just in case.

Failed to generate core.o

In rhel7, 3.10.0-121.el7.x86_64, it can't find the function(or say macro) in_nmi().

make[3]: Entering directory `/usr/src/kernels/3.10.0-121.el7.x86_64'
  CC [M]  /root/kpatch/kpatch/kmod/core/core.o
/root/kpatch/kpatch/kmod/core/core.c: In function ‘kpatch_ftrace_handler’:
/root/kpatch/kpatch/kmod/core/core.c:266:2: error: implicit declaration of function ‘in_nmi’ [-Werror=implicit-function-declaration]
  if (unlikely(in_nmi())) {
  ^
cc1: some warnings being treated as errors
make[4]: *** [/root/kpatch/kpatch/kmod/core/core.o] Error 1

This problem can't be reproduced in F20, lucky.
For safy, we need to add header <linux/preempt_mask.h> to core.c.

changes to __init function not detected

kpatch-build succeeds with the following patch which changes an __init function:

Index: src/fs/proc/meminfo.c
===================================================================
--- src.orig/fs/proc/meminfo.c
+++ src/fs/proc/meminfo.c
@@ -29,6 +29,8 @@ static int meminfo_proc_show(struct seq_
    unsigned long pages[NR_LRU_LISTS];
    int lru;

+   printk("hello\n");
+
 /*
  * display in kilobytes.
  */
@@ -181,6 +183,7 @@ static const struct file_operations memi
 static int __init proc_meminfo_init(void)
 {
    proc_create("meminfo", 0, NULL, &meminfo_proc_fops);
+   printk("hi there\n");
    return 0;
 }
 module_init(proc_meminfo_init);

I would expect it to fail when it detects that the .init.text section changed.

use original System.map + new vmlinux to resolve duplicate symbols

As described in #192.

changed section .rela.parainstructions not selected for inclusion

I meet a new unreconcilable difference:

[root@hp-dl388g8-23 ~]# kpatch-build mlock.patch 
Using cache at /root/.kpatch/3.10.0-121.el7.x86_64/src
Testing patch file
patching file mm/rmap.c
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
rmap.o: changed function: try_to_unmap_file
rmap.o: changed section .rela.parainstructions not selected for inclusion
/usr/local/libexec/kpatch/create-diff-object: unreconcilable difference
ERROR: kpatch build failed. Check /tmp/kpatch-build-1398172719.log for more details.

The patch is:

[root@hp-dl388g8-23 ~]# cat mlock.patch 
diff --git a/mm/rmap.c b/mm/rmap.c
index 009a408..1f95c59 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1384,9 +1384,19 @@ static int try_to_unmap_cluster(unsigned long cursor, unsigned int *mapcount,
                BUG_ON(!page || PageAnon(page));

                if (locked_vma) {
-                       mlock_vma_page(page);   /* no-op if already mlocked */
-                       if (page == check_page)
+                       if (page == check_page) {
+                               /* we know we have check_page locked */
+                               mlock_vma_page(page);
                                ret = SWAP_MLOCK;
+                       } else if (trylock_page(page)) {
+                               /*
+                                * If we can lock the page, perform mlock.
+                                * Otherwise leave the page alone, it will be
+                                * eventually encountered again later.
+                                */
+                               mlock_vma_page(page);
+                               unlock_page(page);
+                       }
                        continue;       /* don't unmap */
                }

-- 
1.7.1

data section .data.unlikely selected for inclusion

I'm seeing this error with patch from #219:

diff --git a/net/core/dev.c b/net/core/dev.c
index d06cd72..b7c1300 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3503,6 +3503,7 @@ ncls:
        case RX_HANDLER_PASS:
            break;
        default:
+           printk("BUG!\n");
            BUG();
        }
    }

# git log -1 --oneline
a705c22 Merge pull request #222 from jpoimboe/special-section-fixes

# uname -r
3.10.0-123.el7.x86_64

...
section .rela.text.net_rx_action is included
 start include_symbol(softnet_data)
 symbol softnet_data is included
 end include_symbol(softnet_data)
 start include_symbol(this_cpu_off)
 symbol this_cpu_off is included
 end include_symbol(this_cpu_off)
 start include_symbol(jiffies)
 symbol jiffies is included
 end include_symbol(jiffies)
 start include_symbol(netdev_budget)
 symbol netdev_budget is included
 end include_symbol(netdev_budget)
 start include_symbol(pv_irq_ops)
 symbol pv_irq_ops is included
 end include_symbol(pv_irq_ops)
 start include_symbol(_raw_spin_lock)
 symbol _raw_spin_lock is included
 end include_symbol(_raw_spin_lock)
 start include_symbol(cpu_number)
 symbol cpu_number is included
 end include_symbol(cpu_number)
 start include_symbol(_raw_spin_unlock)
 symbol _raw_spin_unlock is included
 end include_symbol(_raw_spin_unlock)
 start include_symbol(net_rps_action_and_irq_enable.isra.83)
 symbol net_rps_action_and_irq_enable.isra.83 is included
 end include_symbol(net_rps_action_and_irq_enable.isra.83)
 start include_symbol(dma_issue_pending_all)
 symbol dma_issue_pending_all is included
 end include_symbol(dma_issue_pending_all)
 start include_symbol(__tracepoint_napi_poll)
 symbol __tracepoint_napi_poll is included
 end include_symbol(__tracepoint_napi_poll)
 start include_symbol(__raise_softirq_irqoff)
 symbol __raise_softirq_irqoff is included
 end include_symbol(__raise_softirq_irqoff)
 start include_symbol(napi_gro_flush)
 symbol napi_gro_flush is included
 end include_symbol(napi_gro_flush)
 start include_symbol(__list_del_entry)
 symbol __list_del_entry is included
 end include_symbol(__list_del_entry)
 start include_symbol(__list_add)
 symbol __list_add is included
 end include_symbol(__list_add)
 start include_symbol(.data.unlikely)
 symbol .data.unlikely is included
 section .data.unlikely is included
 end include_symbol(.data.unlikely)
 start include_symbol(napi_complete)
 symbol napi_complete is included
 end include_symbol(napi_complete)
end include_symbol(net_rx_action)

...
dev.o: dev.o: data section .data.unlikely selected for inclusion
dev.o: 1 unsupported section change(s)
/usr/local/libexec/kpatch/create-diff-object: unreconcilable difference

__bug_table section not handled by create-object-diff

In source files that use BUG family macros, there is a special section __bug_table with a corresponding rela section. This text section contains a table of 12-byte entries each with the structure:

1st word - address of instruction pointer at BUG (relocated)
2nd word - address of string with filename (relocated)
3rd word - line number of the BUG

There is a single __bug_table for the entire file and the .rela__bug_table section references each function symbol that contains a BUG.

kpatch-build: better error message for changed asm-offsets.s

Changes to some data structure definitions (e.g. kern_ipc_perm) result in the auto-generated file asm-offsets.s file changing, resulting in the following confusing kpatch-build error:

strip:/root/.kpatch/3.10.0-115.el7.x86_64/obj2/arch/x86/kernel/asm-offsets.s: File format not recognized

We should detect this file changing in kpatch-build and tell the user that a struct definition was changed (which is unsupported).

lookup_local_symbol timekeeping_forward_now.constprop.8 (timekeeping.c)

I'm hitting this error, with following patch:

diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index 1c5b0fc..aaee73f 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -474,6 +474,8 @@ void do_gettimeofday(struct timeval *tv)
 {
        struct timespec now;

+       if (!tv)
+               return;
        getnstimeofday(&now);
        tv->tv_sec = now.tv_sec;
        tv->tv_usec = now.tv_nsec/1000;

/usr/local/libexec/kpatch/create-diff-object: kpatch_create_patches_sections: 1490: lookup_local_symbol timekeeping_forward_now.constprop.8 (timekeeping.c)

# git log -p -1 --oneline
13f02e6 Merge pull request #224 from jpoimboe/distro-specific-vmlinux

# uname -r
3.10.0-123.el7.x86_64

[RFE] print patched functions through sysfs

I have an idea to export patched functions through sysfs, so that could make user/developer to view what is changed. And it also could help to debug.

Therefore I plan to implement this function, like:

# cat /sys/kernel/kpatch/kpatch_meminfo/funcs
[function name]    [old address]    [new address]
mem_proc_show 0xffffffff81228560 -> 0xffffffffa07451d0

Is that useful?

Although no function name information is stored in struct kpatch_func, we could get it from lookup_symbol_name().
Any other thing should be considered? (kprobe maybe :-).)

add-patches-section doesn't handle symbol collisions properly

Given the following patch:

--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -2207,6 +2207,7 @@ KSM_ATTR(sleep_millisecs);
 static ssize_t pages_to_scan_show(struct kobject *kobj,
                                  struct kobj_attribute *attr, char *buf)
 {
+       printk("hello from pages_to_scan_show()!\n");
        return sprintf(buf, "%u\n", ksm_thread_pages_to_scan);
 }

running the output of create-diff-object into add-patches-section results in

found patched function pages_to_scan_show
original function at address ffffffff811a1450 (length 36)

looking at vmlinux:
13403: 0000000000000000      0 FILE    LOCAL  DEFAULT      ABS ksm.c
...
13425: ffffffff81194030     36 FUNC    LOCAL  DEFAULT        1 pages_to_scan_show
...
13723: 0000000000000000      0 FILE    LOCAL  DEFAULT      ABS huge_memory.c
...
13745: ffffffff811a1450     36 FUNC    LOCAL  DEFAULT        1 pages_to_scan_show

add-patches-section is patching the pages_to_scan_show() in huge_memory.c when it should patch the one in ksm.c.

gcc version checks

Hi,

I try to use kpatch on Debian GNU/Linux Wheezy with a 3.14.5 kernel. The kernel compilation works but when kpatch-build try to create the patch, it fail !

+ /usr/local/libexec/kpatch/create-diff-object orig/fs/proc/meminfo.o patched/fs/proc/meminfo.o /usr/lib/debug/lib/modules/3.14.5-kpatch/vmlinux output/fs/proc/meminfo.o
+ tee -a /tmp/kpatch-build-1402941756.log
meminfo.o: meminfo.o: changed section .rela.text not selected for inclusion
meminfo.o: meminfo.o: changed section .rodata.str1.8 not selected for inclusion
meminfo.o: 2 unsupported section change(s)
/usr/local/libexec/kpatch/create-diff-object: unreconcilable difference
+ [[ 2 -eq 0 ]]
+ die
+ [[ -z '' ]]
+ echo 'ERROR: kpatch build failed. Check /tmp/kpatch-build-1402941756.log for more details.'
ERROR: kpatch build failed. Check /tmp/kpatch-build-1402941756.log for more details.
+ exit 1
+ cleanup
+ [[ 1 -eq 0 ]]
+ [[ -e /media/.kpatch/src/applied-patch ]]

The command line used : kpatch-build -s /usr/src/linux-source-3.14.5-kpatch -c /boot/config-3.14.5-kpatch -d meminfo-string.patch

I found some discussions about .rela. issues but everytime it's fixed by a commit. I use the last kpatch version.

Thanks for your help.

changed section .rela__jump_table not selected for inclusion

~ $ cat jumptablefail.patch 
Index: src/kernel/timer.c
===================================================================
--- src.orig/kernel/timer.c
+++ src/kernel/timer.c
@@ -1370,6 +1370,8 @@ static void run_timer_softirq(struct sof
 {
    struct tvec_base *base = __this_cpu_read(tvec_bases);

+   printk("run_timer_softirq\n");
+
    hrtimer_run_pending();

    if (time_after_eq(jiffies, base->timer_jiffies))
~ $ kpatch-build -d jumptablefail.patch
DEBUG mode enabled
Using cache at /home/jpoimboe/.kpatch/3.13.10-200.fc20.x86_64/src
Testing patch file
patching file kernel/timer.c
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
timer.o: changed function: run_timer_softirq
timer.o: changed function: timer_cpu_notify
timer.o: changed section .rela__jump_table not selected for inclusion
/usr/local/libexec/kpatch/create-diff-object: unreconcilable difference
ERROR: kpatch build failed. Check /tmp/kpatch-build-1398801879.log for more details.

subcommand unload not recognized

(latest commit 6006cb0)

# kpatch unload kpatch-meminfo.ko
subcommand unload not recognized
usage: kpatch <command> [<args>]

Valid commands:
   install <file>       install hotpatch module to the kpatch DB
   uninstall <hotpatch> uninstall hotpatch module from the kpatch DB

   load --all           load all installed hotpatch modules into the running kernel
   load <hotpatch>      load installed hotpatch module to the running kernel
   unload <hotpatch>    unload hotpatch module from the running kernel


   info <hotpatch>      show information about an installed hotpatch module

   list                 list installed hotpatch modules

kpatch_create_dynamic_rela_sections: 1563: lookup_local_symbol .data..read_mostly (dev.c)

I'm getting error with following patch for __netif_receive_skb_core:

diff --git a/net/core/dev.c b/net/core/dev.c
index d06cd72..b7c1300 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3503,6 +3503,7 @@ ncls:
        case RX_HANDLER_PASS:
            break;
        default:
+           printk("BUG!\n");
            BUG();
        }
    }

Copying source to /root/.kpatch/src
Testing patch file
checking file net/core/dev.c
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
dev.o: changed function: __netdev_upper_dev_link
dev.o: changed function: __dev_set_promiscuity
dev.o: changed function: rollback_registered_many
dev.o: changed function: net_rx_action
dev.o: changed function: __netif_receive_skb_core
dev.o: changed function: napi_gro_complete
dev.o: changed function: unregister_netdevice_queue
dev.o: changed function: __dev_change_flags
dev.o: changed function: netdev_upper_dev_unlink
dev.o: changed function: netdev_has_upper_dev
dev.o: changed function: netdev_run_todo
dev.o: changed function: dev_set_allmulti
dev.o: changed function: netdev_master_upper_dev_get
dev.o: changed function: netdev_has_any_upper_dev
dev.o: changed function: dev_change_net_namespace
dev.o: changed function: __netdev_update_features
dev.o: changed function: register_netdevice
/usr/local/libexec/kpatch/create-diff-object: kpatch_create_dynamic_rela_sections: 1563: lookup_local_symbol .data..read_mostly (dev.c)
ERROR: kpatch build failed. Check /tmp/kpatch-build-1401370085.log for more details.

Cannot patch self built kernel from fedora srpms

I've built kernel (3.14.2-200.fc20.x86_64) without applying any patches straight from fedora SRPMs. I've booted the resulting binaries, then used kpatch-build to build a kernel patch. Trying to load that into the running kernel however is failing (see below). Can we add documentation what it takes to get this working. Thanks!

kpatch load kpatch-all.ko
loading core module: /usr/local/lib/modules/3.14.2-200.fc20.x86_64/extra/kpatch/kpatch.ko
loading patch module: kpatch-all.ko
insmod: ERROR: could not insert module kpatch-all.ko: Invalid parameters
kpatch: failed to load patch kpatch-all.ko
[root@f20-kim0labs ~]# journalctl | tail
May 17 14:49:13 f20-kim0labs login[376]: ROOT LOGIN ON tty1
May 17 14:49:13 f20-kim0labs systemd[765]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
May 17 14:49:13 f20-kim0labs systemd[765]: Failed to open private bus connection: Failed to connect to socket /run/user/0/dbus/user_bus_socket: No such file or directory
May 17 14:49:13 f20-kim0labs systemd[765]: Starting Default.
May 17 14:49:13 f20-kim0labs systemd[765]: Reached target Default.
May 17 14:49:13 f20-kim0labs systemd[765]: Startup finished in 8ms.
May 17 14:49:13 f20-kim0labs systemd[1]: Started User Manager for 0.
May 17 14:49:14 f20-kim0labs NetworkManager[423]: startup complete
May 17 14:53:06 f20-kim0labs kernel: kpatch: module verification failed: signature and/or required key missing - tainting kernel
May 17 14:53:06 f20-kim0labs kernel: kpatch: can't set ftrace filter at address 0xffffffff81429880
[root@f20-kim0labs ~]# insmod kpatch-all.ko
insmod: ERROR: could not insert module kpatch-all.ko: Invalid parameters

kpatch replace race condition

module_put() is called too early.

symbol address mismatch between vmlinux and running kernel in Ubuntu

Loading a patch module built in Ubuntu fails because the ftrace registration at module load time fails. This is because the addresses of symbols in the unstripped vmlinux do not match the addresses in the running kernel.

symbols with same (empty) name but different type

I came across a kpatch-build error, while trying following patch:

# cat tp_trivial_change_to_function_with_tp.patch
diff --git a/kernel/signal.c b/kernel/signal.c
index 50e4107..2a69b78 100644
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -2196,6 +2196,9 @@ int get_signal_to_deliver(siginfo_t *info, struct k_sigaction *return_ka,
    struct signal_struct *signal = current->signal;
    int signr;

+   if (!current)
+       return 0;
+
    if (unlikely(current->task_works))
        task_work_run();

$ kpatch-build tp_trivial_change_to_function_with_tp.patch -s /root/rpmbuild/BUILD/kernel-3.10.0-123.el7/linux-3.10.0-123.el7.x86_64 --debug
DEBUG mode enabled
Copying source to /root/.kpatch/src
Testing patch file
checking file kernel/signal.c
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
signal.o: symbol info mismatch: 
/usr/local/libexec/kpatch/create-diff-object: unreconcilable difference
ERROR: kpatch build failed. Check /tmp/kpatch-build-1400832771.log for more details.

$ tail /tmp/kpatch-build-1400832771.log
  CHK     include/generated/uapi/linux/version.h
  CHK     include/generated/utsrelease.h
  CALL    /root/.kpatch/src/scripts/checksyscalls.sh
  CC      scripts/mod/devicetable-offsets.s
  GEN     scripts/mod/devicetable-offsets.h
  HOSTCC  scripts/mod/file2alias.o
  HOSTLD  scripts/mod/modpost
  CC      kernel/signal.o
signal.o: symbol info mismatch: 
/usr/local/libexec/kpatch/create-diff-object: unreconcilable difference

Looking at debug output of patched object, I could see 2 symbols with same (empty) name ""
but different type:

=== symbol list (751 entries) ===
sym 00, type 0, bind 0, ndx 00, name 
...
sym 425, type 4, bind 0, ndx 65521, name

I tried following change, which allowed me to progress further:

diff --git a/kpatch-build/create-diff-object.c b/kpatch-build/create-diff-object.c
index 655b05d..9e2a57a 100644
--- a/kpatch-build/create-diff-object.c
+++ b/kpatch-build/create-diff-object.c
@@ -601,7 +601,7 @@ void kpatch_correlate_symbols(struct list_head *symlist1, struct list_head *syml

        list_for_each_entry(sym1, symlist1, list) {
                list_for_each_entry(sym2, symlist2, list) {
-                       if (!strcmp(sym1->name, sym2->name)) {
+                       if (!strcmp(sym1->name, sym2->name) && (sym1->type == sym2->type)) {
                                sym1->twin = sym2;
                                sym2->twin = sym1;
                                /* set initial status, might change */

Debug output:

sym 00, type 0, bind 0, ndx 00, name  (SAME)
...
sym 425, type 4, bind 0, ndx 65521, name  (SAME)
...
signal.o: changed function: do_rt_tgsigqueueinfo
signal.o: changed function: do_rt_sigqueueinfo
signal.o: changed function: get_signal_to_deliver
signal.o: signal.o: changed section .rela__jump_table not selected for inclusion
signal.o: 1 unsupported section change(s)
/root/kpatch/kpatch-build/create-diff-object: unreconcilable difference

but then I hit change in jump_table, which looks unsupported at the moment, according to commit 1622123.

provide ability to set LIBDIR

please, add ability to set LIBDIR manually and use this path in dracut dir and modules dir.

NMI handler live patching

Kpatch currently uses stop_machine to check all task stacks. Indeed, almost all functions never runs in that, but NMI can happen. Suppose that an NMI happens on other cpu while kpatch enabling the function list. It may cause consistency issue.

At this point, I have no good idea to solve this issue. SInce we have nothing to wait all running nmi handler to be done inside stop_machine, it seems that the consistency issue can not be solved.

Add [-c| --config] option to kpatch-build

Because kpatch-build will check the .config file from the directory
specified by "--sourcedir", if not existed, the copy src phase will fail.

So adding [-c| --config] option to kpatch-build maybe a good choice.

create-diff-object: create __mcount_loc section

For ftrace to recognize patched functions, we need to create the __mcount_loc section (and its rela section), which is just a simple array of pointers to the first instruction of every patched function. Normally the kernel recordmcount.pl script does this, but it only works for .text and a few other sections, so it ignores our -ffunction-sections sections.

on F20, unable to create patch

Log file at: http://paste.fedoraproject.org/101699/00703421
I can't see any glaring error

I'm very new to this (most likely user error). This is what I got:
./kpatch-build ~/all.patch -r ~/kernel-3.14.2-200.fc20.src.rpm
Fedora/Red Hat distribution detected
Unpacking kernel source
Testing patch file
checking file drivers/tty/n_tty.c
checking file drivers/tty/tty_buffer.c
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
ERROR: kpatch build failed. Check /tmp/kpatch-build-1400064989.log for more details.

create-diff-object have troubles with special ELF sections

Hi.

When I try to compile my patch:

...
create-diff-object:
...
readdir.o: readdir.o: changed section .rela.altinstructions not selected for inclusion
readdir.o: readdir.o: changed section .fixup not selected for inclusion
readdir.o: readdir.o: changed section .rela.fixup not selected for inclusion
readdir.o: readdir.o: changed section .rela__ex_table not selected for inclusion
readdir.o: 4 unsupported section change(s)
/usr/local/libexec/kpatch/create-diff-object: unreconcilable difference

Solution from Josh:
To fix this we need to add support for 3 special sections:
.altinstructions, .fixup, __ex_table.

For full info, please read https://www.redhat.com/archives/kpatch/2014-May/msg00000.html.

kretprobes / activeness safety incompatibility

This is from an email discussion with @mhiramat.

kretprobes removes the target function's caller from the stack, which means that the kpatch activeness safety check could fail to detect a function on the stack of a process.

I added a dump_stack() call to meminfo_proc_show. Here's the normal
case with no ftrace or kprobes:

[   24.903293]  [<ffffffff8168731c>] dump_stack+0x45/0x56
[   24.903297]  [<ffffffff81228585>] meminfo_proc_show+0x25/0x530
[   24.903334]  [<ffffffff811db90a>] seq_read+0x14a/0x390
[   24.903338]  [<ffffffff8121fa9d>] proc_reg_read+0x3d/0x80
[   24.903340]  [<ffffffff811b89b5>] vfs_read+0x95/0x160
[   24.903342]  [<ffffffff811b94c9>] SyS_read+0x49/0xa0
[   24.903347]  [<ffffffff816962e9>] system_call_fastpath+0x16/0x1b

Here it is with function_graph tracing enabled. Notice that seq_read is
still on the stack:

[   92.606903]  [<ffffffff8168731c>] dump_stack+0x45/0x56
[   92.606907]  [<ffffffff81228585>] meminfo_proc_show+0x25/0x530
[   92.606945]  [<ffffffff81696185>] ftrace_graph_caller+0x85/0x85
[   92.606946]  [<ffffffff811db90a>] seq_read+0x14a/0x390
[   92.606952]  [<ffffffff8121fa9d>] proc_reg_read+0x3d/0x80
[   92.606955]  [<ffffffff811b89b5>] vfs_read+0x95/0x160
[   92.606957]  [<ffffffff811b94c9>] SyS_read+0x49/0xa0
[   92.606959]  [<ffffffff816962e9>] system_call_fastpath+0x16/0x1b

And here it is with a kretprobe enabled. Now seq_read is no longer on
the stack, as it has been replaced with kretprobe_trampoline_holder:

[   45.427144]  [<ffffffff8168731c>] dump_stack+0x45/0x56
[   45.427148]  [<ffffffff81228585>] meminfo_proc_show+0x25/0x530
[   45.427185]  [<ffffffff81690269>] kretprobe_trampoline_holder+0x9/0x9
[   45.427191]  [<ffffffff8121fa9d>] proc_reg_read+0x3d/0x80
[   45.427194]  [<ffffffff811b89b5>] vfs_read+0x95/0x160
[   45.427196]  [<ffffffff811b94c9>] SyS_read+0x49/0xa0
[   45.427199]  [<ffffffff816962e9>] system_call_fastpath+0x16/0x1b

Some options:

kretprobes somehow leaves seq_read on the stack
kpatch can check for the presence of kretprobe_trampoline_holder on all the backtraces. Downside is that it could create a lot of false positive failures.
kpatch uses a (new?) kprobe interface to detect any kretprobes on the target function.

Different compiler causes the kernel build to behave differently

I built the official kernel from kernel-3.13.6-200.fc20.src.rpm on x86_64 platform.
But the address of meminfo_proc_show are different between generated vmlinux and running kernel.
So that the ftrace filter can't be set to this function meminfo_proc_show.

Here is /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/obj/fs/proc/.meminfo.o.cmd:

cmd_fs/proc/meminfo.o := gcc -Wp,-MD,fs/proc/.meminfo.o.d  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.2/include -I/home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include -Iarch/x86/include/generated  -I/home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include -Iinclude -I/home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi -Iarch/x86/include/generated/uapi -I/home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi -Iinclude/generated/uapi -include /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/kconfig.h  -I/home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/fs/proc -Ifs/proc -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -O2 -m64 -mno-mmx -mno-sse -mpreferred-stack-boundary=3 -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -Wframe-larger-than=2048 -Wno-unused-but-set-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -pg -mfentry -DCC_USING_FENTRY -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -DCC_HAVE_ASM_GOTO    -D"KBUILD_STR(s)=\#s" -D"KBUILD_BASENAME=KBUILD_STR(meminfo)"  -D"KBUILD_MODNAME=KBUILD_STR(proc)" -c -o fs/proc/meminfo.o /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/fs/proc/meminfo.c

source_fs/proc/meminfo.o := /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/fs/proc/meminfo.c

deps_fs/proc/meminfo.o := \
    $(wildcard include/config/highmem.h) \
    $(wildcard include/config/mmu.h) \
    $(wildcard include/config/quicklist.h) \
    $(wildcard include/config/memory/failure.h) \
    $(wildcard include/config/transparent/hugepage.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/fs.h \
    $(wildcard include/config/sysfs.h) \
    $(wildcard include/config/smp.h) \
    $(wildcard include/config/fs/posix/acl.h) \
    $(wildcard include/config/security.h) \
    $(wildcard include/config/quota.h) \
    $(wildcard include/config/fsnotify.h) \
    $(wildcard include/config/ima.h) \
    $(wildcard include/config/preempt.h) \
    $(wildcard include/config/epoll.h) \
    $(wildcard include/config/debug/writecount.h) \
    $(wildcard include/config/file/locking.h) \
    $(wildcard include/config/debug/lock/alloc.h) \
    $(wildcard include/config/auditsyscall.h) \
    $(wildcard include/config/block.h) \
    $(wildcard include/config/fs/xip.h) \
    $(wildcard include/config/migration.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/linkage.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/compiler.h \
    $(wildcard include/config/sparse/rcu/pointer.h) \
    $(wildcard include/config/trace/branch/profiling.h) \
    $(wildcard include/config/profile/all/branches.h) \
    $(wildcard include/config/enable/must/check.h) \
    $(wildcard include/config/enable/warn/deprecated.h) \
    $(wildcard include/config/kprobes.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/compiler-gcc.h \
    $(wildcard include/config/arch/supports/optimized/inlining.h) \
    $(wildcard include/config/optimize/inlining.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/compiler-gcc4.h \
    $(wildcard include/config/arch/use/builtin/bswap.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/stringify.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/export.h \
    $(wildcard include/config/have/underscore/symbol/prefix.h) \
    $(wildcard include/config/modules.h) \
    $(wildcard include/config/modversions.h) \
    $(wildcard include/config/unused/symbols.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/linkage.h \
    $(wildcard include/config/x86/32.h) \
    $(wildcard include/config/x86/64.h) \
    $(wildcard include/config/x86/alignment/16.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/wait.h \
    $(wildcard include/config/lockdep.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/list.h \
    $(wildcard include/config/debug/list.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/types.h \
    $(wildcard include/config/uid16.h) \
    $(wildcard include/config/lbdaf.h) \
    $(wildcard include/config/arch/dma/addr/t/64bit.h) \
    $(wildcard include/config/phys/addr/t/64bit.h) \
    $(wildcard include/config/64bit.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/int-ll64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/int-ll64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/bitsperlong.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/bitsperlong.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/bitsperlong.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/posix_types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/stddef.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/stddef.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/posix_types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/posix_types_64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/posix_types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/poison.h \
    $(wildcard include/config/illegal/pointer/value.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/const.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/spinlock.h \
    $(wildcard include/config/debug/spinlock.h) \
    $(wildcard include/config/generic/lockbreak.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/typecheck.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/preempt.h \
    $(wildcard include/config/debug/preempt.h) \
    $(wildcard include/config/preempt/tracer.h) \
    $(wildcard include/config/preempt/count.h) \
    $(wildcard include/config/context/tracking.h) \
    $(wildcard include/config/preempt/notifiers.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/preempt.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/rmwcc.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/percpu.h \
    $(wildcard include/config/x86/64/smp.h) \
    $(wildcard include/config/x86/cmpxchg64.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/kernel.h \
    $(wildcard include/config/preempt/voluntary.h) \
    $(wildcard include/config/debug/atomic/sleep.h) \
    $(wildcard include/config/prove/locking.h) \
    $(wildcard include/config/ring/buffer.h) \
    $(wildcard include/config/tracing.h) \
    $(wildcard include/config/ftrace/mcount/record.h) \
  /usr/lib/gcc/x86_64-redhat-linux/4.8.2/include/stdarg.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/bitops.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/bitops.h \
    $(wildcard include/config/x86/cmov.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/alternative.h \
    $(wildcard include/config/paravirt.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/asm.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/ptrace.h \
    $(wildcard include/config/x86/debugctlmsr.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/segment.h \
    $(wildcard include/config/cc/stackprotector.h) \
    $(wildcard include/config/x86/32/lazy/gs.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/cache.h \
    $(wildcard include/config/x86/l1/cache/shift.h) \
    $(wildcard include/config/x86/internode/cache/shift.h) \
    $(wildcard include/config/x86/vsmp.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/page_types.h \
    $(wildcard include/config/physical/start.h) \
    $(wildcard include/config/physical/align.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/page_64_types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/ptrace.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/ptrace-abi.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/processor-flags.h \
    $(wildcard include/config/vm86.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/processor-flags.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/init.h \
    $(wildcard include/config/broken/rodata.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/paravirt_types.h \
    $(wildcard include/config/x86/local/apic.h) \
    $(wildcard include/config/x86/pae.h) \
    $(wildcard include/config/paravirt/debug.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/desc_defs.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/kmap_types.h \
    $(wildcard include/config/debug/highmem.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/kmap_types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/pgtable_types.h \
    $(wildcard include/config/kmemcheck.h) \
    $(wildcard include/config/mem/soft/dirty.h) \
    $(wildcard include/config/compat/vdso.h) \
    $(wildcard include/config/proc/fs.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/pgtable_64_types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/sparsemem.h \
    $(wildcard include/config/sparsemem.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/spinlock_types.h \
    $(wildcard include/config/paravirt/spinlocks.h) \
    $(wildcard include/config/nr/cpus.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/rwlock.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/ptrace.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/cpufeature.h \
    $(wildcard include/config/x86/debug/static/cpu/has.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/required-features.h \
    $(wildcard include/config/x86/minimum/cpu/family.h) \
    $(wildcard include/config/math/emulation.h) \
    $(wildcard include/config/x86/use/3dnow.h) \
    $(wildcard include/config/x86/p6/nop.h) \
    $(wildcard include/config/matom.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/bitops/find.h \
    $(wildcard include/config/generic/find/first/bit.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/bitops/sched.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/arch_hweight.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/bitops/const_hweight.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/bitops/le.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/byteorder.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/byteorder/little_endian.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/byteorder/little_endian.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/swab.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/swab.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/swab.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/byteorder/generic.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/bitops/ext2-atomic-setbit.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/log2.h \
    $(wildcard include/config/arch/has/ilog2/u32.h) \
    $(wildcard include/config/arch/has/ilog2/u64.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/printk.h \
    $(wildcard include/config/early/printk.h) \
    $(wildcard include/config/printk.h) \
    $(wildcard include/config/dynamic/debug.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/kern_levels.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/dynamic_debug.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/kernel.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/sysinfo.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/percpu.h \
    $(wildcard include/config/have/setup/per/cpu/area.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/threads.h \
    $(wildcard include/config/base/small.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/percpu-defs.h \
    $(wildcard include/config/debug/force/weak/per/cpu.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/thread_info.h \
    $(wildcard include/config/compat.h) \
    $(wildcard include/config/debug/stack/usage.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/bug.h \
    $(wildcard include/config/generic/bug.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/bug.h \
    $(wildcard include/config/bug.h) \
    $(wildcard include/config/debug/bugverbose.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/bug.h \
    $(wildcard include/config/generic/bug/relative/pointers.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/thread_info.h \
    $(wildcard include/config/ia32/emulation.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/page.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/page_64.h \
    $(wildcard include/config/debug/virtual.h) \
    $(wildcard include/config/flatmem.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/range.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/memory_model.h \
    $(wildcard include/config/discontigmem.h) \
    $(wildcard include/config/sparsemem/vmemmap.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/getorder.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/processor.h \
    $(wildcard include/config/m486.h) \
    $(wildcard include/config/xen.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/vm86.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/vm86.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/math_emu.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/sigcontext.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/sigcontext.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/current.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/msr.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/msr.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/msr-index.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/ioctl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/ioctl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/ioctl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/ioctl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/errno.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/errno.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/errno-base.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/cpumask.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/cpumask.h \
    $(wildcard include/config/cpumask/offstack.h) \
    $(wildcard include/config/hotplug/cpu.h) \
    $(wildcard include/config/debug/per/cpu/maps.h) \
    $(wildcard include/config/disable/obsolete/cpumask/functions.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/bitmap.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/string.h \
    $(wildcard include/config/binary/printf.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/string.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/string.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/string_64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/paravirt.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/nops.h \
    $(wildcard include/config/mk7.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/special_insns.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/personality.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/personality.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/cache.h \
    $(wildcard include/config/arch/has/cache/line/size.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/math64.h \
    $(wildcard include/config/arch/supports/int128.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/div64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/div64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/err.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/irqflags.h \
    $(wildcard include/config/trace/irqflags.h) \
    $(wildcard include/config/irqsoff/tracer.h) \
    $(wildcard include/config/trace/irqflags/support.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/irqflags.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/atomic.h \
    $(wildcard include/config/arch/has/atomic/or.h) \
    $(wildcard include/config/generic/atomic64.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/atomic.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/cmpxchg.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/cmpxchg_64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/atomic64_64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/atomic-long.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/bottom_half.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/barrier.h \
    $(wildcard include/config/x86/ppro/fence.h) \
    $(wildcard include/config/x86/oostore.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/spinlock_types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/lockdep.h \
    $(wildcard include/config/lock/stat.h) \
    $(wildcard include/config/prove/rcu.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/rwlock_types.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/spinlock.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/jump_label.h \
    $(wildcard include/config/jump/label.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/jump_label.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/rwlock.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/spinlock_api_smp.h \
    $(wildcard include/config/inline/spin/lock.h) \
    $(wildcard include/config/inline/spin/lock/bh.h) \
    $(wildcard include/config/inline/spin/lock/irq.h) \
    $(wildcard include/config/inline/spin/lock/irqsave.h) \
    $(wildcard include/config/inline/spin/trylock.h) \
    $(wildcard include/config/inline/spin/trylock/bh.h) \
    $(wildcard include/config/uninline/spin/unlock.h) \
    $(wildcard include/config/inline/spin/unlock/bh.h) \
    $(wildcard include/config/inline/spin/unlock/irq.h) \
    $(wildcard include/config/inline/spin/unlock/irqrestore.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/rwlock_api_smp.h \
    $(wildcard include/config/inline/read/lock.h) \
    $(wildcard include/config/inline/write/lock.h) \
    $(wildcard include/config/inline/read/lock/bh.h) \
    $(wildcard include/config/inline/write/lock/bh.h) \
    $(wildcard include/config/inline/read/lock/irq.h) \
    $(wildcard include/config/inline/write/lock/irq.h) \
    $(wildcard include/config/inline/read/lock/irqsave.h) \
    $(wildcard include/config/inline/write/lock/irqsave.h) \
    $(wildcard include/config/inline/read/trylock.h) \
    $(wildcard include/config/inline/write/trylock.h) \
    $(wildcard include/config/inline/read/unlock.h) \
    $(wildcard include/config/inline/write/unlock.h) \
    $(wildcard include/config/inline/read/unlock/bh.h) \
    $(wildcard include/config/inline/write/unlock/bh.h) \
    $(wildcard include/config/inline/read/unlock/irq.h) \
    $(wildcard include/config/inline/write/unlock/irq.h) \
    $(wildcard include/config/inline/read/unlock/irqrestore.h) \
    $(wildcard include/config/inline/write/unlock/irqrestore.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/wait.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/kdev_t.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/kdev_t.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/dcache.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/rculist.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/rcupdate.h \
    $(wildcard include/config/rcu/torture/test.h) \
    $(wildcard include/config/tree/rcu.h) \
    $(wildcard include/config/tree/preempt/rcu.h) \
    $(wildcard include/config/rcu/trace.h) \
    $(wildcard include/config/preempt/rcu.h) \
    $(wildcard include/config/rcu/user/qs.h) \
    $(wildcard include/config/tiny/rcu.h) \
    $(wildcard include/config/debug/objects/rcu/head.h) \
    $(wildcard include/config/rcu/nocb/cpu.h) \
    $(wildcard include/config/no/hz/full/sysidle.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/seqlock.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/completion.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/debugobjects.h \
    $(wildcard include/config/debug/objects.h) \
    $(wildcard include/config/debug/objects/free.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/rcutree.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/rculist_bl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/list_bl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/bit_spinlock.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/lockref.h \
    $(wildcard include/config/arch/use/cmpxchg/lockref.h) \
  include/generated/bounds.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/path.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/stat.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/stat.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/stat.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/time.h \
    $(wildcard include/config/arch/uses/gettimeoffset.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/time.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/uidgid.h \
    $(wildcard include/config/uidgid/strict/type/checks.h) \
    $(wildcard include/config/user/ns.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/highuid.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/list_lru.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/nodemask.h \
    $(wildcard include/config/movable/node.h) \
    $(wildcard include/config/numa.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/numa.h \
    $(wildcard include/config/nodes/shift.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/llist.h \
    $(wildcard include/config/arch/have/nmi/safe/cmpxchg.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/radix-tree.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/rbtree.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/pid.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/mutex.h \
    $(wildcard include/config/debug/mutexes.h) \
    $(wildcard include/config/mutex/spin/on/owner.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/capability.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/capability.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/semaphore.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/fiemap.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/shrinker.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/migrate_mode.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/percpu-rwsem.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/rwsem.h \
    $(wildcard include/config/rwsem/generic/spinlock.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/rwsem.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/percpu.h \
    $(wildcard include/config/need/per/cpu/embed/first/chunk.h) \
    $(wildcard include/config/need/per/cpu/page/first/chunk.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/smp.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/errno.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/errno.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/smp.h \
    $(wildcard include/config/x86/io/apic.h) \
    $(wildcard include/config/x86/32/smp.h) \
    $(wildcard include/config/debug/nmi/selftest.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/mpspec.h \
    $(wildcard include/config/x86/numaq.h) \
    $(wildcard include/config/eisa.h) \
    $(wildcard include/config/x86/mpparse.h) \
    $(wildcard include/config/acpi.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/mpspec_def.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/x86_init.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/bootparam.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/screen_info.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/screen_info.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/apm_bios.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/apm_bios.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/edd.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/edd.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/e820.h \
    $(wildcard include/config/efi.h) \
    $(wildcard include/config/hibernation.h) \
    $(wildcard include/config/memtest.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/e820.h \
    $(wildcard include/config/intel/txt.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/ioport.h \
    $(wildcard include/config/memory/hotremove.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/ist.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/ist.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/video/edid.h \
    $(wildcard include/config/x86.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/video/edid.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/apicdef.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/apic.h \
    $(wildcard include/config/x86/x2apic.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/pm.h \
    $(wildcard include/config/vt/console/sleep.h) \
    $(wildcard include/config/pm.h) \
    $(wildcard include/config/pm/sleep.h) \
    $(wildcard include/config/pm/runtime.h) \
    $(wildcard include/config/pm/clk.h) \
    $(wildcard include/config/pm/generic/domains.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/workqueue.h \
    $(wildcard include/config/debug/objects/work.h) \
    $(wildcard include/config/freezer.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/timer.h \
    $(wildcard include/config/timer/stats.h) \
    $(wildcard include/config/debug/objects/timers.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/ktime.h \
    $(wildcard include/config/ktime/scalar.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/jiffies.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/timex.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/timex.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/param.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/param.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/param.h \
    $(wildcard include/config/hz.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/param.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/timex.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/tsc.h \
    $(wildcard include/config/x86/tsc.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/fixmap.h \
    $(wildcard include/config/paravirt/clock.h) \
    $(wildcard include/config/provide/ohci1394/dma/init.h) \
    $(wildcard include/config/x86/visws/apic.h) \
    $(wildcard include/config/pci/mmconfig.h) \
    $(wildcard include/config/x86/intel/mid.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/acpi.h \
    $(wildcard include/config/acpi/numa.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/acpi/pdc_intel.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/numa.h \
    $(wildcard include/config/numa/emu.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/topology.h \
    $(wildcard include/config/x86/ht.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/topology.h \
    $(wildcard include/config/have/memoryless/nodes.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/mmu.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/realmode.h \
    $(wildcard include/config/acpi/sleep.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/io.h \
    $(wildcard include/config/mtrr.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/iomap.h \
    $(wildcard include/config/has/ioport.h) \
    $(wildcard include/config/pci.h) \
    $(wildcard include/config/generic/iomap.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/pci_iomap.h \
    $(wildcard include/config/no/generic/pci/ioport/map.h) \
    $(wildcard include/config/generic/pci/iomap.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/vmalloc.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/xen/xen.h \
    $(wildcard include/config/xen/dom0.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/xen/interface/xen.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/xen/interface.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/xen/interface_64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/pvclock-abi.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/xen/hypervisor.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/pvclock.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/clocksource.h \
    $(wildcard include/config/arch/clocksource/data.h) \
    $(wildcard include/config/clocksource/watchdog.h) \
    $(wildcard include/config/clksrc/of.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/clocksource.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/vsyscall.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/vsyscall.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/vvar.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/idle.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/io_apic.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/irq_vectors.h \
    $(wildcard include/config/have/kvm.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/pfn.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/blk_types.h \
    $(wildcard include/config/blk/cgroup.h) \
    $(wildcard include/config/blk/dev/integrity.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/fs.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/limits.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/quota.h \
    $(wildcard include/config/quota/netlink/interface.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/percpu_counter.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/dqblk_xfs.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/dqblk_v1.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/dqblk_v2.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/dqblk_qtree.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/projid.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/quota.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/nfs_fs_i.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/fcntl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/fcntl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/fcntl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/fcntl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/mm.h \
    $(wildcard include/config/need/multiple/nodes.h) \
    $(wildcard include/config/sysctl.h) \
    $(wildcard include/config/ppc.h) \
    $(wildcard include/config/parisc.h) \
    $(wildcard include/config/metag.h) \
    $(wildcard include/config/ia64.h) \
    $(wildcard include/config/stack/growsup.h) \
    $(wildcard include/config/numa/balancing.h) \
    $(wildcard include/config/ksm.h) \
    $(wildcard include/config/have/memblock/node/map.h) \
    $(wildcard include/config/have/arch/early/pfn/to/nid.h) \
    $(wildcard include/config/debug/vm/rb.h) \
    $(wildcard include/config/arch/uses/numa/prot/none.h) \
    $(wildcard include/config/debug/pagealloc.h) \
    $(wildcard include/config/memory/hotplug.h) \
    $(wildcard include/config/hugetlbfs.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/gfp.h \
    $(wildcard include/config/zone/dma.h) \
    $(wildcard include/config/zone/dma32.h) \
    $(wildcard include/config/cma.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/mmzone.h \
    $(wildcard include/config/force/max/zoneorder.h) \
    $(wildcard include/config/memory/isolation.h) \
    $(wildcard include/config/memcg.h) \
    $(wildcard include/config/compaction.h) \
    $(wildcard include/config/flat/node/mem/map.h) \
    $(wildcard include/config/no/bootmem.h) \
    $(wildcard include/config/have/memory/present.h) \
    $(wildcard include/config/need/node/memmap/size.h) \
    $(wildcard include/config/sparsemem/extreme.h) \
    $(wildcard include/config/have/arch/pfn/valid.h) \
    $(wildcard include/config/nodes/span/other/nodes.h) \
    $(wildcard include/config/holes/in/zone.h) \
    $(wildcard include/config/arch/has/holes/memorymodel.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/pageblock-flags.h \
    $(wildcard include/config/hugetlb/page.h) \
    $(wildcard include/config/hugetlb/page/size/variable.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/page-flags-layout.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/memory_hotplug.h \
    $(wildcard include/config/have/arch/nodedata/extension.h) \
    $(wildcard include/config/have/bootmem/info/node.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/notifier.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/srcu.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/mmzone.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/mmzone_64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/mmdebug.h \
    $(wildcard include/config/debug/vm.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/topology.h \
    $(wildcard include/config/sched/smt.h) \
    $(wildcard include/config/sched/mc.h) \
    $(wildcard include/config/sched/book.h) \
    $(wildcard include/config/use/percpu/numa/node/id.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/debug_locks.h \
    $(wildcard include/config/debug/locking/api/selftests.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/mm_types.h \
    $(wildcard include/config/split/ptlock/cpus.h) \
    $(wildcard include/config/arch/enable/split/pmd/ptlock.h) \
    $(wildcard include/config/have/cmpxchg/double.h) \
    $(wildcard include/config/have/aligned/struct/page.h) \
    $(wildcard include/config/want/page/debug/flags.h) \
    $(wildcard include/config/aio.h) \
    $(wildcard include/config/mm/owner.h) \
    $(wildcard include/config/mmu/notifier.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/auxvec.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/auxvec.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/auxvec.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/page-debug-flags.h \
    $(wildcard include/config/page/poisoning.h) \
    $(wildcard include/config/page/guard.h) \
    $(wildcard include/config/page/debug/something/else.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/uprobes.h \
    $(wildcard include/config/arch/supports/uprobes.h) \
    $(wildcard include/config/uprobes.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/uprobes.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/pgtable.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/pgtable_64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/pgtable.h \
    $(wildcard include/config/have/arch/soft/dirty.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/page-flags.h \
    $(wildcard include/config/pageflags/extended.h) \
    $(wildcard include/config/arch/uses/pg/uncached.h) \
    $(wildcard include/config/swap.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/huge_mm.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/vmstat.h \
    $(wildcard include/config/vm/event/counters.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/vm_event_item.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/hugetlb.h \
    $(wildcard include/config/arch/want/huge/pmd/share.h) \
    $(wildcard include/config/cgroup/hugetlb.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/hugetlb_inline.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/cgroup.h \
    $(wildcard include/config/cgroups.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/sched.h \
    $(wildcard include/config/sched/debug.h) \
    $(wildcard include/config/no/hz/common.h) \
    $(wildcard include/config/lockup/detector.h) \
    $(wildcard include/config/detect/hung/task.h) \
    $(wildcard include/config/core/dump/default/elf/headers.h) \
    $(wildcard include/config/sched/autogroup.h) \
    $(wildcard include/config/virt/cpu/accounting/native.h) \
    $(wildcard include/config/bsd/process/acct.h) \
    $(wildcard include/config/taskstats.h) \
    $(wildcard include/config/audit.h) \
    $(wildcard include/config/inotify/user.h) \
    $(wildcard include/config/fanotify.h) \
    $(wildcard include/config/posix/mqueue.h) \
    $(wildcard include/config/keys.h) \
    $(wildcard include/config/perf/events.h) \
    $(wildcard include/config/schedstats.h) \
    $(wildcard include/config/task/delay/acct.h) \
    $(wildcard include/config/fair/group/sched.h) \
    $(wildcard include/config/rt/group/sched.h) \
    $(wildcard include/config/cgroup/sched.h) \
    $(wildcard include/config/blk/dev/io/trace.h) \
    $(wildcard include/config/rcu/boost.h) \
    $(wildcard include/config/compat/brk.h) \
    $(wildcard include/config/virt/cpu/accounting/gen.h) \
    $(wildcard include/config/sysvipc.h) \
    $(wildcard include/config/rt/mutexes.h) \
    $(wildcard include/config/task/xacct.h) \
    $(wildcard include/config/cpusets.h) \
    $(wildcard include/config/futex.h) \
    $(wildcard include/config/fault/injection.h) \
    $(wildcard include/config/latencytop.h) \
    $(wildcard include/config/function/graph/tracer.h) \
    $(wildcard include/config/bcache.h) \
    $(wildcard include/config/have/unstable/sched/clock.h) \
    $(wildcard include/config/irq/time/accounting.h) \
    $(wildcard include/config/no/hz/full.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/sched.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/preempt_mask.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/hardirq.h \
    $(wildcard include/config/x86/thermal/vector.h) \
    $(wildcard include/config/x86/mce/threshold.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/irq.h \
    $(wildcard include/config/generic/pending/irq.h) \
    $(wildcard include/config/hardirqs/sw/resend.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/irqreturn.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/irqnr.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/irqnr.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/irq.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/irq_regs.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/irqdesc.h \
    $(wildcard include/config/irq/preflow/fasteoi.h) \
    $(wildcard include/config/sparse/irq.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/hw_irq.h \
    $(wildcard include/config/irq/remap.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/profile.h \
    $(wildcard include/config/profiling.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/sections.h \
    $(wildcard include/config/debug/rodata.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/sections.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/uaccess.h \
    $(wildcard include/config/x86/intel/usercopy.h) \
    $(wildcard include/config/debug/strict/user/copy/checks.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/smap.h \
    $(wildcard include/config/x86/smap.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/uaccess_64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/cputime.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/cputime.h \
    $(wildcard include/config/virt/cpu/accounting.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/cputime_jiffies.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/sem.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/sem.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/ipc.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/ipc.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/ipcbuf.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/ipcbuf.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/sembuf.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/signal.h \
    $(wildcard include/config/old/sigaction.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/signal.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/signal.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/signal.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/signal-defs.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/siginfo.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/siginfo.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/siginfo.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/proportions.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/seccomp.h \
    $(wildcard include/config/seccomp.h) \
    $(wildcard include/config/seccomp/filter.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/seccomp.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/seccomp.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/seccomp_64.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/unistd.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/unistd.h \
    $(wildcard include/config/x86/x32/abi.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/unistd.h \
  arch/x86/include/generated/uapi/asm/unistd_64.h \
  arch/x86/include/generated/asm/unistd_64_x32.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/ia32_unistd.h \
  arch/x86/include/generated/asm/unistd_32_ia32.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/rtmutex.h \
    $(wildcard include/config/debug/rt/mutexes.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/plist.h \
    $(wildcard include/config/debug/pi/list.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/resource.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/resource.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/resource.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/resource.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/resource.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/hrtimer.h \
    $(wildcard include/config/high/res/timers.h) \
    $(wildcard include/config/timerfd.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/timerqueue.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/task_io_accounting.h \
    $(wildcard include/config/task/io/accounting.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/latencytop.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/cred.h \
    $(wildcard include/config/debug/credentials.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/key.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/sysctl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/sysctl.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/assoc_array.h \
    $(wildcard include/config/associative/array.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/selinux.h \
    $(wildcard include/config/security/selinux.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/cgroupstats.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/taskstats.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/prio_heap.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/idr.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/xattr.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/slab.h \
    $(wildcard include/config/slab/debug.h) \
    $(wildcard include/config/failslab.h) \
    $(wildcard include/config/slob.h) \
    $(wildcard include/config/slab.h) \
    $(wildcard include/config/slub.h) \
    $(wildcard include/config/debug/slab.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/kmemleak.h \
    $(wildcard include/config/debug/kmemleak.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/slub_def.h \
    $(wildcard include/config/slub/stats.h) \
    $(wildcard include/config/memcg/kmem.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/kobject.h \
    $(wildcard include/config/debug/kobject/release.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/sysfs.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/kobject_ns.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/kref.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/xattr.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/percpu-refcount.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/cgroup_subsys.h \
    $(wildcard include/config/cgroup/debug.h) \
    $(wildcard include/config/cgroup/cpuacct.h) \
    $(wildcard include/config/cgroup/device.h) \
    $(wildcard include/config/cgroup/freezer.h) \
    $(wildcard include/config/net/cls/cgroup.h) \
    $(wildcard include/config/cgroup/perf.h) \
    $(wildcard include/config/netprio/cgroup.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/mempolicy.h \
    $(wildcard include/config/tmpfs.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/pagemap.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/highmem.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/uaccess.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/hardirq.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/ftrace_irq.h \
    $(wildcard include/config/ftrace/nmi/enter.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/vtime.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/context_tracking_state.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/static_key.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/cacheflush.h \
    $(wildcard include/config/x86/pat.h) \
    $(wildcard include/config/debug/rodata/test.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/cacheflush.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/mempolicy.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/shm.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/shm.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/shmbuf.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/shmbuf.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/shmparam.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/tlbflush.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/hugetlb.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/asm-generic/hugetlb.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/mman.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/mman.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/uapi/asm/mman.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/mman.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/asm-generic/mman-common.h \
    $(wildcard include/config/mmap/allow/uninitialized.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/proc_fs.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/quicklist.h \
    $(wildcard include/config/nr/quick.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/seq_file.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/swap.h \
    $(wildcard include/config/frontswap.h) \
    $(wildcard include/config/memcg/swap.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/memcontrol.h \
    $(wildcard include/config/inet.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/node.h \
    $(wildcard include/config/memory/hotplug/sparse.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/device.h \
    $(wildcard include/config/debug/devres.h) \
    $(wildcard include/config/pinctrl.h) \
    $(wildcard include/config/dma/cma.h) \
    $(wildcard include/config/devtmpfs.h) \
    $(wildcard include/config/sysfs/deprecated.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/klist.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/pinctrl/devinfo.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/pinctrl/consumer.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/pinctrl/pinctrl-state.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/ratelimit.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/device.h \
    $(wildcard include/config/x86/dev/dma/ops.h) \
    $(wildcard include/config/intel/iommu.h) \
    $(wildcard include/config/amd/iommu.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/pm_wakeup.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/fs/proc/internal.h \
    $(wildcard include/config/proc/devicetree.h) \
    $(wildcard include/config/net.h) \
    $(wildcard include/config/proc/sysctl.h) \
    $(wildcard include/config/tty.h) \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/proc_ns.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/linux/binfmts.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/arch/x86/include/asm/exec.h \
  /home/ryan/.kpatch/3.13.6-200.fc20.x86_64/src/include/uapi/linux/binfmts.h \

fs/proc/meminfo.o: $(deps_fs/proc/meminfo.o)

$(deps_fs/proc/meminfo.o):

kpatch-build still need other packages even after yum-builddep kernel

Due to https://bugzilla.redhat.com/show_bug.cgi?id=1040219 , we still need to install elfutils-devel zlib-devel binutils-devel newt-devel python-devel perl-ExtUtils-Embed audit-libs-devel numactl-devel pciutils-devel for rpmbuild -bp, maybe we should also add this in README?

kpatch build is confused by kstrtab symbols

Junliang Li reported this on the mailing list. The following patch fails:

Index: src/drivers/scsi/constants.c
===================================================================
--- src.orig/drivers/scsi/constants.c
+++ src/drivers/scsi/constants.c
@@ -1546,6 +1546,8 @@ void scsi_print_sense(char *name, struct
 {
    struct scsi_sense_hdr sshdr;

+   printk(KERN_INFO "\nkpatch code\n");
+
    scmd_printk(KERN_INFO, cmd, " ");
    scsi_decode_sense_buffer(cmd->sense_buffer, SCSI_SENSE_BUFFERSIZE,
                 &sshdr);

It fails with:

create-diff-object: kpatch_create_symbol_table: 372: local symbol __kstrtab_scsi_show_result starts at section offset 18, expected 0

It's getting confused by the __kstrtab_scsi_show_result symbol, which I think is created because of EXPORT_SYMBOL(scsi_show_result).

build failed for kernel 3.8.13

I tried to build the kpatch source code on kernel 3.8.13, but it's failed, the error message as following:
make -C kpatch-build
make[1]: Entering directory /data/kpatch/kpatch-0.1.5/kpatch-build' make[1]: Nothing to be done forall'.
make[1]: Leaving directory /data/kpatch/kpatch-0.1.5/kpatch-build' make -C kpatch make[1]: Entering directory/data/kpatch/kpatch-0.1.5/kpatch'
make[1]: Nothing to be done for all'. make[1]: Leaving directory/data/kpatch/kpatch-0.1.5/kpatch'
make -C kmod
make[1]: Entering directory /data/kpatch/kpatch-0.1.5/kmod' make -C core clean make[2]: Entering directory/data/kpatch/kpatch-0.1.5/kmod/core'
rm -f -Rf ..o.cmd ..ko.cmd .tmp_versions _.o *.ko *.mod.c
Module.symvers
make[2]: Leaving directory /data/kpatch/kpatch-0.1.5/kmod/core' make -C core make[2]: Entering directory/data/kpatch/kpatch-0.1.5/kmod/core'
make -C /lib/modules/3.8.13-35.1.1.el6uek.x86_64/build M=/data/kpatch/kpatch-0.1.5/kmod/core kpatch.ko
make[3]: Entering directory /usr/src/kernels/3.8.13-35.1.1.el6uek.x86_64' CC [M] /data/kpatch/kpatch-0.1.5/kmod/core/core.o /data/kpatch/kpatch-0.1.5/kmod/core/core.c:158:58: error: macro "hash_for_each_possible_rcu" requires 5 arguments, but only 4 given /data/kpatch/kpatch-0.1.5/kmod/core/core.c: In function ‘kpatch_get_func’: /data/kpatch/kpatch-0.1.5/kmod/core/core.c:158: error: ‘hash_for_each_possible_rcu’ undeclared (first use in this function) /data/kpatch/kpatch-0.1.5/kmod/core/core.c:158: error: (Each undeclared identifier is reported only once /data/kpatch/kpatch-0.1.5/kmod/core/core.c:158: error: for each function it appears in.) /data/kpatch/kpatch-0.1.5/kmod/core/core.c:159: error: expected ‘;’ before ‘if’ /data/kpatch/kpatch-0.1.5/kmod/core/core.c:155: warning: unused variable ‘f’ /data/kpatch/kpatch-0.1.5/kmod/core/core.c:167:43: error: macro "hlist_for_each_entry_continue_rcu" requires 3 arguments, but only 2 given /data/kpatch/kpatch-0.1.5/kmod/core/core.c: In function ‘kpatch_get_prev_func’: /data/kpatch/kpatch-0.1.5/kmod/core/core.c:167: error: ‘hlist_for_each_entry_continue_rcu’ undeclared (first use in this function) /data/kpatch/kpatch-0.1.5/kmod/core/core.c:168: error: expected ‘;’ before ‘if’ /data/kpatch/kpatch-0.1.5/kmod/core/core.c:227:51: error: macro "hash_for_each_rcu" requires 5 arguments, but only 4 given /data/kpatch/kpatch-0.1.5/kmod/core/core.c: In function ‘kpatch_backtrace_address_verify’: /data/kpatch/kpatch-0.1.5/kmod/core/core.c:227: error: ‘hash_for_each_rcu’ undeclared (first use in this function) /data/kpatch/kpatch-0.1.5/kmod/core/core.c:227: error: expected ‘;’ before ‘{’ token /data/kpatch/kpatch-0.1.5/kmod/core/core.c:196: warning: unused variable ‘i’ /data/kpatch/kpatch-0.1.5/kmod/core/core.c:769:52: error: macro "hash_for_each_rcu" requires 5 arguments, but only 4 given /data/kpatch/kpatch-0.1.5/kmod/core/core.c: In function ‘kpatch_register’: /data/kpatch/kpatch-0.1.5/kmod/core/core.c:769: error: ‘hash_for_each_rcu’ undeclared (first use in this function) /data/kpatch/kpatch-0.1.5/kmod/core/core.c:770: error: expected ‘;’ before ‘func’ /data/kpatch/kpatch-0.1.5/kmod/core/core.c:791:52: error: macro "hash_for_each_rcu" requires 5 arguments, but only 4 given /data/kpatch/kpatch-0.1.5/kmod/core/core.c:791: error: expected ‘;’ before ‘{’ token /data/kpatch/kpatch-0.1.5/kmod/core/core.c:789: warning: unused variable ‘safe’ /data/kpatch/kpatch-0.1.5/kmod/core/core.c:789: warning: unused variable ‘kpmod2’ /data/kpatch/kpatch-0.1.5/kmod/core/core.c:835: error: ‘LOCKDEP_STILL_OK’ undeclared (first use in this function) /data/kpatch/kpatch-0.1.5/kmod/core/core.c:835: error: too many arguments to function ‘add_taint’ /data/kpatch/kpatch-0.1.5/kmod/core/core.c:846:52: error: macro "hash_for_each_rcu" requires 5 arguments, but only 4 given /data/kpatch/kpatch-0.1.5/kmod/core/core.c:847: error: expected ‘;’ before ‘func’ /data/kpatch/kpatch-0.1.5/kmod/core/core.c:733: warning: unused variable ‘i’ make[4]: *_\* [/data/kpatch/kpatch-0.1.5/kmod/core/core.o] Error 1 make[3]: **\* [kpatch.ko] Error 2 make[3]: Leaving directory /usr/src/kernels/3.8.13-35.1.1.el6uek.x86_64'
make[2]: *** [kpatch.ko] Error 2
make[2]: Leaving directory/data/kpatch/kpatch-0.1.5/kmod/core' make[1]: **\* [all] Error 2 make[1]: Leaving directory /data/kpatch/kpatch-0.1.5/kmod'
make: *** [build-kmod] Error 2

changed section .rela.smp_locks not selected for inclusion

Hi, I meet following failure when build a upstream patch:

  CALL    /root/.kpatch/3.10.0-115.el7.x86_64/src/scripts/checksyscalls.sh
  CC      scripts/mod/devicetable-offsets.s
  GEN     scripts/mod/devicetable-offsets.h
  HOSTCC  scripts/mod/file2alias.o
  HOSTLD  scripts/mod/modpost
  CC      mm/mlock.o
changed function: mlock_vma_page
kpatch_verify_patchability:730: changed section .rela.smp_locks not selected for inclusion
/usr/local/libexec/kpatch/create-diff-object: unreconcilable difference

My patch is:

diff --git a/mm/mlock.c b/mm/mlock.c
index 79b7cf7..5a9cbcc 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -76,8 +76,6 @@ void clear_page_mlock(struct page *page)
  */
 void mlock_vma_page(struct page *page)
 {
-       BUG_ON(!PageLocked(page));
-
        if (!TestSetPageMlocked(page)) {
                mod_zone_page_state(page_zone(page), NR_MLOCK,
                                    hpage_nr_pages(page));

link-vmlinux-syms doesn't handle symbol collisions properly

Same problem as #53 only with link-vmlinux-syms when pinning addresses into the symbol table of the kernel module.

failed to build under 3.12.18

make[1]: Leaving directory '/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kpatch-build'
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:295:41: warning: 'struct ftrace_ops' declared inside parameter list [enabled by default]
         struct ftrace_ops *fops, struct pt_regs *regs)
                                         ^
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:295:41: warning: its scope is only this definition or declaration, which is probably not what you want [enabled by default]
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:339:15: error: variable 'kpatch_ftrace_ops' has initializer but incomplete type
 static struct ftrace_ops kpatch_ftrace_ops __read_mostly = {
               ^
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:340:2: error: unknown field 'func' specified in initializer
  .func = kpatch_ftrace_handler,
  ^
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:340:2: warning: excess elements in struct initializer [enabled by default]
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:340:2: warning: (near initialization for 'kpatch_ftrace_ops') [enabled by default]
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:341:2: error: unknown field 'flags' specified in initializer
  .flags = FTRACE_OPS_FL_SAVE_REGS,
  ^
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:341:11: error: 'FTRACE_OPS_FL_SAVE_REGS' undeclared here (not in a function)
  .flags = FTRACE_OPS_FL_SAVE_REGS,
           ^
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:341:2: warning: excess elements in struct initializer [enabled by default]
  .flags = FTRACE_OPS_FL_SAVE_REGS,
  ^
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:341:2: warning: (near initialization for 'kpatch_ftrace_ops') [enabled by default]
/var/tmp/paludis/build/sys-kernel-kpatch-scm/work/kpatch-scm/kmod/core/core.c:339:26: warning: 'kpatch_ftrace_ops' defined but not used [-Wunused-variable]
 static struct ftrace_ops kpatch_ftrace_ops __read_mostly = {

source based distributive support

Can somebody explain to me how can i add my own linux distro support (exherbo).

Unified patch management interface required

Current simple module load/unload interface has some problems.

There is no way to describe/manage the dependency among patches. Wish to have a way to describe patch dependency (e.g. patch B requires the kernel with patch A). The kernel version dependency can be solved with module versioning.
To avoid a kernel panic when failing to unpatch, patching/unpatching should be done via sysfs or ioctl which can return errors. module unloading is very critical one. We can lock the patch modules with increment refcount from kpatch module until the unpatching was done successfully.
We have no way to check or list-up what functions are already patched, and which patch module has the patched function.

kpatch-build failed (fc19)

$ kpatch-build foo.patch
Using cache at /home/lilu/.kpatch/3.13.6-100.fc19.x86_64/src
Testing patch file
patching file fs/proc/meminfo.c
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
changed function: meminfo_proc_show
Building patch module: kpatch-foo.ko
ERROR: kpatch build failed. Check /tmp/kpatch-build-1395733172.log for more details.

The last few lines from log file(110KB in total):
https://github.com/lulinqing/misc/blob/master/kpatch-build-1395733172.log

... ...
found patched function meminfo_proc_show
original function at address ffffffff81227a90 (length 1312)
patches_nr = 1
patches_size = 24
relas_size = 24
patches_offset = 71
relas_offset = 80
make -C /home/lilu/.kpatch/3.13.6-100.fc19.x86_64/src M=/tmp/tmp.wAw8XBHOYT/patch kpatch-foo.ko
make[1]: Entering directory `/home/lilu/.kpatch/3.13.6-100.fc19.x86_64/src'
  CC [M]  /tmp/tmp.wAw8XBHOYT/patch/kpatch-patch-hook.o
  LD [M]  /tmp/tmp.wAw8XBHOYT/patch/kpatch-foo.o
/tmp/tmp.wAw8XBHOYT/patch/output.o: file not recognized: File format not recognized
make[3]: *** [/tmp/tmp.wAw8XBHOYT/patch/kpatch-foo.o] Error 1
make[2]: *** [kpatch-foo.ko] Error 2
make[1]: *** [sub-make] Error 2
make[1]: Leaving directory `/home/lilu/.kpatch/3.13.6-100.fc19.x86_64/src'
make: *** [kpatch-foo.ko] Error 2

Test patch I used:

Index: src/fs/proc/meminfo.c
===================================================================
--- src.orig/fs/proc/meminfo.c
+++ src/fs/proc/meminfo.c
@@ -95,7 +95,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
        "Committed_AS:   %8lu kB\n"
        "VmallocTotal:   %8lu kB\n"
        "VmallocUsed:    %8lu kB\n"
-       "VmallocChunk:   %8lu kB\n"
+       "VMALLOCCHUNK:   %8lu kB\n"
 #ifdef CONFIG_MEMORY_FAILURE
        "HardwareCorrupted: %5lu kB\n"
 #endif

Thanks!

Get rid of the stop_machine from kpatch

Since the stop_machine() stops all CPUs at a while, it is not good for some mission critical systems which are the main target of live patching. Thus, we need to apply patch without stop_machine(). The main reason why kpatch requires that is to check the activity of the target functions, in other words, to ensure the target function is NOT in use when patching.

To solve this issue without stop_machine(), I'd like to propose new approach, which directly checks the target functions are in use by using ftrace and kretprobes. Of course, we still need a stack dump checking too.

The basic idea is simple: increment counter at the entry of the function and decrement it at the return of the function. After checking stacks of all processes, if we see the counter is 0, that is safe and go live patching. If not, no go. Moreover, we can also add a latency period after checking the stacks. While the period, if the counter decremented to 0, it can go patching.

To implement the trick, we use atomic_inc_not_zero() and atomic_dec_not_zero(). So the reference value of the counter will be 1.
Here is the patching-side concept code. I'm sure it is very simple.

atomic_t counter;
ftrace_handler() {
  func = kpatch_get_func();
  if (func->updating) {
    ret = atomic_inc_not_zero(&counter);
    if (ret != 0)
       func = kpatch_get_committed_func(); /* we're under checking, use only committed one*/
    else {
       /* we've already checked and it is safe to apply patch */
    }
  }
  if (func)
    regs->ip = func->ip;
}

kretprobe_handler() {
  ret = atomic_dec_not_zero(&counter);
  if (ret == 1 and we already checked all stacks) {
    atomic_cmpxhg(&counter, 1, 0); /* Try to fix the state 1 -> 0 */
  }
}

Note that actually using kretprobes on NMI is not safe now in upstream kernel, so I need to fix that. And also, kretprobes can fail to hooking function return. Anyway, in that case the counter just never reach to 1. Thus we need to set a timer to give it up.

Additional discussion point of this method is that we'd better support incremental binary patch, because it becomes hard to find the point that all target functions are not in use if the number of target function is increased.

Thank you,

kprobes incompatibility

When kprobes registers with ftrace before kpatch does for the same function, kprobes "wins", until the kprobe has been unregistered with ftrace. The kernel kprobe_ftrace_handler function changes regs->ip, which overwrites the value which kpatch had written.

Option 1:

diff --git a/arch/x86/kernel/kprobes/ftrace.c b/arch/x86/kernel/kprobes/ftrace.c
index 23ef5c5..5969176 100644
--- a/arch/x86/kernel/kprobes/ftrace.c
+++ b/arch/x86/kernel/kprobes/ftrace.c
@@ -57,6 +57,7 @@ void __kprobes kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
    struct kprobe *p;
    struct kprobe_ctlblk *kcb;
    unsigned long flags;
+   unsigned long save_ip = regs->ip;

    /* Disable irq for emulating a breakpoint and avoiding preempt */
    local_irq_save(flags);
@@ -80,6 +81,8 @@ void __kprobes kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
         * If pre_handler returns !0, it sets regs->ip and
         * resets current kprobe.
         */
+       if (ip != save_ip)
+           regs->ip = save_ip;
    }
 end:
    local_irq_restore(flags);

With this patch, if both kpatch and a kprobe handler try to modify the IP, kpatch always wins.

Option 2:

Create an FTRACE_OPS_FL_LAST flag which ensures that kpatch is called last, so that any kprobes attempts to update regs->ip will be ignored.

This is another variation on option 1: if both kpatch and a kprobe handler try to modify the IP, kpatch always wins.

Option 3:

Create an FTRACE_OPS_FL_FIRST flag which ensures that kpatch is called first. Then pass the updated regs->ip to the kprobes functions so they can set their breakpoint at the beginning of the replacement function. The downside here is that kprobes modules can override kpatch's redirecting of functions, but that may not be much of an issue.

So if both kpatch and a kprobe handler try to modify the IP, kprobes wins.

Option 4:

Create an FTRACE_OPS_FL_IPMODIFY flag: first to register wins.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.