SUSE Linux Enterprise Server 16.1

Kdump Configuration and Troubleshooting Guide

Publication Date: 02 Jul 2026

WHAT?

Kdump is a Linux kernel crash dumping mechanism that captures a snapshot of system memory (a vmcore file) when the Linux kernel encounters a fatal error (kernel panic).

WHY?

Use Kdump to collect diagnostic data for kernel crash analysis and root-cause investigation. Correctly setting up Kdump and obtaining the memory dump may help SUSE support or kernel developers to debug a potential kernel crash.

EFFORT

The average reading time of this article is approximately 40 minutes.

REQUIREMENTS

Linux fundamentals: Understanding basic Linux commands, file permissions, directory structures and use of the command line.

Revision History: Kdump Configuration and Troubleshooting Guide

1 What is Kdump and how does it work? #

Kdump is a kernel crash dumping mechanism that captures the system’s memory state into a vmcore file when a system crash occurs. A vmcore file is a snapshot of your computer's system memory (RAM) taken at the exact moment the Linux kernel crashed.

1.1 Why is Kdump critical for system crash debugging? #

The primary importance of Kdump lies in its ability to capture a snapshot of a system's memory at the exact moment of a critical failure.

When a Linux kernel experiences a fatal error, syslog or journald usually fail along with it, often leaving no record of what went wrong. Kdump bypasses this limitation by using Kexec to boot a secondary capture kernel in a reserved slice of RAM. The unstable crashed system is replaced with a freshly started and stable capture kernel.

Without this tool, administrators are often left with nothing but a blank screen or a frozen console, making it nearly impossible to diagnose the root cause of intermittent or silent system crashes.

1.2 Understanding the dual-kernel model #

Kdump uses a second isolated kernel referred to as the capture or crash kernel to handle a system crash safely. When the main system kernel fails, you cannot trust it to write its own crash logs to disk— because the kernel memory might be corrupted and the kernel itself is no longer reliable. The dual-kernel approach solves this by jumping into a completely different environment.

The model relies on two distinct kernels residing in memory simultaneously:

The production (primary) kernel is the kernel you use every day. It runs your applications and services.
The capture (crash) kernel is a second copy of the kernel loaded in a small reserved area of RAM. It only starts when the primary kernel panics. Alongside the crash kernel, a special-purpose initramfs image is loaded to the reserved RAM. It is built by the Kdump tool and includes all the drivers, settings, and programs to store the vmcore file.

1.3 What is Kexec and how does it boot the crash kernel? #

Kexec is a system call that functions as a software-defined boot loader, allowing a running kernel to bypass the hardware BIOS/UEFI stage and directly hand over control to a new kernel. By loading the secondary kernel's image and parameters into memory while the system is still active, Kexec performs a warm boot that preserves the state of RAM and significantly reduces downtime. This mechanism is the backbone of the Kdump dual-kernel model, as it provides a reliable way to jump from a crashing production environment into a clean recovery environment for data capture.

The most important component of Kexec is the kexec command. You can load a kernel with Kexec in two ways:

Load the kernel to the address space of a production kernel for a regular reboot:
```
> sudo  kexec -l KERNEL_IMAGE
```
You can later boot to this kernel with the command kexec -e. Instead of using Kexec, you can directly use a program called kexec-bootloader. It finds the default kernel, initrd and command-line options from the boot loader configuration and passes everything to Kexec to load the default kernel properly.
Load the kernel to a reserved area of memory:
```
> sudo  kexec -p KERNEL_IMAGE
```
This kernel is booted automatically when the system crashes.This is what Kdump uses to load the capture kernel.

1.4 What data is captured in a Linux vmcore file? #

A vmcore file is a snapshot of your system's physical memory (RAM) taken at the exact moment the Linux kernel crashed. When kdump is set up on a system, the kdump service loads the capture kernel and initramfs on system boot, using the kexec program, into a pre-reserved area of RAM— the crash kernel area. If at some point the system crashes, the capture kernel is started. Its memory is restricted to the pre-reserved crash kernel area, so none of the memory used by the crashed production kernel is overwritten. The job of the capture kernel and initramfs is to save the contents of the production kernel's memory into a vmcore file.

The vmcore file is a snapshot of the RAM and includes:

The Kernel state: All active kernel data structures, global variables and the call stack, showing what the CPU was executing at the time of the crash.
Process information: A list of every process that was running, including their individual stacks and registers.
Memory pages: Depending on your settings, it can contain the actual data held in RAM by applications.
VMCOREINFO: special section that tells analysis tools how the kernel's memory was laid out so they can make sense of the raw data.

2 Installing and configuring Kdump #

This section describes how to install and configure Kdump.

To install Kdump on the immutable system, run the following command and reboot:

> sudo  transactional-update pkg install kdump

To install Kdump on the mutable system, run the following command:

> sudo  zypper install kdump

This command downloads the following packages:

kdump
kexec-tools
makedumpfile

To store the vmcore file remotely, you can install any of the following recommended packages:

Install the cifs-utils package to store on Samba stores.
Install the lftp package to store on FTP / SFTP servers.
Install the nfs-client package to store on NFS servers.
Install the openssh-clients package to store over SSH.

Important: Managing Kdump and GRUB configuration conflicts

The kdump-commandline service automatically manages your crash memory allocation. It will overwrite any manual crashkernel= entries in GRUB_CMDLINE_LINUX_DEFAULT inside /etc/default/grub unless the automation is explicitly disabled. To prevent this behavior, modify the KDUMP_UPDATE_BOOTLOADER variable to false.

How do you stop Kdump from overwriting GRUB configurations?

To keep your manual crashkernel= settings from being erased, update your configuration with these rules:

Set KDUMP_UPDATE_BOOTLOADER="false" to disable automatic updates.
Keep all crashkernel= parameters out of the GRUB_CMDLINE_LINUX line.
Manual configurations belong exclusively in GRUB_CMDLINE_LINUX_DEFAULT.

Why should you avoid adding crash kernel settings to GRUB_CMDLINE_LINUX?

Adding crashkernel= to GRUB_CMDLINE_LINUX causes system conflicts.

Settings in GRUB_CMDLINE_LINUX apply globally to all boot targets.
This includes the rescue system, which does not run Kdump.
The kdump-commandline service never alters this specific line.
Leaving values here creates direct conflicts with automated service settings.

2.1 Configuring Kdump for non-immutable systems #

To use Kdump, a specific amount of system memory (RAM)—often referred to as crash kernel memory- must be permanently reserved at boot time for the capture kernel and initramfs. The amount of memory to reserve is specified using the crashkernel= kernel command-line parameter. Kdump includes kdumptool calibrate and kdumptool commandline to calculate exactly how much memory needs to be reserved.

When the systemctl enable --now kdump command triggers the main kdump.service, it automatically calls a helper service named kdump-commandline.service before attempting to load the kernel. This helper service verifies if the current kernel command line contains the expected crashkernel= options. The options are:

Fixed memory allocation (e.g. ,256M) reserves a strict hardcoded amount of RAM.
Range-based allocation (e.g. 2G-8G:192M ) dynamically scales the reserved memory based on the total RAM installed in the system.
Automatic allocation ( auto) where the kernel automatically calculates the required memory based on system architecture.
High/low split (e.g. 512M,high crashkernel=128M,low) allocates memory across different addressable zones specifically to accommodate hardware drivers on large enterprise systems.

If the options are missing, the helper service automatically modifies the boot loader configuration to contain the crash kernel options to be applied on the next boot.

Before attempting to load, the kdump.service:

Starts the kdump-commandline.service
Updates the boot loader
Builds the capture initramfs

To verify that the service successfully checked and updated your kernel boot options, inspect the system logs:

> sudo  journalctl -u kdump-commandline.service

Expected output:

  Started Check and update kdump options on the kernel command line.
  Kdump expects these crashkernel= values on the kernel command line:
  crashkernel=72M,low crashkernel=443M,high
  (based on the output of 'kdumptool calibrate')
  No crashkernel= found in /proc/cmdline. Kdump can't be started.
  Updated the command line options in the bootloader.
  Changes will take effect on next boot.

The boot loader has been updated. Reboot the machine to apply the changes and start Kdump.

Other configurations:

The /etc/sysconfig/kdump file is the primary configuration file for managing the Kdump service. It includes essential options such as:

KDUMP_CRASHKERNEL for defining the reserved crash memory size
KDUMP_VERBOSE for increasing debugging log levels
KDUMP_SAVEDIR for specifying the core dump (vmcore) storage location

Example 1: Contents of a /etc/sysconfig/kdump file #

cat /etc/sysconfig/kdump
  ## Path:        System/Kernel/Kdump
  ## Description: Crash Dump Configuration

  ## Type:        string
  ## Default:     "auto"
  ## ServiceRestart:      kdump
  #
  # KDUMP_CRASHKERNEL is used by "kdumptool commandline" to determine
  # the value of the kernel "crashkernel=" command-line options.
  #
  # Possible values are "auto" or one or more "crashkernel=..." values.
  #
  # KDUMP_CRASHKERNEL="auto" makes "kdumptool commandline" use the default
  # values as proposed by "kdumptool calibrate".
  #
  # If the default values are not adequate you may provide a manual setting,
  # e.g., KDUMP_CRASHKERNEL="crashkernel=72M,low crashkernel=300M,high"
  #
  # See also: kdump(5).
  #
  KDUMP_CRASHKERNEL="auto"


  ## Type:        boolean
  ## Default:     "true"
  ## ServiceRestart:      kdump
  #
  # When KDUMP_UPDATE_BOOTLOADER is set to "true" "kdumptool commandline -c -U"
  # called from the kdump-commandline.service  will check if the expected kernel
  # command-line options are present and update the bootloader using *pbl*(8) if
  # not.
  #
  # Also the kdump options are removed from the command line when kdump is
  # uninstalled.
  #
  # When set to false "kdumptool commandline -c -U" will only check the options
  # and report possible warnings but bootloader will not be updated.
  #
  KDUMP_UPDATE_BOOTLOADER="true"

  ## Type:        string
  ## Default:     ""
  ## ServiceRestart:      kdump
  #
  # Kernel Version string for the -kdump kernel. Use an empty string to
  # auto-detect a suitable kernel.
  #
  # See also: kdump(5).
  #
  KDUMP_KERNELVER=""

  ## Type:        integer
  ## Default:     32
  ## ServiceRestart:      kdump
  #
  # Number of CPUs to be used in the kdump environment. You may want to
  # increase the number if computing power is the bottleneck in your setup.
  #
  # If the value is zero, use all available CPUs.
  #
  # Decreasing the number of CPUs lowers the memory required by kdump.
  # Increasing it raises memory usage and may cause failures if crashkernel
  # memory is insufficient.
  #
  # See also: kdump(5).
  #
  KDUMP_CPUS=32

  ## Type:        string
  ## Default:     ""
  ## ServiceRestart:      kdump
  #
  # The kdump commandline is the command line that needs to be passed off to
  # the kdump kernel. If a command line is not specified, the default will be
  # taken from /proc/cmdline and adjusted.
  #
  # WARNING: There are a few options that always should be passed to the kdump
  # kernel. See kdump(5) for details. Don't use that variable, consider using
  # KDUMP_COMMANDLINE_APPEND instead.
  #
  KDUMP_COMMANDLINE=""

  ## Type:        string
  ## Default:     ""
  ## ServiceRestart:     kdump
  #
  # Set this variable if you only want to _append_ values to the default
  # command line string. The string also gets appended if KDUMP_COMMANDLINE
  # is set.
  #
  # For network based dumps, you may have to add a "net_delay" parameter
  # here. Consult the man page for details.
[...]

For more information, refer to man kdump(5) and man kdump(7).

Any modifications made to this configuration file require a manual restart of the Kdump service to take effect. However, if any changes are made specifically to the KDUMP_CRASHKERNEL memory allocation size, a full system reboot is required because these settings alter the system's core boot parameters.

2.2 Configuring Kdump for immutable systems #

On an immutable file system, the standard kdump-commandline.service is unable to directly modify the boot loader configuration. To configure Kdump in these specialized environments, use the transactional-update setup-kdump command instead. Executing this command handles the entire configuration process by automatically installing Kdump, disabling KDUMP_UPDATE_BOOTLOADER, and running the necessary calibration and command-line tools to update the boot loader within a new system snapshot. A reboot is required to boot into the new snapshot and activate the updated Kdump configuration.

If the automatically calculated memory allocation estimated by the calibration tool proves to be insufficient for your system requirements, you can manually override the settings. To specify custom memory sizes directly on an immutable system, you can append the crashkernel parameter to the command by running:

> sudo  transactional-update setup-kdump [--crashkernel=low,high]

Other configurations:

Modifying the /etc/sysconfig/kdump configuration file on an immutable system requires the execution of the transactional-update command and reboot for the changes to take effect. However, if your modifications alter the crashkernel memory allocation size, you must perform an additional step to ensure the changes are safely written to the read-only environment; you need to execute the transactional-update run kdumptool commandline -u command. This applies the new memory settings directly to the boot loader configuration so they can take physical effect during the next system boot.

The /etc/sysconfig/kdump file is the primary configuration file for managing the Kdump service. It includes essential options such as:

KDUMP_CRASHKERNEL for defining reserved crash memory size
KDUMP_VERBOSE for increasing debugging log levels
KDUMP_SAVEDIR for specifying the core dump (vmcore) storage location

Example 2: Contents of a /etc/sysconfig/kdump file #

cat /etc/sysconfig/kdump
      ## Path:        System/Kernel/Kdump
      ## Description: Crash Dump Configuration

      ## Type:        string
      ## Default:     "auto"
      ## ServiceRestart:      kdump
      #
      # KDUMP_CRASHKERNEL is used by "kdumptool commandline" to determine
      # the value of the kernel "crashkernel=" command-line options.
      #
      # Possible values are "auto" or one or more "crashkernel=..." values.
      #
      # KDUMP_CRASHKERNEL="auto" makes "kdumptool commandline" use the default
      # values as proposed by "kdumptool calibrate".
      #
      # If the default values are not adequate you may provide a manual setting,
      # e.g., KDUMP_CRASHKERNEL="crashkernel=72M,low crashkernel=300M,high"
      #
      # See also: kdump(5).
      #
      KDUMP_CRASHKERNEL="auto"


      ## Type:        boolean
      ## Default:     "true"
      ## ServiceRestart:      kdump
      #
      # When KDUMP_UPDATE_BOOTLOADER is set to "true" "kdumptool commandline -c -U"
      # called from the kdump-commandline.service  will check if the expected kernel
      # command-line options are present and update the bootloader using *pbl*(8) if
      # not.
      #
      # Also the kdump options are removed from the command line when kdump is
      # uninstalled.
      #
      # When set to false "kdumptool commandline -c -U" will only check the options
      # and report possible warnings but bootloader will not be updated.
      #
      KDUMP_UPDATE_BOOTLOADER="true"

      ## Type:        string
      ## Default:     ""
      ## ServiceRestart:      kdump
      #
      # Kernel Version string for the -kdump kernel. Use an empty string to
      # auto-detect a suitable kernel.
      #
      # See also: kdump(5).
      #
      KDUMP_KERNELVER=""

      ## Type:        integer
      ## Default:     32
      ## ServiceRestart:      kdump
      #
      # Number of CPUs to be used in the kdump environment. You may want to
      # increase the number if computing power is the bottleneck in your setup.
      #
      # If the value is zero, use all available CPUs.
      #
      # Decreasing the number of CPUs lowers the memory required by kdump.
      # Increasing it raises memory usage and may cause failures if crashkernel
      # memory is insufficient.
      #
      # See also: kdump(5).
      #
      KDUMP_CPUS=32

      ## Type:        string
      ## Default:     ""
      ## ServiceRestart:      kdump
      #
      # The kdump commandline is the command line that needs to be passed off to
      # the kdump kernel. If a command line is not specified, the default will be
      # taken from /proc/cmdline and adjusted.
      #
      # WARNING: There are a few options that always should be passed to the kdump
      # kernel. See kdump(5) for details. Don't use that variable, consider using
      # KDUMP_COMMANDLINE_APPEND instead.
      #
      KDUMP_COMMANDLINE=""

      ## Type:        string
      ## Default:     ""
      ## ServiceRestart:     kdump
      #
      # Set this variable if you only want to _append_ values to the default
      # command line string. The string also gets appended if KDUMP_COMMANDLINE
      # is set.
      #
      # For network based dumps, you may have to add a "net_delay" parameter
      # here. Consult the man page for details.
    [...]

2.3 Testing Kdump #

It is advisable to test Kdump after configuring it by simulating a kernel crash. Otherwise, you may only find out that it does not work when an actual kernel crash occurs, leaving you with no possibility to debug the crash. Ensure no critical workloads are running and no unsaved data is present on the system. Additionally, ensure to sync and unmount file systems:

Force an immediate emergency sync of all cached memory data to your hard drives to prevent data loss:

echo s > /proc/sysrq-trigger

Force an immediate emergency remount of all file systems to read-only mode to prevent data corruption:

echo u > /proc/sysrq-trigger

Then you can simulate a kernel crash:

echo c > /proc/sysrq-trigger

Check if there is a new directory created under your KDUMP_SAVEDIR, which is /var/crash by default. This contains the dmesg and vmcore of the crashed kernel.

3 How to troubleshoot common Kdump issues #

Troubleshooting Kdump is a critical process to ensure that your system can successfully capture a vmcore file during a kernel crash, which is often the only way to diagnose system crashes. When troubleshooting Kdump, the process usually fails at one of three stages: during boot (memory reservation), during the crash when the dump does not start, or during the save process when the file is not written.

3.1 Crash kernel memory reserved is insufficient #

One of the most common reasons Kdump fails is that the amount of crash kernel memory reserved is insufficient. Different system configurations may require more memory than estimated by kdumptool calibrate and set up automatically in the boot loader config by the kdump-commandline.service.

During Kdump, if you see error messages mentioning low memory and invoking the Out of Memory (OOM) killer, this is the likely cause. If you don't see such messages, trying with increased crash kernel reservation is a good first step.

Find the size of the current automatically configured crash kernel reservation by checking the kernel command line:
```
> sudo  cat /proc/cmdline
```
- crashkernel=X M
- crashkernel=Y M,low
- crashkernel=Z M,high
Add up the values of X, Y, and Z to determine the total size of the current reservation in MiB (current).
Manually set the new reservation by editing the /etc/sysconfig/kdump file. Locate and change the value of the KDUMP_CRASHKERNEL variable to double the current size:
```
KDUMP_CRASHKERNEL="crashkernel=<2 * current>M"
```
Reboot the system to apply the new memory parameters:
```
> sudo  reboot
```
Repeat this process a few times until Kdump successfully executes without errors.
Note
Fine-tuning crash kernel memory involves finding the smallest RAM reservation that successfully captures a crash dump without triggering OOM errors in the capture kernel. Fine-tune your final setting using the range between your last working and non-working values to avoid wasting system memory.

4 For more information #

For information on Kdump, refer to the following resources:

Man pages:
- man 7 kdump
- man 5 kdump
Official Linux kernel documentation: https://docs.kernel.org/admin-guide/kdump/kdump.html
Man page for Kexec: https://man7.org/linux/man-pages/man8/kexec.8.html

5 Legal Notice #

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”.

For SUSE trademarks, see https://www.suse.com/company/legal/. All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors, nor the translators shall be held liable for possible errors or the consequences thereof.