Skip to main content

· 4 min read
Pete Johanson

Today, we merged a significant change to the low level sensor code that is used to support encoders. In particular, this paves the way for completing the work on supporting split peripheral sensors/encoders, and other future sensors like pointing devices.

As part of the work, backwards compatibility for existing shields has been retained, but only for a grace period to allow out-of-tree shields to move to the new approach for encoders.

Special thanks to joelspadin for the thorough code review and testing throughout the development of the refactor.

Summary of Changes

The following items have been merged:

  1. Split configuration of hardware details, and behavior configuration to allow more flexible functionality of sensors/encoders, in particular linear encoders that lack detents/"clicks" as they rotate.
  2. Support for upstream Zephyr sensor drivers, including the NRFX QDEC driver that can be used on nRF52 based keyboards.
  3. Sensor data handling changes that pave the way for split sensor handling easily.

Configuration Changes

The major changes to configuration in the devicetree files relates to how the number of steps/triggers for a given encoder are set. In particular, the number of pulses/steps for a given encoder is configured first, allowing ZMK to determine the exact angular degrees of change that is represented by a single pulse on the data lines to that encoder.

Once that angular degrees mapping is completed, now independently there is a configuration setting to control how many triggers of the behavior in the keymap should occur for each full rotation of the sensor. Another way to think of this is "how many degrees of rotation results in a triggering of the sensor behavior in your keymap layer".

Splitting these two parts of the encoder configuration allows greater flexibility, and fine grained control of encoder behavior for linear encoders that don't have fixed detents.

Old Configuration

Previously, an encoder configuration looked like:

    left_encoder: encoder_left {
compatible = "alps,ec11";
label = "LEFT_ENCODER";
a-gpios = <&pro_micro 21 (GPIO_ACTIVE_HIGH | GPIO_PULL_UP)>;
b-gpios = <&pro_micro 20 (GPIO_ACTIVE_HIGH | GPIO_PULL_UP)>;
resolution = <4>;
};

Here, the resolution property was used to indicate how many encoder pulses should trigger the sensor behavior one time. Next, the encoder is selected in the sensors node:

    sensors {
compatible = "zmk,keymap-sensors";
sensors = <&left_encoder &right_encoder>;
};

That was the entirety of the configuration for encoders.

New Configuration

    left_encoder: encoder_left {
compatible = "alps,ec11";
label = "LEFT_ENCODER";
a-gpios = <&pro_micro 21 (GPIO_ACTIVE_HIGH | GPIO_PULL_UP)>;
b-gpios = <&pro_micro 20 (GPIO_ACTIVE_HIGH | GPIO_PULL_UP)>;
steps = <80>;
};

Here, the steps property is now used to indicate how many encoder pulses there are in a single complete rotation of the encoder. Next, the encoder is selected in the sensors node as before, but an additional configuration is used to indicate how many times the encoder should trigger the behavior in your keymap per rotation:

    sensors {
compatible = "zmk,keymap-sensors";
sensors = <&left_encoder &right_encoder>;
triggers-per-rotation = <20>;
};

For tactile encoders that have detents, the triggers-per-rotation would match the number of detents on the encoder. For linear encoders, the value can be chosen to suit your needs.

Zephyr Sensor Drivers

The configuration changes bring ZMK's code in line with how upstream Zephyr sensor drivers handle rotations. This has the added advantage of allowing us to leverage other sensor drivers. On Nordic MCUs, like nRF52840, the NRFX QDEC driver can be used, for example:

&pinctrl {
qdec_default: qdec_default {
group1 {
psels = <NRF_PSEL(QDEC_A, 1, 11)>,
<NRF_PSEL(QDEC_B, 1, 10)>;
bias-pull-up;
};
};
};

// Set up the QDEC hardware based driver and give it the same label as the deleted node.
encoder: &qdec0 {
status = "okay";
led-pre = <0>;
steps = <80>;
pinctrl-0 = <&qdec_default>;
pinctrl-names = "default";
};

The NRFX QDEC driver has the advantage of supporting optical encoders as well, and although it polls, it does so in hardware without waking the MCU core; initial basic power profiling is promising.

Split Sensor/Encoder Support

In addition to the refactors for splitting the configuration, the changes merged included refactors designed to simplify and move forward with the long outstanding feature of supporting encoders on the peripheral side of split keyboards. That work is planned as a follow up.

Deprecation

The old configuration will be supported for a period of one month, and then removed, giving users a grace period to complete the migration to the new separated configuration.

· 9 min read
Pete Johanson

I'm happy to announce that we have completed the work to upgrade ZMK to Zephyr 3.2!

petejohanson did the upgrade work to adjust ZMK for the Zephyr changes, with help from Nicell on the LVGL pieces.

  • Upgrade to LVGL 8.x API, and move to the new Kconfig settings.
  • Tons of RP2040 work.
  • Zephyr core API changes, including DTS label use changes.
  • Move to pinctrl Zephyr subsystem.

Getting The Changes

Use the following steps to update to the latest tooling in order to properly use the new ZMK changes:

User Config Repositories Using GitHub Actions

Existing user config repositories using Github Actions to build will pull down Zephyr 3.2 automatically, however if you created your user config a while ago, you may need to update it to reference our shared build configuration to leverage the correct Docker image.

  1. Replace the contents of your .github/workflows/build.yml with:

    on: [push, pull_request, workflow_dispatch]

    jobs:
    build:
    uses: zmkfirmware/zmk/.github/workflows/build-user-config.yml@main
  2. If it doesn't exist already, add a new file to your repository named build.yaml:

    # This file generates the GitHub Actions matrix
    # For simple board + shield combinations, add them
    # to the top level board and shield arrays, for more
    # control, add individual board + shield combinations to
    # the `include` property, e.g:
    #
    # board: [ "nice_nano_v2" ]
    # shield: [ "corne_left", "corne_right" ]
    # include:
    # - board: bdn9_rev2
    # - board: nice_nano_v2
    # shield: reviung41
    #
    ---

and then update it as appropriate to build the right shields/boards for your configuration.

Upgrade a manual script

If you have a custom GitHub Actions workflow you need to maintain for some reason, you can update the workflow to to use the stable Docker image tag for the build:

  • Open .github/workflows/build.yml in your editor/IDE

  • Change zmkfirmware/zmk-build-arm:2.5 to zmkfirmware/zmk-build-arm:stable wherever it is found

  • Locate and delete the lines for the DTS output step, which is no longer needed:

      - name: ${{ steps.variables.outputs.display-name }} DTS File
    if: ${{ always() }}
    run: |
    if [ -f "build/zephyr/${{ matrix.board }}.pre.tmp" ]; then cat -n build/zephyr/${{ matrix.board }}.pre.tmp; fi
    if [ -f "build/zephyr/zephyr.dts" ]; then cat -n build/zephyr/zephyr.dts; fi

VS Code & Docker (Dev Container)

If you build locally using VS Code & Docker then:

  • pull the latest ZMK main with git pull for your ZMK checkout
  • reload the project
  • if you are prompted to rebuild the remote container, click Rebuild
  • otherwise, press F1 and run Remote Containers: Rebuild Container
  • Once the container has rebuilt and reloaded, run west update to pull the updated Zephyr version and its dependencies.

Once the container has rebuilt, VS Code will be running the 3.2 Docker image.

Local Host Development

The following steps will get you building ZMK locally against Zephyr 3.2:

  • Run the updated toolchain installation steps, and once completed, remove the previously installed SDK version (optional, existing SDK should still work)
  • Install the latest version of west by running pip3 install --user --update west.
  • pull the latest ZMK main with git pull for your ZMK checkout
  • run west update to pull the updated Zephyr version and its dependencies

From there, you should be ready to build as normal!

Known Issues

A few testers have reported inconsistent issues with bluetooth connections on Windows after upgrading, which can be resolved by re-pairing your keyboard by:

  1. Remove the device from Windows.
  2. Clear the profile on your keyboard that is associated with the Windows device by triggering &bt BT_CLR on your keymap while that profile is active.
  3. Restart Windows.
  4. Re-connect Windows to your keyboard.

Windows Battery Reporting Fix

Zephyr 3.2 introduced a new Kconfig setting that can be used to work around a bug in Windows related to battery reporting. Check out our bluetooth config for the full details. The key new configuration that can be set if using Windows is:

CONFIG_BT_GATT_ENFORCE_SUBSCRIPTION=n

Keymap Changes

Due to conflicts with new devicetree node labels added for Zephyr's reset system, the &reset behavior has been renamed to &sys_reset.

All of the in-tree keymaps have been fixed, but you may encounter build failures about duplicate names, requiring you rename the behavior reference in your keymap. Use the Keymap Upgrader and this will get fixed for you automatically.

Board/Shield Changes

The following changes have already been completed for all boards/shields in ZMK main branch. For existing or new PRs, or out of tree boards, the following changes are necessary to properly work with the latest changes.

Move to pinctrl driver

Before this change, setting up the details of pins to use them for peripherals like SPI, I2C, etc. was a mix of platform specific driver code. Zephyr has moved to the newer pinctrl system to unify the handling of pin configuration, with additional flexibility for things like low power modes for those pins, etc.

Board specific shield overlays

The main area this affects existing shields is those with board specific overrides, e.g. <shield>/boards/seeeduino_xiao_ble.overlay, that sets up additional components on custom buses, e.g. addressable RGB LEDs leveraging the SPI MOSI pin.

nRF52 Pin Assignments

Previously in ZMK, we relied on per-driver devicetree source properties to set the alternate pin functions for things like SPI or I2C. For example, here is the I2C bus setup as it was previously on the nice_nano board:

&i2c0 {
compatible = "nordic,nrf-twi";
sda-pin = <17>;
scl-pin = <20>;
};

With the move to the pinctrl system, this setup now look like:

 &i2c0 {
compatible = "nordic,nrf-twi";
pinctrl-0 = <&i2c0_default>;
pinctrl-1 = <&i2c0_sleep>;
pinctrl-names = "default", "sleep";
};

which references the pinctrl configuration:

&pinctrl {
i2c0_default: i2c0_default {
group1 {
psels = <NRF_PSEL(TWIM_SDA, 0, 17)>,
<NRF_PSEL(TWIM_SCL, 0, 20)>;
};
};

i2c0_sleep: i2c0_sleep {
group1 {
psels = <NRF_PSEL(TWIM_SDA, 0, 17)>,
<NRF_PSEL(TWIM_SCL, 0, 20)>;
low-power-enable;
};
};
};

Although slightly more verbose this allows pin configuration infrastructure to be re-used, specify other modes, like sleep, etc. in a standard way across architectures.

Out of Tree Boards/Shields

All of the in-tree boards and shields have been upgraded, but if you maintain/use an out-of-tree board or shield that uses the converted boards and overrides pins for various buses, you may need to switch to pinctrl to match ZMK's new approach.

The approach is the following when updating a board:

  1. Add an entry CONFIG_PINCTRL=y to the <board>_defconfig file in the board directory.

  2. Add a new file with the naming convention <board>-pinctrl.dtsi to your board directory.

  3. In the new file, add your pinctrl entries that set up different pin control configurations for whatever peripherals/buses are needed. Here's the nice!nano file as an example:

    /*
    * Copyright (c) 2022 The ZMK Contributors
    * SPDX-License-Identifier: MIT
    */

    &pinctrl {
    uart0_default: uart0_default {
    group1 {
    psels = <NRF_PSEL(UART_RX, 0, 8)>;
    bias-pull-up;
    };
    group2 {
    psels = <NRF_PSEL(UART_TX, 0, 6)>;
    };
    };

    uart0_sleep: uart0_sleep {
    group1 {
    psels = <NRF_PSEL(UART_RX, 0, 8)>,
    <NRF_PSEL(UART_TX, 0, 6)>;
    low-power-enable;
    };
    };

    i2c0_default: i2c0_default {
    group1 {
    psels = <NRF_PSEL(TWIM_SDA, 0, 17)>,
    <NRF_PSEL(TWIM_SCL, 0, 20)>;
    };
    };

    i2c0_sleep: i2c0_sleep {
    group1 {
    psels = <NRF_PSEL(TWIM_SDA, 0, 17)>,
    <NRF_PSEL(TWIM_SCL, 0, 20)>;
    low-power-enable;
    };
    };
    };
  4. From the main <board>.dts file, add an #include "<board>-pinctrl.dtsi" to have the C-preprocessor combine the files.

  5. Update the various peripheral nodes to use the new pinctrl configurations. For example, the following old configuration:

    &i2c0 {
    compatible = "nordic,nrf-twi";
    sda-pin = <15>;
    scl-pin = <17>;
    };

    would be changed to:

    &i2c0 {
    compatible = "nordic,nrf-twi";
    pinctrl-0 = <&i2c0_default>;
    pinctrl-1 = <&i2c0_sleep>;
    pinctrl-names = "default", "sleep";
    };

Because pinctrl configuration is very dependent on the specific target SoC, you will rarely need to consider it for a shield overlay that leverages a pro micro or XIAO abstraction. As noted, you're more likely to need to fix up pinctrl settings is using a board specific shield overlay, e.g. <shield>/boards/<board>.overlay to set things up.

LVGL Kconfig changes.

With the update to LVGL 8.x, Zephyr now leverages an upstream Kconfig file for most LVGL settings. Due to this, the naming for many existing configs has been adjusted. For any configs moved upstream, the naming mostly involves a prefix change from LVGL_ to the shorter LV_. For any that are still Zephyr specific configs, they are now prefixed with LV_Z_ prefix.

If you maintain or use an out of tree board/shield with a display, the following will need to be changed in your Kconfig files:

  • LVGL_VDB_SIZE -> LV_Z_VDB_SIZE
  • LVGL_DPI -> LV_DPI_DEF
  • LVGL_BITS_PER_PIXEL -> LV_Z_BITS_PER_PIXEL

Other than those specific examples, most other Kconfig values can simply change the LVGL_ prefix to LV_.

Raspberry Pi Pico/RP2040 Support

This Zephyr update allows ZMK to support the new(-ish) RP2040 SoC found in the Raspberry Pi Pico.

note

ZMK does not support wired split communication yet, so RP2040 is only usable for non-split keyboards. To follow progress on wired splits, see #1117.

Supported Controllers

The following RP2040 powered controllers have board definitions for folks to test:

  • Raspberry Pi Pico (rpi_pico)
  • SparkFun Pro Micro RP2040 (sparkfun_pro_micro_rp2040)
  • Adafruit Keyboar/KB2040 (adafruit_kb2040)
  • Seeeduino XIAO RP2040 (seeeduino_xiao_rp2040)
  • Adafruit Qt PY RP2040 (adafruit_qt_py_rp2040)
  • BoardSource blok (boardsource_blok)
  • Elite-Pi (compatible with the sparkfun_pro_micro_rp2040 board)

Upcoming Changes

Display re-init

Zephyr's improved power domain support is a foundation upon which we can provide a proper fix for the longstanding display re-init bug which has prevented ZMK from formally supporting our display code.

There is work still remaining to fully leverage the power domain system within ZMK to fix the bug, but upgrading Zephyr is the first necessary step.

Thanks!

Thanks to all the testers who have helped verify ZMK functionality on the newer Zephyr version.

· 3 min read
Pete Johanson

Two years ago, today, I minted the first ever commit for ZMK:

commit 85c8be89dea8f7a00e8efb06d38e2b32f3459935
Author: Pete Johanson <peter@peterjohanson.com>
Date: Tue Apr 21 16:20:34 2020 -0400

Initial work.

.gitignore | 1 +
.gitmodules | 3 +++
CMakeLists.txt | 40 +++++++++++++++++++++++++++++++++++++++
src/main.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
src/zmk_lib.h | 7 +++++++
zephyr-rust | 1 +
6 files changed, 112 insertions(+)

I will never forget that commit. Not because of the code it contained (please don't look, it's not worth it!), but for what it started.

Working on ZMK has given me the opportunity to reconnect with old friends (@brixmeister was my Gentoo mentor/sponsor when I became a contributor there on my first ever OSS project, and is a current active Zephyr RTOS contributor!), make new ones, and learn so much from the amazing mechanical keyboard community.

First Keyboard

But I'm getting ahead of myself! Back to early ZMK. I present you the first ZMK keyboard:

stm32wb55rg dev kit keyboard

That first "keyboard" taught me a lot. It forced me to dust off my long forgotten, rudimentary electronics knowledge, and gave me my first taste of really combining the physical/tangible with code in a way that years of doing backend API development never had.

I was hooked.

Zephyr RTOS

Early in my brainstorming, I knew I needed a foundation to build upon that would get me "a lot for free." I evaluated several different real-time operating systems (RTOSes) and happened upon Zephyr. It immediately ticked all the boxes I wanted:

  • Robust, open source Bluetooth stack, supporting multiple SoCs. At the time, I was trying out stm32wb thanks to some interest among keyboard designers, but I also so there were other compelling choices that might be a good fit.
  • An open source, non-copyleft license. I am a firm believer in F/OSS, and wanted to use a license that was as unrestricted as possible.
  • Had a lot of core APIs available, so I could focus on the keyboard functionality, not the plumbing. I love tinkering, but I wanted to focus my time on the interesting bits, not infrastructure.

I'm really happy with the choice, it has served us incredibly well the past two years.

Real Keyboard

At some point, somehow, innovaker introduced me to nicell who graciously sent me a few of the early pre-production nice!nano controllers, which I was able to get running on my Kyria. Doing so required the first split code, as well as lots of general improvements.

kyria keyboard

The day I was finally able to type on a wireless, split keyboard running ZMK was deeply momentous for me!

Onward and Upward

We've come a long way since then, with our supported hardware, features and behaviors growing regularly.

ZMK powered keyboards are now available in group buys and in stock at various vendors; compatible controllers have been used in a wide range of builds to empower our users to free themselves from their USB/TRRS cables and move about untethered.

This progress is only possible thanks to all of the contributors who've joined me in the vision for a wireless-first world. I am so grateful for everyone who has given their time to contribute code, answer questions on our Discord server, write more documentation, and especially all the users who have trusted us to make their input devices work.

I can't wait to see what we can accomplish together in the next two years.

· 8 min read
Pete Johanson

Welcome to the fifth ZMK "State Of The Firmware" (SOTF)!

This update will cover all the major activity since SOTF #4. That was over a year ago, so lots to cover!

Recent Activity

Here's a summary of the various major changes since last time, broken down by theme:

Keymaps/Behaviors

Since last time, there have been several new powerful keymap features and behaviors added, including several asked for features, such as tap-dance and macros.

Caps Word

petejohanson added the caps word behavior, i.e. &caps_word, in #823 that allows toggling a mode where all all alpha characters are sent to the host capitalized until a non-alpha, non-"continue list" keycode is sent. This can be useful for typing things like CONFIG_ENABLE_CAPS_WORD without having to hold down shift. This is similar in spirit to using the caps lock key, but with the added benefit of turning itself off automatically.

Key Repeat

petejohanson added the new key repeat behavior in #1034 to allow repeating the last sent key-press again, including any modifiers that were applied to that key press. It can be added to your keymap using the simple &key_repeat reference.

Macros

petejohanson, taking heavy inspiration on the initial work from okke-formsma, added macro support in #1168. Several common patterns are documented, but one example, changing the underglow color as you activate/deactivate a layer, looks like:

ZMK_MACRO(layer_color_macro,
wait-ms = <0>;
tap-ms = <0>;
bindings
= <&macro_press &mo 1>
, <&macro_tap &rgb_ug RGB_COLOR_HSB(128,100,100)>
, <&macro_pause_for_release>
, <&macro_release &mo 1>
, <&macro_tap &rgb_ug RGB_COLOR_HSB(300,100,50)>;
)

Tap Dance

kurtis-lew worked diligently to add the tap-dance behavior in #1139, allowing different behaviors to be invoked based on the number of times a user taps a single key in their keymap, e.g.

/ {
behaviors {
td0: tap_dance_0 {
compatible = "zmk,behavior-tap-dance";
label = "TAP_DANCE_0";
#binding-cells = <0>;
tapping-term-ms = <200>;
bindings = <&kp N1>, <&kp N2>, <&kp N3>;
};
};

keymap {
compatible = "zmk,keymap";

default_layer {
bindings = <
&td0
>;
};
};
};

Conditional Layers

bcat added conditional layers in #830 as a generalized version of the common "adjust layer" pattern on smaller keyboards.

Example:

/ {
conditional_layers {
compatible = "zmk,conditional-layers";
tri_layer {
if-layers = <1 2>;
then-layer = <3>;
};
};
};

Combos

mcrosson added the layer specific combos in #661, so users can make certain combos only triggerable when the layers set for the combo are active.

This is used by the ZMK implementation of ARTSEY extensively.

Sticky Keys

okke-formsma updated sticky keys in #1122 to add the ignore-modifiers; property; when set, sticky keys won't release when other modifiers are pressed. This allows you to combine sticky modifiers, which is popularly used with "callum-style mods".

Hold-Tap Improvements

jmding8 added an additional positional hold-tap configuration in #835 to help certain sequences produce the expected results.

jmding8 also added an additional hold-tap flavor: tap-unless-interrupted in #1018 which works very well with the new positional hold-tap config.

okke-formsma implemented retro-tap hold-tap property in #667

okke-formsma also added quick-tap-ms hold-tap property in #655

Apple Device Compatibility Improvements

Pairing

petejohanson did some sleuthing and fixed a long standing problem with inconsistent pairing with macOS in [#946]](https://github.com/zmkfirmware/zmk/pull/946). With the changes, macOS more reliably pairs with ZMK devices.

Consumer (Media) Codes

Another persistent bug that Apple users experienced was related to crashes and problems with keyboard configurations, that was traced to an issue with ZMK's HID usage that was fixed by petejohanson in #726.

Debounce Enhancements

joelspadin applied some major enhancements to our debouncing approach to allow fine grained control of our debouncing in #888, including allowing eager debouncing which can reduce key press latency.

Split Improvements

Behavior Locality

The long awaited locality enhancement was finally merged by petejohanson in #547, allowing more fine grained control of where certain behaviors are invoked. Some key improvements thanks to the changes:

  • RGB Underglow behaviors now run globally, so enabling/disabling RGB, changing the color, animation, etc. applies to both sides of a split properly.
  • Reset/Bootloader behaviors now run wherever the key was pressed. For example, adding a &bootloader reference to the peripheral side of a split will now put that side of the split into the bootloader when pressed.

Split Connections

petejohanson also added fixes to improve split re-connection for certain scenarios in #984, helping ensure splits properly connect when one side or the other is reset.

Hardware Support

Backlight

bortoz added single color backlight support in #904 for those keyboards that have it as an alternative to RGB underglow.

E-Paper Display (EPD) Driver

petejohanson worked with LOWPROKB to add support for the E-Paper Displays (EPD) in #895 used in keyboards like the Corne-ish Zen.

nRF VDDH Battery Sensing

joelspadin added a new sensor driver to support battery charge calculation by sensing voltage on the VDDH pin on nRF52 chips in #750, which is particularly useful for designs using "high voltage mode" with that SoC.

Miscellaneous

Documentation

dxmh and caksoylar have joined the ZMK organization to help with documentation, and have been doing an amazing job adding new docs, and leading reviewing docs related PRs to free other contributors up to focus on other areas. It's been an incredible addition to ZMK!

NKRO Support

petejohanson's work on the HID foundation also included adding support for full NKRO HID in #726 that can be enabled by adding the following to your .conf file for your config:

CONFIG_ZMK_HID_REPORT_TYPE_NKRO=y

Power Profiler

It's been live for a while, but nicell added an amazing power profiler in #312 to allow users to estimate their battery life for various hardware configurations.

Min/Max Underglow Brightness

malinges added support for configuring min/max underglow brightness in #944 by setting the values in your .conf file as percentages of full:

CONFIG_ZMK_RGB_UNDERGLOW_BRT_MIN=20
CONFIG_ZMK_RGB_UNDERGLOW_BRT_MAX=80

This can be useful to be sure that lowering brightness doesn't set the brightness to zero, and raising the brightness doesn't consume too much power.

Zephyr 3.0

petejohanson helped prepare and test the upgrade of ZMK to Zephyr 3.0 in #1143. The updated Zephyr release brings with it some key BLE stability fixes, as well as various other core improvements that improve ZMK. This was a huge undertaking!

New Shields

New Boards

Board/Shield Metadata

nicell and petejohanson worked together in #883 to settle on a metadata format that is used to document every board and shield. This now drives automatic generation of our supported hardware page and our more nuanced GH Actions automation for testing changes to ZMK.

Coming Soon!

Some items listed in the last coming soon section are still under active development.

  • RP2040 support
  • Peripheral rotary encoder support
  • Caps/Scroll/Num Lock LED support
  • Mouse Keys
  • Wired split support
  • More modular approach to external boards/shields, custom code, user keymaps, etc.
  • More shields and boards

Statistics

Some statistics of interest for ZMK:

  • GitHub (lifetime stats)
    • 105 Contributors
    • 791 Closed PRs
    • 849 Stars
    • 832 Forks
  • Discord Chat
    • 3430 total registered
  • Website (last 30 days)
    • 35.9K page views
    • 3.29K new users

Thanks!

As we approach the two year birthday for ZMK, I am reminded of how far we have come in such a short time, in large part thanks to the amazing community that has grown around it. I am so grateful to have so many contributors, testers, and user believing in the project and helping make it a joy to work on.

· 6 min read
Pete Johanson

I'm happy to announce that we have completed the work to upgrade ZMK to Zephyr 3.0!

petejohanson did the upgrade work to adjust ZMK for the Zephyr changes.

  • Moving to Zephyr's UF2 build integration that was submitted upstream by petejohanson
  • Additional color-mapping property needed for ws2812 LED strep devicetree nodes
  • Zephyr core API changes, including delayed work, USB/HID
  • Adjust for pinctrl changes on stm32
  • Fixes for power management and log formatter changes

Getting The Changes

Use the following steps to update to the latest tooling in order to properly use the new ZMK changes:

User Config Repositories Using GitHub Actions

Existing user config repositories using Github Actions to build will pull down Zephyr 3.0 automatically, however to build properly, the repository needs to be updated to use the stable Docker image tag for the build:

  • Open .github/workflows/build.yml in your editor/IDE

  • Change zmkfirmware/zmk-build-arm:2.5 to zmkfirmware/zmk-build-arm:stable wherever it is found

  • Locate and delete the lines for the DTS output step, which is no longer needed:

      - name: ${{ steps.variables.outputs.display-name }} DTS File
    if: ${{ always() }}
    run: |
    if [ -f "build/zephyr/${{ matrix.board }}.pre.tmp" ]; then cat -n build/zephyr/${{ matrix.board }}.pre.tmp; fi
    if [ -f "build/zephyr/zephyr.dts" ]; then cat -n build/zephyr/zephyr.dts; fi
note

If you created your user config repository a while ago, you may find that your build.yml file instead references a zephyr-west-action-arm custom GitHub Action instead. In this case, the upgrade is not as direct. We suggest that instead you re-create your config repository to get an updated setup using the new automation approach.

VS Code & Docker (Dev Container)

If you build locally using VS Code & Docker then:

  • pull the latest ZMK main with git pull for your ZMK checkout
  • reload the project
  • if you are prompted to rebuild the remote container, click Rebuild
  • otherwise, press F1 and run Remote Containers: Rebuild Container
  • Once the container has rebuilt and reloaded, run west update to pull the updated Zephyr version and its dependencies.

Once the container has rebuilt, VS Code will be running the 3.0 Docker image.

Local Host Development

The following steps will get you building ZMK locally against Zephyr 3.0:

  • Run the updated toolchain installation steps, and once completed, remove the previously installed SDK version (optional, existing SDK should still work)
  • pull the latest ZMK main with git pull for your ZMK checkout
  • run west update to pull the updated Zephyr version and its dependencies

From there, you should be ready to build as normal!

Board/Shield Changes

The following changes have already been completed for all boards/shields in ZMK main branch. For existing or new PRs, or out of tree boards, the following changes are necessary to properly work with the latest changes.

RGB Underglow

Zephyr's WS2812 led_strip driver added a new required property. When adding underglow to a board, you now must also add the additional include #include <dt-bindings/led/led.h> at the top of your devicetree file, and add a color-mapping property like:

led_strip: ws2812@0 {
compatible = "worldsemi,ws2812-spi";
label = "WS2812";

/* SPI */
reg = <0>; /* ignored, but necessary for SPI bindings */
spi-max-frequency = <4000000>;

/* WS2812 */
chain-length = <10>; /* number of LEDs */
spi-one-frame = <0x70>;
spi-zero-frame = <0x40>;
color-mapping = <LED_COLOR_ID_GREEN
LED_COLOR_ID_RED
LED_COLOR_ID_BLUE>;
};
note

Standard WS2812 LEDs use a wire protocol where the bits for the colors green, red, and blue values are sent in that order. If your board/shield uses LEDs that require the data sent in a different order, the color-mapping property ordering should be changed to match.

Display Selection

Zephyr moved to using a chosen node named zephyr,display to select the display device to be used with LVGL, the underlying display library we use.

For example, for a shield with:

&pro_micro_i2c {
status = "okay";

oled: ssd1306@3c {
compatible = "solomon,ssd1306fb";
reg = <0x3c>;
label = "SSD1306";
width = <128>;
height = <32>;
segment-offset = <0>;
page-offset = <0>;
display-offset = <0>;
multiplex-ratio = <31>;
com-invdir;
segment-remap;
com-sequential;
prechargep = <0x22>;
};
};

You would add a chosen node like:

/ {
chosen {
zephyr,display = &oled;
};
};

USB Logging

Zephyr unified the way the console/logging device is selected, removing the hacks that special-cased the USB CDC ACM output. Now, the CDC ACM device is configured in the devicetree as well. To ensure that USB logging properly works with custom board definitions, two sections of the <board>.dts file need updating.

Underneath the USB device, add the CDC ACM node:

&usbd {
status = "okay";
cdc_acm_uart: cdc_acm_uart {
compatible = "zephyr,cdc-acm-uart";
label = "CDC_ACM_0";
};
};

Then, an additional chosen node (near the top of the file) will mark the CDC ACM device as the console:

/ {
chosen {
...
zephyr,console = &cdc_acm_uart;
};
...
};

UF2 Builds

Previously, to get ZMK to build a UF2 image to flash to a given board required adding a CMakeLists.txt file that added a custom post build command. Now, the only thing necessary to have Zephyr build a UF2 is to add the following to your <board>_defconfig file:

CONFIG_BUILD_OUTPUT_UF2=y

If updating an existing board, be sure to remove the previous CMakeLists.txt file to avoid generating the UF2 twice during a west build.

For more details on the implementation, see zephyr#31066.

STM32 Clock Configuration

Clock configuration moved to devicetree as well, out of the Kconfig files. Here is a sample config for a board that uses the HSI for the PLL source:

&clk_hsi {
status = "okay";
};

&pll {
prediv = <1>;
mul = <12>;
clocks = <&clk_hsi>;
status = "okay";
};

&rcc {
clocks = <&pll>;
clock-frequency = <DT_FREQ_M(72)>;
ahb-prescaler = <1>;
apb1-prescaler = <2>;
};

After adding the nodes, be sure to remove the clock/PLL related configuration from the <board>_defconfig file.

Seeeduino XIAO

The Seeed(uino) XIAO has gained in popularity for use on smaller boards, and gained more traction with the release of the new XIAO BLE board, powered by the popular nRF52840 SoC. As part of the 3.0 update, we've also more fully integrated the XIAO and XIAO BLE to make it easier to build keyboard (shields) using either controller.

Future Hardware

One of the exciting items that's one step closer as part of this work is support for Raspberry Pi Pico/RP2040. With Zephyr 3.0 merged, this start the process for getting those controllers/chips supported by ZMK. Follow the issue to keep track of progress. This will also enable us to support the XIAO compatible Adafruit Qt Py RP2040 and XIAO RP2040.

Thanks!

Thanks to all the testers who have helped verify ZMK functionality on the newer Zephyr version.

· One min read
Pete Johanson

As preparation for completing the work to upgrade ZMK to Zephyr 3.0, users with user config repositories who wish to avoid future build failures with their GitHub Actions workflows can take steps to adjust their repositories now.

GitHub Actions needs to use our latest Docker image to ensure continued compatibility with the ZMK codebase on Zephyr 3.0 (and beyond). You should:

  • Open .github/workflows/build.yml in your editor/IDE
  • Change zmkfirmware/zmk-build-arm:2.5 to zmkfirmware/zmk-build-arm:stable wherever it is found

Once the changes are committed and pushed, the build will run as expected.

A future blog post will outline the complete Zephyr 3.0 changes once that work is finalized.

note

If you created your user config repository a while ago, you may find that your build.yml file instead references a zephyr-west-action-arm custom GitHub Action instead. In this case, the upgrade is not as direct. We suggest that instead you re-create your config repository to get an updated setup using the new automation approach.

· 3 min read
Pete Johanson

I'm happy to announce that we have completed the work to upgrade ZMK to Zephyr 2.5!

A big part of this work was some major refactors and improvements by innovaker to our zmk-docker Docker image and GH Actions automation.

  • Faster build times with improved caching.
  • Integration tests which automatically verify new images.
  • PRs to the repo now build properly and run the tests as well.
  • Build images for multiple target architectures, e.g. zmk-build-riscv64, all in parallel.
  • Nightly builds to be sure we're pulling in the latest OS/package updates, to ensure we keep our images up to date, address any reported vulnerabilities, etc.
  • Faster upgrade paths for future Zephyr SDK and Zephyr versions.

In addition, petejohanson did the upgrade work to adjust ZMK for the Zephyr changes.

  • Updated to newer devicetree/driver Zephyr API
  • Adjustment for Zephyr pinmux changes
  • Fixes for power management, LVGL, and formatter changes

Getting The Changes

Use the following steps to update to the latest tooling in order to properly use the new ZMK changes:

User Config Repositories Using GitHub Actions

Existing user config repositories using Github Actions to build will pull down Zephyr 2.5 automatically, and should work, fine as is. However, to upgrade to the newer Docker image, you should:

  • Open .github/workflows/build.yml in your editor/IDE
  • Change zmkfirmware/zmk-build-arm:2.4 to zmkfirmware/zmk-build-arm:2.5 wherever it is found
note

If you created your user config repository a while ago, you may find that your build.yml file instead references a zephyr-west-action-arm custom GitHub Action instead. In this case, the upgrade is not as direct. We suggest that instead you re-create your config repository to get an updated setup using the new automation approach.

VS Code & Docker (Dev Container)

If you build locally using VS Code & Docker then:

  • pull the latest ZMK main with git pull for your ZMK checkout
  • reload the project
  • if you are prompted to rebuild the remote container, click Rebuild
  • otherwise, press F1 and run Remote Containers: Rebuild Container
  • Once the container has rebuilt and reloaded, run west update to pull the updated Zephyr version and its dependencies.

Once the container has rebuilt, VS Code will be running the 2.5 Docker image.

Local Host Development

The following steps will get you building ZMK locally against Zephyr 2.5:

  • Run the updated toolchain installation steps, and once completed, remove the previously installed SDK version (optional, existing SDK should still work)
  • pull the latest ZMK main with git pull for your ZMK checkout
  • run west update to pull the updated Zephyr version and its dependencies

From there, you should be ready to build as normal!

Thanks!

Thanks again to innovaker for all the hard work, and to all the testers who have helped verify ZMK functionality on the newer Zephyr version.

· 7 min read
Pete Johanson

Welcome to the fourth ZMK "State Of The Firmware" (SOTF)!

This update will cover all the major activity since SOTF #3.

Recent Activity

Here's a summary of the various major changes since last time, broken down by theme:

Keymaps/Behaviors

Since last time, there have been several new powerful keymap features and behaviors added, including one of the most asked for features, combos!

Combos

The initial combos work has landed! The amazing okke-formsma has once again delivered another powerful feature for ZMK. Combos are "position based", and are configured in a new toplevel node next to they keymap node in user's keymap files.

An example, that would send the ESC keycode when pressing both the first and second positions on your keyboard:

/ {
combos {
compatible = "zmk,combos";
combo_esc {
timeout-ms = <50>;
key-positions = <0 1>;
bindings = <&kp ESC>;
};
};
};
note

Combos currently are "global", and not scoped to a given active layer. There is future planned work to allow enabling a certain combo for only certain active layers.

Sticky Keys (One-Shot Mods/Layers) Behavior

okke-formsma also contributed the initial "sticky keys" behavior, which can be used for functionality sometimes called "one shot mods" or "one shot layers". In your keymap, this would like like:

&sk LEFT_CONTROL

for a sticky key/modifier, or:

&sl NAV

for a sticky layer.

&to Layer Behavior

mcrosson contributed the new &to layer related behavior. This can be used to completely replace the active layer with a new one.

This is most frequently used when using multiple core base layers with different layouts, e.g. QWERTY and DVORAK, to switch between them.

Grave Escape Behavior

okke-formsma added an implementation of the "Grave Escape" behavior, developing a more generic "mod-morph" behavior to do so. Adding

&gresc

to your keymap will send ESC when pressed on its own, but will send ` when pressed with a GUI or Shift modifier held.

RGB Underglow Color Selection

mcrosson updated the RGB Underglow behavior to allow binding an explicit color selection to a key position.

Keymap Upgrader

joelspadin completed the Keymap Upgrader which can be used to update your keymap to using the latest supported codes, and move away from the old deprecated codes.

If you've made keymap customizations, please make sure to run your keymaps through the upgrader, since the old deprecated codes will be removed in a future version of ZMK.

Displays

There has been lots of work to get display support complete enough for use by end users. Although not quite ready for prime time, it is incredibly close, and we are looking forward to having the last few items completed and the feature documented!

Idle Blanking

petejohanson added idle blanking for displays, which ensures they will go blank, and into low power mode, after a short period of inactivity from the user. This ensures we avoid burn-in for OLEDs, and helps improve battery life.

Battery and Output Widgets

petejohanson implemented the first two complete, dynamic "widgets" for the displays for ZMK, adding a small battery indicator, which includes charging status, and a small output indicator, showing the currently active output (USB or BLE). When using BLE, the indicator also shows the active profile slot, as well as if the profile slot is open, awaiting connection from the paired host, or is actively connected to the host for that profile slot.

Highest Layer Display

mcrosson has contributed the next display widget, showing the highest active layer in the keymap. petejohanson then added a small follow up to allow layers in keymaps to add a label property to each layer, e.g. label = "Nav"; and have that label be displayed in the widget instead of the numeric layer number.

WPM

New contributor allymparker added our fourth widget, a words-per-minute display! This widget work also included creating the core state logic for tracking the WPM.

For now, this widget is only working on the central side of split keyboards.

Miscellaneous

Zephyr 2.4

innovaker is at it again with some crucial core fixes, helping prepare and test the upgrade of ZMK to Zephyr 2.4. The updated Zephyr release brings with it some key BLE stability fixes, as well as various other core improvements that improve ZMK. This was a huge undertaking!

BLE Deadlock Fixes

petejohanson was heads down diagnosing and fixing a deadlock issue on BLE that was frustrating and plaguing many users. After finally pinpointing the underlying root cause, he developed a fix and roped in many testers on Discord to help stress test things before merging.

Central/Peripheral Selection

Previously overriding the selection of left as central, and right as peripheral for wireless splits required making local edits to the configuration files, and maintaining them in a ZMK fork.

petejohanson updated the config files to allow users to override this in their <board>_left.conf/<board>_right.conf files in their user repos.

Improved Docker Containers

As part of the Zephyr 2.4. prep work, innovaker, along with lots of testing and input from mcrosson, developed a brand new pair of Docker images which is now published to Docker Hub as zmkfirmware/zmk-build-arm and zmkfirmware/zmk-dev-arm.

The previously blogged VSCode + Docker integration, as well as our GH Action build automation was all moved over to the new images.

Settings Debounce

nicell contributed settings debounce work, to help avoid unnecessary extra writes to flash when making various changes that should be saved, such as the active BLE profile, external VCC on/off, etc.

New Shields

New Boards

Sponsorship

Since it's inception, quite a few users have inquired whether they could sponsor any of the contributors involved in ZMK. Although we are not intending to directly fund any individual contributors for their work on ZMK, there is good that can come from folks sponsoring ZMK.

You can see the full discussion on #497, but some items that are being considered with sponsorship funds:

  • Hiring a designer to complete the logo/mascot work.
  • Creating stickers to send as thank-yous to first time contributors.
  • Hosting costs for GitHub Pro.
  • Other hosting costs, e.g. Docker Hub.

For anyone looking to contribute, you can find the ZMK Firmware project is now set up on Open Collective.

Coming Soon!

Some items listed in the last coming soon section are still under active development.

  • A power profiler page for the website, to help users estimate their battery life for a given keyboard - Nicell
  • Behavior "locality", allowing improved split usage for things like &sys_reset, and controlling external power and RGB underglow for both sides - petejohanson
  • More modular approach to external boards/shields, custom code, user keymaps, etc.
  • More shields and boards

Statistics

Some statistics of interest for ZMK:

  • GitHub (lifetime stats)
    • 389 Closed PRs
    • 199 Stars
    • 163 Forks
  • Discord Chat
    • 702 total registered
  • Website (last 30 days)
    • 11.5K page views
    • 1K new users

Thanks!

Thanks again to the numerous contributors, testers, and users who have made working on ZMK such a pleasure!

· 6 min read
Pete Johanson

Welcome to the third ZMK "State Of The Firmware" (SOTF)!

This update will cover all the major activity since SOTF #2. This edition comes a bit later than planned, but the amount of features and changes will hopefully make it worth it!

Recent Activity

Here's a summary of the various major changes since last time, broken down by theme:

Keymaps/Behaviors

Tons of activity related to keymaps, so we'll go into more detail this time.

Codes Overhaul

innovaker completely overhauled the set of available codes for keymaps, and simultaneously has created beautiful documentation to help users visualize the codes, and also understand if they are supported on their particular operating system.

This also laid the foundation for the other keymap related changes that are now available.

Modified (Shifted) Codes

okke-formsma added the ability to apply modifiers to a code, e.g.:

&kp LC(C)

which sends Control + c when pressed. This feature is often used on smaller keyboards to achieve "shifted keycodes", e.g. LS(N1) to send a !. To make this easier, in addition to all the normal codes, we now have defines for common shifted codes, e.g. EXCL for !, AT for @, etc.

To learn more, check out the Modifiers documentation.

Simplified Key Press Behavior

In previous versions of ZMK, users needed to be careful to select between the &kp and &cp behaviors in their keymaps, depending on whether the particular keycode they wanted to send was in the "HID consumer page" or not. Forcing users to understand the difference and get this right was awkward and error prone.

petejohanson and innovaker have reduced this complexity. Users can now simply use &kp with all available codes and ZMK will handle sending the right events to the connected host.

Power Management

Several important power management features have been added to ZMK, helping save power for many use cases.

BLE Battery Level Reporting

Nicell added the necessary driver and core code to send BLE battery level notifications to hosts that support displaying them. Testing seems to show this works with Windows and GNOME, but macOS does not display the battery info.

External Power Control

megamind4089 added a new driver and behavior to allow users to toggle (on/off) the external power supplied by boards such as the nRFMicro and nice!nano that have specialized hardware for this purpose.

With this change, you can add

&ext_power EP_TOG

to toggle (on/off) the power to external hardware like RGB underglow or OLEDs. Check out the external power control docs for more info.

Deep Sleep

petejohanson has contributed the initial deep sleep support to ZMK. This work also included some automatic power savings by switching to PORT events on the nRF52 chips, which reduces the idle power draw, even without deep sleep. Deep sleep is currently not turned on by default, but will be soon.

Miscellaneous

Output Selection

joelspadin added output selection to allow selecting whether to send output over USB or BLE if both are connected. This should now help avoid having "double keypresses" when your keyboard is plugged into a host.

Bootloader Corruption Fix

Nicell has already blogged about this, but for those that missed it, a major, and incredibly difficult to pin down bug involving corruption of the bootloader on devices using the Adafruit nRF52 bootloader has been fixed by Nicell. If you've encountered this bug, flashing the latest firmware should prevent it from reoccurring. Unfortunately, due to the nature of this fix, you will need to re-pair your keyboard with your hosts, as the fix involves changing where settings are stored in the flash of the controller.

Official USB Product ID

petejohanson has gotten an official USB product ID assigned to the ZMK Firmware. For anyone looking to uniquely identify a USB device running the ZMK Firmware, you can match on:

  • Vendor ID: 0x1d50
  • Product ID: 0x615e

We are incredibly grateful that Openmoko Inc., in the wake of discontinuing the openmoko projects, has made this an option for OSS projects.

Development: Remote Docker Container Integration

idan contributed VSCode devcontainer integration to make it easier for developers to build and develop ZMK without having to do complicated local toolchain setup and configuration. This also opens up some amazing future flexibility for things like GitHub Codespaces.

There's some follow up tweaks necessary for better supporting using this with user config repositories, which will be available soon.

New Shields

New Boards

Testing

There has been an amazing amount of testing from various users as we develop new features. In particular, we'd like to give a shout out to tominabox1 who has been tireless in providing detailed and thorough testing of changes as they are being developed.

Coming Soon!

Some items listed in the last coming soon section are still under active development.

  • OLED work, including battery and USB/BLE connection status - petejohanson
  • One shot mod/layer behaviors - okke-formsma
  • A power profiler page for the website, to help users estimate their battery life for a given keyboard - Nicell
  • A keymap converter to automatically update keymaps to the new codes and use of &kp everywhere - joelspadin

Statistics

Some statistics of interest for ZMK:

  • GitHub (lifetime stats)
    • 210 Closed PRs
    • 116 Stars
    • 101 Forks
  • Discord Chat
    • 363 total registered
  • Website (last 30 days)
    • 8.5K page views
    • 766 new users

Thanks!

Thanks again to the numerous contributors and users who have made working on ZMK such a pleasure!

· 8 min read
Nick Winans

Recently I was able to fix the "stuck in the bootloader" issue in #322 that had been plaguing us for quite some time. I want to go over what the issue was, how the issue was diagnosed, and how it was fixed.

Background

What exactly is the "stuck in the bootloader" issue? Seemingly randomly, users' keyboards would suddenly stop working and when they would reset their keyboard they would get put into the bootloader instead of back into the firmware. This would require the user to re-flash the firmware again to get into the firmware. That wouldn't be so bad except for the fact that once this occurs, every reset would require the user to re-flash the firmware again. The only way to really fix this issue was to re-flash the bootloader itself, which is a huge pain.

Going into this, all we knew was that this issue was most likely introduced somewhere in the #133, which added Bluetooth profile management. We've had quite a few attempts at trying to recreate the issue, but we never were able to get it to happen consistently.

Diagnosing the issue

This issue had been happening sporadically for the past month, and I finally decided to dig in to see what was going on. We started in the Discord and discussed what was common between all of the people who have experienced this issue. Everyone who had this issue reported that they did quite a bit of profile switching. This lined up with the possible connection to the Bluetooth profile management pull request.

Pinpointing the cause

I had a hunch that this was related to the settings system. The settings system is used by profile Bluetooth switching, and the settings system works directly with the system flash. Based on this hunch, I tried spamming the RGB underglow cycle behavior on my main keyboard. Sure enough after a couple minutes, I got stuck in the bootloader. I was even able to reproduce it again.

This was an important discovery for two reasons. First, I was able to recreate the issue consistently, which meant I could set up logging and more closely monitor what the board was doing. Second, this more or less proved that it was specifically the settings system at fault. Both Bluetooth profile switching and RGB underglow cycling trigger it, and the one common piece is they save their state to settings.

Settings system overview

To understand what's going wrong, we first need to understand how the settings system works. Here's a diagram to explain the flash space that the settings system holds for our nRF52840 based boards (nice!nano, nRFMicro, BlueMicro).

Settings Diagram

The settings flash space lives at the end of the flash of the chip. In this case it starts at 0xF8000 and is 0x8000 bytes long, which is 32KB in more comprehensible units. Then due to the chip's architecture, this flash space is broken into pages, which are 0x1000 bytes in size (4KB).

The backend that carries out the settings save and read operation in ZMK is called NVS. NVS calls these pages sectors. Due to how flash works, you can't write to the same bytes multiple times without erasing them first, and to erase bytes, you need to erase the entire sector of flash. This means when NVS writes to the settings flash if there's no erased space available for the new value, it will need to erase a sector.

Logging discoveries

So first I enabled logging of the NVS module by adding CONFIG_NVS_LOG_LEVEL_DBG=y to my .conf file. I repeated the same test of spamming RGB underglow effect cycle and the resulting logs I got were this:

[00:00:00.000,671] <inf> fs_nvs: 8 Sectors of 4096 bytes
[00:00:00.000,671] <inf> fs_nvs: alloc wra: 3, f70
[00:00:00.000,671] <inf> fs_nvs: data wra: 3, f40
// A bunch of effect cycle spam
[00:02:34.781,188] <dbg> fs_nvs: Erasing flash at fd000, len 4096
// A bunch more effect cycle spam
[00:06:42.219,970] <dbg> fs_nvs: Erasing flash at ff000, len 4096
// A bunch more effect cycle spam
// KABOOM - bootloader issue

So at start up, we can see that the 8 sectors of 4KB are found by NVS properly, however, I wasn't sure what the second and third lines meant, but we'll get back to that. Nonetheless the next two logs from NVS showed erasing the sector at 0xFD000 and then erasing the 0xFF000 sector.

Erased Sectors

It's really odd that the third to last sector and the last sector are erased, and then shortly after the bootloader issue is hit. I really had no explanation for this behavior.

Reaching out to Zephyr

At this point, I nor anyone else working on the ZMK project knew enough about NVS to explain what was going on here. Pete Johanson, project founder, reached out on the Zephyr Project's Slack (ZMK is built on top of Zephyr if you weren't aware). Justin B and Laczen assisted by first explaining that those alloc wra and data wra logs from earlier are showing what data NVS found at startup.

More specifically, data wra should be 0 when it first starts up on a clean flash. As we can see from my earlier logging on a clean flash I was instead getting f40. NVS is finding data in our settings sectors when they should be blank! We were then given the advice to double check our bootloader.

The Adafruit nRF52 Bootloader

Most of the boards the contributors of ZMK use have the Adafruit nRF52 Bootloader, which allows for extremely easy flashing by dragging and dropping .uf2 files onto the board as a USB drive. Every bootloader takes up a portion of the flash, and in the README explains that the first 0x26000 is reserved for the bootloader with the nRF52840, and we've properly allocated that.

However, there isn't a full explanation of the flash allocation of the bootloader in the README. There's a possibility that the bootloader is using part of the same flash area we're using. I reached out on the Adafruit Discord, and Dan Halbert pointed me towards the linker map of the nRF52840. Let's take a look.

FLASH (rx) : ORIGIN = 0xF4000, LENGTH = 0xFE000-0xF4000-2048 /* 38 KB */

BOOTLOADER_CONFIG (r): ORIGIN = 0xFE000 - 2048, LENGTH = 2048

/** Location of mbr params page in flash. */
MBR_PARAMS_PAGE (rw) : ORIGIN = 0xFE000, LENGTH = 0x1000

/** Location of bootloader setting in flash. */
BOOTLOADER_SETTINGS (rw) : ORIGIN = 0xFF000, LENGTH = 0x1000

Here's a diagram to show this a bit better.

Adafruit Bootloader Diagram

We've found the issue! As you can see from the red bar (representing our settings flash area), we've put the settings flash area right on top of the Adafruit bootloader's flash space. Oops!

This also shines some light on why NVS erased 0xFD000 and 0xFF000 sectors. It's possible there was no flash written to 0xFD000 because the bootloader didn't use up all of that space it has, and then there possibly weren't any bootloader settings set yet, so 0xFF000 could be used and erased by NVS too.

After erasing 0xFF000, NVS probably next erased a rather important part of the bootloader that resulted in this issue at hand. In my opinion, we're pretty lucky that it didn't delete an even more vital part of the bootloader. At least we could still get to it, so that we could re-flash the bootloader easily!

The solution

Now that we've found the issue, we can pretty easily fix this. We'll need to move the settings flash area back so that it doesn't overlap with the bootloader. First we calculate the size of the of flash area the bootloader is using.

0x100000 (end of flash) - 0x0F4000 (start of bootloader) = 0xC000 (48KB)

So the bootloader is using the last 48KB of the flash, this means all we need to do is shift back the settings area and code space 0xC000 bytes. We'll apply this to all of the .dts files for the boards that were affected by this issue.

        code_partition: partition@26000 {
label = "code_partition";
- reg = <0x00026000 0x000d2000>;
+ reg = <0x00026000 0x000c6000>;
};


- storage_partition: partition@f8000 {
+ storage_partition: partition@ec000 {
label = "storage";
- reg = <0x000f8000 0x00008000>;
+ reg = <0x000ec000 0x00008000>;
};

And with those changes, we should no longer run into this issue! In the process of these changes, we lost 48KB of space for application code, but we're only using around 20% of it anyways. 🎉