Main purpose¶

In modern world even embedded systems can benefit from virtualization technologies. For example, ARMv8 architecture have built-in virtualization extensions, so both Xen hypervisor and KVM can run on variety of ARMv8-based SoCs. In embedded world it is natural to have some way to build the whole bootable image with one command. But there is no easy way to build multiple different virtual machine images. For example, we want to build Linux-based host VM, Android and another Linux distro as a guest VM. moulin was created exactly for this purpose.

Most of the modern BSPs are Yocto-based Linux distributions with vendor-specific changes. Mobile chips likely will have Android Open Source (AOSP) support as well. So right now moulin is focuses on supporting this two build systems. More will be added later.

Design¶

moulin is not a replacement for make or bitbake. It merely processes project YAML file and generates Ninja build file. So, actual job runner is the Ninja.

On other hand, moulin project files describe the whole build, with source code locations, build options, dependencies between images and optional parameters.

Requirements and Installation¶

moulin requires Python 3.6+ to run. You might need pip or pip3 tool to install it.

moulin generates build.ninja compatible with ninja 1.10+. This is because lower versions of ninja do not support multiple outputs. So if you run ninja and see error message like this

multiple outputs aren't (yet?) supported by depslog

you need to get newer version from https://github.com/ninja-build/ninja/releases.

Also moulin requires pygit2. If it is not installed on your system, pip will try to install it from repository. It may fail in case if you are missing libgit2 development files or if installed libgit2 is too old. So, it is better to install pygit2 globally. For example, on Ubuntu 18.04 you need to install python3-pygit2 package.

moulin source code is stored at GitHub.

Preferred way to install it right now is to put it your local per-user storage using

pip3 install --user git+https://github.com/xen-troops/moulin

command. Make sure that your PATH environment variable includes /home/${USER}/.local/bin.

You can encounter problem with pygit2 installation on some Ubuntu or Debian distribution due to this bug. In this case you will need to use the following command

PIP_IGNORE_INSTALLED=0 pip3 install --user git+https://github.com/xen-troops/moulin

as a workaround.

Alternatively, you can clone the mentioned git repository and invoke moulin.py directly.

YAML format¶

YAML (yet-another-markup-language) is a quite simple tree-like file format that can be easily read by humans and machines. If you are not familiar with it, you can take a look at the official specification or read any of multiple guides that are available in the Internet.

For the most of this documentation we assume that you can read and understand YAML files.

Simple Android build¶

There is example of minimal AOSP build:

desc: "Build Android 11 XenVM image for Renesas Rcar H3"
components:


  android:


    sources:


      - type: repo


        url: https://github.com/xen-troops/android_manifest.git


        rev: android-11-master


        manifest: doma.xml


        depth: 1


        groups: all


        dir: "."


    builder:


      type: android


      env:


        - "TARGET_BOARD_PLATFORM=r8a7795"


      lunch_target: xenvm-userdebug


      target_images:


        - "out/xenvm/userdebug/boot.img"


        - "out/xenvm/userdebug/system.img"

Let's discuss what is going there. This project file consist of two mandatory sections. desc provides description for the build. In this case it tells us that we will build very specific variant of Android 11.

components section is the main part of the file. It describes only one component, which is android, of course. Every component consists of two main parts: sources and builder.

sources describe all code sources that should be fetched prior the building. In this particular case we have only one repo - based source.

builder section configures the actual build. In this example we are building Android, so it has Android-specific options like lunch_target.

If the code above is stored in file android-vm.yaml, you can issue moulin android-vm.yaml. This will create build.ninja file in the same directory. After that, just run ninja and it will use repo to fetch given manifest, synchronize sources and start the build.

Parameterized Yocto build¶

Now let's consider more complex example:

desc: "Xilinx BSP"
variables:


  MACHINE: "this will be overwritten by parameters"
components:


  xilinx-bsp:


    sources:


      - type: git


        url: "git://git.yoctoproject.org/poky"


        rev: gatesgarth


      - type: git


        url: "git://git.yoctoproject.org/meta-xilinx"


        rev: gatesgarth


    builder:


      type: yocto


      conf:


        - [MACHINE, "%{MACHINE}"]


      build_target: core-image-minimal


      layers:


        - "../meta-xilinx/meta-xilinx-bsp"


        - "../meta-xilinx/meta-xilinx-standalone"


        - "../meta-xilinx/meta-xilinx-contrib"


      target_images:


        - "tmp/deploy/images/%{MACHINE}/core-image-minimal-%{MACHINE}.tar.gz"


        - "tmp/deploy/images/%{MACHINE}/zImage"


        - "tmp/deploy/images/%{MACHINE}/boot.bin"
parameters:


  MACHINE:


    desc: "Xilinx device for which we will build"


    # Couple of machines are chosen at random just for demonstration purposes


    zc702-zynq7:


      default: true


      overrides:


        variables:


          MACHINE: zc702-zynq7


    zc706-zynq7:


      overrides:


        variables:


          MACHINE: zc706-zynq7


    qemu-zynq7:


      overrides:


        variables:


          MACHINE: qemu-zynq7

This build file allows you to build Xilinx BSP for one of selected devices(zc702-zynq7, zc706-zynq7, qemu-zynq7). Apart from early discussed desc and components sections we can see two new: variables and parameters. But let's take a look at the component. In this case we are fetching two git repositories. Also we are building Yocto distribution now, we have completely different builder with different option. You can see familiar Yocto settings like list of layers on additional local.conf entries. All those sections are described in detail in reference manual.

variables section describes variables. This is basically strings that can be used in any other part of the file using %{VARIABLE_NAME} syntax. We are using % symbol instead of more familiar $ sign to ensure that it will not clash with bitbake's or make's variables. It is really annoying to escape dollar signs in constructions like the following:

conf:


  - [SSTATE_DIR, "${TOPDIR}/../common_data/sstate"]


  - [DL_DIR, "${TOPDIR}/../common_data/downloads"]

Please note that in this particular example topmost variables section can be omitted, because only one variable will be overwritten by subsequent parameters. But, generally you can define variables there. Also, one variable can refer to another:

variables:


  A: "justA"


  B: "%{A}_%{A}" # will be expanded to "justA_justA"

% itself can be escaped by doubling it:

variables:


  A: "justA"


  B: "%%{A}_%%{A}" # will be expanded to "%{A}_%{A}"

parameters section provides means to parameterize your build. If you have such section in your build file, you can get help using moulin itself:

$ moulin xilinx-bsp.yaml --help-config
usage: moulin.py xilinx-bsp.yaml [--MACHINE {zc702-zynq7,zc706-zynq7,qemu-zynq7}]
Config file description: Xilinx BSP
optional arguments:


    --MACHINE {zc702-zynq7,zc706-zynq7,qemu-zynq7}


                          Xilinx device for which we will build

parameters section consist of one or more parameters, and each parameter can have a number of predefined values, one of which must have default flag set. User can select desired parameter variant with command line:

$ moulin xilinx-bsp.yaml --MACHINE qemu-zynq7

All entries from overrides of the chosen parameter will be applied on top of the build configuration. In this example, it will overwrite MACHINE variable with some meaningful value.

Parameterized build with multiple images¶

The next example is the most complex one. It shows the main reason why moulin was written in the first place:

desc: "Renesas Gen3 build with Xen, Dom0 and DomD"
variables:


  MACHINE: "salvator-x-m3-xt"
common_data:


  sources: &COMMON_SOURCES


    - type: git


      url: "/home/lorc/mnt/ssd2/yovrin-test2/src/poky"


      rev: 424296bf9bb4bae27febf91bce0118df09ce5fa1


    - type: git


      url: "/home/lorc/mnt/ssd2/yovrin-test2/src/meta-virtualization"


      rev: 92cd3467502bd27b98a76862ca6525ce425a8479


    - type: git


      url: "/home/lorc/mnt/ssd2/yovrin-test2/src/meta-arm"


      rev: f7c5e7d5094f65d105d9d580ba59527c25fb0d0f


    - type: git


      url: "/home/lorc/mnt/ssd2/yovrin-test2/src/meta-openembedded"


      rev: f2d02cb71eaff8eb285a1997b30be52486c160ae


  conf: &COMMON_CONF


    - [SSTATE_DIR, "${TOPDIR}/../common_data/sstate"]


    - [DL_DIR, "${TOPDIR}/../common_data/downloads"]


    # This is basically xt_shared_env.inc


    # known domains


    - [XT_DIR_REL_DOM0, "dom0"]


    - [XT_DIR_REL_DOMD, "domd"]


    # these are the folders within domain's root filesystem where all


    # installed artifacts live


    - [XT_DIR_ABS_ROOTFS, "/xt"]


    - [XT_DIR_ABS_ROOTFS_DOM0, "${XT_DIR_ABS_ROOTFS}/${XT_DIR_REL_DOM0}/"]


    - [XT_DIR_ABS_ROOTFS_DOMD, "${XT_DIR_ABS_ROOTFS}/${XT_DIR_REL_DOMD}/"]


    - [XT_DIR_ABS_ROOTFS_DOMA, "${XT_DIR_ABS_ROOTFS}/${XT_DIR_REL_DOMA}/"]


    - [XT_DIR_ABS_ROOTFS_DOMF, "${XT_DIR_ABS_ROOTFS}/${XT_DIR_REL_DOMF}/"]


    - [XT_DIR_ABS_ROOTFS_DOMR, "${XT_DIR_ABS_ROOTFS}/${XT_DIR_REL_DOMR}/"]


    - [XT_DIR_ABS_ROOTFS_DOMU, "${XT_DIR_ABS_ROOTFS}/${XT_DIR_REL_DOMU}/"]


    # these are folder names to be used accross domains to install


    # various types of artifacts


    - [XT_DIR_ABS_ROOTFS_SCRIPTS, "${XT_DIR_ABS_ROOTFS}/scripts"]


    - [XT_DIR_ABS_ROOTFS_CFG, "${XT_DIR_ABS_ROOTFS}/cfg"]


    - [XT_DIR_ABS_ROOTFS_DOM_CFG, "${XT_DIR_ABS_ROOTFS}/dom.cfg"]
components:


  dom0:


    # build-dir is optional


    build-dir: shared-build2


    sources:


      - *COMMON_SOURCES


    builder:


      type: yocto


      work_dir: build-dom0


      conf:


        - *COMMON_CONF


        - [MACHINE, "generic-armv8-xt"]


        - ["PREFERRED_PROVIDER_virtual/kernel", "linux-generic-armv8"]


        # Remove ptest to reduce the build time


        - [DISTRO_FEATURES_remove, "ptest"]


        # For virtualization


        - [DISTRO_FEATURES_append, " virtualization"]


        - [DISTRO_FEATURES_append, " xen"]


        # FIXME: normally bitbake fails with error if there are bbappends w/o recipes


        - [SERIAL_CONSOLES, ""]


        # Disable shared link for GO packages


        - [XT_GUESTS_INSTALL, "domu"]


        - [MACHINEOVERRIDES_append, ":%{MACHINE}"]


        - [TUNE_FEATURES_append, " cortexa57-cortexa53"]


      external_src:


        "domd-install-artifacts": "build-domd/tmp/deploy/images/%{MACHINE}/"


      build_target: core-image-thin-initramfs


      layers:


        - "../poky/meta"


        - "../poky/meta-poky"


        - "../poky/meta-yocto-bsp"


        - "../meta-arm/meta-arm-toolchain"


        - "../meta-openembedded/meta-oe"


        - "../meta-openembedded/meta-networking"


        - "../meta-openembedded/meta-python"


        - "../meta-openembedded/meta-filesystems"


        - "../meta-virtualization"


        # Use inner layers of meta-xt-images and meta-xt-prod-devel


        - "../meta-xt-images/recipes-dom0/dom0-image-thin-initramfs/files/meta-xt-images-extra"


        - "../meta-xt-images/recipes-domx/meta-xt-images-domx/"


        - "../meta-xt-images/machine/meta-xt-images-generic-armv8"


        - "../meta-xt-prod-devel/recipes-dom0/dom0-image-thin-initramfs/files/meta-xt-prod-extra"


      target_images:


        - "tmp/deploy/images/salvator-x/core-image-minimal-salvator-x.ext4"


        - "tmp/deploy/images/salvator-x/bl2-salvator-x.bin"


      # Dependencies from other layers (like domd kernel image for example)


      additional_deps:


        - "build-domd/tmp/deploy/images/%{MACHINE}/Image"


  domd:


    # build-dir is optional


    build-dir: shared-build2


    sources:


      - *COMMON_SOURCES


      - type: git


        url: "/home/lorc/mnt/ssd2/yovrin-test2/src/meta-clang"


        rev: e63d6f9abba5348e2183089d6ef5ea384d7ae8d8


      - type: git


        url: "/home/lorc/mnt/ssd2/yovrin-test2/src/meta-python2"


        rev: c96cfe30701ba191903c5f7d560c3ba667d46c9d


      - type: git


        url: "/home/lorc/mnt/ssd2/yovrin-test2/src/meta-renesas"


        rev: c0a59569d52e32c26de083597308e7bc189675dd


      - type: git


        url: "/home/lorc/mnt/ssd2/yovrin-test2/src/meta-selinux"


        rev: 7af62c91d7d00a260cf28e7908955539304d100d


      - type: git


        url: "/home/lorc/mnt/ssd2/yovrin-test2/src/meta-xt-prod-devel"


        rev: "REL-v6.0"


      - type: git


        url: "/home/lorc/mnt/ssd2/yovrin-test2/src/meta-xt-images"


        rev: "REL-v6.0"


    builder:


      type: yocto


      work_dir: build-domd


      conf:


        - *COMMON_CONF


        - [MACHINE, "%{MACHINE}"]


        - [PREFERRED_VERSION_u-boot_rcar, "v2020.01%"]


        # override console specified by default by the meta-rcar-gen3


        # to be hypervisor's one


        - [SERIAL_CONSOLES, "115200;hvc0"]


        - [XT_GUESTS_INSTALL, "domu"]


      build_target: core-image-weston


      layers:


        - "../poky/meta"


        - "../poky/meta-poky"


        - "../poky/meta-yocto-bsp"


        - "../meta-renesas/meta-rcar-gen3"


        - "../meta-arm/meta-arm-toolchain"


        - "../meta-openembedded/meta-oe"


        - "../meta-openembedded/meta-networking"


        - "../meta-openembedded/meta-python"


        - "../meta-openembedded/meta-filesystems"


        - "../meta-selinux"


        - "../meta-virtualization"


        - "../meta-clang"


        - "../meta-python2"


        # Use inner layers of meta-xt-images and meta-xt-prod-devel


        - "../meta-xt-images/recipes-domd/domd-image-weston/files_rcar/meta-xt-images-extra"


        - "../meta-xt-images/recipes-domx/meta-xt-images-domx/"


        - "../meta-xt-images/recipes-domx/meta-xt-images-vgpu/"


        - "../meta-xt-images/machine/meta-xt-images-rcar-gen3"


        - "../meta-xt-prod-devel/recipes-domd/domd-image-weston/files/meta-xt-prod-extra"


      target_images:


        - "tmp/deploy/images/%{MACHINE}/Image"
parameters:


  # Prebuilt DDK


  USE_PREBUILT_DDK:


    "no":


      default: true


      overrides:


        components:


          domd:


            sources:


              - type: git


                url: "ssh://git@gitpct.epam.com/epmd-aepr/img-proprietary"


                rev: "ef1aa566d74a11c4d2ae9592474030a706b4cf39"


                dir: "proprietary"


            builder:


              conf:


                - [PREFERRED_PROVIDER_gles-user-module, "gles-user-module"]


                - [PREFERRED_VERSION_gles-user-module, "1.11"]


                - [PREFERRED_PROVIDER_kernel-module-gles, "kernel-module-gles"]


                - [PREFERRED_VERSION_kernel-module-gles, "1.11"]


                - [PREFERRED_PROVIDER_gles-module-egl-headers, "gles-module-egl-headers"]


                - [PREFERRED_VERSION_gles-module-egl-headers, "1.11"]


                - [EXTRA_IMAGEDEPENDS_append, " prepare-graphic-package"]


    "yes":


      overrides:


        components:


          domd:


            builder:


              conf:


                - [XT_RCAR_EVAPROPRIETARY_DIR, "./"]


                - [PREFERRED_PROVIDER_virtual/libgles2, "rcar-proprietary-graphic"]


                - [PREFERRED_PROVIDER_virtual/egl, "rcar-proprietary-graphic"]


                - [PREFERRED_PROVIDER_kernel-module-pvrsrvkm, "rcar-proprietary-graphic"]


                - [PREFERRED_PROVIDER_kernel-module-dc-linuxfb, "rcar-proprietary-graphic"]


                - [PREFERRED_PROVIDER_kernel-module-gles, "rcar-proprietary-graphic"]


                - [PREFERRED_PROVIDER_gles-user-module, "rcar-proprietary-graphic"]


                - [PREFERRED_PROVIDER_gles-module-egl-headers, "rcar-proprietary-graphic"]


                - [BBMASK_append, " meta-xt-images-vgpu/recipes-graphics/gles-module/"]


                - [BBMASK_append, " meta-xt-prod-extra/recipes-graphics/gles-module/"]


                - [BBMASK_append, " meta-xt-prod-vgpu/recipes-graphics/gles-module/"]


                - [BBMASK_append, " meta-xt-prod-vgpu/recipes-graphics/wayland/"]


                - [BBMASK_append, " meta-xt-prod-vgpu/recipes-kernel/kernel-module-gles/"]


                - [BBMASK_append, " meta-xt-images-vgpu/recipes-kernel/kernel-module-gles/"]


                - [BBMASK_append, " meta-renesas/meta-rcar-gen3/recipes-kernel/kernel-module-gles/"]


                - [BBMASK_append, " meta-renesas/meta-rcar-gen3/recipes-graphics/gles-module/"]


  # Machines


  MACHINE:


    salvator-x-m3-xt:


      default: true


      overrides:


        variables:


          MACHINE: "salvator-x-m3-xt"


    salvator-x-h3-xt:


      overrides:


        variables:


          MACHINE: "salvator-x-h3-xt"


    h3ulcb-4x2g-kf-xt:


      overrides:


        variables:


          MACHINE: "h3ulcb-4x2g-kf-xt"


        components:


          builder:


            dom0:


              conf:


                - [MACHINEOVERRIDES_append, ":kingfisher"]


            domd:


              sources:


                - type: git


                  url: "/home/lorc/mnt/ssd2/yovrin-test2/src/meta-rcar"


                  rev: a99eb54e9b068375b306fec53f1530f7eb780014


              builder:


                layers:


                  - "../meta-rcar/meta-rcar-gen3-adas"


                conf:


                  #FIXME: patch ADAS: do not use network setup as we provide our own


                  - [BBMASK "meta-rcar-gen3-adas/recipes-core/systemd"]


                  # Remove development tools from the image


                  - [IMAGE_INSTALL_remove " strace eglibc-utils ldd rsync gdbserver dropbear opkg git subversion nano cmake vim"]


                  - [DISTRO_FEATURES_remove " opencv-sdk"]


                  # Do not enable surroundview which cannot be used


                  - [DISTRO_FEATURES_remove " surroundview"]


                  - [PACKAGECONFIG_remove_pn-libcxx "unwind"]

This is an example of real build configuration file. It is still under development, so unlike two previous examples, you would not be able to use this example to make a build. We would not cover it in detail, just give you the list of highlights:

Invoking moulin¶

moulin has one mandatory parameter - file name of build description. It should be in YAML format. You may use a regular local file or URL. moulin detects URL by the presence of a protocol prefix, like https://. If you use the URL to GitHub or another network repository, you can use the URL for the raw file only, not for a web page with that file. For example, this URL points to the correct YAML file: https://raw.githubusercontent.com/xen-troops/meta-xt-prod-devel-rcar/master/prod-devel-rcar.yaml. But the following URL can't be used, because it points to GitHub's web page: https://github.com/xen-troops/meta-xt-prod-devel-rcar/blob/master/prod-devel-rcar.yaml. Pay attention, that file will be downloaded only if a file with the same name doesn't exist in the current folder. This is done to preserve possible local changes made by a user.

As a result, moulin will generate ninja.build file. You can then invoke ninja to perform the actual build. moulin adds generator rule in ninja.build, so it is not mandatory to invoke moulin after you made changes into your YAML file. Ninja will detect any changes to this file and invoke moulin automatically to re-create ninja.build.

If YAML file contains parameters key, it is possible to invoke moulin with additional command line options. Set of this options depends of contents of YAML file and can be viewed using --help-config command line option.

Verbose output¶

Use -v or --verbose command line option to increase verbosity of output. With this option enabled moulin will give more information about what it is doing.

Dumping intermediate state¶

--dump command line option can be used to force moulin to dump intermediate state of processed YAML file. You will see contents of your build config after applying all parameters and expanding all variables. This can come in help during debugging of your build, because you can see what exactly is passed to fetchers and builders.

Internal command line options¶

There is --fetcherdep command line option which is internal, and it is even hidden from -h output. It is used by moulin to generate dynamic dependency files for Ninja, so Ninja can track changes inside components.

This option is not meant to be used by a user.

YAML sections¶

YAML file should consist of number of pre-defined keys/sections which are discussed below. Any unknown keys are ignored. Right now only the following top-level keys are supported:

Minimal Version¶

Optional min_ver section should hold minimal required version of moulin. This is a text string that conforms PEP-440. For example min_ver: "0.2". moulin will compare this with own version and will stop if required version is newer.

Mandatory sections: "desc" and "components"¶

There are only two mandatory sections: desc and components. desc should contain text string that describes the build. It is displayed when moulin invoked with --help-config command line option.

components should contain dictionary where key is a component name and value is an another dictionary with component settings:

components:


    component1:


        build-dir: "component-build-dir" # Optional


        default: true # Optional


        sources:


            ......


        builder:


            .....


    component2:


        sources:


            ......


        builder:


            .....

There are two main parts of each component description: sources and builder.

sources is optional and can contain list of source code definition, which will be fetched prior starting a build:

sources:


 - type: git


   url: "git://git.yoctoproject.org/poky"


   rev: gatesgarth


 - type: repo


   url: https://github.com/xen-troops/android_manifest.git


   rev: android-11-master


   manifest: doma.xml

All supported fetchers are listed in Fetchers section.

builder contains build configuration for a given component. There are multiple builder types supported. They are described in Builders section.

Apart from two mandatory options, component description can contain following optional keys:

Variables¶

variables section is optional. It can contain dictionary of variable's name-value pairs:

variables:


  A: "a"


  B: "1%{A}%{A}" # will be expanded to "1aa"


  C: "2%{B}%{B}" # will be expanded to "21aa1aa"

Variables can be used anywhere in the YAML file. During internal pre-processing all variable references in form of %{variable_name} will be replaced with actual variable value.

% is a special symbol. It can be escaped by doubling it: %%.

Variables should be used to decrease amount of hard-coded values. Good candidates that should be moved to variables are path names, branches, hardware identifiers, etc.

Parameters¶

Often it is desired to have some options for a build. For example one can want to support a number of different HW boards, or to enable additional features. It would be not feasible to have separate YAML for every board-feature combination. This is where parameters come to help. All parameters should be stored in parameters section:

parameters:


  parameter1:


    desc: "parameter 1 description"


    option1:


      default: true


      overrides:


        ...


    option2:


      overrides:


        ...


    option3:


      overrides:


        ...


  parameter2:


    desc: "parameter 2 description"


    option1:


      overrides:


        ...


    option2:


      overrides:


        ...


    option3:


      default: true


      overrides:


        ...

Every parameter should include mandatory desc key. Parameter can have one or more options, one of option should have default flag enabled.

Main part of each option is the overrides section. Contents of this section should correspond to top-level layout of YAML file. All contents of this section will be overlaid on contents of YAML file during pre-processing stage. Rules of this process are:

Basically, this rules follow the intuitive idea of extending/overwriting original config: primitive values will be overwritten, all other values will be extended.

User can chose parameter's options using command line arguments, as described in Invoking moulin section.

Fetchers¶

Fetchers are the moulin plugins responsible for downloading of sources listed in sources section of a component.

moulin will generate phony Ninja target fetch-{component_name} for every component. It can be used to just fetch sources without building anything.

git fetcher¶

git fetcher is used to download code from a remote or local git repositories. There is a full list of supported parameters:

type: git # Selects `git` fetcher
url: "url://for.repository/project.git"
rev: revision_name
dir: "directory/where/store/code"

repo fetcher¶

repo fetcher is used to download code using Google's repo tool. Full list of supported options:

type: repo # Selects `repo` fetcher
url: https://manifest.address/repo.git
rev: manifest-revision
manifest: manifest-file.xml
depth: 1
groups: all
dir: "."

unpack fetcher¶

unpack fetcher is used to unpack already available archives to a specified directory. Example use-case is when need to use 3rd-party code/resources that are not available in git repository. Full list of supported options:

type: unpack # Selects `unpack` fetcher
archive_type: tar
file: my_file.tar.gz
dir: "."

Right now unpack fetcher supports two archive types: tar and zip.

west fetcher¶

west fetcher is used to download code using Zephyr's west meta-tool. Full list of supported options:

type: west # Selects `west` fetcher
url: https://manifest.address/zephyr
rev: manifest-revision
file: manifest-file.yml

For additional details see documentation on west init: https://docs.zephyrproject.org/latest/develop/west/built-in.html#west-init

Regarding installation of west, please see: https://docs.zephyrproject.org/latest/develop/west/install.html

Builders¶

Builders are the moulin plugins responsible for actual image building.

moulin will generate phony Ninja target {component_name} for every component. It can be used to build certain component. Please note that this will not build only given component. Any prerequisites will be fetched and build as well.

Builder configuration heavily depends on builder type and is described in next subsections.

yocto builder¶

Yocto builder is used to build OpenEmbedded-based images. It expects that poky repository is cloned in {build_dir}/poky and uses it's poky/oe-init-build-env script to initialize build environment. Then bitbake-layers tool is used to add additional layers and bitbake used to perform the build.

builder:


  type: yocto       # Should be `yocto`


  work_dir: "build" # Optional


  build_target: core-image-minimal # Mandatory


  conf:             # Mandatory


    - [MACHINE, "machine-name"]


    - [DISTRO_FEATURES_remove, "feature_to_remove"]


    - [DISTRO_FEATURES_append, "feature_to_add"]


  layers:           # Optional


    - "../poky/meta-yocto-bsp"


    - "../meta-other-layer/"


  external_src:     # Optional


    "package-name": "path-to-package-sources"


    "another-package-name": ["path part1", "path part2", "path part3"]


  target_images:    # Mandatory


    - "tmp/deploy/images/machine-name/Image"


  additional_deps:  # Optional


    - "path/to/file/generated/by/other/component"

Mandatory options:

Optional parameters. Those provide advanced features that may be needed if you are building multiple VMs with cross-dependencies.

android builder¶

Android builder is used to build Android Open Source Project (AOSP). It expects that AOSP is present in build directory. In most cases AOSP is cloned using repo fetcher.

builder:


  type: android # Should be 'android'


  env:          # Optional


    - "TARGET_BOARD_PLATFORM=r8a7795"


  lunch_target: xenvm-userdebug


  target_images:


    - "out/xenvm/userdebug/boot.img"


    - "out/xenvm/userdebug/system.img"


  additional_deps:  # Optional


    - "path/to/file/generated/by/other/component"

Mandatory options:

Optional parameters:

android_kernel builder¶

Android Kernel builder is used to build kernel and kernel modules for Android Open Source Project (AOSP). It expects that correct directory layout is present in build directory. In most cases AOSP is cloned using repo fetcher.

builder:


  type: android_kernel # Should be 'android_kernel'


  env:                 # Optional


    - "TARGET_BOARD_PLATFORM=r8a7795"


    - "BUILD_CONFIG=common/build.config.xenvm"


    - "SKIP_MRPROPER=1"


  target_images:


    - "out/android12-5.4/common/arch/arm64/boot/Image"

Mandatory options:

Optional parameters:

archive builder¶

Archive builder does is intended to create archive from other components. It can be used to gather build artifacts, for example. This builder uses tar to create archive files. Archives can be optionally compressed as, tar is invoked with --auto-compress option.

builder:


  type: archive        # Should be 'artchive'


  name: "artifacts.tar.bz2"


  items:


    - "yocto/build/tmp/deploy/images/generic-armv8-xt/Image"


    - "yocto/build/tmp/deploy/images/generic-armv8-xt/uInitramfs"

Mandatory options:

zephyr builder¶

This builder is used to build applications based on Zephyr OS. It uses Zephyr OS meta-tool west. Required code is expected to be fetched by west fetcher.

builder:


  type: zephyr


  board: xenvm


  shields:          # Optional


    - "shield1"


    - "shield2"


  target: samples/synchronization


  work_dir: build_dir


  target_images:


    - "zephyr/build/zephyr/zephyr.bin"


  env:


    - "MY_ENV_VAR=my_value"


  additional_deps:  # Optional


    - "path/to/file/generated/by/other/component"

Mandatory options:

Optional parameters:

Please note that this builder uses --pristine=auto command-line option.

Proper versions of CMake and Zephyr SDK have to be installed on the host.

For additional details please see https://docs.zephyrproject.org/latest/develop/west/build-flash-debug.html#building-west-build

About¶

rouge is a companion tool for moulin. Its purpose is to simplify creation of bootable images. It can create partition table, fill partitions with predefined files or raw data. It supports GPT, ext4fs, raw images, Android sparse images. Further formats can be added if needed.

Right now it can be used only as a separate tool, but there are plans to integrate it into moulin output.

Design Principles¶

rouge shares ideas (and code) with moulin. Thus, it is very similar in configuring and invoking to moulin. It can be used as a stand-alone tool: just provide only images: section in your YAML file. Or you can include this section into the same file, which is used by moulin to share common options or variables. In latter case moulin will generate additional image-{image_name} rules, so you can build images with Ninja.

Requirements¶

To do its job rouge invokes number of external utilities. Most of them are available on every system, except Android tools that usually should be installed separately. There is a list:

Invoking rouge¶

rouge uses the same design ideas as moulin, and part of the command line options are shared with moulin. This includes --help-config, -v and --dump arguments. Please refer to moulin documentation for more details. This document describes only argument specific to rouge.

rouge [-h] [--help-config] [-v] [--dump] [-l] [-f] [-s] [-o FILE]


             [-l | -i image_name]


             build.yaml image_name

Apart from this options, rouge will read and parse all YAML-file related parameters in the same way as moulin does. You can check available parameters with --help-config.

Principles of Operation¶

rouge works in a very simple way. It uses moulin's YAML processor that applies parameters and substitutes variables, then reads images: section, finds requested image specification.

For a given image it checks if all mentioned files are present, then calculates sizes of partitions (if any) and total image size. Then it writes data to a given file/block device according to the specifications.

rouge tries to use sparse files whenever possible. Sparse file is a file with "holes" in it. It allows you to have a huge file that represents whole disk image with tiny bit of actual information in it. This speeds up image creation process and decreases used disk space. If you are writing resulting image file to your SD card manually, try adding conv=sparse option to your dd command line. This will speed up the writing process. If you want to distribute resulting images, take a look at Intel's bmap tool. It allows you to share sparse files across devices.

YAML Sections¶

Shared sections¶

rouge uses the same YAML processing code as moulin so refer to moulin's documentation for the desc, min_ver, variables, parameters sections description. This page describes only parts specific to rouge.

Image specifications¶

Images are specified in the following way:

images:


  image_name_a:


    desc: "Description for the first image"


    image_size: 512 MiB


    type: gpt


    ... block description ...


  image_name_b:


    desc: "Description for the second image"


    type: raw_image


    ... block description ...


  image_name_c:


    desc: "Description for the third image"


    type: empty


    ... block description ...

images: section contains one or more keys, which serve as image names. Every image can have description, which will be displayed when rouge lists available images. type: key is mandatory as it defines type of block. Supported block types as described in the following sections.

Also you may specify the required size of image using image_size:. Please see section 'Size Designation' below for supported notation. If actual size of all partitions will be less than image_size: then image will be blown up to image_size:. If actual size is bigger than specified - error will be printed with explanation like "Actual size (20000) of image is bigger than requested one (10000)."

Block descriptions¶

"Block" is a basic rouge entity that describes one partition or partition table. Some block types can be nested. Supported block types are described below.

Size Designation¶

All block have size parameter. For some block types this parameter is mandatory, for some - optional. Basic unit for size is byte. For example

type: empty
size: 4096

defines empty block with size of 4096 bytes. rouge supports some SI suffixes:

Suffix must be separated from number by space. For example: size: 4 MiB defines size of 4 mebibytes or 4 194 304 bytes.

Empty block¶

Empty block is a block that does not contain any file or raw image. rouge will write nothing into this block if filled: zeroes option is not specified.

type: empty # defines empty block
size: 4096
filled: zeroes

size is mandatory, as rouge can't infer it.

filled is optional, with only zeroes value allowed for now. This option may be used if you need the block to be filled with zeroes. For example, this is used for some Android partitions, like 'rpmbemul'. Use this option only if you really need to. Otherwise you will needlessly increase size and upload time of an image.

Raw Image Block¶

Purpose of this block type is to include any binary data from other file. For example, if your build system creates .ext4 image with root file system, you can use this block to place that image into GPT partition (which is described below).

type: raw_image # defines raw image block
size: 400 MiB
image_path: "some/path/rootfs.ext4"

image_path is mandatory. This is a file to be included into resulting image.

size is optional. If it is omitted, rouge will use size of file. If provided size is smaller than file size, rouge will stop with an error. Thus, you can create block that is bigger than file, but not smaller.

Android Sparse Image Block¶

It is similar to Raw Image Block, but it handles files in Android Sparse image format.

type: android_sparse # defines android sparse block
size: 3000 MiB
image_path: "android/out/target/product/xenvm/userdata.img"

image_path is mandatory. This is a file to be included into resulting image. rouge will call simg2img2 tool to unpack it before writing it to a resulting image.

size is optional. If it is omitted, rouge will use data size, read from the file. If provided size is smaller than read size, rouge will stop with an error. Thus, you can create block that is bigger than unpacked file, but not smaller.

Filesystem Image With Files¶

This block types allows you to create new filesystem with some files included from your disk. This is ideal for creating boot partitions, where you store kernel, initial ramdisk and so on.

type: ext4 # defines ext4 partition block
size: 30 MiB
items:


  "remote_file1": "path/to/local/file1"


  "remote_file2": "path/to/local/file2"


  "remote_file3": "path/to/local/file3"


  "remote_file4": "path/to/local/file4"


  "remote_dir": "path/to/local/directory/"

type is required. Defines the filesystem type, currently ext4 and vfat are supported.

items: section is optional. It defines remote:local mapping of files that should be presented on newly created filesystem. remote part is how the file will be named on new filesystem, while local is a path on your disk. You can specify parent folders for remote and these folders will be created on the destination filesystem. You may specify not only files but directories also. If the local directory contains subdirectories, they will be created unders the remote directory. Older versions of rouge used files: as the name of the section. This name is still possible to use, but it is deprecated. Also only items: can contain directories.

size is optional. rouge will calculate total file size and add some space for the filesystem metadata to determine block size. You can increase size, if wish.

GUID Partition Table (GPT) block¶

This block type defines GPT along with all partitions. In most cases this is will be your top-level block definition. It can (and should) include other blocks, including other GPT. Inner GPT can come in handy in cases when you are creating image that holds data for multiple virtual machines and wish to provide VM with own GPT.

type: gpt # defines GPT block
partitions:


  boot: # partition label


    gpt_type: 21686148-6449-6E6F-744E-656564454649 # BIOS boot partition (kinda...)


    gpt_guid: 8DA63339-0007-60C0-C436-083AC8230900 # Partition GUID


    type: empty


    size: 30 MiB


  rootfs:


    gpt_type: B921B045-1DF0-41C3-AF44-4C6F280D3FAE # Linux aarch64 root


    type: raw_image


    image_path: "rootfs.ext4"

This example defines GPT with two partitions: boot and rootfs. boot is empty block and rootfs includes Raw Image block.

partitions: section is mandatory. It defines list of partitions, where key is a partition label.

Each partition contains definition of other block type plus optional keys:

gpt_type: (which we strongly suggest to provide) key holds GPT Partition Type GUID. List of widely used types can be found on Wikipedia, for example.

gpt_guid: key sets the GPT Partition GUID. By default this GUID is generated automatically to ensure that every partition in the world would have unique identifier. But there are some cases when external software depends on exact value of a partition GUID. In such cases it is possible to hard-code this value. We strongly recommend not to use this key except for the cases when this is neccessary because, accoding to the page 121 of Specification the software that makes copied of GPT-formatted disks and partitions must generate new Unique Partition GUID in each GPT Partition Entry.

rouge will place partitions one after another, aligning partition start to 1 MiB (as per standard recommendation) and partition size to sector size, which defaults to 512 bytes.

Examples¶

The following example provides multiple different images:

min_ver: 0.3
desc: "rouge sample images"
images:


  empty_image:


    desc: "Just empty 32MB file"


    type: empty


    size: 32 MiB


  unpacked_userdata:


    desc: "Unpacked android userspace image"


    type: android_sparse


    image_path: "android/out/target/product/xenvm/userdata.img"


  simple_bootable_sd:


    type: gpt


    desc: "Full SD-card/eMMC image"


    partitions:


      boot:


        gpt_type: 21686148-6449-6E6F-744E-656564454649 # BIOS boot partition (kinda...)


        type: ext4


        size: 30 MiB


        items:


          "Image": "yocto/build/tmp/deploy/images/generic-armv8-xt/Image"


          "initrd": "yocto/build/tmp/deploy/images/generic-armv8-xt/uInitrd"


      domd_rootfs:


        gpt_type: B921B045-1DF0-41C3-AF44-4C6F280D3FAE # Linux aarch64 root


        gpt_guid: 8DA63339-0007-60C0-C436-083AC8230900 # Partition GUID


        type: raw_image


        image_path: "yocto/build-domd/tmp/deploy/images/machine/core-image-weston.ext4"

Component-specific targets¶

As mentioned in chapters "Fetchers" and "Builders" of "User reference manual", moulin generates some helpful targets for components: to fetch sources, to create a configuration, to build the component. For example, if we have component domu, we will have following self-explanatory targets:

Rouge-specific targets¶

If section images: is present in YAML file, moulin will also generate handy image-{image_name} rules. They can be used to invoke rouge with the same build options, as moulin was invoked.

Moulin also generates {image_name}.img.gz and {image_name}.img.bmap targets.

Build inside yocto¶

If your component has yocto builder and you need to work inside the components's build environment then you need to specify corresponding build folder (see work_dir parameter in builder section of required component) as a parameter to oe-init-build-env. For example to build only kernel-module-gles in domd:

$ cd yocto
$ . poky/oe-init-build-env build-domd
$ bitbake kernel-module-gles