Transition to Ansible collections with the help of git submodule

Ansible collections - a new way to package your content

When planning how to organize and distribute Ansible content, we have to get familiar with the (not so new) concept of Ansible collections. Ansible collections can include plugins, modules, filters, roles and more in one gzipped tar archive. Although Ansible collections are created in a very similar way to roles, a recurring question is: How do I efficiently transition to the new content structure?

In this blog post I want to quickly guide you through the process of how to put existing roles into a single Ansible collection by using the git submodule feature, how to use the collection in your playbook and will give you a simple example on how to automatically build Ansible collections on GitLab using their CI/CD capabilities.

In this scenario I’m assuming you already know Ansible and the concept of roles and collections, and will therefore skip the basics. Let’s assume you have a bunch of existing roles and you want to transition over to using Ansible collections with as little effort as possible.

If you want to learn more about Ansible collections and do a full exercise, I suggest to have a look at the Ansible collections workshop.

Prerequisites

To better illustrate how to transition from roles to collections, I setup a bunch of git repositories:

ansible-lamp: provides two playbooks to better explain the concept.
ansible-lamp-apache: example role setting up an Apache Web Server
ansible-lamp-mariadb: example role setting up a MariaDB Database Server
ansible-lamp-php: example role setting up PHP
ansible-lamp-collection: an Ansible collection utilizing the roles ansible-lamp, ansible-mariadb and ansible-php

Note: All these roles are only used for learning purposes and should not be used in a production scenario and will probably not even work for you.

A word on git submodules

Git allows you to have git repositories inside of existing git repositories. This is a very powerful feature which makes it easy to embed an existing repository without the need to manually sync the content.

To add an existing git repository to a sub directory, you can use the following example:

git submodule add <repository URL> <local directory>

This will add a git clone of the specified repository into the directory. Check out the man page or the more detailed section on git submodules in the official Git Book.

When running git pull to update your local repository, this will not check for updates in the submodules. Make sure to use git submodule update --remote to also check for new changes in the submodules. You can also update specific sub modules:

git submodule update --remove <local directory of submodule>

Fully Qualified Collection Name

A Fully Qualified collection Name (FQCN) is built from three components:

the namespace, which is typically the name of the author or organization creating and maintaining the collection
the name of the collection
the name of the module, role, plugin etc. inside the collection

To reference a module, plugin, role etc. you should use the full name to avoid name clashes. Here are a few examples:

1ansible.posix.selinux # Configures the SELinux mode and policy.
2ansible.builtin.debug # This module prints statements during execution and can be useful for debugging variables or expressions without necessarily halting the playbook. (same as the "debug" module in Ansible <= 2.9)
3ansible.builtin.dnf # Installs, upgrade, removes, and lists packages and groups with the `dnf' package manager. (same as "dnf" module in Ansible <= 2.9)

Create the Ansible collection

Let’s first initialize a basic Ansible collection structure. ansible-galaxy can do that for us:

ansible-galaxy collection init <namespace>.<collection name>

In the example below the namespace is cjung and the name of the collection is lamp (since the collection will provide a simple LAMP stack):

ansible-galaxy collection init cjung.lamp

Note: You probably want to use your own name as author.

This creates a directory structure which should look like this:

# tree
.
└── cjung
    └── lamp
        ├── docs
        ├── galaxy.yml
        ├── plugins
        │   └── README.md
        ├── README.md
        └── roles

5 directories, 3 files

Working with dependencies

Ansible collections can have dependencies as well, similar to roles. This can be helpful when reorganizing the roles. Let’s say we have three roles:

Web Server Role
Database Role
General System Hardening Role

Both the Web Server and the Database role will be migrated to a collection. Since they both rely on the hardening role, we could add it to both collections - but that wouldn’t be very efficient and introduce additional effort in maintaining the content. Instead we should create a hardening collection which is defined as a dependency. It’s very simple to do that, by just adding the list of dependencies in our galaxy.yml file.

1dependencies: {
2  - cjung.hardening # this is just an example, this collection does not exist. ansible-galaxy install will try to find a collection with this name automatically
3  - https://gitlab.com/cjung/ansible-lamp-collection/-/raw/master/cjung-lamp-1.0.0.tar.gz
4}

When installing the collection with ansible-galaxy collection install the dependencies will automatically be installed as well.

Add existing Roles to the Collection

As you can see, the Ansible roles are supposed to go into the directory cjung/lamp/roles/. To make our life easier, we can use Git Submodules to add our existing roles.

git submodule add https://gitlab.com/cjung/ansible-lamp-apache.git cjung/lamp/roles/lamp_apache
git submodule add https://gitlab.com/cjung/ansible-lamp-mariadb.git cjung/lamp/roles/lamp_mariadb
git submodule add https://gitlab.com/cjung/ansible-lamp-php.git cjung/lamp/roles/lamp_php

After adding the submodules, make sure to commit your changes.

Use the Ansible Collection in your Playbook

It’s good practice to specify the roles and collections used in your playbook as requirements. If you’re planning to use Red Hat Ansible Tower or AWX, specifying the requirements is a must. Otherwise they will not be automatically installed before running the playbook.

The syntax for a collection requirements file looks like this:

1---
2collections:
3# point to a collection archive
4- https://gitlab.com/cjung/ansible-lamp-collection/-/raw/master/cjung-lamp-1.0.0.tar.gz
5# point to a collection on Ansible Galaxy
6- awx.awx

You can then easily install the collection automatically:

1ansible-galaxy collection install -r collections/requirements.yml

Note: The name of the file has to be collections/requirement.yml relative of your project root directory, to allow Ansible Tower or AWX to automatically install it for you. Also make sure to remove the role from your roles/requirements.yml - or better delete the file entirely, if no longer needed.

Update your Playbooks

Since you’re now using an Ansible collection for your roles, you have to update your playbooks accordingly. This is easier than it might sound.

The shortcut

To make the transition easier, you can use the collections keyword in your roles or playbook, This will tell Ansible to search for the roles (modules, plugins etc.) in the specified Ansible collections.

You simply list all the collections you need, in your playbook:

1- hosts: all
2  collections:
3    - my_namespace.my_collection

Or in your role:

1# myrole/meta/main.yml
2collections:
3  - my_namespace.first_collection
4  - my_namespace.second_collection
5  - other_namespace.other_collection

This is a shortcut to get you started more quickly, but it is not the recommended solution. Instead of using the collections keyword, you should update your Ansible roles and playbooks to use the FQCN.

If you’re using the collections keyword, you rely on how Ansible internally searches for content. Problems can occur if the name of a role or module is found in several collections and is therefore not unique. This can lead to unpredictable results when running your Ansible playbook.

You can avoid this kind of problem by using the FQCN syntax as described in the following chapter.

The proper way

When adding Ansible collection support, a new naming syntax for Ansible modules, roles, plugins, etc. was introduced. This is called the Fully Qualified Collection Name, or FQCN for short.

You can use the Apache role in your playbook like this:

1---
2- name: setup a lamp server
3  hosts: lamp_server
4  become: true
5  roles:
6  - lamp_apache

After switching to a collection, it becomes:

1---
2- name: setup a lamp server
3  hosts: lamp_server
4  become: true
5  roles:
6  - cjung.lamp.lamp_apache

As you can see, the namespace and collection name is put in front of the role name. This concludes all the necessary steps to migrate from Ansible roles to collections.

CI/CD Pipeline

This section is for advanced readers who want to use CI/CD pipelines to automatically build new Ansible collections every time a change is pushed to the Git repository.

If you’re using many different Ansible roles, rebuilding the Ansible collection manually can become a drag. I’m using GitLab quite heavily and built a simple CI/CD pipeline to automate the process.

 1# this is a container image with Ansible and the python modules for AWS, Azure and Google Cloud
 2image: registry.gitlab.com/cjung/podman-examples/ansible-cloud
 3
 4stages:
 5  - check
 6  - build
 7
 8build:
 9  stage: build
10  before_script:
11  # configure git with the variables from the current user profile
12  - git config --global user.name "${GITLAB_USER_NAME}"
13  - git config --global user.email "${GITLAB_USER_EMAIL}"
14  script:
15  # build the actual Ansible collection found in the sub directory
16  - ansible-galaxy collection build cjung/lamp/
17  # add the new file
18  - git add .
19  # commit the file to the git repository - the "skip ci" keyword will make sure you're not creating an endless CI/CD loop
20  - git commit -m "[skip ci] automated build of ansible collection"
21  # push the actual changes
22  - git push "https://${GITLAB_USER_NAME}:${CI_GIT_TOKEN}@${CI_REPOSITORY_URL#*@}" HEAD:master
23  only:
24  - master
25
26# Optionally run ansible-lint on each role to find syntax errors or other issues
27lint-apache:
28  stage: check
29  script:
30  - ansible-lint -r roles/lamp_apache/
31
32lint-mariadb:
33  stage: check
34  script:
35  - ansible-lint -r roles/lamp_mariadb/
36
37lint-php:
38  stage: check
39  script:
40  - ansible-lint -r roles/lamp_php/

To really make this work, you have to set the following variables in your GitLab CI/CD configuration:

CI_GIT_TOKEN: you have to create a personal access token in GitLab and add it as a variable. This is necessary to automatically push a new version of the Ansible collection to the git repository.
GIT_SUBMODULE_STRATEGY: recursive - to make sure GitLab does also clone the git submodules repositories, the variable has tobe set to recursive.

Now you can run the GitLab pipeline manually or push a change into the repository to trigger an automated build. You can also use GitLab hooks to trigger a build when changes are pushed to the individual roles.