cfbs add surf-cfengine-library
The SCL distribution consists of a library and services part. The services are build with the SCL building blocks that can be controlled/configured with json data. The SCL building blocks are, e.g.:
The goal is to setup a library were we can easily install/configure/maintain services. There are many services included and these are used at SURF for our HPC clusters and Office Automation. We hope that this is also useful for others and that will grow as the standard repo for cfengine services. In Ansible they call it playbooks and in Salt terms formulas.
The project started as library for generating configuration files with mustache/json. In our CFengine configuration we used different methods and we wanted a standard method for generating these configuration files. This is the standard that we defined for the mustache/json files and json merging:
$(def.node_template_dir)
templates/scl/$(service_name)
$(def.node_template_dir)/$(service_name)
$(service_name)[json_files]
$(def.node_template_dir)/$(service_name)
$(service_name)[local_generated_json_files]
Both scenarios will be described in the subsection below. For both scenarios you can specify multiple json files. The files will be merged and the last one wins if the same variable name is used, e.g.:
a : 1
a : 2
If the order is { "b.json", "a.json" }
the value of a would be 1
This framework depends on the augments file def.json
, we have developed a multiple augments strategy:
The merge strategy is:
default.json
def.<service_name>_json_files
if defineddef.<service_name>[json_files]
if defineddef.<service_name>_local_generated_json_files
if defineddef.<service_name>[local_generated_json_files]
if defineddef.<service_name>
if defined in def.json or:
def
At our site we have multiple augments files defined:
With scl you can merge at 3 levels
ntp_json_files
(domain|os.json).ntp: {
json_files: [ role.json ]
server: [ ntp.example.org ]
}
If this is merged as one internal def.json
file by CFengine:
ntp_json_files : [ site.json ]
ntp: {
json_files: [ role.json ]
server: [ ntp.example.org ]
}
SCL has a 3 level merge strategy. The ntp_json_files
for the global setting and the service definition we have 2 levels:
json_files
definition that can be used for override the global ones.json_files
attribute setting.The golden rule is to use <service>_json_files
for the global one and the other scl merge options to override settings.
there are three options:
With this you can easily test and build masterfiles configuration. The steps to build your masterfiles:
The “module” is published in the CFEngine build catalogue as surf-cfengine-library. These are the installation instructions:
mkdir scl_masterfiles
cd scl_masterfiles
cfbs init
cfbs add masterfiles
cfbs add https://github.com/basvandervlies/cf_surfsara_lib
(master branch)cfbs add surf-cfengine-library
(stable one)cfbs build
cfbs install
Note: Due to a bug in CFEngine Enterprise this library only works with CFEngine Community or CFEngine Enterprise versions 3.18.2 or 3.20 and greater.
As of January 2022, these unreleased versions are available as nightly builds installable with cf-remote.
pip3 install cf-remote
cf-remote --version master install --hub <hostname>
# for 3.20cf-remote --version 3.18.x install --hub <hostname>
cp -a masterfiles/lib/scl <masterfiles>/lib/scl
cp -a modules/ <masterfiles>/modules/scl
cp -a templates/\* $(sys.workdir)/templates/scl
/lib/scl/stdlib.cf
in your inputsbody common control
{
inputs => {
...
"lib/scl/stdlib.cf",
...
};
}
The def.node_template_dir
variable is set in lib/scl/def.cf
, but can also be set
set in def.json
. The def.json wins, e.g.:
vars:
{
"node_template_dir" : "/etc/node_status/templates"
}
default value is: /var/cfengine/node_templates
For older versions you have to manually add the shorcut templates
to controls/cf_serverd.cf
"$(sys.workdir)/templates"
handle => "server_access_grant_access_templates",
shortcut => "templates",
comment => "Grant access to templates directory",
admit => { @(def.acl) };
See above to add templates shortcut
to cf-serverd.
The documentation is embedded in the source files, and generated:
There are several services setups included with inline documentation. These setups are used in production at SURF.
To enable the service on your system use def.scl_services_enabled
method in def.cf/def.json for
both installations method.
This is the preferred method for MPF and your own framework. With this method you can control which services are run and which files are included, e.g.: def.json
"vars": {
"scl_services_enabled": [
"ntp",
"resolv"
]
}
This will include the service files ntp.cf
and resolv.cf
and run all bundles that have the meta tag
service\_ntp
and service\_resolv
. The bundle run can be protected by an class statement, default is any
, e.g.:
"ntp": {
"run_class": "debian|centos"
}
This will only run on debian or centos hosts.
In this file you can override settings for the services.
"vars": {
"ntp" : {
"server": [ "<your_ip_server1>", "<your_ip_server2>" ]
}
}
You can also specify json setup files:
"vars": {
"tcpwrappers": {
"json_files": [ "allow_ssh.json", "allow_http.json" ]
}
}
or:
"vars": {
"tcpwrappers_json_files": [ "allow_ssh.json", "allow_http.json" ]
}
You can use both definitions at the same same time. The tcpwrappers_json_files
definition is read first and can be overwritten by
the tcpwrappers: { json_files }
definition. See the merge strategy.
For every service you dynamically set classes in the service data, e.g.:
"vars": {
"dhclient": {
"classes": {
"RESOLV_CONF": [ "<short_hostname>" ]
}
}
}
This will set the class DHCLIENT\_RESOLV\_CONF
on host/node r24n2
You can also override settings in this file, e.g.:
vars:
"ntp" data => parsejson( '{ "server" : [ "<your_ip_server1>" ] }' );
vars:
"tcpwrappers" data => parsejson( '{ "json_files": [ "allow_ssh.json", "allow_http.json" ] '} );
If you defined your own def.cf
and do not want the one included in this framework you can set the following class:
SCL_SKIP_DEF_CF_INCLUDE
The SURF CFEngine library also checks for some classes:
templates
directory. This directory must be one level higher than your policy files directory (../templates):-DSCLOCAL
: Copy from local directory the mustache and json file(s)
To test local mustache/json changes in $(def.node_template_dir)
, the copy of the json/mustache file(s) from the policy server can be skipped by:-DSKIP_SCLOCAL
: Skip copying of mustache and json files-DDEBUG_MUSTACHE
(all service bundles)-DDEBUG_ntp