Adaltas

Node Mecano: Common functions for system deployment

Mecano gather a set of functions usually used during system deployment. All the functions share a common API with flexible options.

Functions include “copy”, “download”, “exec”, “extract”, “git”, “link”, “mkdir”, “move”, “remove”, “render”, “service”, “write”. They all share common usages and philosophies:
* Run actions both locally and remotely over SSH.
* Ability to see if an action had an effect through the second argument provided in the callback.
* Common API with options and callback arguments and calling the callback with an error and the number of affected actions.
* Run one or multiple actions depending on option argument being an object or an array of objects.

chmod([goptions], options, callback)

Change the file permissions of a file.

options Command options include:

  • destination Where the file or directory is copied.
  • mode Permissions of the file or the parent directory
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.
  • log Function called with a log related messages.

callback Received parameters are:

  • err Error object if any.
  • modified Number of files with modified permissions.

chmod([goptions], options, callback)

Change the file permissions of a file.

options Command options include:

  • destination Where the file or directory is copied.
  • mode Permissions of the file or the parent directory
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.
  • log Function called with a log related messages.

callback Received parameters are:

  • err Error object if any.
  • modified Number of files with modified permissions.

cp copy([goptions], options, callback)

Copy a file. The behavior is similar to the one of the cp Unix utility. Copying a file over an existing file will overwrite it.

options Command options include:

  • source The file or directory to copy.
  • destination Where the file or directory is copied.
  • not_if_exists Equals destination if true.
  • mode Permissions of the file or the parent directory
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.

callback Received parameters are:

  • err Error object if any.
  • copied Number of files or parent directories copied.

todo: * preserve permissions if mode is true

download([goptions], options, callback)

Download files using various protocols.

When executed locally: the http protocol is handled with the “request” module; the ftp protocol is handled with the “jsftp”; the file protocol is handle with the navite fs module.

options Command options include:

  • source File, HTTP URL, FTP, GIT repository. File is the default protocol if source is provided without any.
  • destination Path where the file is downloaded.
  • force Overwrite destination file if it exists.
  • stdout Writable Stream in which commands output will be piped.
  • stderr Writable Stream in which commands error will be piped.

callback Received parameters are:

  • err Error object if any.
  • downloaded Number of downloaded files

File example

1
2
3
4
mecano.download
  source: 'file://path/to/something'
  destination: 'node-sigar.tgz'
, (err, downloaded) -> ...

HTTP example

1
2
3
4
mecano.download
  source: 'https://github.com/wdavidw/node-sigar/tarball/v0.0.1'
  destination: 'node-sigar.tgz'
, (err, downloaded) -> ...

FTP example

1
2
3
4
5
6
mecano.download
  source: 'ftp://myhost.com:3334/wdavidw/node-sigar/tarball/v0.0.1'
  destination: 'node-sigar.tgz'
  user: "johndoe",
  pass: "12345"
, (err, downloaded) -> ...

File example

exec execute([goptions], options, callback)

Run a command locally or with ssh if host or ssh is provided.

options Command options include:

  • cmd String, Object or array; Command to execute.
  • env Environment variables, default to process.env.
  • cwd Current working directory.
  • uid Unix user id.
  • gid Unix group id.
  • code Expected code(s) returned by the command, int or array of int, default to 0.
  • code_skipped Expected code(s) returned by the command if it has no effect, executed will not be incremented, int or array of int.
  • stdout Writable Stream in which commands output will be piped.
  • stderr Writable Stream in which commands error will be piped.
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.

callback Received parameters are:

  • err Error if any.
  • executed Number of executed commandes.
  • stdout Stdout value(s) unless stdout option is provided.
  • stderr Stderr value(s) unless stderr option is provided.

extract([goptions], options, callback)

Extract an archive. Multiple compression types are supported. Unless specified as an option, format is derived from the source extension. At the moment, supported extensions are ‘.tgz’, ‘.tar.gz’ and ‘.zip’.

options Command options include:

  • source Archive to decompress.
  • destination Default to the source parent directory.
  • format One of ‘tgz’ or ‘zip’.
  • creates Ensure the given file is created or an error is send in the callback.
  • not_if_exists Cancel extraction if file exists.
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.

callback Received parameters are:

  • err Error object if any.
  • extracted Number of extracted archives.

git([goptions], options, callback

options Command options include:

  • source Git source repository address.
  • destination Directory where to clone the repository.
  • revision Git revision, branch or tag.
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.
  • stdout Writable EventEmitter in which command output will be piped.
  • stderr Writable EventEmitter in which command error will be piped.

ini([goptions], options, callback

Write an object as .ini file. Note, we are internally using the ini module. However, there is a subtile difference. Any key provided with value of undefined or null will be disregarded. Within a merge, it get more prowerfull and tricky: the original value will be kept if undefined is provided while the value will be removed if null is provided.

The ini function rely on the write function and accept all of its options. It introduces the merge option which instruct to read the destination file if it exists and merge its parsed object with the one provided in the content option.

options Command options include:

  • append Append the content to the destination file. If destination does not exist, the file will be created. When used with the match and replace options, it will append the replace value at the end of the file if no match if found and if the value is a string.
  • backup Create a backup, append a provided string to the filename extension or a timestamp if value is not a string.
  • content Object to stringify.
  • stringify User defined function to stringify to ini format, default to require('ini').stringify.
  • destination File path where to write content to or a callback.
  • from Replace from after this marker, a string or a regular expression.
  • local_source Treat the source as local instead of remote, only apply with “ssh” option.
  • match Replace this marker, a string or a regular expression.
  • merge Read the destination if it exists and merge its content.
  • replace The content to be inserted, used conjointly with the from, to or match options.
  • source File path from where to extract the content, do not use conjointly with content.
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.
  • stringify Provide your own user-defined function to stringify the content, see ‘misc.ini.stringify_square_then_curly’.
  • separator Default separator between keys and values, default to ” : “.
  • to Replace to before this marker, a string or a regular expression.

krb5_ktadd([goptions], options, callback

Create a new Kerberos principal and an optionnal keytab.

options Command options include:

  • kadmin_server Address of the kadmin server; optional, use “kadmin.local” if missing.
  • kadmin_principal KAdmin principal name unless kadmin.local is used.
  • kadmin_password Password associated to the KAdmin principal.
  • principal Principal to be created.
  • password Password associated to this principal; required if no randkey is provided.
  • randkey Generate a random key; required if no password is provided.
  • keytab Path to the file storing key entries.
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.
  • log Function called with a log related messages.
  • stdout Writable Stream in which commands output will be piped.
  • stderr Writable Stream in which commands error will be piped.

krb5_principal([goptions], options, callback

Create a new Kerberos principal and an optionnal keytab.

options Command options include:

  • kadmin_server Address of the kadmin server; optional, use “kadmin.local” if missing.
  • kadmin_principal KAdmin principal name unless kadmin.local is used.
  • kadmin_password Password associated to the KAdmin principal.
  • principal Principal to be created.
  • password Password associated to this principal; required if no randkey is provided.
  • randkey Generate a random key; required if no password is provided.
  • keytab Path to the file storing key entries.
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.
  • log Function called with a log related messages.
  • stdout Writable Stream in which commands output will be piped.
  • stderr Writable Stream in which commands error will be piped.

krb5_delprinc([goptions], options, callback

Create a new Kerberos principal and an optionnal keytab.

options Command options include:

  • principal Principal to be created.
  • kadmin_server Address of the kadmin server; optional, use “kadmin.local” if missing.
  • kadmin_principal KAdmin principal name unless kadmin.local is used.
  • kadmin_password Password associated to the KAdmin principal.
  • keytab Path to the file storing key entries.
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.
  • log Function called with a log related messages.
  • stdout Writable Stream in which commands output will be piped.
  • stderr Writable Stream in which commands error will be piped.

ldap_acl([goptions], options, callback

options Command options include:

  • to What to control access to as a string.
  • by Who to grant access to and the access to grant as an array (eg: {..., by:["ssf=64 anonymous auth"]})
  • url Specify URI referring to the ldap server, alternative to providing an [ldapjs client] instance.
  • binddn Distinguished Name to bind to the LDAP directory, alternative to providing an [ldapjs client] instance.
  • passwd Password for simple authentication, alternative to providing an [ldapjs client] instance.
  • ldap Instance of an pldapjs client]ldapclt, alternative to providing the url, binddn and passwd connection properties.
  • unbind Close the ldap connection, default to false if connection is an ldapjs client instance.
  • name Distinguish name storing the “olcAccess” property, using the database adress (eg: “olcDatabase={2}bdb,cn=config”).
  • overwrite Overwrite existing “olcAccess”, default is to merge.
  • log Function called with a log related messages.

Resources: http://www.openldap.org/doc/admin24/access-control.html

ldap_index([goptions], options, callback

options Command options include:

  • indexes Object with keys mapping to indexed attributes and values mapping to indices (“pres”, “approx”, “eq”, “sub” and ‘special’).
  • url Specify URI referring to the ldap server, alternative to providing an [ldapjs client] instance.
  • binddn Distinguished Name to bind to the LDAP directory, alternative to providing an [ldapjs client] instance.
  • passwd Password for simple authentication, alternative to providing an [ldapjs client] instance.
  • ldap Instance of an pldapjs client]ldapclt, alternative to providing the url, binddn and passwd connection properties.
  • unbind Close the ldap connection, default to false if connection is an ldapjs client instance.
  • name Distinguish name storing the “olcAccess” property, using the database adress (eg: “olcDatabase={2}bdb,cn=config”).
  • overwrite Overwrite existing “olcAccess”, default is to merge.

Resources

http://www.zytrax.com/books/ldap/apa/indeces.htmlldap_schema([goptions], options, callback)

Register a new ldap schema.

options Command options include:

  • url Specify URI referring to the ldap server, alternative to providing an [ldapjs client] instance.
  • binddn Distinguished Name to bind to the LDAP directory, alternative to providing an [ldapjs client] instance.
  • passwd Password for simple authentication, alternative to providing an [ldapjs client] instance.
  • name Common name of the schema.
  • schema Path to the schema definition.
  • overwrite Overwrite existing “olcAccess”, default is to merge.
  • log Function called with a log related messages.
    ln link([goptions], options, callback)

Create a symbolic link and it’s parent directories if they don’t yet exist.

options Command options include:

  • source Referenced file to be linked.
  • destination Symbolic link to be created.
  • exec Create an executable file with an exec command.
  • mode Default to 0755.

callback Received parameters are:

  • err Error object if any.
  • linked Number of created links.

mkdir([goptions], options, callback)

Recursively create a directory. The behavior is similar to the Unix command mkdir -p. It supports an alternative syntax where options is simply the path of the directory to create.

options Command options include:

  • cwd Current working directory for relative paths.
  • uid Unix user id.
  • gid Unix group id.
  • mode Default to 0755.
  • directory Path or array of paths.
  • destination Alias for directory.
  • exclude Regular expression.
  • source Alias for directory.

callback Received parameters are:

  • err Error object if any.
  • created Number of created directories

Simple usage:

1
2
mecano.mkdir './some/dir', (err, created) ->
  console.info err?.message ? created

Advance usage:

1
2
3
4
5
6
mecano.mkdir
  ssh: options.ssh
  destination: './some/dir'
  uid: 'me'
  gid: 'my_group'
  mode: 0o0777 or '777'

mv move([goptions], options, callback)

Move files and directories.

options Command options include:

  • destination Final name of the moved resource.
  • force Overwrite the destination if it exists.
  • source File or directory to move.

callback Received parameters are:

  • err Error object if any.
  • moved Number of moved resources.

Example

mecano.mv

1
2
3
4
source: __dirname
desination: '/temp/my_dir'
(err, moved) ->
console.info "#{moved} dir moved"

rm remove([goptions], options, callback)

Recursively remove files, directories and links. Internally, the function use the rimraf library.

options Command options include:

  • source File, directory or pattern.
  • destination Alias for “source”.

callback Received parameters are:

  • err Error object if any.
  • removed Number of removed sources.

Example

1
2
mecano.rm './some/dir', (err, removed) ->
  console.info "#{removed} dir removed"

Removing a directory unless a given file exists

1
2
3
4
5
mecano.rm
  source: './some/dir'
  not_if_exists: './some/file'
, (err, removed) ->
  console.info "#{removed} dir removed"

Removing multiple files and directories

1
2
3
4
5
mecano.rm [
  { source: './some/dir', not_if_exists: './some/file' }
  './some/file'
], (err, removed) ->
  console.info "#{removed} dirs removed"

render([goptions], options, callback)

Render a template file At the moment, only the ECO templating engine is integrated.

options Command options include:

  • engine Template engine to use, default to “eco”
  • content Templated content, bypassed if source is provided.
  • source File path where to extract content from.
  • destination File path where to write content to or a callback.
  • context Map of key values to inject into the template.
  • local_source Treat the source as local instead of remote, only apply with “ssh” option.
  • uid File user name or user id
  • gid File group name or group id
  • mode File mode (permission and sticky bits), default to 0666, in the for of {mode: 0o744} or {mode: "744"}

callback Received parameters are:

  • err Error object if any.
  • rendered Number of rendered files.

If destination is a callback, it will be called multiple times with the
generated content as its first argument.

service([goptions], options, callback)

Install a service. For now, only yum over SSH.

options Command options include:

  • name Package name.
  • startup Run service daemon on startup. If true, startup will be set to ‘2345’, use an empty string to not define any run level.
  • yum_name Name used by the yum utility, default to “name”.
  • chk_name Name used by the chkconfig utility, default to “srv_name” and “name”.
  • srv_name Name used by the service utility, default to “name”.

    start Ensure the service is started, a boolean.

    stop Ensure the service is stopped, a boolean.

  • action Execute the service with the provided action argument.
  • stdout Writable Stream in which commands output will be piped.
  • stderr Writable Stream in which commands error will be piped.
  • installed Cache a list of installed services. If an object, the service will be installed if a key of the same name exists; if anything else (default), no caching will take place.
  • updates Cache a list of outdated services. If an object, the service will be updated if a key of the same name exists; If true, the option will be converted to an object with all the outdated service names as keys; if anything else (default), no caching will take place.

callback Received parameters are:

  • err Error object if any.
  • modified Number of action taken (installed, updated, started or stoped).
  • installed List of installed services.
  • updates List of services to update.

touch([goptions], options, callback)

Create a empty file if it does not yet exists.

upload([goptions], options, callback)

Upload a file to a remote location. Options are identical to the “write” function with the addition of the “binary” option.

options Command options include:

  • binary Fast upload implementation, discard all the other option and use its own stream based implementation.
  • from Replace from after this marker, a string or a regular expression.
  • to Replace to before this marker, a string or a regular expression.
  • match Replace this marker, a string or a regular expression.
  • replace The content to be inserted, used conjointly with the from, to or match options.
  • content Text to be written.
  • source File path from where to extract the content, do not use conjointly with content.
  • destination File path where to write content to.
  • backup Create a backup, append a provided string to the filename extension or a timestamp if value is not a string.
  • md5 Validate uploaded file with md5 checksum (only for binary upload for now).
  • sha1 Validate uploaded file with sha1 checksum (only for binary upload for now).

callback Received parameters are:

  • err Error object if any.
  • rendered Number of rendered files.

write([goptions], options, callback)

Write a file or a portion of an existing file.

options Command options include:

  • from Replace from after this marker, a string or a regular expression.
  • local_source Treat the source as local instead of remote, only apply with “ssh” option.
  • to Replace to before this marker, a string or a regular expression.
  • match Replace this marker, a string or a regular expression.
  • replace The content to be inserted, used conjointly with the from, to or match options.
  • content Text to be written, an alternative to source which reference a file.
  • source File path from where to extract the content, do not use conjointly with content.
  • destination File path where to write content to.
  • backup Create a backup, append a provided string to the filename extension or a timestamp if value is not a string.
  • append Append the content to the destination file. If destination does not exist, the file will be created.
  • write An array containing multiple transformation where a transformation is an object accepting the options from, to, match and replace
  • ssh Run the action on a remote server using SSH, an ssh2 instance or an configuration object used to initialize the SSH connection.
  • uid File user name or user id
  • gid File group name or group id
  • mode File mode (permission and sticky bits), default to 0666, in the for of {mode: 0o744} or {mode: "744"}

callback Received parameters are:

  • err Error object if any.
  • written Number of written files.

The option “append” allows some advance usages. If “append” is null, it will add the replace value at the end of the file if no match if found and if the value is a string. When used conjointly with the match and replace options, it gets even more interesting. If append is a string or a regular expression, it will place the “replace” string just after the match. An append string will be converted to a regular expression such as “test” will end up converted as the string “test” is similar to the RegExp /.test.$/mg.

Example replacing part of a file using from and to markers

1
2
3
4
5
6
7
8
mecano.write
  content: 'here we are\n# from\nlets try to replace that one\n# to\nyou coquin'
  from: '# from\n'
  to: '# to'
  replace: 'my friend\n'
  destination: "#{scratch}/a_file"
, (err, written) ->
  # here we are\n# from\nmy friend\n# to\nyou coquin

Example replacing a matched line by a string with

1
2
3
4
5
6
7
mecano.write
  content: 'email=david(at)adaltas(dot)com\nusername=root'
  match: /(username)=(.*)/
  replace: '$1=david (was $2)'
  destination: "#{scratch}/a_file"
, (err, written) ->
  # email=david(at)adaltas(dot)com\nusername=david (was root)

Example replacing part of a file using a regular expression

1
2
3
4
5
6
7
mecano.write
  content: 'here we are\nlets try to replace that one\nyou coquin'
  match: /(.*try) (.*)/
  replace: ['my friend, $1']
  destination: "#{scratch}/a_file"
, (err, written) ->
  # here we are\nmy friend, lets try\nyou coquin

Example replacing with the global and multiple lines options

1
2
3
4
5
6
7
mecano.write
  content: '#A config file\n#property=30\nproperty=10\n#End of Config'
  match: /^property=.*$/mg
  replace: 'property=50'
  destination: "#{scratch}/replace"
, (err, written) ->
  '# A config file\n#property=30\nproperty=50\n#End of Config'

Example appending a line after each line containing “property”

1
2
3
4
5
6
7
8
mecano.write
  content: '#A config file\n#property=30\nproperty=10\n#End of Config'
  match: /^.*comment.*$/mg
  replace: '# comment'
  destination: "#{scratch}/replace"
  append: 'property'
, (err, written) ->
  '# A config file\n#property=30\n# comment\nproperty=50\n# comment\n#End of Config'

Example with multiple transformations

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
mecano.write
  content: 'username: me\nemail: my@email\nfriends: you'
  write: [
    match: /^(username).*$/mg
    replace: "$1: you"
  ,
    match: /^email.*$/mg
    replace: ""
  ,
    match: /^(friends).*$/mg
    replace: "$1: me"
  ]
  destination: "#{scratch}/file"
, (err, written) ->
  # username: you\n\nfriends: me
Fork me on GitHub