Macros

A macro is a named sequence of steps, declared as a workflow, that can be called from another workflow as a regular action.

Defining a Macro

Macros are defined like regular workflows (see the workflow definition syntax for more info):

macro-name:
  parameters: ...
  steps: ...
  return: ...

Any workflow defined in a manifest can operate as a macro. Like any other workflow, a macro can declare parameters and return values. Macro parameters can be passed as part of an expression to the macro’s steps. Macro return values can be expressions involving the macro steps’ output parameters.

For example:

macro-name:
  parameters:
    - macro-host: ...
    - macro-port: ...
  steps:
    - macro-step1:
        action: ...
        parameters:
          some-url: "http://{$.macro-host}:{$.macro-port}/resource" # Expression with macro parameters
        output:
          result-hosts: ...                                         # Step output parameter
    - macro-step2:
        action: ...
        output:
          result-paths: ...                                         # Step output parameter
  return:
    urls: "{$.result-hosts}/{$.result-paths}"                       # Expression with step output parameters

Calling a Macro

To call a macro, use its name as an action:

steps:
  - calling-macro:
      action: macro-name
      ...

Passing Parameters

When calling a macro with parameters, you can provide values for the parameters as follows:

macro-name:
  parameters:
    - macro-param1:                       # Has no default value
        description: Param1 Description
    - macro-param2:                       # Has some default value
        description: Param2 Description
        default: 1

...

steps:
  - calling-macro:
      action: macro-name
      parameters:
        macro-param1: Param1 Value        # Required
        macro-param2: Param2 Value        # Optional

The values MUST be provided for macro parameters that have no default values defined. For macro parameters with defaults, the values can be omitted.

Referencing Return Values

The step that references a macro can define names for the macro’s return values that makes them available in the caller workflow:

provision-macro:
  steps: ...
  return:
    app-hosts: ...
    db-hosts: ...

...

# In the caller workflow:
steps:
  - provision-all:
      action: provision-macro
      output:
        provision-app-hosts: app-hosts
        provision-db-hosts: db-hosts

In the sample above provision-app-hosts and provision-db-hosts are local names for the provision-macro return values.

Scoping Rules

These names have local scope within a macro: macro parameters, phases, step output parameters and macro return values. Return values are local in the sense that they cannot be referenced directly, but must be brought into the caller scope as described in the Referencing Return Values section.

Note that step output parameters cannot be referenced in outer scopes directly; rather, the corresponding macro return values should be provided.

provision-macro:
  steps:
    - provision-db:
        action: ...
        output:
          db-hosts: ips       # 'db-hosts' cannot be used in outer scopes
    - provision-app:
        action: ...
        output:
          app-hosts: ips      # 'app-hosts' cannot be used in outer scopes
  return:
    macro-db-hosts:
      value: "{$.db-hosts}"   # 'macro-db-hosts' is available in the caller scope
    macro-app-hosts:
      value: "{$.app-hosts}"  # 'macro-app-hosts' is available in the caller scope

You are free to use the same identifiers for a macro return value and a step output parameter. There will be no collisions or ambiguity.

return:
  db-hosts: "{$.db-hosts}"
  app-hosts: "{$.app-hosts}"

Advanced Topics

Names Mangling

When an application compiles a manifest, it inlines all calls to macros into flat workflows. To avoid collisions, step names and step output parameters from macros are mangled by prefixing them with the caller step name and the ~ character. Consider the following source manifest:

launch:
  steps:
    - provision-all:
        action: provision-macro
        output:
          provision-app-hosts: app-hosts
          provision-db-hosts: db-hosts
  return:
    apps:
      value: "{$.provision-app-hosts}"
    dbs:
      value: "{$.provision-db-hosts}"

provision-macro:
  steps:
    - provision-db:
        action: ...
        output:
          db-hosts: ips
    - provision-app:
        action: ...
        output:
          app-hosts: ips
  return:
    db-hosts:
      value: "{$.db-hosts}"
    app-hosts:
      value: "{$.app-hosts}"

After compilation, the launch manifest will look similar to the following:

launch:
  steps:
    - provision-all~provision-db:
        action: ...
        output:
          provision-all~db-hosts: ips
    - provision-all~provision-app:
        action: ...
        output:
          provision-all~app~hosts: ips
  return:
    db-hosts:
      value: "{$.provision-all~db-hosts}"
    app-hosts:
      value: "{$.provision-all~app-hosts}"
../_images/mangled.png

Mangled step names are seen on the Jobs tab of the instance screen.

Expressions Substitution

When expanding macros, JSON path expressions will also be expanded.

launch:
  parameters:
    - launch-p1: ...
  steps:
    - launch-s1:
        action: macro
        parameters:
          macro-p1: "{$.launch-p1}"
        output:
          launch-s1-o1: macro-r1
  return:
    launch-r1:
      value: "{$.launch-s1-o1}"

macro:
  parameters:
    - macro-p1: ...
  steps:
    - macro-s1:
        action: basic1
        parameters:
          basic1-p1: "{$.macro-p1}"
        output:
          macro-s1-o1: basic1-r1
    - macro-s2:
        action: basic2
        parameters:
          basic2-p1: "{$.macro-s1-p1}"
        output:
           macro-s2-o1: basic2-r1
  return:
    macro-r1:
      value: "{$.macro-s2-o1}"

The manifest above will be compiled into the following:

launch:
  parameters:
    - launch-p1: ...
  steps:
    - launch-s1~macro-s1:
        action: basic1
        parameters:
          basic1-p1: "{$.launch-p1}"
        output:
          launch-s1~macro-s1-o1: basic1-r1
    - launch-s1~macro-s2:
        action: basic2
        parameters:
          basic2-p1: "{$.launch-s1~macro-s1-p1}"
        output:
           launch-s1~macro-s2-o1: basic2-r1
  return:
    launch-r1:
      value: "{$.launch-s1~macro-s2-o1}"

In the example above, the JSONPath expressions are trivial and satisfy the pattern {$.var-name} –, which includes a single entry starting from the root. There are also non-trivial cases which may take place in manifests (although in fact not all cases can be supported in general). The next example demonstrates several cases where either the substitution source or target (or both) are non-trivial JSONPath expressions.

launch:
  parameters:
    - launch-p1: ...
  steps:
    - launch-s1:
        action: macro
        parameters:
          macro-p1: '{$.launch-p1..bar}'    # source is non-trivial
...
macro:
  parameters:
    - macro-p1: ...
  steps:
    - macro-s1:
        action: basic1
        parameters:
          basic1-p1: "{$.macro-p1}"         # target is trivial
          basic1-p2: "{$.macro-p1[0]}"      # target is non-trivial
        output:
          macro-s1-o1: basic1-r1            # source is trivial
    - macro-s2:
        action: basic2
        parameters:
          basic2-p1: "{$.macro-s1-p1..foo}" # target is non-trivial
...

Although some of the expressions are non-trivial, all of them have a target that starts with a JSONPath part (which always resolves to single data entry). This finds the exact corresponding source for the target and substitutes it entirely, rather than the target’s starting part. Thus, for the example above, the following result will be produced:

launch:
  parameters:
    - launch-p1: ...
  steps:
    - launch-s1~macro-s1:
        action: basic1
        parameters:
          basic1-p1: "{$.launch-p1..bar}"
          basic1-p2: "{$.launch-p1..bar[0]}"
        output:
          launch-s1~macro-s1-o1: basic1-r1
    - launch-s1~macro-s2:
        action: basic2
        parameters:
          basic2-p1: "{$.launch-s1~macro-s1-p1..foo}"

Note

Cases where a target starts with any of part which may resolve to multiple data entries are not supported and such the expressions are left unchanged at compilation time.

Phase Hygienic Macros

The manifest compiler uses the algorithm described below to achieve phase names locality. First, phase names inside a macro are mangled like other local names described in the previous section. Then, two additional points are taken into consideration.

If there is a phase attribute set at the macro call site, then the steps wait until the macro is completed. Thus, the compiler adds the end barrier step at the end of the macro steps sequence and sets the precedingPhases attribute of the barrier step to the list of all phase names in the macro. If some step in the macro has no phase defined, it receives the default value.

Consider the following example:

launch:
  steps:
    - provision-all:
        action: provision-macro        # calling the macro
        phase: provision               # the macro has the defined phase at its call site
    - finalize:
        action: ...
        precedingPhases: [provision]   # awaiting until the 'provision' phase is completed

provision-macro:
  steps:
    - provision-db:                    # the step with an explicitly defined phase
        action: ...
        phase: db-work
    - provision-app:                   # the step with no phase defined
        action: ...
        precedingPhases: [db-work]     # awaiting until the 'db-work' phase is completed

After the compilation, the workflow above will be expanded into the following:

launch:
  steps:
    - provision-all~provision-db:
        action: ...
        phase: provision-all~db-work              # the explicitly defined phase is mangled
    - provision-all~provision-app:
        action: ...
        phase: provision-all~~default             # the default phase is assigned
        precedingPhases: [provision-all~db-work]  # the preceding phases are mangled as well
    - provision-all~~end:                         # the end barrier step
        action: wait                              # uses the action 'wait' for barriers
        phase: provision                          # the barrier step received the phase from the call site
        precedingPhases:                          # awaits until all the phases from the macro are completed
          - provision-all~db-work
          - provision-all~~default
    - finalize:
        action: ...
        precedingPhases: [provision]              # awaits until the end barrier from the macro is completed

Secondly, if there are one or more preceding phases defined at the macro call site, then the macro should wait until all steps with the corresponding phases are completed. Thus, the compiler adds the begin barrier step at the beginning of the macro step sequence and assigns it to an auto-generated phase and the preceding phases from the macro call site. The compiler then adds the barrier step phase name to the precedingPhase attribute for each step in the macro.

Consider the following example:

launch:
  steps:
    - deploy-prerequisites:
        action: ...
        phase: prerequisites
    - provision-all:
        action: provision-macro           # calling the macro
        precedingPhases: [prerequisites]  # the macro has preceding phases defined at its call site

provision-macro:
  steps:
    - provision-db:
        action: ...
        phase: db-work
    - provision-app:
        action: ...
        precedingPhases: [db-work]        # awaiting until the 'db-work' phase is completed

The compiler will expand the workflow above into the following:

launch:
  steps:
    - deploy-prerequisites:
        action: ...
        phase: prerequisites
    - provision-all~~begin:                       # the begin barrier step
        action: wait                              # uses the action 'wait' for barriers
        phase: provision-all~~begin               # the auto-generated phase for the barrier
        precedingPhases: [prerequisites]          # assigns the preceding phase(s) from the call site
    - provision-all~provision-db:
        action: ...
        phase: provision-all~db-work              # the explicitly defined phase is mangled
        precedingPhases: [provision-all~~begin]   # adds the barrier phase to the step preceding phases
    - provision-all~provision-app:
        action: ...
        phase: provision-all~~default
        precedingPhases:
          - provision-all~~begin                  # adds the barrier phase to the step preceding phases
          - provision-all~db-work

Note

  • wait is a special action that does nothing but serve as a junction point where all preceding phases are guaranteed to finish.
  • The double tilde ~~ is used for auto-generated names to ensure they don’t interfere with user-defined names.