Orchestrator Plugin for Backstage

The Orchestrator for Backstage is a mechanism designed to facilitate the implementation and execution of developer self-service flows. It serves as a vital component that enhances and augments the existing scaffolder functionality of Backstage with a more flexible and powerful set of features including long-running and asynchronous flows.

The orchestrator works harmoniously with other Backstage components such as the Software Catalog, permissions, and plugins as well as others. By leveraging its capabilities, organizations can orchestrate and coordinate developer self-service flows effectively.

Context

The Backstage Orchestrator plugin aims to provide a better option to Scaffolder, based on workflows to have a more flexible and powerful tool that addresses the need by streamlining and automating processes, allowing developers to focus more on coding and innovation.

The orchestrator relies on SonataFlow, a powerful tool for building cloud-native workflow applications.

The main idea is to keep the same user experience for users, leveraging the UI components, input forms, and flow that Scaffolder provides, this way it should be straightforward for users and transparent no matter whether using Templates or Workflows, both can live together being compatible with integration points.

The orchestrator controls the flow orchestrating operations/tasks that may be executed in any external service including Scaffolder Actions, this way it is possible to leverage any existing Action hence Software Templates can be easily migrated to workflows opening the door to extend them to more complex use cases.

Capabilities

Advanced core capabilities

Stateful/long-lived
Branching and parallelism
Error management and compensation
Event-driven supporting CloudEvents
Audit logging
Sub-flows
Choreography
Timer/timeout control
Built-in powerful expression evaluation with JQ
Low Code/No code
Cloud-native architecture Kubernetes/OpenShift with Operator support
OpenAPI / REST built-in integration etc.

Client-side tooling

Orchestration visualization / graphical editor
Integration with service catalog/actions
GitHub integration
Form generation
Runtime monitoring of instances
Dashboards
Potential custom integrations (user interaction, notifications, etc.)

For administrators

Installation

The Orchestrator plugin is composed of the following packages:

@red-hat-developer-hub/backstage-plugin-orchestrator-backend package connects the Backstage server to the Orchestrator. For setup process, see Backend Setup
@red-hat-developer-hub/backstage-plugin-orchestrator package contains frontend components for the Orchestrator plugin. For setup process, see Frontend Setup
@red-hat-developer-hub/backstage-plugin-orchestrator-common package contains shared code between the Orchestrator plugin packages.

Prerequisites for running the plugins locally in development mode

Docker up and running

Setting up the Orchestrator as a dynamic plugin in a Helm deployment

Please follow this link for instructions: https://github.com/janus-idp/backstage-showcase/blob/main/docs/dynamic-plugins/installing-plugins.md#installing-external-dynamic-plugins.

Setting up the configuration for the Orchestrator plugin

The following configuration is required for the Orchestrator plugin to work properly:

backend:
  csp:
    frame-ancestors: ['http://localhost:3000', 'http://localhost:7007']
    script-src: ["'self'", "'unsafe-inline'", "'unsafe-eval'"]
    script-src-elem: ["'self'", "'unsafe-inline'", "'unsafe-eval'"]
    connect-src: ["'self'", 'http:', 'https:', 'data:']
orchestrator:
  sonataFlowService:
    baseUrl: http://localhost
    port: 8899
    autoStart: true
    workflowsSource:
      gitRepositoryUrl: https://github.com/parodos-dev/backstage-orchestrator-workflows
      localPath: /tmp/orchestrator/repository
  dataIndexService:
    url: http://localhost:8899

When interacting with an existing SonataFlow infrastructure, the sonataFlowService config section must be entirely omitted and the dataIndexService.url must point to the existing Data Index Service.

For more information about the configuration options, including other optional properties, see the config.d.ts file.

How to use the Git Hub identity provider

The workspaces/orchestrator/app-config.yaml and the workspaces/orchestrator/packages/app/src/App.tsx files contain sufficient configuration to use the Git Hub identity provider.

Feel free to update the users.yaml file to match your identity.

To run, additional secrets needs to be provided:

export AUTH_GITHUB_CLIENT_ID=...fill
export AUTH_GITHUB_CLIENT_SECRET=...fill

yarn dev

Setting up the Orchestrator backend package

Install the Orchestrator backend plugin using the following command:

yarn workspace backend add @red-hat-developer-hub/backstage-plugin-orchestrator-backend

Add the following code to the packages/backend/src/index.ts file:

const backend = createBackend();

/* highlight-add-next-line */
backend.add(
  import('@red-hat-developer-hub/backstage-plugin-orchestrator-backend'),
);

backend.start();

Setting up the Orchestrator frontend package

Install the Orchestrator frontend plugin using the following command:

yarn workspace app add @red-hat-developer-hub/backstage-plugin-orchestrator

Add a route to the OrchestratorPage and the customized template card component to Backstage App (packages/app/src/App.tsx):

/* highlight-add-next-line */
import { OrchestratorPage } from '@red-hat-developer-hub/backstage-plugin-orchestrator';

const routes = (
  <FlatRoutes>
    {/* ... */}
    {/* highlight-add-next-line */}
    <Route path="/orchestrator" element={<OrchestratorPage />} />
  </FlatRoutes>
);

Add the Orchestrator to Backstage sidebar (packages/app/src/components/Root/Root.tsx):

/* highlight-add-next-line */
import { OrchestratorIcon } from '@red-hat-developer-hub/backstage-plugin-orchestrator';

export const Root = ({ children }: PropsWithChildren<{}>) => (
  <SidebarPage>
    <Sidebar>
      <SidebarGroup label="Menu" icon={<MenuIcon />}>
        {/* ... */}
        {/* highlight-add-start */}
        <SidebarItem
          icon={OrchestratorIcon}
          to="orchestrator"
          text="Orchestrator"
        />
        {/* highlight-add-end */}
      </SidebarGroup>
      {/* ... */}
    </Sidebar>
    {children}
  </SidebarPage>
);

Extensible Workflow Execution Form

The orchestrator plugin includes an extensible form for executing forms. For detailed guidance see the Extensible Workflow Execution Form Documentation.

For users

Using the Orchestrator plugin in Backstage

The Orchestrator plugin enhances the Backstage with the execution of developer self-service flows. It provides a graphical editor to visualize workflow definitions, and a dashboard to monitor the execution of the workflows.

Refer to the Quick start to install the Orchestrator using the helm chart and execute a sample workflow through the Red Hat Developer Hub orchestrator plugin UI.

OpenAPI

The plugin provides OpenAPI v2 endpoints definition to facilitate communication between the frontend and backend. This approach minimizes the data that needs to be sent to the frontend, provides flexibility and avoids dependencies on changes in the CNCF serverless specification. It also allows for a seamless transition if there's a need to replace the backend implementation.

In addition, by leveraging on OpenAPI spec, it is possible to generate clients and create CI steps.

OpenAPI specification file is available in orchestrator-common.
OpenAPI specification documentation is available here

orchestrator-common

The typescript client is generated in generated folder from openapi.yaml specification file.

orchestrator-backend

The orchestrator backend can use the generated schema to validate the HTTP requests and responses.

audit log

The orchestrator backend has audit logs for all incoming requests.

For more information about audit logs in RHDH, please refer to the official documentation. The official Log storage OpenShift documentation may also be of interest.

Development instruction

Checkout the backstage-plugin

git clone git@github.com:red-hat-developer-hub/backstage-plugins.git

If you need to change the OpenAPI spec, edit the openapi.yaml according to your needs and then execute from the project root folder:

yarn --cwd plugins/orchestrator-common openapi

This command updates the generated files including API, client and docs.

NOTE: Do not manually edit auto-generated files

If you add a new component in the spec, then you need to export the generated typescript object here. For example, if you define

components:
  schemas:
    Person:
      type: object
      properties:
        name:
          type: string
        surname:
          type: string

then

export type Person = components['schemas']['Person'];

When defining a new endpoint, you have to define the operationId. That id is the one that you can use to implement the endpoint logic.

For example, let's assume you add

paths:
  /names:
    get:
      operationId: getNames
      description: Get a list of names
      responses:
        '200':
          description: Success
          content:
            application/json:
              schema:
               type: array
                items:
                  $ref: '#/components/schemas/Person'

Then you can implement the endpoint in router.ts referring the operationId getNames:

api.register('getNames', async (_c, _req, res: express.Response, next) => {
  // YOUR LOGIC HERE
  const result: Person[] = [
    { name: 'John', surname: 'Snow' },
    { name: 'John', surname: 'Black' },
  ];

  res.status(200).json(result);
});

Workflows	Chart Version	Orchestrator Operator Version
move2kube	1.3.x	1.3.x
create-ocp-project	1.3.x	1.3.x
request-vm-cnv	1.3.x	1.3.x
modify-vm-resources	1.3.x	1.3.x
mta-v7	1.3.x	1.3.x
mtv-migration	1.3.x	1.3.x
mtv-plan	1.3.x	1.3.x
——————–	—————	———————-
mta-analysis	0.3.x	1.2.x
move2kube	0.3.x	1.2.x
create-ocp-project	0.1.x	1.2.x
request-vm-cnv	0.1.x	1.2.x
modify-vm-resources	0.1.x	1.2.x
mta-v6	0.2.x	1.2.x
mta-v7	0.2.37	1.2.x
mtv-migration	0.0.x	1.2.x
mtv-plan	0.0.13	1.2.x

Environment variable	Description	Mandatory
`BACKSTAGE_NOTIFICATIONS_URL`	The backstage server URL for notifications	✅
`NOTIFICATIONS_BEARER_TOKEN`	The authorization bearer token to use to send notifications	✅
`MTA_URL`	The MTA Hub server URL	✅

key	default	description
mta.url	http://mta-ui.openshift-mta.svc.cluster.local:8080	Endpoint (with protocol and port) for MTA
quarkus.rest-client.mta_json.url	${mta.url}/hub	MTA hub api
quarkus.rest-client.notifications.url	${BACKSTAGE_NOTIFICATIONS_URL:http://backstage-backstage.rhdh-operator/api/notifications/}	Backstage notification url
quarkus.rest-client.mta_json.auth.basicAuth.username	username	Username for the MTA api
quarkus.rest-client.mta_json.auth.basicAuth.password	password	Password for the MTA api

Environment variable	Description	Mandatory	Default value
`JIRA_URL`	The Jira server URL	✅
`JIRA_USERNAME`	The Jira server username	✅
`JIRA_API_TOKEN`	The Jira API Token	✅
`JIRA_PROJECT`	The key of the Jira project where the escalation issue is created	❌	`TEST`
`JIRA_ISSUE_TYPE`	The ID of the Jira issue type to be created	✅
`OCP_API_SERVER_URL`	The OpensShift API Server URL	✅
`OCP_API_SERVER_TOKEN`	The OpensShift API Server Token	✅
`ESCALATION_TIMEOUT_SECONDS`	The number of seconds to wait before triggering the escalation request, after the issue has been created	❌	`60`
`POLLING_PERIODICITY`(1)	The polling periodicity of the issue state checker, according to ISO 8601 duration format	❌	`PT6S`

Environment variable	Description	Mandatory	Default value
`MOVE2KUBE_URL`	The move2kube instance server URL	✅
`BACKSTAGE_NOTIFICATIONS_URL`	The backstage server URL for notifications	✅
`NOTIFICATIONS_BEARER_TOKEN`	The authorization bearer token to use to send notifications	✅
`MOVE2KUBE_GET_PLAN_MAX_RETRIES`	The amount of retries to get the plan before failing the workflow	❌	10

Environment variable	Description	Mandatory	Default value
`MOVE2KUBE_API`	The move2kube instance server URL	✅
`SSH_PRIV_KEY_PATH`	The absolute path to the SSH private key	✅
`BROKER_URL`	The knative broker URL	✅
`LOG_LEVEL`	The log level	❌	INFO

Workflow Name	Description
`create-ocp-project`	Sets up an OpenShift Container Platform (OCP) project.
`escalation`	Demos workflow ticket escalation.
`extendable-workflow`	Provides a flexible, extendable workflow setup.
`greeting`	Sample greeting workflow.
`modify-vm-resources`	Modifies resources allocated to virtual machines.
`move2kube`	Workflow for Move2Kube tasks and transformation.
`mta-v7.x`	Migration toolkit for applications, version 7.x.
`mtv-migration`	Migration tasks using Migration Toolkit for Virtualization (MTV).
`mtv-plan`	Planning workflows for Migration Toolkit for Virtualization.
`request-vm-cnv`	Requests and provisions VMs using Container Native Virtualization (CNV).

Documentation

Orchestrator

1 - Quick Start

Quickstart Guide

2 - Architecture

3 - Core Concepts

3.1 - Workflow Types

Infrastructure workflow

Examples:

Assessment workflow

Examples:

Note

4 - Installation

4.1 - Requirements

Operators

OpenShift Serverless Logic operator requirements

RHDH operator requirements

Workflows

How to evaluate resource requirements for your workflow

4.2 - Orchestrator on OpenShift

Orchestrator Documentation

Installing the Orchestrator Helm Operator

Important Note for ARM64 Architecture Users

Prerequisites

Deployment with GitOps

Detailed Installation Guide

From OperatorHub

With Helm (deprecated)

Additional information

Additional Workflow Namespaces

GitOps environment

Deploying PostgreSQL reference implementation

ArgoCD and workflow namespace

Workflow installation

Cleanup

4.3 - Orchestrator on Kubernetes

Orchestrator-k8s helm chart

Usage

Configuration

Secrets as env vars:

Development

4.4 - Orchestrator on existing RHDH instance

Prerequisites

4.5 - Workflows

4.5.1 - Deploy From Helm Repository

Orchestrator Workflows Helm Repository

Usage

Prerequisites

Installation

Installing workflows in additional namespaces

Version Compatability

Helm index

5 - Serverless Workflows

5.1 - Assessment

5.1.1 - MTA Analysis

MTA - migration analysis workflow

Synopsis

Workflow application configuration

Inputs

Output

Dependencies

Runtime configuration

Workflow Diagram

Installation

5.2 - Infrastructure

5.2.1 - Simple Escalation

Simple escalation workflow

Prerequisite

Workflow diagram

Application configuration

How to run

5.2.2 - Move2Kube

Move2kube (m2k) workflow

Context

Design diagram

Workflow

Workflow application configuration

Move2kube workflow

m2k-func serverless function

Components