Internal site. Jolli authentication required to view.
CTRL K
Skip to Content
API ReferencePipeline CRUDPUT /v0/pipelines/{pipeline_name}

Upsert Pipeline

PUT/v0/pipelines/{pipeline_name}Pipeline CRUD

Try it

PUT/v0/pipelines/{pipeline_name}
Authentication
Parameters
Request body (application/json)

Fully update a pipeline if it already exists, otherwise create a new pipeline.

Authentication

  • JSON web token (JWT) or API keyBearer token

Parameters

Path parameters

NameTypeRequiredDescription
pipeline_namestringYesUnique pipeline name

Request body

Content type: application/json

  • object
    Create a new pipeline (POST), or fully update an existing pipeline (PUT). Fields which are optional and not provided will be set to their empty type value (for strings: an empty string `""`, for objects: an empty dictionary `{}`).
    • descriptionstring
    • namestringrequired
    • program_codestringrequired
    • program_configobject
    • runtime_configobject
    • udf_ruststring
    • udf_tomlstring

Response

200Pipeline successfully updatedapplication/json
  • object
    Pipeline information. It both includes fields which are user-provided and system-generated.
    • created_atstring (date-time)required
    • deployment_desired_statusstringrequired
    • deployment_desired_status_sincestring (date-time)required
    • deployment_errorobject
    • deployment_idstring (uuid)
    • deployment_initialobject
    • deployment_resources_desired_statusstringrequired
    • deployment_resources_desired_status_sincestring (date-time)required
    • deployment_resources_statusstringrequired
      Pipeline resources status. ```text /start (early start failed) ┌───────────────────┐ │ ▼ Stopped ◄────────── Stopping /start │ ▲ │ │ /stop?force=true │ │ OR: timeout (from Provisioning) ▼ │ OR: fatal runtime or resource error ⌛Provisioning ────────────│ OR: runtime status is Suspended │ │ │ │ ▼ │ Provisioned ─────────────┘ ``` ### Desired and actual status We use the desired state model to manage the lifecycle of a pipeline. In this model, the pipeline has two status attributes associated with it: the **desired** status, which represents what the user would like the pipeline to do, and the **current** status, which represents the actual (last observed) status of the pipeline. The pipeline runner service continuously monitors the desired status field to decide where to steer the pipeline towards. There are two desired statuses: - `Provisioned` (set by invoking `/start`) - `Stopped` (set by invoking `/stop?force=true`) The user can monitor the current status of the pipeline via the `GET /v0/pipelines/{name}` endpoint. In a typical scenario, the user first sets the desired status, e.g., by invoking the `/start` endpoint, and then polls the `GET /v0/pipelines/{name}` endpoint to monitor the actual status of the pipeline until its `deployment_resources_status` attribute changes to `Provisioned` indicating that the pipeline has been successfully provisioned, or `Stopped` with `deployment_error` being set.
    • deployment_resources_status_detailsobject
    • deployment_resources_status_sincestring (date-time)required
    • deployment_runtime_desired_statusobject
    • deployment_runtime_desired_status_sincestring (date-time)
    • deployment_runtime_statusobject
    • deployment_runtime_status_detailsobject
    • deployment_runtime_status_sincestring (date-time)
    • deployment_statusstringrequired
    • deployment_status_sincestring (date-time)required
    • descriptionstringrequired
    • idstring (uuid)required
      Pipeline identifier.
    • namestringrequired
    • platform_versionstringrequired
    • program_codestringrequired
    • program_configobjectrequired
      Program configuration.
    • program_errorobjectrequired
      Log, warning and error information about the program compilation.
    • program_infoobject
    • program_statusstringrequired
      Program compilation status.
    • program_status_sincestring (date-time)required
    • program_versioninteger (int64)required
      Version number.
    • refresh_versioninteger (int64)required
      Version number.
    • runtime_configobjectrequired
      Global pipeline configuration settings. This is the publicly exposed type for users to configure pipelines.
    • storage_statusstringrequired
      Storage status. The storage status can only transition when the resources status is `Stopped`. ```text Cleared ───┐ ▲ │ /clear │ │ │ │ Clearing │ ▲ │ │ │ InUse ◄───┘ ```
    • storage_status_detailsobject
    • udf_ruststringrequired
    • udf_tomlstringrequired
    • versioninteger (int64)required
      Version number.
201Pipeline successfully createdapplication/json
  • object
    Pipeline information. It both includes fields which are user-provided and system-generated.
    • created_atstring (date-time)required
    • deployment_desired_statusstringrequired
    • deployment_desired_status_sincestring (date-time)required
    • deployment_errorobject
    • deployment_idstring (uuid)
    • deployment_initialobject
    • deployment_resources_desired_statusstringrequired
    • deployment_resources_desired_status_sincestring (date-time)required
    • deployment_resources_statusstringrequired
      Pipeline resources status. ```text /start (early start failed) ┌───────────────────┐ │ ▼ Stopped ◄────────── Stopping /start │ ▲ │ │ /stop?force=true │ │ OR: timeout (from Provisioning) ▼ │ OR: fatal runtime or resource error ⌛Provisioning ────────────│ OR: runtime status is Suspended │ │ │ │ ▼ │ Provisioned ─────────────┘ ``` ### Desired and actual status We use the desired state model to manage the lifecycle of a pipeline. In this model, the pipeline has two status attributes associated with it: the **desired** status, which represents what the user would like the pipeline to do, and the **current** status, which represents the actual (last observed) status of the pipeline. The pipeline runner service continuously monitors the desired status field to decide where to steer the pipeline towards. There are two desired statuses: - `Provisioned` (set by invoking `/start`) - `Stopped` (set by invoking `/stop?force=true`) The user can monitor the current status of the pipeline via the `GET /v0/pipelines/{name}` endpoint. In a typical scenario, the user first sets the desired status, e.g., by invoking the `/start` endpoint, and then polls the `GET /v0/pipelines/{name}` endpoint to monitor the actual status of the pipeline until its `deployment_resources_status` attribute changes to `Provisioned` indicating that the pipeline has been successfully provisioned, or `Stopped` with `deployment_error` being set.
    • deployment_resources_status_detailsobject
    • deployment_resources_status_sincestring (date-time)required
    • deployment_runtime_desired_statusobject
    • deployment_runtime_desired_status_sincestring (date-time)
    • deployment_runtime_statusobject
    • deployment_runtime_status_detailsobject
    • deployment_runtime_status_sincestring (date-time)
    • deployment_statusstringrequired
    • deployment_status_sincestring (date-time)required
    • descriptionstringrequired
    • idstring (uuid)required
      Pipeline identifier.
    • namestringrequired
    • platform_versionstringrequired
    • program_codestringrequired
    • program_configobjectrequired
      Program configuration.
    • program_errorobjectrequired
      Log, warning and error information about the program compilation.
    • program_infoobject
    • program_statusstringrequired
      Program compilation status.
    • program_status_sincestring (date-time)required
    • program_versioninteger (int64)required
      Version number.
    • refresh_versioninteger (int64)required
      Version number.
    • runtime_configobjectrequired
      Global pipeline configuration settings. This is the publicly exposed type for users to configure pipelines.
    • storage_statusstringrequired
      Storage status. The storage status can only transition when the resources status is `Stopped`. ```text Cleared ───┐ ▲ │ /clear │ │ │ │ Clearing │ ▲ │ │ │ InUse ◄───┘ ```
    • storage_status_detailsobject
    • udf_ruststringrequired
    • udf_tomlstringrequired
    • versioninteger (int64)required
      Version number.
400application/json
  • object
    Information returned by REST API endpoints on error.
    • detailsobjectrequired
      Detailed error metadata. The contents of this field is determined by `error_code`.
    • error_codestringrequired
      Error code is a string that specifies this error type.
    • messagestringrequired
      Human-readable error message.
409Cannot rename pipeline as the new name already existsapplication/json
  • object
    Information returned by REST API endpoints on error.
    • detailsobjectrequired
      Detailed error metadata. The contents of this field is determined by `error_code`.
    • error_codestringrequired
      Error code is a string that specifies this error type.
    • messagestringrequired
      Human-readable error message.
500application/json
  • object
    Information returned by REST API endpoints on error.
    • detailsobjectrequired
      Detailed error metadata. The contents of this field is determined by `error_code`.
    • error_codestringrequired
      Error code is a string that specifies this error type.
    • messagestringrequired
      Human-readable error message.
Request
curl -X PUT 'https://api.example.com/v0/pipelines/{pipeline_name}' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "example1",
  "description": "Description of the pipeline example1",
  "runtime_config": {
    "workers": 16,
    "max_rss_mb": null,
    "hosts": 1,
    "storage": {
      "backend": {
        "name": "default"
      },
      "min_storage_bytes": null,
      "min_step_storage_bytes": null,
      "compression": "default",
      "cache_mib": null
    },
    "fault_tolerance": {
      "model": "none",
      "checkpoint_interval_secs": 60
    },
    "cpu_profiler": true,
    "tracing": false,
    "tracing_endpoint_jaeger": "",
    "min_batch_size_records": 0,
    "max_buffering_delay_usecs": 0,
    "resources": {
      "cpu_cores_min": null,
      "cpu_cores_max": null,
      "memory_mb_min": null,
      "memory_mb_max": null,
      "storage_mb_max": null,
      "storage_class": null,
      "service_account_name": null,
      "namespace": null
    },
    "clock_resolution_usecs": 1000000,
    "pin_cpus": [],
    "provisioning_timeout_secs": null,
    "max_parallel_connector_init": null,
    "init_containers": null,
    "checkpoint_during_suspend": true,
    "http_workers": null,
    "io_workers": null,
    "env": {},
    "dev_tweaks": {},
    "logging": null,
    "pipeline_template_configmap": null
  },
  "program_code": "CREATE TABLE table1 ( col1 INT );",
  "udf_rust": null,
  "udf_toml": null,
  "program_config": {
    "profile": "optimized",
    "cache": true,
    "runtime_version": null
  }
}'
Response
{
  "id": "67e55044-10b1-426f-9247-bb680e5fe0c8",
  "name": "example1",
  "description": "Description of the pipeline example1",
  "created_at": "1970-01-01T00:00:00Z",
  "version": 4,
  "platform_version": "v0",
  "runtime_config": {
    "workers": 16,
    "max_rss_mb": null,
    "hosts": 1,
    "storage": {
      "backend": {
        "name": "default"
      },
      "min_storage_bytes": null,
      "min_step_storage_bytes": null,
      "compression": "default",
      "cache_mib": null
    },
    "fault_tolerance": {
      "model": "none",
      "checkpoint_interval_secs": 60
    },
    "cpu_profiler": true,
    "tracing": false,
    "tracing_endpoint_jaeger": "",
    "min_batch_size_records": 0,
    "max_buffering_delay_usecs": 0,
    "resources": {
      "cpu_cores_min": null,
      "cpu_cores_max": null,
      "memory_mb_min": null,
      "memory_mb_max": null,
      "storage_mb_max": null,
      "storage_class": null,
      "service_account_name": null,
      "namespace": null
    },
    "clock_resolution_usecs": 1000000,
    "pin_cpus": [],
    "provisioning_timeout_secs": null,
    "max_parallel_connector_init": null,
    "init_containers": null,
    "checkpoint_during_suspend": true,
    "http_workers": null,
    "io_workers": null,
    "env": {},
    "dev_tweaks": {},
    "logging": null,
    "pipeline_template_configmap": null
  },
  "program_code": "CREATE TABLE table1 ( col1 INT );",
  "udf_rust": "",
  "udf_toml": "",
  "program_config": {
    "profile": "optimized",
    "cache": true,
    "runtime_version": null
  },
  "program_version": 2,
  "program_status": "Pending",
  "program_status_since": "1970-01-01T00:00:00Z",
  "program_error": {
    "sql_compilation": null,
    "rust_compilation": null,
    "system_error": null
  },
  "program_info": null,
  "deployment_error": null,
  "refresh_version": 4,
  "storage_status": "Cleared",
  "storage_status_details": null,
  "deployment_id": null,
  "deployment_initial": null,
  "deployment_status": "Stopped",
  "deployment_status_since": "1970-01-01T00:00:00Z",
  "deployment_desired_status": "Stopped",
  "deployment_desired_status_since": "1970-01-01T00:00:00Z",
  "deployment_resources_status": "Stopped",
  "deployment_resources_status_details": null,
  "deployment_resources_status_since": "1970-01-01T00:00:00Z",
  "deployment_resources_desired_status": "Stopped",
  "deployment_resources_desired_status_since": "1970-01-01T00:00:00Z",
  "deployment_runtime_status": null,
  "deployment_runtime_status_details": null,
  "deployment_runtime_status_since": null,
  "deployment_runtime_desired_status": null,
  "deployment_runtime_desired_status_since": null
}
GET /v0/pipelines/{pipeline_name}PATCH /v0/pipelines/{pipeline_name}