Recompile Pipeline

POST/v0/pipelines/{pipeline_name}/update_runtimePipeline Lifecycle

Try it

POST/v0/pipelines/{pipeline_name}/update_runtime

Authentication

JSON web token (JWT) or API key (http bearer)

Parameters

pipeline_name (path, required)

Recompile a pipeline with the Feldera runtime version included in the currently installed Feldera platform.

Use this endpoint after upgrading Feldera to rebuild pipelines that were compiled with older platform versions. In most cases, recompilation is not required; pipelines compiled with older versions will continue to run on the upgraded platform.

Situations where recompilation may be necessary:

To benefit from the latest bug fixes and performance optimizations.
When backward-incompatible changes are introduced in Feldera. In this case, attempting to start a pipeline compiled with an unsupported version will result in an error.

If the pipeline is already compiled with the current platform version, this operation is a no-op.

Note that recompiling the pipeline with a new platform version may change its query plan. If the modified pipeline is started from an existing checkpoint, it may require bootstrapping parts of its state from scratch. See Feldera documentation for details on the bootstrapping process.

Authentication

JSON web token (JWT) or API keyBearer token

Parameters

Path parameters

Name	Type	Required	Description
`pipeline_name`	string	Yes	Unique pipeline name

Response

200Pipeline successfully updatedapplication/json

object
Pipeline information. It both includes fields which are user-provided and system-generated.
- created_atstring (date-time)required
- deployment_desired_statusstringrequired
- deployment_desired_status_sincestring (date-time)required
- deployment_errorobject
- deployment_idstring (uuid)
- deployment_initialobject
- deployment_resources_desired_statusstringrequired
- deployment_resources_desired_status_sincestring (date-time)required
- deployment_resources_statusstringrequired
  Pipeline resources status. ```text /start (early start failed) ┌───────────────────┐ │ ▼ Stopped ◄────────── Stopping /start │ ▲ │ │ /stop?force=true │ │ OR: timeout (from Provisioning) ▼ │ OR: fatal runtime or resource error ⌛Provisioning ────────────│ OR: runtime status is Suspended │ │ │ │ ▼ │ Provisioned ─────────────┘ ``` ### Desired and actual status We use the desired state model to manage the lifecycle of a pipeline. In this model, the pipeline has two status attributes associated with it: the **desired** status, which represents what the user would like the pipeline to do, and the **current** status, which represents the actual (last observed) status of the pipeline. The pipeline runner service continuously monitors the desired status field to decide where to steer the pipeline towards. There are two desired statuses: - `Provisioned` (set by invoking `/start`) - `Stopped` (set by invoking `/stop?force=true`) The user can monitor the current status of the pipeline via the `GET /v0/pipelines/{name}` endpoint. In a typical scenario, the user first sets the desired status, e.g., by invoking the `/start` endpoint, and then polls the `GET /v0/pipelines/{name}` endpoint to monitor the actual status of the pipeline until its `deployment_resources_status` attribute changes to `Provisioned` indicating that the pipeline has been successfully provisioned, or `Stopped` with `deployment_error` being set.
- deployment_resources_status_detailsobject
- deployment_resources_status_sincestring (date-time)required
- deployment_runtime_desired_statusobject
- deployment_runtime_desired_status_sincestring (date-time)
- deployment_runtime_statusobject
- deployment_runtime_status_detailsobject
- deployment_runtime_status_sincestring (date-time)
- deployment_statusstringrequired
- deployment_status_sincestring (date-time)required
- descriptionstringrequired
- idstring (uuid)required
  Pipeline identifier.
- namestringrequired
- platform_versionstringrequired
- program_codestringrequired
- program_configobjectrequired
  Program configuration.
- program_errorobjectrequired
  Log, warning and error information about the program compilation.
- program_infoobject
- program_statusstringrequired
  Program compilation status.
- program_status_sincestring (date-time)required
- program_versioninteger (int64)required
  Version number.
- refresh_versioninteger (int64)required
  Version number.
- runtime_configobjectrequired
  Global pipeline configuration settings. This is the publicly exposed type for users to configure pipelines.
- storage_statusstringrequired
  Storage status. The storage status can only transition when the resources status is `Stopped`. ```text Cleared ───┐ ▲ │ /clear │ │ │ │ Clearing │ ▲ │ │ │ InUse ◄───┘ ```
- storage_status_detailsobject
- udf_ruststringrequired
- udf_tomlstringrequired
- versioninteger (int64)required
  Version number.

400application/json

object
Information returned by REST API endpoints on error.
- detailsobjectrequired
  Detailed error metadata. The contents of this field is determined by `error_code`.
- error_codestringrequired
  Error code is a string that specifies this error type.
- messagestringrequired
  Human-readable error message.

404Pipeline with that name does not existapplication/json

object
Information returned by REST API endpoints on error.
- detailsobjectrequired
  Detailed error metadata. The contents of this field is determined by `error_code`.
- error_codestringrequired
  Error code is a string that specifies this error type.
- messagestringrequired
  Human-readable error message.

500application/json

object
Information returned by REST API endpoints on error.
- detailsobjectrequired
  Detailed error metadata. The contents of this field is determined by `error_code`.
- error_codestringrequired
  Error code is a string that specifies this error type.
- messagestringrequired
  Human-readable error message.


curl -X POST 'https://api.example.com/v0/pipelines/{pipeline_name}/update_runtime' \
  -H 'Authorization: Bearer YOUR_TOKEN'


const response = await fetch('https://api.example.com/v0/pipelines/{pipeline_name}/update_runtime', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer YOUR_TOKEN'
  }
});
 
const data = await response.json();
console.log(data);


interface ApiResponse {
  // shape your response here
}
 
const response: Response = await fetch('https://api.example.com/v0/pipelines/{pipeline_name}/update_runtime', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer YOUR_TOKEN'
  }
});
 
const data = (await response.json()) as ApiResponse;
console.log(data);


import requests
 
url = "https://api.example.com/v0/pipelines/{pipeline_name}/update_runtime"
headers = {
    "Authorization": "Bearer YOUR_TOKEN"
}
 
response = requests.request("post", url, headers=headers)
print(response.json())


package main
 
import (
	"fmt"
	"io"
	"net/http"
)
 
func main() {
	req, err := http.NewRequest("POST", "https://api.example.com/v0/pipelines/{pipeline_name}/update_runtime", nil)
	if err != nil {
		panic(err)
	}
	req.Header.Set("Authorization", "Bearer YOUR_TOKEN")
 
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
 
	out, _ := io.ReadAll(resp.Body)
	fmt.Println(string(out))
}


{
  "id": "67e55044-10b1-426f-9247-bb680e5fe0c8",
  "name": "example1",
  "description": "Description of the pipeline example1",
  "created_at": "1970-01-01T00:00:00Z",
  "version": 4,
  "platform_version": "v0",
  "runtime_config": {
    "workers": 16,
    "max_rss_mb": null,
    "hosts": 1,
    "storage": {
      "backend": {
        "name": "default"
      },
      "min_storage_bytes": null,
      "min_step_storage_bytes": null,
      "compression": "default",
      "cache_mib": null
    },
    "fault_tolerance": {
      "model": "none",
      "checkpoint_interval_secs": 60
    },
    "cpu_profiler": true,
    "tracing": false,
    "tracing_endpoint_jaeger": "",
    "min_batch_size_records": 0,
    "max_buffering_delay_usecs": 0,
    "resources": {
      "cpu_cores_min": null,
      "cpu_cores_max": null,
      "memory_mb_min": null,
      "memory_mb_max": null,
      "storage_mb_max": null,
      "storage_class": null,
      "service_account_name": null,
      "namespace": null
    },
    "clock_resolution_usecs": 1000000,
    "pin_cpus": [],
    "provisioning_timeout_secs": null,
    "max_parallel_connector_init": null,
    "init_containers": null,
    "checkpoint_during_suspend": true,
    "http_workers": null,
    "io_workers": null,
    "env": {},
    "dev_tweaks": {},
    "logging": null,
    "pipeline_template_configmap": null
  },
  "program_code": "CREATE TABLE table1 ( col1 INT );",
  "udf_rust": "",
  "udf_toml": "",
  "program_config": {
    "profile": "optimized",
    "cache": true,
    "runtime_version": null
  },
  "program_version": 2,
  "program_status": "Pending",
  "program_status_since": "1970-01-01T00:00:00Z",
  "program_error": {
    "sql_compilation": null,
    "rust_compilation": null,
    "system_error": null
  },
  "program_info": null,
  "deployment_error": null,
  "refresh_version": 4,
  "storage_status": "Cleared",
  "storage_status_details": null,
  "deployment_id": null,
  "deployment_initial": null,
  "deployment_status": "Stopped",
  "deployment_status_since": "1970-01-01T00:00:00Z",
  "deployment_desired_status": "Stopped",
  "deployment_desired_status_since": "1970-01-01T00:00:00Z",
  "deployment_resources_status": "Stopped",
  "deployment_resources_status_details": null,
  "deployment_resources_status_since": "1970-01-01T00:00:00Z",
  "deployment_resources_desired_status": "Stopped",
  "deployment_resources_desired_status_since": "1970-01-01T00:00:00Z",
  "deployment_runtime_status": null,
  "deployment_runtime_status_details": null,
  "deployment_runtime_status_since": null,
  "deployment_runtime_desired_status": null,
  "deployment_runtime_desired_status_since": null
}