Pipeline

Pipeline is the data model we use to enrich data. There is no object called “pipeline” you can interact with. Instead, “pipeline” is an internal data representation we use for data enrichment.

A pipeline is made up of:

⛓️‍💥 Pipes and their order
🔢 Fields added by your input
🧮 Logic and conditions

A big flexible table

When thinking of UI-based applications it can help to think of the underlying data model as a table with ordered columns. Each field in your pipeline is a column. So, every property of your input objects is a column and every output field of a pipe is a column too.

A lot of thought has gone into the data model to allow complex use cases. Here’s what’s possible:

🧪 Processing individual rows and cells of tables
℁ Add columns with flexibility
😢 Processing with partially incomplete inputs

Check out this example where we try to build the core functionality of Clay in less than 1000 lines of code (we’re still trying to get below 1000).

Validation

Before pipelines are used to process data, they are checked for validity.

In a valid pipeline:

All pipes have their input requirements met
All logical conditions between pipes can be resolved

Only valid pipelines can be run. If you send an invalid pipeline to our servers, we will respond with a 400 Bad Request status code instantly.

Pipeline request

You can run a pipeline by sending a pipeline request:


const result = await fetch("https://pipe0.com/api/v1/run", {
method: "POST",
headers: {
    "Authorization": `Bearer ${YOUR_API_TOKEN}`,
    "X-Test-Mode": "true" // Remove this header for production requests
},
body: JSON.stringify({
    pipes: [
    {
        name: "PeopleBusinessEmailWaterfallV1"
    },
    {
        name: "CompanyDescriptionGoogleMapsV1"
    }
    ],
    input: [
        {
            name: "John Doe",
            companyName: "Google LLC"
        }
    ]
})
});

When your pipeline is run, input objects get converted to output records. You can see the shape of output records in the pipeline response section.

Order of pipes

You may spend time reading pipe documentation to make sure pipes are added in the correct order - this is not necessary! Internally, we analyze pipelines to find the best pipe order. As a user, the only thing you need to ensure is that the pipeline can be processed. For this all pipes must have their input requirements met.

If your application runs in the browser or is based on Javascript you can use the browser SDK to validate your pipeline before sending a request to our servers.

Async processing

When you run a pipeline, the processing happens asynchronously. The pipeline response includes an id property which you can use to poll the task until it is complete.

Note

If you’re using the browser SDK to validate pipelines you will receive a pipeline response as a validation result. However, this result will neither have an id nor status property.

Pipeline Response

Both endpoints

POST https://pipe0.com/api/v1/run (used to run the pipeline)
GET https://pipe0.com/api/v1/check (used to check the processing state)

return the same response format: A pipeline response.


{
	"success": true,
	"error": null,
	"data": {
		"id": "...",
		"type": "enrichment",
		"status": "pending",
		"order": [
			"15991b91-4b3c-4907-a7cc-b838e223004d",
			"b0015d04-f25a-46ed-8131-4af0f9365880"
		],
		"errors": [],
		"fields": {
			"id": {
				"name": "id",
				"type": "string",
				"addedBy": "input",
				"required": true
			},
			"name": {
				"name": "name",
				"type": "string",
				"addedBy": "input",
				"required": true
			},
			"email": {
				"name": "email",
				"type": "string",
				"addedBy": "input",
				"required": true
			},
			"businessEmail": {
				"name": "businessEmail",
				"type": "string",
				"addedBy": "PeopleBusinessEmailWaterfallV1",
				"required": true
			},
			"linkedInProfileUrl": {
				"name": "linkedInProfileUrl",
				"type": "string",
				"addedBy": "input",
				"required": true
			}
		},
		"records": {
			"15991b91-4b3c-4907-a7cc-b838e223004d": {
				"id": "15991b91-4b3c-4907-a7cc-b838e223004d",
				"errors": [],
				"fields": {
					"id": {
						"meta": null,
						"type": "string",
						"value": "15991b91-4b3c-4907-a7cc-b838e223004d",
						"errors": [],
						"status": "completed",
						"addedBy": "input",
						"resolvedBy": "input"
					},
					"name": {
						"meta": null,
						"type": "string",
						"value": "Jane Smith",
						"errors": [],
						"status": "completed",
						"addedBy": "input",
						"resolvedBy": "input"
					},
					"email": {
						"meta": null,
						"type": "string",
						"value": "jane@me.com",
						"errors": [],
						"status": "completed",
						"addedBy": "input",
						"resolvedBy": "input"
					},
					"businessEmail": {
						"meta": null,
						"type": "string",
						"value": "my@email.com",
						"errors": [],
						"status": "completed",
						"addedBy": "input",
						"resolvedBy": "input"
					},
					"linkedInProfileUrl": {
						"meta": null,
						"type": "string",
						"value": "https://linkedin.com/in/janesmith",
						"errors": [],
						"status": "completed",
						"addedBy": "input",
						"resolvedBy": "input"
					}
				}
			},
			"b0015d04-f25a-46ed-8131-4af0f9365880": {
				"id": "b0015d04-f25a-46ed-8131-4af0f9365880",
				"errors": [],
				"fields": {
					"id": {
						"meta": null,
						"type": "string",
						"value": "b0015d04-f25a-46ed-8131-4af0f9365880",
						"errors": [],
						"status": "completed",
						"addedBy": "input",
						"resolvedBy": "input"
					},
					"name": {
						"meta": null,
						"type": "string",
						"value": "Bob Example",
						"errors": [],
						"status": "completed",
						"addedBy": "input",
						"resolvedBy": "input"
					},
					"email": {
						"meta": null,
						"type": "string",
						"value": "bob@example.com",
						"errors": [],
						"status": "completed",
						"addedBy": "input",
						"resolvedBy": "input"
					},
					"businessEmail": {
						"meta": null,
						"type": "string",
						"value": null,
						"errors": [],
						"status": "queued",
						"addedBy": "PeopleBusinessEmailWaterfallV1",
						"resolvedBy": null
					},
					"linkedInProfileUrl": {
						"meta": null,
						"type": "string",
						"value": "https://linkedin.com/in/bobexample",
						"errors": [],
						"status": "completed",
						"addedBy": "input",
						"resolvedBy": "input"
					}
				}
			}
		},
		"isTestMode": true,
		"enrichments": [
			{
				"name": "PeopleBusinessEmailWaterfallV1",
				"config": {
					"providers": [
						{
							"name": "clearbit"
						}
					],
					"inputFieldNames": {
						"name": "",
						"email": "",
						"random": ""
					},
					"outputFieldNames": {
						"businessEmail": ""
					}
				}
			}
		],
		"organizationId": "di21noyh28g36ie146m5icew",
		"createdAt": "2025-03-06T23:51:38.256Z",
		"updatedAt": "2025-03-06T23:51:38.256Z"
	}
}

Failure modes

If your pipeline was run the endpoint always returns an HTTP 200 OK status. The result of the processing can be read from the global task status as well as the field status’ of each individual record field. Please remember that just because your pipeline has passed validation it does not mean that processing will succeed.