Inputs

Input data is rarely clean. pipe0 sanitizes every input object before processing: it fixes common format errors, regenerates invalid values where it can, and handles missing fields per record instead of failing the whole task.

Input sanitation

Enrichment input usually comes from CRMs, ATSs, or web forms, and carries the mistakes users typed into them.

Cleanup

The following request payload contains common errors but will be processed successfully.

{
  "pipes": [
    {
      "pipe_id": "company:identity@3"
    }
  ],
  "input": [
    {
      "id": 1,
      "name": "Susi Jui",
      "company_name": "Pipe0",
      "company_domain": "https://www.pipe0.com/about", // protocol, www and path get stripped
      "email": "mailto:susi@pipe0.com", // "mailto:" prefix
      "personal_website_url": "wwwww.susi.com" // wwww instead of www
    },
    {
      "id": 2,
      "name": "Tom Schmidt",
      "company_name": "Pipe0",
      "company_domain": "not today" // invalid: expected a domain
    }
  ]
}

Here's how we clean this request:

1. Parse URLs into a consistent format and clean common mistakes
2. Parse email addresses into a consistent format and clean common mistakes
3. Fix obvious typos and remove invalid characters
4. Convert data formats on demand (int to float, float to int, int to string)

Regeneration

In our example, "company_domain": "not today" is not a valid domain.

Because company_domain is an output field of company:identity@3, we can find the correct value.

During processing, company:identity@3 detects that company_domain is of invalid format and replaces it with the correct value.

The result may look like this:

{
    "id": 2,
    "name": "Tom Schmidt",
    "company_name": "Pipe0",
    "company_domain": "pipe0.com" // healed
}

Incomplete data

Input data is often incomplete. Failing the entire task because one input object cannot be processed would be impractical, so validation works per record.

Partially missing input fields

If at least one input object can be processed, pipeline validation passes.

Take the following request payload:

{
  "pipes": [
    {
      "pipe_id": "company:identity@3"
    }
  ],
  "input": [
    {
      "id": 1,
      "name": "Susi Jui",
      "company_name": "Pipe0"
    },
    { // CANNOT be processed by "company:identity@3"
      "id": 2
      // required `company_name` missing
    }
  ]
}

The pipe company:identity@3 requires the input field company_name, which is not present in record id=2. In this case:

• Pipeline validation passes.
• Record id=1 is processed in full.
• Record id=2 has failed fields.

No input object has the required input fields

Another example:

{
  "pipes": [
    {
      "pipe_id": "company:identity@3"
    }
  ],
  "input": [
    { // CANNOT be processed by "company:identity@3"
      "id": 1,
      "name": "Susi Jui"
    },
    { // CANNOT be processed by "company:identity@3"
      "id": 2,
      "name": "Tom Schmidt"
    }
  ]
}

No input object has the required field company_name. The request fails during request validation, before processing starts.

Never fail a task

If you don't want to handle failing tasks, there's an escape hatch: define the expected input fields and set them to null. Pipeline validation passes, the task never fails, and only individual fields fail instead.

Input expansion

When you enrich data with pipe0 you transform your input objects into output records. An input object may look like this:

{
    "id": 2,
    "name": "Tom Schmidt"
}

Some interactions require you to reprocess previously processed fields. For this, it is common to transform your output records back to input objects. By doing so, previous processing information is lost. This includes metadata like the result of a waterfall or UI widgets.

If you pass a plain value to the API, it is always marked as resolved_by: input.

Input expansion is the alternative: pass your inputs fully or partially expanded, as the field value shape of the response object.

{
  "id": 2,
  "name": {
      "value": "Tom Schmidt",
      "status": "completed",
      "type": "string",
      "reason": null,
      "meta": null,
      "ui": {
        "severity": "none"
      }
  }
}

Expanding inputs gives you control but shifts the responsibility of providing valid input states to you.