Skip to main content

Varicent ELT Assistant

Filter

Use the Filter tool to explore a subset of your data. You can also use the  Filter tool to remove unwanted rows.

Input

The Filter tool requires one data input.

Filter tool versions

The Filter tool has two different versions available to use in the pipe. You can choose which behaviour you want to go with, depending on the needs of your pipe.

A pop-up appears asking if you want to continue with the previous behaviour or upgrade to the newest version, with the details:

  • Previous behaviour: The Filter tool in Big Pipe handled NULLs and empty values differently than the Standard Pipe. It didn’t treat two NULLs as equal and returned different results for filters like equals, contains, starts with, and ends with, especially when NULL or empty values were involved. This caused mismatches in results between Big Pipe and Standard Pipe.

  • Updated behaviour: Filtering in Big Pipe is now aligned with Standard Pipe NULLs and empty values are treated consistently across both. Equals, not equals, and contains now work the same, including support for NULL comparisons.

    • For contains, starts with, or ends with: rows with NULLs or empty values are not included in the result.

    • For not contains: rows with NULLs are included.

Configuration

Use the following configuration options to help create your Filter configuration.

Configuring the Filter tool

  1. Go to the Pipes module from the side navigation bar.

  2. From the Pipes tab, click an existing pipe to open, or create a new pipe. To create a new pipe, read the Creating a pipe documentation.

  3. In Pipe builder, add your data source.

  4. Click symon_add_icon.png + Tool.

  5. In the search bar, search for Filter. Click + Add tool.

    Tip

    You can also find the Filter tool in the Clean section.

  6. Connect the tool to your data set.

  7. From the configuration pane, under the If all drop-down menu, select the criteria:

    • If all: If all of the following are true.

    • If any: If any of the following are true.

  8. Under Select column, select the column to use to filter.

  9. Next, select one of the following filters to use:

    • Equal

    • Not Equal

    • Less Than

    • Less Than or Equal

    • Greater Than

    • Greater than or Equal

    • Starts With

    • Ends With

    • Contains

    • Not Contains

    • Is Null

    • Is Not Null

  10. Under Value or text, select the value or text to use, or start typing to find a value or text.

  11. (Optional) Under + click either Statement, optionally add an AND statement, or Group to optionally add an OR statement. Filters data based on the specific conditions. Only rows that meet the conditions in all  AND statements will be included.

Usage example

In this example, we have a data set that contains customer IDs and an NPS score.

CustomerID

NPS

2453

2

3243

4

3452

10

4324

6

5324

8

Let's say we want to explore data only for customers considered "detractors" (a score of 6 or less). We can remove the rows we don't need by using the Filter tool:

  1. In the left side literal or column field, select or type "NPS".

  2. Select or type "Less Than or Equal".

  3. In the right side literal or column field, type "6".

As a result, the data set now looks like this:

CustomerID

NPS

2453

2

3243

4

4324

6