Skip to main content

Varicent ELT Assistant

Amazon S3 import

Connect your Varicent ELT and an Amazon S3 bucket in your AWS account to import your data. 

The Varicent ELT user interface provides two authentication methods, the IAM role and the Access Key ID/Secret Access Key options, to connect to Amazon S3. The Access Key ID/Secret Access Key option can use either temporary or long-term credentials: 

  •  IAM Role: Choose this option when your AWS administrator permits cross-account role access. Your administrator creates a role with access only to the required S3 buckets and configures its trust policy. The IAM role option avoids sharing long-term access keys. For more information, read the Delegate access across AWS accounts using IAM roles documentation.

  • Temporary AWS security credentials use an Access Key ID, Secret Access Key, and a session token. If the IAM role is unavailable, you can connect using temporary credentials, which expire. After expiration, the connection stops working until it's updated with a new credential set. This option is best for temporary or manually managed access, not unattended scheduled pipelines. For more information, read the Using temporary credentials with AWS services documentation.

  • Long-term IAM access key: If neither the IAM role nor temporary AWS security credentials are practical, you can choose the Access Key ID/Secret Access Key option. The access remains valid until it is turned off or deleted. Your administrator must restrict the IAM user to the required bucket, store the secret securely, and rotate it regularly. The secret cannot be recovered, so ensure that you save it. For more information, read the AWS access keys and security guidance documentation.

Varicent ELT Amazon S3 import supports the following file types:

  • CSV

  • Excel

  • Parquet

  • .txt

Importing data

  1. In Varicent ELT, select the Data module from the sidebar navigation pane.

  2. Click Import data.

  3. In the Third party sources section, click Amazon S3.

  4. Click New connection.

  5. On the Connect to your Data page, under Connector name, enter the name of the connection.

  6. Under the Authentication method drop-down, select one of the following options:

    1. IAM role:

      1. Under Account ID, enter the ID of the AWS account that you want to use.

      2. Under Bucket Name, enter the name of the AWS bucket that you connect to.

      3. Under the Region drop-down menu, select the region of your Amazon S3 bucket, and then click Next.

    2. Access Key ID/Secret Access Key:

      1. Under Access Key ID enter a valid Access Key ID.

      2. Under Secret Access Key, enter the valid secret access key.

      3. Under Session Token, optionally enter the session token. 

        For temporary AWS credentials, enter the session token provided by AWS. For long-term credentials, leave this field blank. For more information, see the Use temporary credentials with AWS resources documentation.

        Note

        Any expired credentials must be replaced by editing the connection before browsing or importing from Amazon S3 again.

      4. Under Bucket Name, enter the name of the AWS bucket that you connect to.

      5. Under the Region drop-down menu, select the region of your Amazon S3 bucket, and then click Next.

  7. If you are setting up the IAM role, on the next Connect to your Data page, follow the on-screen steps to set up your account.

    1. Follow the on-screen steps to create a new policy in your AWS account.

      Note

      The policy body is pre-populated with the information that you had entered in the previous step. Use the copy button to copy the code.

    2. Follow the on-screen steps to create a new role in your AWS account.

      Note

      The account ID value depends on the environment you are in.

    3. In the Role Name field, copy over the pre-populated role name and use that exact name of the the role in AWS.

    4. Click Next.

  8. On the Select data page, under Directory, select the directory and the file that you want to import. For example, if you want to select Parquet files, ensure you select the folder that contains them.

  9. Under File type, select the file type that you want to import, such as CSV, Excel, .txt or Parquet.

    Caution

    If you select a directory with Parquet files, ensure that you select the Parquet file type. If you select another type of file, the import will fail.

    Note

    If you select the Parquet file option, you won't have to specify the Delimiter and other configuration options; just follow the import wizard.

    Delimiter, quote, row exclusions, and header only apply to .CSV and .txt files. Excel and Parquet files follow different file-specific steps.

  10. Under Delimiter, select the delimiter that you want to use from the following delimiters in the drop-down field:

    • Comma (,)

    • Pipe (|)

    • Semi-colon (;)

    • Space ( )

    • Tab ( )

    • Tilde (~)

  11. Under Quote, select the type of quotes that you want to use, such as double quotes (").

  12. Under Exclude excess data, optionally specify how many rows to exclude from the beginning of the file in Ignore first, and from the end in Ignore last. Leave both values as 0 if you do not want to exclude any rows.

  13. Under Header, check to designate the first unignored row as the column header. Leave unchecked to have column headers automatically generated, and then click Next.

  14. On the Import data page, review your data and select the data to obfuscate (if any), select Keep out of bound dates to maintain any dates outside the date range, and then click Next.

  15. On the Filter data page, optionally click Filter data to refine your data, or click Import data to continue.

    1. Filter the rows in your data by selecting the column to filter, the filter category, and the value or text.

    2. Click Import data to continue.

    Note

    For more information about how to configure data filters, see: Filter data.

  16. On the Review page, review the data sets that you are importing.

  17. Click Import data.

    The data is available on the My Data page.