Skip to main content

Varicent ELT Assistant

Amazon S3 export

Connect your Varicent ELT and an Amazon S3 bucket in your AWS account to export your data. This connection allows Varicent ELT to write data to Amazon S3.

Set up the connector to be available for use with the Export tool. You only need to set up the connection once. After the connection is set up, Varicent ELT can receive exported files to the Amazon S3 bucket.

The Varicent ELT user interface provides two authentication methods, the IAM role and the Access Key ID/Secret Access Key options to connect to Amazon S3. The Access Key ID/Secret Access Key option can use either temporary or long-term credentials: 

  •  IAM Role: Choose this option when your AWS administrator permits cross-account role access. Your administrator creates a role with access only to the required S3 buckets and configures its trust policy. The IAM role option avoids sharing long-term access keys. For more information, read the Delegate access across AWS accounts using IAM roles documentation.

  • Temporary AWS security credentials use an Access Key ID, Secret Access Key, and a session token. If the IAM role is unavailable, you can connect using temporary credentials, which expire. After expiration, the connection stops working until it's updated with a new credential set. This option is best for temporary or manually managed access, not unattended scheduled pipelines. For more information, read the Using temporary credentials with AWS services documentation.

  • Long-term IAM access key: If neither the IAM role nor temporary AWS security credentials are practical, you can choose the Access Key ID/Secret Access Key option. The access remains valid until it is turned off or deleted. Your administrator must restrict the IAM user to the required bucket, store the secret securely, and rotate it regularly. The secret cannot be recovered, so ensure that you save it. For more information, read the AWS access keys and security guidance documentation.

Create the connection

Create the connection between Varicent ELT and Amazon S3. Create a new Amazon S3 connection or use an existing one.

Creating a new connection

  1. In Varicent ELT , from the sidebar navigation pane, select the Pipe module, and then go to your pipe.

  2. Add the Export tool.

  3. In the Export tool configuration section, click the connection drop-down.

  4. From the connection drop-down, click + New connection.

  5. Create the new Amazon S3 connection for the Export tool.

  6. On the Connect to your Data page, under Connector name, enter the name of the connection.

  7. Under the Authentication method drop-down, select one of the following options: 

    1. IAM role:

      1. Under Account ID, enter the ID of the AWS account that you want to use.

      2. Under Bucket Name, enter the name of the AWS bucket that you connect to.

      3. Under the Region drop-down menu, select the region of your Amazon S3 bucket, and then click Next.

    2. Access Key ID/Secret Access Key:

      1. Under Access Key ID, enter a valid Access Key ID.

      2. Under Secret Access Key, enter the valid secret access key.

      3. Under Session Token, optionally enter the session token.

        For temporary AWS credentials, enter the session token provided by AWS. For long-term credentials, leave this field blank. For more information, see the Use temporary credentials with AWS resources documentation.

        Note

        Any expired credentials must be replaced by editing the connection before exporting from Amazon S3 again.

      4. Under Bucket Name, enter the name of the AWS bucket that you connect to.

      5. Under the Region drop-down menu, select the region of your Amazon S3 bucket, and then click Next.

  8. On the next Connect to your Data page, follow the on-screen steps to set up the IAM role.

    1. Follow the on-screen steps to create a new policy in your AWS account.

      Note

      The policy body is pre-populated with the information that you had entered in the previous step. Use the copy button to copy the code.

    2. Follow the on-screen steps to create a new role in your AWS account.

      Note

      The account ID value depends on the environment you are in.

    3. In the Role Name field, copy over the pre-populated role name and use that exact name of the role that was created in AWS.

    4. Click Finish.

      The new export connection is created and available to use on your pipe page.

  9. For IAM role only, on your pipe page in the Export configuration pane, enter the following information:

    1. Select the existing connection by its connection name.

    2. In the File name field, enter the name of the file.

    3. Click Add timestamp suffix to include the timestamp in the export.

      The format would appear as <filename>-<YYYY-MM-DDThh_mm_ss.msZ>.<extension>. For example, you could have a timestamp of SalesQ3_Update-2023-07-21T16_28_54.672Z.xlsx

    4. In the Export type drop-down, select CSV, Excel, or text.

    5. In the Delimiter drop-down menu, select the delimiter to use.

      Note

      The Delimiter field only appears when the export type is set to text.

    6. In the Export destination drop-down, choose where you want to export the data.

    7. Click Export now.

      You will be redirected to the Create an Export page. For more information, see Exporting data.

Using an existing connection

  1. In Varicent ELT, from the sidebar navigation pane, select the Pipe module, and then go to your pipe.

  2. Add the Export tool.

  3. In the Export tool configuration section, click the Connection drop-down and select an existing Amazon S3 connection.

  4. In the File name field, enter the name of the file.

  5. Click Add timestamp suffix to include the timestamp in the export.

    The format would appear as <filename>-<YYYY-MM-DDThh_mm_ss.msZ>.<extension>. For example, you could have a timestamp of SalesQ3_Update-2023-07-21T16_28_54.672Z.xlsx

  6. In the Export type drop-down, select CSV, Excel, or text.

  7. In the Delimiter drop-down menu, select the delimiter to use.

    Note

    The Delimiter field only appears when the export type is set to text.

  8. In the Export destination drop-down, choose where you want to export the data.

  9. Click Export now.

    You will be redirected to the Create an Export pane. For more information, see Exporting data.