This guide provides instructions on how to set up access to and use Brightloom’s SFTP server, which is located at: sftp.prod.brightloom.com

Please review this guide carefully to plan your ETL process for:

  • Pushing data to Brightloom SFTP

  • Pulling data from Brightloom SFTP

General Process

  1. Gain access to Brightloom’s SFTP server

    1. Generate SSH private key and public key pair. Send the public key to your Brightloom contact

    2. Your Brightloom contact will reply when your server access has been setup with an assigned username and other connection information, usually within 3 business day

    3. Use a standard SFTP client to access the server using your private key, and confirm your access

  2. Transfer data through SFTP (one of the following or both)

    1. Inbound: Transfer data (ex: transaction data, campaign data, etc.) from your system to Brightloom

    2. Outbound: Transfer data (ex: SmartSegment files) from Brightloom to your system

SFTP Server Connection

Brightloom’s SFTP uses SSH key authentication. The SFTP industry standard is to use SSH keys, as SFTP stands for SSH File Transfer Protocol. This is done instead of user/password authentication, as issuing a password and passing it around via email is insecure.

Generate your SSH keys 

There are various ways of generating ssh keys. Remember to always keep the private key private. Never send or store the private key outside a secure environment, including over email. The private key is basically your identity.

Here are some useful pointers on how to generate ssh keys:

After generating SSH keys, you should have saved the information in two files, one for the private key and one for the public key. Send the public key file to your Brightloom contact.

Create SFTP Account

Once the public key has been received, Brightloom will create an SFTP account with your public key and provide you with the connection information:

  • Host: sftp.prod.brightloom.com

  • Port: 22

  • Username: To be assigned by Brightloom. It will be from 3 to 32 characters, consisting of a-z, A-Z, 0-9, underscore, and hyphen. It will not start with a hyphen.

  • Root directory: To be assigned by Brightloom.

Access the SFTP Server

Please confirm access once you have received the connection information from Brightloom. You should use your private key (that’s paired with the public key you provided to Brightloom) to access the SFTP server.

The SFTP server supports any standard SFTP client. Some examples include:

  • OpenSSH – A Macintosh and Linux command line utility.

  • WinSCP – A Windows-only graphical client.

  • Cyberduck – A Linux, Macintosh, and Microsoft Windows graphical client.

  • FileZilla – A Linux, Macintosh, and Windows graphical client.

Here’s an example of accessing the SFTP server using command line interface and the sftp command line client:

  • id_rsa is the private key file, and ~/.ssh/ is where the private key file is stored

  • username will be provided

Data Transfer

You can use Brightloom’s SFTP server to push your data to Brightloom (inbound), or pull SmartSegment files from Brightloom (outbound).

Inbound

If you choose to use Brightloom’s SFTP server to push your data to Brightloom (inbound), once your data arrives to Brightloom SFTP, the data will be ingested into our data lake for processing. In order to ensure a successful data ingestion and processing,

  • Please follow the guidelines below when sending and/or setting up your data uploads

  • Please also confirm how Brightloom should expect to receive your data on Brightloom SFTP by filling out the “SFTP Implementation Worksheet

Data Source

  • Purpose: Help Brightloom understand where the data is extracted from.

  • Recommended: Data source(s) with which would allow you to implement a consistent ETL process for pushing data to Brightloom SFTP.

  • Data provider to confirm: In the “SFTP Implementation Worksheet”, please fill out the “Data Source” column to confirm the data source for each file.

  • Change request: If you are changing the data sources for your uploads, please inform your Brightloom contact about the change.

File List

  • Purpose: Help Brightloom understand what files will be included in your uploads.

  • Recommended: Brightloom prefers to receive separate files that match Brightloom’s standard tables (see “Brightloom CGP Data Requirement” for details.).

  • Data provider to confirm: In the “SFTP Implementation Worksheet”, please fill out the “File List” column with a keyword that allows Brightloom to identify each file.

  • Change request: If you are adding/removing any file from your uploads, please inform your Brightloom contact about the change.

Schedule

  • Purpose: Help Brightloom understand when we can expect to receive data refresh for each file. A scheduled data refresh will help inform and improve Brightloom’s data model and will allow us to measure the results from the executed campaigns.

  • Recommended: Brightloom recommends the data provider to automate the file transfer and send daily data refreshes to Brightloom SFTP. If the file upload is manual, please send the updated data at least once a week.

  • Data provider to confirm: In the “SFTP Implementation Worksheet”, please fill out the “Schedule” column to confirm the data refresh cadence and schedule for each file.

  • Change request: If you are changing the upload schedule for any file, please inform your Brightloom contact about the change.

Directory Structure

Purpose: A consistent directory structure for uploads would allow Brightloom to ingest the data files programmatically.

Recommended:

  1. sftp.prod.brightloom.com is the SFTP server that you will use your username and private key to access.

  2. Once you are connected to the server, in the root folder, you should see an inbound and an outbound folder. Please make sure you upload folders/files to the inbound folder.

  3. In the inbound folder,

    1. If all data files will be uploaded at same cadence and schedule

      1. Please create a <YYYY-MM-DDThhmmZ> timestamp folder as the parent folder of all the data files when you share a new data drop.

      2. The ingestion job references the <YYYY-MM-DDThhmmZ> timestamp to differentiate the data refreshes. It can be the date and time when you pull, or send the data.

      3. ISO 8601 timestamp string is preferred for the folder name (the “Z” suffix is the time zone designator for the zero UTC offset).

    2. If some data files need to be uploaded separately at a different schedule (usually due to data being extracted from different data sources)

      1. Please create a <data source> folder for each data source that will have a different upload schedule

      2. In each <data source> folder, follow steps for scenario 3a.

    3. If there is a need to organize your uploads in a customized directory structure. Please define the directory structure in the “SFTP Implementation Worksheet” and let your Brightloom contact know.

Here’s an example of how the data drops could look like for scenario 3a:

All data files will be included in the daily data feed.

Here’s an example of how the data drops could look like for scenario 3b:

Transaction data files will be uploaded daily

Product/menu data will be uploaded separately at a different schedule

Guests profile data will be uploaded separately at a different schedule

Campaign data will be uploaded separately at a different schedule

Data provider to confirm: In the “SFTP Implementation Worksheet”, please fill out the “Directory Structure” column to confirm the folder structure will be used for each file.

Change request: If you would like to make a change to the directory structure for your uploads, please inform your Brightloom contact about the change.

Data Refresh Method

  • Purpose: Help Brightloom understand what data (full data dumps vs. incremental updates) will be included in each data file so Brightloom can process accordingly.

  • Recommended:

    • Brightloom CGP supports these methods:

    • Full load for all data available (all rows in the table)

    • Full load for the last X days (all rows in the table with a date cutoff)

    • Incremental load (new and updated rows only)

You may use different methods for different files (e.g. incremental load for transactions vs. full load for products.)

  • Data provider to confirm: In the “SFTP Implementation Worksheet”, please fill out the “Refresh Method” column to confirm the method used for each file.

  • Change request: If you would like to change the load method for any of the data files, please inform your Brightloom contact about the change.

File Format

  • Purpose: Help Brightloom understand how each file would look like (file extension, column headers, etc.) so Brightloom can process accordingly.

  • Recommended: Brightloom CGP currently supports CSV and Parquet formats. The data files do not need to be password protected as they are being sent using SSH File Transfer Protocol.

  • Data provider to confirm: In the “SFTP Implementation Worksheet”, please fill out the “File Extension”, “Header Included”, and “Field Delimiter”columns for each file.

  • Change request: If there has been any change in any data file, please inform your Brightloom contact about the change.

Outbound

Brightloom SmartSegment exports are available through CGP console - SmartSegment Explorer. You may choose to also receive the SmartSegment exports through Brightloom SFTP.

File List

When new SmartSegments are generated, there will be two files exported to the outbound folder for each SmartSegment:

  • Treatment group file: <YYYYMMDD>_<Product ID>_<Promo Value>.csv

  • Control group file: <YYYYMMDD>_<Product ID>_<Promo Value>_control.csv

For example, when 10 SmartSegments are generated, there will be 20 CSV files exported to the SFTP outbound folder (see "SmartSegment Export Specification" for file schema details.).

Schedule

Default: By default, SmartSegments will be generated and refreshed in the CGP console on a daily basis. If you choose to enable the export through SFTP, SmartSegment files will be exported to the outbound  folder at the same time.

Change request: If you would like to change the export schedule, please let your Brightloom contact know the preferred schedule. 

Directory Structure

Default:

  1. sftp.prod.brightloom.com is the SFTP server that you will use your username and private key to access.

  2. Once you are connected to the server, in the root folder, you should see an inbound and an outbound folder. Brightloom will export SmartSegment files to the outbound folder directory.

  3. Please pull *.csv files from the outbound folder to process.

  4. Once the files are pulled, it is recommended that you move those files from the outbound folder to a subfolder (e.g. outbound/complete)

    1. Permission to create folders (e.g. outbound/complete) and delete/move files has been enabled for your sftp account.

    2. Over time, the subfolder (e.g. outbound/complete) gets larger and slower as it contains all history. And the outbound folder stays small and fast as it only has files that need to be pulled.

Here’s an example of how the SmartSegment exports could look like:

Change request: If you would like to receive SmartSegment files in a different directory structure, please let your Brightloom contact know the preferred structure. 

_________________________

1 As of Jan. 2020, please use the following algorithms and key sizes as older methods are no longer secure:

  • DSA: It’s unsafe and even no longer supported since OpenSSH version 7, you need to upgrade it!

  • RSA: It depends on the key size. If it has 3072 or 4096-bit length, then you’re good. Less than that, you probably want to upgrade it. The 1024-bit length is even considered unsafe.

  • ECDSA: It depends on how well your machine can generate a random number that will be used to create a signature. There’s also a trustworthiness concern on the NIST curves that is being used by ECDSA

  • Ed25519: It’s the most secure public-key algorithm available today, but not supported by many applications