Tokenize data

This guide explains how to tokenize and detokenize your data with a Skyflow vault.

Concepts

When you tokenize data, you insert a value into a table and get a token that represents the value in return.

When you detokenize data, you submit a token and receive the value in return.

Each column in your vault can have a token format that determines how the column generates tokens for its values:

• UUID deterministic token: This format generates random UUIDs. If a value appears in the column more than once, each instance of the value generates the same token.
• Format preserving deterministic token: This format generates random tokens but preserves the format based on the specified regular expression (regex) expression. If a value appears in the column more than once, each instance of the value generates the same token.
• Format preserving token: This format generates random tokens but preserves the format based on the specified regex expression. Tokens have no connection to the column value.
• UUID token: This format generates random UUIDs. Tokens have no connection to the column value.
• Transient UUID token: This format generates random UUIDs. Tokens have no connection to the column value. This format purges data after a predefined time to live (TTL) expires. If the token expires, detokenization fails. When you update a transient field with a new value, the TTL resets.
• Preserve email domain deterministic token: This format preserves the domain and top level domain (TLD) of an email address. For example, “jane@example.com” might become “yrnakewnmfpanwnfszaejd@email.com”. If a value appears in the column more than once, each instance of the value generates the same token. Only works with columns formatted as the Email data type.

Each column can belong to a column group, which lets columns in the group generate the same deterministic tokens for the same values, regardless which tables the columns belong to. This lets you track common values across tables without exposing yourself to sensitive data.

Prerequisites

Studio

• Sign in to your Skyflow account. If you don’t have an account, sign up for a free trial.

• A vault

API

• Sign in to your Skyflow account. If you don’t have an account, sign up for a free trial.

• A device with the following tools available:

  • A terminal that can run bash commands
  • curl
  • jq 1.6 or greater

• A bearer token to authenticate API calls. For a short-lived token, use the following process. To generate tokens from service accounts, see Authenticate.

  1. In Studio, click your account icon and choose Generate API Bearer Token.

  2. Click Generate Token.

• Skyflow account, vault, and workspace details:

  1. In Studio, click vault menu icon > View vault details.
  2. Note your Account ID, Vault ID, and Vault URL values.

• Your environment’s Management API URL:

  • Trial or Production: https://manage.skyflowapis.com
  • Staging: https://manage.skyflowapis-preview.com

• Set environment variables for your account and vault details:

  export ACCOUNT_ID=$ACCOUNT_ID
  export VAULT_ID=$VAULT_ID
  export VAULT_URL=$VAULT_URL
  export MANAGEMENT_URL=$MANAGEMENT_URL
  export TOKEN=$TOKEN

The following content uses the Quickstart vault template. Adapt your commands accordingly.

Configure tokenization in your vault

The credit_cards table in the Quickstart vault has four columns configured for tokenization:

• cardholder_name, expiry_month, and expiry_year generate UUID deterministic tokens.
• card_number generates format preserving deterministic tokens.

The persons table has with five columns configured for tokenization:

• name, date_of_birth, and state generate UUID deterministic tokens.
• email_address and ssn generate format preserving deterministic tokens.

Studio

You can only set column groups via the Management API.

To set the token format for a column,

1. Navigate into your vault.
2. Click Edit schema.
3. Click the arrow in header of the column you want to tokenize, then click Edit column.
4. Click Continue.
5. In the Tokens section, select the applicable tokenization option for the column.
6. Click Continue through the remaining sections, then click Save column.
7. Repeat these steps for each column you want to tokenize. After you update all the columns you want to tokenize data for, click Publish.

API

You can set the token format for columns or add a column to a column group with the Management API.

To update a column’s token format, you need to update the column’s skyflow.options.default_token_policy option. To add a column to a column group, all columns in the group need the same values for a variety of options. See Column groups.

1. Export $ACCOUNT_ID, $VAULT_ID, $VAULT_URL, and $TOKEN values as described in Prerequisites.

2. Get your current vault information. The following command stores the schemas from the curl response’s .vault.schemas field to a vaultSchemas.json file.

   curl -s -X GET "$MANAGEMENT_URL/v1/vaults/$VAULT_ID" \
     -H "X-SKYFLOW-ACCOUNT-ID: $ACCOUNT_ID" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     | jq -r ".vault.schemas" > vaultSchemas.json

3. Open the vaultSchemas.json file and locate each column you want to update.

   For example, the credit_cards.name column in a Quickstart vault might look like this:

   {
     "name": "name",
     "datatype": "DT_STRING",
     "isArray": false,
     "tags": [
         {
           "name": "skyflow.options.data_type",
           "values": [
             "skyflow.Name"
           ]
         },
         {
           "name": "skyflow.options.default_dlp_policy",
           "values": [
             "REDACT"
           ]
         },
         {
           "name": "skyflow.validation.regular_exp",
           "values": [
             "^$|^[A-za-z ,.'-;]+$"
           ]
         },
         {
           "name": "skyflow.options.default_token_policy",
           "values": [
             "DETERMINISTIC_UUID"
           ]
         }
       ],
       "properties": null,
       "index": 0
     },

4. For each column, add or update the following objects in the .tags array to set the token format and other options for column groups. For each option’s accepted values, see Vault settings.

   This example sets the token type to deterministic UUID tokens:

   {
     "name": "skyflow.options.default_token_policy",
     "values": ["DETERMINISTIC_UUID"]
   }

   This example creates the name column group and sets the column to generate deterministic UUID tokens:

   { "name": "skyflow.options.column_group", "values": ["name"] },
   { "name": "skyflow.options.data_type", "values": ["skyflow.Name"] },
   { "name": "skyflow.options.default_token_policy", "values": ["DETERMINISTIC_UUID"]},
   { "name": "skyflow.options.default_dlp_policy", "values": ["REDACT"]},
   { "name": "skyflow.options.format_preserving_regex", "values": []},
   { "name": "skyflow.options.find_pattern", "values": []},
   { "name": "skyflow.options.replace_pattern", "values": []},
   { "name": "skyflow.options.regular_exp", "values": ["^$|^[A-za-z ,.'-;]+$"]}

   Updating the credit_cards.name column for the name column group looks like this:

   {
     "name": "name",
     "datatype": "DT_STRING",
     "isArray": false,
     "tags": [
       {
         "name": "skyflow.options.data_type",
         "values": ["skyflow.Name"]
       },
       {
         "name": "skyflow.options.column_group",
         "values": ["name"]
       },
       {
         "name": "skyflow.options.default_token_policy",
         "values": ["DETERMINISTIC_UUID"]
       },
       {
         "name": "skyflow.validation.regular_exp",
         "values": ["^$|^[A-za-z ,.'-;]+$"]
       },
       {
         "name": "skyflow.options.default_dlp_policy",
         "values": ["REDACT"]
       },
       {
         "name": "skyflow.options.format_preserving_regex",
         "values": []
       },
       {
         "name": "skyflow.options.find_pattern",
         "values": []
       },
       {
         "name": "skyflow.options.replace_pattern",
         "values": []
       }
     ],
     "properties": null,
     "index": 0
   }

5. Save and close vaultSchemas.json.

6. Update your vault’s schemas, filling the object format expected by Update Vault with the content of vaultSchemas.json:

   curl -s -X PATCH "$MANAGEMENT_URL/v1/vaults/$VAULT_ID" \
     -H "X-SKYFLOW-ACCOUNT-ID: $ACCOUNT_ID" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d "$(jq -r "{ vaultSchema: { schemas: . } }" vaultSchemas.json)"

Tokenize your data

After you configure your vault, it automatically generates tokens based on your tokenization settings. Use the following API calls to insert new records and retrieve existing tokens.

Insert and tokenize your data

Insert a new record and return the skyflow_id and tokens for your data.

Postman

1. Expand the QuickstartVault directory to the credit cards CRUD APIs folder. Choose the Post insert credit_cards method.

2. In the Body, enter values for the parameters. Set tokenization to “true.”

3. Click Send to make the call. Skyflow returns a skyflow_id and tokens for the record you just inserted.

The card_number maintains the standard 16-digit format because of format-preserving deterministic tokenization.

Leave this tab open for quick access to the skyflow_id. In the next section, you’ll use the ID to retrieve your tokenized data.

curl

Schema

In a terminal, run the following command. Replace the $VAULT_URL, $VAULT_ID, and $BEARER_TOKEN parameters with your vault-specific values.

   curl -i -X POST "$VAULT_URL/v1/vaults/$VAULT_ID/credit_cards" \
   -H "Authorization: Bearer $BEARER_TOKEN" \
   -d '{
       "tokenization": true,
       "records": [
           {
               "fields": {
                   "card_number": "'"$CARD_NUMBER"'"
                   "cardholder_name": "'"$CARDHOLDER_NAME"'"
                   "expiry_month": "'"$EXPIRY_MONTH"'"
                   "expiry_year": "'"$EXPIRY_YEAR"'"
               }
           }
       ]
   }'

Skyflow returns a skyflow_id and tokens for the record you just inserted.

{
  "records": [
    {
      "skyflow_id": "$SKYFLOW_ID",
      "tokens": {
        "card_number": "$CARD_NUMBER TOKEN",
        "cardholder_name": "$CARDHOLDER_NAME TOKEN",
        "expiry_month": "$EXPIRY_MONTH TOKEN",
        "expiry_year": "$EXPIRY_YEAR TOKEN"
      }
    }
  ]
}

Sample

In a terminal, run the following command:

   curl -i -X POST 'ebfc9bee4242.vault.skyflowapis.com/v1/vaults/x3d7caf822364170856c6f9d9a3bdf3f/credit_cards' \
   -H 'Authorization: Bearer eyJhbGciOiJSU' \
   -d '{
       "tokenization": true,
       "records": [
           {
               "fields": {
                "card_number": "8887777796",
                "cardholder_name": "jane doe",
                "expiry_month": "11",
                "expiry_year": "2025"
               }
           }
       ]
   }'

Skyflow returns a skyflow_id and tokens for the record you just inserted.

{
  "records": [
    {
      "skyflow_id": "85e21b54-481f-42c6-8054-f3f770a77386",
      "tokens": {
        "card_number": "8263-6743-4832-2862",
        "cardholder_name": "5d25061a-082e-495d-8df5-942f85d96721",
        "expiry_month": "0792b4ab-4974-4b86-9c64-59c601c13d21",
        "expiry_year": "47deed73-c29a-404f-be87-855f7be791ea"
      }
    }
  ]
}

The card_number maintains the standard 16-digit format because of format-preserving deterministic tokenization.

The preceding example uses these variables:

Variable	Type	Description
VAULT_URL	path (string)	Your workspace URL. Found on your Vault Details page.
VAULT_ID	path (string)	The vault ID. Found on your Vault Details page.
BEARER_TOKEN	header (string)	A Bearer token.

Store the skyflow_id for quick access. In the next section, you’ll use the ID to retrieve your tokenized data.

Get tokens for your stored data

Retrieve tokens for data already in the vault with the Get Record API.

Postman

1. Expand the QuickstartVault directory to the credit cards CRUD APIs folder. Choose the GET get credit_cards method.

2. From the Params section, select the checkboxes for the redaction and tokenization rows to enable them. Change the tokenization value to “true.”

3. Copy and paste the skyflow_id value into the Path Variables Value cell.

4. Click Send to make the call. Skyflow returns tokens for the record you specified.

curl

Schema

In a terminal, run the following command. Replace the $VAULT_URL, $VAULT_ID, $SKYFLOW_ID, and $BEARER_TOKEN parameters with your vault-specific values.

curl -i -X GET "$VAULT_URL/v1/vaults/$VAULT_ID/credit_cards/$SKYFLOW_ID?tokenization=true" \
-H "Authorization: Bearer $BEARER_TOKEN"

To return tokens instead of record values, make sure to set tokenization to true.

Skyflow returns tokens for the record you specified.

"fields": {
        "card_number": "$CARD_NUMBER TOKEN",
        "cardholder_name": "$CARDHOLDER_NAME TOKEN",
        "expiry_month": "$EXPIRY_MONTH TOKEN",
        "expiry_year": "$EXPIRY_YEAR TOKEN"
    }

Sample

In a terminal, run the following command:

curl -i -X GET 'ebfc9bee4242.vault.skyflowapis.com/v1/vaults/x3d7caf822364170856c6f9d9a3bdf3f/credit_cards/$85e21b54-481f-42c6-8054-f3f770a77386?tokenization=true' \
-H 'Authorization: Bearer eyJhbGciOiJSU'

Skyflow returns tokens for the record you specified.

"fields": {
        "card_number": "8263-6743-4832-2862",
        "cardholder_name": "5d25061a-082e-495d-8df5-942f85d96721",
        "expiry_month": "0792b4ab-4974-4b86-9c64-59c601c13d21",
        "expiry_year": "47deed73-c29a-404f-be87-855f7be791ea"
    }

The preceding example uses these variables:

Variable	Type	Description
VAULT_URL	path (string)	Your workspace URL. Found on your Vault Details page.
VAULT_ID	path (string)	The vault ID. Found on your Vault Details page.
SKYFLOW_ID	path (string)	The Skyflow ID of the record you want to retrieve.
BEARER_TOKEN	header (string)	A Bearer token.

Next steps

You’ve tokenized your data, retrieved your token data, and detokenized your data.

For more detailed information about reading and writing sensitive data to your vault, see the Data API docs.

Read more about custom vault configurations, policy-based access control, or connecting your data to other first-party and third-party services at the pages below:

• Create a custom vault
• Data governance overview
• Connections overview