# Tokenize data

This guide explains how to tokenize and detokenize your data with a Skyflow vault.

## Concepts

When you **tokenize** data, you insert a value into a table and get a token that represents the value in return.

When you **detokenize** data, you submit a token and receive the value in return.

Each column in your vault can have a [**token format**](/docs/tokenization/) that determines how the column generates tokens for its values:

- **UUID deterministic token**: This format generates random UUIDs. If a value appears in the column more than once, each instance of the value generates the same token.
- **Format preserving deterministic token**: This format generates random tokens but preserves the format based on the specified regular expression (regex) expression. If a value appears in the column more than once, each instance of the value generates the same token.
- **Format preserving token**: This format generates random tokens but preserves the format based on the specified regex expression. Tokens have no connection to the column value.
- **UUID token**: This format generates random UUIDs. Tokens have no connection to the column value.
- **Transient UUID token**: This format generates random UUIDs. Tokens have no connection to the column value. This format purges data after a predefined time to live (TTL) expires. If the token expires, detokenization fails. When you update a transient field with a new value, the TTL resets.
- **Preserve email domain deterministic token**: This format preserves the domain and top level domain (TLD) of an email address. For example, "[jane@example.com](mailto\:jane@example.com)" might become "[yrnakewnmfpanwnfszaejd@email.com](mailto\:yrnakewnmfpanwnfszaejd@email.com)". If a value appears in the column more than once, each instance of the value generates the same token. Only works with columns formatted as the Email data type.

Each column can belong to a [**column group**](https://docs.skyflow.com/docs/tokenization/column-groups), which lets columns in the group generate the same deterministic tokens for the same values, regardless which tables the columns belong to. This lets you track common values across tables without exposing yourself to sensitive data.

## Prerequisites

#### Studio

- [Sign in](https://docs.skyflow.com/docs/resources/sign-in) to your Skyflow account. If you don't have an account,
  [sign up for a free trial](https://www.skyflow.com/try-skyflow).

- [A vault](https://docs.skyflow.com/docs/vaults/create-a-vault)

#### API

- [Sign in](https://docs.skyflow.com/docs/resources/sign-in) to your Skyflow account. If you don't have an account,
  [sign up for a free trial](https://www.skyflow.com/try-skyflow).

- A device with the following tools available:
  - A terminal that can run `bash` commands
  - `curl`
  - [`jq`](https://stedolan.github.io/jq/) 1.6 or greater

- A bearer token to authenticate API calls. For a short-lived token, use the following process. To generate tokens from service accounts, see [Authenticate](/docs/fundamentals/api-authentication/).

1. In Studio, click your account icon and choose **Generate API Bearer Token**.

2. Click **Generate Token**.

- Skyflow account, vault, and workspace details:
1. In Studio, click **vault menu icon > View vault details**.
2. Note your **Account ID**, **Vault ID**, and **Vault URL** values.

- Your environment's Management API URL:
  - Trial or Production: `https://manage.skyflowapis.com`
  - Staging: `https://manage.skyflowapis-preview.com`

- Set environment variables for your account and vault details:

  ```bash
  export ACCOUNT_ID=$ACCOUNT_ID
  export VAULT_ID=$VAULT_ID
  export VAULT_URL=$VAULT_URL
  export MANAGEMENT_URL=$MANAGEMENT_URL
  export TOKEN=$TOKEN
  ```

The following content uses the [Quickstart vault template](https://docs.skyflow.com/docs/vaults/create-a-vault). Adapt your commands accordingly.

## Configure tokenization in your vault

The **credit\_cards** table in the Quickstart vault has four columns configured for tokenization:

- **cardholder\_name**, **expiry\_month**, and **expiry\_year** generate UUID deterministic tokens.
- **card\_number** generates format preserving deterministic tokens.

The **persons** table has with five columns configured for tokenization:

- **name**, **date\_of\_birth**, and **state** generate UUID deterministic tokens.
- **email\_address** and **ssn** generate format preserving deterministic tokens.

#### Studio

You can only set column groups via the [Management API](https://docs.skyflow.com/api/management).

To set the token format for a column,

1. Navigate into your vault.
2. Click **Edit schema**.
3. Click the arrow in header of the column you want to tokenize, then click **Edit column**.
4. Click **Continue**.
5. In the **Tokens** section, select the applicable tokenization option for the column.
6. Click **Continue** through the remaining sections, then click **Save column**.
7. Repeat these steps for each column you want to tokenize. After you update all the columns you want to tokenize data for, click **Publish**.

#### API

You can set the token format for columns or add a column to a column group with the Management API.

To update a column's token format, you need to update the column's `skyflow.options.default_token_policy` option. To add a column to a column group, all columns in the group need the same values for a variety of options. See [Column groups](https://docs.skyflow.com/docs/tokenization/column-groups).

1. Export `$ACCOUNT_ID`, `$VAULT_ID`, `$VAULT_URL`, and `$TOKEN` values as described in [Prerequisites](#prerequisites).

2. Get your current vault information. The following command stores the schemas from the `curl` response's `.vault.schemas` field to a `vaultSchemas.json` file.

   ```bash
   curl -s -X GET "$MANAGEMENT_URL/v1/vaults/$VAULT_ID" \
     -H "X-SKYFLOW-ACCOUNT-ID: $ACCOUNT_ID" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     | jq -r ".vault.schemas" > vaultSchemas.json
   ```

3. Open the `vaultSchemas.json` file and locate each column you want to update.

   For example, the `credit_cards.name` column in a Quickstart vault might look like this:

   ```json
   {
     "name": "name",
     "datatype": "DT_STRING",
     "isArray": false,
     "tags": [
         {
           "name": "skyflow.options.data_type",
           "values": [
             "skyflow.Name"
           ]
         },
         {
           "name": "skyflow.options.default_dlp_policy",
           "values": [
             "REDACT"
           ]
         },
         {
           "name": "skyflow.validation.regular_exp",
           "values": [
             "^$|^[A-za-z ,.'-;]+$"
           ]
         },
         {
           "name": "skyflow.options.default_token_policy",
           "values": [
             "DETERMINISTIC_UUID"
           ]
         }
       ],
       "properties": null,
       "index": 0
     },
   ```

4. For each column, add or update the following objects in the `.tags` array to set the token format and other options for column groups. For each option's accepted values, see [Vault settings](https://docs.skyflow.com/docs/vaults/vault-settings).

   This example sets the token type to deterministic UUID tokens:

   ```json
   {
     "name": "skyflow.options.default_token_policy",
     "values": ["DETERMINISTIC_UUID"]
   }
   ```

   This example creates the `name` column group and sets the column to generate deterministic UUID tokens:

   ```json
   { "name": "skyflow.options.column_group", "values": ["name"] },
   { "name": "skyflow.options.data_type", "values": ["skyflow.Name"] },
   { "name": "skyflow.options.default_token_policy", "values": ["DETERMINISTIC_UUID"]},
   { "name": "skyflow.options.default_dlp_policy", "values": ["REDACT"]},
   { "name": "skyflow.options.format_preserving_regex", "values": []},
   { "name": "skyflow.options.find_pattern", "values": []},
   { "name": "skyflow.options.replace_pattern", "values": []},
   { "name": "skyflow.options.regular_exp", "values": ["^$|^[A-za-z ,.'-;]+$"]}
   ```

   Updating the `credit_cards.name` column for the `name` column group looks like this:

   ```json
   {
     "name": "name",
     "datatype": "DT_STRING",
     "isArray": false,
     "tags": [
       {
         "name": "skyflow.options.data_type",
         "values": ["skyflow.Name"]
       },
       {
         "name": "skyflow.options.column_group",
         "values": ["name"]
       },
       {
         "name": "skyflow.options.default_token_policy",
         "values": ["DETERMINISTIC_UUID"]
       },
       {
         "name": "skyflow.validation.regular_exp",
         "values": ["^$|^[A-za-z ,.'-;]+$"]
       },
       {
         "name": "skyflow.options.default_dlp_policy",
         "values": ["REDACT"]
       },
       {
         "name": "skyflow.options.format_preserving_regex",
         "values": []
       },
       {
         "name": "skyflow.options.find_pattern",
         "values": []
       },
       {
         "name": "skyflow.options.replace_pattern",
         "values": []
       }
     ],
     "properties": null,
     "index": 0
   }
   ```

5. Save and close `vaultSchemas.json`.

6. Update your vault's schemas, filling the object format expected by [Update Vault](https://docs.skyflow.com/api/management/vaults/update-vault) with the content of `vaultSchemas.json`:

   ```bash
   curl -s -X PATCH "$MANAGEMENT_URL/v1/vaults/$VAULT_ID" \
     -H "X-SKYFLOW-ACCOUNT-ID: $ACCOUNT_ID" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d "$(jq -r "{ vaultSchema: { schemas: . } }" vaultSchemas.json)"
   ```

## Tokenize your data

After you configure your vault, it automatically generates tokens based on your tokenization settings. Use the following API calls to insert new records and retrieve existing tokens.

### Insert and tokenize your data

Insert a new record and return the `skyflow_id` and tokens for your data.

#### Postman

1. Expand the QuickstartVault directory to the **credit cards CRUD APIs** folder. Choose the **Post insert credit\_cards** method.

2. In the **Body**, enter values for the parameters. Set tokenization to "true."

3. Click **Send** to make the call. Skyflow returns a `skyflow_id` and tokens for the record you just inserted.

![tokenize-postman-post-insert-body](/media/tokenize-postman-insert-body-fd1d5c08.gif)

The `card_number` maintains the standard 16-digit format because of format-preserving deterministic tokenization.

Leave this tab open for quick access to the `skyflow_id`. In the next section,
you'll use the ID to retrieve your tokenized data.

#### curl

#### Schema

In a terminal, run the following command. Replace the $VAULT\_URL, $VAULT\_ID, and $BEARER\_TOKEN parameters with your vault-specific values.

```bash
   curl -i -X POST "$VAULT_URL/v1/vaults/$VAULT_ID/credit_cards" \
   -H "Authorization: Bearer $BEARER_TOKEN" \
   -d '{
       "tokenization": true,
       "records": [
           {
               "fields": {
                   "card_number": "'"$CARD_NUMBER"'"
                   "cardholder_name": "'"$CARDHOLDER_NAME"'"
                   "expiry_month": "'"$EXPIRY_MONTH"'"
                   "expiry_year": "'"$EXPIRY_YEAR"'"
               }
           }
       ]
   }'
```

Skyflow returns a `skyflow_id` and tokens for the record you just inserted.

```json
{
  "records": [
    {
      "skyflow_id": "$SKYFLOW_ID",
      "tokens": {
        "card_number": "$CARD_NUMBER TOKEN",
        "cardholder_name": "$CARDHOLDER_NAME TOKEN",
        "expiry_month": "$EXPIRY_MONTH TOKEN",
        "expiry_year": "$EXPIRY_YEAR TOKEN"
      }
    }
  ]
}
```

#### Sample

In a terminal, run the following command:

```bash
   curl -i -X POST 'ebfc9bee4242.vault.skyflowapis.com/v1/vaults/x3d7caf822364170856c6f9d9a3bdf3f/credit_cards' \
   -H 'Authorization: Bearer eyJhbGciOiJSU' \
   -d '{
       "tokenization": true,
       "records": [
           {
               "fields": {
                "card_number": "8887777796",
                "cardholder_name": "jane doe",
                "expiry_month": "11",
                "expiry_year": "2025"
               }
           }
       ]
   }'
```

Skyflow returns a `skyflow_id` and tokens for the record you just inserted.

```json
{
  "records": [
    {
      "skyflow_id": "85e21b54-481f-42c6-8054-f3f770a77386",
      "tokens": {
        "card_number": "8263-6743-4832-2862",
        "cardholder_name": "5d25061a-082e-495d-8df5-942f85d96721",
        "expiry_month": "0792b4ab-4974-4b86-9c64-59c601c13d21",
        "expiry_year": "47deed73-c29a-404f-be87-855f7be791ea"
      }
    }
  ]
}
```

The `card_number` maintains the standard 16-digit format because of format-preserving deterministic tokenization.

The preceding example uses these variables:

| Variable       | Type            | Description                                           |
| -------------- | --------------- | ----------------------------------------------------- |
| `VAULT_URL`    | path (string)   | Your workspace URL. Found on your Vault Details page. |
| `VAULT_ID`     | path (string)   | The vault ID. Found on your Vault Details page.       |
| `BEARER_TOKEN` | header (string) | A Bearer token.                                       |

Store the skyflow\_id for quick access. In the next section, you'll use the ID
to retrieve your tokenized data.

### Get tokens for your stored data

Retrieve tokens for data already in the vault with the Get Record API.

#### Postman

1. Expand the QuickstartVault directory to the **credit cards CRUD APIs** folder. Choose the **GET get credit\_cards** method.

2. From the **Params** section, select the checkboxes for the redaction and tokenization rows to enable them. Change the tokenization value to "true."

3. Copy and paste the `skyflow_id` value into the Path Variables **Value** cell.

4. Click **Send** to make the call. Skyflow returns tokens for the record you specified.

![tokenize-postman-get-tokens](/media/tokenize-postman-get-tokens-bd6d1ffe.gif)

#### curl

#### Schema

In a terminal, run the following command. Replace the $VAULT\_URL, $VAULT\_ID, $SKYFLOW\_ID, and $BEARER\_TOKEN parameters with your vault-specific values.

```bash
curl -i -X GET "$VAULT_URL/v1/vaults/$VAULT_ID/credit_cards/$SKYFLOW_ID?tokenization=true" \
-H "Authorization: Bearer $BEARER_TOKEN"
```

To return tokens instead of record values, make sure to set `tokenization` to
true.

Skyflow returns tokens for the record you specified.

```json
"fields": {
        "card_number": "$CARD_NUMBER TOKEN",
        "cardholder_name": "$CARDHOLDER_NAME TOKEN",
        "expiry_month": "$EXPIRY_MONTH TOKEN",
        "expiry_year": "$EXPIRY_YEAR TOKEN"
    }
```

#### Sample

In a terminal, run the following command:

```bash
curl -i -X GET 'ebfc9bee4242.vault.skyflowapis.com/v1/vaults/x3d7caf822364170856c6f9d9a3bdf3f/credit_cards/$85e21b54-481f-42c6-8054-f3f770a77386?tokenization=true' \
-H 'Authorization: Bearer eyJhbGciOiJSU'
```

Skyflow returns tokens for the record you specified.

```json
"fields": {
        "card_number": "8263-6743-4832-2862",
        "cardholder_name": "5d25061a-082e-495d-8df5-942f85d96721",
        "expiry_month": "0792b4ab-4974-4b86-9c64-59c601c13d21",
        "expiry_year": "47deed73-c29a-404f-be87-855f7be791ea"
    }
```

The preceding example uses these variables:

| Variable       | Type            | Description                                           |
| -------------- | --------------- | ----------------------------------------------------- |
| `VAULT_URL`    | path (string)   | Your workspace URL. Found on your Vault Details page. |
| `VAULT_ID`     | path (string)   | The vault ID. Found on your Vault Details page.       |
| `SKYFLOW_ID`   | path (string)   | The Skyflow ID of the record you want to retrieve.    |
| `BEARER_TOKEN` | header (string) | A Bearer token.                                       |

## Next steps

You've tokenized your data, retrieved your token data, and detokenized your data.

For more detailed information about reading and writing sensitive data to your vault, see the [Data API](/api/data/) docs.

Read more about custom vault configurations, policy-based access control, or connecting your data to other first-party and third-party services at the pages below:

- [Create a custom vault](https://docs.skyflow.com/docs/vaults/create-a-vault)
- [Data governance overview](/docs/governance/)
- [Connections overview](/docs/connections/)