Skip to main content

Alation: Register Metadata using API

What is Alation

When your datasets, reports, and other assets are spread across environments (AWS, Azure, GitHub, Power Bi etc), it can take days to hunt down the right information (to find out where is what data). Metadata management is the cornerstone of modern data strategy. Like a skilled librarian, it ensures the right data is cataloged, easy to locate, properly maintained, and aligned with organizational needs. One of the Metadata management solutions is Alation.

Video explanations:

Connectors

Alation provides connectors for over 120 sources:all connectors.

Alation can extract most of the metadata automatically but not all.
Let’s take the example of Denodo & Alation case. Alation has a connector for Denodo.

Supported Authentications methods:

  • Authentication with username and password
  • SSL Authentication

Alation automatically extracts following metadata :

  • List of schemas
  • List of tables
  • List of views
  • List of columns
  • popularity of the data
  • Primary key information for extracted tables
  • Retrieval of data samples from extracted table
  • Retrieval of data samples from extracted columns

But, connector cannot extract:

  • Column comments
  • Column data types
  • Source comments

To compliment the metadata extracted by the connector (to add the metadata that Alation does not automatically extract), API can/should be used.

Create Custom Fields

Before explaining how to use API to add metadata fields to the data sources, I need to explain custom fields.

On the Alation UI, the data sources such as tables, schemas etc. are presented with the metadata fields like title and description. Often times default metadata fields such as title, description etc. is enough, but if you need to add new, custom fields, it is possible.

Alation has built-in fields such as title and description. In addition tot hat, Alation also allows creation of the custom fields. Custom fields are reusable objects.
Custom fields are attached to object templates. A (object) template specifies how metadata is presented for a type of object (table, schema, column etc).
Each object type has its own template.

Almost all the objects of the same type share a common template, in other words, a template applies to all the objects of the same type. For example, all the schemas use the same schema template.

To add a custom field (to a schema).

  1. You must have the Catalog Admin or Server Admin role to customize the Alation catalog.
  2. Create a custom field
  3. Edit the (schema) template by adding the newly created custom field.

graph
(it is AI generated image, there are some inaccuracies in the text)

Note: Alation assigns an ID to every custom field. The title of the custom field might be “Data Sensitivity” and its field_id 1002.

APIs

Upload Logical Metadata (ULM) API

There is an old api, Upload Logical Metadata (ULM) API. This api is deprecated. DO NOT USE IT. (Alation doesn’t recommend using the ULM API.)
Upload Logical Metadata (ULM) API Deprecation

Relational Integration API

To update titles, descriptions and custom fields, use Relational Integration APIs.

Permissions needed: Data Source Viewer or Data Source Admin

API Payload limitation per api call:

Object Type Max Objects in POST Payload
Schema 1,000
Table 1,000
Column 10,000

Update schema example

import requests

url = "https://alation_domain/integration/v2/schema/"

payload = [
    {
        "key": "95.employees", # schema name is employees
        "title": "Schema for employees data.",
        "description": "Stores all the tables related to employees from marketing department.",
        "db_comment": "This schema is part of company database storing important PII data.",
        "custom_fields": [
             {
            "field_id": 101,
            "value": "Finance Team"       // Business Owner custom field
            "op": "replace"
            },
            {
            "field_id": 102,
            "value": "Highly Confidential" // Data Sensitivity custom field
            "op": "replace"
            }
        ]
    }
]
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "TOKEN": "my_secret_token"
}

response = requests.patch(url, json=payload, headers=headers)

print(response.text)
Term Explanation
key Name of a schema/table etc
title title of a schema/table (It exits only in Alation’s catalog, not in the database.)
db_comment Comments on the schema from the data source.
op Operation need to perform for custom field. It can be add, remove or replace. add and remove operation are only apllicable for MULTI_PICKER and OBJECT_SET. replace is applicable for all custom fields.

custom_fields should be array of objects.
Alation assigns an ID to every custom field. The title of the custom field might be “Data Sensitivity” and its field_id 1002.
But, Relational Integration API does NOT allow creation of new custom fields. It can also update them.

These custom fields first should be associated with the schema otype template. You can create and add a new custom field to the schema otype template from Customize Catalog option in the UI.

Custom Field Value API

Custom Field Values Async API can be used to update custom fields.

Note: The Custom Field Value APIs will not allow you to update titles and descriptions on schemas, tables, and columns. Use Relational Integration API instead.

Limit: Maximum of 10,000 objects per request.

import requests

url = "https://alation_domain/integration/v2/custom_field_value/async/"

payload = [
    {
        "field_id": 10006, # custom field id
        "oid": 5,  # id of the object this custom field belongs to
        "otype": "table", # the type of the object this field belongs to
        "ts_updated": "2025-09-10T07:07:33.884Z",
        "value": "my custom field value"
    },
    {
        "field_id": 10007,
        "oid": "7", # id of the object this custom field belongs to
        "otype": "attribute", # the type of the object this field belongs to
        "ts_updated": "2025-09-10T07:07:33.884Z",
        "value": ["high, medium, low"]
    }
]
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "TOKEN": "my_secret_token"
}

response = requests.put(url, json=payload, headers=headers)

print(response.text)
Labels:

Comments

Post a Comment

Popular posts from this blog

Introduction to SQLFluff: How to make your SQL code clean and error-free

Image by Jake Aldridge from Pixabay You know oftentimes, the cause of runtime or compile errors and hours of debugging agony is all due to simply a missing semicolon. Have you ever had such experience? If you had, you are not alone. There are two ways to avoid these unfortunate situations: either become a perfect developer who never makes mistakes, or use helpful tools such as linters that can catch these errors early on. I am nowhere near being a perfect developer who never makes a mistake. In fact, I'm probably the opposite of a perfect developer, so even if I wanted to, I wouldn’t be able to teach you how to become a perfect developer. But what I can teach you is using linters. A Wikipedia defines a linter as a "static code analysis tool used to flag programming errors, bugs, stylistic errors and suspicious constructs." If you're not convinced yet on using linters, consider this scenario: in a large project with multiple members, different people tend to ...
Post a Comment
Read more

WinMerge のセットアップと使う方

WinMerge は、Windows 用のオープン ソースの差分およびマージ ツールです。WinMerge は、フォルダーとファイルの両方を比較し、違いを理解して扱いやすい視覚的なテキスト形式で表示します。この記事でWinMerge のセットアップと使う方を教えます。 source: https://winmerge.org WinMerge をダウンロード WinMerge のウェブサイト に行って、「WinMerge-2.16.44-x64-Setup.exe」ボタンを押し、WinMerge 2.16 をダウンロードしてください。 WinMerge をインストール ダウンロードされたソフトウェアをクリックし、ポップアップ画面で「Next」を押してください 「Languages」部分をスクロールダウンし、「Japanese menus and dialogs」を選択し、「Next」ボタンを押してください ターミナル等からも WinMerge をアクセス出来ようにする為に「Add WinMerge folder to your system path」オプションを選択し、希望によって他のオプション選択してください 「Enable Explorer context menu Integration」オプションを選択したら、フォルダ／ファイルを右キリックし、コンテクストメニューから WinMerge を開くようになります。 「Install」ボタンを押し、「Next」ボタンを押し、その後、「Finish」ボタンを押してください 言語を日本語にする もし WinMerge の言語が日本語じゃなくて、英語なら、「Edit」タブから「Options」を押してください。 ポップアップ画面で右側の下にある「Languages」と言うドロップダウンメニューから日本語を選択し、「OK」ボタンを押してください WinMerge を使う方 「ファイル」タッブから「開く」を押し 参照ボタンを押し、比較したいフォルダ・ファイルを指定 比較したいフォルダを指定する方法: ポップアップ画面から対象のフォルダーを選択し、「Open」を押してくだい 何も選択しないで、「Open」を押してくだい 右側下にある「比較」ボタンを押し ...
Post a Comment
Read more

Git squash merge explained

There are many ways to integrate changes in git: regular / normal git merge, git squash merge, git rebase etc. This article explains git squash merge by comparing it to regular merge. Let’s use below example: In the repository with default main branch, after two commits, a new feature branch is created. Some work happened in feature branch. feature branch now has 2 commits that it shares with main branch, and three exclusive commits (exists only in feature branch). In the meantime, others worked on main branch and added two new commits (exists only in main branch). git log output of the main branch: c72d4a9 ( HEAD - > main ) fourth commit on main 2c3dd61 third commit on main 0c2eec3 second commit on main 9b968e8 first commit on main git log output of the feature branch: 786650f ( HEAD - > feature ) third commit on feature 21cbaf1 second commit on feature 677bc7f first commit on feature 0c2eec3 second commit on main 9b968e8 first commit on mai...
Post a Comment
Read more