BeryX — Connector guide

How to obtain credentials and configure each data source in the connection wizard. Technical contract in docs/ARQUITETURA.md §3.1; implementation in backend/app/infra/connectors/.

Principles common to all sources:

• Read-only: BeryX only reads metadata, aggregates and samples — it never writes to the source. It always uses the vendors' official methods.
• Encrypted credentials: the configuration is encrypted with Fernet before it is persisted and never appears in logs or in API responses.
• Flow in the wizard: create connection → Test connection (diagnostic message if it fails) → Discover datasets → schedule/run scans.
• Fixed egress IP: with Cloud NAT active (DEPLOY_GCP §12), all connections to the sources leave from a fixed IP per region. The wizard shows it ("Our requests come from …") and it is also at GET /api/v1/public/egress-ips. In production it is not yet active: while that endpoint returns configured: false, the wizard hides the note and egress has no fixed IP. Ports per connector, TLS and private connectivity: REDE_ALLOWLIST.md.
• Good neighbour to the DBA: all queries from the SQL connectors go out identified (application_name/program_name/QUERY_TAG and the comment /* DataVault scan <uuid> */), and each connection can have blackout windows, a maximum number of concurrent queries, a maximum sampling and a read replica (replica_host/replica_port in Postgres, MySQL and SQL Server). The report of what was executed can be exported as CSV/JSON. Full guide: DBA.md.
• Optional drivers: each source has its own pip extra. Without the extra installed, the app still works: GET /api/v1/connections/types returns available=false and the type's missing_extra, and the wizard shows "driver not included in this installation" instead of letting you create a connection that would fail the test. The type → extra → modules map lives in backend/app/infra/connectors/extras.py (single source). The same endpoint returns the name (label), the family (group) and the anchor of this guide's section (docs_anchor) for each type — that is where the Help page gets the list of supported connectors; a new heading here that changes an anchor makes tests/test_connection_types.py fail.

Summary

Validation status

Live proof, per connector, that the sequence the product uses works: test_connection → list_tables (or list_documents in the document sources) → get_columns → profile_table (1 table, small sample) → fetch_rows(limit=5) and, when the connector implements it, table_metadata. It runs with backend/scripts/connector_matrix.py (logic in app/infra/connectors/conformity.py) and each row is classified by the strongest tier that passed:

A step that fails because of the network or a timeout is marked 🌐 unreachable (not ❌): an unstable public source does not count as a regression. The CI gate (--fail-on-regression) compares with the baseline docs/verificacao/connector-matrix.json and fails when a real_* row that used to pass comes back with fail — and, on a real source that has never passed in full, when a step that was ✅ comes back ❌. Error messages go through redact(): no DATAVAULT_MATRIX_* value, password, token or private key reaches the JSON, the Markdown or the logs (tested in tests/test_connector_matrix.py).

cd backend
python scripts/connector_matrix.py --tier public,fake --write-docs   # no Docker, no secrets
docker compose -f ../docker-compose.connectors.yml up -d --wait && python ../seed/connectors/seed.py
DATAVAULT_ALLOW_PRIVATE_NETWORKS=true python scripts/connector_matrix.py --tier docker --write-docs

--write-docs rewrites only the table between the markers below and adds the tiers that were run to the baseline (the other tiers stay as they were). A new connector has to appear here — the matrix test fails if a type in the CONNECTOR_REGISTRY has no row.

Honest notes (2026-09-25):

• Production image without Kafka, Pub/Sub and Fabric. The build's CONNECTOR_EXTRAS (cd.yml, gcp_deploy.sh, security.yml — identical, tested) does not include kafka, pubsub or fabric (pyodbc + msodbcsql18): in production these types appear in the wizard as "driver not included in this installation". Including them is a separate decision.
• Pub/Sub does not use the emulator. The connector talks only to the fixed Google endpoints (anti-SSRF) and does not accept PUBSUB_EMULATOR_HOST; relaxing the guard to make a test pass is out of the question. The row stays fake_transport + real_sandbox (when there are secrets). The docker job therefore validates 9 types in real_docker (postgres, mysql, mssql/SQL Server 2022, oracle/Oracle 23ai Free, mongodb/MongoDB 7, s3/MinIO, adls/Azurite, sftp/atmoz/sftp, kafka/Redpanda) — GCS has the same fixed-endpoint design as Pub/Sub and is left out for the same reason.
• SQL Server (mssql): real_docker only in the matrix job. There is no public SQL Server endpoint; the real proof is the SQL Server 2022 from docker-compose.connectors.yml (seed/connectors/mssql.sql, login with db_datareader only), run in the docker job (--expect-min real_docker:9 + tests/test_mssql_docker.py, which proves that the INSERT fails on the server). Proof from 2026-09-25: run 36093070996 (commit c2a8c4c = the code of 1296f65 + the workflow restricted to the mssql service, because the minio/minio image is no longer public on Docker Hub), with the 14 tests of tests/test_mssql_docker.py green in the url and fields modes, including the INSERT refused by the server for the db_datareader login. The Azure SQL/Synapse mode with Entra ID stays fake_transport (assembly of odbc_connect verified over a fake transport) until there are DATAVAULT_MATRIX_MSSQL_* secrets from an Azure sandbox.
• Oracle (oracle): real_docker only in the matrix job, no run yet. There is no public Oracle. The real proof is the Oracle 23ai Free (gvenzl/oracle-free:23-slim) from docker-compose.connectors.yml, with seed/connectors/oracle.sql and the datavault_ro user with only CREATE SESSION + SELECT. It runs in the docker job (--expect-min real_docker:9 + tests/test_oracle_docker.py): connection, discovery without system schemas, types, a profile with CLOB and BLOB, rows, the catalogue estimate and the INSERT refused with ORA-01456 on a table where the GRANT would let it through. The connector went in without that run, because it was built on a machine without Docker: the row stays driver_missing until the job's first green run, which is left for a separate card.
• Azure Blob/ADLS (adls): Azurite outside compose in the 2026-10-09 proof. The baseline's real_docker attempt ran against Azurite 3.35.0 installed with npm (the version of the compose image), because the card's machine had no Docker; the attempt's source says so. The docker job runs the same seed against the mcr.microsoft.com/azure-storage/azurite:3.35.0 image, plus tests/test_adls_docker.py. The real_public attempt reads the Azure Open Datasets holidays Parquet with anonymous access.
• SFTP (sftp): local OpenSSH outside compose in the 2026-10-09 proof. The baseline's real_docker attempt ran against macOS's OpenSSH 10.2 (internal-sftp -R, Ed25519 key), because the card's machine had no Docker; the attempt's source says so. The docker job runs tests/test_sftp_docker.py and the matrix against the compose sftp service (atmoz/sftp, chrooted user, host key generated on each run and pinned by its fingerprint). The real_public attempt connects to Rebex's public test server (test.rebex.net, user demo, read-only), which only holds a readme.txt: it proves the pinned key, the listing and reading one document, with no tables.
• SAP (sap): the real proof is only the sandbox with an API key. The real source for sap in the matrix is the SAP Business Accelerator Hub sandbox (S/4HANA Cloud's API_BUSINESS_PARTNER), with the DATAVAULT_MATRIX_SAP_APIKEY secret; without it, the row stays fake_transport. The sandbox uses the api_key mode, so the basic and oauth2_client_credentials modes have no proof against a real SAP in the matrix. odata runs in the public tier against the OData.org reference services (v2 and v4).
• MongoDB (mongodb): real_docker in the matrix job, optional real_sandbox. There is no public MongoDB for the matrix. The real proof is the mongo:7 from docker-compose.connectors.yml (seed/connectors/mongodb/, with mixed types, missing fields, a duplicated order_id, dates and nested documents, and a user with the read role only), run in the docker job with tests/test_mongodb_docker.py. That test also proves that the server refuses an insert from that user. With the DATAVAULT_MATRIX_MONGODB_URI and DATAVAULT_MATRIX_MONGODB_DATABASE secrets of a free Atlas M0 cluster, the sandbox tier also runs against Atlas. The baseline's real_docker attempt of 2026-10-09 ran against the official mongod 7.0.14 binary with the same seed, outside compose, because the card's machine had no Docker; the attempt's source says so.
• RNAcentral (public postgres): catalogue yes, data no. Since 2026-09-25 the public user reader has no USAGE on any schema (permission denied for schema rnacen) and information_schema.columns comes back empty. The connector now reads the columns through pg_attribute + format_type, which never touches pg_collation, and so test + discover + columns pass: that is what validate_prod.sh checks. profile and rows fail on EBI's side. The row stays unverified in the public tier and the full proof for postgres is the one from the docker job (real_docker). The gate watches the steps that pass today, and if columns goes back to ❌ the job fails. We evaluated FlyBase's public Chado as an alternative, but it sits behind a pooler that rejects the default_transaction_read_only startup parameter, and the read-only guard is not relaxed to make a test pass. Supporting poolers (PgBouncer/Supavisor) without losing the mechanical read-only is separate work.

The generated table below keeps its Portuguese labels: Conector = connector, Classificação = classification, Estado = status, Fonte = source, Data = date, Extra na imagem prod = extra in the production image, sim/não = yes/no; in the legend, ok = passed, falhou = failed, fonte inacessível (rede/timeout — não conta como regressão) = source unreachable (network/timeout, not counted as a regression), não se aplica = not applicable, não corrido = not run.

Legenda: ✅ ok · ❌ falhou · 🌐 fonte inacessível (rede/timeout — não conta como regressão) · — não se aplica · · não corrido.

BigQuery

What you need: a Google Cloud service account with read access to the project/dataset and its JSON key.

Step by step:

1. In the Google Cloud console, select the project → IAM & Admin → Service Accounts → Create service account (e.g. datavault-reader).
2. Grant the minimum roles:
   • BigQuery Job User (roles/bigquery.jobUser) at project level — needed to run profiling queries;
   • BigQuery Data Viewer (roles/bigquery.dataViewer) — can be granted only on the dataset(s) to analyse (Dataset → Sharing → Permissions) instead of the whole project, for least privilege.
3. On the service account: Keys → Add key → Create new key → JSON. Download the file.
4. In the wizard: fill in project (the project ID, e.g. praxis-analytics), optionally dataset, and paste the full contents of the JSON file into the credentials_json field (including the curly braces). credentials_path (the path of a file on the server) is accepted only when the file connectors are enabled (DATAVAULT_ENABLE_FILE_CONNECTORS, off in production).

Credentials always explicit (2026-09 sweep, A2): without credentials_json the connection is refused — BeryX never uses the server's own credentials (ADC/Cloud Run service account). The same applies to GCS (service_account_info with the JSON key, or anonymous: true for public buckets) and S3 (access_key_id + secret_access_key, or anonymous: true).

Guided wizard (GRAPH-ONBOARD): for BigQuery and GCS (with credentials), step 2 of the new connection shows the Google Cloud Registration guide (API enabled → service account → read roles → JSON key → fields) and the Test access button (POST /connections/check, config not saved). Here the test is a single step (connection, the connector's test_connection); the UI links the message to the likely step of the guide (API disabled, 403/permissions, JSON/invalid_grant, resource not found).

Installation: pip install "datavault-backend[bigquery]"

Known limitations:

• Profiling runs queries in BigQuery — there is a cost per bytes read, limited by the caps in the next section.
• Dataset names are case-sensitive; the dataset must be in the region the project expects.
• If credentials_json and credentials_path are both filled in, the pasted JSON takes precedence.
• The key must belong to a service account ("type": "service_account"); other credential types (e.g. external_account) are refused.
• The connection test validates authentication, the project and that the dataset exists (when given), with a distinct message for each case.
• The cost caps are config fields, set when the connection is created. To change them on an existing connection, recreate it or adjust the installation values (DATAVAULT_BIGQUERY_*, in CONFIGURACAO.md), which apply to every connection without its own value.

Cost of a scan, caps and partitions

BigQuery on-demand bills the bytes of the columns each query reads, and LIMIT does not reduce them. Without caps, a scan of a 2 TB table with 100 columns could bill tens of TB. Since card #293, BeryX limits the cost before reading and not only afterwards (the cost telemetry, COSTX, measures what has already been billed).

Cost model per table. The scan counts in passes. Each pass reads, in the worst case, the whole table, with the size from __TABLES__.size_bytes (logical bytes, the basis of the on-demand bill). It is an upper bound, because each query reads only its own columns.

COUNT(*) reads no columns and COUNT(DISTINCT) adds no bytes (it uses slots, which matters under capacity pricing). Correlations and co-nulls (deep), segments, the unique rule and the SQL rules are one query each. They are left out of the estimate, but are subject to the per-query cap.

Metadata. Per table and scan, BeryX reads __TABLES__ and three INFORMATION_SCHEMA views (COLUMNS, TABLE_OPTIONS, PARTITIONS). According to Google's documentation, each query to INFORMATION_SCHEMA bills at least 10 MB on-demand. The read is cached for the duration of the scan, and the automatic monitors reuse it.

Caps (fields of the connection config; without their own value, the installation's values apply):

Degradation in order. Above the per-scan cap, the profile degrades, in this order:

1. It skips the COUNT(DISTINCT). This saves no bytes, but it saves slots.
2. It replaces the per-column samples and the raw rows with one query each, SELECT … TABLESAMPLE SYSTEM (p PERCENT), with p calculated to read ~1 GB.
3. It skips the histograms.

Step 2 also applies when a full pass already exceeds the per-query cap, because a SELECT * would be refused. If it still does not fit, a date-partitioned table moves to the partition window. A table without time partitions is refused before any query to the data, with the caps message. What was decided is stored in metrics_json.cost_guard of the table profile: the caps, scanned_bytes, estimated_bytes, planned_bytes, degraded and sample_percent. The log records scan_cost_guarded.

Partitions. BeryX reads INFORMATION_SCHEMA.COLUMNS.is_partitioning_column, TABLE_OPTIONS.require_partition_filter and PARTITIONS (rows, logical bytes and last write per partition). It profiles only the partitions of the last partition_window_days days (today included) in two cases:

• when the table requires a partition filter (require_partition_filter = true, common practice in enterprise accounts; without this, the first COUNT(*) failed and the dataset ended up failed with no explanation);
• when the whole table exceeds one of the caps and is partitioned by date.

The partitioning can be by a DATE/TIMESTAMP/DATETIME column or by ingestion time, with the _PARTITIONTIME pseudocolumn. The filter is coluna >= <início da janela> (column >= start of the window), with a constant, so that BigQuery prunes the partitions. The window applies to the profile and to the subsequent passes of the scan: distributions, segments, the unique rule (which then states that it did not count the whole table) and the rows.

SQL rules (expression, custom_sql) run the rule's SQL as it is. On a table with a mandatory filter, the partition filter has to be in the rule itself. Without it, or above the per-query cap, the rule gets the error in Portuguese saying what to change in the rule or in the connection, and the scan continues. Outside a scan, a one-off read of rows from a table above the per-scan cap is not refused: it goes through TABLESAMPLE.

The windowed profile follows the contract of sampled profiles (#290): the row_count is that of the whole table (sum of PARTITIONS, or __TABLES__), and the column metrics are those of the window (sampled, sample_rows). That way, volume and drift remain comparable between scans. metrics_json.cost_guard.window stores column, days, since, reason (require_partition_filter, max_bytes_per_query or max_bytes_per_scan), the number of partitions and the rows in the window. A table partitioned by integer range with a mandatory filter has no time window, and the profile is refused with a message explaining why.

Automatic monitors. table_metadata now uses INFORMATION_SCHEMA.PARTITIONS: row_count is the sum of the partitions and last_modified the last write. If the view is not readable, it uses __TABLES__. According to Google's documentation, the PARTITIONS view is still pre-GA.

Actionable errors. "Query exceeded limit for bytes billed" becomes "A tabela 'x' excede o teto de 50 GB por query desta conexão (maximum_bytes_billed): o BigQuery recusou a query sem a faturar" ("Table 'x' exceeds this connection's cap of 50 GB per query (maximum_bytes_billed): BigQuery refused the query without billing it"), followed by what to do (raise max_bytes_per_query, reduce the window or partition the table by date). "Cannot query over table … partition elimination" becomes a message about metadata permissions. The dataset ends up failed, with the reason in Portuguese in the live scan view and in the scan summary (stats.dataset_errors). A cap refusal is never swallowed by the profile's fallbacks, because that would give a "completed" profile without metrics.

Permissions. COLUMNS, TABLE_OPTIONS and PARTITIONS require bigquery.tables.get and bigquery.tables.list, already included in BigQuery Data Viewer. Without them, the profile proceeds without a window or per-partition counts. On a table with a mandatory filter, the message says which permissions are missing.

Behaviour change (upgrade). A table that read more than 50 GB in a single query is now profiled by window (if it is date-partitioned) or fails with the cap message. A table whose estimate exceeds 200 GB degrades. To keep the previous behaviour, raise the caps in the installation or in the connection.

Databricks

What you need: an active SQL Warehouse, a Personal Access Token (PAT) and the warehouse's HTTP path.

Step by step:

1. PAT: in the workspace, top right corner → User Settings → Developer → Access tokens → Generate new token. Copy the token (it is only shown once). Preferably use a user/service principal with read-only access to the catalogs to be analysed.
2. HTTP path: SQL Warehouses → (your warehouse) → Connection details → HTTP path (format /sql/1.0/warehouses/<id>).
3. Host: on the same screen, the Server hostname (e.g. adb-1234567890123456.7.azuredatabricks.net or dbc-abc12345-de67.cloud.databricks.com). You can paste it with or without https:// — it is normalised.
4. In the wizard: host, access_token, http_path; optionally catalog and schema (Unity Catalog) to limit discovery.

Installation: pip install "datavault-backend[databricks]"

Known limitations:

• The warehouse must be running: a stopped warehouse with auto-stop can make the first test fail with a timeout — try again after 1-2 minutes.
• Unity Catalog permissions required: USE CATALOG, USE SCHEMA and SELECT on the tables.
• PATs expire according to the workspace policy — renew them and update the connection.
• The connection test tells these apart in its message: invalid token vs wrong host vs wrong http_path.

Running in your Databricks (optional, ADR 0005 step B)

To measure whole-row duplicates, checks and Cramér's V on the full table without bringing rows out of the workspace, BeryX can submit the public package beryx-profiler as a one-time run in your account. The job reads the tables and returns only aggregated metrics over HTTPS to the BeryX API.

Status: does not run in scans yet. The package, the Jobs API client and the ingestion exist in the code, but no scan submits the job yet; until then, in Databricks too, row rules and duplicates use the sample described in Security. The rest of this section describes how it works.

• Declared footprint: an ephemeral run (runs/submit, does not create a persistent Job) named beryx-profiler <id>, visible in the history. Nothing is uploaded to DBFS, Volumes or workspace files; the code comes from PyPI, and pip only installs the wheel with the version and SHA-256 we pinned. Without that pinned hash, the run is not submitted. The job never writes data.
• Dedicated credential (config remote_jobs, encrypted with Fernet, never returned by the API): token of a service principal — not the warehouse PAT.

  "remote_jobs": {
    "token": "<token do SP>",
    "compute": {"mode": "serverless"}
  }
  

  compute.mode: existing_cluster (cluster_id), policy (policy_id, spark_version, node_type_id, num_workers?) or serverless. host and catalog are optional (by default, those of the connection).
• Minimum SP permissions: USE CATALOG, USE SCHEMA and SELECT on the tables; on compute, CAN_ATTACH_TO on the cluster or CAN_USE on the cluster policy or serverless jobs enabled. No CAN_MANAGE, no workspace admin and no writing to catalogs.
• Network: outbound HTTPS from the workspace to the BeryX API (DATAVAULT_API_PUBLIC_BASE_URL).
• Run parameters: visible to anyone who can see the run in the workspace (run page and runs/get). They carry the spec, the ingestion URL and a single-use dvw_* claim code, never the token that delivers the metrics. The job generates that token and registers it with the code before reading any table. If someone else uses the code, the run fails and the metrics are discarded.
• Cost: DBUs in your account; each run has timeout_seconds (30 min by default) and is cancelled (runs/cancel) if it exceeds it.
• Jobs API calls (closed allowlist in the client): POST runs/submit, POST runs/cancel, GET runs/get, GET runs/get-output, GET scim/v2/Me. Any other call is blocked before it leaves BeryX.

Snowflake

What you need: a dedicated read-only user, with its own role and warehouse.

Step by step (run as ACCOUNTADMIN/SECURITYADMIN):

CREATE ROLE IF NOT EXISTS DATAVAULT_READER;
GRANT USAGE ON WAREHOUSE <wh> TO ROLE DATAVAULT_READER;
GRANT USAGE ON DATABASE <db> TO ROLE DATAVAULT_READER;
GRANT USAGE ON ALL SCHEMAS IN DATABASE <db> TO ROLE DATAVAULT_READER;
GRANT SELECT ON ALL TABLES IN DATABASE <db> TO ROLE DATAVAULT_READER;
GRANT SELECT ON FUTURE TABLES IN DATABASE <db> TO ROLE DATAVAULT_READER;
CREATE USER IF NOT EXISTS datavault_svc PASSWORD='<forte>' DEFAULT_ROLE=DATAVAULT_READER;
GRANT ROLE DATAVAULT_READER TO USER datavault_svc;

The account is the account identifier (e.g. xy12345.eu-west-1 or org-conta), visible in Admin → Accounts or in the Snowsight URL.

Wizard fields: account, user, password, database; optional schema, warehouse, role.

Installation: pip install "datavault-backend[snowflake]"

Known limitations: the warehouse consumes credits during profiling (use a dedicated XS with a short auto-suspend); without role, the user's default is used.

Dynamics 365 (Dataverse)

What you need: an app registration in Entra ID with a secret, and an application user in the Dataverse environment with a read profile.

Step by step:

1. App registration: entra.microsoft.com → Identity → Applications → App registrations → New registration (e.g. datavault-reader). Note the Application (client) ID (client_id) and the Directory (tenant) ID (tenant_id) from the Overview page.
2. Secret: in the app → Certificates & secrets → New client secret. Copy the value (client_secret) — it is only shown once.
3. API permission (for consent): API permissions → Add a permission → Dynamics CRM → Delegated → user_impersonation → Grant admin consent. Note: for the server-to-server flow used by BeryX, what actually grants access is the next step.
4. Application user in Dataverse (essential): admin.powerplatform.microsoft.com → Environments → (your environment) → Settings → Users + permissions → Application users → + New app user → choose the app registration → assign a security role with read on the entities to be analysed (e.g. a custom read-only role, or Basic User + read on the relevant tables).
5. Organisation URL (org_url): the environment's address, e.g. https://a-tua-org.crm4.dynamics.com (visible in the admin center or in the browser). It can be pasted with or without https://.

Wizard fields: org_url, tenant_id, client_id, client_secret; optional entities — a list (or a comma-separated string) of entity set names (plural, e.g. accounts,contacts,leads,opportunities,incidents). Default: accounts, contacts, leads, opportunities.

Installation: pip install "datavault-backend[dynamics]"

Known limitations:

• row_count uses OData's $count=true, which Dataverse caps at 5,000 — counts above that appear truncated.
• Profiling is done on a sample (the first ~1,000 rows via $top), not on the whole table; null and distinct percentages are relative to the sample.
• The Web API has throttling (request limits per user); scans of many entities may slow down.
• Each Web API response has a 30 MB cap, and the 30 s timeout counts for the whole request. Scan rows arrive in pages of 1,000 records. A response above the cap fails with «Resposta Dataverse demasiado grande».
• The connection test does token + WhoAmI and distinguishes: invalid secret vs wrong client_id/tenant_id vs missing application user in the environment (403).
• The connection test message goes to the connection's last error, which every workspace member sees. Since card #610 it goes through the same filter as the SQL connectors: the client_secret value and the password of an org_url with user:password@ come out as ***. Proof: backend/tests/test_dynamics_connection_redaction.py.

Confluence / Jira (Atlassian Cloud, document source)

Two document sources on the same Atlassian Cloud account (email + API token), the same credentials as the Jira ticket integration in Notifications. There are no tables and no tabular scans: content comes in through the Documents button (or POST /connections/{id}/documents).

Credentials — on the customer side:

1. A dedicated Atlassian account (e.g. datavault@empresa.pt) with read access only: View permission on the Confluence spaces to be analysed and Browse projects on the Jira projects to be read.
2. With that account, at https://id.atlassian.com/manage-profile/security/api-tokens → Create API token. Copy the token (ATATT…), which is only shown once. The connector authenticates with Basic auth (email + token) at the site address.
3. In the wizard: site (e.g. a-tua-org.atlassian.net, with or without https://), the account's email, the api_token and, optionally, the space (Confluence, space_key) or the project (Jira, project_key). Through the API, Jira also accepts jql.

Confluence — how it works: lists the current pages of the space (without space_key, all the pages the account can see), 50 per request, up to the analysis limit (200 by default, maximum 1,000). Each page is a document confluence/<id>/<título>.html with the body in storage format (HTML) and the date of the latest version. Cap of 5 MB per page. Signals: off by default, because internal documentation does not go to the LLM unless someone asks for it. mine_signals: true in the configuration turns them on.

Jira — how it works: searches via GET /rest/api/3/search/jql (the legacy /search responds 410). The effective JQL is project = <project_key> combined with the jql from the configuration, ordered by updated DESC (or by the ORDER BY of the jql). Without a project or JQL, it uses updated >= -30d. At most 500 issues per analysis, paginated by nextPageToken. Each issue is a document jira/<CHAVE>/<resumo>.txt with summary, status, labels, description (ADF → text) and the 30 most recent comments, these with the name of the commenter. Cap of 2 MB per issue (the text is cut). Signals: on by default, because that is what the source exists for. Each issue goes through the SIGNALS pipeline (pre-filter by DQ glossary → extraction → linking to the catalogue → idempotent upsert per issue) and the signal links to https://<site>/browse/<CHAVE>. mine_signals: false turns them off.

Connection test diagnostics: Confluence reads GET /wiki/rest/api/space?limit=1 and Jira GET /rest/api/3/myself (the message says which account it authenticated with). 401 = email or token rejected (in Jira also 403, which includes missing permissions), 404 = wrong site, space or JQL. A response that is not JSON (an SSO, proxy or maintenance page) has its own message. In Jira, so do 429 (rate limit) and 410 (discontinued endpoint).

Security: only GETs, through the ReadOnlyHttpClient (anti-SSRF + DNS pinning), to the given site, which also goes through the guard when the connection is created. The project key is validated (^[A-Z][A-Z0-9_]{0,9}$) before it enters the JQL. Page ids and issue keys are validated before they enter the URL.

Installation: none — both connectors only use httpx, which is a base dependency. They ship in the production image.

Known limitations:

• Atlassian Cloud only: the paths are the Cloud ones (/wiki/rest/api, /rest/api/3), and Server/Data Center are left out.
• In Confluence, pages only: blog posts, attachments and comments are not read.
• In Jira, 500 issues and 30 comments per issue in each analysis. A 429 from Jira is not retried: during the listing it fails the analysis; when reading an issue it skips that issue.
• Validation status: fake_transport only (see the matrix above). There is no proof yet against a real Atlassian instance.

SharePoint / Teams (Microsoft Graph, document source)

Registration decision: ADR 0003, option A — the app is registered by the customer in their own tenant, with the least privilege possible (Sites.Selected). The files of a Teams channel live in the team's SharePoint site: a team site counts as a site. Teams messages are out of scope.

What you need: someone with the Cloud Application Administrator role (or Privileged Role Administrator) for the consent and a SharePoint Administrator (or site owner) for the per-site grant.

Step by step:

1. Entra ID → App registrations → New registration (single tenant, no redirect URI). Note the Application (client) ID and the Directory (tenant) ID.
2. API permissions → Microsoft Graph → Application → Sites.Selected → Grant admin consent. Do not use Sites.Read.All or Files.Read.All.
3. Certificates & secrets → New client secret (validity ≤ 12 months). Certificate: card GRAPH-CERT.
4. Grant read to the app on each site you want: Graph POST /sites/{site-id}/permissions with roles: ["read"] (Graph Explorer) or Grant-PnPAzureADAppSitePermission -Permissions Read.
5. In the wizard: site URLs (one per line, e.g. https://a-tua-org.sharepoint.com/sites/Qualidade), tenant_id, client_id, client_secret and, optionally, a folder inside each library (folder). Through the API, also drive_names (list of libraries to include).

Installation: pip install "datavault-backend[sharepoint]" (msal, the same library as Dynamics).

How it works: document source — there are no tables and no tabular scans. The Documents button walks through the libraries of each site (subfolders up to 5 levels, @odata.nextLink pagination) and each PDF/DOCX/TXT/MD/HTML file goes into the document analysis (UNSTR). Cap of 20 MB per file, checked before the download.

Security: only GETs through the ReadOnlyHttpClient (anti-SSRF guard + DNS pinning); each @odata.nextLink must point to https://graph.microsoft.com/v1.0/ (lesson A1); the download follows the 302 to *.sharepoint.com without the Authorization header. The only outbound request that does not go through that client is the token POST that msal makes to login.microsoftonline.com.

Connection test diagnostics: invalid/expired secret, wrong client_id or tenant_id (AADSTS codes), 403 = missing read grant on the site (Sites.Selected), 404 = wrong site URL. Throttling (429/503) honours Retry-After (up to 2 attempts).

Guided wizard (GRAPH-ONBOARD): in step 2 of the new connection, the Registration guide shows the 5 steps above with a textual "screen" of the portal (menus in English), the exact fields (with a copy button) and the common errors at each step. The Test access button sends the config, still unsaved, to POST /connections/check and receives one result per step, in the guide's order:

A step that depends on another one that failed comes back skipped. Warnings do not block. The token and the secret never go into messages or logs (the onboarding_access_checked log carries only the workspace, the type, ok and the failed steps).

Google Drive (Drive API v3, document source)

Registration decision: ADR 0003, option A — the credential is the customer's, in their own GCP project, and BeryX always requests only the scope https://www.googleapis.com/auth/drive.readonly. The document base (limits, contract, helpers) is the same as for SharePoint: app/infra/connectors/documents.py.

Three credential modes (credentials_json), in the recommended order:

1. Service account + sharing (default): the JSON key of an SA with no IAM roles; the customer shares the folders/Shared Drives with the SA's email as Viewer. The reach is exactly what was shared.
2. Service account + domain-wide delegation (only if the Workspace sharing policy blocks iam.gserviceaccount.com): the same key + subject (email of the impersonated technical user). In the Admin Console → Security → API controls → Domain-wide delegation: the SA's numeric Unique ID with only drive.readonly. It is broader (it sees everything the subject sees): use a user with access only to the right folders.
3. The customer's OAuth app: JSON "type": "authorized_user" (client_id, client_secret of the OAuth app registered in the customer's project + refresh_token of a technical user). The refresh requests only drive.readonly; if that scope was not consented to, Google refuses. An app in Testing mode has its refresh token expire in 7 days — publish it as Internal.

Any other type (e.g. external_account) is refused at creation: never ADC, never server credentials.

Step by step (mode 1):

1. The customer's GCP project → APIs & Services → Library → enable the Google Drive API.
2. IAM & Admin → Service Accounts → create the SA with no roles.
3. Keys → Add key → JSON (or, later, Workload Identity Federation — card GRAPH-CERT).
4. In Drive: Share each folder (or add a Viewer member to the Shared Drive) with the SA's email.
5. In the wizard: the credential JSON, folders (one URL https://drive.google.com/drive/folders/<id> or id per line, up to 20), optional shared_drive_id, optional subfolder (folder, a path of names inside each root) and, only in mode 2, the impersonated user.

Installation: pip install "datavault-backend[gdrive]" (google-auth + requests, the official transport for the refresh).

How it works: document source — no tables and no tabular scans. The Documents button walks each root with files.list ('<pasta>' in parents and trashed = false, supportsAllDrives/includeItemsFromAllDrives, corpora=drive on the Shared Drive), subfolders up to 5 levels, pagination by pageToken and a cap of 500 requests. PDF/DOCX/TXT/MD/HTML files come in via files.get?alt=media; Google Docs come in exported as HTML and Google Slides as text (files.export); Sheets, Forms and shortcuts are left out. Paths gdrive/<file-id>/<nome>. Cap of 20 MB per file, checked before the download when Drive declares the size (and always on the body). The prefix requested in the analysis is a subfolder inside the configured roots — the reach never goes beyond what was shared.

Security: only GETs through the ReadOnlyHttpClient (anti-SSRF guard + DNS pinning) to https://www.googleapis.com/drive/v3; pagination is a parameter, never a returned URL. Folder/file ids are validated (^[A-Za-z0-9_-]+$) before they enter the URL or the q query; subfolder names are escaped in q. The token_uri of the pasted JSON is ignored and fixed to https://oauth2.googleapis.com/token (a tampered JSON would send the signed JWT or the refresh token to another host); a universe_domain ≠ googleapis.com is refused. The only outbound request outside the Drive API is that google-auth POST. Private key, refresh token and access token never go into messages or logs.

Guided wizard and diagnostics: the Google Drive Registration guide (API → SA → key → sharing → optional delegation → fields) and Test access return one result per step:

Throttling (429/503 and 403 userRateLimitExceeded/rateLimitExceeded) honours Retry-After (up to 2 attempts, wait ≤ 10 s).

Google Sheets (Sheets API v4, each tab is a dataset)

Each tab of a Google spreadsheet is a dataset named «spreadsheet title/tab» (card #213). The credential is Google Drive's, with the same three modes: the JSON key of a service account with the spreadsheet shared as Viewer, the same key with subject (domain-wide delegation) or the authorized_user JSON of a customer OAuth app. The same code pins the JSON's token_uri and validates the universe_domain (GoogleApiMixin, in app/infra/connectors/gdrive.py). Implementation: app/infra/connectors/gsheets.py.

Config: credentials_json, spreadsheet_ids (ids or URLs https://docs.google.com/spreadsheets/d/<id>/…, one per line, up to 20; ids ^[A-Za-z0-9_-]+$), header_row? (header row, 1 by default, up to 1000), sheets? (tab filter by name) and subject?. Creation returns 422 (netguard.GoogleSheetsConfigError) for a credential that is not service_account/authorized_user, a universe other than googleapis.com, a malformed id or URL and a header_row outside 1–1000.

Scopes: spreadsheets.readonly for the Sheets API. The modification date comes from Drive (files.get?fields=modifiedTime), which does not accept that scope: that request carries a separate token with drive.metadata.readonly. With domain-wide delegation, authorise both scopes in the Admin Console. Without the second one, or without the Drive API enabled, reading carries on and freshness is not measured.

Step by step (service account):

1. Customer's GCP project → APIs & Services → Library → enable the Google Sheets API and, for freshness, the Google Drive API.
2. IAM & Admin → Service Accounts → create the SA with no roles and the JSON key (as for Drive).
3. In the spreadsheet: Share with the SA's email, role Viewer.
4. In the wizard: credential JSON, spreadsheets (one URL or id per line), header row and, if you want, the tabs to read.

Installation: pip install "datavault-backend[gdrive]", the Drive extra (google-auth + requests). There is no new extra.

How it reads:

• spreadsheets.get (only sheets.properties) lists the tabs. Only grid tabs (sheetType GRID) become datasets: charts and BigQuery connections have no cells. If two spreadsheets share a title, the second one gets the start of its id in the name.
• values.get with valueRenderOption=UNFORMATTED_VALUE and dateTimeRenderOption=SERIAL_NUMBER, in blocks of 10,000 rows that never pass the grid's rowCount, up to 100,000 rows per tab (ROW_LIMIT; above that the profile says sampled). Reading stops at the first block that is entirely empty.
• Columns: the text of the header row. An empty cell gives the column its letter (C); a repeated name gets _2. Numbers and booleans arrive typed. A spreadsheets.get with the effectiveFormat.numberFormat.type mask over the first 200 rows tells which columns are dates (DATE/DATE_TIME in the majority): there, the serial number becomes a date in the spreadsheet's time zone, and TIME cells become HH:MM:SS. Coercion and profiling (nulls, distinct values, min/max, mean, samples) are the CSV connector's (coerce_values and profile_columns), in memory: 00123 stays text.
• row_count counts the non-empty rows below the header. Empty ones go to source_metrics.blank_rows and cells with a formula error (#N/A, #REF!…) to source_metrics.error_cells.
• An empty tab, one without a header or one with only the header is still a dataset: the profile carries the warning in source_metrics.warnings and nothing raises.
• Freshness: the first date column. Without one, the spreadsheet's Drive modifiedTime applies (per spreadsheet, not per tab), which is also table_metadata.last_modified.
• Each scan has 500 HTTP requests. Once the budget runs out, the next tab fails with that message.

Security: GETs only, through the ReadOnlyHttpClient (anti-SSRF + DNS pinning), to https://sheets.googleapis.com/v4/spreadsheets and https://www.googleapis.com/drive/v3/files. The hosts are fixed in code and nothing from the config goes into them. The tab name is quoted ('Sheet1'!1:10000) and encoded in the path. The module has no values.update, append or batchUpdate: tests/test_gsheets_connector.py proves it with a fake transport that refuses POST/PUT.

Test access: Google Drive's steps, with the Sheets API:

Throttling (429/503) honours Retry-After, as for Drive.

Slack and Teams — chat channels as a source of signals (CHATSIG, ADR 0008)

An extension of SIGNALS: conversation threads (message + replies) in channels chosen by the customer go into the same pipeline as Jira tickets (pre-filter by DQ glossary → Haiku or heuristic extraction → linking to the catalogue → idempotent upsert per source_ref with content_hash). Each thread is a document slack/<canal>-<ts>/… or teams/<chave>-<mensagem>/…; the corpus appears in the catalogue as documentos/canais and the signals on the Signals page. It runs with the Documents button (or POST /connections/{id}/documents).

Privacy — opt-in per channel, never DMs:

• Only the channels listed in channels are read; without a list, the connection is refused. A path forged for another channel is refused before any request.
• Slack: direct message ids (D…) are refused in the config; conversations.info confirms that each channel is neither a DM nor a group conversation (is_im/is_mpim) before reading messages. Bot tokens (xoxb-) only: a user token would give access to the DMs of whoever generated it.
• Teams: team channels only (19:…@thread.tacv2); chat ids (1:1, group, meeting) are refused and no /chats endpoint is called.
• Authors never go into the text sent to the AI (neither names nor ids); mentions of people become @utilizador. Messages from bots/apps, system messages and deleted messages are ignored — BeryX's own alerts in a channel do not turn into "complaints".
• Limited window: lookback_days (30 by default, max. 90); up to 20 channels per connection and 100 replies per thread. mine_signals: false turns off mining while keeping the connection.

Slack — on the customer side:

1. https://api.slack.com/apps → Create New App (from scratch) in the organisation's workspace.
2. OAuth & Permissions → Bot Token Scopes: channels:history, channels:read and, only if there are private channels to mine, groups:history, groups:read. Never im:* or mpim:* ("Test access" warns if they are present).
3. Install to Workspace and copy the Bot User OAuth Token (xoxb-…).
4. In each chosen channel: /invite @nome-da-app (a bot token only reads channels it is in).
5. In BeryX: the token, the channels (#dados-problemas, id C0… or link …/archives/C0…) and, optionally, the workspace address (a-tua-org.slack.com) for the signal links.

Teams — on the customer side (the same app registered for SharePoint will do; see ADR 0003/0008):

1. Minimum permission: RSC ChannelMessage.Read.Group (Application), granted by the owner of each team when installing the app in that team. Tenant-wide alternative: ChannelMessage.Read.All (Application) with admin consent. No Chat.* permission.
2. Channel messages are a Microsoft protected API: the customer submits the access request (Protected APIs in Microsoft Teams form) for the app in their tenant.
3. In BeryX: tenant_id, client_id, client_secret and the channels — one link per line ("Get link to channel" in Teams) or <team-id>/<channel-id>.

Read-only: Slack only through read methods of the official Web API (auth.test, conversations.info|list|history|replies) via GET; Teams through GETs to Graph v1.0. Both go through the ReadOnlyHttpClient (anti-SSRF + pinning); tokens and secrets never appear in logs or errors.

Apache Kafka (topics as datasets)

What you need: the bootstrap servers and a read-only account — ACL Read + Describe on the topics to monitor and, if you want an application's lag, Describe on its consumer group. On Confluent Cloud: a service account API key with those roles (SASL_SSL + PLAIN). For Avro: the Schema Registry URL and API key (read-only).

Installation: pip install "datavault-backend[kafka]" (confluent-kafka, the official client built on librdkafka). Avro is decoded with no extra dependencies; the Schema Registry is read over HTTP.

How it works (v1, no continuous consumption): each configured topic (or, without topics, every non-internal topic — up to 200) is a dataset. Each scan reads a sample of the last sample_size messages (default 500, maximum 5000), spread across the partitions from the tail (up to 64 partitions, 30 s budget), and profiles them as rows: the top-level fields of the JSON/Avro record are columns (nested objects and lists stay as JSON text), plus _kafka_partition, _kafka_offset and _kafka_timestamp. row_count = retained messages (sum of high − low watermark).

Its own metrics (in profile_results.metrics_json.source_metrics of the table row):

• last_message_age_seconds / latest_message_at — event freshness; the profile's latest_timestamp is the timestamp of the most recent message, so the freshness rule and the automatic monitors work with nothing special;
• malformed_pct / malformed_reasons — messages whose value cannot be read as an object (invalid_json, not_an_object, invalid_avro, schema_not_found, schema_registry_missing, missing_schema_id, unsupported_schema_type); tombstones (null value) are counted separately in tombstones;
• sample_lag — messages left unread at the head of the partitions when the sample finished (produced during the scan, or reading cut short by the budget; sample_timed_out);
• consumer_group_lag — with consumer_group, the real lag of that application (high watermark − committed offset), read with OffsetFetch;
• schema_conformity — with schema_registry_url, how many messages in the sample conform to the registered schema (checked, violations, violation_pct, reasons, schema_ids, subject); null without a registry. See "Rules on streams" below.

The automatic monitors use only watermarks (change_counter = sum of the high watermarks) and the timestamp of the last message in each partition.

Formats: value_format = auto (default: Confluent wire format → Schema Registry; otherwise JSON), json or avro. Registry JSON schemas are supported; PROTOBUF is outside v1 (it counts as malformed).

Rules on streams (card #103). Four rule types at topic level, evaluated on each scan's sample; when they fail they open issues (rule_violation, with the type in subtype) and count towards the dataset verdict (GET /datasets/{id}/verdict) exactly like the other rules — a failing critical rule gives fail.

Schema conformity: messages in the Confluent wire format are validated against the schema of the id they carry — Avro that decodes is conformant (invalid_avro and schema_not_found are violations); JSON schemas validate the usual subset of JSON Schema (type, required, properties, additionalProperties: false, enum/const, items, anyOf/oneOf/allOf, local $ref). JSON messages without framing are validated against the latest version of the subject <tópico>-value (TopicNameStrategy), whether it is JSON Schema or Avro (fields without a default are required; extra fields are tolerated). Reasons: missing_required, type_mismatch, enum_mismatch, unexpected_field, no_branch_matched, invalid_json, invalid_avro, schema_not_found. Without a registry, or without any message that has a schema, the rule fails with the explanation in details.error (just like freshness without a timestamp). Only counters and codes leave the connector — never payloads.

Onboarding: on a topic's first scan, besides the generic rules, max_malformed_pct (≤ 1%, warning) and — if the registry validated any message — schema_conformity (0%, warning) are created.

Examples (POST /api/v1/rules, or the rules section of the GitOps YAML):

{"dataset_id": "<orders>", "rule_type": "max_event_age", "params": {"max_age_seconds": 300}, "severity": "critical"}
{"dataset_id": "<orders>", "rule_type": "max_event_age", "params": {"max_age_seconds": 900, "column": "event_time"}}
{"dataset_id": "<orders>", "rule_type": "schema_conformity", "params": {"max_violation_pct": 0.5}}
{"dataset_id": "<orders>", "rule_type": "required_fields", "params": {"fields": ["order_id", "customer.id"], "max_violation_pct": 0}}
{"dataset_id": "<orders>", "rule_type": "max_malformed_pct", "params": {"max_pct": 1}}

rules:
  - dataset: orders
    type: max_event_age
    params: {max_age_seconds: 300}
    severity: critical
  - dataset: orders
    type: required_fields
    params: {fields: [order_id, customer.id]}

Reading the result: observed is the age in seconds (max_event_age) or the observed percentage (the others); details carries source (broker_timestamp, event_timestamp on a log file, rows or profile), reasons per reason, missing_by_field per field and checked_count. max_event_age and required_fields also work outside streams (they use latest_timestamp / the column profile); schema_conformity only on streams (Kafka and Pub/Sub — see below) and max_malformed_pct also on the log files in buckets.

Security / read-only: the consumer uses assign() — it never joins a consumer group, never triggers rebalances and never commits (enable.auto.commit=false). The librdkafka config is built from an allowlist (no arbitrary property from the customer gets through: ssl.ca.location, plugins or Kerberos commands would be LFI/RCE); a custom CA only as PEM (ssl_ca_pem). Bootstrap servers, the brokers advertised in the metadata and the Schema Registry go through the anti-SSRF guard; the registry is read only by GET on the ReadOnlyHttpClient and requires https. Payloads are never logged; the metrics only store counters.

Google Pub/Sub (topics as datasets — Kafka's counterpart)

What you need: a service account JSON key from your GCP project and, on each topic to monitor, a pull subscription dedicated to BeryX with the label datavault-sampler=true:

gcloud pubsub subscriptions create datavault-orders --topic=orders \
  --labels=datavault-sampler=true --message-retention-duration=1h --expiration-period=never
gcloud pubsub subscriptions add-iam-policy-binding datavault-orders \
  --member=serviceAccount:datavault@<projeto>.iam.gserviceaccount.com --role=roles/pubsub.subscriber

Service account roles: roles/pubsub.subscriber only on the sampling subscriptions; roles/pubsub.viewer on the project (to read the subscriptions' configuration and the topics' schemas); roles/monitoring.viewer if you want the backlog of your consumers' subscriptions.

Installation: pip install "datavault-backend[pubsub]" (google-cloud-pubsub + google-cloud-monitoring, the official clients). Avro reuses the streaming base's own decoder.

Decision — how to sample without stealing messages. In Pub/Sub, reading means doing a pull on a subscription, and each subscription has its own cursor. Alternatives considered:

The connector refuses, before any pull, subscriptions without the label datavault-sampler=true, push subscriptions, export subscriptions (BigQuery/Cloud Storage) or detached subscriptions, and two sampling subscriptions on the same topic. Seek is used, yes, but only on the dedicated subscription: before each sample it rewinds to agora − sample_window_seconds (default 3600; 0 turns it off). This discards old backlog — without it, a daily scan would sample yesterday's messages and freshness would raise false alarms — and, with retain_acked_messages or retention on the topic, it restores the window already read.

How it works: each subscription in subscriptions (id, or projects/<p>/subscriptions/<id>) gives a dataset named after the topic (schema = project). Each scan pulls up to sample_size messages (default 500, maximum 5000, 30 s budget, stops after two empty pulls) and acknowledges them; the profile, columns, rule rows and count use the same sample (a single pull per scan). Columns = top-level JSON/Avro fields, plus _pubsub_message_id and _pubsub_publish_time. row_count = messages in the sample. The automatic monitors (table_metadata) never pull — they would consume the dedicated subscription on every cycle.

Metrics (source_metrics, kind: "pubsub") — the core is the same as Kafka's (streaming.sample_metrics), so the stream rules, freshness and the onboarding suggestions work the same way:

• latest_message_at / last_message_age_seconds — from publish_time; the profile's latest_timestamp = the most recent one;
• malformed_pct / malformed_reasons (invalid_json, not_an_object, invalid_avro, schema_not_found, schema_unavailable, missing_schema, unsupported_schema_type); messages without data (attributes only) count in tombstones;
• consumer_group_lag — with consumer_subscriptions (your consumers' real subscriptions), the sum of num_undelivered_messages of those that read the topic, read from Cloud Monitoring (never a pull on them); per-subscription detail in consumer_subscriptions and consumer_oldest_unacked_age_seconds;
• schema_conformity — on topics with a Pub/Sub schema, from each message's googclient_schemaname/googclient_schemaencoding/googclient_schemarevisionid attributes: BINARY Avro that decodes is conformant; JSON is validated against that revision's Avro schema. Binary Protobuf is left out (malformed, unsupported_schema_type). null on topics without a schema — the rule explains why;
• sampling_subscription, sample_window_start, subscription_filter (the sample respects the subscription's filter), ack_failed.

Security / read-only: it never publishes, and never creates, alters or deletes resources; the only writes are to the cursor of the dedicated subscription (ack and seek), which is checked by its label before any pull/seek. Fixed Google endpoints (no hosts taken from the config); service account key only (token_uri pinned to oauth2.googleapis.com, universe googleapis.com), never the server's ADC — validated when the connection is created. Payloads, the key and tokens are never logged.

Microsoft Fabric (SQL analytics endpoint)

What you need: an Entra ID service principal with access to the Fabric workspace, Microsoft's ODBC 18 driver on the system, and the SQL analytics endpoint of the Lakehouse/Warehouse.

Step by step:

1. App registration + secret: same as steps 1-2 of Dynamics (you can reuse the same app) → tenant_id, client_id, client_secret.
2. Enable service principals in Fabric: app.fabric.microsoft.com → Settings → Admin portal → Tenant settings → Developer settings → "Service principals can use Fabric APIs" → Enabled (the tenant admin can restrict it to a security group that includes the app).
3. Workspace access: in the workspace → Manage access → + Add people or groups → add the service principal with the Viewer role (enough for reading via the SQL endpoint).
4. Endpoint and database: in the Lakehouse/Warehouse → Settings → SQL analytics endpoint → SQL connection string — this is the server (e.g. xxxx-yyyy.datawarehouse.fabric.microsoft.com); database is the name of the Lakehouse/Warehouse as it appears in the workspace.
5. System ODBC driver (required, in addition to the pip extra):
   • macOS: brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release && HOMEBREW_ACCEPT_EULA=Y brew install msodbcsql18
   • Debian/Ubuntu: follow Microsoft's instructions and sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18

Wizard fields: server, database, tenant_id, client_id, client_secret (optional: port, odbc_driver).

Installation: pip install "datavault-backend[fabric]" + the ODBC driver above.

Known limitations:

• A Lakehouse's SQL analytics endpoint is read-only by nature (ideal for BeryX), but Delta data can take a few minutes to show up in the endpoint after a write.
• System schemas (INFORMATION_SCHEMA, sys, queryinsights) are excluded from table discovery.
• The connection requires Encrypt=yes (TLS) — BeryX sets it automatically; environments with TLS inspection may need the correct certificates on the system.
• Without the system ODBC driver, the connection test returns the brew/apt installation instructions.
• Checksums, HLL sketches and the row sample by key hash use HASHBYTES('SHA2_256', CAST(col AS VARCHAR(max))): Fabric Warehouse has no NVARCHAR (its VARCHAR is UTF-8). That is why the checksum_dialect is fabric and data-diff does not compare Fabric checksums with those of a SQL Server (the text bytes are different). If the source refuses the hash sample, the scan uses the first rows and records sample_hash_fallback in the worker log.

SQL Server / Azure SQL / Synapse (mssql)

The source for Portuguese ERPs (Primavera, PHC, Sage) and the Microsoft mid-market. A single type covers SQL Server on-premises (2016+), Azure SQL Database/Managed Instance and Synapse (dedicated and serverless SQL pools) — all of them speak T-SQL through the same dialect.

Two authentication modes (the wizard picks one based on whether tenant_id is present):

In both: application_intent: "readonly" (optional) sends ApplicationIntent=ReadOnly, which in an Availability Group with readable replicas or in Azure SQL read scale-out routes the session to the read replica (which refuses writes and puts no load on the primary). host also accepts the Azure portal format (tcp:servidor.database.windows.net,1433). Other protocol prefixes (np:, lpc:, admin:) are refused. The anti-SSRF guard (netguard) validates the host after removing the tcp: prefix, the ,1433 port and the named instance (\\INSTANCIA), when the connection is created and again at runtime before each connection.

Read-only login (SQL Server / Azure SQL with SQL authentication) — SQL Server has no read-only session mode equivalent to Postgres's default_transaction_read_only: the guarantee comes from the GRANTs. Create a login with db_datareader only:

-- in master (SQL Server) — on Azure SQL, in the master database of the logical server
CREATE LOGIN datavault_ro WITH PASSWORD = '<password forte>';
-- in EACH database to analyse
USE <base>;
CREATE USER datavault_ro FOR LOGIN datavault_ro;
ALTER ROLE db_datareader ADD MEMBER datavault_ro;
-- optional: row counts from the partition catalogue (without this it uses sys.partitions)
GRANT VIEW DATABASE STATE TO datavault_ro;   -- SQL Server 2022/Azure SQL: VIEW DATABASE PERFORMANCE STATE

On Azure SQL you can instead create a contained user directly in the database (CREATE USER datavault_ro WITH PASSWORD = '…') and connect as that user.

Service principal (Azure SQL / Synapse):

1. App registration + secret: same as steps 1-2 of Dynamics (you can reuse the app) → tenant_id, client_id, client_secret.
2. Entra admin on the server: the logical server must have an Entra ID administrator set (portal → SQL server → Microsoft Entra ID → Set admin).
3. User in the database, connected as the Entra admin:

   CREATE USER [nome-da-app] FROM EXTERNAL PROVIDER;
   ALTER ROLE db_datareader ADD MEMBER [nome-da-app];
   

   Synapse dedicated pool: the same, in the pool's database. Synapse serverless: in the database created over the lake, and the service principal also needs to read the storage (Storage Blob Data Reader) for the views/external tables.
4. Firewall: add BeryX's outbound IP to the firewall rules of the logical server (error 40615 = IP refused).
5. System ODBC Driver 18 (instructions in the Fabric section) + pip install "datavault-backend[mssql,fabric]".

Not available in the production image. The BeryX image (cd.yml/gcp_deploy.sh) ships the [mssql] extra (pymssql), but not the [fabric] extra (pyodbc) nor the system ODBC Driver 18. In that installation GET /connections/types returns unavailable_modes: {"entra": "fabric"} and the wizard disables the "Entra ID" button, stating which extra is missing. When self-hosted, install both to enable the mode.

TLS: always. In Entra mode (ODBC) it sends Encrypt=yes;TrustServerCertificate=no — the certificate is verified. In SQL auth mode pymssql requests mandatory encryption (encryption=require), but the FreeTDS built into the wheels does not validate the server certificate (there is no CA configurable per connection): the session is encrypted, but not protected against a man-in-the-middle with a fake certificate. For full certificate verification use Entra ID mode (or a mssql+pyodbc:// url with a SQL login, which requires the ODBC driver).

What the connector does:

• Discovery excludes the sys, INFORMATION_SCHEMA and guest schemas and the fixed db_* roles.
• Profiling by T-SQL push-down: counts, nulls, distinct values, min/max, mean/standard deviation, zeros/negatives; quantiles via NTILE(20) (T-SQL's PERCENTILE_CONT only exists as a window function); bit is left out of min/max; text lengths are not computed (LEN() ignores trailing spaces — misleading numbers). varbinary is left out of distinct values/min/max; exotic types (xml, geography, sql_variant, hierarchyid) degrade to the null count without failing the scan.
• Cheap metadata (automatic monitors): row count from sys.dm_db_partition_stats (heap/clustered index, no scan; with a db_datareader-only login it falls back to sys.partitions). Only in the mssql connector: Fabric keeps the previous behaviour (no catalogue count) because these views have not been validated on a Fabric Warehouse. sys.tables.modify_date is exposed as the last schema change — it changes with DDL, not with INSERT/UPDATE, so it is never used as data freshness; freshness comes from a time column in the table.
• Driver query timeout (pymssql's timeout / pyodbc's Connection.timeout) taken from DATAVAULT_CONNECTOR_STATEMENT_TIMEOUT_S — there is no session statement_timeout in T-SQL.
• Checksums and HLL sketches (data-diff/INCR) from the first 4 bytes of HASHBYTES('SHA2_256', CAST(col AS NVARCHAR(max))) (card #317). CHECKSUM is not used: it follows the collation, and in a CI collation "A" and "a" would give the same hash — a false "equal" in data-diff.
• Row sample by primary key hash (see Row sample); without a key, the first rows in the order of the 1st column.
• Password and client_secret never appear in error messages (they are removed even when the driver echoes the connection string).

Full url: mssql+pymssql://user:pass@host:1433/base or mssql+pyodbc://… — only with the params in the mssql dialect allow-list (table in the PostgreSQL / MySQL / SQLite section). With pymssql, connect_timeout and applicationintent=readonly are translated into the driver's kwargs.

Out of scope: CDC, Change Tracking and sys.dm_exec_* (lineage/queries); Windows/Kerberos authentication and ActiveDirectoryPassword in the UI.

Oracle Database (oracle)

It is the database of many legacy ERPs, banks, telcos and the public sector. The connector uses python-oracledb in thin mode: it speaks the Oracle Net protocol in pure Python, without Instant Client or system libraries, so it works in the production image ([oracle] extra). It needs Oracle Database 12.1 or later, the thin mode minimum.

Fields: host (name or IP only), port (1521), service_name or sid, user, password and, optionally, schemas (a list, or comma-separated text). The connection always goes to the given host and port, with an explicit service or SID: the connector does not read TNS aliases, tnsnames.ora or wallets from the server. A host with a port, path, ?, ( or = (Easy Connect with parameters, a (DESCRIPTION=…) descriptor) is refused. The anti-SSRF guard validates that same host when the connection is created and again at runtime.

Read-only user. Create a user with only CREATE SESSION and SELECT on the tables to analyse. Avoid SELECT ANY TABLE, which opens every schema, including Oracle's own.

CREATE USER datavault_reader IDENTIFIED BY "<password forte>";
GRANT CREATE SESSION TO datavault_reader;
-- uma linha por tabela ou vista a analisar
GRANT SELECT ON hr.employees TO datavault_reader;
GRANT SELECT ON hr.departments TO datavault_reader;

In a schema with many tables, generate the GRANTs from the catalogue (as the schema owner or a DBA) and run the result. A table created later needs its own GRANT.

SELECT 'GRANT SELECT ON ' || owner || '.' || table_name || ' TO datavault_reader;'
FROM all_tables WHERE owner = 'HR';

Mechanical read-only. Besides the GRANTs, every connector transaction starts with SET TRANSACTION READ ONLY (the engine's begin listener). Inside it, Oracle refuses INSERT, UPDATE, DELETE and MERGE with ORA-01456, even when the user has the GRANT. DDL commits implicitly and leaves that transaction: against CREATE or DROP only the missing privilege counts, so the user must not have more than the GRANTs above. In a read-only transaction every query sees the database at the same instant. A long profile of a heavily changed table can hit ORA-01555 ("snapshot too old"), as any long consistent read can.

What the connector does:

• Discovery: the schemas with ALL_USERS.ORACLE_MAINTAINED = 'N' plus the user's own. If ALL_USERS lacks that column, all of them except a fixed list of Oracle schemas (SYS, SYSTEM, XDB, MDSYS, CTXSYS, OUTLN, DBSNMP, AUDSYS, WMSYS, OLAPSYS, ORDSYS, LBACSYS, DVSYS, GSMADMIN_INTERNAL, APEX_* and others). With schemas, only those. Names follow the SQLAlchemy convention: an unquoted identifier, upper case in the catalogue (HR.EMPLOYEES), shows in lower case (hr.employees). Reflection quotes them; no name is interpolated into the SQL. Tables, views and materialized views come with their kind: a view gets the limited profile of views (see Keys, partitions and object type).
• Types: NUMBER(p, 0) and INTEGER are integers. NUMBER(p, s) and NUMBER without precision are numeric (they can hold decimals) and show as NUMERIC(p, s) and NUMERIC. DATE carries a time and counts as a temporal column. CLOB, NCLOB, BLOB and LONG stay out of COUNT(DISTINCT), MIN/MAX and ORDER BY, which Oracle refuses on those types (ORA-00932, ORA-00997), and also out of LENGTH, the empty and whitespace-only counts and the not_null rule's treat_as_null: nulls are counted and the sample rows carry the value.
• Push-down profiling: counts, nulls, distincts, min/max, mean and standard deviation (STDDEV_POP), zeros and negatives, text lengths (LENGTH, in characters); quantiles by NTILE(20). LIMIT goes out as FETCH FIRST n ROWS ONLY.
• Cheap metadata (automatic monitors): the count is ALL_TABLES.NUM_ROWS, marked as an estimate (row_count_estimated, with LAST_ANALYZED in row_count_as_of), because it is the count from the last DBMS_STATS run and not the current one. When ALL_TAB_MODIFICATIONS is visible, the DML since that run (INSERTS + UPDATES + DELETES) serves as a write counter. Oracle records that view with a delay and clears it on every run, so its TIMESTAMP is not used as data freshness.
• Timeout: Oracle has no session statement_timeout. The connector sets oracledb's call_timeout (per round trip to the database) from DATAVAULT_CONNECTOR_STATEMENT_TIMEOUT_S. When it expires, the driver interrupts the call on the server and raises DPY-4024, which is never retried. Only a lost connection is retried (retries table below).
• Row sample: up to DATAVAULT_SAMPLE_FULL_SCAN_MAX_ROWS rows, the first rows in primary-key order (or by the first non-LOB column); above that, the first rows Oracle returns, with no ORDER BY, so the whole table is not sorted. This connector has no hash sampling, checksums or HLL sketches: data-diff compares only the rows read by key, and a table above DATAVAULT_PROFILE_DISTINCT_MAX_ROWS gets no approximate distinct count from a sketch. It has no push-down SQL rules either.
• The password never appears in error messages.

TLS: the connection is TCP without TLS. This version supports neither TCPS with a wallet nor Native Network Encryption, which oracledb only has in thick mode: a server that requires NNE refuses the connection with DPY-3001. Until then, connect over a private network (VPN, interconnect).

Full url: oracle+oracledb://user:pass@host:1521/?service_name=ORCLPDB1 or oracle+oracledb://user:pass@host:1521/SID, only with service_name, sid and connect_timeout (seconds, oracledb's tcp_connect_timeout). The driver is always oracledb in thin mode. The other oracledb.connect parameters stay out: config_dir and wallet_location read files from the server, https_proxy and protocol change the destination or the transport.

Out of scope: thick mode (Instant Client), TCPS and wallets, Kerberos/RADIUS, SELECT ANY TABLE, V$SQL/AWR (query lineage) and native SAMPLE (p) sampling.

MongoDB (collections as datasets)

Each collection in a database is a dataset. MongoDB does not store a schema: BeryX infers it from a random sample of the collection and measures fields with mixed types and missing fields. Implementation in app/infra/connectors/mongodb.py.

Connection: uri or fields.

In both modes: database (the database to read), collections? (a list; empty = the database's collections, up to 200, without system.*, views and time-series) and sample_size? (documents per sample: 1,000 by default, 10,000 at most).

TLS, in both modes: tls_mode? and ssl_ca_pem?, with the Postgres and MySQL rules (TLS). Without tls_mode, the connection is verify-full: encrypted, with the server's certificate and hostname verified against certifi's public CAs. An instance with a private CA takes the CA's PEM in ssl_ca_pem. Connecting without verifying the certificate (require) or without TLS (disable) needs explicit confirmation and is recorded in the audit log.

Only these options pass from the uri: authSource, replicaSet, tls (or ssl, which must match the tls_mode: a tls=false needs the disable tls_mode), readPreference, connectTimeoutMS, serverSelectionTimeoutMS and retryReads. The Atlas string carries retryWrites, w and appName, which are accepted and dropped. Any other option refuses the connection with 422, including tlsCAFile and tlsCertificateKeyFile (they read files on the server) and directConnection.

Read-only user. Create it in the admin database, with the read role on the database to analyse only:

// mongosh, connected as an administrator
use admin
db.createUser({
  user: "datavault_ro",
  pwd: passwordPrompt(),
  roles: [{ role: "read", db: "loja" }]
})

With the user in admin, the authSource is admin, the default of the fields mode. The read role covers everything the connector reads, $collStats included.

MongoDB Atlas:

1. Database Access → Add New Database User: password authentication and the read privilege on the database to analyse only (or the Only read any database role).
2. Network Access → IP Access List: add BeryX's egress IP. Without it, the connection times out ("Não foi possível contactar o MongoDB").
3. Connect → Drivers: copy the mongodb+srv://… string into the uri field, with the password from step 1.

What the connector reads (official pymongo driver):

• ping in the connection test, with the latency, and list_collection_names in discovery;
• a random sample through aggregate([{"$sample": {"size": N}}]), for the columns and the profile;
• estimated_document_count for the count, from the collection's metadata and without reading documents;
• $collStats for the size; if the user cannot read it, the profile carries on without a size;
• list_indexes and, for freshness, one find().sort().limit(1) tied to an index by hint: one per indexed field (up to 10; in the profile, only date fields) and another for _id. Each one reads the last entry of the index. Indexes where that does not happen are left out: partial, hidden, with a collation, and wildcard, text, hashed or geo indexes;
• find().limit() for the rows the rules read.

It calls nothing else: tests/test_mongodb_connector.py wires the connector to a fake client that records every call and fails on any other. The read preference is secondaryPreferred by default (reads go to a secondary when there is one), retryWrites is off and the sample, the count and the rows find have a 60 s cap on the server (maxTimeMS); each freshness read has 5 s.

Inferred schema:

• columns = top-level fields, in the order they appear in the sample (up to 500), with _id first and always as text;
• type = the type of most values. Integers and decimals (int, long, double, Decimal128) count as the same number: a field with both becomes float, with no conflict;
• nested objects and arrays come in as JSON text, as in Kafka; subfields do not become columns.

Source metrics (source_metrics, kind: "mongodb"), per field in fields:

fields_with_type_conflicts lists the fields with conflicts. estimated_document_count, storage and newest_insert_at (the time of the most recent _id ObjectId) complete the metrics. The column metrics (min/max, mean, lengths) are those of the majority type, over the sample. null_count counts the documents without the field and the explicit null values; a value of another type does not count as null, it counts in type_conflict_pct. distinct_count counts the present values of every type, by type and value, like a MongoDB unique index: 1 and 1.0 are the same value, 1 and "1" are two. Nulls and distinct values thus share the same basis, and a fully distinct key with a few values in text does not show up with repeats in the unique rule.

Count and sample. row_count comes from estimated_document_count and the profile marks it as an estimate (row_count_estimated) and as sampled (sample_rows). When the sample reads the whole collection (fewer documents than sample_size), the count is exact and the profile is not sampled.

Freshness. In a sampled profile, the maximum of an indexed date field comes from the index, exact, and replaces the sample's maximum (the sample's value stays in exact_column_metrics). In the automatic monitors, last_modified is the most recent of those maxima and the time of the most recent _id ObjectId; dates more than 24 h in the future (deadlines, expiry dates) are left out. Without date indexes, the rules' freshness comes from the sample's maximum.

Security:

• the uri never reaches the driver: mongodb_target, in app/infra/connectors/netguard.py, extracts hosts, credentials and options, and the connector opens the MongoClient with that result only;
• every host goes through the anti-SSRF guard when the connection is created and again before every connection. On a mongodb+srv://, BeryX resolves the SRV record, refuses targets outside the cluster's domain and validates each target; from the TXT record it only takes authSource and replicaSet;
• pymongo resolves each name through the guard (netguard.guarded_getaddrinfo, card #564): a single resolution per connection, refused if any address is internal, and the driver connects to the addresses of that resolution without going back to DNS. A name that switches to the internal network between validation and connection (DNS rebinding) never gets to open a socket. This applies to every connection the driver makes, monitoring included. TLS still verifies the certificate against the name. A pymongo older than 4.12 is refused, because it lacks the point where the guard goes in;
• the members the replica set announces go through the same guard: the driver's monitoring only contacts them, without credentials, if the resolution passes; operations, and with them authentication, only go to the members the guard accepts (server_selector). If a member lands on a blocked destination, the connection is closed and refused;
• local sockets are always refused, also with DATAVAULT_ALLOW_PRIVATE_NETWORKS=true: in the uri and the host, a path (/tmp/mongodb-27017.sock) or a host ending in .sock, with or without a port, which pymongo would connect to over a Unix socket without resolving the name; inside the driver, a .sock member announced by the replica set never gets to open a socket, not even in monitoring, which contacts it before the server_selector (card #602);
• with verify-full, pymongo receives tlsCAFile with the certifi bundle or with the customer's CA, written to a temporary 0600 file that is deleted on close() (tls.mongodb_tls_args). The uri's tlsCAFile is still refused, because it would read a file on the server;
• documents never go to the logs.

Out of scope: views and time-series collections, subfields as columns, change streams and X.509, LDAP or Kerberos authentication (authMechanism is outside the accepted options).

Salesforce

What you need: a user (ideally read-only), password and security token.

Step by step:

1. Security token: while signed in → avatar → Settings → My Personal Information → Reset My Security Token — it arrives by email. (If the organisation uses trusted IP ranges, the token may not be required, but the connector always asks for it.)
2. Recommended: an integration user with a read-only Permission Set on the objects to analyse.
3. domain: login (production) or test (sandbox).

Wizard fields: username, password, security_token, domain.

Installation: pip install "datavault-backend[salesforce]"

Known limitations: the MVP exposes the Account, Contact, Lead and Opportunity sObjects; basic profiling (counts via SOQL, no per-column metrics); it consumes the org's API calls.

Error messages (card #611). The connection test message goes to the connection's last error. When no object in the scan is analysed, the connector's last error goes to the scan's error. Workspace members see both. simple-salesforce puts the requested URL and the response body in the error, so these messages go through the SQL connectors' filter (card #590): the value of the password, the security_token and the open session's token comes out as ***. The password also comes out like that in the form the login sends it, escaped with html.escape (an & becomes &). Proof: backend/tests/test_salesforce_connection_redaction.py.

CSV

What you need: a local (or mounted) directory with .csv files — each file is treated as a table.

Wizard fields: path (directory).

Installation: works without extras (stdlib); pip install "datavault-backend[csv]" adds pandas to split the text faster. The types are the same with and without pandas.

How the file is read (card #289, in app/infra/connectors/csv_file.py):

• Encoding: the BOM decides first (UTF-8 or UTF-16, Excel's "Unicode Text"). Without a BOM, it tries utf-8 → cp1252 → latin-1. A latin-1 file whose only non-ASCII characters are accented letters is also valid cp1252 and is reported as cp1252, with the same text.
• Delimiter: decided by csv.Sniffer over the first 64 KiB, only among , ; tab and |. When there is no verdict (a single-column file), it stays ,. The encoding and the delimiter are kept in the table profile (file_format, e.g. {"delimiter": ";", "encoding": "cp1252"}).
• Types: the coercion is BeryX's, not pandas' (app/infra/connectors/profiling.py). A value only becomes an integer if it has no leading zeros ("0" yes, "08001" no), so postcodes, SKUs, NIF (Portuguese tax number) and phone numbers stay text. Only the unquoted empty field is NULL: N/A and - are values and stay visible in the profile. As in Postgres' COPY … CSV, an unquoted empty field is NULL and a quoted "" is empty text (empty_count), on Python 3.12 or later (the Docker image uses 3.13; on 3.11 the reader cannot tell them apart and both count as NULL); in a file whose header is fully quoted and that has no unquoted empty field, the writer quoted everything and "" counts as NULL. In a numeric, date or boolean column, '' is always NULL (app/infra/connectors/csv_file.py, parse_csv_lines and coerce_table). GCS/S3 read CSV files with the same reader. NaN/inf in a numeric column count as float, with nan_count kept separate from nulls. Dates only in ISO-8601, and either all with a time zone or all without; a column that mixes the two stays text, because no time zone is inferred. The literals true/false (True, TRUE…) are booleans.
• Caps: the profile reads up to DATAVAULT_CSV_PROFILE_MAX_ROWS rows (1,000,000) and DATAVAULT_CSV_PROFILE_MAX_BYTES bytes (256 MB). Beyond that, the profile covers the first rows and states sampled and sample_rows (the UI shows "profile on a sample of N rows").
• Row sample (row rules, duplicates; card #317): one row in every k of those the profile reads, starting from semente mod k — it spreads across the part of the file that is read instead of staying in the first 5,000.

Known limitations: Portuguese decimals (1.234,56) and dd/mm/aaaa dates stay text (no locale is inferred; the profile's format mask describes them). Numbers with more than ~16 significant digits lose precision (float). In a CSV an empty string cannot be told apart from NULL. The case-by-case detail is in the adversarial corpus.

Google Cloud Storage / Amazon S3 (files as tables)

Data lake in object storage. Each *.parquet, *.csv or *.csv.gz file under bucket/prefix is a table, named after its path relative to the prefix (without the extension). Event and log files (*.jsonl, *.ndjson, *.log and the same in .gz) are tables too, with one line per event (see below). CSV files are read as in the CSV connector: encoding and delimiter from the start of the file (64 KiB), dtype=str in pandas and the same type coercion. The same file gives the same profile on disk and in a bucket. Parquet files keep their native types. Text columns are called text in this connector. Shared implementation in app/infra/connectors/object_storage.py (fsspec).

Credentials always explicit (A2): either the customer's credential, or anonymous: true for public buckets. Without either, the connection is refused: BeryX never uses the server's own credentials (Cloud Run ADC, boto chain). anonymous must be a boolean ("false" as text is refused).

Google Cloud Storage — on the customer's side:

1. In the Google Cloud console: IAM & Admin → Service Accounts → Create service account (e.g. datavault-reader), with no project-level roles.
2. On the bucket: Permissions → Grant access to the service account with Storage Object Viewer (roles/storage.objectViewer), which lists and reads objects and writes nothing.
3. On the service account: Keys → Add key → Create new key → JSON.
4. In the wizard: bucket (without gs://), optional prefix (e.g. gold/vendas) and the full content of the JSON in the "Service account JSON" field (it is sent as service_account_info). For a public bucket, tick "Anonymous access" and leave the key blank.

With credentials, step 2 of the new connection shows the Google Cloud Registration guide and the Test access button (see the BigQuery section).

Amazon S3 (and compatible services) — on the customer's side:

1. A read-only IAM user (or role with access keys), with a policy along these lines:

   {
     "Version": "2012-10-17",
     "Statement": [
       {"Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::<bucket>"},
       {"Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::<bucket>/<prefixo>/*"}
     ]
   }
   

2. Security credentials → Create access key → access_key_id and secret_access_key.
3. In the wizard: bucket, optional prefix, the two keys (or "Anonymous access" for public AWS Open Data buckets), optional region (e.g. eu-west-1) and optional endpoint_url for S3-compatible services (MinIO, Cloudflare R2, …).

Fields: bucket, prefix?, max_files? (200 by default) and the credentials above. protocol is not configurable: a gcs connector only speaks gcs/gs and an s3 one only s3 (without this, file or http would give local reads or SSRF).

Installation: pip install "datavault-backend[gcs]" or pip install "datavault-backend[s3]" (both are in the production image). PDFs for document analysis also need the [documents] extra; scanned PDFs (with no text layer) need the [ocr] extra and the system's Tesseract (all three are in the production image).

Known limitations:

• Discovery lists at most max_files files (200 by default), in alphabetical order of path. Parquet/CSV files take the cap first and event and log files only after them, so a new .log never takes the place of a dataset already in the catalogue. An event file with the name of a Parquet/CSV file (app.jsonl next to app.csv) is left out; between two event files with the same name (app.jsonl and app.jsonl.gz) the first in alphabetical order stays.
• The profile reads the first 100,000 rows of each file (Parquet files only through the row groups needed). The column metrics (nulls, distinct values, mean, deviation, samples) come from that sample, not from the whole file, and the profile says sampled/sample_rows when the file went on further — each column carries from_sample (the UI shows "profile on a sample of N rows"). It profiles at most 120 columns (DATAVAULT_PROFILE_MAX_COLUMNS), chosen by the wide-table priority; the catalogue keeps all of them.
• The row_count is always for the whole file, even in a sampled profile: the row_count_min rule, the volume monitor and the row_count_drop drift see the real number. Parquet: num_rows from the footer, without reading data; when every row group has statistics, the nulls (any type) and the min/max (integers, floats, timestamps and dates) of each column become exact and the column lists them in exact_metrics — strings keep the sample's min/max (writers truncate those statistics). CSV/CSV.GZ: a second streaming pass (csv.reader; cells with line breaks inside quotes count as one row) up to DATAVAULT_OBJECT_STORAGE_COUNT_MAX_BYTES (512 MB, measured on every read in the file's bytes and, for a .gz, also in the decompressed bytes — a highly compressible .gz expands thousands of times and the count stops at the first measurement that passes the ceiling, even in the middle of a row); above that the count is extrapolated from the observed ratio (rows counted × file size ÷ file bytes consumed — for a .gz, the compressed bytes that the inflate consumed) and the profile says row_count_estimated (the UI shows "estimated row count"; the volume drift says so in its description). Files under 100,000 rows get no second read; a truncated or corrupt .gz, or a cell over 64 Mi characters, after the sample does not fail the profile (the count stops there and stays estimated). The count raises the field limit of the worker's csv module to 64 Mi characters (process-wide; it stays raised). Note: in Parquet float columns pandas merges NaN with NULL in the sample — with statistics in the footer, null_count counts only NULL.
• Distinct values, the mean, zeros and value samples still come from the sample. So anything that compares them with the volume uses the rows read, not the whole file's row_count. That is the case for a column's unique rule, key suggestions, the profile's alerts and semantic types, the synthetic data generator and the rule estimate in the browser. In a Parquet file with exact nulls from the footer, the column keeps the sample's nulls in sample_null_count. Without this basis, a unique key in a file over 100,000 rows showed up full of repeats (method).
• A CSV that is ASCII in the first 64 KiB and cp1252 further on fails on read with the pandas message, instead of silently swapping characters.
• Parquet and CSV files over 2 GB are refused at profiling ("Partition the source"). Event and log files have no such cap: they are streamed only up to the line cap.
• No cheap automatic monitors: there is no catalogue metadata. Freshness comes from the file's first temporal column.
• Each scan makes GET requests and transfers data from the bucket, and the cloud provider charges for them.
• The S3 endpoint_url goes through the anti-SSRF guard. In the SaaS it must be a public endpoint; a MinIO on a private network is only reachable when self-hosted with DATAVAULT_ALLOW_PRIVATE_NETWORKS=true.

Event and log files (card #183): .jsonl, .ndjson, .log, .jsonl.gz, .ndjson.gz and .log.gz (archived webhooks, access logs, application logs). Each line is an event, read in this order: JSON, syslog (RFC 5424 and RFC 3164), nginx/apache (combined and common) and logfmt key=value (at least two pairs). A line that fits none of them counts as malformed, with the reason (invalid_json, not_an_object, unparseable, line_too_long above 1 MiB). Logs in the default format of Python's logging or of log4j (2026-10-09 10:00:00,123 INFO app: …) fit none of them: every line counts as unparseable and the max_malformed_pct rule always fails. To measure them, the application has to write its logs as JSON or logfmt. A timestamp that falls outside the years 1 to 9999 in UTC (9999-12-31T23:59:59-03:00, for example) or carries an offset of 24 hours or more does not make the line malformed: the event counts, without a readable timestamp. In an access log the date is part of the format, so such a date makes the line count as unparseable. The columns come from the event keys, as in a Kafka topic: the first 1,000 distinct keys, in the order they appear (columns_capped says whether there were more). Cloud Logging exports to GCS are .json files and are not read: a .json file in a bucket may hold one event per line or a whole document. Implementation in app/infra/connectors/event_files.py.

• Reading: the first DATAVAULT_LOGFILE_MAX_LINES lines (50,000 by default), in 64 KiB blocks and with .gz decompressed as a stream: the file is never held whole in memory. Reading also stops at 256 MiB decompressed or after 30 s. If the file went on, the profile says sampled and the line count is extrapolated from the bytes read (row_count_estimated). A .gz truncated after lines were read does not fail the profile: reading stops there.
• Metrics (source_metrics with kind: "logfile"): dominant format, lines read, % malformed and reasons, % of lines with a readable timestamp, % of line pairs in chronological order, largest gap between events (seconds), distribution by level (debug, info, warning, error, critical and other for a level that is not recognised), number of distinct key sets (more than one points to a schema change between app versions), the 20 most frequent message templates with the count of each and pii_kinds, the number of messages with each kind of personal data (email, NIF with a valid check digit, Portuguese IBAN and mobile number, payment card passing Luhn).
• Templates: a template is the message with digits → <n>, UUID → <uuid>, hexadecimal → <hex>, IP → <ip> and email → <email>. People's names and the rest of the text stay as they are: "Pagamento recusado para Maria Santos" is left unchanged. With real samples in prompts (the default mode), the root-cause diagnosis sends the templates to Claude, like the column samples. With synthetic samples, the AI gets only the counts, without the text.
• Rules: the stream rules apply as in a topic (max_malformed_pct, max_event_age, required_fields). max_event_age and the dataset's freshness use the most recent event in the lines read. In a file larger than the read cap, those are the first lines, and a file that grows at the bottom looks older than it is (the same happens with a CSV). Onboarding suggests max_malformed_pct at 1%. schema_conformity does not apply: a file has no registered schema.
• Personal data: when messages carry personal data, the scan opens a log_pii automatic monitor issue (warning) with only the counts per kind. Once marked "ignored", it does not reopen on that dataset.
• UI: the dataset detail shows the "Log file" panel with the metrics, the levels as bars and the template table.

Documents in the same bucket: PDF, DOCX, TXT, MD and HTML files under the prefix are not tables. They go through document analysis (Documents button, UNSTR), with a ceiling of 50 MB per document.

Scanned PDFs (local OCR, card #418): a PDF with no text layer goes through OCR in the worker (the image's Tesseract 5, Portuguese, English and Spanish), page by page, with a page ceiling (DATAVAULT_DOCUMENTS_OCR_MAX_PAGES, 50) and a per-document deadline (DATAVAULT_DOCUMENTS_OCR_TIMEOUT_SECONDS, 180 s), within a per-scan budget (DATAVAULT_DOCUMENTS_OCR_SCAN_BUDGET_SECONDS, 3 h). The document is marked with the method and the average confidence, and the corpus opens the problem "N documents read by OCR". OCR is local: the document does not leave the installation, and nothing is written back to the bucket. This applies to every document source (GCS, S3, SharePoint, Google Drive, Confluence). Without OCR installed, the PDF stays out of the index with "needs OCR" (method).

Images in the same bucket (card #182): with include_images: true on the connection (the "Include images in the document analysis" box in the wizard; off by default), JPEG, PNG, WebP, GIF, TIFF and BMP files also go through document analysis. This applies to GCS, S3, SharePoint and Google Drive. Each image is profiled in the worker with Pillow only, no GPU and no LLM: format, width × height, corrupt or truncated file, EXIF (whether there is GPS, the capture date and the device make and model, never the serial number), SHA-256 and a 64-bit dHash. The corpus gets an images block with exact duplicates (SHA-256) and near duplicates (dHash within DATAVAULT_IMAGE_NEAR_DUP_HAMMING bits, 8 by default), formats, resolutions, % with GPS, % corrupt and the age by EXIF against the file date. It opens the problems image_corrupt, image_near_duplicates and image_gps_pii. A photo with GPS coordinates is personal location data, so the open or acknowledged problem blocks readiness for RAG and for training, also in the data policy's min_ai_readiness rule. An image above DATAVAULT_IMAGE_MAX_PIXELS (50 MP) is not decoded and is marked "too large". Pixels and EXIF text do not go to the database, the LLM or the logs, and the dataset's "Images" section shows metrics only, with no thumbnails. Without the [images] extra (Pillow, which the production image already has through the [ocr] extra), images count as not analysable and the scan still completes. Images and documents share the limit of the analysis request (200 by default, up to 1000). On GCS and S3, the cut follows the alphabetical order of the paths; on SharePoint and Drive, the listing order. To keep images from pushing documents out, raise the limit or point the prefix at a folder with images only.

Azure Blob Storage / ADLS Gen2 (files as tables)

Data lake in Azure: Blob Storage and ADLS Gen2 (accounts with a hierarchical namespace), including the storage account of a Synapse workspace and the external locations of Databricks on Azure. The adls connector (AdlsConnector in app/infra/connectors/object_storage.py) is the GCS and S3 code on top of adlfs, with the fsspec protocol abfs. The container plays the bucket: each Parquet, CSV or .csv.gz file under container/prefix is a table. Event and log files, documents (PDF, DOCX, TXT, MD, HTML), OCR, images (include_images) and limitations follow the GCS and S3 section.

One credential, always explicit (A2). The connection carries exactly one of these: the account key (account_key), a SAS token (sas_token), a service principal (tenant_id + client_id + client_secret) or anonymous: true for public containers. With none, or with two, creation is refused with 422. BeryX never uses the server's credentials. adlfs always receives anon=True, so without a credential it stays anonymous and never falls back to DefaultAzureCredential (managed identity, Azure CLI, environment variables). The connector refuses to connect if the server has an AZURE_STORAGE_* variable set that adlfs would read instead of the connection's credential, such as AZURE_STORAGE_CONNECTION_STRING. Error messages give only the names of fields and variables, never the value.

Service principal (recommended) — on the customer's side:

1. In Entra ID: App registrations → New registration (e.g. datavault-reader) and, under Certificates & secrets, a client secret with an expiry date.
2. On the container: Access Control (IAM) → Add role assignment → Storage Blob Data Reader for the app. The role lists and reads blobs and writes nothing. Assigned on the container, not the account, the app does not see the other containers.
3. In the wizard: the storage account (account_name, the part before .blob.core.windows.net), the container, the optional prefix and, under «Service principal (Entra ID)», the ID of the tenant, the application ID and the secret.

SAS token: generate it on the container (Shared access tokens) with only the Read and List permissions and an expiry date, and paste the token (it starts with sv= and includes sig=), not the URL. When it expires, the connection test and the scans fail until you paste a new one.

Account key: it works, but the key gives full access to the account, writes included. The connector only reads (find, info and open('rb')); a service principal or a SAS limit what a leaked credential allows.

Public container: «Acesso anónimo» (anonymous access), for containers with the Container access level. The «Feriados públicos — Azure Open Datasets» preset uses it.

Sovereign clouds: endpoint_suffix accepts core.windows.net (the default), core.usgovcloudapi.net (Azure US Government) and core.chinacloudapi.cn (Azure China, 21Vianet), and refuses anything else. The host is always <account_name>.blob.<endpoint_suffix>, and the service principal asks the Entra ID of the same cloud for the token.

Azurite (development and CI): endpoint_url (e.g. http://127.0.0.1:10000/devstoreaccount1) points the connector at the emulator. It is accepted only with DATAVAULT_ALLOW_PRIVATE_NETWORKS=true; in production it is refused with 422. With endpoint_url, the credential is the key, the SAS or anonymous access.

Fields: account_name, container, prefix?, max_files? (200 by default), include_images?, endpoint_suffix? and one credential. protocol is not configurable: abfs, abfss and az are ignored (the connector always speaks abfs) and any other value is refused.

Installation: pip install "datavault-backend[adls]" (adlfs, which brings azure-storage-blob and azure-identity, plus pandas and pyarrow). It ships in the production image. Without the extra, the connector answers with the pip instruction.

OneLake and Synapse: Microsoft Fabric's OneLake speaks the ADLS Gen2 API on its own host (onelake.dfs.fabric.microsoft.com), outside the clouds this connector accepts. For a Fabric Lakehouse or Warehouse, use the fabric connector. In a Synapse workspace, point adls at the storage account of the workspace and at the file system holding the files; the SQL pool tables are read through mssql.

Network: an account with a firewall (Networking → Enabled from selected virtual networks and IP addresses) must accept BeryX's egress IP (see the top of this guide).

SFTP (partner and bank file drops)

Banks, logistics partners, insurers and EDI networks deliver files into an SFTP folder. The sftp connector (SftpConnector in app/infra/connectors/sftp.py) reads that folder with the GCS and S3 code, on top of the fsspec protocol sftp (paramiko). The root_path folder plays the bucket: each Parquet, CSV or .csv.gz file under root_path/prefix is a table, and so are event and log files. PDF, DOCX, TXT, MD and HTML go to the document analysis. Reading, OCR, images (include_images) and limitations follow the GCS and S3 section.

"Today's file did not arrive". Each file's modification date (the mtime from the SFTP stat) is the last modification the automatic monitors store on every scan. When the partner replaces the same file on each delivery (e.g. pagamentos.csv), the freshness monitor learns the delivery cadence and opens an issue when the next one is late. A file with the date in its name (pagamentos_20261009.csv) is a new dataset on each delivery, with a single date, and that monitor does not see the delay.

Pinned server key. host_key_fingerprint is the SHA256 fingerprint of the server key, in the format OpenSSH prints:

ssh-keyscan -t ed25519 sftp.parceiro.example | ssh-keygen -lf -
# 256 SHA256:… sftp.parceiro.example (ED25519)

Confirm the value with the partner through another channel: ssh-keyscan asks the server itself and does not prove it is the right one. The wizard accepts the whole line or just the SHA256:…. The connector compares it with the key the server presents, before authenticating. If it does not match, the connection drops without sending the password or reading data, and the connection test names the key type the server presented. paramiko negotiates the Ed25519 key first; on a server without one, use -t ecdsa or -t rsa. If the partner only gives you the RSA or ECDSA fingerprint of a server that also has Ed25519, paste the whole ssh-keygen -lf line, which ends with the type (e.g. (RSA)): the connector then negotiates only keys of that type. In production (DATAVAULT_ENV=prod) the field is required and creating the connection without it is rejected with 422. Outside production, a connection created through the API without the field accepts the key unverified, and the connection test shows the fingerprint the server presented; the wizard always asks for it. The MD5 format is not accepted.

Only the connection's credential. The connection carries password or the private key in PEM (private_key_pem, with passphrase if it is encrypted), never both. Paste the key content, which starts with -----BEGIN: a file path is rejected with 422. The connector never reads the BeryX server's known_hosts, keys or SSH agent. It accepts Ed25519, ECDSA and RSA keys. Error messages name only the fields.

Read-only, inside root_path. The connector lists folders, runs stat and opens files in rb. Writing, deleting, renaming, creating folders and downloading to the BeryX server's disk are blocked in the connector's own filesystem. root_path is an absolute path (e.g. /outbound) and prefix a subfolder of it, both without ... Every folder listed and every file read or queried (stat) must sit inside root_path, also after the server resolves links: a prefix that is a link pointing outside is refused. Symbolic links are neither listed nor opened.

On the partner's side (or your team's):

1. A user just for BeryX, locked to the delivery folder and read-only. In OpenSSH: ChrootDirectory for the folder and ForceCommand internal-sftp -R (-R puts the SFTP server in read-only mode). On a managed SFTP service, a read-only policy for that user.
2. A credential for that user. With a key, generate a pair just for this (ssh-keygen -t ed25519 -f datavault_sftp), put the .pub in the user's authorized_keys and paste the private key into the wizard.
3. The server key fingerprint (above).
4. If the server filters by IP, allow BeryX's egress IP (see the top of this guide).

Network: host is only the name or the IP, without sftp://, port or path, and the port is an integer (22 by default). The anti-SSRF guard checks the name and each resolved IP before opening the socket, and paramiko connects to the IP that passed the guard. In production, private networks and loopback are rejected; the cloud metadata endpoint is always rejected.

Fields: host, port?, username, password or private_key_pem (+ passphrase?), host_key_fingerprint, root_path, prefix?, max_files? (200 by default), include_images?. protocol is not configurable: only sftp; fsspec's ssh alias and any other value are rejected.

Installation: pip install "datavault-backend[sftp]" (paramiko, plus pandas and pyarrow). It ships in the production image. Without the extra, the connector answers with the pip instruction.

FTPS and FTP are not supported. fsspec's ftp protocol does not do TLS reliably, and FTP without TLS sends the password in the clear. Ask a partner that only has FTPS for SFTP or a bucket.

PostgreSQL / MySQL / SQLite

• PostgreSQL (postgres, postgres extra): create a read-only user (GRANT CONNECT ON DATABASE, GRANT USAGE ON SCHEMA, GRANT SELECT ON ALL TABLES IN SCHEMA ...). Fields: host, port, database, user, password, or a full url.
• MySQL/MariaDB (mysql, mysql extra): GRANT SELECT ON <db>.* TO 'datavault'@'%';. Same fields as Postgres.
• SQLite (sqlite, included): path to the .db file — meant for tests and demos, not for production.

Read replica (Postgres, MySQL and SQL Server): replica_host and replica_port (optional) put all of BeryX's connections on the replica, with the same credentials and no fallback to the primary. Details in DBA.md.

Connection test message (card #590). In the SQL connectors (Postgres, MySQL/MariaDB, SQLite, Snowflake, BigQuery, Databricks, Fabric, SQL Server and Oracle), the driver's error goes through a filter before it reaches the test result and the connection's last error, which every workspace member sees. The values of the config's password, tokens and secrets, the whole url and the url's password, both decoded and percent-encoded, come out as ***. In pairs such as password=… or PWD=… the value goes, a ://user:password@ DSN loses the password and a PEM block goes entirely. The host and the user stay, because they say where the connection failed. Proof: backend/tests/test_connection_message_redaction.py.

Read errors (card #612). The same filter runs on the errors the SQL connectors return when listing tables and columns, reading a table's structure, reading and counting rows, computing checksums, evaluating an SQL rule or a remediation's trial run and reading the FinOps telemetry of Snowflake, Databricks and BigQuery. These errors reach the result of table discovery, the scan's error and the rules' result. In rules, the error's first line goes through the filter before the literal mask, which cuts at an unclosed quote and at 300 characters. Proof: backend/tests/test_sql_read_error_redaction.py, with a guard that fails an exception text outside the filter in an SQL connector's method.

TLS (Postgres and MySQL/MariaDB)

Since card #449, every Postgres or MySQL connection has a tls_mode:

• Server CA (ssl_ca_pem). Without it, the certificate must come from a public CA (the Mozilla bundle, via certifi). RDS, Cloud SQL, Azure Database and internal CAs are private: paste the CA certificate in PEM (text that starts with -----BEGIN CERTIFICATE-----). A bundle with several CAs also works, up to 256 KB, which is enough for the RDS global bundle. The PEM is encrypted with the rest of the config and never appears in the API, the logs or the audit log. It is only accepted with verify-full. File paths and private keys are refused with 422.
• Drivers. In Postgres, libpq receives sslmode and sslrootcert. A custom CA goes into a temporary 0600 file, which lives as long as the engine and is deleted at the end. libpq's sslrootcert=system is not used: the image's psycopg2-binary embeds a libpq 17 with an OpenSSL that looks for the system CAs in a folder from the wheel's build that does not exist. In MySQL, PyMySQL (≥ 1.2) receives an SSLContext with the CA in memory and refuses to connect if the server does not announce TLS. With tls_mode, the connection always uses the connector's official driver (postgresql:// or postgresql+psycopg2://; mysql+pymysql:// or mariadb+pymysql://).
• Full url. The URL's sslmode is only accepted if it is the same as the tls_mode. allow, prefer and verify-ca are refused: the first two fall back to clear text without warning and the third does not check the hostname. sslrootcert, sslcert and sslkey remain outside the allow-list.
• Read replica. With verify-full, the replica's certificate must be valid for the replica_host.
• MongoDB (#211). MongoDB connections have the same tls_mode, the same ssl_ca_pem and the same confirmation. There are no MongoDB connections from before the rule: without tls_mode, they connect with verify-full. pymongo receives tls and tlsCAFile (the certifi bundle or the custom CA in a 0600 file) and, with require, tlsAllowInvalidCertificates and tlsAllowInvalidHostnames. The uri's tls must match the tls_mode. Details in the MongoDB section. Proof: backend/tests/test_mongodb_tls.py (driver options, API and audit log) and backend/tests/test_mongodb_tls_docker.py, against the compose's mongodb-tls (tls profile, TLS only, with the postgres-tls certificate).
• Connections created before #449. They have no tls_mode and keep the old behaviour: libpq in prefer and PyMySQL in "preferred" mode, which encrypt if the server allows it, without verifying, and can fall back to clear text. The app shows them as "TLS not verified", and the API returns tls_mode: null and tls_verified: false. To verify them, recreate the connection. Changing the default for these connections is the owner's decision (ENCRIPTACAO_RESIDENCIA.md §8).
• Public sources in the gallery. The RNAcentral Postgres and the Ensembl MySQL do not accept TLS. The Rfam MySQL uses the self-signed certificate that MySQL generates. The presets come with disable and require, and the app asks for the same confirmation before connecting them.
• Proof. backend/tests/test_connector_tls.py covers the kwargs per mode and PyMySQL with real TLS against a fake MySQL server. backend/tests/test_postgres_tls_docker.py runs against a Postgres with a self-signed certificate: the postgres-tls service in docker-compose.connectors.yml, in the matrix's docker tier.

Mechanical guarantees (beyond the GRANTs): Postgres sessions open with default_transaction_read_only=on and SQLite with mode=ro. A full url must use the connector's own dialect (e.g. postgresql://… in a postgres connector) and only accepts the query params in the dialect's allow-list (any other is refused with 422, pointing to the structured fields; the rule is also applied at runtime to connections already saved):

Left out, among others: host/hostaddr, options, local_infile, init_command, odbc_connect, credentials_path and authenticator.

Destination revalidated on connect (card #339): every host and URL in the config goes back through the anti-SSRF guard before the connection is opened, once per engine (BaseConnector.assert_destination_allowed), and not only when the connection is saved. A connection saved before a new rule, or a name that has since started resolving to the internal network (DNS rebinding), is refused before any TCP connection. The name is resolved once and, if any of the IPs (IPv4 or IPv6) is internal, the connection is refused. The scan marks the dataset as failed with the reason "Destino de conexão não permitido neste ambiente: '' ()", without the IP. Where the driver allows it, the connector connects only to the validated IPs, in the resolver's order (if the first refuses the connection, it tries the next), and the driver does not resolve the name again:

With private networks blocked (production), a host that does not resolve is refused (netguard.check_host, card #564), when the connection is created and on each revalidation: accepting it let the driver resolve it later, without the guard, and the name could meanwhile point to the internal network. In the Snowflake and BigQuery url, the "host" is the account and the project, not a network name: it still goes through the guard, but it does not have to resolve. The drivers with a pinned IP (psycopg2, psycopg and pymysql) also refuse a path (local Unix socket) and a comma-separated list of hosts: the guard would not have validated the destination the driver was going to connect to. With DATAVAULT_ALLOW_PRIVATE_NETWORKS=true (development), the local socket and a name that does not resolve connect as before, without a pinned IP.

Standard deviation: computed by the native function STDDEV_POP (Postgres, MySQL/MariaDB, BigQuery, Snowflake, Databricks, Oracle) or STDEVP (SQL Server, Azure SQL, Synapse, Fabric). In SQLite it is a Python aggregate registered on each connection: exact for integers and Welford for the rest, in the same query as the other aggregates. The old formula E[x²] − E[x]² in floating point cancelled itself out in high-magnitude columns and returned 0.0 for an epoch-millis column with a spread of 100 (the correct value is 28.9). It remains only as a fallback for unknown dialects. A mean or deviation that the source returns as NaN/±Inf (e.g. float8 with 'NaN' in Postgres) is shown as "unknown" in the profile.

Statement timeout and retries

DATAVAULT_CONNECTOR_STATEMENT_TIMEOUT_S (120 s) is the maximum time of each query on every SQL source. The source cancels the query on the server, so a runaway aggregate does not keep the warehouse running until the vendor's limit (2 days on Snowflake). Each dialect receives it its own way (app/infra/connectors/timeouts.py, card #346):

With 0, no dialect gets a limit. SQL rules and the remediation rehearsal change the session limit only for their query and restore the connection's limit at the end. A MySQL or Databricks that refuses the SET (old version) still connects, with the server's limit, and logs connector_statement_timeout_unsupported.

Retries. A transient source error used to leave the dataset failed in that scan and a gap in the drift and anomaly history. Now a table's profile, incremental profile, rows, metadata and count are retried up to DATAVAULT_CONNECTOR_RETRY_ATTEMPTS times (3), waiting up to 2 s and 4 s (exponential, at most 30 s, with jitter). Only what is transient in each dialect is retried (app/infra/connectors/retry.py):

Timeouts are never retried (the query cancelled for time would cost the same again), nor authentication and permission errors, BeryX's own refusals (bytes cap, destination refused by the anti-SSRF guard) or a configuration error. SQLite never retries. A dataset's retries add up to at most DATAVAULT_CONNECTOR_RETRY_BUDGET_S (60 s, waits and repeated attempts): once the budget runs out, the dataset fails with the last error. Each retry is logged as connector_retry with the scan's correlation_id, the attempt and only the error's class name (the driver message may carry data). When there were retries, the scan summary keeps the total in retries. An error solved by a retry does not count towards the connection's circuit breaker.

Wide tables

Each aggregate is a scan of the column, and in a warehouse it is billed. So a table with more than DATAVAULT_PROFILE_MAX_COLUMNS columns (120) profiles only 120. Until card #289 these were the first 120 and the dataset's catalogue kept only those: in a 300-column SAP/CRM table, 180 disappeared from the catalogue and the schema change monitor (SCHEMACHG) was blind to them. Now:

• the catalogue keeps every column, read in the same pass as the profile (without a second trip to the source);
• the profiled columns are chosen by priority: keys (PK, id, *_id), date/time columns, names with personal data (email, nif, iban, telefone, …) and then schema order. The choice is deterministic and stable between scans, and it is the same in the incremental profile. It lives in app/infra/connectors/profiling.py;
• the table profile records profiled_columns, skipped_columns and column_count (the catalogue). The UI shows "120 of 300 columns profiled" and marks the rest as "not profiled";
• the schema diff compares full catalogues. The schema_change issue of a wide table says how many columns were profiled and that the diff covers the whole catalogue. On the first scan after this version the previous profile does not yet have the snapshot of the catalogue, and the diff compares only the columns that profile saw. This way ~180 false "new" columns do not appear, and the issue's note explains it.

Row sample by dialect

Rules that need rows (pattern, required_fields) and duplicates read up to 5,000 rows per table (ROW_FETCH_LIMIT, always below the source's maximum sampling); label quality reads with the same method, up to its own ceiling. Since card #317 those rows are a sample chosen by the source itself, with a fixed seed per dataset — the same from scan to scan, so that duplicate clusters and pattern violations do not open and close without the data changing. SELECTs only:

p comes from the profile's row_count (~1.5× the limit, trimmed in Python by the row hash); a view or foreign table in PostgreSQL refuses TABLESAMPLE and the sample falls back to the first rows, with the sample_native_fallback event in the worker log. The profile records the method in metrics_json["sample"] (method, rows, fraction, seed, deterministic) and the app shows it in a badge next to the columns; the duplicates issue and the issues of the pattern and validity rules say so in their text. A manual scan with "New sample" (resample in POST /scans) uses another seed for that scan only.

The profile's sample_values (up to 20 per column, for semantic types and PII detection) come from one sample of 1,000 rows by the same method, instead of one SELECT DISTINCT … LIMIT 20 per column: the SQL profile runs 2 + ⌈colunas/10⌉ queries per table, where it used to run 1 + ⌈colunas/10⌉ + colunas (in a 120-column table, 120 extra queries; in BigQuery, 120 billed reads). Some columns go back to ask for their values separately: those that are almost all NULL and those with few categories that the sample did not see in full, so that the accepted-values proposal does not miss a rare category. With the BigQuery cost guard degraded, none of them does. In the incremental profile the sample stays within the new window and is never TABLESAMPLE, which runs before the WHERE and would read the whole table: it goes by the key hash or by the first rows by the temporal column. The 32-bit hash per dialect (hashtext, CRC32, dv_hash, FARM_FINGERPRINT, HASH, xxhash64, HASHBYTES) also serves the data-diff checksums and the HLL sketches of the incremental profile, which now exist in BigQuery, Snowflake, Databricks and SQL Server. The text the hash reads is faithful to the value. In BigQuery, ARRAY, STRUCT, JSON and BYTES go through TO_JSON_STRING. In SQL Server and Fabric, datetime, float, money and binaries go through CONVERT with an explicit style, because CAST lost seconds and digits. A column that the source cannot write as text gets no checksum, and the data-diff compares it via the sample. Details and limitations in Methods › Sampling.

Keys, partitions and object type

Since card #347 BeryX reads what the source declares about each object. It reads only the catalogue, never data, at discovery and on every scan. A part the account cannot read stays empty and the scan goes on.

In BigQuery, Snowflake and Databricks the keys are informational: the source does not enforce them. BeryX uses them as a key, never as proof of uniqueness (the unique rule measures it).

Permissions. None new. In BigQuery the INFORMATION_SCHEMA views require bigquery.tables.get and bigquery.tables.list, already included in BigQuery Data Viewer. That is up to five queries per dataset and scan (MATERIALIZED_VIEWS only if the dataset has materialized views, CONSTRAINT_COLUMN_USAGE only if it has foreign keys), each billed at least 10 MB.

Where it shows. The catalogue shows the type next to the name (views, materialized views, external tables) and the «PK», «partition» and «clustering» marks on the columns. The API returns kind on each dataset and primary_key, partition and clustering on each column of the detail. The unique rule suggested on the first scan sits on the declared primary key; the names (id, *_id) remain the fallback. A declared foreign key becomes a declared edge in the lineage, ahead of the naming convention. The data-diff uses the base's primary key when the request has no key_columns, and the matching key suggestion puts first the primary key both sources declare with the same names.

Views and external tables. Every query to a view runs its SQL in full, and every query to an external table reads the files. By default they profile at basic. The COUNT(*) query runs over the whole object and also returns the minimum and maximum of the date and time columns: the count stays exact, because the volume monitors compare it between scans, and freshness comes from the most recent row, not from the first rows read. The other aggregates and the samples come only from the first 100,000 rows, without COUNT(DISTINCT), and above that the profile says sampled. In an insert-only table the first rows are almost always the oldest: nulls, empty text, sentinels, numeric ranges and samples of the new rows go unmeasured. When a large view moves from the full profile to the limited one, its metrics may jump once on the next scan. In BigQuery LIMIT does not reduce the bytes billed for a view, and the connection's maximum_bytes_billed cap still refuses a query that exceeds it. For the full profile, an admin sets the dataset's depth to standard or deep in Datasets → Bulk operations; basic keeps the limited profile.

Materialized views. They profile as tables. In the automatic monitors, freshness (last_modified) is the last refresh: MATERIALIZED_VIEWS.last_refresh_time in BigQuery and refreshed_on from SHOW MATERIALIZED VIEWS in Snowflake.

Adversarial profiling corpus

The cases below are not exotic; they are the day-to-day of a Portuguese or Spanish ERP. The corpus exists so that we can say "trust the numbers" with proof. It lives in backend/tests/fixtures/adversarial/ and has two parts: a generator (build_corpus.py, with each case's data and each column's declared SQL type) and the expectations, one YAML per case in cases/. The format is in the folder's README.md. The test backend/tests/test_adversarial_corpus.py generates the corpus in a temporary directory, profiles each case through the CSV connector (on the pandas and stdlib paths, which must agree) and through SQLite, and compares with the YAML. The tolerances are explicit: rel=1e-9 by default and abs=1e-6 on the epoch standard deviation. Every profile in the corpus must also save as strict JSON.

Recorded (and not fixed) limitations. Each YAML states in known_limitations what the current behaviour does not solve yet:

• PT decimals and dd/mm/aaaa dates stay text;
• 9999-12-31 and future dates go into the column's maximum and into the latest_timestamp that the connector returns (freshness no longer uses them: since card #292 the domain leaves those columns out of the choice of freshness column and raises the future_timestamps alert);
• NFC and NFD count as distinct values;
• the two 01:30s of the clock change are the same value without a time zone;
• DECIMAL(38,10) loses precision in float and in SQLite's REAL;
• SQLite stores NaN as NULL, so nan_count is only measured in files.

First scan after this version (transition). File columns (CSV, GCS, S3) whose type was wrong change once: integer to string/text for codes with leading zeros or with sentinels, and text to datetime for ISO dates in buckets. SCHEMACHG records it as type_changed. It is not suppressed, because the previous type was wrong and the issue documents the correction.

How #207 runs the same corpus against real sources. The real per-connector validation matrix (#207) reuses CASES and the YAML files without copying them:

1. Load: for each case with the SQL target, it creates the table with each column's sql_type translated into the dialect. For example, INTEGER → bigint/BIGINT/INT64; REAL → double precision/DOUBLE/FLOAT64; DECIMAL(38,10) → numeric(38,10)/DECIMAL(38,10)/BIGNUMERIC; TIMESTAMP → timestamp/DATETIME/DATETIME. Then it inserts the same rows with driver parameters, never by interpolation. Postgres 16 and MySQL 8 run in the docker-compose.connectors.yml of the matrix's docker job; BigQuery runs in the sandbox tier, with the DATAVAULT_MATRIX_BIGQUERY_* secrets.
2. Profile: runs with the real connector (postgres, mysql, bigquery), via profile_table, as in a scan.
3. Comparison: the test_adversarial_corpus.py runner already merges all with the target's section. #207 adds postgres:, mysql: and bigquery: sections to the YAML files, only with what changes per dialect: type names, the exact precision of numeric(38,10), the NaN that Postgres stores in float8, and the nan_count via native SQL, which is in #207's scope. The rest (all) must pass as it is.
4. What only a real source proves: the native STDDEV_POP in the epoch_millis case (Postgres, MySQL, BigQuery) and the information_schema fallback with CAST(:s AS CHAR) on a real MySQL. Until this version CAST(:s AS TEXT) failed on MySQL and reflection silently fell back to []. Today only the text of the query is covered by a test. Lossless reading of DECIMAL(38,10) is also still missing.

SAP

The sap connector (card #212) reads SAP S/4HANA and SAP Gateway through the OData services, v2 and v4, that SAP itself publishes. Each entity set of the service is a table. It sits on the common OData base (app/infra/connectors/odata.py), the same as the generic OData connector and Dynamics 365. The SAP-specific part is in app/infra/connectors/sap.py.

Service URL (base_url): the root of an OData service, always over https. Each connection reads one service; to read another, create another connection.

• On-premise (SAP Gateway): https://<host>/sap/opu/odata/sap/API_BUSINESS_PARTNER. The last segment is the technical name of the service.
• S/4HANA Cloud: https://<tenant>-api.s4hanacloud.ondemand.com/sap/opu/odata/sap/API_BUSINESS_PARTNER.

Authentication (auth):

• basic (the default): username and password of a communication user dedicated to BeryX.
• oauth2_client_credentials: token_url, client_id, client_secret and scope (optional). The token request goes only to the token_url, which must be https and goes through the anti-SSRF guard.
• api_key: api_key and api_key_header (optional; APIKey by default, the header of the SAP Business Accelerator Hub sandbox). It is the mode for trying things out against the sandbox.

The none mode (no credentials) is refused for sap. The password, the client_secret and the api_key are never shown back and never appear in logs.

Other fields:

• sap_client (optional): the SAP client, with three digits (e.g. 100). It goes as sap-client=100 on every request.
• entity_sets (optional): a list, or comma-separated text, of the entity sets to read (e.g. A_BusinessPartner,A_BusinessPartnerAddress). By default, all of those in $metadata, up to 200.
• sample_size (optional): rows in the profile sample, from 1 to 5000 (1000 by default).

Authorisations — on the customer's side:

1. On-premise: activate the service in Gateway (transaction /IWFND/MAINT_SERVICE) and give the communication user a role with the S_SERVICE authorisation for that service. In the back end, the role only needs display authorisations on the data the service exposes.
2. S/4HANA Cloud: create a communication arrangement for the communication scenario that exposes the service, with a communication user, and use that user's credentials in the basic mode.

Connection test: test_connection reads $metadata and asks for one row ($top=1) of the first entity. When it fails, it says what to fix:

Read-only: requests to the service are GET only, through the ReadOnlyHttpClient (anti-SSRF guard and DNS pinning). The connector never asks for the CSRF token (X-CSRF-Token: Fetch), which SAP only requires for writes. With oauth2_client_credentials, the only POST is the token request, to the token_url.

Columns, count and sample:

• Columns and types come from $metadata (EDMX), with the Edm types mapped to BeryX's.
• The row count comes from the service: $inlinecount=allpages in OData v2, $count in v4 and, if the response does not carry the count, GET <EntitySet>/$count.
• Column metrics are computed on a sample of sample_size rows ($top, with $format=json), as in Dynamics: null and distinct percentages are relative to the sample.
• v2 /Date(…)/ dates and Edm.DateTimeOffset values become UTC. Edm.Int64 and Edm.Decimal, which v2 sends as text, become numbers. This applies to the profile and to the rows that scans read (odata.py: convert_value, row_value). In the rows, Edm.Decimal stays an exact Decimal, as in the SQL connectors.
• Pagination follows __next (v2) and @odata.nextLink (v4) only when the link has the same origin (scheme, host and port) as the base_url. A link to another origin is refused.
• The connector asks for rows in blocks of 1,000 ($top). If the service serves a block in smaller pages (Northwind uses pages of 20 and of 500), the connector follows the links. When the whole block has arrived, it asks for the next one with $skip, up to sample_size (in the profile) or the scan's row limit (odata.py: _collect).

Validation on save (422): base_url missing or not https, unknown auth or none, missing authentication fields, sap_client without three digits, token_url not https, invalid api_key_header, invalid names in entity_sets and sample_size outside 1 to 5000.

Installation: no extra. The connector only uses the base dependencies (httpx).

Network: in production, the anti-SSRF guard refuses destinations in private networks. An on-premise Gateway must be reachable over https from BeryX's egress IP (see the top of this guide).

Out of scope: a direct connection to the HANA database (hdbcli). BeryX reads SAP through the OData services.

Process integration with SAP and Salesforce

"Does it integrate with our SAP?" — the short answer:

• We read SAP S/4HANA and SAP Gateway over OData, Salesforce, Dynamics 365 and the databases of ERPs that run on SQL Server — Primavera, PHC and Sage. Other OData v2 and v4 services are read through the generic OData connector.
• We never write to the ERP or the CRM, nor do we install plugins inside SAP or Salesforce: that would mean giving up read-only access. The fix comes out in SQL/PySpark or as a Pull Request, for your process to approve and apply.
• We do not validate the record at the moment it is saved (inside the SAP or Salesforce transaction): we measure what was saved, in the next scan. Validating at the point of entry, with writes, is the territory of MDM and quality suites (SAP MDG, Informatica).

OData (generic)

The odata connector (card #212) reads OData v2 and v4 services that have no connector of their own in BeryX: a company's internal OData services or public services. Each entity set is a table. It is the base of SAP (app/infra/connectors/odata.py) and behaves the same way: GET only, columns from $metadata, count from the service, profile on a sample and pagination only within the same origin as the base_url. The connection test reads $metadata and asks for one row of the first entity.

Fields:

• base_url (https): the root of the service, where <base_url>/$metadata answers (e.g. https://services.odata.org/V2/Northwind/Northwind.svc).
• auth: none by default, for public services with no credentials. The basic, oauth2_client_credentials and api_key modes take the same fields as SAP.
• sap_client, entity_sets and sample_size: as in SAP.

Limits: only the entity sets declared in the root $metadata (odata.py: parse_metadata, list_tables). A path through another entity, such as Company('…')/Customers, is left out. Business Central has not been validated.

Public services: with auth none, such as the OData.org reference services, https://services.odata.org/V2/Northwind/Northwind.svc/ (v2) and https://services.odata.org/TripPinRESTierService/ (v4). They are the odata sources in the connector matrix (public tier).

Validation on save (422): the same as SAP, except that none is accepted.

Installation: no extra (httpx).

Dedicated connectors: for the Dynamics 365 Dataverse use dynamics, and for SAP use sap.

REST/JSON API (generic)

The rest_api connector (card #269) reads REST APIs and microservices that return JSON and have no connector of their own. Each configured GET endpoint is a dataset, profiled on a sample (app/infra/connectors/rest_api.py). The connection test asks for the first page of the first endpoint.

Fields:

• base_url (https): the root of the API, with no query (e.g. https://api.example.com/v1).
• auth: none by default; bearer with token; basic with username and password; api_key with api_key in a header of its own (api_key_header, X-API-Key by default). The key never goes in the query: a secret in the URL ends up in proxy logs.
• endpoints: the list of endpoints, each a path (/users) or an object with path, name?, records_path?, params?, pagination?, total_path? and contract_path?. The name is the dataset name; without it, it comes from the path.
• records_path: where the list sits in the payload, in simple JSONPath ($.data, $.data.items[*]). Without it, the list is the payload itself or the first usual key holding objects (data, items, results, records, value, entries).
• pagination: type = none, offset (offset_param and limit_param), page (page_param, page_start 0 or 1), cursor (cursor_path in the body and cursor_param in the request) or link (Link header with rel="next", or next_path in the body); page_size up to 1000. It applies to every endpoint and each endpoint can have its own.
• total_path: where the API states the total of records (e.g. $.meta.total); without it, the row count is what the sample read.
• allowed_hosts: extra hosts that pagination links may point to (e.g. a CDN). They get the requests without the credential.
• sample_size (1000 by default, at most 5000) and max_pages (50 by default, at most 500): caps on reading each endpoint.

How it reads: GET only, through the ReadOnlyHttpClient (anti-SSRF with DNS pinning). Reading stops when a page comes back empty or short, when a cursor repeats, after max_pages pages or when the sample is full (rest_api.py: _collect). The top-level fields of each record are the columns; nested objects and lists stay as JSON text in the rows and take the object or array type in the catalog.

Host allowlist: every request, including pagination links and each hop of a redirect, only goes out to the base_url host (at the same origin) or to a host in allowed_hosts; any other destination is refused before the request (rest_api.py: _check_host). The credential only goes to the base_url origin. An allowlist host that resolves to an internal IP is still blocked by the anti-SSRF guard.

API contract: with contract (an OpenAPI 3.x or Swagger 2.0 document, in JSON or YAML), each profile compares the schema of the endpoint's GET 200 response (the contract_path, or the path itself) with the sample records, field by field, nested ones included (morada.cidade). The scan opens a schema_change issue with source contract and the downstream impact when there is drift: a required field that came in no record ("removed"), a field the contract does not declare ("new") or a type outside the declared one (app/infra/connectors/api_contract.py; app/domain/scans/schema_changes.py: record_contract_drift). An optional field missing from the sample does not count, and neither does a sample with no records. The schema of one record is reached through the path the list came from: the records_path, or the one detection picked ($.data in a {"data": […]} envelope) (rest_api.py: locate_records). Only the document's own $ref are followed: the contract never makes the connector fetch anything. A $ref cycle, as in a tree, is cut where it loops back, and each schema is read only once. Reading has caps: 64 levels of nested allOf/anyOf/oneOf and 1,000,000 steps, where every element walked counts (a property, a required name or a type entry copied, a $ref hop, a paths, content or messages key), even when several schemas share it through a $ref or a YAML anchor (api_contract.py: _SchemaReader). YAML merge keys (<<) copy at most 1,000,000 pairs (api_contract.py: _ContractLoader). On Kafka topics, the same contract takes an AsyncAPI 2.x or 3.x document and compares the payload of the channel message named after the topic (kafka.py: _contract_check).

Validation on save (422): base_url in https and with no credentials in the URL, auth mode and the mode's fields, paths that start with / with no .. or query, simple JSONPath, a known pagination type (cursor needs cursor_path), an api_key_header that is not a protocol header, allowed_hosts with host names only, integer caps and a contract readable within the reading caps, with every schema read already on save (api_contract.py: validate_contract). On a Kafka topic, a contract that is not AsyncAPI, or that goes past those caps, also gives 422.

Installation: no extra (httpx).

Limits: no OAuth2 or session cookies, and no search POST (an API that only lists through POST is left out). The Link header read is the RFC 8288 one.

Tickets (Jira, Linear, ServiceNow, Azure DevOps, GitHub, GitLab)

This is not a data source: it is the way out to the team's ticket tool. BeryX creates the ticket from an issue, incident or suggestion (the «Create ticket» button, bulk tickets and the assistant card) and keeps the link on the entity. The integration is set up in Notifications → Tickets (admin). «Test connection» validates the config as saving does and reads the provider's targets without creating anything (POST /notification-channels/tickets/connection-test). The six providers meet the same TicketProvider contract (app/infra/tickets/base.py: validate_config, create, get_status, list_targets) and are listed in app/infra/tickets/registry.py.

Writes: each provider writes to create the ticket. During sync (card #381), Jira and Linear also comment on an already linked ticket (comment, on by default) and, with transition in the integration's policy, close or reopen it, only for tickets in the integration's project (Jira) or team (Linear); none edits other fields. Two POSTs do not write: Linear reads through GraphQL queries and ServiceNow in OAuth mode exchanges the credentials for a token at POST /oauth_token.do. Two-way incident sync with ServiceNow is left to card #255.

Security: the config is encrypted with Fernet (TicketIntegration.config_encrypted) and the API never returns it. The host the customer configures (the Jira site, the ServiceNow instance, the Azure DevOps Server base_url, the GitHub Enterprise or GitLab self-managed api_url) goes through the anti-SSRF guard on save and on every request, with IP pinning and no redirects (app/infra/network/outbound.py). Errors carry an excerpt of the provider's response without the credentials, and the providers write no logs (app/infra/tickets/http.py).

Timeouts and retries: each request has 20 s. Reads (targets, status and the OAuth token) try up to 3 times on 429 and 5xx, waiting 0.5 s and 1 s, or the Retry-After up to 5 s. Creation is not repeated, because after a 5xx the ticket may already exist. A 429 on creation goes up with the Retry-After and bulk tickets retry with their own backoff (app/domain/integrations/ticket_bulk.py).

GitHub and GitLab with the git integration: instead of its own token, the config can carry git_integration_id. The token (or the GitHub App) and the api_url then come from the git integration of the same workspace and the same provider, and its repository is the default.

Interop

ODCS contracts ↔ datacontract-cli (DCSPEC)

The ODCS v3.1 contract that BeryX exports (GET /datasets/{id}/contract.yaml, domain/contracts/service.py) is verified against datacontract-cli, the reference tool of the Bitol ecosystem. The verification is optional and does not run in the normal CI:

• Local suite backend/tests/test_interop_datacontract.py (marked slow): exports the contract of a fixture dataset with every rule type, runs the CLI and re-imports the result with our importers (domain/rules/importers.py). Without the datacontract binary the tests are skipped.

  python -m venv /tmp/dc && /tmp/dc/bin/pip install -r backend/requirements-interop.txt
  cd backend && DATACONTRACT_BIN=/tmp/dc/bin/datacontract python -m pytest -m slow tests/test_interop_datacontract.py
  

• CI job .github/workflows/interop.yml (workflow_dispatch only, with the CLI version as input): installs the CLI in an isolated venv and runs the suite with DATACONTRACT_REQUIRED=1 (a missing binary fails instead of skipping). Run it before raising ODCS_API_VERSION or touching the ODCS export/import.

datacontract-cli is a DEV-only dependency (requirements-interop.txt, pinned): it brings its own versions of pydantic/sqlglot, so it stays in a separate venv and never goes into the production image.

Result (datacontract-cli 1.2.1, ODCS v3.1.0) — criterion: lossless round-trip on the fields we support.

Incompatibilities found

1. Our bug, fixed: the latency SLA exports element: <schema>.<tabela> (physicalName) and the ODCS importer read the dot as tabela.coluna — the freshness rule came back tied to a column that does not exist (encomendas). The importer now recognises name/physicalName before extracting the column (regression test in test_rule_interop.py).
2. export sodacl (CLI): invalidValues with arguments.pattern comes out as invalid_count(col) = 0 without the regex — the pattern is lost. The sql check (range) comes out as a user-defined SodaCL metric (our SodaCL importer declares it in skipped) and the latency SLA is not emitted. nullValues, duplicateValues, required and rowCount survive.
3. export dbt-models (CLI): only emits not_null (from required/nullValues); duplicateValues does not become unique. Without servers in the contract, the data_type values come out in the CLI's default dialect (NUMBER, TIMESTAMP_TZ) — for dbt, prefer our contract.dbt.yml, which maps from logicalType and emits unique.
4. Table in the conversion: the CLI's SodaCL uses the physicalName (vendas.encomendas) and dbt/SQL uses the name (encomendas). Dataset matching on import accepts both, but it is worth knowing which one appears in each artefact.
5. Logical types: we export TIMESTAMP as logicalType: date (the physicalType keeps the real type). ODCS v3.1 has timestamp; it was not changed here because compare_contract_to_schema and the dbt mapping depend on the current value — a candidate for its own card.