BeryX — Capabilities by evaluation criterion

Public document. It organises what BeryX does by the criteria analysts use to evaluate data quality tools (profiling, monitoring, parsing and normalisation, matching, cleansing, enrichment, metadata management, orchestration and experience for business users). It is the technical source for the website's /datavault/rfp page: each section says what exists, where it is in the code and which tests prove it — and, when it is partial, what is missing.

Security and compliance are in SEGURANCA.md. Coverage by quality dimension (DAMA, ISO 8000, ISO/IEC 25012) and by DAMA-DMBOK area is in COBERTURA_QUALIDADE.md.

Profiling

Automatic discovery and profile of every table in all connected sources: counts, nulls, distinct values, minimum/maximum/mean/standard deviation, cardinality, distributions and limited samples.

• More than 20 source types: warehouses (BigQuery, Snowflake, Databricks, Fabric, Azure Synapse), operational databases (PostgreSQL, MySQL, Oracle Database, SQL Server and Azure SQL — the Primavera, PHC and Sage ERPs run on the last two), MongoDB (schema inferred from a sample), business applications (SAP S/4HANA and SAP Gateway over OData, Salesforce, Dynamics 365 and other OData v2 and v4 services), files, Google Sheets (each tab is a dataset), object storage (GCS, S3, Azure Blob and ADLS Gen2), SFTP, streaming (Kafka, Pub/Sub) and document sources — list and setup in the connector guide.
• Incremental profiling: on large tables with a time column, the re-scan reads only the new window and combines it with the previous profile (with a periodic full profile).
• Documents (SharePoint, Google Drive) have their own document quality profile.

Tests: backend/tests/test_incremental_profiling.py, backend/tests/test_profile_distributions.py.

Monitoring

• Automatic monitors by default on every table in the connection (up to 500 per connection in each scan), with no configuration and from each source's metadata (in databases, only from the catalogue): freshness (learns the update cadence), volume (modified z-score over the history) and schema changes, depending on what each source exposes (method).

• Drift and anomaly detection: columns added/removed, type changes, null spikes, volume drops, per-column distribution drift (from the quantiles and categories already profiled, method), seasonal and per-segment anomalies.

• Published methodology: formulas, weights, thresholds and limitations of the score, the anomalies, the duplicates, the trust score and the sampling in METODOS.md, with a "How we calculate" section in the app.

• Rule engine with 16 rule types: nulls (not_null), uniqueness (unique), patterns (pattern), ranges (range), freshness (freshness), minimum volume (row_count_min), check-digit validity (valid_identifier, a registry of identifiers per country) — NIF/NIPC (Portuguese tax numbers), NISS, NIB and Cartão de Cidadão (Portuguese ID card); DNI, NIE and company NIF from Spain; CPF, CNPJ (including the alphanumeric one) and PIS/PASEP from Brazil; AHV/AVS and UID from Switzerland; NHS number and GB VAT from the United Kingdom; IBAN with the country's structure and length (SWIFT registry and Angola); Luhn. Without a published check digit, structure only: SSN, EIN and ITIN from the US, NINO and Companies House from the United Kingdom, NIF and BI from Angola —, reference lists (in_reference_set: the bundled ISO lists or the workspace's reference data registry, with versions, code mappings between systems and a coherence check across datasets), conditions between columns of the same row in a closed expression language — "end_date >= start_date", "total = sum(subtotal, vat)", "if country = 'PT' then valid(nif, 'nif')" — (expression, with the check digit of NIF, NISS, CPF, CNPJ, DNI, NIE, NHS number, AHV/AVS and UID verified in the source itself), custom read-only SQL — a SELECT of the violating rows, with zero or up to N expected rows — (custom_sql), referential integrity (referential: orphan keys against another dataset, by pushdown on the same connection or an in-memory anti-join between connections, proposed by DETGEN for each lineage edge), accuracy against authoritative sources (accuracy, below) and, on streams, event age (max_event_age), conformity with the Schema Registry (schema_conformity), required fields (required_fields) and unreadable messages (max_malformed_pct). Expression and SQL rules run in the source, over the whole table, and only on SQL sources; on files, APIs and streams they are "not evaluable". The customer's SQL goes through a guard — a single SELECT, no writes, only functions from a list and only tables of the connection — and runs in a transaction that is always rolled back, with read-only mode and a maximum time on the session where the engine has them (DBA.md). Custom Python checks remain out. Rules arrive through industry packs (e-commerce, financial, healthcare, SaaS), through suggestions from the profile and the workspace's acceptance history — they only go in when an administrator accepts them — or through the import of dbt (singular tests too, which become custom SQL rules), Great Expectations, SodaCL and ODCS (what it converts).

• Accuracy against authoritative sources (accuracy rule, card #266). A Portuguese tax number with the right check digit may not exist: the rule asks a source whether the value exists. VAT number in the European Commission's VIES (Member States and Northern Ireland), CP7 postcode and locality in GeoAPI.pt (Portuguese postal service data), phone number in the country's numbering plan with the phonenumbers library (E.164, type and country, no network), email by syntax and the domain's MX record (DNS over HTTPS, nothing is sent to the mailbox) and countries and currencies in the published ISO lists, downloaded with periodic refresh. It is opt-in: an admin allows each provider in PUT /api/v1/workspace/accuracy and, without that, no value leaves and the rule stays "not evaluated". Requests are GET only, through the connectors' read-only HTTP client with the anti-SSRF guard; answers are cached in the worker, per workspace, each provider has a minimum interval between requests and each scan has a lookup cap (DATAVAULT_ACCURACY_MAX_LOOKUPS_PER_SCAN). What fails opens an accuracy issue with masked examples. UK and Swiss VAT numbers stay unverified: the official services (HMRC and UID) require POST (backend/app/domain/quality/accuracy.py, backend/app/domain/scans/accuracy.py).

• Portuguese rules only with Portuguese-shaped data. What is specific to Portugal no longer applies to data with another shape (backend/app/domain/quality/pt_shape.py):

  • the CP4 list (4-digit Portuguese postcodes) requires a majority of CP7 (full Portuguese postcodes) in the sample (NNNN-NNN, also with a space or no separator). A column that only stores the CP4 ("1000") no longer gets that rule, because it cannot be told apart from a Swiss code. A Japanese postcode written without a hyphen ("1000001") still passes as Portuguese;
  • the NIF and the postcode from the e-commerce pack on vat_number, tax_id, zip or postal_code columns require the same shape in the last scan's sample. A British VAT number, an American EIN or a Swiss UID does not have the shape of the NIF;
  • in phone matching, a number without a country code matches the same number with a foreign country code ("923 456 789" with "+244 923 456 789", backend/app/domain/quality/matching.py). With no foreign country codes in the key, +351 applies to the 9 digits. With the data country PT (the default), the mirror case also matches: a Portuguese mobile "912 345 678" is linked to "+244 912 345 678" when that is the only number in the key with the same 9 digits. With another country on one side (see the data country, below), a number without a country code on that side gets that country's code (+34 in Spain) and never +351;
  • in personal data detection, a phone number from another country counts as a phone: with +, with the Brazilian DDD area code ("(11) 91234-5678") or, in a column with a phone name, by the digits alone. The name counts when it has a phone word (telefone_alt, phone_number, tlm) or when it is only celular, contato, contacto or whatsapp; canal_contacto and is_mobile do not count. Dates, decimals, the CP7 and the CEP (Brazilian postcode) never count as a phone. It no longer passes as an address, and it stays PII (backend/app/domain/compliance/pii_map.py). Without a phone name, a national number also counts when it is written with separators and grouped like a phone number of PT, Spain, Brazil, Angola, the US, Switzerland or the United Kingdom ("020 7946 0958", "612 34 56 78"), in 80% of the sample. In an identifier column (sku, order_ref) it does not count (#552).

  In NN/NN/YYYY dates, the correction only reads the month first when some value has the 2nd field above 12. Without proof of the order there is no UPDATE, only the warning.

• Data country (#513). The workspace has a country (ISO 3166-1 alpha-2, PT by default) and the connection and the dataset can override it. The scan uses the one from the lowest level that has it (backend/app/domain/catalog/data_country_service.py). It is not data residency. With a country other than PT (backend/app/domain/quality/data_country.py):

  • the semantic type no longer gives phone_pt or postal_code_pt, and DETGEN does not propose the CTT CP4 list or the Portuguese phone pattern (backend/app/domain/quality/semantic_types.py, backend/app/domain/rules/generation.py);
  • phone numbers become the phone type with the country's region, when most of the sample are numbers assigned in that country's numbering plan. The plan comes from phonenumbers, with no network (backend/app/domain/quality/phones.py, #516). A value with a decimal part never counts. Without a phone hint in the column name, only numbers written with «+», «00» or separators count, grouped like a phone number of that plan: the case number «2024/12345» does not count. A column of a numeric type is not phone (backend/app/domain/quality/semantic_types.py);
  • standardization does not propose +351 or the CP7, addresses are not read as Portuguese and there is no district and municipality enrichment (backend/app/domain/insights/standardization.py);
  • phone standardization proposes the country's E.164, for example +34 in a Spanish dataset. A number written with «+» or «00» is only converted with the country's calling code. Numbers from another country stay as they are, for manual review, and count towards the tolerance of the rule that protects the column. In Portugal the same holds for +351 (backend/app/domain/insights/phone_standardization.py);
  • in matching, each phone number becomes E.164 in the numbering plan of its side's country. «923 456 789» from an Angolan dataset matches «+244 923 456 789» and not «+351 923 456 789». Brazilian numbers with the 9th digit match «+55…», also with the trunk 0 or the carrier code. In the United Kingdom, Switzerland and the US the national format matches the international one;
  • NIF, NISS, CC and IBAN are still recognised, because the check digit proves the identifier. The PII map still catches a phone number in a column with a phone name, from the samples.

  On a SQL source, a table with a country column (pais, country…) holding ISO codes of two or more countries, one of them PT, gets for the NIF or NISS column the proposal se pais = 'PT' então válido(nif, 'nif') when the sample mixes values with and without the Portuguese check digit (at least 30% with it): the check digit is only required of the Portuguese rows, at the source.

• SLOs per dataset and lineage-based incidents (a problem upstream groups the affected datasets downstream).

• Alerts in-app, email, Slack, Teams and webhook, with on-call rotations and escalation. Each channel has an optional scope (domains, connections, datasets, issue types, automatic monitors), so that each team receives only its own, with a preview of the last 30 days.

Tests: backend/tests/test_auto_monitors.py, backend/tests/test_schema_changes.py, backend/tests/test_anomalies.py, backend/tests/test_quality_rules.py, backend/tests/test_slo.py, backend/tests/test_oncall_escalation.py, backend/tests/test_alert_routing.py.

Parsing and normalisation

Everything is a proposal. BeryX measures the profile's sample and generates the SQL in the connection's dialect and the PySpark. The customer is the one who applies it, and nothing is written to the source.

• Semantic types detected automatically per column, from the values and the name: email, URL, UUID, phone, NIF, NISS, IBAN, Cartão de Cidadão, postcode, country, currency, dates, percentages, amounts, names and addresses (backend/app/domain/quality/semantic_types.py). Types with a fixed format give a suggested pattern or check-digit rule (backend/app/domain/rules/generation.py).

• Format standardisation: phone numbers, postcodes, NIF, IBAN and dates stored in mixed formats are converted to the canonical form. Step 1 counts the rows per mask and step 2 is a proposed UPDATE. On acceptance, BeryX proposes the pattern rule that stops the old formats from coming back (backend/app/domain/insights/standardization.py).

• Decomposition of composite fields by locale rules (pt, es, en), with no external services (backend/app/domain/quality/parsing.py):

  • addresses → street, number, floor, postcode, locality and country;
  • full names → title, first name, surnames (with "de/da/del/y") and last surname, also in the "Surname, Name" form;
  • phone numbers → country code, national number, extension and E.164;
  • NIF or DNI/NIE/CIF together with the name → identifier validated by the check digit, and name;
  • dates in free text (months and days of the week in Portuguese, Spanish and English) → ISO date;
  • amounts with currency → decimal value and ISO currency code.

  The locale and the day/month order are read from the values themselves. A column with an atomic semantic type (NIF, IBAN, email, postcode, ISO date…) is never treated as composite, and without a hint in the name a phone number is only recognised with "+", "00" or separators (ATOMIC_SEMANTIC_KINDS in backend/app/domain/quality/parsing.py). For each column, the report gives the success rate in the sample, the coverage of each component and the masks of the values that do not decompose, never the values. In a test corpus with mixed formats, 30 of 30 PT/ES addresses, 30 of 30 names, 29 of 30 phone numbers and 30 of 30 dates are decomposed.

• Normalisation proposal (parse_standardize suggestion): a read-only SELECT with the derived columns and the count of recognised rows, plus the equivalent PySpark. The derived columns are only filled in the rows that the count gives as recognised and stay NULL in the others. There is never an UPDATE (backend/app/domain/insights/parsing_sql.py). A column that already has standardisation does not get the decomposition. Like any accepted suggestion, it opens a pull request in the customer's repository (cleansing and remediation).

• In the app: the "Parsing" panel in the dataset detail shows the detected components, the coverage and the formats that fail. The "Propose normalisation" button shows the suggestion's SQL and PySpark. The API is GET /api/v1/datasets/{id}/parsing.

• Detection of variants of the same value in a categorical column (upper/lower case, spaces), with a normalisation proposal in SQL and PySpark.

• ODCS data contracts with types and formats, which can be validated in CI (quality gate in CI).

Limits: the SQL only uses functions that exist in PostgreSQL, BigQuery, Snowflake, Databricks and MySQL. SQL Server/Fabric and SQLite do not have REGEXP_REPLACE and get only the PySpark. The data_iso column comes out as text and is only filled with days that exist in the calendar. In the SQL, the NIF and the DNI/NIE are recognised by their shape; the check digit is only validated in the sample report. For phone numbers, the recognised country codes are those of a fixed list (Portugal, Spain, United Kingdom, USA, France, Germany, Italy, Switzerland, Luxembourg, Belgium, Netherlands, Ireland, Brazil, Angola, Mozambique and Cape Verde), and the others are not decomposed.

Tests: backend/tests/test_parsing_composite.py (corpus and components), backend/tests/test_parsing_sql.py (SQL compiled in PostgreSQL and BigQuery, derived columns NULL outside the recognised rows, PySpark and API), backend/tests/test_postgres_only.py (the SQL run on PostgreSQL in the backend-postgres job), backend/tests/test_suggestions_standardization.py, backend/tests/test_suggestions.py, backend/tests/test_quality_rules.py.

Matching and deduplication

Everything is a proposal. BeryX scores and groups the records, the reviewer's decision is recorded in BeryX and the consolidation comes out as read-only SQL and PySpark. Nothing is written to the source.

• Detection of exact (hash) and approximate (text similarity with rapidfuzz) duplicates within a dataset, grouped into clusters that are reviewed, merged or ignored in the UI, with a consolidation proposal in SQL (deduplication ROW_NUMBER) and PySpark.
• Matching between two datasets, including from different sources (e.g. customers in the CRM and in the ERP — entity resolution): exact, normalised (email, phone, NIF, name, postcode) or approximate keys, with pairs, unmatched rows, ambiguous rows and coverage, on the "Matching" page (backend/app/domain/quality/matching.py, method). For each pair, the record that would remain (the most complete, then the most recent, then side A), with the reason and a diagnostic SELECT.
• Matching rules configurable per field, saved per pair of datasets (backend/app/domain/quality/matching_rules.py): for each field, the comparator, the normaliser, the weight and the minimum similarity. The comparators are exact, normalized, jaro_winkler, token (token_sort_ratio), numeric with a tolerance and date with a window in days (backend/app/domain/quality/matching_comparators.py). Blocking is by key, by prefix or by a phonetic code for Portuguese, and there are two thresholds: match (0.85 by default) and review (0.70). In the UI, the columns on each side are chosen from the schema of the two datasets, and fields (up to 12) and blocking rules (up to 4) can be added and removed. With no fields chosen, the rule is built from the suggested keys.
• Clusters reviewed in the UI: each row is linked to its best candidate and the links form clusters across the two datasets. A cluster is in "match" when its weakest link passes the match threshold, and in "review" when it falls between the two thresholds. In the "Matching: rules and clusters" section of the "Between datasets" view of Duplicates, each cluster shows the similarity of each field and is marked as merged or ignored. The decision is tied to the cluster's rows, not to the run: it holds in the following runs while the cluster has the same rows, and it goes into the audit trail.
• Survivorship rules per field (backend/app/domain/quality/survivorship.py): priority source, most recent (by the update column of each dataset, which the suggestion detects by name; without both, the rule is refused), most complete or most frequent, with ties broken by the order of the sources and then of the rows. The consolidated record is only proposed, and each field says which source and which row the value comes from and by which rule. It is exported as a read-only SELECT or as PySpark with no writes, and it can go on as a suggestion that, once accepted, opens a pull request in the customer's repository (cleansing and remediation).
• Metrics per rule set: candidate and scored pairs, clusters in each band, match rate, review decisions (merged, ignored, pending) and the precision observed in those decisions.
• Labelled benchmark (synthetic, Portuguese): with the rules suggested for CRM × ERP and the default thresholds, precision 1.0 and recall 0.85 on 10,000 rows, and precision 1.0 and recall 0.83 on 2,000 (method).

Rule runs happen outside the scan, on the matching queue, over a deterministic sample of up to 50,000 rows of each dataset. With personal data columns in the rules, the run asks for confirmation, as in key matching. The clusters store the scores and a hash of each row. The example values are truncated, with personal data masked, and retention deletes them.

Limits: matching runs in BeryX over that sample, with no pushdown to the source. Each rule set compares two datasets. The thresholds are not tuned automatically from the decisions: the observed precision is shown and whoever manages the rules decides whether to change them. A block with more than 200 rows on one side generates no candidates and is counted. The phonetic code is our own reduction of Portuguese spelling, not a published algorithm. In the exported SQL, the JOIN uses only the equality fields and each row is an A × B pair, with the same choice the app makes for that pair; the pairs that depend on similarity fields, and the choice among more than two records of a cluster, are in the app's clusters.

By design, we are not an MDM hub: we do not maintain a persisted master record or write the golden record to the sources (what we do not do).

Tests: backend/tests/test_duplicates.py, backend/tests/test_duplicate_constellation.py, backend/tests/test_matching.py, backend/tests/test_record_matching.py, backend/tests/test_matching_bench.py, backend/tests/test_matching_rules.py, backend/tests/test_matching_rules_api.py.

Cleansing and remediation

By design: we propose, we do not write.

BeryX proposes the fix — generated SQL and PySpark, with estimated savings and analysed risk (safe vs breaking) — and the customer applies it. We do not ask for write credentials and we never write to the source.

• Remediation as a pull request: the fix opens a PR in the customer's GitHub/GitLab repository (SQL/PySpark models and dbt tests), for review through the process that already exists.
• Governed flow: proposed → accepted → applied → verified, with an audit trail and automatic verification on the next scan.

Tests: backend/tests/test_git_remediation.py, backend/tests/test_remediation_lifecycle.py.

Enrichment

Everything is a proposal. BeryX adds to each value of a column what a reference source says about it, with its origin, and generates the SQL in the connection's dialect and the PySpark. The customer is the one who applies it, by pull request or SQL after approving the suggestion, and nothing is written to the source.

• Six enrichment types, chosen by the column's semantic type in the last profile and, for the Spanish postcode, by the column name (backend/app/domain/quality/enrichment.py):
  • Portuguese postcode (CP7) → locality, municipality and district, in GeoAPI.pt (data from CTT, the Portuguese postal service, and INE, the statistics office);
  • Spanish postcode → locality and autonomous community, in OpenStreetMap's public Nominatim, which receives only the postcode;
  • company tax number (NIF) → name and address registered in the European Commission's VIES. Only the NIF of a legal person (first digit 5, 6 or 9) is looked up. The NIF of a natural person is out of scope and does not leave;
  • address → latitude and longitude, in a Nominatim instance of the customer's own. Addresses do not go to the public Nominatim, whose usage policy asks that no personal data be sent to it;
  • country → ISO 3166-1 alpha-2 code, and currency → ISO 4217 code, from the ISO lists versioned in BeryX, with no external lookup.
• Provenance: each proposed value carries the source, the lookup date and the confidence. Each proposal gives the coverage in the sample, that is, the rows with a proposed value among the rows with a value in the column. A value the source does not know, that goes over the lookup cap or whose source did not answer gets no proposal, with the reason. In VIES, a field published as --- counts as having no value.
• Proposal (enrichment suggestion, backend/app/domain/quality/enrichment_service.py): step 1 counts, read-only, the rows to enrich, and can be rehearsed on the source (rehearsing a fix). Step 2 is a proposed UPDATE on a column of your choice, which only fills the rows where it is empty, or, with no column chosen, a SELECT that joins the proposed fields. The SQL's reference table only carries values the source confirmed: postcodes, company NIFs and ISO codes. For geocoding, the SQL carries no addresses: step 2 joins a table of coordinates that the customer loads. Like any accepted suggestion, it can go through a pull request to the customer's repository (cleansing and remediation). Scans do not delete a proposal awaiting approval: only a new proposal for the same column and type updates it. Once applied, the suggestion does not become verified on its own, because the scan does not measure enrichment: it is marked as verified by hand, on the suggestion card.
• Opt-in per provider: no value leaves before an administrator allows the provider in the workspace, under "Rules → Enrichment providers" or in PUT /api/v1/workspace/enrichment. The authorisation records who consented and when. Without authorisation, the sample is not even read. It is separate from the authorisation of the accuracy rule. What leaves for each provider is in SEGURANCA.md.
• Geocoding only in the customer's instance (nominatim_self_hosted): to allow it, the administrator gives the instance URL (nominatim_url) and confirms that the addresses may be personal data (confirm_personal_data). Changing the URL asks for that confirmation again, and withdrawing the authorisation deletes the URL. Without a configured and allowed instance, no address leaves. The URL goes through the anti-SSRF guard when it is saved and on every lookup. An instance behind authentication is not supported.
• API: GET /api/v1/datasets/{id}/enrichment lists the candidate columns, with the type and provider of each; POST /api/v1/datasets/{id}/enrichment/proposals reads a sample of that column only, queries the provider and saves the suggestion (backend/app/api/v1/enrichment.py).

Limits: the proposal runs over a sample of the column, and the values that do not appear in it are left without a proposal. Each proposal has a cap on external lookups, and the public Nominatim answers 1 request per second. For the Spanish postcode, the public Nominatim returns a single result, so a code shared by several localities gives only one of them. GeoAPI.pt and the public Nominatim are free services with no contracted service level. The public Nominatim's values come from OpenStreetMap, under the ODbL licence, which asks whoever publishes them for attribution. Beyond the list above, BeryX does not add third-party data to the records: there is no paid postal service, commercial company database or geocoding outside the customer's Nominatim instance.

Tests: backend/tests/test_enrichment.py.

Metadata management

• Automatic catalogue of all sources, with domains/data products, owner and members.

• Lineage: foreign keys, the SQL of views (at column level), the dbt manifest.json and third-party OpenLineage events — OpenLineage.

• Publishing to the customer's catalogue: quality verdicts in OpenMetadata and DataHub, as "Data Quality Metric" assets in Collibra (Import API v2) and, in Microsoft Purview, the verdict, score, scan date and BeryX link as table attributes in the Data Map (Collibra and Purview validated with a simulated transport, validation in a customer tenant pending) — external catalogue.

• Sensitivity classification per column (personal data) and README per dataset generated and approved by a human.

• Catalogue and metadata quality (governance KPIs), per workspace and per domain: % of datasets with owner, description, classification, domain, contract and rules; outdated descriptions (the schema changed afterwards), owners who are inactive or have left, datasets with no usage and no owner, and docs for columns that no longer exist. Each problem comes with the suggestion through the channel that solves it (generated README, import from the external catalogue), with targets, daily trend and CSV export (GET /catalog/metadata-quality, backend/app/domain/catalog/metadata_quality.py).

• Business glossary (Team plan or above): terms such as "Net revenue", with definition, synonyms, domain and owner, linked to the datasets and columns that implement them. Approving a term or a link is for an admin or the owner of the term's domain, and it goes into the audit log. The term inherits the quality of the approved columns: the datasets' score and the unresolved issues of each column, with the explanation ("is at 51.7 because of 2 issue(s)…"). The suggester proposes columns by name, by synonyms, by semantic type and by dbt description, without an AI key; with AI, the suggestion is marked as AI-generated. A suggestion is never created already approved. Approved terms go into the ODCS contract (businessName and tags), into datavault.yaml, into the catalogue search, into the assistant (search_glossary) and into the MCP server (glossary://terms) (backend/app/domain/catalog/glossary.py, backend/app/api/v1/glossary.py). In the app, the Glossary page searches, creates, edits and approves terms and reviews the suggestions, and the dataset's column table shows each column's terms (accepted with a solid border, proposed with a dashed one), with "+ term" and the notice of links still to review (frontend/src/pages/Glossario.tsx, frontend/src/components/DatasetTermChips.tsx).

• Owner and steward per dataset: each dataset can have its own owner and steward. Without its own owner, it inherits the owner of the first domain that has one. This rule is the same in the detail, in the dictionary, in search and in the metadata KPIs (effective_owner in backend/app/domain/catalog/metadata.py). The steward is not inherited. Whoever can write to a dataset can change its owner and steward; only an admin changes a domain's owner. The dataset owner does not grant permissions.

• Tags per dataset and per column: the workspace chooses a tag policy. In free mode it accepts any tag; in controlled mode, only tags on the list can be added (the ones the target already had stay), and only an admin changes it. Editing the tags replaces the manual ones and keeps those from the external catalogue. Changing the owner, the steward or the tags goes into the audit trail (PATCH /datasets/{id}/owners, PUT /datasets/{id}/tags, PUT /catalog/tag-policy). In the app, the dataset details page has the “Owner, steward and tags” panel, with the history and the diff between dates, and Datasets › Metadata has the tag policy.

• Exportable data dictionary per dataset, domain or workspace, in CSV, JSON, Markdown, OpenMetadata and DataHub. The dictionary carries descriptions, types, sensitivity, approved terms, tags, owner, steward, active rules and the domains' SLA. Terms are only exported in the plans with the glossary. The OpenMetadata file has one PUT /api/v1/tables (CreateTableRequest) body per table and, separately, what that request does not accept: the owner and the steward by email (the request requires the user's id in OpenMetadata), the domains, the SLA and the rules. Tags and terms go into the DataVault classification and glossary, which must exist in OpenMetadata. The DataHub file has UPSERT MCPs for datahub ingest, with the owner as BUSINESS_OWNER and the steward as DATA_STEWARD. The FQN or URN comes from the catalogue sync mapping, if the connection has one. It is a file: BeryX does not send descriptions or tags to the customer's catalogue. In the CSV, cells that start with =, +, - or @ are neutralised. Each request carries up to 2,000 datasets; with more, the response carries X-Dictionary-Truncated: true (GET /datasets/{id}/dictionary, GET /domains/{id}/dictionary, GET /catalog/dictionary, backend/app/domain/catalog/dictionary.py).

• Metadata history: every change is recorded with the what, who, when, before and after. It covers changes of owner, steward and tags, sensitivity set by hand, term-to-column links created or decided through the API and edits to a term's fields. The history returns the 500 most recent changes in the window, with X-History-Truncated: true when there are more; to go further back, pass in before the id of the oldest one returned. The diff between two dates takes into account all the changes in the window and gives, per field, the value at the start and at the end (GET /datasets/{id}/metadata/history, GET /datasets/{id}/metadata/diff, GET /catalog/metadata-history, GET /catalog/metadata-diff). It is also included in the workspace export. The history starts with this version: earlier audit and sensitivity events were not converted.

• Cross-cutting search (GET /search?q=): datasets, columns, glossary terms, tags and owners or stewards. On Postgres it combines FTS with an escaped LIKE; on SQLite it uses only the LIKE. Columns match by name and by description, also with accented words. Datasets in domains restricted for the person searching do not appear in any of the kinds (backend/app/domain/catalog/metadata_search.py).

What is missing: the owner and the tags, the dictionary, the history and the search do not yet have a page in the app and are used through the API. Left out of the history are the creation of terms, the automatically suggested links, what datavault.yaml applies and the automatic sensitivity classification. With the external catalogue connected, the glossary terms read in the last import are counted, but those terms do not go into the BeryX glossary. The import also does not write dataset tags.

Tests: backend/tests/test_column_lineage.py, backend/tests/test_dbt_manifest.py, backend/tests/test_openlineage_consume.py, backend/tests/test_catalog_sync.py, backend/tests/test_column_sensitivity.py, backend/tests/test_dataset_readme.py, backend/tests/test_metadata_quality.py, backend/tests/test_glossary.py, backend/tests/test_metadata_management.py, frontend/src/pages/Glossario.dom.test.tsx, frontend/src/components/DatasetTermChips.dom.test.tsx.

Critical data elements and regulatory dossier

• Critical data elements (CDE): a dataset or a column marked for BCBS 239, Solvency II or another regulation, with a reference ("Principle 3", "Article 19", the line of the report to the supervisor), owner and rationale. An admin or the owner of one of the dataset's domains declares it.
• Regulatory dossier, per regulation and requirement: principles 3 to 6 of BCBS 239; Article 82 of Directive 2009/138/EC and Article 19 of Delegated Regulation (EU) 2015/35, for Solvency II. For each element it gives the quality controls per required dimension, the last result of each rule, the open issues, compliance with the domain's SLA, one-hop lineage (at column level), the ODCS contract and the gaps: no rule, no result in the last 30 days, no owner, no SLA. The "CDE coverage" KPI counts the required dimensions that have a control and a result.
• It is exported in Markdown, CSV and PDF, and shared through the reports' read-only link. It is deterministic, without generative AI and with metadata only. No status claims compliance: the dossier points out gaps and does not certify (GET /api/v1/cde/dossier, backend/app/domain/compliance/regulatory_dossier.py).

What is missing: accuracy shows up as a gap in every element, because no BeryX rule type counts in that dimension today (integrity is covered with a referential rule). This is the verified-accuracy proposal in the quality coverage. The interface is also missing: the "CDE" chip on the dataset, the dossier page and the KPI in the domain dashboard. For now, the dossier is used through the API and the reports.

Tests: backend/tests/test_regulatory_dossier.py, backend/tests/test_cde.py.

Orchestration and workflow

• Orchestrators: dbt Cloud webhook (scan at the end of each job) and Airflow package with a quality gate in a DAG — orchestrators.
• Integrators: health of Fivetran and Airbyte pipelines (read-only, official APIs) — failed or late sync per destination dataset, latency and source↔destination reconciliation, and the sync in root-cause diagnosis — integrators.
• BI tools: Power BI, Tableau and Looker (read-only, official APIs) — failed or late refreshes as issues, dashboards in lineage and in issue impact with their owner, and a warning for dashboards nobody has viewed — BI tools.
• CI/CD: rules and contracts gate in the pipeline, data-diff between base and candidate — quality gate in CI.
• Agents: MCP server for Claude and other MCP clients — MCP server.
• Record validation at the point of entry: the integrator or the pipeline sends up to 5,000 records to POST /api/v1/datasets/{id}/validate before writing them and gets a verdict for each one from the dataset's active rules (not_null, pattern, range, required_fields, valid_identifier with the identifiers of the per-country registry, in_reference_set with the ISO lists and the workspace's own), the reason code of each failure and the batch verdict. Rules over the whole dataset (unique, freshness…) are marked as not applicable. Nothing is stored or sent to an LLM and the source is not read: rejecting or quarantining stays on the customer's side. dvk_* token or SDK (dv.validate.records) — record validation.
• IDE: VS Code extension with inline quality in dbt/SQL files — score, issues, contract and PII per table, per-column profile on hover, policy alerts and the branch's data-diff — VS Code extension.
• Configuration as code: deterministic export/import of the workspace configuration in YAML (datavault.yaml), to review in code review and promote between environments. The file carries the rules (rules), the domains (domains), the data policy (data_policy), the AI policy (llm_policy), the alert channels (channels), the scheduled reports (report_schedules), the business glossary with the approved links (glossary) and, per connection, the scan schedule and the activation of automatic monitors (scan_schedules, auto_monitors) and the source environment (connection_environments) — with no secrets or ids. The import first shows the plan (the diff of what changes) and applies all or nothing. The customer runs it from their own CI with the SDK (beryx config plan on the PR, beryx config apply on merge, over /api/v1/workspace/config) and a "Configuration as code" token; changes to the AI and data policies and to the connections' environment stay with an admin. The operator also runs it with backend/scripts/workspace_config.py, which runs alongside the backend (in the pipeline). Through the API, rules, domains, channels and scheduled reports are managed resource by resource at /api/v1/iac with the same token — this is what the Terraform provider uses (integrations/terraform). Monitors defined in other tools (e.g. montecarlo.yml) are not imported: rules come in through the import formats — dbt, Great Expectations, SodaCL and ODCS (what it converts) — and go out as dbt, SodaCL and GX: dbt tests, SodaCL checks or a Great Expectations suite (GET /api/v1/datasets/{id}/rules/export?format=dbt|sodacl|gx; the whole workspace in a single SodaCL file through GET /api/v1/rules/export). In SodaCL and GX, only what the import reads back as the same rule is written; what is left out (stream rules, check digits, SQL rules outside dbt, tolerances above zero) comes in the file with its reason, in the SodaCL header and in meta.beryx.unsupported in GX (what goes out).
• Issue workflow with statuses, assignees, rotations and scheduling of scans and reports.

Tests: backend/tests/test_orchestrators.py, backend/tests/test_data_diff.py, backend/tests/test_mcp_server.py, backend/tests/test_workspace_config_gitops.py, backend/tests/test_iac_api.py, backend/tests/test_ide.py, backend/tests/test_record_validation_api.py, integrations/vscode/test/.

Experience for business users

• Reports per persona (engineering, management, executive) with estimated impact in euros, in PDF, schedulable and shareable via a read-only link.
• Governance dashboard for the CDO: the programme's effectiveness in one place — coverage (owner, documentation, classification, contracts, rules), cycle time and MTTR of critical issues, compliance with SLO/SLA and deadlines, adoption of suggestions and of "Never again" rules, score per dimension, critical incidents and usage —, each KPI with a target per workspace or organisation, traffic light, monthly trend and drill-down by domain; export of the executive scorecard (CSV) and of the evidence for the procurement pack (Markdown) (GET /governance, backend/app/domain/governance/kpis.py).
• Natural-language assistant over the workspace's quality data.
• Live scan: the first scan is shown step by step; first result in minutes.
• Interface in Portuguese, English and Spanish; public quality badge per dataset.

Tests: backend/tests/test_reports_v2.py, backend/tests/test_report_pdf.py, backend/tests/test_assistant_chat.py, backend/tests/test_public_badge.py, backend/tests/test_governance.py.