Introduction

The document site for light-portal application.

Architecture

Design

Light Portal is an application that connect the providers to the consumers, and it contains many components or applications. Each component will have some API endpoints and a user interface in the portal view single page application.

To allow the users to understand each component in detail in term of design, we have collected all the design documents in this section.

Portal View

Mutliple Environment

This document outlines the necessary changes to configure portal view to work dynamically across different environments (sdx, dev, non-prod, prod) using environment-specific configuration.

1. Environment Variables Setup

Create .env File

Create environment-specific .env files in project root:

# Environment variables
# VITE_BASE_PATH is used as the base URL prefix for API calls.
VITE_BASE_PATH=/bff/admin/
# VITE_PORTAL_URL is the full absolute URL where the frontend static files are served
VITE_PORTAL_URL=https://sdx.lightapi.net/bff

Required Environment Variables

• VITE_BASE_PATH: Defines the sub-path where your application is deployed.
• VITE_PORTAL_URL: The API endpoint base URL.

Benefits of .env Configuration

• Switch environments without code changes
• Maintain a single codebase for all environments

2. Vite Configuration Changes

File: vite.config.js Location: Project root

Required Change:

import { defineConfig, loadEnv } from 'vite';
import react from '@vitejs/plugin-react';

export default defineConfig(({ mode }) => {
  const env = loadEnv(mode, process.cwd(), '');

  return {
    plugins: [react()],
    base: env.VITE_BASE_PATH || "/",
    // ... other configurations
  };
});

Why This Change is Necessary?

The Problem Without base Configuration

When your application is deployed to a sub-path rather than the domain root, all asset references break.

What base Affects

The base configuration controls how Vite prefixes:

• Static asset URLs (JavaScript, CSS, images, fonts)
• Client-side routing paths
• Public folder references

Example: Without vs With base

Without base Configuration:

• App hosted at: https://example.com/portal/
• Vite generates: <script src="/assets/index.js">
• Browser requests: https://example.com/assets/index.js
• Result: 404 Not Found ❌

With base: “/portal/”:

• App hosted at: https://example.com/portal/
• Vite generates: <script src="/portal/assets/index.js">
• Browser requests: https://example.com/portal/assets/index.js
• Result: Success ✅

3. React Router Configuration Changes

File: App.tsx Location: src/App.tsx

Required Change:

import { BrowserRouter } from 'react-router-dom';

function App() {
  const basename = import.meta.env.VITE_BASE_PATH || "/";

  return (
    <BrowserRouter basename={basename}>
      {/* Your app routes and components */}
    </BrowserRouter>
  );
}

export default App;

What basename Does

The basename prop tells React Router the base URL prefix for all routes in your application.

Routing Behavior Comparison

Why It’s Required

When your app is hosted at a sub-path (e.g., https://example.com/portal/), React Router needs to know that /portal is the deployment prefix, not part of your route definitions.

Without basename:

1. You define <Route path="/dashboard" />
2. User visits /portal/dashboard
3. React Router sees /portal/dashboard → no match → Route Not Found ❌

With basename="/portal":

1. React Router strips /portal from the URL
2. Sees /dashboard → matches your route → Success ✅

4. API Call Configuration

Current Behavior Issue

Without a configured base URL, the browser constructs API request URLs relative to the current page origin.

Example:

• App running at: https://example.com/portal/dashboard
• API call: fetch('/api/users')
• Browser sends request to: https://example.com/api/users

This may work in some cases but breaks when:

• API is hosted on a different domain/subdomain
• API has a different base path
• Cross-environment consistency is needed

Solution: Custom Fetch Wrapper

File: src/utils/fetchClient.js

const BASE_URL = import.meta.env.VITE_API_BASE_URL || "";

/**
 * Custom fetch wrapper with automatic base URL prefixing
 * @param {string} endpoint - API endpoint path (e.g., '/api/users')
 * @param {Object} options - Fetch options (method, headers, body, etc.)
 * @returns {Promise} - Response JSON
 */
async function fetchClient(endpoint, options = {}) {
  const url = `${BASE_URL}${endpoint}`;

  const defaultHeaders = {
    "Content-Type": "application/json",
  };

  const config = {
    ...options,
    headers: {
      ...defaultHeaders,
      ...options.headers,
    },
  };

  const response = await fetch(url, config);

  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }

  return response.json();
}

export default fetchClient;

Usage Example

import fetchClient from './utils/fetchClient';

// GET request
const users = await fetchClient('/api/users');

// POST request
const newUser = await fetchClient('/api/users', {
  method: 'POST',
  body: JSON.stringify({ name: 'John Doe', email: '[email protected]' }),
});

// With custom headers
const data = await fetchClient('/api/protected', {
  headers: {
    'Authorization': `Bearer ${token}`,
  },
});

Benefits

• Consistency: All API calls use the same base URL
• Environment flexibility: Different API endpoints per environment
• Maintainability: Single place to update API configuration
• Error handling: Centralized response validation

5. Build and Deployment Steps

Step 1: Change the .env variables for specific environment (sdx, dev, prod etc.)

VITE_BASE_PATH=/ (base path)
VITE_PORTAL_URL=https://example.com (endpoint URL)

Step 2: Build

npm run build

MCP Registry Design for AI Gateway

This document outlines the registration and management strategy for Model Context Protocol (MCP) tools within the AI Gateway.

Registration Strategy: The Hybrid Model

For a robust and scalable AI Gateway, the recommended approach is a Hybrid Model: Register the MCP Server as the primary entity, but manage and expose the Tools individually.

This approach balances the technical requirements of connectivity with the operational requirements of governance, security, and performance.

1. Register the Server (The “Connection” Layer)

The MCP Server should be treated as the source of truth and the primary unit of connectivity.

• Centralized Configuration: Authentication (API keys, OAuth), base URLs, transport protocols (SSE or Stdio), and environment variables are defined at the server level.
• Connectivity Management: A single server acts as a wrapper around related APIs. Registering tools individually would create significant overhead and redundant connections.
• Lifecycle & Health Monitoring: If an MCP server goes down, all its tools become unavailable. It is more efficient to monitor health and availability at the server level.
• Dynamic Discovery: The MCP protocol includes a tools/list capability. By registering the server, the gateway can automatically sync and discover new tools when the server is updated, eliminating the need for manual registration of every new function.

2. Expose Tools Individually (The “Governance” Layer)

While the gateway connects to the server, it should expose and manage tools as individual objects. This is crucial for:

• Granular Permissions (RBAC): Access control can be applied at the tool level. For example, a “Finance” team might be granted access to a get-invoice tool but restricted from a modify-ledger tool, even if both reside on the same ERP server.
• Context Window Optimization: Large Language Models (LLMs) have limited context windows. Sending all 50 tools from a large server to an LLM wastes tokens and increases the “lost in the middle” effect. The Gateway should allow for the activation of specific subsets of tools for a given AI session or agent.
• Rate Limiting & Cost Control: High-compute or high-cost tools (e.g., generate-video) can be rate-limited or billed differently compared to lightweight tools (e.g., get-weather).
• Safety & Compliance: Metadata can be attached to individual tools to flag them as Read-Only, Destructive, or Sensitive, enabling specific security flows (like “Human-in-the-loop” approvals) for risky operations.

Recommended Architecture: The “Catalog” Pattern

The implementation should follow a “Catalog” or “App Store” pattern:

1. Provider/Server Registration: An admin registers a server (e.g., “The GitHub MCP Server”) with its credentials.
2. Automated Discovery: The Gateway calls the server’s list_tools method and populates a tool catalog.
3. Governance & Activation: Admins “enable” specific tools for specific model configurations or user groups.
4. Routing Layer: When a model requests a tool, the Gateway resolves the request to the owning Server and handles the underlying communication.

Comparison of Approaches

Data Model & Schema Design

The AI Gateway leverages the existing API registry schema used by light-gateway, with specific enhancements to accommodate the unique requirements of the MCP protocol.

Conceptual Mapping

Core Tables & Enhancements

To support MCP, the following schema adjustments are implemented:

1. API Version (Server Connection)

The api_version_t table is enhanced to store transport-level configurations for stdio or SSE connections.

ALTER TABLE api_version_t ADD COLUMN transport_config TEXT;
-- JSON Example for transport_config: 
-- {"transport": "stdio", "command": "npx", "args": ["-y", "@mcp/server-google"]}

2. API Endpoint (Tool Definition)

The api_endpoint_t table acts as the tool registry. We relax the traditional HTTP method constraints and add fields for MCP tool metadata.

-- Allow 'call' as a valid operation for MCP tools
ALTER TABLE api_endpoint_t DROP CONSTRAINT api_endpoint_t_http_method_check;
ALTER TABLE api_endpoint_t ADD CHECK ( http_method IN ( 'delete', 'get', 'patch', 'post', 'put', 'call' ) );

-- Store the Tool Schema (for LLM validation) and Metadata (for safety flags)
ALTER TABLE api_endpoint_t ADD COLUMN tool_schema TEXT;   -- JSON Schema of the tool inputs
ALTER TABLE api_endpoint_t ADD COLUMN tool_metadata TEXT; -- e.g., {"destructive": true, "read_only": false}

Full Registry Schema Reference

-- API Definition (The MCP Server)
CREATE TABLE api_t (
    host_id                 UUID NOT NULL,
    api_id                  VARCHAR(16) NOT NULL,
    api_name                VARCHAR(128) NOT NULL,
    api_desc                VARCHAR(1024),
    api_status              VARCHAR(32) NOT NULL,
    active                  BOOLEAN NOT NULL DEFAULT TRUE,
    PRIMARY KEY (host_id, api_id)
);

-- API Version (The Connection/Transport)
CREATE TABLE api_version_t (
    host_id                 UUID NOT NULL,
    api_version_id          UUID NOT NULL,
    api_id                  VARCHAR(16) NOT NULL,
    api_version             VARCHAR(16) NOT NULL,
    api_type                VARCHAR(7) NOT NULL,    -- 'mcp', 'openapi', etc.
    transport_config        TEXT,                   -- MCP-specific connection data
    spec                    TEXT,                   -- Full tool manifest (optional)
    active                  BOOLEAN NOT NULL DEFAULT TRUE,
    PRIMARY KEY(host_id, api_version_id),
    FOREIGN KEY(host_id, api_id) REFERENCES api_t(host_id, api_id) ON DELETE CASCADE
);

-- API Endpoint (The Individual Tool)
CREATE TABLE api_endpoint_t (
    host_id              UUID NOT NULL,
    endpoint_id          UUID NOT NULL,
    api_version_id       UUID NOT NULL,
    endpoint             VARCHAR(1024) NOT NULL,  -- Tool Name
    http_method          VARCHAR(10),             -- 'call' for MCP
    endpoint_name        VARCHAR(128) NOT NULL,
    endpoint_desc        VARCHAR(1024),
    tool_schema          TEXT,                    -- Input parameter validation
    tool_metadata        TEXT,                    -- Safety and cost metadata
    active               BOOLEAN NOT NULL DEFAULT TRUE,
    PRIMARY KEY(host_id, endpoint_id),
    FOREIGN KEY(host_id, api_version_id) REFERENCES api_version_t(host_id, api_version_id) ON DELETE CASCADE
);

Tool Metadata & Synchronization

Populating the api_endpoint_t table involves coordinating data from the MCP Server with operational policies defined within the AI Gateway.

Sources of Metadata

The metadata for each tool is synthesized from three primary sources:

1. Standard MCP Server Response (Automated)

When the Gateway performs a tools/list call, the MCP server provides the baseline technical definition for each tool.

• Source Fields: name, description, inputSchema.
• Mapping: These are mapped directly to endpoint, endpoint_desc, and tool_schema respectively.

2. Gateway Operational Enrichment (Manual/Policy)

Since the standard MCP protocol does not include operational flags (like safety or cost), the AI Gateway manages these in the tool_metadata JSON column.

• Administrative Enrichment: Platform admins use the Gateway UI to tag specific tools. Common tags include:
  • destructive: true: Triggers a warning or confirmation flow.
  • human_approval_required: true: Places the request in a queue for manual sign-off.
  • cost_tier: "high": Used for rate-limiting or internal billing.
• Heuristic Auto-Tagging: The Gateway can automatically infer metadata based on patterns. For example, any tool starting with get_ or list_ is auto-flagged as read_only: true.

3. Protocol Extensions (Custom)

The MCP specification allows for additional properties in the tool object. If a custom MCP server includes an extra metadata or annotations block, the Gateway’s synchronization logic can be configured to capture and store these directly.

Synchronization Workflow

The following lifecycle ensures the Gateway’s registry remains accurate:

1. Connection: The Gateway establishes a connection to the server using the transport_config.
2. Discovery (Sync): The Gateway calls tools/list and performs an “upsert” for all tools found.
   • Existing tools have their tool_schema and endpoint_desc updated.
   • New tools are created with a default active status and baseline tool_metadata.
3. Review: An administrator reviews the newly discovered tools in the Gateway dashboard.
4. Governance Policy: The administrator “enables” the tool for specific roles and configures any required safety metadata (e.g., flagging the drop_table tool as destructive).
5. LLM Execution: When a model calls the tool, the Gateway uses the stored tool_schema for pre-flight validation and the tool_metadata to enforce security policies.

Too Many Pages/Forms

The portal has accumulated many pages, generated forms, custom admin screens, and feature-specific entry points. The sidebar can expose these pages, but it does not help a user understand which pages are required to finish a real business task. The MCP Gateway quick start wizard is a useful experiment, but it also shows the limitation of a rigid linear wizard: real tasks have optional steps, pre-existing data, and multiple valid starting points.

This document proposes a task-oriented navigation layer for portal-view.

Problem

Users currently need to know the portal information architecture before they can complete a task. For example, onboarding an API to MCP Gateway may require some combination of:

• create or select an API
• create or select an API version
• link the API version to a gateway or sidecar instance
• select MCP tools
• configure access control
• revisit instance, API, or role administration later

The same pattern exists across other areas. A task is not a single route; it is a sequence of related pages and forms. The current navigation model makes users pick pages first, then infer the task process themselves.

Current MCP Wizard Observation

The MCP Gateway wizard already has useful building blocks:

• flowConfig.tsx keeps step metadata in one place.
• McpServerForm.tsx renders a generic wizard shell.
• useMcpPrefill.ts can resume from URL context such as apiId, apiVersionId, and instanceApiId.
• Several steps are marked skippable.

However, the wizard is still too rigid:

• Step order is linear even when the task is naturally conditional.
• Initial step selection relies on hard-coded step numbers.
• Optional work is represented as skip buttons instead of task state.
• The wizard duplicates or wraps existing forms instead of treating existing pages/forms as first-class task steps.
• The solution is specific to MCP Gateway and does not help users navigate the rest of the portal.

Design Goals

• Let users start from a task, not a page name.
• Keep existing pages and generated forms as the source of truth.
• Support multiple entry points into the same task.
• Detect what has already been completed and show only relevant next actions.
• Support optional, required, blocked, complete, and skipped steps.
• Preserve role-based visibility and host-specific context.
• Allow users to leave a task, return later, and continue from context.
• Make the approach reusable for MCP, API publishing, access control, deployment, config promotion, migration, and admin workflows.

Non-Goals

• Do not replace every admin page with a wizard.
• Do not create a separate custom form for each task if an existing generated form already works.
• Do not use the sidebar as the only navigation surface.
• Do not force a strict step sequence when the data model allows safe jumping.

Proposed Solution

Add a task-oriented navigation layer above the current pages/forms.

The main pieces are:

1. Task Center
2. Task Registry
3. Task Progress Resolver
4. Task Navigation Shell
5. Global Search and Command Palette
6. Contextual Next Actions

Task Center

The Task Center is a page where users choose what they want to accomplish. It should group work by intent, not by implementation table.

Example task groups:

• API Marketplace
  • Register a new API
  • Add an API version
  • Publish an API
  • Review API details
• MCP Gateway
  • Onboard an existing API to MCP Gateway
  • Register a standalone MCP server
  • Configure MCP tools
  • Configure MCP access control
• Access Control
  • Create role
  • Assign permissions
  • Configure endpoint access
• Platform Operations
  • Register controller/gateway instance
  • Link API version to instance
  • Promote configuration
• Portal Administration
  • Manage host users
  • Export/import portal data
  • Convert migration snapshot

Each task card should show:

• title
• short description
• required role
• common starting object, such as API, instance, host, or client
• progress status when the current context is known
• primary action such as Start, Continue, Review, or Fix Missing Step

Task Registry

Introduce a registry that describes tasks and steps declaratively. This is the generalized version of the current MCP flowConfig.tsx, but it should route to existing pages/forms instead of rendering every step inside one wizard.

Example TypeScript shape:

export type TaskDefinition = {
  id: string;
  title: string;
  description: string;
  category: string;
  roles?: string[];
  keywords: string[];
  entryPoints: TaskEntryPoint[];
  steps: TaskStep[];
};

export type TaskStep = {
  id: string;
  title: string;
  description?: string;
  required: boolean;
  dependsOn?: string[];
  route: (ctx: TaskContext) => string;
  formId?: string;
  completeWhen?: TaskCompletionCheck;
  visibleWhen?: TaskVisibilityCheck;
  blockedWhen?: TaskBlockedCheck;
};

The task registry should live close to portal navigation code, for example:

src/tasks/taskRegistry.ts
src/tasks/taskTypes.ts
src/tasks/resolvers/
src/pages/tasks/TaskCenter.tsx
src/pages/tasks/TaskDetail.tsx

Page And Form Metadata

To make search and tasks work well, pages and generated forms need metadata.

For generated forms, the metadata can come from Forms.json plus a small registry override when the form title is not enough.

For custom pages, add a route/page registry:

export type PageDefinition = {
  route: string;
  title: string;
  description?: string;
  category: string;
  roles?: string[];
  keywords: string[];
  entities?: string[];
};

This registry can feed:

• sidebar sections
• Task Center
• command palette
• page breadcrumbs
• contextual next actions

The important rule is that page/form metadata should be reused, not copied into each wizard.

Task Progress Resolver

A task should not blindly ask users to complete steps that are already done. Each task can have a resolver that checks the current host and entity context.

For MCP Gateway, the resolver can check:

• API exists
• API version exists
• instance API link exists
• MCP tool configuration exists
• access control exists

The UI then marks each step:

• Complete
• Required
• Optional
• Blocked
• Skipped
• Needs review

The resolver should use existing query endpoints where possible. The first implementation can query on page load. Later, it can cache per task/session.

Task Navigation Shell

Instead of a full-screen wizard that owns all steps, use a task shell that can wrap or accompany existing pages.

Recommended behavior:

• A task detail page shows the checklist and current state.
• Selecting a step navigates to the existing page/form with task context in the URL or router state.
• The target page shows a compact “Task” panel or return link.
• After save, the user can return to the checklist or continue to the next recommended step.

Example URL:

/app/form/createService?task=mcp-onboard-api&returnTo=/app/tasks/mcp-onboard-api

This keeps existing page behavior intact while adding guided navigation.

Global Search And Command Palette

The portal should have a global launcher. It should search tasks, pages, forms, and entities.

Examples:

• “onboard mcp”
• “create api”
• “auth client”
• “relation type”
• “instance api”
• “export snapshot”

Search results should be role-aware and host-aware.

Result types:

• Task
• Page
• Form
• Entity
• Recent item

This is the fastest way to help expert users without forcing them through a wizard.

Contextual Next Actions

Detail pages should expose next actions based on the current entity.

Examples:

• API detail
  • Add version
  • Link version to gateway
  • Configure MCP tools
  • Configure access control
• Instance detail
  • Link API version
  • Configure MCP tools
  • View gateway servers
• Auth client detail
  • Assign owner
  • Review sessions
  • Review audit
• Snapshot export
  • Convert snapshot
  • Import snapshot

These actions should come from the same task registry, not from one-off buttons hard-coded on every page.

MCP Gateway Example

The MCP Gateway quick start can be rebuilt as a task:

Task: Onboard API to MCP Gateway

Steps:
1. Select or create API
2. Select or create API version
3. Choose deployment mode
4. Link API version to gateway or sidecar instance
5. Select MCP tools
6. Configure access control

Step behavior:

• API selection is required unless apiId is already provided.
• API version is required unless apiVersionId is already provided.
• Spec upload is optional and only shown when creating a new API/version.
• Deployment mode is required when the version is not linked.
• Gateway selection is required only for centralized deployment.
• Tool selection is optional if users only want to register the server first.
• Access control is optional but should be shown as a recommended final step.

This task can support several entry points:

/app/tasks/mcp-onboard-api
/app/tasks/mcp-onboard-api?apiId=...
/app/tasks/mcp-onboard-api?apiId=...&apiVersionId=...
/app/tasks/mcp-onboard-api?instanceApiId=...

The UI should not rely on fixed step numbers. It should compute visible steps from the task context and completion state.

Task State

Start with client-side state:

• URL query parameters for entity context
• sessionStorage for in-progress task context
• existing backend records for real completion state

Later, add persisted task state if needed:

• user id
• host id
• task id
• context JSON
• skipped step ids
• last active step
• updated timestamp

Persisting task state should not become the source of truth for business data. It should only remember navigation state and user choices. Completion should be derived from actual portal records.

Sidebar Role

The sidebar should become smaller and more stable. It should expose major areas, not every page/form.

Recommended sidebar sections:

• Home
• Tasks
• Marketplace
• MCP Gateway
• Operations
• Administration

Deep links should still exist, but they should be discoverable through search, contextual actions, and task detail pages.

Implementation Plan

Phase 1: Inventory And Metadata

• Create page/form metadata registry.
• Add task registry types.
• Register the most-used pages and forms.
• Add global search over registered tasks/pages/forms.

Phase 2: Task Center

• Add /app/tasks.
• Add task category cards.
• Add task detail checklist page.
• Implement client-side task context with URL parameters and session storage.

Phase 3: MCP Gateway Task

• Convert the current MCP wizard flow into mcp-onboard-api task definition.
• Reuse existing MCP components for the pages that still need custom UI.
• Replace hard-coded step numbers with resolver-driven visible steps.
• Add return-to-task behavior after saves.

Phase 4: Contextual Actions

• Add task actions to API detail and instance detail pages.
• Add task actions to access control and config pages where appropriate.
• Use the task registry to drive action visibility.

Phase 5: Broader Rollout

• Add tasks for API publishing, config promotion, host/user management, and snapshot export/import.
• Reduce sidebar clutter once task/search usage is available.
• Add persisted task state only if session storage is not enough.

Risks And Mitigations

Recommendation

Keep the MCP Gateway wizard as a prototype, but do not build more isolated wizards in the same style. The long-term solution should be:

• a task registry
• a Task Center
• resolver-driven progress
• global search
• contextual next actions
• reuse of existing pages and generated forms

This gives new users guided paths while still letting experienced users jump directly to the page or form they already know.

User Filter

As more portal users manage their own APIs, clients, instances, schedules, and configuration records, giving every operator a broad admin role becomes too coarse. A broad admin can see and modify records created by other admins on the same host. This document proposes an incremental owner-scoped filtering model for portal-view.

The first step is a UI-side filter based on the user recorded on each row, such as update_user. This is not a complete security boundary. The same rule must eventually be enforced in the query and command services with fine-grained authorization from the rule engine. The UI implementation is still useful because it improves day-to-day user experience and gives us a concrete policy shape to move into the service layer.

Problem

Portal admin pages were originally designed for a small set of trusted operators. Many tables expose all host-scoped records once the user can access the admin page.

That model creates problems as adoption grows:

• application owners need to manage their own APIs, clients, and instances
• broad admin roles expose unrelated records from other teams
• users can accidentally edit or delete records owned by another user or team
• creating one role per page, such as api-admin or instance-admin, still does not solve row ownership
• service-layer fine-grained authorization is not available everywhere yet

The immediate need is to let users use admin-like pages while limiting the rows they see and act on.

Current Experiment

Schedule.tsx is the first experimental page. The idea is:

• users can access the schedule admin surface
• normal users only see schedules where updateUser matches their user id
• global admins or schedule admins can still see all schedules
• the updateUser column can be hidden for normal users
• create/update/delete actions are available only on the visible set

One implementation detail matters: ownership filters must be added before the request payload serializes the filters array.

const apiFilters = [];

if (ownedOnly && userId) {
  apiFilters.push({ id: "updateUser", value: userId });
}

const cmdData = {
  filters: JSON.stringify(apiFilters),
};

Adding the filter after cmdData.filters is built will not send it to the backend.

Design Goals

• Allow regular users to manage records they created or updated.
• Avoid giving every self-service user broad all-record admin visibility.
• Keep the admin table implementation familiar and incremental.
• Centralize the owner filter logic instead of duplicating it page by page.
• Make the UI rule match the future service-layer rule as closely as possible.
• Preserve host scoping and existing role-based page visibility.
• Avoid presenting UI-side filtering as a security boundary.

Non-Goals

• Do not claim UI filtering is sufficient authorization.
• Do not replace service-layer rule-engine enforcement.
• Do not solve full team ownership in the first UI-only pass.
• Do not migrate every admin page in one large change.
• Do not overload update_user as the permanent ownership model if a better owner field exists or can be added.

Ownership Model

There are several possible ownership signals. They should be treated in this order of preference.

update_user is acceptable for the first UI experiment because many tables already have it. However, it has an important semantic problem: ownership moves to whoever last updated the row. If Alice creates an API and Bob updates it, Bob becomes the owner under an update_user rule.

The long-term model should add explicit owner fields where needed:

owner_user_id
owner_position_id

owner_group_id is intentionally deferred. Groups are still useful for flat team membership, but position ownership fits the portal authorization model better when access should follow the organization hierarchy. owner_org_id is also deferred because normal portal records are already scoped by host_id, and host_t links back to org_t through the host domain. Add organization-level ownership only if a future cross-host/global ownership use case requires it.

Do not add created_by and updated_by as authorization fields in Phase 4. The existing update_user and update_ts columns remain the last-updater audit trail. If creator audit becomes important, add create_user and create_ts as audit fields later, not as substitutes for stable ownership.

Until explicit owner columns exist, each page should declare which field is used for interim UI owner filtering.

Role Model

Use one page per entity type, but separate page visibility from row scope.

Do not give every user account access to every admin page. Only pages that are safe for self-service ownership should be exposed to user, and each of those pages must apply the owner filter and action guards.

The admin role can be repurposed as the global all-record role once the sidebar stops using it as a broad menu marker. Role checks must use exact role tokens. A role such as schedule-admin must not match admin through substring checks.

Access Modes

The UI should support three access modes.

Owner-Scoped Admin

This is the default self-service mode. The user can open admin pages, but rows are filtered to records they own.

Example:

roles: user
scope: owned
filter: updateUser = current user id

All-Scope Admin

This is for operators who can see and manage every record on the current host.

Example roles:

admin
schedule-admin

The default all-scope role is admin. Page-specific roles such as schedule-admin can opt a user into all-record visibility for one area. Do not use platform-admin as a global all-scope role because the portal already has a Platform Admin page for deployment platform management.

Read-Only or Support View

Some users may need to see records without modifying them. This can be added later with separate flags:

canReadAll = true
canWriteOwned = true
canWriteAll = false

Proposed UI Architecture

Add a small ownership-scope helper used by admin pages.

Example shape:

type OwnershipScopeOptions = {
  roles?: string | null;
  userId?: string | null;
  ownerField: string;
  allScopeRoles?: string[];
};

type OwnershipScope = {
  ownedOnly: boolean;
  ownerFilter: { id: string; value: string } | null;
  canWriteAll: boolean;
};

Example usage:

import {
  applyOwnershipFilter,
  defaultAllScopeRoles,
  ownershipScope,
} from "../utils/ownershipScope";

const ownership = ownershipScope({
  roles,
  userId,
  ownerField: "updateUser",
  allScopeRoles: [...defaultAllScopeRoles, "schedule-admin"],
});

const apiFilters = applyOwnershipFilter(columnFiltersWithoutActive, ownership);

This helper should live near other portal navigation/task utilities or in a small access utility module, for example:

src/utils/ownershipScope.ts

or:

src/tasks/accessScope.ts

The helper should not call the backend. It only computes the UI filter and UI capabilities from the current user state.

Sidebar Behavior

The sidebar should not use admin as a marker on every admin menu link. That made the whole Administration group disappear for normal users and prevented owner-scoped self-service pages from being reachable.

Recommended behavior:

• admin users see every Administration link.
• non-admin users see only Administration links explicitly marked with user or a matching entity role, such as role: "user schedule-admin".
• only add user to a link after that page has owner-scoped filtering and action guards.
• remove role: "admin" from individual menu links.
• use exact role-token matching instead of string includes, so schedule-admin does not accidentally grant admin.

At the Phase 3 rollout point, the following Administration links are safe to expose to user because the pages apply the shared owner-scope helper and action guards:

• API Admin
• API Detail
• OAuth Auth Client and Client Token
• App Admin
• Instance Admin, Runtime Instance, and instance relationship pages
• Schedule Admin
• Workflow Definition

Configuration, platform admin, user/role admin, workflow process/task/audit pages, and lower-volume metadata pages should remain admin-only until they have the same owner-scope treatment or a separate support/read-only policy.

Admin Page Behavior

For an owner-scoped user:

• add the owner filter before the query payload is serialized
• hide the owner column if it does not add useful information
• show a small scope label such as “My records”
• keep create actions available
• allow update/delete only for rows matching the ownership rule
• preserve normal table sorting, pagination, and global filter behavior

For an all-scope admin:

• do not add the owner filter
• show a scope label such as “All host records”
• show the owner/update columns
• allow existing admin actions

For a user without enough context:

• if userId is missing, do not run an owner-scoped query
• show a clear message that user context is required
• avoid falling back to all-record visibility

Action-Level Guard

List filtering is not enough for a good UI. Row actions should also check the same scope.

Example:

const canUpdateRow =
  ownership.canWriteAll ||
  row.original.updateUser === userId;

For rows the user cannot modify:

• hide destructive actions, or
• disable them with a tooltip explaining the scope

Even after service-layer authorization is implemented, the UI should keep these guards so users understand why an action is unavailable.

Phase 4 Ownership Columns

For high-value entity tables, add canonical owner columns directly on the entity row:

owner_user_id UUID NULL
owner_position_id VARCHAR(128) NULL

Recommended constraints where the table has host_id:

FOREIGN KEY (host_id, owner_user_id)
  REFERENCES user_host_t(host_id, user_id)

FOREIGN KEY (host_id, owner_position_id)
  REFERENCES position_t(host_id, position_id)

Both owner columns should be nullable during migration. New records should get owner_user_id from the authenticated user on the service side by default. Do not trust a browser-submitted owner user id unless the caller has permission to assign ownership.

owner_position_id should be optional on create. The UI can show a host position dropdown populated from the user’s allowed positions. If the user has exactly one effective position and the page is configured for position ownership, the UI can default to that position. If the user has multiple positions, require an explicit choice when position ownership is desired.

For portal forms, the optional position owner field should be exposed as ownerPositionId and backed by the existing position label dynaselect query. The form action uses the position/getPositionLabel endpoint, which is backed by the queryPositionLabel persistence method and returns the id/label pairs needed by the select control.

Do not expose ownerUserId as a normal create/update form field. The command path must derive owner_user_id from the authenticated user in the event context. If an owner-transfer use case is needed later, implement it as a separate command with explicit authorization and audit behavior.

Normal update forms may update owner_position_id when the page allows the caller to choose or clear the owning position. update_user changes on every update and remains audit metadata. owner_user_id should not change on normal update; it changes only through an explicit owner-transfer action restricted to the current owner, admin, or the relevant entity-admin role.

Existing rows should be migrated conservatively:

• if update_user can be resolved to a user in the host, it can be used as an initial owner_user_id
• leave owner_position_id null unless there is a reliable source for the owning position
• rows with no owner columns populated should be treated as unassigned legacy rows, visible only to all-scope admins until an owner is assigned

Service-Layer Target

The UI filter is an interim step. The durable solution belongs in the query and command services.

The service layer should eventually:

• derive user id, roles, host id, and scopes from JWT claims
• ignore client-supplied owner filters as an authorization source
• inject owner predicates into query handlers based on the authenticated user
• reject update/delete commands when the user does not own the row and lacks all-scope permission
• use rule-engine policies for exceptions and domain-specific ownership

Once service-side owner enforcement is implemented, the UI should no longer be the source of authorization predicates. The service should inject the ownership predicate from authenticated user context and rule-engine decisions.

The UI should still keep owner-aware behavior for usability:

• show “My records” or “Admin View” scope labels
• hide or show owner columns based on the user’s scope
• disable update/delete actions that the current user cannot take
• optionally send a simple view hint such as scope=owned or scope=all

The service must treat any UI-supplied scope or owner filter as a hint only. It must ignore, override, or reject filters that would expand the caller’s authorized scope.

For owner-scoped users, the service-side predicate should be an OR condition:

owner_user_id = current_user_id
OR owner_position_id IN current_user_effective_positions

For all-scope admins, such as admin or the relevant entity-admin role, the service should omit this owner predicate and return all rows within the normal host scope.

The UI and backend should share the same policy concepts:

host scope
entity type
owner field
owned-only permission
all-record permission
read vs write capability

Position hierarchy must be resolved by the service layer or rule engine. A JWT claim such as pos=ai-engineer only grants exact-position access unless the service expands it to effective positions from position_t and user_position_t. If hierarchy is enabled, the effective position set should include inherited positions according to the existing position inheritance rules.

Rows with owner_position_id IS NULL are not position-owned. A user can still see the row if owner_user_id matches their user id. Rows where both owner_user_id and owner_position_id are null are unassigned legacy rows and should not be visible to normal owner-scoped users by default.

Rule Engine Direction

The rule engine can express policies such as:

user can read API when api.owner_user_id == user.user_id
user can update API when api.owner_user_id == user.user_id
admin can read all APIs on host
admin can update all APIs on host
api-admin can read all APIs on host
api-admin can update all APIs on host
support can read all APIs but cannot update

For tables that do not yet have explicit ownership fields, the policy can temporarily map ownership to update_user.

Rollout Plan

Phase 1: Fix Schedule Experiment

• Fix filter ordering so updateUser is included in the request.
• Use roles plus user id to decide owner-scoped vs all-scope mode.
• Add action-level guards for update/delete.
• Keep the current route behavior unchanged.

Phase 2: Add Reusable UI Helper

• Create a shared ownership-scope helper.
• Add unit-level coverage if the repo has a practical test pattern.
• Document default all-scope roles.
• Keep owner field configurable per page.

Phase 3: Apply To High-Value Admin Pages

Start with pages where users commonly manage their own records:

• API admin
• API detail/version admin
• OAuth clients
• client apps
• instances
• instance API links
• schedules
• workflow definitions

Then expand to lower-volume metadata pages.

Current implementation status:

• src/utils/ownershipScope.ts centralizes exact role matching, owner-scope calculation, owner filter injection, and owner-column hiding.
• Sidebar access now exposes only scoped links to user or matching entity-admin roles, while exact admin continues to see all Administration links.
• API pages use admin and api-admin for all-record scope, with user limited by updateUser.
• OAuth client pages use admin and oauth-client-admin for all-record scope, with user limited by updateUser.
• Client app pages use admin and app-admin for all-record scope, with user limited by updateUser.
• Instance pages use admin and instance-admin for all-record scope, with user limited by updateUser.
• Schedule pages use admin and schedule-admin for all-record scope, with user limited by updateUser.
• Workflow Definition uses admin and workflow-admin for all-record scope, with user limited by updateUser.
• Task/page search registries use exact role-token checks so schedule-admin or another entity-admin role does not accidentally match global admin, while exact admin still has global visibility.

Deferred from this phase:

• Workflow Process, Task, Worklist, Work, Audit, and Trace remain admin-only until their ownership rules are defined and implemented.
• Configuration and platform pages remain admin-only because their ownership model is not yet defined.
• User and role administration remain admin-only because exposing them to self-service users would require a separate delegated-administration model.

Phase 4: Add Explicit Ownership Fields

Where update_user is too weak, add proper owner fields through the database and services.

Candidate fields:

owner_user_id
owner_position_id

Apply these first to the high-value tables that already have owner-scoped admin pages. Keep the fields nullable during migration, default owner_user_id from the authenticated user on create, and make owner transfer explicit.

Current implementation status:

• portal-db adds nullable owner_user_id and owner_position_id columns to the high-value portal tables used by the owner-scoped admin pages.
• The migration backfills owner_user_id from update_user only when update_user is already a UUID. Non-UUID audit values remain unassigned instead of blocking the migration.
• A database insert trigger defaults owner_user_id from update_user for new rows when the command path writes the authenticated user id into update_user.
• Query projections for the scoped UI pages now return ownerUserId and ownerPositionId, and UUID filtering recognizes ownerUserId.
• portal-view now uses ownerUserId for ownership checks on action controls. The UI no longer sends an owner filter for service-enforced pages because service-side scope must include both direct user ownership and position ownership.
• Owner-aware create/update forms expose optional ownerPositionId with a host-scoped position dynaselect backed by queryPositionLabel.
• Command schemas allow optional ownerPositionId for the owner-aware create and update commands. They do not accept ownerUserId; owner_user_id comes from the authenticated event user.
• light-portal persistence writes owner_user_id from the event user on create and writes owner_position_id from ownerPositionId on create/update.
• Schedule query is the first service-enforced owner-scope path. Non all-scope users are filtered by owner_user_id = current_user_id OR owner_position_id IN effective positions based on authenticated audit context.

Remaining rollout work:

• Add explicit owner-transfer commands instead of changing ownership through normal update forms.

Phase 5: Enforce In Services

• Add query-side owner predicates.
• Add command-side ownership checks.
• Move policy decisions into rule-engine configuration.
• Keep the UI filters as usability hints, not authorization.

Current implementation status:

• Query-side owner predicates are implemented for Schedule, API, API Version, App, OAuth Client, Client Token, Instance, Instance API, Instance API Path Prefix, Instance App, Instance App API, Runtime Instance, and Workflow Definition.
• Query handlers derive scope from the authenticated audit attachment. Users with the global admin role or the entity-specific all-scope role bypass the owner predicate; other users are scoped by user id or effective positions.
• The UI keeps owner-aware action guards, but it does not send the owner filter as a request filter for service-enforced pages. That keeps position-owned rows visible when the service grants access by owner_position_id.
• The db-provider keeps backward-compatible query methods and adds owner-aware overloads so query services can roll forward independently.

Remaining service rollout work:

• Add command-side ownership checks before update/delete actions.
• Add explicit owner-transfer commands and audit events.
• Move the all-scope role and position hierarchy decisions from Java guards into rule-engine policy once the service-side rule context is ready.

Future Improvement: Entity Access Grants

Do not introduce a generic ownership table in Phase 4. It adds query joins, pagination complexity, and weaker referential integrity before we have a clear sharing use case.

A generic table can be added later for secondary grants, sharing, and delegated administration. It should supplement the canonical owner columns rather than replace them.

Possible future shape:

entity_access_t
  host_id
  entity_type
  entity_id
  principal_type   -- user, position, group, role
  principal_id
  access_level     -- owner, maintainer, viewer

Use this only when we need use cases such as:

• share one API with another position or group
• give support read-only access to a selected set of records
• delegate maintenance without transferring the canonical owner
• manage record-specific exceptions from an Access Admin page

Risks And Mitigations

Recommendation

Use owner-scoped filtering as the first UI step, but centralize it immediately. Do not copy the Schedule.tsx logic into every page by hand.

The recommended path is:

1. fix the schedule filter ordering
2. introduce a reusable ownership-scope helper
3. apply it to the most common self-service admin pages
4. add explicit owner fields where update_user is not good enough
5. enforce the same rules in query and command services through the rule engine

This gives users a safer admin experience now while creating a clear migration path to real fine-grained authorization.

Contextual Help Links

portal-view has many pages, generated forms, task flows, and admin tables. Even with the task-oriented navigation work, users still need page-specific and form-specific help when they are making a decision or filling a field. This document proposes a contextual help-link model for pages and forms.

Problem

Users often need help at the exact point where they are working:

• what this page is for
• when to use this form
• what required fields mean
• which optional fields matter
• what permissions or ownership rules apply
• what happens after submit
• how this page fits into a larger task

Today, help is usually outside the UI context. Users must know where to look, which document applies, and which page or form name maps to the screen in front of them.

Design Goals

• Add a clear help entry point to every major page and generated form.
• Keep help content close to the product documentation source of truth.
• Avoid bloating the portal-view application bundle with documentation.
• Allow documentation-only updates without rebuilding portal-view.
• Make help links declarative so page, form, and task metadata can drive them.
• Keep link identifiers stable even if routes or component names change.
• Support future documentation search, related topics, and task-specific help.
• Preserve the ability to run the app locally with a configurable docs base URL.

Non-Goals

• Do not build a full documentation authoring system inside portal-view.
• Do not duplicate long user guides in component source files.
• Do not block a page or form rollout because full documentation is missing.
• Do not use contextual help as a replacement for better labels, validation, or field-level error messages.

Documentation Location Decision

The help content should live in light-portal-doc. portal-view should store only metadata that points to the relevant help page.

Recommended split:

light-portal-doc
  src/help/portal-view/
    pages/
    forms/
    tasks/
    concepts/

portal-view
  page registry, task registry, and form metadata with help ids or help paths

Why light-portal-doc

Pros:

• Keeps user-facing documentation in the documentation repo.
• Allows documentation changes without rebuilding or redeploying portal-view.
• Avoids increasing the app bundle with markdown content.
• Supports documentation search, navigation, publishing, and review workflows.
• Allows the same help content to be linked from support tickets, onboarding, release notes, and external docs.
• Fits the existing pattern where portal-view design docs already live in light-portal-doc.

Cons:

• Requires stable published URLs.
• Requires a configurable docs base URL for local and deployed environments.
• Can drift from UI behavior unless we add link validation and ownership rules.

Why Not portal-view/docs

Pros:

• Easy to review UI and docs in one PR.
• Help content can be tightly coupled to the component version.
• Local development does not need a separate docs deployment.

Cons:

• Documentation-only changes require app rebuilds and deployments.
• Large markdown content can bloat the frontend repo and build context.
• It is harder to provide a proper documentation navigation/search experience.
• It encourages implementation notes and user help to mix in the same repo.

Recommendation: use light-portal-doc for content and keep portal-view limited to stable link metadata.

Help Content Structure

Create a user-facing help tree separate from design docs:

src/help/portal-view/
  pages/
    api-admin.md
    api-detail.md
    instance-admin.md
    schedule-admin.md
  forms/
    create-api.md
    update-api.md
    create-client.md
    update-instance.md
  tasks/
    mcp-onboard-api.md
    register-standalone-mcp-server.md
  concepts/
    ownership-and-positions.md
    hosts-and-user-hosts.md
    api-versioning.md

Use page-level help for screen orientation and form-level help for submission semantics. Use concept help for reusable explanations that should not be copied into many page/form documents.

URL Strategy

Help URLs should be stable and human-readable.

Recommended public URL shape:

/help/portal-view/pages/api-admin
/help/portal-view/forms/create-api
/help/portal-view/tasks/mcp-onboard-api
/help/portal-view/concepts/ownership-and-positions

Do not make the public URL depend on React route internals or component names. If a route changes from /app/api to another route later, the help URL should not need to change.

portal-view should build the absolute link from a runtime config value:

PORTAL_DOC_BASE_URL=https://doc.lightapi.net

or for Vite:

VITE_PORTAL_DOC_BASE_URL=https://doc.lightapi.net

Local development can point to a local docs server:

VITE_PORTAL_DOC_BASE_URL=http://localhost:3000

Metadata Contract

Use a stable help id or help path in the app metadata. A help path is more direct and easier to validate.

Page registry example:

{
  id: "api-admin",
  title: "API Admin",
  route: "/app/apis",
  helpPath: "/help/portal-view/pages/api-admin",
}

Task registry example:

{
  id: "mcp-onboard-api",
  title: "Onboard API to MCP Gateway",
  helpPath: "/help/portal-view/tasks/mcp-onboard-api",
}

Form metadata example:

{
  "formId": "createApi",
  "helpPath": "/help/portal-view/forms/create-api",
  "actions": []
}

If we need indirection later, we can change to helpId and resolve it through a small registry:

{
  helpId: "forms.create-api"
}

Start with helpPath because it is simple, transparent, and works well with static documentation.

Portal UI Behavior

Each page and generated form should have a small help action in a predictable location.

Recommended behavior:

• open help in a new browser tab
• use an external-link icon or help icon with an accessible label
• keep the help action near the page title or form title
• if a form is opened inside a task shell, prefer form-specific help first and show task help as a secondary link
• if no specific help exists yet, fall back to the nearest page or concept help

Example resolution order for a form opened from a task:

1. form helpPath
2. current task helpPath
3. current page helpPath
4. generic portal help landing page

Do not render a broken link. If a help path is missing, hide the action or show the fallback help link.

Generated Forms

Generated forms should support a top-level helpPath field in Forms.json. The renderer can read it and show a help action in the form header.

For example:

{
  "formId": "createSchedule",
  "helpPath": "/help/portal-view/forms/create-schedule",
  "schema": {},
  "form": []
}

Field-level help can be added later, but it should not be the first step. Many field descriptions can stay in the JSON schema title/description. Use field-level help only for fields where a short description is not enough, such as security, ownership, deployment, or advanced configuration fields.

Possible future field shape:

{
  "key": "ownerPositionId",
  "helpPath": "/help/portal-view/concepts/ownership-and-positions"
}

Task-Aware Help

The task-oriented navigation layer should support task help separately from page or form help. A user working on the same form may need different context depending on the task.

Example:

• createApi opened from “Register a new API” links to create API form help.
• createApi opened from “Onboard API to MCP Gateway” can also link to MCP onboarding task help.

The UI should pass task context through existing task URL parameters and layout state, then render both links when useful:

Help: Create API
Related: Onboard API to MCP Gateway

Authoring Guidelines

Each page help document should include:

• what the page is used for
• who can access it
• what records are visible
• common actions
• links to related forms and tasks

Each form help document should include:

• when to use the form
• what happens after submit
• required fields
• important optional fields
• ownership and permission behavior
• validation or troubleshooting notes

Keep help content user-facing. Do not put implementation details, class names, or database internals in the main help body unless they are truly needed for an operator.

Validation

To prevent link drift, add a lightweight validation step once the first help docs exist.

Validation should check:

• every helpPath in portal-view points to a markdown source in light-portal-doc
• every high-value page has page help
• every high-value form has form help
• no help path uses a route-specific or component-specific unstable name

This can start as a script in light-portal-doc or a shared CI check that accepts both repo paths.

Rollout Plan

Phase 1: Documentation Structure

• Create src/help/portal-view/pages.
• Create src/help/portal-view/forms.
• Create src/help/portal-view/tasks.
• Create src/help/portal-view/concepts.
• Add placeholder help pages for the high-value admin pages and forms.

Phase 2: App Metadata

• Add optional helpPath to pageRegistry.ts.
• Add optional helpPath to taskRegistry.ts.
• Add optional top-level helpPath to generated form metadata.
• Add a docs base URL runtime config.

Phase 3: UI Components

• Add a reusable help-link component.
• Render page help near page titles.
• Render form help in the generated form header.
• Render task help in the task navigation shell.
• Add fallback behavior when a specific help link is missing.

Phase 4: Coverage And Validation

• Add help paths for all self-service owner-scoped admin pages.
• Add help paths for all high-value create/update forms.
• Add a validation script for help path coverage and broken links.
• Add missing docs over time as pages move into the task-oriented model.

Initial Scope

Start with the pages and forms most likely to be used by self-service users:

• API Admin and API Detail
• create/update API
• create/update API Version
• App Admin
• create/update App
• OAuth Client and Client Token
• create/update Client
• create Client Token
• Instance Admin and relationship pages
• create/update Instance
• create Instance API
• create/update Instance API Path Prefix
• create Instance App
• create Instance App API
• Schedule Admin
• create/update Schedule
• Workflow Definition
• create/update Workflow Definition

Then expand to admin-only pages after their ownership and access model is clear.

MVP Decisions

Use these decisions for the first implementation.

Missing Help Links

Do not hide the help action when a specific page, form, or task help path is missing. Fall back to the generic portal-view help landing page:

/help/portal-view/index

This keeps the UI consistent. A missing specific help page should degrade to general help instead of making the help affordance disappear.

Help Presentation

Open help in a new browser tab for the MVP. Do not build an embedded markdown viewer, side drawer, or iframe-based documentation panel in the first version.

This keeps portal-view small and avoids adding documentation rendering, iframe, routing, and panel-state complexity to the app. A side panel can be revisited later if users need in-page help while editing long forms.

JSON Schema Descriptions

Do not auto-generate full form help pages from JSON schema descriptions. Schema titles and descriptions are best used for inline labels, helper text, or field-level tooltips.

Form-level help should explain why the form exists, when to use it, what happens after submit, and how the form fits into a larger workflow. It should not simply repeat field types and required flags.

Documentation Versioning

Use latest documentation URLs for the MVP. Do not introduce release-versioned help URLs in the first implementation.

The portal will likely support both cloud SaaS deployments and enterprise on-premise deployments. SaaS users normally interact with the latest deployed portal, but enterprise customers may run older portal versions for a longer period. Versioned docs are therefore a good future requirement, but they should not block the first help-link rollout.

Keep helpPath values relative and version-neutral:

/help/portal-view/forms/create-api

Then versioning can be introduced later by changing only the configured docs base URL:

PORTAL_DOC_BASE_URL=https://doc.lightapi.net/v2.0

This keeps the app metadata stable while allowing SaaS to use latest docs and on-premise builds to point at version-specific documentation.

Future Enhancements

In-Page Help Drawer

Add an optional in-page help drawer after the helpPath metadata is stable and the first new-tab implementation has proven useful.

The drawer should be opt-in, not the default for every form. Long or complex configuration forms can declare:

{
  "helpPath": "/help/portal-view/forms/update-instance",
  "inPageHelp": true
}

When enabled, the UI can render a right-side drawer that displays the help document through an iframe or a lightweight markdown renderer. This avoids constant tab switching for complex forms while keeping the MVP simple.

Field-Level Help Paths

Add field-level help paths sparingly for complex fields and architectural concepts. Standard fields should continue to use JSON schema titles, descriptions, helper text, or tooltips.

Example future field metadata:

{
  "key": "ownerPositionId",
  "helpPath": "/help/portal-view/concepts/ownership-and-positions"
}

The UI can render a small help icon next to the field label when a field-level helpPath exists. Good candidates include ownership, security, OAuth token exchange, deployment target, transport configuration, and workflow definition fields.

Versioned Documentation

Add release-versioned documentation when multiple portal versions must be supported at the same time, especially for on-premise enterprise deployments.

The relative helpPath values should remain unchanged. The deployment or build configuration should select the versioned docs base URL:

SaaS/latest:
PORTAL_DOC_BASE_URL=https://doc.lightapi.net

On-premise v2.0:
PORTAL_DOC_BASE_URL=https://doc.lightapi.net/v2.0

This gives cloud deployments a simple latest-docs experience and gives enterprise deployments a path to version-matched help without changing portal-view metadata.

Recommendation

Store user-facing help content in light-portal-doc and add declarative helpPath metadata in portal-view. This keeps documentation maintainable and publishable while allowing every page, form, and task to provide context-aware help from the UI.

Event Processing Notifications

Portal commands are event driven. After a command is submitted, one or more CloudEvents are written to event_store_t and outbox_message_t. The hybrid-query event consumer later processes the outbox rows and updates the projection tables used by portal-view.

The notification page in the user profile is intended to show the user the recent processing result for those events. Today the table and read path exist, but notification_t is not populated consistently, so the page cannot provide meaningful status.

Current State

The command path already writes events through the common command handler:

1. The command handler validates and enriches the request.
2. It builds one or more CloudEvents.
3. It inserts those events into event_store_t and outbox_message_t.
4. The command returns before the query-side projection has necessarily run.

The query side can run through either event-processing pipeline, selected by configuration:

• Pg-notify pipeline: DbEventConsumerStartupHook polls outbox_message_t, uses the table’s gapless c_offset, groups rows by transaction_id, and writes failed transactions to the database dead_letter_queue.
• Kafka pipeline: a connector publishes rows from outbox_message_t to Kafka. PortalEventConsumerStartupHook consumes those records, groups records by the command-side transaction_id, and produces failed transactions to the Kafka DLQ topic when DLQ is enabled.

Both pipelines eventually call PortalDbProvider.handleEvent(conn, event). handleEvent dispatches the event to the projection method for that event type. Because both pipelines process the same outbox-backed events, they should share the same user-facing notification status model.

The notification pieces are partially present:

• notification_t exists in portal-db.
• NotificationDataPersistenceImpl can query notification_t.
• NotificationServiceImpl can insert a notification row.
• user-query exposes getNotification.
• portal-view has a notification table page.

The current gap is that notification rows are not created at the central event processing boundary.

There is also a separate UI error in MailMenu: it calls getPrivateMessage, whose handler currently returns an empty response. That explains the browser error Unexpected end of JSON input, but it is separate from the notification status design.

Goals

• Show the current user the latest event processing results in the profile notification page.
• Record both successful and failed projection processing.
• Preserve event processing correctness even if notification insertion fails.
• Keep notification creation centralized instead of adding calls to every projection method.
• Support commands that emit multiple events.
• Make the read API filter by host and user by default.
• Keep enough diagnostic data to debug failed projections.
• Keep notification writes idempotent so event replay is safe.

Non-Goals

• Do not replace event_store_t, outbox_message_t, or dead_letter_queue.
• Do not use notifications as the source of truth for projection state.
• Do not build a real-time push channel in the first phase.
• Do not add notification logic manually to every projection method.
• Do not expose other users’ processing history to non-admin users.

Recommended Design

Use notification_t as an operational projection-status table. The command side creates PENDING rows at the central event publication boundary, and the hybrid-query event consumer updates those rows with the processing result.

The primary processing-result write point should be the centralized outbox consumer path, around the call to PortalDbProvider.handleEvent(conn, event).

Recommended lifecycle:

command handler
  -> event_store_t
  -> outbox_message_t
  -> notification_t PENDING row
  -> response to caller

hybrid-query consumer
  -> read outbox_message_t
  -> handleEvent(conn, event)
  -> projection table write
  -> notification_t status row

For command-side publication, insert or update one notification row for each CloudEvent with status PENDING in the same transaction that writes event_store_t and outbox_message_t. Leave event_partition and event_offset null for this first insert, because the consumer has not observed the event position yet.

For successful projection processing, update the notification row for the CloudEvent to status SUCCEEDED and populate event_partition and event_offset from the active processor’s outbox position.

For failed projection processing, insert or update one notification row for each failed CloudEvent with status FAILED or DLQ, and store the exception message. Populate event_partition and event_offset when the processor has that information.

Status Model

Use one explicit status field. Do not keep is_processed; this feature is being implemented for the first time, and a boolean cannot distinguish pending, success, retry, DLQ, and skipped outcomes.

Recommended statuses:

The UI should show the status labels, not the underlying event pipeline. The same status meanings apply to both pg-notify and Kafka processing.

Schema

The existing table is close, but it is too small for operational status and has nonce as INTEGER while event tables use BIGINT.

Recommended table shape:

CREATE TABLE notification_t (
    id                  UUID NOT NULL,
    host_id             UUID NOT NULL,
    user_id             UUID NOT NULL,
    nonce               BIGINT NOT NULL,
    event_class         VARCHAR(255) NOT NULL,
    event_json          TEXT NOT NULL,
    event_ts            TIMESTAMP WITH TIME ZONE NULL,
    process_ts          TIMESTAMP WITH TIME ZONE NOT NULL,
    status              VARCHAR(16) NOT NULL,
    error               VARCHAR(2048) NULL,
    aggregate_id        VARCHAR(255) NULL,
    aggregate_type      VARCHAR(255) NULL,
    aggregate_version   BIGINT NULL,
    event_partition     INTEGER NULL,
    event_offset        BIGINT NULL,
    transaction_id      UUID NULL,
    read_ts             TIMESTAMP WITH TIME ZONE NULL,
    PRIMARY KEY (host_id, id),
    FOREIGN KEY (host_id) REFERENCES host_t(host_id) ON DELETE CASCADE
);

user_id is intentionally not a foreign key to user_t. PENDING rows are inserted on the command side before projection tables are updated, so enforcing that projection FK would break commands such as user creation before the projection catches up.

Recommended indexes:

CREATE INDEX idx_notification_user_process_ts
    ON notification_t (host_id, user_id, process_ts DESC);

CREATE INDEX idx_notification_status_process_ts
    ON notification_t (host_id, status, process_ts DESC);

CREATE INDEX idx_notification_transaction
    ON notification_t (host_id, transaction_id);

CREATE INDEX idx_notification_event_position
    ON notification_t (host_id, event_partition, event_offset);

CREATE INDEX idx_notification_unread_failure
    ON notification_t (host_id, user_id, process_ts DESC)
    WHERE read_ts IS NULL AND status IN ('FAILED', 'DLQ');

event_partition and event_offset are intentionally generic processing position fields. They are useful for operator diagnostics, but the UI should not label them as pg-notify or Kafka details. In the pg-notify processor, event_partition is the configured logical consumer partition and event_offset is outbox_message_t.c_offset. In the Kafka processor, event_partition and event_offset are the consumed Kafka record partition and offset.

Both columns are nullable. PENDING rows should leave them empty at initial insert time. They are filled later by the pg-notify or Kafka processor when the processing result changes the row to SUCCEEDED, FAILED, DLQ, or SKIPPED.

transaction_id remains a UUID because it is generated by the command side and used by both event processors.

Do not store pipeline name, source topic/channel name, or DLQ destination in notification_t. Those are implementation details of the configured event pipeline. Operators can use service configuration and logs when they need pipeline-specific diagnostics.

For existing installations, ship this as a patch:

ALTER TABLE notification_t ALTER COLUMN nonce TYPE BIGINT;
ALTER TABLE notification_t ADD COLUMN IF NOT EXISTS status VARCHAR(16);
ALTER TABLE notification_t ADD COLUMN IF NOT EXISTS event_ts TIMESTAMP WITH TIME ZONE;
ALTER TABLE notification_t ADD COLUMN IF NOT EXISTS aggregate_id VARCHAR(255);
ALTER TABLE notification_t ADD COLUMN IF NOT EXISTS aggregate_type VARCHAR(255);
ALTER TABLE notification_t ADD COLUMN IF NOT EXISTS aggregate_version BIGINT;
ALTER TABLE notification_t ADD COLUMN IF NOT EXISTS event_partition INTEGER;
ALTER TABLE notification_t ADD COLUMN IF NOT EXISTS event_offset BIGINT;
ALTER TABLE notification_t ADD COLUMN IF NOT EXISTS transaction_id UUID;
ALTER TABLE notification_t ADD COLUMN IF NOT EXISTS read_ts TIMESTAMP WITH TIME ZONE;

ALTER TABLE notification_t DROP CONSTRAINT IF EXISTS notification_t_user_id_fkey;
ALTER TABLE notification_t DROP COLUMN IF EXISTS is_processed;
ALTER TABLE notification_t ALTER COLUMN status SET NOT NULL;

CREATE INDEX IF NOT EXISTS idx_notification_unread_failure
    ON notification_t (host_id, user_id, process_ts DESC)
    WHERE read_ts IS NULL AND status IN ('FAILED', 'DLQ');

Write Path

Notification writes need an explicit transaction policy. A single rule cannot cover every status:

• Failure and DLQ notifications must be durable even when projection writes are rolled back.
• Success notifications must not claim success until the projection write has committed.
• Notification write failures should not break projection processing.

Use a REQUIRES_NEW style helper for notification writes that must survive a projection rollback. In plain JDBC, this means opening a separate connection with its own commit/rollback boundary.

For success rows, there are two safe options:

1. Commit the projection transaction first, then write SUCCEEDED in a separate notification transaction.
2. Write SUCCEEDED inside the projection transaction, but wrap it in a savepoint and treat notification insert failure as non-fatal.

The first option is the recommended default because notification failures cannot roll back projection updates. The tradeoff is a small window where projection has committed but the success notification is missing. That is acceptable because event_store_t remains the source of truth and success notifications are user feedback, not projection correctness.

Recommended service methods:

void recordPending(Map<String, Object> event, UUID transactionId);

void recordSuccess(Map<String, Object> event, EventMetadata metadata);

void recordFailure(Map<String, Object> event, EventMetadata metadata, String error, String status);

recordPending should participate in the command-side transaction that writes event_store_t and outbox_message_t. It may store transaction_id, because that value is generated by the command side, but it must leave event_partition and event_offset null. recordSuccess and recordFailure should use the event-processing transaction policy described below.

EventMetadata should carry only pipeline-neutral data that is not inside the CloudEvent map:

• eventPartition: the active processor’s partition value. For pg-notify this is the configured logical consumer partition; for Kafka this is the consumed Kafka record partition.
• eventOffset: the active processor’s offset value. For pg-notify this is outbox_message_t.c_offset; for Kafka this is the consumed Kafka record offset.
• transactionId: the command-side transaction UUID used by both processors.

Both consumers should build this metadata before calling handleEvent, so the failure path still has offset and transaction context after the projection transaction is rolled back.

Use an idempotent upsert:

INSERT INTO notification_t (...)
VALUES (...)
ON CONFLICT (host_id, id) DO UPDATE SET
    process_ts = EXCLUDED.process_ts,
    status = EXCLUDED.status,
    error = EXCLUDED.error,
    event_partition = EXCLUDED.event_partition,
    event_offset = EXCLUDED.event_offset,
    transaction_id = EXCLUDED.transaction_id;

This makes replay and fallback processing safe.

Success Handling

In the normal batch path:

begin projection transaction
for each event from the active pipeline:
  parse CloudEvent
  handleEvent(conn, event)
commit projection transaction

for each successfully committed event:
  recordSuccess(event, metadata) in separate notification transaction

Do not write SUCCEEDED before the projection transaction commits unless it is part of the same transaction. If it is written in the same transaction, a projection rollback must roll back the success row too.

The implementation can keep an in-memory list of successfully applied events while processing the batch. After commit, loop through that list and upsert the success notifications. If a success notification write fails, log it and continue; do not retry the projection.

Failure Handling

Failure rows must be written outside the failed projection transaction.

In fallback mode, processing is retried per transaction. For a failed transaction:

begin projection transaction
savepoint projection_attempt
  process transaction events
on exception:
  rollback to projection_attempt or rollback projection transaction
  write failed events to database DLQ or Kafka DLQ topic
  recordFailure(event, metadata, error, "DLQ") in separate notification transaction

For pg-notify, the DLQ destination is the database dead_letter_queue table. For Kafka, the DLQ destination is the configured Kafka DLQ topic. The failure notification can be committed with the database DLQ transaction for pg-notify, or in a separate notification transaction immediately after the Kafka DLQ produce request is accepted. The key requirement is that it must not be part of the projection work that is being rolled back.

If the database connection enters an unrecoverable error state, close it and open a fresh connection for the DLQ and failure notification writes.

If the payload cannot be parsed as a CloudEvent, the consumer may not know the CloudEvent id or event type. In that case, the DLQ remains the primary failure record. If the consumer metadata still has host, user, partition, offset, and transaction id, the consumer can create a diagnostic notification with a generated id, but this should be treated as a best-effort operational row.

Pending Handling

PENDING is part of phase one. Add pending rows at the central command-side publication boundary that writes event_store_t and outbox_message_t.

The pending notification should be written in the same command-side transaction as the event-store and outbox rows. If the command rolls back, the pending notification must roll back too. Do not add pending writes to individual command handlers.

At this stage, the notification row should contain command-known fields only: CloudEvent id, host id, user id, nonce, event class, event JSON, event timestamp, and transaction id. The processor-owned event_partition and event_offset fields remain null until event processing updates the row.

Read API

Keep getNotification as the main query endpoint, but tighten its contract.

Recommended request fields:

{
  "hostId": "uuid",
  "userId": "uuid",
  "offset": 0,
  "limit": 25,
  "status": "SUCCEEDED",
  "eventClass": "ClientCreatedEvent",
  "nonce": "123",
  "fromTs": "2026-05-08T00:00:00Z",
  "toTs": "2026-05-08T23:59:59Z",
  "error": "duplicate key"
}

Recommended response:

{
  "total": 1,
  "notifications": [
    {
      "id": "uuid",
      "hostId": "uuid",
      "userId": "uuid",
      "nonce": 123,
      "eventClass": "ClientCreatedEvent",
      "status": "SUCCEEDED",
      "processTs": "2026-05-08T16:12:00Z",
      "aggregateId": "host|client",
      "aggregateType": "Client",
      "aggregateVersion": 2,
      "transactionId": "uuid",
      "eventPartition": 0,
      "eventOffset": 1001,
      "error": null,
      "eventJson": "{...}"
    }
  ]
}

eventPartition and eventOffset are intentionally displayed as generic position fields regardless of which event pipeline is configured. The main list can hide them by default and show them in the detail view.

userId is a filter on getNotification, not a separate endpoint contract. The profile notification page should always send the logged-in user’s userId. The admin notification page can omit userId to request host-wide results, or pass a specific userId to narrow the host-wide view to one user.

Authorization rules:

• Normal users can query only their own token user_id within the selected hostId. If the request omits userId, the backend should apply the token user_id; if the request supplies another userId, the backend should reject it or override it with the token user_id.
• Admin users can query all users for the host by omitting userId, or filter to a specific user by providing userId.
• The backend should enforce this using token claims, not only UI filters.

Portal View

The profile notification page should become a processing-status view.

Recommended columns:

• Time
• Status
• Event
• Aggregate
• Nonce
• Error
• Details

Recommended default filters:

• hostId from the selected host.
• userId from the logged-in user.
• No default status filter.
• Most recent first.
• Last 25 rows.

The UI should display concise summaries and keep full eventJson behind an expandable detail row or dialog.

Show all events associated with the user, including successful, failed, and derived events. Derived events should be visible as their own rows instead of being collapsed under the original command transaction.

The current processFlag filter should be replaced by status. No is_processed compatibility mapping is needed because this feature has not yet started populating notification_t.

The header MailMenu should not call getPrivateMessage unless that handler is restored. For notification status, add a small notification badge endpoint or reuse getNotification with limit = 5.

The header badge should count only unread failure notifications, such as FAILED and DLQ, and display the count in red when the count is greater than zero.

In the list, FAILED and DLQ status badges should also use red styling.

Phase 2 adds two narrow user-query RPCs:

• getUnreadNotificationCount: returns unread FAILED and DLQ notifications for the current hostId and userId.
• markFailureNotificationsRead: sets read_ts on unread FAILED and DLQ notifications for the current hostId and userId.

The header uses the count endpoint for its badge and marks failures read when the user opens the notification menu. The notification page also marks failures read when it is opened.

Admin Notification Page

Phase 3 should add a separate admin notification page instead of overloading the profile notification page. The recommended location is:

• Route: /app/event/notifications
• Menu: Administration -> Event Admin -> Notifications

This page should reuse the same notification table and getNotification read API, but with admin defaults:

• hostId from the selected host.
• No default userId filter, so admins see host-wide results.
• Default status filter for FAILED and DLQ, with an option to show all statuses.
• Filters for userId, eventClass, status, transactionId, aggregateId, processing position, time range, and error text.
• No unread badge behavior and no call to markFailureNotificationsRead.

The page should clearly identify itself as an admin view, such as “Admin View: Host Notifications”. Host-wide access must still be enforced by the backend using token roles.

Operational Cleanup

Notifications are operational history. They should not grow forever.

Recommended retention:

• Keep successful notifications for 30 to 90 days.
• Keep failed and DLQ notifications longer, such as 180 days.
• Allow host-level configuration later if needed.

Cleanup should be implemented as a generic operational cleanup process, not as notification-specific UI or command-handler logic. The first cleanup target is notification_t, but the same framework should also support other operational tables such as message_t for private messages.

Recommended implementation:

• Add an OperationalCleanupStartupHook on the query side.
• Run cleanup on a fixed interval, such as daily, with config-driven enablement, interval, batch size, and per-target retention days.
• Use a single cleanup coordinator that owns multiple cleanup targets. Each target defines its table, timestamp column, status/type conditions if needed, retention duration, and batch delete SQL.
• Use a database lock, such as a PostgreSQL advisory lock or a dedicated cleanup lock row, so only one service instance performs cleanup at a time.
• Delete in bounded batches to avoid long table locks and large transactions.
• Use a separate database connection and transaction for cleanup work.
• Log cleanup failures and continue service startup; cleanup failure must not block query APIs or event processing.

Do not use schedule_t directly for this cleanup. That scheduler is business workflow infrastructure that emits events into event_store_t and outbox_message_t. Operational cleanup is local maintenance and should stay out of the event-processing path.

Example notification cleanup:

WITH doomed AS (
    SELECT host_id, id
    FROM notification_t
    WHERE (status IN ('SUCCEEDED', 'SKIPPED') AND process_ts < ?)
       OR (status IN ('FAILED', 'DLQ') AND process_ts < ?)
    ORDER BY process_ts
    LIMIT ?
)
DELETE FROM notification_t n
USING doomed d
WHERE n.host_id = d.host_id
  AND n.id = d.id;

Private-message cleanup can be another target using message_t.send_time:

WITH doomed AS (
    SELECT host_id, from_id, nonce
    FROM message_t
    WHERE send_time < ?
    ORDER BY send_time
    LIMIT ?
)
DELETE FROM message_t m
USING doomed d
WHERE m.host_id = d.host_id
  AND m.from_id = d.from_id
  AND m.nonce = d.nonce;

Recommended default cleanup targets:

Do not delete recent PENDING notifications. Old PENDING rows should be treated as an operational signal first because they may indicate that the event consumer is stopped or lagging. If a hard cap is needed later, make it a separate, longer retention policy.

Snapshot and Promotion

notification_t should be treated as an operational table, not a promoted projection table.

It should be excluded from global snapshot export and conversion alongside event_store_t, outbox_message_t, dead_letter_queue, log_counter, and consumer_offsets.

Rollout Plan

Phase 1: Make Notifications Useful

• Add status and diagnostic columns to notification_t.
• Add pipeline-neutral event_partition, event_offset, and transaction_id metadata.
• Change NotificationService to support separate notification transactions.
• Insert PENDING rows at the central command-side outbox publication boundary.
• Insert SUCCEEDED rows after successful handleEvent.
• Insert DLQ rows in fallback failure handling.
• Update getNotification to support status and correct timestamp fields.
• Update portal-view to use status, default to the current user, and show all user-associated events including derived events.

Phase 2: Improve User Feedback

• Add an unread marker with read_ts.
• Add a small header badge query for unread FAILED and DLQ notifications and render the badge in red.
• Mark unread failure notifications as read when the user opens the header menu or the notification page.

Phase 3: Operations

• Add a generic operational cleanup startup hook with retention targets for notification_t and message_t.
• Make cleanup configurable by enablement, interval, batch size, and per-target retention days.
• Add a database lock so only one service instance runs cleanup at a time.
• Add an admin notification page under Event Admin that uses getNotification without a userId filter for host-wide failures.
• Add dashboards or alerts for repeated DLQ statuses.

Risks and Mitigations

API Marketplace Catalog

Context

The portal already has a Marketplace navigation group and an api-marketplace page registry entry. The current API administration page is table-oriented and is useful for owners, but it is not a consumer catalog. A Marketplace API catalog should let users discover APIs by business category, capability, protocol, lifecycle status, and governance metadata.

API create and update forms already use the standardized taxonomy fields:

• categoryIds for selected category identifiers.
• tagIds for selected tag identifiers.
• getCategoryLabelByType with entityType = "api" for category options.
• getTagLabelByType with entityType = "api" for tag options.

The service query layer also returns categoryIds, categories, tagIds, and tags for API rows. The catalog should use those fields for display and filtering instead of reintroducing the legacy apiTags string field.

Goals

• Add a Marketplace menu entry for an API catalog.
• Use database-backed categories and tags, not hard-coded UI lists.
• Keep categories and tags reusable across future catalog pages.
• Keep API create/update forms as the source of truth for taxonomy assignment.
• Give consumers a browse-first experience instead of an admin table.
• Support deep links from a catalog listing to API detail, versions, endpoints, runtime bindings, and owner actions.
• Preserve host scope and ownership rules already used by API administration.

Non-Goals

• Do not replace API administration pages with the catalog.
• Do not store display names in API rows when they can be resolved from category_t, tag_t, entity_category_t, and entity_tag_t.
• Do not use the old apiTags field for catalog filtering.
• Do not make taxonomy values static frontend constants.
• Do not expose private tenant APIs through a public catalog without an explicit visibility and authorization decision.

Current Building Blocks

User Experience

The first screen under Marketplace should be the usable catalog, not a landing page. The recommended route is:

/app/marketplace/api

The sidebar can keep the existing Marketplace group, but its children should move from API-type-only links to intent-based entries:

• API Catalog
• API Clients
• JSON Schema
• YAML Rule
• Schema Form

The API Catalog page should provide:

• Search across API id, name, description, business group, line of business, capability, platform, git repository, categories, and tags.
• Category tabs or a category rail based on getCategoryLabelByType.
• Grouped tag filters based on getTagLabelByType.
• Filter chips for active category and tag selections.
• A compact card or list row per API with name, description, status, categories, tags, owner, business group, and latest version summary.
• Actions to view details, review versions, create a new version, update the API metadata, and open related runtime or access-control pages when the user has permission.

The catalog should support an Uncategorized bucket for active APIs without category assignments. This avoids hiding incomplete data and gives admins an easy cleanup target.

Categories And Tags

Categories should be stable browse buckets. Tags should be flexible facets. Both are stored with entityType = "api" so the same tag names can be reused for other entity types without forcing cross-catalog semantics.

Recommended initial API categories:

The stored category_name must stay lower-case and URL-friendly. The display labels above are UI labels derived from those values.

Recommended initial API tag groups:

Stored tag names must stay lower-case and URL-friendly. If a display label needs capitalization, the UI should format it or the label endpoint should provide a separate display field later.

Tags without tag_group_code or tag_group_label should be shown under a General filter group in the catalog UI. Configured groups should appear first by group_sort_order; the General group should appear after configured groups, matching the current label query behavior where null group sort values sort last.

Data Flow

Catalog filter option loading:

portal-view
  -> category/getCategoryLabelByType(hostId, entityType = "api")
  -> tag/getTagLabelByType(hostId, entityType = "api")

Catalog result loading:

portal-view
  -> service/getApi(hostId, offset, limit, active, filters, globalFilter, sorting)
  -> api rows with categoryIds, categories, tagIds, tags

The catalog should prefer server-side pagination and filtering. Client-side filtering is acceptable only for a small first pass because it breaks as soon as the API count exceeds one fetched page.

Query Contract

The existing getApi contract already supports filters, globalFilter, sorting, offset, limit, hostId, and active. To make the catalog work well at scale, add first-class filter support for taxonomy fields:

{
  "hostId": "01964b05-552a-7c4b-9184-6857e7f3dc5f",
  "offset": 0,
  "limit": 20,
  "active": true,
  "categoryIds": ["..."],
  "tagIds": ["..."],
  "tagMatch": "all",
  "globalFilter": "payment"
}

Recommended semantics:

• categoryIds uses OR semantics by default. An API in any selected category is returned.
• tagIds should support tagMatch = "all" and tagMatch = "any".
• Category and tag filters should use EXISTS against entity_category_t and entity_tag_t with entity_type = 'api' and active = TRUE.
• Display arrays should continue to be returned as categories and tags.
• Form update payloads should continue to submit identifiers only through categoryIds and tagIds.

Page Design

The API Catalog page can be implemented as a dedicated page rather than trying to stretch the current API admin table.

Proposed files:

src/pages/marketplace/ApiCatalog.tsx
src/pages/marketplace/components/ApiCatalogFilters.tsx
src/pages/marketplace/components/ApiCatalogCard.tsx
src/pages/marketplace/hooks/useApiCatalog.ts

Page state:

• search text
• selected category ids
• selected tag ids
• tag match mode
• active status
• pagination
• sorting
• view mode, either compact list or card grid

Catalog state should be URL-driven from Phase 1. Search text, selected categories, selected tags, tag match mode, active status, sorting, and pagination should be encoded in the query string so users can refresh the page, use browser navigation, and share filtered catalog URLs. Example:

/app/marketplace/api?q=payment&category=public-api&tag=oauth2&tag=mtls&tagMatch=all&page=1

The page should still reuse existing infrastructure:

• fetchClient for portal query calls.
• useUserState for host and user context.
• buildTaskAwareRoute for deep links.
• ownership utilities for update/delete action visibility.
• TaskActionPanel for publisher/admin next actions.
• pageRegistry and contextual help metadata.

Routing And Navigation

Add or update these portal-view entries:

The existing /app/marketplace route can redirect to /app/marketplace/api or remain a broader Marketplace landing page later. For the first API catalog implementation, redirecting keeps the behavior simple.

Backend Changes

The backend already persists API category and tag relationships. The main backend change is query filtering:

1. Extend service-query spec for optional categoryIds, tagIds, and tagMatch.
2. Update GetApi to pass those optional fields to the DB provider.
3. Update PortalDbProvider#getApi and ApiServicePersistenceImpl#getApi.
4. Add SQL predicates over entity_category_t and entity_tag_t.
5. Verify or add compound indexes for taxonomy filtering.
6. Add tests for category-only, tag-any, tag-all, combined taxonomy filters, and APIs with no taxonomy assignments.

The existing join-table indexes are useful for entity lookups and label resolution, but catalog filtering also needs indexes that start with filter fields. Before implementing Phase 2, verify the query plan and add indexes if needed:

CREATE INDEX idx_entity_tag_filter
ON entity_tag_t (entity_type, tag_id, entity_id)
WHERE active = TRUE;

CREATE INDEX idx_entity_category_filter
ON entity_category_t (entity_type, category_id, entity_id)
WHERE active = TRUE;

For tagMatch = "all", prefer a single grouped subquery over generating one EXISTS predicate per selected tag when the selected tag set can grow. A common shape is to filter entity_tag_t by selected tag ids, group by entity_id, and require COUNT(DISTINCT tag_id) = selectedTagCount.

The query response should continue to include both identifiers and labels:

{
  "apiId": "0001",
  "apiName": "Petstore",
  "categoryIds": ["..."],
  "categories": ["public-api"],
  "tagIds": ["..."],
  "tags": ["openapi", "oauth2"]
}

Implementation Phases

Phase 1: Catalog Page

• Add the API Catalog route and Marketplace menu entry.
• Load category and tag options from the existing label endpoints.
• Load APIs with service/getApi.
• Render search, category filter, grouped tag filter, and API list/card results.
• Store catalog filters, search text, sorting, and pagination in the URL query string.
• Use current query response labels for display.
• Deep-link to existing API detail and update forms.

Phase 2: Server-Side Taxonomy Filters

• Add categoryIds, tagIds, and tagMatch to service-query.
• Implement SQL filtering in ApiServicePersistenceImpl.
• Keep current table filtering support for admin use.
• Add DB provider and handler tests.

Phase 3: Catalog Polish

• Add API detail summary panels with versions, endpoint count, runtime exposure, and access-control hints.
• Add help docs and task links.
• Add optional counts per category and tag if the catalog needs faceted counts.

Open Questions

• Should Marketplace API Catalog show only active APIs by default? The recommendation is yes, with an admin-visible inactive filter.
• Should unauthenticated users ever see catalog data? The recommendation is no until a separate public visibility model is designed.
• Should category selection be single-select or multi-select? The recommendation is multi-select OR semantics for flexibility.
• Should tags use all-match or any-match by default? The recommendation is all for precision, with a visible toggle if users need broader searches.
• Should OpenAPI tags imported from specs automatically create API catalog tags? The recommendation is no for the first pass. Spec tags are often endpoint-level groupings and should not automatically become curated catalog taxonomy.

Agent Skill And API Endpoint Discovery

Problem

The GenAI chat flow has two separate concepts that are easy to confuse:

• The light-gateway MCP endpoint is the runtime server that lists and executes tools. An agent should call the gateway for tools/list and tools/call. A listed tool may be backed by a downstream MCP server or by a gateway-routed HTTP/OpenAPI endpoint.
• Portal-query is the catalog service for skills, tools, and agent assignments. The agent should read this catalog through the genai-query API, cache it locally, and search it during chat.
• The controller registry remains a runtime control-plane service for registration, discovery, and cache-management commands. It should not own the portal skill/tool catalog and should not execute downstream MCP or REST calls.

During chat, light-agent should use its local catalog cache to find relevant skills, then call tools/list on the gateway to verify executable tools. Tool execution still goes through the gateway. If the catalog cache is empty or stale, the agent should refresh it from portal-query. If portal-query is temporarily unavailable, the agent should still be able to use the gateway tool list directly.

The missing piece is a portal-managed catalog that explains which API endpoints exist, which endpoint projections are invokable by agents, which skills they belong to, and which agents are allowed or expected to use those skills. Without that catalog, the agent can list executable gateway tools, but it has no domain guidance beyond each tool description.

Goals

• Keep the gateway as the runtime source of truth for MCP tool execution.
• Keep direct gateway tools/list and tools/call working even when no skills have been authored.
• Treat API endpoints as the generic capability unit. MCP tools, OpenAPI operations, JSON-RPC methods, and future protocol operations should all become endpoint-level capabilities before they are exposed to agents.
• Populate a portal endpoint and tool catalog from API version parsing, LightAPI descriptions, gateway-discovered MCP tools, manually pasted MCP tools/list payloads, and gateway-routed REST tools.
• Let portal users create skills that contain instructions and curated tool selections.
• Let portal users assign skills to agent definitions.
• Use the genai-query API and spec as the portal-query access surface for skills, tools, and agent assignments.
• Let the agent cache the effective catalog locally and reload it when controller cache-management invalidation is triggered.
• Make skills useful for progressive disclosure without requiring every MCP tool to be wrapped before it can be called.
• Store semantic routing metadata for endpoint capabilities so the agent or portal-query can perform macro-filtering, keyword search, vector ranking, context viability checks, and safety filtering.

Non-Goals

• Do not move MCP request routing or downstream REST calls into the controller.
• Do not implement skill/search in controller-rs. Controller-rs can invalidate the agent cache, but portal-query owns catalog reads.
• Do not use config-server as the first delivery path for the skills/tools catalog. The agent can fetch from portal-query and cache locally.
• Do not require every gateway tool to have a skill before it is executable.
• Do not replace the existing MCP Gateway registry design. This design extends it with agent-facing skill curation.
• Do not implement embeddings in the first phase. Keyword search is enough for the initial local catalog search.
• Do not limit the catalog to MCP tools. The UI may use “tool” when referring to LLM tool-calling, but the persistent capability model should be endpoint first.
• Do not use skill assignments as the only authorization control. Gateway policy and downstream authorization still apply at execution time.

Concepts

Target Architecture

The target flow keeps runtime execution and control-plane metadata separate.

Portal UI
  -> writes api_endpoint_t, tool_t, tool_param_t, skill_t, skill_tool_t, skill_workflow_t, agent_skill_t

light-gateway /mcp
  -> lists executable tools from mcp-router.tools and downstream MCP servers
  -> executes tools/call against downstream MCP or REST services

light-workflow
  -> owns deterministic multi-step workflow execution, task state, and audit events

portal-query genai-query API
  -> serves skill/tool/agent-skill catalog reads from portal data

controller-rs portal registry
  -> registers agents and sends cache-management invalidation commands

light-agent
  -> loads assigned skills and mapped tools from portal-query
  -> caches the effective catalog locally
  -> searches cached skills during chat
  -> lists executable tools from light-gateway
  -> calls selected tools through light-gateway

For the account-agent example:

1. The gateway exposes account tools such as getAccounts and getAccountByNo.
2. Portal stores the canonical endpoint rows in api_endpoint_t.
3. Portal publishes selected endpoint rows into tool_t as agent-invokable capabilities.
4. An operator creates an “Account Management” skill in skill_t.
5. Portal links that skill to the account tools through skill_tool_t.
6. Portal assigns the skill to the account agent through agent_skill_t.
7. At startup or cache reload, the agent reads the assigned catalog through genai-query and caches it locally.
8. At chat time, the agent searches its local catalog cache.
9. The agent combines matched skill instructions with the gateway tool definitions.
10. Any tool call still goes to light-gateway tools/call.

Source Of Truth

The gateway is the runtime source of truth for executable tools. If a tool is not available from the gateway, the agent should not be able to execute it just because it exists in the portal database.

api_endpoint_t is the canonical portal endpoint catalog. It stores the endpoint identity, protocol method, path, logical tool schema, endpoint description, and raw tool metadata for one API version.

tool_t is the agent-facing projection of an endpoint. It stores the tool name, agent description, implementation type, optional endpoint reference, response schema, active flag, semantic routing fields, and semantic embedding. The full metadata object should still be preserved in api_endpoint_t.tool_metadata for import/export and agent cache payloads.

The portal database is the control-plane catalog. It stores:

• operator-friendly descriptions,
• skill instructions,
• agent assignments,
• governance metadata,
• cached or imported tool schemas.

Tool sync should be idempotent. The recommended unique identity is:

host_id + api_version_id + endpoint

Gateway exposure is a separate deployment selection. The catalog should sync all endpoint rows for an API version, then let the user choose which endpoint/tool projections are deployed to a specific gateway instance.

For runtime-executable projections, the gateway identity is:

hostId + serviceId + envTag

The access token used for portal catalog or gateway deployment APIs should carry matching host, sid, and env claims. Portal-query must verify those claims against the requested hostId, serviceId, and envTag before returning or changing catalog data.

Runtime verification means checking whether an endpoint projection is actually listed by a deployed gateway through tools/list. This should be done against the selected gateway instance when an operator is preparing or reviewing a gateway deployment. A later host-wide diagnostics view can aggregate all registered gateways, but phase 2 does not need host-wide verification as the default.

Runtime verification is not part of the persistence projection. The persistence layer should store catalog state, endpoint/tool metadata, and inactive drift state, but it should not call a live gateway. The portal UI, deployment review flow, or a diagnostics endpoint should call the selected gateway’s tools/list with the operator or service credential, compare the returned tool names and schemas with the catalog, and surface the result as deployment drift.

If a previously imported endpoint or tool disappears from the gateway, the sync process should mark the catalog projection inactive instead of deleting it immediately. This preserves skill mappings and gives operators a clear drift signal.

Current Data Model

The database already has the main tables needed for this design:

• skill_t: skill name, description, content_markdown, embedding placeholder, version, and active flag.
• tool_t: agent-facing tool catalog with name, description, implementation metadata, endpoint reference, and response schema.
• tool_param_t: parameter-level metadata and validation schema.
• agent_skill_t: maps agent definitions to skills.
• skill_tool_t: maps skills to tools for progressive disclosure.
• api_endpoint_t: MCP or REST endpoint metadata, including tool_schema and tool_metadata.
• wf_definition_t: stores workflow definitions as YAML for the light-workflow runtime.

Phase 3.5 should add a skill-to-workflow mapping table rather than storing workflow YAML inside skill_t. The recommended table is:

The current phase 2 persistence path can preserve semantic metadata in api_endpoint_t.tool_metadata before dedicated routing columns exist. That is acceptable for import/export compatibility and for small catalogs searched from the agent’s local cache. It should not be treated as the final indexed search shape. Before portal-query performs database-side macro-filtering over large catalogs or before vector ranking becomes a production dependency, promote the high-use routing fields to first-class columns or indexed relationships and backfill them from tool_metadata.

The existing MCP Registry design already maps MCP tools into api_endpoint_t. OpenAPI parsing also creates endpoint rows. This design uses tool_t as the agent-facing catalog row and links it back to api_endpoint_t when the tool originates from an API endpoint.

Recommended mapping for gateway-imported tools:

tool_t.implementation_type should be a standardized enum aligned with the LightAPI Description execution model. Endpoint-backed tools should use a LightAPI endpoint implementation type rather than preserving every downstream transport as a different tool implementation. The downstream protocol remains in the endpoint and LightAPI request metadata.

Recommended first enum values:

For lightapi_endpoint, execution still goes through gateway tools/call when the endpoint is exposed to a gateway. The source protocol, such as MCP, OpenAPI, JSON-RPC, OpenRPC, or gRPC, belongs in api_endpoint_t, tool_metadata, and the LightAPI request description.

Endpoint-First Capability Model

Agents and skills should operate over endpoint capabilities, not only over MCP tools. MCP remains the runtime protocol for tool-calling through the gateway, but the catalog should support any endpoint that can be represented as an agent-invokable capability.

Recommended capability layers:

1. api_endpoint_t: canonical endpoint row for the API version.
2. tool_t: agent-facing executable projection of the endpoint.
3. tool_param_t: normalized top-level input parameters derived from the endpoint’s JSON Schema.
4. skill_tool_t: curated relationship between a skill and a tool projection, including per-skill overrides such as priority, examples, or approval notes.
5. agent_skill_t: assignment of skills to agent definitions.

This model supports these source types:

tool_param_t should be generated from the logical input schema, not from wire transport details alone. For OpenAPI, the logical input schema should merge path parameters, query parameters, and request body into one object. For MCP, the logical input schema is the MCP inputSchema. For JSON-RPC, it is the logical params schema.

Semantic Routing Metadata

The customer-required semantic routing fields should be first-class indexed catalog data, not only JSON metadata. They are used for macro-filtering before expensive keyword, vector, or LLM ranking, so the common filter fields must be queryable through normal portal-query indexes.

Recommended indexed fields or relationships:

• domain and semantic namespace,
• sensitivity tier,
• semantic weight,
• target personas,
• active state,
• source protocol and implementation type,
• portal category and tag relationships.

Recommended phase 2 column names for endpoint and tool projections:

The full structured payload should still be preserved in api_endpoint_t.tool_metadata so LightAPI import/export, gateway config generation, and agent cache payloads have one portable metadata object.

Recommended api_endpoint_t.tool_metadata shape:

{
  "routing": {
    "domain": "finance.accounts",
    "category": "account-management",
    "semanticNamespace": "prod.accounts.core",
    "targetPersonas": ["account-agent", "customer-support-agent"],
    "semanticDescription": "Retrieves account profile and status information when a user asks about an existing account.",
    "semanticKeywords": ["account lookup", "customer account", "balance", "status"],
    "contextRequirements": {
      "requiredInputs": ["accountNo"],
      "requiredContext": ["host_id"]
    },
    "dependencies": [
      {
        "endpoint": "/v1/accounts/{accountNo}@get",
        "relation": "frequently_chained_after"
      }
    ],
    "semanticWeight": 0.75,
    "sensitivityTier": "Internal-Only",
    "fallbackEndpoint": "/v1/accounts@get",
    "embedding": {
      "model": "tool-description-embedding",
      "source": "semanticDescription"
    }
  },
  "safety": {
    "read_only": true,
    "destructive": false,
    "humanApprovalRequired": false
  }
}

Recommended ownership:

The first semantic search implementation can work from the agent’s local cache:

1. Filter by host, active flag, assigned skill, domain, namespace, target persona, and sensitivity tier.
2. Exclude endpoints whose required context is not available in the current workflow or chat state.
3. Rank by keyword matches over skill text, endpoint name, tool name, description, semantic keywords, and LightAPI capability text.
4. When embeddings are populated, combine vector similarity with the keyword score and multiply by semanticWeight.
5. Call gateway tools/list and intersect the ranked set with currently executable tools before exposing schemas to the LLM.

Embedding Recommendation

Keep the first production embedding dimension at 384 because the current Postgres vector column is already VECTOR(384) and the first catalog use case is routing over short endpoint descriptions, not long document retrieval.

Recommended model strategy:

• Use a provider abstraction with configured embedding_model, embedding_dimension, and embedding_source.
• For OpenAI-hosted embeddings, use text-embedding-3-small with the dimensions parameter set to 384.
• For on-prem or firewall-restricted deployments, use a local embedding service that is configured to emit 384-dimensional vectors.
• Store enough metadata to know how a vector was created: model, dimension, source text hash, source field, and generated timestamp.
• Re-embed when the semantic description, keywords, domain, or model config changes.

The portal catalog write path should remain in the portal service layer that owns api_endpoint_t and tool_t persistence. Because the current portal command/query services are Java, the Java side should own transactions, versioning, and persistence of embedding results. A Rust service or worker can still generate embeddings behind an internal API or queue consumer, especially if local model performance is better there. In that model, Java requests or consumes the vector and writes it through the normal portal persistence path.

LightAPI Description Enrichment

LightAPI Description should be the preferred enrichment source for endpoint capabilities. OpenAPI and MCP tools/list are good at initial extraction, but LightAPI adds the agent-oriented context needed for high-accuracy routing:

• endpoint identity and stable endpointId
• domain, tags, lifecycle, visibility, and capability group
• logical input schema and request mapping
• result schema and result cases
• examples and behavior notes
• progressive disclosure metadata
• agent-facing descriptions, personas, keywords, context requirements, and guardrails

Recommended merge priority for endpoint metadata:

1. Portal operator overrides.
2. Endpoint-level LightAPI Description.
3. API-level inherited LightAPI Description context.
4. OpenAPI/OpenRPC/protobuf/MCP source extraction.
5. Gateway runtime tools/list discovery.

This keeps runtime discovery useful while letting curated LightAPI descriptions provide richer semantic routing without hand-authoring every endpoint as an independent skill.

Phase 2 persistence should be treated as the receiver for this metadata, not as the extractor. The openapi-parser, a LightAPI Description parser, or a dedicated ingestion worker must emit the enriched endpoint payload on the API version event. At minimum, the event payload for each endpoint should include:

• endpointId, endpoint identity, protocol, method, path, name, and description,
• logical toolSchema generated from the LightAPI operation input contract,
• toolMetadata.routing with namespace, domain, capability group, personas, keywords, context requirements, sensitivity tier, and semantic weight where present,
• toolMetadata.safety from LightAPI safety, visibility, idempotency, and destructive-operation hints,
• response schema or result metadata when it is available for the tool projection.

If the parser only emits the base OpenAPI or MCP fields, the catalog remains valid but only has low-enrichment metadata. The phase 2 implementation should record that as an ingestion gap, not as a persistence defect.

Portal Catalog Contract

The agent should read skills and tools through the genai-query API in portal-query. The source spec is:

genai-query/src/main/resources/spec.yaml

The current spec already includes catalog endpoints for the main entities:

• getAgentSkill and getFreshAgentSkill
• getSkill and getFreshSkill
• getSkillTool and getFreshSkillTool
• getSkillDependency and getFreshSkillDependency
• getTool and getFreshTool
• getToolParam and getFreshToolParam

Phase 2 should add a dedicated effective catalog endpoint instead of forcing the agent to compose many generic query endpoints. The endpoint should still live in genai-query, not controller-rs.

Recommended endpoint behavior:

• verify the caller’s token claims before reading catalog rows,
• require request host_id, service_id, and env_tag,
• match token host, sid, and env claims to those request values,
• return only endpoint/tool projections valid for that host, service, and environment,
• include active endpoint metadata, tool schemas, safety metadata, routing metadata, and skill mappings relevant to the agent,
• support a freshness or version field so the agent can cache the result.

The agent should cache the returned structure locally:

{
  "host_id": "00000000-0000-0000-0000-000000000000",
  "agent_def_id": "00000000-0000-0000-0000-000000000000",
  "catalog_version": 42,
  "skills": [
    {
      "skill_id": "00000000-0000-0000-0000-000000000000",
      "name": "Account Management",
      "description": "Use account tools to inspect and manage customer accounts.",
      "content_markdown": "Prefer read-only tools before create or update tools.",
      "tools": [
        {
          "tool_id": "00000000-0000-0000-0000-000000000000",
          "endpoint_id": "00000000-0000-0000-0000-000000000000",
          "name": "getAccounts",
          "endpoint": "/v1/accounts@get",
          "api_type": "openapi",
          "description": "List account summaries.",
          "input_schema": {
            "type": "object",
            "properties": {}
          },
          "routing_metadata": {
            "domain": "finance.accounts",
            "semanticNamespace": "prod.accounts",
            "semanticKeywords": ["account list", "customer accounts"],
            "sensitivityTier": "Internal-Only"
          },
          "safety": {
            "read_only": true,
            "destructive": false
          }
        }
      ]
    }
  ]
}

For phase 2, the agent definition identity is the agent API version identity. agent_definition_t.agent_def_id stores the same UUID as api_version_t.api_version_id; the table is an agent-specific profile extension for model and runtime settings, not a second standalone agent registry. The agent display name comes from api_t.api_name, so agent_definition_t does not duplicate the API name. API Admin continues to own the API/API-version lifecycle, Instance Admin continues to own deployed instances, and the Agent Definition page edits the profile for that API version.

The previous registry skill/search response shape was:

{
  "skills": [
    {
      "skill_id": "00000000-0000-0000-0000-000000000000",
      "name": "Account Management",
      "description": "Use account tools to inspect and manage customer accounts.",
      "tool_name": "getAccounts",
      "input_schema": {
        "type": "object",
        "properties": {}
      }
    }
  ]
}

That flattened shape can remain as an internal compatibility DTO while the agent is migrated, but it should not be the long-term external contract. The target cache shape should support a skill with multiple tools:

{
  "skills": [
    {
      "skill_id": "00000000-0000-0000-0000-000000000000",
      "name": "Account Management",
      "description": "Use account tools to inspect and manage customer accounts.",
      "content_markdown": "Prefer read-only tools before create or update tools.",
      "tools": [
        {
          "name": "getAccounts",
          "description": "List account summaries.",
          "input_schema": {
            "type": "object",
            "properties": {}
          }
        }
      ]
    }
  ]
}

Migration rule:

• Remove the controller-rs skill/search placeholder.
• The agent can temporarily accept both the flattened shape and the nested tools shape while its portal-query client is being migrated.
• After migration, the nested effective catalog shape becomes the preferred internal cache contract.

Agent identity can come from token claims, configured agent definition, or request fields. If inference is not enough, pass explicit fields to the portal-query catalog call:

{
  "agent_def_id": "00000000-0000-0000-0000-000000000000",
  "host_id": "00000000-0000-0000-0000-000000000000",
  "service_id": "com.networknt.account-agent-1.0.0",
  "env_tag": "dev"
}

Runtime Behavior

The agent should treat the portal catalog as helpful guidance, not as a hard dependency for basic tool use.

Recommended behavior:

1. At startup, call the genai-query API to load the effective agent catalog.
2. Cache the catalog locally under host_id, agent identity, and catalog version.
3. During chat, search the local catalog with the user prompt.
4. If matched skills are returned, add skill instructions to the prompt context.
5. If matched skills include tool mappings, prefer those tools for the LLM tool list.
6. Call gateway tools/list to verify executable tools and obtain the current runtime schemas.
7. Intersect skill-selected tool names with gateway-listed tools.
8. If no skills match, or the local catalog is unavailable, fall back to gateway tools/list.
9. Execute all LLM tool calls through gateway tools/call.

When portal data changes, controller cache management can invalidate the agent’s local catalog cache. Reload behavior should match the agent’s initial loading strategy:

• if the agent loads the catalog during startup, invalidation should trigger an eager reload so the next chat request sees current metadata;
• if the agent loads the catalog on the first request, invalidation can clear the cache and let the next request reload lazily.

This keeps the account-agent usable before the portal skill catalog is fully populated and avoids making controller-rs part of the catalog query or execution path.

Portal UI

Endpoint Catalog And Tool Projection

The catalog UI should be endpoint-first but still show the tool projection that agents will see. It should let operators:

• browse api_endpoint_t rows by API, API version, endpoint, method, source, and active state,
• import or resync endpoint capabilities from OpenAPI, MCP tools/list, manually pasted MCP tools payloads, LightAPI descriptions, and selected gateway runtime surfaces,
• publish selected endpoint rows into tool_t as agent-invokable tools,
• generate or refresh tool_param_t rows from the logical input schema,
• see tool name, description, input schema, downstream endpoint, API type, semantic namespace, domain, personas, sensitivity tier, and runtime executable state,
• compare catalog metadata against source specs and current gateway tools/list,
• mark missing endpoint projections inactive,
• override operator-facing descriptions without changing gateway config,
• review and edit semantic routing metadata such as keywords, context requirements, fallback endpoint, priority weight, read-only, destructive, sensitive, or human-approval-required.

The first implementation should not depend only on live gateway access. It can import from the endpoint rows produced by API version parsing, including manual MCP tools/list JSON pasted into the API version spec field. Gateway tools/list should then be used to verify which imported projections are currently executable by a deployed gateway.

Skill Editor

The Skill Editor should let operators:

• create and update skill_t rows,
• write content_markdown instructions,
• link tools through skill_tool_t,
• set tool access level and per-skill config,
• preview which tools the skill would expose for a sample prompt,
• optionally link the skill to one or more workflow definitions,
• activate or deactivate skills.

Skill content should be short and operational. It should describe when to use the skill, how to interpret the tools, and any sequencing rules. It should not contain secrets.

Workflow-backed Skills

Some skills are only guidance plus a curated tool set. Other skills need a repeatable process that calls several tools, branches on results, waits for human input, runs assertions, or leaves an audit trail. Those skills should use light-workflow as the orchestration layer.

The boundary is:

Workflow-backed skills should be optional. Use a workflow when the skill represents a durable or regulated process, such as API onboarding, approval, validation, remediation, scheduled live testing, or a multi-step operation with clear checkpoints. Do not require workflow backing for simple skills that only guide an agent toward one tool call or open-ended exploration.

The workflow definition remains canonical in wf_definition_t.definition as YAML. The skill workspace should link to the definition through skill_workflow_t and should reuse the generic workflow editor described in Workflow Editor. The skill workspace can constrain the editor with skill context, but it should not implement its own workflow runtime.

For workflow-backed skills, skill_tool_t becomes the allowed tool set. A save-time validator should reject workflow steps that reference a gateway tool not linked to the skill, unless the step is explicitly marked as a future or external dependency. This keeps progressive disclosure, operator review, and workflow execution aligned.

Recommended Skill Workspace tabs:

Agent Skill Assignment

The Agent Skill Assignment UI should let operators:

• select an agent definition,
• assign one or more active skills through agent_skill_t,
• set priority and sequence,
• preview the final skill list for that agent,
• verify that each assigned skill still has at least one executable gateway tool.

Portal-query And Agent Cache Implementation

Catalog lookup should be implemented through the genai-query API. The agent should fetch the assigned active catalog, cache it locally, and run progressive disclosure search against the cache.

Phase 5 implements this for the Rust light-agent only. Other agent runtimes can adopt the same genai-query contract later, but they are not part of the Phase 5 implementation scope.

Initial algorithm:

1. Resolve host_id from the agent runtime configuration and the catalog request token. Resolve agent_def_id from LIGHT_AGENT_AGENT_DEF_ID or LIGHT_AGENT_API_VERSION_ID. Resolve service_id and env_tag from the registered Rust agent service config.
2. Call genai-query getEffectiveAgentCatalog.
3. The endpoint loads active agent_skill_t rows and linked active skill_t, skill_tool_t, tool_t, tool_param_t, and skill_workflow_t rows.
4. Build a nested effective catalog grouped by skill, with each skill carrying its mapped tools, schemas, endpoint identity, safety flags, and routing metadata.
5. Cache the effective catalog locally with catalogVersion and catalogHash.
6. During chat, macro-filter cached entries by agent persona, domain, namespace, sensitivity tier, active state, and available workflow context.
7. Rank cached entries by simple text matching over skill_t.name, skill_t.description, skill_t.content_markdown, tool_t.name, tool_t.description, endpoint name, endpoint description, and semantic keywords.
8. Intersect the final candidate list with gateway tools/list before exposing tool schemas to the LLM.

Controller cache management should invalidate this local cache when portal catalog data changes. After invalidation, the agent reloads from portal-query.

Later algorithm:

• Add vector search over skill_t.description_embedding and tool_t.description_embedding.
• Add vector search over endpoint semantic descriptions and LightAPI capability text.
• Include skill dependency expansion from skill_dependency_t.
• Use dependency mappings and fallback endpoints for chain planning, prefetch, and failure repair.
• Include inactive or missing-tool diagnostics for portal admin views, not for normal agent search.

Gateway Implementation

The gateway should keep the MCP data-plane contract stable:

• tools/list returns the executable tool set for the caller.
• tools/call routes by tool name to downstream MCP servers or REST services.
• Gateway policy remains authoritative at execution time.
• Gateway does not depend on skill_t or agent_skill_t to execute tools.

The gateway can expose an administrative sync endpoint later, but the first portal sync can call the existing MCP tools/list endpoint with an operator or service credential.

mcp-router.tools in values.yml should stay a runtime execution projection, not the full semantic registry. It should include the fields the gateway needs to list and call tools, plus safety metadata that must be enforced at runtime. Richer semantic routing metadata should stay in portal-query and the agent cache unless the gateway needs it for a concrete runtime policy decision.

Security Rules

• Skill assignment narrows what the agent should offer to the LLM, but it does not grant runtime authorization by itself.
• Gateway access control, endpoint scopes, OAuth token claims, and downstream service authorization still decide whether a tool call is allowed.
• Tool schemas and descriptions are not trusted input. They should be validated before storing and escaped when rendered.
• Skill content must not contain secrets, tokens, private keys, or passwords.
• A stale catalog row must not make a removed gateway tool executable.
• A stale local agent cache must be intersected with gateway tools/list before exposing tools to the LLM.
• Controller cache invalidation only forces reload; it does not grant access to catalog rows or executable tools.
• Sensitive or destructive tool metadata should be enforced by the gateway or a policy layer, not only by prompt instructions.
• Sensitivity tier must be checked before catalog disclosure. An agent without clearance for Restricted-PII should not receive the endpoint description or schema even if a skill references it.
• Context requirements are not only prompt hints. If required context is missing, the endpoint should be excluded or routed to an ask/workflow step that obtains the missing value.

Failure Handling

Phased Implementation

Phase 1: Preserve Direct MCP Baseline

• Keep agent tool execution through gateway tools/call.
• Remove the controller-rs skill/search placeholder before it becomes a dependency.
• Ensure agent falls back to gateway tools/list when no catalog cache is available.
• Keep direct gateway tools/list and tools/call working without portal skills.

Phase 2: API Endpoint Catalog Sync

• Add portal UI for endpoint-first import and resync.
• Use existing API version parsing to populate api_endpoint_t for OpenAPI and MCP tools, including manual MCP tools/list payloads accepted in the API version spec field.
• Sync all endpoint rows for the API version into the endpoint catalog. Do not limit the catalog to the endpoints currently selected for one gateway instance.
• Import or refresh LightAPI Description metadata for endpoint enrichment.
• Publish selected endpoint rows into tool_t as agent-facing tool projections.
• Generate tool_param_t from each endpoint’s logical input schema.
• Link every API-origin tool projection back to api_endpoint_t.endpoint_id.
• Store semantic routing metadata in indexed endpoint/tool fields and preserve the full metadata payload in api_endpoint_t.tool_metadata.
• If the first code slice only writes tool_metadata, keep that as a compatibility step and add the indexed routing-column migration before database-side macro-filtering or production vector ranking is enabled.
• Let users select which endpoint projections should be exposed to a specific gateway instance. This deployment selection is separate from endpoint catalog sync.
• Verify runtime executability outside persistence with gateway tools/list for the selected gateway instance when a gateway is reachable.
• Mark disappeared or non-executable projections inactive instead of deleting them.
• Add drift indicators for schema, description, safety metadata, and semantic routing metadata changes.

Phase 3: Skill Authoring

• Keep the existing skill_t CRUD page as the phase 3 authoring surface.
• Add skill-scoped category and tag assignment to the create/update skill forms. The UI should use dropdowns populated from the existing portal taxonomy where entity_type = 'skill'.
• Persist skill categories through entity_category_t and skill tags through entity_tag_t; do not add tags or categories columns to skill_t.
• Implement skill save as a composite command: one event updates the skill row and one taxonomy event replaces the selected category/tag associations for the same skill.
• Keep content_markdown as the instruction body. YAML or JSON skill files are import/export envelopes; if full structured skill authoring is introduced later, add a nullable JSONB skill-spec column beside content_markdown instead of replacing it.
• Keep embeddings optional.

Phase 3.5: Skill Workspace And Structured Authoring

• Add a richer Skill Workspace with Overview, Tools, Workflow, Preview, and Test tabs.
• Add tool linking workflows for skill_tool_t and formalize skill_tool_t.config for per-skill tool overrides.
• Add workflow-backed skill support through skill_workflow_t, with wf_definition_t.definition kept as the canonical workflow YAML.
• Reuse the generic Workflow Editor in the Workflow tab for YAML editing, step preview, validation, and test runs.
• Add validation that workflow tool-call steps reference tools linked to the skill through skill_tool_t.
• Add “create skill from LightAPI/tool” flows that can generate a draft skill, link relevant tools, and optionally create a starter workflow definition.
• Add YAML/JSON import/export for structured skill documents. Normalize YAML to JSON for storage when a persisted structured payload is needed, while keeping Markdown instructions in content_markdown.

Phase 4: Agent Assignment

• Add portal UI for agent_skill_t.
• Let operators assign active skills to agent definitions.
• Add an Agent Definition assignment entry point in addition to the existing agent_skill_t table page, so operators can manage assigned skills from the agent context.
• Add a batch assignment composite command that emits one AgentSkillCreatedEvent per selected skill.
• Add validation that assigned skills have at least one active direct skill_tool_t link. A workflow-backed skill does not satisfy this by having only skill_workflow_t; the workflow must use the skill’s linked tools.
• Enforce assignment validation in command handlers and mirror the same checks as UI preflight feedback.
• Treat sequence_id as the deterministic effective prompt/display order and priority as a ranking weight for later catalog/search behavior.

Phase 5: Real Skill Search

• Add the dedicated genai-query getEffectiveAgentCatalog endpoint with token verification against host, sid, and env claims.
• The endpoint returns the active nested catalog for one hostId + agentDefId + serviceId + envTag: agent metadata, assigned skills, tags, categories, skill config, mapped tools, tool params, routing/safety fields, workflow references, catalogVersion, and catalogHash.
• Implement the Rust light-agent portal-query client using that endpoint.
• Build and cache the nested effective catalog inside the Rust agent.
• Start with local macro-filtering and keyword matching over cached skills, endpoint metadata, and tool projections.
• Intersect selected catalog tool names with gateway tools/list; execute only through gateway tools/call.
• Wire controller cache-management invalidation to clear the Rust agent catalog cache. The next chat request lazily reloads from portal-query.
• If portal-query is unavailable or no agent definition ID is configured, the Rust agent falls back to direct gateway tools/list without portal catalog filtering.
• Add vector ranking after 384-dimensional embeddings are populated and combine it with semanticWeight.

Phase 6: Semantic Routing And Governance

• Support the Rust light-agent only. Other agent runtimes can adopt the same catalog and diagnostics contracts later.
• Use the normalized sensitivity tiers public, internal, confidential, and restricted. Treat missing or unknown tool tiers as internal.
• Enforce sensitivity-tier disclosure before portal-query returns the effective catalog to the agent. Tools blocked by policy are omitted from the returned tools list and surfaced as diagnostics for admin review.
• Block destructive or approval-required tools unless the skill/tool policy names an approval workflow. Until workflow-owned approval state exists, the current active row plus aggregate version remains the catalog versioning authority.
• Keep gateway tools/list and tools/call as the runtime source of truth. The Rust agent must still intersect catalog-selected tools with live gateway tools/list.
• Add Rust-agent diagnostics that compare the effective catalog against gateway tools/list at /diagnostics/tools, showing catalog tools missing from the gateway, gateway tools outside the catalog, and policy-blocked catalog tools.
• Enforce the same destructive, approval-required, and sensitivity metadata at the gateway before tools/call execution. A blocked call should include auditInfo fields and gateway debug/warn logs with the tool name, endpoint, tier, policy reason, and approval state.
• Do not write catalog-disclosure audit records into audit_log_t; it is reserved for workflow. Phase 6 uses auditInfo so the existing audit log file path captures blocked gateway decisions. A generic audit table can be added in a later governance phase if file logging is not enough.

Resolved Phase 2 Decisions

• Phase 2 endpoint catalog sync covers all endpoint rows for an API version. Gateway exposure is a separate step where users select which endpoint/tool projections to deploy to a specific gateway instance.
• Runtime verification means checking the selected gateway instance’s tools/list response to confirm that a deployed endpoint projection is executable there. It is not the same as endpoint catalog sync and should be implemented in the portal UI, deployment review flow, or diagnostics layer, not inside the persistence projection.
• Gateway exposure identity is hostId + serviceId + envTag. The token used for portal APIs must carry matching host, sid, and env claims.
• tool_t.implementation_type should be standardized and aligned with the LightAPI Description execution model. Endpoint-backed tools should use the standardized endpoint implementation type, with downstream protocol stored in endpoint and LightAPI metadata.
• High-use semantic routing fields should be indexed columns or indexed relationships, with the full structured payload preserved in api_endpoint_t.tool_metadata. JSON-only persistence is only an interim import/export-compatible shape for small catalogs or local-cache search.
• LightAPI Description enrichment requires an upstream parser or ingestion worker to emit enriched endpoint payloads. The persistence layer can store tool_schema, tool_metadata.routing, and tool_metadata.safety, but it does not derive those fields from the raw LightAPI document by itself.
• Endpoint category and tag classification should reuse the existing portal tag and category system.
• Embeddings should start at 384 dimensions to match the current VECTOR(384) schema. Use a provider abstraction so hosted OpenAI embeddings or local embedding services can be swapped without changing the catalog schema.
• genai-query should expose a dedicated effective catalog endpoint. Its token verification must match request host_id, service_id, and env_tag against token host, sid, and env claims.
• Cache reload behavior depends on the loading strategy. Startup-loaded catalogs should eagerly reload after invalidation. First-request-loaded catalogs can reload lazily on the next request.
• Phase 2 focuses on tool and endpoint metadata. Skill-specific metadata and per-skill tool config should be designed later with the skill authoring phase.
• Phase 3 uses the existing taxonomy join tables for skill tags and categories. Skill files may be YAML or JSON, but the database should keep content_markdown for the instruction body; a structured JSONB skill-spec column belongs in a later full authoring/import phase if it becomes needed.

Resolved Phase 3.5 Decisions

• Use light-workflow for workflow-backed skills that need durable multi-tool orchestration, approvals, assertions, retries, scheduled tests, or audit history.
• Do not force every skill into a workflow. Skills remain the discovery and guidance layer, and simple skills can stay instruction-and-tool based.
• Keep light-gateway as the runtime tool execution path. Workflow tasks that call tools should still use gateway-visible tool identities and should not bypass gateway policy.
• Keep workflow definitions in wf_definition_t.definition as YAML. Link skills to workflow definitions through skill_workflow_t instead of embedding workflow definitions in skill_t.
• Treat skill_tool_t as the allowed tool set for workflow-backed skills. Save-time validation should flag workflow tool calls that are not linked to the skill.
• Build the workflow authoring UI as a generic reusable editor first, then embed it inside the Skill Workspace with skill-aware reference filtering and validation.

Resolved Phase 4 Decisions

• An assignable skill must be active and must have at least one active direct tool link through skill_tool_t. Active skill_workflow_t rows are useful orchestration metadata, but they do not replace the direct allowed-tool set.
• Workflow-backed skill assignment should also rely on the Phase 3.5 validator: workflow tool-call references must resolve to tools linked through skill_tool_t.
• Validation must be enforced server-side by createAgentSkill, updateAgentSkill, and the batch assignment composite command. The Portal UI should run the same checks as preflight feedback, but UI checks are not authoritative.
• Keep the existing AgentSkill table page and add an Agent Definition assignment context so operators can assign and inspect skills from the agent they are configuring.
• Batch assignment should be a composite command that creates multiple AgentSkillCreatedEvent events from one request.
• sequence_id controls deterministic ordering when building the agent’s effective skill prompt/catalog. priority is reserved as a ranking weight for later effective-catalog and search behavior.
• Live gateway runtime executability checks are not part of Phase 4 persistence validation. Keep them as a diagnostics or governance item that compares cataloged/assigned tools with the selected gateway instance’s tools/list response before deployment or runtime enablement.

Recommendation

Implement this as a progressive control-plane enhancement. The gateway remains the execution path, and portal-authored skills become the agent guidance layer served by portal-query. The agent should cache the effective catalog locally and reload it after controller cache-management invalidation. This lets MCP tools work immediately through tools/list and tools/call, while still giving portal operators a clean path to organize tools into skills, assign those skills to agents, and improve retrieval over time.

Workflow Editor

Purpose

The Workflow Editor is the generic Portal authoring surface for light-workflow definitions. It should replace the raw textarea-only workflow definition experience with a structured editor that still preserves YAML as the canonical workflow definition stored in wf_definition_t.definition.

The editor is reusable. It can be opened from the Workflow Definition page, embedded in the Skill Workspace, or used by future task-specific authoring flows such as API onboarding, scheduled live tests, and remediation playbooks.

Design Boundary

light-workflow owns workflow execution, task state, retries, waiting human tasks, and audit events. The Portal editor authors definitions and starts test runs, but it must not implement its own workflow runtime.

The gateway remains the runtime tool execution path. Workflow steps that invoke tools should reference gateway-visible tools or endpoint descriptions and then execute through the same runtime path used by agents.

The editor should not duplicate endpoint contracts. API, MCP, JSON-RPC, gRPC, and other endpoint details belong in LightAPI descriptions, OpenAPI/OpenRPC documents, protobuf metadata, or the portal endpoint catalog. Workflow tasks reference those descriptions and provide step-level wiring, guards, exports, and error handling.

Current State

The current Portal implementation already has the persistence and generic CRUD surface needed for a first editor:

• wf_definition_t stores namespace, name, version, and definition.
• workflow-command exposes create, update, delete, and start workflow commands.
• workflow-query exposes workflow definition reads.
• portal-view has a Workflow Definition table and generic create/update forms whose definition field is a YAML textarea.

The first Workflow Editor can therefore be an incremental UI improvement over the existing definition CRUD and start workflow command.

Goals

• Keep workflow YAML as the canonical persisted artifact.
• Provide a readable step outline or graph next to the YAML editor.
• Validate definitions before save and before test runs.
• Let users discover and reference endpoint descriptions, gateway tools, skills, rules, and human task types from a side panel.
• Support workflow definition create, update, import, export, and start-test flows.
• Make the editor embeddable so skill authoring can use the same workflow authoring component with skill-specific constraints.
• Preserve owner scoping and existing Portal command/query conventions.

Non-Goals

• Do not execute workflow logic in Portal View.
• Do not make skills the workflow runtime.
• Do not store workflow YAML in skill_t.
• Do not require a visual drag-and-drop graph before the editor is useful.
• Do not copy full API contracts into workflow steps when endpoint descriptions can be referenced.
• Do not fork or embed the Apache KIE Serverless Logic Web Tools as the first implementation path. They are useful reference material for CNCF Serverless Workflow concepts, but they are tightly coupled to the strict upstream spec and would be expensive to adapt for Light-Fabric agentic extensions.

Authoring Model

The editor should maintain two synchronized representations:

All saves should serialize from the YAML source or from a parsed model that round-trips to the same specification format. If the visual editor changes a step, it should update the YAML and keep the YAML visible.

The editor should support progressive enhancement:

1. YAML editor plus parsed step outline.
2. Step palette and property panel that edit YAML safely.
3. Read-only graph preview.
4. Drag-and-drop graph editing once round-trip behavior is reliable.

Implementation Architecture

The recommended implementation is a custom React editor built from focused building blocks:

The workflow YAML or JSON document remains the source of truth. CodeMirror edits parse into the editor store. The parsed workflow model is then projected into React Flow nodes and edges. React Flow edits update the same model and then serialize back to the YAML document.

This avoids adding a second large browser editor runtime to portal-view, which already uses CodeMirror for Markdown and OpenAPI JSON/YAML editing. It also avoids fighting a visualizer that only understands the strict CNCF Serverless Workflow schema, while still letting Portal define first-class visual treatments for Light-Fabric task types such as agent, mcp, ask, assert, rule, switch, and future LLM or approval-oriented steps.

CodeMirror should use a custom JSON Schema derived from the CNCF Serverless Workflow schema plus Light-Fabric agentic extensions. For JSON definitions, use a CodeMirror 6 JSON Schema integration such as codemirror-json-schema to provide linting, autocomplete, and hover details. For YAML definitions, reuse the existing portal-view CodeMirror YAML setup where possible and add schema validation through a YAML language-server bridge or equivalent worker-backed integration. The goal is Monaco-like schema assistance without Monaco’s bundle cost.

React Flow should not own the persisted shape. It owns layout, selection, edge creation, and node interaction. The persisted workflow definition should remain independent of the canvas library so a future editor or CLI can read the same definitions.

Recommended sync behavior:

1. Parse CodeMirror content into a typed workflow model when the YAML is valid.
2. Preserve text edits and show problems when YAML is invalid; do not destroy the user’s in-progress text.
3. Project valid workflow models to React Flow nodes and edges.
4. Let graph edge changes update transition targets in the model.
5. Let property-panel changes update the model through schema-aware controls.
6. Serialize model changes back into the YAML document using stable formatting.
7. Keep conflict handling explicit when source edits and graph edits race.

Mermaid can be used for documentation or a lightweight read-only preview, but it is not the long-term authoring surface. JSONForms can be useful inside property panels, but it should not replace the graph/source editor combination.

Layout

Recommended first layout:

The generic Workflow Definition page can use the full layout. The Skill Workspace can embed the same editor with a narrower reference scope and a skill-aware validation profile.

Step Palette

The editor should understand the task types defined by the Light-Fabric agentic workflow design:

The palette should create minimal valid YAML fragments. Users can then edit the full YAML when advanced options are needed.

Reference Panel

The editor should help authors reference existing catalog objects instead of typing fragile identifiers by hand:

• workflow definitions and versions,
• LightAPI endpoint descriptions,
• API endpoints and tool projections,
• gateway-visible MCP tools,
• rule definitions,
• agent definitions,
• skills and skill-linked tools when the editor is embedded in the Skill Workspace.

For generic workflow authoring, the reference panel can show all objects the current user is allowed to read. For skill authoring, it should filter tools to the skill’s linked tools and flag references outside that set.

Validation

Validation should run in layers:

Runtime diagnostics should be separate from persistence validation. A workflow definition can be saved before a gateway is reachable, but the editor should make missing runtime executability visible before test or deployment.

Test Runner

The editor should support a test panel that starts a workflow instance through the existing workflow start command and then reads instance events and task state through the workflow query APIs.

The test panel should support:

• JSON workflow input,
• start run,
• event stream or polling view,
• current context and output preview,
• waiting task completion for ask or approval steps,
• assertion and rule failure display,
• gateway or endpoint call failure display,
• rerun with the same input.

The test runner is a client of light-workflow; it does not execute workflow steps in the browser.

Skill Workspace Integration

Phase 3.5 skill authoring should embed the Workflow Editor rather than create a second skill-specific workflow UI.

Recommended integration:

1. The Skill Workspace has a Workflow tab.
2. The tab lets the user choose none or workflow-backed.
3. In workflow-backed mode, the user can select an existing workflow definition or create a draft definition.
4. The link is stored in skill_workflow_t.
5. The editor reference panel filters tool references to the tools linked by skill_tool_t.
6. Validation rejects or warns on workflow tool calls not present in the skill’s allowed tool set.
7. The Test tab starts the linked workflow with sample JSON input and displays the same workflow events used by the generic editor.

This keeps the skill as a discovery and guidance artifact while light-workflow owns deterministic orchestration.

Data And API Changes

The first generic editor can reuse existing workflow definition APIs. Later phases should add editor-friendly endpoints only when they remove real UI complexity.

Phase B adds the validation endpoint and keeps the reference catalog composed from existing read models. A single combined catalog endpoint remains optional if the multiple list queries become noisy or slow.

Server-side validation should be authoritative. Client-side validation is useful for responsiveness but should not be the only guard before saving or testing a workflow definition.

Phased Implementation

Phase A: Structured YAML Editor

• Add a generic Workflow Editor component and route.
• Replace create/update workflow definition textarea navigation with the editor where practical.
• Keep YAML visible and canonical.
• Reuse the existing portal-view CodeMirror editor stack with the Light-Fabric workflow schema for YAML/JSON validation, autocomplete, hover help, folding, and parse markers.
• Parse YAML client-side to render a step outline and problems panel.
• Add import/export and basic validation before save.

Phase B: Catalog-Aware Authoring

• Add a reference panel for endpoint descriptions, tools, rules, agents, and workflow definitions.
• Add a step palette that inserts valid YAML snippets.
• Add schema-backed property panels for selected steps. Use dropdowns for catalog references and constrained enums instead of free-text fields where Portal already has authoritative labels.
• Add server-side validation through validateWfDefinition.
• Add runtime diagnostics that compare MCP tool references with gateway tools/list or the Rust agent /diagnostics/tools endpoint when a gateway target is selected.

Phase C: Test And Worklist Integration

• Add a test runner panel backed by light-workflow start and query APIs.
• Show workflow events, current task state, waiting human tasks, assertions, and final output.
• Let users complete ask tasks from the test panel.
• Link failed test runs to remediation tasks or worklist entries.

Phase C uses the existing Portal workflow command/query boundary. The editor starts a test run through workflow/startWorkflow, then refreshes getProcessInfo, getTaskInfo, getTaskAsst, getWorklist, and getAuditLog for the returned wfInstanceId. The test panel completes a waiting human task through workflow/completeTask, which preserves the structured response in the event data and materializes the task as completed through the existing TaskInfoUpdatedEvent projection.

The panel should expose remediation links instead of silently creating production work. Failed process or task rows can open a prefilled remediation task form, and task assignments can jump to the workflow worklist with the current workflow instance context.

Phase D: Visual Graph Editing

• Add a React Flow graph preview after the outline is stable.
• Represent Light-Fabric task types with custom React Flow nodes and explicit transition edges.
• Add drag-and-drop graph editing only after YAML/model round-trip behavior is reliable.
• Keep YAML as the source of truth even when visual editing is enabled.

Phase D adds the graph as a projection of the parsed YAML model, not a separate persisted representation. The graph reads steps, tasks, states, or do containers and renders one custom React Flow node per detected step. Node styling reflects the Light-Fabric task type, and the graph can overlay runtime task status from the Phase C test-run read models when the workflow task id matches a graph step id.

Explicit transition fields such as next, then, to, and transition become solid graph edges. Ordered fallback edges are shown as dashed edges so authors can distinguish model transitions from inferred sequence. Creating an edge in React Flow updates the source step’s transition in YAML, and deleting an explicit edge removes that transition target from YAML. Dragging nodes changes only the authoring layout in the browser session; it does not mutate the saved workflow definition.

The graph must continue to tolerate partial or invalid authoring states. If the YAML cannot be parsed into a known workflow container, the editor keeps the source editor and validation panels usable and shows an empty graph state rather than blocking authoring.

Recommendation

Build the generic Workflow Editor before the Skill Workspace embeds workflow authoring. The skill UI should provide context and constraints, while the workflow editor provides YAML editing, step preview, validation, and test runs for every workflow authoring use case in Portal.

Portal Catalog Scope

Problem

Light Portal supports multiple tenants through host_id and can also host multiple runtime environments in one portal instance. A common deployment shape is:

Within an organization or a cloud deployment, operators need a catalog for APIs, API endpoints, tools, skills, schemas, rules, workflows, categories, and tags. Some catalog entries are reusable platform knowledge. Other entries are tenant-owned, environment-bound, or tied to a concrete gateway deployment.

The main design question is whether Light Portal should clone catalog rows into every host/tenant, or maintain one shared catalog per portal instance and expose it through a separate single page application and virtual host.

The recommended answer is neither full cloning nor a UI-only split. The portal should model catalog scope explicitly:

• shared catalog definitions use global scope,
• tenant-specific definitions and overrides use host scope,
• environment-specific runtime bindings use host plus environment scope,
• a separate SPA may expose the same backend catalog, but it should not become the catalog authority.

Goals

• Avoid duplicating the full catalog for every tenant.
• Prevent catalog drift between tenants and between portal instances.
• Preserve tenant isolation for private APIs, private skills, secrets, access control, and runtime bindings.
• Let dev and sit share one portal instance while still keeping their runtime endpoint targets separate.
• Let stg and prd share another portal instance while keeping production controls stricter.
• Support an effective catalog query that combines global definitions with host-specific rows and environment-specific bindings.
• Reuse existing portal-query APIs and the genai-query catalog direction for agent-facing skills and tools.
• Keep light-gateway as the runtime MCP execution path for tools/list and tools/call.
• Support promotion or import/export between portal instances instead of relying on ad hoc row copies.

Non-Goals

• Do not clone every global catalog row into every tenant by default.
• Do not make a separate SPA the source of truth for catalog data.
• Do not bypass host-scoped authorization just because a catalog item is global.
• Do not put secrets, client credentials, runtime tokens, or deployment state in global catalog rows.
• Do not move MCP tool execution from light-gateway into portal-query, controller-rs, or the catalog UI.
• Do not require every MCP or API endpoint to be wrapped in a skill before the gateway can expose it as a runtime tool.

Current Model

The database already contains both global-capable and host-scoped patterns.

category_t and tag_t have nullable host_id. A null host_id means the category or tag is global. A non-null host_id means the row belongs to one host. Their unique indexes already separate global uniqueness from host-specific uniqueness.

The query behavior for category and tag labels returns both host-specific rows and global rows for a host. This is the right shape for taxonomy and catalog organization metadata.

Other catalog entities are currently host-scoped:

• api_t
• api_version_t
• api_endpoint_t
• agent_definition_t
• skill_t
• tool_t
• tool_param_t
• agent_skill_t
• skill_tool_t
• skill_dependency_t

Those tables use host_id NOT NULL and most query paths filter by host_id = ?. This is correct for private tenant data and runtime-bound data, but it is too narrow for reusable platform catalog definitions if the only sharing mechanism is row replication.

Design Decision

Use a scoped catalog inside Light Portal.

The portal backend remains the source of truth. The catalog UI can be part of the existing portal SPA or exposed through another SPA/virtual host, but both UI surfaces must read and write through the same portal-query and command APIs.

The durable model is:

global catalog definition
  -> host enablement or host override
    -> environment runtime binding

This model allows one shared definition for reusable knowledge and separate tenant or environment controls where isolation matters.

Scope Types

Global rows are reusable definitions. Host rows are ownership and isolation. Environment rows are runtime selection.

Catalog Entity Guidance

Effective Catalog

Consumers should not need to manually merge global and host rows. Portal-query should expose an effective catalog read model for each host and runtime context.

The effective catalog request should include:

• hostId
• serviceId when the catalog is for a gateway, agent, or runtime service
• envTag when the result is environment-specific
• optional agentDefId when the result is for an agent
• optional filters for entity type, category, tag, protocol, routing domain, or capability

The effective catalog response should include:

• global definitions visible to the caller,
• host-specific definitions visible to the caller,
• host overrides that shadow global defaults,
• environment bindings for the requested envTag,
• active state and catalog version or freshness metadata,
• category and tag labels from both global and host-specific taxonomy rows,
• enough provenance to show whether a row came from global scope, host scope, or an environment binding.

Recommended precedence:

environment binding > host override > global definition

This keeps shared definitions stable while allowing host and environment customization.

Data Model Direction

For tables that already support nullable host_id, keep the current pattern:

host_id IS NULL  -> global/shared row
host_id = ?      -> host-specific row

For strictly host-scoped catalog tables, do not simply make every host_id nullable without checking foreign keys and runtime assumptions. Some tables are correctly host-scoped because they point to tenant-owned APIs, credentials, gateway endpoints, or agent assignments.

Use one of these patterns per entity:

1. Nullable host_id on the definition table when the entity can safely be global and all references can resolve global plus host rows.
2. Separate template and binding tables when the definition is global but enablement is tenant-specific.
3. Keep the current host-scoped table when the entity is inherently tenant or runtime bound.

For reusable skills and tools, the safest long-term shape is template plus binding:

catalog_skill_template_t
  -> host_skill_t or skill_t host override
    -> agent_skill_t assignment

catalog_tool_template_t
  -> host tool projection
    -> skill_tool_t mapping
    -> gateway runtime tools/list verification

If the implementation starts smaller, it can add nullable global scope to selected catalog definition tables first, but the query contract must still return the effective catalog and indicate scope provenance.

Separate SPA Or Virtual Host

A separate SPA deployed with LightAPI and sign-in as another BFF virtual host is useful as a catalog presentation surface. It can provide a marketplace-style view for shared APIs, tools, skills, schemas, rules, and workflows.

It should not own separate catalog state.

Recommended use:

• browse global catalog definitions,
• request enablement for a host,
• compare host overrides with global definitions,
• review environment bindings,
• publish or promote catalog versions between portal instances.

Avoid using the separate SPA to bypass tenant-aware portal APIs. The BFF should still pass authenticated requests to portal-query or command APIs, and those APIs must enforce host, service, environment, and role checks.

Environment Handling

Within one portal instance, environments should be runtime bindings, not cloned catalog universes.

For a dev/sit instance:

• one shared catalog can describe a capability,
• dev and sit get separate env_tag bindings,
• runtime endpoints can differ through target_host, service_id, instance, deployment, or gateway registration,
• a tool can be visible in both environments but executable only where the gateway lists it.

For a stg/prd instance:

• stg and prd can share approved global definitions,
• production enablement should require stricter workflow or authorization,
• secrets, tokens, OAuth clients, runtime instances, and deployment state remain environment-specific,
• catalog promotion into prd should preserve stable IDs and versions.

Promotion Between Portal Instances

The boundary between dev/sit and stg/prd is an instance boundary. Treat it as a promotion boundary, not as live replication between tenants.

Recommended promotion flow:

1. Author or import catalog definitions in the lower portal instance.
2. Review and approve the global or host-scoped definitions.
3. Export selected catalog rows with their versions and dependencies.
4. Import into the target portal instance.
5. Resolve environment bindings for stg or prd.
6. Verify runtime exposure through the selected light-gateway tools/list.
7. Activate the target bindings.

Promotion should be idempotent. A repeated import of the same catalog version should update or confirm the same target definition instead of creating duplicates.

Security And Authorization

Global catalog visibility does not mean global execution permission.

Authorization must be checked at these layers:

• portal UI and BFF authentication,
• portal-query read authorization,
• command API write authorization,
• host and environment claim matching,
• category/tag visibility when private taxonomy is used,
• gateway tools/list availability,
• gateway tools/call policy,
• downstream service authorization.

For runtime catalog reads used by gateways and agents, the token should include host, sid, and, when environment-specific data is requested, env. The query handler should compare those claims with the requested hostId, serviceId, and envTag.

UI Guidance

The portal UI should show catalog scope explicitly:

• Global
• Host
• Environment

For list pages, include filters for scope, environment, category, tag, active state, and source protocol. For detail pages, show whether a host row inherits from a global definition, overrides it, or is private to the host.

For destructive changes, make the target scope clear. Updating a global catalog definition can affect many hosts, while updating a host override should affect only that host.

Migration Approach

1. Keep the existing category and tag nullable host_id behavior.
2. Add effective catalog read APIs before broad schema changes so callers have a stable contract.
3. Identify which catalog entities need global definitions versus host-only rows.
4. Add template or nullable-scope tables for reusable definitions.
5. Add host enablement or override tables for tenant-specific activation.
6. Add environment binding views or APIs for dev, sit, stg, and prd.
7. Add import/export or snapshot support for promotion between portal instances.
8. Update portal-view to expose scope and provenance.
9. Keep existing host-scoped APIs working during the migration.

Open Questions

• Should global reusable skills and tools use nullable host_id in the existing tables, or separate template tables with host bindings?
• Which catalog entities require approval workflow before production activation?
• Should category and tag assignment tables store additional scope metadata, or is scope fully inherited from the referenced category or tag?
• What stable external identity should be used during cross-instance catalog promotion when UUIDs differ between portal databases?
• Should portal-query expose one broad effective catalog endpoint or multiple entity-specific effective endpoints?

OAuth Kafka

Token Exchange

This document outlines the design decisions and implementation details for supporting multiple token exchange flows in the oauth-kafka module.

Comparison of Detection Methods

When implementing token exchange (RFC 8693), the server must determine which identity provider (IdP) issued the subject_token to verify it correctly and map claims.

Implementation Strategy

Our implementation in ProviderIdTokenPostHandler uses Option 4: Client Context as the primary strategy:

1. Database-Driven Configuration: A new column token_ex_type has been added to the auth_client_t table to specify the supported exchange type for each client.

   ALTER TABLE auth_client_t ADD COLUMN token_ex_type VARCHAR(64);
   

2. Supported Exchange Types:
   • msal: Microsoft Authentication Library based exchange.
   • ccac: Client Credentials to Authorization Code exchange.
3. Flow Determination: Instead of relying on client-supplied parameters like requested_token_type, the server retrieves the token_ex_type from the client context in the database to decide which handler to use. This ensures that only authorized exchange types are performed for each specific client.

Recommendation

For the light-portal ecosystem:

• Option 4: Client Context is the selected method. It provides the highest level of security by ensuring that token exchange flows are explicitly configured and restricted on a per-client basis in the database.
• token_ex_type should be populated for any client that requires token exchange functionality. Clients without this configuration will not be allowed to perform token exchange.

Future Considerations

• Implement automated issuer discovery if the number of external providers grows.
• Support “opaque” token exchange by integrating with introspection endpoints of external IdPs.
• Extend the auth_client_t configuration to support multiple allowed exchange types per client if needed.

OAuth Audit

The OAuth services keep authorization codes and refresh tokens as operational state. These rows are short lived and are now written directly to auth_code_t and auth_refresh_token_t instead of being created through the general event store. This avoids high-volume login and refresh-token churn in event_store_t and outbox_message_t.

Audit and login history are recorded separately in append-oriented OAuth audit tables.

Goals

• Show administrators who is currently online.
• Show a user the last login time and session history.
• Track refresh-token rotation and rejected refresh attempts.
• Preserve enough history for support and security review without storing raw secrets.
• Keep the hot login and token-refresh path simple and transactional.

Tables

auth_session_t stores one row per login session. It is the current and historical session summary.

• session_id identifies the browser/device session.
• login_ts, last_refresh_ts, logout_ts, and expires_ts describe the session lifetime.
• status is ACTIVE, LOGGED_OUT, EXPIRED, or REVOKED.
• refresh_count is incremented on each successful refresh-token rotation.
• ip_address, user_agent, and device_id are optional request context fields.

auth_session_audit_t stores append-only auth audit entries.

• LOGIN_SUCCEEDED
• LOGIN_FAILED
• AUTH_CODE_ISSUED
• AUTH_CODE_CONSUMED
• REFRESH_TOKEN_ISSUED
• REFRESH_TOKEN_ROTATED
• REFRESH_TOKEN_REJECTED
• LOGOUT
• SESSION_EXPIRED
• SESSION_REVOKED

auth_refresh_token_t.session_id links the currently valid refresh token to the session that owns it. This removes ambiguity when the same user is logged in from multiple browsers or devices.

Audit rows keep session_id as data, but do not use a hard foreign key to auth_session_t. Audit history must remain groupable by session even if operational session rows are later archived or removed.

Login Flow

When /oauth2/{providerId}/code authenticates the user:

1. Insert the authorization code into auth_code_t.
2. Insert an ACTIVE session into auth_session_t.
3. Insert LOGIN_SUCCEEDED and AUTH_CODE_ISSUED audit rows.
4. Include the session_id in the auth code row so the token exchange can attach the refresh token to the same session.

Failed logins write LOGIN_FAILED with the available host, provider, client, request metadata, and failure reason.

Authorization Code Exchange

When grant_type=authorization_code succeeds:

1. Delete the consumed auth code from auth_code_t.
2. Insert the refresh token into auth_refresh_token_t with the auth code’s session_id.
3. Insert AUTH_CODE_CONSUMED and REFRESH_TOKEN_ISSUED audit rows.

Refresh Token Rotation

When grant_type=refresh_token succeeds, the service performs one transaction:

1. Insert the replacement refresh token.
2. Delete the previous refresh token with its expected aggregate version.
3. Update auth_session_t.last_refresh_ts and increment refresh_count.
4. Insert REFRESH_TOKEN_ROTATED with the old and new token ids.

If a refresh token is missing, invalid, or belongs to the wrong client, the service writes REFRESH_TOKEN_REJECTED when enough context is available. Raw refresh-token values must not be stored in audit metadata.

Admin Revocation

Administrators can kick out a user by revoking the user’s current refresh token. Operationally, deleting the refresh token is enough to stop the session from renewing once the current access token expires. The audit/session model adds explicit session state to that behavior.

The revocation operation must run as one transaction:

1. Find the refresh token row and its session_id.
2. Delete the refresh token from auth_refresh_token_t.
3. Update auth_session_t:
   • status = 'REVOKED'
   • logout_ts = CURRENT_TIMESTAMP
   • end_reason = 'ADMIN_REVOKED'
4. Insert SESSION_REVOKED into auth_session_audit_t.

The database patch provides revoke_auth_session_by_refresh_token(host_id, refresh_token, admin_user, reason) for this workflow. Admin screens should call the revoke operation instead of issuing a plain refresh-token delete when the intent is to kick out a logged-in user.

If the refresh token has no session_id, the operation still deletes the token and returns NULL. This preserves backward compatibility with refresh-token rows created before session tracking.

Admin Queries

Current online users:

SELECT *
FROM auth_session_t
WHERE status = 'ACTIVE'
  AND (expires_ts IS NULL OR expires_ts > CURRENT_TIMESTAMP);

User login history:

SELECT *
FROM auth_session_t
WHERE host_id = $1
  AND user_id = $2
ORDER BY login_ts DESC;

Session duration:

SELECT
    login_ts,
    COALESCE(logout_ts, last_refresh_ts, CURRENT_TIMESTAMP) - login_ts AS duration
FROM auth_session_t
WHERE host_id = $1
  AND session_id = $2;

Retention

auth_session_t can be retained longer than operational token tables. auth_session_audit_t should use a retention policy appropriate for the deployment, for example 90 days or one year. Retention jobs should delete audit rows by event_ts and optionally archive them before deletion.

Multi-Tenant

Database Schema

Adding a host_id to every table is one approach, but it does lead to composite primary keys and can impact performance. Using UUIDs as primary keys, even in a multi-tenant environment, is another viable option with its own set of trade-offs. Let’s examine both strategies:

1. Host ID on Every Table (Composite Primary Keys)

Schema: Each table would have a host_id column, and the primary key would be a combination of host_id and another unique identifier (e.g., user_id, endpoint_id).

CREATE TABLE user_t (
    host_id UUID NOT NULL,  -- References hosts table
    user_id INT NOT NULL, 
    -- ... other columns
    PRIMARY KEY (host_id, user_id),
    FOREIGN KEY (host_id) REFERENCES hosts_t(host_id)
);

Pros:

• Data Isolation: Clear separation of data at the database level. Easy to query data for a specific tenant.

• Backup/Restore: Simplified backup and restore procedures for individual tenants.

Cons:

• Composite Primary Keys: Can lead to more complex queries, especially joins, as you always need to include the host_id. Can affect query optimizer performance.

• Storage Overhead: host_id is repeated in every row of every table, adding storage overhead.

• Index Impact: Composite indexes can sometimes be less efficient than single-column indexes.

2. UUIDs as Primary Keys (Shared Tables)

Schema: Tables use UUIDs as primary keys. A separate table (tenant_resources_t) maps UUIDs to tenants.

CREATE TABLE user_t (
    user_id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    -- ... other columns
);


CREATE TABLE tenant_resource_t(
    host_id UUID NOT NULL,
    resource_type varchar(255) NOT NULL, --e.g., 'user', 'api_endpoint'
    resource_id UUID NOT NULL,
    PRIMARY KEY(host_id, resource_type, resource_id),
    FOREIGN KEY (host_id) REFERENCES hosts_t(host_id)
);

Pros:

• Simplified Primary Keys: Easier to manage single-column UUID primary keys. Simpler joins.

• Reduced Storage Overhead: No need to repeat host_id in every table.

• Application Logic: Multi-tenancy is handled mostly in the application logic by querying tenant_resources_t to ensure a user belongs to the correct tenant, adding a layer of flexibility. (This is also a con if not carefully implemented.)

Cons:

• Data Isolation (slightly reduced): Data is logically separated but resides in shared tables. Robust application logic is essential to prevent data leakage between tenants.

• Backup/Restore (more complex): Backing up/restoring for a single tenant requires filtering based on the tenant_resources_t table.

• Query Performance (potential issue): Queries might require joining with tenant_resources_t table which will add a bit overhead. Proper indexing and query optimization become crucial.

3. Hybrid Approach (Recommended in many cases)

• Combine the strengths of both approaches.

• Use UUIDs as primary keys for most tables for simplicity.

• Add host_id to tables where data isolation is paramount (e.g., tables containing sensitive financial or personal data) or where frequent tenant-specific queries are performed. This provides a more granular level of control and allows for optimized queries when needed.

• Use the tenant_resources_t table to maintain an overview of which resources belong to which tenant, supporting the application logic’s multi-tenancy enforcement.

Choosing the Right Strategy:

• Small Number of Tenants, Low Data Volume: UUIDs might be sufficient.

• Strict Data Isolation Requirements, Frequent Tenant-Specific Queries, High Data Volume: host_id on every relevant table or a hybrid approach is often preferable.

• Flexibility in Multi-Tenancy Implementation: Favors the UUID approach or Hybrid approach, with logic predominantly handled in the application layer.

Additional Considerations:

• Database Support: Check if database (PostgreSQL) has specific features for multi-tenancy.

• Indexing: Carefully plan indexes for optimal performance in any multi-tenant scenario.

• Schema Design: Consider creating views for tenant-specific data access to simplify queries in the application layer.

• Security: Implement robust security measures to prevent data leakage between tenants, regardless of the chosen approach.

The hybrid approach usually provides the best balance between performance, data isolation, and development complexity in real-world multi-tenant applications. You have more direct control where strict isolation is required and can maintain the simplicity of UUIDs where it’s beneficial. Using tenant_resources_t for application logic enforcement offers flexibility and a central point of management for tenant resource association. This approach also prepares the application for potential future expansion and different multi-tenancy demands as the application grows.

Should host_id part of the PK

This is a classic multi-tenancy design question. Both approaches have implications. Let’s analyze them:

Option 1: Current Approach - PK (host_id, instance_id)

• Pros:
  • Excellent for Tenant-Specific Queries: This is the biggest advantage. Queries like SELECT ... FROM instance_t WHERE host_id = ? AND ... or SELECT ... FROM instance_t WHERE host_id = ? ORDER BY instance_id can directly and efficiently use the primary key index. The index is naturally ordered by tenant first, then by instance within that tenant. This improves data locality for a specific tenant’s data.
  • Clear Logical Grouping: The primary key explicitly represents the concept that an instance belongs to a specific host (tenant).
  • Enforces Uniqueness Per Tenant: Guarantees that instance_id is unique within a given host_id. (Although UUIDv7 makes global collisions highly unlikely anyway).
• Cons:
  • Wider Primary Key: The PK is 32 bytes (16+16).
  • Wider Foreign Keys: Any table referencing instance_t would need both host_id and instance_id as its foreign key columns.
  • Slightly Larger Secondary Indexes: Other indexes on instance_t will implicitly include both PK columns, making them slightly larger than if the PK was just 16 bytes.

Option 2: Alternative - PK (instance_id)

• Pre-requisite: This only works if your application guarantees that instance_id is globally unique across all hosts/tenants. Given you’re using UUIDv7, this is a safe assumption in practice, but the schema wouldn’t enforce uniqueness per host explicitly via the PK itself.
• Pros:
  • Narrower Primary Key: The PK is only 16 bytes.
  • Simpler Foreign Keys: Tables referencing instance_t only need a single instance_id column for the foreign key.
  • Slightly Smaller Secondary Indexes: Other indexes on the table will be marginally smaller.
• Cons:
  • Requires Separate Index for Tenant Queries: You would absolutely need a separate index on (host_id, instance_id) (or at least (host_id)) for efficient tenant-specific queries (WHERE host_id = ?). Without it, querying for a specific tenant’s data would require less efficient scans. This index would likely be a UNIQUE index anyway to enforce the logical relationship: CREATE UNIQUE INDEX instance_t_host_instance_idx ON instance_t (host_id, instance_id);
  • Potential Reduced Locality: While the separate index helps, the primary key index itself (based only on instance_id) might interleave data from different tenants physically, potentially slightly reducing cache efficiency for queries scanning many instances for a single tenant compared to the composite PK approach.

Recommendation:

Stick with the composite primary key: PRIMARY KEY(host_id, instance_id).

Reasoning:

1. Performance for Core Use Case: In multi-tenant systems, filtering by the tenant identifier (host_id) is almost always the primary access pattern. Having host_id as the leading column in the PK index directly optimizes this critical path.
2. Index Necessity: Even if you chose instance_id as the sole PK, you would still need to create an index on (host_id, instance_id) for performance. Making this essential index the primary key index is often the most straightforward and efficient approach.
3. Clarity: The composite key clearly reflects the logical relationship and ownership.
4. Cost: The “cost” of a 32-byte PK vs. a 16-byte PK is often negligible compared to the performance gains achieved by aligning the PK index with the dominant query patterns in a multi-tenant architecture. The impact on FKs and secondary indexes is real but usually acceptable.

Using just instance_id as the PK prioritizes global uniqueness and FK simplicity over optimizing tenant-specific queries directly via the PK index. In most multi-tenant scenarios, optimizing tenant queries is more important.

Citus PostgreSQL Extension

Citus, now fully integrated into PostgreSQL as a distributed database extension, can be very helpful in scaling your multi-tenant application, especially if you anticipate significant data growth and high query loads. Here’s how Citus can fit into your use case and the factors to consider:

How Citus Helps:

• Horizontal Scalability: Citus allows you to distribute the data across multiple PostgreSQL nodes (servers), enabling horizontal scaling. This is crucial for handling increasing data volumes and query loads in a multi-tenant environment.

• Improved Query Performance: By distributing data and queries, Citus can significantly improve the performance of many types of queries, especially analytical queries that operate on large datasets. This is particularly beneficial if we have tenants with substantially different data volumes or query patterns.

• Shard Placement by Tenant: One of the most effective ways to use Citus for multi-tenancy is to shard the data by host_id (or a tenant ID). This means that all data for a given tenant resides on the same shard (a subset of the distributed database). This allows for efficient tenant isolation and simplifies queries for tenant-specific data.

• Simplified Multi-Tenant Queries: When sharding by tenant, queries that filter by host_id become very efficient because Citus can route them directly to the appropriate shard. This eliminates the need for expensive scans across the entire database.

• Flexibility: Citus supports various sharding strategies, allowing you to choose the best approach for the data and query patterns. You can even use a hybrid approach, distributing some tables while keeping others replicated across all nodes for faster access to shared data.

Example (Sharding by Tenant):

Create a distributed table: When creating tables (e.g., user_t, api_endpoint_t, etc.), we would declare them as distributed tables in Citus, using the host_id as the distribution column:

CREATE TABLE user_t (
    host_id UUID NOT NULL,
    user_id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    -- ... other columns
) DISTRIBUTE BY HASH (host_id);

Querying: When querying data for a specific tenant, include the host_id in the WHERE clause:

SELECT * FROM users_t WHERE host_id = 'your-tenant-id';

Citus will automatically route this query to the shard containing the data for that tenant, resulting in much faster query execution.

Citus Cost:

• Citus Open Source: The Citus open-source extension is free to use and is included in the PostgreSQL distribution. We can self-host and manage it.

• Azure CosmosDB for PostgreSQL (Managed Citus): Microsoft offers a fully managed cloud service called Azure CosmosDB for PostgreSQL, which is built on Citus. This service has usage-based pricing, and the cost depends on factors like the number of nodes, storage, and compute resources used. This managed option reduces the operational overhead of managing Citus yourself.

Recommendation:

Don’t automatically add host_id to every table just because we are using Citus. Carefully analyze the data model, query patterns, and multi-tenancy requirements.

• Distribute tables by host_id (tenant ID) when data locality and isolation are paramount, and we want to optimize tenant-specific queries.

• Consider replicating smaller, frequently joined tables to avoid unnecessary joins and host_id overhead.

• Use a central mapping table (tenant_resources_t) to manage tenant-resource associations and enforce multi-tenancy rules in the application logic where appropriate.

This more nuanced approach provides a balance between the benefits of distributed data with Citus and avoiding unnecessary complexity or performance overhead from overusing host_id. Choose the Citus deployment model (self-hosted open source or managed cloud service) that best suits our needs and budget.

Primary Key Considerations in a Distributed Citus Environment

When a table includes host_id (due to sharding requirements), it is important to include host_id as part of the primary key. This ensures proper functioning and optimization within the Citus distributed database.

1. Distribution Column Requirement
   In Citus, the distribution column (e.g., host_id) must be part of the primary key. This is essential for routing queries and distributing data correctly across shards.

2. Uniqueness Enforcement

   • The primary key enforces uniqueness across the entire distributed database.
   • For example, if user_id is unique only within a tenant (host), then (host_id, user_id) is required as the primary key to ensure uniqueness across all shards.

3. Data Locality and Co-location
   Including host_id in the primary key ensures that all rows for the same tenant (identified by the same host_id) are stored together on a single shard. This provides:

   • Efficient Joins: Joins between tables related to the same tenant can be performed locally on a single shard, avoiding expensive cross-shard data transfers.
   • Optimized Queries: Queries filtering by host_id are efficiently routed to the appropriate shard.

4. Referential Integrity
   If other tables reference the users_t table and are also distributed by host_id, including host_id in the primary key of users_t is essential to maintain referential integrity across shards.

Multi-Host User Session Management

In a multi-host environment where multiple hosts reside on the same server, users must associate with one host at a time. The session management is handled as follows:

1. Host Association on Login:

   • Once a user logs in, a host cookie is returned, derived from the JWT token.
   • The user’s session defaults to the associated host in the cookie.

2. Switching Hosts:

   • If a user wishes to switch to another host, they can:
     • Access the User Menu to select a different host.
     • Log out of the current session.
   • During the next login, the session will be tied to the newly selected host.

3. Host in API Requests:

   • For all API requests sent to the server, the host is typically included as part of the request payload.
   • For login users, the host is in the JWT token as a custom claim.
   • For guest users, the default host is used until the user is signed in.
   • This ensures proper routing and handling of requests in a multi-host environment.

By associating users to a specific host for each session, this approach ensures clear separation of data and responsibilities across hosts, while providing users the flexibility to switch hosts as needed.

Event Header

As the portal is based on the event sorucing, all events will be responsible for populating the database. So, they need to be separated by host_id as well. In the event header, we have one unique id which is generated when event is created. Also, it has host_id and user_id in the EventId which is included in every events.

Reference and Shared Tables

In an application there are some data that is shared by all tenants. For example, the dropdown options on the UI and business validation. We call them reference data and have defined several tables to manage them centrally. For each reference data type, there is a logical table defined in the ref_table_t and marked as common or not. Common means the table can be shared with other tenants. Otherwise, it is only private for the owner tenant.

Some other entities are very similar but they cannot be fit into the reference tables. For example, category_t table contains all the category definitions for different entities. These tables are designed with an optional host_id. Here is an exmaple.

CREATE TABLE category_t (
    category_id          VARCHAR(22) NOT NULL,   -- unique id to identify the category
    host_id              VARCHAR(22),            -- null mean global category
    entity_type          VARCHAR(50) NOT NULL,   -- the version of the schema
    category_name        VARCHAR(126) NOT NULL,  -- category name, must be url friendly.
    category_desc        VARCHAR(1024) NOT NULL, -- decription
    parent_category_id   VARCHAR(22) REFERENCES category_t(category_id) ON DELETE SET NULL, -- parent category id, null if there is no parent.
    sort_order           INT DEFAULT 0,          -- sort order on the UI
    update_user          VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_ts            TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (category_id)
);

-- 1. Unique index for GLOBAL categories (where host_id IS NULL)
-- Ensures uniqueness of (entity_type, category_name, parent_category_id) ONLY when host_id is NULL
CREATE UNIQUE INDEX idx_category_unique_global
ON category_t (entity_type, category_name, parent_category_id)
NULLS NOT DISTINCT -- Handles NULLs in parent_category_id correctly
WHERE host_id IS NULL;

-- 2. Unique index for TENANT-SPECIFIC categories (where host_id IS NOT NULL)
-- Ensures uniqueness of (host_id, entity_type, category_name, parent_category_id)
-- for rows that belong to a specific host.
CREATE UNIQUE INDEX idx_category_unique_tenant
ON category_t (host_id, entity_type, category_name, parent_category_id)
NULLS NOT DISTINCT -- Handles NULLs in parent_category_id correctly
WHERE host_id IS NOT NULL;


CREATE INDEX idx_category_entity_type ON category_t (entity_type);
CREATE INDEX idx_category_parent ON category_t (parent_category_id);
CREATE INDEX idx_category_name ON category_t (category_name);
CREATE INDEX idx_category_host_id ON category_t (host_id);

On the UI, the host_id will be auto populated according to the associated host_id by the user in readonly mode. There is a checkbox “Is Global Category” in the form. If checked, the backend service will have an FGA rule to ensure that the user is admin and the host_id will be removed in the event. This works for both create and update.

When viewing categories, the super admin might see all categories by default, possibly with a column or indicator showing the host_id (or “Global”). Filters should allow viewing global only, or a specific tenant’s categories.

Tenant Admin / Host Owner:

When a tenant admin accesses the category management UI, their context is fixed to their own host_id.

They should only be able to create/edit categories associated with their specific host_id.

The UI should not offer them the option to create/edit global categories or categories for other hosts. The host_id is implicitly set or displayed as read-only based on their logged-in context.

When viewing categories, they should see their own tenant-specific categories plus all applicable global categories. The UI should clearly differentiate between these (e.g., using grouping, labels, icons).

System Integration

System integrations must preserve the same identity and tenant boundaries as interactive portal workflows. The integration token is not only an access credential; it is also the source of audit metadata, event metadata, row filtering, and host scoping.

Command Side

The command side uses event sourcing. Every accepted command writes one or more domain events, and those events are later projected into query-side tables. Because events become the durable system of record, command calls need a stable user identity and host identity.

For command APIs, use an authorization code token whenever possible. The token must contain the real portal user id so command handlers can derive the correct userId, host, nonce, and CloudEvent metadata. This is the preferred path for browser flows, operator tools, and integrations that can act on behalf of a known user.

If the integration has no user session in the request context, do not submit anonymous command events. First onboard a real user in the system for the integration actor or service account. That user becomes the durable audit principal for the commands emitted by the integration.

After the user is onboarded, create an auth client for the integration and set custom claims that carry the command identity:

{
  "host": "<host-id>",
  "elm": "<integration-user-email>",
  "uid": "<integration-user-id>",
  "uty": "<user-type>"
}

The uid claim must reference the onboarded user. The host claim must match the tenant boundary where commands are allowed to run. The elm and uty claims should match the onboarded user’s email and user type so downstream authorization, audit, and support workflows can identify the actor without guessing.

For an integration auth client whose type is trusted, the client application can call Light OAuth with the client_credentials grant type when the auth client has these custom claims configured. Light OAuth issues a token that carries the custom claims, allowing the token to act as an id-token-like access token for command APIs, similar to the user-bearing token produced by the authorization code grant. This path is only acceptable for trusted client types because the client, not an interactive browser session, is asserting the user and host identity through the auth client configuration.

Command-side integration rules:

• Prefer an authorization code token tied to the real interactive user.
• Use a dedicated onboarded integration user only when no user session exists.
• For non-session integrations, use a trusted auth client with custom claims and request the token from Light OAuth with grant_type=client_credentials.
• Do not use a token that lacks a usable userId/uid for event-sourced commands.
• Do not allow non-trusted clients to mint user-bearing command tokens from client credentials.
• Keep host ownership explicit; never infer host scope from the client id alone.
• Treat the auth client and custom claims as deployment configuration, not as a substitute for user onboarding.

Query Side

The query side serves read models built from command-side events and operational tables. Query APIs do not create domain events, do not allocate command nonces, and should not mutate event-sourced state.

Query integrations still need authorization and tenant scoping. The request token must provide enough identity to determine the host and the effective user or service account. For user-scoped reads, use the same authorization code token or integration-user token described for the command side so row and column filters can apply consistently.

If authorization code flow is not available for a query integration, the client_credentials flow is acceptable only for auth clients whose type is trusted. The token must carry host, sid, and, when environment-specific data is requested, env. Here sid is the service id for the gateway, agent, or other Light-Fabric runtime calling portal-query. Query handlers must compare these claims with the requested hostId, serviceId, and optional envTag before returning service-scoped data.

Light-Fabric ecosystem components such as gateways and agents may use a long-lived token for query-side access when the token was issued through this trusted client_credentials path. That access is not general portal read access. It is limited to query endpoints built for those runtime components, such as gateway, agent, discovery, or catalog endpoints, and those endpoints must enforce the claim match before returning data.

For host-scoped or service-level reads, a client token can be used only when the auth client type is trusted and the token carries the required host, service, and environment claims. The query service should apply the same host boundary as the command side and return only data visible to that actor. A missing user session may reduce the allowed result set, but it must not broaden access.

Query-side integration rules:

• Read from projected/query tables; do not write command events from query handlers.
• Resolve host scope from the validated token claims and request parameters.
• When authorization code flow is unavailable, accept client_credentials only from auth clients whose type is trusted.
• Require host and sid token claims; require env when the endpoint or request is environment-scoped.
• Match token host, sid, and optional env to requested hostId, serviceId, and optional envTag.
• Allow long-lived Light-Fabric runtime tokens only on endpoints designed for gateways, agents, and similar ecosystem components.
• Do not use long-lived runtime tokens for broad user-facing query access.
• Apply user, role, position, group, attribute, and fine-grained filters when the endpoint requires them.
• Use the onboarded integration user for auditability when a human user is not present.
• Keep query tokens least-privileged; read-only integrations should not receive command scopes.

Portal Event

Light Portal is using event sourcing and CQRS. Any update to the system will generate an event and there are hundreds of event types.

All events are in Avro format and will be pushed to a Kafka cluster for stream processing. Each event has an EventId that contains common info for events and it is reside in light-kafka repo.

Here is one of the events in the light-portal.

{
  "type": "record",
  "name": "ApiRuleCreatedEvent",
  "namespace": "net.lightapi.portal.market",
  "fields": [
    {
      "name": "EventId",
      "type": {
        "type": "record",
        "name": "EventId",
        "namespace": "com.networknt.kafka.common",
        "fields": [
          {
            "name": "id",
            "type": "string",
            "doc": "a unique identifier"
          },
          {
            "name": "nonce",
            "type": "long",
            "doc": "the number of the transactions for the user"
          },
          {
            "name": "timestamp",
            "type": "long",
            "default": 0,
            "doc": "time the event is recorded"
          },
          {
            "name": "derived",
            "type": "boolean",
            "default": false,
            "doc": "indicate if the event is derived from event processor"
          }
        ]
      }
    },
    {
      "name": "hostId",
      "type": "string",
      "doc": "host id"
    },
    {
      "name": "apiId",
      "type": "string",
      "doc": "api id"
    },
    {
      "name": "ruleIds",
      "type": {
        "type": "array",
        "items": "string"
      },
      "doc": "one or many rule ids that link to the apiId"
    }
  ]
}

Kafka Key

When pushing events into a Kafka topic, the record key will be used to distribute record between different Kafka partitions. Here is the key selection for the system.

• multi-tenent

The key will be the hostId

• single-tenent

The key will be the userId

Promotion or Replay

Promotion approaches

1. When promote from dev to sit, we can export all event from dev and update the event json file and then replay to the sit.
2. We can import the original event json from dev to sit and then update some on the sit host.

Promotable Event Type

There are two type of events: configurable event vs transactional event. We should only promote the configurable events from dev to sit. Not the deployment logs from dev to sit. We need a table to define the promotable event types.

Reference Table

When building a web application, there would be a lot of dropdown selects in forms. The form itself only cares about the id and label list to render the form and only the id will be submitted to the backend API for single select and several ids for multiple select.

To save the effort to create many similar tables, we can craete a set of tables for all dropdowns. For some of the reference tables, dropdown should be the same across all hosts and we can set common flag to ‘Y’ so that they are shared by all hosts. If the dropdown values might be different between hosts, we can create a reference table per host and link the reference table with host in a separate table that support sharding.

Reference Schema

CREATE TABLE ref_host_t (
  table_id             VARCHAR(22) NOT NULL,
  host_id              VARCHAR(22) NOT NULL,
  PRIMARY KEY (table_id, host_id),
  FOREIGN KEY (table_id) REFERENCES ref_table_t (table_id) ON DELETE CASCADE,
  FOREIGN KEY (host_id) REFERENCES host (host_id) ON DELETE CASCADE
);

CREATE TABLE ref_table_t (
  table_id             VARCHAR(22) NOT NULL, -- UUID genereated by Util
  table_name           VARCHAR(80) NOT NULL, -- Name of the ref table for lookup.
  table_desc           VARCHAR(1024) NULL,
  active               CHAR(1) NOT NULL DEFAULT 'Y', -- Only active table returns values
  editable             CHAR(1) NOT NULL DEFAULT 'Y', -- Table value and locale can be updated via ref admin
  common               CHAR(1) NOT NULL DEFAULT 'Y', -- The drop down shared across hosts
  PRIMARY KEY(table_id)
);


CREATE TABLE ref_value_t (
  value_id              VARCHAR(22) NOT NULL,
  table_id              VARCHAR(22) NOT NULL,
  value_code            VARCHAR(80) NOT NULL, -- The dropdown value
  start_time            TIMESTAMP NULL,       
  end_time              TIMESTAMP NULL,
  display_order         INT,                  -- for editor and dropdown list.
  active                VARCHAR(1) NOT NULL DEFAULT 'Y',
  PRIMARY KEY(value_id),
  FOREIGN KEY table_id REFERENCES ref_table_t (table_id) ON DELETE CASCADE
);


CREATE TABLE value_locale_t (
  value_id              VARCHAR(22) NOT NULL,
  language              VARCHAR(2) NOT NULL,
  value_desc            VARCHAR(256) NULL, -- The drop label in language.
  PRIMARY KEY(value_id,language),
  FOREIGN KEY value_id REFERENCES ref_value_t (value_id) ON DELETE CASCADE
);



CREATE TABLE relation_type_t (
  relation_id           VARCHAR(22) NOT NULL,
  relation_name         VARCHAR(32) NOT NULL, -- The lookup keyword for the relation.
  relation_desc         VARCHAR(1024) NOT NULL,
  PRIMARY KEY(relation_id)
);



CREATE TABLE relation_t (
  relation_id           VARCHAR(22) NOT NULL,
  value_id_from         VARCHAR(22) NOT NULL,
  value_id_to           VARCHAR(22) NOT NULL,
  active                VARCHAR(1) NOT NULL DEFAULT 'Y',
  PRIMARY KEY(relation_id, value_id_from, value_id_to)
  FOREIGN KEY relation_id REFERENCES relation_type_t (relation_id) ON DELETE CASCADE,
  FOREIGN KEY value_id_from REFERENCES ref_value_t (value_id) ON DELETE CASCADE,
  FOREIGN KEY value_id_to REFERENCES ref_table_t (value_id) ON DELETE CASCADE
);

Authentication & Authorization

Light-Portal is a single-page application (SPA) that utilizes both the OAuth 2.0 Authorization Code and Client Credentials flows.

The following pattern illustrates the end-to-end process recommended by the Light Platform for an SPA interacting with downstream APIs.

Sequence Diagram

sequenceDiagram
    participant PortalView as Portal View
    participant LoginView as Login View
    participant Gateway as Light Gateway
    participant OAuthKafka as OAuth-Kafka
    participant AuthService as Auth Service
    participant ProxySidecar as Proxy Sidecar
    participant BackendAPI as Backend API

    PortalView ->> LoginView: 1. Signin redirect
    LoginView ->> OAuthKafka: 2. Authenticate user
    OAuthKafka ->> AuthService: 3. Authenticate User<br/>(Active Directory<br/>for Employees)<br/>(CIF System<br/>for Customers)
    AuthService ->> OAuthKafka: 4. Authenticated
    OAuthKafka ->> OAuthKafka: 5. Generate auth code
    OAuthKafka ->> PortalView: 6. Redirect with code
    PortalView ->> Gateway: 7. Authorization URL<br/>with code param
    Gateway ->> OAuthKafka: 8. Create JWT access<br/>token with code
    OAuthKafka ->> OAuthKafka: 9. Generate JWT<br/>access token<br/>with user claims
    OAuthKafka ->> Gateway: 10. Token returns<br/>to Gateway
    Gateway ->> PortalView: 11. Token returns<br/>to Portal View<br/>in Secure Cookie
    PortalView ->> Gateway: 12. Call Backend API
    Gateway ->> Gateway: 13. Verify the token
    Gateway ->> OAuthKafka: 14. Create Client<br/>Credentials token
    OAuthKafka ->> OAuthKafka: 15. Generate Token<br/>with Scopes
    OAuthKafka ->> Gateway: 16. Return the<br/>scope token
    Gateway ->> Gateway: 17. Add scope<br/>token to<br/>X-Scope-Token<br/>Header
    Gateway ->> ProxySidecar: 18. Invoke API
    ProxySidecar ->> ProxySidecar: 19. Verify<br/>Authorization<br/>token
    ProxySidecar ->> ProxySidecar: 20. Verify<br/>X-Scope-Token
    ProxySidecar ->> ProxySidecar: 21. Fine-Grained<br/>Authorization
    ProxySidecar ->> BackendAPI: 22. Invoke<br/>business API
    BackendAPI ->> ProxySidecar: 23. Business API<br/>response
    ProxySidecar ->> ProxySidecar: 24. Fine-Grained<br/>response filter
    ProxySidecar ->> Gateway: 25. Return response
    Gateway ->> PortalView: 26. Return response

1. When a user visits the website to access the single-page application (SPA), the Light Gateway serves the SPA to the user’s browser. Each single page application will have a dedicated Light Gateway instance acts as a BFF. By default, the user is not logged in and can only access limited site features. To unlock additional features, the user can click the User button in the header and select the Sign In menu. This action redirects the browser from the Portal View to the Login View, both served by the same Light Gateway instance.

2. On the Login View page, the user can either input a username and password or choose Google/Facebook for authentication. When the login form is submitted, the request is sent to the Light Gateway with the user’s credentials. The Gateway forwards this request to the OAuth Kafka service.

3. OAuth Kafka supports multiple authenticator implementations to verify user credentials. Examples include authenticating via the Light Portal user database, Active Directory for employees, or CIF service for customers.

4. Once authentication is successfully completed, the OAuth Kafka responds with the authentication result.

5. Upon successful authentication, OAuth Kafka generates an authorization code (a UUID associated with the user’s profile).

6. OAuth Kafka redirects the authorization code back to the browser at the Portal View via the Gateway.

7. Since the Portal View SPA lacks a dedicated redirect route for the authorization code, the browser sends the code as a query parameter in a request to the Gateway.

8. The StatelessAuthHandler in the Gateway processes this request, initiating a token request to OAuth Kafka to obtain a JWT access token.

9. OAuth Kafka generates an access token containing user claims in its custom JWT claims. The authorization code is then invalidated, as it is single-use.

10. The access token is returned to the Gateway.

11. The StatelessAuthHandler in the Gateway stores the access token in a secure cookie and sends it back to the Portal View.

12. When the Portal View SPA makes requests to backend APIs, it includes the secure cookie in the API request sent to the Gateway.

13. The StatelessAuthHandler in the Gateway validates the token in the secure cookie and places it in the Authorization header of the outgoing request.

14. If the token is successfully validated, the TokenHandler in the Gateway makes a request to OAuth Kafka for a client credentials token, using the path prefix of the API endpoint.

15. OAuth Kafka generates a client credentials token with the appropriate scope for accessing the downstream service.

16. The client credentials token is returned to the Gateway.

17. The TokenHandler in the Gateway inserts this token into the X-Scope-Token header of the original request.

18. The Gateway routes the original request, now containing both tokens, to the downstream proxy sidecarof the backend API.

19. The proxy sidecar validates the Authorization token, verifying its signature, expiration, and other attributes.

20. The proxy sidecar also validates the X-Scope-Token, ensuring its signature, expiration, and scope are correct.

21. Once both tokens are successfully validated, the proxy sidecar enforces fine-grained authorization rules based on the user’s custom security profile contained in the Authorization token.

22. If the fine-grained authorization checks are passed, the proxy sidecar forwards the request to the backend API.

23. The backend API processes the request and sends the full response back to the proxy sidecar.

24. The proxy sidecar applies fine-grained filters to the response, reducing the number of rows and/or columns based on the user’s security profile or other policies.

25. The proxy sidecar returns the filtered response to the Gateway.

26. The Gateway forwards the response to the Portal View, allowing the SPA to render the page.

Fine-Grained Authorization

What is Fine-Grained Authorization?

Fine-grained authorization (FGA) refers to a detailed and precise control mechanism that governs access to resources based on specific attributes, roles, or rules. It’s also known as fine-grained access control (FGAC). Unlike coarse-grained authorization, which applies broader access policies (e.g., “Admins can access everything”), fine-grained authorization allows for more specific policies (e.g., “Admins can access user data only if they belong to the same department and the access request is during business hours”).

Key Features

• Granular Control: Policies are defined at a detailed level, considering attributes like user role, resource type, action, time, location, etc.
• Context-Aware: Takes into account dynamic conditions such as the time of request, user’s location, or other contextual factors.
• Flexible Policies: Allows the creation of complex, conditional rules tailored to the organization’s needs.

Why Do We Need Fine-Grained Authorization?

1. Enhanced Security

By limiting access based on detailed criteria, fine-grained authorization minimizes the risk of unauthorized access or data breaches.

2. Regulatory Compliance

It helps organizations comply with legal and industry-specific regulations (e.g., GDPR, HIPAA) by ensuring sensitive data is only accessible under strict conditions.

3. Minimized Attack Surface

By restricting access to only the required resources and operations, fine-grained authorization reduces the potential impact of insider threats or compromised accounts.

4. Improved User Experience

Enables personalized access based on roles and permissions, ensuring users see only what they need, which reduces confusion and improves productivity.

5. Auditing and Accountability

Detailed access logs and policy enforcement make it easier to track and audit who accessed what, when, and why, fostering better accountability.

Examples of Use Cases

• Healthcare: A doctor can only view records of patients they are treating.
• Government: A government employee can access to data and documents based on security clearance levels and job roles.
• Finance: A teller can only access transactions related to their assigned branch.
• Enterprise Software: Employees can edit documents only if they own them or have been granted editing permissions.

Fine-Grained Authorization in API Access Control

In API access control, fine-grained authorization governs how users or systems interact with specific API endpoints, actions, and data. This approach ensures that access permissions are precisely tailored to attributes, roles, and contextual factors, enabling a secure and customized API experience. As the Light Portal is a platform centered on APIs, the remainder of the design will focus on the API access control context.

Early Approaches to Fine Grained Authorization

Early approaches to fine grained authorization primarily involved Access Control Lists (ACLs) and Role-Based Access Control (RBAC). These methods laid the foundation for more sophisticated access control mechanisms that followed. Here’s an overview of these primary approaches:

Access Control Lists (ACLs):

• ACLs were one of the earliest forms of fine grained authorization, allowing administrators to specify access permissions on individual resources for each user or group of users.

• In ACLs, permissions are directly assigned to users or groups, granting or denying access to specific resources based on their identities.

• While effective for small-scale environments with limited resources and users, ACLs became cumbersome as organizations grew. Maintenance issues arose, such as the time required to manage access to an increasing number of resources for numerous users.

Role-Based Access Control (RBAC):

• RBAC emerged as a solution to the scalability and maintenance challenges posed by ACLs. It introduced the concept of roles, which represent sets of permissions associated with particular job functions or responsibilities.

• Users are assigned one or more roles, and their access permissions are determined by the roles they possess rather than their individual identities.

• RBAC can be implemented with varying degrees of granularity. Roles can be coarse-grained, providing broad access privileges, or fine-grained, offering more specific and nuanced permissions based on organizational needs.

• Initially, RBAC appeared to address the limitations of ACLs by providing a more scalable and manageable approach to access control.

Both ACLs and RBAC have their shortcomings:

• Maintenance Challenges: While RBAC offered improved scalability compared to ACLs, it still faced challenges with role management as organizations expanded. The proliferation of roles, especially fine grained ones, led to a phenomenon known as role explosion where the number of roles grew rapidly, making them difficult to manage effectively.

• Security Risks: RBAC’s flexibility also posed security risks. Over time, users might accumulate permissions beyond what they need for their current roles, leading to a phenomenon known as permission creep. This weakened overall security controls and increased the risk of unauthorized access or privilege misuse.

Following the discussion of early approaches to fine grained authorization, it’s crucial to acknowledge that different applications have varying needs for authorization.

Whether to use fine grained or coarse-grained controls depends on the specific project. Controlling access becomes trickier due to the spread-out nature of resources and differing levels of detail needed across components. Let’s delve into the differentiating factors:

Standard Models for Implementing FGA

There are several standard models for implementing FGA:

• Attribute-Based Access Control (ABAC): In ABAC, access control decisions are made by evaluating attributes such as user roles, resource attributes (e.g., type, size, status), requested action, current date and time, and any other relevant contextual information. ABAC allows for very granular control over access based on a wide range of attributes.

• Policy-Based Access Control (PBAC): PBAC is similar to ABAC but focuses more on defining policies than directly evaluating attributes. Policies in PBAC typically consist of rules or logic that dictate access control decisions based on various contextual factors. While ABAC relies heavily on data (attributes), PBAC emphasizes using logic to determine access.

• Relationship-Based Access Control (ReBAC): ReBAC emphasizes the relationships between users and resources, as well as relationships between different resources. By considering these relationships, ReBAC provides a powerful and expressive model for describing complex authorization contexts. This can involve the attributes of users and resources and their interactions and dependencies.

Each of these models offers different strengths and may be more suitable for different scenarios. FGA allows for fine grained control over access, enabling organizations to enforce highly specific access policies tailored to their requirements.

Streamlining FGA by Implementing Rule-Based Access Control:

ABAC (Attribute-Based Access Control) focuses on data attributes, PBAC (Policy-Based Access Control) centers on logic, and ReBAC (Relationship-Based Access Control) emphasizes relationships between users and resources. But what if we combined all three to leverage the strengths of each? This is the idea behind Rule-Based Access Control (RuBAC).

By embedding a lightweight rule engine, we can integrate multiple rules and actions to achieve the following:

• Optimize ABAC: Reduce the number of required attributes since not all rules depend on them. For example, a standard rule like “Customer data can only be accessed during working hours” can be shared across policies.

• Flexible Policy Enforcement: Using a rule engine makes access policies more dynamic and simpler to manage.

• Infer Relationships: Automatically deduce relationships between entities. For instance, the rule engine could grant a user access to a file if they already have permission for the containing folder.

Principle of Least Privilege

The principle of least privilege access control widely referred to as least privilege, and PoLP is the security concept in which user(s) (employee(s)) are granted the minimum level of access/permissions to the app, data, or system that is required to perform his/her job functions.

To ensure PoLP is effectively enforced, we’ve compiled a list of best practices:

• Conduct a thorough privilege audit: As we know, visibility is critical in an access environment, so conducting regular or periodic access audits of all privileged accounts can help your team gain complete visibility. This audit includes reviewing privileged accounts and credentials held by employees, contractors, and third-party vendors, whether on-premises, accessible remotely, or in the cloud. However, your team must also focus on default and hard-coded credentials, which IT teams often overlook.

• Establish the least privilege as the default: Start by granting new accounts the minimum privileges required for their tasks and eliminate or reconfigure default permissions on new systems or applications. Further, use role-based access control to help your team determine the necessary privileges for a new account by providing general guidelines based on roles and responsibilities. Also, your team needs to update and adjust access level permissions when the user’s role changes; this will help prevent privilege creep.

• Enforce separation of privileges: Your team can prevent over-provisioning by limiting administrator privileges. Firstly, segregate administrative accounts from standard accounts, even if they belong to the same user, and isolate privileged user sessions. Then, grant administrative privileges (such as read, write, and execute permissions) only to the extent necessary for the user to perform their specific administrative tasks. This will help your team prevent granting users unnecessary or excessive control over critical systems, which could lead to security vulnerabilities or misconfigurations.

• Provide just-in-time, limited access: To maintain least-privilege access without hindering employee workflows, combine role-based access control with time-limited privileges. Further, replace hard-coded credentials with dynamic secrets or use one-time-use/temporary credentials. This will help your team grant temporary elevated access permissions when users need it, for instance, to complete specific tasks or short-term projects.

• Keep track and evaluate privileged access: Continuously monitor authentications and authorizations across your API platform and ensure all the individual actions are traceable. Additionally, record all authentication and authorizaiton sessions comprehensively, and use automated tools to swiftly identify any unusual activity or potential issues. These best practices are designed to enhance the security of your privileged accounts, data, and assets while ensuring compliance adherence and improving operational security without disrupting user workflows.

OpenAPI Specification Extensions

OpenAPI uses the term security scheme for authentication and authorization schemes. OpenAPI 3.0 lets you describe APIs protected using the following security schemes. The fine-grained authorization is just another layer of security and it is natural to define the fine-grained authorization in the same specification. It can be done with OpenAPI specification extensions.

Extensions (also referred to as specification extensions or vendor extensions) are custom properties that start with x-, such as x-logo. They can be used to describe extra functionality that is not covered by the standard OpenAPI Specification. Many API-related products that support OpenAPI make use of extensions to document their own attributes, such as Amazon API Gateway, ReDoc, APIMatic and others.

As OpenAPI specification openapi.yaml is loaded during the light-4j startup, the extensions will be available at runtime in cache for each endpoint just like the scopes definition. The API owner can define the following two extensions for each endpoint:

• x-request-access: This section allows designer to specify one or more rules as well as one or more security attributes for the input of the rules. For example, roles, location etc. The rule result will decide if the user has access to the endpoint based on the security attributes from the JWT token in the request chain.

• x-response-filter: This section is similar to the above; however, it works on the response chain. The rule result will decide which row or column of the response JSON will return to the user based on the security profile from the JWT token.

Example of OpenAPI specification with fine-grained authorization.

paths:
  /accounts:
    get:
      summary: "List all accounts"
      operationId: "listAccounts"
      x-request-access:
        rule: "account-cc-group-role-auth"
        roles: "manager teller customer"
      x-response-filter:
        rule: "account-row-filter"
        teller:
          status: open
        customer:
          status: open
          owner: @user_id
        rule: "account-col-filter"
          teller: ["num","owner","type","firstName","lastName","status"]
          customer: ["num","owner","type","firstName","lastName"]
      security:
      - account_auth:
        - "account.r"

FGA Rules for AccessControlHandler

With the above specification loaded during the runtime, the rules will be loaded during the server startup for the service as well. In the Rule Registry on the light-portal, we have a set of built-in rules that can be picked as fine-grained policies for each API. Here is an example of rule for the above specification in the x-request-access.

account-cc-group-role-auth:
  ruleId: account-cc-group-role-auth
  host: lightapi.net
  description: Role-based authorization rule for account service and allow cc token and transform group to role.
  conditions:
    - conditionId: allow-cc
      variableName: auditInfo
      propertyPath: subject_claims.ClaimsMap.user_id
      operatorCode: NIL
      joinCode: OR
      index: 1
    - conditionId: manager
      variableName: auditInfo
      propertyPath: subject_claims.ClaimsMap.groups
      operatorCode: CS
      joinCode: OR
      index: 2
      conditionValues:
        - conditionValueId: manager
          conditionValue: admin
    - conditionId: teller
      variableName: auditInfo
      propertyPath: subject_claims.ClaimsMap.groups
      operatorCode: CS
      joinCode: OR
      index: 3
      conditionValues:
        - conditionValueId: teller
          conditionValue: frontOffice
    - conditionId: allow-role-jwt
      variableName: auditInfo
      propertyPath: subject_claims.ClaimsMap.roles
      operatorCode: NNIL
      joinCode: OR
      index: 4
  actions:
    - actionId: match-role
      actionClassName: com.networknt.rule.FineGrainedAuthAction
      actionValues:
        - actionValueId: roles
          value: $roles

All rules are managed by the light-portal and shared by all the services. In addition, developers can create their customized rules for their own services.

Response Filter

There are two type of filters. Row and Column.

Row

For row filter, we need to check the condition defined for some of the properties in order to make the filter decision. In database, for each endpoint, we have colName, operator and colValue defined for the condition.

The operator supports the following enum: [“=”,“!=”,“<”,“>”,“<=”,“>=”,“in”,“not in”, “range”]

For the colValue, we do support variables from the jwt token with @. For example, @eid will be replaced with the eid claim from the jwt token.

Col

For column filter, we need to include a list of columns or exclude a list of columns in json format.

[“accountNo”,“firstName”,“lastName”]

or

![“status”]

JSON Schema Registry

JSON Schema is a declarative language that provides a standardized way to describe and validate JSON data.

What it does

JSON Schema defines the structure, content, data types, and constraints of JSON documents. It’s an IETF standard that helps ensure the consistency and integrity of JSON data across applications.

How it works

JSON Schema uses keywords to define data properties. A JSON Schema validator checks if JSON documents conform to the schema.

What it’s useful for

• Describing existing data formats
• Validating data as part of automated testing
• Submitting client data
• Defining how a record should be organized

What is a JSON Schema Registry

The JSON Schema Registry provides a centralized service for your JSON schemas with RESTful endpoints for storing and retrieving JSON schemas.

When using data in a distributed application with many RESTful APIs, it is important to ensure that it is well-formed and structured. If data is sent without prior validation, errors may occur on the services. A schema registry provides a way to ensure that the data is validated before it is sent and validated after it is received.

A schema registry is a service used to define and confirm the structure of data that is sent between consumers and providers. In a schema registry, developers can define what the data should look like and how it should be validated. The schemas can be utilized in the OpenAPI specifications to ensure that schemas can be externalized.

Schema records can also help ensure forward and backward compatibility when changes are made to the data structure. When a schema record is used, the data transfered with more schema information that can be used to ensure that applications reading the data can interpret it.

Given the API consumers and providers can belong to different groups or organizations, it is necessary to have a centralized service to manage the schemas so that they can be shared between them. This is why we have implemented this service as part of the light-portal.

Schema Specification Version

The registry is heterogeneous registry as it can store schemas of different schema draft versions. By default the registry is configured to store schemas of Draft 2020-12. When a schema is added, the version which is currently is set, is what the schema is saved as.

The following list contains all supported specification versions.

• Draft 4
• Draft 6
• Draft 7
• 2019-09
• 2020-12

Schema Version

Once a schema is registed into the registry, it will be assigned as version 1. Each time it is updated, the version number will increase 1. When the schema is retrieve, the version number can be part of the URL to indicate that exact version will be retrieved. If version number is not in the URL, the latest version will be retrieved.

Access Endpoint

Table Structure

Light Controller

YAML Rule Registry

React Schema Form

React Schema Form is a form generator based on JSON Schema and form definitions from Light Portal. It renders UI forms to manipulate database entities, and form submissions are automatically hooked into an API endpoint.

Debugging a Component

Encountering a bug in a react-schema-form component can be challenging since the source code may not be directly visible. To debug:

1. Set up the Light Portal server if dropdowns are loaded from the server.
2. Use the example app in the same project to debug.

Use a Local Alias with Vite

Vite allows creating an alias to point to your library’s src folder. Update the vite.config.ts in your example app:

import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';
import path from 'path';

export default defineConfig({
  plugins: [react()],
  resolve: {
    alias: {
      'react-schema-form': path.resolve(__dirname, '../src'), // Adjust the path to point to the library's `src` folder
    },
  },
});

Use a Link Script in package.json

Update the example app’s package.json file. In the dependencies section, replace the library’s version with a local path:

{
  "dependencies": {
    "react-schema-form": "file:../src"
  }
}

Library Entry Point

Vite requires an entry point file, typically named index.js or index.ts, in your library’s src folder. Ensure that your library’s src folder includes a properly configured index.js file, like this:

export { default as SchemaForm } from './SchemaForm'
export { default as ComposedComponent } from './ComposedComponent'
export { default as utils } from './utils'
export { default as Array } from './Array'

Without a correctly named and configured entry file, components like SchemaForm may not be imported properly.

Update index.html

If you change the entry point file from main.js to index.js, ensure you update the reference in the index.html file located in the root folder. For example:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Vite + React</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/index.js"></script>
  </body>
</html>

Sync devDependencies from peerDependencies

When the source code in src is used directly by the example app, the peerDependencies in the example app won’t work for react-schema-form components. To address this, copy the peerDependencies into the devDependencies section of react-schema-form’s package.json. For example:

  "devDependencies": {
    "@babel/runtime": "^7.26.0",
    "@codemirror/autocomplete": "^6.18.2",
    "@codemirror/language": "^6.10.6",
    "@codemirror/lint": "^6.8.2",
    "@codemirror/search": "^6.5.7",
    "@codemirror/state": "^6.4.1",
    "@codemirror/theme-one-dark": "^6.1.2",
    "@codemirror/view": "^6.34.2",
    "@emotion/react": "^11.13.5",
    "@emotion/styled": "^11.13.5",
    "@eslint/js": "^9.13.0",
    "@lezer/common": "^1.2.3",
    "@mui/icons-material": "^6.1.6",
    "@mui/material": "^6.1.6",
    "@mui/styles": "^6.1.6",
    "@types/react": "^18.3.1",
    "@uiw/react-markdown-editor": "^6.1.2",
    "@vitejs/plugin-react": "^4.3.3",
    "codemirror": "^6.0.1",
    "eslint": "^9.13.0",
    "eslint-plugin-react": "^7.37.2",
    "eslint-plugin-react-hooks": "^5.0.0",
    "eslint-plugin-react-refresh": "^0.4.14",
    "gh-pages": "^6.2.0",
    "globals": "^15.11.0",
    "react": "^18.3.1",
    "react-dom": "^18.3.1",
    "vite": "^6.0.3"
  },
  "peerDependencies": {
    "@babel/runtime": "^7.26.0",
    "@codemirror/autocomplete": "^6.18.2",
    "@codemirror/language": "^6.10.6",
    "@codemirror/lint": "^6.8.2",
    "@codemirror/search": "^6.5.7",
    "@codemirror/state": "^6.4.1",
    "@codemirror/theme-one-dark": "^6.1.2",
    "@codemirror/view": "^6.34.2",
    "@emotion/react": "^11.13.5",
    "@emotion/styled": "^11.13.5",
    "@lezer/common": "^1.2.3",
    "@mui/icons-material": "^6.1.6",
    "@mui/material": "^6.1.6",
    "@mui/styles": "^6.1.6",
    "@types/react": "^18.3.1",
    "@uiw/react-markdown-editor": "^6.1.2",
    "codemirror": "^6.0.1",
    "react": "^18.3.1",
    "react-dom": "^18.3.1"
  },

Additionally, ensure the peerDependencies are also synced with the dependencies section of the example app’s package.json. This step allows react-schema-form components to load independently and work seamlessly during development.

Update Source Code

After completing all the updates, perform a clean install for both react-schema-form and the example app. Then, start the server from the example folder using the following command:

yarn dev

Whenever you modify a react-schema-form component, simply refresh the browser to reload the example application and see the updated component in action.

Debug with Visual Studio Code

You can debug the component using Visual Studio Code. There are many tutorials available online that explain how to debug React applications built with Vite, which can help you set up breakpoints, inspect components, and track down issues effectively.

Component dynaselect

dynaselect is a component that renders a dropdown select, either from static options or options loaded dynamically from a server via an API endpoint. It is a wrapper of material ui Autocomplete component. Below is an example form from the example app that demonstrates how to use this component.

{
  "schema": {
    "type": "object",
    "title": "React Component Autocomplete Demo Static Single",
    "properties": {
      "name": {
        "title": "Name",
        "type": "string",
        "default": "Steve"
      },
      "host": {
        "title": "Host",
        "type": "string"
      },
      "environment": {
        "type": "string",
        "title": "Environment",
        "default": "LOCAL",
        "enum": [
          "LOCAL",
          "SIT1",
          "SIT2",
          "SIT3",
          "UAT1",
          "UAT2"
        ]
      },
      "stringarraysingle": {
        "type": "array",
        "title": "Single String Array",
        "items": {
          "type": "string"
        }
      },
      "stringcat": {
        "type": "string",
        "title": "Joined Strings"
      },
      "stringarraymultiple": {
        "type": "array",
        "title": "Multiple String Array",
        "items": {
          "type": "string"
        }
      }
    },
    "required": [
      "name",
      "environment"
    ]
  },
  "form": [
    "name",
    {
      "key": "host",
      "type": "dynaselect",
      "multiple": false,
      "action": {
        "url": "https://localhost/portal/query?cmd=%7B%22host%22%3A%22lightapi.net%22%2C%22service%22%3A%22user%22%2C%22action%22%3A%22listHost%22%2C%22version%22%3A%220.1.0%22%7D"
      }
    },
    {
      "key": "environment",
      "type": "dynaselect",
      "multiple": false,
      "options": [
        {
          "id": "LOCAL",
          "label": "Local"
        },
        {
          "id": "SIT1",
          "label": "SIT1"
        },
        {
          "id": "SIT2",
          "label": "SIT2"
        },
        {
          "id": "SIT3",
          "label": "SIT3"
        },
        {
          "id": "UAT1",
          "label": "UAT1"
        },
        {
          "id": "UAT2",
          "label": "UAT2"
        }
      ]
    },
    {
      "key": "stringarraysingle",
      "type": "dynaselect",
      "multiple": false,
      "options": [
        {
          "id": "id1",
          "label": "label1"
        },
        {
          "id": "id2",
          "label": "label2"
        },
        {
          "id": "id3",
          "label": "label3"
        },
        {
          "id": "id4",
          "label": "label4"
        },
        {
          "id": "id5",
          "label": "label5"
        },
        {
          "id": "id6",
          "label": "label6"
        }
      ]
    },
    {
      "key": "stringcat",
      "type": "dynaselect",
      "multiple": true,
      "options": [
        {
          "id": "id1",
          "label": "label1"
        },
        {
          "id": "id2",
          "label": "label2"
        },
        {
          "id": "id3",
          "label": "label3"
        },
        {
          "id": "id4",
          "label": "label4"
        },
        {
          "id": "id5",
          "label": "label5"
        },
        {
          "id": "id6",
          "label": "label6"
        }
      ]
    },
    {
      "key": "stringarraymultiple",
      "type": "dynaselect",
      "multiple": true,
      "options": [
        {
          "id": "id1",
          "label": "label1"
        },
        {
          "id": "id2",
          "label": "label2"
        },
        {
          "id": "id3",
          "label": "label3"
        },
        {
          "id": "id4",
          "label": "label4"
        },
        {
          "id": "id5",
          "label": "label5"
        },
        {
          "id": "id6",
          "label": "label6"
        }
      ]
    }
  ]
}

Dynamic Options from APIs

The host is a string type field rendered as a dynaselect with multiple set to false. The options for the select are loaded via an API endpoint, with the action URL provided. Note that the cmd query parameter value is encoded because it contains curly brackets {}.

To encode and decode the query parameter value, you can use the following tool:

Encoder/Decoder Tool

Encoded:

%7B%22host%22%3A%22lightapi.net%22%2C%22service%22%3A%22user%22%2C%22action%22%3A%22listHost%22%2C%22version%22%3A%220.1.0%22%7D

Decoded:

{"host":"lightapi.net","service":"user","action":"listHost","version":"0.1.0"}

When using the example app to test the react-schema-form with APIs, you need to configure CORS on the light-gateway. Ensure that CORS is enabled only on the light-gateway and not on the backend API, such as hybrid-query.

Here is the example in values.yml for the light-gateway.

# cors.yml
cors.enabled: true
cors.allowedOrigins:
  - https://devsignin.lightapi.net
  - https://dev.lightapi.net
  - https://localhost:3000
  - http://localhost:5173
cors.allowedMethods:
  - GET
  - POST
  - PUT
  - DELETE

Single string type

For the environment field, the schema defines the type as string, and the form definition specifies multiple: false to indicate it is a single select.

The select result in the model looks like the following:

{
  "environment": "SIT1",
}

Single string array type

For the stringarraysingle field, the schema defines the type as a string array, and the form definition specifies multiple: false to indicate it is a single select.

The select result in the model looks like the following:

{
  "stringarraysingle": [
    "id3"
  ],	
}

Multiple string type

For the stringcat field, the schema defines the type as a string, and the form definition specifies multiple: true to indicate it is a multiple select.

The select result in the model looks like the following:

{
	"stringcat": "id2,id4"
}

Multiple string array type

For the stringarraymultiple field, the schema defines the type as a string array, and the form definition specifies multiple: true to indicate it is a multiple select.

The select result in the model looks like the following:

{
  "stringarraymultiple": [
    "id2",
    "id5",
    "id3"
  ],	
}

User Management

User Type

The user_type field is a critical part of the user security profile in the JWT token and can be leveraged for fine-grained authorization. In a multi-tenant environment, user_type is presented as a dropdown populated from the reference table configured for the organization. It can be dynamically selected based on the host chosen during the user registration process.

Supported Standard Dropdown Models

1. Employee and Customer

   • Dropdown values: E (Employee), C (Customer)
   • Default model for lightapi.net host.
   • Suitable for most organizations.

2. Employee, Personal, and Business

   • Dropdown values:
     • E (Employee)
     • P (Personal)
     • B (Business)
   • Commonly used for banks where personal and business banking are separated.

Database Configuration

• The user_type field is nullable in the user_t table by default.
• However, you can enforce this field as mandatory in your application via the schema and UI configuration.

On-Prem Deployment

In on-premise environments, the user_type can determine the authentication method:

• Employees: Authenticated via Active Directory.
• Customers: Authenticated via a customer database.

This flexibility allows organizations to tailor the authentication process based on their specific needs and user classifications.

Handling Users with Multi-Host Access

There are two primary ways to handle users who belong to multiple hosts:

1. User-Host Mapping Table:

user_t: This table would not have a host_id and would store core user information that is host-independent. The user_id would be unique across all hosts.

user_host_t (or user_tenant_t): This would be a mapping table to represent the many-to-many relationship between users and hosts.

-- user_t (no host_id, globally unique user_id)
CREATE TABLE user_t (
    user_id UUID PRIMARY KEY DEFAULT uuid_generate_v4(), -- UUID is recommended
    -- ... other user attributes (e.g., name, email) 
);

-- user_host_t (mapping table)
CREATE TABLE user_host_t (
    user_id UUID NOT NULL,
    host_id UUID NOT NULL,
    -- ... other relationship-specific attributes (e.g., roles within the host)
    PRIMARY KEY (user_id, host_id),
    FOREIGN KEY (user_id) REFERENCES user_t (user_id) ON DELETE CASCADE,
    FOREIGN KEY (host_id) REFERENCES host_t (host_id) ON DELETE CASCADE -- Assuming you have a hosts_t
);

2. Duplicating User Records (Less Recommended):

user_t: You would keep host_id in this table, and the primary key would be (host_id, user_id).

User Duplication: If a user needs access to multiple hosts, you would duplicate their user record in users_t for each host they belong to, each with a different host_id.

Why User-Host Mapping is Generally Preferred:

• Data Integrity: Avoids data duplication and the potential for inconsistencies that come with it. If a user’s core information (e.g., name, email) changes, you only need to update it in one place in user_t.

• Flexibility: Easier to add or remove a user’s access to hosts without affecting their core user data.

• Querying: While you’ll need joins to get a user’s hosts or a host’s users, these joins are straightforward using the mapping table.

• Scalability: Better scalability as your user base and the number of hosts they can access grow.

Distributing Tables in a Multi-Host User Scenario:

With the user-host mapping approach:

• user_t: This table would likely be a reference table in Citus (replicated to all nodes) since it does not have a host_id for distribution.

• user_host_t: This table would be distributed by host_id.

• Other tables (e.g., employees_t, api_endpoints_t, etc.): These would be distributed by host_id as before.

When querying, you would typically:

• Start with the user_hosts_t table to find the hosts a user has access to.

• Join with other tables (distributed by host_id) based on the host_id to retrieve tenant-specific data.

Choosing the Right user_id Primary Key:

Here’s a comparison of the options for the user_id primary key in user_t:

1. UUID (user_id)

• Pros:
  • Globally Unique: Avoids collisions across hosts or when scaling beyond the current setup.
  • Security: Difficult to guess or enumerate.
  • Scalability: Well-suited for distributed environments like Citus.
• Cons:
  • Storage: Slightly larger storage size compared to integers.
  • Readability: Not human-readable, which can be inconvenient for debugging.
• Recommendation:
  This is generally the best option for a user_id in a multi-tenant, distributed environment.

2. Email (email)

• Pros:
  • Human-Readable: Easy to identify and manage.
  • Login Identifier: Often used as a natural login credential.
• Cons:
  • Uniqueness Challenges: Enforcing global uniqueness across all hosts may require complex constraints or application logic.
  • Changeability: If emails change, cascading updates can complicate the database.
  • Security: Using emails as primary keys can expose sensitive user data if not handled securely.
  • Performance: String comparisons are slower than those for integers or UUIDs.
• Recommendation:
  Not recommended as a primary key, especially in a multi-tenant or distributed setup.

3. User-Chosen Unique ID (e.g., username)

• Pros:
  • Human-Readable: Intuitive and user-friendly.
• Cons:
  • Uniqueness Challenges: Enforcing global uniqueness is challenging and may require complex constraints.
  • Changeability: Users may request username changes, causing cascading update issues.
  • Security: Usernames are easier to guess or enumerate compared to UUIDs.
• Recommendation:
  Not recommended as a primary key in a multi-tenant, distributed environment.

In Conclusion:

• Use a User-Host Mapping Table:
  This is the best approach to handle users who belong to multiple hosts in a multi-tenant Citus environment.

• Use UUID for user_id:
  UUIDs are the most suitable option for the user_id primary key in user_t due to their global uniqueness, security, and scalability.

• Distribute by host_id:
  Distribute tables that need sharding by host_id, and ensure that foreign keys to distributed tables include host_id.

• Use Reference Tables:
  For tables like user_t that don’t have a host_id, designate them as reference tables in Citus.

This approach provides a flexible and scalable foundation for managing users with multi-host access in your Citus-based multi-tenant application.

User Tables

Using a single user_t table with a user_type discriminator is a good approach for managing both employees and customers in a unified way. Adding optional referral relationships for customers adds a nice dimension as well. Here’s a suggested table schema in PostgreSQL, along with explanations and some considerations:

user_t (User Table): This table will store basic information common to both employees and customers.

CREATE TABLE user_t (
    user_id                   VARCHAR(24) NOT NULL,
    email                     VARCHAR(255) NOT NULL,
    password                  VARCHAR(1024) NOT NULL,
    language                  CHAR(2) NOT NULL,
    first_name                VARCHAR(32) NULL,
    last_name                 VARCHAR(32) NULL,
    user_type                 CHAR(1) NULL, -- E employee C customer or E employee P personal B business
    phone_number              VARCHAR(20) NULL,
    gender                    CHAR(1) NULL,
    birthday                  DATE NULL,
    country                   VARCHAR(3) NULL,
    province                  VARCHAR(32) NULL,
    city                      VARCHAR(32) NULL,
    address                   VARCHAR(128) NULL,
    post_code                 VARCHAR(16) NULL,
    verified                  BOOLEAN NOT NULL DEFAULT false,
    token                     VARCHAR(64) NULL,
    locked                    BOOLEAN NOT NULL DEFAULT false,
    nonce                     BIGINT NOT NULL DEFAULT 0,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL
);

ALTER TABLE user_t ADD CONSTRAINT user_pk PRIMARY KEY ( user_id );

ALTER TABLE user_t ADD CONSTRAINT user_email_uk UNIQUE ( email );

user_host_t (User to host relationship or mapping):

CREATE TABLE user_host_t (
    host_id                   VARCHAR(24) NOT NULL,
    user_id                   VARCHAR(24) NOT NULL,
    -- other relationship-specific attributes (e.g., roles within the host)
    PRIMARY KEY (host_id, user_id),
    FOREIGN KEY (user_id) REFERENCES user_t (user_id) ON DELETE CASCADE,
    FOREIGN KEY (host_id) REFERENCES host_t (host_id) ON DELETE CASCADE
);

employee_t (Employee Table): This table will store employee-specific attributes.

CREATE TABLE employee_t (
    host_id                   VARCHAR(22) NOT NULL,
    employee_id               VARCHAR(50) NOT NULL,  -- Employee ID or number or ACF2 ID. Unique within the host. 
    user_id                   VARCHAR(22) NOT NULL,
    title                     VARCHAR(255) NOT NULL,
    manager_id                VARCHAR(50), -- manager's employee_id if there is one.
    hire_date                 DATE,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, employee_id),
    FOREIGN KEY (host_id, user_id) REFERENCES user_host_t(host_id, user_id) ON DELETE CASCADE,
    FOREIGN KEY (host_id, manager_id) REFERENCES employee_t(host_id, employee_id) ON DELETE CASCADE
);

customer_t (Customer Table): This table will store customer-specific attributes.

CREATE TABLE customer_t (
    host_id                   VARCHAR(24) NOT NULL,
    customer_id               VARCHAR(50) NOT NULL,
    user_id                   VARCHAR(24) NOT NULL,
    -- Other customer-specific attributes
    referral_id               VARCHAR(22), -- the customer_id who refers this customer. 
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, customer_id),
    FOREIGN KEY (host_id, user_id) REFERENCES user_host_t(host_id, user_id) ON DELETE CASCADE,
    FOREIGN KEY (host_id, referral_id) REFERENCES customer_t(host_id, customer_id) ON DELETE CASCADE
);

position_t (Position Table): Defines different positions within the organization for employees.

CREATE TABLE position_t (
    host_id                   VARCHAR(22) NOT NULL,
    position_id               VARCHAR(22) NOT NULL,
    position_name             VARCHAR(255) UNIQUE NOT NULL,
    description               TEXT,
    inherit_to_ancestor       BOOLEAN DEFAULT FALSE,
    inherit_to_sibling        BOOLEAN DEFAULT FALSE,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, position_id)
);

user_position_t (Employee Position Table): Links employees to their positions with effective dates.

CREATE TABLE employee_position_t (
    host_id                   VARCHAR(22) NOT NULL,
    employee_id               VARCHAR(50) NOT NULL,
    position_id               VARCHAR(22) NOT NULL,
    position_type             CHAR(1) NOT NULL, -- P position of own, D inherited from a decendant, S inherited from a sibling.
    start_date                DATE NOT NULL,
    end_date                  DATE,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, employee_id, position_id),
    FOREIGN KEY (host_id, position_id) REFERENCES position_t(host_id, position_id) ON DELETE CASCADE
);

Authorization Strategies

In order to link users to API endpoints for authorization, we will adpot the following approaches with a rule engine to enforce the policies in the sidecar of the API with access-control middleware handler.

A. Role-Based Access Control (RBAC)

This is a common and relatively simple approach. You define roles (e.g., “admin,” “editor,” “viewer”) and assign permissions to those roles. Users are then assigned to one or more roles.

Role Table:

CREATE TABLE role_t (
    host_id                   VARCHAR(22) NOT NULL,
    role_id                   VARCHAR(22) NOT NULL,
    role_name                 VARCHAR(255) UNIQUE NOT NULL,
    description               TEXT,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, role_id)
);

Role-Endpoint Permission Table:

CREATE TABLE role_permission_t (
    host_id                   VARCHAR(32) NOT NULL,
    role_id                   VARCHAR(32) NOT NULL,
    endpoint_id               VARCHAR(64) NOT NULL,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, role_id, endpoint_id),
    FOREIGN KEY (host_id, role_id) REFERENCES role_t(host_id, role_id) ON DELETE CASCADE,
    FOREIGN KEY (endpoint_id) REFERENCES api_endpoint_t(endpoint_id) ON DELETE CASCADE
);

Role-User Assignment Table:

CREATE TABLE role_user_t (
    host_id                   VARCHAR(22) NOT NULL,
    role_id                   VARCHAR(22) NOT NULL,
    user_id                   VARCHAR(22) NOT NULL,
    start_date DATE NOT NULL DEFAULT CURRENT_DATE,
    end_date DATE,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, role_id, user_id, start_date),
    FOREIGN KEY (user_id) REFERENCES user_t(user_id) ON DELETE CASCADE,
    FOREIGN KEY (host_id, role_id) REFERENCES role_t(host_id, role_id) ON DELETE CASCADE
);

B. User-Based Access Control (UBAC)

This approach assigns permissions directly to users, allowing for very fine-grained control. It’s more flexible but can become complex to manage if you have a lot of users and endpoints. It should only be used for temporary access.

User-Endpoint Permissions Table:

CREATE TABLE user_permission_t (
    user_id                   VARCHAR(22) NOT NULL,
    host_id                   VARCHAR(22) NOT NULL,
    endpoint_id               VARCHAR(22) NOT NULL,
    start_date DATE NOT NULL DEFAULT CURRENT_DATE,
    end_date DATE,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (user_id, host_id, endpoint_id),
    FOREIGN KEY (user_id) REFERENCES user_t(user_id) ON DELETE CASCADE,
    FOREIGN KEY (endpoint_id) REFERENCES api_endpoint_t(endpoint_id) ON DELETE CASCADE
);

C. Group-Based Access Control (GBAC)

You can group users into teams or departments and assign permissions to those groups. This is useful when you want to manage permissions for sets of users with similar access needs.

Groups Table:

CREATE TABLE group_t (
    host_id                   VARCHAR(32) NOT NULL,
    group_id                  VARCHAR(32) NOT NULL,
    group_name                VARCHAR(255) UNIQUE NOT NULL,
    description               TEXT,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, group_id)
);

Group-Endpoint Permission Table:

CREATE TABLE group_permission_t (
    host_id                   VARCHAR(32) NOT NULL,
    group_id                  VARCHAR(32) NOT NULL,
    endpoint_id               VARCHAR(32) NOT NULL,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, group_id, endpoint_id),
    FOREIGN KEY (host_id, group_id) REFERENCES group_t(host_id, group_id) ON DELETE CASCADE,
    FOREIGN KEY (endpoint_id) REFERENCES api_endpoint_t(endpoint_id) ON DELETE CASCADE
);

Group-User Membership Table:

CREATE TABLE group_user_t (
    host_id                   VARCHAR(22) NOT NULL,
    group_id                  VARCHAR(22) NOT NULL,
    user_id                   VARCHAR(22) NOT NULL,
    start_date DATE NOT NULL DEFAULT CURRENT_DATE,
    end_date DATE,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, group_id, user_id, start_date),
    FOREIGN KEY (user_id) REFERENCES user_t(user_id) ON DELETE CASCADE,
    FOREIGN KEY (host_id, group_id) REFERENCES group_t(host_id, group_id) ON DELETE CASCADE
);

D. Attribute-Based Access Control (ABAC)

Attribute Table:

CREATE TABLE attribute_t (
    host_id                   VARCHAR(22) NOT NULL,
    attribute_id              VARCHAR(22) NOT NULL,
    attribute_name            VARCHAR(255) UNIQUE NOT NULL, -- The name of the attribute (e.g., "department," "job_title," "project," "clearance_level," "location").
    attribute_type            VARCHAR(50) CHECK (attribute_type IN ('string', 'integer', 'boolean', 'date', 'float', 'list')), -- Define allowed data types
    description               TEXT,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, attribute_id)
);

2. Attribute User Table:

CREATE TABLE attribute_user_t (
    host_id                   VARCHAR(22) NOT NULL,
    attribute_id              VARCHAR(22) NOT NULL,
    user_id                   VARCHAR(22) NOT NULL, -- References users_t
    attribute_value           TEXT, -- Store values as strings; you can cast later
    start_date                DATE NOT NULL DEFAULT CURRENT_DATE,
    end_date                  DATE,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, attribute_id, user_id, start_date),
    FOREIGN KEY (user_id) REFERENCES user_t(user_id) ON DELETE CASCADE,
    FOREIGN KEY (host_id, attribute_id) REFERENCES attribute_t(host_id, attribute_id) ON DELETE CASCADE
);

3. Attribute Permission Table:

CREATE TABLE attribute_permission_t (
    host_id                   VARCHAR(32) NOT NULL,
    attribute_id              VARCHAR(32) NOT NULL,
    endpoint_id               VARCHAR(32) NOT NULL, -- References api_endpoints_t
    attribute_value           TEXT,
    update_user               VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_timestamp          TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (host_id, attribute_id, endpoint_id),
    FOREIGN KEY (endpoint_id) REFERENCES api_endpoint_t(endpoint_id) ON DELETE CASCADE,
    FOREIGN KEY (host_id, attribute_id) REFERENCES attribute_t(host_id, attribute_id) ON DELETE CASCADE
);

How it Works:

1. Define Attributes: Define all relevant attributes in attribute_t. Think about all the properties of your users, resources, and environment that might be used in access control decisions.

2. Assign Attributes to Users: Populate attribute_user_t to associate attribute values with users.

3. Assign Attributes to Endpoints: Populate attribute_permission_t to associate attribute values with API endpoints.

4. Write Policies: Create policy rules in rule engine. These rules should use the attribute names defined in attribute_t.

5. Policy Evaluation (at runtime):

• The policy engine receives the subject (user), resource (API endpoint), and action (HTTP method) of the request.

• The engine retrieves the relevant attributes from the user_attribute_t and attribute_permission_t tables.

• The engine evaluates the policy rule from the relevant policies against the attributes.

• Based on the policy evaluation result, access is either granted or denied.

Key Advantages of ABAC:

• Fine-Grained Control: Express very specific access rules.

• Centralized Policy Management: Policies are stored centrally and can be easily updated.

• Flexibility and Scalability: Adapts easily to changing requirements.

• Auditing and Compliance: Easier to audit and demonstrate compliance.

Format of attributes in JWT token:

Unlike roles, groups and positions that can be concatanated as a string, an attribut is a key/value pair. We need to format multiple attributes into a string and put it into a token.

Challenges

• Spaces: The primary issue is that simple key-value pairs like key1:value1 key2:value2 will not work when value contain spaces.

• Escaping: We need a way to escape characters that may confuse the parser, for example if the value also contains a :.

• Readability: The format should be reasonably readable for debugging and human consumption.

• Parsing: The format should be easy to parse on the application side.

Options

1. Comma-Separated Key-Value Pairs with Escaping:

• Format: key1=value1,key2=value2_with_spaces,key3=value3,with,commas

• Escaping: Use backslash \ to escape commas and backslashes within the values. You can also escape spaces to make it more clear \

• Pros: Simple to implement, relatively easy to parse using splitting by comma and then by =.

• Cons: Can become hard to read with complex values, requires proper escaping, will become unreadable if \ need to be escaped.

2. Custom Delimiter and Escaping:

• Format: key1^=^value1~key2^=^value2 with spaces~key3^=^value3~

• Delimiter: Use ^=^ as delimiter for key and value and use ~ for different attributes.

• Pros: You can avoid many escaping issues and keep spaces, easier to read than comma separated values.

• Cons: Need to choose delimiter carefully to make sure it is unique.

3. URL-Encoded Key-Value Pairs:

• Format: key1=value1&key2=value+with+spaces&key3=value3%2Cwith%2Ccommas

• Pros: Well-established standard, handles spaces and special characters well.

• Cons: Requires URL encoding and decoding, slightly more overhead, can be less readable.

• Recommended Approach: Custom Delimiter with Simple Escaping

We recommend the Custom Delimiter with Simple Escaping approach for your use case. It’s a good balance between simplicity, readability, and the ability to handle spaces within values. It avoids the need to rely on complex URL encoding and also avoids the unreadability issue of using comma with backslash escaping.

JWT Security Claims

Using the tables defined above, follow these steps to create an authorization code token with user security claims:

1. uid
   The entity_id (e.g., employee_id for employees and customer_id for customers) should be assigned to the uid claim in the JWT. This uid will be used by the response transformer to filter the response for the user and must represent a business identifier.

   Examples:

   • Employee: Use the ACF2 ID as the uid.
   • Customer: Use the CIF ID as the uid (e.g., in a banking context).

2. role
   Include a list of roles associated with the user.

3. grp
   Add a list of groups the user belongs to.

4. att
   Include a list of key-value pairs representing user attributes.

5. pos Include a list of positions for the user.

6. host The host of the user.

Example Token

eyJraWQiOiJUal9sX3RJQlRnaW5PdFFiTDBQdjV3IiwiYWxnIjoiUlMyNTYifQ.eyJpc3MiOiJ1cm46Y29tOm5ldHdvcmtudDpvYXV0aDI6djEiLCJhdWQiOiJ1cm46Y29tLm5ldHdvcmtudCIsImV4cCI6MTczNDA2NDU5NSwianRpIjoicEs4WEtDZkU1aVFSdWdlQThJWXBwZyIsImlhdCI6MTczNDA2Mzk5NSwibmJmIjoxNzM0MDYzODc1LCJ2ZXIiOiIxLjAiLCJ1aWQiOiJzaDM1IiwidXR5IjoiRSIsImNpZCI6ImY3ZDQyMzQ4LWM2NDctNGVmYi1hNTJkLTRjNTc4NzQyMWU3MiIsImNzcmYiOiItTUN4OGhZRlF1bVZ3NFZkRDVHbEd3Iiwic2NwIjpbInBvcnRhbC5yIiwicG9ydGFsLnciLCJyZWYuciIsInJlZi53Il0sInJvbGUiOiJhZG1pbiB1c2VyIiwiYzEiOiIzNjEiLCJjMiI6IjY3IiwiZ3JwIjoiZGVsZXRlIGluc2VydCBzZWxlY3QgdXBkYXRlIiwiYXR0IjoiY291bnRyeV49XkNBTn5wZXJhbmVudCBlbXBsb3llZV49XnRydWV-c2VjdXJpdHlfY2xlYXJhbmNlX2xldmVsXj1eMiIsInBvcyI6IkFQSVBsYXRmb3JtRGVsaXZlcnkiLCJob3N0IjoiTjJDTXcwSEdRWGVMdkMxd0JmbG4yQSJ9.Gky_rR9hreP04GZm-0H_HBBAeDIPhQ9tsNuZclUzTdkMrYay40kcNk4jWkPdMcxfIfIbGj2eqSQgNhkBuym2yc6HsRF0nukZhYSGklVNXFe3R-0DdKwxxWyqvXyWDvrQtme0ttT2tYGTRRCZXnHDRMUFeDSz7kVjjIj3WymjFyxWBnWnBOjYqDL34652Fb8c7hWME0nSxbWO0ZvPRDhRM-l0nDGNm2ojq-3sjaU_pRywYahXP-wtnNSLwvctFgONPWSM9Ie6FqwRmYBFVo8OE0VdTRvUfnO4mL1O2UbTfxzbNJFv4HP1mSZG_SSB5j3t_RuZLfUMIajFi105ze2PUg

And the payload:

{
  "iss": "urn:com:networknt:oauth2:v1",
  "aud": "urn:com.networknt",
  "exp": 1734064595,
  "jti": "pK8XKCfE5iQRugeA8IYppg",
  "iat": 1734063995,
  "nbf": 1734063875,
  "ver": "1.0",
  "uid": "sh35",
  "uty": "E",
  "cid": "f7d42348-c647-4efb-a52d-4c5787421e72",
  "csrf": "-MCx8hYFQumVw4VdD5GlGw",
  "scp": [
    "portal.r",
    "portal.w"
  ],
  "role": "admin user",
  "c1": "361",
  "c2": "67",
  "grp": "delete insert select update",
  "att": "country^=^CAN~peranent employee^=^true~security_clearance_level^=^2",
  "pos": "APIPlatformDelivery",
  "host": "N2CMw0HGQXeLvC1wBfln2A"
}

Group and Position Management

Define Groups Related to User Category

You can create groups that align with teams, departments, or other organizational units. These groups are relatively static and reflect the overall organizational structure. Use a separate table, group_t, as described earlier, to store these groups. Groups can be applied to all users regardless of their user type.

Use the Employee Reporting Structure to Manage Positions

Positions are similar to groups in managing user permissions, but they leverage the organizational reporting structure to propagate permissions between team members and their direct manager.

• Position Flags

  Each position in the position_t table has two flags:

• inherit_to_ancestor: Determines if the position is inherited by a subordinate.
• inherit_to_sibling: Determines if the position is inherited by team members (siblings) under the same manager.

• Responsibilities

  The application is responsible for propagating positions:

• Between Siblings: Assigning inherited positions to team members under the same manager.
• To the Manager: Assigning inherited positions to the direct manager.

• User Interface for Position Management

  A user interface (UI) can be implemented to simplify position management:

• Feature: List all potential inherited positions for selection when adding a new user or changing a manager.
• Functionality: Allow administrators to choose specific positions to inherit for users and managers dynamically.

Use Both Groups and Positions

You can choose to use both groups and positions for your organization. However, you need to ensure that groups and positions categorize users across different dimensions. In general, groups should be used for customers, while positions should be used for employees.

User Login Query

Here is the query to run against the database tables upon a user login request:

SELECT
    u.user_id,
    u.user_type,
    CASE
        WHEN u.user_type = 'E' THEN e.employee_id
        WHEN u.user_type = 'C' THEN c.customer_id
        ELSE NULL
    END AS entity_id,
    CASE WHEN u.user_type = 'E' THEN string_agg(DISTINCT p.position_name, ' ' ORDER BY p.position_name) ELSE NULL END AS positions,
    string_agg(DISTINCT r.role_name, ' ' ORDER BY r.role_name) AS roles,
    string_agg(DISTINCT g.group_name, ' ' ORDER BY g.group_name) AS groups,
     CASE
        WHEN COUNT(DISTINCT at.attribute_name || '^=^' || aut.attribute_value) > 0 THEN string_agg(DISTINCT at.attribute_name || '^=^' || aut.attribute_value, '~' ORDER BY at.attribute_name || '^=^' || aut.attribute_value)
        ELSE NULL
    END AS attributes
FROM
    user_t AS u
LEFT JOIN
    user_host_t AS uh ON u.user_id = uh.user_id
LEFT JOIN
    role_user_t AS ru ON u.user_id = ru.user_id
LEFT JOIN
    role_t AS r ON ru.host_id = r.host_id AND ru.role_id = r.role_id
LEFT JOIN
    attribute_user_t AS aut ON u.user_id = aut.user_id
LEFT JOIN
    attribute_t AS at ON aut.host_id = at.host_id AND aut.attribute_id = at.attribute_id
LEFT JOIN
    group_user_t AS gu ON u.user_id = gu.user_id
LEFT JOIN
    group_t AS g ON gu.host_id = g.host_id AND gu.group_id = g.group_id
LEFT JOIN
    employee_t AS e ON uh.host_id = e.host_id AND u.user_id = e.user_id
LEFT JOIN
    customer_t AS c ON uh.host_id = c.host_id AND u.user_id = c.user_id
LEFT JOIN
    employee_position_t AS ep ON e.host_id = ep.host_id AND e.employee_id = ep.employee_id
LEFT JOIN
    position_t AS p ON ep.host_id = p.host_id AND ep.position_id = p.position_id
WHERE
    u.email = '[email protected]'
GROUP BY
    u.user_id, u.user_type, e.employee_id, c.customer_id;

And here is an example result from the test database:

utgdG50vRVOX3mL1Kf83aA  E   sh35    APIPlatformDelivery admin user  delete insert select update country^=^CAN~peranent employee^=^true~security_clearance_level^=^2

Parse Attribute String

The query above returns attributes in a customized format. These attributes can be parsed using the Util.parseAttributes method available in the light-4j utility module

Portal View and Default Role

Given the flexibility of fine-grained authorization approaches, users can choose one or more methods to suit their business requirements. However, in scenarios where RBAC (Role-Based Access Control) is not utilized, the role claim may not exist in the custom claims of the JWT token.

Handling Missing role in JWT

For the portal-view application, at least one role is required to filter menu items. To address cases where no roles are present in the JWT:

1. Default Role Assignment:
   If the role claim is absent in the JWT, the system will:

   • Assign a default role, "user", to ensure compatibility.
   • Include this role in a roles field in the browser cookie.

2. Cookie Roles Field:

   • The roles field in the cookie will contain a single role: "user".
   • This ensures the portal-view can still function as expected by displaying the appropriate menu items for users.

Example Workflow

1. A user authenticates, and their JWT is generated without a role claim.
2. During authentication handling:
   • The StatelessAuthHandler checks for the presence of the role claim.
   • If no roles are found, the "user" role is added to the roles field in the cookie.
3. The portal-view reads the roles field from the cookie to filter menu items appropriately.

This approach provides a seamless experience while maintaining compatibility with applications requiring roles for authorization or UI customization.

Private Messages

Problem

Portal users need a way to exchange private messages from the user profile without exposing email addresses to each other. The sender should only need a recipient user id or a display-safe user label. The backend can resolve email internally when it needs to send an external notification, but email must not be part of the user-facing message contract.

Current State

The current codebase already has a partial private-message skeleton:

• user-command exposes lightapi.net/user/sendMessage/0.1.0.
• The sendMessage request contains userId, subject, and content.
• light-portal defines PrivateMessageSentEvent.
• portal-db defines message_t.
• portal-view has a mail menu, a private messages page, and a privateMessage form.
• user-query exposes lightapi.net/user/getPrivateMessage/0.1.0.

The current implementation is not complete enough to support production use:

• GetPrivateMessage has its real implementation commented out and currently returns null.
• SendMessage resolves the recipient through queryUserById, then stores the whole response as toEmail. That lookup currently returns too much user data, including email and sensitive fields that should not be exposed through a peer messaging flow.
• SendMessage does not put fromId into event data, but the projection code reads fromId from event data.
• The message_t table now has host_id NOT NULL, but the projection insert does not write host_id.
• The table is inbox-style storage, keyed by sender and nonce, and does not model conversations, read state, participant visibility, or per-user delete.
• The UI mostly relies on the mail menu response and navigation state. The messages page should load its own data from the query API.
• The existing private-message tests are disabled stubs.

Goals

• Let one logged-in user send a message to another portal user without knowing or seeing the recipient email.
• Keep the message model host-scoped so tenant boundaries are explicit.
• Derive sender identity from the authorization token, not from form input.
• Store user ids in message records and events. Do not store recipient email in the message projection unless a short migration bridge requires it.
• Support an inbox page, unread badge, conversation view, reply, read state, and per-user hide/delete.
• Keep email notification as an optional side effect that resolves the recipient email internally.
• Provide a path from the existing message_t skeleton to a conversation-based model without breaking existing UI routes immediately.

Non-Goals

• Do not build group chat in the first phase.
• Do not expose email addresses in message APIs, events, UI state, or task context.
• Do not use private messages as an audit or support-ticket system.
• Do not implement WebSocket or SSE push in the first phase. Polling is enough until the read/write model is stable.
• Do not make public user lookup broader as part of this feature.

Privacy Rules

Private messages should be user-id based at every external boundary.

The UI may show:

• Display name.
• Avatar or initials.
• User id when no better label exists.
• Message subject, preview, content, and timestamps.

The UI must not show:

• Sender email.
• Recipient email.
• Password, token, nonce, or other profile internals from user_t.

The backend may resolve recipient email only inside trusted server code for external email notification. That internal lookup should return the minimum fields required, ideally user_id, email, current host membership, and a display label.

Recommended Data Model

For a chat-like experience, introduce conversation identity instead of treating each message as an isolated inbox row.

CREATE TABLE private_conversation_t (
    host_id              UUID NOT NULL,
    conversation_id      UUID NOT NULL,
    participant_low_id   UUID NOT NULL,
    participant_high_id  UUID NOT NULL,
    created_ts           TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
    last_message_id      UUID NULL,
    last_message_ts      TIMESTAMP WITH TIME ZONE NULL,
    PRIMARY KEY (host_id, conversation_id),
    UNIQUE (host_id, participant_low_id, participant_high_id),
    FOREIGN KEY (host_id) REFERENCES host_t(host_id) ON DELETE CASCADE
);

participant_low_id and participant_high_id are the two sorted user ids. This gives each pair of users one stable conversation per host without relying on email.

CREATE TABLE private_message_t (
    host_id          UUID NOT NULL,
    message_id       UUID NOT NULL,
    conversation_id  UUID NOT NULL,
    from_user_id     UUID NOT NULL,
    to_user_id       UUID NOT NULL,
    subject          VARCHAR(256) NULL,
    content          TEXT NOT NULL,
    send_ts          TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY (host_id, message_id),
    FOREIGN KEY (host_id, conversation_id)
        REFERENCES private_conversation_t(host_id, conversation_id)
        ON DELETE CASCADE
);

CREATE TABLE private_message_state_t (
    host_id      UUID NOT NULL,
    message_id   UUID NOT NULL,
    user_id      UUID NOT NULL,
    read_ts      TIMESTAMP WITH TIME ZONE NULL,
    deleted_ts   TIMESTAMP WITH TIME ZONE NULL,
    PRIMARY KEY (host_id, message_id, user_id),
    FOREIGN KEY (host_id, message_id)
        REFERENCES private_message_t(host_id, message_id)
        ON DELETE CASCADE
);

The state table keeps read and delete behavior per participant. A user deleting a message should hide it from that user only. It should not erase the other participant’s copy.

Recommended indexes:

CREATE INDEX idx_private_conversation_last_message
    ON private_conversation_t (host_id, participant_low_id, participant_high_id, last_message_ts DESC);

CREATE INDEX idx_private_message_conversation_ts
    ON private_message_t (host_id, conversation_id, send_ts DESC);

CREATE INDEX idx_private_message_to_user_ts
    ON private_message_t (host_id, to_user_id, send_ts DESC);

CREATE INDEX idx_private_message_state_unread
    ON private_message_state_t (host_id, user_id)
    WHERE read_ts IS NULL AND deleted_ts IS NULL;

If the first implementation needs to reuse message_t, treat it as a migration bridge only. Add from_user_id, to_user_id, message_id, read_ts, and per-user delete columns, then migrate to the conversation tables once the API contract is stable.

Event Model

Keep the event-driven command/query pattern. A message send should create a CloudEvent and the query-side projection should update the private-message tables.

Recommended event data:

{
  "hostId": "019...",
  "conversationId": "019...",
  "messageId": "019...",
  "fromUserId": "019...",
  "toUserId": "019...",
  "subject": "Question about the API",
  "content": "Can you take a look at this?"
}

fromUserId and hostId are derived from the token. toUserId, subject, and content come from validated request data. conversationId can be generated by the command side after looking up or creating the pair conversation, or it can be derived during projection from the participant pair.

Do not put toEmail into PrivateMessageSentEvent. Email notification should be a separate trusted server-side action.

API Contracts

Send Message

Keep the existing sendMessage action name for compatibility, but change the contract to be user-id based.

{
  "toUserId": "019...",
  "conversationId": "019...",
  "subject": "Question about the API",
  "content": "Can you take a look at this?"
}

conversationId is optional. If absent, the backend resolves or creates the conversation for the current user and toUserId.

Server responsibilities:

• Require an authorization-code token.
• Derive fromUserId from the token.
• Derive hostId from the active user host.
• Validate that toUserId belongs to the same host.
• Reject empty content and enforce size limits.
• Optionally reject self-messages unless a product decision allows notes to self.
• Write the event through the existing command event-store path.
• Send optional external email notification after the command is accepted.

Conversation List

Add or evolve a query endpoint for the inbox list.

{
  "offset": 0,
  "limit": 25
}

The backend derives hostId and userId from the token. The response should include only conversations involving the current user.

{
  "total": 1,
  "conversations": [
    {
      "conversationId": "019...",
      "otherUserId": "019...",
      "otherUserLabel": "Jane Smith",
      "lastMessageTs": "2026-05-08T13:30:00Z",
      "lastMessagePreview": "Can you take a look at this?",
      "unreadCount": 2
    }
  ]
}

Conversation Messages

{
  "conversationId": "019...",
  "offset": 0,
  "limit": 50
}

The backend validates that the current user is one of the participants.

{
  "conversationId": "019...",
  "messages": [
    {
      "messageId": "019...",
      "fromUserId": "019...",
      "fromUserLabel": "Jane Smith",
      "subject": "Question about the API",
      "content": "Can you take a look at this?",
      "sendTs": "2026-05-08T13:30:00Z",
      "read": false
    }
  ]
}

Unread Count

The mail badge should call a count endpoint instead of loading all messages.

{
  "count": 3
}

Mark Read and Delete

markPrivateConversationRead should mark unread rows in private_message_state_t for the current user and conversation.

deletePrivateMessage or hidePrivateConversation should set deleted_ts for the current user only.

Operational Cleanup

Private messages are user content, not operational status rows. They should not be hard-deleted only because they are old while either participant can still see them.

The operational cleanup job may purge active private-message rows only when all participant state rows for the message have deleted_ts set and the latest deleted_ts is older than privateMessageRetentionDays.

Cleanup responsibilities:

• Select purge candidates from private_message_t joined to private_message_state_t.
• Require every participant state row for the message to have deleted_ts set.
• Use MAX(deleted_ts) as the retention clock so the grace period starts after the last participant deletes the message.
• Delete private_message_state_t rows first, then delete the private_message_t row in the same transaction.
• Leave private_conversation_t rows in place so the participant pair keeps a stable conversation identity if a new message is sent later.
• Skip private-message cleanup when privateMessageRetentionDays is less than or equal to zero.

The cleanup job should not purge visible messages, partially deleted messages, or recently deleted-by-all messages. A separate maximum retention policy for undeleted private messages would need an explicit product/security decision.

Authorization

The command and query handlers must not trust user ids supplied by the client for the current user. The current user is always the token subject.

Rules:

• A sender can send only as themself.
• A user can read only conversations where they are a participant.
• A user can mark read or delete only their own state rows.
• Admin visibility should be a separate explicit support/admin endpoint if it is needed later.
• Cross-host messaging should be rejected in the first phase. If cross-host messaging is later needed, the contract must model the recipient host explicitly and pass a product/security review.

Portal View

Use the current profile surfaces but make them data-driven:

• MailMenu should poll unread count and show a small list of recent conversations only after the menu opens.
• /app/messages should fetch conversation data directly. It should not depend on location.state from MailMenu.
• The privateMessage form should use toUserId, not userId, to avoid confusing recipient identity with the current user.
• Reply should prefill toUserId and optionally conversationId.
• User-facing labels should come from a display-safe user label endpoint.
• Empty inbox, loading, and error states should be explicit.

The first UI can be an inbox plus conversation thread. Real-time typing, presence, attachments, and rich-text editing are later enhancements.

Migration Plan

Phase 0: Stop the Broken Behavior

• Make GetPrivateMessage return valid JSON even before the new model is complete.
• Fix the existing projection insert to include host_id if message_t remains in use.
• Ensure SendMessage stores sender identity from the token.
• Stop using broad queryUserById output as a recipient email value.

Phase 1: User-ID Based Backend

• Add the conversation/message/state tables.
• Update PrivateMessageSentEvent to use fromUserId and toUserId.
• Add a trusted recipient resolver that returns only internal fields needed for validation and optional email notification.
• Implement conversation list, conversation messages, unread count, mark-read, and hide/delete APIs.

Phase 2: Portal View

• Update the mail badge to use unread count.
• Update /app/messages to load data directly.
• Update the privateMessage form and reply paths to use toUserId.
• Remove email assumptions from task context and UI state.

Phase 3: Cleanup

• Remove to_email from the active private-message path.
• Remove disabled private-message tests and replace them with focused coverage.
• Ensure operational cleanup targets the active private-message tables and purges only messages deleted by all participants after the retention window.
• Add optional push delivery later if polling becomes insufficient.

Testing

Backend tests should cover:

• Sender is derived from token and cannot be spoofed.
• Recipient must belong to the current host.
• Message event contains user ids, not emails.
• Projection writes host-scoped conversation and message rows.
• Inbox query returns only conversations for the current user.
• Conversation query rejects non-participants.
• Unread count increments for the recipient and clears after mark-read.
• Delete/hide affects only the current user’s state.
• Operational cleanup purges only messages deleted by all participants after retention and keeps visible, partially deleted, and recently deleted messages.

Frontend tests should cover:

• Mail menu shows unread count without loading full inbox.
• Messages page fetches its own data.
• Reply pre-populates recipient context without email.
• Empty and error states do not produce JSON parse failures.

Open Questions

• Should users be able to send messages to themselves as private notes?
• Should profile pages expose a “Message” action only for users in the same host, or should some cross-host flows be allowed?
• Should email notification include the sender display label, or only say that a portal message was received?
• Should any maximum retention policy apply to undeleted private messages?
• Should administrators have a separate support/audit view, and under what permission?

Config Server

Default Config Properties

For each config class in light-4j modules, we use annotations to generate schemas for the config files with default values, comments and validation rules.

As one time step, we also generate events to input all the properties into the light-portal. These events will create a base-line of the config properties with default values. All events in this first time population doesn’t have a version.

For each version release, we will create and attach an event.json file with the change to the properties. Most likely, we will add some properties with default values for each release. All events in the is file will have a version associated. Once played on the portal, updates for the version will be populated.

On the portal ui, we load all properties and default values from database with a union of the base-line properties and all versions below and equal to the current version.

Instance Config Snapshot

Once a logical instance is created on the light-portal, we need to provide the product_version_id which will map to a specific product version. We also need to provide runtime configuration and deployment configuration for the instance to start the server and deploy it to a target environment. During the configuration updates, it might be a process of discovery and may take several revisit to complete. If a user makes a mistake, he/she might want to rollback the previous changes to a snapshot version to start it over again. During the deployment, we also need to save and tag the snapshot version so that we can rollback to the previous deployment configuration snapshot in case of deployment failure.

The above requirements force us to create a table that is record all the commit for the config updates at instance level. It is like a GitHub commit to group several updates together. The user needs to explicitly click the commit button on the UI to allow the server to run the query to populate the snapshot table to create a new snapshot id.

Durng the deployment, the deployment serivce will invoke the config server to force a commit and also link that commit to a deployment id just like a tag in GitHub.

To meet the requirement above, we need to design tables to store immutable snapshots associated with a commitId/snapshotId to proivde reliable rollback points.

Snapshot tables

CREATE TABLE config_snapshot_t (
    snapshot_id                 UUID NOT NULL, -- Primary Key, maybe UUIDv7 for time ordering
    snapshot_ts                 TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
    snapshot_type               VARCHAR(32) NOT NULL, -- e.g., 'DEPLOYMENT', 'USER_SAVE', 'SCHEDULED_BACKUP'
    description                 TEXT,                 -- User-provided description or system-generated info
    user_id                     UUID,                 -- User who triggered it (if applicable)
    deployment_id               UUID,                 -- FK to deployment_t if snapshot_type is 'DEPLOYMENT'
    -- Scope columns define WHAT this snapshot represents:
    scope_host_id               UUID NOT NULL,      -- Host context (always needed)
    scope_config_phase          CHAR(1) NOT NULL,   -- config phase context(required)
    scope_environment           VARCHAR(16),        -- Environment context (if snapshot is env-specific)
    scope_product_id            VARCHAR(8)          -- Product id context
    scope_product_version       VARCHAR(12)         -- Product version context
    scope_service_id            VARCHAR(512)        -- Service id context
    scope_api_id                VARCHAR(16)         -- Api id context
    scope_api_version           VARCHAR(16)         -- Api version context
    PRIMARY KEY(snapshot_id),
    FOREIGN KEY(deployment_id) REFERENCES deployment_t(deployment_id) ON DELETE SET NULL,
    FOREIGN KEY(user_id) REFERENCES user_t(user_id) ON DELETE SET NULL,
    FOREIGN KEY(scope_host_id) REFERENCES host_t(host_id) ON DELETE CASCADE
);

-- Index for finding snapshots by type or scope
CREATE INDEX idx_config_snapshot_scope ON config_snapshot_t (scope_host_id, scope_config_phase, scope_environment, 
    scope_product_id, scope_product_version, scope_service_id, scope_api_id, scope_api_version, snapshot_type, snapshot_ts);
CREATE INDEX idx_config_snapshot_deployment ON config_snapshot_t (deployment_id);


CREATE TABLE config_snapshot_property_t (
    snapshot_property_id        UUID NOT NULL,         -- Surrogate primary key for easier referencing/updates if needed
    snapshot_id                 UUID NOT NULL,         -- FK to config_snapshot_t
    config_id                   UUID NOT NULL,         -- The config id
    property_id                 UUID NOT NULL,         -- The final property id 
    property_name               VARCHAR(64) NOT NULL,  -- The final property name
    property_type               VARCHAR(32) NOT NULL,  -- The property type
    property_value              TEXT,                  -- The effective property value at snapshot time
    value_type                  VARCHAR(32),           -- Optional: Store the type (string, int, bool...) for easier parsing later
    source_level                VARCHAR(32),           -- e.g., 'instance', 'product_version', 'environment', 'default'
    PRIMARY KEY(snapshot_property_id),
    FOREIGN KEY(snapshot_id) REFERENCES config_snapshot_t(snapshot_id) ON DELETE CASCADE
);

-- Unique constraint to ensure one value per key within a snapshot
ALTER TABLE config_snapshot_property_t
    ADD CONSTRAINT config_snapshot_property_uk UNIQUE (snapshot_id, config_id, property_id);

-- Index for quickly retrieving all properties for a snapshot
CREATE INDEX idx_config_snapshot_property_snapid ON config_snapshot_property_t (snapshot_id);


-- Snapshot of Instance API Overrides
CREATE TABLE snapshot_instance_api_property_t (
    snapshot_id         UUID NOT NULL,
    host_id             UUID NOT NULL,
    instance_api_id     UUID NOT NULL,
    property_id         UUID NOT NULL,
    property_value      TEXT,
    update_user         VARCHAR (255) NOT NULL,
    update_ts           TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY(snapshot_id, host_id, instance_api_id, property_id), -- Composite PK matches original structure + snapshot_id
    FOREIGN KEY(snapshot_id) REFERENCES config_snapshot_t(snapshot_id) ON DELETE CASCADE
);
CREATE INDEX idx_snap_iapi_prop ON snapshot_instance_api_property_t (snapshot_id);


-- Snapshot of Instance App Overrides
CREATE TABLE snapshot_instance_app_property_t (
    snapshot_id         UUID NOT NULL,
    host_id             UUID NOT NULL,
    instance_app_id     UUID NOT NULL,
    property_id         UUID NOT NULL,
    property_value      TEXT,
    update_user         VARCHAR (255) NOT NULL,
    update_ts           TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY(snapshot_id, host_id, instance_app_id, property_id),
    FOREIGN KEY(snapshot_id) REFERENCES config_snapshot_t(snapshot_id) ON DELETE CASCADE
);
CREATE INDEX idx_snap_iapp_prop ON snapshot_instance_app_property_t (snapshot_id);

-- Snapshot of Instance App API Overrides
CREATE TABLE snapshot_instance_app_api_property_t (
    snapshot_id         UUID NOT NULL,
    host_id             UUID NOT NULL,
    instance_app_id     UUID NOT NULL,
    instance_api_id     UUID NOT NULL,
    property_id         UUID NOT NULL,
    property_value      TEXT,
    update_user         VARCHAR (255) NOT NULL,
    update_ts           TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY(snapshot_id, host_id, instance_app_id, instance_api_id, property_id),
    FOREIGN KEY(snapshot_id) REFERENCES config_snapshot_t(snapshot_id) ON DELETE CASCADE
);
CREATE INDEX idx_snap_iaappi_prop ON snapshot_instance_app_api_property_t (snapshot_id);


-- Snapshot of Instance Overrides
CREATE TABLE snapshot_instance_property_t (
    snapshot_id         UUID NOT NULL,
    host_id             UUID NOT NULL,
    instance_id         UUID NOT NULL,
    property_id         UUID NOT NULL,
    property_value      TEXT,
    update_user         VARCHAR (255) NOT NULL,
    update_ts           TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY(snapshot_id, host_id, instance_id, property_id),
    FOREIGN KEY(snapshot_id) REFERENCES config_snapshot_t(snapshot_id) ON DELETE CASCADE
);
CREATE INDEX idx_snap_inst_prop ON snapshot_instance_property_t (snapshot_id);


-- Snapshot of Environment Overrides (If needed for rollback)
CREATE TABLE snapshot_environment_property_t (
    snapshot_id         UUID NOT NULL,
    host_id             UUID NOT NULL,
    environment         VARCHAR(16) NOT NULL,
    property_id         UUID NOT NULL,
    property_value      TEXT,
    update_user         VARCHAR (255) NOT NULL,
    update_ts           TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY(snapshot_id, host_id, environment, property_id),
    FOREIGN KEY(snapshot_id) REFERENCES config_snapshot_t(snapshot_id) ON DELETE CASCADE
);
CREATE INDEX idx_snap_env_prop ON snapshot_environment_property_t (snapshot_id);

CREATE TABLE snapshot_product_property_t (
    snapshot_id         UUID NOT NULL,
    product_id          VARCHAR(8) NOT NULL,
    property_id         UUID NOT NULL,
    property_value      TEXT,
    update_user         VARCHAR (255) NOT NULL,
    update_ts           TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY(snapshot_id, product_id, property_id),
    FOREIGN KEY(snapshot_id) REFERENCES config_snapshot_t(snapshot_id) ON DELETE CASCADE
);
CREATE INDEX idx_snap_prd_prop ON snapshot_product_property_t (snapshot_id);

CREATE TABLE snapshot_product_version_property_t (
    snapshot_id         UUID NOT NULL,
    host_id             UUID NOT NULL,
    product_version_id  UUID NOT NULL,
    property_id         UUID NOT NULL,
    property_value      TEXT,
    update_user         VARCHAR (255) NOT NULL,
    update_ts           TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY(snapshot_id, host_id, product_version_id, property_id),
    FOREIGN KEY(snapshot_id) REFERENCES config_snapshot_t(snapshot_id) ON DELETE CASCADE
);
CREATE INDEX idx_snap_pv_prop ON snapshot_product_version_property_t (snapshot_id);

How to generate rollback events

There are two options to generate rollback events or compensate events.

Option 1. With historical events.

1. Identify Target State: You have a snapshot_id representing the desired historical state.
2. Find Snapshot Timestamp: Get the snapshot_ts from config_snapshot_t for the target snapshot_id.
3. Query Events: Find all configuration events in your event store that:
   • Occurred after the snapshot_ts.
   • Relate to the specific scope (host, instance, environment, etc.) being rolled back.
4. Generate Compensating Events: For each event found in step 3, create its logical inverse (a “compensating event”). For example:
   • InstancePropertyUpdated { propertyId: X, newValue: B, oldValue: A } -> InstancePropertyUpdated { propertyId: X, newValue: A, oldValue: B } (Requires storing oldValue in the original event).
   • InstancePropertyCreated { propertyId: X, value: A } -> InstancePropertyDeleted { propertyId: X, value: A } (Requires storing the value in the delete event for potential future rollback).
   • InstancePropertyDeleted { propertyId: X, value: A } -> InstancePropertyCreated { propertyId: X, value: A } (Requires storing the value in the delete event).
5. Order Compensating Events: Sort the generated compensating events in the reverse chronological order of the original events they are compensating for.
6. Replay Compensating Events: Apply these ordered compensating events through your event handling system.

Conceptually, this is a valid approach often used in event sourcing patterns (related to compensating transactions). However, it comes with significant challenges and complexities:

Challenges & Considerations:

1. Generating Perfect Inverse Events: This is the hardest part.
   • Requires Rich Events: Your original events must contain enough information to construct their inverse. For updates, you need the oldValue. For creations, the delete needs the key. For deletions, the create needs the deleted value. If your current events don’t store this, you cannot reliably generate compensating events this way.
   • Complexity: For multi-step or complex operations, determining the exact inverse sequence can be non-trivial.
2. Order of Operations: Compensating events MUST be applied in strict reverse order. Getting this wrong can lead to incorrect states.
3. State Dependencies: Event handlers sometimes make assumptions about the state before the event is applied. Replaying compensating events might encounter unexpected states if other unrelated changes have occurred or if the reverse logic isn’t perfect, potentially causing handler errors.
4. Performance: Querying potentially thousands of events, generating inverses, and replaying them might be slow, especially if the time gap between the snapshot and the present is large.
5. Snapshot Data Not Used: This approach doesn’t directly leverage the known good state stored in config_snapshot_property_t. It relies solely on the ability to perfectly reverse subsequent events.
6. Idempotency: Compensating event handlers should ideally be idempotent (applying them multiple times has the same effect as applying them once), although this is hard to guarantee for inverse operations.

Option 2: Diff-based event generation.

1. Get Target State: Fetch key-values from config_snapshot_property_t for snapshot_id. (TargetState)
2. Get Current State: Run aggregation query for the current configuration. (CurrentState)
3. Calculate Diff: Find differences between TargetState and CurrentState.
4. Generate Corrective Events: Create events to transform CurrentState into TargetState.
   • If key is in TargetState but different/missing in CurrentState -> Generate Upsert[Level]Property event with the value from TargetState (applied at the highest relevant override level for the scope).
   • If key is in CurrentState but missing in TargetState -> Generate Delete[Level]Property event for the override that’s currently providing the value (likely the highest relevant override level).
5. Apply Events: Apply these corrective events.

Why the Diff-Based Approach is Often Preferred for Snapshot Rollback:

• Uses Known Good State: It directly uses the guaranteed state from the snapshot table.
• Less Reliant on Event Reversibility: It doesn’t matter if the original events are perfectly reversible or store old values. It focuses on achieving the target state from the current state.
• Potentially Fewer Events: Might generate fewer events than reversing a long history, focusing only on the net changes needed.
• More Direct: The generated events directly aim to establish the target state, which can feel less fragile than relying on reversing history.

Conclusion:

While method of reversing events since the snapshot is a recognized event sourcing pattern, it’s often more complex and potentially fragile for the specific task of rolling back to a known snapshot state compared to the diff-based corrective event generation method.

The diff-based method leverages the snapshot data directly and focuses on achieving the target state, making it generally more robust and often easier to implement correctly, as it doesn’t require perfectly reversible events.

How to create the snapshot

Let’s clarify how the scope_* columns in config_snapshot_t relate to the query that generates the snapshot and the override tables (*_property_t).

The Purpose of scope_* Columns:

The scope_* columns in config_snapshot_t serve one primary purpose: To record the specific context for which the snapshot was generated. They define what set of effective configuration values are stored in the associated config_snapshot_property_t rows.

Think of them as the input parameters that were used to run the aggregation query when the snapshot was created.

How They Are Used in the Snapshot Generation Query:

You do not need one scope_* column for every *_property_t table. Instead, the values you store in the scope_* columns are the parameters you pass into your aggregation query’s WHERE clauses to filter the rows from the relevant override tables according to the desired context.

Let’s refine the query strategy using the scope_* concept and aim for a more efficient query than repeated NOT EXISTS clauses (using ROW_NUMBER() or DISTINCT ON).

Example Scenario: Snapshotting for a specific Instance

Let’s say you want to create a snapshot for a specific instance_id on a specific host_id.

1. Input Parameters:

   • p_host_id (UUID)
   • p_instance_id (UUID)

2. Derive Related IDs (Inside your snapshot creation logic/service):

   • You’ll need to query instance_t to get the associated product_version_id, environment, etc., for this instance.
   • Query product_version_t to get product_id.
   • Let’s call these derived values v_product_version_id, v_environment, v_product_id.

3. config_snapshot_t Record:

   • Generate a snapshot_id (e.g., UUIDv7).
   • snapshot_ts: CURRENT_TIMESTAMP
   • snapshot_type: e.g., ‘DEPLOYMENT’
   • scope_host_id: p_host_id
   • scope_instance_id: p_instance_id
   • scope_environment: v_environment (Store the derived environment for clarity, even though it came from the instance)
   • scope_product_version_id: v_product_version_id (Store for clarity)
   • scope_product_id: v_product_id (Store for clarity)
   • (Other scope_* columns like scope_instance_api_id would be NULL for this instance-level snapshot)

4. Aggregation Query (Using ROW_NUMBER()): This query uses the input parameters (p_host_id, p_instance_id) and the derived values (v_product_version_id, v_environment, v_product_id) to find the highest priority value for each property_id.

WITH – Parameters derived before running this query: – p_host_id UUID – p_instance_id UUID – v_product_version_id UUID (derived from p_instance_id) – v_environment VARCHAR(16) (derived from p_instance_id) – v_product_id VARCHAR(8) (derived from v_product_version_id)

– Find relevant instance_api_ids and instance_app_ids for the target instance RelevantInstanceApis AS ( SELECT instance_api_id FROM instance_api_t WHERE host_id = ? – p_host_id AND instance_id = ? – p_instance_id ), RelevantInstanceApps AS ( SELECT instance_app_id FROM instance_app_t WHERE host_id = ? – p_host_id AND instance_id = ? – p_instance_id ),

– Pre-process Instance App API properties with merging logic Merged_Instance_App_Api_Properties AS ( SELECT iaap.property_id, CASE cp.value_type WHEN ‘map’ THEN COALESCE(jsonb_merge_agg(iaap.property_value::jsonb), ‘{}’::jsonb)::text WHEN ‘list’ THEN COALESCE((SELECT jsonb_agg(elem ORDER BY iaa.update_ts) – Order elements based on when they were added via the link table? Or property update_ts? Assuming property update_ts. Check data model if linking time matters more. FROM jsonb_array_elements(sub.property_value::jsonb) elem WHERE jsonb_typeof(sub.property_value::jsonb) = ‘array’ ), ‘[]’::jsonb)::text – Requires subquery if ordering elements – Subquery approach for ordering list elements by property timestamp: /* COALESCE( (SELECT jsonb_agg(elem ORDER BY prop.update_ts) FROM instance_app_api_property_t prop, jsonb_array_elements(prop.property_value::jsonb) elem WHERE prop.host_id = iaap.host_id AND prop.instance_app_id = iaap.instance_app_id AND prop.instance_api_id = iaap.instance_api_id AND prop.property_id = iaap.property_id AND jsonb_typeof(prop.property_value::jsonb) = ‘array’ ), ‘[]’::jsonb )::text / ELSE MAX(iaap.property_value) – For simple types, MAX can work if only one entry expected, otherwise need timestamp logic – More robust for simple types: Pick latest based on timestamp / (SELECT property_value FROM instance_app_api_property_t latest WHERE latest.host_id = iaap.host_id AND latest.instance_app_id = iaap.instance_app_id AND latest.instance_api_id = iaap.instance_api_id AND latest.property_id = iaap.property_id ORDER BY latest.update_ts DESC LIMIT 1) */ END AS effective_value FROM instance_app_api_property_t iaap JOIN config_property_t cp ON iaap.property_id = cp.property_id JOIN instance_app_api_t iaa ON iaa.host_id = iaap.host_id AND iaa.instance_app_id = iaap.instance_app_id AND iaa.instance_api_id = iaap.instance_api_id – Join to potentially use its timestamp for ordering lists WHERE iaap.host_id = ? – p_host_id AND iaap.instance_app_id IN (SELECT instance_app_id FROM RelevantInstanceApps) AND iaap.instance_api_id IN (SELECT instance_api_id FROM RelevantInstanceApis) GROUP BY iaap.host_id, iaap.instance_app_id, iaap.instance_api_id, iaap.property_id, cp.value_type – Group to aggregate/merge ),

– Pre-process Instance API properties Merged_Instance_Api_Properties AS ( SELECT iap.property_id, CASE cp.value_type WHEN ‘map’ THEN COALESCE(jsonb_merge_agg(iap.property_value::jsonb), ‘{}’::jsonb)::text WHEN ‘list’ THEN COALESCE((SELECT jsonb_agg(elem ORDER BY prop.update_ts) FROM instance_api_property_t prop, jsonb_array_elements(prop.property_value::jsonb) elem WHERE prop.host_id = iap.host_id AND prop.instance_api_id = iap.instance_api_id AND prop.property_id = iap.property_id AND jsonb_typeof(prop.property_value::jsonb) = ‘array’), ‘[]’::jsonb)::text ELSE (SELECT property_value FROM instance_api_property_t latest WHERE latest.host_id = iap.host_id AND latest.instance_api_id = iap.instance_api_id AND latest.property_id = iap.property_id ORDER BY latest.update_ts DESC LIMIT 1) END AS effective_value FROM instance_api_property_t iap JOIN config_property_t cp ON iap.property_id = cp.property_id WHERE iap.host_id = ? – p_host_id AND iap.instance_api_id IN (SELECT instance_api_id FROM RelevantInstanceApis) GROUP BY iap.host_id, iap.instance_api_id, iap.property_id, cp.value_type ),

– Pre-process Instance App properties Merged_Instance_App_Properties AS ( SELECT iapp.property_id, CASE cp.value_type WHEN ‘map’ THEN COALESCE(jsonb_merge_agg(iapp.property_value::jsonb), ‘{}’::jsonb)::text WHEN ‘list’ THEN COALESCE((SELECT jsonb_agg(elem ORDER BY prop.update_ts) FROM instance_app_property_t prop, jsonb_array_elements(prop.property_value::jsonb) elem WHERE prop.host_id = iapp.host_id AND prop.instance_app_id = iapp.instance_app_id AND prop.property_id = iapp.property_id AND jsonb_typeof(prop.property_value::jsonb) = ‘array’), ‘[]’::jsonb)::text ELSE (SELECT property_value FROM instance_app_property_t latest WHERE latest.host_id = iapp.host_id AND latest.instance_app_id = iapp.instance_app_id AND latest.property_id = iapp.property_id ORDER BY latest.update_ts DESC LIMIT 1) END AS effective_value FROM instance_app_property_t iapp JOIN config_property_t cp ON iapp.property_id = cp.property_id WHERE iapp.host_id = ? – p_host_id AND iapp.instance_app_id IN (SELECT instance_app_id FROM RelevantInstanceApps) GROUP BY iapp.host_id, iapp.instance_app_id, iapp.property_id, cp.value_type ),

– Combine all levels with priority AllOverrides AS ( – Priority 10: Instance App API (highest) - Requires aggregating the merged results if multiple app/api combos apply to the instance SELECT m_iaap.property_id, – Need final merge/latest logic here if multiple app/api combos apply to the SAME instance_id and define the SAME property_id – Assuming for now we take the first one found or need more complex logic if merge is needed again at this stage – For simplicity, let’s assume we just take MAX effective value if multiple rows exist per property_id for the instance MAX(m_iaap.effective_value) as property_value, – This MAX might not be right for JSON, need specific logic if merging across app/api combos is needed here 10 AS priority_level FROM Merged_Instance_App_Api_Properties m_iaap – No additional instance filter needed if CTEs were already filtered by RelevantInstanceApps/Apis linked to p_instance_id GROUP BY m_iaap.property_id – Group to handle multiple app/api links potentially setting the same property for the instance

UNION ALL

-- Priority 20: Instance API
SELECT
    m_iap.property_id,
    MAX(m_iap.effective_value) as property_value, -- Similar merge concern as above
    20 AS priority_level
FROM Merged_Instance_Api_Properties m_iap
GROUP BY m_iap.property_id

UNION ALL

-- Priority 30: Instance App
SELECT
    m_iapp.property_id,
    MAX(m_iapp.effective_value) as property_value, -- Similar merge concern
    30 AS priority_level
FROM Merged_Instance_App_Properties m_iapp
GROUP BY m_iapp.property_id

UNION ALL

-- Priority 40: Instance
SELECT
    ip.property_id,
    ip.property_value,
    40 AS priority_level
FROM instance_property_t ip
WHERE ip.host_id = ? -- p_host_id
  AND ip.instance_id = ? -- p_instance_id

UNION ALL

-- Priority 50: Product Version
SELECT
    pvp.property_id,
    pvp.property_value,
    50 AS priority_level
FROM product_version_property_t pvp
WHERE pvp.host_id = ? -- p_host_id
  AND pvp.product_version_id = ? -- v_product_version_id

UNION ALL

-- Priority 60: Environment
SELECT
    ep.property_id,
    ep.property_value,
    60 AS priority_level
FROM environment_property_t ep
WHERE ep.host_id = ? -- p_host_id
  AND ep.environment = ? -- v_environment

UNION ALL

-- Priority 70: Product (Host independent)
SELECT
    pp.property_id,
    pp.property_value,
    70 AS priority_level
FROM product_property_t pp
WHERE pp.product_id = ? -- v_product_id

UNION ALL

-- Priority 100: Default values
SELECT
    cp.property_id,
    cp.property_value, -- Default value
    100 AS priority_level
FROM config_property_t cp
-- Optimization: Filter defaults to only those applicable to the product version?
-- JOIN product_version_config_property_t pvcp ON cp.property_id = pvcp.property_id
-- WHERE pvcp.host_id = ? AND pvcp.product_version_id = ?

), RankedOverrides AS ( SELECT ao.property_id, ao.property_value, ao.priority_level, ROW_NUMBER() OVER (PARTITION BY ao.property_id ORDER BY ao.priority_level ASC) as rn FROM AllOverrides ao WHERE ao.property_value IS NOT NULL – Exclude levels where the value was NULL (unless NULL is a valid override) ) – Final Selection for Snapshot Table SELECT – snapshot_id needs to be added here or during INSERT cfg.config_name || ‘.’ || cp.property_name AS property_key, ro.property_value, cp.property_type, cp.value_type – Include ro.priority_level AS source_priority if storing provenance FROM RankedOverrides ro JOIN config_property_t cp ON ro.property_id = cp.property_id JOIN config_t cfg ON cp.config_id = cfg.config_id WHERE ro.rn = 1;

5.  **Populate `config_snapshot_property_t`:** Insert the results of this query into `config_snapshot_property_t`, using the `snapshot_id` generated in step 3.

**Key Takeaways:**

*   The `scope_*` columns define the *context* of the snapshot.
*   The values for these `scope_*` columns are used as *parameters* within the `WHERE` clauses of the aggregation query that *generates* the snapshot data.
*   You don't need a `scope_*` column per override table. You need columns representing the different *dimensions* or *levels* by which you might want to define a snapshot's context (host, instance, environment, product version, etc.).
*   The aggregation query uses these parameters to filter the relevant rows from each override table and then determines the highest priority value using `UNION ALL` and a ranking mechanism (`ROW_NUMBER()` or `DISTINCT ON`).

This approach keeps the `config_snapshot_t` table focused on metadata and context, while the query handles the complex logic of applying that context to the various override tables to produce the effective configuration for `config_snapshot_property_t`.

### Config Phase

In the config_t table, there is a config_phase column to separate different stages of api/app life cycles. For example, config for codegen, config for runtime, config for deployment. 

Given your two main use cases:

1.  **Service Startup:** Needs the *runtime* (`'R'`) configuration.
2.  **Deployment Rollback:** Needs to potentially restore the state required for *deployment* (`'D'`) and the resulting *runtime* (`'R'`) configuration from that point in time. (Generator `'G'` configs are usually less relevant for deployment/runtime rollbacks).

Here are the options and the recommended approach:

**Option 1: Phase-Specific Snapshots (Separate Records)**

*   **How:** Add `scope_config_phase CHAR(1)` to `config_snapshot_t`.
*   **Snapshot Creation:** When a snapshot event occurs (e.g., pre-deployment):
    *   Generate a `snapshot_id_D` (e.g., using UUIDv7).
    *   Run the aggregation query with `config_phase = 'D'`.
    *   Store results in `config_snapshot_property_t` linked to `snapshot_id_D`.
    *   Create metadata in `config_snapshot_t` for `snapshot_id_D` with `scope_config_phase = 'D'`.
    *   Generate *another* `snapshot_id_R`.
    *   Run the aggregation query with `config_phase = 'R'`.
    *   Store results in `config_snapshot_property_t` linked to `snapshot_id_R`.
    *   Create metadata in `config_snapshot_t` for `snapshot_id_R` with `scope_config_phase = 'R'`.
    *   You'd need a way to link `snapshot_id_D` and `snapshot_id_R` to the same logical event (e.g., same `related_deployment_id`).
*   **Pros:** Very explicit separation. Querying for a specific phase's snapshot is straightforward.
*   **Cons:** Requires multiple runs of the aggregation query. Doubles the metadata rows in `config_snapshot_t`. Complicates linking phases related to the same event. Less efficient.

**Option 2: Single Snapshot, Phase Included in Properties (Recommended)**

*   **How:** Do **not** add `scope_config_phase` to `config_snapshot_t`. Instead, add `config_phase CHAR(1)` to `config_snapshot_property_t`.
*   **Snapshot Creation:**
    *   Generate a single `snapshot_id`.
    *   Create one metadata row in `config_snapshot_t` representing the overall scope and time (without phase).
    *   **Modify the Aggregation Query:**
        *   **Remove** the `WHERE c.config_phase = ?` filter entirely.
        *   **SELECT** the `c.config_phase` value in the final `SELECT` statement.
    *   Run this modified query *once*. It will calculate the effective properties across *all* phases applicable to the scope.
    *   Store the results in `config_snapshot_property_t`, populating the new `config_phase` column for each property based on the phase of the `config_t` record from which it originated.
*   **`config_snapshot_property_t` Structure:**
    ```sql
    CREATE TABLE config_snapshot_property_t (
        -- ... other columns ...
        config_phase        CHAR(1) NOT NULL, -- Phase this property belongs to
        property_key        TEXT NOT NULL,
        property_value      TEXT,
        property_type       VARCHAR(32),
        value_type          VARCHAR(32),
        -- ...
        PRIMARY KEY(snapshot_property_id), -- Or PK(snapshot_id, config_phase, property_key)? Needs thought.
        FOREIGN KEY(snapshot_id) REFERENCES config_snapshot_t(snapshot_id) ON DELETE CASCADE
    );
    -- Ensure uniqueness within a snapshot for a given key *and phase*
    ALTER TABLE config_snapshot_property_t
        ADD CONSTRAINT config_snapshot_property_uk UNIQUE (snapshot_id, config_phase, property_key);
    -- Index for lookup by snapshot and phase
    CREATE INDEX idx_config_snapshot_property_snap_phase ON config_snapshot_property_t (snapshot_id, config_phase);
    ```
*   **Pros:**### commitConfigInstance

Let's outline the structure of your `commitConfigInstance` service method and the necessary SQL INSERT statements using JDBC.

This involves several steps within a single database transaction:

1.  **Generate Snapshot ID:** Create a new UUID for the snapshot.
2.  **Derive Scope IDs:** Query live tables (`instance_t`, `product_version_t`, etc.) based on the input `hostId` and `instanceId` to get other relevant scope identifiers (`environment`, `productId`, `productVersionId`, `serviceId`, etc.).
3.  **Insert Metadata:** Insert a record into `config_snapshot_t`.
4.  **Aggregate Effective Config:** Run the complex aggregation query (using `ROW_NUMBER()` or similar) to get the final effective properties.
5.  **Insert Effective Config:** Insert the results from step 4 into `config_snapshot_property_t`.
6.  **Snapshot Override Tables:** For each relevant live override table (`instance_property_t`, `instance_api_property_t`, etc.), select its current state (filtered by scope) and insert it into the corresponding `snapshot_*_property_t` table.
7.  **Commit/Rollback:** Commit the transaction if all steps succeed, otherwise roll back.

**Java Service Method Structure (Conceptual)**

```java
import com.github.f4b6a3.uuid.UuidCreator; // For UUIDv7 generation
import javax.sql.DataSource; // Assuming you have a DataSource injected
import java.sql.*;
import java.time.OffsetDateTime;
import java.util.*;

public class ConfigSnapshotService {

    private final DataSource ds;
    // Inject DataSource via constructor

    // Pre-compile your complex aggregation query (modify based on previous examples)
    private static final String AGGREGATE_EFFECTIVE_CONFIG_SQL = """
        WITH AllOverrides AS (
            -- Priority 10: Instance App API (merged) ...
            -- Priority 20: Instance API (merged) ...
            -- Priority 30: Instance App (merged) ...
            -- Priority 40: Instance ...
            -- Priority 50: Product Version ...
            -- Priority 60: Environment ...
            -- Priority 70: Product ...
            -- Priority 100: Default ...
        ),
        RankedOverrides AS (
           SELECT ..., ROW_NUMBER() OVER (PARTITION BY ao.property_id ORDER BY ao.priority_level ASC) as rn
           FROM AllOverrides ao WHERE ao.property_value IS NOT NULL
        )
        SELECT
            c.config_phase,   -- Phase from config_t
            cfg.config_id,    -- Added config_id
            cp.property_id,   -- Added property_id
            cp.property_name, -- Added property_name
            cp.property_type,
            cp.value_type,
            cfg.config_name || '.' || cp.property_name AS property_key, -- Keep for logging/debug? Not needed in snapshot table itself
            ro.property_value,
            ro.priority_level -- To determine source_level
        FROM RankedOverrides ro
        JOIN config_property_t cp ON ro.property_id = cp.property_id
        JOIN config_t cfg ON cp.config_id = cfg.config_id
        WHERE ro.rn = 1;
    """; // NOTE: Add parameters (?) for host_id, instance_id, derived IDs etc.

    public Result<String> commitConfigInstance(Map<String, Object> event) {
        // 1. Extract Input Parameters
        UUID hostId = (UUID) event.get("hostId");
        UUID instanceId = (UUID) event.get("instanceId");
        String snapshotType = (String) event.getOrDefault("snapshotType", "USER_SAVE"); // Default type
        String description = (String) event.get("description");
        UUID userId = (UUID) event.get("userId"); // May be null
        UUID deploymentId = (UUID) event.get("deploymentId"); // May be null

        if (hostId == null || instanceId == null) {
            return Failure.of(new Status(INVALID_PARAMETER, "hostId and instanceId are required."));
        }

        UUID snapshotId = UuidCreator.getTimeOrderedEpoch(); // Generate Snapshot ID (e.g., V7)

        Connection connection = null;
        try {
            connection = ds.getConnection();
            connection.setAutoCommit(false); // Start Transaction

            // 2. Derive Scope IDs
            // Query instance_t and potentially product_version_t based on hostId, instanceId
            DerivedScope scope = deriveScopeInfo(connection, hostId, instanceId);
            if (scope == null) {
                connection.rollback(); // Rollback if instance not found
                return Failure.of(new Status(OBJECT_NOT_FOUND, "Instance not found for hostId/instanceId."));
            }

            // 3. Insert Snapshot Metadata
            insertSnapshotMetadata(connection, snapshotId, snapshotType, description, userId, deploymentId, hostId, scope);

            // 4 & 5. Aggregate and Insert Effective Config
            insertEffectiveConfigSnapshot(connection, snapshotId, hostId, instanceId, scope);

            // 6. Snapshot Individual Override Tables
            // Use INSERT ... SELECT ... for efficiency
            snapshotInstanceProperties(connection, snapshotId, hostId, instanceId);
            snapshotInstanceApiProperties(connection, snapshotId, hostId, instanceId);
            snapshotInstanceAppProperties(connection, snapshotId, hostId, instanceId);
            snapshotInstanceAppApiProperties(connection, snapshotId, hostId, instanceId); // Requires finding relevant App/API IDs first
            snapshotEnvironmentProperties(connection, snapshotId, hostId, scope.environment());
            snapshotProductVersionProperties(connection, snapshotId, hostId, scope.productVersionId());
            snapshotProductProperties(connection, snapshotId, scope.productId());
            // Add others as needed

            // 7. Commit Transaction
            connection.commit();
            logger.info("Successfully created config snapshot: {}", snapshotId);
            return Success.of(snapshotId.toString());

        } catch (SQLException e) {
            logger.error("SQLException during snapshot creation for instance {}: {}", instanceId, e.getMessage(), e);
            if (connection != null) {
                try {
                    connection.rollback();
                } catch (SQLException ex) {
                    logger.error("Error rolling back transaction:", ex);
                }
            }
            return Failure.of(new Status(SQL_EXCEPTION, "Database error during snapshot creation."));
        } catch (Exception e) { // Catch other potential errors (e.g., during scope derivation)
             logger.error("Exception during snapshot creation for instance {}: {}", instanceId, e.getMessage(), e);
             if (connection != null) {
                 try { connection.rollback(); } catch (SQLException ex) { logger.error("Error rolling back transaction:", ex); }
             }
            return Failure.of(new Status(GENERIC_EXCEPTION, "Unexpected error during snapshot creation."));
        } finally {
            if (connection != null) {
                try {
                    connection.setAutoCommit(true); // Restore default behavior
                    connection.close();
                } catch (SQLException e) {
                    logger.error("Error closing connection:", e);
                }
            }
        }
    }

    // --- Helper Methods ---

    // Placeholder for derived scope data structure
    private record DerivedScope(String environment, String productId, String productVersion, UUID productVersionId, String serviceId /*, add API details if needed */) {}

    private DerivedScope deriveScopeInfo(Connection conn, UUID hostId, UUID instanceId) throws SQLException {
        // Query instance_t LEFT JOIN product_version_t ... WHERE i.host_id = ? AND i.instance_id = ?
        // Extract environment, service_id from instance_t
        // Extract product_id, product_version from product_version_t (via product_version_id in instance_t)
        // Return new DerivedScope(...) or null if not found
        String sql = """
            SELECT i.environment, i.service_id, pv.product_id, pv.product_version, i.product_version_id
            FROM instance_t i
            LEFT JOIN product_version_t pv ON i.host_id = pv.host_id AND i.product_version_id = pv.product_version_id
            WHERE i.host_id = ? AND i.instance_id = ?
        """;
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, hostId);
            ps.setObject(2, instanceId);
            try (ResultSet rs = ps.executeQuery()) {
                if (rs.next()) {
                    return new DerivedScope(
                        rs.getString("environment"),
                        rs.getString("product_id"),
                        rs.getString("product_version"),
                        rs.getObject("product_version_id", UUID.class),
                        rs.getString("service_id")
                    );
                } else {
                    return null; // Instance not found
                }
            }
        }
    }

    private void insertSnapshotMetadata(Connection conn, UUID snapshotId, String snapshotType, String description,
                                        UUID userId, UUID deploymentId, UUID hostId, DerivedScope scope) throws SQLException {
        String sql = """
            INSERT INTO config_snapshot_t
            (snapshot_id, snapshot_ts, snapshot_type, description, user_id, deployment_id,
             scope_host_id, scope_environment, scope_product_id, scope_product_version_id, -- Changed col name
             scope_service_id /*, scope_api_id, scope_api_version - Add if applicable */)
            VALUES (?, CURRENT_TIMESTAMP, ?, ?, ?, ?, ?, ?, ?, ?, ? /*, ?, ? */)
            """;
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, snapshotId);
            ps.setString(2, snapshotType);
            ps.setString(3, description);
            ps.setObject(4, userId);         // setObject handles null correctly
            ps.setObject(5, deploymentId);   // setObject handles null correctly
            ps.setObject(6, hostId);
            ps.setString(7, scope.environment());
            ps.setString(8, scope.productId());
            ps.setObject(9, scope.productVersionId()); // Store the ID
            ps.setString(10, scope.serviceId());
            // Set API scope if needed ps.setObject(11, ...); ps.setString(12, ...);
            ps.executeUpdate();
        }
    }


    private void insertEffectiveConfigSnapshot(Connection conn, UUID snapshotId, UUID hostId, UUID instanceId, DerivedScope scope) throws SQLException {
         String insertSql = """
            INSERT INTO config_snapshot_property_t
            (snapshot_property_id, snapshot_id, config_phase, config_id, property_id, property_name,
             property_type, property_value, value_type, source_level)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
            """;
        // Prepare the aggregation query
        try (PreparedStatement selectStmt = conn.prepareStatement(AGGREGATE_EFFECTIVE_CONFIG_SQL);
             PreparedStatement insertStmt = conn.prepareStatement(insertSql)) {

             // Set ALL parameters for the AGGREGATE_EFFECTIVE_CONFIG_SQL query
             int paramIndex = 1;
             // Example: set parameters based on how AGGREGATE_EFFECTIVE_CONFIG_SQL is structured
             // selectStmt.setObject(paramIndex++, hostId);
             // selectStmt.setObject(paramIndex++, instanceId);
             // ... set derived scope IDs (productVersionId, environment, productId) ...
             // ... set parameters for all UNION branches and potential subqueries ...

             try (ResultSet rs = selectStmt.executeQuery()) {
                int batchCount = 0;
                while (rs.next()) {
                    insertStmt.setObject(1, UuidCreator.getTimeOrderedEpoch()); // snapshot_property_id
                    insertStmt.setObject(2, snapshotId);
                    insertStmt.setString(3, rs.getString("config_phase"));
                    insertStmt.setObject(4, rs.getObject("config_id", UUID.class));
                    insertStmt.setObject(5, rs.getObject("property_id", UUID.class));
                    insertStmt.setString(6, rs.getString("property_name"));
                    insertStmt.setString(7, rs.getString("property_type"));
                    insertStmt.setString(8, rs.getString("property_value"));
                    insertStmt.setString(9, rs.getString("value_type"));
                    insertStmt.setString(10, mapPriorityToSourceLevel(rs.getInt("priority_level"))); // Map numeric priority back to level name

                    insertStmt.addBatch();
                    batchCount++;

                    if (batchCount % 100 == 0) { // Execute batch periodically
                        insertStmt.executeBatch();
                    }
                }
                 if (batchCount % 100 != 0) { // Execute remaining batch
                     insertStmt.executeBatch();
                 }
             }
         }
    }

    // Helper to map priority back to source level name
    private String mapPriorityToSourceLevel(int priority) {
        return switch (priority) {
            case 10 -> "instance_app_api"; // Adjust priorities as used in your query
            case 20 -> "instance_api";
            case 30 -> "instance_app";
            case 40 -> "instance";
            case 50 -> "product_version";
            case 60 -> "environment";
            case 70 -> "product";
            case 100 -> "default";
            default -> "unknown";
        };
    }


    // --- Methods for Snapshotting Individual Override Tables ---

    private void snapshotInstanceProperties(Connection conn, UUID snapshotId, UUID hostId, UUID instanceId) throws SQLException {
        String sql = """
            INSERT INTO snapshot_instance_property_t
            (snapshot_id, host_id, instance_id, property_id, property_value, update_user, update_ts)
            SELECT ?, host_id, instance_id, property_id, property_value, update_user, update_ts
            FROM instance_property_t
            WHERE host_id = ? AND instance_id = ?
            """;
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, snapshotId);
            ps.setObject(2, hostId);
            ps.setObject(3, instanceId);
            ps.executeUpdate();
        }
    }

    private void snapshotInstanceApiProperties(Connection conn, UUID snapshotId, UUID hostId, UUID instanceId) throws SQLException {
         // Find relevant instance_api_ids first
        List<UUID> apiIds = findRelevantInstanceApiIds(conn, hostId, instanceId);
        if (apiIds.isEmpty()) return; // No API overrides for this instance

        String sql = """
            INSERT INTO snapshot_instance_api_property_t
            (snapshot_id, host_id, instance_api_id, property_id, property_value, update_user, update_ts)
            SELECT ?, host_id, instance_api_id, property_id, property_value, update_user, update_ts
            FROM instance_api_property_t
            WHERE host_id = ? AND instance_api_id = ANY(?) -- Use ANY with array for multiple IDs
            """;
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, snapshotId);
            ps.setObject(2, hostId);
            // Create a SQL Array from the List of UUIDs
            Array sqlArray = conn.createArrayOf("UUID", apiIds.toArray());
            ps.setArray(3, sqlArray);
            ps.executeUpdate();
            sqlArray.free(); // Release array resources
        }
    }

    // Similar methods for snapshotInstanceAppProperties, snapshotInstanceAppApiProperties...
    // These will need helper methods like findRelevantInstanceApiIds/findRelevantInstanceAppIds

    private void snapshotEnvironmentProperties(Connection conn, UUID snapshotId, UUID hostId, String environment) throws SQLException {
        if (environment == null || environment.isEmpty()) return; // No environment scope
        String sql = """
             INSERT INTO snapshot_environment_property_t
             (snapshot_id, host_id, environment, property_id, property_value, update_user, update_ts)
             SELECT ?, host_id, environment, property_id, property_value, update_user, update_ts
             FROM environment_property_t
             WHERE host_id = ? AND environment = ?
             """;
         try (PreparedStatement ps = conn.prepareStatement(sql)) {
             ps.setObject(1, snapshotId);
             ps.setObject(2, hostId);
             ps.setString(3, environment);
             ps.executeUpdate();
         }
    }

     private void snapshotProductVersionProperties(Connection conn, UUID snapshotId, UUID hostId, UUID productVersionId) throws SQLException {
         if (productVersionId == null) return;
         String sql = """
              INSERT INTO snapshot_product_version_property_t
              (snapshot_id, host_id, product_version_id, property_id, property_value, update_user, update_ts)
              SELECT ?, host_id, product_version_id, property_id, property_value, update_user, update_ts
              FROM product_version_property_t
              WHERE host_id = ? AND product_version_id = ?
              """;
          try (PreparedStatement ps = conn.prepareStatement(sql)) {
              ps.setObject(1, snapshotId);
              ps.setObject(2, hostId);
              ps.setObject(3, productVersionId);
              ps.executeUpdate();
          }
     }

     private void snapshotProductProperties(Connection conn, UUID snapshotId, String productId) throws SQLException {
         if (productId == null || productId.isEmpty()) return;
         String sql = """
               INSERT INTO snapshot_product_property_t
               (snapshot_id, product_id, property_id, property_value, update_user, update_ts)
               SELECT ?, product_id, property_id, property_value, update_user, update_ts
               FROM product_property_t
               WHERE product_id = ?
               """;
           try (PreparedStatement ps = conn.prepareStatement(sql)) {
               ps.setObject(1, snapshotId);
               ps.setString(2, productId);
               ps.executeUpdate();
           }
     }

    // --- Helper method to find associated instance_api_ids ---
    private List<UUID> findRelevantInstanceApiIds(Connection conn, UUID hostId, UUID instanceId) throws SQLException {
        List<UUID> ids = new ArrayList<>();
        String sql = "SELECT instance_api_id FROM instance_api_t WHERE host_id = ? AND instance_id = ?";
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, hostId);
            ps.setObject(2, instanceId);
            try (ResultSet rs = ps.executeQuery()) {
                while(rs.next()) {
                    ids.add(rs.getObject("instance_api_id", UUID.class));
                }
            }
        }
        return ids;
    }
     // --- Add similar helper for findRelevantInstanceAppIds ---
     // --- Add similar helper for findRelevantInstanceAppApiIds (if needed) ---

}

SQL INSERT Statements:

1. config_snapshot_t:

   INSERT INTO config_snapshot_t
   (snapshot_id, snapshot_ts, snapshot_type, description, user_id, deployment_id,
    scope_host_id, scope_environment, scope_product_id, scope_product_version_id, scope_service_id /*, ... other scope cols */)
   VALUES (?, CURRENT_TIMESTAMP, ?, ?, ?, ?, ?, ?, ?, ?, ? /*, ... */)
   

   (Parameters: snapshotId, snapshotType, description, userId, deploymentId, hostId, environment, productId, productVersionId, serviceId, …)

2. config_snapshot_property_t: (Executed in a loop/batch)

   INSERT INTO config_snapshot_property_t
   (snapshot_property_id, snapshot_id, config_phase, config_id, property_id, property_name,
    property_type, property_value, value_type, source_level)
   VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
   

   (Parameters: new UUID, snapshotId, phase, configId, propertyId, propName, propType, propValue, valType, sourceLevelString)

3. snapshot_instance_property_t:

   INSERT INTO snapshot_instance_property_t
   (snapshot_id, host_id, instance_id, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, instance_id, property_id, property_value, update_user, update_ts
   FROM instance_property_t
   WHERE host_id = ? AND instance_id = ?
   

   (Parameters: snapshotId, hostId, instanceId)

4. snapshot_instance_api_property_t:

   INSERT INTO snapshot_instance_api_property_t
   (snapshot_id, host_id, instance_api_id, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, instance_api_id, property_id, property_value, update_user, update_ts
   FROM instance_api_property_t
   WHERE host_id = ? AND instance_api_id = ANY(?) -- Parameter is a SQL Array of relevant instance_api_ids
   

   (Parameters: snapshotId, hostId, SQL Array of instance_api_ids)

5. snapshot_instance_app_property_t:

   INSERT INTO snapshot_instance_app_property_t
   (snapshot_id, host_id, instance_app_id, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, instance_app_id, property_id, property_value, update_user, update_ts
   FROM instance_app_property_t
   WHERE host_id = ? AND instance_app_id = ANY(?) -- Parameter is a SQL Array of relevant instance_app_ids
   

   (Parameters: snapshotId, hostId, SQL Array of instance_app_ids)

6. snapshot_instance_app_api_property_t:

   INSERT INTO snapshot_instance_app_api_property_t
   (snapshot_id, host_id, instance_app_id, instance_api_id, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, instance_app_id, instance_api_id, property_id, property_value, update_user, update_ts
   FROM instance_app_api_property_t
   WHERE host_id = ?
     AND instance_app_id = ANY(?) -- SQL Array of relevant instance_app_ids
     AND instance_api_id = ANY(?) -- SQL Array of relevant instance_api_ids
   

   (Parameters: snapshotId, hostId, SQL Array of instance_app_ids, SQL Array of instance_api_ids)

7. snapshot_environment_property_t:

   INSERT INTO snapshot_environment_property_t
   (snapshot_id, host_id, environment, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, environment, property_id, property_value, update_user, update_ts
   FROM environment_property_t
   WHERE host_id = ? AND environment = ?
   

   (Parameters: snapshotId, hostId, environment)

8. snapshot_product_version_property_t:

   INSERT INTO snapshot_product_version_property_t
   (snapshot_id, host_id, product_version_id, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, product_version_id, property_id, property_value, update_user, update_ts
   FROM product_version_property_t
   WHERE host_id = ? AND product_version_id = ?
   

   (Parameters: snapshotId, hostId, productVersionId)

9. snapshot_product_property_t:

   INSERT INTO snapshot_product_property_t
   (snapshot_id, product_id, property_id, property_value, update_user, update_ts)
   SELECT ?, product_id, property_id, property_value, update_user, update_ts
   FROM product_property_t
   WHERE product_id = ?
   

   (Parameters: snapshotId, productId)

Important Considerations:

• Transaction Management: Ensure all these inserts happen within a single atomic transaction.
• Error Handling: Robust error handling and rollback are critical.
• Performance: The aggregation query can be heavy. Consider its performance impact. The INSERT ... SELECT statements should be reasonably efficient if the source tables are properly indexed on their scope columns.
• Parameterization: Use PreparedStatement for all queries to prevent SQL injection and improve performance.
• Scope Derivation Logic: Ensure the logic in deriveScopeInfo and the helpers like findRelevantInstanceApiIds is correct and handles cases where relationships might not exist (e.g., an instance with no associated APIs).
• UUID Generation: Use a suitable UUID generator (like UUIDv7 via uuid-creator).
• Logging: Add clear logging for snapshot creation success/failure.
• Refinement: You’ll need to adapt the AGGREGATE_EFFECTIVE_CONFIG_SQL query based on the final version using ROW_NUMBER and correctly incorporate parameters.
  • Only one aggregation query run per snapshot event.
  • Captures the complete multi-phase state at that point under one snapshot_id.
  • Simpler config_snapshot_t metadata.
  • Facilitates rollbacks that might need to restore both ‘D’ and ‘R’ phase configurations.
• Cons:
  • Requires modifying the aggregation query.
  • Consuming services (like startup) need to add AND config_phase = 'R' when querying config_snapshot_property_t.

Conclusion:

Option 2 is the better approach. Create a single snapshot representing the point-in-time state for the defined scope, but modify your aggregation query to:

1. Remove the config_phase = ? filter.
2. Select c.config_phase in the final output.
3. Store this config_phase along with the effective property_key and property_value in the config_snapshot_property_t table.

This is more efficient for snapshot creation and provides a complete view for rollbacks. Consumers simply need to add a phase filter when reading the specific configuration they need (e.g., 'R' for runtime).

Modified Aggregation Query Snippet (Final SELECT):

-- (Inside the final SELECT statement of the modified aggregation query)
SELECT
    -- snapshot_id needs to be added here or during INSERT
    c.config_phase,   -- <-- Select the phase
    cfg.config_name || '.' || cp.property_name AS property_key,
    ro.property_value,
    cp.property_type,
    cp.value_type
    -- Include ro.priority_level AS source_priority if storing provenance
FROM RankedOverrides ro
JOIN config_property_t cp ON ro.property_id = cp.property_id
JOIN config_t cfg ON cp.config_id = cfg.config_id
-- Make sure the join to config_t (aliased as c or cfg) is available here to get the phase
-- This join likely happened earlier to get config_name anyway.
WHERE ro.rn = 1;

commitConfigInstance

Okay, let’s outline the structure of your commitConfigInstance service method and the necessary SQL INSERT statements using JDBC.

This involves several steps within a single database transaction:

1. Generate Snapshot ID: Create a new UUID for the snapshot.
2. Derive Scope IDs: Query live tables (instance_t, product_version_t, etc.) based on the input hostId and instanceId to get other relevant scope identifiers (environment, productId, productVersionId, serviceId, etc.).
3. Insert Metadata: Insert a record into config_snapshot_t.
4. Aggregate Effective Config: Run the complex aggregation query (using ROW_NUMBER() or similar) to get the final effective properties.
5. Insert Effective Config: Insert the results from step 4 into config_snapshot_property_t.
6. Snapshot Override Tables: For each relevant live override table (instance_property_t, instance_api_property_t, etc.), select its current state (filtered by scope) and insert it into the corresponding snapshot_*_property_t table.
7. Commit/Rollback: Commit the transaction if all steps succeed, otherwise roll back.

Java Service Method Structure (Conceptual)

import com.github.f4b6a3.uuid.UuidCreator; // For UUIDv7 generation
import javax.sql.DataSource; // Assuming you have a DataSource injected
import java.sql.*;
import java.time.OffsetDateTime;
import java.util.*;

public class ConfigSnapshotService {

    private final DataSource ds;
    // Inject DataSource via constructor

    // Pre-compile your complex aggregation query (modify based on previous examples)
    private static final String AGGREGATE_EFFECTIVE_CONFIG_SQL = """
        WITH AllOverrides AS (
            -- Priority 10: Instance App API (merged) ...
            -- Priority 20: Instance API (merged) ...
            -- Priority 30: Instance App (merged) ...
            -- Priority 40: Instance ...
            -- Priority 50: Product Version ...
            -- Priority 60: Environment ...
            -- Priority 70: Product ...
            -- Priority 100: Default ...
        ),
        RankedOverrides AS (
           SELECT ..., ROW_NUMBER() OVER (PARTITION BY ao.property_id ORDER BY ao.priority_level ASC) as rn
           FROM AllOverrides ao WHERE ao.property_value IS NOT NULL
        )
        SELECT
            c.config_phase,   -- Phase from config_t
            cfg.config_id,    -- Added config_id
            cp.property_id,   -- Added property_id
            cp.property_name, -- Added property_name
            cp.property_type,
            cp.value_type,
            cfg.config_name || '.' || cp.property_name AS property_key, -- Keep for logging/debug? Not needed in snapshot table itself
            ro.property_value,
            ro.priority_level -- To determine source_level
        FROM RankedOverrides ro
        JOIN config_property_t cp ON ro.property_id = cp.property_id
        JOIN config_t cfg ON cp.config_id = cfg.config_id
        WHERE ro.rn = 1;
    """; // NOTE: Add parameters (?) for host_id, instance_id, derived IDs etc.

    public Result<String> commitConfigInstance(Map<String, Object> event) {
        // 1. Extract Input Parameters
        UUID hostId = (UUID) event.get("hostId");
        UUID instanceId = (UUID) event.get("instanceId");
        String snapshotType = (String) event.getOrDefault("snapshotType", "USER_SAVE"); // Default type
        String description = (String) event.get("description");
        UUID userId = (UUID) event.get("userId"); // May be null
        UUID deploymentId = (UUID) event.get("deploymentId"); // May be null

        if (hostId == null || instanceId == null) {
            return Failure.of(new Status(INVALID_PARAMETER, "hostId and instanceId are required."));
        }

        UUID snapshotId = UuidCreator.getTimeOrderedEpoch(); // Generate Snapshot ID (e.g., V7)

        Connection connection = null;
        try {
            connection = ds.getConnection();
            connection.setAutoCommit(false); // Start Transaction

            // 2. Derive Scope IDs
            // Query instance_t and potentially product_version_t based on hostId, instanceId
            DerivedScope scope = deriveScopeInfo(connection, hostId, instanceId);
            if (scope == null) {
                connection.rollback(); // Rollback if instance not found
                return Failure.of(new Status(OBJECT_NOT_FOUND, "Instance not found for hostId/instanceId."));
            }

            // 3. Insert Snapshot Metadata
            insertSnapshotMetadata(connection, snapshotId, snapshotType, description, userId, deploymentId, hostId, scope);

            // 4 & 5. Aggregate and Insert Effective Config
            insertEffectiveConfigSnapshot(connection, snapshotId, hostId, instanceId, scope);

            // 6. Snapshot Individual Override Tables
            // Use INSERT ... SELECT ... for efficiency
            snapshotInstanceProperties(connection, snapshotId, hostId, instanceId);
            snapshotInstanceApiProperties(connection, snapshotId, hostId, instanceId);
            snapshotInstanceAppProperties(connection, snapshotId, hostId, instanceId);
            snapshotInstanceAppApiProperties(connection, snapshotId, hostId, instanceId); // Requires finding relevant App/API IDs first
            snapshotEnvironmentProperties(connection, snapshotId, hostId, scope.environment());
            snapshotProductVersionProperties(connection, snapshotId, hostId, scope.productVersionId());
            snapshotProductProperties(connection, snapshotId, scope.productId());
            // Add others as needed

            // 7. Commit Transaction
            connection.commit();
            logger.info("Successfully created config snapshot: {}", snapshotId);
            return Success.of(snapshotId.toString());

        } catch (SQLException e) {
            logger.error("SQLException during snapshot creation for instance {}: {}", instanceId, e.getMessage(), e);
            if (connection != null) {
                try {
                    connection.rollback();
                } catch (SQLException ex) {
                    logger.error("Error rolling back transaction:", ex);
                }
            }
            return Failure.of(new Status(SQL_EXCEPTION, "Database error during snapshot creation."));
        } catch (Exception e) { // Catch other potential errors (e.g., during scope derivation)
             logger.error("Exception during snapshot creation for instance {}: {}", instanceId, e.getMessage(), e);
             if (connection != null) {
                 try { connection.rollback(); } catch (SQLException ex) { logger.error("Error rolling back transaction:", ex); }
             }
            return Failure.of(new Status(GENERIC_EXCEPTION, "Unexpected error during snapshot creation."));
        } finally {
            if (connection != null) {
                try {
                    connection.setAutoCommit(true); // Restore default behavior
                    connection.close();
                } catch (SQLException e) {
                    logger.error("Error closing connection:", e);
                }
            }
        }
    }

    // --- Helper Methods ---

    // Placeholder for derived scope data structure
    private record DerivedScope(String environment, String productId, String productVersion, UUID productVersionId, String serviceId /*, add API details if needed */) {}

    private DerivedScope deriveScopeInfo(Connection conn, UUID hostId, UUID instanceId) throws SQLException {
        // Query instance_t LEFT JOIN product_version_t ... WHERE i.host_id = ? AND i.instance_id = ?
        // Extract environment, service_id from instance_t
        // Extract product_id, product_version from product_version_t (via product_version_id in instance_t)
        // Return new DerivedScope(...) or null if not found
        String sql = """
            SELECT i.environment, i.service_id, pv.product_id, pv.product_version, i.product_version_id
            FROM instance_t i
            LEFT JOIN product_version_t pv ON i.host_id = pv.host_id AND i.product_version_id = pv.product_version_id
            WHERE i.host_id = ? AND i.instance_id = ?
        """;
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, hostId);
            ps.setObject(2, instanceId);
            try (ResultSet rs = ps.executeQuery()) {
                if (rs.next()) {
                    return new DerivedScope(
                        rs.getString("environment"),
                        rs.getString("product_id"),
                        rs.getString("product_version"),
                        rs.getObject("product_version_id", UUID.class),
                        rs.getString("service_id")
                    );
                } else {
                    return null; // Instance not found
                }
            }
        }
    }

    private void insertSnapshotMetadata(Connection conn, UUID snapshotId, String snapshotType, String description,
                                        UUID userId, UUID deploymentId, UUID hostId, DerivedScope scope) throws SQLException {
        String sql = """
            INSERT INTO config_snapshot_t
            (snapshot_id, snapshot_ts, snapshot_type, description, user_id, deployment_id,
             scope_host_id, scope_environment, scope_product_id, scope_product_version_id, -- Changed col name
             scope_service_id /*, scope_api_id, scope_api_version - Add if applicable */)
            VALUES (?, CURRENT_TIMESTAMP, ?, ?, ?, ?, ?, ?, ?, ?, ? /*, ?, ? */)
            """;
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, snapshotId);
            ps.setString(2, snapshotType);
            ps.setString(3, description);
            ps.setObject(4, userId);         // setObject handles null correctly
            ps.setObject(5, deploymentId);   // setObject handles null correctly
            ps.setObject(6, hostId);
            ps.setString(7, scope.environment());
            ps.setString(8, scope.productId());
            ps.setObject(9, scope.productVersionId()); // Store the ID
            ps.setString(10, scope.serviceId());
            // Set API scope if needed ps.setObject(11, ...); ps.setString(12, ...);
            ps.executeUpdate();
        }
    }


    private void insertEffectiveConfigSnapshot(Connection conn, UUID snapshotId, UUID hostId, UUID instanceId, DerivedScope scope) throws SQLException {
         String insertSql = """
            INSERT INTO config_snapshot_property_t
            (snapshot_property_id, snapshot_id, config_phase, config_id, property_id, property_name,
             property_type, property_value, value_type, source_level)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
            """;
        // Prepare the aggregation query
        try (PreparedStatement selectStmt = conn.prepareStatement(AGGREGATE_EFFECTIVE_CONFIG_SQL);
             PreparedStatement insertStmt = conn.prepareStatement(insertSql)) {

             // Set ALL parameters for the AGGREGATE_EFFECTIVE_CONFIG_SQL query
             int paramIndex = 1;
             // Example: set parameters based on how AGGREGATE_EFFECTIVE_CONFIG_SQL is structured
             // selectStmt.setObject(paramIndex++, hostId);
             // selectStmt.setObject(paramIndex++, instanceId);
             // ... set derived scope IDs (productVersionId, environment, productId) ...
             // ... set parameters for all UNION branches and potential subqueries ...

             try (ResultSet rs = selectStmt.executeQuery()) {
                int batchCount = 0;
                while (rs.next()) {
                    insertStmt.setObject(1, UuidCreator.getTimeOrderedEpoch()); // snapshot_property_id
                    insertStmt.setObject(2, snapshotId);
                    insertStmt.setString(3, rs.getString("config_phase"));
                    insertStmt.setObject(4, rs.getObject("config_id", UUID.class));
                    insertStmt.setObject(5, rs.getObject("property_id", UUID.class));
                    insertStmt.setString(6, rs.getString("property_name"));
                    insertStmt.setString(7, rs.getString("property_type"));
                    insertStmt.setString(8, rs.getString("property_value"));
                    insertStmt.setString(9, rs.getString("value_type"));
                    insertStmt.setString(10, mapPriorityToSourceLevel(rs.getInt("priority_level"))); // Map numeric priority back to level name

                    insertStmt.addBatch();
                    batchCount++;

                    if (batchCount % 100 == 0) { // Execute batch periodically
                        insertStmt.executeBatch();
                    }
                }
                 if (batchCount % 100 != 0) { // Execute remaining batch
                     insertStmt.executeBatch();
                 }
             }
         }
    }

    // Helper to map priority back to source level name
    private String mapPriorityToSourceLevel(int priority) {
        return switch (priority) {
            case 10 -> "instance_app_api"; // Adjust priorities as used in your query
            case 20 -> "instance_api";
            case 30 -> "instance_app";
            case 40 -> "instance";
            case 50 -> "product_version";
            case 60 -> "environment";
            case 70 -> "product";
            case 100 -> "default";
            default -> "unknown";
        };
    }


    // --- Methods for Snapshotting Individual Override Tables ---

    private void snapshotInstanceProperties(Connection conn, UUID snapshotId, UUID hostId, UUID instanceId) throws SQLException {
        String sql = """
            INSERT INTO snapshot_instance_property_t
            (snapshot_id, host_id, instance_id, property_id, property_value, update_user, update_ts)
            SELECT ?, host_id, instance_id, property_id, property_value, update_user, update_ts
            FROM instance_property_t
            WHERE host_id = ? AND instance_id = ?
            """;
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, snapshotId);
            ps.setObject(2, hostId);
            ps.setObject(3, instanceId);
            ps.executeUpdate();
        }
    }

    private void snapshotInstanceApiProperties(Connection conn, UUID snapshotId, UUID hostId, UUID instanceId) throws SQLException {
         // Find relevant instance_api_ids first
        List<UUID> apiIds = findRelevantInstanceApiIds(conn, hostId, instanceId);
        if (apiIds.isEmpty()) return; // No API overrides for this instance

        String sql = """
            INSERT INTO snapshot_instance_api_property_t
            (snapshot_id, host_id, instance_api_id, property_id, property_value, update_user, update_ts)
            SELECT ?, host_id, instance_api_id, property_id, property_value, update_user, update_ts
            FROM instance_api_property_t
            WHERE host_id = ? AND instance_api_id = ANY(?) -- Use ANY with array for multiple IDs
            """;
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, snapshotId);
            ps.setObject(2, hostId);
            // Create a SQL Array from the List of UUIDs
            Array sqlArray = conn.createArrayOf("UUID", apiIds.toArray());
            ps.setArray(3, sqlArray);
            ps.executeUpdate();
            sqlArray.free(); // Release array resources
        }
    }

    // Similar methods for snapshotInstanceAppProperties, snapshotInstanceAppApiProperties...
    // These will need helper methods like findRelevantInstanceApiIds/findRelevantInstanceAppIds

    private void snapshotEnvironmentProperties(Connection conn, UUID snapshotId, UUID hostId, String environment) throws SQLException {
        if (environment == null || environment.isEmpty()) return; // No environment scope
        String sql = """
             INSERT INTO snapshot_environment_property_t
             (snapshot_id, host_id, environment, property_id, property_value, update_user, update_ts)
             SELECT ?, host_id, environment, property_id, property_value, update_user, update_ts
             FROM environment_property_t
             WHERE host_id = ? AND environment = ?
             """;
         try (PreparedStatement ps = conn.prepareStatement(sql)) {
             ps.setObject(1, snapshotId);
             ps.setObject(2, hostId);
             ps.setString(3, environment);
             ps.executeUpdate();
         }
    }

     private void snapshotProductVersionProperties(Connection conn, UUID snapshotId, UUID hostId, UUID productVersionId) throws SQLException {
         if (productVersionId == null) return;
         String sql = """
              INSERT INTO snapshot_product_version_property_t
              (snapshot_id, host_id, product_version_id, property_id, property_value, update_user, update_ts)
              SELECT ?, host_id, product_version_id, property_id, property_value, update_user, update_ts
              FROM product_version_property_t
              WHERE host_id = ? AND product_version_id = ?
              """;
          try (PreparedStatement ps = conn.prepareStatement(sql)) {
              ps.setObject(1, snapshotId);
              ps.setObject(2, hostId);
              ps.setObject(3, productVersionId);
              ps.executeUpdate();
          }
     }

     private void snapshotProductProperties(Connection conn, UUID snapshotId, String productId) throws SQLException {
         if (productId == null || productId.isEmpty()) return;
         String sql = """
               INSERT INTO snapshot_product_property_t
               (snapshot_id, product_id, property_id, property_value, update_user, update_ts)
               SELECT ?, product_id, property_id, property_value, update_user, update_ts
               FROM product_property_t
               WHERE product_id = ?
               """;
           try (PreparedStatement ps = conn.prepareStatement(sql)) {
               ps.setObject(1, snapshotId);
               ps.setString(2, productId);
               ps.executeUpdate();
           }
     }

    // --- Helper method to find associated instance_api_ids ---
    private List<UUID> findRelevantInstanceApiIds(Connection conn, UUID hostId, UUID instanceId) throws SQLException {
        List<UUID> ids = new ArrayList<>();
        String sql = "SELECT instance_api_id FROM instance_api_t WHERE host_id = ? AND instance_id = ?";
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, hostId);
            ps.setObject(2, instanceId);
            try (ResultSet rs = ps.executeQuery()) {
                while(rs.next()) {
                    ids.add(rs.getObject("instance_api_id", UUID.class));
                }
            }
        }
        return ids;
    }
     // --- Add similar helper for findRelevantInstanceAppIds ---
     // --- Add similar helper for findRelevantInstanceAppApiIds (if needed) ---

}

SQL INSERT Statements:

1. config_snapshot_t:

   INSERT INTO config_snapshot_t
   (snapshot_id, snapshot_ts, snapshot_type, description, user_id, deployment_id,
    scope_host_id, scope_environment, scope_product_id, scope_product_version_id, scope_service_id /*, ... other scope cols */)
   VALUES (?, CURRENT_TIMESTAMP, ?, ?, ?, ?, ?, ?, ?, ?, ? /*, ... */)
   

   (Parameters: snapshotId, snapshotType, description, userId, deploymentId, hostId, environment, productId, productVersionId, serviceId, …)

2. config_snapshot_property_t: (Executed in a loop/batch)

   INSERT INTO config_snapshot_property_t
   (snapshot_property_id, snapshot_id, config_phase, config_id, property_id, property_name,
    property_type, property_value, value_type, source_level)
   VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
   

   (Parameters: new UUID, snapshotId, phase, configId, propertyId, propName, propType, propValue, valType, sourceLevelString)

3. snapshot_instance_property_t:

   INSERT INTO snapshot_instance_property_t
   (snapshot_id, host_id, instance_id, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, instance_id, property_id, property_value, update_user, update_ts
   FROM instance_property_t
   WHERE host_id = ? AND instance_id = ?
   

   (Parameters: snapshotId, hostId, instanceId)

4. snapshot_instance_api_property_t:

   INSERT INTO snapshot_instance_api_property_t
   (snapshot_id, host_id, instance_api_id, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, instance_api_id, property_id, property_value, update_user, update_ts
   FROM instance_api_property_t
   WHERE host_id = ? AND instance_api_id = ANY(?) -- Parameter is a SQL Array of relevant instance_api_ids
   

   (Parameters: snapshotId, hostId, SQL Array of instance_api_ids)

5. snapshot_instance_app_property_t:

   INSERT INTO snapshot_instance_app_property_t
   (snapshot_id, host_id, instance_app_id, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, instance_app_id, property_id, property_value, update_user, update_ts
   FROM instance_app_property_t
   WHERE host_id = ? AND instance_app_id = ANY(?) -- Parameter is a SQL Array of relevant instance_app_ids
   

   (Parameters: snapshotId, hostId, SQL Array of instance_app_ids)

6. snapshot_instance_app_api_property_t:

   INSERT INTO snapshot_instance_app_api_property_t
   (snapshot_id, host_id, instance_app_id, instance_api_id, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, instance_app_id, instance_api_id, property_id, property_value, update_user, update_ts
   FROM instance_app_api_property_t
   WHERE host_id = ?
     AND instance_app_id = ANY(?) -- SQL Array of relevant instance_app_ids
     AND instance_api_id = ANY(?) -- SQL Array of relevant instance_api_ids
   

   (Parameters: snapshotId, hostId, SQL Array of instance_app_ids, SQL Array of instance_api_ids)

7. snapshot_environment_property_t:

   INSERT INTO snapshot_environment_property_t
   (snapshot_id, host_id, environment, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, environment, property_id, property_value, update_user, update_ts
   FROM environment_property_t
   WHERE host_id = ? AND environment = ?
   

   (Parameters: snapshotId, hostId, environment)

8. snapshot_product_version_property_t:

   INSERT INTO snapshot_product_version_property_t
   (snapshot_id, host_id, product_version_id, property_id, property_value, update_user, update_ts)
   SELECT ?, host_id, product_version_id, property_id, property_value, update_user, update_ts
   FROM product_version_property_t
   WHERE host_id = ? AND product_version_id = ?
   

   (Parameters: snapshotId, hostId, productVersionId)

9. snapshot_product_property_t:

   INSERT INTO snapshot_product_property_t
   (snapshot_id, product_id, property_id, property_value, update_user, update_ts)
   SELECT ?, product_id, property_id, property_value, update_user, update_ts
   FROM product_property_t
   WHERE product_id = ?
   

   (Parameters: snapshotId, productId)

Important Considerations:

• Transaction Management: Ensure all these inserts happen within a single atomic transaction.
• Error Handling: Robust error handling and rollback are critical.
• Performance: The aggregation query can be heavy. Consider its performance impact. The INSERT ... SELECT statements should be reasonably efficient if the source tables are properly indexed on their scope columns.
• Parameterization: Use PreparedStatement for all queries to prevent SQL injection and improve performance.
• Scope Derivation Logic: Ensure the logic in deriveScopeInfo and the helpers like findRelevantInstanceApiIds is correct and handles cases where relationships might not exist (e.g., an instance with no associated APIs).
• UUID Generation: Use a suitable UUID generator (like UUIDv7 via uuid-creator).
• Logging: Add clear logging for snapshot creation success/failure.
• Refinement: You’ll need to adapt the AGGREGATE_EFFECTIVE_CONFIG_SQL query based on the final version using ROW_NUMBER and correctly incorporate parameters.

rollbackConfigInstance

Okay, here’s the rollbackConfigInstance method implementing the DELETE/INSERT strategy to restore the state of instance-level and related sub-level overrides from a snapshot.

Assumptions:

• “Rolling back an instance” means restoring the overrides defined specifically for that instance and its associated APIs, Apps, and App-API combinations. It does not modify higher-level overrides (Environment, Product Version, Product).
• The snapshot_*_property_t tables accurately store the state of the corresponding live tables at the time the snapshot was taken.
• The necessary helper methods like findRelevantInstanceApiIds, findRelevantInstanceAppIds exist (examples provided).

import com.github.f4b6a3.uuid.UuidCreator; // If needed for audit logging ID
import javax.sql.DataSource;
import java.sql.*;
import java.util.*;

public class ConfigRollbackService {

    private final DataSource ds;
    // Inject DataSource via constructor

    // --- SQL Templates ---

    // DELETE Statements (Targeting LIVE tables)
    private static final String DELETE_INSTANCE_PROPS_SQL = "DELETE FROM instance_property_t WHERE host_id = ? AND instance_id = ?";
    private static final String DELETE_INSTANCE_API_PROPS_SQL = "DELETE FROM instance_api_property_t WHERE host_id = ? AND instance_api_id = ANY(?)";
    private static final String DELETE_INSTANCE_APP_PROPS_SQL = "DELETE FROM instance_app_property_t WHERE host_id = ? AND instance_app_id = ANY(?)";
    private static final String DELETE_INSTANCE_APP_API_PROPS_SQL = "DELETE FROM instance_app_api_property_t WHERE host_id = ? AND instance_app_id = ANY(?) AND instance_api_id = ANY(?)";

    // INSERT ... SELECT Statements (From SNAPSHOT tables to LIVE tables)
    private static final String INSERT_INSTANCE_PROPS_SQL = """
        INSERT INTO instance_property_t
        (host_id, instance_id, property_id, property_value, update_user, update_ts)
        SELECT host_id, instance_id, property_id, property_value, update_user, update_ts
        FROM snapshot_instance_property_t
        WHERE snapshot_id = ? AND host_id = ? AND instance_id = ?
        """;
    private static final String INSERT_INSTANCE_API_PROPS_SQL = """
        INSERT INTO instance_api_property_t
        (host_id, instance_api_id, property_id, property_value, update_user, update_ts)
        SELECT host_id, instance_api_id, property_id, property_value, update_user, update_ts
        FROM snapshot_instance_api_property_t
        WHERE snapshot_id = ? AND host_id = ? AND instance_api_id = ANY(?)
        """;
     private static final String INSERT_INSTANCE_APP_PROPS_SQL = """
        INSERT INTO instance_app_property_t
        (host_id, instance_app_id, property_id, property_value, update_user, update_ts)
        SELECT host_id, instance_app_id, property_id, property_value, update_user, update_ts
        FROM snapshot_instance_app_property_t
        WHERE snapshot_id = ? AND host_id = ? AND instance_app_id = ANY(?)
        """;
    private static final String INSERT_INSTANCE_APP_API_PROPS_SQL = """
        INSERT INTO instance_app_api_property_t
        (host_id, instance_app_id, instance_api_id, property_id, property_value, update_user, update_ts)
        SELECT host_id, instance_app_id, instance_api_id, property_id, property_value, update_user, update_ts
        FROM snapshot_instance_app_api_property_t
        WHERE snapshot_id = ? AND host_id = ? AND instance_app_id = ANY(?) AND instance_api_id = ANY(?)
        """;

    public Result<String> rollbackConfigInstance(Map<String, Object> event) {
        // 1. Extract Input Parameters
        UUID snapshotId = (UUID) event.get("snapshotId");
        UUID hostId = (UUID) event.get("hostId");
        UUID instanceId = (UUID) event.get("instanceId");
        UUID userId = (UUID) event.get("userId"); // For potential auditing
        String description = (String) event.get("rollbackDescription"); // Optional reason

        if (snapshotId == null || hostId == null || instanceId == null) {
            return Failure.of(new Status(INVALID_PARAMETER, "snapshotId, hostId, and instanceId are required."));
        }

        Connection connection = null;
        List<UUID> currentApiIds = null;
        List<UUID> currentAppIds = null;

        try {
            connection = ds.getConnection();
            connection.setAutoCommit(false); // Start Transaction

            // --- Pre-computation: Find CURRENT associated IDs for DELETE scope ---
            // It's generally safer to delete based on current relationships and then
            // insert based on snapshot relationships if they could have diverged.
            currentApiIds = findRelevantInstanceApiIds(connection, hostId, instanceId);
            currentAppIds = findRelevantInstanceAppIds(connection, hostId, instanceId);
            // Note: InstanceAppApi requires both lists.

            logger.info("Starting rollback for instance {} (host {}) to snapshot {}", instanceId, hostId, snapshotId);

            // --- Execute Deletes from LIVE tables ---
            executeDelete(connection, DELETE_INSTANCE_PROPS_SQL, hostId, instanceId);

            if (!currentApiIds.isEmpty()) {
                executeDeleteWithArray(connection, DELETE_INSTANCE_API_PROPS_SQL, hostId, currentApiIds);
                // Also delete AppApi props related to these APIs if apps also exist
                if (!currentAppIds.isEmpty()) {
                     executeDeleteWithTwoArrays(connection, DELETE_INSTANCE_APP_API_PROPS_SQL, hostId, currentAppIds, currentApiIds);
                }
            }

            if (!currentAppIds.isEmpty()) {
                executeDeleteWithArray(connection, DELETE_INSTANCE_APP_PROPS_SQL, hostId, currentAppIds);
                 // AppApi props deletion might have already happened above if APIs existed.
                 // If only apps existed but no APIs, delete AppApi here (redundant if handled above)
                 // Generally safe to run the AppApi delete again if needed, targeting only appIds.
                 // For simplicity, we assume the AppApi delete targeting both arrays covers necessary cases.
            }


            // --- Execute Inserts from SNAPSHOT tables ---
            executeInsertSelect(connection, INSERT_INSTANCE_PROPS_SQL, snapshotId, hostId, instanceId);

            // For array-based inserts, we need the IDs *from the snapshot time*
            // However, the SELECT inside the INSERT query implicitly filters by snapshot_id AND the array condition,
            // so it should correctly only insert relationships that existed in the snapshot.
            // We still use the *current* IDs to DEFINE the overall scope of instance being affected,
            // but the INSERT...SELECT filters correctly based on snapshot content.
            if (!currentApiIds.isEmpty()) { // Use currentApiIds to decide IF we run the insert query
                executeInsertSelectWithArray(connection, INSERT_INSTANCE_API_PROPS_SQL, snapshotId, hostId, currentApiIds);
                 if (!currentAppIds.isEmpty()) {
                    executeInsertSelectWithTwoArrays(connection, INSERT_INSTANCE_APP_API_PROPS_SQL, snapshotId, hostId, currentAppIds, currentApiIds);
                }
            }
            if (!currentAppIds.isEmpty()) { // Use currentAppIds to decide IF we run the insert query
                executeInsertSelectWithArray(connection, INSERT_INSTANCE_APP_PROPS_SQL, snapshotId, hostId, currentAppIds);
                // Redundant AppApi insert if handled above? No, the INSERT uses the AppId filter.
                // If only apps existed at snapshot time, this covers it.
            }

            // --- Optional: Audit Logging ---
            // logRollbackActivity(connection, snapshotId, hostId, instanceId, userId, description);


            // --- Commit Transaction ---
            connection.commit();
            logger.info("Successfully rolled back instance {} (host {}) to snapshot {}", instanceId, hostId, snapshotId);
            return Success.of("Rollback successful to snapshot " + snapshotId);

        } catch (SQLException e) {
            logger.error("SQLException during rollback for instance {} to snapshot {}: {}", instanceId, snapshotId, e.getMessage(), e);
            if (connection != null) {
                try {
                    connection.rollback();
                     logger.warn("Transaction rolled back for instance {} snapshot {}", instanceId, snapshotId);
                } catch (SQLException ex) {
                    logger.error("Error rolling back transaction:", ex);
                }
            }
            return Failure.of(new Status(SQL_EXCEPTION, "Database error during rollback operation."));
        } catch (Exception e) { // Catch other potential errors
             logger.error("Exception during rollback for instance {} to snapshot {}: {}", instanceId, snapshotId, e.getMessage(), e);
             if (connection != null) {
                 try { connection.rollback(); } catch (SQLException ex) { logger.error("Error rolling back transaction:", ex); }
             }
            return Failure.of(new Status(GENERIC_EXCEPTION, "Unexpected error during rollback operation."));
        } finally {
            if (connection != null) {
                try {
                    connection.setAutoCommit(true); // Restore default behavior
                    connection.close();
                } catch (SQLException e) {
                    logger.error("Error closing connection:", e);
                }
            }
        }
    }

    // --- Helper Methods for Execution ---

    private void executeDelete(Connection conn, String sql, UUID hostId, UUID instanceId) throws SQLException {
         try (PreparedStatement ps = conn.prepareStatement(sql)) {
             ps.setObject(1, hostId);
             ps.setObject(2, instanceId);
             int rowsAffected = ps.executeUpdate();
             logger.debug("Deleted {} rows from {} for instance {}", rowsAffected, getTableNameFromDeleteSql(sql), instanceId);
         }
    }

    private void executeDeleteWithArray(Connection conn, String sql, UUID hostId, List<UUID> idList) throws SQLException {
        if (idList == null || idList.isEmpty()) return; // Nothing to delete if list is empty
        try (PreparedStatement ps = conn.prepareStatement(sql)) {
            ps.setObject(1, hostId);
            Array sqlArray = conn.createArrayOf("UUID", idList.toArray());
            ps.setArray(2, sqlArray);
            int rowsAffected = ps.executeUpdate();
            logger.debug("Deleted {} rows from {} for {} IDs", rowsAffected, getTableNameFromDeleteSql(sql), idList.size());
            sqlArray.free();
        }
    }

    private void executeDeleteWithTwoArrays(Connection conn, String sql, UUID hostId, List<UUID> idList1, List<UUID> idList2) throws SQLException {
        if (idList1 == null || idList1.isEmpty() || idList2 == null || idList2.isEmpty()) return;
         try (PreparedStatement ps = conn.prepareStatement(sql)) {
             ps.setObject(1, hostId);
             Array sqlArray1 = conn.createArrayOf("UUID", idList1.toArray());
             Array sqlArray2 = conn.createArrayOf("UUID", idList2.toArray());
             ps.setArray(2, sqlArray1);
             ps.setArray(3, sqlArray2);
             int rowsAffected = ps.executeUpdate();
             logger.debug("Deleted {} rows from {} for {}x{} IDs", rowsAffected, getTableNameFromDeleteSql(sql), idList1.size(), idList2.size());
             sqlArray1.free();
             sqlArray2.free();
         }
    }


    private void executeInsertSelect(Connection conn, String sql, UUID snapshotId, UUID hostId, UUID instanceId) throws SQLException {
         try (PreparedStatement ps = conn.prepareStatement(sql)) {
             ps.setObject(1, snapshotId);
             ps.setObject(2, hostId);
             ps.setObject(3, instanceId);
             int rowsAffected = ps.executeUpdate();
              logger.debug("Inserted {} rows into {} from snapshot {}", rowsAffected, getTableNameFromInsertSql(sql), snapshotId);
         }
    }

     private void executeInsertSelectWithArray(Connection conn, String sql, UUID snapshotId, UUID hostId, List<UUID> idList) throws SQLException {
         if (idList == null || idList.isEmpty()) return; // No scope to insert for
         try (PreparedStatement ps = conn.prepareStatement(sql)) {
             ps.setObject(1, snapshotId);
             ps.setObject(2, hostId);
             Array sqlArray = conn.createArrayOf("UUID", idList.toArray());
             ps.setArray(3, sqlArray);
             int rowsAffected = ps.executeUpdate();
             logger.debug("Inserted {} rows into {} from snapshot {} for {} IDs", rowsAffected, getTableNameFromInsertSql(sql), snapshotId, idList.size());
             sqlArray.free();
         }
     }

     private void executeInsertSelectWithTwoArrays(Connection conn, String sql, UUID snapshotId, UUID hostId, List<UUID> idList1, List<UUID> idList2) throws SQLException {
         if (idList1 == null || idList1.isEmpty() || idList2 == null || idList2.isEmpty()) return;
         try (PreparedStatement ps = conn.prepareStatement(sql)) {
             ps.setObject(1, snapshotId);
             ps.setObject(2, hostId);
             Array sqlArray1 = conn.createArrayOf("UUID", idList1.toArray());
             Array sqlArray2 = conn.createArrayOf("UUID", idList2.toArray());
             ps.setArray(3, sqlArray1);
             ps.setArray(4, sqlArray2);
             int rowsAffected = ps.executeUpdate();
              logger.debug("Inserted {} rows into {} from snapshot {} for {}x{} IDs", rowsAffected, getTableNameFromInsertSql(sql), snapshotId, idList1.size(), idList2.size());
             sqlArray1.free();
             sqlArray2.free();
         }
     }


    // --- Helper methods to find associated IDs (same as before) ---
    private List<UUID> findRelevantInstanceApiIds(Connection conn, UUID hostId, UUID instanceId) throws SQLException {
        // ... implementation ...
    }
    private List<UUID> findRelevantInstanceAppIds(Connection conn, UUID hostId, UUID instanceId) throws SQLException {
        // ... implementation ...
    }

    // --- Optional: Helper to get table name from SQL for logging ---
    private String getTableNameFromDeleteSql(String sql) {
        // Simple parsing, might need adjustment
        try { return sql.split("FROM ")[1].split(" ")[0]; } catch (Exception e) { return "[unknown table]"; }
    }
    private String getTableNameFromInsertSql(String sql) {
        try { return sql.split("INTO ")[1].split(" ")[0]; } catch (Exception e) { return "[unknown table]"; }
    }

     // --- Optional: Audit Logging Method ---
    // private void logRollbackActivity(Connection conn, UUID snapshotId, ...) throws SQLException { ... }

}

Explanation:

1. Parameter Extraction: Gets snapshotId, hostId, instanceId.
2. Transaction Control: Sets setAutoCommit(false) at the start and uses commit() or rollback().
3. Find Current Related IDs: Queries instance_api_t and instance_app_t to find the currently associated APIs and Apps for the target instance. This is needed to define the scope for the DELETE operations.
4. Execute Deletes: Uses helper methods to execute DELETE statements against the live override tables (instance_property_t, instance_api_property_t, etc.), filtering by hostId, instanceId, and the arrays of currentApiIds / currentAppIds where applicable. This clears the current overrides within the specific scope.
5. Execute Inserts: Uses helper methods to execute INSERT INTO [live_table] SELECT ... FROM [snapshot_table] statements.
   • These statements select data from the snapshot_*_property_t tables.
   • They are filtered by the target snapshotId.
   • Crucially, they are also filtered by hostId, instanceId, and the relevant instance_api_id / instance_app_id arrays. This ensures only the overrides relevant to this instance within that snapshot are inserted.
6. SQL Arrays: Uses connection.createArrayOf("UUID", ...) to pass lists of IDs efficiently to PostgreSQL’s ANY(?) operator. Remember to free() the array resources.
7. Helpers: Includes helper methods for executing the different types of DELETE/INSERT statements and for finding the related API/App IDs.
8. Error Handling: Uses try-catch-finally to manage the transaction and connection state.
9. Logging: Added basic logging for start, success, errors, and rows affected.
10. Scope Focus: This implementation focuses the rollback on the instance and its direct sub-levels (API, App, App-API). It assumes rolling back an instance doesn’t automatically mean rolling back shared Environment or Product Version overrides.

This approach provides a clean and relatively robust way to restore the specific override state from your snapshots using the DELETE/INSERT strategy. Remember to test thoroughly!

Instance File Config Phase

Overview

instance_file_t stores instance-specific files that are not modeled as standard config_property_t rows. Examples include API specifications such as openapi.yaml and custom certificates or supporting files.

The config snapshot model currently separates two kinds of file data:

• Standard files are flattened into config_snapshot_property_t.
• Non-standard instance files are copied into snapshot_instance_file_t.

The /config-server/files endpoint must return both sets. It already filters standard files by config_phase through config_snapshot_property_t.config_phase, but instance_file_t and snapshot_instance_file_t do not currently carry config_phase. That makes it impossible to union the two sources while preserving runtime, deployment, and generator phase semantics.

Problem

When a service starts through DefaultConfigLoader, it calls /config-server/files with host, serviceId, and envTag. The endpoint resolves the current snapshot and returns the files that should be written into /config.

For the sidecar case, openapi.yaml exists in both instance_file_t and snapshot_instance_file_t, but it does not exist in config_snapshot_property_t. Since the current /files query reads only config_snapshot_property_t, the response does not include openapi.yaml, and the sidecar cannot write it to /config.

The correct endpoint behavior is:

1. Read standard files from config_snapshot_property_t.
2. Read non-standard files from snapshot_instance_file_t.
3. Filter both sources by the requested config phase.
4. Return one filename-to-base64-content map.

Decision

Add config_phase to both runtime and snapshot instance file tables:

• instance_file_t.config_phase
• snapshot_instance_file_t.config_phase

The allowed values should match config_t.config_phase:

• G: generator
• D: deployment
• R: runtime

The default value for existing and new rows should be R, because current instance files are consumed by runtime startup unless explicitly marked otherwise.

Schema Changes

Runtime Table

ALTER TABLE instance_file_t
  ADD COLUMN config_phase CHAR(1) NOT NULL DEFAULT 'R';

ALTER TABLE instance_file_t
  ADD CHECK (config_phase IN ('G', 'D', 'R'));

ALTER TABLE instance_file_t
  DROP CONSTRAINT IF EXISTS instance_file_uk;

ALTER TABLE instance_file_t
  ADD CONSTRAINT instance_file_uk
    UNIQUE (host_id, instance_id, config_phase, v_file_name);

The unique constraint must include config_phase so the same filename can exist separately for runtime and deployment if needed.

Snapshot Table

ALTER TABLE snapshot_instance_file_t
  ADD COLUMN config_phase CHAR(1) NOT NULL DEFAULT 'R';

ALTER TABLE snapshot_instance_file_t
  ADD CHECK (config_phase IN ('G', 'D', 'R'));

CREATE INDEX idx_snap_inst_file_phase
  ON snapshot_instance_file_t (snapshot_id, config_phase, file_type, active);

The primary key can remain (snapshot_id, host_id, instance_file_id) because instance_file_id identifies the copied runtime row. The phase-aware index supports config-server lookups.

Migration

Existing rows should be backfilled to runtime:

UPDATE instance_file_t
SET config_phase = 'R'
WHERE config_phase IS NULL;

UPDATE snapshot_instance_file_t
SET config_phase = 'R'
WHERE config_phase IS NULL;

If a historical custom file was actually intended for deployment or generator use, it must be corrected explicitly after migration. There is no reliable way to infer that from the current schema.

Snapshot Creation

create_snapshot must copy config_phase from instance_file_t into snapshot_instance_file_t.

Current copy shape:

INSERT INTO snapshot_instance_file_t (
    snapshot_id, host_id, instance_file_id, instance_id, file_type,
    file_name, file_value, file_desc, expiration_ts,
    aggregate_version, active, update_user, update_ts
)
SELECT
    p_snapshot_id, t.host_id, t.instance_file_id, t.instance_id, t.file_type,
    t.file_name, t.file_value, t.file_desc, t.expiration_ts,
    t.aggregate_version, t.active, t.update_user, t.update_ts
FROM instance_file_t t
WHERE t.host_id = p_host_id
  AND t.instance_id = p_instance_id
  AND t.active = TRUE;

Target copy shape:

INSERT INTO snapshot_instance_file_t (
    snapshot_id, host_id, instance_file_id, instance_id, config_phase,
    file_type, file_name, file_value, file_desc, expiration_ts,
    aggregate_version, active, update_user, update_ts
)
SELECT
    p_snapshot_id, t.host_id, t.instance_file_id, t.instance_id, t.config_phase,
    t.file_type, t.file_name, t.file_value, t.file_desc, t.expiration_ts,
    t.aggregate_version, t.active, t.update_user, t.update_ts
FROM instance_file_t t
WHERE t.host_id = p_host_id
  AND t.instance_id = p_instance_id
  AND t.active = TRUE;

Snapshot creation should continue copying all active instance files for the instance. Consumers filter by phase when reading.

Config Server Query

The /files endpoint should union standard files and non-standard instance files for the current snapshot.

Standard files:

SELECT
    p.source_level AS source,
    c.config_name,
    p.property_name,
    p.value_type,
    p.property_value,
    10 AS source_rank
FROM config_snapshot_property_t p
JOIN config_snapshot_t cs ON cs.snapshot_id = p.snapshot_id
JOIN config_t c ON c.config_id = p.config_id
JOIN host_t h ON cs.host_id = h.host_id
WHERE h.sub_domain || '.' || h.domain = ?
  AND cs.current = TRUE
  AND p.config_phase = ?
  AND p.property_type = 'File'
  AND cs.service_id = ?
  AND cs.environment = ?

Non-standard instance files:

SELECT
    'instance_file' AS source,
    'files' AS config_name,
    f.file_name AS property_name,
    'string' AS value_type,
    f.file_value AS property_value,
    100 AS source_rank
FROM snapshot_instance_file_t f
JOIN config_snapshot_t cs
  ON cs.snapshot_id = f.snapshot_id
 AND cs.host_id = f.host_id
 AND cs.instance_id = f.instance_id
JOIN host_t h ON h.host_id = cs.host_id
WHERE h.sub_domain || '.' || h.domain = ?
  AND cs.current = TRUE
  AND f.config_phase = ?
  AND f.file_type = 'File'
  AND f.active = TRUE
  AND cs.service_id = ?
  AND cs.environment = ?

The implementation can combine these with UNION ALL. If the same filename appears in both sources, the instance file should win because it is the instance-specific override. Java can enforce this by inserting standard rows first and custom rows second into the response map. SQL can enforce it with source_rank and DISTINCT ON (property_name) if the response is assembled directly from a result set.

The same model should be applied to /certs with property_type = 'Cert' and file_type = 'Cert', because instance_file_t.file_type already supports certificates.

API and Event Changes

All create, update, query, and replay paths for instance files should include configPhase.

Required behavior:

• New create/update requests accept configPhase.
• Missing configPhase defaults to R for backward compatibility.
• Created and updated events include configPhase.
• Replay of historical events defaults missing configPhase to R.
• Query responses expose configPhase.
• UI forms and grids allow the operator to choose or filter by phase.

Code Impact

Expected implementation surfaces:

• portal-db/postgres/ddl.sql
• portal-db/postgres/ddl-dbvis.sql
• New portal-db/postgres/patch_*.sql
• portal-db/postgres/sp_tr_fn.sql
• light-portal/db-provider persistence for create, update, query, snapshot, clone, and replay flows
• light-config-server snapshot /files and /certs query behavior through ConfigServerQueryPersistenceImpl
• portal-service/crates/portal-core snapshot file and cert queries
• portal-service/apps/config-server response assembly if duplicate precedence is handled outside SQL
• portal-view schemas/forms/pages for instance files

Validation

Minimum checks:

1. Create or migrate an instance file named openapi.yaml with config_phase = 'R'.
2. Create a snapshot for the instance.
3. Verify snapshot_instance_file_t has the same config_phase.
4. Call /config-server/files?host=dev.lightapi.net&serviceId=...&envTag=dev.
5. Confirm the response contains both standard files such as logback.xml and non-standard files such as openapi.yaml.
6. Start a sidecar with DefaultConfigLoader and confirm /config/openapi.yaml is written.

Regression tests should cover:

• Existing instance files default to runtime.
• Same filename can exist in different phases.
• /files filters out non-matching phases.
• Custom instance files override standard files with the same filename.
• Java and Rust config-server implementations return the same file keys.

Out of Scope

This change does not move non-standard files into config_snapshot_property_t. Keeping them in snapshot_instance_file_t preserves the distinction between modeled config properties and instance-specific file artifacts.

Deployment

Deployment service allows users to deploy and manage their configured light products. This service is used by the application and api developers and operations.

The deployment service contains pipeline management, platform management and deployment management. It also integrates with product management and instance management services.

Timestamp

Okay, let’s break down the best way to persist Java’s OffsetDateTime in PostgreSQL.

1. Best Database Column Type: TIMESTAMP WITH TIME ZONE (or TIMESTAMPTZ)

This is unequivocally the best choice in PostgreSQL for storing OffsetDateTime objects. Here’s why:

• Preserves the Instant: OffsetDateTime represents a specific instant in time with an offset from UTC. TIMESTAMPTZ is designed precisely for this.
• UTC Normalization: When you insert a value into a TIMESTAMPTZ column, PostgreSQL uses the provided offset to normalize the timestamp and stores it internally as UTC. This is crucial for correctly representing the absolute point in time, regardless of the original offset.
• Automatic Conversion on Retrieval: When you select data from a TIMESTAMPTZ column, PostgreSQL automatically converts the stored UTC value back to the current session’s timezone setting (TimeZone parameter). Your JDBC driver then maps this appropriately.
• Avoids Ambiguity: Using TIMESTAMPTZ prevents the ambiguity that can arise with TIMESTAMP WITHOUT TIME ZONE, where the lack of offset/timezone information can lead to incorrect interpretations depending on server and client settings.

Why NOT TIMESTAMP WITHOUT TIME ZONE (or TIMESTAMP)?

• This type stores the date and time literally as provided, discarding any timezone or offset information.
• If you store an OffsetDateTime’s local date/time part into this column, you lose the offset, making it impossible to know the exact instant it represents globally. This is generally incorrect for OffsetDateTime.

2. How to Convert (JDBC)

Modern JDBC drivers (PostgreSQL JDBC driver versions supporting JDBC 4.2+, which is most versions used today) handle the conversion automatically and correctly when you use the appropriate methods.

Persisting (Saving):

• Use PreparedStatement.setObject(int parameterIndex, OffsetDateTime value)

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;

// Assume 'conn' is your established JDBC Connection
public void saveEventTime(Connection conn, int eventId, OffsetDateTime eventTime) throws SQLException {
    // Use TIMESTAMPTZ in your table definition
    String sql = "UPDATE events SET event_timestamp = ? WHERE id = ?";

    try (PreparedStatement pstmt = conn.prepareStatement(sql)) {
        // Use setObject for OffsetDateTime - the driver handles conversion
        pstmt.setObject(1, eventTime); // Pass the OffsetDateTime directly
        pstmt.setInt(2, eventId);
        pstmt.executeUpdate();
    }
}

// Example Usage:
// OffsetDateTime nowWithOffset = OffsetDateTime.now(); // Uses system default offset
// OffsetDateTime specificTime = OffsetDateTime.of(2023, 10, 27, 10, 30, 0, 0, ZoneOffset.ofHours(-4));
// saveEventTime(connection, 1, specificTime);

The JDBC driver sends the OffsetDateTime (including its offset) to PostgreSQL. PostgreSQL’s TIMESTAMPTZ type normalizes this to UTC for storage.

Retrieving (Loading):

• Use ResultSet.getObject(int columnIndex, OffsetDateTime.class) or ResultSet.getObject(String columnLabel, OffsetDateTime.class)

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.time.OffsetDateTime;

public OffsetDateTime loadEventTime(Connection conn, int eventId) throws SQLException {
    String sql = "SELECT event_timestamp FROM events WHERE id = ?";
    OffsetDateTime eventTime = null;

    try (PreparedStatement pstmt = conn.prepareStatement(sql)) {
        pstmt.setInt(1, eventId);
        try (ResultSet rs = pstmt.executeQuery()) {
            if (rs.next()) {
                // Use getObject with the target class - the driver handles conversion
                eventTime = rs.getObject("event_timestamp", OffsetDateTime.class);
            }
        }
    }
    return eventTime;
}

// Example Usage:
// OffsetDateTime retrievedTime = loadEventTime(connection, 1);
// if (retrievedTime != null) {
//     System.out.println("Retrieved: " + retrievedTime);
//     // Note: The offset might be different from the original if your
//     // JVM's default timezone or JDBC connection timezone differs
//     // from the original offset, but it represents the SAME instant in time.
// }

When retrieving, PostgreSQL sends the stored UTC timestamp. The JDBC driver, knowing the target type is OffsetDateTime and considering the session’s timezone setting, converts the UTC value back into an appropriate OffsetDateTime object representing the correct instant.

Summary:

1. Database Column: Use TIMESTAMP WITH TIME ZONE (TIMESTAMPTZ).
2. Persisting (Java -> DB): Use PreparedStatement.setObject(index, yourOffsetDateTime).
3. Retrieving (DB -> Java): Use ResultSet.getObject(column, OffsetDateTime.class).
4. JDBC Driver: Ensure you are using a modern PostgreSQL JDBC driver that supports JDBC 4.2 / Java 8 Time API.
5. Session Timezone: Be aware that the OffsetDateTime retrieved might have an offset corresponding to the client/session’s timezone setting, but it will represent the same exact instant as the one stored (because it was normalized to UTC).

Tag

Let’s design a tagging system for your light-portal entities. Tags are typically non-hierarchical keywords or labels that you can assign to entities for flexible organization and discovery, complementing categories.

1. Database Design (PostgreSQL)

For a flexible and efficient tagging system, we’ll use two main tables: a central tags table and a join table entity_tags to create a many-to-many relationship between entities and tags.

a) tag Table: Stores the definitions of the tags themselves.

CREATE TABLE tag_t (
    tag_id        VARCHAR(22) NOT NULL,         -- Unique ID for the tag
    host_id       VARCHAR(22),                  -- null means global tag 
    tag_name      VARCHAR(100) UNIQUE NOT NULL, -- Tag name (e.g., "featured", "urgent", "api", "documentation") - Enforce uniqueness
    tag_desc      VARCHAR(1024),                -- Optional description of the tag
    update_user   VARCHAR(255) DEFAULT SESSION_USER NOT NULL,
    update_ts     TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY (tag_id)
);

-- Index for efficient lookup by tag_name (common search/filter)
CREATE INDEX idx_tags_tag_name ON tags_t (tag_name);

• tag_id: Unique identifier for each tag.
• tag_name: The actual tag value (e.g., “featured”). UNIQUE NOT NULL constraint ensures tag names are unique across the system (global tags in this design).
• tag_desc: Optional description for the tag.
• update_user, update_ts: Standard audit columns.
• UNIQUE (tag_name): Important constraint to ensure tag names are unique. This makes tag management simpler and consistent.

b) entity_tags_t Join Table (Many-to-Many Relationship): Links entities to tags.

CREATE TABLE entity_tags_t (
    entity_id   VARCHAR(22) NOT NULL,      -- ID of the entity (schema, product, document, etc.)
    entity_type VARCHAR(50) NOT NULL,     -- Type of the entity ('schema', 'product', 'document', etc.)
    tag_id      VARCHAR(22) NOT NULL REFERENCES tags_t(tag_id) ON DELETE CASCADE, -- Foreign key to tags_t

    PRIMARY KEY (entity_id, entity_type, tag_id) -- Composite primary key to prevent duplicate tag assignments to the same entity
);

-- Indexes for efficient queries
CREATE INDEX idx_entity_tags_tag_id ON entity_tags_t (tag_id);        -- Find entities by tag
CREATE INDEX idx_entity_tags_entity ON entity_tags_t (entity_id, entity_type); -- Find tags for an entity

• entity_id: ID of the entity being tagged.
• entity_type: Type of the entity (must match the types you use for categories and other entity-related tables).
• tag_id: Foreign key referencing the tags_t table.
• Composite Primary Key (entity_id, entity_type, tag_id): Ensures that an entity of a specific type cannot be associated with the same tag multiple times.
• ON DELETE CASCADE: If a tag is deleted from tags_t, all associations in entity_tags_t are automatically removed. Consider ON DELETE RESTRICT if you want to prevent tag deletion if it’s still in use.

2. Service Endpoints

You’ll need service endpoints to manage tags themselves and to manage the associations between tags and entities.

a) Tag Management Endpoints (Likely in a TagService or Admin-Specific Service):

• POST /tags - Create a new tag
  • Request Body (JSON):

    {
      "tagId": "uniqueTagId123",  // Optional - let backend generate if not provided
      "tagName": "featured",      // Required - unique tag name
      "tagDesc": "Items that are highlighted or promoted" // Optional
    }
    

  • Response: 201 Created, with Location header (URL of the new tag) and response body (created tag JSON).
• GET /tags - List all tags (with pagination, filtering, sorting - similar to getCategory endpoint)
  • Query Parameters: offset, limit, tagName, tagDesc, etc.
  • Response: 200 OK, JSON array of tag objects (with total count).
• GET /tags/{tagId} - Get a specific tag by ID
  • Path Parameter: tagId
  • Response: 200 OK, tag object in JSON. 404 Not Found if not exists.
• PUT /tags/{tagId} - Update an existing tag
  • Path Parameter: tagId
  • Request Body (JSON): (Same structure as POST, but tagId in the path is used for identification)
  • Response: 200 OK, updated tag object in JSON. 404 Not Found if tag not found.
• DELETE /tags/{tagId} - Delete a tag
  • Path Parameter: tagId
  • Response: 204 No Content. 404 Not Found if tag not found.

b) Entity Tag Association Endpoints (Likely within Entity-Specific Services like SchemaService, ProductService):

• (Within POST /schemas, PUT /schemas/{schemaId}, etc. entity creation/update endpoints):
  • Request Body for creating or updating an entity should include a field (e.g., tagIds: ["tagId1", "tagId2"]) to specify the tags to associate with the entity.
  • Service logic (like in the updated createSchema and updateSchema methods) will handle updating the entity_tags_t table (deleting old links and inserting new ones) within the same transaction as the entity creation/update.
• GET /schemas/{schemaId}/tags (or /products/{productId}/tags, etc.) - Get tags associated with a specific entity
  • Path Parameter: schemaId (or productId, etc.)
  • Response: 200 OK, JSON array of tag objects associated with the entity.
• PUT /schemas/{schemaId}/tags (or similar) - Replace tags associated with an entity (Less common, often handled within the entity update endpoint directly)
  • Path Parameter: schemaId
  • Request Body (JSON): { "tagIds": ["tagIdA", "tagIdB"] } - list of tag IDs to associate.
  • Response: 200 OK, updated entity object (or just 204 No Content).

c) Entity Filtering/Search Endpoints:

• GET /schemas (or /products, /documents, etc.) - List entities, now with tag filtering:
  • Query Parameter: tagNames (or tagIds, or tags - choose one and be consistent), e.g., tagNames=featured,api&tagNames=urgent (multiple tags to filter by).
  • Backend logic: Modify the getSchema (or getProduct, getDocument, etc.) service methods to:
    1. Parse the tagNames parameter (could be comma-separated, multiple parameters, etc.).
    2. Modify the SQL query to include a JOIN with entity_tags_t and tags_t and add a WHERE clause to filter by the provided tag names. You might need to use EXISTS or IN subqueries for efficient filtering by multiple tags.

Example Query for Filtering Schemas by Tags (using PostgreSQL EXISTS):

SELECT schema_t.*, ... -- Select schema columns
FROM schema_t
WHERE EXISTS (
    SELECT 1
    FROM entity_tags_t et
    INNER JOIN tags_t t ON et.tag_id = t.tag_id
    WHERE et.entity_id = schema_t.schema_id
      AND et.entity_type = 'schema'
      AND t.tag_name IN (?, ?, ?) -- Parameterized tag names list
);

UI Considerations:

• Tag Management UI: Similar to category management, likely an admin section to create, edit, delete tags.
• Tag Assignment UI:
  • Entity creation/edit forms should include a tag selection component (e.g., tag input with autocomplete, checkboxes, tag pills).
  • Allow users to search/browse existing tags and assign them.
• Tag Filtering/Browsing UI:
  • Display tags prominently (tag cloud, list, filters).
  • Clicking/selecting a tag should filter the entity lists to show only entities associated with that tag.

Benefits of this Tagging System:

• Flexible Organization: Tags are free-form and non-hierarchical, allowing for more flexible and ad-hoc categorization than categories alone.
• Discoverability: Improves search and filtering capabilities, making it easier for users to find relevant entities.
• Metadata Enrichment: Tags add valuable metadata to entities.
• Scalability: The database design is efficient for querying and managing tags and associations even with a large number of entities and tags.

This design provides a solid foundation for a tagging system. You can further refine it based on your specific requirements, such as adding tag groups, permissions for tag management, or more advanced search capabilities.

UUID

In the light-portal database, we are using UUID for most of the keys in order to support event replay between multiple environments. To balance database performance with the need for URL-friendly, we are using the PostgreSQL native UUID type for the key.

CREATE TABLE your_table (
    id UUID PRIMARY KEY,
    -- other columns
);

The PostgreSQL can only generate UUIDv4 and it causes index locality problem. So we are using Java to generate UUIDv7 which is Time-Ordered UUID. These embed a timestamp, making them roughly sequential and significantly improving index locality and insert performance. You’ll need a library for this.

import com.github.f4b6a3.uuid.UuidCreator;
import java.util.UUID;

// In your entity or service
UUID primaryKey = UuidCreator.getTimeOrderedEpoch(); // UUIDv7
// Store this 'primaryKey' directly.

In light-4j utility module, we have a UuidUtil class that can generate the UUIDv7 and also encode/decode to base64 string.

Here is the class.

package com.networknt.utility;

import com.github.f4b6a3.uuid.UuidCreator;
import java.util.Base64;
import java.util.UUID;
import java.nio.ByteBuffer;

public class UuidUtil {

    // Use Java 8's built-in Base64 encoder/decoder
    private static final Base64.Encoder URL_SAFE_ENCODER = Base64.getUrlEncoder().withoutPadding();
    private static final Base64.Decoder URL_SAFE_DECODER = Base64.getUrlDecoder();

    public static UUID getUUID() {
        return UuidCreator.getTimeOrderedEpoch(); // UUIDv7
    }

    /**
     * Generate a UUID and encode it to a URL-safe Base64 string.
     *
     * @return A URL-safe Base64 encoded UUID string.
     */
    public static String uuidToBase64(UUID uuid) {
        ByteBuffer bb = ByteBuffer.wrap(new byte[16]);
        bb.putLong(uuid.getMostSignificantBits());
        bb.putLong(uuid.getLeastSignificantBits());
        return URL_SAFE_ENCODER.encodeToString(bb.array());
    }

    /**
     * Decode a URL-safe Base64 string back to a UUID.
     *
     * @param base64 A URL-safe Base64 encoded UUID string.
     * @return The decoded UUID.
     */
    public static UUID base64ToUuid(String base64) {
        byte[] bytes = URL_SAFE_DECODER.decode(base64);
        ByteBuffer bb = ByteBuffer.wrap(bytes);
        long high = bb.getLong();
        long low = bb.getLong();
        return new UUID(high, low);
    }

}

Composit key vs Surrogate UUID key

Composite key with 5 or more columns

User the following three tables as examples. We have composite key with 5 columns and some of them are varchar types in product version_property_t table. Is is a good idea to create UUID keys for config_property_t and product_version_t?

-- each config file will have a config_id reference and this table contains all the properties including default. 
CREATE TABLE config_property_t (
    config_id                 UUID NOT NULL,
    property_name             VARCHAR(64) NOT NULL,
    property_type             VARCHAR(32) DEFAULT 'Config' NOT NULL,
    light4j_version           VARCHAR(12), -- only newly introduced property has a version.
    display_order             INTEGER,
    required                  BOOLEAN DEFAULT false NOT NULL,
    property_desc             VARCHAR(4096),
    property_value            TEXT,
    value_type                VARCHAR(32),
    property_file             TEXT,
    resource_type             VARCHAR(30) DEFAULT 'none',
    update_user               VARCHAR(255) DEFAULT SESSION_USER NOT NULL,
    update_ts                 TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP NOT NULL
);

ALTER TABLE config_property_t
    ADD CHECK ( property_type IN ( 'Cert', 'Config', 'File') );


COMMENT ON COLUMN config_property_t.property_value IS
    'Property Default Value';

COMMENT ON COLUMN config_property_t.value_type IS
    'One of string, boolean, integer, float, map, list';

COMMENT ON COLUMN config_property_t.resource_type IS
  'One of none, api, app, app_api, api|app_api, app|app_api, all';

ALTER TABLE config_property_t ADD CONSTRAINT config_property_pk PRIMARY KEY ( config_id, property_name );



CREATE TABLE product_version_t (
    host_id                     UUID NOT NULL,
    product_id                  VARCHAR(8) NOT NULL,
    product_version             VARCHAR(12) NOT NULL, -- internal product version 
    light4j_version             VARCHAR(12) NOT NULL, -- open source release version
    break_code                  BOOLEAN DEFAULT false, -- breaking code change to upgrade to this version.
    break_config                BOOLEAN DEFAULT false, -- config server need this to decide if clone is allowed for this version. 
    release_note                TEXT,
    version_desc                VARCHAR(1024),
    release_type                VARCHAR(24) NOT NULL, -- Alpha Version, Beta Version, Release Candidate, General Availability, Production Release
    current                     BOOLEAN DEFAULT false,
    version_status              VARCHAR(16) NOT NULL, 
    update_user                 VARCHAR (255) DEFAULT SESSION_USER NOT NULL,
    update_ts                   TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP NOT NULL,
    PRIMARY KEY(host_id, product_id, product_version)
);


-- customized property for product version within the host.
CREATE TABLE product_version_property_t (
    host_id          UUID NOT NULL,
    product_id       VARCHAR(8) NOT NULL,
    product_version  VARCHAR(12) NOT NULL,
    config_id        UUID NOT NULL,
    property_name    VARCHAR(64) NOT NULL,
    property_value   TEXT,
    property_file    TEXT,
    update_user      VARCHAR (126) DEFAULT SESSION_USER NOT NULL,
    update_ts        TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP NOT NULL
);

ALTER TABLE product_version_property_t
    ADD CONSTRAINT product_version_property_pk PRIMARY KEY ( host_id, 
                                                             product_id,
                                                             product_version,
                                                             config_id,
                                                             property_name);

Replacing the wide, multi-column composite primary key in product_version_property_t with one based on surrogate UUID keys from the parent tables is generally a very good idea for several reasons, primarily related to performance and maintainability.

Let’s break down the pros and cons:

Current Situation (Composite Keys):

• config_property_t PK: (config_id UUID, property_name VARCHAR(64)) - Size: 16 + (up to 64 + overhead) bytes. Reasonably sized.
• product_version_t PK: (host_id UUID, product_id VARCHAR(8), product_version VARCHAR(12)) - Size: 16 + (8 + overhead) + (12 + overhead) bytes. Getting wider.
• product_version_property_t PK: (host_id UUID, product_id VARCHAR(8), product_version VARCHAR(12), config_id UUID, property_name VARCHAR(64)) - Size: 16 + (8+oh) + (12+oh) + 16 + (64+oh) bytes. This is quite wide.

Problems with the Wide Composite Key in product_version_property_t:

1. Large Primary Key Index: The B-tree index for this 5-column PK will be large. Larger indexes consume more disk space, more memory (reducing cache efficiency), and take longer to scan during queries, updates, and deletes.
2. Large Foreign Keys (Implicit): If any other table were to reference product_version_property_t (unlikely given its nature, but possible), the foreign key would also need all 5 columns.
3. Larger Secondary Indexes: Any other indexes you create on product_version_property_t will typically include the primary key columns implicitly (in PostgreSQL leaf nodes). A wide PK bloats all other indexes on the table, magnifying the performance impact.
4. Update Overhead: Updating any of the PK columns (though generally discouraged) is very expensive. More importantly, index maintenance during INSERTs and DELETEs is heavier with wider keys.
5. Join Complexity: Joining on 5 columns is syntactically more verbose.

Proposed Solution (Surrogate UUID Keys):

1. Add Surrogate Key to config_property_t:

   • Add a new column: cp_id UUID PRIMARY KEY DEFAULT gen_random_uuid() (or use UUIDv7 generator).
   • Add a UNIQUE constraint on the original natural key: ALTER TABLE config_property_t ADD CONSTRAINT config_property_nat_key UNIQUE (config_id, property_name); (This is CRITICAL to maintain data integrity).
   • Size of new PK: 16 bytes.

2. Add Surrogate Key to product_version_t:

   • Add a new column: pv_id UUID PRIMARY KEY DEFAULT gen_random_uuid() (or UUIDv7).
   • Add a UNIQUE constraint on the original natural key: ALTER TABLE product_version_t ADD CONSTRAINT product_version_nat_key UNIQUE (host_id, product_id, product_version); (CRITICAL).
   • Size of new PK: 16 bytes.

3. Modify product_version_property_t:

   • Remove the existing 5-column PK.
   • Replace the 5 individual columns used for the PK/FK relationship with two new columns: pv_id UUID and cp_id UUID.
   • Create the new Primary Key: ALTER TABLE product_version_property_t ADD CONSTRAINT product_version_property_pk PRIMARY KEY (pv_id, cp_id);
   • Add Foreign Key constraints:

     ALTER TABLE product_version_property_t
         ADD CONSTRAINT product_version_property_fk_pv
         FOREIGN KEY (pv_id) REFERENCES product_version_t(pv_id) ON DELETE CASCADE; -- Or appropriate action
     
     ALTER TABLE product_version_property_t
         ADD CONSTRAINT product_version_property_fk_cp
         FOREIGN KEY (cp_id) REFERENCES config_property_t(cp_id) ON DELETE CASCADE; -- Or appropriate action
     

   • Size of new PK: 16 + 16 = 32 bytes.

Advantages of the Proposed Solution:

1. Significantly Smaller PK Index: The PK index on product_version_property_t shrinks from potentially >100 bytes wide to just 32 bytes. This is the biggest win.
2. Smaller Secondary Indexes: All other indexes on product_version_property_t will also be smaller.
3. Improved Query Performance: Smaller indexes mean faster scans, better cache utilization, and quicker lookups, updates, and deletes involving the PK or indexes on product_version_property_t.
4. Simpler Joins: Joining product_version_property_t to its parent tables now only requires joining on the single pv_id or cp_id columns.
5. Simpler Foreign Keys: The FK definitions are simpler.

Disadvantages/Considerations:

1. Need for UNIQUE Constraints: You absolutely must add the UNIQUE constraints on the original natural keys in the parent tables (config_property_t, product_version_t) to prevent duplicate entries based on business logic. This adds two new indexes.
2. Storage Overhead (Parent Tables): You add 16 bytes per row to config_property_t and product_version_t for the new UUID PKs.
3. Migration Effort: You need to:
   • Add the new UUID columns to the parent tables.
   • Populate them with unique UUIDs for all existing rows.
   • Add the new FK columns (pv_id, cp_id) to product_version_property_t.
   • Update product_version_property_t to populate these new columns by joining back to the parent tables based on the original composite keys.
   • Drop the old PK and FK constraints (or the old columns if they are no longer needed besides the value overrides).
   • Add the new PK and FK constraints.
   • Add the UNIQUE constraints on the parent tables.
4. Lookups by Natural Key: Queries that previously used the composite PK directly on the parent tables (e.g., SELECT ... FROM product_version_t WHERE host_id = ? AND product_id = ? AND product_version = ?) will now use the new UNIQUE index instead of the PK index. Performance should still be very good, but it’s technically hitting a different index.

Conclusion:

Yes, it is a good idea. The performance benefits gained from dramatically reducing the size of the primary (and consequently, secondary) indexes on the frequently accessed product_version_property_t table likely outweigh the disadvantages (extra storage on parent tables, migration effort, extra unique indexes). The simplification of joins and foreign keys is also a significant maintainability improvement.

Recommendation:

Proceed with the plan to introduce surrogate UUID keys (preferably UUIDv7 for generation if using a library) on config_property_t and product_version_t, and change product_version_property_t to use these two UUIDs as its composite primary key. Do not forget to add the UNIQUE constraints on the original natural keys in the parent tables.

Composite key with 3 or 4 columns

If a composite key with 3 or 4 columns, it is less definitive than the 5-column case. It moves into a “it depends” territory, requiring weighing the pros and cons more carefully based on specifics.

Here’s a breakdown of factors to consider for 3 or 4 column composite primary keys:

Arguments for Sticking with the Composite Natural Key (CNK):

1. Simplicity (Potentially): No need for an extra surrogate key column and an extra UNIQUE index on the natural key columns. The schema might feel slightly less cluttered if the natural key is intuitive and stable.
2. Reduced Storage (Parent Table): Avoids adding 16 bytes per row for the UUID PK in the table itself.
3. Meaningful Key: The PK components have inherent business meaning, which can sometimes be useful for direct queries or understanding relationships without extra joins (though the UNIQUE index on the SUK approach provides this lookup too).
4. Migration Cost: Avoids the effort of adding columns, backfilling data, and changing referencing tables.

Arguments for Refactoring to a Surrogate UUID Key (SUK):

1. Index Size (Still Relevant): This is the biggest factor.
   • Calculate the Width: Add up the maximum potential size of the 3 or 4 columns in the CNK.
     • UUID: 16 bytes
     • INT: 4 bytes
     • BIGINT: 8 bytes
     • VARCHAR(N): N bytes + 1 or 4 bytes overhead (depending on length)
     • TIMESTAMP: 8 bytes
     • BOOLEAN: 1 byte
   • Compare: Compare the calculated width to the typical width of a surrogate key reference (16 bytes for one UUID, or 32 bytes if the child table needs two UUIDs like in your product_version_property_t example).
   • Threshold: If the CNK width starts exceeding ~32-40 bytes, the performance benefits of a narrower SUK (especially for secondary indexes and joins) become increasingly attractive. Even a 3-column key like (UUID, VARCHAR(8), VARCHAR(12)) is already 16 + (8+1) + (12+1) = 38 bytes. A 4-column key is almost certainly wider.
2. Secondary Index Bloat: Remember, all other indexes on the table implicitly include the PK columns. A wide CNK makes every index larger, impacting cache efficiency and scan speed across the board. This effect is magnified if you have many secondary indexes.
3. Join Performance/Simplicity: Joining on a single UUID column is generally faster and syntactically simpler than joining on 3 or 4 columns, especially if some are strings.
4. Foreign Key Simplicity: Tables referencing this table only need to store a single UUID column as the foreign key, rather than 3 or 4 columns. This significantly reduces storage and complexity in child tables.
5. Immutability/Stability: Surrogate keys are inherently stable. If there’s any chance the values in the natural key columns might need to change (which is generally bad practice for PKs but sometimes unavoidable), using a SUK provides crucial insulation.

Decision Rubric (Heuristics):

• Scenario 1: Key is mostly small, fixed-width types (e.g., 3 INTs, 2 INTs + UUID). Total width < ~32 bytes.
  • Lean towards: Keeping the CNK might be acceptable, especially if the table isn’t heavily referenced or doesn’t have many secondary indexes. The benefits of SUK are less pronounced.
• Scenario 2: Key includes even one moderately sized VARCHAR (e.g., VARCHAR(20+)) or multiple smaller VARCHARs. Total width > ~32-40 bytes.
  • Lean towards: Refactoring to SUK. The performance benefits from reduced index sizes (primary and secondary) and simpler joins/FKs likely start to outweigh the costs.
• Scenario 3: Key includes wide VARCHARs (VARCHAR(50+)).
  • Lean towards: Strongly consider refactoring to SUK. Benefits are significant.
• Scenario 4: The table is frequently referenced by other tables via FK.
  • Lean towards: Refactoring to SUK to simplify the child tables.
• Scenario 5: The table has many secondary indexes.
  • Lean towards: Refactoring to SUK to keep those indexes smaller.
• Scenario 6: Any component of the natural key is potentially volatile.
  • Lean towards: Refactoring to SUK for stability.

Conclusion:

For 3 or 4 columns, it’s less of a slam dunk than for 5 columns, but refactoring to a surrogate UUID key is still often beneficial, especially if VARCHARs are involved or the total key width exceeds roughly 32-40 bytes. The key drivers are the reduction in primary and secondary index size and the simplification of foreign keys and joins.

Carefully calculate the width of your specific 3 or 4 column composite keys and consider the usage patterns of the table before deciding.

Atomic Events vs Composite Events

Event Design

This is a fundamental design decision in any Event Sourcing system, and the choice has significant long-term consequences for your system’s flexibility, maintainability, and clarity.

The overwhelming consensus and best practice is to design atomic events.

Let’s break down why, and then look at the trap of composite events.

1. Atomic Events (The Recommended Approach)

An atomic event represents a single, granular, immutable fact that has occurred in your system. It is the smallest possible unit of change that has meaning to the business domain.

Example Scenario: A user updates their profile by changing their name and their shipping address on the same form.

With atomic events, this single user action (the Command) would result in two separate events being written to the stream:

1. CustomerNameChanged { customerId: "123", newName: "Jane Doe" }
2. CustomerAddressChanged { customerId: "123", newAddress: "..." }

Why Atomic Events are Superior:

a) Maximum Flexibility and Reusability:

• Targeted Consumers: You can have different parts of your system (projections, process managers, other microservices) subscribe to only the events they care about. The shipping department only needs to know about CustomerAddressChanged, while the marketing department might only care about CustomerNameChanged. With a composite event, both would have to subscribe and parse the larger event to see if the part they care about was updated.
• Future-Proofing: Six months from now, you might need to build a new feature that triggers a welcome kit to be sent when a customer provides an address for the first time. It’s trivial to add a new consumer for the CustomerAddressChanged event.

b) Clear and Unambiguous Intent:

• Each event has a single, well-defined purpose. The name of the event, like ItemAddedToCart, tells you exactly what happened.
• A composite event like ProfileUpdated is ambiguous. Did the name change? The address? The phone number? The password? You have to inspect the payload to understand the actual change, which makes the event stream harder to read and debug.

c) Simplified Event Handlers (Projections):

• Your event handlers become small and simple. The logic for handling CustomerAddressChanged is focused purely on updating the address.
• With a composite event, your handler becomes a complex series of if statements: if (event.newName != null) { updateName(); } if (event.newAddress != null) { updateAddress(); } .... This is brittle and hard to maintain.

d) Better for Versioning:

• Events are immutable and stored forever. Your system will evolve. It’s much easier to version a small, focused event (e.g., add an optional field to CustomerAddressChanged_v2) than it is to version a large, complex composite event where any one of its dozen fields might change.

2. Composite Events (The Anti-Pattern)

A composite event attempts to bundle multiple state changes into a single, large event. It’s often a result of developers mapping the event directly to the UI form or the incoming command.

Using the same scenario (updating name and address):

A composite event might look like this:

• CustomerProfileUpdated { customerId: "123", name: "Jane Doe", address: "...", phoneNumber: null }

The Problems with this Approach:

• Loss of Granularity: As explained above, you lose the ability to react to specific changes.
• Ambiguous Intent: The event name doesn’t tell the whole story.
• Sparse Payloads: What if the user only changed their name? The address and phoneNumber fields in the event payload would be null or unchanged. Your consumers have to handle these sparse, optional fields, leading to more complex logic.
• False Cohesion: It groups things together that only changed at the same time but are not necessarily part of the same business fact.

3. The Key Insight: The Role of the Aggregate

You might be thinking, “But the name and address change must be atomic! What if the system crashes after writing the first event but before the second?”

This is where the Aggregate from Domain-Driven Design (DDD) comes in. The Aggregate is the consistency boundary.

Here is the correct flow:

1. Command: A single UpdateCustomerProfileCommand is sent to the system. It contains both the new name and the new address.
2. Aggregate: The Customer Aggregate receives the command. It validates the business rules (e.g., the name isn’t empty, the address is valid).
3. Event Generation: If the rules pass, the Aggregate’s method produces a list of atomic events: [CustomerNameChanged, CustomerAddressChanged].
4. Atomic Persistence: The Event Store then takes this list of events and persists them to the event stream in a single, atomic transaction. It’s all or nothing. You will never have a CustomerNameChanged in the stream without the corresponding CustomerAddressChanged from the same command.

Key Takeaway: The Command can be composite (representing a single user gesture), but it should be resolved into a set of atomic Events. The atomicity of writing these events is guaranteed by your Aggregate and Event Store, not by bundling them into a single event type.

How to Design Good Atomic Events (A Checklist)

• Name it in the Past Tense: UserRegistered, OrderShipped, PasswordReset. It’s a fact that has already happened.
• Capture Business Intent: Don’t just record a CRUD-like change. ProductPriceUpdated is okay, but PriceAdjustedForSale is better because it captures the why.
• Ensure it’s a Complete Fact: Include all necessary data for a consumer to understand the event without having to look up previous state. For example, ItemAddedToCart should include productId, quantity, and priceAtTimeOfAdding, not just productId.
• Include Causation and Correlation IDs: Add metadata to your events. Who triggered this change (userId)? What command caused it (causationId)? What overall business process is this part of (correlationId)?
• Think “What happened?” not “What changed?”: An event is a story. OrderSubmitted is a great event. A composite event like OrderStateChanged { oldState: "Pending", newState: "Submitted" } is far less expressive.

Final Verdict

Always design atomic events.

Use the Aggregate pattern to ensure that multiple atomic events resulting from a single command are persisted transactionally. This approach gives you a system that is far more flexible, maintainable, readable, and ready for future evolution.

Transaction

When using Kafka to handle the events, we need to handle the transactions on both producer side and consumer side along with database transation to ensure multiple events can be handled together. The choice you make here directly impacts the atomicity, ordering, and resilience of your entire system.

Let’s break down the options and the correct way to handle transactions on both the producer and consumer side.

The Correct Approach: Multiple Events, One Transaction

The best practice, by far, is to push multiple, individual event messages per command, wrapped in a single Kafka producer transaction.

This approach maintains the benefits of atomic events (flexibility, clarity) while guaranteeing the all-or-nothing atomicity you need.

How to Implement This (Producer Side)

Your service that processes the command and talks to the Aggregate will do the following:

1. Generate Events: The Customer Aggregate generates the list of atomic events: [CustomerNameChanged, CustomerAddressChanged].
2. Use Kafka Producer Transactions: This is the key feature that makes it all work.

Here is a conceptual code flow (using Java-like syntax):

// IMPORTANT: Configure your producer for transactions and idempotence
// props.put("transactional.id", "my-unique-transactional-id");
// props.put("enable.idempotence", "true");
KafkaProducer<String, Event> producer = new KafkaProducer<>(props);

// The list of events from your Aggregate
List<Event> events = customerAggregate.handle(updateProfileCommand);

// 1. Initialize the transaction
producer.initTransactions();

try {
    // 2. Begin the transaction
    producer.beginTransaction();

    // The Aggregate ID (e.g., "customer-123") is the Kafka Key
    String aggregateId = customerAggregate.getId();

    for (Event event : events) {
        // 3. Send EACH event as a SEPARATE message.
        // CRUCIAL: All events for this transaction MUST have the same key.
        // This ensures they all go to the same partition and are consumed in order.
        producer.send(new ProducerRecord<>("customer-events-topic", aggregateId, event));
    }

    // 4. Commit the transaction.
    // This makes all messages in the transaction visible to consumers atomically.
    producer.commitTransaction();

} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
    // These are fatal errors, we should close the producer
    producer.close();
} catch (KafkaException e) {
    // 5. If anything goes wrong, abort. None of the messages will be visible.
    producer.abortTransaction();
}

producer.close();

Why this is the best way:

• Atomicity Guaranteed: Kafka guarantees that consumers will either see ALL the messages from commitTransaction or NONE of them (if you abortTransaction).
• Ordering Guaranteed: By using the same key (aggregateId) for all events in the transaction, you ensure they are written to the same partition in the exact order you sent them. Your consumer will read them in that same order.
• Consumer Flexibility: Your stream processors can now consume individual, meaningful events. A shipping-related processor can filter for and process only CustomerAddressChanged events, completely ignoring CustomerNameChanged.

How to Process Events Transactionally (Consumer Side)

Now, how does your streams processor populate the database tables while maintaining consistency? This is often called the “Transactional Outbox” pattern, but in reverse—a “Transactional Inbox”.

The goal is to atomically update the database AND commit the Kafka offset. You never want to commit an offset for a message whose database update failed.

Here is the standard, robust pattern for a custom consumer/streams processor:

1. Disable Kafka Auto-Commit: This is the most important step. Your application must take manual control of committing offsets. In your consumer configuration, set enable.auto.commit=false.

2. Consume and Process in Batches:

// This is a conceptual loop for your consumer
while (true) {
    // 1. Poll for a batch of records. Kafka gives you a batch.
    ConsumerRecords<String, Event> records = consumer.poll(Duration.ofMillis(1000));

    if (records.isEmpty()) {
        continue;
    }

    // Get your database connection
    Connection dbConnection = database.getConnection();
    dbConnection.setAutoCommit(false); // Start manual DB transaction management

    try {
        // 2. Process each record in the polled batch
        for (ConsumerRecord<String, Event> record : records) {
            Event event = record.value();
            // Apply the change to the database based on the event type
            processEvent(event, dbConnection);
        }

        // 3. If all events in the batch were processed successfully, commit the database transaction
        dbConnection.commit();

        // 4. IMPORTANT: Only after the DB commit succeeds, commit the Kafka offset.
        // This tells Kafka "I have successfully and durably processed all messages up to this point."
        consumer.commitSync();

    } catch (SQLException e) {
        // 5. If the DB update fails, rollback the DB transaction...
        dbConnection.rollback();
        // ...and DO NOT commit the Kafka offset.
        // The consumer will re-poll and re-process this same batch of messages later.
        // This is why your processing logic MUST be idempotent.
        System.err.println("Database update failed. Rolling back. Will retry batch.");
        // You might want to seek to the beginning of the failed batch to be explicit
        // consumer.seek(record.topic(), record.partition(), record.offset());
    } finally {
        dbConnection.close();
    }
}

It is possible to handle transactions in a Kafka Streams processor, but it requires using the low-level Processor API and is significantly more complex than the standard consumer approach. You cannot achieve this with the high-level DSL (.map(), .filter(), etc.) alone.

If your processor’s only job is to read from Kafka and write to a database: Use the Plain Kafka Consumer. It is simpler, more direct, less error-prone, and purpose-built for this task. You are essentially building a custom, lightweight Kafka Connect sink.

The Critical Need for Idempotency

Because a failure can occur after the DB commit but before the Kafka offset commit, your application might restart and re-process the same batch of events.

Your database update logic must be idempotent. This means running the same update multiple times produces the same result as running it once.

Examples of Idempotent Operations:

• INSERT with a primary key: INSERT INTO customers (...) VALUES (...) ON DUPLICATE KEY UPDATE ... (MySQL) or INSERT ... ON CONFLICT ... DO UPDATE ... (PostgreSQL).
• UPDATE statements: UPDATE customers SET name = 'Jane Doe' WHERE customer_id = '123'. Running this 5 times is the same as running it once.
• Using Versioning: Store a version or last_processed_event_id in your database table.

  UPDATE customers
  SET name = 'Jane Doe', version = 2
  WHERE customer_id = '123' AND version = 1;
  

  If the update tries to run again, the WHERE clause will not match, and no rows will be affected.

Why Not Put a List of Events in One Message?

This is an anti-pattern that solves one problem (producer atomicity) by creating many more downstream.

• Loss of Meaning: The fundamental unit is the event, not a list of events. A Kafka message should represent one fact.
• Consumer Complexity: Every single consumer now has to be written to expect a list. It has to deserialize the list and loop through it.
• No Filtering: A consumer who only cares about CustomerAddressChanged still has to receive and parse the entire message containing the CustomerNameChanged event, only to discard it. This is inefficient and tightly couples your consumers to the producer’s batching behavior.
• Versioning Hell: Versioning a list of events is much harder than versioning a single event.