Unified TDX Context Handling Design

Date: 2025-11-10 Issue: #78JIRA: DEVAI-402Status: Implementation In Progress

Deprecated: TD_API_KEY and .env file storage

This design note describes the original implementation using TD_API_KEY environment variables and .env files for credential storage. As of v0.5.0:

Keychain storage: tdx uses the system keychain (macOS Keychain, Windows Credential Manager, Linux Secret Service) for secure credential storage instead of .env files.
New environment variables: Use TDX_API_KEY and TDX_API_KEY_<PROFILE> instead of TD_API_KEY. The legacy TD_API_KEY is still supported but deprecated and will show a warning.

Summary

The tdx CLI currently requires explicit flags for many context parameters (database, agent, segment, workspace, account). This becomes cumbersome for users who work across multiple Treasure AI accounts, projects, and shell sessions.

This design introduces a unified, predictable context system based on:

Profiles (long-lived identities)
Project-local config (per-repo defaults)
PID-based session overrides (active shell context)
Global config (fallback defaults)
Environment variables used only for credentials

The goal is to reduce repeated command flags, eliminate hidden sources of state, and provide a consistent user experience across environments.

Background / Problems Today

Users working on multiple TD accounts must repeatedly specify --account, --database, --agent, etc.
Environment variables can produce opaque behavior and are hard to debug.
Project-local defaults are not standardized.
Switching accounts or databases inside a shell is inconvenient.

We need a structured, layered context model that is predictable, explicit, and ergonomic.

Proposed Context Model (Layered Resolution)

Final context is resolved in the following priority order:

1. CLI flags (--database, --parent-segment, --llm-project, --llm-agent, --profile)
2. Session context (PID-scoped; set via `tdx use ...`)
3. Project config   (tdx.json in project root)
4. Profile config   (~/.config/tdx/profiles/<name>/tdx.json)
5. Global config    (~/.config/tdx/tdx.json)
6. Built-in defaults

Credential resolution happens separately via:

TD_API_KEY / TD_API_KEY_<SITE> environment variables
Profile .env file (~/.config/tdx/profiles/<name>/.env)
Global .env file (~/.config/tdx/.env)

IMPORTANT: API keys are NEVER stored in .json files for security reasons.

1. Profiles

Profiles represent long-lived account settings and live under:

~/.config/tdx/profiles/<name>/tdx.json  (configuration)
~/.config/tdx/profiles/<name>/.env      (credentials - 0600 permissions)

Profile Configuration Example (tdx.json)

json

{
  "site": "us01",
  "database": "customer360",
  "llm_project": "Analytics",
  "llm_agent": "DataAnalyst"
}

Profile Credentials Example (.env)

bash

TD_API_KEY=key_id/secret
TD_API_KEY_US01=key_id/secret_for_us01

Users select a profile via:

bash

tdx --profile prod ...
tdx use profile prod
tdx profile use prod

2. Project Config

Per-project defaults stored in:

project/tdx.json

Example:

json

{
  "database": "customer360",
  "parent_segment": "blue_users",
  "llm_project": "CustomerAnalytics"
}

These defaults apply whenever commands are run inside the project directory.

Security: Project config files should NEVER contain API keys or credentials since they may be committed to version control.

3. PID-Based Session Context

A shell receives a temporary context stored at:

~/.config/tdx/sessions/<ppid>.json

Users modify session-level context:

bash

tdx use database analytics
tdx use parent_segment vip_users
tdx use llm_project Growth
tdx use llm_agent MarketingAnalyst

Session files are automatically cleaned up when the PID no longer exists.

This makes interactive work much easier:

bash

tdx use database analytics
tdx query "SELECT * FROM events LIMIT 10"   # no need for --database

4. Global Config

Contains persistent defaults and the active profile:

~/.config/tdx/tdx.json      (configuration)
~/.config/tdx/.env          (credentials - 0600 permissions)

Example (tdx.json):

json

{
  "profile": "work",
  "site": "us01"
}

Example (.env):

bash

TD_API_KEY=default_key_id/secret

5. Environment Variables: Credentials Only

We restrict env vars to use for authentication only:

bash

TD_API_KEY=xxxx
TD_API_KEY_US01=xxxx
TD_API_KEY_JP01=xxxx

No env vars control database, profile, region, agent, or other context.

This avoids hidden state and aligns with modern CLI best practices.

Configuration Schema

tdx.json Parameters (Explicit Naming)

All tdx.json files (project, profile, global) use the same schema with simple, explicit parameter names:

json

{
  "site": "us01",                    // TD site: us01, jp01, eu01, ap02
  "database": "my_database",         // Default database name
  "parent_segment": "segment_name",  // Parent segment name
  "llm_project": "MyProject",        // LLM project name
  "llm_agent": "AgentName",          // LLM agent name
  "profile": "production"            // Active profile (global config only)
}

Parameter Definitions:

site: Treasure Data site identifier (us01, jp01, eu01, ap02)
database: Default database name for queries
parent_segment: Parent segment name (replaces ambiguous "segment" or "segmentId")
llm_project: LLM project name (used with llm_agent for agent reference)
llm_agent: LLM agent name within the project
profile: Active profile name (only valid in global config)

.env File Format (Credentials Only)

bash

# Default API key
TD_API_KEY=key_id/secret

# Site-specific API keys (optional overrides)
TD_API_KEY_US01=key_id/secret_us01
TD_API_KEY_JP01=key_id/secret_jp01
TD_API_KEY_EU01=key_id/secret_eu01
TD_API_KEY_AP02=key_id/secret_ap02

Security Requirements:

File permissions: 0600 (owner read/write only)
Never commit to version control
Validate format: {key_id}/{secret} with / separator

Session + Project Interaction Example

bash

# In a project with:
#   tdx.json → { "database": "customer360" }

tdx use database analytics
tdx query "SELECT COUNT(*) FROM events"

→ database = analytics     # session override

After cleaning the PID session (closing the shell):

bash

tdx query "SELECT COUNT(*) FROM events"
→ database = customer360   # project default

Predictable, transparent, and debuggable.

New Commands

Use Command (Session Management)

bash

# Show current resolved context
tdx use

# Output example:
# Current Context:
#   Site: us01
#   Database: analytics
#   Parent Segment: vip_users
#   LLM Project: Growth
#   LLM Agent: MarketingAnalyst
#   API Key: abcd...wxyz

# Show resolution sources (debugging)
tdx use --debug

# Output example with sources:
# Current Context (with sources):
#   Site: us01 (profile: production)
#   Database: analytics (session)
#   Parent Segment: vip_users (project: tdx.json)
#   LLM Project: Growth (global)
#   LLM Agent: MarketingAnalyst (session)
#   API Key: abcd...wxyz (env: TD_API_KEY_US01)

# Set session-level overrides
tdx use database <name>
tdx use parent_segment <name>
tdx use llm_project <name>
tdx use llm_agent <name>
tdx use profile <name>

# Examples:
tdx use database analytics
tdx use parent_segment vip_users
tdx use llm_project Growth
tdx use llm_agent MarketingAnalyst
tdx use profile production

# Clear session context
tdx use --clear

Profile Commands

bash

# List all profiles
tdx profiles

# Output example:
# Available Profiles:
#   * production (active)
#     development
#     staging

# Show profile details
tdx profile show [name]

# Output example:
# Profile: production
#   Site: us01
#   Database: customer360
#   LLM Project: Analytics
#   API Key: abcd...wxyz (from .env)

# Switch active profile
tdx profile use <name>

# Create new profile (interactive)
tdx profile create <name>

# Delete profile (with confirmation)
tdx profile delete <name>

File Structure

# Project-level (safe to commit)
project/
  tdx.json                  # Project defaults (no secrets)

# User-level (NOT committed)
~/.config/tdx/
  tdx.json                  # Global config
  .env                      # Global credentials (0600)

  profiles/
    production/
      tdx.json              # Profile config
      .env                  # Profile credentials (0600)
    development/
      tdx.json
      .env

  sessions/
    12345.json              # PID-based session context
    67890.json

Security Considerations

File Permissions

Configuration files (tdx.json):
- Permissions: 0644 (readable by all, writable by owner)
- Safe to commit to version control (project-level)
- Never contain credentials
Credential files (.env):
- Permissions: 0600 (readable/writable by owner only)
- NEVER commit to version control
- Store in user-level directories only
Directories:
- ~/.config/tdx/profiles/<name>/: 0700
- ~/.config/tdx/sessions/: 0700

Credential Storage

Validation: Error if API key found in any .json file
Display: Mask API keys in output (show only first/last 4 chars)
Project isolation: No project-local .env files to prevent accidental commits
Future enhancement: OS keychain integration (macOS Keychain, Linux Secret Service, Windows Credential Manager)

Session Cleanup

Auto-cleanup expired sessions on CLI initialization
Session expiration: 24 hours of inactivity
Manual cleanup: tdx sessions clean

Implementation Architecture

Core Modules

src/core/profile.ts - Profile CRUD operations
src/core/project-config.ts - Project config discovery (search upward for tdx.json)
src/core/session.ts - PID-based session management
src/core/context.ts - Unified context resolution with priority
src/core/credentials.ts - Multi-source credential resolution

Utilities

src/utils/process.ts - PPID detection (get parent shell PID)
src/utils/file-permissions.ts - Secure file permission management

Commands

src/commands/context-command.ts - Show context
src/commands/use-command.ts - Set session values
src/commands/profile-command.ts - Profile management

Integration Points

src/cli.ts - Add context resolution before creating TDX instance
src/core/config.ts - Refactor to integrate with new context system
src/commands/command.ts - Add context to CommandContext interface
src/sdk/index.ts - Accept context defaults in SDKConfig

Migration & Backward Compatibility

Phase 1: Non-Breaking Additions

Add new config types and loaders (profile, project, session)
Add credential resolver with backward compatibility
Add new commands (context, use, profiles) as opt-in features

Phase 2: Migrate Existing Users

Keep existing ~/.config/tdx/.env as global credentials
Provide tdx migrate command for auto-migration to profiles
Maintain CLI flag priority (always highest)

Phase 3: Documentation

Update README.md with new commands
Provide migration guide with examples
Add troubleshooting section for context resolution

Testing Strategy

Unit Tests

src/core/profile.test.ts: Profile CRUD operations
src/core/project-config.test.ts: Project config discovery (upward search)
src/core/session.test.ts: Session management and cleanup
src/core/context.test.ts: Context resolution priority
src/core/credentials.test.ts: Multi-source credential resolution

Integration Tests

tests/context-integration.test.ts: End-to-end context flow
tests/profile-workflow.test.ts: Complete profile workflow
Mock filesystem for tests
Test all resolution priority combinations

Future Enhancements

OS Keychain Integration: Store credentials in native OS keychain
Shell Integration: tdx shell wrapper for better PID management
Auto-detection: Detect context from directory markers
Context Export: tdx use --json for scripting
Context History: Track context changes over time
Validation: tdx use validate to check configuration

Open Questions / Feedback

Should we support automatic "directory enter" hook to load project context (opt-in)?
Should we provide shell aliases for common use cases?
Should we add tdx use diff to compare contexts between environments?
Should profiles support inheritance (e.g., staging extends production)?

References

GitHub Issue: https://github.com/treasure-data/tdx/issues/78
JIRA Ticket: https://treasure-data.atlassian.net/browse/DEVAI-402
Related: Context management patterns from kubectl, terraform, docker contexts

Unified TDX Context Handling Design ​

Summary ​

Background / Problems Today ​

Proposed Context Model (Layered Resolution) ​

1. Profiles ​

Profile Configuration Example (tdx.json) ​

Profile Credentials Example (.env) ​

2. Project Config ​

3. PID-Based Session Context ​

4. Global Config ​

5. Environment Variables: Credentials Only ​

Configuration Schema ​

tdx.json Parameters (Explicit Naming) ​

.env File Format (Credentials Only) ​

Session + Project Interaction Example ​

New Commands ​

Use Command (Session Management) ​

Profile Commands ​

File Structure ​

Security Considerations ​

File Permissions ​

Credential Storage ​

Session Cleanup ​

Implementation Architecture ​

Core Modules ​

Utilities ​

Commands ​

Integration Points ​

Migration & Backward Compatibility ​

Phase 1: Non-Breaking Additions ​

Phase 2: Migrate Existing Users ​

Phase 3: Documentation ​

Testing Strategy ​

Unit Tests ​

Integration Tests ​

Future Enhancements ​

Open Questions / Feedback ​

References ​

Unified TDX Context Handling Design

Summary

Background / Problems Today

Proposed Context Model (Layered Resolution)

1. Profiles

Profile Configuration Example (tdx.json)

Profile Credentials Example (.env)

2. Project Config

3. PID-Based Session Context

4. Global Config

5. Environment Variables: Credentials Only

Configuration Schema

tdx.json Parameters (Explicit Naming)

.env File Format (Credentials Only)

Session + Project Interaction Example

New Commands

Use Command (Session Management)

Profile Commands

File Structure

Security Considerations

File Permissions

Credential Storage

Session Cleanup

Implementation Architecture

Core Modules

Utilities

Commands

Integration Points

Migration & Backward Compatibility

Phase 1: Non-Breaking Additions

Phase 2: Migrate Existing Users

Phase 3: Documentation

Testing Strategy

Unit Tests

Integration Tests

Future Enhancements

Open Questions / Feedback

References