tdx

Journey Round Trip Analysis ​

Executive Summary ​

This document analyzes the journey round trip logic in TDX CLI, explaining how data flows from API → YAML (pull) → API (push) and identifying potential issues, particularly around activations and local segments.

What is Journey Round Trip? ​

A "journey round trip" refers to the ability to:

  1. Pull journeys from Treasure AI CDP API to local YAML files
  2. Push those YAML files back to the API
  3. Maintain stability - pulling and immediately pushing should result in minimal/no changes

Key Challenge ​

Journeys contain embedded segments and embedded activations with auto-generated IDs that need to be:

  • Converted to human-readable names during pull
  • Matched back to server IDs during push
  • Kept stable across multiple pull/push cycles

Architecture Overview ​

Data Flow ​

PULL: API → YAML
┌─────────────────┐
│ Journey API     │
│ ├─ id: 123      │
│ ├─ entryCriteria│
│ │  └─ segmentId │──────────┐
│ │     "456"     │          │
│ └─ steps        │          │
│    └─ activation│          │
│       Id: "789" │──────┐   │
└─────────────────┘      │   │
         │               │   │
         │ Resolve IDs   │   │
         │ to names      │   │
         ▼               │   │
┌─────────────────┐      │   │
│ journey.yml     │      │   │
│ segments:       │◀─────┘   │
│   entry_72ee:   │          │
│     rule: ...   │          │
│ activations:    │◀─────────┘
│   to_td:        │
│     connection  │
└─────────────────┘

PUSH: YAML → API
┌─────────────────┐
│ journey.yml     │
│ segments:       │
│   entry_72ee:   │
│     rule: {...} │──────────┐
│ activations:    │          │
│   to_td: {...}  │──────┐   │
└─────────────────┘      │   │
         │               │   │
         │ Match by      │   │
         │ content       │   │
         ▼               │   │
┌─────────────────┐      │   │
│ Journey API     │      │   │
│ ├─ entryCriteria│◀─────┘   │
│ │  └─ segmentId │          │
│ │     "456"     │          │
│ └─ steps        │◀─────────┘
│    └─ activation│
│       Id: "789" │
└─────────────────┘

Core Components ​

1. Embedded Segments (Journey-Local Segments) ​

Characteristics:

  • Have kind=3 (SEGMENT_KIND_JOURNEY)
  • Auto-generated names: journey_entry_criteria_72ee903f-0644-42f5-a927-118442868213
  • Used for: entry criteria, decision branches, wait conditions, goals

Pull Logic (src/sdk/segment/pull-workflow.ts:385-387):

typescript
const finalYaml = buildMergedJourneyDef(versionDetails, existingRuleToName);
  • Detects journey-local segments via pattern matching
  • Generates clean names: entry_criteria_72ee (type + short hash)
  • Matches segments by rule content to preserve names across pulls
  • Embeds segment definitions in YAML segments section

Push Logic (src/sdk/segment/push-workflow.ts:1322-1386):

typescript
export function buildLocalExpandedRuleMappings(journey: JourneyDef)
  • Builds expanded rule mappings from local YAML
  • Compares server segment rules (expanded) with local rules
  • Matches by content hash (not name)
  • Creates new segments only if no match found

Matching Strategy:

  • Expanded rule comparison - Rules are recursively expanded to include referenced segment content
  • Normalization - YAML → API → YAML to ensure identical structure
  • Content hash - Deterministic hash of expanded rule for matching

2. Embedded Activations (Journey-Local Activations) ​

Characteristics:

  • Created via /entities/journeys/{journeyId}/activations
  • Referenced in activation steps within journey stages
  • Must be created AFTER journey exists (need journeyId)

Pull Logic (src/sdk/segment/pull-workflow.ts:546-580):

typescript
resolveActivation: async (activationId: string) => {
  const definition = journeyActivationStepToYaml(activationStep, ...);
  return { name: finalName, isJourneyLocal: true, definition };
}

Push Logic (src/sdk/segment/push-workflow.ts:2799-2841):

typescript
resolveEmbeddedActivationId: async (name: string, definition: ActivationDef) => {
  // Match by name first
  const byName = journeyActivationNameToId.get(definition.name);
  if (byName) return byName;

  // Match by signature (content)
  const signature = activationSignature(definition);
  const bySignature = activationSignatureToId?.get(signature);
  if (bySignature) return bySignature;

  // Create new activation
  const created = await deps.journey.createActivation(journeyId, createRequest);
}

Matching Strategy (src/sdk/segment/push-workflow.ts:227-241):

typescript
function activationSignature(definition: ActivationDef): string {
  const normalized = {
    connection: definition.connection,
    all_columns: definition.all_columns,
    columns: definition.columns,
    behavior: definition.behavior,
    schedule: definition.schedule,
    notification: definition.notification,
    connector_config: definition.connector_config,
    activation_template_id: definition.activation_template_id,
  };
  return stableStringify(normalized);
}
  • Signature matching - Ignores name field for rename stability
  • Content-based - Compares connection, config, columns, etc.

3. The 3-Step Journey Creation Workflow ​

Problem: New journeys with embedded resources have circular dependencies:

  • Entry criteria references segment IDs that don't exist yet
  • Segments need journeyId to be created
  • Activations need journeyId to be created

Solution: 3-step workflow (src/sdk/segment/push-workflow.ts:2994-3104)

Step 1: Create Journey with Placeholders ​

typescript
// Create journey without entry criteria
const createRequest = JSON.parse(JSON.stringify(apiRequest));
stripPendingIds(createRequest);  // Remove __pending_*__ placeholders
const createdJourney = await deps.journey.create(createRequest);
  • Removes all __pending_segment__ and __pending_act_*__ placeholders
  • Creates journey shell without entry criteria
  • Gets journeyId for step 2

Step 2: Create Embedded Resources ​

typescript
// Create embedded segments
for (const [name, { definition }] of pendingEmbeddedSegments) {
  const uniqueSegmentName = generateDeterministicJourneySegmentName(...);
  const segmentId = await deps.segment.createJourneySegment({
    name: uniqueSegmentName,
    journeyId,  // Now we have this!
    rule: yamlRuleToApi(resolvedRule, true),
  });
  embeddedSegmentNameToId.set(name, segmentId);
}

// Create embedded activations
for (const [name, definition] of pendingEmbeddedActivations) {
  const created = await deps.journey.createActivation(journeyId, createReq);
  embeddedActivationNameToId.set(name, created.id);
}
  • Creates segments with journeyId association
  • Creates activations with journeyId
  • Builds mapping from local names to real IDs

Step 3: Update Journey with Real IDs ​

typescript
resolvePendingIdsInRequest(
  updateRequest,
  apiRequest,
  stageNameToId,
  embeddedSegmentNameToId,
  embeddedActivationNameToId,
);
await deps.journey.update(journeyId, updateRequest);
  • Resolves all pending IDs to real IDs
  • Updates journey with entry criteria and activation references

Consolidation Logic (Multi-Version Journeys) ​

Problem ​

Journeys can have multiple versions (v1, v2, v3) with potentially identical segments/activations. Without consolidation, we'd get: to_td, to_td_2, to_td_3 (duplicates)

Solution (src/sdk/segment/journey-utils.ts:383-438) ​

Segment Consolidation ​

typescript
function consolidateSegmentsByContent(
  versionSegments: Array<Record<string, EmbeddedSegmentDef>>,
  existingRuleToName?: Map<string, string[]>,
): ConsolidatedSegments {
  for (const [name, segment] of Object.entries(segments)) {
    const contentKey = serializeRule(segment.rule);
    const existingName = contentKeyToName.get(contentKey);

    if (existingName) {
      // Map to existing
      nameMapping.set(name, existingName);
    } else {
      // Prefer name from existing file if available
      const canonicalName = existingRuleToName?.get(contentKey)?.[0] || name;
      merged[canonicalName] = segment;
      nameMapping.set(name, canonicalName);
    }
  }
}

Activation Consolidation ​

typescript
function consolidateActivationsByContent(
  versionActivations: Array<Record<string, ActivationDef>>,
): ConsolidatedActivations {
  const contentKey = JSON.stringify({
    name: activation.name,
    connection: activation.connection,
    connector_config: activation.connector_config,
  });

  if (contentKeyToName.has(contentKey)) {
    nameMapping.set(name, contentKeyToName.get(contentKey));
  } else {
    merged[name] = activation;
    contentKeyToName.set(contentKey, name);
  }
}

Reference Updates ​

After consolidation, all references must be updated:

  • entry_criteria.segment references
  • decision_point branch segment references
  • wait condition segment references
  • activation step references
  • goal.segment references
  • Segment-to-segment references (include/exclude)

Potential Issues ​

🔴 CRITICAL: Wait Condition Segments Not Resolved in 3-Step Workflow ​

Location: src/sdk/segment/push-workflow.ts:3144-3277

Problem: The resolvePendingIdsInRequest() function resolves pending IDs for:

  • ✅ Goal segments (line 3166-3178)
  • ✅ Entry criteria (line 3189-3197)
  • ✅ Exit criteria (line 3200-3212)
  • ✅ Milestones (line 3215-3223)
  • ✅ Decision point branches (line 3258-3273)
  • ✅ Activation steps (line 3237-3256)
  • ❌ Wait condition segments - MISSING!

Code:

typescript
function resolvePendingIdsInRequest(...) {
  // ... resolves goal, entry, exit, milestone, decision points, activations

  // MISSING: Wait condition resolution
  // if (step.type === 'WaitStep' && step.waitCondition?.segmentId) {
  //   const actualId = resolvePendingSegment(step.waitCondition.segmentId, ...);
  //   if (actualId) step.waitCondition.segmentId = actualId;
  // }
}

Impact:

  • New journeys with wait conditions referencing embedded segments will fail
  • The pending ID __pending_segment__ won't be resolved
  • API will reject the request or create invalid wait conditions

Evidence:

  • Wait conditions ARE stripped in stripPendingIds() (line 3110-3139) - but NOT EXPLICITLY, likely missed
  • Wait conditions ARE collected during segment reference collection (journey-yaml.ts:556-558, 624-626, 785-787, 880-882)
  • Wait conditions ARE converted during YAML→API (journey-yaml.ts:1710)

Fix Required: Add wait condition resolution in resolvePendingIdsInRequest():

typescript
// After decision point handling, before closing step loop
if (step.type === 'WaitStep' && step.waitCondition?.segmentId) {
  if (originalStep?.waitCondition?.segmentId &&
      isPendingSegmentId(originalStep.waitCondition.segmentId)) {
    const actualId = resolvePendingSegment(
      originalStep.waitCondition.segmentId,
      `stage '${stage.name}' wait condition`,
    );
    if (actualId) {
      step.waitCondition.segmentId = actualId;
    }
  }
}

🟡 MEDIUM: Activation Signature Doesn't Include name Field ​

Location: src/sdk/segment/push-workflow.ts:227-241

Problem: The activation signature intentionally excludes the name field for "rename stability", but this could cause issues:

typescript
function activationSignature(definition: ActivationDef): string {
  const normalized = {
    connection: definition.connection,
    // NOTE: Excludes definition.name!
    ...
  };
}

Scenario:

  1. User has activation: { name: "Send to TD", connection: "td-connection", ... }
  2. User renames it in YAML: { name: "Send to Treasure", connection: "td-connection", ... }
  3. Push workflow matches by signature → finds existing activation
  4. Result: Activation is NOT updated with new name

Is this intentional?

  • Pro: Prevents creating duplicate activations when user renames in YAML
  • Con: Actual activation name on server won't match YAML name
  • Verdict: This seems intentional based on comment "rename stability", but could be confusing

Recommendation:

  • Document this behavior clearly
  • OR: Include name in signature but add explicit rename detection

🟡 MEDIUM: Activation Consolidation Uses Different Keys Than Signature ​

Location:

  • Consolidation: src/sdk/segment/journey-utils.ts:116-120
  • Signature: src/sdk/segment/push-workflow.ts:227-241

Problem: Consolidation uses:

typescript
const contentKey = JSON.stringify({
  name: activation.name,        // INCLUDED
  connection: activation.connection,
  connector_config: activation.connector_config,
});

Signature uses:

typescript
const normalized = {
  connection: definition.connection,
  all_columns: definition.all_columns,     // INCLUDED
  columns: definition.columns,              // INCLUDED
  behavior: definition.behavior,            // INCLUDED
  schedule: definition.schedule,            // INCLUDED
  notification: definition.notification,    // INCLUDED
  connector_config: definition.connector_config,
  activation_template_id: definition.activation_template_id,  // INCLUDED
};

Impact:

  • Consolidation might merge activations that signature matching wouldn't
  • Consolidation includes name but signature doesn't
  • Consolidation only checks 3 fields, signature checks 8 fields

Example:

  • Activation A: { name: "to_td", connection: "td", columns: ["email"] }
  • Activation B: { name: "to_td", connection: "td", columns: ["user_id"] }
  • Consolidation → MERGES (only checks name, connection, connector_config)
  • Signature → DIFFERENT (checks columns field)

Recommendation:

  • Unify the matching logic
  • Use the same fields for both consolidation and signature matching

🟡 MEDIUM: Goal Segment Reference Only Updated in Top-Level ​

Location: src/sdk/segment/journey-utils.ts:416-423

Code:

typescript
// Update goal.segment reference if present
let goal = firstYaml.goal;
if (goal?.segment) {
  const consolidated = segmentNameMapping.get(goal.segment);
  if (consolidated) {
    goal = { ...goal, segment: consolidated };
  }
}

Problem: Goals can also be defined per-journey-version (not just top-level):

yaml
journeys:
  - version: v1
    goal:
      segment: entry_criteria_abc  # This goal is NOT updated!
  - version: v2
    goal:
      segment: entry_criteria_def

Evidence:

  • Journey entry can have goal field (types/journey-def.ts:JourneyEntryDef)
  • Goal references are collected from entries (journey-yaml.ts:569-571)
  • Goal references are replaced in entries (journey-yaml.ts:637-639)
  • But consolidation only updates top-level goal!

Impact:

  • Per-version goal segment references won't be consolidated
  • Could reference wrong segment name after consolidation

Fix Required:

typescript
// Update goal references in journey entries
for (const journey of journeys) {
  if (journey.goal?.segment) {
    const consolidated = segmentNameMapping.get(journey.goal.segment);
    if (consolidated) {
      journey.goal.segment = consolidated;
    }
  }
}

🟢 LOW: Segment Reference Resolution in Rules Could Miss Edge Cases ​

Location: src/sdk/segment/child-segment-yaml.ts:257-264, 324-330

Code:

typescript
// Pull: Only resolve if it looks like an ID (numeric)
if (/^\d+$/.test(ref.segment)) {
  const name = await resolveSegmentName(ref.segment);
  return { ...ref, segment: name };
}

// Push: Only resolve if it looks like a name (not numeric)
if (!/^\d+$/.test(ref.segment)) {
  const id = await resolveSegmentId(ref.segment);
  return { ...ref, segment: id };
}

Problem: What if a user names their segment with only numbers? "12345"

  • Pull: Would try to resolve as ID (might fail or resolve wrong segment)
  • Push: Wouldn't resolve (thinks it's already an ID)

Impact:

  • Low probability (users unlikely to use all-numeric names)
  • Would cause confusion if it happens

Recommendation:

  • Use a more robust detection method
  • Check context (embedded segment vs. reference)
  • Or: Always pass type hint to resolution functions

🟢 LOW: Stage Name Matching is Case-Insensitive But Could Be More Robust ​

Location: Multiple places use stage.name.toLowerCase() for matching

Examples:

  • src/sdk/segment/push-workflow.ts:3024
  • src/sdk/segment/push-workflow.ts:2857-2858

Code:

typescript
stageNameToId.set(stage.name.toLowerCase(), stage.id);

const stage = targetJourney.attributes.journeyStages.find(
  (s) => s.name.toLowerCase() === stageName.toLowerCase(),
);

Problem:

  • What if stage names differ in whitespace? "Stage One" vs "Stage One"
  • What if special characters? "Stage-1" vs "Stage 1"

Impact:

  • Very low (users unlikely to have such similar names)
  • Would cause stage resolution failures

Recommendation:

  • Normalize stage names (trim, collapse whitespace)
  • Or: Use more robust comparison (slugify)

Round-Trip Stability Mechanisms ​

✅ Content-Based Matching ​

  • Segments matched by expanded rule content (not names)
  • Prevents spurious changes when names differ

✅ Deterministic Naming ​

  • Local segment names preserved via existingRuleToName mapping
  • Short hash from UUID ensures uniqueness

✅ Consolidation ​

  • Duplicates across versions merged using content keys
  • Prevents to_td, to_td_2, to_td_3 proliferation

✅ Placeholder IDs ​

  • Pending resources marked with __pending_*__ prefix
  • Allows YAML→API conversion before resources exist

✅ Email/Connection Resolution ​

  • User IDs converted to emails for readability
  • Connection IDs converted to names

Testing Recommendations ​

Unit Tests Needed ​

  1. Wait Condition Resolution

    • Test: New journey with wait condition segment → 3-step workflow
    • Verify: Wait condition segmentId is resolved in step 3

  2. Activation Signature Consistency

    • Test: Same activation content → same signature
    • Test: Different columns → different signature

  3. Consolidation Logic

    • Test: Multiple versions with same segment → consolidated
    • Test: Multiple versions with same activation → consolidated
    • Test: Goal references updated in journey entries

  4. Edge Cases

    • Test: All-numeric segment names
    • Test: Stage names with special characters/whitespace
    • Test: Journey with all types of segment references (entry, decision, wait, goal)

Integration Tests Needed ​

  1. Full Round Trip

    • Pull journey → Modify nothing → Push
    • Verify: No changes detected

  2. Complex Journey

    • Journey with: entry criteria, decision points, wait conditions, activations, goal
    • Verify: All references resolved correctly

  3. Multi-Version Journey

    • Journey with 3 versions, duplicate segments
    • Verify: Segments consolidated, references updated

Key Files Reference ​

Core Logic ​

  • src/sdk/segment/pull-workflow.ts - Pull orchestration (lines 183-596)
  • src/sdk/segment/push-workflow.ts - Push orchestration (lines 2578-3277)
  • src/sdk/segment/journey-yaml.ts - API ↔ YAML conversion
  • src/sdk/segment/journey-utils.ts - Journey utilities and consolidation

Supporting ​

  • src/sdk/journey.ts - Journey SDK API
  • src/sdk/segment/child-segment-yaml.ts - Segment/activation conversion
  • src/sdk/segment/types/journey-def.ts - Type definitions

Documentation ​

  • docs/design-notes/push-workflow.md - Architecture overview

Summary ​

The journey round trip logic is sophisticated and handles many complex scenarios:

  • ✅ Embedded segments and activations
  • ✅ Multi-version journeys
  • ✅ Content-based matching
  • ✅ 3-step creation workflow
  • ✅ Consolidation of duplicates

Issues Fixed ​

✅ RESOLVED: Wait condition segments in 3-step workflow ​

  • Commit: 2ffe951
  • Added wait condition segment resolution in resolvePendingIdsInRequest()
  • Added wait condition stripping in stripPendingIds()

✅ RESOLVED: Goal references in journey entries ​

  • Commit: 2ffe951
  • Updated buildMergedJourneyDef() to consolidate per-version goal.segment refs

✅ RESOLVED: Activation signature vs consolidation mismatch ​

  • Commit: 6f83bf2
  • Unified activation matching using stableStringify() for consistent key ordering
  • Both consolidation and signature now use same 8 fields

✅ RESOLVED: Non-deterministic activation name suffixes ​

  • Commit: 0229da2
  • Removed _2 suffix logic from resolveActivation() in both pull and push
  • Let consolidation handle all deduplication deterministically

✅ RESOLVED: Rule normalization for stable condition ordering ​

  • Commit: 6f83bf2
  • Apply deepSortObject() to rules during consolidation
  • Ensures consistent condition order in stored YAML

Remaining Issues ​

🟡 TODO: Segment description mismatches ​

  • Different versions may have different descriptions
  • Need to handle description consolidation or selection

🟡 TODO: Segment matching edge cases ​

  • Some segments not matching correctly between versions
  • Example: excluded_10ca vs excluded_4587 duplication

🟢 LOW: Numeric segment name edge case ​

  • Segments named with only numbers might confuse ID vs name detection

🟢 LOW: Stage name matching robustness ​

  • Could normalize whitespace and special characters for better matching

Copyright 2024-present Treasure AI