Manual Lineage Management

Overview

Manual lineage allows you to explicitly define upstream and downstream relationships between individual assets using their unique identifiers. This approach is ideal for documenting custom business logic relationships, cross-system connections, and exceptional cases that aren't captured by automated scanning.

Understanding Manual Lineage

What is Manual Lineage?

Manual lineage enables you to define explicit, individual relationships between specific assets when automated discovery isn't sufficient or when you need to document business logic relationships. This provides complete control over lineage definition for complex scenarios.

When to Use Manual Lineage

Business Logic Relationships: - Calculated tags that derive values from multiple sources - Virtual assets that represent business concepts rather than physical infrastructure - Aggregated metrics that combine data from various systems - Custom transformations implemented in external applications

Cross-Domain Connections: - Data flows between different organizational units - Connections across security boundaries - Integration points with third-party systems - Manual data transfer processes

Complex Configurations: - Non-standard interface implementations - Custom data collection scripts - Specialized calculation engines - Unique networking configurations

Legacy Systems: - Document relationships in older systems without modern lineage capabilities - Bridge gaps in automated discovery - Maintain historical data flow documentation

How to Create Manual Lineage Relationships

Step 1: Access Lineage Management

Navigate to Lineage Management from the main navigation menu
Select the Manual Lineage tab
Review the section description to understand the feature

Step 2: Identify Asset Unique IDs

Before creating relationships, you need the unique identifiers for both assets:

Finding Unique IDs: 1. Navigate to the Explorer page 2. Search for or browse to your target assets 3. Click on an asset to view its details 4. Copy the Unique ID from the asset information panel 5. Repeat for both upstream and downstream assets

Unique ID Format Examples: - PI Data Archive tags: \\PIServers[ServerName]\PointName - AF Elements: \\AFServers[ServerName]\DatabaseName\ElementPath - AF Attributes: \\AFServers[ServerName]\DatabaseName\ElementPath|AttributeName

Step 3: Define the Relationship

Enter Upstream Asset: Paste or type the unique ID of the data source asset
Enter Downstream Asset: Paste or type the unique ID of the data consumer asset
Verify Information: Double-check both IDs for accuracy
Submit: Click "Add Lineage Relationship" to create the connection

Step 4: Verify Creation

The new relationship will appear in the "Existing Manual Relationships" table
Verify that both asset IDs are correct
Check that the relationship direction (upstream → downstream) is accurate

Managing Manual Lineage Relationships

Viewing Existing Relationships

The manual lineage table displays: - Upstream Asset: The source of the data - Downstream Asset: The consumer of the data - Actions: Delete option for each relationship

Deleting Relationships

Locate the relationship in the table
Click the delete (trash) icon in the Actions column
Confirm the deletion when prompted
The relationship will be immediately removed

Best Practices for Manual Lineage

Documentation Standards

Maintain Consistency: Use a consistent naming convention for asset IDs
Document Business Reasons: Keep records of why each manual relationship was created
Track Changes: Maintain a record of who created each relationship and when
Regular Reviews: Establish a schedule for reviewing and validating manual relationships

Accuracy Guidelines

Verify Asset IDs: Always verify asset IDs before submitting relationships
Test Relationships: Ensure the relationship accurately represents actual data flow
Use External Documentation: Maintain detailed explanations for complex relationships
Team Coordination: Work with team members to avoid duplicate or conflicting relationships

Maintenance Procedures

Change Management: Review manual relationships when system configurations change
Asset Lifecycle: Update or remove relationships when assets are decommissioned
System Updates: Validate relationships after system upgrades or migrations
Audit Schedule: Establish regular audits for manual lineage accuracy

Common Use Cases

Business Logic Documentation

Calculated Tags: - Document how derived metrics are calculated from source data - Show dependencies between calculated values and their inputs - Track complex formulas that span multiple systems

Virtual Assets: - Define relationships for business concepts that don't have physical counterparts - Connect logical groupings to their constituent elements - Document organizational hierarchies reflected in data

Cross-System Integration

Third-Party Systems: - Document data flows to and from external applications - Track integration points with vendor systems - Maintain lineage across security boundaries

Manual Processes: - Document human-mediated data transfers - Track batch processing and scheduled data movements - Maintain lineage for exception handling processes

Complex Infrastructure

Custom Interfaces: - Document non-standard data collection methods - Track custom scripts and automation - Maintain lineage for specialized hardware connections

Legacy Systems: - Bridge gaps in automated discovery capabilities - Document historical data flows - Maintain continuity during system modernization

Troubleshooting Manual Lineage

Common Issues

Invalid Asset IDs: - Problem: System cannot find the specified asset - Solution: Verify the unique ID format and ensure the asset exists in the system - Prevention: Copy IDs directly from asset detail pages

Wrong Direction: - Problem: Upstream and downstream are reversed - Solution: Delete the incorrect relationship and recreate with proper direction - Prevention: Carefully consider data flow direction before creating relationships

Duplicate Relationships: - Problem: Multiple relationships exist for the same asset pair - Solution: Remove duplicate entries, keeping only the most accurate - Prevention: Check existing relationships before creating new ones

Performance Considerations

Large Number of Relationships: - Monitor system performance with extensive manual lineage data - Consider using PI Interface lineage for repetitive patterns - Optimize by grouping related assets when possible

Complex Queries: - Limit lineage depth when exploring relationships - Use filtering to focus on specific asset types or domains - Consider performance impact when creating deep relationship chains

Integration with Other Features

Explorer Page Enhancement

Related Assets: Lineage relationships enhance asset browsing and discovery
Context Understanding: Data flow context improves asset management decisions
Navigation: Easy movement between related assets

Issue Troubleshooting

Root Cause Analysis: Trace data quality problems to their source using lineage
Impact Assessment: Understand which assets are affected by upstream issues
Resolution Planning: Use lineage to plan comprehensive fixes

Reporting and Analytics

Compliance Reports: Include lineage information in regulatory documentation
System Analysis: Analyze data flow patterns for optimization opportunities
Business Intelligence: Support analytics initiatives with comprehensive data provenance