Contents

Cancel

Recommended Articles

  1. unify-apps

    Indexing

    Unify AI

    Transform raw content into searchable knowledge through AI-powered indexing and vector embeddings

  2. unify-apps

    IMAP

    Unify Integrations

    Integrate your app with IMAP to enable seamless email synchronization, real-time access, and enhanced communication workflows.

  3. unify-apps

    Preview Your Work

    Unify Automations

    Effortlessly review & monitor your automation’s performance

  4. unify-apps

    QuickBooks

    Unify Integrations

    Integrate your app with QuickBooks to streamline accounting, automate invoicing, and manage finances effortlessly

  5. unify-apps

    FTP/FTPS

    Unify Integrations

    Connect your app with FTP/FTPS to automate secure file transfers and streamline data exchange across systems.

  6. unify-apps

    Facebook Ads

    Unify Integrations

    Connect your app with Facebook Ads to automate campaign management, optimize ad performance, and track marketing success.

  7. unify-apps

    Snowflake

    Unify Automations

    Connect to Snowflake for fast, scalable cloud data warehousing and analytics

  8. unify-apps

    Button

    Unify Applications

    Create interactive elements with ease using buttons

  9. unify-apps

    BambooHR

    Unify Automations

    Integrate your app with BambooHR to streamline HR management, automate employee data processing, and enhance onboarding workflows

  10. unify-apps

    Build your first automation

    Unify Automations

    Learn how to build your automation step by step

  11. unify-apps

    User Management

    Unify Applications

    Efficiently manage user roles and permissions

  12. unify-apps

    Microsoft Teams

    Unify Automations

    Connect your app with Microsoft Teams to enhance communication, automate workflows, and foster collaboration across your organization

  13. unify-apps

    Jira

    Unify Automations

    Use Jira to plan, track, and manage your agile and software development projects

  14. unify-apps

    Pre Processing

    Unify AI

    Optimize query processing through intelligent rephrasing, retrieval, and ranking to deliver accurate AI responses

  15. unify-apps

    Google Calendar

    Unify Integrations

    Integrate your app with Google Calendar to streamline scheduling, automate event management, and improve team coordination

  16. unify-apps

    SAP HANA

    Unify Integrations

    Connect your app with SAP HANA to leverage in-memory computing for real-time data processing and advanced analytics at scale.

#
Unify Data
Logo
Create your first Data Pipeline
Logo
Settings & Deployment

Settings & Deployment

Logo

4 min READ

After completing the object selection and schema mapping, the final steps in creating your data pipeline are configuring the pipeline settings and deploying it.
These steps are crucial for determining how your pipeline will operate and making it live.

Configure Pipeline Settings

Navigate to the "Settings" tab in the UnifyData interface. Here, you'll find several important options to configure:

  1. Auto Retry

    • When enabled, UnifyData will automatically retry processing any records that fail during the pipeline execution. Enable this for more robust pipelines, especially when dealing with unstable data sources or network connections.

  2. Data Ingestion Mode
    This setting determines what data will be processed by your pipeline.

    • Historic and Live Data

      Processes all existing data and continues to process new data.Best for initial data migrations or when you need a full data sync.

    • Live Data Only

      Only processes new data from the moment the pipeline is activated.Useful for ongoing data synchronization after an initial data load.

    • Historic Data Only

      Processes only existing data up to the point of pipeline activation.Useful for one-time data migrations or creating point-in-time snapshots.

      Image
      Image
  3. Load Schedule
    This setting defines how often your pipeline will run.

    • Real Time
      Continuously processes data as it becomes available. Best for use cases requiring up-to-the-minute data accuracy.

    • Interval
      Allows you to set a specific frequency for the pipeline to run (e.g., hourly, daily, weekly). Useful for batch processing or when real-time data is not necessary.

Deployment

This is the process of making your configured pipeline live and operational.Once you've configured your settings, you're ready to deploy your pipeline.

  1. Review all your configurations. Check your source and destination connections, object selections, schema mappings, and settings.

  2. Click the "Deploy" button, located in the top right corner of the screen.

  3. It  will perform a final validation of your pipeline configuration. If any issues are detected, you'll be notified and you can make corrections.

  4. Once deployed, you'll see an ON/OFF toggle appear, usually at the top left of the screen. This toggle allows you to activate or pause your pipeline as needed.

Monitor Your Deployed Pipeline

After deployment and activation, it's crucial to monitor your pipeline's performance:

  1. Check the pipeline logs regularly for any errors or warnings.

  2. Monitor the data in your destination to ensure it's being populated as expected.

  3. Keep monitoring system resources, especially for real-time or frequently running pipelines.