Contents

Cancel

Recommended Articles

  1. unify-apps

    Indexing

    Unify AI

    Transform raw content into searchable knowledge through AI-powered indexing and vector embeddings

  2. unify-apps

    Snowflake as Destination

    Unify Data

    Load and transform your data seamlessly into Snowflake's cloud data warehouse with UnifyApps' native connectivity and flexible staging options.

  3. unify-apps

    Quentn

    Unify Integrations

    Integrate your app with Quentn to automate marketing campaigns, manage contacts, and optimize customer engagement.

  4. unify-apps

    Connector SDK

    Platform Tools

    Create custom connectors within the Unify platform to unify workflows

  5. unify-apps

    JobNimbus

    Unify Integrations

    Integrate your application with JobNimbus to manage leads, track jobs, and streamline your workflow processes efficiently

  6. unify-apps

    IMAP

    Unify Integrations

    Integrate your app with IMAP to enable seamless email synchronization, real-time access, and enhanced communication workflows.

  7. unify-apps

    Basin

    Unify Integrations

    Basin transforms form handling from a backend headache into a simple integration—capture submissions, trigger automations, and manage data flows without touching server code, giving you more time to build what matters

  8. unify-apps

    Preview Your Work

    Unify Automations

    Effortlessly review & monitor your automation’s performance

  9. unify-apps

    QuickBooks

    Unify Integrations

    Integrate your app with QuickBooks to streamline accounting, automate invoicing, and manage finances effortlessly

  10. unify-apps

    FTP/FTPS

    Unify Integrations

    Connect your app with FTP/FTPS to automate secure file transfers and streamline data exchange across systems.

  11. unify-apps

    Data-Sync by Avoid Duplicate Operations Setting

    Unify Data

    Prevent infinite loops in bidirectional data synchronization by creating record hashes that ensure one-way data flow across connected systems.

  12. unify-apps

    Filters

    Unify Applications

    Enable users to refine, search, and sort data effortlessly across dashboards and datasets

  13. unify-apps

    Insided

    Unify Integrations

    Integrate your app with Insided to enhance customer engagement, streamline community management, and drive self-service support.

  14. unify-apps

    Reverse Polling

    Unify Data

    Reverse Polling technique efficiently retrieves recent data from APIs that return results in chronological order (oldest first), optimizing pagination and data processing strategies when working with time-ordered data sources.

  15. unify-apps

    Facebook Ads

    Unify Integrations

    Connect your app with Facebook Ads to automate campaign management, optimize ad performance, and track marketing success.

  16. unify-apps

    Duplicate Field

    Unify Integrations

    Create independent copies of your data fields to enable multiple mappings while preserving original values for validation and complex workflows.

  17. unify-apps

    Gainsight

    Unify Integrations

    Integrate your app with Gainsight to enhance customer success, automate engagement workflows, and drive retention

  18. unify-apps

    Simplesat

    Unify Integrations

    Integrate your app with Simplesat to collect real-time customer feedback, measure satisfaction, and improve service quality.

  19. unify-apps

    Livestorm

    Unify Integrations

    Integrate your app with Livestorm to streamline webinar hosting, automate event management, and enhance audience engagement.

  20. unify-apps

    Application Connectors

    Unify Data

    Instantly leverage 30+ pre-built application connectors to extract, transform, and load your business-critical data between systems with UnifyApps' no-code integration platform.

    Logo
    Pipeline Deployment
    Logo
    Pipeline Logs
Unify Data
Logo
Overview
Logo
Salesforce as Source Polling Tuning

Salesforce as Source Polling Tuning

Logo

3 mins READ

Polling settings in UnifyApps data pipelines allow you to optimize how frequently your pipeline checks for and processes new or changed data from Salesforce. Proper tuning of these settings ensures efficient data synchronization while minimizing API usage and system resource consumption.

Understanding Salesforce Source Polling

When configuring Salesforce as a source in your data pipeline, the polling mechanism:

  • Periodically queries Salesforce for new or modified records

  • Processes these records in controlled batches

  • Synchronizes the data to your destination system

  • Manages API usage within Salesforce's rate limits

Configuring Polling Settings for Salesforce

Image
Image

To configure polling settings for your Salesforce source:

  1. Navigate to the Settings tab in your pipeline

  2. Locate the Polling Settings section

  3. Configure the following parameters:

    • Frequency: The time unit for polling intervals (seconds, minutes, hours)

    • Interval: The number of units between polling operations

    • Batch Size: The maximum number of records processed in each polling cycle

How Polling Settings Impact Performance: Example

Let's examine how different polling configurations affect a Salesforce to PostgreSQL pipeline:

Example: Banking CRM Data Synchronization

Scenario: Customer Data Integration

  • Salesforce contains customer account information

  • Updates occur throughout the business day at varying rates

  • Pipeline synchronizes data to a banking database

Configuration Options and Impacts:

High-Frequency Polling (1 minute interval, 100 batch size)

Benefits:

  • Near real-time data availability

  • Smaller, more frequent batches reduce processing spikes

Considerations:

  • Higher Salesforce API consumption

  • More frequent connection overhead

  • May exceed API limits during peak times

Medium-Frequency Polling (15 minute interval, 250 batch size)

Benefits:

  • Balanced approach for most use cases

  • Reasonable data latency

  • Moderate API consumption

Considerations:

  • Some data synchronization delay acceptable

  • Good for steady, moderate update volumes

Low-Frequency Polling (60 minute interval, 1000 batch size)

Benefits:

  • Minimizes API consumption

  • Fewer connection cycles

  • Efficiently processes larger batches

Considerations:

  • Longer data latency

  • Potential for larger processing spikes

  • Better for systems with infrequent updates

Optimizing Salesforce Polling Settings

  1. Analyze Update Patterns Monitor when and how frequently your Salesforce data changes:

    Morning: High update volume during business start

    Midday: Moderate, steady updates

    Evening: Low update activity

    Adjust polling frequency accordingly, with more frequent polling during high-activity periods.

  2. Consider Salesforce API Limits Salesforce enforces API request limits per 24-hour period:

    • Enterprise Edition: 1,000,000 calls

    • Professional Edition: 25,000 calls

    • Developer Edition: 15,000 calls

      Calculate your polling impact:

      (24 hours / polling interval) × number of objects synced = daily API calls

  3. Balance Latency vs. Resource Usage For different data types:

    Customer contact updates: Higher frequency (5-15 minutes)

    Transaction records: Medium frequency (15-30 minutes)

    Reference data: Lower frequency (hourly or daily)

  4. Batch Size Considerations Optimize batch size based on:

    Small batches (50-100): For complex records with many fields

    Medium batches (100-500): For standard business objects

    Large batches (500-1000+): For simple, high-volume data

Best Practices for Salesforce Polling

  1. Stagger Multiple Pipeline Schedules If running multiple Salesforce pipelines, offset their polling schedules to avoid simultaneous API load:

    Pipeline 1: Start at :00 minutes

    Pipeline 2: Start at :15 minutes

    Pipeline 3: Start at :30 minutes

    Pipeline 4: Start at :45 minutes

  2. Implement Off-Hours Processing for Large Volumes Schedule intensive historical loads during off-peak hours:

    Initial data loads: Weekends or overnight

    Large batch synchronizations: Early morning (2-5 AM)

  3. Monitor and Adjust Regularly review pipeline performance metrics:

    • Processing time per batch

    • API usage trends

    • Error rates at different polling frequencies

    • Data latency requirements vs. actual performance

By carefully tuning your polling settings for Salesforce source pipelines, you can achieve an optimal balance between data freshness, system performance, and API efficiency. Regular monitoring and adjustment of these settings ensure your data integration remains robust as your business needs evolve.