Help Docs

Google Cloud BigQuery Data Transfer monitoring integration

Monitor your Google Cloud BigQuery Data Transfer resources to track active and completed transfer runs, monitor row counts processed, and observe run durations. Google Cloud BigQuery Data Transfer monitor ensures your scheduled data imports get completed reliably.

Benefits

Monitoring BigQuery Data Transfer ensures that your service remains reliable, secure, and high-performing. Key benefits include:

  • Identify stalled or failed transfer runs before they impact downstream reporting.
  • Track run durations and row counts to fine tune transfer schedules.
  • Automated alerts and IT automation help maintain SLA compliance and reduce downtime.
  • Consolidate BigQuery Data Transfer metrics with other GCP services for a single pane of glass view.
  • Bulk configuration and agent upgrades ensure smooth monitoring across multiple transfer configurations.

Use cases

Here are some use cases where Google Cloud BigQuery Data Transfer monitoring will be helpful:

  • Monitor active and completed transfer runs to detect stalled data imports in real time.
  • Capture run duration trends to quickly identify degrading transfer performance.
  • Visualize row counts processed and peak load periods to plan capacity and optimize scaling.
  • Set thresholds for run duration to ensure contractual obligations are met.
  • Track completed run counts to strengthen data freshness governance.
  • Integrate BigQuery Data Transfer monitoring with other GCP services (BigQuery, Cloud Storage, Data Studio) for holistic visibility.

Setup and configuration

  • Adding Google Cloud BigQuery Data Transfer while configuring a new Google Cloud monitor

    If you have not configured a Google Cloud monitor yet, add one by following the steps below:

    1. Go to Cloud > GCP > Add GCP Monitor or Admin > Cloud Monitoring > Google Cloud Platform(GCP).
    2. Provide a unique display name for identification purposes.
    3. Upload the JSON file that contains the private key of the service account to authenticate and perform resource discovery.
    4. Select BigQuery Data Transfer from the Select the Resources for Monitoring list.
    5. Select existing Notification Profiles, User Alerts Groups, Tags, and IT Automation Templates or add new ones. You can also integrate alarms with your preferred third-party service.
    6. Click Start GCP Monitoring.
  • Adding Google Cloud BigQuery Data Transfer to an existing Google Cloud monitor

    If you already have a Google Cloud monitor configured for the service account, you can add Google Cloud BigQuery Data Transfer by following the steps below:

    1. Go to Cloud > GCP and select your GCP monitor.
    2. Click the hamburger Hamburger icon icon next to Service View and select Edit, which brings you to the Edit GCP Monitor page.
    3. On the Edit GCP Monitor page, select BigQuery Data Transfer from the Select the Resources for Monitoring list and click Save.
    4. After successful configuration, go to Cloud > GCP > BigQuery Data Transfer. Now you can view the discovered Google Cloud BigQuery Data Transfer resources.
Note

It will take approximately five minutes to discover new GCP resources.

Licensing

Each Google Cloud BigQuery Data Transfer monitor consumes one basic monitor license.

Polling frequency

Google Cloud BigQuery Data Transfer monitor collects minute-wise metric data, and the statuses of your Google BigQuery Data Transfer resources are reported every five minutes.

Supported metrics

Metric nameDescriptionStatisticUnit
Active Runs The number of transfer runs currently in progress for a transfer configuration. Average Count
Completed Runs The number of transfer runs that have completed for a transfer configuration. Total Count
Last Execution Job Rows The number of rows processed by the most recent transfer run. Average Count
Run Duration The duration of transfer runs for a transfer configuration. Average Seconds

Threshold configuration

  • Global configuration
    1. In the web client, go to the Admin section on the left navigation pane.
    2. Select Configuration Profiles from the left pane and select Threshold and Availability from the drop-down menu.
    3. Click Add Threshold Profile in the top-right corner.
    4. For Monitor Type, select BigQuery Data Transfer.
    5. Specify the threshold values for the required metrics, then click Save.
  • Monitor-level configuration
    1. In the web client, go to Cloud > GCP > BigQuery Data Transfer.
    2. Select a resource you would like to set a threshold for, then click the hamburger Hamburger icon icon.
    3. Select Edit, which directs you to the Edit BigQuery Data Transfer Monitor page.
    4. Specify the threshold values for the required metrics, then click Save.

IT Automation

IT Automation tools automatically resolve performance degradation issues. These tools react to events proactively, rather than waiting for manual intervention. The IT Automation tools automate repetitive tasks and automatically remediate threshold breaches. The alarm engine continually evaluates system events for which thresholds are set and executes the mapped automation when there is a breach.

How to configure IT automation for a monitor

Configuration Rules

Editing multiple monitors to associate different monitor groups or add a different tag can be a tedious process. With Configuration Rules, you can automate the configuration settings of your monitoring resources. Create custom rules to track configuration changes continuously and achieve the ideal configuration settings.

How to add Configuration Rules

Summary

The Summary tab will give you the performance data organized by time for the metrics listed above. To view the summary:

  1. Go to Cloud > GCP > BigQuery Data Transfer.
  2. Select a resource.
  3. Click the Summary tab.

Configuration Details

The Configuration Details tab provides details on the configurations of application instances. To get the configuration details:

  1. Go to Cloud > GCP > BigQuery Data Transfer.
  2. Select a resource.
  3. Click the Configuration Details tab.

Reports

Gain in-depth data about the various parameters of your monitored resources and accentuate your service performance using our insightful reports.

To view reports for a Google BigQuery Data Transfer resource:

  1. Go to the Reports section on the left navigation pane.
  2. Select BigQuery Data Transfer from the menu on the left.
  3. You can find the Availability Summary Report, Performance Report, and Inventory Report for one selected monitor. Or you can get the Summary Report, Availability Summary Report, Health Trend Report, and Performance Report for all the Google BigQuery Data Transfer monitors.

You can also get reports from the Summary tab of the Google BigQuery Data Transfer monitor:

  1. Click the Summary tab.
  2. Get the Availability Summary Report of the monitor by clicking Availability.
  3. You can also find the Performance Report of the monitor by clicking any chart title.

Related content

Was this document helpful?

Would you like to help us improve our documents? Tell us what you think we could do better.


We're sorry to hear that you're not satisfied with the document. We'd love to learn what we could do to improve the experience.


Thanks for taking the time to share your feedback. We'll use your feedback to improve our online help resources.

Shortlink has been copied!