Google Cloud TPU monitoring integration
Monitor your Google Cloud TPU nodes to track Matrix Multiply Unit (MXU) utilization, TensorCore idle time, host CPU and memory usage, and network throughput. Google Cloud TPU monitor lets you ensure your machine learning training and inference workloads are using accelerator capacity efficiently.
Benefits
Monitoring Cloud TPU ensures that your service remains reliable, secure, and high-performing. Key benefits include:
- Identify underutilized TPU capacity before it wastes training budget.
- Track host CPU and memory usage to fine-tune the data pipeline feeding the TPU.
- Automated alerts and IT automation help maintain SLA compliance and reduce downtime.
- Consolidate Cloud TPU metrics with other GCP services for a single pane of glass view.
- Bulk configuration and agent upgrades ensure smooth monitoring across multiple TPU nodes.
Use cases
Here are some use cases where Google Cloud TPU monitoring will be helpful:
- Monitor MXU utilization and TensorCore idle duration to detect underutilized accelerators in real time.
- Capture host CPU and memory usage trends to quickly identify input-pipeline bottlenecks.
- Visualize network throughput to plan capacity and optimize data loading.
- Set thresholds for MXU utilization to ensure contractual obligations are met.
- Track node health and state to strengthen training job reliability.
- Integrate Cloud TPU monitoring with other GCP services (Vertex AI, Compute Engine, Cloud Storage) for holistic visibility.
Setup and configuration
- Adding Google Cloud TPU while configuring a new Google Cloud monitor
If you have not configured a Google Cloud monitor yet, add one by following the steps below:
- Go to Cloud > GCP > Add GCP Monitor or Admin > Cloud Monitoring > Google Cloud Platform (GCP).
- Provide a unique display name for identification purposes.
- Upload the JSON file that contains the private key of the service account to authenticate and perform resource discovery.
- Select Cloud TPU from the Select the Resources for Monitoring list.
- Select existing Notification Profiles, User Alerts Groups, Tags, and IT Automation Templates or add new ones. You can also integrate alarms with your preferred third-party service.
- Click Start GCP Monitoring.
- Adding Google Cloud TPU to an existing Google Cloud monitor
If you already have a Google Cloud monitor configured for the service account, you can add Google Cloud TPU by following the steps below:
- Go to Cloud > GCP and select your GCP monitor.
- Click the hamburger
icon next to Service View and select Edit, which brings you to the Edit GCP Monitor page. - On the Edit GCP Monitor page, select Cloud TPU from the Select the Resources for Monitoring list and click Save.
- After successful configuration, go to Cloud > GCP > Cloud TPU. Now you can view the discovered Google Cloud TPU resources.
It will take approximately five minutes to discover new GCP resources.
Licensing
Each Google Cloud TPU monitor consumes one basic monitor license.
Polling frequency
Google Cloud TPU monitor collects minute-wise metric data, and the statuses of your Google Cloud TPU resources are reported every five minutes. TPU metrics are collected with a 3-minute delay.
Supported metrics
| Metric name | Description | Statistic | Unit |
|---|---|---|---|
| MXU Utilization | The utilization of the TPU's Matrix Multiply Unit (MXU), the core compute engine used for machine learning workloads. | Average | Percentage (%) |
| TensorCore Idle Duration | The longest time, in seconds, that any TensorCore on the node has been continuously idle. A value that keeps growing means the TPU isn’t receiving work. | Maximum | Seconds |
| Host CPU Utilization | The CPU utilization of the host machine attached to the TPU. | Average | Percentage (%) |
| Host Memory Usage | The amount of memory used on the host machine attached to the TPU. | Average | Bytes |
| Received Bytes per Second | The rate of inbound network traffic to the TPU host. | Total | Bytes/second |
| Sent Bytes per Second | The rate of outbound network traffic from the TPU host. | Total | Bytes/second |
Configuration details
| Attribute | Description |
|---|---|
| Resource Name | The display name of the TPU node. |
| Node ID | The unique identifier of the TPU node. |
| State | The current operational state of the TPU node (for example, READY or CREATING). |
| Health | The current health status reported for the TPU node. |
| Accelerator Type | The TPU accelerator type and topology configured for this node. |
| TPU Type | The generation or type of TPU hardware used by this node. |
| Topology | The physical topology of TPU chips allocated to this node. |
| Runtime Version | The TPU software runtime version running on this node. |
| API Version | The TPU API version used to create this node. |
| Spot VM | Indicates whether this node is a Spot (preemptive) TPU. |
| Network | The VPC network this TPU node is attached to. |
| Sub network | The sub network this TPU node is attached to. |
| External IPs Enabled | Indicates whether this TPU node has external IP addresses enabled. |
| Created Time | The date and time this TPU node was created. |
Health Symptoms
This tab lists the problems Google Cloud has detected on the TPU node, such as out-of-memory errors, unexpected reboots or hardware issues. The details shown here are Symptom, Worker, Details and Creation date. The data is read live from Google Cloud each time you open the tab. The tab is empty when the node is healthy.
Threshold configuration
- Global configuration
- In the web client, go to the Admin section on the left navigation pane.
- Select Configuration Profiles from the left pane and select Threshold and Availability from the drop-down menu.
- Click Add Threshold Profile in the top-right corner.
- For Monitor Type, select Cloud TPU.
- Specify the threshold values for the required metrics, then click Save.
- Monitor-level configuration
- In the web client, go to Cloud > GCP > Cloud TPU.
- Select a resource you would like to set a threshold for, then click the hamburger icon.
- Select Edit, which directs you to the Edit Cloud TPU Monitor page.
- Specify the threshold values for the required metrics, then click Save.
TPU health can turn the monitor Down even while the node is running:
- Up: READY, CREATING, STARTING, RESTARTING, REIMAGING, or UNHIDING.
- Down: STOPPED, STOPPING, PREEMPTED, TERMINATED, REPAIRING, DELETING, or HIDING.
Monitor status shifts to Down when health is bad if the Health field is TIMEOUT or UNHEALTHY, even if the state is READY. The reason shown is Google’s health description.
IT Automation
IT Automation tools automatically resolve performance degradation issues. These tools react to events proactively rather than waiting for manual intervention. The IT Automation tools automate repetitive tasks and automatically remediate threshold breaches. The alarm engine continually evaluates system events for which thresholds are set and executes the mapped automation when there is a breach.
Configuration Rules
Editing multiple monitors to associate different monitor groups or add a different tag can be a tedious process. With Configuration Rules, you can automate the configuration settings of your monitoring resources. Create custom rules to track configuration changes continuously and achieve the ideal configuration settings.
Summary
The Summary tab will give you the performance data organized by time for the metrics listed above. To view the summary:
- Go to Cloud > GCP > Cloud TPU.
- Select a resource.
- Click the Summary tab.
Configuration Details
The Configuration Details tab provides details on the configurations of application instances. To get the configuration details:
- Go to Cloud > GCP > Cloud TPU.
- Select a resource.
- Click the Configuration Details tab.
Reports
Gain in-depth data about the various parameters of your monitored resources and accentuate your service performance using our insightful reports.
To view reports for a Google Cloud TPU resource:
- Go to the Reports section on the left navigation pane.
- Select Cloud TPU from the menu on the left.
- You can find the Availability Summary Report, Performance Report, and Inventory Report for one selected monitor. Or you can get the Summary Report, Availability Summary Report, Health Trend Report, and Performance Report for all the Google Cloud TPU monitors.
You can also get reports from the Summary tab of the Google Cloud TPU monitor:
- Click the Summary tab.
- Get the Availability Summary Report of the monitor by clicking Availability.
- You can also find the Performance Report of the monitor by clicking any chart title.