Google Cloud Composer monitoring integration
Monitor your Google Cloud Composer environments to track scheduler health, DAG parsing performance, task queue backlogs, worker evictions, and the health of the underlying Airflow metadata database. Google Cloud Composer monitor lets you keep your managed Apache Airflow workflows running reliably.
Benefits
Monitoring Cloud Composer ensures that your service remains reliable, secure, and high-performing. Key benefits include:
- Identify scheduler or database health issues before they stall your entire pipeline.
- Track DAG parse times and task queue backlogs to fine-tune environment sizing.
- Automated alerts and IT automation help maintain SLA compliance and reduce downtime.
- Consolidate Cloud Composer metrics with other GCP services for a single pane of glass view.
- Bulk configuration and agent upgrades ensure smooth monitoring across multiple environments.
Use cases
Here are some use cases where Google Cloud Composer monitoring will be helpful:
- Monitor environment and database health to detect failures in real time.
- Capture failed task and zombie task counts to quickly identify unreliable DAGs.
- Visualize task queue length and worker pod evictions to plan capacity and optimize scaling.
- Set thresholds for DAG parse time and parse errors to ensure contractual obligations are met.
- Track scheduler heartbeat rate to strengthen pipeline reliability.
- Integrate Cloud Composer monitoring with other GCP services (BigQuery, Dataflow, Cloud Storage) for holistic visibility.
Setup and configuration
- Adding Google Cloud Composer while configuring a new Google Cloud monitor
If you have not configured a Google Cloud monitor yet, add one by following the steps below:
- Go to Cloud > GCP > Add GCP Monitor or Admin > Cloud Monitoring > Google Cloud Platform (GCP).
- Provide a unique display name for identification purposes.
- Upload the JSON file that contains the private key of the service account to authenticate and perform resource discovery.
- Select Cloud Composer from the Select the Resources for Monitoring list.
- Select existing Notification Profiles, User Alerts Groups, Tags, and IT Automation Templates or add new ones. You can also integrate alarms with your preferred third-party service.
- Click Start GCP Monitoring.
- Adding Google Cloud Composer to an existing Google Cloud monitor
If you already have a Google Cloud monitor configured for the service account, you can add Google Cloud Composer by following the steps below:
- Go to Cloud > GCP and select your GCP monitor.
- Click the hamburger
icon next to Service View and select Edit, which brings you to the Edit GCP Monitor page. - On the Edit GCP Monitor page, select Cloud Composer from the Select the Resources for Monitoring list and click Save.
- After successful configuration, go to Cloud > GCP > Cloud Composer. Now you can view the discovered Google Cloud Composer resources.
It will take approximately five minutes to discover new GCP resources.
Licensing
Each Google Cloud Composer monitor consumes one basic monitor license.
Polling frequency
Google Cloud Composer metrics are collected with a three-minute delay, so the latest data is about 3 minutes old. Environment Health uses a five-minute window.
Supported metrics
| Metric name | Description | Statistic | Unit |
|---|---|---|---|
| Environment Health | Indicates whether the Cloud Composer environment is reporting as healthy. | Average | Percentage (%) |
| Database Health | Indicates whether the environment's Airflow metadata database is reporting as healthy. | Average | Percentage (%) |
| DAG Bag Size | The number of DAGs currently loaded by the Airflow scheduler. | Average | Count |
| Scheduler Heartbeats per Second | The rate of heartbeats emitted by the Airflow scheduler, used to confirm it is alive. | Total | Count/second |
| Task Queue Length | The number of tasks currently queued and waiting to be picked up by a worker. | Average | Count |
| Finished Tasks per second | The rate at which task instances are completed. | Total | Count/second |
| Database Disk Usage | The amount of disk space used by the environment's Airflow metadata database. | Average | Bytes |
| Web Server Health | Indicates whether the Airflow web server is reporting as healthy. | Average | Percentage (%) |
| Worker Pod Evictions per Second | The rate at which Airflow worker pods are evicted, often due to resource pressure. | Total | Count/second |
| Failed Tasks | The number of task instances that finished in a failed state. | Total | Count |
| Unfinished Task Instances | The number of task instances that have not yet reached a terminal state. | Average | Count |
| DAG Parse Errors | The number of errors encountered while parsing DAG files. | Total | Count |
| DAG Parse Time | The total time taken to parse all DAG files. | Average | Seconds |
| Database CPU Utilization | The CPU utilization of the environment's Airflow metadata database. | Average | Percentage (%) |
| Database Memory Utilization | The memory utilization of the environment's Airflow metadata database. | Average | Percentage (%) |
| Database Disk Utilization | The disk utilization of the environment's Airflow metadata database. | Average | Percentage (%) |
| Celery Workers | The number of Celery worker processes currently running in the environment. | Average | Count |
| Scheduler Pod Evictions | The number of times the Airflow scheduler pod was evicted. | Total | Count |
| Zombie Tasks Killed | The number of zombie tasks (tasks whose worker process died unexpectedly) killed by the scheduler. | Total | Count |
Configuration details
| Attribute | Description |
|---|---|
| Resource Name | The display name of the Cloud Composer environment. |
| State | The current state of the environment (for example, RUNNING or UPDATING). |
| Image Version | The Cloud Composer and Airflow image version running in this environment. |
| Environment Size | The size tier of the environment (for example, Small, Medium, or Large). |
| Resilience Mode | Indicates whether high-resilience mode is enabled for this environment. |
| Airflow URI | The URL of the Airflow web interface for this environment. |
| DAGs Folder | The Cloud Storage path where this environment's DAG files are stored. |
| Storage Bucket | The Cloud Storage bucket associated with this environment. |
| Web Server Plugins Mode | Indicates whether web server plugins are enabled for this environment. |
| Metadata Retention (Days) | The number of days Airflow task metadata is retained. |
| Networking Type | The networking configuration type used by this environment (for example, public or private IP). |
| Network | The VPC network this environment is deployed in. |
| Subnetwork | The subnetwork this environment is deployed in. |
| Scheduler Count | The number of Airflow scheduler instances configured for this environment. |
| Scheduler CPU | The number of vCPUs allocated to each Airflow scheduler. |
| Scheduler Memory (GB) | The amount of memory, in GB, allocated to each Airflow scheduler. |
| Minimum Workers | The minimum number of Airflow worker instances this environment can scale down to. |
| Maximum Workers | The maximum number of Airflow worker instances this environment can scale up to. |
| Worker CPU | The number of vCPUs allocated to each Airflow worker. |
| Worker Memory (GB) | The amount of memory, in GB, allocated to each Airflow worker. |
| Web Server CPU | The number of vCPUs allocated to the Airflow web server. |
| Web Server Memory (GB) | The amount of memory, in GB, allocated to the Airflow web server. |
| DAG Processors | The number of standalone DAG processor instances configured for this environment. |
| Triggerers | The number of Airflow triggerer instances configured for this environment, used for deferrable operators. |
| Unique ID | The unique identifier assigned to this environment. |
| Created Time | The date and time this environment was created. |
| Updated Time | The date and time this environment was last updated. |
Workloads
This tab lists the Airflow components running in the environment: scheduler, web server, workers, triggerer, DAG processor and Redis. The columns are Name, Type, Status, Status Message and Details. The data is read live from Google Cloud each time you open the tab.
This tab can be empty for older Cloud Composer 1 environments, because the workloads API is meant for Composer 2 and later.
Threshold configuration
- Global configuration
- In the web client, go to the Admin section on the left navigation pane.
- Select Configuration Profiles from the left pane and select Threshold and Availability from the drop-down menu.
- Click Add Threshold Profile in the top-right corner.
- For Monitor Type, select Cloud Composer.
- Specify the threshold values for the required metrics, then click Save.
- Monitor-level configuration
- In the web client, go to Cloud > GCP > Cloud Composer.
- Select a resource you would like to set a threshold for, then click the hamburger icon.
- Select Edit, which directs you to the Edit Cloud Composer Monitor page.
- Specify the threshold values for the required metrics, then click Save.
Monitor status is decided based on the status of Cloud Composer:
- Up: RUNNING, CREATING, or UPDATING.
- Down: ERROR, HIBERNATED, or DELETING.
The alert will be displayed with a reason such as Cloud Composer environment is hibernated.
IT Automation
IT Automation tools automatically resolve performance degradation issues. These tools react to events proactively rather than waiting for manual intervention. The IT Automation tools automate repetitive tasks and automatically remediate threshold breaches. The alarm engine continually evaluates system events for which thresholds are set and executes the mapped automation when there is a breach.
Configuration Rules
Editing multiple monitors to associate different monitor groups or add a different tag can be a tedious process. With Configuration Rules, you can automate the configuration settings of your monitoring resources. Create custom rules to track configuration changes continuously and achieve the ideal configuration settings.
Summary
The Summary tab will give you the performance data organized by time for the metrics listed above. To view the summary:
- Go to Cloud > GCP > Cloud Composer.
- Select a resource.
- Click the Summary tab.
Configuration Details
The Configuration Details tab provides details on the configurations of application instances. To get the configuration details:
- Go to Cloud > GCP > Cloud Composer.
- Select a resource.
- Click the Configuration Details tab.
Reports
Gain in-depth data about the various parameters of your monitored resources and accentuate your service performance using our insightful reports.
To view reports for a Google Cloud Composer resource:
- Go to the Reports section on the left navigation pane.
- Select Cloud Composer from the menu on the left.
- You can find the Availability Summary Report, Performance Report, and Inventory Report for one selected monitor. Or you can get the Summary Report, Availability Summary Report, Health Trend Report, and Performance Report for all the Google Cloud Composer monitors.
You can also get reports from the Summary tab of the Google Cloud Composer monitor:
- Click the Summary tab.
- Get the Availability Summary Report of the monitor by clicking Availability.
- You can also find the Performance Report of the monitor by clicking any chart title.