AI / ML

AWS has introduced an Apache Spark troubleshooting agent for Amazon EMR on EKS, enabling data engineers to diagnose job failures using natural language queries. The agent automatically analyzes Spark History Server data, distributed executor logs, and cluster configurations to identify root causes like memory errors, data skew, and resource contention. It provides automated root cause analysis and PySpark code recommendations, eliminating the need to manually sift through logs. This feature extends the troubleshooting agent's coverage to include EMR on EC2 and EMR Serverless, accessible via the console or CLI.

• Diagnose EMR on EKS Spark failures via natural language instead of manual log inspection.
• Agent analyzes History Server data, executor logs, and cluster configs for root cause analysis.
• Receives automated code recommendations to fix issues like memory errors and data skew.
• Now available across all EMR deployment options: EC2, Serverless, and EKS.
• Access directly from the EMR on EKS console or via the AWS CLI.

Amazon SageMaker Feature Store now supports high-throughput batch ingestion via BatchWriteRecord, allowing multiple records across feature groups in a single request. Users can also discover stored records using ListRecords without pre-known identifiers. Additionally, the offline store now allows creating tables and databases with custom names for better cataloging.

• Use BatchWriteRecord for lower latency and fewer API calls during high-volume feature ingestion.
• ListRecords enables record discovery in feature groups without needing specific record IDs.
• Custom naming for offline store tables and databases improves data organization and cataloging.
• Batch writes target the online store to accelerate real-time feature availability for models.

Amazon SageMaker Unified Studio Workflows now includes 19 new operators for services like Amazon Bedrock, S3 Tables, S3 Vectors, and AWS Glue Data Catalog. These additions enable users to orchestrate tasks such as managing Bedrock guardrails and provisioning S3 resources directly within the visual workflow creator. By integrating these capabilities, the update reduces the need for custom integration code or switching between different AWS consoles. This expansion allows data workers to build more comprehensive workflows that include triggering MWAA Serverless runs and managing catalog metadata.

• No custom code needed for Bedrock guardrails, S3 Tables/Vectors, and Glue Catalog tasks.
• Visual workflow creator now supports 19 new operators across four major AWS services.
• Centralize orchestration by triggering MWAA Serverless runs directly from SageMaker.
• Reduced context switching between consoles for data engineering workflows.