AWS Glue Data Catalog
- What it is
- A persistent, central metadata repository for all your data assets — a technical catalog of databases, tables, schemas, partitions, and locations.
- Hive metastore compatible.
- Why it matters
- It is the single source of truth that lets Amazon Athena, Amazon Redshift Spectrum, Amazon EMR, and AWS Glue jobs all query the same data in Amazon S3 with the same schema definitions.
- Without it, each engine would need its own schema definition of the same files.
- How it gets populated
- Glue crawlers scan S3 (or JDBC sources), infer schema and data types, and create or update table definitions on a schedule.
- Related
- AWS Lake Formation builds on the Data Catalog to add fine-grained (table, column, row) permissions across a data lake.
- Not to be confused with
- AWS Glue itself — the ETL engine. The Data Catalog is the metadata layer it populates and reads.