56.22 Future of DataFrames and Data Systems

Takeaways

SQLGlot is a no-dependency SQL parser, transpiler, optimizer, and engine. It can be used to format SQL or translate between 21 different dialects like DuckDB, Presto / Trino, Spark / Databricks, Snowflake, and BigQuery. It aims to read a wide variety of SQL inputs and output syntactically and semantically correct SQL in the targeted dialects.
Ibis is the portable Python dataframe library:
- Fast local dataframes (via DuckDB by default)
- Lazy dataframe expressions
- Interactive mode for iterative data exploration
- Compose Python dataframe and SQL code
- Use the same dataframe API for 20+ backends
- Iterate locally and deploy remotely by changing a single line of code
A Future Multi-Engine Stack
- An execution engine tailored for different data scales
  - <= 1TB: DuckDB and Friends
  - 1 - 10TB: Spark, Dask, Ray, etc
  - 10TB: GPU-accelerated Processing
- A portable language front end (Ibis, Malloy, PRQL, or a standard transpilable SQL)
- Arrow-native API and wire transport
Open source developers get feedback after 6 or 12 months.
Pandas is not going anywhere. Polars does not support time series data yet. Also ChatGPT knows more on Pandas than Polars.
Essential DS skills:
- SQL
- Designing schema
- Organizing data
- Data storage, partitioning
Columnar data format innovation to make parquet better he is thinking about.