DataExpert-io

DataExpert-io/data-engineer-handbook

This is a repo with links to everything you'd ever want to learn about data engineering

44.4kstars 9.4kforks 100+contributors Jupyter Notebook

README.md

# The Data Engineering Handbook
<a href="https://trendshift.io/repositories/8755" target="_blank"><img src="https://trendshift.io/api/badge/repositories/8755" alt="DataExpert-io%2Fdata-engineer-handbook | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>

This repo has all the resources you need to become an amazing data engineer!

### Join the free Databricks AI boot camp on August 3rd [here](https://learn.dataexpert.io/program/the-one-week-beginners-databricks-boot-camp-7129) and signup for Databricks Free Edition [here](https://signup.databricks.com/?provider=DB_FREE_TIERutm_source=github&utm_medium=video&utm_campaign=DataExpert)

## Getting started

If you are new to data engineering, start by following this [2024 breaking into data engineering roadmap](https://blog.dataengineer.io/p/the-2024-breaking-into-data-engineering)

If you are here for the [4-week free beginner boot camp](https://learn.dataexpert.io/program/the-absolute-beginner-data-engineering-boot-camp-starting-august-7th-6453/details) you can check out:
- [introduction](beginner-bootcamp/introduction.md)
- [software needed](beginner-bootcamp/software.md)

If you are here for the [6-week free intermediate boot camp](https://learn.dataexpert.io/program/free-community-boot-camp/details) you can check out
- [introduction](intermediate-bootcamp/introduction.md)
- [software needed](intermediate-bootcamp/software.md)

For more applied learning:
- Check out the [projects](projects.md) section for more hands-on examples!
- Check out the [interviews](interviews.md) section for more advice on how to pass data engineering interviews!
- Check out the [books](books.md) section for a list of high quality data engineering books
- Check out the [communities](communities.md) section for a list of high quality data engineering communities to join
- Check out the [newsletter](newsletters.md) section to learn via email

## Resources

### Great [list of over 25 books](books.md)

Top 3 must read books are:
- [Fundamentals of Data Engineering](https://www.amazon.com/Fundamentals-Data-Engineering-Robust-Systems/dp/1098108302/)
- [Designing Data-Intensive Applications](https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321/)
- [Designing Machine Learning Systems](https://www.amazon.com/Designing-Machine-Learning-Systems-Production-Ready/dp/1098107969)

### Great [list of over 10 communities to join](communities.md):

Top must-join communities for DE:
- [DataExpert.io Community Discord](https://discord.gg/JGumAXncAK)
- [Data Talks Club Slack](https://datatalks.club/slack)
- [Data Engineer Things Community](https://www.dataengineerthings.org/)

Top must-join communities for ML:
- [AdalFlow Discord](https://discord.com/invite/ezzszrRZvT)
- [Chip Huyen MLOps Discord](https://discord.gg/dzh728c5t3)

### Companies:

- Orchestration  
  - [Mage](https://www.mage.ai)
  - [Astronomer](https://www.astronomer.io)
  - [Prefect](https://www.prefect.io)
  - [Dagster](https://www.dagster.io)
  - [Airflow](https://airflow.apache.org/)
  - [Kestra](https://kestra.io/) 
  - [Shipyard](https://www.shipyardapp.com/)
  - [Hamilton](https://github.com/dagworks-inc/hamilton)
- Data Lake / Cloud
  - [Tabular](https://www.tabular.io)
  - [Microsoft](https://www.microsoft.com)
  - [Databricks](https://www.databricks.com/company/about-us)
  - [Onehouse](https://www.onehouse.ai)
  - [Delta Lake](https://delta.io/)
  - [Ilum](https://ilum.cloud/)
  - [DuckLake](https://ducklake.select/)
  - [Apache Iceberg](https://iceberg.apache.org/)
  - [Apache Polaris](https://polaris.apache.org/)
  - [Lakekeeper](https://lakekeeper.io/)
- Data Warehouse
  - [Snowflake](https://www.snowflake.com/en/)
  - [Firebolt](https://www.firebolt.io/)
  - [Databend](https://www.databend.com/)
- Data Quality
  - [dbt](https://www.getdbt.com/)
  - [Metaplane](https://www.metaplane.dev/)
  - [Gable](https://www.gable.ai)
  - [Great Expectations](https://www.greatexpectations.io)
  - [Streamdal](https://streamdal.com)
  - [Coalesce](https://coalesce.io/)
  - [Soda](https://www.soda.io/)
  - [DQOps](https://dqops.com/)
  - [HEDDA.IO](https://hedda.io)
  - [Dingo](https://github.com/MigoXLab/dingo)
- Education Companies
  - [DataExpert.io](https://www.dataexpert.io)
  - [LearnDataEngineering.com](https://www.learndataengineering.com)
  - [AlgoExpert](https://www.algoexpert.io)
  - [ByteByteGo](https://www.bytebytego.com)
- Analytics / Visualization
  - [Preset](https://www.preset.io)
  - [Starburst](https://www.starburst.io)
  - [Metabase](https://www.metabase.com/)
  - [Looker Studio](https://lookerstudio.google.com/overview)
  - [Tableau](https://www.tableau.com/)
  - [Power BI](https://powerbi.microsoft.com/)
  - [Hex](https://hex.ai/)
  - [Apache Superset](https://superset.apache.org/)
  - [Evidence](https://evidence.dev)
  - [Redash](https://redash.io/)
  - [Lightdash](https://lightdash.com/)
- Data Integration
  - [Cube](https://cube.dev)
  - [Fivetran](https://www.fivetran.com)
  - [Airbyte](https://airbyte.io)
  - [dlt](https://dlthub.com/)
  - [Sling](https://slingdata.io/)
  - [Meltano](https://meltano.com/)
  - [Estuary](https://estuary.dev/)
  - [Arpe.io](https://arpe.io/)
- Semantic Layers
  - [Cube](https://cube.dev)
  - [dbt Semantic Layer](https://www.getdbt.com/product/semantic-layer) 
- Modern OLAP
  - [Apache Druid](https://druid.apache.org/)
  - [ClickHouse](https://clickhouse.com/)
  - [Apache Pinot](https://pinot.apache.org/)
  - [Apache Kylin](https://kylin.apache.org/)
  - [DuckDB](https://duckdb.org/)
  - [QuestDB](https://questdb.io/)
  - [StarRocks](https://www.starrocks.io/)
- LLM application library
  - [AdalFlow](https://github.com/SylphAI-Inc/AdalFlow)
  - [LangChain](https://github.com/langchain-ai/langchain)
  - [LlamaIndex](https://github.com/run-llama/llama_index)
- Real-Time Data
  - [Aggregations.io](https://aggregations.io)
  - [Responsive](https://www.responsive.dev/)
  - [RisingWave](https://risingwave.com/)
  - [Striim](https://www.striim.com/)
- Data Lineage
  - [OpenLineage](https://openlineage.io/)

### Data Engineering blogs of companies:

- [Netflix](https://netflixtechblog.com/tagged/big-data)
- [Uber](https://www.uber.com/blog/houston/data/?uclick_id=b2f43229-f3f4-4bae-bd5d-10a05db2f70c)
- [Databricks](https://www.databricks.com/blog/category/engineering/data-engineering)
- [Airbnb](https://medium.com/airbnb-engineering/data/home)
- [Amazon AWS Blog](https://aws.amazon.com/blogs/big-data/)
- [Microsoft Data Architecture Blogs](https://techcommunity.microsoft.com/t5/data-architecture-blog/bg-p/DataArchitectureBlog)
- [Microsoft Fabric Blog](https://blog.fabric.microsoft.com/)
- [Oracle](https://blogs.oracle.com/datawarehousing/)
- [Meta](https://engineering.fb.com/category/data-infrastructure/)
- [Onehouse](https://www.onehouse.ai/blog)
- [Estuary Blog](https://estuary.dev/blog/)

### Data Engineering Whitepapers:

- [A Five-Layered Business Intelligence Architecture](https://ibimapublishing.com/articles/CIBIMA/2011/695619/695619.pdf)
- [Lakehouse:A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf)
- [Big Data Quality: A Data Quality Profiling Model](https://link.springer.com/chapter/10.1007/978-3-030-23381-5_5)
- [The Data Lakehouse: Data Warehousing and More](https://arxiv.org/abs/2310.08697)
- [Spark: Cluster Computing with Working Sets](https://dl.acm.org/doi/10.5555/1863103.1863113)
- [The Google File System](https://research.google/pubs/the-google-file-system/)
- [Building a Universal Data Lakehouse](https://www.onehouse.ai/whitepaper/onehouse-universal-data-lakehouse-whitepaper)
- [XTable in Action: Seamless Interoperability in Data Lakes](https://arxiv.org/abs/2401.09621)
- [MapReduce: Simplified Data Processing on Large Clusters](https://research.google/pubs/mapreduce-simplified-data-processing-on-large-clusters/)
- [Tidy Data](https://vita.had.co.nz/papers/tidy-data.pdf)
- [Data Engineering Whitepapers](https://www.ssp.sh/brain/data-engineering-whitepapers/)

### Social Media Accounts

Here's the mostly comprehensive list of data engineering creators: 
**(You have to have at least 5k followers somewhere to be added!)**

#### YouTube 
| Name                        | YouTube Channel                                                                                         | Follower Count |
|----------------------------|---------------------------------------------------------------------------------------------------------|---------------:|
| ByteByteGo                 | [ByteByteGo](https://www.youtube.com/c/ByteByteGo)                                             | 1,000,000+     |
| Data with Baraa            | [Data with Baraa](https://www.youtube.com/@DataWithBaraa)                                       | 195,000+     |
| Zach Wilson                | [Data with Zach](https://www.youtube.com/@eczachly_)                                          | 150,000+       |
| Shashank Mishra            | [E-learning Bridge](https://www.youtube.com/@shashank_mishra)                                   | 100,000+       |
| Seattle Data Guy           | [Seattle Data Guy](https://www.youtube.com/c/SeattleDataGuy)                                  | 100,000+       |
| TrendyTech                 | [TrendyTech](https://www.youtube.com/c/TrendytechInsights)                                   | 100,000+       |
| Darshil Parmar             | [Darshil Parmar](https://www.youtube.com/@DarshilParmar)                                       | 100,000+       |
| Andreas Kretz              | [Andreas Kretz](https://www.youtube.com/c/andreaskayy)                                          | 100,000+       |
| The Ravit Show             | [The Ravit Show](https://youtube.com/@theravitshow)                                           | 100,000+       |
| Guy in a Cube              | [Guy in a Cube](https://www.youtube.com/@GuyInACube)                                            | 100,000+       |
| Adam Marczak               | [Adam Marczak](https://www.youtube.com/@AdamMarczakYT)                                         | 100,000+       |
| nullQueries                | [nullQueries](https://www.youtube.com/@nullQueries)                                             | 100,000+       |
| TECHTFQ by Thoufiq         | [TECHTFQ by Thoufiq](https://www.youtube.com/@techTFQ)                                         | 100,000+       |
| SQLBI                       | [SQLBI](https://www.youtube.com/@SQLBI)                                                     | 100,000+       |
| Alex Freberg               | [Alex The Analyst](https://www.youtube.com/@AlexTheAnalyst)                                     | 100,000+       |
| Ankur Ranjan               | [Big Data Show](https://www.youtube.com/@TheBigDataShow)                                        | 100,000+       |
| Prashanth Kumar Pandey     | [ScholarNest](https://www.youtube.com/@ScholarNest)                                              | 77,000+        |
| ITVersity                  | [ITVersity](https://www.youtube.com/@itversity)                                                  | 67,000+        |
| Soumil Shah                | [Soumil Shah](https://www.youtube.com/@SoumilShah)                                               | 50,000         |
| Ansh Lamba                 | [Ansh Lamba](https://www.youtube.com/@AnshLambaJSR)                                              | 18,000+        |
| Azure Lib                  | [Azure Lib](https://www.youtube.com/@azurelib-academy)                                        | 10,000+        |
| Advancing Analytics        | [Advancing Analytics](https://www.youtube.com/@AdvancingAnalytics)                               | 10,000+        |
| Kahan Data Solutions       | [Kahan Data Solutions](https://www.youtube.com/@KahanDataSolutions)                               | 10,000+        |
| Ankit Bansal               | [Ankit Bansal](https://youtube.com/@ankitbansal6)                                               | 10,000+        |
| Mr. K Talks Tech           | [Mr. K Talks Tech](https://www.youtube.com/channel/UCzdOan4AmF65PmLLks8Lmww)                      | 10,000+        |
| Samuel Focht               | [Python Basics](https://www.youtube.com/@PythonBasics)                                           | 10,000+        |
| Mehdi Ouazza              | [Mehdio DataTV](https://www.youtube.com/@mehdio)                                                    | 3,000+         |
| Alex Merced                | [Alex Merced Data](https://www.youtube.com/@alexmerceddata_)                                            | N/A           |
| John Kutay                 | [John Kutay](https://www.youtube.com/@striiminc) | N/A           |
| Emil Kaminski              | [Databricks For Professionals](https://www.youtube.com/@DatabricksPro)                           | 5,000+          |

#### LinkedIn

| Name                      | LinkedIn Profile                                                                                         | Follower Count |
|--------------------------|----------------------------------------------------------------------------------------------------------|---------------:|
| Zach Wilson              | [Zach Wilson](https://www.linkedin.com/in/eczachly)                                                     | 400,000+       |
| Chip Huyen               | [Chip Huyen](https://www.linkedin.com/in/chiphuyen/)                                    | 250,000+       |
| Shashank Mishra          | [Shashank Mishra](https://www.linkedin.com/in/shashank219/)                                     | 100,000+       |
| Seattle Data Guy         | [Ben Rogojan](https://www.linkedin.com/in/benjaminrogojan)                                        | 100,000+       |
| TrendyTech               | [Sumit Mittal](https://www.linkedin.com/in/bigdatabysumit/)                                   | 100,000+       |
| Darshil Parmar           | [Darshil Parmar](https://www.linkedin.com/in/darshil-parmar/)                                   | 100,000+       |
| Andreas Kretz            | [Andreas Kretz](https://www.linkedin.com/in/andreas-kretz)                                     | 100,000+       |
| ByteByteGo (Alex Xu)     | [Alex Xu](https://www.linkedin.com/in/alexxubyte)                                             | 100,000+       |
| Azure Lib (Deepak Goyal) | [Deepak Goyal](https://www.linkedin.com/in/deepak-goyal-93805a17/)                              | 100,000+       |
| Alex Freberg             | [Alex Freberg](https://www.linkedin.com/in/alex-freberg/)                                     | 100,000+       |
| SQLBI (Marco Russo)      | [Marco Russo](https://www.linkedin.com/in/sqlbi)                                                  | 50,000+        |
| Ankit Bansal             | [Ankit Bansal](https://www.linkedin.com/in/ankitbansal6/)                                        | 50,000+        |
| Marc Lamberti            | [Marc Lamberti](https://www.linkedin.com/in/marclamberti)                                       | 50,000+        |
| Ankur Ranjan             | [Ankur Ranjan](https://www.linkedin.com/in/thebigdatashow/)                                       | 48,000+        |
| ITVersity (Durga Gadiraju)| [Durga Gadiraju](https://www.linkedin.com/in/durga0gadiraju/)                                   | 48,000+        |
| Prashanth Kumar Pandey   | [Prashanth Kumar Pandey](https://www.linkedin.com/in/prashant-kumar-pandey/)                       | 37,000+        |
| Alex Merced              | [Alex Merced](https://www.linkedin.com/in/alexmerced)                                           | 30,000+        |
| Ijaz Ali                 | [Ijaz Ali](https://www.linkedin.com/in/ijaz-ali-6aaa87122/)                                       | 24,000+        |
| Mehdi Ouazza             | [Mehdi Ouazza](https://www.linkedin.com/in/mehd-io/)                                        | 20,000+        |
| Ananth Packkildurai      | [Ananth Packkildurai](https://www.linkedin.com/in/ananthdurai/)                                    | 18,000+        |
| Ansh Lamba               | [Ansh Lamba](https://www.linkedin.com/in/ansh-lamba-793681184/)                                    | 13,000+        |
| Manojkumar Vadivel       | [Manojkumar Vadivel](https://www.linkedin.com/in/manojvsj/)                                        | 12,000+        |
| Advancing Analytics      | [Simon Whiteley](https://www.linkedin.com/in/simon-whiteley-uk/)                                  | 10,000+        |
| Li Yin                   | [Li Yin](https://www.linkedin.com/in/li-yin-ai/)                                                  | 10,000+        |
| Jaco van Gelder          | [Jaco van Gelder](https://www.linkedin.com/in/jwvangelder/)                                       | 10,000+        |
| Joseph Machado           | [Joseph Machado](https://www.linkedin.com/in/josephmachado1991/)                                  | 10,000+        |
| Eric Roby                | [Eric Roby](https://www.linkedin.com/in/codingwithroby/)                                           | 10,000+        |
| Simon Späti              | [Simon Späti](https://www.linkedin.com/in/sspaeti/)                                            | 10,000+        |
| Constantin Lungu         | [Constantin Lungu](https://www.linkedin.com/in/constantin-lungu-668b8756)                         | 10,000+        |
| Lakshmi Sontenam         | [Lakshmi Sontenam](https://www.linkedin.com/in/shivaga9esh)                                      | 9,500+         |
| Dani Pálma               | [Daniel Pálma](https://www.linkedin.com/in/danthelion/)                                          | 9,000+         |
| Soumil Shah              | [Soumil Shah](https://www.linkedin.com/in/shah-soumil/)                                          | 8,000+         |
| Arnaud Milleker          | [Arnaud Milleker](https://www.linkedin.com/in/arnaudmilleker/)                                    | 7,000+         |
| Dimitri Visnadi          | [Dimitri Visnadi](https://www.linkedin.com/in/visnadi/)                                    | 7,000+         |
| Lenny                    | [Lenny A](https://www.linkedin.com/in/lennyardiles/)                                         | 6,000+         |
| Dipankar Mazumdar        | [Dipankar Mazumdar](https://www.linkedin.com/in/dipankar-mazumdar/)                                 | 5,000+         |
| Daniel Ciocirlan         | [Daniel Ciocirlan](https://www.linkedin.com/in/danielciocirlan)                                    | 5,000+         |
| Hugo Lu                  | [Hugo Lu](https://www.linkedin.com/in/hugo-lu-confirmed/)                                           | 5,000+         |
| Tobias Macey             | [Tobias Macey](https://www.linkedin.com/in/tmacey)                                                 | 5,000+         |
| Marcos Ortiz             | [Marcos Ortiz](https://www.linkedin.com/in/mlortiz)                                             | 5,000+         |
| Julien Hurault           | [Julien Hurault](https://www.linkedin.com/in/julienhuraultanalytics/)                               | 5,000+         |
| John Kutay               | [John Kutay](https://www.linkedin.com/in/johnkutay/)                                               | 5,000+         |
| Hassaan Akbar            | [Hassaan Akbar](https://www.linkedin.com/in/ehassaan)                                              | 5,000+         |
| Subhankar                | [Subhankar](https://www.linkedin.com/in/subhankarumass/)                                            | 5,000+         |
| Nitin                    | [Nitin](https://www.linkedin.com/in/tomernitin29/)                                                        | N/A           |
| Hassaan                    | [Hassaan](https://www.linkedin.com/in/shassaan/)                                                        | 5000+           |
| Javier de la Torre             | [Javier](www.linkedin.com/in/javier-de-la-torre-medina)                                                        | 5000+           |

#### X/Twitter

| Name              | X/Twitter Profile                                                 | Follower Count |
|-------------------|------------------------------------------------------------------|---------------:|
| ByteByteGo        | [alexxubyte](https://twitter.com/alexxubyte/)            | 100,000+       |
| Dan Kornas        | [@dankornas](https://www.twitter.com/dankornas)           | 66,000+        |
| Zach Wilson       | [EcZachly](https://www.twitter.com/EcZachly)          | 30,000+        |
| Seattle Data Guy  | [SeattleDataGuy](https://www.twitter.com/SeattleDataGuy)   | 10,000+        |
| SQLBI             | [marcorus](https://x.com/marcorus)                       | 10,000+        |
| Joseph Machado    | [startdataeng](https://twitter.com/startdataeng)         | 5,000+         |
| Alex Merced       | [@amdatalakehouse](https://www.twitter.com/amdatalakehouse)      | N/A           |
| John Kutay        | [@JohnKutay](https://x.com/JohnKutay)                            | N/A           |
| Mehdi Ouazza      | [mehd_io](https://x.com/mehd_io)                                 | N/A           |

#### Instagram

| Name           | Instagram Profile                                                                   | Follower Count |
|----------------|--------------------------------------------------------------------------------------|---------------:|
| Sundas Khalid  | [sundaskhalidd](https://www.instagram.com/sundaskhalidd)                              | 300,000+       |
| Zach Wilson    | [eczachly](https://www.instagram.com/eczachly)                             | 150,000+       |
| Andreas Kretz  | [learndataengineering](https://www.instagram.com/learndataengineering)          | 5,000+         |
| Alex Merced    | [@alexmercedcoder](https://www.instagram.com/alexmercedcoder)                       | N/A           |

#### TikTok

| Name            | TikTok Profile                                                                   | Follower Count |
|-----------------|----------------------------------------------------------------------------------|---------------:|
| Zach Wilson     | [@eczachly](https://www.tiktok.com/@eczachly)                            | 70,000+        |
| Alex Freberg    | [@alex_the_analyst](https://www.tiktok.com/@alex_the_analyst)             | 10,000+        |
| Mehdi Ouazza    | [@mehdio_datatv](https://www.tiktok.com/@mehdio_datatv)                          | N/A           |

### Great Podcasts

- [The Data Engineering Show](https://www.dataengineeringshow.com/)
- [Data Engineering Podcast](https://www.dataengineeringpodcast.com/)
- [DataTopics](https://www.datatopics.io/)
- [The Data Engineering Side Of Data](https://podcasts.apple.com/us/podcast/the-engineering-side-of-data/id1566999533)
- [DataWare](https://www.ascend.io/dataaware-podcast/)
- [The Data Coffee Break Podcast](https://www.deezer.com/us/show/5293247)
- [The Datastack show](https://datastackshow.com/)
- [Intricity101 Data Sharks Podcast](https://www.intricity.com/learningcenter/podcast)
- [Drill to Detail with Mark Rittman](https://www.rittmananalytics.com/drilltodetail/)
- [Analytics Power Hour](https://analyticshour.io/)
- [Catalog & cocktails](https://listen.casted.us/public/127/Catalog-%26-Cocktails-2fcf8728)
- [Datatalks](https://datatalks.club/podcast.html)
- [Data Brew by Databricks](https://www.databricks.com/discover/data-brew)
- [The Data Cloud Podcast by Snowflake](https://rise-of-the-data-cloud.simplecast.com/)
- [What's New in Data](https://www.striim.com/podcast/)
- [Open||Source||Data by Datastax](https://www.datastax.com/resources/podcast/open-source-data)
- [Streaming Audio by confluent](https://developer.confluent.io/podcast/)
- [The Data Scientist Show](https://podcasts.apple.com/us/podcast/the-data-scientist-show/id1584430381)
- [MLOps.community](https://podcast.mlops.community/)
- [Monday Morning Data Chat](https://open.spotify.com/show/3Km3lBNzJpc1nOTJUtbtMh)
- [The Data Chief](https://www.thoughtspot.com/data-chief/podcast)
- [The Joe Reis Show](https://open.spotify.com/show/3mcKitYGS4VMG2eHd2PfDN)
- [Data Bytes](https://open.spotify.com/show/6VbjON5Ck9QYInBnmoqrDE)
- [Super Data Science: ML & AI Podcast with Jon Krohn](https://open.spotify.com/show/1n8P7ZSgfVLVJ3GegxPat1)

### Great [list of 20+ newsletters](newsletters.md)

Top must follow newsletters for data engineering:
- [DataEngineer.io Newsletter](https://blog.dataengineer.io)
- [Joe Reis](https://joereis.substack.com)
- [Start Data Engineering](https://www.startdataengineering.com)
- [Data Engineering Weekly](https://www.dataengineeringweekly.com)
- [Data Engineer Things](https://dataengineerthings.substack.com/)

### Glossaries:
- [Data Engineering Vault](https://www.ssp.sh/brain/data-engineering/)
- [Airbyte Data Glossary](https://glossary.airbyte.com/)
- [Data Engineering Wiki by Reddit](https://dataengineering.wiki/Index)
- [Seconda Glossary](https://www.secoda.co/glossary/)
- [Glossary Databricks](https://www.databricks.com/glossary)
- [Airtable Glossary](https://airtable.com/shrGh8BqZbkfkbrfk/tbluZ3ayLHC3CKsDb)
- [Data Engineering Glossary by Dagster](https://dagster.io/glossary)

### Design Patterns

- [Cumulative Table Design](https://www.github.com/DataExpert-io/cumulative-table-design)
- [Microbatch Deduplication](https://www.github.com/EcZachly/microbatch-hourly-deduped-tutorial)
- [The Little Book of Pipelines](https://www.github.com/EcZachly/little-book-of-pipelines)
- [Data Developer Platform](https://datadeveloperplatform.org/architecture/)

### Courses / Academies

- [DataExpert.io course](https://www.dataexpert.io) use code **HANDBOOK10** for a discount!
- [LearnDataEngineering.com](https://www.learndataengineering.com)
- [Technical Freelancer Academy](https://www.technicalfreelanceracademy.com/) Use code **zwtech** for a discount!
- [IBM Data Engineering for Everyone](https://www.edx.org/learn/data-engineering/ibm-data-engineering-basics-for-everyone)
- [PrepNPlaced Open Learning](https://www.prepnplaced.com/open-learning) - 153 hours of free SQL, Python, PySpark, and Power BI lessons with MCQs and a free skill test
- [Qwiklabs](https://www.qwiklabs.com/)
- [DataCamp](https://www.datacamp.com/)
- [Udemy Courses from Shruti Mantri](https://www.udemy.com/user/shruti-mantri-5/)
- [Rock the JVM](https://rockthejvm.com/) teaches Spark (in Scala), Flink and others
- [Data Engineering Zoomcamp by DataTalksClub](https://datatalks.club/)
- [Efficient Data Processing in Spark](https://josephmachado.podia.com/efficient-data-processing-in-spark)
- [Scaler](https://www.scaler.com/)
- [DataTeams - Data Engingeer hiring platform](https://www.datateams.ai/)
- [Udemy Courses from Daniel Blanco](https://danielblanco.dev/links)
- [DeepLearning.AI Data Engineering Professional Certificate](https://www.coursera.org/professional-certificates/data-engineering)

### Certifications Courses

- [Google Cloud Certified - Professional Data Engineer](https://cloud.google.com/certification/data-engineer)
- [Databricks - Certified Associate Developer for Apache Spark](https://www.databricks.com/learn/certification/apache-spark-developer-associate)
- [Databricks - Data Engineer Associate](https://www.databricks.com/learn/certification/data-engineer-associate)
- [Databricks - Data Engineer Professional](https://www.databricks.com/learn/certification/data-engineer-professional)
- [Microsoft DP-203: Data Engineering on Microsoft Azure](https://learn.microsoft.com/en-us/credentials/certifications/exams/dp-203/?tab=tab-learning-paths)
- [Microsoft DP-600: Fabric Analytics Engineer Associate](https://learn.microsoft.com/credentials/certifications/fabric-analytics-engineer-associate/)
- [Microsoft DP-700: Fabric Data Engineer Associate](https://learn.microsoft.com/en-us/credentials/certifications/fabric-data-engineer-associate/?practice-assessment-type=certification)
- [AWS Certified Data Engineer - Associate](https://aws.amazon.com/certification/certified-data-engineer-associate/)

Repository info

Default branch: main
Latest version: —
Created: 2023-11-20
Last activity: 2026-08-04
License: —
Open issues: 48
Watchers: 488

Topics

apachespark awesome bigdata data dataengineering sql

Contributors

EcZachly

205 contributions

ry-v1

39 contributions

isangwanrahul

22 contributions

evil-in

10 contributions

damaradiprabowo

8 contributions