What is this delta lake thing?

Hello! excellent video! It is recommended in the first bronze layer to save the data in parquet and in the following two in delta? thank you

Atomicity not automicity

ACID = "Atomicity" not "Automicity". Thanks for the video.

Yes explore the BOG boots on the ground

Interesting. What is the benefit of using this vs creating incremental loading within your merge statements? Are there more costs associated with using a delta lake? Additionally, will this pick-up changes from my source?

This video is gold, makes it easier to understand spark and delta lake - kudos!

@Guy in a cube - Can we do this same thing in ADF - Mapping dataflow?

Can an API hosted on an App service in anyway fetch Delta tables data ? thanks

Again, another great video from the great series (Azure synapse analytics)
Thanks a lot guys(in the cube), you are amazing

How do you handle change to the source system in a Delta lake? For example: when a source table adds 3 columns and drops two?

Do you know what is the difference between lake databases and delta lake project? Both seem to have roughly the same functionality - I can use Spark to do ETL tasks - and then use spark pools as well as serverless sql pools to query data.

Great video series for getting started with the topic. Probably the video is already in production but as a follow up to the series I can imagine it could be interesting to see how powerful the functionality of delta is. What exactly does the time travel feature look like. For me it was impressive to see how granular you can jump back in time and roll back changes to rows but also structural changes to a table. If we want to look at it more from an ETL perspective, maybe a look at the change data feed would be interesting.

Regardless of how you continue this series I am very excited because your hands-on way of approaching these things takes the hurdle out of many to begin their journey.

Hello, thanks for putting out great content and useful videos.

Delta is certainly cool, however, after having a deeper look: Delta time travel does not seem to be a replacement for a proper Type2 SCD modelled data, since:
- there is a limited data retention for the delta log (30 days), it can be extended of course
- you can't leverage that time travel when using Serverless SQL Pool (which is how I'd expose Delta tables to Power BI)
- or have I missed something obvious?

Furthermore - the SQL / pySpark interoperability works to an extent, for example Synapse Spark SQL doesn't support SQL based time travel (SELECT * FROM TABLE AS OF VERSOIN N) - this has to be done via pySpark. On the bright side - pySpark is not that hard to pick up, takes getting used to, but it's quite powerful :)

Now only add support for Delta for the Workspace-created Lake Database! :)

Cheers

Seems complex 🤔

A for Atomicity I think 😊

Previously I would not have perquet files, previously I would have a SQL-Server. What problem does a delta lake solve compared to just using a SQL-Server?

Adam looks different today...

You should do videos about machine learning models in Synapse

PySQL should be a thing! 😁