Комментарии:
Hello! excellent video! It is recommended in the first bronze layer to save the data in parquet and in the following two in delta? thank you
ОтветитьAtomicity not automicity
ОтветитьACID = "Atomicity" not "Automicity". Thanks for the video.
ОтветитьYes explore the BOG boots on the ground
ОтветитьInteresting. What is the benefit of using this vs creating incremental loading within your merge statements? Are there more costs associated with using a delta lake? Additionally, will this pick-up changes from my source?
ОтветитьThis video is gold, makes it easier to understand spark and delta lake - kudos!
Ответить@Guy in a cube - Can we do this same thing in ADF - Mapping dataflow?
ОтветитьCan an API hosted on an App service in anyway fetch Delta tables data ? thanks
ОтветитьAgain, another great video from the great series (Azure synapse analytics)
Thanks a lot guys(in the cube), you are amazing
How do you handle change to the source system in a Delta lake? For example: when a source table adds 3 columns and drops two?
ОтветитьDo you know what is the difference between lake databases and delta lake project? Both seem to have roughly the same functionality - I can use Spark to do ETL tasks - and then use spark pools as well as serverless sql pools to query data.
ОтветитьGreat video series for getting started with the topic. Probably the video is already in production but as a follow up to the series I can imagine it could be interesting to see how powerful the functionality of delta is. What exactly does the time travel feature look like. For me it was impressive to see how granular you can jump back in time and roll back changes to rows but also structural changes to a table. If we want to look at it more from an ETL perspective, maybe a look at the change data feed would be interesting.
Regardless of how you continue this series I am very excited because your hands-on way of approaching these things takes the hurdle out of many to begin their journey.
Hello, thanks for putting out great content and useful videos.
Delta is certainly cool, however, after having a deeper look: Delta time travel does not seem to be a replacement for a proper Type2 SCD modelled data, since:
- there is a limited data retention for the delta log (30 days), it can be extended of course
- you can't leverage that time travel when using Serverless SQL Pool (which is how I'd expose Delta tables to Power BI)
- or have I missed something obvious?
Furthermore - the SQL / pySpark interoperability works to an extent, for example Synapse Spark SQL doesn't support SQL based time travel (SELECT * FROM TABLE AS OF VERSOIN N) - this has to be done via pySpark. On the bright side - pySpark is not that hard to pick up, takes getting used to, but it's quite powerful :)
Now only add support for Delta for the Workspace-created Lake Database! :)
Cheers
Seems complex 🤔
ОтветитьA for Atomicity I think 😊
ОтветитьPreviously I would not have perquet files, previously I would have a SQL-Server. What problem does a delta lake solve compared to just using a SQL-Server?
ОтветитьAdam looks different today...
ОтветитьYou should do videos about machine learning models in Synapse
ОтветитьPySQL should be a thing! 😁
Ответить