In this discussion, Madison talks about the skills that aspiring analytics engineers should prioritize and shares insights into the importance of networking and job search, along with tips on interview processes.

Vu Trinh details Discord's evolution from using in-house data orchestration solutions to adopting modern tools like Dagster and dbt. Initially, Discord used their internal system, Derived, to manage complex data transformations. However, as their needs grew, they found limitations in Derived's flexibility.

Harish Govindaraju discusses the challenges data organizations face in transitioning from delivering traditional data outputs to creating true data products. He emphasizes that a data product includes not just data but also essential metadata ensuring usability, reliability, and governance. He also explores the importance of clear ownership, data quality, lineage, security, and interoperability.

Robert S. Seiner emphasizes the critical importance of understanding and managing organizational data through a comprehensive Data Asset Inventory. This inventory is more than just a list; it’s a detailed catalog that includes internal and external data, metadata, and contextual information.

Brittany discusses the recurring issue of companies frequently switching Business Intelligence (BI) tools without addressing the root causes of their data challenges. She argues that the high costs and complexities involved in implementing new BI platforms often don't yield long-term benefits, as the core problem lies in the inability of these tools to scale with evolving data models.

Ananth Packkildurai provides a comprehensive guide to evaluating Change Data Capture (CDC) tools, which are crucial for real-time data integration and analytics in modern data architectures. He emphasizes CDC's relevance in unified data environments, its role in modernizing legacy systems, and its importance for real-time business demands.

Marc Delbaere explores the integration of data lineage and data value to maximize business impact. He draws a parallel to the Japanese term "Tsundoku" to highlight the wasted potential of unused data catalogues. Delbaere introduces "Data Value Lineage" as a process of ensuring that all data activities are linked to business value.

Pavel Samsonov critiques Nike's shift to a data-driven strategy, orchestrated by CEO John Donahue on McKinsey's advice, which led to a $25B market cap loss. This approach favored easily measurable data, ignoring the qualitative insights critical for effective marketing. Samsonov argues that relying solely on quantitative data can obscure important nuances, leading to misguided decisions.

This blog differentiates between a Data Governance Clinic, an intensive workshop offering personalized advice, and a Data Governance Mastermind, a peer-to-peer session for ongoing support and collaborative problem-solving to help audiences understand what they need.

In his latest blog, Seattle Data Guy emphasizes the importance of foundational skills beyond specific technologies in the data field. He highlights the value of thinking in systems, developing data intuition, fostering team growth, and maintaining constant curiosity.

Torsten Walbaum explores how perfectionism can hinder careers in high-growth companies, particularly for Data Scientists. He argues that delivering results quickly is often more valuable than striving for perfection, as businesses prioritize speed over flawless work.

Bernd Wessely challenges the current practices in data engineering, highlighting confusion over its definition and the problematic integration of business logic. He argues that data engineering should focus solely on the movement, manipulation, and management of data, leaving business logic to application developers.

Tobias Macey, in conversation with Tom Baeyens, discussed the importance of data contracts in ensuring data reliability in the latest episode of Data Engineering Podcast. They compare data observability, which is reactive and focuses on diagnosing issues, with data testing, which is preventive and ensures data quality across handover points.

Mikkel Dengsøe explores the composition of data teams across 40 top companies in the US and Europe. He emphasizes that there’s no universal ideal ratio for data roles, as different companies have distinct needs and structures.

Kris Peeters discusses the complexity of modern data platforms and the challenges of using a "bag of tools" approach, which requires developers to master multiple tools and scripts. He advocates for creating a unified interface that simplifies the user experience, reduces cognitive load, and supports diverse use cases, such as data warehousing and data science.

Host Richie Cotton discusses with Gunther Hagleitner how large language models (LLMs) can significantly boost data teams' productivity through LLM-assisted coding. The conversation focuses on the technology, talent, and processes needed to harness these productivity gains. Key insights include adjusting workflows to integrate LLMs and understanding the evolving skillsets required for data practitioners in this new landscape.

This article talks about key trends like the rise of real-time data processing, increased adoption of data warehouses and lakes, the importance of data observability for pipeline reliability, and a shift toward modular and reusable data architectures.

Barr Moses says that many data quality initiatives fail due to poor incentive structures and lack of focus on what truly matters. Effective data qulaity scorecards need to prioritize critical data, standardize quality measurement, and incentivize both producers and consumers. Successful initiatives ensure both data producers and consumers trust and adopt the data provided, leveraging modern tools like data observability solutions and data catalogs to streamline and sustain these efforts.

Sandeep Bose discusses the pitfalls of traditional data lake strategies, hilighting their high costs, long timelines, and fragmented data organization. Instead of achieving a unified data repository, many enterprises end up with hard-to-navigate data silos. Bose argues for a shift towards domain-driven data architectures and modern technologies like Generative AI (GenAI) and Machine Learning (ML).

Dan McKone emphasizes the importance of using data and analytics not just to support but to directly derive business strategy, especially during critical decision-making moments. Despite the widespread availability of data, many leaders rely too much on intuition, leading to less effective decisions. He highlights examples like HP's strategic split, Spotify's subscription model, and Starbucks' location strategy as successes of data-driven decision-making.

In the latest episode of the Data Pancakes, the host explores the complexities of defining data products due to varying interpretation and lack of standardization. He discusses three perspectives on data products: business, technical, and user.

Eric Sandosham advises smaller organizations on building data analytics capabilities effectively. Instead of focusing on efficiency, they should tackle flawed decision-making by examining assumptions about revenue, risk, and customer experience. The recommended approach is to start with Diagnostic Analytics to identify data gaps, then move to Predictive Analytics for tool standardization, and finally to Descriptive Analytics to enhance data use and storytelling.

Cassie Kozyrkov discusses teh evolving role of data scientists in the age of AI, emphasizing their critical importance in enterprise-scale automation. She argues that while AI tools evolve, the foundational skills of data scientists remain essential for ensuring effective, safe and reliable AI systems.

Learn if your data team really need gen AI to begin with!

A simple and jargon-free framework for data modeling.

Dylan Anderson emphasizes the critical importance of data modelling in today's rush towards new technologies. He argues that neglecting data modelling leads to short-term fixes and inefficiencies. Data modelling, spanning conceptual, logical, and physical phases, aligns business processes with data structures, enabling better decision-making and operational efficiency.

In the latest Data Engineering podcast episode, host Tobias Macey discussed the evolution and challenges of product management for data platforms with Praveen Gujar, LinkedIn’s Director of Product. They explored the importance of clean, reliable data and the role of product managers in bridging engineering and business teams.

Wannes Rosiers explores the distinction between data catalogs & data product catalogs within the context of modern data management. He highlights that while traditional data catalogs focus on data structure and definitions, data product catalogs expand this concept to include the entire user experience and functionality.

In this interesting read, Nicola talks about focusing on data governance strategies and how to align them with your organisation’s strategies.

Sarah Levy emphasizes the necessity for data teams to approach AI-driven analytics cautiously. With the launch of Tableau’s Einstein Copilot, the integration of generative AI promises enhanced data exploration. Sarah highlights that effective data modelling and governance are crucial for leveraging AI in analytics.

In this blog, Ananth Packkildurai explores the modern data stack's origins, rise, and fall. Initially emerging from various innovations and industry shifts, tools like Apache Yarn and S3 paved the way for scalable data processing.

Wasteman discusses best practices for creating accurate and reliable software-based financial systems. He covers fundamental financial definitions, goals of accounting systems, engineering principles, and best practices (use of integers for financial amounts, granularity for currency conversions, consistent rounding methodologies).

Théo Tortorici discusses how combining a semantic layer with AI data scientists facilitates easy data adoption. A semantic layer abstracts complex data into user-friendly terms, ensuring consistent metrics across an organization. He addresses business problems like consistent reporting, & reduced dependency on data teams.

The Seattle Data Guy emphasizes the challenges new data leaders face due to a lack of coaching and guidance. His blog offers advice on becoming successful in data leadership roles, such as understanding the business, communicating in business terms, and ensuring data quality.

In the latest episode of the Dataframed podcast, Shuang Li, Group Product Manager at Box, shared insights on building a scalable data platform with the host Adel Nehme. Shuang talked about cloud migration challenges, data ingestion, and processing pipelines.

Théo Tortorici discusses the benefits of integrating a semantic layer with AI data assistants for easy data adoption. He resolves business problems like inconsistent reporting and reliance on data teams, and technical issues such as data complexity and scalability.

Arup Nanda discusses various approaches to structuring data architecture teams based on evolving data landscapes. He contrasts traditional monolithic database designs with modern decoupled architectures like microservices and streaming systems.

Steven Macleod joins Joe to discuss the importance of operational data modeling, the flaws in current data modeling practices, and the challenges of data modeling at startups and in 3rd party tools.

Arslan Shahid guides through building an AI-powered “Auto-Analyst” system designed to assist data scientists and analysts by automating data analysis tasks. Arslan details the design and functionality of various agent, statistical analytics agent, machine learning agent, and code combiner agent.

Joe Reis compares the evolution of mixed martial arts (MMA) to the field of data modelling. He argues that MMA fighters must be proficient in multiple fighting styles, and data professionals must master various data modelling techniques.

Learn the two macro trends are forcing data modeling into a multi-disciplinary practice.

Daniel Lambert, M.Sc. discusses the critical importance of modern data architecture for successful AI projects. He highlights the exponential growth of data, necessitating agile and flexible architectures capable of handling vast amounts of information from diverse sources.

Mikkel Dengsøe discusses the critical role of data observability across all data pipeline stages. He emphasizes real-time issue detection in data ingestion, storage, transformation, and analysis to uphold data quality and reliability.

Hugo Lu draws attention to the emerging cracks in the Modern Data Stack (MDS), highlighting three key issues. Firstly, there’s a shift towards all-in-one data platforms like Databricks and Snowflake. Secondly, there’s a decline in support quality and increasing service bugs among MDS providers. Thirdly, competitive pressures are intensifying as major players like Snowflake encroach on their partners’ territories

Pedram Navid discusses the evolution of data roles, emphasizing the shift from data scientists to data engineers and now to data platform engineers. Initially, data scientists were key; then, data engineers emerged to handle data ingestion and transformation. Today, data platform engineers focus on creating frameworks allowing others to build custom pipelines.

Get a detailed perspective on the reasons to invest into semantic data layer with Olga.

This session highlights David’s journey in the data space with crucial insights into the role of semantics in the improving and optimizing AI-driven data analytics.

Willem Koenders emphasizes the importance of data governance despite it not being a primary concern for most startups. He discusses how startups face risks like data breaches and scalability issues amid rapid growth and limited resources.

Julien Hurault discusses the challenges of maintaining data pipelines and offers strategies for effective bug-fixing and backfilling. He emphasizes that pipelines often break due to various external factors. Julien stresses the importance of designing them with failure in mind. He outlines a three-step workflow for debugging: reverting data to the state at the time of failure, reproducing the bug in a safe environment, and backfilling correct data.

In this session, Nicola offers insights into the current state of data governance amidst the AI inclinations and shares actionable strategies to implement data governance in your orgs.

Timo Dechau, Manuel Cellier, and Tangi Gouez discuss the customer cycle, building internal data products, and the difference between a dashboard and data products, which are different from a product data analyst perspective.

Learn the power of operationalising analytical systems.

This battle is about the correct definition of “Data Product”. Juha represents a more maximalist position and Malcolm a more minimalist one. Both come up with arguments to support their views. The Distinguished Judges then make their determination – but, as always, the final judgement is yours.

Stephanie Kirmer discusses data privacy in AI development, focusing on data localizaiton. Data localization refers to storing data within the geographic location it was created to comply with various data privacy laws. Kirmer explains the importance of knowing where data is stored due to legal, contractual, and performance reasons.

When it comes to Data Governance, the one million dollar question is still “what’s the value? what do you deliver?”

Ergest Xheblati discusses how self-service analytics can alleviate the burden on data teams by reducing the influx of questions and requests. Xheblati argues that current self-service tools, like text-SQL, are ineffective because busy executives lack the time and expertise for proper analysis.

Kai Waehner discusses the Shift Left architecture as a solution to challenges in data integration, moving from batch processing and Reverse ETL to real-time data products. He emphasizes using Apache Kafka, Flink, and Iceberg to unify transactional and analytical workloads, reducing costs and improving flexibility.

Howard Chi discusses the evolution from traditional monolithic data systems to composable data systems, which support open standards and modular components. He says this shift allows for better interoperability, reusability, and customization, leveraging technologies like Apache Iceberg, Apache Arrow, and DataFusion.

In his latest blog, Oliver Wulveryck explored data contracts as pivotal tools in data management, particularly in facilitating AI development. He introduces Bitol’s Open Data Contract Standard and demonstrates its application through practical examples.

Sanjeev Mohan discussed how GenAI is revolutionizing data transformation in business. Traditional methods, reliant on manual coding, are inadequate for modern data demands. Mohan proposes a new approach using AI-driven “copilots” and agentic architecture. Copilots automate repetitive tasks and enhance data quality, allowing intelligent agents to focus on decision-making and problem-solving, he added.

Martin Chesbrough discusses the integration of operational and analytical data within data platforms, emphasizing the need for a cohesive approach to derive insights and improve organizational performance. He critiques the “data exhaust pipe” method, where data is indiscriminately dumped into warehouses.

Anna Ricci discusses the importance of hybrid encryption techniques in enhancing data security. She explains the basic encryption concepts and highlights the limitations of symmetric and asymmetric encryption. Hybrid encryption combines both methods, using asymmetric encryption for secure key exchange and symmetric encryption for efficient data processing.

Wannes Rosiers highlights the latest panel discussion on data products, emphasizing their definition, design, and implementation. The panel, featuring insights from industry experts like Jean-Georges Perrin and Nikki Bueno De Mesquita, defines data products as units of data with lifecycle management designed to create value and be reusable across different use cases.

In this blog, learn if data modeling enough?

Sven Balnojan PHD emphasizes the effective use of data in decision-making, contrasting modern data overload with historical strategic use by figures like Sun Tzu. He talks about four key rules: invest heavily in acquiring data, focus on getting the right data, use data to expand options, and finally, make quick decisions. He stresses that data should provide options rather than definitive answers.

Jan Meskens discusses the limitations of relying solely on technology for successful data projects. He argues that many organizations invest in advanced data platforms only to find that the platform doesn’t address the underlying issues, such as inefficient data team structures, bureaucratic processes, and a lack of data literacy.

This is the second part of the Snowflake series that discusses how Snowflake approaches monetisation, its limitations and puts forth the strategy to leverage data products atop Snowflake for improved data sharing, monetisation, and UX.

Andy Sawyer discusses various data warehouse modelling approaches to optimize data analysis and querying. He begins by highlighting the limitation of transactional databases for analytics and introduces data warehouses as a solution.

In the latest episode of the Women in Data Podcast, Karen Jean-Francois and Cecilia Oliveira delve into frequent challenges in data analytics and mitigation strategies. They discuss everything from comprehending requests and validating data accuracy to documenting workflows and delivering consistent updates in detail.

Juha joins Joe Reis to discuss the power of conceptual data modeling, why the data modeling world is broken today, data products, and much more.

Samir Sharma talks to Saurabh Gupta on the latest episode of The Data Strategy Show. They shared views about the alignment of data strategy with business goals, having a key executive sponsor, and the ability to deliver value in a short span. Saurabh also shared his views about the data strategy in zero-budget, with a focus on productization and software practices.

In the latest episode of the Data Engineering Podcast, Tobias Macey speaks with Nicola Aksham. In the episode, Aksham talks about the practical steps of building out a data governance practice within the organization. She also shared about her career choices and how she transitioned into data governance by accident.

Torsten Walbaum shares ten key lessons learned from working in data analytics roles at companies like Uber, Meta, and various startups. The insights include the importance of storytelling with data, developing strong business acumen, maintaining objectivity, combining data with primary research, and being sceptical of overly positive data.

In the latest blog post of The Seattle Data Guy, Benjamin Rogojan talked about various data modelling techniques used by companies to capture and analyze data for business insights. He focused on methods like Slowly Changing Dimension [SCD], particularly Type 2, which tracks changes over time by inserting new rows.

Learn how continuous integration and continuous deployment on tools like airflow is doomed to fail in the long-run.

This article walks us through the benefits of building data products with existing Snowflake to optimise the costs and the step-wise guide to doing so for effective monetisation of data and improved business outcomes.

In this session, David along with the co-speakers engage in a conversation around the classic individual contributor vs manager conundrum, migrating between warehouses, and reactive vs proactive data workflows.

Solmaz Shahalizadeh, former VP of Shopify, provides fifteen tips to enhance a startup’s data strategy. She emphasizes the importance of building a strong data infrastructure, collecting and unifying relevant data, and enabling teams to interact with data independently.

Wannes Rosiers discusses the rationale behind imec’s decision to buy rather than build their own data platform. He highlights the different perspectives of business, IT, and governance teams in such decisions and explains how imec's R&D and IT departments prefer purchasing modular tools to avoid reinventing the wheel.

Barr Moses discusses the evolution of data quality management in the context of the increasing importance of data and AI. She compares three methodologies: data testing, data monitoring, and data observability.

Benn Stancil discusses the common journey of data tool companies, highlighting how they start with innovative, customized solutions but often end up becoming traditional business intelligence tools due to market demands and investor pressures. He emphasizes that despite initial intentions to remain unique, these companies inevitably add features like dashboards and drag-and-drop visualizations to appeal to a broader audience.

This article talks about the importance of interoperability and focuses on the integration of tools like dbt alongside David’s experiences in building data stacks at his org, highlighting the value of open-source solutions for creating resilient and interoperable systems.

Peter Baumann talks about the importance of fostering a robust Data Culture within organizations to make their data strategies effective. He emphasizes that Data Culture, although hard to measure, is crucial for the successful implementation of data-related initiatives such as data governance, architecture, and analytics.

Learn the difference between data catalog and data cataloging through this insightful post of Olga.

Matt Weingarten discusses the need for realigning data teams around specific data products to improve ownership and technical alignment. Matt emphasizes on maintaining strong communication across teams to manage technical debt and ensure collective progress.

In his recent blog, Zach Wilson discusses the challenges data engineers face in getting promoted beyond the senior engineer level. He shares some practical strategies to overcome these barriers, including self-advocacy, upskilling, building strong relationships, and focusing on impactful projects. He emphasizes the need for data engineers to demonstrate broader technical and leadership skills to advance their careers.

Willem Koenders says that data maturity models measure an organization's development in data capabilities across dimensions like strategy, leadership, and technology. However, these models often overlook the actual adoption and implementation of these capabilities. He argues that having the right tools and processes is insufficient without ensuring they are actively used.

In this video, the host try a role-playing approach with Juha Korpela as a consultant describing the benefits - a somewhat out-of-touch CDAO at a medium-sized manufacturing company.

This article from Charlotte’s substack newsletter, offers practical tips on implementing, launching and optimizing data governance across your orgs.

Chris Tabb joins us to chat about something on his mind - platform gravity. The direction seems be the unified platform that combines applications, data, and algorithms. Chris is here to shed light on this trend.

This is a two-part brief to lay out the blueprint for ensuring a successful AI journey.

Salma Bakouk talks about the challenges of managing complexity and disorder, termed "data entropy," within modern data organizations. She explains entropy from thermodynamics and information theory perspectives, relating it to the increasing disarray in data systems as they scale.

Learn from David's experience and detailed perspective on why data leaders need to have an impact.

In this blog, Willem Koenders discusses the integration of Generative AI and data products in the pharmaceutical industry. He emphasizes their role in transforming data into actionable business decisions. Koenders also explore how Generative AI enhances data management processes, such as data acquisition, transformation, consumption, and operations.

In the latest episode of the Joe Reis Show, Gordon Wong emphasizes that data teams should create success metrics to guide their work. This means tracking things like customer engagement and usage, and measuring the value they add to the business. By doing this, data teams can show their impact, Wong added.

In the latest episode of Data Engineering Podcast, host Tobias Macey shares his experiences with release management reflecting upon the present challenges he faces. He also talked about various factors that contribute to the complexity of the problem.

In his latest post, Yerachmiel Feltzman discusses how to significantly reduce BigQuery storage costs by adjusting default configurations, particularly the “time travel” feature. Time travel allows data retrieval from previous states but increases storage costs.

Abraham Thomas discusses data asset pricing, noting its value depends on the use case and user. Unique data holds high value, while functional substitutes can impact uniqueness. Traditional software pricing models often fail, with structured volume and quality being more relevant. Wrapping data in software or services aids pricing.

In this blog, Lei Sun and Mohammad Islam unveiled Uber's strategy that helped the company achieve data categorization at scale leveraging various AI/ML techniques. They explored data categorization's crucial role in privacy and security, detailing their approach to auto-categorization with DataK9.

Maicol Nicolini provides a practical guide to performing Exploratory Data Analysis (EDA) for time series forecasting using Python. He discusses the importance of EDA in understanding data characteristics and informing feature engineering. Nicolini outlines a structured EDA template that includes steps like descriptive statistics, time plots, seasonal plots, box plots, time series decomposition, and lag analysis.