• png logo

How to use Databricks Lakehouse and Responsible AI

Gain practical skills in using the Databricks Lakehouse platform and knowledge of AI ethics to meet the demands of the industry.

Two colleagues in an office looking at a computer screen with the woman pointing at the screen

How to use Databricks Lakehouse and Responsible AI

  • 4 weeks

  • 4 hours per week

  • Digital certificate when eligible

  • Intermediate level

Find out more about how to join this course

  • Duration

    4 weeks
  • Weekly study

    4 hours
  • 100% online

    How it works
  • Unlimited subscription

    $244.99 for a whole yearLearn more

Master the fundamentals of the Databricks Lakehouse platform

On this four-week course, you’ll delve into the Databricks Lakehouse platform to understand its functionality and architecture.

You’ll gain hands-on experience using the platform to develop practical skills in managing large-scale data operations, developing AI solutions, and implementing ethical AI practices.

Armed with these skills, you’ll be able to build robust, responsible AI systems. This will make you a valuable asset in the rapidly evolving field of AI and data science.

Gain an understanding of data transformation and Data Live tables

You’ll delve into data transformation and pipelines to understand how to develop efficient data pipelines using Data Live tables and Jobs.

Learning the best practices, you’ll discover how to harness data transformation for effective decision-making.

Unpack responsible Generative AI

As the AI industry continues to evolve, it’s important to consider how users can do this responsibly.

On week three of the course, you’ll explore AI ethics in detail and unpack real-world examples so you can evaluate the ethical implications of GenAI technologies.

Gain fundamental skills in Unity Catalog

Finally, you’ll learn how to use data governance strategies with Unity Catalog to help you master advanced skills in Databricks Lakehouse.

Learning from industry experts and through hands-on exercises, you’ll finish the course with the technical skills and ethical considerations needed in the AI industry.

Syllabus

  • Week 1

    Databricks Lakehouse Platform Fundamentals

    • Databricks Lakehouse Platform

      Learn to leverage Databricks and AI for advanced data analytics. Master the Lakehouse platform, build end-to-end ML pipelines, and responsibly integrate LLMs into your workflows. Gain hands-on experience with clusters and more.

    • Data Transformation with Apache Spark

      Explore data transformation using Apache Spark in Databricks. Learn to set up development environments, use CLI tools, and work with notebooks. Covers multi-language support, repos, and practical exercises in various languages.

    • Data Management with Delta Lake

      Learn data management with Delta Lake. Topics: Spark SQL, Catalog Explorer, table creation, querying external sources, Delta Lake pipelines, ACID transactions, Z-Ordering. Includes lab, tutorials, and quiz on RStudio and COPY INTO

  • Week 2

    Data Transformation and Pipelines

    • Data Pipelines with Delta Live Tables

      Explore Delta Live Tables for automated data pipelines. Learn key components, pipeline types, Auto Loader configuration, event querying, and end-to-end examples. Includes vacuum and garbage collection, plus hands-on labs

    • Workloads with Jobs

      Explore Databricks Jobs for orchestrating workloads. Learn about multi-task workflows, dependencies, job history, dashboards, and failure handling. Includes demos, labs, and quizzes to reinforce concepts and practical application.

    • Data Access with Unity Catalog

      Explore Unity Catalog for data access and governance. Learn about catalogs, metastores, and best practices. Hands-on with Python quickstart and object security. Includes external lab, dynamic catalog building, reflection, and quiz

  • Week 3

    Responsible Generative AI

    • AI Ethics of Generative Models

      Explore the ethical implications of generative AI. Learn about profit sharing, tragedy of the commons, game theory, and regulatory challenges. Examine issues of bias, negative externalities, and perfect competition.

    • Evaluating Real-World Performance of LLMs

      Learn to assess Large Language Model performance in real-world scenarios. Implement Elo rating systems in Python, Rust, R, and Julia. Gain hands-on experience through coding labs.

    • Exploring Production LLM Workflows

      Dive into practical LLM deployment using Lorax and Skypilot. Understand Ludwig for model fine-tuning. Apply these tools to fine-tune Mistral-7b and launch Mixtral.

  • Week 4

    Local LLMOps

    • Getting Started with local models

      Explore local AI models with llamafile. Learn to set up and run models like Mixtral, understand system metrics, and get hands-on experience. Dive into key concepts like Whisper.cpp.

    • Getting Started with Rust Candle

      Lesson 2 introduces Rust Candle, covering basic implementation, Starcoder exploration, and Whisper transcription. It delves into remote AWS development, AI security topics like sleeper agents and data poisoning.

    • Using Rust Candle

      Explore Rust Candle for LLM applications. Learn serverless inference, CLI & chat inference, and using Star Coder. Implement Rust Candle on AWS GPU. Includes readings, reflections, and a quiz to reinforce learning.

When would you like to start?

Start straight away and join a global classroom of learners. If the course hasn’t started yet you’ll see the future date listed below.

  • Available now

Learning on this course

On every step of the course you can meet other learners, share your ideas and join in with active discussions in the comments.

What will you achieve?

By the end of the course, you‘ll be able to...

  • Create solutions that use Databricks for data engineering and ML workloads
  • Create and design ML pipelines
  • Develop solutions with Llamafile and other local LLMs like Mixtral
  • Critique ethical issues with the use of Generative AI including the tragegy of the commons and negative externalities.

Who is the course for?

This course is designed for data professionals, software engineers, and AI enthusiasts looking to master the Databricks Lakehouse Platform and explore responsible AI practices.

It’s ideal for those seeking to advance their careers in data engineering, machine learning, and AI ethics.

The course serves professionals in tech, finance, healthcare, and research sectors who want to implement scalable data solutions and understand the ethical implications of generative AI.

Who will you learn with?

Noah Gift

Founder of Pragmatic AI Labs & Executive in Residence at Duke MIDS and Duke AIPI. Former Bay Area CTO and author of multiple O'Reilly books.

Who developed the course?

svg logo

Pragmatic AI Labs

Learn from leading instructors from top universities with real-world industry experience.

Achieve your goals with practical courses designed with a vocational focus. Our engaging programs provide you with the knowledge and tools to succeed in your career and have a positive impact.

Ways to learn

Buy this course

Subscribe & save

Limited access

Choose the best way to learn for you!

$79/one-off payment

$244.99 for a whole year

Automatically renews

Free

Fulfill your current learning needDevelop skills to further your careerSample the course materials
Access to this courseticktick

Access expires 5 Mar 2025

Access to 1,000+ coursescrosstickcross
Learn at your own paceticktickcross
Discuss your learning in commentstickticktick
Tests to check your learningticktickcross
Certificate when you're eligiblePrinted and digitalDigital onlycross
Continue & Upgrade

Cancel for free anytime

Ways to learn

Choose the best way to learn for you!

Subscribe & save

$244.99 for a whole year

Automatically renews

Develop skills to further your career

  • Access to this course
  • Access to 1,000+ courses
  • Learn at your own pace
  • Discuss your learning in comments
  • Tests to boost your learning
  • Digital certificate when you're eligible

Cancel for free anytime

Buy this course

$79/one-off payment

Fulfill your current learning need

  • Access to this course
  • Learn at your own pace
  • Discuss your learning in comments
  • Tests to boost your learning
  • Printed and digital certificate when you’re eligible

Limited access

Free

Sample the course materials

  • Access expires 5 Mar 2025

Find out more about certificates, Unlimited or buying a course (Upgrades)

Sale price available until 3 March 2025 at 23:59 (UTC). T&Cs apply.

Find out more about certificates, Unlimited or buying a course (Upgrades)

Sale price available until 3 March 2025 at 23:59 (UTC). T&Cs apply.

Learning on FutureLearn

Your learning, your rules

  • Courses are split into weeks, activities, and steps to help you keep track of your learning
  • Learn through a mix of bite-sized videos, long- and short-form articles, audio, and practical activities
  • Stay motivated by using the Progress page to keep track of your step completion and assessment scores

Join a global classroom

  • Experience the power of social learning, and get inspired by an international network of learners
  • Share ideas with your peers and course educators on every step of the course
  • Join the conversation by reading, @ing, liking, bookmarking, and replying to comments from others

Map your progress

  • As you work through the course, use notifications and the Progress page to guide your learning
  • Whenever you’re ready, mark each step as complete, you’re in control
  • Complete 90% of course steps and all of the assessments to earn your certificate

Want to know more about learning on FutureLearn? Using FutureLearn

Do you know someone who'd love this course? Tell them about it...