Das Digital Digest

Overview

Initializing search

Home
AI & ML
Data Engineering & Analytics
Cloud & DevOps
Enterprise Solutions
Projects
Resources

Das Digital Digest

Home
AI & ML
AI & ML
- History
- Running Stable Diffusion Locally
Data Engineering & Analytics
Data Engineering & Analytics
- Apache Airflow
  Apache Airflow
- Spark & Databricks
  Spark & Databricks
  - Spark
    Spark
    
    Concepts
    
    Narrow vs Wide
    
    Architecture
    
    Cache & Persist
    
    Broadcast Variables
    
    Data Skew
    
    Missing Values
    
    Windows Functions
    
    Partitioning
    
    RDD vs DataFrame
    
    Scala Guide
    
    Interview Q&A
    
    Shuffle
    
    Databases & Catalogs
    
    Common Topics
    
    Hive Integration
    
    Setup Guide
  - Databricks
    Databricks
    
    Concepts
    
    Catalogs
    
    Authentication
    
    ADLS Mount
    
    Secret Scope
    
    SQL
    
    Magic Commands
    
    Delta Lake
    
    Project Example
  - ADLS Integration
    ADLS Integration
    
    Setup
    
    Summary
  - Hive
    Hive
    
    Concepts
- Stream Processing
  Stream Processing
  - Introduction
  - Event Hubs
    Event Hubs
    
    Overview
    
    vs Kafka
    
    Hello EventHubs
    
    Local Emulator
  - Processing Options
  - Kinesis Integration
- SQL
  SQL
  - Basics
  - Advanced Topics
  - Windows Functions
  - SSIS
    SSIS
    
    Overview
    
    Input/Output
    
    Flat File Source
    
    Project 1
    
    Project 2
  - SSRS
  - Versions
  - DBT
    DBT
    
    Overview
    
    Setup Project
    
    Connections
Cloud & DevOps
Cloud & DevOps
- Microsoft Fabric
  Microsoft Fabric
  - Getting Started
  - Features
    Features
    
    Data Warehouse
    
    Real-Time Analytics
    
    Data Science
    
    Data Factory
  - Development
    Development
    
    Direct Lake
    
    PySpark & SQL
    
    Pandas vs Spark
    
    DataFrame Inspection
    
    KQL
    
    Spark Streaming
  - Projects
    Projects
    
    ETL with OPG
    
    ETL with PySpark
    
    E2E Project
  - Administration
  - Q&A
- DevOps
  DevOps
  - GitHub Actions
    GitHub Actions
    
    Concepts
    
    Hello Actions
    
    Sample Workflows
  - Azure DevOps
    Azure DevOps
    
    Pipelines
    
    Self-Hosted Agents
    Self-Hosted Agents
    
    Windows
    
    Windows Container
    
    Linux Container
  - ADF CI/CD
    ADF CI/CD
    
    Overview
    
    Implementation
  - Tools Comparison
  - Scenarios
  - Bicep
- Containers
  Containers
  - Docker & Kubernetes
    Docker & Kubernetes
    
    Docker
    Docker
    
    Concepts
    
    VS Code Setup
    
    Spark & Hadoop
    Spark & Hadoop
    
    Overview
    
    PySpark
    
    Bitnami Cluster
    
    With MSSQL
    
    Hadoop
    Hadoop
    
    Concepts
    
    Cluster Setup
    
    Hive
    Hive
    
    Concepts
    
    With Postgres
    
    With MySQL
    
    Apache Setup
  - MongoDB
  - Kafka
  - Airflow
- Azure Services
  Azure Services
  - Synapse
    Synapse
    
    Concepts
    
    Pools
    
    ETL Pipelines
    
    Copy Data Tool
    
    Integration Runtime
    
    Database Types
    
    Lake DB
    
    Storage Evolution
    
    CETAS
  - Data Factory
    Data Factory
    
    Projects
    
    Local to ADLS
    
    PySpark Warehouse
    
    REST API
    
    Monitoring
    
    Copy Pipeline
  - Q&A
Enterprise Solutions
Enterprise Solutions
- Microsoft 365
  Microsoft 365
  - SharePoint
    SharePoint
    
    Events
    
    Forms vs PowerApps
    
    Mini Role
    
    Evolution
    
    vs Other ECM
    
    Farm
    Farm
    
    Consolidation
    
    2007 Upgrade
    
    2016 Upgrade
    
    Migration
    Migration
    
    SPMT
    
    Documentum
    
    WSS3
    
    Integration
    Integration
    
    OAuth Python
  - Licensing
- Power Platform
  Power Platform
  - Hello Platform
  - Admin Central
  - Components
    Components
    
    Hello Dataverse
    
    Hello Dynamics
    
    Model Apps
    
    Power Pages
    
    Power Automate
    
    Calculation Groups
  - Features
    Features
    
    Document Intelligence
    Document Intelligence
    
    Overview
    
    Automation
    
    With Cognitive Search
    
    Q&A
    
    Syntex
    Syntex
    
    Enable
    
    Document Library
    
    ECM Capture
  - Integration
    Integration
    
    Custom Connectors
    
    On-Premise Gateway
  - Security
  - Q&A
- MongoDB
  MongoDB
- Python
  Python
  - Basics
  - Data Structures
    Data Structures
    
    Sets
    
    Tuples
    Tuples
    
    Basic
    
    Advanced
    
    Lists
    
    Dictionaries
  - Functions
    Functions
    
    Lambda
    
    Built-in
    
    Overview
    
    Summary
  - Control Flow
    Control Flow
    
    If-Elif-Else
    
    Operators
    
    For Loops
    
    Enumerate
    
    Range
  - Advanced
    Advanced
    
    With Statement
    
    Testing
    
    Main Function
    
    Decorators
    
    Arguments
    
    Error Handling
    
    Diff & Patch
    
    Debugging
    
    Formatting
  - Integrations
    Integrations
    
    Linux
    
    PySpark
    
    Graph API
  - Scripts
Projects
Projects
- Azure Sky Weather
  Azure Sky Weather
  - Overview
  - Ingestion
    Ingestion
    
    HTTP Triggered
    HTTP Triggered
    
    Functions
    
    Azure Functions
    
    Timer Triggered
  - Transformation
- StreamKraft
  StreamKraft
  - Overview
- Sparkzure
  Sparkzure
  - Overview
- JSON Projects
  JSON Projects
  - Validator
    Validator
    
    Python
    
    Azure Function
  - Azure SDK
  - To Parquet
  - To Hive
- ETL Projects
  ETL Projects
- Trading
  Trading
  - SPY ETF Recommender
- Setup Guides
  Setup Guides
Resources
Resources
- MkDocs
  MkDocs
  - Github Deployment
- Miscellaneous
  Miscellaneous
  - Markdown
    Markdown
    
    Basics
    
    Colors
    
    PDF Export
  - Cloud
    Cloud
    
    GCP Usage
    
    Azure Budget
  - System
    System
    
    Ubuntu Benefits
    
    VMware Free
    
    VS Code Tips
    
    Background Apps
  - Data
    Data
    
    Datasets
    
    Fact vs Dimension

Overview

layout: default title: spark-hive-hadoop parent: Docker nav_order: 4 has_children: true