Data Engineering
525 skills in Data & AI > Data Engineering
devops
Deploy and manage cloud infrastructure on Cloudflare (Workers, R2, D1, KV, Pages, Durable Objects, Browser Rendering), Docker containers, and Google Cloud Platform (Compute Engine, GKE, Cloud Run, App Engine, Cloud Storage). Use when deploying serverless functions to the edge, configuring edge computing solutions, managing Docker containers and images, setting up CI/CD pipelines, optimizing cloud infrastructure costs, implementing global caching strategies, working with cloud databases, or building cloud-native applications.
media-processing
Process multimedia files with FFmpeg (video/audio encoding, conversion, streaming, filtering, hardware acceleration) and ImageMagick (image manipulation, format conversion, batch processing, effects, composition). Use when converting media formats, encoding videos with specific codecs (H.264, H.265, VP9), resizing/cropping images, extracting audio from video, applying filters and effects, optimizing file sizes, creating streaming manifests (HLS/DASH), generating thumbnails, batch processing images, creating composite images, or implementing media processing pipelines. Supports 100+ formats, hardware acceleration (NVENC, QSV), and complex filtergraphs.
databases
Work with MongoDB (document database, BSON documents, aggregation pipelines, Atlas cloud) and PostgreSQL (relational database, SQL queries, psql CLI, pgAdmin). Use when designing database schemas, writing queries and aggregations, optimizing indexes for performance, performing database migrations, configuring replication and sharding, implementing backup and restore strategies, managing database users and permissions, analyzing query performance, or administering production databases.
github-archive
Investigate GitHub security incidents using tamper-proof GitHub Archive data via BigQuery. Use when verifying repository activity claims, recovering deleted PRs/branches/tags/repos, attributing actions to actors, or reconstructing attack timelines. Provides immutable forensic evidence of all public GitHub events since 2011.
compression-optimizer
Compression Optimizer - Auto-activating skill for Data Pipelines. Triggers on: compression optimizer, compression optimizer Part of the Data Pipelines skill category.
data-catalog-updater
Data Catalog Updater - Auto-activating skill for Data Pipelines. Triggers on: data catalog updater, data catalog updater Part of the Data Pipelines skill category.
data-partitioner
Data Partitioner - Auto-activating skill for Data Pipelines. Triggers on: data partitioner, data partitioner Part of the Data Pipelines skill category.
dbt-model-generator
Dbt Model Generator - Auto-activating skill for Data Pipelines. Triggers on: dbt model generator, dbt model generator Part of the Data Pipelines skill category.
data-lineage-tracker
Data Lineage Tracker - Auto-activating skill for Data Pipelines. Triggers on: data lineage tracker, data lineage tracker Part of the Data Pipelines skill category.
dagster-pipeline-creator
Dagster Pipeline Creator - Auto-activating skill for Data Pipelines. Triggers on: dagster pipeline creator, dagster pipeline creator Part of the Data Pipelines skill category.
airflow-dag-generator
Airflow Dag Generator - Auto-activating skill for Data Pipelines. Triggers on: airflow dag generator, airflow dag generator Part of the Data Pipelines skill category.
data-quality-checker
Data Quality Checker - Auto-activating skill for Data Pipelines. Triggers on: data quality checker, data quality checker Part of the Data Pipelines skill category.
beam-pipeline-builder
Beam Pipeline Builder - Auto-activating skill for Data Pipelines. Triggers on: beam pipeline builder, beam pipeline builder Part of the Data Pipelines skill category.
flink-job-creator
Flink Job Creator - Auto-activating skill for Data Pipelines. Triggers on: flink job creator, flink job creator Part of the Data Pipelines skill category.
vertex-ai-pipeline-creator
Vertex Ai Pipeline Creator - Auto-activating skill for GCP Skills. Triggers on: vertex ai pipeline creator, vertex ai pipeline creator Part of the GCP Skills skill category.
prefect-flow-builder
Prefect Flow Builder - Auto-activating skill for Data Pipelines. Triggers on: prefect flow builder, prefect flow builder Part of the Data Pipelines skill category.
klingai-async-workflows
Build asynchronous video generation workflows with Kling AI. Use when integrating video generation into larger systems or pipelines. Trigger with phrases like 'klingai async', 'kling ai workflow', 'klingai pipeline', 'async video generation'.
airflow-operator-creator
Airflow Operator Creator - Auto-activating skill for Data Pipelines. Triggers on: airflow operator creator, airflow operator creator Part of the Data Pipelines skill category.
windsurf-cicd-github-actions
Generate and maintain GitHub Actions with Cascade assistance. Activate when users mention "github actions", "ci/cd pipeline", "workflow automation", "continuous integration", or "deployment pipeline". Handles CI/CD configuration with AI assistance. Use when working with windsurf cicd github actions functionality. Trigger with phrases like "windsurf cicd github actions", "windsurf actions", "windsurf".
klingai-batch-processing
Process multiple video generation requests efficiently with Kling AI. Use when generating multiple videos or building content pipelines. Trigger with phrases like 'klingai batch', 'kling ai bulk', 'multiple videos klingai', 'klingai parallel generation'.