185: Workflow Orchestrators
Intro topic: Asymmetric ReturnsNews/Links:NanoChat by Andrej Karpathyhttps://github.com/karpathy/nanochatPydantic AIhttps://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/1000th Starlink this yearhttps://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/ChatGPT Apps SDKhttps://openai.com/index/introducing-apps-in-chatgpt/Book of the ShowPatrickThe Will of the Many by James Islingtonhttps://amzn.to/43IfU8QJasonInterview with DHH (Founder of Ruby on Rails)https://www.youtube.com/watch?v=vagyIcmIGOQPatreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrickFactoriohttps://www.factorio.com/ Jasonnip.io Topic: Workflow OrchestratorsWhyBatch jobs (embarrassingly parallel)Long-running tasks (e.g. transcoding video)Checkpointing/resumingHowMessage QueuesContainerizationWorker Pools & AutoscalingHistory & BackfillSteps to run workflows:Containerize the workflow definition and send to the cloudContainerize all the individual tasksSubmit job(s)ExamplesAirflowLegacy but dominantDagsterGreat UX for python developersTemporal: https://temporal.io/The new hotnessRayLow-level but very powerfulKubeflowDesigned for ML workflows, integrated dashboard
★ Support this podcast on Patreon ★
]]
---
Programming Throwdown
Patrick Wheeler and Jason Gauci
184: Asynchronous Programming
186: Becoming a Manager
Download Audio File
Intro topic: Asymmetric Returns
News/Links:
- NanoChat by Andrej Karpathy
- https://github.com/karpathy/nanochat
- Pydantic AI
- https://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/
- 1000th Starlink this year
- https://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/
- ChatGPT Apps SDK
- https://openai.com/index/introducing-apps-in-chatgpt/
Book of the Show
- Patrick
- The Will of the Many by James Islington
- https://amzn.to/43IfU8Q
- Jason
- Interview with DHH (Founder of Ruby on Rails)
- https://www.youtube.com/watch?v=vagyIcmIGOQ
Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h
Tool of the Show
- Patrick
- Factorio
- https://www.factorio.com/
- Jason
- nip.io
Topic: Workflow Orchestrators
- Why
- Batch jobs (embarrassingly parallel)
- Long-running tasks (e.g. transcoding video)
- Checkpointing/resuming
- How
- Message Queues
- Containerization
- Worker Pools & Autoscaling
- History & Backfill
- Steps to run workflows:
- Containerize the workflow definition and send to the cloud
- Containerize all the individual tasks
- Submit job(s)
- Examples
- Airflow
- Legacy but dominant
- Dagster
- Great UX for python developers
- Temporal: https://temporal.io/
- The new hotness
- Ray
- Low-level but very powerful
- Kubeflow
- Designed for ML workflows, integrated dashboard
---
[Original source](https://www.programmingthrowdown.com/episodes/185-workflow-orchestrators)