Surprise! One More for the Year π
Hello again! I truly thought the previous session would be the final Dueweekly issue of the year, but I found a little more time to sneak in one last release. Since I had the content ready, I figuredβwhy not?
I know this arrives just as the holiday season begins, so I want to wish you and your family a wonderful time together. Stay safe, enjoy the break, and take this time to recharge for a big year ahead.
Grab a seat and enjoy this final holiday edition. Cheers! π»
Architecture
1. Wallarm - DNS Load Balancing and Failover
- This excellent article tackles the topic of DNS, its role in load balancing, and how to ensure high availability and failover using this technique. This material is particularly interesting and valuable for anyone eager to learn more about networking, routing, and especially operating enormous infrastructure. The article provides a deep dive into using DNS for load balancing, failover, and the methodologies required to implement them. While reading the entire piece may be time-consuming, you will encounter new and valuable concepts, such as Rotational DNS Assignment, the Dump-Bag Technique, and Capacity-Aware DNS Assignment.
- You will also gain knowledge about the advantages of DNS Load Balancing and Failover and be provided with strategies to efficiently optimize your DNS implementation, covering how to select the right approach, methods for improvement, and other related considerations.
Career / Story
- This article was a fortunate find while scrolling through my news feed. First and foremost, I want to give a shout-out to Mr. Nguyen Phong (https://www.facebook.com/phonhnx/), the founder of CloudExam.pro, a platform dedicated to AWS Certification training and practice. I am truly impressed and inspired by his spirit in building this platform from the ground up. I hope this message reaches him, and I wish him great success. The article is written in Vietnamese and chronicles his journey and challenges while building the platform from scratch. Although there is currently no commercial alignment, the AWS bill he receivedβthough not substantialβhighlights a tough question for any builder or non-profit indie developer: How do you ensure the platform becomes successful, and is the price for this trade-off truly worth it?
- In this blog post, he introduces the features of his platform, details how he constructed the infrastructure on AWS, and outlines the next steps he needs to take. If you are an indie hacker, an open-source developer, or an entrepreneur building a business from scratch in the technology sector, I believe you will both learn from and empathize with the challenges one must face when stepping into this marketplace.
Data Engineer
1. Blog - The 9 Ways to Move Data Kafka β Iceberg
2. Blog - Kafka topics to Iceberg Tables
- Both articles focus on the topic of moving data from Kafka into a Data Lakehouse environment, specifically leveraging Iceberg, which is a leading modern data stack technology today. First, the authors highlight the issues and challenges encountered when migrating data out of Kafka, addressing concerns like Data Freshness, Table Maintenance, Partitioning, and other related complexities. They then explore various methods for dealing with and maintaining Kafka data by copying it to a Lakehouse format like Iceberg.
- Throughout the articles, you will gain further insight into the appropriate frameworks and open-source tools that can help address this use case. If you work with modern data stacks like Kafka and Iceberg, reading these articles is highly recommended, as they provide useful information and solutions for solving similar problems in your own environment.
Kubernetes
1. RedHat - Global Load Balancer Approaches π (Recommended)
- This article presents an overview of Global Load Balancer (GLB) solutions when designing a multi-cluster architecture, particularly within Kubernetes and OpenShift environments. It explores various methods for approaching GLB, covering design methodologies and automation techniques, which is highly beneficial for those seeking to learn.
- You will be introduced to two main GLB options: DNS-Based and Anycast-Based. The DNS-Based method aligns with what Iβve previously discussed (refer to the article The Story of Mine about Multi-Region Architecture for more examples and proposals). This approach allows you to adapt any DNS management service (e.g., Route 53, Cloudflare) for domain resolution, though it commonly encounters challenges related to TTL (Time-To-Live) and unbalanced traffic patterns, which are often observed during DNS-Based GLB configuration.
- The article also discusses the Anycast-Based approach, which involves new networking techniques like IP/BGP (https://www.cloudflare.com/learning/security/glossary/what-is-bgp/) and Anycast Cloud Services (e.g., AWS Global Accelerator (AGA) or Google Cloud Network Load Balancer). The author links this approach to MetalLB (https://metallb.io/) for BGP routing, providing excellent illustrations for easier comprehension. I believe this blog will furnish you with numerous keywords and methodologies for approaching and researching your GLB implementation.
- Furthermore, for a Cloud Native approach within Kubernetes, the article again highlights k8gbβthe βstarβ of Kubernetes GLBβalong with another implementation developed by Red Hat called global-load-balancer-operator (https://github.com/redhat-cop/global-load-balancer-operator). Both tools are highly effective and compatible with various setups. By delving deeper into these tools, you will gain further expertise in GLB and routing massive traffic between regions for building Multi-Cluster Kubernetes implementations.
2. Youtube - Ensuring high availability with global load balancing in Kubernetes - Matthias Hauber π (Recommended)
- This is an excellent video from the ContainerDays Conference (https://www.containerdays.io/), providing a session and insights on how to tackle problems related to DNS, CDN, and Global Load Balancing (GLB) for Kubernetes, especially in a multi-cluster deployment. The presenter outlines several potential approaches, such as Cluster Mesh (not a true GLB solution) and CDN, before focusing on k8gb as the current best option for this situation, given its foundation on CoreDNS and ExternalDNS.
- During the video, you will learn how traffic can be distributed globally to the nearest Kubernetes cluster, the challenges encountered in this process, and a demonstration that proves the concept works, providing effective geographic load balancing. Watching this video will offer valuable insights and knowledge on adapting these concepts and understanding how large-scale Kubernetes clusters manage traffic globally using DNS.
3. Youtube - Kubernetes Multi Cluster DNS with External DNS, CoreDNS and Etcd
- This is a great video that offers excellent insight into multi-cluster DNS, a critical and engaging topic in Kubernetes architecture. You will learn the rationale for building a unified DNS across multiple clusters and identify the key technology stack required for your Kubernetes DNS solution.
- The video comprehensively covers DNS and networking within Kubernetes, spanning external, internal, and local resolution, with a focus on CoreDNS (the current backbone of Kubernetes DNS). Additionally, it provides a tutorial on setting up ExternalDNS to seamlessly route external traffic to internal services, integrated with CoreDNS and Etcd.
- The source code for this implementation is available here: https://github.com/morrismusumi/kubernetes/tree/main/external-dns.
4. Thenewstack - An End-to-End Cloud Native Observability Framework
- Full-Stack Observability: This article is an excellent resource for anyone looking to architect a comprehensive observability stack. It is a critical topic for professionals focused on on-call rotations and incident response. The content emphasizes native Kubernetes frameworks and their specific roles within the ecosystemβfor example, implementing distributed tracing using OpenTelemetry and the OTLP (OpenTelemetry Protocol).
- Implementation & Automation: The guide also showcases best practices for managing Kubernetes via Cloud Provider APIs and integrating observability within CI/CD pipeline stages. Additionally, it explores advanced workflows, such as exporting logs to AI/ML models for automated root-cause analysis and remediation.