Quote
Hello guy, Itβs me again, after two week read and explore about tech world, letβs see what topic I actually follow and maybe you can digest another from mine. Hope you can find useful information and get more innovation, now letβs digest.
Architecture
- Blogger tells about the story behind the big company like Netflix about how to control spike traffic and Infrastructure strategies when they applied it for their AWS Cloud
- They deal with couple of technologies from AWS, such as Open Connect - Global CDN, S3, EC2 Spot instance, Eureka, Ribbon, Chao Engineer, Serverless and moreover to handle huge traffic and build success nowadays
- Netflix doesnβt have own Datacenter but they leverage AWS for becoming own architecture for them to building the Netflix platform, with combination many AI Driven, they can build scaling system in second for spike connection.
- If you want to find more about Netflix Architecture, You can enjoy this article. BTW, you can find to Giaosucan - Netflix Content to learn more about Netflix in Vietnamese - Big shout out to Giaosucan
2. Medium - How to Build a Scalable and Efficient Internal Developer Platform (IDP) from Scratch
- Learn how to setup completely IDP, and how to organization IDP for your project to reduce complexity for developer
- Again and again, IDA is new platform and generation for Platform Engineer or DevOps to focus and try to build for isolating multiple tech component inside only Portal and leverage itself to build product
- Article covers multiple layers inside IDP for building each components inside and suggest couple of IDP from big tech, and covey new future trends for IDP in next stage
3. Dev.to - 9 Caching Strategies for System Design Interviews
- Introduce and emphasize Caching is one of important things in software design
- List some caching levels, styles and strategies for easier approaching who want to get closer with caching
- Suggest you about useful keyword for researching and exploring with Cache
4. Medium - Top 10 System Design Concepts Every Programmer should learn
5. Dev.to - The Software Design /System Design Interview Preparation RoadMap (with Resources)
- Articles bring you most of concept inside software and design system whatever from small to large maybe applied
- Go through each system design, you will have more information by explanation of author about each type, what use-case used for and what keyword you can dive deep to explore and deal with your idea.
- In another ways, you can have couple of suggestion for starting SA by following roadmap and resources attachment
Containerization
1. Medium - Stop Managing Docker Like Itβs 2020: Three Tools That Changed Everything
- Introduce a couple of new tools for managing and diving deepest into docker image for explore and inspect about what else can optimize with your Docker Image. These tools include Lazy Docker, Dive and Watchtower
- In good ways, you can use
Watchtowerto automation update new image for your docker, it will build new strategy to GitOps and you should follow this way.
Data Engineer
1. Airbyte - Data Pipeline vs. ETL: Optimize Data Flow (Beginnerβs Guide)
- The comparison btw Data pipeline and ETL with detail explaining and key differences, you can easier understand use case for using once of them
- Give you more information each data stage in progress for Data Pipeline and ETL
2. Airbyte - What is Data Extraction? Data Extraction Tools and Techniques
- Explain the theory of Data Extraction and list main stage of this progress
- Give you couple of methods and techniques related Data Extraction
- Comparison btw Data Extraction and Data Mining
- Challenge and how we can handle extraction for empowering business
- Tools and usage for applying ETL for Data Extraction, and learn how to automate that one
- Relate to best practices in Data Extraction
- In the end, you will have couple of example real world with applying Data Extraction progress
3. Airbyte - What is Data Pipeline: Benefits, Types, & Examples
- Explain the theory of Data Pipeline with couple of benefits of them and what components inside data pipelines, such as Origin, Destination, Dataflow, Storage, Workflow and Monitoring
- List a couple of data pipeline styles with explaining
- Covey some example for relating what use-case should use data pipelines
4. Medium - dbt in Production: Airflow with Kubernetes
- Learn about challenge when use DBT inside the scaling system, with many things problems can be occur, such as conflict package, managed airflow, β¦
- They want to bring DBT with concept Airflow and Kubernetes, because they will use
KubernetesPodOperatorfor scale up DBT and use this pod created new DBT for each task - List about couples of tech debt need to solve when use this strategies for logging and save artifact
5. Medium - Airflow: Why is nothing working?
- We will learn about problem of Bluecore when setup airflow and use
SubDagOperatorsto cause Deadlock for Data Pipeline with scale upCelerybut not solving - They try to explain why they configure wrong and what happen inside their system and give the solution for tackling problem. From step to simple disassemble, they know about the SubDagOperator mess up with executing child Dag and parrent Dag should wait before child completely and child dags not finish and cause deadlock.
- They replace with new custom Operator for executing single task and anything worked
6. Medium - Weβre All Using Airflow Wrong and How to Fix It
- List some challenge and problems when work with Airflow, such as Functional task with operator, hard to debug with Airflow Operator and architecture of dags when use worker for executing (Refresh installing progress anytime).
- For fix problem, they try to use
KubernetesOperatorfor trigger creating new pod for multitasking in multiple different requirements and prevent conflict, with detailing evidence
7. Youtube - Orchestrating Airbyte and dbt with Airflow
- Video is a tutorial which try to help you build Dag in Airflow used both technology Airbyte and DBT
- Write to use
DBTin good way with use BashExecutor inside Airflow and we manage the ETL inside Airbyte - But itβs is use Docker and maybe different when you use it with Kubernetes
8. Airflow - Best Practices of Airflow
9. Medium - Top 10 Apache Airflow Best Practices for Data Engineers
- These articles cover about how to implement and structure your Airflow in good way to ensure performance and efficiency to handle multiple situations with this tool
- Tricks and tips are written DAGs with optimize and leverage to multiple executor, providers to boost workflow
Developer
1. Python - Package index mirrors and caches
- List of couple techniques of Python to keep your package in Offline mode via cache, you can install back again your package with cache priority
- Introduce command and how to use PIp for caching and there are some project out there can help you do some stuff like this.
2. Blog - DIY node_modules cache for Docker in your CI
- Learn about strategy to use one base image to store node_module and reuse on the step build and release for new one, you can optimize time to read from base docker image locally instead from npm
- The author will share how to implement step by step to retrieve that one with Dockerfile related
3. Gradle - Speed Up Maven Build
- One of another blogs talk about cache in build time because this implementation will help you reduce the complexity, time for rebuild whole project again
- Use some techniques of
Mavenextension via Develocity to handle logics and boost time to build your project depend on this technique
4. Medium - Advanced Caching Strategies in Node.js: Boosting Performance and Scalability
- Introduce some caching strategies in Node.js to help you reduce the load on databases and external APIs, caching allows applications to deliver data faster, handle larger traffic loads, and improve the overall user experience.
- You can learn multiple tools and how to leverage these tools to implement cache strategies and do some good way integration with Nodejs project
5. CI/CD caching with Bitrise: What is a cache, and why should you care about caching?
6. Bitrise - CI/CD caching with Bitrise: Dependency caching with Bitrise
- Deliver some idea and reason why you should use cache for CI/CD progress, If you use this gadget, you will boost your build time to next gen and thatβs really good way to rapid release for your project
- Bitrise suggest you about how to use dependency caching, thatβs really one of caching strategies to helpful any build for retrieve the local cache depend on key-value. One of thing consideration is you should have bit huge memory and cpu for loading your cache into your application
Kubernetes
1. GitGuardian - How to Handle Secrets in Helm
- Article is a guideline to help you imagine how to integrate helm-secrets into deployments for protecting secrets variables with cross team used GitOps Strategies
- Use
helm-secretswithSOPSfor encrypting/decrypting in bidirection - Use external operator in Kubenetes Cluster for automatically retrieves secrets from secret managers like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, etc.
- Relate to couple of method to protect secrets with Vault Operator and try to update pod with new secrets updating withΒ Reloader
Linux Collection
1. Medium - 100 Essential Commands, Scripts, and Hacks : The DevOps Engineerβs Survival Guide
- List of multiple useful command for helping you boost time to debug and troubleshoot the problem inside Linux machine
- You can find a new command, and itβs really powerful for helping you solving problem with detail explanation of Author