Protegendo e Recuperando Dados Perdidos - Git, Backup, BTRFS

Tech1 of 14

Fabio Akita: Git is Non-Negotiable

Git is an essential tool for all programmers, regardless of their company's adoption. Even for personal projects, initializing a Git repository with 'git init' and making frequent commits is crucial for version control and data safety. This practice prevents data loss from accidental deletions or system failures.

The Perils of Partition Loss: Superblock Corruption

Partition corruption, often due to hardware failure or human error, can render data inaccessible. A critical component, the superblock, contains vital information about the file system's structure. Its corruption prevents the OS from mounting the partition, necessitating recovery efforts.

Fabio Akita: Beware of AliExpress SSD Scams

Cheap, unbranded SSDs from platforms like AliExpress are often scams, using inferior SD card components or rigged controllers that lie about capacity. These drives are unreliable and prone to data corruption, making them unsuitable for critical data storage.

Superblock Recovery with fsck

When a partition's superblock is corrupted, recovery is possible by using the 'fsck' command with the '-b' option to restore from backup superblocks. This process, while not guaranteed, can often re-establish file system integrity and allow access to data.

Akita's Rule: Trust No Failed Hardware

Hardware that has failed once should never be trusted again with important data. The recommended action is to replace it entirely, treating it as junk, or repurpose it only for non-critical tasks like storing games.

The Peril of Data Deletion

When a file is deleted, its index entry is removed, but the data blocks remain on the disk until overwritten. This means accidental deletions are recoverable if the drive is immediately unmounted to prevent further writes. A bit-by-bit backup using tools like DD is crucial for safe recovery attempts.

The Unavoidable Necessity of Backups

Despite the common neglect, backups are the single most effective way to prevent data loss from hardware failure, accidental deletion, or malicious attacks like ransomware. Even a simple external HDD is a far better solution than no backup at all, offering a crucial safety net against digital disasters.

Modern File Systems: Checksums for Integrity

Unlike older file systems, modern ones like APFS, NTFS, ZFS, and BTRFS incorporate checksums to detect data corruption at the file or block level. This allows systems to identify when data has been altered, providing a critical alert that the data is untrustworthy and prompting recovery from backups.

Copy-on-Write (CoW) and Snapshots: The Git of File Systems

Copy-on-Write, a core feature of file systems like BTRFS and ZFS, functions similarly to Git branches. When a file is modified, new blocks are written without overwriting the old ones, preserving the original data. This enables efficient snapshots that consume minimal extra space, providing a robust mechanism for versioning and recovery.

BTRFS Installation and Configuration on Ubuntu

While Manjaro offers BTRFS out-of-the-box, Ubuntu requires manual partitioning for BTRFS. This involves creating EFI and boot partitions, then dedicating the rest to BTRFS, ideally with subvolumes for root ('@') and home ('@home'). Special attributes can prevent snapshots from consuming excessive space in volatile directories like /var/log.

Timeshift: The Instant Recovery System

Timeshift provides an instant, local backup solution that allows users to revert their system to a previous state, effectively undoing problematic updates or software installations without needing external drives or reinstallation. This acts as a crucial last line of defense against accidental data loss or system corruption.

The Docker Space-Hogging Menace

Docker images, especially when combined with BTRFS snapshots that don't automatically reclaim space, can rapidly consume massive amounts of disk space. This occurs because each snapshot may retain copies of Docker images, leading to exponential space usage that can fill even large drives unexpectedly.

Ransomware Defense: Beyond Simple Backups

Ransomware can infect and encrypt files over days, potentially corrupting even external backups if they are connected during the encryption process. A robust defense requires layered backups, including immutable or versioned cloud storage and local snapshots (like BTRFS), to ensure recovery from encrypted files.

Layered Security and Contingency Planning

True data security relies on multiple layers of defense, assuming that any single layer might fail. By implementing Git, local snapshots, external backups, and potentially cloud solutions, users create a contingency plan that ensures data resilience against various threats, from minor errors to major catastrophes.

Protegendo e Recuperando Dados Perdidos - Git, Backup, BTRFS

FAFabio Akita1h 11m

FA

Criptografia na Prática - Certificados, BitTorrent, Git, Bitcoin

Tech1 of 6

Fabio Akita: The Necessity of Disk Encryption

Disk encryption, using algorithms like AES-256, is crucial for protecting data on laptops against physical theft. Relying solely on OS login is insufficient as it doesn't protect against direct hardware access. Modern CPUs accelerate encryption, making it a practical security measure.

Criptografia na Prática - Certificados, BitTorrent, Git, Bitcoin

FAFabio Akita1h 14m

FA

Configurando Docker Compose, Postgres, com Testes de Carga - Parte Final da Rinha de Backend

Tech1 of 13

Akita: The Rinha's Value is in Diverse Implementations

The true value of the Rinha de Backend event lies not in solving it quickly, but in studying the diverse approaches participants took to solve the same problem. Examining different implementations, like Lauro Appelt's PHP version, provides invaluable learning opportunities for developers.

Docker Compose: Essential Infrastructure Documentation

Docker Compose is indispensable for modern development, acting as living documentation for a project's infrastructure by automating the setup of dependencies like databases. Every real project, regardless of size, should include a working Dockerfile and docker-compose.yaml.

NGINX as a Load Balancer: Configuration and Network Modes

Implementing NGINX as a reverse proxy for load balancing requires careful configuration, especially regarding worker processes and connections. The choice between Docker's host network mode and bridge mode significantly impacts performance and cross-platform compatibility, with host mode often proving superior for load testing but problematic on non-Linux systems.

Akita: Host Network Mode Challenges on Mac/Windows

While Docker's host network mode offers performance benefits, it is Linux-exclusive. On macOS and Windows, it fails due to the underlying virtualization layers, necessitating the use of bridge mode for general development or alternative solutions like WSL 2 for a more native Linux environment.

Configuring Gatling for Load Testing

Setting up Gatling involves installing Java (preferably version 17 or older), downloading Gatling, and configuring the `run-test.sh` script to point to the Gatling binary directory. The simulation scripts, written in Scala or Kotlin, define scenarios using test data from TSV files, specifying HTTP requests (POST for creation, GET for search) and expected response codes.

Host Network Mode Performance Gains

Switching Docker services to host network mode, as demonstrated with Lucas Weis's Node.js project, significantly reduces request errors (like 'Premature Close') and increases throughput. This mode bypasses Docker's virtual network, allowing direct binding to the host's localhost, which proved more performant than the default bridge mode for the Rinha de Backend challenge.

Akitaonrails: Benchmarking is Essential

The 'Rinha de Backend' challenge highlighted the importance of load testing and benchmarking. Tools like Gatling are invaluable for making informed decisions about whether to optimize code or scale infrastructure, providing data to justify the effort and expected improvements.

Akitaonrails: Optimizing Partial Searches with Full-Text Search

Partial searches using ILIKE with leading/trailing wildcards are inefficient as they prevent index usage, forcing full table scans. PostgreSQL's full-text search capabilities, particularly using GiST indexes with trigram operators (gist_trgm_ops), can tokenize fields and enable efficient partial matching, drastically reducing search times.

Akitaonrails: Analyzing Query Performance with EXPLAIN ANALYZE

PostgreSQL's 'EXPLAIN ANALYZE' command is vital for understanding query execution plans. It reveals whether queries use efficient index scans or slow sequential scans, providing concrete data (like microseconds vs. milliseconds) to diagnose and fix performance issues.

Akitaonrails: Tuning PostgreSQL Connection Pools

Configuring PostgreSQL's `max_connections` requires careful balancing. Each connection consumes RAM (2-4MB), so excessively high limits can exhaust resources. Load testing is essential to determine the optimal number of connections needed to support the application's workload without wasting memory or causing connection wait times.

Connection Pool Performance Under Load

Load testing reveals that database connection pools are highly effective, with active connections rarely exceeding 40 even under heavy load. The majority of connections remain idle, and spikes only occur during extreme performance degradation (over 1 second response time), indicating that connection pool size is often not the primary bottleneck. The optimal pool size must be determined through empirical testing with tools like pgAdmin and Gatling, rather than relying on theoretical formulas. This pragmatic approach avoids over-engineering and ensures efficient resource utilization.

The Pragmatic Programmer's Mantra: Measure, Then Optimize

The core lesson from the Rinha de Backend challenge is to embrace pragmatism: write the simplest code that solves the problem, measure its performance rigorously, and only then optimize based on empirical data. Over-engineering and premature optimization are common pitfalls. This data-driven approach, involving benchmarking, profiling, and load testing, is essential for gaining control over software behavior and ensuring that optimizations actually improve performance rather than degrade it.

Thread vs. Fiber Models and Connection Pooling

Languages utilizing native threads (like Java, Rust) can manage database connections more efficiently due to thread-safe connection pools, requiring fewer simultaneous connections. Interpreted languages with a Global Interpreter Lock (GIL) or similar mechanisms (like Node.js, Python) often require separate connection pools per process or thread, potentially leading to higher overall connection counts. Understanding these underlying concurrency models is crucial for correctly configuring connection pools and optimizing database interactions across different language ecosystems.

Configurando Docker Compose, Postgres, com Testes de Carga - Parte Final da Rinha de Backend

FAFabio Akita1h 36m

FA

Guia DEFINITIVO de Aprendendo a Aprender | A maior BRONCA da sua vida [RATED R]

Education1 of 12

Akita: The Tech Market Bubble Has Burst

The tech industry has been in a prolonged bubble for over a decade, fueled by irrational exuberance and venture capital. This unsustainable growth, characterized by startups operating at a loss and prioritizing expansion over profit, has now burst due to the COVID-19 crisis, leading to layoffs and a market correction. Those who relied on hype rather than genuine skill will suffer.

The Flawed Education System: Courses and Certifications

Traditional educational approaches, including online courses and certifications, are fundamentally flawed. They often provide superficial, step-by-step instructions rather than fostering deep understanding or critical thinking. This system conditions students to 'scrape by' and produces graduates unprepared for the real demands of the job market, especially when the market corrects.

The Superiority of Public Universities and Real-World Experience

Public universities, due to their rigorous selection processes and focus on foundational subjects, tend to produce graduates with stronger potential than private institutions. Furthermore, practical experience, especially through internships in a structured environment, is crucial for developing actual professional competence, far exceeding the value of mere certifications.

Critique of Traditional Courses

Traditional courses, with their structured materials, step-by-step instructions, and passive learning format, are deemed ineffective. They fail to filter for genuine passion and can lead to complacency, producing students who only aim to pass rather than truly learn.

The Value of Foundational Knowledge

Mastering fundamental concepts like processes, threads, and memory allocation is crucial for programmers, enabling them to adapt to new languages and tools. Without this foundation, developers become slaves to the first technology they learn.

Programming Languages as Tools, Not Religions

Programming languages are merely tools designed to serve their users, not objects of blind loyalty. The speaker emphasizes that true programmers should not be defined by their preferred language and must be adaptable, learning new tools as needed.

Learning to Learn: The Art of 'Figuring It Out'

The core of learning to learn lies in developing the ability to 'figure things out' independently, embracing challenges, and persisting through frustration and pain without external validation. This self-driven process is the true filter for success.

The Importance of English in Tech

English proficiency is paramount in the tech world, as most cutting-edge information and open-source projects emerge in English first. Relying solely on translated materials leads to delays and missed opportunities.

École 42: A New Educational Paradigm

École 42, a nonprofit coding school with global branches, employs a peer-to-peer learning methodology starting with fundamental subjects like Shell, C, and Git, before moving to more common languages, challenging traditional educational structures.

The 42 SP Model: No Teachers, Just Peers

The 42 SP educational model fundamentally rejects traditional teaching structures, eliminating classes and teachers. Instead, it fosters a collaborative environment where students learn by teaching and helping each other, with progress tracked through a gamified system of 'galaxies' and levels. This peer-driven approach is crucial for developing practical skills and a deep understanding, as students must actively engage with the material and their peers to advance. The speaker concludes that this self-directed, community-based learning is essential for true mastery.

Non-Profit Status and Student Selection

42 SP operates as a non-profit organization, funded by donations rather than tuition fees. This financial model allows the school to be highly selective, admitting students based on potential and drive, and even removing those who do not actively participate or collaborate. The speaker emphasizes that this contrasts sharply with commercial courses where paying students might be retained regardless of engagement. The speaker concludes that this non-profit structure enables 42 SP to prioritize genuine learning and collaboration over financial incentives, ensuring resources are used effectively by motivated individuals.

The Value of In-Person Learning for Beginners

The speaker strongly advocates for the necessity of in-person interaction for individuals starting their careers, contrasting it with the limitations of remote work for novices. They argue that the collaborative and problem-solving environment at 42 SP, which requires physical presence, is crucial for developing essential skills that cannot be replicated online. The speaker concludes that while remote work suits experienced professionals, beginners thrive and learn more effectively through direct, in-person engagement and mentorship.

Guia DEFINITIVO de Aprendendo a Aprender | A maior BRONCA da sua vida [RATED R]

FAFabio Akita1h 5m

FA

Desbloqueando o "Algoritmo" do Twitter - Introdução a Grafos

Tech1 of 17

s1: Incomplete Code, Inconclusive Analysis

The released Twitter algorithm code is incomplete, with a second part missing and potential omissions in configuration files. This makes definitive analysis difficult, as we can only perform static analysis without knowing how the code actually runs in production. Claims based solely on this incomplete code should be treated with caution.

s1 Critiques Public Reaction to Code Release

The public's reaction to the Twitter code release, particularly focusing on trivial details like 'author_is_elon,' is characterized as embarrassing and indicative of 'bikeshedding.' The speaker argues that such discussions are a waste of time and distract from understanding the actual technical complexities.

s1: The Evolution of Twitter's Recommendation Systems

Twitter's recommendation efforts began around 2010 with the 'Who To Follow' system, initially using an in-memory graph processing engine called Cassovary. This system relied on follower graphs loaded from HDFS, which in turn ingested data from FlockDB, Twitter's first graph database. The evolution from relational databases to specialized graph databases demonstrates the increasing complexity and scale of their needs.

Speaker: From Yellow Pages to PageRank - The Web's Search Revolution

Before Google, finding information online relied on rudimentary directories akin to Yellow Pages. The invention of PageRank by Larry Page and Sergey Brin revolutionized this by using the hyperlink structure of the web itself to rank pages based on a 'random surfer' model, effectively killing manual directory efforts and establishing a new paradigm for search relevance.

Speaker: HITS Algorithm - Distinguishing Hubs from Authorities

The HITS algorithm, developed by Jon Kleinberg, improved upon PageRank by assigning two scores: 'hub' (pages that link out) and 'authority' (pages that receive many links from hubs). This distinction is crucial for differentiating between influencers who direct traffic and genuine authorities on a topic, offering a more nuanced view of web page importance.

Speaker: Twitter's SALSA-Based Recommendation Engine and Homophily

Twitter adapted SALSA to build its early recommendation engine, creating a bipartite graph where 'hubs' are users' trusted circles and 'authorities' are users they interact with. By ranking these connections and applying the principle of homophily (people with similar interests cluster together), the system generated 'similar to you' recommendations.

Speaker: The Shift to Hadoop and the MapReduce Paradigm

As data volumes grew, Twitter moved towards Hadoop and MapReduce for its recommendation engine. MapReduce, a fundamental Big Data processing model, involves partitioning data across multiple servers, processing chunks in parallel (Map), and then aggregating results (Reduce), enabling scalability beyond single machines.

Speaker: Graph Processing Challenges in Hadoop and 'One Pass PPR'

Graphs proved challenging for Hadoop's MapReduce due to their iterative nature and the need for frequent data shuffling. To overcome this, Twitter adapted personalized PageRank into 'One Pass PPR,' reducing iterations and shuffling by partitioning data by user ID and using caching, significantly improving performance in a distributed environment.

Snowflake: Twitter's ID Strategy

Twitter developed an open-source project called Snowflake for generating unique IDs. Unlike simple auto-incrementing integers, Snowflake IDs are prefixed with a timestamp, server identifier, and a sequence number. This design facilitates grouping by time, locating data within the server cluster, and crucially, prevents ID collisions across distributed systems, making it a robust solution for large-scale applications.

Protobufs vs. JSON: Efficiency in Communication

While JSON is user-friendly for human interaction and APIs, it's inefficient for system-to-system communication due to its text-based nature and embedded schema. Protocol Buffers (Protobufs) and Thrift, in contrast, generate compact binary formats that are significantly smaller and faster for computers to process. This efficiency is critical for high-throughput systems like Twitter, where terabytes of data are transferred daily, leading to substantial savings in bandwidth and processing power.

Twitter's Migration to Scala: A Strategic Move

In the mid-2000s, facing massive growth, Twitter migrated from Ruby on Rails to Scala. This decision, often misunderstood as 'Rails doesn't scale,' was driven by the need for a more robust, performant language capable of handling extreme scale and complex distributed systems. Scala, running on the mature JVM, offered a blend of object-oriented and functional programming paradigms, making it suitable for building components like queue systems and graph databases that were nascent at the time.

Vector Space Model for Relevance Ranking

Relevance in search and recommendations is computed using the Vector Space Model, a concept from Linear Algebra. Documents and queries are represented as vectors in a high-dimensional space, where dimensions correspond to words. Cosine similarity, calculated via dot product, measures the angle between these vectors, indicating their relevance. A higher cosine similarity score signifies greater relevance, forming the basis for algorithms like PageRank and the underlying technology in search engines and recommendation systems.

Approximate Nearest Neighbor (ANN) for Performance

For large datasets, finding the exact nearest neighbor in a high-dimensional space is computationally infeasible. Approximate Nearest Neighbor (ANN) algorithms, such as Locality-Sensitive Hashing (LSH) or KD-trees, offer a performance boost by trading a small degree of precision for significant speed improvements. Twitter's SimCluster-ANN project utilizes these techniques to efficiently find similar tweets and users, optimizing recommendation and search functionalities.

SimClusters: Uncovering Tweet and User Communities

Twitter's SimClusters project uses a custom matrix factorization algorithm to identify communities of users and tweets. Users and tweets are represented as vectors in an embedding space, allowing for similarity calculations. This system, updated regularly, groups around 145,000 communities, enabling the discovery of influential users and relevant content based on shared interests and interactions.

Speaker: The Recommendation System's Architecture

The Twitter recommendation system is a multi-stage process involving candidate sourcing, ranking via machine learning models (PyTorch), and final filtering. Projects like GraphJet, SimClusters, and RealGraph contribute to feature engineering and pre-training, while the Home Mixer and visibility filters implement censorship policies. The system blends 'In-Network' and 'Out-of-Network' tweets, with the cr-mixer project playing a key role in this blend.

Speaker: The Frustration of Incomplete Release

The analysis is significantly hampered by the incomplete release of Twitter's code, specifically the absence of essential Bazel Workspace files needed for compilation. This, coupled with a lack of automated test files, makes it impossible to run, test, or fully validate the system's behavior. The speaker expresses frustration, stating that without these components, 'we have no way to compile or even try to run anything.'

Speaker: Code Quality and Legacy Systems

Despite being over 10 years old, the leaked Scala codebase for Twitter's recommendation system exhibits surprising quality, featuring short methods, low complexity, good naming conventions, and organized configuration. This contrasts with other leaked codebases, like Twitch's, and suggests a strong engineering foundation. The speaker notes that maintaining such a legacy system without automated tests would be 'complete stupidity,' implying their existence is crucial.

Desbloqueando o "Algoritmo" do Twitter - Introdução a Grafos

FAFabio Akita1h 29m

FA

Esqueça Metodologias "Ágeis" | [Rated R]

Tech1 of 6

Fabio Akita: The Prostitution of Agile

The term 'Agile' has been prostituted, becoming a marketable noun for consultancies rather than an adjective describing a way of being. This distortion has led to a loss of focus on the original values and practices, turning it into a 'zombie' unrecognizable from its intent.

Esqueça Metodologias "Ágeis" | [Rated R]

FAFabio Akita42m