Well, it’s been a while xD Since my last entry a lot has changed, so let’s cut to the chase!

Docker

The majority of my contributions to Guix has been packaging Docker’s dependencies. We keep the progress in this Milestone created by Hellseher. I submit my contributions in batches of about 20 packages each and, until now, I’ve submitted 7 batches, totalizing 92 commits, the most recent one being the PR #7580.

As I commented in my first journal entry, Guix is a source based distribution. So its package repository is actually a set of package recipes (also called package definitions) that describes how to compile a given software. In this way, to “package” a software, I need to write its package recipe and submit it to the Guix git repository. Each recipe is submitted as a commit.

Fortunaly, there is a tool, namely guix import, to help write this package definitions. So the first thing I did was to run guix import go --recursive github.com/moby/moby. The command guix import go fetches data from a given package (in this case, github.com/moby/moby) and produces its recipe and the flag --recursive calls guix import for each unpackaged dependency of the target project recursively. In this way, after a very long time fetching all the necessary data, guix import generated a very huge file containing all the recipes!

The next step is to find a recipe in this file which has all dependencies already packaged, test if it actually compiles (remember that some tests from the upstream package may fail) and submit it to guix.

This process is very tiresome and unmotivating:

• It is tiresome just because the amount of packages in the generated file is overwhelming and the dependency graph makes it a mess to choose a package to work with.

• It is specially unmotivating, because this generated file doesn’t explain to you the context of each package. To better ilustrate this point, let me draw an example: In the third batch, I package go-github-com-tonistiigi-go-csvvalue, a Go library to read csv values. Why in the hell would Docker use it? Even if indirectly? How does this package fit in the dependency graph? I mean, I know some packages depend on in, I can find them in the generated file, but theses others packages are also os disconnected from the ideia of Docker that it seems a waste of time to package them.

It is not enough to aimlessly work on this packages. I had to organize myself in a way to keep my focus and motivation!

I’ll write a dedicated post about it, because I really think it deserves a post by itself and because I’m already kind of tired of writing xD

Getting into the Guix’s Go team

After some batches, I wanted to contribute something other than a package definition. So I decided to pick the bug #3968 to fix!

Since I wasn’t in a team, Hellseher couldn’t assign the issue to me, so he asked me to join the Go team!

I was very happy with this and commited my named into the teams.scm file and created a PR. After that, the Hellseher merged my commit adding me to the Go team.

I’ll write a post about this bug once I fix it!

Conclusion

I’m very happy with my progress contributions in the Guix community! :D

O que se segue é uma lista de propostas de temas para meu doutorado em HPC. Após conversar com meu futuro orientador, focamos no tema de reprodutibilidade. O que me levou a participar do ACM REP 2025.

Originalmente esse texto foi disponibilizado como uma pagina própria nesse blog. Decidi transformá-lo em um post, como qualquer outro.

Metodologia para análise experimental de algoritmos em grafos

Para minha dissertação de mestrado implementei o algoritmo HDT para resolver o problema de conexidade em grafos dinâmicos.

Fizemos um experimento comparando com outros algoritmos seguindo esse framework de 2022. No entanto, devido a problemas de metodologia, não escrevemos um capítulo sobre isso na dissertação.

Lemos o artigo A Theoretician’s Guide to the Experimental Analysis of Algorithms de David S. Johnson. Boa introdução sobre o autor: slides

1. Faça experimentos dignos de nota Isto é; investigue questões de real interesse da comunidade.
2. Relacione seu artigo com a literatura
3. Faça testes em um conjunto de instâncias que permita se chegar a conclusões gerais
4. Projete os experimentos de forma eficiente e efetiva
5. Use implementações suficientemente eficientes
6. Garanta a reprodutibilidade
7. Garanta a comparabilidade
8. Reporte a história completa
9. Chegue a conclusões bem-justificadas
10. Apresente os dados de modo informativo

Em 2017, um novo algoritmo para o problema de conexidade dinâmica foi publicado. Ele adiciona mais estruturas de dados sobre HDT, obtendo menor consumo assintótico, mas com constantes maiores. Uma ideia de artigo é implementar esse algoritmo e fazer um experimento comparativo. Os autores desse artigo publicaram uma versão mais longa (56 paginas) explicando em detalhes como as estruturas de dados funcionam.

Em HPC, essa metodologia ainda se preocupa com mais fatores, Large scale graph processing systems: survey and an experimental evaluation.

Criar distro GNU/Linux focada em HPC

Livro Operating Systems for Supercomputers and High Performance Computing

Artigos:

• The Multikernel: A new OS architecture for scalable multicore systems
• A Multi-Kernel Survey for High-Performance Computing

User space

• Lembrei da apresentação: “Ambientes computacionais para a reprodutibilidade de softwares (de pesquisa) com GNU Guix e Software Heritage” de maio/2024 com o Prof. Alexandre Abdo

Guix System tem uma equipe focada em HPC e muitos princípios importantes para HPC:

• reprodutibilidade
• implantação de uma frota de maquinas

Tenho dois patches aceitos no Guix:

• Update httpd: 78570 as 9f33cb8825
• Update dropbear: 78600 as ac88ea15c7

E atualmente estou trabalhando para atualizar o pacote Docker: issue 74746.

Outros gerenciadores de pacotes podem ser encontrados no repositório awesome HPC.

Kernel

• Kitten
  • Feito pela Sandia labs
  • slides sobre o kernel
• McKernel
• mOS
  • Um fork do Linux feito pela intel para HPC.
  • Multi-kernel
  • Alguns artigos sobre:
  • Sua arquitetura
  • Avaliação de desempenho
  • Integrando containers ao mOS
  • Poster
• Barrelfish:
  • Feito pela ETH Zurich e Microsoft R&D

DevOps e engenharia de plataforma para HPC

A ideia é propor as práticas de DevOps e engenharia de plataforma para o ambiente acadêmico.

• Artigo: Continuous Integration and Delivery for HPC: Using Singularity and Jenkins
• Artigo: Continuous Integration for HPC with Github Actions and Tapis
• Artigo: Singularity: Scientific containers for mobility of compute
• Artigo: Containers in HPC: a survey
• Artigo: Containerization for High Performance Computing Systems: Survey and Prospects
• Artigo Shifter: Containers for HPC
• Artigo: A COMPARATIVE STUDY OF PLATFORM ENGINEERING TOOLS: IMPLICATIONS FOR SYSTEM DESIGN AND SCALABILITY

It is been a while since I got time to contribute. In this entry, I will report about some progress that I got about two months ago.

Docker

I succeeded to skip the failing test that I commented in my last journal entry. The problem was in the parameter BUILDFLAGS in the unit test file that referenced libsystemd. A simple substitution, as shown below, fixes the problem.

(substitute* "hack/test/unit"
                (("BUILDFLAGS=.*")
                 (string-append "BUILDFLAGS=(-tags 'netgo')\n")))

I then reported my progress to the related issue. After some days, Hellseher replied to me and suggested that I collaborate on codesberg, the new git web instance used by Guix’s community.

Hellseher also suggested to ‘devendor’ Docker. Some Golang projects, such as Docker, store all dependencies in a folder called vendor. This folder “hides” the dependencies from the Guix package system, so it is a good practice to remove this folder and declare all dependencies as inputs in the packages’s recipe.

The problem is that Docker contains a lot of dependencies, so it is a lot of work to devendor it. I intend to start adding these dependecies in the next week.

• Workflow é uma serie de passos computacionais e de manipulação de dados.
• “Workflows are becoming the paradigm of choice for HPC applications.”

Como reproduzir workflows?

• Mini-Apps: são proxies que replicam com fidelidade o comportamento de workflows. O apresentador mostrou esse estudo de caso.
• Nexflow é um sistem de gerenciamento de Workflow. Houve uma palestra apresentando a plataforma um uma live no canal do youtube do LNCC.

HPC

Benchmark

Fazem benchmark para:

• Comunicar cargas de trabalho para vendors.
• Projetar sistemas e monitorar progresso.
• Validar ferramentas:
  • Compiladores.
  • Debugadores.
  • Ferramentas de corretude e desempenho.

Os 3 R’s em benchmark em HPC:

• Repeat (mesmo hardware, ambiente e aplicação):
  • Verificar se o benchmark ainda compila e roda no sistema.
  • Repassar as tarefas de benchmark para outro usuário.
  • Avaliar desempenho ao longo do envelhecimento do sistema.
  • Coletar ruído de variabilidade.
• Replicate (mesmo hardware e aplicação, ambiente de software muda)
  • Verificar se o benchmark ainda compila e roda no sistema atualizado.
  • Validar desempenho do sistema atualizado.
  • Avalizar diferenças de desempenho entre diferentes opções de software (ex: gcc, clang, tcc)
• Reproduce (hardware e ambiente diferem)
  • Verificar se o benchmark compila e roda no sistema diversos.
  • Avalizar desempenho em diversos sistemas.
  • Explorar possíveis benefícios de fazer upgrade de hardware.

No artigo Towards Collaborative Continuous Benchmarking for HPC, os autores apresentam benchpark, um framework que benchmark de clusters HPC. O framework possibilita formalizar os 3 R’s descritos acima.

Reproducibility initiative in SC’24

Artigo Implementing a Reproducibility Initiative in HPC: Experiences from SC24

SC24:

• Artifact Description e Artifact Evaluation appendix são obrigatórios.
• Reproducibility Challenge no Student Cluster Competition: Estudantes tentam reproduzir papers publicados na SC do ano anterior. Começou na SC16.
• Reproducers: 101 para 102 papers
• NEW: Reproducibility Report:
  • pequeno paper publicado pelo reprodutor descrevendo o que foi reproduzido.
  • é publicado como “company report” ao paper original.
  • possui um template.
  • limite de tempo de 8h (excluindo computação).
  • repo github.
• Badges:
  • Artifact available: significa que está permanentemente arquivado, verificado que não é bem verdade nesse artigo.
  • Artifact Functional: artefato consegue ser executado.
  • Results Reproduced: resultados chaves podem ser reproduzidos por outros.
• Recomendações:
  • Focar em availability e functionality no momento da publicação.
  • Focar em replicability e reproducibility em journals e tracks específicos.

Reprodutibilidade é difícil:

• Consome muito tempo dos reproducers
• Em HPC:
  • Escala dos computadores
  • Objetos avaliadors (como desempenho)

Próximas leituras

• Implementing a Reproducibility Initiative in HPC: Experiences from SC24
• Reproducibility, Replicability and Repeatability: A survey of reproducible research with a focus on high performance computing
• Longevity of Artifacts in Leading Parallel and Distributed Systems Conferences: a Review of the State of the Practice in 2023

The Purely Functional Software Deployment Model in Clusters

O modelo de implantação puramente funcional foi introduzido

Para HPC a implementação GNU Guix é mais madura, eles tem uma equipe de HPC dedicada. Activity Report

• Painless Transposition of Reproducible Distributed Environments with NixOS Compose
• Reproducible and User-Controlled Software Environments in HPC with Guix

No dia 5 desse mês participei de um evento promovido pela AWS sobre como usar nuvem (especialmente a deles) para acelerar pesquisa. Além de adesivos, levei algumas coisas comigo desse evento e gostaria de compartilhar.

Esse evento foi dividido em duas partes. De manhã houve palestras sobre diferentes usos de AWS para pesquisa e estudos de casos com pesquisadores e de tarde houve um workshop.

sim@cloud

Para mim, o destaque das palestras foi a apresentação da professora Lucia Drummond sobre o SIM@CLOUD, que é um framework que otimiza custo de simulações feitos na nuvem usando instâncias spot da AWS.

Essas instâncias possuem um desconto consideravel em seu custo, as vezes de até 90%, pois usam uma capacidade ociosa dos datacenters. No entanto, quando a AWS precisar dessa capacidade, ela vai deletar sua máquina após um aviso prévio de 2 minutos. Por tanto sua aplicação deve ser tolerante a falhas e estar preparada para tratar esse aviso de desligamento. SIM@CLOUD automatiza a migração de simulações entre instâncias spot, assim mitigando o problema de desligamento prématuro.

O time dela já usou essa ferramenta com a Petrobras para rodar algumas simulações de forma mais econômica. Em tese a ferramenta é open-source, mas estou com dificuldade para encontrar o código fonte da mesma. Se eu encontrar o código, vou editar esse texto e adicionar um link para ele.

Pesquisando sobre o tema encontrei os seguintes artigos que parecem interessantes:

• Smart spot instances for the supercloud;
• Achieving Performance and Availability Guarantees with Spot Instances;
• High Performance Computing in Clouds: Moving HPC Applications to a Scalable and Cost-Effective Environment: Livro do time da autora.

Workshop

No workshop, usamos o jupiter notebook integrado a AWS. O material utilizado está disponivel nesse repositório. Em particular, utilizamos esse notebook.

Percorremos o básico de containers e subimos uma aplicação de comparação de DNA na infra da AWS. Ainda não tive tempo para revisitar o material após o workshop, se fizer isso, atualizo esse tempo com mais comentários sobre.

I got my first two accepted patches:

• Update httpd: 78570 as 9f33cb8825
• Update dropbear: 78600 as ac88ea15c7

Total accepted patches: 2

Docker

In my last post I commented that I was working on updating Guix’s Docker package to version v28. I got the build phase to succeed, but failed in the check phase due to a new unit test that was introduced.

I got tired trying to make this test succeed or to avoid this test a all. So I decided to share my work on the related issue and focus my energy on lower hanging fruits to clear my mind.

Patches

As a low hanging fruit, I decided to update an existing package, it is a very simple thing to do, you just need to:

1. Update the versionfield in the recipe;
2. Update the tarball sha256 hash;
3. Check if the recipe still build the package;
4. Fix any problem that arises.

In Docker’s case, the last packaged version is two years old. So there are a lot more tests to patch in the recipe. So step 4 needs a lot of work.

We can find packages that need to be updated using the guix refresh command. Just be aware that it takes some time, because it fetches a lot of metadata from the internet.

$ guix refresh
nongnu/packages/wine.scm:25:13: winetricks would be upgraded from 20230212 to 20250102
nongnu/packages/wine.scm:145:13: dxvk would be upgraded from 2.0 to 2.6.1
nongnu/packages/wasm.scm:109:4: wasm32-wasi-clang would be upgraded from 15.0.7 to 20.1.6
nongnu/packages/wasm.scm:129:13: wasm32-wasi-libcxx would be upgraded from 15.0.7 to 20.1.6
nongnu/packages/vpn.scm:15:13: zerotier would be upgraded from 1.12.2 to 1.14.2
nongnu/packages/video.scm:84:13: intel-media-driver would be upgraded from 25.1.4 to 2018Q2.1
nongnu/packages/video.scm:119:2: intel-media-driver-nonfree would be upgraded from 25.1.4 to 2018Q2.1
nongnu/packages/video.scm:211:2: obs-with-cef would be upgraded from 30.1.2 to 31.0.3
nongnu/packages/video.scm:26:13: ffmpeg-nvenc would be upgraded from 6.1.2 to 7.1.1

After looking at my options, I choosed to update httpd (Apache), because the last packaged version was reasonably recent, so I guessed that step 4 wouldn’t need any work and I was right, I just updated the version and sha256 and the build recipe was successful. The patch is simple and was submited as issue 78570. A commiter reviewed and accepted my patch! and commited as commit 9f33cb8825, this is my first accepted patch. But there was a problem, I was credited as “arthurhdrodrigues— via Guix-patches via” instead of Arthur Rodrigues or just arthurhdrodrigues.

I have some theories about this:

• I didn’t configured a name in my email; and
• I didn’t use git send-email to submit the patches, I just copy-and-pasted the patch as in a plain text email. So some meta-data differed, to be precise the Cc: field in the email header using git send-email was set to my email, as shown below.

From 8f5469ee21374ac5712ac9d2e0c5e46d59e8bd9c Mon Sep 17 00:00:00 2001                                                                                                   
Message-ID: <8f5469ee21374ac5712ac9d2e0c5e46d59e8bd9c.1749085808.git.arthurhdrodrigues@proton.me>
From: Arthur Rodrigues <arthurhdrodrigues@proton.me>
Date: Wed, 4 Jun 2025 22:07:40 -0300
Subject: [PATCH] docker
 
Change-Id: I6f1ee9bfa263f9ebb9b75c6d8c2b1110d36b040c
---
 gnu/packages/docker.scm | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
 
diff --git a/gnu/packages/docker.scm b/gnu/packages/docker.scm
index b03019de08..bd341ded42 100644
--- a/gnu/packages/docker.scm
+++ b/gnu/packages/docker.scm

After fixing these problems, I submited another patch that updates dropbear, a lightweight ssh server. This second patch was also accepted and commited as ac88ea15c7 and this time I was credited correctly!

Now I’m ready to update more packages and go back to work with Docker v28 :D

Por exemplo, a primeira apresentação com nome Experiências práticas e inspiradoras de Infraestruturas Digitais Autônomas, apresentou o projeto Nova CanudOS, que visa divultar técnicas de infraestruturas autônomas. Concomitante a essa apresentação, houve outra sobre homelab que não pude comparecer, mas pesquisei sobre o palestrantes e achei seu blog pessoal.

Já sobre fediverso, o que se destacou para mim foi a onde.social. Uma iniciativa de um rede social descentralizada brasileira, você pode ver o resumo da apresentação ou a pagina no plantaformas.

Guix System is a fairly new GNU System or GNU/Linux distribution, the pure GNU version ships with the Hurd kernel, while the Linux version ships with linux-libre a linux fork with all of the binary blobs, obfuscated code and portions of code under proprietary licenses removed.

It is the GNU implementation of the pure functional deployment model, inaugurated by Nix and NixOS.

As I can see, Guix cares much more about reprodutibility than NixOS and it has a dedicated HPC team. But since it is less mature, it is kind of rough around the edges, so I decided to contribute to it!

First bug report

My first contact with the community was this bug report, which was quickly closed, because it was not considered a bug.

First patch

Guix is, by default, a source based distribution, thus a package is actually a recipe to build a given software. This recipes are stored in the folder gnu/packages.

My first submitted patch proposed the addition of the Golang library jwt. The recipe for Golang libraries are very simple, because no compilation is required, to install a Golang library, you just need to download its source files.

Before sending the patch to the mailing list I checked the file golang-xyz.scm, that constains a lot of Golang libraries and didn’t find jwt. But after submitting it, a contributer pointed out that this project is already present using guix’s packages search tool webpage. This web page shows the location of the library as being golang-crypto.scm, a simple grep in guix’s source code confirms that. So my patch was not accepted.

Current Contribution

Now I’m updating the Docker package to version 28.1.1, the current package is version 20.10.27, which is two years old. This update is tracked in the issue 74746.

Future Contributions

In the futere I also plan to add the package act and ghidra. Also configure a default wallpaper for Gnome, currently there is no default wallpaper and a blank dark blue backgroud is the default behavior.

The basic idea is to develop a compatibility layer to run linux modules on redox-os. It’d be something similar to Wine, but to be a layer between linux and redox-os.

The objective is to “port” all linux’s device drivers to redox at once.

The idea

I had this interesting idea some weeks ago and I don’t know if it is possible, so I’ll write about it here and maybe some day work toward a proof of concept.

Linux is the most used kernel in the world and Mars. This popularity implies in (and is possible by) a large range of supported machines and thus in drivers implemented for linux. Actually the majority of kernel code is device drivers.

But it is not the only open source OS out there. Redox-os is a fairly new kernel/OS written in Rust that uses the microkernel architecture. It’d be nice if redox-os supported as much hardware as linux, but it lacks the necessary drivers and of course rewrite all linux’s drivers is a massive task to implement and maintain.

Then I came up with an idea that I must point out again that I don’t know if it is possible. The idea is to develop a compatibility layer to run linux modules on redox-os.

This compability layer would provide a virtualized enviroment for linux modules to run in redox-os just as another userspace process. It wouldn’t be a virtual machine, because we are not virtualizing hardware. It’d be something similar to Wine, but this layer would virtualize not only the linux’s syscalls (as Wine implements the Windows syscalls), but also all other resources a module needs (such as global .

The problem is that I don’t know which resources a linux module needs. I presume that there should be some global variables and internal API that this layer needs to virtualize.

If this layer succeeds, it will port all linux’s device drivers to redox-os at once with reduced maintenance cost, that is, instead of maintaining all drivers, we just maintain the compatibility layer.

Digitalizei algumas revistas antigas, da época que que se vendia CDs que permitiam acesso a internet.

Você pode acessá-las nesse link

Como hobby eu escrevo alguns comentários sobre elas nesse página do blog.