Roni Frantchi, OpenTofu | 20 December 2023
OpenTofu v1.6.0-rc1, the final stage before the first stable release, is out today. It follows the quick succession of its alpha and beta versions, on the road to an expected General Availability release on January 10, 2024, right after the holidays.
This release includes bug fixes, stability improvements, and updates to documentation. Importantly, this version highlights the stability of our new public registry, which debuted with the v1.6.0-beta1 release three weeks ago, and has been extensively tested by us and the early adopters from the OpenTofu community.
From Idea to Release Candidate in 4 Months
The OpenTofu journey has been a whirlwind, spanning from a proposition to a release candidate in just four months.
This is a major milestone for us, but also a bittersweet moment. Looking back on the beginning of our journey, there was hope that HashiCorp would hear our appeal, reverse its decision and restore balance to the ecosystem.
Yet, here we are … and everything that transpired after those initial weeks of surprise, hope, and disappointment can only be characterized as a case study of collaboration. Peers, community, and competitors all united to work together to preserve an open-source option for Infrastructure-as-Code.
To recap, here are the major milestones so far:
As of December 18th, OpenTofu had surpassed 31,000 downloads, had 60 committers, and had seen over 1,000 pull requests and issues. The project’s main repository has also amassed over 16,450 GitHub stars on top of the 36,200+ GitHub stars for the manifesto.
Now, with that out of the way, let’s talk shop.
The Registry Challenge
From the moment the OpenTF fork was announced, it was evident that a new public registry would be needed—an open-source substitute for the Terraform one, which is no longer accessible for non-Terraform projects following the TOS changes.
Similar to its predecessor in function, this new registry would have to be a highly available package resolution service for all providers and modules used by OpenTofu. Additionally, it had to meet other specific criteria, including:
- The registry had to be as self-sufficient as possible and require as little maintenance as possible.
- Anticipating increasing demand, it needed to be built for high availability and perform well at scale.
- It should smoothly transition from HashiCorp's registry, following the 'drop-in replacement' approach.
- Whatever we went with, it had to be open-source.
Brewing Together
Homebrew is the de facto standard package manager on macOS – just as APK or RPM is to Linux or Chocolatey/Winget is to Windows. Its simple nature, open-source package database at its core, and efficiency for packaging applications inspired something similar that would be ideal for our needs, while its popularity served as evidence of its scalability.
As we were discussing this, Homebrew lead Mike McQuaid, who has been working on that project for more than 14 years, took notice of our conversation and joined the fray, contributing helpful insights about the repository structure:
For me, this experience highlighted the importance of the Open Source concept and reminded me that without its inherently collaborative nature, engagements like that would be few and far between.
This instinctive and uninhibited camaraderie is what Open Source is all about.
Implementation
Amongst other things, Mike’s advice for us was to fragment the repo, to improve performance:
“We went through a ‘sharding’ process recently so that we have formulae/casks in our largest repositories split into subdirectories…The ‘git’ performance is dramatically better in this format.”
Following up on it, we built the registry (https://registry.opentofu.org) as a collection of alphabetized subdirectories broken down by namespace. These were hydrated with information from all providers and modules currently available on GitHub, each with its own JSON file with metadata and other specifics.
Importantly, using static files and hosting them on a Cloudflare R2 public bucket also lets us take full advantage of its CDN capabilities, ensuring performance and high availability with over 94% cache hit rates (on that note, huge THANKS to CloudFlare for sponsoring this project!).
Outside of our own tests, the four most requested files were:
/.well-known/terraform.json/v1/providers/hashicorp/null/versions/v1/providers/hashicorp/template/versions/v1/providers/hashicorp/aws/versions
With the registry live, we set up a GitHub Action to scan for updates to indexed resources. We also introduced an IssueOps process for adding new providers. With it in place, new submissions would be auto-processed and auto-validated when they go into the registry, with Issues providing context and transparency.
We’re currently exploring a dedicated UI for package and documentation browsing.
Next Stop: Jan 10th
With the holidays in full swing, we decided to roll out a release candidate first and schedule the GA for January 10th. Meanwhile, you can already start testing OpenTofu by simply following the directions provided here. It is a drop-in replacement so Terraform users will feel right at home, which is the whole idea.
If you are interested in contributing to the project, please check out these contribution guidelines, and don’t forget to join our Slack community.
Happy holidays!
P.S.
While concluding the work on this post, and the upcoming release, we learned that Mitchell Hashimoto, HashiCorp’s legendary co-founder, announced that he will leave the company after 11 years.
Mitchell’s work on projects such as Vagrant, Consul, Vault, and of course Terraform is a source of inspiration for us and many others within the Open Source community.
The entire OpenTofu team extends its heartfelt gratitude to Mitchell for all of his numerous contributions, as we eagerly anticipate learning about the next steps in his already remarkable journey.