We are excited to announce the general availability of the MetaSource (or MDAPI 4) in both the staging and production Fedora Infrastructure environments. The release includes an architectural rewrite of the MDAPI from Python to Go, making it a performant source of RPM repositories metadata as a REST service with 1:1 API compatibility. More details about the developments and acknowledgements are below.

Rewrite

The project faced a critical challenge due to its dependence on SQLite3-based RPM repositories metadata which were deprecated by the Fedora Linux 41 release. Rather than applying temporary workarounds on the existing codebase, we took this opportunity to fundamentally redesign the project. The improvements included optimized processing of XML-based RPM repositories metadata, swifter response towards HTTP REST operations, interactive documentation for ecosystem experience, stellar coverage across functional testcases among other things. The move from Python to Go programming language allowed us to take advantage of the performance benefits and resource efficiency – all while ensuring that the solution stays simple enough to maintain for a high-throughput HTTP service for a cloud-native first deployment.

Comparison

Sustained querying

Services	Sample Count	Total Duration	Per request Duration
MDAPI MDAPI v3.1.7	1000 requests	63 minutes, 3 seconds	3.7828 seconds
MetaSource MDAPI v4.0.0	1000 requests	42 minutes, 30 seconds	2.5505 seconds

Services	Average Memory	Minimum Memory	Maximum Memory
MDAPI MDAPI v3.1.7	157.09 MiB 160,861 KiB	99.16 MiB 101,544 KiB	204.34 MiB 209,248 KiB
MetaSource MDAPI v4.0.0	109.72 MiB 112,350 KiB	81.66 MiB 83,616 KiB	174.52 MiB 178,712 KiB

MetaSource (or MDAPI v4.0.0) performs roughly 33% faster than MDAPI v3.1.7 while using about 30% lesser memory than that on sustained querying operations. This means that MetaSource would be able to address approx 50% additional requests without furthering resource consumption. Please note that the results may vary depending on unknown variables like network bandwidth and querying nature.

Concurrent querying

Services	Sample Duration	Total Count	Per request Duration
MDAPI MDAPI v3.1.7	500 seconds	148 requests	7.3615 seconds
MetaSource MDAPI v4.0.0	500 seconds	310 requests	3.5530 seconds

Services	Average Memory	Minimum Memory	Maximum Memory
MDAPI MDAPI v3.1.7	217.41 MiB 222,625 KiB	136.73 MiB 140,008 KiB	289.93 MiB 296,888 KiB
MetaSource MDAPI v4.0.0	187.02 MiB 191,510 KiB	130.68 MiB 133,816 KiB	257.44 MiB 263,624 KiB

MetaSource (or MDAPI v4.0.0) performs roughly 52% faster than MDAPI v3.1.7 while using about 14% lesser memory than that on concurrent querying operations. This means that MetaSource would be able to address approx 110% additional requests without furthering resource consumption. Please note that the results may vary depending on unknown variables like network bandwidth and querying nature.

Appeal

This project would not have been possible without the help of Akashdeep Dhar, Kevin Fenzi, David Kirwan, Michal Konecny, James Antill, Steve Milner, Shounak Dey and countless others who have either contributed to Fedora Infrastructure projects ecosystem or the createrepo_c project. We ask the readers to contribute to the project by developing, maintaining, testing or documenting the project.