Thanos

Term 1 · Mar-May · 1 project

• Add support for new PromQL aggregations

  LFX Mentorship · Listed in the official cohort

  Source IssueLFX portal

  Thanos ( has its own PromQL (Prometheus ( querying language) engine. The original PromQL engine recently added support for new aggregations. We are missing support for them in the Thanos PromQL engine ( In this project you will implement support for limitk and limit_ratio. See issue ( This will unblock users who need this functionality. The project is interesting because you will learn how query engines are implemented, about distributed query execution.

  • Go programming language experience

Term 3 · Sep-Nov · 1 project

• Add support for hedged requests

  LFX Mentorship · Listed in the official cohort

  @GiedriusSSource IssueLFX portal

  The long-tail requests sometimes are inevitable between the Thanos Store Gateway and an external cache. Lowering the timeouts between the store-gateway and the cache service isn't a greater way to address this problem. Using a HTTP client to issue hedged requests to object storage and other parts of Thanos' stack could help reduce tail latency by a lot.

  • Go
  • HTTP
  • Prometheus
  • Thanos

Term 2 · Jun-Aug · 1 project

• Compactor: Display TODO plan in UI

  LFX Mentorship · Listed in the official cohort

  @MichaHoffmannSource IssueLFX portal

  In the Thanos Compactor UI there is visibility of the global block list and loaded block list. But it would be useful to also have a list of planned and currently running compaction groups in order to see what exactly is in progress. This way it would be easier to diagnose what the Thanos Compactor is currently working on, and possibly what is delaying the progress of compactions. This is especially useful if you have large TSDB blocks in S3 buckets that take time to get compacted.

  • Go
  • React
  • some familiarity with Prometheus and Thanos

Term 3 · Sep-Nov · 2 projects

• Implement fan-out query observability in Thanos

  LFX Mentorship · Listed in the official cohort

  @GiedriusSSource IssueLFX portal

  In the previous mentorship sessions we added the foundation required for query observability in Thanos's new promql-engine and hooked it up in the UI. We now have the foundation to record telemetry from our query engine as well such as time consumed per operator. This project aims to expand on this and add more metadata to the query execution, both on the promql-engine operator tree level and Thanos Query Select() calls for fan-out query observability. Once we have this metadata, we would like to visualize it in the Query UI.

  • Go
  • React.js with TypeScript
  • Git + GitHub
  • Any Prometheus/PromQL/Thanos understanding is a plus

• Release the distributed query engine in Thanos

  LFX Mentorship · Listed in the official cohort

  @fpetkovskiSource IssueLFX portal

  The Thanos engine is capable of executing queries in a distributed manner, by pushing down aggregations to other querier nodes. This querying mode is not yet integrated well in the UI and is not exposed to users. The goal of this project is to add the needed integrations to the Thanos UI and officially release the feature to end users.

  • Go
  • React.js with TypeScript
  • Git + GitHub
  • Any Prometheus/PromQL/Thanos understanding is a plus

Term 2 · Jun-Aug · 1 project

• Continuation of add query observability for the new engine

  LFX Mentorship · Listed in the official cohort

  @saswatamcodeSource IssueLFX portal

  We have added solid foundation for query observability in the new engine during the previous LFX mentorship term. Let's continue the awesome work by Pradyumna by implementing other features.

  • Go
  • React

Term 1 · Mar-May · 3 projects

• Add query observability for new promql engine

  LFX Mentorship · Listed in the official cohort

  @GiedriusSSource IssueLFX portal

  The new Thanos Promql Engine lacks observability down to operator level and we don't have a way to track each operator's performance. This project aims to extend the Explain method of each operator, and return an operator tree with time taken recorded. Then Thanos Query UI could then visualize the operator trace.

  • Go
  • React

• Querying Apache Parquet files with PromQL

  LFX Mentorship · Listed in the official cohort

  @fpetkovskiSource IssueLFX portal

  The new Thanos PromQL Engine has a sufficient separation between the syntax tree and the execution plan to allow us to query arbitrary data sources. In this project we would like to explore ways to query data stored in Apache Parquet files.

  • Go

• Series Cardinality API

  LFX Mentorship · Listed in the official cohort

  @yeya24Source IssueLFX portal

  Prometheus has a TSDB stats API which contains information about series cardinality and the API is supported by Thanos. However, it can only return 10 results per stats, which is not flexible to track the arbitrary metrics. This project aims to design and implement APIs that expose cardinalities. Stretch goal can be to add cardinality explorer page to Thanos UI.

  • Go
  • React

Term 3 · Sep-Nov · 2 projects

• Load balancing of API communication in Thanos

  LFX Mentorship · Listed in the official cohort

  @bwplotkaSource IssueLFX portal

  Thanos uses gRPC for the majority of network communication. It performs fanouts and sharding of different queries to multiple nodes in a distributed system. Unfortunately, due to the nature of the gRPC, a conventional TCP-based load balancer (e.g. K8s Service) is not enough to distribute requests equally to multiple replicas of the same stateless node. As a result, there is a need to figure out the pragmatic way for Thanos users to load balance requests to multiple backends either by gRPC client load balancing or by guides and integration with popular load balancing proxies like nginx, caddy or envoy.

  • Go
  • HTTP
  • gRPC

• Receive: Support for tenant-specific external labels

  LFX Mentorship · Listed in the official cohort

  @fpetkovskiSource IssueLFX portal

  Tenants in Thanos Receivers currently get one external label which indicates their tenant ID. We would like to implement attaching arbitrary external labels to each Thanos Tenant. This functionality is useful for various different use cases, such as improving performance when querying data for tenants which share the same labels.

  • Go

Term 0 · 2022 Summer (Jun-Aug) · 1 project

• Implement Unified Endpoint Discovery

  LFX Mentorship · Listed in the official cohort

  @bwplotkaSource IssueLFX portal

  Thanos Querier microservice is one of the core component of the Thanos system responsible for asking relevent data stores for metrics, labels, metadata, exemplars, targets, alerts and more and merging their results. One of the key challenges to Querier configuration is telling which endpoints it should talk to and what APIs it should expect. For this we proposed the Unified Endoint Discovery idea, which allows consistency and easy to use configuration across all APIs. This project is meant to continue the implementation of this proposal and make sure it works well for all the edge cases using our e2e testing framework.

  • Go
  • DNS

Term 0 · 2022 Spring (Mar-May) · 1 project

• Run a community Thanos demo instance

  LFX Mentorship · Listed in the official cohort

  @GiedriusSSource IssueLFX portal

  Thanos is a distributed system that has a user interface written in React. Let's create a community instance with continuous integration for easy testing of how Thanos works. Also, it could serve as a testing ground for new React components. A server is provided by CNCF (<

  • Linux
  • Ansible
  • Python
  • Shell Scripting

Term 0 · 2021 Fall (Sep-Nov) · 3 projects

• Add groupcache (and improve) backend for the caching bucket in Thanos Store

  LFX Mentorship · Listed in the official cohort

  @GiedriusSSource IssueLFX portal

  This project is about implementing the groupcache back-end for the caching bucket. Caching bucket is a generic mechanism for caching requests to remote object storage. All of the other caching mechanisms currently suffer from the cache stampede problem and it is impossible to "properly" solve this problem with them. This is where groupcache comes in. Some work has already been done so you will have to something work off of. Groupcache also needs some improvements with regards to fetching multiple keys at once to improve the performance even more. In the end your work will make not just Thanos faster but also cheaper because less requests will need to be made to the remote object storage.

  • Go

• Add metrics to track the progress for compaction and downsampling

  LFX Mentorship · Listed in the official cohort

  @yeya24Source IssueLFX portal

  This project is about improving the observability of the Thanos compactor component to help user track the current progress. Thanos compactor usually has to deal with large TSDB blocks for compaction and downsampling. When the data are large, it takes a long time to finish the compaction or downsampling. Now users have to check the bucket UI manually to see whether the work is done. It would be better to have a more fine-grained way to tell the progress of compaction and downsampling. We can have some metrics to track the work and even integrate the stats into the UI.

  • Go

• Migrate Thanos to the New Protocol Buffers v2 API

  LFX Mentorship · Listed in the official cohort

  @squatSource IssueLFX portal

  This project is about updating and optimizing the protocol buffers implementation used for communication throughout the Thanos project. Thanos relies heavily on gRPC and protocol buffers for communication between its many components, and now with the release of the new Golang API for protocol buffers and the deprecation of the old API, Thanos should be updated to benefit from the improvements in the ecosystem. As part of this project, the protocol buffers generator used in Thanos will be migrated to one that supports the new API and brings other improvements, such as reducing memory allocations. This project will help make Thanos faster and ensure that we stay up to date with the latest libraries.

  • Go
  • protocol buffers

Term 0 · 2021 Summer (Jun-Aug) · 2 projects

• Descriptive API definitions using OpenAPI and Protobuf

  LFX Mentorship · Listed in the official cohort

  @bwplotkaSource IssueLFX portal

  In order to improve Thanos usage for users, we would like to define our APIs, both HTTP and gRPC, in protobuf/OpenAPI and expose the automatically generated documentation in the website. We also want to define the configuration of our components in protobuf. This would allow users to use tools for documentation, validation, type checking and even code generation to use our APIs efficiently. During this project we also expect collaboration with the Prometheus project to implement similar improvements on Prometheus' side. Optionally we would like to work on the index page on every Thanos component server that will expose those resources for easier debug.

  • Go
  • Protocol Buffers
  • Yaml (:

• Enhanced Block Viewer UI

  LFX Mentorship · Listed in the official cohort

  @onpremSource IssueLFX portal

  The Thanos BlockViewer UI has proven to be an essential part of the debuggability story for the Thanos project. It allows administrators to see the exact state of data in Object Storage in a provider-agnostic way. This project is about extending this UI with richer features, context, and actions to improve observability and increase control.

  • React
  • TypeScript
  • Go
  • ObjectStorage

Term 0 · 2021 Spring (Mar-May) · 4 projects

• gRPC Exemplars API

  LFX Mentorship · Listed in the official cohort

  @squatSource IssueLFX portal

  Exemplars are an amazing solution that allows linking metrics to logs, traces, and more! Recently Prometheus added support to Exemplars as defined by OpenMetrics API. In Thanos with our powerful deployment flexibility, we can allow federating Exemplars up to multi-cluster, global level! During this task mentee will develop together with mentors a new gRPC API that allows to access Prometheus exemplars on Thanos level. This is a work item bringing novel and edge technology to the open-source, which will enormously help Thanos users. During this mentorship, you will learn a lot about programming in Go, distributed Systems, gRPC Observability, Prometheus, Thanos!

  • Go
  • gRPC

• Multi-Tenant Instrumentation for Thanos operations

  LFX Mentorship · Listed in the official cohort

  @yashrsharma44Source IssueLFX portal

  Thanos can store and serve the data for multiple tenants at once. However, currently, Thanos does not always provide the needed introspective information about actions related to the tenant (e.g external labels). Allowing admins to obtain tenants’ information on per tenant queries, operations and ingestion would give actionable insight and answer questions such as: What data is used/queried the most for a tenant X? During this mentorship, you will implement logic that will enormously improve the experience of running multi-tenant Thanos on the scale. You will learn more about Go, instrumentation, multitenancy, APIs, and SRE concepts like SLOs.

  • Go
  • Prometheus (basic)
  • Instrumentation (basic)

• Stateless Ruler

  LFX Mentorship · Listed in the official cohort

  @bwplotkaSource IssueLFX portal

  Thanos Ruler is a critical component in Thanos that is responsible for the alert evaluation and recording rules. However, a few extensive rules can create a significant amount of resulting time-series, limiting the scalability of Thanos Rule, as it uses a single embedded TSDB. Recording/Alerting Rules are a substantial piece of monitoring infrastructure, so we want to ensure users can operate Rulers and scale them in an easy way. There is no way to scale rule evaluation and storage today except functionally sharding rules onto multiple instances of the Thanos Ruler component. Luckily, we have already solved scaling storage of time-series across various processes using Thanos Receiver. To scale rule evaluations and storage, during this mentorship, you will have a chance to implement the proposal that allows the Thanos rule component to have a stateless mode, storing results of queries by sending them to a Thanos receive hash-ring instead of storing them locally. You will learn about Go, Time-series databases, distributed system design, Prometheus, and of course Thanos.

  • Go

• Vertical Block Sharding

  LFX Mentorship · Listed in the official cohort

  @bwplotkaSource IssueLFX portal

  Current Thanos topology is generally horizontally scalable. However, the use cases and approaches of deploying Thanos shifted through time. While initially, Thanos was enabling ingestion through sidecars, now it’s not uncommon to see Thanos receiver usage. This means that the invariant of definite size TSDB block is no longer true. With offline deduplication and arbitrary Receive tenants data can be ingested into huge, often hundreds GB size TSDB blocks. This makes it harder to scale compaction and query operation on top of such blocks. The idea of this work is to vertically split larger blocks into smaller ones with the common scaling technique called sharding. As a mentee, we will guide you to make progress towards this goal by teaming up with experienced developers to deliver transparent automation for vertical block sharding! We are looking forward to working with you! During this mentorship, you will learn a lot about programming in Go, distributed Systems, TimeSeries Database, Prometheus, Thanos!

  • Go

Term 0 · 2020 Q3–Q4 · 3 projects

• Receive: Hashring Update Improvements

  CommunityBridge · Listed in the official cohort

  @squatSource IssueCommunityBridge

  Currently, any change to the hashring configuration file will trigger all Thanos Receive nodes to flush their multi-TSDBs, causing them to enter an unready state until the flush is complete. This unavailability during a flush allows for a clear state transition, however it can result in downtimes on the order of five minutes for every configuration change. We propose modifying how the Thanos Receive component re-configures itself after the hashring configuration file has changed so that the system experiences no downtime.

  • Thanos
  • Timeseries database
  • Go

• UI: Enhancements

  CommunityBridge · Listed in the official cohort

  @kakkoyunSource IssueCommunityBridge

  The Thanos project has recently migrated its UI to one built on re-usable and shareable components written in React, with the goal of fostering collaboration with the broader Prometheus community. As part of this proposal, we would like create further UI components like status, configuration and discovery pages for better debuggability as well as basic benchmarking and regressions test. Additionally the plan is to collaborate with the Prometheus community to continue building a shared UI component library and to contribute upstream to Prometheus so that it can leverage these components in its UI.

  • React
  • JavaScript
  • Go

• UI: Extending BlockViewer

  CommunityBridge · Listed in the official cohort

  @prmsrswtSource IssueCommunityBridge

  Thanos BlockViewer UI proven to be essential part of debuggability story for the Thanos proejct. It allows to see exact state of data in Object Storage in an provider agnostic way. This project is about extending this UI with richer features, context and actions to increase observability and control. As part of this proposal, we would like to also contribute the same BlockViewer to Prometheus community to make sure it gives the same value for Prometheus project.

  • React
  • JavaScript
  • Go
  • ObjectStorage

Term 0 · 2020 Q2 · 4 projects

• Complete Katacoda tutorials

  CommunityBridge · Listed in the official cohort

  @bwplotkaSource IssueCommunityBridge

  This is a great project for a writer with some passion for Docker and Kubernetes, interested in doing interactive docs: specifically learning to build Katacoda Course. Currently, we have a single Katacoda tutorial, which helps Thanos users to start up quickly and understand the concepts. To help improve the user experience further we have planned five more tutorials. The Katacoda tutorials are written in YAML format and don't require a lot of custom tooling. We have a moderate amount of docs on our website thanos.io, which should help you to get started quickly. Also, there are many youtube videos explaining the concepts. We can offer friendly community support and technical guidance and feedback. We're looking for one contributor to explore interactive tutorials and learn some technical concepts along the way. We are definitely not looking for perfection on the first try. Writing interactive tutorials: Intro: Downsampling and unlimited metric retention for Prometheus Intro: Global and meta alerts with Thanos Advanced: Connecting remote Prometheuses to Thanos using simple Envoy setup. Advanced: Using Prometheus remote write to stream metrics to Thanos Advanced: Query low tail latency with low cost: Introducing caching to Thanos

  • Distributed systems
  • Linux
  • Kubernetes
  • Docker

• Per Request Query Tracking and Limiting

  CommunityBridge · Listed in the official cohort

  @bwplotkaSource IssueCommunityBridge

  Thanos is at the very high-level: durable and cheap database capable of storing a very large amount of metric data. This means that querying that data can be expensive. Imagine someone queries years of data for millions of series. No matter how fast or efficient our indexing is, even the amount of data being passed through the network can get expensive. This is why users should have control, potentially someday per tenant to tell what query was performed, for how long and how much data it was using, and what happened or error. Users should also be able to limit such large queries.

  • go
  • distributed systems
  • Linux

• Thanos Query Frontend

  CommunityBridge · Proposed, acceptance not recorded

  @bwplotkaSource Issue

  As per proposal we plan to embedd Cortex query frontend as new Thanos command. With this project you will have opportunity to work in both Cortex and Thanos codebase to improve both worlds while creating shiny new command in Thanos project for this microservice proxy.

  • go
  • distributed systems
  • caching

• Versioned Website Docs

  CommunityBridge · Listed in the official cohort

  @squatSource IssueCommunityBridge

  The Thanos website includes a documentation area that contains per-component docs and configuration built by rendering the markdown files from the tip of master of the Thanos repository. This frequently causes confusion for users, as breaking changes are often merged into master that conflict with the APIs of previous releases. To solve this user-facing issue, we would like to allow the website to show a different set of docs for every Thanos release. This project will involve designing a documentation structure and version picker in the website to select different versions of documents. We will also need to design a workflow for managing docs that integrates with our Git workflow, i.e. updating corresponding docs on pull requests, cherry-picks, etc.

  • go
  • HTML
  • JavaScript
  • CSS

Term 0 · 2020 Q1 · 1 project

• Improving read write coordination for object storage and end to end benchmarking tests on demand through CI

  CommunityBridge · Listed in the official cohort

  Source