Facebook Thrift

Thrift is a serialization and RPC framework for service communication. Thrift enables these features in all major languages, and there is strong support for C++, Python, Hack, and Java. Most services at Facebook are written using Thrift for RPC, and some storage systems use Thrift for serializing records on disk.

Facebook Thrift is not a distribution of Apache Thrift. This is an evolved internal branch of Thrift that Facebook re-released to the open source community in February 2014. Facebook Thrift was originally released closely tracking Apache Thrift but is now evolving in new directions. In particular, the compiler was rewritten from scratch and the new implementation features a fully asynchronous Thrift server. Read more about these improvements in the ThriftServer documentation.

You can also learn more about this project in the original Facebook Code blog post.

Table of Contents

• About Thrift
  • A Code Generator
  • A Serialization Framework
  • An RPC Framework
• Building
  • Build with CMake
  • getdeps.py
  • Thrift Files
  • Python Build (thrift-python)
• C++ Static Reflection
• C++ Server Metrics

About Thrift

At a high level, Thrift is three major things:

A Code Generator

Thrift has a code generator which generates data structures that can be serialized using Thrift, and client and server stubs for RPC, in different languages.

A Serialization Framework

Thrift has a set of protocols for serialization that may be used in different languages to serialize the generated structures created from the code generator.

An RPC Framework

Thrift has a framework to frame messages to send between clients and servers and to call application-defined functions when receiving messages in different languages.

There are several key goals for these components:

• Ease of use: Thrift takes care of the boilerplate of serialization and RPC and enables the developer to focus on the schema of the system's serializable types and on the interfaces of the system's RPC services.

• Cross-language support: Thrift enables intercommunication between different languages. For example, a Python client communicating with a C++ server.

• Performance: Thrift structures and services enable fast serialization and deserialization, and its RPC protocol and frameworks are designed with performance as a feature.

• Backwards compatibility: Thrift allows fields to be added to and removed from serializable types in a manner that preserves backward and forward compatibility.

Building

Build with CMake

The CMake build fetches any dependency it cannot find installed, so a clone and two commands are enough:

git clone https://github.com/facebook/fbthrift
cd fbthrift
cmake -B _build
cmake --build _build

This needs CMake 3.24 or newer, a C++20 compiler, and zlib and Zstd installed. Everything else, including Folly, Fizz, Wangle, mvfst and their dependencies, is used as-is if already on the system and fetched otherwise. Alternatively, install the system packages first:

./build/fbcode_builder/getdeps.py install-system-deps --recursive fbthrift

The build puts its outputs in _build:

• _build/bin/thrift: The Thrift compiler binary to generate client and server code.
• _build/lib/libthriftcpp2.a: Library for clients and servers.

Tests are not built by default. Configure with -DTHRIFT_TESTS=ON to build them. They need GTest and, unless THRIFT_BENCHMARKS is off, Google Benchmark installed; neither is fetched:

cmake -B _build -DTHRIFT_TESTS=ON
cmake --build _build
(cd _build && ctest)

If configure had to fetch Fizz, Wangle, mvfst or xxHash, it turns install rules off, because an installed Thrift would refer to libraries that exist only in the build tree. The configure output lists them; install them and reconfigure to get an installable Thrift.

CMake options:

• THRIFT_RPC (default ON): build the RPC transport stack and its dependencies (Fizz, Wangle, mvfst).
• THRIFT_TESTS (default OFF): build the tests.
• THRIFT_BENCHMARKS (default ON): build the benchmarks along with the tests.
• THRIFT_PYTHON (default OFF): build the thrift-python library and extensions; requires Cython.

getdeps.py

getdeps.py also builds Thrift's sibling projects and is what CI uses. Prefer the CMake build above unless you need that.

# Build, using system dependencies if available
./build/fbcode_builder/getdeps.py --allow-system-packages build fbthrift

getdeps.py will invoke cmake etc and put output in its scratch area (you can see in logs, and can override with --scratch-path), including installed/fbthrift/bin/thrift and installed/fbthrift/lib/libthriftcpp2.a.

If you want to invoke cmake again to iterate, there is a helpful run_cmake.py output in the scratch build/fbthrift directory.

Thrift Files

When using thrift and the CMake build system, include: ThriftLibrary.cmake in your project. This includes the following macro to help building Thrift files:

thrift_library(
  #file_name
  #services
  #language
  #options
  #file_path
  #output_path
)

This generates a library called file_name-<language>. That is, for Test.thrift compiled as cpp2, it will generate the library Test-cpp2. This should be added as a dependency to any source or header file that contains an include to generated code.

Python Build (thrift-python)

Build and install the thrift-python wheel for Python development.

Container Build with Docker BuildKit (Recommended)

docker buildx build -t fbthrift-python-build -f .devcontainer/Containerfile .
docker run -it -v $(pwd):/fbthrift -w /fbthrift fbthrift-python-build bash

Build and Test Commands

apt-get update
python3 build/fbcode_builder/getdeps.py install-system-deps --recursive fbthrift-python
python3 build/fbcode_builder/getdeps.py build fbthrift-python
python3 build/fbcode_builder/getdeps.py test fbthrift-python
pip3 install $(python3 build/fbcode_builder/getdeps.py show-inst-dir fbthrift-python)/share/thrift/wheels/thrift-*.whl
python3 -c "from thrift.python.types import StructMeta; print('thrift-python installed successfully')"

Note: The wheel is portable and can be installed into any compatible Python environment (Python 3.8+).

Other Ways to Build

Non-Container Build:

sudo apt-get update
sudo apt-get install -y python3-dev patchelf
pip3 install cython auditwheel
python3 build/fbcode_builder/getdeps.py --allow-system-packages install-system-deps --recursive fbthrift-python
python3 build/fbcode_builder/getdeps.py --allow-system-packages build fbthrift-python
python3 build/fbcode_builder/getdeps.py --allow-system-packages test fbthrift-python

Build with Conda:

conda create -n thrift-python-build python=3.10
conda activate thrift-python-build
conda install cython=3.2.2
pip3 install auditwheel cmake==3.28.0
sudo apt-get update
sudo apt-get install -y patchelf
python3 build/fbcode_builder/getdeps.py --allow-system-packages install-system-deps --recursive fbthrift-python
python3 build/fbcode_builder/getdeps.py --allow-system-packages build fbthrift-python
python3 build/fbcode_builder/getdeps.py --allow-system-packages test fbthrift-python

Building Containers with Other Runtimes:

Legacy Docker:

docker build -t fbthrift-python-build -f .devcontainer/Containerfile .
docker run -it -v $(pwd):/fbthrift -w /fbthrift fbthrift-python-build bash

Podman:

podman build -t fbthrift-python-build -f .devcontainer/Containerfile .
podman run -it -v $(pwd):/fbthrift -w /fbthrift fbthrift-python-build bash

Install the wheel (for all non-container builds):

pip3 install $(python3 build/fbcode_builder/getdeps.py show-inst-dir fbthrift-python)/share/thrift/wheels/thrift-*.whl

C++ Static Reflection

Information regarding C++ Static Reflection support can be found under the static reflection library directory, in the corresponding README file.

C++ Server Metrics

To collect runtime stats from a Thrift server, e.g. the number of active requests/connections, the C++ Thrift server supports an observer API that installs callbacks at a set of specific execution points in the server.

To expose collected metrics out of the server process, one way is to use fb303 interfaces, see fb303 Github repo.

License

Facebook Thrift is Apache 2.0 licensed.