High-Performance Ingestion with Python and Swarm64DB

. </div>

Abstract

Tags: python

Swarm64DB is a hardware-accelerated plugin for PostgreSQL and other RDBMS. By using Swarm64DB in combination with PostgreSQL, Python and the right scaling mechanism, we are able to push the ingestion throughput into areas where Python can easily compete with compiled languages. The talk highlights the architecture of our solution and showcases a real world use-case..

Description

In August 2016, a benchmark about asyncpg, a new PostgreSQL client library for Python was published. It highlighted ingestion speeds of up to 900k rows per second for a synthetic benchmark with a single byte per row. However, in realistic use-cases the ingestion speed is usually below that. Surprisingly it is not Python which is the limiting factor but rather the database itself. To make traditional RDBMS ready for Big Data, high performing OLAP and fast analysis on high speed data streams, Swarm64 created Swarm64DB, a hardware-accelerated plugin for PostgreSQL and other RDBMS. By using Swarm64DB in combination with PostgreSQL, Python and the right scaling mechanism, we are able to push the ingestion throughput into areas where Python can easily compete with compiled languages. The talk highlights the architecture of our solution and showcases a real world use-case.