Scale with NSQ: a realtime distributed messaging system

June 01, 2015 12:22

Berlin Buzzwords 2015

NSQ is an open source realtime distributed messaging platform designed to operate at scale and handle billions of messages per day. It is built entirely in Go and promotes distributed and decentralized topologies without single points of failure. This enables fault tolerance and high availability coupled with a reliable message delivery guarantee.

NSQ was built by the engineering team at Bitly to support its high volume even stream processing architecture. It powers our near real-time analytics systems and is now used by a range of companies including Docker, Stripe and BuzzFeed.

During this talk we will answer questions such as why was NSQ built and how it fits into the larger picture. We will also take a look at some code examples. By using real-life scenarios that we faced at Bitly, we will discusss how NSQ deals with concepts of availability, performance, latency, fault tolerance and how it's enabled us to scale effectively. We will also discuss why Go was an excellent language to use for building NSQ.



Georgi Knox


more decks of the speaker