A Trip to Bulgaria and Release of the Sublimely Magnificent Node.js RM Webserver Mark III

My trip to Bulgaria has come to an end! I’ve been back in Belgium for a bit, processing the experience of Bulgaria and thinking about what I want to learn from it in the long term. I think there’s a word for precisely this kind of thing, but I can’t seem to remember it. Internalizing, pondering, evaluating, … and everything in between.

It only lasted for one week, but traveling to different countries pushes me as far out of my old comfort zone as I can imagine, so taking time for it is a must. Trying to take steps that are too big can move you backward rather than forward.

One small week with a lot of new and unexpected experiences.

Having only been there for a week, I can’t describe what life in Bulgaria is really like. Even my short time has provided me with plenty of experiences that I’ll take with me to the future, and even those experiences I can’t properly describe. I could approximate them, but rather than mess up the translation of the experience I just challenge you to visit Sofia yourself.

But here are some pictures to share with all of you!

Next up: Nope, not Reykjavík, Iceland! At least not yet. Having been to Bulgaria, and having taken the time to process everything, I’ve come to realize that my next best step is not to travel to the next unknown place yet. Unfamiliar places sound challenging and fun, but now is the time to go somewhere a little bit more familiar and to allow part of my life to settle. A place to keep my sense of home before I travel to the next unfamiliar place. That’s my real step forward.

So next up: Vienna, Austria!

And today marks the devkat release of The Sublimely Magnificent Node.js RM Webserver Mark III.

We’ve finished the last integrations and it’s ready to take over the Flask/werkzeug webserver’s task of serving dynamic data for RocketMap, except a tad more efficiently than Flask/werkzeug.

For those who don’t know, people who support RocketMap development via devkat’s Patreon get early access to our private Gitlab code repositories. Supporters of the rank “Grapefruit” (shh, don’t ask why it’s called that) get direct access to our development repositories, and “Not A Grapefruit” (in short: NAGs) supporters have an early release testing repository. NAGs get access to releases for two weeks to test the new releases; after those two weeks, it gets published to the open source community.

Update 20/11: As of today, the project is open source and released on Github! You can visit the project here. Future updates and reworks still follow the devkat release cycle described above. Big thanks to all of our Patreon supporters who made this possible in the first place.

To summarize, on top of the original version of the Node.js web server, we’ve worked on reworking the web server to be even better:

  • replaced Express.js with restify for an API-focused structure,
  • comes with built-in DTrace support,
  • comes with built-in configurable request throttling (rate limiting, default 5 req./s rate, 10 req./s burst),
  • added logging library for better configurability,
  • added HTTPS support,
  • removed Sequelize ORM in favor of the more “raw” approach with node-mysql, improving query performance by removing overhead and getting more control over SQL queries

Here are our latest benchmarks, straight from a production environment, tested with ApacheBench. Keep in mind that these benchmarks aren’t 1-on-1 implementation translations and they don’t even have the same set of features. This shouldn’t be used as a measure for Flask/werkzeug’s performance, it’s only meant to give realistic expectations for RocketMap hosters.

Flask/werkzeug:
 Time taken for tests: 201.058 seconds
 Complete requests: 10000
 Requests per second: 49.74 [#/sec] (mean)
 Time per request: 20.106 [ms] (mean, across all concurrent requests)
 100% 87801 (longest request)

The Sublimely Magnificent Node.js RM Webserver Mark III:
 Time taken for tests: 2.178 seconds
 Complete requests: 10000
 Requests per second: 4591.51 [#/sec] (mean)
 Time per request: 0.218 [ms] (mean, across all concurrent requests)
 100% 1222 (longest request)

As usual, these benchmarks depend on a lot of variables (e.g. hardware, OS, software versions, settings) so everyone will get different results, but I’m sure you’ll see some great improvements. 😉

Enjoy! ♥

New Releases and This Thing Called a Digital Nomad

Hi everyone 👋

How are you doing?

Part One: Let’s get you up-to-date.

If you’ve been following our activity on our Discord server you’ll know life has been busy.

Last week, we released RocketMap 4.1.0 (click here for full changelog), which included 4 policy updates, 2 breaking changes, 4 new features, 13 enhancements, 6 bugfixes and 3 documentation updates. It was a smooth release and brought some good stuff to RocketMap (especially –dump, which auto-uploads environment information to hastebin.com, for the heroes in our #help chat). We’re happy that all of you were happy!

Our activity is going great, even more than a year later.

RocketMap Visitors 26/09 - 09/10

We also had some fun with a fan art contest for our resident cow member of the community (“BishCow”, the bish cow). Everyone could submit their own fan art with “BishCow” as primary subject for a chance to win a copy of the Humble THQ Nordic PlayStation Bundle. One of our regulars, “Kartul”, won the contest with the following entry, hand-made by adding hundreds of Miltank (Pokémon) images on Google Maps.

A gallery of all entries is available right here!

Kartully wins!

Things aren’t always super happy and positive, this time as we decided to disclose that GoMan, a paid service that was being used by some of our users was intentionally breaking their customers’ security and privacy by downgrading HTTPS requests to HTTP to read/modify their clients’ requests.

GoMan proxies announcement

There was some backlash against the disclosure by some of the service’s users, but the positive support was overwhelming. We’d do it again in a heartbeat, it was the right decision. No ragrets.

Part Two: New & future release.

Since we’re not just a community of BishCow fan artists 😏, we did get more work done! Ha! A moment of complete surprise, I bet you’re staring at your computer screen in unreserved amazement.

🥁 … drum rolls … 🥁

(New) Release One: The Sublimely Magnificent Node.js RM Webserver Mark III.

I’ll admit, the name doesn’t really get across how awesome it is, but I couldn’t come up with a better one.

As with the Sublimely Magnificent Jecht Shot Mark III, there is no revision two, sorry!

The new version is immediately available on the private Gitlab for all “Grapefruit”-rank users on the devkat Patreon.

Compared with the previous version, which was based on Express.js + Sequelize, the new one:

  • uses restify instead of Express.js for a REST API-focused structure
  • comes with more advanced/configurable logging
  • supports HTTPS (yay security!)
  • has built-in dtrace support
  • now has built-in request throttling (default 5 req./s rate, 10 req./s burst)
  • removed Sequelize ORM in favor of the more “raw” approach with node-mysql, improving query performance by removing overhead

The Sublimely Magnificent Node.js RM Webserver Mark III is more efficient than its previous version and has a much better load profile and scaling thanks to the combination of old implementations (e.g. load limiter) with the new (request throttling).

If you’re not sure what the difference between those two is: a load limiter rejects requests when the overall server load is too high: accepting new requests queues more work while server load goes over its limit, leading to permanently delayed responses until the server crashes. Request throttling is a hard limit on the number of requests per second per IP which limits the maximum effect on server load per visitor.

Five Point Plant

source: http://ivanwlam.com

(Future) Release Two: RaidAPI.

We’ve been wanting to do something more for the Pokémon Go community as a whole rather than only improving the technical aspects of devkat and RocketMap, so we’ve decided to work on a RaidAPI.

In short, the RaidAPI is a plug-and-play community raid organization tool.

Similar to Google Analytics (or jQuery), you plug our JavaScript library into your website and you get access to all of its features. Sign up to join a raid at a specific time, sign out of a raid, get information about raid signups, …

After including the library, every website can call the internal methods directly via JavaScript, even websites that aren’t hosting RocketMap’s front-end. We wanted to be as inclusive as possible.

The RaidAPI library also includes pre-built user interfaces for all actions (sign up to a raid, see a raid’s status or signups, …), served via automatically sized modals to easily support all platforms and devices, even for non-RocketMap websites.

Best of all: this is entirely optional. All internal methods are publicly accessible and can be plugged into any website or user interface.

The RaidAPI will be fully implemented in all RocketMap websites, with an opt-in option. To put everyone’s mind at easy, rest assured we don’t store any of your server’s data (e.g. gyms or raids): we only store the required data to match a user to his/her signup.

Oh, and the data isn’t limited per website!

The RaidAPI is built to work across websites: to share data, and make raid participation global, bringing a more accessible real life community to everyone around the world, on all websites in near real-time.

It’s also entirely free, for everyone!

We’re working on some experiments such as a small banner advertisement at the bottom of our own interface which makes the whole platform self-sustainable so the platform can stay free for everyone, regardless of the amount of visitors. This is still in a very early testing phase, so all options are open.

Online Communities

source: https://xkcd.com/256/

Last Part: This Thing Called a Digital Nomad.

This part is something I haven’t told a lot of people yet.

Starting next Wednesday (two days from now), I’m starting my journey and leaving my home country (Belgium) permanently.

Yup, it’s finally happening.

There are a lot of things that led me to this point, and even more things that fuel me today, but in order not to bore you to death with a wall of text, I’ll keep it short.

Let’s compare it to what the internet calls “a digital nomad”.

The idea that someone travels from country to country, staying for an undetermined amount of time to learn and live the local culture and finding exactly those things/experiences that you can’t predict beforehand.

Previously, I’ve met some great friends who I’ve learned a whole lot from, and who are still supporting me. I’ve learned from those experiences and today, I want to be open-minded enough to meet peoples from all across the world. I hope to rely on the support they give me to take steps towards a new and exciting future.

I want to learn about the things that I can’t see from my (safe) comfort zone behind my computer at home. To see what I haven’t seen and to be who I haven’t been. To experience who I can be when I let myself.

Regardless of what happens or what I learn, I know that I’ll be happy to be able to look back on everything and say I did my best.

Calvin and Hobbes: Big Sunny Field

source: Calvin & Hobbes

The past months I’ve done a lot of preparation work to make this possible, especially in my freelancing work and by working on my own projects.

I don’t want this to sound too much like a “goodbye” article, so let’s look at the future. When possible, I’m going to share those future moments with you. Whether you’ve been here for a long time or you’re new, I’ll really hope you’ll stick around.

By doing my best, I hope to encourage some of you to take important (yet hard) steps in your own life that might be holding you back. And if you have your own stories to tell, I hope you’ll share them with me as well via the comments on this article, via email or even via Twitter (@SebVercammen).

The first stops are: Sofia, Bulgaria (next Wednesday, 18/10), followed by Reykjavík, Iceland, and Vienna, Austria. These places will take up the next few months, and we’ll decide on new areas as time goes on – I want to be able to stay long enough to have experienced the local life.

I don’t know what the future will bring, but I’m excited.

As always, devkat and RocketMap will continue to be maintained and updated.

source: GIF Keyboard

Benchmarking Node.js (Express) Webserver vs RocketMap’s Flask/werkzeug

One of RocketMap’s biggest limiting factors has always been its primitive development webserver (Flask/werkzeug). Because of its synchronous design (even the official docs specify it’s a development webserver and not intended to be used on production systems), it can’t be expected to properly support the load of a public, online webserver.

As RocketMap’s community grew and it started being used for projects and events, and it couldn’t even handle the webserver load created by just family and friends, it became clear that it needed to be replaced.

So I wrote a Node.js webserver with Express.

To keep it short, I:

  • separated the data API from the front-end’s static components: the new webserver only handles dynamic data requests,
  • added docs on the repository to use nginx (Linux)/apache2 (Windows) to serve static files (which nginx does amazingly well on Linux),
  • optimized SQL queries and added query limits,
  • added sorting to the queries to first send items closest to the center of the requested viewport,
  • removed whitespace from the JSON response (RocketMap originally sent “beautified” responses, the default of Flask’s jsonify),
  • added gzip compression,
  • added a load limiter that gracefully rejects new requests if the system is overloaded,
  • changed the webserver to use a persistent database connection pool (default 1 to 5 connections) instead of Flask/werkzeug creating hundreds of database connections (during our benchmark, we had to increase our connection limit to 1000 for Flask/werkzeug),
  • added a process manager with Node.js’ cluster to leverage multiprocessing on each CPU core (configurable), easily disabled to replace with a more robust process manager if preferred.

I’ve shared most of this information on Discord while I was building it, but the benchmarks and results are new for everyone.

About the load limiter.

The load limiter changes how the webserver behaves under impossible load. Impossible load is when the load on the webserver is above the limit of requests it can handle. Understanding how a webserver handles impossible load is important, because it determines how your webserver will behave when things unavoidably (or unexpectedly) go wrong.

Instead of continuing to queue up requests and wanting to handle all of them (but not being able to keep up), we intentionally fail gracefully by rejecting the request instantly (HTTP 503 – Service Unavailable). This article on Mozilla’s blog goes into more detail: Building A Node.JS Server That Won’t Melt – A Node.JS Holiday Season, part 5.

We care mostly about the results, so here is the webserver’s behavior when the load goes above the maximum and we’re not using a load limiter:

Webserver w/o load limiter

and this is how it behaves when we use a load limiter:

Webserver w/ load limiter

Since a load limiter will fail gracefully, not all responses are HTTP 200 OK, but the goal is to keep our servers alive during impossible load, not to respond to all requests (which is impossible, as the requests are intentionally above the limit of what the webserver can handle).

The benchmarks.

Note: These benchmarks aren’t meant to test each and every individual component that was used in building the new webserver. The comparison between Flask/werkzeug and Node.js/Express is not a fair one because they serve entirely different purposes (and werkzeug shouldn’t be used in any production environment), but these benchmarks show practical and direct results for our RocketMap users, giving an example of the real change in performance if they switch from Flask/werkzeug to the new Node.js webserver.

ApacheBench was used for benchmarking, with 1000 requests and 100 concurrency. The dataset (MariaDB database) was manually crafted to return the exact same dataset for both platforms (1000 active items in the viewport).

First, the total time our benchmark took:

Flask/werkzeug: 172.135s
devkat w/o load limiter: 19.609s
devkat w/ load limiter: 15.331s

The difference with the load limiter is because 900 out of 1000 requests were handled, while the other 100 requests failed gracefully due to the server being busy. This behavior reduces mean response time on handled requests due to the event loop being less busy.

Mean requests per second:

Flask/werkzeug: 5.81 RPS
devkat: 54 RPS

Transfer rate (w/ concurrency):

Flask/werkzeug: 4586.3 Kib/s
devkat: 31916.41 Kib/s

The mean time (in ms) it took to serve requests (load limiter is insignificant here, as it only comes into play when we’re overloaded):

Requests served

Flask/werkzeug’s behavior over time worsens while our new webserver remains more consistent. The fastest response time (request fully completed) for our new webserver (without load limiter) was 392ms, and slowest was 2492ms. For Flask/werkzeug, the fastest was 10845ms and the slowest was 23364ms.

 

That’s a success. 👍

Charity Fundraising: The Ocean Cleanup

Current charity donations: € 1553

End date: June 30th. Donate here.

I want to talk about something important to me.

Before I can get to my main point, I would like to share some background with you.

For too many years, my life was focused (almost exclusively) on technology. Learning to code, spending hours upon hours online each day. Exploring, learning, feeding the never-ending hunger for more information because of my excitement when I first discovered some people in an online chat who happened to be writing code. They had the ability to think of something and then actually build it. I thought it was amazing, and it absorbed me.

Problem: when you do something too much, it’s too much.

And for a long time, I forgot how to do everything else. I forgot how to human.

Computer Problems

source: https://xkcd.com/722/

Until recently.

As you might know, I freelance as a full stack developer (and software engineering), I build my own projects (startups) and I try to learn new technologies as often as I can. I try not to stand still. You would think that these things are relevant, or that they would somehow explain why/how things changed for me, but the truth is they’re not – and they don’t. But as I’ve come to notice, the most significant changes in my life have been due to the most absurd or unimaginable things possible.

However, I can describe the things that I do, the things that I focus on, and hope that it helps to explain who I was (or am) as a person when these things happen(ed). It’s the best I can do.

I can’t remember where I’ve first read this quote, but life is what happens while you are busy making other plans(According to Wikipedia, it was Allen Saunders.)

And so I was confronted with how to human.

I met some people who are nothing like me.

I’m going to try to explain who they are – or at least the effect they’ve had on me – as best as I can, but written text isn’t the best at conveying emotion, so for the next part I hope you try to imagine it as if it would be happening to you. I hope you can open up.

They aren’t involved as much in technology, they aren’t focused on building commercial projects, we don’t work in the same fields and we’re not even from the same country. They just happen to be exchange students and they’re working to get their Master’s degree in human ecology. And I somehow just happened to meet them.

Even though I was an absolute stranger to them, they chose not to be a stranger to me. They chose to be the most honest, open, selfless and supportive people I’ve ever met or seen, more than I’ve ever seen anyone even try to be. If you would ask me, I would not be able to come up with any reason for why I would have “deserved” it or for why it ultimately happened. It just did.

Change, especially significant personal change, can be scary and overwhelming… and it was. It was so overwhelming that it hit me like a truck. It was so much, so fast.

But they didn’t mind, and they always stayed themselves.

I thought I was doing pretty well, taking care of myself. But meeting them made it clear to me that I was forgetting something important. I wasn’t standing still to seriously think about everything and everyone else. I was (unintentionally) selfish because of my inaction.

Everything

source: https://xkcd.com/968/

They taught me a lot. In so little time, they changed me.

One of the things they reminded me of, is that we should remember to care for each other more. We shouldn’t just close our eyes and act as if we don’t notice. We shouldn’t say we don’t care, just because it’s not happening to us personally. We shouldn’t distract ourselves with trivial things so we don’t have to look at the hard stuff.

Today I want to give back to them. They are the reason I’m sharing all this with you, and I’m hoping that we can make something greater together.

I want to show them that their efforts are worth something, and that everyone can change the world in their own way if they truly want to.

The birthday of one of them happens to be next Monday the 12th and I want to give this as a gift. The gift of letting them know they’re making real change, that they’re bringing positive change to a lot of lives. The gift of significant change and real impact.

And I need your help to do it.

I’m hosting a charity fundraiser for The Ocean Cleanup.

I want to give something back that helps people improve our world, to help build a better future.

In short, The Ocean Cleanup develops advanced technologies to rid the world’s oceans of plastic. There are over 5 trillion pieces of plastic currently littering the ocean. Ocean currents concentrate plastic in five areas in the world: the subtropical gyres, also known as the world’s “ocean garbage patches”. Once in these patches, the plastic will not go away by itself.

Ocean garbage patches

source: https://www.theoceancleanup.com/

Since 2013, The Ocean Cleanup has improved its original design and plans and has gathered over 31.5 million USD in donations.

Some past milestones (read more here):

On May 11th, they unveiled the results of two years of work: an improvement to their design that will enable them to start extracting plastic within the next 12 months. With the first deployment in the Great Pacific Garbage Patch in the first half of 2018, they will start cleaning up two years ahead of schedule.

The Ocean Cleanup solves an important problem on a massive scale by intelligently using their technology. It’s a solution to a real problem that can be used in a matter of months.

From today on, we’re hosting a public fundraiser to gather as many donations as we can within this month (until the 30th).

To help The Ocean Cleanup, you can donate via this link: fundraiser has ended.

(We chose to use Paypal because other payment processors, including GoFundMe, take 4%+ of all donations.)

To start, I’ve donated € 350 myself and I’m adding more in a few days. This month, I will also give away 100% of contributions that I usually receive via Patreon.

Every day, I’ll update the total amount of raised donations at the top of this article.

I just hope that you’ll consider joining us in donating. Together, we can do more than when we are alone.

Please consider sharing the article (at the top of this article are some buttons!).

 

To anyone who has helped, from the bottom of my heart, thank you. ♥

devkat Progress Report: Webserver & Front-end

Since my last article about things going slightly wrong (PR 1900), things have already improved. First off, an immense thank you to everyone who decided to support us. It’s thanks to your support that we can keep working on what we love, in the way that we do things best.

To jump right into it, here are the first two major points we’re already working on:

  1. We’re replacing RocketMap’s Flask dev webserver (runs on Python).
  2. We’re building an entirely new modular front-end that will replace all current webpages with a much faster alternative.

1. We’re replacing RocketMap’s Flask dev webserver.

Flask is a development webserver that runs on Python – it’s not fast, it’s not particularly good either, and it’s even recommended to use only for development and never for production. Using it during RocketMap’s development was fine, as it was only used by developers, but ever since a community was built around RocketMap that has changed and it’s time to move forward to a proper webserver.

We’re replacing Flask with a fully asynchronous Node.js webserver on Express.js that uses Handlebars for templating (with enabled view caching), Sequelize for ORM, and that supports gzip compression and load limiting with toobusy-js.

In short, your webserver will support much more concurrent connections (Node.js’ asynchronous design is perfect for it), it will be able to scale up properly (and can easily be used with load balancing behind nginx) and the load limiter will automatically refuse requests when it notices the load on your server is too high, so instead of queuing up more requests waiting to be handled and further increasing the load, it will handle the requests it can handle and tell the rest to try again later.

Using nginx as a reverse proxy for Flask already helped support more users, but this new webserver will give you a codebase that can support concurrency on a much bigger scale. You no longer need to fiddle around to “get it to work”: it will now work properly by default and you can further scale up/optimize with other technologies such as nginx reverse proxy w/ caching.

2. We’re building an entirely new modular front-end (with additional JS abstraction layer) that will replace all webpages.

Yup, it’s happening.

Gone bootstrap, gone the finicky switches and nearly non-existent mobile support, gone the never-ending lag when too many Pokémon are on the map.

With a focus on responsive design, mobile support, minimal file size and performance, we’ve changed the old way of developing the front-end. With an entire new stack (Bourbon, Neat, npm and other modules), we’re writing responsive SCSS code to convert to minimal CSS (the file size is a fraction of what it used to be).

And what does the “modular” in the title mean? Well, we’ve built several optional customizable blocks into the front-end code, meaning you get a few blocks in which you can put your own HTML/CSS/JS code without having to edit our files. You can customize the front-end while still being able to update your codebase (git pull) without conflicts. For now, there is a customizable header title & icon, two customizable header content blocks (top left and top right corners of the screen), and optional customizable menu items.

This includes an additional abstraction layer in our JavaScript code to make it easier to control/manage the map and its markers from code outside of the repository’s code.

Besides the entire UI rework, it also includes a rewrite from scratch of all front-end JavaScript code with a focus on performance. Something that used to be very lacking has now become the main focus. The most important steps involve stepping away from slow libraries (e.g. jQuery) and using native JS, and rewriting the Google Maps markers to be rendered via WebGL.

If you’re not sure what this all means, just know that it’ll support millions of active markers without lagging; you’ll be able to zoom out to see the world map without having to hide Pokémon markers. Even on mobile.

(While right now it can’t even support a few tens of thousands without lagging on a desktop…)

303: It's Compiling

source: https://xkcd.com/303/

And as usual, there’s more coming.

The two points we’re working on are already massive changes compared to what has been changed in RocketMap’s history, and we’re going to need a fair bit of time to complete it, but these are the first steps towards an immensely more stable and more efficient RocketMap.

We have much more on our list of to-dos, some of which are smaller ones that we’ll work on during development of the above two points (e.g. the auto-verification mailserver we’ve recently released for all of our Patrons), and we’ll continue to keep you updated.

Remember that you can find us on our Discord, and that we’d love to hear your thoughts on these changes. ♥

 

…to boldly go where no man has gone before.

Enter devkat

By now, you know the daily routine: Niantic forces an API update, pgoapi gets updated, RocketMap implements the update. Or Niantic is silent for a while and RocketMap focuses on reworking old code, reviewing new code, and implementing new features. It sounds simple enough.

As my previous articles have explained, software development comes with its challenges, and open source adds some extra challenges. You might have noticed that things don’t always go smoothly: the merge of PR 1900 (SpeedScan improvements, by rocketrobot) and its unfortunate circumstances.

This article hopes to address these concerns and will explain our long term vision. Everyone currently in devkat has helped write and improve it, including the people you’ve known for a long time (Thunderfox, Dreggy, j4k3y and others).

For those who weren’t around, and for those who want an explanation.

RocketMap has a large workload but not enough people to finish everything instantly. This is, in itself, not an issue: having a work queue can actually be better than finishing everything instantly, as you can prepare a certain throughput of completed work per week rather than having moments where you’re just twiddling your thumbs. There’s always something interesting to work on.

For pull requests (open source work submitted by volunteers) to be accepted/merged, the only official requirement is two approvals on Github (preferably by a known #pr Discord member). Another requirement is to use common sense: there are often cases were two approvals are simply not enough.

Unfortunately, software code isn’t so simple that two people can click the “Approve” button and that we can expect everything to go perfectly for everyone if we decide to merge the changes. Computer Science is not something that can be based on “but it works for me!”, and this is often forgotten.

Which brings us back to PR 1900.

1900 was a large pull requests that didn’t do one thing, it tried to fix 12 separate issues in a single pull request. Common sense (or experience) would dictate to separate the pull request into 12 separate PRs and to review each one separately. Unfortunately, again, RocketMap is not an organization with official processes – until now, it never tried to be, in favor of being as open as it can be, meaning RocketMap was the result of all combined efforts put in by all the volunteers. A noble thought, but not always an efficient one.

Play Work

Source: http://management.curiouscatblog.net/2007/07/15/the-joy-of-work/

Besides having a large PR to review, the “approvals” on Github started pouring in: people who simply pulled the changes into their local installation, who ran it, and then approved the PR because “hey, it works for me!”, entirely ignoring the fact that thousands of other people are running our code on different types of setups with different kinds of hardware and configurations, and ignoring code quality and efficiency.

I want to repeat it one last time: Computer Science is not based on “hey, it works for me!”. (And maintaining a project to uphold a certain level of quality is not “perfectionism”, it’s just not being lazy.)

But with the few volunteers working on RocketMap, we ran into an issue: we didn’t have people around that could go into detail to properly review PR 1900. I could, but besides the large amount of time that goes into RocketMap (for no real life return), I also have a life to maintain (Oh, and I just became an uncle! My sister just had a baby, Elias.).

With no other option, I offered to review PR 1900 in its entirety, with the immense side note that it would take a lot of time and that people would need to be patient.

Patience is however not a virtue because it’s always easy, and this was demonstrated in the days while I was reviewing PR 1900. Even the people who had been contributing to RocketMap and who knew inside & out what I was up to (including my holiday, its duration, and I was even online on Discord while on a mountain with my snowboard strapped to my feet in France), ended up PM’ing me and poking me to merge 1900, fully aware that it would take much more time (I repeated this often enough).

Which brought us to a point where one thing became obvious: people didn’t want a review, they wanted a merge.

Mimi and Eunice - Patience

Source: http://mimiandeunice.com/2010/08/09/patience/

Those who really wanted a review (which includes rocketrobot, the author of the PR), were immensely patient and understanding. I thank those people for that. But everyone else was so obsessed with getting it merged because they personally liked the PR and “hey, it worked for them!” that they lost sight of everything else.

So I merged PR 1900 without finishing my review.

One main reason: RocketMap is open source. As far as our development went, everyone should have been equal. But I realized that this wasn’t the case: when it came to taking responsibility and carrying workload on our shoulders, I realized that everyone simply expected me to continue being responsible for the majority of what got merged. And this is wrong: an Approval on Github brings responsibility with it – you can’t approve something without also saying that you’re giving your word that you’ve done everything necessary to properly review the code, and that you’re ready to be partially reponsible for the merge.

PR 1900 had 12 different people who approved the PR, two people who just discussed it and one who said not to merge (me). 12 people…

So I merged.

Of course things went wrong. While it “worked for some”, there were inevitable side effects (which we knew beforehand would happen), and for others it broke their map entirely. And so everyone was suddenly hit with reality.

There are some who have held me personally responsible and who are angered at me for what happened. I can understand why, but I also feel that those points of view are ignoring what really happened for weeks. I will say that it’s immensely unfortunate that we annoyed so many people, but I also won’t apologize for my decision. I explained my thinking, and I still stand behind it.

So it happened – now we learn and improve.

It’s frustrating that it happened.

So how do fix it for the future?

First, I believe that, if RocketMap has done one thing very right, it’s that it has always continued to struggle for what it believes is right. The staff, the volunteers, even the community, has occasionally struggled but it was always in order to eventually learn and to improve. In some contexts, a struggle is exactly the best situation you can have, because it means no one is controlling anyone and everyone has the opportunity to participate.

But for software development, a struggle leads to delays, and we want to fix this. We want RocketMap, in every sense of what RocketMap has become, to be an enjoyable experience.

Enter devkat.

So far, you’ve only occasionally heard of devkat. In simple terms, it’s the group of people that has been responsible for several of the responsibility, coding work, merges, and brainstorming contributions surrounding RocketMap. For the public, it has (seemingly) only been the group that hosts vote.devkat.org, because that has been the only time so far that we used the devkat name to publish something.

And devkat is how we want to solve the RocketMap problems: structure, official processes, sustainability. All of it, we want to tackle it.

We’ll work to:

  1. Regularly brainstorm to improve how we work and what we work on.
  2. Have a real development roadmap, structure and formal processes. Eventually we hope to have more frequent public communication (roadmap, milestones, todos, releases).
  3. Contribute better and well-tested code to RocketMap.
  4. Increase the amount of hours we can work on RocketMap. More support means we can trade in more of our regular working hours for RocketMap.

I’ve said it before: by separating devkat from RocketMap, RocketMap can continue to be what it is, freely. And devkat can continue to experiment without being constrained to what people have come to expect (even demand) from RocketMap.

We’ve been working on devkat for a few months now behind the scenes because we always wanted to create a development environment that works best for us. We wanted to solve some concrete issues that had already shown up, and we wanted to have a place and the people to solve issues that might pop up in the future. What happened with PR 1900 was a good reminder and a sign for us that it’s time to continue and to publish what we’ve been working on.

I’ll personally be working exclusively with devkat. I’ll still be around RocketMap as often as I used to, but all of my code will go towards devkat’s way of doing things – a proper way of working will benefit all of us. I feel that we can discuss for a long time how it’s – at least in some ways – unfortunate that money is what brings sustainability, but ultimately it’s the way the world works and I’d rather be realistic and adapt than hold an idealistic view that will eventually kill the project.

In a few concrete steps (explained in detail afterwards):

  1. When we publish things made by devkat, we’ll officially announce it on devkat rather than silently contributing to RocketMap. This will help keep a clear focus on how we’ve been doing and what we can improve in devkat specifically.
  2. The core group of devkat will forever stay a small group: growth is not our focus, code quality and a better RocketMap is. We’ll be accessible via our Discord server (http://discord.devkat.org). You can always apply if you feel it’s the right thing to do, but accepting or even replying is entirely at our own discretion – our focus comes first.
  3. To solve the problem of not having formal processes, leading to untested/unreviewed code (e.g. PR 1900) or other roadblocks, devkat will host its own code on a private repository which we’ll maintain in a more formal way. There will be a way for contributors to formally contribute which will maintain our code quality, and there will be a way for interested people to participate in testing or using the code before it’s publicly released to RocketMap (more on this later). This repository will contain code at least two weeks before it gets released publicly to RocketMap.
  4. Finished contributions to the private devkat repository will be tested and reviewed in private for at least two weeks before releasing the code to the main RocketMap repository (per pull request, as usual). Because of how we’ll work, we’re more than ready to take full responsibility for any devkat merges (so none of that PR 1900 again).
  5. When available, we’ll provide paid hourly support for everything related to devkat code, setting up RocketMap, 3rd party tools (e.g. account gens) and unreleased/unmerged code (e.g. pull requests or devkat’s experimental code), from A to Z, including debugging and accessing private/custom setups. A quote can always be requested.

You can already visit the devkat Discord server or its Patreon profile.

RocketMap itself will stay exactly as it is. It’s still an open organization, it still stays on Github, it still stays free (and remember: devkat releases for free to RocketMap two weeks after it publishes to its private repositories). If you want a more formal explanation, devkat is a separation of concerns and responsibilities, to allow devkat to experiment without having to affect what RocketMap is as an organization and community.

#3 and #4: devkat Patreon, Roles and Private Repository

devkat’s private repository and closed group will maintain a specific level of code quality. Stricter rules and development processes, things that can’t (or that we don’t always want to) expect or enforce in an open source project like RocketMap. We want to try to make RocketMap better by doing what we do best, and we want to do this by creating a sustainable, better environment for our developers, which will let us contribute better code to RocketMap.

Via Patreon pledges (devkat Patreon profile), you can financially support our efforts and get read access to the private repository (more details in the pledge options on Patreon), and we use Paypal for the private support (we set up a unique invoice for each request, just PM us on devkat’s Discord or send an e-mail to hello@devkat.org).

1. To solve the problem of sustainability, devkat will offer paid read access to its private repository.

Some of our Patreon pledges includes read access to our private repository. By contributing, you get instant access to all finished code merges at least two weeks before it gets released publicly to RocketMap. Our higher Patreon contributors who want to support us more can also get access to our experimental code – this is similar to Github’s “pull requests”, except that it’ll be access to exclusively devkat code before it gets released to the devkat private repository for basic pledgers.

After you’ve pledged and you’ve received your Discord rank, just PM us on Discord or send an e-mail to hello@devkat.org to request access to the repository. We only send your login details after you’ve requested them, so make sure to let us know you want access.

2. We’ll provide paid hourly support for RocketMap, devkat, and 3rd party tools.

As another way to support the development, some of our members will be open to setting things up for you when it gets too difficult. Whether it’s the basics of getting set up, helping you purchase whatever is required (proxies, hashing keys, servers) or helping you debug your custom installation/configuration, we can help you out.

If you need our help, PM us on Discord or send an e-mail to hello@devkat.org. We’ll give you a quote, and all invoices will be sent via Paypal (We’re a registered company, we pay our taxes. It’s all good. ♥).

3. Volunteer contributors can earn bounties by contributing to devkat.

To motivate the developers who take the time to do things properly, we’re going to work with a bounty system. When you, as a developer, want to write code for RocketMap, we want to give you two options:

  1. You release to RocketMap as usual and go through the old process.
  2. You contact us via devkat’s Discord and you release to devkat’s private repository. Your code will go devkat’s usual process of review by its members, followed by a thorough testing process, on to the paid read-only repository for our supportive community members. Two weeks later, it’s at its final stage and has been properly reviewed, after which it gets merged publicly into RocketMap.

Bounty Hunter Mary Poppins

If you choose to work the devkat way, you’ll earn a bounty for your work. All bounties depend on the type and volume of your work, and everything surrounding the bounties is communicated before you submit anything to devkat. No surprises. We also welcome any volunteers who simply want to contribute code to be reviewed by devkat members.

PR contributors who have an open PR and want to move it to devkat before it gets merged can do so by PM’ing us on Discord. Just reminder that it’s first come first served, we can’t (and won’t) take up more work than we can handle.

As an important side note, contributing code does not require you to have a paid Patreon pledge. We received a question about this, and we hope this clears it up.

4. And once we get the ball rolling, we invest more into RocketMap.

Public projects, hosting servers, distributing data, investing in our own RE team for a free API, writing dynamic/polymorphic systems to permanently block scrapers, … everything we can try, we will try.

RocketMap is what we’ve always been and what we’ll always be about. We’ll continue to work on improving RocketMap and on contributing new items that will make it easier to set up and use, so we can get back to a time where people used the maps to play together as groups of friends and family. Only this time, more professionally.

It’s time to make RocketMap great again.

Donald Trump

We’re not going to build any walls (at least not unless they make sense for RocketMap), but I believe we can use a bit of their optimism to work towards a more positive message: making RocketMap great(er).

Continuously experimenting and trying new things to learn from, so we can see for ourselves what really works.

Thank you for your continued support. ♥

 

Today I don’t leave you with a song but with a message.

I Got 25.5 Million Visits in 45 Days

Twenty-five million, five hundred thousand visits in only 45 days. Let that sink in for a minute.

25.5 Million Pageviews

If the entire country of Belgium would visit my website once per day, it would take them three days to reach that amount of visits.

Every single person in Belgium, their fathers and mothers, their grandmothers and grandfathers, all of the babies and everyone’s friends – and three days.

Google Analytics Pageviews

Wait, what project?

I recently launched a project about Pokémon Go (RareSpawns.be). The concept is simple: in the Pokémon games, there are some Pokémon that are considered rare; they are hard to get or rarely appear in the game at all.

In Pokémon Go, a rare Pokémon is even rarer than in the regular games because we can’t control the spawns: we have to wait for Niantic to create one, and we’ve all seen what happens when this happens in a dense area.

A rare Pokémon can move the world.

The website is a realtime, crowdsourced platform that shows where some of the rare Pokémon have spawned in the world – it includes their coordinates, their potential (called their IV, displayed as a percentage) and the in-game moves the Pokémon can use.

The secret to getting so many visitors?

There is no secret.

The project did what it had to and it supports a huge amount of concurrent visitors without delay. During the peak of the activity, I remodeled and rewrote the entire program’s structure to support the activity and the inevitable 24/7 DDoS attacks – in the end, it took ~2 weeks of full-time development to make it what it is today (the tech stack deserves a separate article).

I launched, went to bed with ~250 concurrent users, and I woke up to a few thousand.

I worked, went to bed again, and woke up with hundreds of thousands of visits.

Google Analytics Realtime

What did we learn? The more I go to bed, the more visits I get.

People loved it so much, they started to attack it.

After a certain amount of popularity, the attacks started. 24/7 DDoS attacks that just never stopped.

After the rework of the structure, I had separated all of the work/responsibilities between multiple instances. There was no single point of failure and the most important ones (i.e. the data entry points and API) were no longer running on the same instance as the instance which serves the real-time feed on the front-end.

In simpler terms, it means they could attack all they want and they could even send it all as web traffic, it still wouldn’t increase the load on the most important instances. Oh, and the front-end feed was being served separately from the webserver that was hosting the files – every server/instance does exactly what it has to, nothing more.

OVH DDoS

And the number of active database connections I was using during all of this? One. A single connection. Built almost entirely on Node.js, I needed only one instance with a single database connection to handle all of it.

And I’ve got to hand it to my host: OVH has an amazing infrastructure.

Up next…

It’s been amazing. I’ve learned a lot and I couldn’t have hoped for it to get as popular as it did.

I’ve met a lot of interesting people along the way (PokemonGo-Map team, pogoapi and all of our regulars) and I want to add a special thank you for the people that have always helped me to stay on track and work on my goals every single day.

Up next is a completely different project. Another day, another challenge. 🙂

And what about you?

Do you have any projects you’d like to talk about, or any challenges in your near future?

Let me know in the comments, I want to know all about it!

Psst, in the meantime, enjoy this song from twenty one pilots. Time to relax.