A Trip to Bulgaria and Release of the Sublimely Magnificent Node.js RM Webserver Mark III

My trip to Bulgaria has come to an end! I’ve been back in Belgium for a bit, processing the experience of Bulgaria and thinking about what I want to learn from it in the long term. I think there’s a word for precisely this kind of thing, but I can’t seem to remember it. Internalizing, pondering, evaluating, … and everything in between.

It only lasted for one week, but traveling to different countries pushes me as far out of my old comfort zone as I can imagine, so taking time for it is a must. Trying to take steps that are too big can move you backward rather than forward.

One small week with a lot of new and unexpected experiences.

Having only been there for a week, I can’t describe what life in Bulgaria is really like. Even my short time has provided me with plenty of experiences that I’ll take with me to the future, and even those experiences I can’t properly describe. I could approximate them, but rather than mess up the translation of the experience I just challenge you to visit Sofia yourself.

But here are some pictures to share with all of you!

Next up: Nope, not Reykjavík, Iceland! At least not yet. Having been to Bulgaria, and having taken the time to process everything, I’ve come to realize that my next best step is not to travel to the next unknown place yet. Unfamiliar places sound challenging and fun, but now is the time to go somewhere a little bit more familiar and to allow part of my life to settle. A place to keep my sense of home before I travel to the next unfamiliar place. That’s my real step forward.

So next up: Vienna, Austria!

And today marks the devkat release of The Sublimely Magnificent Node.js RM Webserver Mark III.

We’ve finished the last integrations and it’s ready to take over the Flask/werkzeug webserver’s task of serving dynamic data for RocketMap, except a tad more efficiently than Flask/werkzeug.

For those who don’t know, people who support RocketMap development via devkat’s Patreon get early access to our private Gitlab code repositories. Supporters of the rank “Grapefruit” (shh, don’t ask why it’s called that) get direct access to our development repositories, and “Not A Grapefruit” (in short: NAGs) supporters have an early release testing repository. NAGs get access to releases for two weeks to test the new releases; after those two weeks, it gets published to the open source community.

Update 20/11: As of today, the project is open source and released on Github! You can visit the project here. Future updates and reworks still follow the devkat release cycle described above. Big thanks to all of our Patreon supporters who made this possible in the first place.

To summarize, on top of the original version of the Node.js web server, we’ve worked on reworking the web server to be even better:

  • replaced Express.js with restify for an API-focused structure,
  • comes with built-in DTrace support,
  • comes with built-in configurable request throttling (rate limiting, default 5 req./s rate, 10 req./s burst),
  • added logging library for better configurability,
  • added HTTPS support,
  • removed Sequelize ORM in favor of the more “raw” approach with node-mysql, improving query performance by removing overhead and getting more control over SQL queries

Here are our latest benchmarks, straight from a production environment, tested with ApacheBench. Keep in mind that these benchmarks aren’t 1-on-1 implementation translations and they don’t even have the same set of features. This shouldn’t be used as a measure for Flask/werkzeug’s performance, it’s only meant to give realistic expectations for RocketMap hosters.

Flask/werkzeug:
 Time taken for tests: 201.058 seconds
 Complete requests: 10000
 Requests per second: 49.74 [#/sec] (mean)
 Time per request: 20.106 [ms] (mean, across all concurrent requests)
 100% 87801 (longest request)

The Sublimely Magnificent Node.js RM Webserver Mark III:
 Time taken for tests: 2.178 seconds
 Complete requests: 10000
 Requests per second: 4591.51 [#/sec] (mean)
 Time per request: 0.218 [ms] (mean, across all concurrent requests)
 100% 1222 (longest request)

As usual, these benchmarks depend on a lot of variables (e.g. hardware, OS, software versions, settings) so everyone will get different results, but I’m sure you’ll see some great improvements. 😉

Enjoy! ♥

devkat Progress Report: Webserver & Front-end

Since my last article about things going slightly wrong (PR 1900), things have already improved. First off, an immense thank you to everyone who decided to support us. It’s thanks to your support that we can keep working on what we love, in the way that we do things best.

To jump right into it, here are the first two major points we’re already working on:

  1. We’re replacing RocketMap’s Flask dev webserver (runs on Python).
  2. We’re building an entirely new modular front-end that will replace all current webpages with a much faster alternative.

1. We’re replacing RocketMap’s Flask dev webserver.

Flask is a development webserver that runs on Python – it’s not fast, it’s not particularly good either, and it’s even recommended to use only for development and never for production. Using it during RocketMap’s development was fine, as it was only used by developers, but ever since a community was built around RocketMap that has changed and it’s time to move forward to a proper webserver.

We’re replacing Flask with a fully asynchronous Node.js webserver on Express.js that uses Handlebars for templating (with enabled view caching), Sequelize for ORM, and that supports gzip compression and load limiting with toobusy-js.

In short, your webserver will support much more concurrent connections (Node.js’ asynchronous design is perfect for it), it will be able to scale up properly (and can easily be used with load balancing behind nginx) and the load limiter will automatically refuse requests when it notices the load on your server is too high, so instead of queuing up more requests waiting to be handled and further increasing the load, it will handle the requests it can handle and tell the rest to try again later.

Using nginx as a reverse proxy for Flask already helped support more users, but this new webserver will give you a codebase that can support concurrency on a much bigger scale. You no longer need to fiddle around to “get it to work”: it will now work properly by default and you can further scale up/optimize with other technologies such as nginx reverse proxy w/ caching.

2. We’re building an entirely new modular front-end (with additional JS abstraction layer) that will replace all webpages.

Yup, it’s happening.

Gone bootstrap, gone the finicky switches and nearly non-existent mobile support, gone the never-ending lag when too many Pokémon are on the map.

With a focus on responsive design, mobile support, minimal file size and performance, we’ve changed the old way of developing the front-end. With an entire new stack (Bourbon, Neat, npm and other modules), we’re writing responsive SCSS code to convert to minimal CSS (the file size is a fraction of what it used to be).

And what does the “modular” in the title mean? Well, we’ve built several optional customizable blocks into the front-end code, meaning you get a few blocks in which you can put your own HTML/CSS/JS code without having to edit our files. You can customize the front-end while still being able to update your codebase (git pull) without conflicts. For now, there is a customizable header title & icon, two customizable header content blocks (top left and top right corners of the screen), and optional customizable menu items.

This includes an additional abstraction layer in our JavaScript code to make it easier to control/manage the map and its markers from code outside of the repository’s code.

Besides the entire UI rework, it also includes a rewrite from scratch of all front-end JavaScript code with a focus on performance. Something that used to be very lacking has now become the main focus. The most important steps involve stepping away from slow libraries (e.g. jQuery) and using native JS, and rewriting the Google Maps markers to be rendered via WebGL.

If you’re not sure what this all means, just know that it’ll support millions of active markers without lagging; you’ll be able to zoom out to see the world map without having to hide Pokémon markers. Even on mobile.

(While right now it can’t even support a few tens of thousands without lagging on a desktop…)

303: It's Compiling

source: https://xkcd.com/303/

And as usual, there’s more coming.

The two points we’re working on are already massive changes compared to what has been changed in RocketMap’s history, and we’re going to need a fair bit of time to complete it, but these are the first steps towards an immensely more stable and more efficient RocketMap.

We have much more on our list of to-dos, some of which are smaller ones that we’ll work on during development of the above two points (e.g. the auto-verification mailserver we’ve recently released for all of our Patrons), and we’ll continue to keep you updated.

Remember that you can find us on our Discord, and that we’d love to hear your thoughts on these changes. ♥

 

…to boldly go where no man has gone before.

Enter devkat

By now, you know the daily routine: Niantic forces an API update, pgoapi gets updated, RocketMap implements the update. Or Niantic is silent for a while and RocketMap focuses on reworking old code, reviewing new code, and implementing new features. It sounds simple enough.

As my previous articles have explained, software development comes with its challenges, and open source adds some extra challenges. You might have noticed that things don’t always go smoothly: the merge of PR 1900 (SpeedScan improvements, by rocketrobot) and its unfortunate circumstances.

This article hopes to address these concerns and will explain our long term vision. Everyone currently in devkat has helped write and improve it, including the people you’ve known for a long time (Thunderfox, Dreggy, j4k3y and others).

For those who weren’t around, and for those who want an explanation.

RocketMap has a large workload but not enough people to finish everything instantly. This is, in itself, not an issue: having a work queue can actually be better than finishing everything instantly, as you can prepare a certain throughput of completed work per week rather than having moments where you’re just twiddling your thumbs. There’s always something interesting to work on.

For pull requests (open source work submitted by volunteers) to be accepted/merged, the only official requirement is two approvals on Github (preferably by a known #pr Discord member). Another requirement is to use common sense: there are often cases were two approvals are simply not enough.

Unfortunately, software code isn’t so simple that two people can click the “Approve” button and that we can expect everything to go perfectly for everyone if we decide to merge the changes. Computer Science is not something that can be based on “but it works for me!”, and this is often forgotten.

Which brings us back to PR 1900.

1900 was a large pull requests that didn’t do one thing, it tried to fix 12 separate issues in a single pull request. Common sense (or experience) would dictate to separate the pull request into 12 separate PRs and to review each one separately. Unfortunately, again, RocketMap is not an organization with official processes – until now, it never tried to be, in favor of being as open as it can be, meaning RocketMap was the result of all combined efforts put in by all the volunteers. A noble thought, but not always an efficient one.

Play Work

Source: http://management.curiouscatblog.net/2007/07/15/the-joy-of-work/

Besides having a large PR to review, the “approvals” on Github started pouring in: people who simply pulled the changes into their local installation, who ran it, and then approved the PR because “hey, it works for me!”, entirely ignoring the fact that thousands of other people are running our code on different types of setups with different kinds of hardware and configurations, and ignoring code quality and efficiency.

I want to repeat it one last time: Computer Science is not based on “hey, it works for me!”. (And maintaining a project to uphold a certain level of quality is not “perfectionism”, it’s just not being lazy.)

But with the few volunteers working on RocketMap, we ran into an issue: we didn’t have people around that could go into detail to properly review PR 1900. I could, but besides the large amount of time that goes into RocketMap (for no real life return), I also have a life to maintain (Oh, and I just became an uncle! My sister just had a baby, Elias.).

With no other option, I offered to review PR 1900 in its entirety, with the immense side note that it would take a lot of time and that people would need to be patient.

Patience is however not a virtue because it’s always easy, and this was demonstrated in the days while I was reviewing PR 1900. Even the people who had been contributing to RocketMap and who knew inside & out what I was up to (including my holiday, its duration, and I was even online on Discord while on a mountain with my snowboard strapped to my feet in France), ended up PM’ing me and poking me to merge 1900, fully aware that it would take much more time (I repeated this often enough).

Which brought us to a point where one thing became obvious: people didn’t want a review, they wanted a merge.

Mimi and Eunice - Patience

Source: http://mimiandeunice.com/2010/08/09/patience/

Those who really wanted a review (which includes rocketrobot, the author of the PR), were immensely patient and understanding. I thank those people for that. But everyone else was so obsessed with getting it merged because they personally liked the PR and “hey, it worked for them!” that they lost sight of everything else.

So I merged PR 1900 without finishing my review.

One main reason: RocketMap is open source. As far as our development went, everyone should have been equal. But I realized that this wasn’t the case: when it came to taking responsibility and carrying workload on our shoulders, I realized that everyone simply expected me to continue being responsible for the majority of what got merged. And this is wrong: an Approval on Github brings responsibility with it – you can’t approve something without also saying that you’re giving your word that you’ve done everything necessary to properly review the code, and that you’re ready to be partially reponsible for the merge.

PR 1900 had 12 different people who approved the PR, two people who just discussed it and one who said not to merge (me). 12 people…

So I merged.

Of course things went wrong. While it “worked for some”, there were inevitable side effects (which we knew beforehand would happen), and for others it broke their map entirely. And so everyone was suddenly hit with reality.

There are some who have held me personally responsible and who are angered at me for what happened. I can understand why, but I also feel that those points of view are ignoring what really happened for weeks. I will say that it’s immensely unfortunate that we annoyed so many people, but I also won’t apologize for my decision. I explained my thinking, and I still stand behind it.

So it happened – now we learn and improve.

It’s frustrating that it happened.

So how do fix it for the future?

First, I believe that, if RocketMap has done one thing very right, it’s that it has always continued to struggle for what it believes is right. The staff, the volunteers, even the community, has occasionally struggled but it was always in order to eventually learn and to improve. In some contexts, a struggle is exactly the best situation you can have, because it means no one is controlling anyone and everyone has the opportunity to participate.

But for software development, a struggle leads to delays, and we want to fix this. We want RocketMap, in every sense of what RocketMap has become, to be an enjoyable experience.

Enter devkat.

So far, you’ve only occasionally heard of devkat. In simple terms, it’s the group of people that has been responsible for several of the responsibility, coding work, merges, and brainstorming contributions surrounding RocketMap. For the public, it has (seemingly) only been the group that hosts vote.devkat.org, because that has been the only time so far that we used the devkat name to publish something.

And devkat is how we want to solve the RocketMap problems: structure, official processes, sustainability. All of it, we want to tackle it.

We’ll work to:

  1. Regularly brainstorm to improve how we work and what we work on.
  2. Have a real development roadmap, structure and formal processes. Eventually we hope to have more frequent public communication (roadmap, milestones, todos, releases).
  3. Contribute better and well-tested code to RocketMap.
  4. Increase the amount of hours we can work on RocketMap. More support means we can trade in more of our regular working hours for RocketMap.

I’ve said it before: by separating devkat from RocketMap, RocketMap can continue to be what it is, freely. And devkat can continue to experiment without being constrained to what people have come to expect (even demand) from RocketMap.

We’ve been working on devkat for a few months now behind the scenes because we always wanted to create a development environment that works best for us. We wanted to solve some concrete issues that had already shown up, and we wanted to have a place and the people to solve issues that might pop up in the future. What happened with PR 1900 was a good reminder and a sign for us that it’s time to continue and to publish what we’ve been working on.

I’ll personally be working exclusively with devkat. I’ll still be around RocketMap as often as I used to, but all of my code will go towards devkat’s way of doing things – a proper way of working will benefit all of us. I feel that we can discuss for a long time how it’s – at least in some ways – unfortunate that money is what brings sustainability, but ultimately it’s the way the world works and I’d rather be realistic and adapt than hold an idealistic view that will eventually kill the project.

In a few concrete steps (explained in detail afterwards):

  1. When we publish things made by devkat, we’ll officially announce it on devkat rather than silently contributing to RocketMap. This will help keep a clear focus on how we’ve been doing and what we can improve in devkat specifically.
  2. The core group of devkat will forever stay a small group: growth is not our focus, code quality and a better RocketMap is. We’ll be accessible via our Discord server (http://discord.devkat.org). You can always apply if you feel it’s the right thing to do, but accepting or even replying is entirely at our own discretion – our focus comes first.
  3. To solve the problem of not having formal processes, leading to untested/unreviewed code (e.g. PR 1900) or other roadblocks, devkat will host its own code on a private repository which we’ll maintain in a more formal way. There will be a way for contributors to formally contribute which will maintain our code quality, and there will be a way for interested people to participate in testing or using the code before it’s publicly released to RocketMap (more on this later). This repository will contain code at least two weeks before it gets released publicly to RocketMap.
  4. Finished contributions to the private devkat repository will be tested and reviewed in private for at least two weeks before releasing the code to the main RocketMap repository (per pull request, as usual). Because of how we’ll work, we’re more than ready to take full responsibility for any devkat merges (so none of that PR 1900 again).
  5. When available, we’ll provide paid hourly support for everything related to devkat code, setting up RocketMap, 3rd party tools (e.g. account gens) and unreleased/unmerged code (e.g. pull requests or devkat’s experimental code), from A to Z, including debugging and accessing private/custom setups. A quote can always be requested.

You can already visit the devkat Discord server or its Patreon profile.

RocketMap itself will stay exactly as it is. It’s still an open organization, it still stays on Github, it still stays free (and remember: devkat releases for free to RocketMap two weeks after it publishes to its private repositories). If you want a more formal explanation, devkat is a separation of concerns and responsibilities, to allow devkat to experiment without having to affect what RocketMap is as an organization and community.

#3 and #4: devkat Patreon, Roles and Private Repository

devkat’s private repository and closed group will maintain a specific level of code quality. Stricter rules and development processes, things that can’t (or that we don’t always want to) expect or enforce in an open source project like RocketMap. We want to try to make RocketMap better by doing what we do best, and we want to do this by creating a sustainable, better environment for our developers, which will let us contribute better code to RocketMap.

Via Patreon pledges (devkat Patreon profile), you can financially support our efforts and get read access to the private repository (more details in the pledge options on Patreon), and we use Paypal for the private support (we set up a unique invoice for each request, just PM us on devkat’s Discord or send an e-mail to hello@devkat.org).

1. To solve the problem of sustainability, devkat will offer paid read access to its private repository.

Some of our Patreon pledges includes read access to our private repository. By contributing, you get instant access to all finished code merges at least two weeks before it gets released publicly to RocketMap. Our higher Patreon contributors who want to support us more can also get access to our experimental code – this is similar to Github’s “pull requests”, except that it’ll be access to exclusively devkat code before it gets released to the devkat private repository for basic pledgers.

After you’ve pledged and you’ve received your Discord rank, just PM us on Discord or send an e-mail to hello@devkat.org to request access to the repository. We only send your login details after you’ve requested them, so make sure to let us know you want access.

2. We’ll provide paid hourly support for RocketMap, devkat, and 3rd party tools.

As another way to support the development, some of our members will be open to setting things up for you when it gets too difficult. Whether it’s the basics of getting set up, helping you purchase whatever is required (proxies, hashing keys, servers) or helping you debug your custom installation/configuration, we can help you out.

If you need our help, PM us on Discord or send an e-mail to hello@devkat.org. We’ll give you a quote, and all invoices will be sent via Paypal (We’re a registered company, we pay our taxes. It’s all good. ♥).

3. Volunteer contributors can earn bounties by contributing to devkat.

To motivate the developers who take the time to do things properly, we’re going to work with a bounty system. When you, as a developer, want to write code for RocketMap, we want to give you two options:

  1. You release to RocketMap as usual and go through the old process.
  2. You contact us via devkat’s Discord and you release to devkat’s private repository. Your code will go devkat’s usual process of review by its members, followed by a thorough testing process, on to the paid read-only repository for our supportive community members. Two weeks later, it’s at its final stage and has been properly reviewed, after which it gets merged publicly into RocketMap.

Bounty Hunter Mary Poppins

If you choose to work the devkat way, you’ll earn a bounty for your work. All bounties depend on the type and volume of your work, and everything surrounding the bounties is communicated before you submit anything to devkat. No surprises. We also welcome any volunteers who simply want to contribute code to be reviewed by devkat members.

PR contributors who have an open PR and want to move it to devkat before it gets merged can do so by PM’ing us on Discord. Just reminder that it’s first come first served, we can’t (and won’t) take up more work than we can handle.

As an important side note, contributing code does not require you to have a paid Patreon pledge. We received a question about this, and we hope this clears it up.

4. And once we get the ball rolling, we invest more into RocketMap.

Public projects, hosting servers, distributing data, investing in our own RE team for a free API, writing dynamic/polymorphic systems to permanently block scrapers, … everything we can try, we will try.

RocketMap is what we’ve always been and what we’ll always be about. We’ll continue to work on improving RocketMap and on contributing new items that will make it easier to set up and use, so we can get back to a time where people used the maps to play together as groups of friends and family. Only this time, more professionally.

It’s time to make RocketMap great again.

Donald Trump

We’re not going to build any walls (at least not unless they make sense for RocketMap), but I believe we can use a bit of their optimism to work towards a more positive message: making RocketMap great(er).

Continuously experimenting and trying new things to learn from, so we can see for ourselves what really works.

Thank you for your continued support. ♥

 

Today I don’t leave you with a song but with a message.