lae's gravatar

いらっしゃいませ! I'm Musee, often referred to as lae on the interwebs. I'm an infrastructure engineer at FireEye by day and sysadmin/developer by night who mostly just hacks things here and there nowadays. I deal a lot with Ansible as well.

Thoughts on 2018

Looks like this is my first real journal entry in over five years (the last 2 entries are translated works) so I guess let me start off by saying, this is my attempt at bringing some life back. This is a text heavy entry, mainly because I don't have time to filter through photos before the New Year. I was planning to have the entire thing in Japanese as well, but due to lack of time, I'll have to come back after the New Year for a full translation.

Anyway, this is mainly going to be a snapshot of my events, activities and new interests in 2018, including some introspection, as well as some news and what expectations I have for myself going forward in 2019. If you don't like reading about other people's lives (especially mine), go watch a virtual youtuber or something. This article is split into topical sections, feel free to skip what you might not be interested in.

5年間半ここの日記に(え?日記だったけ?)本質の記入を書きませんでしたね… 最近のはただの英訳した記事の投稿でした。 なので予め言うんでが、これからここの活躍を増やしたいと思います。 この記事はほぼテキストのみとなります。新年まであんまり時間がないから カメラロールと弄りたくない気分です。あと、大体英語のみとなります… でも来年この記事をまた編集するので、そのとき和訳や画像を入れとくと思います。 でもけっこう予想以上長くなりました…

それじゃ、今日の目標はつまり、今年なんのイベントを参加したのか、なんの活躍を したのか、なんの趣味を得たのか…後は今年の反省とか、一部の発表とか、来年の 抱負や期待の話もします。一人の忘年会って感じかな?まあ私の人生に興味を持ってる 方がいらっしゃいましたら(そんな人居ましたっけ?)、是非読んでください。 本記事は話題に限るいくつかの章に分けています。興味ない章は気軽にスキップ。

Cycling

It feels kind of late, but I finally went on my first cycling tour this year! I took a few weeks off at my old job in March and biked from Tokyo to Kyoto (東京→京都) on a fixed gear road bike (Wabi Cycle's Road Pro). I posted a lot of photos on Twitter with the #laedventure hashtag so maybe check that out. I technically began the tour by going to GeiKa (芸カ), a regularly held event where people typically sell fanmade Aikatsu! books and other goods (like keychains), this time being held at the Ota City Industrial Plaza PiO (大田区産業プラザPiO) in Kawasaki (川崎) - let's just say I ended up spending way more cash than I should have (I took a train back to Tokyo to drop my spoils at my friend's place and sleep there for the night).

I then left Kawasaki on the 5th of March and arrived in Kyoto on the 15th, but took a 2 day break (3/7-3/10) in Shizuoka (静岡) due to rain and another 2 (3/11-3/14) in Nagoya (名古屋) to sightsee - I wanted to check out the planetarium but it just happened to be closed for a holiday.... The shortest cycling route between Tokyo/Kyoto is around 500km but I ended up biking approximately 830km in total, making detours and touring around different cities, including Nagoya and my A Silent Voice (聲の形) pilgrimage in Ōgaki (大垣). Not counting inner city travel, I averaged biking around 82km for 4h35m a day for 8 days, with the longest being 123km between Hamamatsu (浜松) and Nagoya, which I guess isn't that bad for a first tour? At the time, I was pretty used to riding fixed gear because I'd commute to work on one, so the actual tour wasn't that hard on my body, but there were several sections I either had to walk (uphill) or abuse my brakes (downhill). I can't really suggest doing a fixed tour (at least, a fully fixed tour) if the route contains mountains. It's probably fine for some European tours, though, and while I'm not planning one for 2019, it might just happen in 2020.

Anyway, after this tour I really fell in love with touring. I had already started watching JaYoe's videos on Youtube about his recumbent cycling tour from Korea to Japan and elsewhere, but later on I also picked up watching the Long Riders! anime, which isn't so much about touring as it is about brevets (long distance biking events). Now, one of my life goals is actually to eventually participate in Paris–Brest–Paris, where you basically bike from Paris to Brest and back to Paris, mainly self-supported, within 90 hours. I'm not sure I can make the next one in 2019, but I'm gonna do my best to participate in the 2023 event. You might see me post about smaller brevets or races in the meantime, though.

After that, I did one long distance bike ride back in the states with a co-worker from Monterey to San Jose. Biking with someone else you know sure is a different experience, I have to say, especially if they're someone more familiar with the area. While I didn't get to do another ride with a partner this year, I'm hoping I can find some others to regularly bike with next year.

I quit my job in July and on the spur of the moment decided to do a tour in Hokkaido (北海道) in August. This one is pretty fresh on my mind so instead of going into too much detail here, I'll write a series of entries instead of making this section even longer - but basically, I took a Shinkansen (bullet train) to Hokkaido and started my tour from Sapporo (札幌), went westwards through the Daisetsuzan (大雪山) mountain range all the way to Shibetsu (標津町) on the west coast, then went through the Shiretoko (知床) national park/mountain range, biked along the northern coast all the way to Cape Sōya (宗谷岬, the northernmost point in Japan) to Wakkanai (稚内), took a detour to visit the island of Rishiri (利尻), down the northeast coast to Rumoi (留萌), back to Sapporo to hang out for a bit because of the typhoon warnings, then to Tomakomai (苫小牧) to take the very first ferry out of Hokkaido after all of the cancelled departures, for a total of approximately 1780km. I somehow made it out a few hours before the Eastern Ibuki earthquake that left all of Hokkaido without power...

The entire tour lasted from August 14 to September 5, then a ride from Ōarai (大洗) back to Tokyo on the 6th (ended up getting home after midnight). Not including inner city travel days, this time I biked an average of ~96km over 4h27m per day for 17 days - no doubt this was helped by the fact that I switched to the freewheel on my rear tire (so it was easier to handle high grade terrain) - still a single gear though, and I'm honestly not sure if I really need to change that. The food is fucking amazing around Hokkaido, so definitely go if you have a chance. I personally recommend visiting Rumoi and Shibetsu/Rausu (羅臼), as well as Rishiri Island (and probably neighboring Rebun (礼文)). If you're on a budget, consider taking the ferry from Oarai to Tomakomai to get to Hokkaido instead of taking the Shinkansen (or local trains).

I had already had a desire to leave America and live somewhere else, but my experiences in Hokkaido ended up solidifying my desire to live specifically in Japan. A country formed around a small set of islands (while it does have it's own set of disadvantages, like the earthquake from earlier) really sets the field for being able to live a comfortable life. I'll probably end up retiring in Hokkaido, whenever that time comes. Keep bugging me to write that article series on my tour, though (but check out #laedventure (hokkaido filter)).

Apart from that, I attempted a challenge to bike from Nagoya to Tokyo in 24 hours after hanging out with some mahjong acquaintances for a day, but only ended up biking 255km of the 365km distance in 20.5 hours before calling it quits, 10 of which I ended up either resting or sleeping since I hadn't slept much after playing mahjong.

For the whole of 2018, I've biked slightly over 5000km, which is honestly a huge leap from ~2000km last year. I don't have any large tours planned for 2019 and might not actually be able to schedule one, but I'm still pretty excited to target 5000km for next year, too! Instead of getting most of that distance from a month-long tour, I'll be targeting shorter but more frequent long distance rides (e.g. 250km every other weekend or something), since it'll soon become a lot easier to do.

Virtual Youtubers

Yes, this gets a section. I think a lot of you are already familiar but basically they're what they sound like - youtubers who post videos not of theirselves directly but using either a 2D or 3D "virtual" avatar, typically using motion tracking (for full 3D they usually have several motion sensors, but there's a bunch that just use Live2D and face tracking with a webcam), and they typically have some sort of backstory/are a different character from their real selves. Like, look at Sally Amaki and her character Fujima Sakura to get some idea.

Anyway, I never knew how much I needed vtubers as a form of entertainment and emotional support/stress relief in my life until this year. Aikatsu used to fill in this gap in 2014, but OG Aikatsu is pretty much over nowadays (well, Ichigo is finally getting a scale figure, I guess?) and Aikatsu Friends is more just entertainment like other anime and not really filling in the gap anymore. Kizuna Ai was basically a vanguard/gateway into the world of vtubers late last year (if I recall correctly) for me, and now I'm technically subscribed to around 30 vtubers (as well as some other youtubers, like Dogen and GYARI - since I rarely used YouTube before this year), but there's a few that I want to highlight as having a particularly impactful influence on me.

First, @CierraRunis. I found out about her around Halloween through her Honeymoon Un Deux Trois cover (funnily enough, I found out through the Aikatsu-related Discord server I'm particularly active, which has a vtuber-specific channel), which is a Vocaloid song with lyrics primarily in Japanese but a couple of lines in English. When I first listened to it, I was like, "Holy shit is this English? How is it so perfect?"―I went and watched her introduction video and it turns out her native language is EnglishBrescian and she's learning Japanese and Chinese! I was totally not expecting this when I originally followed her on Twitter, but this buildup happened and now we're mutuals? Cierra's fans are called classmates (since she's a student at Overidea Academy) but I also consider her as my role model when it comes to speaking Japanese/Chinese (and well, Obamama too I guess, as well as Dogen from earlier). Not planning to do so in 2019, but I'll learn Chinese! Eventually! (Though Korean may be first....)

And Cierra, if you're reading this, happy new year! I hope you get to make more covers and other videos in 2019, and get a Switch to play games with Obamama and friends! I know you play a lot of otome games, and that people often ask you to post videos/stream playing those, but given the nature of those games (like, it's kind of difficult to commentate) don't feel obligated to do so. 今年もよろしくおねがいします!

Then there's YuNi (YuNi's channel), a virtual singer. I found out about her after Oda Nobuhime's support video, when YuNi suddenly stopped tweeting (she left this) and posting videos to secretly work on her first original song with YUC'e. All of her covers are really well done! I ended up getting my first piece of VR equipment (an Oculus Go) in order to participate in her first VR concert, where she debuted her second original song, Winter Berry, and I don't regret it at all. I'm planning another article on the concert experience, but I'm looking forward to VARK (the startup behind the VR concert tech) bringing us a better VR concert experience in 2019 (will we get a 2 hour+ concert next year?), and YuNi bringing more original songs!

Other honorable mentions include Soyokaze Ame, a vtuber who speaks in Hakata dialect and is aiming to be a professional voice actress (she's pretty decent at idle talk, too, and I'm looking forward to her next travel video), the Game Club (basically a bunch of high schoolers...playing games?) and Cocoa (sister to one of the Game Club members and is a really good singer), Natori Sana, a nurse cutie who sucks at video games, Nekomiya Hinata, a cat who will fucking demolish you in FPS games, and Kaguya Luna, ...what the fuck? Also, when will Anima Lyon post their next video?

I'm waiting for my friend ひめるめる to make their debut as a vtuber. ひめるめるさん、VTuberデビュー待ってますよ。もう登録したよー

Gaming

I didn't game as much as I would've liked to. Not because I didn't have time, just mainly because I was too lazy (to e.g. reboot into Windows). While getting a Surface Go (Windows tablet) helped a bit for getting me to play visual novels (I finished planetarian and started playing 9-nine-そらいろそらうたそらのおと, can't remember if I finished any other VNs), I'm hoping that I can setup my Linux desktop so that I can use PCI passthrough in a Windows VM for my GPU, which would let me still have a Linux desktop environment for daily use without having to clean up running applications in order to use Windows with native graphical performance.

As for games I did play - I'm now at 10.40 potential in Arcaea, and I think I'm (only) skill level 6 in Sound Voltex, but can clear several level 16 charts. I haven't really played SDVX in the latter half of 2018, though, but this should change next year and maybe, just maybe, I'll be able to play level 18 charts? I also started playing Atelier Online but it's really not keeping my interest, especially since the events require way too much time to rank in, and it's not as fully voiced as I'd hope. I also started playing PUBG as a chance to chat with some Japanese folks I know. I completed all of the Cytus II story at one point (there's new content now that I haven't gotten around to yet) and uh, unlocked Brain Power. I've also pretty much quit playing LLSIF (a good thing).

I didn't make it to 3dan after all, but I made a new Tenhou account and got to shodan in 77 games and am now currently 2dan (the closest I got to 3dan was 790/800). Next year I'll definitely get to 3dan, maybe even 5dan if I play often enough, but I think I just might create another account instead. While I passed the qualifiers, I unfortunately didn't get past the first round in the 2018 Online Riichi Championship. Better luck next time.

For 2019, I'm hoping to get around to finishing more visual novels to improve my Japanese language comprehension, get used to playing first person shooters again so I can play PUBG with my friends and not hold them back that bad, and eventually finish playing all of the Atelier Arland series and play Atelier Lulua when it comes out.

Anime/Manga/Idols

I really need to finish this entry quickly, so this section should really be more detailed than it is. I recently started logging my shows again, but on AniList - it's still pretty incomplete because a) I didn't login to MAL to get a list to import and b) I don't really remember every show I've watched since I stopped logging in 2014.

Yuru Camp and A Place Further Than the Universe are amazing shows.

Steins;Gate Zero is pretty good, too. Unfortunate about the lack of Dr. Pepper though.

Hey you, go read the A Silent Voice manga. It's good. I reread it this year. I also started reading FukaBoku (不可解なぼくのすべてを), which basically is about someone who is genderqueer. Hopefully it gets localized by someone. I'm thinking of buying more digital manga to read for Japanese practice, as well.

I made a new friend, completely unrelated to anyone else I know, at Ayaka Ohashi's birthday event, and he's also a software engineer (studying at Tohoku University)!

I contributed to @yuzuki_suemizu's flower stand for Hoshizaka Kana at Aikatsu! Music Festa in Budokan (and also became friends), and also went to an Aikatsu offline meeting at the very beginning of the year (with people from Kirakiratter) as well as a Aikatsu-themed DJ party. This was a last minute decision, but I dropped the Fuji Five Lakes cycling event and also went to Aikatsu 5th. Message of a Rainbow is such a good song....

I rushed to Mimorin's release live in Yokohama for her fourth album, tone., right after Comiket. I can't think of words to describe it, but I'm really glad I follow Mimorin.

I didn't get to see Wake Up! Girls at all this year, but I have tickets for a concert in January, and hopefully I can make it to their final live at SSA.

I'm slowly getting into 22/7.

Work and the Future

Y'all probably know this, but I'm currently unemployed. I quit my last role as an infrastructure engineer at FireEye after a stagnant year passed―I didn't see much prospect for my future year and totally felt like I overstayed my welcome, seeing as how all my coworkers were either laid off or quit theirselves. My plan was, at the time of leaving, to finish up a little over a year worth of classes at community college and transfer to Berkeley, complete a degree in CS, then have better prospects of getting a decent paying job as well as an easier time to fulfil work visa requirements in Japan. I planned my Japan vacation a little too long to be able to start this past Fall quarter so was going to start next month. However, that plan is now going completely down the drain.

I've verbally accepted an offer to work at a company in Tokyo. Several of my Japanese followers are familiar with this company, and it's one that I have a lot of respect for, but am currently keeping under wraps. Further details will be discussed after the New Year, regarding visa application and moving in general, but I will be moving to Tokyo hopefully by March.

I'm expecting the following for 2019 from a professional standpoint:

  • Write more software in Rust, possibly at my new company
  • Start seriously using Kubernetes
  • Continue maintenance of my open source Ansible roles (contributors wanted!)
  • Create and maintain roles for Monacoin software
    • Possibly also bring up CI and other infrastructure for the community
    • Code review some of the software forks WakiP created in 2018
  • Relearn AWS to the point I can come up with architecture designs easily
  • Start building out a serious home lab for experiments (Does anyone know how to buy server hardware for cheap in Tokyo? Homelabbing doesn't seem popular in Japan, or I'm not using the correct search terms.)
  • Learn how to leverage "serverless" in an application design (the name is bad, but the concept is...useful?)
  • Get as close to JLPT N1 proficiency as possible, since I'll also be writing documentation in Japanese

I will mostly be doing infrastructure-related tasks at my new company, but will also be learning TypeScript to understand and contribute to their software code base. And hopefully, I'll last here longer than FireEye. If any of my future coworkers are reading this, あけおめことよろ!入社後おた柔らかにお願いします…!

2019 is going to be an exciting year, and I'm going to do my best to log everything, or at least do a better job at my 2019 reflection post! Happy New Year!

Feel free to send me questions―I'll answer them (if I can) on Twitter.

Notes from Zaif Attack

The following is primarily a translation of this blog post.

On September 20, 2018, Tech Bureau sent out a notice that they suspended deposits and withdrawals for three currencies (BTC, MONA, BCH) on the Zaif cryptocurrency exchange due to unauthorized access to its systems. This post is an aggregation of the details of that event.

Press Releases

Tech Bureau

Incident Timeline

TimeEvent
2018.09.14 between 17:00-19:00Approximately 6.7 billion JPY worth of assets were withdrawn without authorization.
2018.09.17Tech Bureau detected an anomaly within the environment.
- eveningTech Bureau suspended withdrawals/deposits for 3 currencies on Zaif.
2018.09.18Tech Bureau identified they had suffered a hacking incident.
- same dayTech Bureau reported the incident to the local finance bureau and started filing papers with the authorities.
- same dayThe official Zaif Twitter account tweeted that customer financial assets are safe.
- same dayIn accordance with the Payment Services Act, the FSA issued a Request for Report to Tech Bureau.
Post-identificationTech Bureau enters into a contract with Fisco for financial support.
Post-identificationTech Bureau enters into a contract with CAICA for assistance in improving security.
2018.09.20 ~2amTech Bureau issues a press release declaring that deposits/withdrawals were suspended due to a hacking operation.
- same dayThe Japan Cryptocurrency Business Association appealed for a member to perform an emergency inspection.
- same dayThe FSA sent an on-site inspection crew to Tech Bureau.
2018.09.21ETA for the FSA to issue a report on its investigation about the status of customer assets to the cryptocurrency exchange's traders.

Damage

  • Approximately 6.7 billion JPY worth of 3 different currencies were withdrawn externally without authorization.
  • Withdrawals and deposits for the 3 affected currencies have been suspended since the evening of 17 September.

Itemization of damages

Tech Bureau's own assets~2.2 billion JPY
Customer assets~4.5 billion JPY
  • Tech Bureau has shown that they can cover the 4.5b loss of customer assets through financial assistance from the FDAG subsidiary.

Information around the Zaif hack itself

  • Funds were withdrawn from the server managing the Zaif hot wallet.
  • Tech Bureau is still investigating the exact method of intrusion, but it doesn't look like they'll publicly announce it as a protective measure.

Details on the unauthorized transactions

Total (estimated) damages on the 3 currencies

CurrencyAmount transferredJPY conversionUSD conversion
Bitcoin5966 BTC4.295 billion JPY38.207 million USD
MonacoinUnder investigation, but sources estimate 6,236,810 MONA650 million JPY5.782 million USD
Bitcoin CashUnder investigation, but sources estimate 42,327 BCH2.019 billion JPY17.954 million USD

Assumed recipient addresses of the hack

CurrencyAddressTime of transaction
Bitcoin1FmwHh6pgkf4meCMoqo8fHH3GNRF571f9w2018.09.14, between 17:33:27 and 18:42:30
Bitcoin Cashqrn0jwaq3at5hhxsne8gmg5uemudl57r05pdzu2nyd2018.09.14, between 17:33:15 and 17:51:24
MonacoinMBEYH8JuAHynTA7unLjon7p7im2U9JbitV2018.09.14, between 17:39:01 and 18:54:10

work in progress



Disclaimer: I make no guarantees of the accuracy of the above article.
Please see the official press releases and/or PR department at Zaif. I am also not affiliated with Zaif or any of the companies mentioned in this article.

A Practical Behind the Scenes, Running Mastodon at Scale (Translation)

The following is a translation of this pixiv inside article.

Good morning! I'm harukasan, the technical lead for ImageFlux. 3 days ago at Pixiv, on April 14, we decided to do a spontaneous launch of Pawoo—and since then I've found myself constantly logged into Pawoo's server environment. Our infrastructure engineers have already configured our monitoring environment to monitor Pawoo as well as prepared runbooks for alert handling. As expected, we started receiving alerts for the two days following launch and, despite it being the weekend, found ourselves working off hours on keeping the service healthy. After all, no matter the environment, it's the job of infrastructure engineers to react to and resolve problems!

pawoo.net Architecture

Let's take a look at the architecture behind Pawoo. If you perform a dig, you'll find that it's hosted on AWS. While we do operate a couple hundred physical servers here at Pixiv, it's not really that possible to procure and build up new ones so quickly. This is where cloud services shine. nojio, an infrastructure engineer who joined us this April, and konoiz, a recent graduate with 2 years of experience, prepared the following architecture diagram pretty quickly.

Pawoo Architecture Diagram Using as many of the services provided by AWS as we could, we were able to bring up this environment in about 5 hours and were able to launch the service later that day.

Dropping Docker

One can pretty easily bring up Mastodon using Docker containers via docker-compose, but we decided to not use Docker in order to separate services and deploy to multiple instances. It's a lot of extra effort to deal with volumes and cgroups, to name a few, when working with Docker containers - it's not hard to find yourself in sticky situations, like "Oh no, I accidentally deleted the volume container!" Mastodon does also provide a Production Guide for deploying without Docker.

So, after removing Docker from the picture, we decided to let systemd handle services. For example, the systemd unit file for the web application looks like the following:

Description=mastodon-web
After=network.target

[Service]
Type=simple
User=mastodon
WorkingDirectory=/home/mastodon/live
Environment="RAILS_ENV=production"
Environment="PORT=3000"
Environment="WEB_CONCURRENCY=8"
ExecStart=/usr/local/rbenv/shims/bundle exec puma -C config/puma.rb
ExecReload=/bin/kill -USR1 $MAINPID
TimeoutSec=15
Restart=always

[Install]
WantedBy=multi-user.target

For RDB, Redis and the load balancer, we decided to use their AWS managed service counterparts. That way, we could quickly prepare a redundant multi-AZ data store. Since ALB supports WebSocket, we could easily distribute streaming as well. We're also utilizing S3 as our CDN/uploaded file store.

Utilizing AWS' managed services, we were able to launch Pawoo as fast as we could, but this is where we began to run into problems.

Tuning nginx

At launch, we had stuck with the default settings for nginx provided by the distro, but it didn't take too long before we started seeing HTTP errors returned so I decided to tweak the config a bit. That said, the important settings to increase are worker_rlimit_nofile and worker_connections.

user www-data;
worker_processes 4;
pid /run/nginx.pid;
worker_rlimit_nofile 65535;

events {
  worker_connections 8192;
}

http {
  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  sendfile on;
  tcp_nopush on;
  keepalive_timeout 15;
  server_tokens off;

  log_format global 'time_iso8601:$time_iso8601\t'
                  'http_host:$host\t'
                  'server_name:$server_name\t'
                  'server_port:$server_port\t'
                  'status:$status\t'
                  'request_time:$request_time\t'
                  'remote_addr:$remote_addr\t'
                  'upstream_addr:$upstream_addr\t'
                  'upstream_response_time:$upstream_response_time\t'
                  'request_method:$request_method\t'
                  'request_uri:$request_uri\t'
                  'server_protocol:$server_protocol\t'
                  'body_bytes_sent:$body_bytes_sent\t'
                  'http_referer:$http_referer\t'
                  'http_user_agent:$http_user_agent\t';

  access_log /var/log/nginx/global-access.log global;
  error_log /var/log/nginx/error.log warn;

  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

Afterward, without changing a lot of settings, nginx started to work pretty well. This and other ways to optimize nginx are written in my book, "nginx実践入門" (A practical introduction to nginx).

Configure Connection Pooling

PostgreSQL, which Mastodon uses, by nature forks a new process for every connection made to it. As a result, it's a very expensive operation to reconnect. This is the biggest difference Postgres has from MySQL.

Rails, Sidekiq, and the nodejs Streaming API all provide the ability to use a connection pool. These should be set to an appropriate value for the environment, keeping in mind the number of instances. If you suddenly increase the number of application instances to e.g. handle high load, the database server will cripple (or should I say, became crippled). For Pawoo, we're using AWS Cloud Watch to monitor the number of connections to RDS.

As the number of connections increased, our RDS instance would become more and more backed up, but it was easy to bring it back to stability just by scaling the instance size upwards. You can see that CPU usage has been swiftly quelled after maintenance events in the graph below:

RDS Graph

Increasing Process Count for Sidekiq

Mastodon uses Sidekiq to pass around messages, though it was originally designed to be a job queue. Every time someone toots, quite a few tasks are enqueued. The processing delay that comes from Sidekiq has been a big problem since launch, so finding a way to deal with this is probably the most important part of operating a large Mastodon instance.

Mastodon uses 4 queues by default (we're using a modified version with 5 queues for Pawoo - see issue):

  • default: for processing toots for display when submitted/received, etc
  • mail: for sending mail
  • push: for sending updates to other Mastodon instances
  • pull: for pulling updates from other Mastodon instances

For the push/pull queues, the service needs to contact the APIs of other Mastodon instances, so when another Mastodon instance is slow or unresponsive, this queue can become backlogged, which then causes the default queue to become backlogged. To prevent this, run a separate Sidekiq instance for each queue.

Sidekiq provides a CLI flag that lets you specify what queue to process, so we use this to run multiple instances of Sidekiq on a single server. For example, one unit file looks like this:

[Unit]
Description=mastodon-sidekiq-default
After=network.target

[Service]
Type=simple
User=mastodon
WorkingDirectory=/home/mastodon/live
Environment="RAILS_ENV=production"
Environment="DB_POOL=40"
ExecStart=/usr/local/rbenv/shims/bundle exec sidekiq -c 40 -q default # defaultキューだけにする
TimeoutSec=15
Restart=always

[Install]
WantedBy=multi-user.target

The most congested queue is the default queue. Whenever a user that has a lot of followers toots, a ginormous number of tasks are dropped into the queue, so if you can't process these tasks immediately, the queue becomes backlogged and everyone notices a delay in their timeline. We're using 720 threads for processing the default queue on Pawoo, but this is a big area for introducing and discussing performance improvements in.

Changing the Instance Type

We weren't quite sure of what kind of load to expect at launch, so we decided to use a standard instance type and change it around after figuring out how Mastodon uses its resources. We started out with instances from the t- family, then switched to using the c4- family after distinguishing that heavy load was occurring every time an instance's CPU credits ran out. We're probably going to move to using spot instances in the near future to cut down costs.

Contributing to Mastodon

Now, we've been mainly trying to improve Mastodon performance by changing aspects of the infrastructure behind it, but modifying the software is the more effective way of achieving better performance. That said, several engineers here at Pixiv have been working to improve Mastodon and have submitted PRs upstream.

A list of submitted Pull Requests:

We actually even have a PR contributed by someone who's just joined the company this month fresh out of college! It's difficult to showcase all of the improvements that our engineers have made within this article, but we expect to continue to submit further improvements upstream.

Summary

We've only just begun but we expect Pawoo to keep growing as a service. Upstream has been improving at great momentum, so we expect that there will be changes to the application infrastructure in order to keep up.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

The translator of this article can be found on Mastodon at lae@kirakiratter.

Increasing Maximum Volume with ALSA

Since I don't have a set of monitors for my desktop, I use a pair of headphones often as speakers for audio. This setup usually is insufferable because of low audio, though mplayer has a "softvol" plugin that lets you increase your sound output's decibel level - and since I hardly needed sound for anything other than music or video, this solution worked perfectly (of course, for anything else I could put on my headphones).

That solution sufficed for me until recently, and I found out ALSA actually has a softvol plugin that lets you set max_dB (usually 0 by default). This can be done in /etc/asound.conf or ~/.asoundrc with the following definitions:

pcm.softvol {
        type softvol
        slave.pcm "cards.pcm.default"
        control {
                name "Software"
                card 0
        }
        max_dB 20.0
}
pcm.!default {
        type            plug
        slave.pcm       "softvol"
}

Depending on your configuration, the line slave.pcm "cards.pcm.default" and card 0 may need modification - you can run aplay -Ll to list your devices and card indices. Changes will take effect upon restarting applications that use sound.

This will create a "Software" control in applications like alsamixer, which will let you increase the decibel level up to 20dB (though there isn't any indication of what the decibel level is at other than percentages). Since I didn't specify min_dB, it defaults to -51dB.

You can also control the left and right channels independently, which is useful when you need volume to be louder in one speaker or headphone in a stereo setup.

Happy π Day!

And there you have it.

As a present, here's an illustration of Japan National Route 20 from the promotional video of 言の葉の庭 (The Garden of Words). It's a new film directed by Makoto Shinkai scheduled to air in theaters (Japan only, though) on 31 May. Feel free to use this as your wallpaper.

Route 20 illustration in 言の葉の庭

In other news, I open-sourced my Showtimes app a few weeks ago. You can see it in action at Commie Subs.

I'll just sit here and continue to stare at the Raspberry Pi I recieved last year but still have yet to use for anything.