Page MenuHomePhabricator

Recent Posts

Production Excellence #44: May 2022

Written by Krinkle on Thu, Jun 16, 12:07 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

GitLab-a-thon!

Written by brennen on Tue, May 31, 6:30 PM in Doing the needful.

Release Engineering's "GitLab-a-thon" sprint for May 10th-24th (roughly) focused on the mechanics of migrating a Wikimedia service to GitLab, setting up a CI pipeline, building container images from that service, and publishing images to the Wikimedia registry. We selected the Blubber project as a good candidate for experimentation:

Read more...

Production Excellence #43: April 2022

Written by Krinkle on May 12 2022, 9:00 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Should Vector be responsive?

Written by Jdlrobson on May 9 2022, 10:31 PM in Leave it to the prose.

Here I share some thoughts around the history of "responsive" MediaWiki skins and how we might want to think about it for Vector.

Read more...

Section translation migrated to Vue 3

Written by santhosh on Apr 26 2022, 5:55 AM in Language and Translation.

Section translation is a mobile first adaptation of ContentTranslation tool. It helps editors to translate sections from a source article to its corresponding article in another language using easy to use UI in a mobile interface. Translating content involves many steps such as choosing the right article, languages, sections to translate, cross checking with the existing article, selecting sentence, translating with the help of Machine translation, editing it and finally publishing. Designing and building such a complex workflow in the small mobile screen is a very challenging project.

Read more...

Production Excellence #42: March 2022

Written by Krinkle on Apr 21 2022, 9:29 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

What We Learned from Trainsperiment Week

Written by thcipriani on Apr 20 2022, 8:15 PM in Doing the needful.

Developers should own the process of putting their code into production. They should decide when to deploy, monitor their deployment, and make decisions about rollback.

Read more...

Sixth Phase — Reaching Destination

Life is like riding a bicycle. To keep your balance you must keep moving. — Albert Einstien

Read more...

Fifth Phase - Experience and Learning

Anyone who has never made a mistake has never tried anything new — Albert Einstein

Read more...

A Trainsperiments Week Reflection

Written by dduvall on Apr 1 2022, 2:29 AM in Doing the needful.

Over here in the Release-Engineering-Team, Train Deployment is usually a rotating duty. We've written about it before, so I won't go into the exact process, but I want to tell you something new about it.

Read more...

Outreachy, December 2021-March 2022

Every time I finish a mentorship, I read my blog posts about previous mentorships. Almost every time I finish a mentorship I write that I'll probably take a break from mentoring for a while, but then I don't. 😅

Read more...

Fourth Phase — Career Opportunities

“Peace cannot be kept by force. It can only be achieved by understanding.” — Albert Einstein

Read more...

Production Excellence #41: February 2022

Written by Krinkle on Mar 15 2022, 12:59 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

GitLab: Rethinking how we handle access control

Written by brennen on Mar 4 2022, 10:44 PM in Doing the needful.

I'll start with a bit of general administrivia. First, our migration of Wikimedia code review & CI to GitLab continues, and we're mindful that people could use regular updates on progress. Second, I need to think through some stuff about the project, and doing that in writing is helpful for all involved. I'm going to try writing occasional blog entries here for both purposes.

Read more...

Lessons Learned in Software Testing: A Context-Driven Approach

Lessons Learned in Software Testing signed by Bret Pettichord.

Read more...

Diving Into Our Deployment Data

Written by thcipriani on Feb 15 2022, 9:17 PM in Doing the needful.

If you’ve ever experienced the pride of seeing your name on MediaWiki's contributor list, you've been involved in our deployment process (whether you knew it or not).

Read more...

Production Excellence #40: January 2022

Written by Krinkle on Feb 4 2022, 4:32 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Third Phase – Midpoint Progress in the journey

“Simplicity is the ultimate sophistication” - Leonardo da Vinci

Read more...

Second Phase – Everybody struggles and what does the audience think about it?

Failures and setbacks are always part of life and sometimes it feels more like a roller coaster. You can either enjoy it or just spend the whole time screaming in fear. Those ups and downs in roller coasters are more similar to failures and success in life. If you think about it, it’s more like a journey where you have a starting point and a destination.

Read more...

First Phase – Introduce yourself

“A journey of a thousand miles begins with a single step” - Lao Tzu

Read more...

Production Excellence #39: December 2021

Written by Krinkle on Jan 17 2022, 10:16 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Production Excellence #38: November 2021

Written by Krinkle on Dec 12 2021, 1:34 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Lessons from 16 Years of Working Remotely

I have been working remotely since 2005. Recently, an intern has asked me for advice about it. I've decided to write a blog post. I hope the advice will be useful to a wide range of people. From people new to remote work (possibly because of COVID) to people with years of experience.

Read more...

Production Excellence #37: October 2021

Written by Krinkle on Nov 5 2021, 2:05 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Benchmarking MediaWiki with PHPBench

Written by kostajh on Oct 28 2021, 1:54 PM in Doing the needful.

This post gives a quick introduction to a benchmarking tool, phpbench, ready for you to experiment with in core and skins/extensions.[1]

Read more...

Production Excellence #36: September 2021

Written by Krinkle on Oct 21 2021, 11:31 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Blog Post Writing Club

Smokers pressure you to light one, drinkers pressure you to drink one, we pressure you to publish one. -- Hrvoje Šimić

Read more...

On Benefits of Team Chores

It has been my philosophy of life that difficulties vanish when faced boldly.
Isaac Asimov

Read more...

How we deploy code

Written by thcipriani on Sep 27 2021, 6:44 PM in Doing the needful.

Last week I spoke to a few of my Wikimedia Foundation (WMF) colleagues about how we deploy code—I completely botched it. I got too complex too fast. It only hit me later—to explain deployments, I need to start with a lie.

Read more...

macOS vs Ubuntu Linux Part 2

In Blog Post: macOS vs Ubuntu Linux I've written about a fun experiment. I've dual booted an MacBook Air with macOS and Ubuntu. Then, I ran the same npm scripts on both operating systems. To make things more interesting, I've ran the scripts both on the host operating system and inside a Docker container. Running scripts on Ubuntu was so faster, that I wanted to dual boot my iMac too. At the time, I could not dual boot the iMac because my disk refused to be partitioned. In the last couple of weeks, I've formatted disks on both machines and did a clean install of both macOS and Ubuntu. It's time to run the experiment again!

Read more...

Production Excellence #35: August 2021

Written by Krinkle on Sep 8 2021, 3:53 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

The Phoenix project book: reflections and thoughts

The phoenix is a long-lived, immortal bird associated with Greek mythology (with analogs in many cultures) that cyclically regenerates or is otherwise born again.
Phoenix (mythology) - Wikipedia

Read more...

Google Summer of Code 2021

Today is the last day of Google Summer of Code 2021. Vidhi Mody and Soham Parekh were mentoring Sahil Grewal on updating WebdriverIO to version 7 in Wikimedia repositories.

Read more...

Production Excellence #34: July 2021

Written by Krinkle on Aug 19 2021, 3:49 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

QA Support for Deployment Train

The Deployments page sets the working framework for QA workflow when it comes to deployment. Although the details and the level of QA work might be different from team to team depending on a specific team's workflow, specific projects and, even, specific phases of a project, the below is an outline of what seems to be, in a sense, typical QA activities for a weekly deploy cadence.

Beta cluster - the start
Read more...

Production Excellence #33: June 2021

Written by Krinkle on Jul 14 2021, 3:34 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Shrinking the tasks backlog

Written by hashar on Jul 2 2021, 3:05 PM in Doing the needful.

The release engineering team triages tasks flagged Release-Engineering-Team on a weekly basis. It is an all hands on deck one hour meeting in which we pick tasks one by one and find out what to do with them. We have started with more than a hundred of them and are now down to just a dozen or so, most filed since the last meeting.

Read more...

Testing search in MediaSearch - Part II

“But what am I going to see?
I don't know. In a certain sense, it depends on you.”
― Stanislaw Lem, Solaris

Read more...

Page object pattern

There are two main purposes of the page object pattern.

Read more...

Production Excellence #32: May 2021

Written by Krinkle on Jun 21 2021, 1:31 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Creating a vue.js based skin with server side rendering

Written by Jdlrobson on May 27 2021, 9:54 PM in Leave it to the prose.

For WMF staff's inspiration week, I decided to take a step back from my work building out a new skin architecture and a redesign of Vector and put myself into the shoes of a skin developer to see if the changes my team had made life easier. As a secondary objective, I was interested in how a MediaWiki skin could be written in Vue.js and what the challenges were to get there.

Read more...

Most Interesting Bugs

What makes a bug interesting? Any bug, by definition, has an element of a surprise (otherwise, it wouldn't be a bug). But not all bugs are equal in their ability to be surprising. Some bugs do stand out, presenting two essential elements of a surprise - unexpectedness and/or unusual manifestation. Based on my own classification criteria, I define a bug to be the most interesting if it is either (or both)
(1) difficult to find
(2) has a quite unexpected (mostly disruptive) impact

Read more...

macOS vs Ubuntu Linux

I do most of my work on a 27-inch iMac from 2012. It's a good desktop machine.

Read more...

Production Excellence #31: April 2021

Written by Krinkle on May 13 2021, 3:49 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Improvements to Email QA

INTRO
When running QA on emails, the most important things to check are the dynamic content populating correctly for different user types, and ensuring there are no layout issues. Especially in an increasingly mobile world, it's important to review how an email will be displayed on a large variety of devices at once.

Read more...

Breakpoint at BrowserStack conferences (2020-2021) - Part 2 Focus on Exploratory testing

Since I was impressed with the depth of presentations at Breakpoint conference (at Browserstack) in July 2021 - see my post Breakpoint at BrowserStack conferences (2020-2021) - reflections and thoughts: Part 1 - Focus on Automation, it was really exciting to see that the next conference (March 2021) would be focusing on the intriguing topic of exploratory testing. The the presentations titles (here is the list of all presentations) looked equally exciting, for example

  • "Manual testing is not dead...just the definition"
  • "Whole team holistic quality"
  • "Contemporary Exploratory Testing"
  • "Building a testing culture"
Read more...

Debug Selenium Tests: logLevel

This tip works if you're targeting:

Read more...

Production Excellence #30: March 2021

Written by Krinkle on Apr 3 2021, 12:20 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Breakpoint at BrowserStack conferences (2020-2021) - Part 1 Focus on Automation

“Improving daily work is even more important than doing daily work.”
― Gene Kim, The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

Read more...

Tracking memory issue in a Java application

Written by hashar on Mar 12 2021, 9:38 AM in Doing the needful.

One of the critical pieces of our infrastructure is Gerrit. It hosts most of our git repositories and is the primary code review interface. Gerrit is written in the Java programming language which runs in the Java Virtual Machine (JVM). For a couple years we have been struggling with memory issues which eventually led to an unresponsive service and unattended restarts. The symptoms were the usual ones: the application responses being slower and degrading until server side errors render the service unusable. Eventually the JVM terminates with:

Read more...

Production Excellence #29: February 2021

Written by Krinkle on Mar 6 2021, 1:03 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Eventlogging journey

“I felt in need of a great pilgrimage, so I sat still for three days.” Hafiz

Read more...

Production Excellence #28: January 2021

Written by Krinkle on Feb 19 2021, 6:45 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Production Excellence #27: December 2020

Written by Krinkle on Feb 4 2021, 5:46 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Testing search in MediaSearch - Part 1

The secret of getting ahead is getting started. The secret of getting started is breaking your complex overwhelming tasks into small manageable tasks, and then starting on the first one.
—Mark Twain

Read more...

Exempla Docent - Testing instrumentation on Special:Homepage (QA perspective)

Two previous posts - Exempla Docent Part 1 and Part 2 outlined QA approach to testing functionality of the Suggested edits (SE) module on Special:Homepage. Part 1 explored testing ORES model-articletopic logic implementation and Part 2 described testing user workflows.

Read more...

Struggle, the road to growth

It has been more than 3 weeks into my Outreachy internship with Wikimedia foundations. The internship started well and the project that I'm working on is about evaluating Microsoft playwright as a possible replacement to the current automation testing framework being used. Week one was mostly about setting up the Wikimedia core by forking and cloning the Wikimedia core repository from Github. In order to simplify continuous integration, we are using Github as our code hosting platform to evaluate playwright instead of using Gerrit. The setup involved the following steps;

  • Forking the Wikimedia repository
  • Cloning the repository, setting up and running it on my local machine
  • Connecting my forked with upstream
  • Configuring CI.
Read more...

Production Excellence #26: November 2020

Written by Krinkle on Dec 15 2020, 9:49 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Runnable runbooks

Written by mmodell on Dec 11 2020, 11:51 PM in Doing the needful.

Recently there has been a small effort on the Release-Engineering-Team to encode some of our institutional knowledge as runbooks linked from a page in the team's wiki space.

Read more...

To dream a dream. My Outreachy Journey

The year 2020 has been a year of massive change in the entire world, there are mixed feelings of loss, confusion among others, but all in all, there is always hope that keeps us moving forward. I must say that being accepted as an Outreachy intern has been that ray of light at the end of the tunnel that I needed to end the year and begin the new year. Outreachy is a paid, remote internship program with the goal to support people from groups underrepresented in tech. Starting my career in the field of software engineering has been a journey of hard work, persistence, and seizing every opportunity since where I come from such opportunities are rare and the support for women's engagement in technology is quite low.

Read more...

Exempla Docent - testing UI for Suggested edits module

In Part 1 of Exempla Docent for QA practices, some approaches to testing ORES model articletopic were explored. This post, as Part 2, will present an overview on testing Suggested edits module (SE) - the UI that presents the ORES articletopic logic to users (more info on Newcomers tasks on Special:Homepage).

Read more...

Engineering Productivity Virtual Offsite October 2020

October 26-29 2020 was my team's second virtual offsite. We've had many offsites, but the first virtual one was in May 2020. The structure of this offsite was similar to the one in May. About four hours of sessions every day, from Monday to Thursday.

Read more...

Production Excellence #25: October 2020

Written by Krinkle on Nov 24 2020, 5:13 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Spin up a basic local Phabricator instance with Docker

If you want to experiment with Phabricator and/or the Phabricator APIs, it can be convenient to have a local instance to play with.

Read more...

Exempla Docent - testing ORES 'articletopic' model

ORES provides machine learning as a service for Wikimedia projects. The ORES model articletopic was used for the Growth team project - Suggested edits for newcomers on Special: Homepage.

Read more...

Production Excellence #24: September 2020

Written by Krinkle on Oct 23 2020, 11:51 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Visual Studio Code + Neovim

I've started using Vim in 2014. I've read a few books, wrote a few articles and gave a few talks on Vim. I really like editing text with Vim, but I was never happy with it's file management, extensions, things like that.

Read more...

Google Summer of Code, June-August 2020

June and July were pretty busy. I was on vacation the majority of August. Interns and other mentors were busy even then. For more introduction, read my post Blog Post: Google Summer of Code, February-May 2020.

Read more...

CI now updates your deployment-charts

Written by jeena on Sep 24 2020, 5:34 PM in Doing the needful.

If you're making changes to a service that is deployed to Kubernetes, it sure is annoying to have to update the helm deployment-chart values with the newest image version before you deploy. At least, that's how I felt when developing on our dockerfile-generating service, blubber.

Read more...

Google Summer of Code, February-May 2020

In February 2020 I've noticed an e-mail that Wikimedia is participating in Google-Summer-of-Code. Unfortunately, I've ignored the e-mail and soon forgot about it.

Read more...

Production Excellence #23: July & August 2020

Written by Krinkle on Sep 23 2020, 6:10 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

RPKI Origin Validation

Written by ayounsi on Aug 10 2020, 1:02 PM in Routing knowledge.

Since the late 90s, databases named Internet Routing Registries (IRR) have been trying to fulfill that (single) source of truth role. Unfortunately, they are subject to a lot of issues: fragmentation (many existing databases, not all equally well-maintained), security (some databases allow anyone to “claim” a prefix) and complexity (for the network operators). They also contain a lot of inaccurate data that have accumulated over time.

Read more...

Internal anycast

Written by ayounsi on Aug 7 2020, 9:48 AM in Routing knowledge.

This project brought two major changes to our infrastructure. Firstly, servers that used to be fronted by LVS for load balancing are now peering directly with our routers. Secondly, we have started using IP anycast for a highly critical service: recursive DNS.

Read more...

Phabricator Search Backend Changes

Written by mmodell on Jul 29 2020, 5:45 PM in Phabricating Phabricator.

Phabricator upstream has implemented a search engine that does not depend on an external full-text index service. It's been the default in Phabricator for quite some time, however, we have been using the ElasticSearch engine for a few years now due to previous issues with the old engine. Specifically, the old phabricator fulltext engine depended on MySQL's built-in fulltext index functionality. Unfortunately, the fulltext engine in MySQL had inconsistent performance and often returned low quality search results.

Read more...

All code is built

Written by Niedzielski on Jul 28 2020, 1:27 PM in Leave it to the prose.

HEADER CAPTION: The head of the Statue of Liberty on exhibit at the Paris World's Fair, 1878. The statue was built in France ahead of time, shipped overseas in crates, and then assembled in New York. Image by Albert Fernique / public domain.

Read more...

Production Excellence #22: June 2020

Written by Krinkle on Jul 23 2020, 3:25 AM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Fanboying Cypress

Software development is pretty agile in its nature in that things normally tend to move pretty quickly. However, the faster you move, the more things break. As a codebase grows in size, its pieces become more and more complex, with every line adding a potential bug. In Wikimedia Foundation, we keep a handle on this through rigorous amounts of testing. Manual testing requires a lot of effort especially when you have a large core repository with a plethora of plugins and extensions that need to be tested. One of the hot frameworks on the scene is Cypress, a complete end to end testing solution.

Read more...

GSoCpedia: The journey so far

“Imagine a world in which every single human being can freely share in the sum of all knowledge.”
~ Wikimedia Foundation

Read more...

Addressing bug from 2019: information about private, security-related Phab tickets

Written by Dsharpe on Jul 6 2020, 5:04 PM in Wikimedia Security Team.

Today, we are writing to share the discovery and squashing of a bug that occurred earlier this year. This particular bug was also one of the rare instances in which we kept a Phabricator ticket private to address a security issue. To help address questions about when and why we make a security-related ticket private, we’re also sharing some insight into what happens when a private ticket about a security issue is closed.

Read more...

Faster source code fetches thanks to git protocol version 2

Written by hashar on Jul 6 2020, 10:57 AM in Doing the needful.

In 2015 I noticed git fetches from our most active repositories to be unreasonably slow, sometimes up to a minute which hindered fast development and collaboration. You can read some of the debugging details I have conducted at the time on T103990. Gerrit upstream was aware of the issue and a workaround was presented though we never went to implement it.

Read more...

Production Excellence #21: May 2020

Written by Krinkle on Jun 24 2020, 8:00 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Celebrating 600,000 commits for Wikimedia

Written by Jdforrester-WMF on May 29 2020, 10:47 PM in Doing the needful.

Earlier today, the 600,000th commit was pushed to Wikimedia's Gerrit server. We thought we'd take this moment to reflect on the developer services we offer and our community of developers, be they Wikimedia staff, third party workers, or volunteers.

Read more...

Production Excellence #20: April 2020

Written by Krinkle on May 14 2020, 4:10 PM in Doing the needful.

How are we doing on that strive for operational excellence during these unprecedented times?

Read more...

Moving Milestones

Written by mmodell on May 1 2020, 10:39 AM in Phabricating Phabricator.

This week a long-standing "bug" has been fixed in Phabricator.

Read more...

Production Excellence #19: February 2020

Written by Krinkle on Mar 24 2020, 9:40 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

Coming to Terms with Change

Written by Ladsgroup on Mar 23 2020, 3:54 PM in The Sum of All Data.

We are dropping the wb_terms table, you might ask why we are doing it and why it’s such an important endeavour.

Read more...

The best documentation automation can buy

Written by Niedzielski on Mar 22 2020, 2:24 PM in Leave it to the prose.

HEADER CAPTION: Screenshot from Wikimedia's famous Visual Editor. The typo "documenation" has a red squiggly line under it indicating the spell checker has automatically detected a spelling error by the author.

Read more...

Production Excellence #18: January 2020

Written by Krinkle on Feb 28 2020, 7:39 PM in Doing the needful.

How’d we do in our strive for operational excellence last month? Read on to find out!

Read more...

New names for everyone!

Written by Andrew on Feb 18 2020, 9:14 PM in Clouds & Unicorns.

The Cloud Services team is in the process of updating and standardizing the use of DNS names throughout Cloud VPS projects and infrastructure, including the Toolforge project. A lot of this has to do with reducing our reliance on the badly-overloaded term 'Labs' in favor of the 'Cloud' naming scheme. The whole story can be found on this Wikitech proposal page. These changes will be trickling out over the coming weeks or months, but one change you might notice already.

Read more...

Parsoid in PHP, or There and Back Again

Written by cscott on Feb 12 2020, 3:56 PM in Wikitext & Parsing.

Authors: S.Subramanya Sastry, C.Scott Ananian; Parsing Team; Wikimedia Foundation

Read more...

Changes to Security Team Workflow

Written by JBennett on Feb 3 2020, 8:07 PM in Wikimedia Security Team.

In an effort to create a repeatable, streamlined process for consumption of security services the Security Team has been working on changes and improvements to our workflows. Much of this effort is an attempt to consolidate work intake for our team in order to more effectively communicate status, priority and scheduling. This is step 1 and we expect future changes as our tooling, capabilities and processes mature.

Read more...

Why does building a skin require PHP knowledge?

Written by Jdlrobson on Feb 3 2020, 6:37 AM in Leave it to the prose.

One of my longstanding pet peeves is that skin development for MediaWiki is so hard. I propose a radical change to how skins are installed and ask for feedback.

Read more...

14 January 2020 security incident on Phabricator

Written by Dsharpe on Jan 16 2020, 10:20 PM in Wikimedia Security Team.

On 14 January 2020, staff at the Wikimedia Foundation discovered that a data file exported from the Wikimedia Phabricator installation, our engineering task and ticket tracking system, had been made publicly available. The file was leaked accidentally; there was no intrusion. We have no evidence that it was ever viewed or accessed. The Foundation's Security team immediately began investigating the incident and removing the related files. The data dump included limited non-public information such as private tickets, login access tokens, and the second factor of the two-factor authentication keys for Phabricator accounts. Passwords and full login information for Phabricator were not affected -- that information is stored in another, unaffected system.

Read more...

Production Excellence #17: December 2019

Written by Krinkle on Jan 10 2020, 2:51 AM in Doing the needful.

How’d we do in our strive for operational excellence in November and December? Read on to find out!

Read more...