Matrix Project Hacked And Undergoes Extensive Infrastructure Rebuild - If you're a matrix.org user change your password now.

Matrix Project is an open source end-to-end encrypted messaging protocol that allows anyone to self-host a messaging service on their own servers. The service was a favorite among opensource developers, including many instant messengers, VoIP, WebRTC, bots and IoT communication. We ourselves were looking to it for IoT messaging.

The sad news is that hacker got access to key servers of the project. According to Matrix project unknown attackers exploited a sandbox bypass vulnerability in its production infrastructure on 4th of April via an outdated, vulnerable version of Jenkins automation server.

After taking down and fixing the other server on production, the Matrix Project found out that their DNS was hijacked and pointing to an defacement server hosted on Github.
Since then the team realized that the stolen encrypted password hashes were exfiltrated from the production database, Matrix.org forced to log out all users and strongly advised them to change their passwords immediately.

“Forensics are ongoing; so far we've found no evidence of large quantities of data being downloaded. The attacker did have access to the production database, so unencrypted content (including private messages, password hashes and access tokens) may be compromised,”"This was a difficult choice to make. We weighed the risk of some users losing access to encrypted messages against that of all users' accounts being vulnerable to hijack via the compromised access tokens, We hope you can see why we made the decision to prioritize account integrity over access to encrypted messages, but we're sorry for the inconvenience this may have caused." said the project management, “Forensics are ongoing; so far we've found no evidence of large quantities of data being downloaded. The attacker did have access to the production database, so unencrypted content (including private messages, password hashes and access tokens) may be compromised,”


The Press Release by the Matrix Project (You may also find the latest updates via the link);

We have discovered and addressed a security breach. (Updated 2019-04-12)

by Matthew Hodgson | Apr 11, 2019 | Uncategorized | 0 comments

Here's what you need to know.

TL;DR: An attacker gained access to the servers hosting Matrix.org. The intruder had access to the production databases, potentially giving them access to unencrypted message data, password hashes and access tokens. As a precaution, if you're a matrix.org user you should change your password now.
The matrix.org homeserver has been rebuilt and is running securely; bridges and other ancillary services (e.g. this blog) will follow as soon as possible. Modular.im homeservers have not been affected by this outage.

The security breach is not a Matrix issue.

The hacker exploited a vulnerability in our production infrastructure (specifically a slightly outdated version of Jenkins). Homeservers other than matrix.org are unaffected.

How does this affect me?

We have invalidated all of the active access tokens for users on Matrix.org - all users have been logged out.
Users with Matrix.org accounts should:

• Change your password now - no plaintext Matrix passwords were leaked, but weak passwords could still be cracked from the hashed passwords
• Change your NickServ password (if you're using IRC bridging) - there's no evidence bridge credentials were compromised, but if you have given the IRC bridges credentials to your NickServ account we would recommend changing this password

And as a reminder, it's good practice to:

• Review your device list regularly - make sure you recognise all of the devices connected to your account
• Always make sure you enable E2E encryption for private conversations

What user data has been accessed?

Forensics are ongoing; so far we've found no evidence of large quantities of data being downloaded. The attacker did have access to the production database, so unencrypted content (including private messages, password hashes and access tokens) may be compromised.

What has not been affected?

• Source code and packages have not been impacted based on our initial investigations. However, we will be replacing signing keys as a precaution.
• Modular.im servers are not affected, based on our initial analysis
• Identity server data does not appear to have been compromised

The target appeared to be internal credentials for onward exploits, not end user information from the matrix.org homeserver.

You might have lost access to your encrypted messages.

As we had to log out all users from matrix.org, if you do not have backups of your encryption keys you will not be able to read your encrypted conversation history. However, if you use server-side encryption key backup (the default in Riot these days) or take manual key backups, you’ll be okay.
This was a difficult choice to make. We weighed the risk of some users losing access to encrypted messages against that of all users' accounts being vulnerable to hijack via the compromised access tokens. We hope you can see why we made the decision to prioritise account integrity over access to encrypted messages, but we're sorry for the inconvenience this may have caused.

What happened?

We were using Jenkins for continuous integration (automatically testing our software). The version of Jenkins we were using had a vulnerability (CVE-2019-1003000, CVE-2019-1003001, CVE-2019-1003002) which allowed an attacker to hijack credentials (forwarded ssh keys), giving access to our production infrastructure. Thanks to @jaikeysarraf for drawing this to our attention.

Timeline

March 13th Updated 2019-04-12 11:00 UTC

• Attacker compromises Jenkins CI server

April 4th Updated 2019-04-12 11:00 UTC

• Attacker gains access to production infrastructure by hijacking a forwarded SSH agent logging into the compromised Jenkins worker

April 9th

• Jenkins vulnerability brought to our attention by @jaikeysarraf

April 10th

• Investigation identified the compromised machines and the full scope of the attack
• Jenkins was removed
• Attacker's access to compromised machines was removed

April 11th

• Matrix.org was taken offline and production infrastructure fully rebuilt
• Having fully flushed out the attacker, external communication was published informing users and advising on next steps
• Matrix.org homeserver restored, with bridges and ancillary services (e.g. this blog) following as soon as possible

Update 2019-04-12

At around 5am UTC on Apr 12, the attacker used a cloudflare API key to repoint DNS for matrix.org to a defacement website (https://github.com/matrixnotorg/matrixnotorg.github.io). The API key was known compromised in the original attack, and during the rebuild the key was theoretically replaced. However, unfortunately only personal keys were rotated, enabling the defacement. We are currently doublechecking that all compromised secrets have been rotated.
The rebuilt infrastructure itself is secure, however, and the DNS issue has been solved without further abuse. If you have already changed your password, you do not need to do so again.
The defacement confirms that encrypted password hashes were exfiltrated from the production database, so it is even more important for everyone to change their password. We will shortly be messaging and emailing all users to announce the breach and advise them to change their passwords. We will also look at ways of non-destructively forcing a password reset at next login.
The attacker has also posted github issues detailing some of their actions and suggested remediations at https://github.com/matrix-org/matrix.org/issues/created_by/matrixnotorg.
This confirms that GPG keys used for signing packages were compromised. These keys are used for signing the synapse debian repository (AD0592FE47F0DF61), and releases of Riot/Web (E019645248E8F4A1). Both keys have now been revoked. The window of compromise for the keys started from April 4th; there have been no Synapse releases since then. There has been one release of Riot/Web (1.0.7), however as the key was passphrased and based on our initial analysis of the release, we believe it to be secure.

What are we doing to prevent this in future?

Once things are back up and running we will retrospect on this incident in detail to identify the changes we need to make. We will provide a proper postmortem, including follow-up steps; meanwhile we are obviously going to take measures to improve the security of our production infrastructure, including patching services more aggressively and more regular vulnerability scans.

Unlimited Private Repositories On GitHub For Free Github Accounts.

We are avid users of Github and since our major products were opensource, we did not pay for private repos. For those, we had our own GIT server running. But it looks like things are changing.
GitHub  always offered free accounts but the caveat was that your code had to be public. To get private repositories, you had to pay. Starting tomorrow, that limitation is gone. Free GitHub users now get unlimited private projects with up to three collaborators.

The amount of collaborators is really the only limitation here and there’s no change to how the service handles public repositories, which can still have unlimited collaborators.

Talking about teams, GitHub also today announced that it is changing the name of the GitHub Developer suite to ‘GitHub Pro.’ The company says it’s doing so in order to “help developers better identify the tools they need.”

But what’s maybe even more important is that GitHub Business Cloud and GitHub Enterprise (now called Enterprise Cloud and Enterprise Server) have become one and are now sold under the ‘GitHub Enterprise’ label and feature per-user pricing.

Good one on you Github and Microsoft.

WhiteSource Bolt for GitHub Will Keep Your Projects On Github, Open Source Or Otherwise, Trouble Free

More and more Open source projects are facing vulnerability issues. According to "The State of Open Source Vulnerability Management" white paper (Download link below) It is evident that most vulnerabilities are found in popular projects.
Data shows that 32% of the top 100 in part thanks to the community reporting, the number of reported vulnerabilities in open source projects is on the rise, rising to 51% in 2017 from the previous year. Most open source projects are vulnerable to at least one issue due to multi level component use, meaning that developers have their work cut out for them, as long as they are using these components in their products. 

WhiteSource Bolt for GitHub also support open source app with some restrictions. The WhiteSource will allow you to monitor one opensource project, provided you meet the following requirements. If you need to add more projects, it is possible with a paid account. They do have multi level systems to match your requirements.

Free Open Source Project Support:

1) The license is limited to analysis of one open source product developed and published by your organization.

2) You must put a link to WhiteSource on the project home page with credit to WhiteSource for analyzing your code along with the following statement:
"WhiteSource is providing free open source analysis solution to this project, as part of its on-going support to the open source community."

3) You agree that WhiteSource can publish your organization and/or oss project name and logo.

4) You agree that WhiteSource can use and publish your data at our discretion

Even if you think it is not for you, visit them, you might like the resources they provide. One good place to start would be "The State of Open Source Vulnerability Management" We will be requesting one of our Guardocs related opensource project to be monitored.