The largest Git repo on the planet

The largest Git repo on the planet

This site uses cookies for analytics, personalized content and ads. By continuing to browse this site, you agree to this use. Learn more
Server & Tools Blogs > Developer Tools Blogs > Brian Harrys blog
Sign in Menu
Brian Harrys blog
Everything you want to know about VSTS and Farming

Brian Harrys blog
The largest Git repo on the planet
★★★★★★★★★★★★★★★
05/24/2017 by Brian Harry MS // 61 Comments

0
0
It’s been 3 months since I first wrote about our efforts to scale Git to extremely large projects and teams with an effort we called “Git Virtual File System”. As a reminder, GVFS, together with a set of enhancements to Git, enables Git to scale to VERY large repos by virtualizing both the .git folder and the working directory. Rather than download the entire repo and checkout all the files, it dynamically downloads only the portions you need based on what you use.

A lot has happened and I wanted to give you an update. Three months ago, GVFS was still a dream. I don’t mean it didn’t exist – we had a concrete implementation, but rather, it was unproven. We had validated on some big repos but we hadn’t rolled it out to any meaningful number of engineers so we had only conviction that it was going to work. Now we have proof.

Today, I want to share our results. In addition, we’re announcing the next steps in our GVFS journey for customers, including expanded open sourcing to start taking contributions and improving how it works for us at Microsoft, as well as for partners and customers.

Windows is live on Git

Over the past 3 months, we have largely completed the rollout of Git/GVFS to the Windows team at Microsoft.

As a refresher, the Windows code base is approximately 3.5M files and, when checked in to a Git repo, results in a repo of about 300GB. Further, the Windows team is about 4,000 engineers and the engineering system produces 1,760 daily “lab builds” across 440 branches in addition to thousands of pull request validation builds. All 3 of the dimensions (file count, repo size and activity), independently, provide daunting scaling challenges and taken together they make it unbelievably challenging to create a great experience. Before the move to Git, in Source Depot, it was spread across 40+ depots and we had a tool to manage operations that spanned them.

As of my writing 3 months ago, we had all the code in one Git repo, a few hundred engineers using it and a small fraction (<10%) of the daily build load. Since then, we have rolled out in waves across the engineering team.

The first, and largest, jump happened on March 22nd when we rolled out to the Windows OneCore team of about 2,000 engineers. Those 2,000 engineers worked in Source Depot on Friday, went home for the weekend and came back Monday morning to a new experience based on Git. People on my team were holding their breath that whole weekend, praying we weren’t going be pummeled by a mob of angry engineers who showed up Monday unable to get any work done. In truth, the Windows team had done a great job preparing backup plans in case of mishap and, thankfully, we didn’t have to use any of them.

Much to my surprise, quite honestly, it went very smoothly and engineers were productive from day one. We had some issues, no doubt. For instance, Windows, because of the size of the team and the nature of the work, often has VERY large merges across branches (10,000’s of changes with 1,000’s of conflicts). We discovered that first week that our UI for pull requests and merge conflict resolution simply didn’t scale to changes that large. We had to scramble to virtualize lists and incrementally fetch data so the UI didn’t just hang. We had it resolved within a couple of days and overall, sentiment that week was much better than we expected.

One of the ways we measured our success was by doing surveys of the engineering team. The main question we asked was “How satisfied are you?” but, of course, we also mined a lot more detail. Two weeks into the rollout, our first survey resulted in:

gitsurvey

I’m not going to jump up and down and celebrate those numbers, but for a team that had just had their whole life changed, had to learn a new way of working and were living through a transition that was very much a work in progress, I felt reasonably good about it. Yes, it’s only 251 survey responses out of 2,000 people but welcome to the world of trying to get people to respond to surveys. 🙂

Another way we measured success was to look at “engineering activity” to see if people were still getting their work done. For instance, we measured number of “checkins” to official branches. Of course, half the team was still on Source Depot and half had moved to Git so we looked at combined activity over time. In the chart below you can see the big drop in Source Depot checkins and the big jump in Git pull requests but overall the sum of the two stayed reasonable consistent. We felt that the data showed that the system was working and there were no major blockers.

activity

On April 22nd, we onboarded the next wave of about 1,000 engineers. And then on May 12th we onboarded another 300-400. Each successive wave followed roughly the same pattern and we now have about 3,500 of the roughly 4,000 Windows engineers on Git. The remaining teams are currently working to deadlines and trying to figure out when is the best time to schedule their move, but I expect, in the next few months we’ll complete the full engineering team.

The scale the system is operating at is really amazing. Let’s look at some numbers…

There are over 250,000 reachable Git commits in the history for this repo, over the past 4 months.
8,421 pushes per day (on average)
2,500 pull requests, with 6,600 reviewers per work day (on average)
4,352 active topic branches
1,760 official builds per day
As you can see, it’s just a tremendous amount of activity over an immensely large codebase.

GVFS performance at scale

If you look at those satisfaction survey numbers, you’ll see there are people who aren’t happy yet. We have lots of data on why and there are many reasons – from tooling that didn’t support Git yet to frustration at having to learn something new. But, the top issue is performance, and I want to drill into that. We knew when we rolled out Git that lots of our performance work wasn’t done yet and we also learned some new things along the way. We track the performance of some of the key Git operations. Here is data collected by telemetry systems for the ~3,500 engineers using GVFS.

performance

You see the “goal” (which was designed to be a worst case, the system isn’t usable if it’s slower than this value, not a “this is where we want to be” value). You also see the 80th percentile result for the past 7 days and the delta from the previous 7 days (you’ll notice everything is getting slower – more on that in a minute).

For context, if we tried this with “vanilla Git”, before we started our work, many of the commands would take 30 minutes up to hours and a few would never complete. The fact that most of them are less than 20 seconds is a huge step but it still sucks if you have to wait 10-15 seconds for everything.

When we first rolled it out, the results were much better. That’s been one of our key learnings. If you read my post that introduced GVFS, you’ll see I talked about how we did work in Git and GVFS to change many operations from being proportional to the number of files in the repo to instead be proportional to the number of files “read”. It turns out that, over time, engineers crawl across the code base and touch more and more stuff leading to a problem we call “over hydration”. Basically, you end up with a bunch of files that were touched at some point but aren’t really used any longer and certainly never modified. This leads to a gradual degradation in performance. Individuals can “clean up” their enlistment but that’s a hassle and people don’t, so the system gets slower and slower.

That led us to embark upon another round of performance improvements we call “O(modified)” which changes the proportionality of many key commands to instead be proportional to the number of files I’ve modified (meaning I have current, uncommitted edits on). We are rolling these changes out to the org over the next week so I don’t have broad statistical data on the results yet but we do have good results from some early pilot users.

I don’t have all the data but I’ve picked a few examples from the table above and copied the performance results into the column called “O(hydrated)”. I’ve added another column called O(modified) with the results for the same commands using the performance enhancements we are rolling out next week. All the numbers are in seconds. As you can see we are getting performance improvements across the board – some are small, some are ~2X and status is almost 5X faster. We’re very optimistic these improvements are going to move the needle on perf perception. I’m still not fully satisfied (I won’t be until Status is under 1 second), but it’s fantastic progress.

beforeafter

Another key performance area that I didn’t talk about in my last post is distributed teams. Windows has engineers scattered all over the globe – the US, Europe, the Middle East, India, China, etc. Pulling large amounts of data across very long distances, often over less than ideal bandwidth is a big problem. To tackle this problem, we invested in building a Git proxy solution for GVFS that allows us to cache Git data “at the edge”. We have also used proxies to offload very high volume traffic (like build servers) from the main Visual Studio Team Services service to avoid compromising end user’s experiences during peak loads. Overall, we have 20 Git proxies (which, BTW, we’ve just incorporated into the existing Team Foundation Server Proxy) scattered around the world.

To give you an idea of the effect, let me give an example. The Windows Team Services account is located in an Azure data center on the west coast of the US. Above you saw that the 80th percentile for Clone for a Windows engineer is 127 seconds. Since a high percentage of our Windows engineers are in Redmond, that number is dominated by them. We ran a test from our North Carolina office (which is both further away and has a much lower bandwidth network). A clone from North Carolina with no proxy server took almost 25 minutes. With a proxy configured and up to date, it took 70 seconds (faster than Redmond because the Redmond team doesn’t use a proxy and they have to go hundreds of miles over the internet to the Azure data center). 70 seconds vs almost 25 minutes is an almost 95% improvement. We see similar improvements when GVFS “faults in” files as they are accessed.

Overall Git with GVFS is completely usable at crazy large scale and the results are proving that our engineers are effective. At the same time, we have a lot of work to do to get the performance to the point that our engineers are “happy” with it. The O(modified) work rolling out next week will be a big step but we have months of additional performance work still on the backlog before we can say we’re done.

To learn more about the details of the technical challenges we’ve faced in scaling Git and getting good performance, check out the series of articles that Saeed Noursalehi is writing on scaling Git and GVFS. It’s fascinating to read.

Trying GVFS yourself

GVFS is an open source project and you are welcome to try it out. All you need to do is download and install it, create a Visual Studio Team Services account with a Git repo in it and you are ready to go. Since we initially published GVFS, we’ve made some good progress. Some of the key changes include:

We’ve started doing regular updates to the published code base – moving towards “development in the open”. As of now, all our latest changes (including the new O(modified) work) are published to the public repo and we will be updating it regularly.
When we first published, we were not ready to start taking external contributions. With this milestone today, we are now, officially ready to start. We feel like enough of the basic infrastructure is in place that people can start picking it up and moving it forward with us. We welcome anyone who wants to pitch in and help.
GVFS relies on a Windows filesystem driver we call GVFlt. Until now, the drop of that driver that we made available was unsigned (because it was very much a work in progress). That clearly creates some friction in trying it out. Today, we released a signed version of GVFlt that will eliminate that friction (for instance, you no longer need to disable BitLocker to install it). Although we have a signed GVFlt driver, that’s not the long term delivery method. We expect this functionality to be incorporated into a future shipping version of Windows and we are still working through those details.
Starting with our talk at Git Merge, we’ve begun engaging with the broader Git community on the problem of scaling Git and GVFS, in particular. We’ve had some great conversations with other large tech companies (like Google and Facebook) who have similar scaling challenges and we are sharing our experiences and approaches. We have also worked with several of the popular Git clients to make sure they work well with GVFS. These include:
Atlassian SourceTree – SourceTree was the first tool to validate with GVFS and have already released an update with a few changes to make it work well.
Tower – The Tower Git team is excited to add GVFS support and they are already working on include GVFS in the Windows version of their app. It will be available as a free update in the near future.
Visual Studio – Of course, it would be good for our own Visual Studio Git integration to work well with GVFS too. We are including GVFS support in VS 2017.3 and the first preview with the necessary support will be available in early June.
Git for Windows – As part of our effort to scale Git, we have also made a bunch of contributions to Git for Windows (the Git command line) and that includes support for GVFS. Right now, we still have a private fork of Git for Windows but, over time, we are working to get all of those changes contributed back to the mainline.
Summary

We’re continuing to push hard on scaling Git to large teams and code bases at Microsoft. A lot has happened in the 3 months since we first talked about the effort. We’ve…

Successfully rolled it out to 3,500 Windows engineers
Made some significant performance improvements and introduced Git proxies
Updated the open source projects with the latest code and opened it for contributions
Provided a signed GVFlt driver to make trying it out easier
Worked with the community to begin to build support into popular tools – like SourceTree, Tower, Visual Studio, etc.
Published some articles with more insights into the technical approach we are taking to scale Git and GVFS.
This is an exciting transition for Microsoft and a challenging project for my team and the Windows team. I’m elated at the progress we’ve made and humbled by the work that remains. If you too find there are times where you need to work with very large codebases and, yet you really you really want to move to Git, I encourage you to give GVFS a try. For now, Visual Studio Team Services is the only backend implementation that supports the GVFS protocol enhancements. We will add support in a future release of Team Foundation Server if we see enough interest and we have talked to other Git services who have some interest in adding support in the future.

Thanks and enjoy.

Brian

Download Visual Studio
Download TFS
Visual Studio Team Services
Search

Search MSDN with Bing

Search this blog Search all blogs
Subscribe Blog via Email
Subscribe to this blog and receive notifications of new posts by email.

Email Address
Subscribe! Unsubscribe
Tags
announcement Application Insights Codeplex Eclipse HockeyApp msdn Performance Personal RCA TestPro TFS TFS Dogfood statistics TFService Visual Studio VSOnline VS Team Services VSTS
Archives
March 2018 (5)
February 2018 (3)
All of 2018 (10)
All of 2017 (58)
All of 2016 (56)
All of 2015 (58)
All of 2014 (88)
All of 2013 (108)
All of 2012 (82)
All of 2011 (78)
All of 2010 (143)
All of 2009 (120)
All of 2008 (124)
All of 2007 (177)
All of 2006 (74)
All of 2005 (12)
All of 2004 (1)
Tags
VS Team Services
Join the conversation
Add Comment
2 years ago

Mike-EEE
Wow… your farming posts are fun, Brian. But this is where you truly earn your bacon. Guess I should pay attention whether you run a pig farm or not. 😛 In any case, incredible post.

2 years ago

Thomas Ricker
WOW!

JUST WOW!

MIND BLOWN.

2 years ago

Bartosz
Amazing, nice work! But how current GVFS performance compares to previous solution (Source Depot)?

2 years ago

Sam Atwell
Are you guys talking to other Git client developers than the ones you mentioned? Tooling is one of the more important tools for something to be picked up everyone.

I am thinking specifically of GitHub Desktop, SmartGit and TortoiseGit as they are probably the 3 biggest Git clients (other than the ones you already mentioned).

2 years ago

Brian Harry MS
@Bartosz, We have. It depends a great deal on the operation. SourceDepot was much faster at some things – like “sd opened”, the equivalent of “git status”. sd opened was < .5s. git status is at 2.6s now. But SD was much slower at some other things – like branching. Creating a branch in SD would take hours. In Git, it's less than a minute. I saw a mail from one of our engineers at one point saying they'd been putting off doing a big refactoring for 9 months because the branch mechanics in SD would have been so cumbersome and after the switch to Git they were able to get the whole refactoring done in a topic branch in no time.

On an operation, by operation basis, SD is still much faster than our Git/GVFS solution. We're still working on it to close the gap but I'm not sure it will ever get as fast at everything. The broader question, though is about overall developer productivity and we think we are on a path to winning that.

Brian

2 years ago

Brian Harry MS
@Sam, yes, we are working with a bunch of Git clients. I focused on the ones that have made good progress. There are others that are very interested but aren’t close to having something and others that are waiting to see how much interest there is. To drive some of this, the developers of the world will have to put in their own vote.

Brian

2 years ago

Jeremy
Really cool work.

Will there ever be a discussion on what the Git branching / release strategy is for Windows? I think a lot of people coming from the monolithic enterprise world (especially those of us dealing with multiple parallel release streams) struggle with how to model such workflows in Git.

2 years ago

Zach
@sam, I’ve been doing this for 10 years and have yet to even hear of smartgit and tortoisegit.

2 years ago

Brian Harry MS
@Jeremy, I’d be happy to write something at some point. I’ll give a short summary here. Windows is still in transition from the “old” long term branch proliferation model to a newer solution with many fewer branches and work happening much closer to master. That said, they are likely too big of an org to ever get too one branch. We’ll see. So they still have branches off master for each of the “major” Windows orgs and a schedule for managing code motion between them The result is about 400 long term branches. That’s a big reduction from where they were at one point but not where they want to be.

We had to decouple the process to evolve branching structure and code flow from the migration to Git so we are using more long term branches than a Git team normally would.

My team (the Visual Studio Team Services/Team Foundation Server team) is much smaller (hundreds of engineers rather than thousands) works in a model that is closer to what I’d recommend as a north star for most teams. All works happens as close to master as possible and we have no (or very few – count them on 1 hand) long term development branches other than master. People create short lived topic branches, do their work and merge to master quickly. We “branch for release” by creating a new branch for each release we ship and all the servicing work for the release happens in the release branch.

Brian

2 years ago

Ben the Builder
Any chance I could do a pull request 🙂

2 years ago

TJ
# Ahh. This makes sense.
#This. But in reverse – sorry, i’ve been at it too long but thanks!

2 years ago

Peter Dave Hello
Hi Brian,

Thanks for this cool experience sharing, I’m also experiencing git performance issue on cdnjs* project, which has about a 88GB large repository(3.5G .git directory), 40k commits, 3.9M files and 0.3M directories, we use large memory, fast processor, SSD and git sparseCheckout in different scenarios to speed up our git operations, but it’s still not a happy speed like the “git status” will still take 30 seconds(slower the first time if there is no cache for filesystem), so it’s very excited to see the GVFS work from Microsoft, which may be the potential solution to our very same problem, it’ll be great if you guys will consider supporting GVFS on unix-like systems like FreeBSD or Linux distributions, thanks again!

https://cdnjs.com
https://github.com/cdnjs/cdnjs

2 years ago

Jay
How often do you anticipate having to delete the current git repo and downloading a fresh copy?

2 years ago

Gurinder
In a previous MS article I read, I was under the impression the Windows team was using Team Services (online) + VS2017. Is that the “…UI for pull requests and merge conflict resolution…” you were referring to or did you have to build a custom UI to handle GVFS and the scale of your pulls/merges? I ask because when you consider Git at scale you also need to consider git tools at scale – can they handle them and can the developer effectively work with the data being shown to them as a result of git’s data model.

2 years ago

Allan
Hi Brian,

Great write-up — thanks for the insights. Do you have a sense of when GitHub support will land?

Allan

2 years ago

Brian Harry MS
@Peter, we will be working on a Mac/Linux port of our work shortly.
@Jay, No more frequently than you normally would with Git. We don’t think of that as part of the solution.
@Gurinder, They do use Team Services – the same one our external customers use too. It supports the GVFS endpoints. I’m going to write some more about the tooling changes and I’ll talk about the merge conflict experience. We’ve written a new one using our extensibility points. We plan to make it available to external customers in the next couple of months.
@Allan, I don’t. We’ve shared our work with GitHub but they have not told us if they have any plans or what they are.

Brian

2 years ago

Sean Beseler
all I can say is wow

2 years ago

Matěj Cepl
Welcome to the Light Side! Anyway, two comments:

1. How many changes there were between the last commit in the local checkout and the remote with that git pull? A minute is quote a long time (and just barely shorter than git clone, which seems suspicious). If it was a realistic morning fetch of all commits from the previous night, then it is strange.

2. I know nobody wants to change their workflow because of tools, but my experience is that git really works better with small commits. Is it correct that the long commit and push times are caused by large commits? And on the top of that commit is not the last moment one touches git repo. With switch to git one gets ability to use tools like git bisect and those are quite useless with too large commits. I really liked this blogpost from of mine colleagues who used to work on the project management of Xorg (quite large project in the free software world) https://who-t.blogspot.cz/2009/12/on-commit-messages.html (also other blogposts in the workflow tag).

2 years ago

Michael Lerro
Very interesting, thanks.

Love that you guys are releasing your work as open source and sharing with the greater community

2 years ago

Lewis Cowles
It’s awesome Microsoft is using and contributing to open-source solutions. The satisfaction-survey part made me chuckle, but I’d imagine once the roll-out is complete stretch goals and future iterations will be able to address tooling that makes git largely invisible.

2 years ago

Troy SK
Access or it didn’t happen! 😉

2 years ago

Brian Harry MS
@Troy, I’m afraid I don’t understand your issue. Could to share more?

Brian

2 years ago

Scott Brickey
after reading some of the considerations/goals/etc of GVFS, I’d be curious whether prefetching could be given recommendations from the TFS server, based on ML identified trends within the repo… perhaps some files are being used more than others (when new features are being added, such as vista’s UAC, which probably involve a bunch of debug step-into), or by certain users (ideally groups, but unsure how you’d identify that within TFS) focusing on specific areas.

Seems like the ML would be one of those larger investments (like SCVMM), but if it’s effective, would really be useful for the uber-large repos (on-prem if necessary), or potentially just really nice performance benefit for VSO subscribers.

2 years ago

Brian Harry MS
@Scott, yes, I suspect we could do some predictive modeling and optimize the cached state. We don’t yet. At some point, we might look into that.

Brian

2 years ago

Gustavo
Is it better to have a single repo? Devs are not so good at respecting boundaries if everything is logically accesible and the end result is projects depending on hundreds of projects instead of librarries which is more efficient. I have experienced this problem with a team of around 150 Devs, I guess it’s worse with thousands of Devs contributing.

I could see benefits or having a Nano repo, and then layers of repos to get to a Full version.

2 years ago

deepender07@gmail.com
This is great step for microsoft..!
GVFS is indeed a requirement for large codebases.

2 years ago

Victor C Nwafor
wow….. can’t wait to be a part of this awesome creatives…

2 years ago

Brian Harry MS
@Gustavo, Let me shed a little more light on this. Some explanation gets cut in an attempt to keep the “story” short and digestable. It turns out the Windows and Devices Group (WDG) that is responsible for Windows, Xbox, Phone, HoloLens and all the extended platform pieces around them actually has quite a large number of Git repos. I don’t have an exact count but it’s in the high hundreds to low thousands of repos. The focus of GVFS has been on the “OS” repo which is the core operating system. We looked very hard at decomposing it and we found that our workflow just was not amenable to that. You might checkout the discussion on Hacker News and elsewhere and find that other large engineering companies like Google and Facebook reached similar conclusion about their core platforms and have adopted solutions with the same general aim as ours.

Brian

2 years ago

Marc-Andre Poitras
Are there any plans to host Team Services in other Azure data centers? We are limited to using the on-premise version because of data residency issues. We have 2 data centers in Canada which would be great if we could have Team Services run there.

Thanks for all the awesome work. And keep the Farm stories coming!

2 years ago

ex v-miczer
Is Bing division also switching from Source Depot to Git?

2 years ago

mpwags
WOW, nice work folks!

2 years ago

Christopher Nelson
Google has the largest single repository. Check your facts:

Google’s monolithic software repository, which is used by 95% of its software developers worldwide, meets the definition of an ultra-large-scale system, providing evidence the single-source repository model can be scaled successfully.

The Google codebase includes approximately one billion files and has a history of approximately 35 million commits spanning Google’s entire 18-year existence. The repository contains 86TB of data, including approximately two billion lines of code in nine million unique source files.

https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext

2 years ago

Greg
With that quantity of developers pushing to the same repository, how is anyone ever “current enough” to successfully push. I have seen even with 10 developers, this happen (using simplified branch terminology):

1. Finish up a task branch (all current with Master at that moment)
2. Merge task branch into Master
3. Push Master
4. Whoops – you are no longer current. Someone pushed before you.
5. Pull, merge, get current again.
6. Push Master
7. Whoops – you are no longer current. Someone pushed before you.
8. ……

I can only imagine with 4K devs how challenging that could be.

Greg

2 years ago

Philip Patrick
First of all – nice work! But of course there are questions 🙂 What is particularly interesting is how did you migrated the history. We are currently on TFVC and would like to switch to Git in TFS, but would like to preserve all history. There is git-tf tool that we will surely try out, but maybe you have better idea?
Another question – to my understanding that means your are not really working in distributed VC, since everything is virtualized and will have server connection to bring the code down to dev’s machine, or do I miss something?

2 years ago

Brian Harry MS
@Christopher, I said the largest Git repo. You are correct that Google has long had a very large mono-repo. We’ve talked to them about it a fair bit over the years. But their mono-repo is not in Git. Hence I stand by my claim (at least on that account). They do use Git for Android (and I’m sure elsewhere) but none of those repos approach the size.

Brian

2 years ago

Saeed Noursalehi
@Matěj, good questions.

1. That pull number shows the 80th percentile time from our telemetry, so it’s not measuring one specific pull. That said, the majority of the pull time is dominated by two things:
– We have added a pre-command step on fetch and pull to download all commits and trees from the server, for all commits since the last fetch/pull. This allows us to have those lightweight objects present locally, but not download the blobs until they are needed. This can take some time, depending on how many new commits there are, and in the Windows repo, the answer is always: a lot of new commits.
– The merge that happens during pull is one of those commands that gets slower and slower the more people’s repos get hydrated (because those telemetry numbers don’t reflect our O(modified) changes, which were just released this week).

You compared the time to clone, but clone is simpler in some ways because it only has to download a packfile of commits and trees (again, no blobs) but doesn’t have to do a merge since we start out with a pure virtual projection of the repo.

2. Some of the commits can be large (e.g. merges between two teams’ branches), but the vast majority of commits are small. The long git commands aren’t really caused by “big commits”. Most of the cost comes from the fact that, with 3M+ files in the repo, the git index is huge and takes time to process. Plus, with the previously deployed GVFS, the more the repo was hydrated, the more those commands also had to do IO in the working directory. We expect many of those commands to be much faster now with O(modified), but telemetry will tell the real story.

2 years ago

Roman Frołow
@Greg

Use my script

https://github.com/rofrol/git-helpers/blob/master/git-prmp

1. update of master without checkout to master
git fetch -f origin ${branch_to_merge_on}:${branch_to_merge_on}
2. then rebase
git rebase ${branch_to_merge_on}
3. merge to master without checkout to master
git fetch . HEAD:${branch_to_merge_on}
4. push
git push origin ${branch_to_merge_on}

2 years ago

Brian Harry MS
@Philip, Actually, they decided to start “fresh” and did not migrate the history. They moved all the “tip” source code over and will leave Source Depot around indefinitely for people who need to go back and look at older revisions.

Brian

2 years ago

Edilson Azevedo
Cloud Team – BrazilHuE

2 years ago

MoiMoi666
Hi,

Many thanks for the blog post. I appreciate these insights so much!

There is still something not really clear to me: using such a large Git mono-repo, how does it affect the build time? I would expect very long compile time because of the immense source code. Which strategy is applied in these cases?

2 years ago

Brian Harry MS
@MoiMoi666, Yes, the Windows build is VERY long if you build the whole thing. The team goes to great lengths to ensure that most people don’t have to build the whole thing. We also have sophisticated build technology to parallelize and cache builds. Most developers don’t need to build too much of it too frequently so it’s not crippling but it does create friction. Once we wrap up our Git work, I’ll start to blog about some of the work we are doing in the build space.

Brian

2 years ago

Brian Harry MS
@ex v-miczer, Yes, most of Bing has already moved to Git. They did so without GVFS though they have 1 or 2 repos big enough that they’d like to use GVFS. We’ve just been asking them to wait until we get the Windows team successful. A couple of weeks ago, I asked that we go start engaging with the Bing Team on GVFS because I think we are ready.

Brian

2 years ago

Brian Harry MS
@Marc-Andre Poitras, Yes. We will be expanding into more data centers in the second half of this year and Canada is on our list of top candidates.

Brian

2 years ago

Saeed Noursalehi
@Greg, you’re absolutely right that with 4000 devs, there’s no way each dev could pull, merge, and push fast enough to ever get their changes in. We even struggled with that on a team of 300. The way we’ve handled that is that no one pushes directly to a shared branch like master. Not only do you have this race to contend with, but it’s also difficult to maintain quality if people can just push code to master at any time. So we require pull requests into all important branches, both to enforce policies, and also to allow the server to handle that pull/merge/push race for you. I’ll talk about how we did this in more detail in a follow up article to this one: https://www.visualstudio.com/learn/technical-scale-challenges/

2 years ago

Wilson
Hi,

Is the driver downloadable via nuget ( https://www.nuget.org/packages/Microsoft.GVFS.GvFlt/ ) ?

thanks.

2 years ago

RicarDog
Hello Brian, you said in other posts that MS intends to keep both DVCS (Git) and centralized VCS (TFVC) options, however the roadmap for TFVC is not so clear, and some VS documentation already refer to it as “legacy” (https://www.visualstudio.com/learn/migrate-from-tfvc-to-git/).

I work on a game dev company and I believe most professionals agree that a centralized VCS is the best option for game development, due to the high volume of non-mergeable binary assets that benefit from locking control, ease of use by non-technical guys such as artists, etc. That’s why Perforce is still very popular in the game industry.

So could you please elaborate on your plans for TFVC? Is it still being supported and developed? Are there plans for improvements / new features to TFVC and its tools or is it really a legacy system that will be kept as-is?

Please, do not turn it into a second-class citizen in the VCS world. I know the industry is walking towards DVCS but there are still many non-legacy use cases that make TFVC a better option. Thanks!

2 years ago

Wilson
In case anyone wants to download the driver, it is really at nuget: https://www.nuget.org/packages/Microsoft.GVFS.GvFlt/

cheers

2 years ago

Brian Harry MS
@RicarDog, You are right that they are scenarios where a CVS is better than a DVCS and that is part of why we support both. We do continue to invest in both (though, certainly in the last year or so, the investment in Git has been much higher). We recently, though, for instance, added support for TFVC for Azure CI/CD. The VSCode team is working on adding TFVC support right now too. We’re working on an updated Windows Shell extension for TFVC. And more. I’ll talk to the team about the use of the word “legacy” in the docs. I suspect the word was used lightly. The context is people moving to Git from any other version control system.

Brian

2 years ago

Jason
Very cool!

2 years ago

Robert Pancoast
As Engineers we require exact precision for collaboration.

GIT and Visual Studio Team Services also work well for Electrical schematics, not just plain-text!
The Microsoft backed security is priceless.

Very well done Microsoft.

2 years ago

Daniel Vicarel
GVFS sounds awesome, but so does the Git LFS project being worked on by GitHub/Atlassian (https://git-lfs.github.com/). I’m stunned that you’ve only given LFS a single passing mention in the two articles that you’ve written about scaling Git so far. LFS has been publicly lauded since 2015, so I don’t know how you can act like Microsoft is the first or only organization to tackle the Git large-file/large-repo problem. You may not have said this explicitly, but totally ignoring the obvious competition is just about as bad. I personally would need an honest comparison of GVFS to LFS, and/or to other large-scale VCSs like Perforce, before I could take GVFS any more seriously. That said, these past two articles have been a very clear and informative, so I look forward to any further information from you on this topic.

2 years ago

Brian Harry MS
@Daniel, We are very familiar with LFS. We have worked with GitHub on it and support it both in Team Services and in TFS. We recently added support for the new locking mechanism introduced in LFS.

You’re right that I didn’t talk a lot about it and maybe I should have. I’ll see if Saeed can cover it in more depth in his technical articles.

At I high level GVFS and LFS have some overlap but not a ton. LFS – Large File Storage is specifically designed to address problem with Git storing large file content (both current and historical) that can cause repos to bloat and performance to degrade. GVFS is a much more general solution that addresses large numbers of branches, large numbers of files and extended history in addition to large files. I’m not saying it is the answer to everything or that it is for everyone. It’s a solution to a set of problems for which there are currently no other solutions (for Git) that I am aware of.

Brian

2 years ago

Geoff S
Just wondering what GUI client (if any) is in use by most of the MS developers ?

2 years ago

Brian Harry MS
@Geoff S, I haven’t looked to see what the ratios are. In general, engineers at Microsoft are pretty heavily command line based for source control. A lot of people use VS though. All of the major Git GUIs have some adoption – SourceTree, GitKracken, GitHub Desktop, Tower, …

Brian

2 years ago

André Villar
I’d be super interested on an article describing the decision to use git. What other VCS where considered, what improvements would those systems need as git needed GVFS, etc.

2 years ago

cj
I’m curious. When commits are merged into their target branch, are the commits rebased onto the target branch? Or are they just plain old merged in, with a new merge commit created? At my last place of employment, they used the rebasing strategy, and that cleaned up the branch lines considerably. Otherwise if you use a tool to show commit logs, you can get thousands of vertical line showing branching activity (Which can be a real pain).

2 years ago

Brian Harry MS
@Cj. Git merging strategy is the source of endless debate. There are pros and cons to various approaches but people tend to have a very strong preference for one approach or another. We support them all and even have branch policies that allow you to enforce what the merge strategy is in a given branch. Inside Microsoft, as with the broader world, different teams have chosen different strategies. I haven’t done a survey to assess which strategy is used the most.

Brian

2 years ago

Brian Harry MS
@Andre, I talked about it some in this post: https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-git-and-some-back-story/

Brian

2 years ago

Brian_Patino
Fantastic read! I’m super excited about the future of windows. Thanks for the great article 🙂

1 year ago

Harvey Harris
I’m learning slowly, but I’m sure I really appreciate your experience! Many developers and open source projects use Git and GitHub to code version control.

2 weeks ago

Sajom
agario unblocked thank you good post.

© 2019 Microsoft Corporation.

Terms of Use Trademarks Privacy & Cookies

Increase your Linux server Internet speed with TCP BBR congestion control

Increase your Linux server Internet speed with TCP BBR congestion control

Skip to content

nixCraft
MENU
Increase your Linux server Internet speed with TCP BBR congestion control
last updated February 23, 2019 in Categories Cloud Computing
I recently read that TCP BBR has significantly increased throughput and reduced latency for connections on Google’s internal backbone networks and google.com and YouTube Web servers throughput by 4 percent on average globally – and by more than 14 percent in some countries. The TCP BBR patch needs to be applied to the Linux kernel. The first public release of BBR was here, in September 2016. The patch is available to any one to download and install. Another option is using Google Cloud Platform (GCP). GCP by default turned on to use a cutting-edge new congestion control algorithm named TCP BBR.

Requirements for Linux server Internet speed with TCP BBR
Make sure that your Linux kernel has the following option compiled as either module or inbuilt into the Linux kerne:

CONFIG_TCP_CONG_BBR
CONFIG_NET_SCH_FQ
You must use the Linux kernel version 4.9 or above. On a Debian/Ubuntu Linux type the following grep command/egrep command:
$ grep ‘CONFIG_TCP_CONG_BBR’ /boot/config-$(uname -r)
$ grep ‘CONFIG_NET_SCH_FQ’ /boot/config-$(uname -r)
$ egrep ‘CONFIG_TCP_CONG_BBR|CONFIG_NET_SCH_FQ’ /boot/config-$(uname -r)

Sample outputs:

Fig.01: Make sure that your Linux kernel has TCP BBR option setup
Fig.01: Make sure that your Linux kernel has TCP BBR option setup

I am using Linux kernel version 4.9.0-8-amd64 on a Debian and 4.18.0-15-generic on an Ubuntu server. If above options not found, you need to either compile latest kernel or install the latest version of Linux kernel using the apt-get command/apt command.
Run test before you enable TCP BBR to improve network speed on Linux
Type the following command on Linux server:
# iperf -s

How to enable TCP BBR to improve network speed on Linux server test
Execute the following on your Linux client:
$ iperf -c gcvm.backup -i 2 -t 30

How to Boost Linux Server Internet Speed with TCP BBR

How to enable TCP BBR congestion control on Linux
Edit the /etc/sysctl.conf file or create a new file in /etc/sysctl.d/ directory:
$ sudo vi /etc/sysctl.conf

OR
$ sudo vi /etc/sysctl.d/10-custom-kernel-bbr.conf

Append the following two lines:
net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr

Save and close the file i.e. exit from the vim/vi text editor by typing :x!. Next you must either reboot the Linux box or reload the changes using the sysctl command:
$ sudo reboot

OR
$ sudo sysctl –system

Sample outputs:

* Applying /etc/sysctl.d/10-console-messages.conf …
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-custom.conf …
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
* Applying /etc/sysctl.d/10-ipv6-privacy.conf …
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf …
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-link-restrictions.conf …
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/10-lxd-inotify.conf …
fs.inotify.max_user_instances = 1024
* Applying /etc/sysctl.d/10-magic-sysrq.conf …
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf …
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_syncookies = 1
* Applying /etc/sysctl.d/10-ptrace.conf …
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf …
vm.mmap_min_addr = 65536
* Applying /etc/sysctl.d/99-sysctl.conf …
* Applying /etc/sysctl.conf …
You can verify new settings with the following sysctl command. Run:
$ sysctl net.core.default_qdisc
net.core.default_qdisc = fq
$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr

Test BBR congestion control on Linux
In my testing between two long distance Linux server with Gigabit ports connected to the Internet, I was able to bump 250 Mbit/s into 800 Mbit/s. You can use tools such as the wget command to measure bandwidths speed:
$ wget https://your-server-ip/file.iso

I also noticed I was able to push almost 100 Mbit/s for my OpenVPN traffic. Previously I was able to push up to 30-40 Mbit/s only. Overall I am quite satisfied with TCP BBR congestion control option for my Linux box.

Linux TCP BBR test with iperf
The iperf is a commonly used network testing tool for TCP/UDP data streams. It measures the throughput of the network. This tool can validate the importance of Linux TCP BBR settings.

Type command on Linux server with TCP BBR congestion control enables
# iperf -s

Sample outputs:

————————————————————
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
————————————————————
[ 4] local 10.128.0.2 port 5001 connected with AAA.BB.C.DDD port 46978
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-30.6 sec 127 MBytes 34.7 Mbits/sec
Type command on Linux/Unix client
$ iperf -c YOUR-Linux-Server-IP-HERE -i 2 -t 30

Sample output when connected to TCP BBR congestion enabled on Linux:

————————————————————
Client connecting to gcp-vm-nginx-www1, TCP port 5001
TCP window size: 45.0 KByte (default)
————————————————————
[ 3] local 10.8.0.2 port 46978 connected with xx.yyy.zzz.tt port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 2.0 sec 4.00 MBytes 16.8 Mbits/sec
[ 3] 2.0- 4.0 sec 8.50 MBytes 35.7 Mbits/sec
[ 3] 4.0- 6.0 sec 10.9 MBytes 45.6 Mbits/sec
[ 3] 6.0- 8.0 sec 16.2 MBytes 68.2 Mbits/sec
[ 3] 8.0-10.0 sec 5.29 MBytes 22.2 Mbits/sec
[ 3] 10.0-12.0 sec 9.38 MBytes 39.3 Mbits/sec
[ 3] 12.0-14.0 sec 8.12 MBytes 34.1 Mbits/sec
[ 3] 14.0-16.0 sec 8.12 MBytes 34.1 Mbits/sec
[ 3] 16.0-18.0 sec 8.38 MBytes 35.1 Mbits/sec
[ 3] 18.0-20.0 sec 6.75 MBytes 28.3 Mbits/sec
[ 3] 20.0-22.0 sec 8.12 MBytes 34.1 Mbits/sec
[ 3] 22.0-24.0 sec 8.12 MBytes 34.1 Mbits/sec
[ 3] 24.0-26.0 sec 9.50 MBytes 39.8 Mbits/sec
[ 3] 26.0-28.0 sec 7.00 MBytes 29.4 Mbits/sec
[ 3] 28.0-30.0 sec 8.12 MBytes 34.1 Mbits/sec
[ 3] 0.0-30.3 sec 127 MBytes 35.0 Mbits/sec
Conclusion
Bottleneck Bandwidth and RTT (BBR) congestion control pre and post average stats collected for 30 seconds using the iperf command:

PRE BBR: Transfer: 27.5 MBytes. Bandwidth: 7.15 Mbits/sec
POST BBR: Transfer: 127 MBytes. Bandwidth: 35.0 Mbits/sec
BBR is, in my opinion, one of the most critical improvements to Linux networking stacks in recent years. This page demonstrated how to enable and set up BBR on Linux based system. For more information see the following pages:

Measure Network Performance: Find Bandwidth, Jitter, Datagram Loss With Iperf
BBR info from Google
TCP BBR congestion control comes to GCP. Your Internet just got faster
SHARE ON Facebook Twitter

Posted by: Vivek Gite
The author is the creator of nixCraft and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter.

Your support makes a big difference:
I have a small favor to ask. More people are reading the nixCraft. Many of you block advertising which is your right, and advertising revenues are not sufficient to cover my operating costs. So you can see why I need to ask for your help. The nixCraft takes a lot of my time and hard work to produce. If everyone who reads nixCraft, who likes it, helps fund it, my future would be more secure. You can donate as little as $1 to support nixCraft:

Become a Supporter Donate via Paypal

Tagged as: Tags Debian, Debian Linux, Linux, Sysadmin, Ubuntu
Start the discussion at http://www.nixcraft.com
Historical Comment Archive
Comments 12 comment
Bogdan Stoica says: July 24, 2017 at 10:22 am
I have tried that but the download/upload speed is garbage. I was able to download with about 30 MB/sec using kernel 3.x and without TCP BBR activated. Installed kernel 4.x and enabled TCP BBR but the download speed was about 1MB/sec (from the exact same source). So I have decided to remove kernel 4.x and stick to the old 3.x kernel.

TJ says: July 24, 2017 at 11:12 am
Are you using a dedicated server or cloud server/vps? May I know your distro version?

Bogdan Stoica says: July 24, 2017 at 11:16 am
Tried on a vps first, CentOS 7, gigabit connection. It might be because of the settings on the dedicated server where the vps was created (for testing purposes only). I will check more anyway.

mimi89999 says: July 22, 2017 at 4:53 pm
Is that good for a Raspberry Pi running as a home server with a 60/6 Mbit/s Internet link? Would this improve performance?

meh says: July 22, 2017 at 3:35 pm
uhm well… i’m not sure it’s actually better than cubic after reading this http://blog.cerowrt.org/post/a_bit_about_bbr/

johnp says: July 24, 2017 at 9:44 pm
As much as I like CeroWrt, they should really date their posts:
http://blog.cerowrt.org/post/bbrs_basic_beauty/

KenP says: July 22, 2017 at 6:37 am
So is this server-side only? Will it make any difference if I compiled the kernel on my desktop linux box at hone? Stupid question?

Vivek Gite says: July 22, 2017 at 1:54 pm
No. It is server side only.

KenP says: July 24, 2017 at 4:09 am
Thanks Vivek

ren says: July 26, 2017 at 4:20 pm
I know you already answered this but, I was reading cerowrt’s article above (the one posted by johnp) and saw the tcp_upload test showing stable bandwidth and more mb/s for bbr vs cubic (the default in several desktop distros)… so wouldn’t this be useful for desktops too?

Ivan Baldo says: July 22, 2017 at 12:53 am
I am using it with fq_codel instead of fq, isn’t that better?
Maybe you could try to benchmark with it to see if there is any difference.
Otherwise, nice article!

Stephen says: August 1, 2017 at 6:49 pm
https://github.com/systemd/systemd/issues/5090

Still, have a question? Get help on our forum!

Post navigation
Previous post:
Previous post:Download of The Day: Fedora Linux 26
Next post:
Next post:How to generate RSS 2.0 feed quickly using Python
nixCraft @2000-2019 nixCraft. All rights reserved.
PRIVACY
TERM OF SERVICE
CONTACT/EMAIL
DONATIONS
SEARCH

12 Concepts That Will Level Up Your JavaScript Skills

12 Concepts That Will Level Up Your JavaScript Skills

To make Medium work, we log user data and share it with processors. To use Medium, you must agree to our Privacy Policy, including cookie policy.
Homepage
Hacker Noon
HOMELATESTMEET @ GITHUB SF HQ?TOP🎧CONTRIBUTE$$$ CLOSING SOON $$$
12 Concepts That Will Level Up Your JavaScript Skills
Go to the profile of Nick Scialli
Nick Scialli
Feb 21

JavaScript is a complex language. If you’re a JavaScript developer at any level, it’s important to understand its foundational concepts. This article tackles 12 concepts that are critical for any JS developer to understand, but in no way represents the full breadth of what a JS developer should know.

Note: If you enjoy this article, please give it a clap 👏 (or 50!) to help spread the word!

I will be continuously updating this list in a Github repository called JS Tips & Tidbits. Please star ⭐ and share if you want to follow along!

1. Value vs. Reference Variable Assignment
Understanding how JavaScript assigns to variables is foundational to writing bug-free JavaScript. If you don’t understand this, you could easily write code that unintentionally changes values.

JavaScript always assigns variables by value. But this part is very important: when the assigned value is one of JavaScript’s five primitive type (i.e., Boolean, null, undefined, String, and Number) the actual value is assigned. However, when the assigned value is an Array, Function, or Object a reference to the object in memory is assigned.

Example time! In the following snippet, var2 is set as equal to var1. Since var1 is a primitive type (String), var2 is set as equal to var1’s String value and can be thought of as completely distinct from var1 at this point. Accordingly, reassigning var2 has not effect on var1.

let var1 = ‘My string’;
let var2 = var1;
var2 = ‘My new string’;
console.log(var1);
// ‘My string’
console.log(var2);
// ‘My new string’
Let’s compare this with object assignment.

let var1 = { name: ‘Jim’ }
let var2 = var1;
var2.name = ‘John’;
console.log(var1);
// { name: ‘John’ }
console.log(var2);
// { name: ‘John’ }
One might see how this could cause problems if you expected behavior like primitive assignment! This can get especially ugly if you create a function that unintentionally mutates an object.

2. Closures
Closure is an important JavaScript pattern to give private access to a variable. In this example, createGreeter returns an anonymous function that has access to the supplied greeting, “Hello.” For all future uses, sayHello will have access to this greeting!

function createGreeter(greeting) {
return function(name) {
console.log(greeting + ‘, ‘ + name);
}
}
const sayHello = createGreeter(‘Hello’);
sayHello(‘Joe’);
// Hello, Joe
In a more real-world scenario, you could envision an initial function apiConnect(apiKey) that returns some methods that would use the API key. In this case, the apiKey would just need to be provided once and never again.

function apiConnect(apiKey) {
function get(route) {
return fetch(`${route}?key=${apiKey}`);
}
function post(route, params) {
return fetch(route, {
method: ‘POST’,
body: JSON.stringify(params),
headers: {
‘Authorization’: `Bearer ${apiKey}`
}
})
}
return { get, post }
}
const api = apiConnect(‘my-secret-key’);
// No need to include the apiKey anymore
api.get(‘http://www.example.com/get-endpoint&#8217;);
api.post(‘http://www.example.com/post-endpoint&#8217;, { name: ‘Joe’ });
3. Destructuring
Don’t be thrown off by JavaScript parameter destructuring! It’s a common way to cleanly extract properties from objects.

const obj = {
name: ‘Joe’,
food: ‘cake’
}
const { name, food } = obj;
console.log(name, food);
// ‘Joe’ ‘cake’
If you want to extract properties under a different name, you can specify them using the following format.

const obj = {
name: ‘Joe’,
food: ‘cake’
}
const { name: myName, food: myFood } = obj;
console.log(myName, myFood);
// ‘Joe’ ‘cake’
In the following example, destructuring is used to cleanly pass the person object to the introduce function. In other words, destructuring can be (and often is) used directly for extracting parameters passed to a function. If you’re familiar with React, you probably have seen this before!

const person = {
name: ‘Eddie’,
age: 24
}
function introduce({ name, age }) {
console.log(`I’m ${name} and I’m ${age} years old!`);
}
console.log(introduce(person));
// “I’m Eddie and I’m 24 years old!”
4. Spread Syntax
A JavaScript concept that can throw people off but is relatively simple is the spread operator! In the following case, Math.max can’t be applied to the arr array because it doesn’t take an array as an argument, it takes the individual elements as arguments. The spread operator … is used to pull the individual elements out the array.

const arr = [4, 6, -1, 3, 10, 4];
const max = Math.max(…arr);
console.log(max);
// 10
5. Rest Syntax
Let’s talk about JavaScript rest syntax. You can use it to put any number of arguments passed to a function into an array!

function myFunc(…args) {
console.log(args[0] + args[1]);
}
myFunc(1, 2, 3, 4);
// 3
6. Array Methods
JavaScript array methods can often provide you incredible, elegant ways to perform the data transformation you need. As a contributor to StackOverflow, I frequently see questions regarding how to manipulate an array of objects in one way or another. This tends to be the perfect use case for array methods.

I will cover a number of different array methods here, organized by similar methods that sometimes get conflated. This list is in no way comprehensive: I encourage you to review and practice all of them discussed on MDN (my favorite JavaScript reference).

map, filter, reduce
There is some confusion around the JavaScript array methods map, filter, reduce. These are helpful methods for transforming an array or returning an aggregate value.

map: return array where each element is transformed as specified by the function
const arr = [1, 2, 3, 4, 5, 6];
const mapped = arr.map(el => el + 20);
console.log(mapped);
// [21, 22, 23, 24, 25, 26]
filter: return array of elements where the function returns true
const arr = [1, 2, 3, 4, 5, 6];
const filtered = arr.filter(el => el === 2 || el === 4);
console.log(filtered);
// [2, 4]
reduce: accumulate values as specified in function
const arr = [1, 2, 3, 4, 5, 6];
const reduced = arr.reduce((total, current) => total + current);
console.log(reduced);
// 21
find, findIndex, indexOf
The array methods find, findIndex, and indexOf can often be conflated. Use them as follows.

find: return the first instance that matches the specified criteria. Does not progress to find any other matching instances.
const arr = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
const found = arr.find(el => el > 5);
console.log(found);
// 6
Again, note that while everything after 5 meets the criteria, only the first matching element is returned. This is actually super helpful in situations where you would normally break a for loop when you find a match!

findIndex: This works almost identically to find, but rather than returning the first matching element it returns the index of the first matching element. Take the following example, which uses names instead of numbers for clarity.
const arr = [‘Nick’, ‘Frank’, ‘Joe’, ‘Frank’];
const foundIndex = arr.findIndex(el => el === ‘Frank’);
console.log(foundIndex);
// 1
indexOf: Works almost identically to findIndex, but instead of taking a function as an argument it takes a simple value. You can use this when you have simpler logic and don’t need to use a function to check whether there is a match.
const arr = [‘Nick’, ‘Frank’, ‘Joe’, ‘Frank’];
const foundIndex = arr.indexOf(‘Frank’);
console.log(foundIndex);
// 1
push, pop, shift, unshift
There are a lot of great array method to help add or remove elements from arrays in a targeted fashion.

push: This is a relatively simple method that adds an item to the end of an array. It modifies the array in-place and the function itself returns the item added to the array.
let arr = [1, 2, 3, 4];
const pushed = arr.push(5);
console.log(arr);
// [1, 2, 3, 4, 5]
console.log(pushed);
// 5
pop: This removes the last item from an array. Again, it modifies the array in place. The function itself returns the item removed from the array.
let arr = [1, 2, 3, 4];
const popped = arr.pop();
console.log(arr);
// [1, 2, 3]
console.log(popped);
// 4
shift: This removes the first item from an array. Again, it modifies the array in place. The function itself returns the item removed from the array.
let arr = [1, 2, 3, 4];
const shifted = arr.shift();
console.log(arr);
// [2, 3, 4]
console.log(shifted);
// 1
unshift: This adds one or more elements to the beginning of an array. Again, it modifies the array in place. Unlike a lot of the other methods, the function itself returns the new length of the array.
let arr = [1, 2, 3, 4];
const unshifted = arr.unshift(5, 6, 7);
console.log(arr);
// [5, 6, 7, 1, 2, 3, 4]
console.log(unshifted);
// 7
splice, slice
These methods either modify or return subsets of arrays.

splice: Change the contents of an array by removing or replacing existing elements and/or adding new elements. This method modifies the array in place.
The following code sample can be read as: at position 1 of the array, remove 0 elements and insert b.

let arr = [‘a’, ‘c’, ‘d’, ‘e’];
arr.splice(1, 0, ‘b’)
slice: returns a shallow copy of an array from a specified start position and before a specified end position. If no end position is specified, the rest of the array is returned. Importantly, this method does not modify the array in place but rather returns the desired subset.
let arr = [‘a’, ‘b’, ‘c’, ‘d’, ‘e’];
const sliced = arr.slice(2, 4);
console.log(sliced);
// [‘c’, ‘d’]
console.log(arr);
// [‘a’, ‘b’, ‘c’, ‘d’, ‘e’]
sort

sort: sorts an array based on the provided function which takes a first element and second element argument. Modifies the array in place. If the function returns negative or 0, the order remains unchanged. If positive, the element order is switched.
let arr = [1, 7, 3, -1, 5, 7, 2];
const sorter = (firstEl, secondEl) => firstEl – secondEl;
arr.sort(sorter);
console.log(arr);
// [-1, 1, 2, 3, 5, 7, 7]
Phew, did you catch all of that? Neither did I. In fact, I had to reference the MDN docs a lot while writing this — and that’s okay! Just knowing what kind of methods are out there with get you 95% of the way there.

7. Generators
Don’t fear the *. The generator function specifies what value is yielded next time next() is called. Can either have a finite number of yields, after which next() returns an undefined value, or an infinite number of values using a loop.

function* greeter() {
yield ‘Hi’;
yield ‘How are you?’;
yield ‘Bye’;
}
const greet = greeter();
console.log(greet.next().value);
// ‘Hi’
console.log(greet.next().value);
// ‘How are you?’
console.log(greet.next().value);
// ‘Bye’
console.log(greet.next().value);
// undefined

And using a generator for infinite values:

function* idCreator() {
let i = 0;
while (true)
yield i++;
}
const ids = idCreator();
console.log(ids.next().value);
// 0
console.log(ids.next().value);
// 1
console.log(ids.next().value);
// 2
// etc…
8. Identity Operator (===) vs. Equality Operator (==)
Be sure to know the difference between the identify operator (===) and equality operator (==) in JavaScript! The ==operator will do type conversion prior to comparing values whereas the === operator will not do any type conversion before comparing.

console.log(0 == ‘0’);
// true
console.log(0 === ‘0’);
// false
9. Object Comparison
A mistake I see JavaScript newcomers make is directly comparing objects. Variables are pointing to references to the objects in memory, not the objects themselves! One method to actually compare them is converting the objects to JSON strings. This has a drawback though: object property order is not guaranteed! A safer way to compare objects is to pull in a library that specializes in deep object comparison (e.g., lodash’s isEqual).

The following objects appear equal but they are in fact pointing to different references.

const joe1 = { name: ‘Joe’ };
const joe2 = { name: ‘Joe’ };
console.log(joe1 === joe2);
// false
Conversely, the following evaluates as true because one object is set equal to the other object and are therefore pointing to the same reference (there is only one object in memory).

const joe1 = { name: ‘Joe’ };
const joe2 = joe1;
console.log(joe1 === joe2);
// true
Make sure to review the Value vs. Reference section above to fully understand the ramifications of setting a variable equal to another variable that’s pointing to a reference to an object in memory!

10. Callback Functions
Far too many people are intimidated by JavaScript callback functions! They are simple, take this example. The console.log function is being passed as a callback to myFunc. It gets executed when setTimeout completes. That’s all there is to it!

function myFunc(text, callback) {
setTimeout(function() {
callback(text);
}, 2000);
}
myFunc(‘Hello world!’, console.log);
// ‘Hello world!’
11. Promises
Once you understand JavaScript callbacks you’ll soon find yourself in nested “callback hell.” This is where Promises help! Wrap your async logic in a Promise and resolve on success or reject on fail. Use “then” to handle success and catch to handle failure.

const myPromise = new Promise(function(res, rej) {
setTimeout(function(){
if (Math.random() {
setTimeout(() => res(‘Hello world!’), 2000);
})
async function myFunc() {
const greeting = await greeter;
console.log(greeting);
}
myFunc();
// ‘Hello world!’
Conclusion
If you didn’t know any of these 12 concepts, you likely have grown at least a little in your knowledge of JavaScript! And if you knew them all, then hopefully this was a chance to practice and grow your knowledge. What other concepts do you think are critical? Let me know in the comments.

JavaScriptProgrammingFront End DevelopmentNodejsSoftware Development
Go to the profile of Nick Scialli
Nick Scialli
Husband, dog dad, software engineer, and data science enthusiast. Powered by ☕️. http://www.twitter.com/nas5w http://github.com/nas5w

Hacker Noon
Hacker Noon
how hackers start their afternoons.

More from Hacker Noon
A hacker intercepted your WiFi traffic, stole your contacts, passwords, & financial data.
Go to the profile of Patrick F. Wilbur
Patrick F. Wilbur
Feb 19
More from Hacker Noon
Four Startup Engineering Killers
Go to the profile of Nemil Dalal
Nemil Dalal
Feb 19
More from Hacker Noon
Why Doesn’t Anyone in Crypto Think About Distribution?
Go to the profile of Asheesh Birla
Asheesh Birla
Feb 13
Responses
Conversation between Eric Boyle and Nick Scialli.
Go to the profile of Eric Boyle
Eric Boyle
Feb 21
console.log(sliced); // [‘b’, ‘c’]
This would actually output [‘c’, ‘d’] because you’re starting at index 2 which is ‘c’ not ‘b’.

I know it is minor. I was just excited to catch something as someone who has been focusing on teaching myself JavaScript.

Go to the profile of Nick Scialli
Nick Scialli
Feb 21
Fixed… and it’s not minor at all! You probably prevented some people from a lot of confusion so thank you!

Conversation between Jesús Molina and Nick Scialli.
Go to the profile of Jesús Molina
Jesús Molina
Feb 21
Hey nice article. I’m learning JS, and I’m confused with the closure thing.
So, when you write sayHello(‘Joe’), would that be the same as createGreeter(‘Hello’,’Joe’);?
Go to the profile of Nick Scialli
Nick Scialli
Feb 21
Thanks for reading Jesús!

The createGreeter function takes only one argument and then returns a function that takes one argument, so it would actually be the same as writing this: createGreeter(‘Hello’)(‘Joe’)

Applause from Nick Scialli (author)
Go to the profile of Justin Meyer
Justin Meyer
Feb 21
Math.max(…arr) -> Math.max.apply(null, arr)
For old school folks.

Conversation between Travis Lawrence and Nick Scialli.
Go to the profile of Travis Lawrence
Travis Lawrence
Feb 21
You can pass parameters to the setTimeout callback by adding those parameters to thesetTimeout 3rd
I know the point of that example is ‘callbacks’ and not ‘setTimeout’ but just wanted to share!
Go to the profile of Nick Scialli
Nick Scialli
Feb 21
Thanks Travis! I’m debating making this change but not sure if it’s more or less clear than it currently is.

All the networks. Found by Everyone.

All the networks. Found by Everyone.

home
View
Uploads
Info
Stats
Tools
Login
Frequently Asked Questions
What is This Thing For?

We consolidate location and information of wireless networks world-wide to a central database, and have user-friendly desktop and web applications that can map, query and update the database via the web.

We currently accept files in any of:

DStumbler: text output
G-Mon: .kml, .txt output
inSSIDer: kml output
Kismac: native (.kismac), text, kml output
Kismet: .csv, .xml, .netxml, .gps, .gpsxml, .nettxt, CWGD output
MacStumbler: plist xml, wiscan format
NetStumbler: native (.ns1), text, wiscan, summary
Pocket Warrior: Text output
Wardrive-Android: kml output
WiFiFoFum: kml, kmz output
WiFi-Where: ns1, kml, csv output. Now available on Cydia (iphone/ipad jailbreak)
Wigle Wifi Wardriving: csv output
Consolidated.db: This is an sqlite file that is synced from an iphone/ipad to a host computer (prior to iOS 4.3.3). OSX details. Windows details.
To have your record removed from our database, or if you have any questions or suggestions, send an email to: WiGLE-admin[at]WiGLE.net or irc chat on #wigle at wigle.net:6667. If you want a record removed from the database, please include the BSSID (Mac Address) of the network in question!

Who are you people?

arkasha focuses on front-end stuff as well as serving as cruise activities directory
bobzilla is the guy who designed the DB, makes everything work, wrote the Android client, and continues to keep this project going
uhtu fixes all the things that arkasha and bobzilla do wrong
thuddwhir wrote the vector-based map generation (SquiGLE)
wos wrote the Mac OSX native client (TinGLE)
All are members of Mimezine.

How does triangulation work?

The “triangulation” is actually just an average of the latitudes and longitudes gathered using the signal strength (squared) as a weight. This is probably more correctly called “weighted-centroid trilateration.” This assumes that signal strength will change at the inverse square of the distance. This is reasonable as long is you don’t get a one-sided view of the network (i.e. only sample it from one side) since it will be skewed in that direction.

How do I Get Started?

WiGLE.net is a submission-based catalog of wireless networks. Submissions are not paired with actual people; rather name/password identities which people use to associate their data. It’s basically a “gee isn’t this neat” engine for learning about the spread of wireless computer usage.

WiGLE concerns itself with 802.11a/b/g/n and cellular networks right now, which can be collected via the WiGLE WiFi Wardriving tool on android. We also have a bluetooth stumbling client for Android, but we do not maintain a catalog of bluetooth networks.

The first step in using WiGLE is to create a username for yourself. You don’t have to submit anything other than a made-up email, username and password, validation is immediate, and we will not contact you (unless you wanna chat on our message boards). This will give you access to our query engine and software downloads.

Once you’ve signed up, you’re free to send us wireless network traces (in any of our listed formats, usually pairings of wireless sample, names and network hardware addresses (for uniqueness), data/SNR triples and GPS coordinates) or enter networks manually. Note, your username gets “credit” for these, but of course some people don’t want their networks listed (various reasons), so we delist these immediately upon request. Once you make a User Name, you can look at the submissions statistics page to see how users stack up.

The easiest way to start stumbling is to use our Android tools, available at:
Android app on Google Play or (or search for “wiglewifi” on an Android device)

To view networks, you can 1.) ask the website, 2.) download the java client. The clients are particularly fun to look at, but require either a java-1.6-and-up machine (windows, sunOS, MacOS-X, linux, mostly) or a windows box, for the new windows-native prototype. This will superimpose “points” from a live query onto a map of an area. Maps can be downloaded in “packs” from our mapping engine and are installed simply by unzipping them into your client installation directory. Mappacks are created and served by-state-by-county, or in the case of large cities, by-state-by-city. If we haven’t generated a map for your area of interest yet, ask for it, and come back after the rendering engine’s had a minute or 5 to think about it.

If your network is in WiGLE and you don’t like it, we’ll take it out immediately, but you should look into making your network harder to detect AND more secure; remember that you’re the one bombarding passers-by with your signal. We aren’t affiliated directly with any particular community or interest (other than our own), but we applaud the efforts of the people who wrote the stumbling software that feeds our project, the people looking to use wireless in innovative ways, and especially the community of people who just dig wireless network access and dig sharing it. (freenets)

What has this project been used for?

Overall, WiGLE aims to show people about wireless in a more-technical capacity then your average static map or newspaper article.

Educating the Public: When WiGLE started in 2001 the default for wifi was to have no encryption at all. An end user had a choice to turn on weak encryption (40-bit WEP), or pay extra for slightly less weak encryption (128-bit WEP). Most users didn’t realize that their private networks could be accessed by anyone in the area (at various distances with antennas and amplifiers). WiGLE, and Wardriving in general, helped to educate users and put pressure on manufacturers to make network security better and easier. Now a days WPA2 is the standard and defaults to “on” for most manufactured devices.

Research Projects: We are often contacted by graduate students and professors to provide additional access or slices of data for research projects around wifi and security, which we attempt to help in any way we can. We’ve heard of research into subjects like WEP/WPA encryption distribution, information leaks from SSID probes, teaching wireless security concepts, geolocation from BSSID’s and more.

Site Surveys: Security professionals use WiGLE to monitor what businesses they consult for are broadcasting over wifi radios. If you have an insecure network you are vulnerable, whether or not it appears in WiGLE. But WiGLE could let you know that an employee or attacker has set up a rogue wifi router on your private network.

Journalism: WiGLE is a handy resource for journalists looking for data points on Wifi proliferation, statistics and security. WiGLE keeps statistics on manufacturer distribution, top SSID’s, encryption modes, channel selection, geographic distribution and much more.

Washington Post
SF Gate
CNET
Wardriving: Drive, Detect, Defend (book)
Finding Useable Networks: Using the “Free Net” search WiGLE users can find legal-to-use open networks near to their current location. They can also find local government open wireless networks, a cable company wireless offering that they are already paying for, or a local coffee shop that has a network available for use.

A Fun Hobby: Many users enjoy trying to detect as many networks as possible, to compete on the leaderboard and see parts of their city/town that they’ve had no reason to visit before. Teams can compete to see which car can find the most networks in a set amount of time.

What’s the License for All This?

The official legal End User License Agreement. We basically just offer no warranties on our software, and don’t want our software to be used for unlicensed commercial gain.

What does that mean for academic/non-profit research or artistic usage?

We believe that one of the best uses for this project is to promote education, research, and awareness! Please email us at WiGLE-admin[at]WiGLE.net, include an explanation of your intended usage, needs, and a registered WiGLE username.

Commercial Use

To help fund WiGLE operations, we offer licenses to a subset of the data derived from postings that our users have permitted us to use for commercial purposes. As a matter of policy, we will not comment on future, past, or hypothetical customers. To discuss a commercial license, please email us at WiGLE-admin[at]WiGLE.net

We additionally refuse to claim or disclaim any involvement with the alleged moon landings, area 51 and ‘bigfoot’.

Why Don’t You Offer Non US Maps for the Java Client?

We’d love to offer world-wide, street-level mapping information, but there’s nothing free for our use. WiGLE relies on the TIGER and VMAP0 mapping data sets for our current maps, which allows us to produce our maps without intellectual property entanglement or licensing fees. We’re certainly interested in hearing from you about candidate datasets which we can use under similar terms, but we’re not necessarily going to write a new parser for each country/city/neighborhood in the world. See The forums for an up-to-date status on our search and instructions for composing your own raster mappacks. We’d urge you to consider contributing to Open Street Map, and if anyone has a tutorial on how to build mappacks from OSM tiles, let us know!

How Can I Change My Password?

If you’ve forgotten your password, we recommend you contact us via WiGLE-admin[at]WiGLE.net and include your registration information for proof of identity or you can use our Reset Password Tool.

SOCIAL
WIKI
TWITTER
IRC
FACEBOOK
SITE INFORMATION
FAQ
END-USER AGREEMENT
PRIVACY
OUR TODO LIST
/DEV/RANDOM
CAFEPRESS GEAR
LINKS
USER MANAGEMENT
RESET PASSWORD
REGISTER
NEWS
FORUMS
NEWS RSS
STATS RSS
WiGLE respects your privacy. To have records of your access point removed from our database, or if you have any questions or suggestions, send an email to: WiGLE-admin[at]WiGLE.net (please include BSSID (MAC) in removal requests). We’re also on IRC: at WiGLE.net:6667

Copyright 2001-2019 bobzilla && arkasha && uhtu

Christian Haschek’s blog

Christian Haschek’s blog

HOME
ABOUT ME
PUBLICATIONS
THIS BLOG IN THE MEDIA
DRAFTS
Futurelopment
a blog about security projects teaching programming and other stuff

SORT BY DATE SORT BY VIEWS

I scanned the whole country of Austria and this is what I’ve found
IP cameras, printers, industrial controls to name a few..
73,559February 08th 2019
The curious case of the Raspberry Pi in the network closet
how we found, analyzed (with the help of Reddit) and in the end caught the culprit of a malicious device in our network
299,532January 16th 2019
Build your own datacenter with PXE and Alpine
Who needs docker when you can run old laptops as blade servers
5,858January 13th 2019
Teaching in minecraft
logic gates have never looked so good
1,377January 10th 2019
Making a smart meter out of a dumb one for 4 $
and displaying it on an awesome dashboard
2,905October 19th 2018
So you have been scammed
a small guide on how to strike back and be a bad scamee
2,273August 31st 2018
How to fight child pornography with a Raspberry Pi and deep learning
Finding illegal content with just 10 Watts of power
7,519June 17th 2018
Reverse engineering your mobile banking app
to make a PHP script that automates checking your balances
5,312April 05th 2018
Raspberry Pi + Deep Learning home security system
From start to finish
6,542March 27th 2018
Raspberry Pi controlled cactus lamp
adding power to a cheap lamp
1,562January 03rd 2018
Teaching your fish tank LEDs new tricks
for just a few bucks and you can even make some virtual clouds
1,168September 09th 2017
API Heaven ICO
finally an ICO for an existing product
1,571August 16th 2017
Making an awesome dashboard for your crypto currencies in 3 steps
using InfluxDB, Grafana and PHP
8,824August 03rd 2017
How to defend your website with ZIP bombs
the good old methods still work today
128,170July 05th 2017
Creating the safest ETH paper wallet with a Raspberry Pi
with a printer and without an internet connection
8,382June 22nd 2017
Accepting Ethereum payments without external services
You don’t need to pay for merchant systems
8,916June 09th 2017
DIY vertical hydroponic system
grow vegetables in small spaces
4,204April 23rd 2017
Making a simple Raspberry Pi Bitcoin/Ethereum trading bot
with rocket.chat and Slack status reporting
25,024April 17th 2017
Is Guetzli useful in production?
It’s much better than google promised – 51% smaller images
2,121March 24th 2017
Finding bombs in the forest
treasure hunters make it easy to find them
1,805October 25th 2016
How a scammer stole 500$ from me and in the end begged me not to tell his parents
Thank god not all scammers are professionals
175,467September 08th 2016
Forget Pay2Win – the time has come for Learn2Win
how to reward players in any game for academic achievements
4,308June 04th 2016
My door sends me chat messages
Raspberry Pi powered door sensor and Rocket.Chat bot with some neat info
21,882April 24th 2016
DIY cloud lamp for fun and profit
ok, just for fun.. no profit involved
5,197February 17th 2016
Letting students mod school computers
with some amazing results
2,418January 22nd 2016
How to visualize RADIUS connections
by parsing Windows Server log files
8,242January 15th 2016
Raspberry Pi controlled power sockets
it’s as easy as it sounds
5,858December 27th 2015
My company just turned 10 and it was quite a ride
also: about the time Thomas Schranz and I almost invented Dropbox
3,707November 14th 2015
That (not so) awesome time the police raided my home
lesson learned: Don’t click on links random people send you
30,733November 08th 2015
Water cooled NAS made from spare parts
with Macguyverish solutions for common problems
4,488September 21st 2015
DIY: Cheap wall mounted, water cooled PC for 51$
with LED backlight and and all the good stuff
33,025September 10th 2015
Let’s analyze over twenty thousand proxies
It won’t get much better
4,821July 04th 2015
Analyzing 443 free proxies – Only 21% are not shady
what about the other 79%?
33,549June 21st 2015
What would happen if you’d embed a remote JPEG with a HTTP link but the remote server is forcing HTTPS?
1,745June 20th 2015
Neue Reifeprüfung analyzed
The simple way to get a good grade
1,498June 04th 2015
Repairing a closed loop CPU water cooler with aquarium tubing
air bubbles… those f***ing air bubbles
5,117March 29th 2015
Raspberry Pi 2 wireless temperature box
The π² + wifi + 1wire sensor = pure awesomeness
4,090March 16th 2015
Chromebooks for Work & Classrooms
are they worth it?
2,986January 02nd 2015
Announcing: Socialcube LITE
My proof-of-concept evolved to a intuitive platform which is free for every educational institution
1,645December 28th 2014
Why hackits are the first thing I teach new classes
they represent everything I love about computer science
3,919December 09th 2014
My XP-based grading system
a modern approach to grading in the year 2014
21,261December 06th 2014
Raspberry Pi controlled LEDs
to visualize the up-state of my servers
13,018August 22nd 2014
IKEA Server Rack
nope, not a lack rack
6,643July 24th 2014
DIY hydroponic pot for ~36€
Easy to build and awesome for your plants
4,525May 06th 2014
Temperature visualization
from 6 months of data
2,421February 11th 2014
Raspberry Pi temperature monitor
with a nice graph and live updates
4,836July 28th 2013
Why are free proxies free?
because it’s an easy way to infect thousands of users and collect their data
178,879May 29th 2013
My door sends me emails
Simple and cheap Raspberry Pi burglar alarm
11,752February 26th 2013
HTML5 heat spreading simulation
with a simple JS algorithm
1,876April 08th 2012
hackthissite.org ascii code picture challenge
2,135January 09th 2012
PHP script that filters moods from twitter
..how depressed is the web?
1,363January 08th 2012

Copyright © Christian Haschek 2019
christian@haschek.at pgp

Blogheim.at Logo

A space error: $370 million for an integer overflow

A space error: $370 million for an integer overflow

How Not To Code
A space error: $370 million for an integer overflow
hownot2code
2 years ago
Start. 37 seconds of flight. KaBOOM! 10 years and 7 billion dollars are turning into dust.

image1

The programmers were to blame for everything.

Four satellites, 2,600 lb, of the Cluster scientific program (study of the solar radiation and Earth’s magnetic field interaction) and a heavy-lift launch vehicle Ariane 5 turned into “confetti” June 4, 1996.

The previous model-rocket Ariane 4 has been successfully launched more than 100 times. What could go wrong?

Apparently, to conquer space, one should know Ada language well.

Dossier
Ariane 5 is a European expendable heavy lift launch vehicle that is part of the Ariane rocket family. It is used to deliver payloads into geostationary transfer orbit (GTO) or low Earth orbit (LEO), can launch two-three satellites, and up to eight micro satellites at a time.

The project history

It was created in 1984-1995 by a European Space Agency (EKA, ESA), the main developer – French Centre National d’Etudes Spatiales (CNES). The program participants’ were 10 European countries, the project cost was 7 billion US dollars (46.2% – contribution of France).

About a thousand industrial firms took part in the creation of the rocket. The prime contractor is a European company, Airbus Defence and Space (Airbus Group unit, “Airbus Group”, Paris). The marketing for Ariane 5 was done by a French company, Arianespace (Evry), with which ESA signed an agreement November 25, 1997.

image2

Vehicle description

Ariane 5 is a two-stage heavy class booster rocket. Length — 52-53 m, maximum diameter — 5.4 m, starting weight: 775-780 tonnes (depending on the configuration).

The first stage is equipped with a liquid rocket engine Vulcain 2 (“Volcano-2”; the first three versions of the missile were made of Vulcain), and the second is HM7B (for the version of Ariane 5 ECA) or Aestus (for Ariane 5 ES). Vulcain 2 and HM7B engines run on a mixture of hydrogen and oxygen, and are manufactured by a French company Snecma (a part of “Safran” group, Paris)

Aestus uses non volatile fuel – a mixture of the MMH propellants with Nitrogen tetroxide oxidizer. The engine was developed by a German company Daimler Chrysler Aerospace AG (DASA, Munich).

In addition, attached to the sides were two solid rocket booster accelerators (manufacturer-Europropulsion, Suresnes, France; a joint venture between Safran Group and the Italian company Avio), which provide more than 90% of torque starting at the beginning, delivering 90% of the thrust during the first launch phases. In the version of the Ariane 5 ES, the second stage may not be available when outputting the payloads into low anchor orbit.

On-board computers
http://www.ruag.com/space/products/digital-electronics-for-satellites-launchers/on-board-computers

Investigation

The day after the catastrophe, the General Director of the European Space Agency (ESA), and Chairman of the French National Centre for space research (CNES) issued a decree on the formation of an independent Commission to investigate the circumstances and causes of this emergency, which included well-known experts and scholars from all interested European countries.

The Commission began its work on June 13, 1996 and on 19 July they released its exhaustive report (PDF), which immediately became available on the net.

The Commission had telemetry data, trajectory data, as well as recorded optical observations of the course of the flight.

The explosion occurred at an altitude of approximately 4 km, and the debris was scattered over an area of about 12 square km in the savanna and the surrounding swamps. The Comission studied the testimonies of numerous specialists and examined the production and operational documentation.

Technical details of the accident
The position and orientation of the booster in space were measured by an Inertial Reference Systems — IRS, a part of which is a built-in computer, which evaluates the angles and speeds based on the information provided by the onboard Inertial Platform, equipped with laser gyroscopes and accelerometers. The data from IRS were passed by a special bus for the onboard computer, which provided the necessary information for the implementation of the flight program and managed directly – through the hydraulic and servo mechanism – the solid booster accelerators and cryogenic engines.

Duplication of the equipment was used to ensure the reliability of Flight Control Systems. Therefore, two IRS systems (one – active and the other is its hot standby) with identical hardware and software were operating in parallel. As soon as the onboard computer detected that the “active” IRS withdrew from a regular mode, it immediately switched to another. There were also two on-board computers.

Significant phases of development process

7 minutes before the scheduled launch there was detected an infringement of “visibility criterion”. Therefore, the start was postponed by an hour.

LT (Launch Time) = 9 o’clock. At 33 min. 59 sec. local time, the “launch window” was “caught” again and finally, the vehicle launched and was running in a normal mode until LT+37 seconds.

In the following several seconds there was a dramatic deviation from the given missile trajectory that ended in an explosion.

At LT+39 seconds, because of high aerodynamic load due to the “angle of attack” exceeding 20 degrees, the starting accelerators separated from its main stage, which triggered the missile Autodestruct System.

The change of the angle of attack happened because of a malfunction in the nozzle rotation of the solid accelerators, which was caused by a command from an on-board computer based on the information from the active Navigation System (IRS 2).

Some of this information was incorrect in principle: what has been interpreted as flight details was actually diagnostic information from the IRS 2 firmware.

The built-in computer IRS2 passed incorrect data, because it diagnosed a contingency, having “caught” an exception that was thrown by one of the software modules.

At the same time the on-board computer could not switch to the backup system IRS 1 because it had already ceased to function during the previous cycle (which took 72 milliseconds) – for the same reason as the IRS 2.

An exception “thrown” by an IRS program, resulted from the conversion of data from a 64-bit floating point format to a 16-bit signed integer, which led to “Operand Error”.

The error occurred in a component that is meant only for performing “adjustment” of the Inertial Platform. This software module generates significant results only until the moment LT+7 seconds of the detachment from the launch pad. After the rocket soared up, the module could no longer affect the module.

“The adjustment function” had to be active (according to the established results) for 50 seconds after the initiation of the “flight mode” on the Navigation System bus (the moment LT-3 seconds), was performed.

The “Operand Error” occurred because of an unexpectedly large magnitude of BH (Horizontal Bias — a horizontal skew), evaluated by the internal function based on the value of “horizontal speed” measured by the Platform sensors.

The BH magnitude served as an indicator of the precision of the Platform positioning. The BH magnitude turned out to be much greater than it was expected, because the trajectory of the Ariane 5 at the early stage was significantly different from the flight path of the Ariane 4 (where this software module was previously used), which led to a much higher “horizontal velocity”.

The final action that had fatal consequences was the processor work termination. Thus, the whole Navigation System ceased to function. It was technically impossible to resume its actions.

The researchers were able to reproduce this chain of events using computer modeling, combined with other research materials and experiments this allowed them to conclude that the causes and the circumstances of the accident are fully identified.

The causes and origins of the accident
The initial requirement to continue the adjustment after the rocket takeoff, was embedded for more than 10 years before the fateful events, when they designed the early Ariane models.

The flight could be cancelled just several seconds before the flight, for example, in the interval of LT-9, for example, when the IRS started the “flight mode”, and LT-5 seconds, when there was a command to perform several operations with the rocket equipment.

In the case of an unexpected cancellation of the takeoff, it was necessary to quickly return to the countdown mode – and not to repeat all the installation operations from the beginning, including the bringing of the Inertial Platform (an operation, requiring 45 min. – the time when the “launch window” would be lost).

It was stated that in case the launch was cancelled, 50 seconds after the LT-9 would be enough for the equipment on the Earth to regain full control over the Inertial Platform without data loss – the Platform could stop the transference that was initiated and the corresponding software module would register all the information about its condition, which will help to return to the original position (in case the rocket is still on the launch pad). Once, in 1989, during start number 33 of the Ariane 4 rocket, this peculiarity was successfully activated.

image14

However, the Ariane 5, in contrast to the previous model had a fundamentally different scenario of pre-flight actions — so different that the work of the fateful software module after the launch time made no sense at all. However, the module was used again without any modifications.

ADA Language

image15

The investigation revealed that this software module contained seven variables involved in type conversion operations. It turned out that the developers performed the analysis for the vulnerability of all operations, capable of throwing an exception.

It was their conscious action – to add adequate protection to four variables, and leave three of them – including BH – unprotected. The ground for this decision was the certainty that overflow is not possible in these variables in general.

This confidence was supported by the evaluations, showing that the expected range of physical parameters that was taken as the basis for the determination of the values of the mentioned variables can never lead to an undesirable situation. And it was true — but for the trajectory evaluated for Ariane 4.

The new generation Ariane 5 rocket launched on an entirely different trajectory, for which no evaluations were carried out. Meanwhile, it turned out that the “horizontal velocity” (together with the initial acceleration) exceeded the estimated (for Ariane 4) more than five times.

The protection of all 7 (including BH) variables wasn’t provided because the maximum workload for the IRS computer was declared as 80%. The developers had to look for ways to reduce unnecessary evaluation expenses, and they weakened the protection in that fragment where theoretically the accident could not happen. When it occurred, then the exception handling mechanism was activated, which turned out to be completely inadequate.

This mechanism supposes three main steps.

The information about the contingency should be transmitted via the bus to the onboard computer OBC.
In parallel it was written – together with the whole context – to the reprogramming memory EEPROM (during the investigation it was possible to restore it and read the contents)
The work of IRS processor should have been aborted.
The last action was a fatal one; it led to the accident despite the fact that the situation was quite normal (even though there was an exception generated due to unsecured overflow).

image17

Conclusion
The defect on the Ariane 5was the result of several factors. There were many stages during development and testing when the defect could have been detected.

The programming module was reused in a new environment where the conditions of functioning were significantly different from the requirements of the program module. These requirements have not been revised.
The system identified and detected an error. Unfortunately, the specification of the error-handling mechanism was inappropriate and caused the final destruction.
The erroneous module was never properly tested in the new environment – neither the hardware, nor the level of system integration. Therefore, the flaws in the development and implementation were not detected.
image19

From the report of the commission:

The main task during the development of Ariane 5 was the reducing of the occasional accident. The exception thrown was not a random accident, but an error in the structure. The exception was detected, but handled incorrectly, because of the point of view that a program should be considered correct, until the opposite is shown. The Commission holds the opposite view, that the software should be considered erroneous, until the best practical current methods demonstrate its correctness.

Happy ending
image21

Despite this failure, there were 4 more satellites, Cluster II built and put into orbit on the rocket Soyuz-U/Fregat in the year 2000.

This accident attracted the attention of the public, politicians, and the heads of organizations to the high risks connected with the usage of complex computational systems, which increased investment into research aimed at improving the reliability of life-critical systems. The following automatic analysis of the Ariane code (written in Ada) was the first case when the static analysis was used in the scope of a large project using the abstract interpretation technique.

Sources
Report Ariane 501 — Presentation of Inquiry Board report
Telles, Matt The Science of Debugging
Class 25: Software Disasters
Ariane 5 – Chronicle of a failure
ARIANE 5 — The Software Reliability Verification Process
Safety in Software — now more important than ever
Static Analysis and Verification of Aerospace Software by Abstract Interpretation
ADA source code
This article was originally published (in Russian) at the website habrahabr.ru. The article was translated and published at our blog with the author’s permission.

Categories: Bugs in C/C++ projects
Tags: bugs, coding, cpp, programming, space
Leave a Comment
How Not To Code
Blog at WordPress.com.

Back to top

Special Report – Inside the UAE’s secret hacking team of U.S. mercenaries

Special Report – Inside the UAE’s secret hacking team of U.S. mercenaries

Directory of sites
Login
Contact
Support
World
Business
Markets
Politics
TV

Search…
SPECIAL REPORTSJANUARY 30, 2019 / 12:24 PM / 8 DAYS AGO
Special Report – Inside the UAE’s secret hacking team of U.S. mercenaries
Christopher Bing, Joel Schectman
28 MIN READ

WASHINGTON (Reuters) – Two weeks after leaving her position as an intelligence analyst for the U.S. National Security Agency in 2014, Lori Stroud was in the Middle East working as a hacker for an Arab monarchy.

She had joined Project Raven, a clandestine team that included more than a dozen former U.S. intelligence operatives recruited to help the United Arab Emirates engage in surveillance of other governments, militants and human rights activists critical of the monarchy.

Stroud and her team, working from a converted mansion in Abu Dhabi known internally as “the Villa,” would use methods learnt from a decade in the U.S intelligence community to help the UAE hack into the phones and computers of its enemies.

Stroud had been recruited by a Maryland cybersecurity contractor to help the Emiratis launch hacking operations, and for three years, she thrived in the job. But in 2016, the Emiratis moved Project Raven to a UAE cybersecurity firm named DarkMatter. Before long, Stroud and other Americans involved in the effort say they saw the mission cross a red line: targeting fellow Americans for surveillance.

“I am working for a foreign intelligence agency who is targeting U.S. persons,” she told Reuters. “I am officially the bad kind of spy.”

The story of Project Raven reveals how former U.S. government hackers have employed state-of-the-art cyber-espionage tools on behalf of a foreign intelligence service that spies on human rights activists, journalists and political rivals.

Interviews with nine former Raven operatives, along with a review of thousands of pages of project documents and emails, show that surveillance techniques taught by the NSA were central to the UAE’s efforts to monitor opponents. The sources interviewed by Reuters were not Emirati citizens.

The operatives utilized an arsenal of cyber tools, including a cutting-edge espionage platform known as Karma, in which Raven operatives say they hacked into the iPhones of hundreds of activists, political leaders and suspected terrorists. Details of the Karma hack were described in a separate Reuters article today.

An NSA spokesman declined to comment on Raven. An Apple spokeswoman declined to comment. A spokeswoman for UAE’s Ministry of Foreign Affairs declined to comment. The UAE’s Embassy in Washington and a spokesman for its National Media Council did not respond to requests for comment.

The UAE has said it faces a real threat from violent extremist groups and that it is cooperating with the United States on counterterrorism efforts. Former Raven operatives say the project helped NESA break up an ISIS network within the Emirates. When an ISIS-inspired militant stabbed to death a teacher in Abu Dhabi in 2014, the operatives say, Raven spearheaded the UAE effort to assess if other attacks were imminent.

Various reports have highlighted the ongoing cyber arms race in the Middle East, as the Emirates and other nations attempt to sweep up hacking weapons and personnel faster than their rivals. The Reuters investigation is the first to reveal the existence of Project Raven, providing a rare inside account of state hacking operations usually shrouded in secrecy and denials.

The Raven story also provides new insight into the role former American cyberspies play in foreign hacking operations. Within the U.S. intelligence community, leaving to work as an operative for another country is seen by some as a betrayal. “There’s a moral obligation if you’re a former intelligence officer from becoming effectively a mercenary for a foreign government,” said Bob Anderson, who served as executive assistant director of the Federal Bureau of Investigation until 2015.

While this activity raises ethical dilemmas, U.S. national security lawyers say the laws guiding what American intelligence contractors can do abroad are murky. Though it’s illegal to share classified information, there is no specific law that bars contractors from sharing more general spycraft knowhow, such as how to bait a target with a virus-laden email.

The rules, however, are clear on hacking U.S. networks or stealing the communications of Americans. “It would be very illegal,” said Rhea Siers, former NSA deputy assistant director for policy.

The hacking of Americans was a tightly held secret even within Raven, with those operations led by Emiratis instead. Stroud’s account of the targeting of Americans was confirmed by four other former operatives and in emails reviewed by Reuters.

The FBI is now investigating whether Raven’s American staff leaked classified U.S. surveillance techniques and if they illegally targeted American computer networks, according to former Raven employees interviewed by federal law enforcement agents. Stroud said she is cooperating with that investigation. No charges have been filed and it is possible none will emerge from the inquiry. An FBI spokeswoman declined to comment.

PURPLE BRIEFING, BLACK BRIEFING
Stroud is the only former Raven operative willing to be named in this story; eight others who described their experiences would do so only on condition of anonymity. She spent a decade at the NSA, first as a military service member from 2003 to 2009 and later as a contractor in the agency for the giant technology consultant Booz Allen Hamilton from 2009 to 2014. Her speciality was hunting for vulnerabilities in the computer systems of foreign governments, such as China, and analysing what data should be stolen.

In 2013, her world changed. While stationed at NSA Hawaii, Stroud says, she made the fateful recommendation to bring a Dell technician already working in the building onto her team. That contractor was Edward Snowden.

“He’s former CIA, he’s local, he’s already cleared,” Stroud, 37, recalled. “He’s perfect!” Booz and the NSA would later approve Snowden’s transfer, providing him with even greater access to classified material.

Two months after joining Stroud’s group, Snowden fled the United States and passed on thousands of pages of top secret program files to journalists, detailing the agency’s massive data collection programs. In the maelstrom that followed, Stroud said her Booz team was vilified for unwittingly enabling the largest security breach in agency history.

“Our brand was ruined,” she said of her team.

In the wake of the scandal, Marc Baier, a former colleague at NSA Hawaii, offered her the chance to work for a contractor in Abu Dhabi called CyberPoint. In May 2014, Stroud jumped at the opportunity and left Booz Allen.

CyberPoint, a small cybersecurity contractor headquartered in Baltimore, was founded by an entrepreneur named Karl Gumtow in 2009. Its clients have included the U.S. Department of Defense, and its UAE business has gained media attention.

In an interview, Gumtow said his company was not involved in any improper actions.

Stroud had already made the switch from government employee to Booz Allen contractor, essentially performing the same NSA job at higher pay. Taking a job with CyberPoint would fulfil a lifelong dream of deploying to the Middle East and doing so at a lucrative salary. Many analysts, like Stroud, were paid more than $200,000 a year, and some managers received salaries and compensation above $400,000.

She understood her new job would involve a counterterrorism mission in cooperation with the Emiratis, a close U.S. ally in the fight against ISIS, but little else. Baier and other Raven managers assured her the project was approved by the NSA, she said. With Baier’s impressive resume, including time in an elite NSA hacking unit known as Tailored Access Operations, the pledge was convincing. Baier did not respond to multiple phone calls, text messages, emails, and messages on social media.

In the highly secretive, compartmentalized world of intelligence contracting, it isn’t unusual for recruiters to keep the mission and client from potential hires until they sign non-disclosure documents and go through a briefing process.

When Stroud was brought into the Villa for the first time, in May 2014, Raven management gave her two separate briefings, back-to-back.

In the first, known internally as the “Purple briefing,” she said she was told Raven would pursue a purely defensive mission, protecting the government of the UAE from hackers and other threats. Right after the briefing ended, she said she was told she had just received a cover story.

She then received the “Black briefing,” a copy of which was reviewed by Reuters. Raven is “the offensive, operational division of NESA and will never be acknowledged to the general public,” the Black memo says. The NESA, or National Electronic Security Authority, was the UAE’s version of the NSA.

Stroud would be part of Raven’s analysis and target-development shop, tasked with helping the government profile its enemies online, hack them and collect data. Those targets were provided by the client, NESA, now called the Signals Intelligence Agency.

The language and secrecy of the briefings closely mirrored her experience at the NSA, Stroud said, giving her a level of comfort.

The information scooped up by Raven was feeding a security apparatus that has drawn international criticism. The Emirates, a wealthy federation of seven Arab sheikhdoms with a population of 9 million, is an ally of neighbour Saudi Arabia and rival of Iran.

Like those two regional powers, the UAE has been accused of suppressing free speech, detaining dissidents and other abuses by groups such as Human Rights Watch. The UAE says it is working closely with Washington to fight extremism “beyond the battlefield” and is promoting efforts to counter the “root causes” of radical violence.

Raven’s targets eventually would include militants in Yemen, foreign adversaries such as Iran, Qatar and Turkey, and individuals who criticized the monarchy, said Stroud and eight other former Raven operatives. Their accounts were confirmed by hundreds of Raven program documents reviewed by Reuters.

Under orders from the UAE government, former operatives said, Raven would monitor social media and target people who security forces felt had insulted the government.

“Some days it was hard to swallow, like [when you target] a 16-year-old kid on Twitter,” she said. “But it’s an intelligence mission, you are an intelligence operative. I never made it personal.”

The Americans identified vulnerabilities in selected targets, developed or procured software to carry out the intrusions and assisted in monitoring them, former Raven employees said. But an Emirati operative would usually press the button on an attack. This arrangement was intended to give the Americans “plausible deniability” about the nature of the work, said former Raven members.

TARGETING ‘GYRO’ AND ‘EGRET’
Stroud discovered that the program took aim not just at terrorists and foreign government agencies, but also dissidents and human rights activists. The Emiratis categorized them as national security targets.

Following the Arab Spring protests and the ousting of Egyptian President Hosni Mubarak in 2011, Emirati security forces viewed human rights advocates as a major threat to “national stability,” records and interviews show.

Lori Stroud is pictured in her home at an unknown location in the U.S., September 27, 2018. Picture taken September 27, 2018. REUTERS/Joel Schectman
One of the program’s key targets in 2012 was Rori Donaghy, according to former Raven operatives and program documents. Donaghy, then 25, was a British journalist and activist who authored articles critical of the country’s human rights record. In 2012, he wrote an opinion piece for the Guardian criticizing the UAE government’s activist crackdown and warning that, if it continued, “those in power face an uncertain future.”

Before 2012, the former operatives said, the nascent UAE intelligence-gathering operation largely relied on Emirati agents breaking into the homes of targets while they were away and physically placing spyware on computers. But as the Americans built up Raven, the remote hacking of Donaghy offered the contractors a tantalizing win they could present to the client.

Because of sensitivity over human rights violations and press freedom in the West, the operation against a journalist-activist was a gamble. “The potential risk to the UAE Government and diplomatic relations with Western powers is great if the operation can be traced back to UAE,” 2012 program documents said.

To get close to Donaghy, a Raven operative should attempt to “ingratiate himself to the target by espousing similar beliefs,” the cyber-mercenaries wrote. Donaghy would be “unable to resist an overture of this nature,” they believed.

Posing as a single human rights activist, Raven operatives emailed Donaghy asking for his help to “bring hope to those who are long suffering,” the email message said.

The operative convinced Donaghy to download software he claimed would make messages “difficult to trace.” In reality, the malware allowed the Emiratis to continuously monitor Donaghy’s email account and Internet browsing. The surveillance against Donaghy, who was given the code name Gyro, continued under Stroud and remained a top priority for the Emirates for years, Stroud said.

Donaghy eventually became aware that his email had been hacked. In 2015, after receiving another suspicious email, he contacted a security researcher at Citizen Lab, a Canadian human rights and digital privacy group, who discovered hackers had been attempting for years to breach his computer.

Reached by phone in London, Donaghy, now a graduate student pursuing Arab studies, expressed surprise he was considered a top national security target for five years. Donaghy confirmed he was targeted using the techniques described in the documents.

“I’m glad my partner is sitting here as I talk on the phone because she wouldn’t believe it,” he said. Told the hackers were American mercenaries working for the UAE, Donaghy, a British citizen, expressed surprise and disgust. “It feels like a betrayal of the alliance we have,” he said.

Stroud said her background as an intelligence operative made her comfortable with human rights targets as long as they weren’t Americans. “We’re working on behalf of this country’s government, and they have specific intelligence objectives which differ from the U.S., and understandably so,” Stroud said. “You live with it.”

Prominent Emirati activist Ahmed Mansoor, given the code name Egret, was another target, former Raven operatives say. For years, Mansoor publicly criticized the country’s war in Yemen, treatment of migrant workers and detention of political opponents.

In September 2013, Raven presented senior NESA officials with material taken from Mansoor’s computer, boasting of the successful collection of evidence against him. It contained screenshots of emails in which Mansoor discussed an upcoming demonstration in front of the UAE’s Federal Supreme Court with family members of imprisoned dissidents.

Raven told UAE security forces Mansoor had photographed a prisoner he visited in jail, against prison policy, “and then attempted to destroy the evidence on his computer,” said a Powerpoint presentation reviewed by Reuters.

Citizen Lab published research in 2016 showing that Mansoor and Donaghy were targeted by hackers — with researchers speculating that the UAE government was the most likely culprit. Concrete evidence of who was responsible, details on the use of American operatives, and first-hand accounts from the hacking team are reported here for the first time.

Mansoor was convicted in a secret trial in 2017 of damaging the country’s unity and sentenced to 10 years in jail. He is now held in solitary confinement, his health declining, a person familiar with the matter said.

Mansoor’s wife, Nadia, has lived in social isolation in Abu Dhabi. Neighbours are avoiding her out of fear security forces are watching.

They are correct. By June 2017 Raven had tapped into her mobile device and given her the code name Purple Egret, program documents reviewed by Reuters show.

To do so, Raven utilized a powerful new hacking tool called Karma, which allowed operatives to break into the iPhones of users around the world.

Karma allowed Raven to obtain emails, location, text messages and photographs from iPhones simply by uploading lists of numbers into a preconfigured system, five former project employees said. Reuters had no contact with Mansoor’s wife.

Karma was particularly potent because it did not require a target to click on any link to download malicious software. The operatives understood the hacking tool to rely on an undisclosed vulnerability in Apple’s iMessage text messaging software.

In 2016 and 2017, it would be used against hundreds of targets across the Middle East and Europe, including governments of Qatar, Yemen, Iran and Turkey, documents show. Raven used Karma to hack an iPhone used by the Emir of Qatar, Sheikh Tamim bin Hamad al-Thani, as well as the phones of close associates and his brother. The embassy of Qatar in Washington did not respond to requests for comment.

WHAT WASHINGTON KNEW
Former Raven operatives believed they were on the right side of the law because, they said, supervisors told them the mission was blessed by the U.S. government.

Although the NSA wasn’t involved in day-to-day operations, the agency approved of and was regularly briefed on Raven’s activities, they said Baier told them.

CyberPoint founder Gumtow said his company was not involved in hacking operations.

“We were not doing offensive operations. Period,” Gumtow said in a phone interview. “If someone was doing something rogue, then that’s painful for me to think they would do that under our banner.”

Instead, he said, the company trained Emiratis to defend themselves through a program with the country’s Ministry of Interior.

A review of internal Raven documents shows Gumtow’s description of the program as advising the Interior Ministry on cyber defence matches an “unclassified cover story” Raven operatives were instructed to give when asked about the project. Raven employees were told to say they worked for the Information Technology and Interoperability Office, the program document said.

Providing sensitive defence technologies or services to a foreign government generally requires special licenses from the U.S. State and Commerce Departments. Both agencies declined to comment on whether they issued such licenses to CyberPoint for its operations in the UAE. They added that human rights considerations figure into any such approvals.

But a 2014 State Department agreement with CyberPoint showed Washington understood the contractors were helping launch cyber surveillance operations for the UAE. The approval document explains CyberPoint’s contract is to work alongside NESA in the “protection of UAE sovereignty” through “collection of information from communications systems inside and outside the UAE” and “surveillance analysis.”

One section of the State Department approval states CyberPoint must receive specific approval from the NSA before giving any presentations pertaining to “computer network exploitation or attack.” Reuters identified dozens of such presentations Raven gave to NESA describing attacks against Donaghy, Mansoor and others. It’s unclear whether the NSA approved Raven’s operations against specific targets.

The agreement clearly forbade CyberPoint employees from targeting American citizens or companies. As part of the agreement, CyberPoint promised that its own staff and even Emirati personnel supporting the program “will not be used to Exploit U.S. Persons, (i.e. U.S. citizens, permanent resident aliens, or U.S. companies.)” Sharing classified U.S. information, controlled military technology, or the intelligence collection methods of U.S. agencies was also prohibited.

Gumtow declined to discuss the specifics of the agreement. “To the best of my ability and to the best of my knowledge, we did everything as requested when it came to U.S. rules and regulations,” he said. “And we provided a mechanism for people to come to me if they thought that something that was done was wrong.”

An NSA spokesman declined to comment on Project Raven.

A State Department spokesman declined to comment on the agreement but said such licenses do not authorize people to engage in human rights abuses.

By late 2015, some Raven operatives said their missions became more audacious.

For instance, instead of being asked to hack into individual users of an Islamist Internet forum, as before, the American contractors were called on to create computer viruses that would infect every person visiting a flagged site. Such wholesale collection efforts risked sweeping in the communications of American citizens, stepping over a line the operators knew well from their NSA days.

U.S. law generally forbids the NSA, CIA and other U.S. intelligence agencies from monitoring U.S. citizens.

Working together with managers, Stroud helped create a policy for what to do when Raven swept up personal data belonging to Americans. The former NSA employees were instructed to mark that material for deletion. Other Raven operatives would also be notified so the American victims could be removed from future collection.

As time went on, Stroud noticed American data flagged for removal show up again and again in Raven’s NESA-controlled data stores.

Still, she found the work exhilarating. “It was incredible because there weren’t these limitations like there was at the NSA. There wasn’t that bullshit red tape,” she said. “I feel like we did a lot of good work on counterterrorism.”

DARKMATTER AND DEPARTURES
When Raven was created in 2009, Abu Dhabi had little cyber expertise. The original idea was for Americans to develop and run the program for five to 10 years until Emirati intelligence officers were skilled enough to take over, documents show. By 2013, the American contingent at Raven numbered between a dozen and 20 members at any time, accounting for the majority of the staff.

In late 2015, the power dynamic at the Villa shifted as the UAE grew more uncomfortable with a core national security program being controlled by foreigners, former staff said. Emirati defence officials told Gumtow they wanted Project Raven to be run through a domestic company, named DarkMatter.

Slideshow (13 Images)
Raven’s American creators were given two options: Join DarkMatter or go home.

At least eight operatives left Raven during this transition period. Some said they left after feeling unsettled about the vague explanations Raven managers provided when pressed on potential surveillance against other Americans.

DarkMatter was founded in 2014 by Faisal Al Bannai, who also created Axiom, one of the largest sellers of mobile devices in the region. DarkMatter markets itself as an innovative developer of defensive cyber technology. A 2016 Intercept article reported the company assisted UAE’s security forces in surveillance efforts and was attempting to recruit foreign cyber experts.

The Emirati company of more than 650 employees publicly acknowledges its close business relationship to the UAE government, but denies involvement in state-backed hacking efforts.

Project Raven’s true purpose was kept secret from most executives at DarkMatter, former operatives said.

DarkMatter did not respond to requests for comment. Al Bannai and the company’s current chief executive, Karim Sabbagh, did not respond to interview requests. A spokeswoman for the UAE Ministry of Foreign Affairs declined to comment.

Under DarkMatter, Project Raven continued to operate in Abu Dhabi from the Villa, but pressure escalated for the program to become more aggressive.

Before long, senior NESA officers were given more control over daily functions, former Raven operatives said, often leaving American managers out of the loop. By mid-2016, the Emirates had begun making an increasing number of sections of Raven hidden from the Americans still managing day-to-day operations. Soon, an “Emirate-eyes only” designation appeared for some hacking targets.

FBI QUESTIONS
By 2016, FBI agents began approaching DarkMatter employees reentering the United States to ask about Project Raven, three former operatives said.

The FBI wanted to know: Had they been asked to spy on Americans? Did classified information on U.S. intelligence collection techniques and technologies end up in the hands of the Emiratis?

Two agents approached Stroud in 2016 at Virginia’s Dulles airport as she was returning to the UAE after a trip home. Stroud, afraid she might be under surveillance by the UAE herself, said she brushed off the FBI investigators. “I’m not telling you guys jack,” she recounted.

Stroud had been promoted and given even more access to internal Raven databases the previous year. A lead analyst, her job was to probe the accounts of potential Raven targets and learn what vulnerabilities could be used to penetrate their email or messaging systems.

Targets were listed in various categories, by country. Yemeni targets were in the “brown category,” for example. Iran was grey.

One morning in spring 2017, after she finished her own list of targets, Stroud said she began working on a backlog of other assignments intended for a NESA officer. She noticed that a passport page of an American was in the system. When Stroud emailed supervisors to complain, she was told the data had been collected by mistake and would be deleted, according to an email reviewed by Reuters.

Concerned, Stroud began searching a targeting request list usually limited to Raven’s Emirati staff, which she was still able to access because of her role as lead analyst. She saw that security forces had sought surveillance against two other Americans.

When she questioned the apparent targeting of Americans, she received a rebuke from an Emirati colleague for accessing the targeting list, the emails show. The target requests she viewed were to be processed by “certain people. You are not one of them,” the Emirati officer wrote.

Days later, Stroud said she came upon three more American names on the hidden targeting queue.

Those names were in a category she hadn’t seen before: the “white category” — for Americans. This time, she said, the occupations were listed: journalist.

“I was sick to my stomach,” she said. “It kind of hit me at that macro level realizing there was a whole category for U.S. persons on this program.”

Once more, she said she turned to manager Baier. He attempted to downplay the concern and asked her to drop the issue, she said. But he also indicated that any targeting of Americans was supposed to be done by Raven’s Emirate staff, said Stroud and two other people familiar with the discussion.

Stroud’s account of the incidents was confirmed by four other former employees and emails reviewed by Reuters.

When Stroud kept raising questions, she said, she was put on leave by superiors, her phones and passport were taken, and she was escorted from the building. Stroud said it all happened so quickly she was unable to recall the names of the three U.S. journalists or other Americans she came across in the files. “I felt like one of those national security targets,” she said. “I’m stuck in the country, I’m being surveilled, I can’t leave.”

After two months, Stroud was allowed to return to America. Soon after, she fished out the business card of the FBI agents who had confronted her at the airport.

“I don’t think Americans should be doing this to other Americans,” she told Reuters. “I’m a spy, I get that. I’m an intelligence officer, but I’m not a bad one.”

By Christopher Bing and Joel Schectman in Washington. Editing by Ronnie Greene, Jonathan Weber and Michael Williams

Our Standards:The Thomson Reuters Trust Principles.
MIDDLE EAST & NORTH AFRICAFEBRUARY 7, 2019 / 2:30 PM / UPDATED 19 MINUTES AGO
Leaked files reveal Iran’s post-revolution crackdown on journalists
Luke Baker
3 MIN READ

PARIS (Reuters) – The Iranian government arrested, imprisoned or executed at least 860 journalists in the three decades between the Islamic revolution in 1979 and 2009, according to documents leaked to media monitoring group Reporters Sans Frontieres (RSF).

At a news conference in Paris attended by Iranian human rights lawyer Shirin Ebadi, recipient of the 2003 Nobel Peace Prize, RSF said whistleblowers had passed on 1.7 million records detailing judicial proceedings against an array of citizens, including minorities, government opponents and journalists.

RSF secretary-general Christophe Deloire said the group had spent months cross-checking the records with its own documented cases and those of other NGOs, and had established that hundreds of journalists had been targeted by the state.

“The file is a register of all the arrests, imprisonments and executions carried out by the Iranian authorities in the Tehran area over three decades,” RSF said.

Representatives of the Iranian government were not immediately reachable for comment on Thursday, a holiday in Iran. But last week Iranian authorities reiterated that there were no political prisoners being held in the country.

RSF released the report to coincide with the 40th anniversary of the Islamic revolution that brought Ayatollah Ruhollah Khomeini to power.

“After months of detailed research work on the file’s entries, RSF is in a position to say that at least 860 journalists and citizen-journalists were arrested, imprisoned and in some cases executed by the Iranian regime between 1979 and 2009, the period on which RSF focused its research.”

Deloire said his organisation would refer the file to the United Nations high commissioner for human rights in the hope further steps could be taken to hold Iran to account.

“The very existence of this file and its millions of entries show not only the scale of the Iranian regime’s mendacity…but the relentless machinations it used for 40 years to persecute men and women for their opinions or their reporting,” he said.

Last month, Amnesty International issued a report accusing the Iranian authorities of a crackdown on dissent during 2018 with more than 7,000 people arrested, among them students, journalists, environmental campaigners and lawyers.

Slideshow (3 Images)
In its analysis, RSF said it had identified at least four journalists who were executed, including Simon Farzami, a Swiss-Iranian of Jewish origin who was bureau chief of French news agency Agence France-Presse when he was arrested in 1980.

Among the 860 were 218 women.

Beyond the journalists rounded up or imprisoned, RSF said the files showed 61,900 political prisoners had been held since the 1980s, with more than 500 of them aged between 15 and 18.

It said the files added to evidence of a massacre in 1988 in which around 4,000 political prisoners were executed on the orders of Khomeini between July and September. Iran has always denied that any such massacre took place.

Writing by Luke Baker; Additional reporting by Michaela Cabrera in Paris and Parisa Hafezi in Dubai; Editing by

Our Standards:The Thomson Reuters Trust Principles.
AppsNewslettersAdvertise with UsAdvertising GuidelinesCookiesTerms of UsePrivacy
All quotes delayed a minimum of 15 minutes. See here for a complete list of exchanges and delays.

© 2019 Reuters. All Rights Reserved.

COOKIES AUF DIESER WEBSITE
Wir verwenden Cookies, um Ihr Erlebnis auf unserer Website so angenehm wie möglich zu gestalten. Weitere Informationen über unsere Datenerfassungspraktiken finden Sie in unserer Datenschutzrichtlinie. Durch Klicken auf Akzeptieren erkären Sie sich mit der Verwendung von Cookies für die in unserem Tool zur Cookie-Kontrolle aufgeführten Zwecke einverstanden.