CLI: improved

CLI: improved

HomeSearchLatestPrevious
CLI: improved
I’m not sure many web developers can get away without visiting the command line. As for me, I’ve been using the command line since 1997, first at university when I felt both super cool l33t-hacker and simultaneously utterly out of my depth.

Over the years my command line habits have improved and I often search for smarter tools for the jobs I commonly do. With that said, here’s my current list of improved CLI tools.

READER DISCOUNTSave $80 on terminal.training

I’ve published 38 videos for new developers, designers, UX, UI, product owners and anyone who needs to conquer the command line today.

$19 – only from this link

Ignoring my improvements
In a number of cases I’ve aliased the new and improved command line tool over the original (as with cat and ping).

If I want to run the original command, which is sometimes I do need to do, then there’s two ways I can do this (I’m on a Mac so your mileage may vary):

\cat # ignore aliases named “cat” – explanation: https://stackoverflow.com/a/16506263/22617
command cat # ignore functions and aliases
bat > cat
cat is used to print the contents of a file, but given more time spent in the command line, features like syntax highlighting come in very handy. I found ccat which offers highlighting then I found bat which has highlighting, paging, line numbers and git integration.

The bat command also allows me to search during output (only if the output is longer than the screen height) using the / key binding (similarly to less searching).

Simple bat output

I’ve also aliased bat to the cat command:

alias cat=’bat’
💾 Installation directions

prettyping > ping
ping is incredibly useful, and probably my goto tool for the “oh crap is X down/does my internet work!!!”. But prettyping (“pretty ping” not “pre typing”!) gives ping a really nice output and just makes me feel like the command line is a bit more welcoming.

/images/cli-improved/ping.gif

I’ve also aliased ping to the prettyping command:

alias ping=’prettyping –nolegend’
💾 Installation directions

fzf > ctrl+r
In the terminal, using ctrl+r will allow you to search backwards through your history. It’s a nice trick, albeit a bit fiddly.

The fzf tool is a huge enhancement on ctrl+r. It’s a fuzzy search against the terminal history, with a fully interactive preview of the possible matches.

In addition to searching through the history, fzf can also preview and open files, which is what I’ve done in the video below:

For this preview effect, I created an alias called preview which combines fzf with bat for the preview and a custom key binding to open VS Code:

alias preview=”fzf –preview ‘bat –color \”always\” {}'”
# add support for ctrl+o to open selected file in VS Code
export FZF_DEFAULT_OPTS=”–bind=’ctrl-o:execute(code {})+abort'”
💾 Installation directions

htop > top
top is my goto tool for quickly diagnosing why the CPU on the machine is running hard or my fan is whirring. I also use these tools in production. Annoyingly (to me!) top on the Mac is vastly different (and inferior IMHO) to top on linux.

However, htop is an improvement on both regular top and crappy-mac top. Lots of colour coding, keyboard bindings and different views which have helped me in the past to understand which processes belong to which.

Handy key bindings include:

P – sort by CPU
M – sort by memory usage
F4 – filter processes by string (to narrow to just “node” for instance)
space – mark a single process so I can watch if the process is spiking
htop output

There is a weird bug in Mac Sierra that can be overcome by running htop as root (I can’t remember exactly what the bug is, but this alias fixes it – though annoying that I have to enter my password every now and again):

alias top=”sudo htop” # alias top and fix high sierra bug
💾 Installation directions

diff-so-fancy > diff
I’m pretty sure I picked this one up from Paul Irish some years ago. Although I rarely fire up diff manually, my git commands use diff all the time. diff-so-fancy gives me both colour coding but also character highlight of changes.

diff so fancy

Then in my ~/.gitconfig I have included the following entry to enable diff-so-fancy on git diff and git show:

[pager]
diff = diff-so-fancy | less –tabs=1,5 -RFX
show = diff-so-fancy | less –tabs=1,5 -RFX
💾 Installation directions

fd > find
Although I use a Mac, I’ve never been a fan of Spotlight (I found it sluggish, hard to remember the keywords, the database update would hammer my CPU and generally useless!). I use Alfred a lot, but even the finder feature doesn’t serve me well.

I tend to turn the command line to find files, but find is always a bit of a pain to remember the right expression to find what I want (and indeed the Mac flavour is slightly different non-mac find which adds to frustration).

fd is a great replacement (by the same individual who wrote bat). It is very fast and the common use cases I need to search with are simple to remember.

A few handy commands:

fd cli # all filenames containing “cli”
fd -e md # all with .md extension
fd cli -x wc -w # find “cli” and run `wc -w` on each file
fd output

💾 Installation directions

ncdu > du
Knowing where disk space is being taking up is a fairly important task for me. I’ve used the Mac app Disk Daisy but I find that it can be a little slow to actually yield results.

The du -sh command is what I’ll use in the terminal (-sh means summary and human readable), but often I’ll want to dig into the directories taking up the space.

ncdu is a nice alternative. It offers an interactive interface and allows for quickly scanning which folders or files are responsible for taking up space and it’s very quick to navigate. (Though any time I want to scan my entire home directory, it’s going to take a long time, regardless of the tool – my directory is about 550gb).

Once I’ve found a directory I want to manage (to delete, move or compress files), I’ll use the cmd + click the pathname at the top of the screen in iTerm2 to launch finder to that directory.

ncdu output

There’s another alternative called nnn which offers a slightly nicer interface and although it does file sizes and usage by default, it’s actually a fully fledged file manager.

My ncdu is aliased to the following:

alias du=”ncdu –color dark -rr -x –exclude .git –exclude node_modules”
The options are:

–color dark – use a colour scheme
-rr – read-only mode (prevents delete and spawn shell)
–exclude ignore directories I won’t do anything about
💾 Installation directions

tldr > man
It’s amazing that nearly every single command line tool comes with a manual via man , but navigating the man output can be sometimes a little confusing, plus it can be daunting given all the technical information that’s included in the manual output.

This is where the TL;DR project comes in. It’s a community driven documentation system that’s available from the command line. So far in my own usage, I’ve not come across a command that’s not been documented, but you can also contribute too.

TLDR output for ‘fd’

As a nicety, I’ve also aliased tldr to help (since it’s quicker to type!):

alias help=’tldr’
💾 Installation directions

ack || ag > grep
grep is no doubt a powerful tool on the command line, but over the years it’s been superseded by a number of tools. Two of which are ack and ag.

I personally flitter between ack and ag without really remembering which I prefer (that’s to say they’re both very good and very similar!). I tend to default to ack only because it rolls of my fingers a little easier. Plus, ack comes with the mega ack –bar argument (I’ll let you experiment)!

Both ack and ag will (by default) use a regular expression to search, and extremely pertinent to my work, I can specify the file types to search within using flags like –js or –html (though here ag includes more files in the js filter than ack).

Both tools also support the usual grep options, like -B and -A for before and after context in the grep.

ack in action

Since ack doesn’t come with markdown support (and I write a lot in markdown), I’ve got this customisation in my ~/.ackrc file:

–type-set=md=.md,.mkd,.markdown
–pager=less -FRX
💾 Installation directions: ack, ag

Futher reading on ack & ag

jq > grep et al
I’m a massive fanboy of jq. At first I struggled with the syntax, but I’ve since come around to the query language and use jq on a near daily basis (whereas before I’d either drop into node, use grep or use a tool called json which is very basic in comparison).

I’ve even started the process of writing a jq tutorial series (2,500 words and counting) and have published a web tool and a native mac app (yet to be released).

jq allows me to pass in JSON and transform the source very easily so that the JSON result fits my requirements. One such example allows me to update all my node dependencies in one command (broken into multiple lines for readability):

npm i $(echo $(\
npm outdated –json | \
jq -r ‘to_entries | .[] | “\(.key)@\(.value.latest)”‘ \
))
The above command will list all the node dependencies that are out of date, and use npm’s JSON output format, then transform the source JSON from this:

{
“node-jq”: {
“current”: “0.7.0”,
“wanted”: “0.7.0”,
“latest”: “1.2.0”,
“location”: “node_modules/node-jq”
},
“uuid”: {
“current”: “3.1.0”,
“wanted”: “3.2.1”,
“latest”: “3.2.1”,
“location”: “node_modules/uuid”
}
}
…to this:

node-jq@1.2.0
uuid@3.2.1
That result is then fed into the npm install command and voilà, I’m all upgraded (using the sledgehammer approach).

Honourable mentions
Some of the other tools that I’ve started poking around with, but haven’t used too often (with the exception of ponysay, which appears when I start a new terminal session!):

ponysay > cowsay
csvkit > awk et al
noti > display notification
entr > watch
What about you?
So that’s my list. How about you? What daily command line tools have you improved? I’d love to know.

Posted 23-Aug 2018 under web.

Want more?
Posts, web development learnings & insights, exclusive workshop and training discounts and more, direct to your inbox.

I won’t send you any spam, and you can unsubscribe at any time. Powered by ConvertKit

Your name
Your name
Your email address
Email address
We use this field to detect spam bots. If you fill this in, you will be marked as a spammer.

I’d like to receive the free command line mini course.
Subscribe
Comments

Archives
All years
2018
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
Links
About Remy
Work with Remy
Subscribe
Ethos (WIP)
House Rules
The Attic
RSS feed
Search
Random post
On GitHub
On Twitter
Presentations
Remy Sharp
I’m a JavaScript developer working professionally on the web since 1999. I run my own consultancy, build products, run training, speak at conferences and curate the UK’s best JavaScript conference.

License
All content by Remy Sharp and under creative commons and code under MIT license.

All code and content for this blog is available as open source on GitHub.

The Fallacy of Federal Gun Laws

The Fallacy of Federal Gun Laws

AUTHOR: ANTONIO GARCÍA MARTÍNEZANTONIO GARCÍA MARTÍNEZ
IDEAS
08.15.1808:00 AM
HOW 3-D PRINTING EXPOSES THE FALLACY OF FEDERAL GUN LAWS

THOM LANG/GETTY IMAGES
DEFENSE DISTRIBUTED, AN Austin-based company notorious for 3-D printing a gun, recently won its risky landmark case against the federal government, allowing it to host digital gun designs on the internet. Though a federal judge blocked the ruling with a temporary injunction earlier this month, if the company’s legal strategy holds, it verges on mad genius. The reasoning: If a well-accepted First Amendment protects bits in the form of speech and code, and a contested Second Amendment protects atoms in the form of guns, then by turning guns into code (ie, atoms into bits) we get First Amendment protection for guns too.

Antonio García Martínez (@antoniogm) is an Ideas contributor for WIRED. Previously he worked on Facebook’s early monetization team, where he headed its targeting efforts. His 2016 memoir, Chaos Monkeys, was a New York Times best seller and NPR Best Book of the Year.

This Defense Distributed story—with its confusing twists into First Amendment jurisprudence, computer-controlled manufacturing, and 3-D printing (and more on that in a moment)—cannot be fully understood without understanding the past half-century in firearms innovation. This new technology is completely undermining the way we’ve traditional regulated guns, and will overturn our nation’s longstanding political compromise around them.

But first, what is a gun exactly?

The revolver above is my grandfather’s Smith and Wesson .38 Police Special (inherited), once the standard police sidearm. It’s a forged hunk of metal, and close to nothing on it can be modified without serious gunsmithing—and, even then, it’ll never be more than a medium-frame, six-shot revolver.

Red – gun. Green – not gun ANTONIO GARCIA-MARTINEZ
Let’s fast-forward to the present.

This is a disassembled AR15. It is modular in every way, and every bit of it can be swapped out like so many Legos. Whatever your opinions on gun control, the AR15 is a masterpiece of open source, modular design—by changing out a part for one with a similar function but different design, both military and civilian owners can endlessly modify it into almost anything, from a submachine gun to a sniper rifle.

Red – gun. Green – not gun ANTONIO GARCIA-MARTINEZ
You might ask: If the modern military weapon is just Legos, then what part is the “gun,” the specific piece of hardware with unique rules around sales and ownership that makes it different from, say, a chainsaw or blender?

The federal bureaucrat responds: The “gun” is the thing with the serial numbers on it. In the case of AR-pattern rifles, that’s what’s known as the lower receiver (or just “lower”). It’s the lower part that encases the rifle’s action, where the pistol grip attaches and a magazine goes in.

Everything else is an accessory, even though it actually does the work of a gun, namely loading a cartridge into the action, striking the primer, which ignites the gunpowder, launching the projectile and hot gasses out of the barrel, and, finally, getting ready to do so again.

This is the official “gun” part of an AR. Everything else is really, truly, not considered a “gun.”

The decision to anoint the lower receiver a gun is a momentous bureaucratic decision. It means that everything in the “non-gun” category, which is most of the actual gun, can be bought and sold unrestricted online, like so many shoes on Zappos. Right now, you can buy everything in the “non-gun” category with zero checks online.

Particularly in light of the initial discussion around advanced manufacturing, you might ask: If the “gun” is this relatively small piece of metal or polymer, then why can’t I make it myself somehow?

Let’s answer that via a thought experiment:

Say I took a rectangular hunk of aluminum and machined out a hole like the magazine well of the AR.

Then I machined out the trigger opening.

Then I machined out the base where the pistol grip attaches and every other detail in the “gun” part (see graphic).

At what point did the hunk of metal become a “gun”?

This isn’t mere philosophizing. It’s a real issue in gun regulation. The federal bureaucrat responds again: That piece of aluminum became the “gun” when you got 81 percent of the way to finishing it. After that point, it must be bought and sold as a gun, with all the restrictions that apply.

What does the enormous US market in gun parts do? It sells you an 80 percent finished piece of metal, which is officially not a “gun” (though it really looks like one) on the internet, no questions asked (link provided for reference, not an endorsement). This is known in the gun world as an “80 percent lower.”

The aspiring unregistered gun owner then finishes the last 20 percent at home, with a drill press and/or a milling machine. Combined with the “non-gun” parts bought freely online, they now have a fully functioning AR15. Nobody knows that gun exists, and it is completely untraceable.

Some retailers (such as the one linked above) offer you the entire package: every non-gun piece required and an 80 percent finished “gun” (again, really just the lower receiver), plus they’ll even include some machining jigs to make the finishing easier. This is the IKEA model of firearms, which even has its own IKEA Effect: New gun owners feel disproportionately attached to a gun they “made,” even though really they just put it together, often with the same hex keys you’d use on an IKEA bookshelf. Anyone with a credit card and address can buy this, and nobody on the planet is the wiser, save the retailer, who is not required to report anything, since they’re just selling “accessories.”

The entire gun debate is really about that last 20 percent of the bureaucratic “gun,” and how and where it happens. This is where Defense Distributed really comes in. They sell you a gadget that makes that last 20 percent step easier than doing it yourself, via a computer-controlled milling technology that’s been around forever. It isn’t a fundamentally new gun technology, but it is an important last-mile advance.

I mention 3-D printing nowhere. That’s because it’s irrelevant, and most of what you read about 3-D-printed guns is noise. 3-D printing is a relatively immature manufacturing technology, and it won’t be making a modern, repeating firearm any time soon. (At least not the home 3-D printers that are cheaply available.) The only reason 3-D printing is even in the story is because Defense Distributed, geniuses at marketing, created a toy 3-D-printed product that could fire one round (before maybe exploding in your hand) as a way to plant the meme in the heads of people (and politicians) who know nothing about guns.

So why is everyone freaking out, including the attorneys general of several states that are suing Defense Distributed to stop? First, they are misunderstanding the technology, and believing Star Trek replicator fantasies around 3-D printing and guns. Second, and more seriously, advances in gun modularity and manufacturing now make most gun regulation obsolete.

To understand why technologies like those of Defense Distributed are poised to make gun regulation obsolete, we have to understand how gun regulation historically worked. In states like California and New York that wish to limit access to dangerous weapons, regulation has focused on certain features associated with military-style firearms, like pistol grips, quick-release magazines, folding stocks, and so on. In a world where guns were largely static and hard to modify, and the existence of such features indicated whether they were destined for sporting or military use, this approach made sense.

With this new modularity of firearms, this has essentially already happened. Consider the .50 Barrett M107 sniper rifle. You may have seen the weapon in 2010’s Academy Award for Best Picture winner, The Hurt Locker, where it’s used by US soldiers to eliminate Iraqi snipers from an astonishing distance. It fires a projectile the size of your thumb, and can kill a man from over a mile away. In spirit, the weapon is illegal in California. In actual fact, it’s legal with the right modifications that only slightly impact functionality. Gun regulation fails.

Why does this odd status quo exist?

Our current gun laws are a necessary compromise among pro- and anti-gun extremes, plus a large middle that wants some gun control but not an outright ban. The NRA zealot is placated by Democratic rhetoric around banning only “weapons of war” paired with the technical knowledge that they can tolerably dodge most Blue State gun laws via the modular technology described above. The pro-gun-control Blue Staters are placated because politicians are “doing something,” and thanks mostly to ignorance about how modern guns work, think their gun laws are actually stopping the distribution of firearms when they increasingly resemble security theater.

Defense Distributed’s ultimate goal is to kick the final, weak leg out from under this tenuous political agreement, and force a reckoning with the state of firearms technology. When the last-mile problem of untraceable, unregistered guns has finally been “solved,” even politicians can’t maintain the charade of effective gun control.

Whether Defense Distributed manages to spark the conflict now or advancing technology eventually sparks it later, this country will come to a total impasse on guns in which the existing compromise becomes untenable. What then?

Foundational ideological conflicts in this country have always been resolved by one side co-opting some feature of the federal government (for example, the US Army, the Supreme Court, ICE) to force the other side to accept its interpretation of the Constitution. Depending on the party in power, both Democrats and Republicans have attempted just that, such as by tightening gun laws via the Federal Assault Weapons Ban signed by Bill Clinton or, more recently, through failed attempts by House Republicans to expand concealed-carry rights nationally.

Alternatively, faced with this and other divergent opinions on key issues like abortion and immigration, the union reverts to a pre–Civil War level of federalism, where the US becomes a loose association of states, and every state goes off their own constitutional interpretation. In that scenario, the Blue States pursue the gun-banning strategy, turning themselves into a US version of Canada. Their pro-gun residents will rail online about “From my cold, dead hands!” and then quietly move elsewhere (to some extent, this is already happening). The pro-gun states will maintain their pro-gun stance, whatever the gun violence rates may be.

It’ll be weird and inconsistent, but we’ll prefer this new awkward compromise to the alternatives: “Calexit,” another Civil War, or the dissolution of the 242-year-old experiment in democracy we call the United States of America.

More Great WIRED Stories
Want to get better at PUBG? Ask PlayerUnknown himself
Hacking a brand new Mac remotely, right out of the box
The super-secret sand that makes your phone possible
Climate change’s looming mental health crisis
Silicon Valley’s playbook to help avoid ethical disasters
Looking for more? Sign up for our daily newsletter and never miss our latest and greatest stories
RELATED VIDEO

DESIGN
3D Printing’s Future
Ceo of 3DSystems Abe Reichental and a couple of aspiring 3D engineers discuss the future place and capabilities of 3D Printing.

#DEFENSE DISTRIBUTE#3-D PRINTING#GUN#FIRST AMENDMENT#TECHNOLOGY
VIEW COMMENTS
SPONSORED STORIES
POWERED BY OUTBRAIN

OE24.AT
Drama um sechsfache Mutter: Nach einem Schnupfen verlor sie Arme & Beine

HYPER TECH
400.000 verkaufte Exemplare: Der Rucksack, um den sich viele Deutsche derzeit reißen

INDIA TODAY
WhatsApp tells Indian govt it can’t trace messages to original sender

GEEKSMATE
How to Detect if Someone is Stealing Your WiFi?

incise.org: tinywm

incise.org: tinywm

TinyWM is a tiny window manager that I created as an exercise in minimalism. It is also maybe helpful in learning some of the very basics of creating a window manager. It is only around 50 lines of C. There is also a Python version using python-xlib.

It lets you do four basic things:

Move windows interactively with Alt+Button1 drag (left mouse button)
Resize windows interactively with Alt+Button3 drag (right mouse button)
Raise windows with Alt+F1 (not high on usability I know, but I needed a keybinding in there somewhere)
Focus windows with the mouse pointer (X does this on its own)
Download
http://incise.org/files/dev/tinywm-1.3.tgz
Github
Known to be packaged in
Debian
Ubuntu
FreeBSD
CRUX
TinyWM around the web
SkatOSWM
In Lisp
someone hacking it
Is this cool, or what?
In a car PC
This guy is doing something or another with it
Ada version
XCB version
Some Processing-related project at the Rhode Island School of Design
Chicken Scheme version
Giles Orr’s Window Manager Report
An extension for Tiny Core Linux
Possum window manaager
SmallWM
See Also
not so tiny window managers
whimsy
The source
Here is tinywm.c from the most recent release, 1.3:


/* TinyWM is written by Nick Welch <mack@incise.org>, 2005.
 *
 * This software is in the public domain
 * and is provided AS IS, with NO WARRANTY. */

#include <X11/Xlib.h>

#define MAX(a, b) ((a) > (b) ? (a) : (b))

int main()
{
    Display * dpy;
    Window root;
    XWindowAttributes attr;
    XButtonEvent start;
    XEvent ev;

    if(!(dpy = XOpenDisplay(0x0))) return 1;

    root = DefaultRootWindow(dpy);

    XGrabKey(dpy, XKeysymToKeycode(dpy, XStringToKeysym("F1")), Mod1Mask, root,
            True, GrabModeAsync, GrabModeAsync);
    XGrabButton(dpy, 1, Mod1Mask, root, True, ButtonPressMask, GrabModeAsync,
            GrabModeAsync, None, None);
    XGrabButton(dpy, 3, Mod1Mask, root, True, ButtonPressMask, GrabModeAsync,
            GrabModeAsync, None, None);

    for(;;)
    {
        XNextEvent(dpy, &ev);
        if(ev.type == KeyPress && ev.xkey.subwindow != None)
            XRaiseWindow(dpy, ev.xkey.subwindow);
        else if(ev.type == ButtonPress && ev.xbutton.subwindow != None)
        {
            XGrabPointer(dpy, ev.xbutton.subwindow, True,
                    PointerMotionMask|ButtonReleaseMask, GrabModeAsync,
                    GrabModeAsync, None, None, CurrentTime);
            XGetWindowAttributes(dpy, ev.xbutton.subwindow, &attr);
            start = ev.xbutton;
        }
        else if(ev.type == MotionNotify)
        {
            int xdiff, ydiff;
            while(XCheckTypedEvent(dpy, MotionNotify, &ev));
            xdiff = ev.xbutton.x_root - start.x_root;
            ydiff = ev.xbutton.y_root - start.y_root;
            XMoveResizeWindow(dpy, ev.xmotion.window,
                attr.x + (start.button==1 ? xdiff : 0),
                attr.y + (start.button==1 ? ydiff : 0),
                MAX(1, attr.width + (start.button==3 ? xdiff : 0)),
                MAX(1, attr.height + (start.button==3 ? ydiff : 0)));
        }
        else if(ev.type == ButtonRelease)
            XUngrabPointer(dpy, CurrentTime);
    }
}

Here is annotated.c, which is just tinywm.c with a lot of comments explaining what is going on. This should give you a reasonable idea of how everything works.

/* TinyWM is written by Nick Welch <mack@incise.org>, 2005.
 *
 * This software is in the public domain
 * and is provided AS IS, with NO WARRANTY. */

/* much of tinywm's purpose is to serve as a very basic example of how to do X
 * stuff and/or understand window managers, so i wanted to put comments in the
 * code explaining things, but i really hate wading through code that is
 * over-commented -- and for that matter, tinywm is supposed to be as concise
 * as possible, so having lots of comments just wasn't really fitting for it.
 * i want tinywm.c to be something you can just look at and go "wow, that's
 * it?  cool!"  so what i did was just copy it over to annotated.c and comment
 * the hell out of it.  ahh, but now i have to make every code change twice!
 * oh well.  i could always use some sort of script to process the comments out
 * of this and write it to tinywm.c ... nah.
 */

/* most X stuff will be included with Xlib.h, but a few things require other
 * headers, like Xmd.h, keysym.h, etc.
 */
#include <X11/Xlib.h>

#define MAX(a, b) ((a) > (b) ? (a) : (b))

int main()
{
    Display * dpy;
    Window root;
    XWindowAttributes attr;

    /* we use this to save the pointer's state at the beginning of the
     * move/resize.
     */
    XButtonEvent start;

    XEvent ev;


    /* return failure status if we can't connect */
    if(!(dpy = XOpenDisplay(0x0))) return 1;

    /* you'll usually be referencing the root window a lot.  this is a somewhat
     * naive approach that will only work on the default screen.  most people
     * only have one screen, but not everyone.  if you run multi-head without
     * xinerama then you quite possibly have multiple screens. (i'm not sure
     * about vendor-specific implementations, like nvidia's)
     *
     * many, probably most window managers only handle one screen, so in
     * reality this isn't really *that* naive.
     *
     * if you wanted to get the root window of a specific screen you'd use
     * RootWindow(), but the user can also control which screen is our default:
     * if they set $DISPLAY to ":0.foo", then our default screen number is
     * whatever they specify "foo" as.
     */
    root = DefaultRootWindow(dpy);

    /* you could also include keysym.h and use the XK_F1 constant instead of
     * the call to XStringToKeysym, but this method is more "dynamic."  imagine
     * you have config files which specify key bindings.  instead of parsing
     * the key names and having a huge table or whatever to map strings to XK_*
     * constants, you can just take the user-specified string and hand it off
     * to XStringToKeysym.  XStringToKeysym will give you back the appropriate
     * keysym or tell you if it's an invalid key name.
     *
     * a keysym is basically a platform-independent numeric representation of a
     * key, like "F1", "a", "b", "L", "5", "Shift", etc.  a keycode is a
     * numeric representation of a key on the keyboard sent by the keyboard
     * driver (or something along those lines -- i'm no hardware/driver expert)
     * to X.  so we never want to hard-code keycodes, because they can and will
     * differ between systems.
     */
    XGrabKey(dpy, XKeysymToKeycode(dpy, XStringToKeysym("F1")), Mod1Mask, root,
            True, GrabModeAsync, GrabModeAsync);

    /* XGrabKey and XGrabButton are basically ways of saying "when this
     * combination of modifiers and key/button is pressed, send me the events."
     * so we can safely assume that we'll receive Alt+F1 events, Alt+Button1
     * events, and Alt+Button3 events, but no others.  You can either do
     * individual grabs like these for key/mouse combinations, or you can use
     * XSelectInput with KeyPressMask/ButtonPressMask/etc to catch all events
     * of those types and filter them as you receive them.
     */
    XGrabButton(dpy, 1, Mod1Mask, root, True, ButtonPressMask, GrabModeAsync,
            GrabModeAsync, None, None);
    XGrabButton(dpy, 3, Mod1Mask, root, True, ButtonPressMask, GrabModeAsync,
            GrabModeAsync, None, None);

    for(;;)
    {
        /* this is the most basic way of looping through X events; you can be
         * more flexible by using XPending(), or ConnectionNumber() along with
         * select() (or poll() or whatever floats your boat).
         */
        XNextEvent(dpy, &ev);

        /* this is our keybinding for raising windows.  as i saw someone
         * mention on the ratpoison wiki, it is pretty stupid; however, i
         * wanted to fit some sort of keyboard binding in here somewhere, and
         * this was the best fit for it.
         *
         * i was a little confused about .window vs. .subwindow for a while,
         * but a little RTFMing took care of that.  our passive grabs above
         * grabbed on the root window, so since we're only interested in events
         * for its child windows, we look at .subwindow.  when subwindow
         * None, that means that the window the event happened in was the same
         * window that was grabbed on -- in this case, the root window.
         */
        if(ev.type == KeyPress && ev.xkey.subwindow != None)
            XRaiseWindow(dpy, ev.xkey.subwindow);
        else if(ev.type == ButtonPress && ev.xbutton.subwindow != None)
        {
            /* now we take command of the pointer, looking for motion and
             * button release events.
             */
            XGrabPointer(dpy, ev.xbutton.subwindow, True,
                    PointerMotionMask|ButtonReleaseMask, GrabModeAsync,
                    GrabModeAsync, None, None, CurrentTime);

            /* we "remember" the position of the pointer at the beginning of
             * our move/resize, and the size/position of the window.  that way,
             * when the pointer moves, we can compare it to our initial data
             * and move/resize accordingly.
             */
            XGetWindowAttributes(dpy, ev.xbutton.subwindow, &attr);
            start = ev.xbutton;
        }
        /* the only way we'd receive a motion notify event is if we already did
         * a pointer grab and we're in move/resize mode, so we assume that. */
        else if(ev.type == MotionNotify)
        {
            int xdiff, ydiff;

            /* here we "compress" motion notify events.  if there are 10 of
             * them waiting, it makes no sense to look at any of them but the
             * most recent.  in some cases -- if the window is really big or
             * things are just acting slowly in general -- failing to do this
             * can result in a lot of "drag lag."
             *
             * for window managers with things like desktop switching, it can
             * also be useful to compress EnterNotify events, so that you don't
             * get "focus flicker" as windows shuffle around underneath the
             * pointer.
             */
            while(XCheckTypedEvent(dpy, MotionNotify, &ev));

            /* now we use the stuff we saved at the beginning of the
             * move/resize and compare it to the pointer's current position to
             * determine what the window's new size or position should be.
             *
             * if the initial button press was button 1, then we're moving.
             * otherwise it was 3 and we're resizing.
             *
             * we also make sure not to go negative with the window's
             * dimensions, resulting in "wrapping" which will make our window
             * something ridiculous like 65000 pixels wide (often accompanied
             * by lots of swapping and slowdown).
             *
             * even worse is if we get "lucky" and hit a width or height of
             * exactly zero, triggering an X error.  so we specify a minimum
             * width/height of 1 pixel.
             */
            xdiff = ev.xbutton.x_root - start.x_root;
            ydiff = ev.xbutton.y_root - start.y_root;
            XMoveResizeWindow(dpy, ev.xmotion.window,
                attr.x + (start.button==1 ? xdiff : 0),
                attr.y + (start.button==1 ? ydiff : 0),
                MAX(1, attr.width + (start.button==3 ? xdiff : 0)),
                MAX(1, attr.height + (start.button==3 ? ydiff : 0)));
        }
        /* like motion notifies, the only way we'll receive a button release is
         * during a move/resize, due to our pointer grab.  this ends the
         * move/resize.
         */
        else if(ev.type == ButtonRelease)
            XUngrabPointer(dpy, CurrentTime);
    }
}

And here’s tinywm.py. XCheckTypedEvent has no equivalent in python-xlib, so it is commented out. It doesn’t affect functionality, except that responsiveness is worse when you are moving/resizing (especially resizing a large window).

# TinyWM is written by Nick Welch <mack@incise.org>, 2005.
#
# This software is in the public domain
# and is provided AS IS, with NO WARRANTY.

from Xlib.display import Display
from Xlib import X, XK

dpy = Display()
root = dpy.screen().root

root.grab_key(XK.string_to_keysym("F1"), X.Mod1Mask, 1,
        X.GrabModeAsync, X.GrabModeAsync)
root.grab_button(1, X.Mod1Mask, 1, X.ButtonPressMask,
        X.GrabModeAsync, X.GrabModeAsync, X.NONE, X.NONE)
root.grab_button(3, X.Mod1Mask, 1, X.ButtonPressMask,
        X.GrabModeAsync, X.GrabModeAsync, X.NONE, X.NONE)

while 1:
    ev = root.display.next_event()

    if ev.type == X.KeyPress and ev.child != X.NONE:
        ev.window.circulate(X.RaiseLowest)
    elif ev.type == X.ButtonPress and ev.child != X.NONE:
        ev.child.grab_pointer(1, X.PointerMotionMask|X.ButtonReleaseMask,
                X.GrabModeAsync, X.GrabModeAsync, X.NONE, X.NONE, X.CurrentTime)
        attr = ev.child.get_geometry()
        start = ev
    elif ev.type == X.MotionNotify:
        #while(XCheckTypedEvent(dpy, MotionNotify, &ev));
        xdiff = ev.root_x - start.root_x
        ydiff = ev.root_y - start.root_y
        ev.window.configure(
            x = attr.x + (start.detail == 1 and xdiff or 0),
            y = attr.y + (start.detail == 1 and ydiff or 0),
            width = max(1, attr.width + (start.detail == 3 and xdiff or 0)),
            height = max(1, attr.height + (start.detail == 3 and ydiff or 0)))
    elif ev.type == X.ButtonRelease:
        dpy.ungrab_pointer(X.CurrentTime)

Two bit history

Two bit history

About
This is a blog about computer history intended primarily for computer people. While there is a lot of writing out there about the history of computing for a general audience, there is much less for a technical audience—which I think is a shame, because there are so many interesting historical questions that might only occur to somebody who designs and builds software every day, questions like Where did JSON come from? and Why are man pages still a thing?

My name is Sinclair Target. You can find more information about me here. If I’ve gotten something horrifically wrong, which is entirely possible, you can email me.

Now anyone can train Imagenet in 18 minutes

Now anyone can train Imagenet in 18 minutes

fast.ai
Making neural nets uncool again

Home
About
Our MOOC
Posts by Topic
© fast.ai 2018. All rights reserved.

Now anyone can train Imagenet in 18 minutes
Written: 10 Aug 2018 by Jeremy Howard
Note from Jeremy: I’ll be teaching Deep Learning for Coders at the University of San Francisco starting in October; if you’ve got at least a year of coding experience, you can apply here.

A team of fast.ai alum Andrew Shaw, DIU researcher Yaroslav Bulatov, and I have managed to train Imagenet to 93% accuracy in just 18 minutes, using 16 public AWS cloud instances, each with 8 NVIDIA V100 GPUs, running the fastai and PyTorch libraries. This is a new speed record for training Imagenet to this accuracy on publicly available infrastructure, and is 40% faster than Google’s DAWNBench record on their proprietary TPU Pod cluster. Our approach uses the same number of processing units as Google’s benchmark (128) and costs around $40 to run.

DIU and fast.ai will be releasing software to allow anyone to easily train and monitor their own distributed models on AWS, using the best practices developed in this project. The main training methods we used (details below) are: fast.ai’s progressive resizing for classification, and rectangular image validation; NVIDIA’s NCCL with PyTorch’s all-reduce; Tencent’s weight decay tuning; a variant of Google Brain’s dynamic batch sizes, gradual learning rate warm-up (Goyal et al 2018, and Leslie Smith 2018). We used the classic ResNet-50 architecture, and SGD with momentum.

Background
Four months ago, fast.ai (with some of the students from our MOOC and our in person course at the Data Instituate at USF) achieved great success in the DAWNBench competition, winning the competition for fastest training of CIFAR-10 (a small dataset of 25,000 32×32 pixel images) overall, and fastest training of Imagenet (a much larger dataset of over a million megapixel images) on a single machine (a standard AWS public cloud instance). We previously wrote about the approaches we used in this project. Google also put in a very strong showing, winning the overall Imagenet speed category using their “TPU Pod” cluster, which is not available to the public. Our single machine entry took around three hours, and Google’s cluster entry took around half an hour. Before this project, training ImageNet on the public cloud generally took a few days to complete.

We entered this competition because we wanted to show that you don’t have to have huge resources to be at the cutting edge of AI research, and we were quite successful in doing so. We particularly liked the headline from The Verge: “An AI speed test shows clever coders can still beat tech giants like Google and Intel.”

However, lots of people asked us – what would happen if you trained on multiple publicly available machines. Could you even beat Google’s impressive TPU Pod result? One of the people that asked this question was Yaroslav Bulatov, from DIU (the Department of Defense’s Silicon Valley-based experimental innovation unit.) Andrew Shaw (One of our DAWNBench team members) and I decided to team up with Yaroslav to see if we could achieve this very stretch goal. We were encouraged to see that recently AWS had managed to train Imagenet in just 47 minutes, and in their conclusion said: “A single Amazon EC2 P3 instance with 8 NVIDIA V100 GPUs can train ResNet50 with ImageNet data in about three hours [fast.ai] using Super-Convergence and other advanced optimization techniques. We believe we can further lower the time-to-train across a distributed configuration by applying similar techniques.” They were kind enough to share the code for this article as open source, where we found some helpful network configuration tips there to ensure we got the most out of the Linux networking stack and the AWS infrastructures.

Experiment infrastructure
Iterating quickly required solving challenges such as:

How to easily run multiple experiments across multiple machines, without having a large pool of expensive instances running constantly?
How to conveniently take advantage of AWS’s spot instances (which are around 70% cheaper than regular instances) but that need to be set up from scratch each time you want to use them?
fast.ai built a system for DAWNBench that included a Python API for launching and configuring new instances, running experiments, collecting results, and viewing progress. Some of the more interesting design decisions in the systems included:

Not to use a configuration file, but instead configuring experiments using code leveraging a Python API. As a result, we were able to use loops, conditionals, etc to quickly design and run structured experiments, such as hyper-parameter searches
Writing a Python API wrapper around tmux and ssh, and launching all setup and training tasks inside tmux sessions. This allowed us to later login to a machine and connect to the tmux session, to monitor its progress, fix problems, and so forth
Keeping everything as simple as possible – avoiding container technologies like Docker, or distributed compute systems like Horovod. We did not use a complex cluster architecture with separate parameter servers, storage arrays, cluster management nodes, etc, but just a single instance type with regular EBS storage volumes.
Independently, DIU faced a similar set of challenges and developed a cluster framework, with analogous motivation and design choices, providing the ability to run many large scale training experiments in parallel. The solution, nexus-scheduler, was inspired by Yaroslav’s experience running machine learning experiments on Google’s Borg system.

The set of tools developed by fast.ai focused on fast iteration with single-instance experiments, whilst the nexus-scheduler developed by DIU was focused on robustness and multi-machine experiments. Andrew Shaw merged parts of the fast.ai software into nexus-scheduler, so that we had the best pieces of each, and we used this for our experiments.

Using nexus-scheduler helped us iterate on distributed experiments, such as:

Launching multiple machines for a single experiment, to allow distributed training. The machines for a distributed run are automatically put into a placement group, which results in faster network performance
Providing monitoring through Tensorboard (a system originally written for Tensorflow, but which now works with Pytorch and other libraries) with event files and checkpoints stored on a region-wide file system.
Automating setup. Various necessary resources for distributed training, like VPCs, security groups, and EFS are transparently created behind the scenes.
AWS provides a really useful API that allowed us to build everything we needed quickly and easily. For distributed computation we used NVIDIA’s excellent NCCL library, which implements ring-style collectives that are integrated with PyTorch’s all-reduce distributed module. We found that AWS’s instances were very reliable and provided consistent performance, which was very important for getting best results from the all-reduce algorithm.

The first official release of nexus-scheduler, including the features merged from the fast.ai tools, is planned for Aug 25th.

Analyzing network utilization using Tensorboard
Analyzing network utilization using Tensorboard
A simple new training trick: rectangles!
I mentioned to Andrew after DAWNBench finished that I thought deep learning practitioners (including us!) were doing something really dumb: we were taking rectangular images (such as those used in Imagenet) and cropping out just the center piece when making predictions. Or a (very slow) alternative widely used is to pick 5 crops (top and bottom left and right, plus center) and average the predictions. Which leaves the obvious question: why not just use the rectangular image directly?

A lot of people mistakenly believe that convolutional neural networks (CNNs) can only work with one fixed image size, and that that must be rectangular. However, most libraries support “adaptive” or “global” pooling layers, which entirely avoid this limitation. It doesn’t help that some libraries (such as Pytorch) distribute models that do not use this feature – it means that unless users of these libraries replace those layers, they are stuck with just one image size and shape (generally 224×224 pixels). The fastai library automatically converts fixed-size models to dynamically sized models.

I’ve never seen anyone try to train with rectangular images before, and haven’t seen them mentioned in any research papers yet, and none of the standard deep learning libraries I’ve found support this. So Andrew went away and figured out how to make it work with fastai and Pytorch for predictions.

The result was amazing – we saw an immediate speedup of 23% in the amount of time it took to reach the benchmark accuracy of 93%. You can see a comparison of the different approaches in this notebook, and compare the accuracy of them in this notebook.

Snippet of the Jupyter Notebook comparing different cropping approaches. ‘Center Crop Image’ is the original photo, ‘FastAi rectangular’ is our new method, ‘Imagenet Center’ is the standard approach, and ‘Test Time Augmentation’ is an example from the multi-crop approach.
Snippet of the Jupyter Notebook comparing different cropping approaches. ‘Center Crop Image’ is the original photo, ‘FastAi rectangular’ is our new method, ‘Imagenet Center’ is the standard approach, and ‘Test Time Augmentation’ is an example from the multi-crop approach.
Progressive resizing, dynamic batch sizes, and more
One of our main advances in DAWNBench was to introduce progressive image resizing for classification – using small images at the start of training, and gradually increasing size as training progresses. That way, when the model is very inaccurate early on, it can quickly see lots of images and make rapid progress, and later in training it can see larger images to learn about more fine-grained distinctions.

In this new work, we additionally used larger batch sizes for some of the intermediate epochs – this allowed us to better utilize the GPU RAM and avoid network latency.

Recently, Tencent published a very nice paper showing 1 million images are significant, such as:

Organizations with large image libraries, such as radiology centers, car insurance companies, real estate listing services, and e-commerce sites, can now create their own customized models. Whilst with transfer learning using so many images is often overkill, for highly specialized image types or fine-grained classification (as is common in medical imaging) using larger volumes of data may give even better results
Smaller research labs can experiment with different architectures, loss functions, optimizers, and so forth, and test on Imagenet, which many reviewers expect to see in published papers
By allowing the use of standard public cloud infrastructure, no up-front capital expense is required to get started on cutting-edge deep learning research.
Next steps
Unfortunately, big companies using big compute tend to get far more than their fair share of publicity. This can lead to AI commentators coming to the conclusion that only big companies can compete in the most important AI research. For instance, following Tencent’s recent paper, OpenAI’s Jack Clark claimed “for all the industries talk about democratization it’s really the case that advantages accrue to people with big computers”. OpenAI (and Jack Clark) are working to democratize AI, such as through the excellent OpenAI Scholars program and Jack’s informative Import AI newsletter; however, this misperception that success in AI comes down to having bigger computers can distort the research agenda.

I’ve seen variants of the “big results need big compute” claim continuously over the last 25 years. It’s never been true, and there’s no reason that will change. Very few of the interesting ideas we use today were created thanks to people with the biggest computers. Ideas like batchnorm, ReLU, dropout, adam/adamw, and LSTM were all created without any need for large compute infrastructure. And today, anyone can access massive compute infrastructure on demand, and pay for just what they need. Making deep learning more accessible has a far higher impact than focusing on enabling the largest organizations – because then we can use the combined smarts of millions of people all over the world, rather than being limited to a small homogeneous group clustered in a couple of geographic centers.

We’re in the process of incorporating all of the best practices used in this project directly into the fastai library, including automating the selection of hyper-parameters for fast and accurate training.

And we’re not even done yet – we have some ideas for further simple optimizations which we’ll be trying out. We don’t know if we’ll hit the sub seven minute mark of Tencent, since they used a faster network than AWS provides, but there’s certainly plenty of room to go faster still.

Special thanks to the AWS and PyTorch teams who helped us by patiently answering our questions throughout this project, and for the wonderfully pragmatic products that they’ve made available for everyone to use!

This post is tagged: [ technical ] (click a tag for more posts in that category).
Related Posts
What HBR Gets Wrong About Algorithms and Bias 07 Aug 2018
Google’s AutoML: Cutting Through the Hype 23 Jul 2018
An Opinionated Introduction to AutoML and Neural Architecture Search 16 Jul 2018

Hacker Finds Hidden ‘God Mode’ on Old x86 CPUs

Hacker Finds Hidden ‘God Mode’ on Old x86 CPUs

Hacker Finds Hidden ‘God Mode’ on Old x86 CPUs
Paul Wagenseil | Aug 9, 2018 5:06 pm ET

LAS VEGAS — Some x86 CPUs have hidden backdoors that let you seize root by sending a command to an undocumented RISC core that manages the main CPU, security researcher Christopher Domas told the Black Hat conference here Thursday (Aug. 9).

The command — “.byte 0x0f, 0x3f” in Linux — “isn’t supposed to exist, doesn’t have a name, and gives you root right away,” Domas said, adding that he calls it “God Mode.”

The backdoor completely breaks the protection-ring model of operating-system security, in which the OS kernel runs in ring 0, device drivers run in rings 1 and 2, and user applications and interfaces (“userland”) run in ring 3, furthest from the kernel and with the least privileges. To put it simply, Domas’ God Mode takes you from the outermost to the innermost ring in four bytes.

“We have direct ring 3 to ring 0 hardware privilege escalation,” Domas said. “This has never been done.”

That’s because of the hidden RISC chip, which lives so far down on the bare metal that Domas half-joked that it ought to be thought of as a new, deeper ring of privilege, following the theory that hypervisors and chip-management systems can be considered ring -1 or ring -2.

“This is really ring -4,” he said. “It’s a secret, co-located core buried alongside the x86 chip. It has unrestricted access to the x86.”

The good news is that, as far as Domas knows, this backdoor exists only on VIA C3 Nehemiah chips made in 2003 and used in embedded systems and thin clients. The bad news is that it’s entirely possible that such hidden backdoors exist on many other chipsets.

“These black boxes that we’re trusting are things that we have no way to look into,” he said. “These backdoors probably exist elsewhere.”

Domas discovered the backdoor, which exists on VIA C3 Nehemiah chips made in 2003, by combing through filed patents. He found one — US8341419 — that mentioned jumping from ring 3 to ring 0 and protecting the machine from exploits of model-specific registers (MSRs), manufacturer-created commands that are often limited to certain chipsets.

Domas followed the “trail of breadcrumbs,” as he put it, from one patent to another and figured out that certain VIA chipsets were covered by the patents. Then he collected many old VIA C3 machines and spent weeks fuzzing code.

He even built a testing rig consisting of seven Nehemiah-based thin clients hooked up to a power relay that would power-cycle the machines every couple of minutes, because his fuzzing attempts would usually crash the systems. After three weeks, he had 15 GB of log data — and the instructions to flip on the backdoor in the hidden RISC chip.

“Fortunately, we still need ring 0 access to start the launch process, right?” Domas asked. “No. Some of the VIA C3 x86 processors have God Mode enabled by default. You can reach it from userland. Antivirus software, ASLR and all the other security mitigations are useless.”

Domas has put all his research, plus tools to check whether your VIA C3 CPU might have an undocumented coprocessor and to disable the coprocessor by default, up on his GitHub page at https://github.com/xoreaxeaxeax/rosenbridge.

Author Bio
Paul Wagenseil, Senior editor, security, privacy and gaming
Paul Wagenseil is a senior editor at Tom’s Guide focused on security and privacy. That’s all he’s going to tell you unless you meet him in person.

Paul Wagenseil, Senior editor, security, privacy and gaming on
FOLLOW US

Copyright © 2018
All Rights Reserved.