DEC64: Decimal Floating Point

DEC64: Decimal Floating Point

DEC64 is a number type. It can precisely represent decimal fractions with 16 decimal places, which makes it very well suited to all applications that are concerned with money. It can represent values as gargantuan as 3.6028797018963967E+143 or as measly as 1.0E-127, which makes it well suited to most scientific applications. It can provide very fast performance on integer values, eliminating the need for a separate int type and avoiding the terrible errors than can result from int truncation.

DEC64 is intended to be the only number type in the next generation of application programming languages.

DEC64 represents numbers as 64 bit values composed of 2 two’s complement components: a 56 bit coefficient and an 8 bit exponent. The coefficient is in the high order end, and the exponent is in the low order end. The coefficient’s decimal point is between bits 8 and 7.

63 8 7 0
coefficient exponent
The coefficient is in the range -36028797018963968 .. 36028797018963967. The exponent is in the range -127 .. 127. Numbers may not use an exponent containing -128. The value of a number is obtained from this formula:

value = coefficient * 10exponent
Normalization is not required, and is usually not desired. Integers can have an exponent of 0 as long as the coefficient is less than 36 quadrillion. Addition of numbers with equal exponents could be performed in a single machine cycle.

There are 255 possible representations of zero. They are all considered to be equal.

There is a special value called nan that has a coefficient of 0 and an exponent of -128. The result of division by zero is nan. nan is also the result of operations that produce results that are too large to be represented. nan is equal to itself.

When an arithmetic operation has an input with an exponent of -128, the result will be nan. Applications are free to use the coefficient as they wish when the exponent is -128, since in that case the coefficient has no arithmetic significance. One possible use is to store object pointers in the coefficient.

DEC64 can be implemented efficiently in hardware or software.

Conversion to and from textual representations is simple and straightforward and free of the complexities that binary floating formats must wrestle with to minimize the inevitable errors caused by the fundamental incompatibility of the binary and decimal systems. DEC64 instead uses an internal representation that is very compatible with the E notation.

To convert an int to DEC64, shift it left 8 bits. To unpack a coefficient, shift it right 8 bits with sign extension. The exponent can be unpacked at no cost on x64 architecture because the least significant byte can be accessed directly.

There is a fast path for addition of integers in software that takes only 5 instructions while also providing for not-a-number and overflow protection.

; Add rdx to rax.

mov cl,al ; load the exponent of rax into cl
or cl,dl ; then or the two exponents together
jnz slow_path ; if both are zero, take the fast path
add rax,rdx ; add the coefficients together
jo overflow ; if there was no overflow, we are done
The fast path for addition in hardware should take only 1 cycle when the two exponents are equal to each other and there is no overflow. The fast path for multiplication in hardware takes the time it takes to do a 56*56 signed multiply when there is no overflow.

A reference implementation is available at It provides the DEC64 elementary operations.

Conversion between DEC64 and strings is trivially easy. This is demonstrated by dec64_string.

The elementary functions (sine, log, sqrt, etc) are demonstrated by dec64_math.

The idea of using powers of ten instead of powers of two is not new. For example,

Floating point subroutines and interpretive systems for early machines were coded by D. J. Wheeler and others, and the first publication of such routines was in The Preparation of Programs for an Electronic Digital Computer by Wilkes, Wheeler, and Gill (Reading, Mass.: Addison-Wesley, 1951), subroutines A1-A11, pages 35-37 and 105-117. It is interesting to note that floating decimal subroutines are described here, although a binary computer was being used; in other words, the numbers were represented as 10ef, not 2ef, and therefore the scaling operations required multiplication or division by 10.

The Art of Computer Programming, Volume 2: Seminumerical Algorithms, Third Edition by Donald Knuth (Addison-Wesley, 1998), page 226.

The book Knuth cited may have been the first software book. It described some of the libraries and conventions of Maurice Wilkes’s EDSAC, one of the first generation of Von Neumann machines. Some of its subroutines used a numeric format that was very similar to DEC64.

Floating point was so important that support for it was moved into hardware for better performance. This led to the development of binary floating point because a shift could be implemented much more easily than a divide by 10. It was discovered that by biasing the exponent and moving it to the position just after the sign bit that floating point numbers could be compared with integer opcodes, a nifty optimization. It was also discovered that because normalization always left a 1 bit in the most significant position of the significand, that that bit could be omitted, providing an additional bit of significance.

The Burroughs 5000 series had a floating point format in which an exponent of zero allowed the mantissa to be treated as an ordinary integer. DEC64 incorporates that brilliant idea.

Languages for scientific computing like FORTRAN provided multiple floating point types such as REAL and DOUBLE PRECISION as well as INTEGER, often also in multiple sizes. This was to allow programmers to reduce program size and running time. This convention was adopted by later languages like C and Java. In modern systems, this sort of memory saving is pointless. By giving programmers a choice of number types, programmers are required to waste their time making choices that don’t matter. Even worse, making a bad choice can lead to a loss of accuracy or destructive bugs. This is a bad practice that is very deeply ingrained.

Binary floating point trades away familiarity and decimal compatibility for performance. This made it unsuitable for business languages like COBOL. Decimal fractions cannot be represented accurately in binary floating point, which is a problem for programs that interact with humans, and is dangerous in programs that manipulate money. Exactness is required, so most business processing used BCD (Binary Coded Decimal) in which each digit is encoded in 4 bits. That created some inefficiency, but benefited from allowing a shift by 4 bits in place of the more complex divide by 10. For a time, mainframes could be ordered with optional floating point units for scientific computing, and optional BCD units for business computing.

The BASIC language eliminated much of the complexity of FORTRAN by having a single number type. This simplified the programming model and avoided a class of errors caused by selection of the wrong type. The efficiencies that could have gained from having numerous number types proved to be insignificant.

Business Basic was a dialect of BASIC that was developed by Basic/Four Corporation for its small business minicomputers. It used decimal floating point, much like the EDSAC, so the language could be used for both scientific and business applications. Business Basic could do everything that BASIC could do, and it could also handle money.

Intel undertook an ambitious architecture that was marketed as iAPX432. A decimal number type was considered for the 432 but rejected in favor of conventional binary floating point. The 432 failed, but its floating point unit was salvaged and repackaged as the 8087, a numeric coprocessor for the 8086. The 8087’s number types became the basis of the IEEE 754 floating point standard. The standard was so successful that it destroyed the market for decimal computing. A later revision of IEEE 754 attempted to remedy this, but the formats it recommended were so inefficient that it has not found much acceptance. DEC64 is a better alternative.


John Regehr, Professor of Computer Science, University of Utah, USA

John Regehr, Professor of Computer Science, University of Utah, USA


John Regehr, Professor of Computer Science, University of Utah, USA

Undefined Behavior in 2017
This post is jointly authored by Pascal Cuoq and John Regehr.

Recently we’ve heard a few people imply that problems stemming from undefined behaviors (UB) in C and C++ are largely solved due to ubiquitous availability of dynamic checking tools such as ASan, UBSan, MSan, and TSan. We are here to state the obvious — that, despite the many excellent advances in tooling over the last few years, UB-related problems are far from solved — and to look at the current situation in detail.

Valgrind and most of the sanitizers are intended for debugging: emitting friendly diagnostics regarding undefined behaviors that are executed during testing. Tools like this are exceptionally useful and they have helped us progress from a world where almost every nontrivial C and C++ program executed a continuous stream of UB to a world where quite a few important programs seem to be largely UB-free in their most common configurations and use cases.

The problem with dynamic debugging tools is that they don’t do anything to help us to cope with the worst UBs: the ones that we didn’t know how to trigger during testing, but that someone else has figured out how to trigger in deployed software — while exploiting it. The problem reduces to doing good testing, which is hard. Tools like afl-fuzz are great but they barely begin to scratch the surface of large programs that process highly structured inputs.

One way to sidestep problems in testing is to use static UB-detection tools. These are steadily improving, but sound and precise static analysis is not necessarily any easier than achieving good test coverage. Of course the two techniques are attacking the same problem — identifying feasible paths in software — from opposite sides. This problem has always been extremely hard and probably always will be. We’ve written a lot elsewhere about finding UBs via static analysis; in this piece our focus is on dynamic tools.

The other way to work around problems in testing is to use UB mitigation tools: these turn UB into defined behavior in production C and C++, effectively gaining some of the benefits of a safe programming language. The challenge is in engineering mitigation tools that:

don’t break our code in any corner cases,
have very low overhead,
don’t add effective attack surfaces, for example by requiring programs to be linked against a non-hardened runtime library,
raise the bar for determined attackers (in contrast, debugging tools can afford to use heuristics that aren’t resistant to adversaries),
compose with each other (in contrast, some debugging tools such as ASan and TSan are not compatible, necessitating two runs of the test suite for any project that wants to use both).
Before looking at some individual kinds of UB, let’s review the our goals here. These apply to every C and C++ compiler.

Goal 1: Every UB (yes, all ~200 of them, we’ll give the list towards the end of this post) must either be documented as having some defined behavior, be diagnosed with a fatal compiler error, or else — as a last resort — have a sanitizer that detects that UB at runtime. This should not be controversial, it’s sort of a minimal requirement for developing C and C++ in the modern world where network packets and compiler optimizations are effectively hostile.

Goal 2: Every UB must either be documented as having some defined behavior, be diagnosed with a fatal compiler error, or else have an optional mitigation mechanism that meets the requirements above. This is more difficult; it necessitates, for example, production-grade memory safety. We like to think that this can be achieved in many execution environments. OS kernels and other maximally performance-critical code will need to resort to more difficult technologies such as formal methods.

The rest of this piece will look at the current situation for various classes of undefined behaviors. We’ll start with the big ones.

Spatial Memory Safety Violations
Background: Accessing out-of-bounds storage and even creating pointers to that storage are UB in C and C++. The 1988 Morris Worm gave us an early hint of what the next N years would be like. So far we know that N >= 29, and probably N will end up being about 75.

Debugging: Valgrind and ASan are both excellent debugging tools. For many use cases ASan is the better choice because it has much less overhead. Both tools retain the representation of addresses as 32- or 64-bit values, and reserve forbidden red zones around valid blocks. This is a robust and compatible approach: it interoperates seamlessly with non-instrumented binary libraries and also supports existing code that relies on pointers being convertible to integers.

Valgrind, working from executable code, cannot insert red zones between stack variables because stack layout is implicitly hard-coded in the offsets of instructions that access the stack, and it would be an impossibly ambitious project to remap stack addresses on the fly. As a result, Valgrind has only limited support for detecting errors in manipulating storage on the stack. ASan works during compilation and inserts red zones around stack variables. Stack variables are small and numerous, so address space and locality considerations prevent the use of very large red zones. With default settings, the addresses of two adjacent local int variables x and y end up separated by 16 bytes. In other words, the verifications done by ASan and Valgrind are only for one memory layout, and the memory layout for which the verifications are done is different from the memory layout of the uninstrumented execution.

A minor weakness of ASan and Valgrind is that they can miss undefined behaviors that get optimized away before the instrumentation has a chance to run, as in this example.

Mitigation: We’ve long had partial mitigation mechanisms for memory unsafety, including ASLR, stack canaries, hardened allocators, and NX. More recently, production-grade CFI (control flow integrity) has become available. Another interesting recent development is pointer authentication in ARMv8.3. This paper has a good overview of memory safety mitigations.

A serious drawback of ASan as a mitigation tool is illustrated here:

$ cat asan-defeat.c

char a[128];
char b[128];

int main(int argc, char *argv[]) {
strcpy(a + atoi(argv[1]), “owned.”);
printf(“%s\n”, b);
return 0;
$ clang-4.0 -O asan-defeat.c
$ ./a.out 128
$ clang-4.0 -O -fsanitize=address -fno-common asan-defeat.c
$ ./a.out 160
In other words, ASan simply forces an attacker to compute a different offset in order to corrupt a target memory region. (Thanks to Yury Gribov for pointing out that we should be using the -fno-common flag to ASan.)

To mitigate this kind of undefined behavior, real bounds checking must be performed, as opposed to only verifying that each memory access lands in some valid region. Memory safety is the gold standard here. Although there is much academic work on memory safety, some showing apparently reasonable overheads and good compatibility with existing software, it has not yet seen widespread adoption. Checked C is a very cool project to keep an eye on in this space.

Summary: Debugging tools for this class of error are very good. Good mitigations are available but this class of bug can only be reliably stopped by full memory/type safety.

Temporal Memory Safety Violations
Background: A “temporal memory safety violation” is any use of a memory location after its lifetime has ended. This includes addresses of automatic variables outliving these variables; use-after-free, where a dangling pointer is accessed for reading or writing; and, double free, which can be just as harmful in practice, since free() modifies metadata that is usually adjacent to the block being freed. If the block has already been freed, these writes can fall on memory used for any other purpose and, in principle, can have as much consequence as any other invalid write.

Debugging: ASan is designed to detect use-after-free bugs, which often lead to hard-to-reproduce, erratic behavior. It does so by placing freed memory blocks in a quarantine, preventing their immediate reuse. For some programs and inputs, this can increase memory consumption and decrease locality. The user can configure the size of the quarantine in order to trade false positives for resource usage.

ASan can also detect addresses of automatic variables surviving the scope of these variables. The idea is to turn automatic variables into heap-allocated blocks, that the compiler automatically allocates when execution enters the block, and frees (while retaining them in a quarantine) when execution leaves the block. This option is turned off by default, because it makes programs even more memory-hungry.

The temporal memory safety violation in the program below causes it to behave differently at the default optimization level and at -O2. ASan can detect a problem in the program below with no optimization, but only if the option detect_stack_use_after_return is set, and only if the program was not compiled with optimization.

$ cat temporal.c

int *G;

int f(void) {
int l = 1;
int res = *G;
G = &l;
return res;

int main(void) {
int x = 2;
G = &x;
printf(“%d\n”, f());
$ clang -Wall -fsanitize=address temporal.c
$ ./a.out
$ ASAN_OPTIONS=detect_stack_use_after_return=1 ./a.out
==5425==ERROR: AddressSanitizer: stack-use-after-return …
READ of size 4 at 0x0001035b6060 thread T0
$ clang -Wall -fsanitize=address -O2 temporal.c
$ ./a.out
$ ASAN_OPTIONS=detect_stack_use_after_return=1 ./a.out
$ clang -v
Apple LLVM version 8.0.0 (clang-800.0.42.1)

In some other examples, the sanitizer’s failure to detect UB that has been “optimized out” can be argued to be harmless, since the optimized-out UB has no consequence. This is not the case here! The program is meaningless in any case, but the unoptimized program behaves deterministically and works as if the variable x had been declared static, whereas the optimized program, in which ASan does not detect any foul play, does not behave deterministically and reveals an internal state that is not supposed to be seen:

$ clang -Wall -O2 temporal.c
$ ./a.out
$ ./a.out
$ ./a.out
Mitigation: As discussed above, ASan is not intended for hardening, but various hardened allocators are available; they use the same quarantining strategy to render use-after-free bugs unexploitable.

Summary: Use ASan (together with “ASAN_OPTIONS=detect_stack_use_after_return=1” for the test cases that are small enough to allow it). Vary optimization levels in case some compilations catch errors that others don’t.

Integer Overflow
Background: Integers cannot underflow, but they can overflow in both directions. Signed integer overflow is UB; this includes INT_MIN / -1, INT_MIN % -1, negating INT_MIN, shift with negative exponent, left-shifting a one past the sign bit, and (sometimes) left-shifting a one into the sign bit. Division by zero and shift by >= bitwidth are UB in both the signed and unsigned flavors. Read more here.

Debugging: LLVM’s UBSan is very good for debugging integer-related undefined behaviors. Because UBSan works near the source level, it is highly reliable. There are some quirks relating to compile-time math; for example, this program traps as C++11 but not as C11; we believe this follows the standards but haven’t looked into it closely. GCC has its own version of UBSan but it isn’t 100% trustworthy; here it looks like constants are being folded before the instrumentation pass gets to run.

Mitigation: UBSan in trapping mode (on hitting UB, process aborts w/o printing a diagnostic) can be used for mitigation. It is usually reasonably efficient and it doesn’t add attack surface. Parts of Android use UBSan to mitigate integer overflows (including unsigned overflows, which of course are not undefined). Although integer overflows are generic logic errors, in C and C++ they are particularly harmful because they often lead to memory safety violations. In a memory-safe language they tend to do much less damage.

Summary: Integer undefined behaviors are not very difficult to catch; UBSan is the only debugging tool you’re likely to ever need. An issue with mitigating integer UBs is the overhead. For example, they cause SPEC CPU 2006 to run about 30% slower. There is plenty of room for improvement, both in eliminating overflow checks that cannot fire and in making the remaining checks less obstructive to the loop optimizers. Someone with resources should push on this.

Strict Aliasing Violations
Background: The “strict aliasing rules” in the C and C++ standards allow the compiler to assume that if two pointers refer to different types, they cannot point to the same storage. This enables nice optimizations but risks breaking programs that take a flexible view of types (roughly 100% of large C and C++ programs take a flexible view of types somewhere). For a thorough overview see Sections 1-3 of this paper.

Debugging: The state of the art in debugging tools for strict aliasing violations is weak. Compilers warn about some easy cases, but these warnings are extremely fragile. libcrunch warns that a pointer is being converted to a type “pointer to thing” when the pointed object is not, in fact, a “thing.” This allows polymorphism though void pointers, but catches misuses of pointer conversions that are also strict aliasing violations. With respect to the C standard and C compilers’ interpretation of what it allows them to optimize in their type-based alias analyses, however, libcrunch is neither sound (it does not detect some violations that happen during the instrumented execution) nor complete (it warns about pointer conversions that smell bad but do not violate the standard).

Mitigation: This is easy: pass the compiler a flag (-fno-strict-aliasing) that disables optimizations based on strict aliasing. The result is a C/C++ compiler that has an old-school memory model where more or less arbitrary casts between pointer types can be performed, with the resulting code behaving as expected. Of the big three compilers, it is only LLVM and GCC that are affected, MSVC doesn’t implement this class of optimization in the first place.

Summary: Correctness-sensitive code bases need significant auditing: it is always suspicious and dangerous to cast a pointer to any type other than a char *. Alternatively, just turn off strict-aliasing-based optimizations using a flag and make sure that nobody ever builds the code without using this flag.

Alignment Violations
Background: RISC-style processors have tended to disallow memory accesses where the address is not a multiple of the size of the object being accessed. On the other hand, C and C++ programs that use unaligned pointers are undefined regardless of the target architecture. Historically we have been complacent about this, first because x86/x64 support unaligned accesses and second because compilers have so far not done much to exploit this UB.

Even so, here is an excellent blog post explaining how the compiler can break code that does unaligned accesses when targeting x64. The code in the post violates strict aliasing in addition to violating the alignment rules, but the crash (we verified it under GCC 7.1.0 on OS X) occurs even when the -fno-strict-aliasing flag is passed to the compiler.

Debugging: UBSan can detect misaligned memory accesses.

Mitigation: None known.

Summary: Use UBSan.

Loops that Neither Perform I/O nor Terminate
Background: A loop in C or C++ code that neither performs I/O nor terminates is undefined and can be terminated arbitrarily by the compiler. See this post and this note.

Debugging: No tools exist.

Mitigation: None, besides avoiding heavily-optimizing compilers.

Summary: This UB is probably not a problem in practice (even if it is moderately displeasing to some of us).

Data Races
Background: A data race occurs when a piece of memory is accessed by more than one thread, at least one of the accesses is a store, and the accesses are not synchronized using a mechanism such as a lock. Data races are UB in modern flavors of C and C++ (they do not have a semantics in older versions since those standards do not address multithreaded code).

Debugging: TSan is an excellent dynamic data race detector. Other similar tools exist, such as the Helgrind plugin for Valgrind, but we have not used these lately. The use of dynamic race detectors is complicated by the fact that races can be very difficult to trigger, and worse this difficulty depends on variables such as the number of cores, the thread scheduling algorithm, whatever else is going on on the test machine, and on the moon’s phase.

Mitigation: Don’t create threads.

Summary: This particular UB is probably a good idea: it clearly communicates the idea that developers should not count on racy code doing anything in particular, but should rather use atomics (that cannot race by definition) if they don’t enjoy locking.

Unsequenced Modifications
Background: In C, “sequence points” constrain how early or late a side-effecting expression such as x++ can take effect. C++ has a different but more-or-less-equivalent formulation of these rules. In either language, unsequenced modifications of the same value, or an unsequenced modification and use of the same value, results in UB.

Debugging: Some compilers emit warnings for obvious violations of the sequencing rules:

$ cat unsequenced2.c
int a;

int foo(void) {
return a++ – a++;
$ clang -c unsequenced2.c
unsequenced2.c:4:11: warning: multiple unsequenced modifications to ‘a’ [-Wunsequenced]
return a++ – a++;
^ ~~
1 warning generated.
$ gcc-7 -c unsequenced2.c -Wall
unsequenced2.c: In function ‘foo’:
unsequenced2.c:4:11: warning: operation on ‘a’ may be undefined [-Wsequence-point]
return a++ – a++;
However, a bit of indirection defeats these warnings:

$ cat unsequenced.c

int main(void) {
int z = 0, *p = &z;
*p += z++;
printf(“%d\n”, z);
return 0;
$ gcc-4.8 -Wall unsequenced.c ; ./a.out
$ gcc-7 -Wall unsequenced.c ; ./a.out
$ clang -Wall unsequenced.c ; ./a.out
Mitigation: None known, though it would be almost trivial to define the order in which side effects take place. The Java Language Definition provides an example of how to do this. We have a hard time believing that this kind of constraint would meaningfully handicap any modern optimizing compiler. If the standards committees can’t find it within their hearts to make this happen, the compiler implementors should do it anyway. Ideally, all major compilers would make the same choice.

Summary: With a bit of practice, it is not too difficult to spot the potential for unsequenced accesses during code reviews. We should be wary of any overly-complex expression that has many side effects. This leaves us without a good story for legacy code, but hey it has worked until now, so perhaps there’s no problem. But really, this should be fixed in the compilers.

A non-UB relative of unsequenced is “indeterminately sequenced” where operations may happen in an order chosen by the compiler. An example is the order of the first two function calls while evaluating f(a(), b()). This order should be specified too. Left-to-right would work. Again, there will be no performance loss in non-insane circumstances.

TIS Interpreter
We now change gears and take a look at the approach taken by TIS Interpreter, a debugging tool that looks for undefined behavior in C programs as it executes them line by line. TIS Interpreter runs programs much more slowly than the LLVM-based sanitizers, and even much more slowly than Valgrind. However, TIS Interpreter can usefully be compared to these sanitizers: it works from the source code, leaves the problem of coverage to test suites and fuzzing tools, and identifies problems along the execution paths that it has been provided inputs for.

A fundamental difference between TIS Interpreter and any single sanitizer is that TIS Interpreter’s goal is, along the execution paths it explores, to be exhaustive: to find all the problems that ASan, MSan, and UBSan are designed to find some of (give or take a couple of minor exceptions that we would be delighted to discuss at great length if provoked). For example, TIS Interpreter identifies unsequenced changes to overlapping memory zones within an expression, such as (*p)++ + (*q)++ when the pointers p and q alias. The problem of the unspecified order of function calls in a same expression, that TIS Interpreter orders without warning when a different order could produce a different result, is a known limitation that will eventually be fixed.

TIS Interpreter’s approach to detecting memory safety errors differs sharply from ASan’s and Valgrind’s in that it doesn’t find errors for a specific heap layout, but rather treats as an error any construct that could lead the execution to behave differently depending on memory layout choices. In other words, TIS Interpreter has a symbolic view of addresses, as opposed to the concrete view taken by Valgrind and ASan. This design choice eliminates the “only the instrumented version of the program is safe, and the instrumented version behaves differently from the deployed version” problem. The occasional C program is written to behave differently depending on the memory layout (for instance if addresses are fed to hash functions or used to provide a total ordering between allocated values). TIS Analyzer warns that these programs are doing this (which is always good to know); sometimes, tweaks make it possible to analyze them in TIS Interpreter anyway, but the resulting guarantees will be weaker.

It is sometimes useful, for debugging purposes, to see the first UB that occurs in an execution. Consider a loop in which MSan warns that uninitialized memory is being used, and in which ASan warns about an out-of-bounds read. Is the out-of-bounds read caused by the incorporation of uninitialized memory in the computation of the index, or is the use of uninitialized memory caused by the index being computed wrongly? One cannot use both ASan and MSan at the same time, so this is a mystery that developers need to solve for themselves. The value of looking for all undefined behaviors at the same time is in this case the confidence that the first undefined behavior seen is not a symptom of a previous undefined behavior. Another advantage is finding undefined behavior that one was not looking for.

Detection of strict aliasing violations in TIS Interpreter is being worked on, following as much as possible the C standard and the interpretation of C compiler designers (which can be observed in each compiler’s translation of well-chosen examples).

But What About the Rest of the Undefined Behaviors?
Let’s take a quick look at the contents of Appendix J.2: a non-normative, non-exhaustive list of undefined behaviors in C. Keep in mind that no equivalent list has ever been created for C++, as far as we know.

First, we’ll list the UBs that we’ve discussed explicitly in this post:

The execution of a program contains a data race (
An object is referred to outside of its lifetime (6.2.4).
The value of a pointer to an object whose lifetime has ended is used (6.2.4).
The value of an object with automatic storage duration is used while it is indeterminate (6.2.4, 6.7.9, 6.8).
Conversion to or from an integer type produces a value outside the range that can be represented (
An lvalue does not designate an object when evaluated (
Conversion between two pointer types produces a result that is incorrectly aligned (
A side effect on a scalar object is unsequenced relative to either a different side effect on the same scalar object or a value computation using the value of the same scalar object (6.5).
An exceptional condition occurs during the evaluation of an expression (6.5).
An object has its stored value accessed other than by an lvalue of an allowable type (6.5).
The operand of the unary * operator has an invalid value (
The value of the second operand of the / or % operator is zero (6.5.5).
Addition or subtraction of a pointer into, or just beyond, an array object and an integer type produces a result that does not point into, or just beyond, the same array object (6.5.6).
Addition or subtraction of a pointer into, or just beyond, an array object and an integer type produces a result that points just beyond the array object and is used as the operand of a unary * operator that is evaluated (6.5.6).
Pointers that do not point into, or just beyond, the same array object are subtracted (6.5.6).
An array subscript is out of range, even if an object is apparently accessible with the given subscript (as in the lvalue expression a[1][7] given the declaration int a[4][5]) (6.5.6).
The result of subtracting two pointers is not representable in an object of type ptrdiff_t (6.5.6).
An expression is shifted by a negative number or by an amount greater than or equal to the width of the promoted expression (6.5.7).
An expression having signed promoted type is left-shifted and either the value of the expression is negative or the result of shifting would be not be representable in the promoted type (6.5.7).
Pointers that do not point to the same aggregate or union (nor just beyond the same array object) are compared using relational operators (6.5.8).
An object is assigned to an inexactly overlapping object or to an exactly overlapping object with incompatible type (
And second, those that we have not addressed:

A ‘‘shall” or ‘‘shall not” requirement that appears outside of a constraint is violated (clause 4).
A nonempty source file does not end in a new-line character which is not immediately preceded by a backslash character or ends in a partial preprocessing token or comment (
Token concatenation produces a character sequence matching the syntax of a universal character name (
A program in a hosted environment does not define a function named main using one of the specified forms (
A character not in the basic source character set is encountered in a source file, except in an identifier, a character constant, a string literal, a header name, a comment, or a preprocessing token that is never converted to a token (5.2.1).
An identifier, comment, string literal, character constant, or header name contains an invalid multibyte character or does not begin and end in the initial shift state (
The same identifier has both internal and external linkage in the same translation unit (6.2.2).
A trap representation is read by an lvalue expression that does not have character type (
A trap representation is produced by a side effect that modifies any part of the object using an lvalue expression that does not have character type (
The operands to certain operators are such that they could produce a negative zero result, but the implementation does not support negative zeros (
Two declarations of the same object or function specify types that are not compatible (6.2.7).
A program requires the formation of a composite type from a variable length array type whose size is specified by an expression that is not evaluated (6.2.7).
Demotion of one real floating type to another produces a value outside the range that can be represented (
A non-array lvalue with an incomplete type is used in a context that requires the value of the designated object (
An lvalue designating an object of automatic storage duration that could have been declared with the register storage class is used in a context that requires the value of the designated object, but the object is uninitialized. (
An lvalue having array type is converted to a pointer to the initial element of the array, and the array object has register storage class (
An attempt is made to use the value of a void expression, or an implicit or explicit conversion (except to void) is applied to a void expression (
Conversion of a pointer to an integer type produces a value outside the range that can be represented (
A pointer is used to call a function whose type is not compatible with the referenced type (
An unmatched ‘ or ” character is encountered on a logical source line during tokenization (6.4).
A reserved keyword token is used in translation phase 7 or 8 for some purpose other than as a keyword (6.4.1).
A universal character name in an identifier does not designate a character whose encoding falls into one of the specified ranges (
The initial character of an identifier is a universal character name designating a digit (
Two identifiers differ only in nonsignificant characters (
The identifier __func__ is explicitly declared (
The program attempts to modify a string literal (6.4.5).
The characters ‘, \, “, //, or /* occur in the sequence between the delimiters, or the characters ‘, \, //, or /* occur in the sequence between the ” delimiters, in a header name preprocessing token (6.4.7).
For a call to a function without a function prototype in scope, the number of ∗ arguments does not equal the number of parameters (
For call to a function without a function prototype in scope where the function is defined with a function prototype, either the prototype ends with an ellipsis or the types of the arguments after promotion are not compatible with the types of the parameters (
For a call to a function without a function prototype in scope where the function is not defined with a function prototype, the types of the arguments after promotion are not compatible with those of the parameters after promotion (with certain exceptions) (
A function is defined with a type that is not compatible with the type (of the expression) pointed to by the expression that denotes the called function (
A member of an atomic structure or union is accessed (
A pointer is converted to other than an integer or pointer type (6.5.4).
An expression that is required to be an integer constant expression does not have an integer type; has operands that are not integer constants, enumeration constants, character constants, sizeof expressions whose results are integer constants, or immediately-cast floating constants; or contains casts (outside operands to sizeof operators) other than conversions of arithmetic types to integer types (6.6).
A constant expression in an initializer is not, or does not evaluate to, one of the following: an arithmetic constant expression, a null pointer constant, an address constant, or an address constant for a complete object type plus or minus an integer constant expression (6.6).
An arithmetic constant expression does not have arithmetic type; has operands that are not integer constants, floating constants, enumeration constants, character constants, or sizeof expressions; or contains casts (outside operands to size operators) other than conversions of arithmetic types to arithmetic types (6.6).
The value of an object is accessed by an array-subscript [], member-access . or −>, address &, or indirection * operator or a pointer cast in creating an address constant (6.6).
An identifier for an object is declared with no linkage and the type of the object is incomplete after its declarator, or after its init-declarator if it has an initializer (6.7).
A function is declared at block scope with an explicit storage-class specifier other than extern (6.7.1).
A structure or union is defined as containing no named members, no anonymous structures, and no anonymous unions (
An attempt is made to access, or generate a pointer to just past, a flexible array member of a structure when the referenced object provides no elements for that array (
When the complete type is needed, an incomplete structure or union type is not completed in the same scope by another declaration of the tag that defines the content (
An attempt is made to modify an object defined with a const-qualified type through use of an lvalue with non-const-qualified type (6.7.3).
An attempt is made to refer to an object defined with a volatile-qualified type through use of an lvalue with non-volatile-qualified type (6.7.3).
The specification of a function type includes any type qualifiers (6.7.3).
Two qualified types that are required to be compatible do not have the identically qualified version of a compatible type (6.7.3).
An object which has been modified is accessed through a restrict-qualified pointer to a const-qualified type, or through a restrict-qualified pointer and another pointer that are not both based on the same object (
A restrict-qualified pointer is assigned a value based on another restricted pointer whose associated block neither began execution before the block associated with this pointer, nor ended before the assignment (
A function with external linkage is declared with an inline function specifier, but is not also defined in the same translation unit (6.7.4).
A function declared with a _Noreturn function specifier returns to its caller (6.7.4).
The definition of an object has an alignment specifier and another declaration of that object has a different alignment specifier (6.7.5).
Declarations of an object in different translation units have different alignment specifiers (6.7.5).
Two pointer types that are required to be compatible are not identically qualified, or are not pointers to compatible types (
The size expression in an array declaration is not a constant expression and evaluates at program execution time to a nonpositive value (
In a context requiring two array types to be compatible, they do not have compatible element types, or their size specifiers evaluate to unequal values (
A declaration of an array parameter includes the keyword static within the [ and ] and the corresponding argument does not provide access to the first element of an array with at least the specified number of elements (
A storage-class specifier or type qualifier modifies the keyword void as a function parameter type list (
In a context requiring two function types to be compatible, they do not have compatible return types, or their parameters disagree in use of the ellipsis terminator or the number and type of parameters (after default argument promotion, when there is no parameter type list or when one type is specified by a function definition with an identifier list) (
The value of an unnamed member of a structure or union is used (6.7.9).
The initializer for a scalar is neither a single expression nor a single expression enclosed in braces (6.7.9).
The initializer for a structure or union object that has automatic storage duration is neither an initializer list nor a single expression that has compatible structure or union type (6.7.9).
The initializer for an aggregate or union, other than an array initialized by a string literal, is not a brace-enclosed list of initializers for its elements or members (6.7.9).
An identifier with external linkage is used, but in the program there does not exist exactly one external definition for the identifier, or the identifier is not used and there exist multiple external definitions for the identifier (6.9).
A function definition includes an identifier list, but the types of the parameters are not declared in a following declaration list (6.9.1).
An adjusted parameter type in a function definition is not a complete object type (6.9.1).
A function that accepts a variable number of arguments is defined without a parameter type list that ends with the ellipsis notation (6.9.1).
The } that terminates a function is reached, and the value of the function call is used by the caller (6.9.1).
An identifier for an object with internal linkage and an incomplete type is declared with a tentative definition (6.9.2).
The token defined is generated during the expansion of a #if or #elif preprocessing directive, or the use of the defined unary operator does not match one of the two specified forms prior to macro replacement (6.10.1).
The #include preprocessing directive that results after expansion does not match one of the two header name forms (6.10.2).
The character sequence in an #include preprocessing directive does not start with a letter (6.10.2).
There are sequences of preprocessing tokens within the list of macro arguments that would otherwise act as preprocessing directives (6.10.3).
The result of the preprocessing operator # is not a valid character string literal (
The result of the preprocessing operator ## is not a valid preprocessing token (
The #line preprocessing directive that results after expansion does not match one of the two well-defined forms, or its digit sequence specifies zero or a number greater than 2147483647 (6.10.4).
A non-STDC #pragma preprocessing directive that is documented as causing translation failure or some other form of undefined behavior is encountered (6.10.6).
A #pragma STDC preprocessing directive does not match one of the well-defined forms (6.10.6).
The name of a predefined macro, or the identifier defined, is the subject of a #define or #undef preprocessing directive (6.10.8).
An attempt is made to copy an object to an overlapping object by use of a library function, other than as explicitly allowed (e.g., memmove) (clause 7).
A file with the same name as one of the standard headers, not provided as part of the implementation, is placed in any of the standard places that are searched for included source files (7.1.2).
A header is included within an external declaration or definition (7.1.2).
A function, object, type, or macro that is specified as being declared or defined by some standard header is used before any header that declares or defines it is included (7.1.2).
A standard header is included while a macro is defined with the same name as a keyword (7.1.2).
The program attempts to declare a library function itself, rather than via a standard header, but the declaration does not have external linkage (7.1.2).
The program declares or defines a reserved identifier, other than as allowed by 7.1.4 (7.1.3).
The program removes the definition of a macro whose name begins with an underscore and either an uppercase letter or another underscore (7.1.3).
An argument to a library function has an invalid value or a type not expected by a function with variable number of arguments (7.1.4).
The pointer passed to a library function array parameter does not have a value such that all address computations and object accesses are valid (7.1.4).
The macro definition of assert is suppressed in order to access an actual function (7.2).
The argument to the assert macro does not have a scalar type (7.2).
The CX_LIMITED_RANGE, FENV_ACCESS, or FP_CONTRACT pragma is used in any context other than outside all external declarations or preceding all explicit declarations and statements inside a compound statement (7.3.4, 7.6.1, 7.12.2).
The value of an argument to a character handling function is neither equal to the value of EOF nor representable as an unsigned char (7.4).
A macro definition of errno is suppressed in order to access an actual object, or the program defines an identifier with the name errno (7.5).
Part of the program tests floating-point status flags, sets floating-point control modes, or runs under non-default mode settings, but was translated with the state for the FENV_ACCESS pragma ‘‘off” (7.6.1).
The exception-mask argument for one of the functions that provide access to the floating-point status flags has a nonzero value not obtained by bitwise OR of the floating-point exception macros (7.6.2).
The fesetexceptflag function is used to set floating-point status flags that were not specified in the call to the fegetexceptflag function that provided the value of the corresponding fexcept_t object (
The argument to fesetenv or feupdateenv is neither an object set by a call to fegetenv or feholdexcept, nor is it an environment macro (,
The value of the result of an integer arithmetic or conversion function cannot be represented (,,,,,, 7.22.1).
The program modifies the string pointed to by the value returned by the setlocale function (
The program modifies the structure pointed to by the value returned by the localeconv function (
A macro definition of math_errhandling is suppressed or the program defines an identifier with the name math_errhandling (7.12).
An argument to a floating-point classification or comparison macro is not of real floating type (7.12.3, 7.12.14).
A macro definition of setjmp is suppressed in order to access an actual function, or the program defines an external identifier with the name setjmp (7.13).
An invocation of the setjmp macro occurs other than in an allowed context (
The longjmp function is invoked to restore a nonexistent environment (
After a longjmp, there is an attempt to access the value of an object of automatic storage duration that does not have volatile-qualified type, local to the function containing the invocation of the corresponding setjmp macro, that was changed between the setjmp invocation and longjmp call (
The program specifies an invalid pointer to a signal handler function (
A signal handler returns when the signal corresponded to a computational exception (
A signal occurs as the result of calling the abort or raise function, and the signal handler calls the raise function (
A signal occurs other than as the result of calling the abort or raise function, and the signal handler refers to an object with static or thread storage duration that is not a lock-free atomic object other than by assigning a value to an object declared as volatile sig_atomic_t, or calls any function in the standard library other than the abort function, the _Exit function, the quick_exit function, or the signal function (for the same signal number) (
The value of errno is referred to after a signal occurred other than as the result of calling the abort or raise function and the corresponding signal handler obtained a SIG_ERR return from a call to the signal function (
A signal is generated by an asynchronous signal handler (
A function with a variable number of arguments attempts to access its varying arguments other than through a properly declared and initialized va_list object, or before the va_start macro is invoked (7.16,,
The macro va_arg is invoked using the parameter ap that was passed to a function that invoked the macro va_arg with the same parameter (7.16).
A macro definition of va_start, va_arg, va_copy, or va_end is suppressed in order to access an actual function, or the program defines an external identifier with the name va_copy or va_end (7.16.1).
The va_start or va_copy macro is invoked without a corresponding invocation of the va_end macro in the same function, or vice versa (7.16.1,,,
The type parameter to the va_arg macro is not such that a pointer to an object of that type can be obtained simply by postfixing a * (
The va_arg macro is invoked when there is no actual next argument, or with a specified type that is not compatible with the promoted type of the actual next argument, with certain exceptions (
The va_copy or va_start macro is called to initialize a va_list that was previously initialized by either macro without an intervening invocation of the va_end macro for the same va_list (,
The parameter parmN of a va_start macro is declared with the register storage class, with a function or array type, or with a type that is not compatible with the type that results after application of the default argument promotions (
The member designator parameter of an offsetof macro is an invalid right operand of the . operator for the type parameter, or designates a bit-field (7.19).
The argument in an instance of one of the integer-constant macros is not a decimal, octal, or hexadecimal constant, or it has a value that exceeds the limits for the corresponding type (7.20.4).
A byte input/output function is applied to a wide-oriented stream, or a wide character input/output function is applied to a byte-oriented stream (7.21.2).
Use is made of any portion of a file beyond the most recent wide character written to a wide-oriented stream (7.21.2).
The value of a pointer to a FILE object is used after the associated file is closed (7.21.3).
The stream for the fflush function points to an input stream or to an update stream in which the most recent operation was input (
The string pointed to by the mode argument in a call to the fopen function does not exactly match one of the specified character sequences (
An output operation on an update stream is followed by an input operation without an intervening call to the fflush function or a file positioning function, or an input operation on an update stream is followed by an output operation with an intervening call to a file positioning function (
An attempt is made to use the contents of the array that was supplied in a call to the setvbuf function (
There are insufficient arguments for the format in a call to one of the formatted input/output functions, or an argument does not have an appropriate type (,,,
The format in a call to one of the formatted input/output functions or to the strftime or wcsftime function is not a valid multibyte character sequence that begins and ends in its initial shift state (,,,,,
In a call to one of the formatted output functions, a precision appears with a conversion specifier other than those described (,
A conversion specification for a formatted output function uses an asterisk to denote an argument-supplied field width or precision, but the corresponding argument is not provided (,
A conversion specification for a formatted output function uses a # or 0 flag with a conversion specifier other than those described (,
A conversion specification for one of the formatted input/output functions uses a length modifier with a conversion specifier other than those described (,,,
An s conversion specifier is encountered by one of the formatted output functions, and the argument is missing the null terminator (unless a precision is specified that does not require null termination) (,
An n conversion specification for one of the formatted input/output functions includes any flags, an assignment-suppressing character, a field width, or a precision (,,,
A % conversion specifier is encountered by one of the formatted input/output functions, but the complete conversion specification is not exactly %% (,,,
An inv alid conversion specification is found in the format for one of the formatted input/output functions, or the strftime or wcsftime function (,,,,,
The number of characters transmitted by a formatted output function is greater than INT_MAX (,,,
The result of a conversion by one of the formatted input functions cannot be represented in the corresponding object, or the receiving object does not have an appropriate type (,
A c, s, or [ conversion specifier is encountered by one of the formatted input functions, and the array pointed to by the corresponding argument is not large enough to accept the input sequence (and a null terminator if the conversion specifier is s or [) (,
A c, s, or [ conversion specifier with an l qualifier is encountered by one of the formatted input functions, but the input is not a valid multibyte character sequence that begins in the initial shift state (,
The input item for a %p conversion by one of the formatted input functions is not a value converted earlier during the same program execution (,
The vfprintf, vfscanf, vprintf, vscanf, vsnprintf, vsprintf, vsscanf, vfwprintf, vfwscanf, vswprintf, vswscanf, vwprintf, or vwscanf function is called with an improperly initialized va_list argument, or the argument is used (other than in an invocation of va_end) after the function returns (,,,,,,,,,,,,
The contents of the array supplied in a call to the fgets or fgetws function are used after a read error occurred (,
The file position indicator for a binary stream is used after a call to the ungetc function where its value was zero before the call (
The file position indicator for a stream is used after an error occurred during a call to the fread or fwrite function (,
A partial element read by a call to the fread function is used (
The fseek function is called for a text stream with a nonzero offset and either the offset was not returned by a previous successful call to the ftell function on a stream associated with the same file or whence is not SEEK_SET (
The fsetpos function is called to set a position that was not returned by a previous successful call to the fgetpos function on a stream associated with the same file (
A non-null pointer returned by a call to the calloc, malloc, or realloc function with a zero requested size is used to access an object (7.22.3).
The value of a pointer that refers to space deallocated by a call to the free or realloc function is used (7.22.3).
The alignment requested of the aligned_alloc function is not valid or not supported by the implementation, or the size requested is not an integral multiple of the alignment (
The pointer argument to the free or realloc function does not match a pointer earlier returned by a memory management function, or the space has been deallocated by a call to free or realloc (,
The value of the object allocated by the malloc function is used (
The value of any bytes in a new object allocated by the realloc function beyond the size of the old object are used (
The program calls the exit or quick_exit function more than once, or calls both functions (,
During the call to a function registered with the atexit or at_quick_exit function, a call is made to the longjmp function that would terminate the call to the registered function (,
The string set up by the getenv or strerror function is modified by the program (,
A command is executed through the system function in a way that is documented as causing termination or some other form of undefined behavior (
A searching or sorting utility function is called with an invalid pointer argument, even if the number of elements is zero (7.22.5).
The comparison function called by a searching or sorting utility function alters the contents of the array being searched or sorted, or returns ordering values inconsistently (7.22.5).
The array being searched by the bsearch function does not have its elements in proper order (
The current conversion state is used by a multibyte/wide character conversion function after changing the LC_CTYPE category (7.22.7).
A string or wide string utility function is instructed to access an array beyond the end of an object (7.23.1, 7.28.4).
A string or wide string utility function is called with an invalid pointer argument, even if the length is zero (7.23.1, 7.28.4).
The contents of the destination array are used after a call to the strxfrm, strftime, wcsxfrm, or wcsftime function in which the specified length was too small to hold the entire null-terminated result (,,,
The first argument in the very first call to the strtok or wcstok is a null pointer (,
The type of an argument to a type-generic macro is not compatible with the type of the corresponding parameter of the selected function (7.24).
A complex argument is supplied for a generic parameter of a type-generic macro that has no corresponding complex function (7.24).
At least one field of the broken-down time passed to asctime contains a value outside its normal range, or the calculated year exceeds four digits or is less than the year 1000 (
The argument corresponding to an s specifier without an l qualifier in a call to the fwprintf function does not point to a valid multibyte character sequence that begins in the initial shift state (
In a call to the wcstok function, the object pointed to by ptr does not have the value stored by the previous call for the same wide string (
An mbstate_t object is used inappropriately (7.28.6).
The value of an argument of type wint_t to a wide character classification or case mapping function is neither equal to the value of WEOF nor representable as a wchar_t (7.29.1).
The iswctype function is called using a different LC_CTYPE category from the one in effect for the call to the wctype function that returned the description (
The towctrans function is called using a different LC_CTYPE category from the one in effect for the call to the wctrans function that returned the description (
Most of these items are already detected, could be detected easily, or would be detected as a side effect of solving UBs that we discussed in detail. In other words, a few basic technologies, such as shadow memory and run-time type information, provide the infrastructure needed to detect a large fraction of the hard-to-detect UBs. Alas it is difficult to make shadow memory and run-time type information fast.

What is the modern C or C++ developer to do?

Be comfortable with the “easy” UB tools — the ones that can usually be enabled just by adjusting a makefile, such as compiler warnings and ASan and UBSan. Use these early and often, and (crucially) act upon their findings.
Be familiar with the “hard” UB tools — those such as TIS Interpreter that typically require more effort to run — and use them when appropriate.
Invest in broad-based testing (track code coverage, use fuzzers, etc.) in order to get maximum benefit out of dynamic UB detection tools.
Perform UB-aware code reviews: build a culture where we collectively diagnose potentially dangerous patches and get them fixed before they land.
Be knowledgeable about what’s actually in the C and C++ standards since these are what compiler writers are going by. Avoid repeating tired maxims like “C is a portable assembly language” and “trust the programmer.”
Unfortunately, C and C++ are mostly taught the old way, as if programming in them isn’t like walking in a minefield. Nor have the books about C and C++ caught up with the current reality. These things must change.

Good luck, everyone.

We’d like to thank various people, especially @CopperheadOS on Twitter, for discussing these issues with us.

Wojciech Muła — website

Wojciech Muła — website

articles — optimization, SIMD, algorithms
notes — various technical notes (or rants)
bird photos
artykuły — asembler x86 (ex. asmcorner), grafika, algorytmy
projekty — pydvi2svg, pysame, aspell-python, sse2string
programy i skrypty — Python, Javascript, C
skrypty shellowe — które wykorzystuję u siebie
notatki (polski)
tłumaczenia tekstów
opowiastki zen
buddyjski słownik ortograficzny,
archiwum grupy dyskusyjnej
projekty ( emulator maszyny RAM)


zdjęcia na

wikipedia, wikibooks

A short report from code::dive 2017
Finally I sorted out the photos of birds I have taken over years. Here’s the link.
updates to Base64 encoding & decoding using AVX512BW instructions — add usage of instruction VPMULTISHIFT from AVX512VL extension
updates to SIMD-friendly algorithms for substring searching — results from ARMv8 (ARM Cortex A57)
updates to SIMD-friendly algorithms for substring searching — results from ARMv7 (Raspberry Pi 3)
new article ARM Neon and Base64 encoding & decoding — I have bought Raspberry Pi 3
updates to Base64 decoding with SIMD instructions — added a slightly shorter pshufb-based lookup
updates to SIMD-friendly algorithms for substring searching — results from Bulldozer CPU
new article AVX512 — first bit set in a large array
updates to Base64 encoding & decoding using AVX512BW instructions — use order of fields imposed by the base64 standard
Base64 encoding with SIMD instructions — added description of pack and lookup procedures using XOP instructions
new article SWAR check if all chars are digits
updates to Base64 encoding with SIMD instructions, Base64 decoding with SIMD instructions and AVX512F base64 coding and decoding — use order of fields imposed by the base64 standard
new article: Population count using XOP instructions — use instruction VPTERNB from AMD XOP can help a little
updates to Base64 decoding with SIMD instructions — refreshed measurments, added results from AMD Bulldozer
updates to Base64 encoding with SIMD instructions — added results from AMD Bulldozer
updates to Base64 decoding with SIMD instructions — I spotted an unnecessary instruction
new article: SIMD-friendly algorithms for substring searching
Paper by Daniel Lemire, Nathan Kurz and me: Faster Population Counts using AVX2 Instructions
updates to Base64 encoding & decoding using AVX512BW instructions — faster unpacking
updates to Conversion numbers to binary representation — a new clever SWAR procedure, by Larry Bank
updates to: AVX512: ternary functions evaluation — more about bit-select
Archive: 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006, 2005, 2004, 2003

Boosting numpy: Why BLAS Matters

Boosting numpy: Why BLAS Matters

I recently noticed that the same code on the same machine had vastly different run times in different virtual environments. This looked a little suspicious to me. Digging deeper, I realized that the reason for this was that numpy was linked against different BLAS libraries in different virtual environments, which had tremendous effects on the performance. However, taking a closer look on what implementation is actually used under the hood on your machine, and possibly compiling an implementation that is tailored to your very system, can yield significant speed–ups almost for free, without changing a single line of your code.

BLAS is the low–level part of your system that is responsible for efficiently performing numerical linear algebra, i.e. all the heavy number crunching. More precisely:

Basic Linear Algebra Subprograms (BLAS) is a specification that prescribes a set of low-level routines for performing common linear algebra operations such as vector addition, scalar multiplication, dot products, linear combinations, and matrix multiplication. They are the de facto standard low-level routines for linear algebra libraries; the routines have bindings for both C and Fortran. Although the BLAS specification is general, BLAS implementations are often optimized for speed on a particular machine, so using them can bring substantial performance benefits.

Together with something like LAPACK, it is the cornerstone of today’s scientific software packages.

LAPACK (Linear Algebra Package) is a standard software library for numerical linear algebra. It provides routines for solving systems of linear equations and linear least squares, eigenvalue problems, and singular value decomposition. It also includes routines to implement the associated matrix factorizations such as LU, QR, Cholesky and Schur decomposition.

Now, as mentioned above, BLAS & LAPACK are merely defining a standard interface, but what really matters is the implementation you choose. There are many different implementations under different licenses, which all have their advantages and disadvantages. It can sometimes be difficult to choose the most beneficial implementation for your machine.

Different Implementations
By default, you will most likely only have the standard BLAS & LAPACK implementation on your system (e.g. Arch Linux packages blas/cblas and lapack). While this situation is certainly sufficient if you don’t care about performance and works out of the box, is has considerable shortcomings compared to more recent implementation, e.g. not fully supporting multi–core computations. We will see how its overall performance compares to other implementations below.

Automatically Tuned Linear Algebra Software (ATLAS)
The main purpose of the ATLAS project is to provide a portable high–performance implementation for many different platforms. To achieve this, they use a complex mechanism to automatically determine the optimal parameters for the respective hardware during compilation. ATLAS has become quite popular in recent years, since it uses the BSD license and is platform independent. When compiled properly, it probably provides a reasonably good BLAS library for most performance–critical applications on most platforms.

On Arch Linux you can use the atlas-lapack package from the repository, which will download and compile ATLAS on your machine. However, before starting the compilation, make sure to disable CPU throttling in your OS, since this will mess up the timing when benchmarking ATLAS during compilation. To do this, I had to run sudo cpupower frequency-set -g performance on my machine. Additionally, I had to disable intel_pstate which actually required me to reboot my system. After this, the compilation process ran smooth and ATLAS determined the optimal settings for my machine, which ended up taking a couple of hours.

Intel Math Kernel Library (Intel MKL)
The Intel Math Kernel Library (Intel MKL) is a proprietary library that is hand–optimized specifically for Intel processors. As of February 2017, it is free of charge for some use cases. You can read more about the Community Licensing here.

If you are using Arch Linux you can obtain it by installing the intel-mkl package, otherwise search your repository or download it directly from Intel’s developer page. The compilation process is straightforward, just follow the instructions provided.

OpenBLAS is another popular open–source implementation that is based on a fork of GotoBLAS2. They claim in their FAQ that OpenBLAS achieves a performance comparable to Intel MKL for Intel’s Sandy Bridge CPUs. Furthermore, OpenBLAS is well–known for its multi-threading features and apparently scales very nicely with the number of cores.

Obtain a copy from or GitHub and follow the installation instructions. This will walk you through the compilation and installation process.

Time for Benchmarks
A reliable comparison can only be based on hard numbers. For this purpose, I drafted a small Python script based on an existing script that I found in this Stackoverflow thread. Here it is:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

from __future__ import print_function

import numpy as np
from time import time

# Let’s take the randomness out of random numbers (for reproducibility)

size = 4096
A, B = np.random.random((size, size)), np.random.random((size, size))
C, D = np.random.random((size * 128,)), np.random.random((size * 128,))
E = np.random.random((int(size / 2), int(size / 4)))
F = np.random.random((int(size / 2), int(size / 2)))
F =, F.T)
G = np.random.random((int(size / 2), int(size / 2)))

# Matrix multiplication
N = 20
t = time()
for i in range(N):, B)
delta = time() – t
print(‘Dotted two %dx%d matrices in %0.2f s.’ % (size, size, delta / N))
del A, B

# Vector multiplication
N = 5000
t = time()
for i in range(N):, D)
delta = time() – t
print(‘Dotted two vectors of length %d in %0.2f ms.’ % (size * 128, 1e3 * delta / N))
del C, D

# Singular Value Decomposition (SVD)
N = 3
t = time()
for i in range(N):
np.linalg.svd(E, full_matrices = False)
delta = time() – t
print(“SVD of a %dx%d matrix in %0.2f s.” % (size / 2, size / 4, delta / N))
del E

# Cholesky Decomposition
N = 3
t = time()
for i in range(N):
delta = time() – t
print(“Cholesky decomposition of a %dx%d matrix in %0.2f s.” % (size / 2, size / 2, delta / N))

# Eigendecomposition
t = time()
for i in range(N):
delta = time() – t
print(“Eigendecomposition of a %dx%d matrix in %0.2f s.” % (size / 2, size / 2, delta / N))
This performs some matrix multiplication, vector–vector multiplication, singular value decomposition (SVD), Cholesky factorization and Eigendecomposition, and averages the timing results (which are of course arbitrary) over multiple runs. This is how you can find out which BLAS implementation numpy is using under the hood:

import numpy as np
You can run the script successively using different BLAS implementations, either in different virtual environments or using a different BLAS configuration in site.cfg. For a fair comparison, you should set your CPU to a constant clock frequency and not perform any other heavy lifting while the scripts are running, especially when they make use of multiple cores on your machine.

Let’s find out how the different implementation compare in terms of performance on my laptop!

Dotted two 4096×4096 matrices in 64.22 s.
Dotted two vectors of length 524288 in 0.80 ms.
SVD of a 2048×1024 matrix in 10.31 s.
Cholesky decomposition of a 2048×2048 matrix in 6.74 s.
Eigendecomposition of a 2048×2048 matrix in 53.77 s.
Dotted two 4096×4096 matrices in 3.46 s.
Dotted two vectors of length 524288 in 0.73 ms.
SVD of a 2048×1024 matrix in 2.02 s.
Cholesky decomposition of a 2048×2048 matrix in 0.51 s.
Eigendecomposition of a 2048×2048 matrix in 29.90 s.
Intel MKL
Dotted two 4096×4096 matrices in 2.44 s.
Dotted two vectors of length 524288 in 0.75 ms.
SVD of a 2048×1024 matrix in 1.34 s.
Cholesky decomposition of a 2048×2048 matrix in 0.40 s.
Eigendecomposition of a 2048×2048 matrix in 10.07 s.
Dotted two 4096×4096 matrices in 3.97 s.
Dotted two vectors of length 524288 in 0.74 ms.
SVD of a 2048×1024 matrix in 1.96 s.
Cholesky decomposition of a 2048×2048 matrix in 0.46 s.
Eigendecomposition of a 2048×2048 matrix in 32.95 s.
Today’s Lesson
We can see that there are tremendous differences in run time. Unsurprisingly, the default implementation seems to be slowest in most regards (by far!). Intel’s Math Kernel Library (Intel MKL) performs best, which is again not truly surprising, given the fact that I use an Intel i5 processor on my machine. ATLAS and OpenBLAS are both within the same ballpark and doing reasonably well. All implementations, except for the default implementation, make use of all cores on my machine, which can explain some but not all of the performance gains. You can control the number of threads used by setting the respective environment variables (e.g. OMP_NUM_THREADS).

The difference between slowest and fastest is quite remarkable. If the low–level math computation is actually your bottleneck, and you are not already making use of all cores on your machine, you can expect speed–ups up to a factor of 2525 (in my case) or something within that order of magnitude. This is roughly equivalent of getting the run time down from a full nightly run, i.e. from leaving the office in the afternoon to the first cup of coffee in the morning, to only half an hour.

Further Thoughts
Always keep an eye on what BLAS library numpy is actually using on your machine. Use to find out about this.
By the way, it also matters which BLAS implementation scipy is linked against.
If you care about performance, whatever BLAS library you decide to use in the end should be tailored to your machine’s very hardware. It you don’t mind the trouble, go ahead and compile it yourself.
As a rule of thumb, when your machine has a reasonably modern Intel processor, then you’re probably best off using Intel MKL.
It seems that, once again, the Anaconda distribution saves your ass by providing sane defaults, i.e. shipping with Intel MKL built-in in my case. If you’ve obtained numpy through conda you are probably doing fine.
There are more libraries such as Eigen or Boost that should be worth to look at. If you have an AMD processor, take a look at ACML.
Building NumPy and SciPy with Intel MKL
No Cost Options for Intel® Parallel Studio XE, Support Yourself, Royalty-Free
Anaconda 2.5 Release – Now with MKL Optimizations
Anaconda Documentation: MKL Optimizations
Performance Comparison of OpenBLAS and Intel® Math Kernel Library in R
Benchmark OpenBLAS, Intel MKL vs ATLAS
Numpy/Scipy with Intel® MKL and Intel® Compilers

Daniel Ellsberg: The Rolling Stone Interview

Daniel Ellsberg: The Rolling Stone Interview

Daniel Ellsberg: The Rolling Stone Interview

The man who leaked the Pentagon Papers talks about how much Mr. Nixon likes to win— how he changed our form of government, used awesome police powers to eliminate opponents and cynically prolonged the Vietnam War four more years

Daniel Ellsberg, THE DICK CAVETT SHOW – Shoot Date: January 28th, 1972. ABC Photo Archives/ABC via Getty
By Jann S. Wenner
November 8, 1973
More News
The Poison Harvest: Agent Orange and the Legacy of Vietnam
Rory Kennedy Explores Vietnam’s Messy End in ‘Last Days’ Doc
All Stories
Daniel Ellsberg was perhaps the first highly placed official (at one time the deputy Assistant Secretary of Defense) to have ever left the inner government and then reveal, with top-secret documents, its closely guarded secret operations. As befits a man who risks his reputation and ruin, to fight a corrupt and unlawful government, he is vain, egocentric and completely convinced of the lightness of his action.

It is a grandiose attitude – one that seems to have especially offended the press. It supposes the power of truth, of the man who speaks it, and the moral example it sets. Ellsberg quotes Madison’s statement that, “Knowledge will forever govern ignorance and the people who mean to be their own governors must take care to arm themselves with the power that knowledge gives.”

I decided to avoid, as far as possible, discussion of his several years in Vietnam, the Pentagon Papers trial and his personal life. The interview was directed toward an exploration and understanding of his intellectual life, his experiences in the secret decision-making processes inside the Defense Department, and conclusions about the motives and methods of the inner government.

His answers were intricately detailed, with many ideas interwoven and cross-referenced into a dissertation of extraordinary complexity, some 500 typewritten pages long. During a week-long siege, it was roughly organized into two sections and the first part was edited, re-edited, and ultimately reduced in half (by myself, David Felton and Bill Sievert). The organization here is not how the interview was originally conducted. For example, the discussion of Henry Kissinger did not actually begin the interview. As we go to press, Part II remains in the wings, unedited and unorganized. We plan to put it into shape and publish it soon.

“We were facing a massive and urgent threat to our remaining democratic institutions, a coup on the eve of its completion. People who carried out this coup are still in power, starting with the president.”

There is a natural tendency to be suspicious. To some degree, we have all been affected by the notion that no matter how necessary and important Daniel Ellsberg’s act was, to risk a life in prison, vilification as a traitor, and personal slander, there “must have been something else behind it.”

But examine his statement: a dark picture of what has been occurring in American government. He acted on one of the basic democratic beliefs, that “a man can make a difference.”

What was your relationship with Henry Kissinger?
He had been at the Center for International Affairs at Harvard, and before I went to Rand in 1959, I gave a couple of seminars to his group, discussions about strategy and politics. Over the years I would see him occasionally at a conference or at a Rand symposium. It was not a personal relationship, nor even a close professional relationship, but an intermittent business or analytical association.

I had a very negative attitude toward him because he was pushing the idea of limited nuclear war as a substitute not only for all-out war but also for no nuclear war. He thought that if we forewent the possibility of nuclear weapons the world would be taken over by stronger nations; and that if we limited our options to threats of unrestrained nuclear war, the prospect would be so horrendous that we would be paralyzed and unable to use nuclear weapons at all. He thought the proper strategy was to build and threaten to use, as appropriate, small tactical nuclear weapons the size of the Hiroshima bombs and up to 10 times that size.

Kissinger has no originality whatsoever as an intellect. I read all of his writings, since they were within the field that I was working in, and thought of them as extremely derivative. They were well-written, good expositions of other people’s ideas and often contained analytic criticism. He changed his sources from book to book and the quality of the thinking pretty closely reflected the quality of his current sources.

His first book was admired by Nixon who gave it a great boost with a photograph on the front page of The New York Times of him going into a meeting of the National Security Council with it under his arm. This was the limited nuclear war book, strongly influenced by Edward Teller, General Gavin, a few other Air Force exponents of this limited war concept, and a major source was Bernard Brodie, a Rand colleague; another was Bill Kaufman. Later, he wrote a book – one that almost reversed, temporarily, the drift of his thinking on nuclear weapons – which was very strongly influenced by Rand associates of mine, such as Albert Wohlstetter, Herman Kahn and Tom Schelling. Some of his articles on arms control were taken almost verbatim from work by Tom Schelling.

The sources would never be directly acknowledged. And he had a trick of covering himself by including people in his bibliography, but in entirely misleading ways. He’d include references to secondary works by these people, or works from which he had not drawn, but no mention whatever of the works he was paraphrasing. He wanted to be thought of not only as an able intellect – which he is in the sense of an expositor and critic, which alone would have been enough for an academic career – but also as an original person, a creative person. His solution to that problem must have put him under a certain tension over the years.

It’s not unlike what must be Nixon’s own tension, the shame and guilt of having succeeded by major deceits and never wholly on his own ability or worth.
I have a very strong feeling that Nixon and Kissinger are similar personalities and feel a great affinity and attraction on that basis. Each of them may be the other’s best friend, at least during business hours. Kissinger – and surely Nixon, too – has a very strong ideological belief in the efficacy and legitimacy of the threat of violence as a tool of power and as a way of “establishing world order.”

I’ve heard him profess sympathy for revolutionary aims, say of the NLF, combined with a sad judgment on the “tragic” implications of revolutionary efforts – that they are unstabilizing, reckless and lead to world disorder unintended by the revolutionary forces – thus the “necessity” for suppressing them, despite the fact that one can sympathize with some of their motives. I think he really wants to have the tragic function of suppressing violently idealistic movements and sees no limits to the amount of violence that is acceptable for him to use to counter threats to world order.

One can guess – and I’m neither a psychiatrist nor a producer of official psychological profiles – that Nixon and Kissinger are people who have very strong desires not only to threaten, but to inflict violence. Kissinger can’t be a rebel, he couldn’t conceive of taking part in violence directed against “authority”; but by all evidence he wants very much to be a party to violence. There’s no question that he likes to issue threats. I would suspect very strongly that he wants some of his threats to fail, so that they have to be carried out.

This is how they come to associate with persons like G. Gordon Liddy.
Yeah, Liddy again is on the side of counterrevolution, like E. Howard Hunt, outlaws on the side of the police. It’s the psychology of a “bad cop”: To adapt a remark by Garry Wills, with each of these men it’s as if the Sheriff of Nottingham had fantasies not of arresting Robin Hood but of mugging him.

It’s not difficult to move from Liddy up to his supervisors like Ehrlichman and fairly describe Ehrlichman as a thug; but would you say it’s fair to describe Nixon and Kissinger as thugs?
After Cambodia and Laos, I’ve always privately thought of Henry Kissinger as a murderer. We’re not talking of persons who burglarize this or that office, but of persons who dropped four million tons of bombs on Indochina. Words are hardly adequate to define people who took those choices and took them in the years 1969 to 1972. They were not confronting Joseph Stalin or Adolf Hitler, nor did they act under whatever misconceptions about Ho Chi Minh may have lingered in the late Fifties or early Sixties. They took those decisions after Lyndon Johnson and Robert MacNamara themselves dropped two million tons, failed, and were thrown out of office; they proceeded to drop four million more tons after 1969, having been elected mostly by people who expected them to end the war…

The story that’s not yet been written – perhaps now it will be – is how Nixon came to manage and complete a massive hoax during that four-year period that he was in the process of ending the war without a victory and had every intention of ending it as fast as possible. It was a marvelously contrived deception. In fact, it led me to have a good deal of respect, from a technical point of view, for the manager of that hoax. After reading Joe McGuiness’ book The Selling of the President, people tended to sneer at the competence of advertising men in campaign politics, but they did an almost miraculous job selling the prolongation of the War to the American public from 1969 to 1973, and beyond.

And Henry Kissinger?
He was a major part of that selling campaign.

When did you come in contact with Kissinger after Rand?
He came to Vietnam in 1966 as a consultant to Henry Cabot Lodge. I was very impressed that he took my advice – which I gave to nearly every visitor but which few of them took – to avoid official briefings and talking to anyone in the presence of his boss or agency head; instead, seek out people who had been around, who were known to know a lot about Vietnam, talk to them privately and separately, and get from them the names of other people and talk to those other people separately. And to talk to the Vietnamese as much as possible. MacNamara never did any of these things in all the trips he made, but always talked to district advisers in the presence of the general in charge and never seemed to realize how much he was being fooled.

Kissinger did see the people I suggested. He is a talented and incisive questioner; he takes notes, listens carefully, and learns very well. In a couple of brief visits he did learn an unusual amount. He became appropriately skeptical and pessimistic and compared to Walt Rostow or others in Washington, he had a pretty realistic picture of the unlikelihood of much improvement. So, that was promising.

I was with him in a couple of conferences during 1967, and he was expressing a view far in advance of any mainstream political figure at that point, namely that our only objective in Vietnam should be an assurance of what he called the decent interval before the Communists took over so that we would not be humiliated at home or in our foreign affairs by an abrupt, naked failure.

McCarthy and Robert Kennedy still felt called on to talk about a negotiated solution with, at most, a coalition government. They were not willing to talk about unilateral withdrawal or acceptance of Communist takeover anytime. Kissinger’s dovish description was exceeded only by people like Abbie Hoffman or Dave Dellinger, who were calling for immediate withdrawal.

Finally, by the way, Kissinger expressed thoroughly and openly a total contempt for Richard Nixon. He worked for Rockefeller and was willing to say things about Nixon, such as his famous statement at the 1968 Republican convention, “Richard Nixon is not fit to be president.” That seemed a little indiscrete for someone in politics.

And a few months later he was appointed foreign policy adviser to the president-elect…
He’d been appointed Special Assistant for National Security Affairs and, in readiness for the first National Security Council meeting, he asked for a study of options on Vietnam. The president of Rand, Harry Rowen, suggested me for the job.

Kissinger accepted me with the first reservation ever expressed during my career as to my discretion. He did not want it known that he had turned to Rand for advice, an outside group, known to be relatively dovish within the defense community. And particularly he did not want it known that I was associated with the study since by that time I was a critic of our involvement. I was surprised to have that question raised; my career had been based on handling secrets and using discretion. He told an official at Rand that he had benefited greatly from our discussions in Vietnam, but was “on the other side of the fence,” and “saw things differently.”

Did you speak personally before you undertook the study?
No. I worked for several weeks putting together the options paper [National Security Memorandum One] and flew with it to his offices, a set of apartments they were using with Xerox machines and typewriters in the Hotel Pierre, on Christmas Day, 1968. We spent a couple of days going over the memorandum.

I suggested that he put a bunch of questions to the various parts of the bureaucracy and ask for parallel, rather than coordinated, answers so that he could compare the discrepancies and get a sense of what the uncertainties and conflicts were – the contradictions. I worked on these questions for him. I wanted him to see how much argument there was.

Did you find yourself liking him?
He tends to be fairly ingratiating, and has a habit of being quite flattering to a person in the presence of associates. Just after the election he gave several talks at Rand, and at one point said to the group, in my presence, “I learned more from Dan Ellsberg in Vietnam than from any other person.” It might have been true, but it was also characteristic of him to say something like that in your presence.

So he is a flatterer…
It’s, let’s say, a nice habit, which, however, is counteracted by certain other traits.

I believed Kissinger was well clued-in on the realistic, pessimistic prospects in Vietnam and that he would be a good adviser to Nixon. However, there was one ominous signal that I didn’t pay too much attention to at the time: He felt that escalation of the war had not been spelled out enough in our discussion of options.

I did take the occasion in my three days there to try to ‘inoculate’ him against the effects of the secret information he was about to start receiving. I had often thought of having a chance to warn somebody new to government, just about to receive a lot of high clearances, and to pass on one of the lessons I had learned not only by participating in the government, but also by studying a lot of the earlier crisis decision-making. I doubt he would remember the conversation.

“Henry, there’s something I would like to tell you, for what it’s worth, something good for a person that’s entering a job like this to know. You’ve been a consultant for a long time and you’ve dealt a great deal with top-secret information. But you are about to receive a whole slew of clearances, maybe 15 or 20 of them higher than top secret.

“I’ve had these myself, as you know, and I’ve known people who’ve acquired them, and I have a pretty good sense of what the effects of receiving these clearances are on a person who didn’t previously know they even existed, and of reading the information that will now become available to you.

“First, you will feel like a fool for having studied, written and talked about these subjects – for having criticized and analyzed decisions made by presidents – for years without having known of the existence of all this inside information. In particular, you’ll feel foolish for having rubbed shoulders for over a decade with officials and consultants who had access to all this information that you didn’t even know they had, and that they kept that secret from you.

“You will feel like a fool, and it will last for about two weeks. Then, after you read all this daily intelligence input – estimates, analyses and so forth, and become used to using what amounts to whole libraries of hidden information which is much more closely held than mere top-secret data – you will forget there was ever a time when you didn’t have it, and you’ll be aware only of the fact that you have it now and others don’t… and that all those other people are fools.

“Over a longer period of time – not too long, but a matter of two or three years – you will eventually become aware of the limitations of this information: There is a great deal that it doesn’t tell you, it is often inaccurate and it can lead you astray just as much as The New York Times can. But that takes quite a while…

“In the meantime, it will become very hard for you to learn from anybody who doesn’t have these clearances, because you’ll be thinking to yourself as you listen to them: ‘What would this man be telling me if in fact he knew what I know? Would he be giving me the same advice, or would it totally change his recommendations?’ And that mental exercise is so torturous that after a while you give it up and you just stop listening. I’ve seen this with my superiors, my colleagues and myself.

“You will deal with a person who doesn’t have those clearances – since you must carefully lie to him about what you know – from the point of view of what do you want him to believe and what impression do you want him to go away with. In effect, how to manipulate him. You’ll give up trying to assess what he has to say, and the danger is you’ll become something like a moron. You’ll become incapable of learning from most people in the world, no matter how much experience or knowledge they may have of their particular area and which may be much greater than yours.”

He thanked me and said it was interesting. It was hard for him to fully appreciate because he didn’t yet have these clearances and their effect on you is quite spectacular, learning about operations that the president can call into being at his will that you didn’t imagine were permitted to any human being.

I said to Henry that I thought of this secret information as something like the potion Circe gave to the wanderers who happened on her island that turned men into swine.

And your feeling after this talk?
As a staff person or consultant, you always feel you’ve gotten your reward if a person has listened to what you had to say and seemed to pay attention – that’s as much as you hope for, and that it will some day have some effect.

What was the fate of your study?
National Security Memorandum One went through one more draft. At the request of Kissinger, the one option for unilateral withdrawal was deleted, which meant that all the alternatives had the property of keeping us in Vietnam. I was, meanwhile, working on this set of questions and answers on Vietnam from various agencies of the bureaucracy.

The answers which amounted to about 1,000 pages showed the range of disagreement and pessimistic attitudes about the performance of the Vietnamese army, and the possibility of stopping infiltration by mining Haiphong or bombing the North. They came from agencies like the CIA, the intelligence branch of the State Department and the civilians in the Defense Department, who would not normally be called on to give direct opinions on these subjects to the president. By this device of asking for parallel reports, I had ensured that the new president got the most realistic estimates any president had ever gotten on these subjects.

They showed clearly, despite some disagreements, that the Vietnamese army would never be capable of withstanding North Vietnamese assault without U.S. bombing and U.S. ground troops. And Nixon was told very flatly by most of the responses, except for the Air Force of course, that bombing of Laos was having no effect whatever, and that the mining of Haiphong would likewise have no effect. We now know that the same month the questions were finished, March 1969, the secret bombing of Cambodia began.

I left town with this satisfied consultant’s feeling of having done all I could to bring realistic information to the attention of the new president and to Henry Kissinger, and with considerable expectation that they would act on this and get us out.

It was the kind of job I would have done for any president, even George Wallace. It seemed that one couldn’t go wrong in improving the understanding the president had of Vietnam politics. Later I came to question the usefulness of even this kind of relationship.

When I left the White House, I made a number of recommendations for new studies. One of them had to do with a study of what the word “accommodations” might mean as used by different agencies, and why that would be bad for the United States. Another one was to conduct an urgent and intensive study of the impact of our artillery and bombing operations on the Vietnamese people with an eye to the possibility of greatly reducing these operations or cutting them out entirely. Another was the adequacy of our information on civilian casualties. No one had ever tried to collect it.

Kissinger sent the word back to me through Morton Halperin that these were very useful suggestions but we had asked enough questions for now. And they were all set aside; they never did get around to asking or answering them. There are some things that these officials know they don’t want to know.

I left Washington without any expectation of working again for the administration. Or desire. They sent me a bunch of forms at Rand to fill out for a White House clearance, since I had been working there without a specific White House clearance, but I never filled them out.

I saw Halperin in Washington in June shortly after Nixon’s most conciliatory speech on the war, and he told me, “For the first time, I’m satisfied with the Vietnam policy of the United States.” Although Nixon had not committed himself to total withdrawal, Halperin was very confident that by the end of the year he would have accepted that idea.

But in June or July, the Russians recognized the Provisional Revolutionary Government, making it clear they would not cooperate with the U.S. in bringing pressure on the North to negotiate a mutual withdrawal. It was then that Kissinger began his secret talks with Hanoi, more or less having given up with the Russians.

The policy that they had come into office with went bankrupt at that point. In Richard Whalen’s book, Catch the Falling Flag, Whalen describes conversations in late ’67 with Nixon – for whom he was the main speechwriter on Vietnam – where he proposed to Nixon that he threaten the mining of Haiphong. This would supposedly present the Russians with a crisis like the Cuban missile crisis, because of their shipping, and encourage them to bring pressure on Hanoi for a settlement. Nixon bought that strategy, but it failed. Still Nixon and Kissinger didn’t believe it; they went ahead with their strategy.

I called Halperin in late June of ’69 with a question that was new to me: “What’s your estimate of the number of Vietnamese who would rather see peace even under a Communist government than see the war continue?”

“Oh, 90%,” he said.

“Do you think your boss thinks that?” I asked.

“I’ve never discussed that with him precisely, but I would guess that he did.”

“Then how can we justify continuing this a day longer, whether to get mutual withdrawal or graceful ending or anything else? I don’t discount some usefulness in papering over our defeat and so forth, but how can we justify killing another Vietnamese when our own guess is that nearly all of them want the war over?”

“Well,” he said, “that’s a good question… let me think about that.”

That was the moment I began to see the need to end the war most urgently. From mid ’67 on, I had been for ending the war “as soon as possible,” but I still had a willingness to see it prolonged by weeks or months in the course of negotiations, in hopes of a somewhat better solution that would leave us with a less controversial ending and perhaps less of a domestic backlash. But by mid-1969 I began to see that domestic politics couldn’t excuse it. I finally saw continuation as immoral, not just mistaken.

In September ’69 I learned from Halperin that the policy had not gone as he had hoped in June; that Nixon and Kissinger had chosen one of the options we had laid out earlier – not the excluded option for unilateral withdrawal – but the option to win the war. A disastrous choice. But not yet public; they were not yet fully committed.

I went to some people at Rand who had been for unilateral withdrawal all along, and said, “I’m with you now; what shall we do about it?” They proposed a letter to The New York Times, calling for unilateral withdrawal which, by the way, no one in mainstream politics had publicly proposed at that point.

I said what we needed was a study that would lay out the facts more exhaustively than a letter, but they said a study would never get cleared out of the Defense Department. The only way we could get past the clearance process was in a letter.

In the course of our drafting the letter, former New York Senator Charles Goodell proposed a Congressional cutoff of funds by the end of 1970. It was the first proposal of that kind by a politician. We went ahead with our plan because Goodell wasn’t recognized as an expert, and our letter would add some authoritative support to his position.

The publication of that letter was as controversial at Rand and in the Defense community probably as my leak of the Pentagon Papers was later. It was a bombshell among our associates – and was very widely quoted in the Moratorium. Meanwhile, without telling anyone at Rand, I started to copy the Pentagon Papers to give them to the Senate.

What was your next contact with Kissinger?
Just after my second marriage in September, 1970, I cut my honeymoon in half to return for an appointment with him. He didn’t keep it although I did see him about 10 days later. I thought it was a good chance to lay several things on him.

He was as bad as I’d ever imagined he was. My earlier feelings were based on his attitude toward a fairly hypothetical situation toward nuclear war. But now we were confronting a man who was managing the actual destruction of Cambodia.

I hadn’t yet heard a great deal about what his own personal role was in all this. It wasn’t completely clear how much of this was Nixon, and how much Kissinger; only that he was implementing a disastrous policy. At that point, I didn’t have the sense of revulsion at the thought of meeting him, which I did acquire later.

When a mutual friend who had an appointment with Kissinger proposed bringing me along, I agreed. It was worth encouraging him to read the Pentagon Papers so that he might discover that pursuing escalation had been talked through before in just as conspiratorial terms, and that it hadn’t worked. Maybe he could learn from that.

Also, I knew that his policy depended on its outlines being invisible to the American public. So I wanted to warn him, in effect, that the trend of policy was visible, at least to some people, including me, who were telling other people about it.

In other words, I thought of leaking information into the White House about what was actually visible from the outside to try to make them understand that their policy was foreseeable. The more foreseeable it was the less viable it might appear to him.

What is San Clemente like?
We went in through a gateway and a voice came out of nowhere like the voice of God from a loudspeaker on top of the guardhouse, telling us where to park. I may be confusing this with parking lots when I would visit friends in prison, but maybe they just borrowed the technology for the White House. It’s my memory that this unseen eye was controlling your movements.

You go into an outer waiting office just like a dentist’s waiting room, but with large color photographs of Nixon lining the wall. In fact, the official photographer stopped in the lobby and chatted with us until he dashed out the door as a pink golf cart went by. There was one person aboard driving it like a little electric Disneyland car about seven and a half miles per hour. It was Nixon, scowling, and looking very grim. His shoulders were hunched over and he was piloting this thing like the engineer of a toy train. Right behind him was another pink golf cart being driven by Bebe Rebozo, and behind that, a third pink cart with two Secret Service men. A convoy.

Finally we saw Kissinger for lunch on a little patio. General Haig was at the table. As we all said hello, Kissinger, in his usual fashion, turned to my friend and said, “You know, I have learned more from Dan Ellsberg…” and I thought he was going to repeat his statement about Vietnam, but he seemed to hesitate, and then said, “about bargaining than from any other person.”

I was taken aback. I didn’t know what he was referring to, although my academic specialty had been “bargaining theory.” And suddenly I remembered that 11 years earlier when I had given a series of talks on “The Art of Coercion,” I had also given a couple of those lectures to Kissinger’s seminar at Harvard. “You have a very good memory,” I said. And he replied, “They were good lectures.”

When I rethought that incident later, it made the hair on the back of my neck stand up. The lectures I had given had to do with Hitler’s blackmail of Austria and Czechoslovakia in the late Thirties, which had allowed him to take over those countries just by threatening their destruction. One of those lectures was “The Theory and Practice of Blackmail.” And another was called, “The Political Uses of Madness.”

News leaks about the Cambodian invasion, obviously coming from off-the-record backgrounders by Kissinger, had revealed a major motive of the invasion was to convince the Russians and the Chinese that our decision-making was unpredictable, and that since we could do something so apparently unpredictable and crazy as invade Cambodia, they could not count on our reasonableness or prudence in a crisis.

That was Hitler’s conscious policy: the threat of unpredictability. I had described it in my lectures as being a possibly effective, but extremely dangerous strategy. It was a commitment to madness. To realize – not that Kissinger had learned this tactic from me, which is very doubtful, but that such a thought truly was in his mind, enough so that he remembered the analogous thesis that I had presented 10 years earlier – this was chilling. It confirmed the nature of his policy and where it might go.

We talked for just a moment before lunch was served. My friend immediately got him into Vietnam, but Kissinger said, “Well, we are not here to talk about Vietnam.” He looked at me quite nervously and made it clear he didn’t want to talk in front of me. I assumed that it was because he wanted to lie to my friend in ways that wouldn’t have been easy in my presence. Kissinger began drumming on the table with his fingertips and then suddenly said, “Tell you what, Dan, why don’t you and General Haig have lunch together while we talk on other matters, then we will all get together.” So, he actually did, after all, pass me off to Haig, with whom I went off to the other side of the house and had lunch.

Haig was very pleasant and relatively forthcoming. I decided that I would try my strategy on him of “leaking in” the Kissinger strategy. We talked about an hour when Kissinger joined us. He said he wanted to talk with me and we should set up a meeting for his next trip out.

You went to San Clemente in the middle of your honeymoon and he evaded you…
The night before I went, a bunch of Rand people came to my house, including some of the people who had written the letter to The New York Times. One of them was an expert on the Vietcong. It was his thesis that the Vietcong, and I’m speaking in particular of the Southerners but also of the North Vietnamese, simply were not susceptible to coercion. He said, “The Hanoi government may be the most popular government in the world,” and that the people were unique in that you couldn’t get them to criticize their regime. The Vietcong defectors, and even the few North Vietnamese defectors, might criticize their immediate superiors or might be mad that they hadn’t gotten a promotion, but none of them, even under probing, would say anything critical about the way the system worked. They regarded their cause as just. Their main reason for leaving was problems in their family or the general hardness of their life; and they felt ashamed for having defected. He said I should tell Kissinger that.

Kissinger himself mentioned at one point in our brief conversation that he had not been able to talk to any real experts on Vietnam – “there were none” – and I gave him this man’s name, and urged him to get him down from Rand. But he never called him.

And then?
I went to San Clemente again, and again wasn’t able to see him. But on one last visit I did see him. I was kept waiting, and when I finally got in he said he could only see me for half an hour. He started out by saying, “I’m very worried about the Middle East situation,” which was very much in the news. Kissinger had been leaking a lot of his critical thoughts about [Secretary of State William P.] Rogers’ handling of those negotiations, which Kissinger was not then allowed into. “I’m afraid that that situation may blow up.”

“Well, Henry,” I said, “I want to talk to you about your Indochina policy… I think that may blow up.”

I had in mind possibly handing him a sheet of paper with the Nixon strategy written on it. The night before I had worked to reduce it to a single page, which was fairly complex, and I laid it all down: the slow reduction of forces, threats, demonstrative actions like Cambodia, the likelihood of future invasions, the ultimate mining of Haiphong, and the deliberate deception of the public.

As I recited the policy, he looked at me with very narrowed eyes, in a way that assured me I was not on the wrong track, but he made no response. He drummed his fingers on the table and said, “I do not want to discuss our policy; let us turn to another subject.”

We went on to a discussion of the Pentagon Papers. I asked if he knew about the MacNamara study on Vietnam and he said he did. [I didn’t know then that he had actually been a consultant in the first month of the study.] “Do you have a copy of it in the White House?” He said he did.

“Have you read it?”

“No, should I?”

“Well, I very strongly think that you should. I want to urge you, as a practical matter at least, to read the summaries, which are only a few pages at the start of each volume. Together they add up to 60 pages, which make a very readable story. You really should make the effort.”

“But do we really have anything to learn from this study?”

My heart sank. The major lesson of the study was that each person repeated the same patterns in decision-making and pretty much the same policy as his predecessor, without even knowing it. I thought, My God! He’s in the same state of mind as all the other makers of decisions in this long process, each of whom thought that history had started with his administration, and had nothing to learn from earlier ones.

I was quite depressed, but I went on to say, “Well, I certainly do think so; it is 20 years of history and there’s a great deal to be learned from it.”

“But after all,” he said, “we make decisions very differently now.” And that capped my depression.

“Cambodia didn’t seem all that different.” And he looked uncomfortable and sort of fidgeted in his chair.

“You must understand, Cambodia was undertaken for very complicated reasons.”

“Henry, there hasn’t been a rotten decision in this area for 20 years which was not undertaken for very complicated reasons. And they were usually the same complicated reasons.” Each of these people thought that, unlike their predecessors, they had very special considerations that called for this particular judgment and escalation – reasons that they were hiding from the public, reasons that their predecessors had hidden from them as members of the public. Each person in that office thought his predecessors had made wrong decisions for stupid ideological and geopolitical reasons, whereas he was making the same decisions for quite different, very clever, domestic political reasons. And so, year after year, the war went on.

As he said goodbye he told me, “Now… this has not been long enough for us to talk. I’m very anxious to see you again. Can you come to Washington?”


“Next week, next Thursday.”

Well, that day we were due to be moving from our house to Cambridge, and I said, “Not Thursday because my wife would have to do all the moving by herself.”

“But it’s very important, I must see you then…”

I never did learn why he put so much urgency on seeing me that week. We talked about it a minute and finally I said, “Well, Henry, it can’t make that much of a difference. You know, we’ve waited a year and a half for this discussion. But I could see you the following week after I move, and I’m at your disposal from that time on.” And he said, “We would pay for your travel… you could not be officially a consultant of course…” – Kissinger is very jealous of the use of the word “consultant” – “but we could pay your transportation east.” So I said that I would call him as soon as I got east.

I did call him – a date was set to see him and then, just an hour or so before I was due to go to Washington, I got a call from his secretary that it was postponed and she set another date, and that, in turn, was postponed, and then they set a third date. So I said to his secretary, “Look, I don’t want to come down when it’s obvious he doesn’t have time to see me,” and she said, “No, he wants to see you.” I called half an hour before I left and it was postponed.

They asked me to set another date, and I never called again. The only reason he wanted to see me was so he could say he listened to “everyone – a whole range of opinion – for example, Dan Ellsberg, among others,” and I decided I would just let it drop.

Did you see him again?
Yes, one more time. But in between came the Son Tay raid, and the renewal of bombing raids on the North. I had now been back east for a couple of months and had happily plunged myself into reading for the book I was supposed to do at MIT.

The general feeling about was that, having learned the lesson on Cambodia, they would change their policy and really couldn’t do that again. I remember a meeting I attended at which Bill Bundy spoke and told us that he’d learned his lesson. And I asked Bundy what would stop him from doing it again, and he said, “The kids wouldn’t allow it,” which was a common attitude. The kids would save us without the parents having to do anything. And it was kind of a willingness on the part of dove parents to see their children be the cannon fodder on the demonstration barricades, just as the hawk parents were willing to see their sons inducted for Vietnam. I’d been going around campuses quite a bit at that point, and I just wasn’t sure if Bundy was right.

So, on the day of the 1970 Yale game, the first football game I’d been to in 15 years or so, a day our letter came out in The New York Times – Bill Bundy was at the same game, representing Yale – I was on my way to the Crimson for a post-game party. And on my way there was a sign spray-painted on one of the buildings… “Hanoi bombed.”

Pat [Ellsberg’s wife] and I were due to go to Newport but instead I began calling everyone I knew to see what could be organized as a protest. I’d known from Cambodia that it was likely to go all the way, and now the final chapter was on us.

So, I called and called all over the coast, and, of course, got the attitude of “Saturday evening, forget it.” It was clear that even rallies were not possible. So at this point, in late November, I went into full-time activity.

One opportunity that came for me to speak was an invitation to testify at a trial of the so-called Minnesota Eight, who had been caught in a draft-board raid in Minneapolis. Noam Chomsky, who had been invited, wasn’t able to go, so he recommended me. I immediately flew out with a bunch of the Pentagon Papers in my suitcase, in hopes that I would be able to give a kind of testimony that would make it appropriate to offer these Papers as evidence in a trial. Then the defense could subpoena the Papers from me and get them into the court record. At least, the judge would read them. He might or might not release them to the public, but at least they would be part of the court record and would be seen by an appeals court, and might, somehow, get judicial reaction on the war.

The night before the trial, the lawyer defending the Minnesota Eight questioned me on my background to establish me as an expert in court. He asked if I had ever worked for this administration. I said I had, but I couldn’t talk about that because for two years I had kept my mouth shut at the request of Henry Kissinger, and it would embarrass Rand if it came out. I just couldn’t mention it, even though it would add greatly to my authority in court.

The next morning I phoned my wife, and she mentioned that I’d received a call from Don Oberdorfer of The Washington Post. I had an hour to kill, so I decided to call Oberdorfer. Incidentally, the FBI later turned up and questioned the guy I was staying with about this call. Oberdorfer said he was doing a two-year wrap-up on the Nixon policies, including Vietnam, and he had asked Kissinger what the origin of the current policy in Vietnam had been. Kissinger had told him, he said, that, “Ironically, certain people who are now great critics of the administration had been crucial in the development of the policy,” in particular a guy named Ellsberg.

“Kissinger mentioned me?” I asked.

“Yes, definitely,” he said, “that’s where I got it. He also mentioned Halperin and Schelling,” who by now also were public critics of the policy. “But he said you had been involved in the study of alternative policies and questions.”

I was absolutely amazed by this. I asked, “What did Kissinger say the policy was?” Oberdorfer said, “It’s the policy of negotiating in Hanoi while withdrawing the troops from Vietnam.” I said, “Look, if that were the policy, I would still be at Rand, and Mort Halperin would still be in the White House. This guy is trying to smear me as being implicated in his rotten policy.” I then explained to him what I thought the policy was, including the threat of escalation, the residual force and so forth, and Oberdorfer was very surprised. (After talking to a lot of other people, he concluded that I was right, that he had been misled by Kissinger.)

I went into court a couple hours later, and when I was asked, I said I had just learned that Henry Kissinger had revealed it on the record, and that I did work as a consultant for the Nixon administration. It was the first time I’d ever mentioned that.

As it was, I didn’t get to offer my documents as evidence; when I said I had participated in official lies, the judge cut me off and turned very angrily to the lawyer and said, “I told you that I would not allow any testimony on this stand critical of the government.” This was in front of the jury. Since they were trying defendants charged with taking actions based on a fairly critical attitude of government policy, this was a rather bizarre constraint on the testimony.

About a month later, I had the chance to go to a conference of MIT students and businessmen who were critical of the war, which was going to lead off with a backgrounder by Henry Kissinger. I’d heard so much, indirectly, of these famous backgrounders, that I wanted to hear for myself how he described his policy. After dinner Kissinger gave his talk, which revealed to me what an enormously effective PR man he is – extremely smooth.

Did he know you were there?
As we went into the meeting, he saw me and came across the room to shake hands. He said, “I must apologize if I embarrassed you at all with Don Oberdorfer.”

“Oh, no, Henry, no problem at all.”

We then went into the meeting of about 60 people. These were businessmen who were not quite as committed as the Business Executives Move for Peace, but were critical of the war. A lot of them were relatively liberal Republicans. The idea of the conference was to bridge the generation gap between students and businessmen. It had the very interesting name, Runnymede – the place where the barons met to present demands to the king. It would not have attracted me except for the opportunity of hearing Kissinger.

He spoke about the “tragedy” of revolutionary movements having these unfortunate unstabilizing consequences, and the “tragic” need to sit on them forcibly. In the question period, one student asked why the rate of withdrawal was not faster and Henry burst out after a couple of these questions and said, “You’re questioning me as if our policy was to stay in Vietnam. But our policy is to get out of Vietnam. We are trending down the war in Vietnam, and I assure you that the war will continue to trend down.”

After quite a bit more questioning, which he handled very convincingly and with great poise, I got up. I had decided on a very careful phrasing of one question to ask him. I figured it was the last time I would be speaking to him.

“I have one question, but I want to make a comment first. You have said that the White House is not a place for moral philosophizing, but in fact, the White House does educate the people by everything that it does and everything it says and doesn’t say. Specifically, tonight you are expressing values when you tell us that the war is trending down and will continue to trend down, and then mention in that connection U.S. casualties and U.S. troop presence. You failed to mention Indochinese casualties, or refugees, or bombing tonnages, which in fact are trending up. You tell the American people that they need not and ought not care about our impact on the Indochinese people, and encourage them to support decisions that ignore that impact.

“So, I have one question for you: What is your best estimate of the number of Indochinese that we will kill, pursuing your policy in the next 12 months?”

He was completely stunned. It was very startling to the audience because it was the first time he’d shown any break in his poise at all. He almost turned around and paced for a second, then looked at me very penetratingly and said, accusingly, “That is a very cleverly worded question.”

I got somewhat angry at that. “I am not trying to be clever at all. That is a very fundamental issue. Can you answer it?”

He thought for a minute, there was silence, and he said, “You are accusing us of a racist policy.”

“Race is not the issue here. Let me put it – how many human beings will we kill under our policy, in the next 12 months?”

He paused again, and then burst out, “What are your alternatives?”

“Dr. Kissinger,” I said, very firmly, “I know the language of alternatives and options very well, and it has nothing to do with this question. I’m asking you how you estimate the consequences of your policy in the next 12 months, if you have an answer. Do you have an estimate, or not?”

There was another long silence, and the student who was hosting the evening got up and said, “Well it’s been a long evening, and I think we’ve had enough questions now. Perhaps we should let Dr. Kissinger go back to Washington.”

The businessmen there were pretty angered by this episode – not so much because they opposed the war, but because they wanted to get on good terms with the people of their children’s generation. They wrote quite a heavy anti-bombing resolution, describing it as murderous and criminal – it was amazing for these people to write.

At the very end, when they were voting on this resolution, I suggested that since these people were fairly wealthy men who were big political contributors to both parties, they should make a resolution that they would donate no money to the political campaign of anyone who was not willing to make a public commitment to ending the bombing and the war.

This absolutely tore the meeting to pieces. They were all furious, Republicans and Democrats alike, because, as they read it, I was challenging them to put themselves on the line in front of these students and friends of their children. I was surprised; it hadn’t occurred to me that they wouldn’t be willing to make such a commitment.

The next morning we were having a session in which I described the Nixon policy to a small working group that included Osborne Elliott, the editor of Newsweek. And I talked about the likelihood that the next move, at best, would be the invasion of Laos, and at worst, the invasion of North Vietnam. And Elliott was extremely skeptical of what I said. He said they had no indication of any such threat of escalation.

After a break, Elliott came back and said, “Ellsberg, apparently there is something in what you were saying. I have talked to my New York office, who just heard from our Saigon office that a total embargo has been put on news from South Vietnam.” It turned out that at the moment Henry Kissinger was telling us that “the war is trending down,” the pre-invasion bombardment of Laos had begun.

Anyway, that, of course, was the last time I was ever inclined to see Kissinger.

What role do you think that Kissinger played in your prosecution?
There was a frequent story at that time that Kissinger had been the pusher of the prosecution, because he was worried about the China negotiations. I could never take that seriously at all. For one thing, the China trip was announced only a month after the Papers started coming up. Besides, the secrecy of those negotiations was, as far as we can tell, entirely our desire, not the Chinese desire. But that didn’t rule out that he might have been the pusher for some other reason.

Did Kissinger ask for the psychiatric profile on you?
CIA Director Richard Helms has testified it was Kissinger’s request that a psychiatric profile be done on me. That came to Helms directly from David Young, who was Kissinger’s assistant and who remained formally on his staff even though he was working with the Plumbers. If Kissinger did ask for the psychiatric profile, it means that he read it, and that would confirm what I felt sure of anyway: I’m sure the president read it. [Bernard Melloy, the CIA psychiatrist who supervised the psychological profile of Ellsberg, has also testified it was his understanding, from Young, that the request came from Kissinger and Ehrlichman. Kissinger has flatly denied any role.]

There is certainly a dramatic circumstantial case to be made that Kissinger was aware of the burglary prior to its taking place.
Given that he asked for that profile, read it and was dissatisfied with it. The burglary was a direct response to his dissatisfaction. There’s every reason to think that Young reported everything to his former and future boss; that’s the way the system works.

Would you say it’s fair to see Kissinger and Ehrlichman as relatively equivalent figures in the administration?
You mean Ehrlichman, Haldeman and Kissinger.

Haldeman’s sort of a staff man, in the middle, but Ehrlichman and Kissinger…
I think Haldeman wasn’t just staff. I see more policy. Haldeman on domestic politics, which is the heart, the central problem, is certainly a key person.

As key as Kissinger in foreign affairs?

Would it be fair to say the way they all operate is the same? How about Kissinger’s phone taps?
Look, Kissinger lied when it came out about those taps. His first reaction was to say he had had nothing to do with it. Then it came out that he had been the one who specified who was to be tapped. Likewise, the story that Kissinger was concerned with leaks is absurd when you look at the people, especially the newsmen, who he had tapped. These were people to whom stories were leaked almost entirely by Kissinger: Marvin Kalb, Henry Brandon, Joe Kraft.

What are we to make of that? Part of his thinking was given by a White House aide who said Kissinger wanted to see whether, behind his back, these journalists and his own White House assistants were loyal to him. And he wanted to find out if these people had other sources that were contradicting what he was saying.

Nobody has made this inference: Imagine the ability Kissinger acquired to manipulate those individuals with whom he was working and talking to daily or weekly, when he secretly knew all they were saying to their other sources and associates. In other words, Kissinger knew by these wiretap logs whether he was getting through to those guys, what to emphasize, how to change his pitch, how to counter a position credibly, and very persuasively, in a way the newsmen would never recognize was based on secret knowledge. And it was precisely the people who were most friendly to him that he had a great desire to be able to control.

In fact, the leaks from the FBI indicated that he had asked directly for those taps and that they had insisted on presidential authority for such taps. That was when he turned to the president and got him involved.

So Kissinger may have been the man who instigated the Plumber program, far more than Ehrlichman.
Definitely, yes. Ehrlichman was not involved in initiating those tasks. And the monitoring of them was first done by Haig and Kissinger for the president, with copies to Haldeman, then later directly for Haldeman. And Young, the head Plumber, came right from Kissinger’s personal staff.

Kissinger got into domestic policy more than has yet come out, and he may well have been involved in far more tapping. At a certain point, they became worried about Hoover’s knowledge of these wiretaps and shifted them away from the FBI. They say the FBI stopped its wiretaps in February of ’71, but they never have explained why. I suspect that they shifted wiretapping to a different agency – the Army Signal Corps, the White House communications center, local police or some other agency.

I’m virtually sure that they were directly wiretapping me either by that point, or after June, when the Papers came out. There have been stories in The Washington Post that I was personally wiretapped from May on, and I don’t see how it could be otherwise.

Nevertheless, they always did deny it, and that’s one of the things that remains to come out. I also suspect that they tapped a lot of other people they wouldn’t want to admit to – probably every Senator who potentially was running for the presidency, and the Senate’s anti-war leadership. And that is something like 15 percent of the senators.

How has Kissinger managed to escape the curse of Watergate?
He puts out to everybody that he wasn’t involved, and nothing much has come out directly. He runs the best public relations operation in this administration and possibly in any administration.

How does that make you feel?
It confirms a lot of what I know about media people: A lot of them aspire to be part of the executive branch, rather than a fourth, independent branch of government. Kissinger offers them that role, very seductively. He does it, in particular, with publishers and editors, and the influence trickles down to managing editors and writers.

So, he’s been pretty immune. And, the man who, with his boss, has dropped more bombs than any human being in history, bugging and lying as necessary, is perceived as a peacemaker, as a lovable wit, a charming fellow, as anything but the murderous creep that he obviously is.

How would you evaluate Kissinger’s role in this administration in foreign policy?
My impression is that he deserves no serious credit – nothing but very modest thanks – for the best of the policy events he’s been associated with; and he deserves the deepest condemnation for much of his work.

To start with the best: I base this on conversations with people who were in the administration, on columnists’ leaks, and backgrounders. It seems probable that the credit for the change in our China relations, which is certainly a very good event, goes to the Chinese – particularly to Chou En Lai – and to the pressure put on them by their conflict with the Russians. They made the policy choice to change their relationship with the U.S. as a counterbalance to Russia.

Secondly, credit goes to Richard Nixon. There’s a lot of indications that Nixon conceived the possibility of a change in our relations with China long before he was associated with Kissinger and before he became president. As early as 1967 or 1968, he spoke about it and hinted at it in his Foreign Affairs article. So my suspicion is that on the U.S. side, it was primarily Richard Nixon, and that Kissinger just went along.

Did Kissinger bring anything to it?
I would think essentially nothing. What is there to it, once both sides have made the decision? There was nothing involved that called for any particular amount of negotiations. Essentially the agreement was for a change in atmosphere to have a willingness to talk to each other. The content was far less important. As far as we are aware, there was no deal that came out of it respecting Vietnam. The only real complexity would have been managing the impact of our changed relationships with Taiwan and Japan, and that was handled miserably.

What about his actions in relation to Russia?
His major achievement was to delay the SALT [Strategic Arms Limitations] agreement by about three years. He could have had, in 1969, the basic achievement in terms of an agreement that had been hammered out internally in the U.S. government in 1968. The impression was that he was unwilling to go with that agreement in 1969 because it was a Johnson agreement, and had to be reworked. The crucial effect of the delay was to encourage the Russians to proceed with the development of MIRV [Multiple Independent Re-Entry Vehicles], which is very bad for humanity, indeed. It could possibly have been averted if we had been willing to make an agreement in 1969. I wouldn’t put all the blame on this administration, for Johnson did not focus on MIRV either. But still, the crucial time was 1969, when we insisted on going ahead with our own testing. Kissinger never had any particular appetite for Arms Limitation agreements of any kind. He was not an enthusiast for them, nor had he any confidence in them. So, I think his effect there is terrible, absolutely terrible.

On Vietnam, Kissinger’s effect has been not only disastrous but criminal in every respect. The judgments involved were stupid, blind, arrogant, criminal and murderous, and the murderous implications of the policy were all realized. We’re talking about men who loosed four million tons of bombs on Indochina.

In the Vietnam talks you had some role for a skillful negotiator and I would assume that, in terms of talent and ability, Kissinger is a capable negotiator; but how much leeway was there? The basic objectives, the constraints set by both the president and himself, precluded any real deal for the first three years; so his talent counted for nothing during that period.

In the end, when they were willing to settle on both sides, it’s very doubtful that his skill counted for anything to speak of. As we can see, all the fancy flourishes in the negotiations seemed to have been totally facade. As a “diplomat,” it was only in the administration’s domestic diplomacy with newspaper publishers and columnists and with congressmen that Kissinger’s conman talents were indispensable and totally successful in mobilizing support, buying time, dissembling the true nature of the policy: “selling” an endless, escalating war.

What were your feelings when the bombing of Hanoi was announced, after the “peace is at hand” statement and the election?
Despair. Horror. I found that I had reserves of naivete left in me after all the years in Vietnam: I really concluded after Kissinger’s announcement on October 26th that it was close to impossible they could toy with the beliefs of a nation to the extent of announcing just before an election that peace was at hand without being ready to back it up.

When the bombing came at Christmas, it took me completely by surprise. In retrospect, I think the bombing had an extremely cynical purpose which was largely achieved: to make it appear that Nixon’s four-year strategy of reliance on bombing and threat of bombing had, in fact, been effective in bringing about the settlement.

In October, there had been no escalation for some time. In fact, the rate of bombing had dropped, and there was no indication that a settlement represented anything but a lowering of aims on both sides – which is what it did represent. It was very widely understood that the mining of Haiphong had accomplished literally nothing, nor had four years of bombing. So Nixon finally prepared to accept a settlement in which the North Vietnamese did not withdraw their troops, which had been his essential demand ever since 1969. But he felt uneasy at signing a settlement that quite obviously came from having lowered his demands.

So you think they bombed Hanoi and killed all those people to make a failed policy look like a success – and not even facing an election?
By design. With the assumption that Nixon would later sign a settlement – essentially the same that had been available before the bombing – that he could then attribute to the bombing and count on a public presumption that not only had this bombing helped, but all earlier bombing had helped as well. Unfortunately, I think his tactical judgment was right; it did teach a “lesson” that’s both mistaken and evil.

The Hanoi bombing was one of the worst acts in the history of civilization. And I don’t say that rhetorically. I wouldn’t bother to argue whether it was precisely worse than one or another act of mass murder in our history. It’s not a matter of body counts or scale. In terms of the exact political and cultural context, above all in terms of a war that had been essentially settled in October, by a country whose public not only accepted such a settlement but had just given the man who had announced that settlement the largest election landslide in our history – that B-52 bombing in highly populated areas was one of the coldest, most murderous acts that any ruler has ever undertaken.

I couldn’t help remembering this when I watched Nixon in front of the POWs, in the midst of their entertainment, proposing a toast to the brave fliers of the B-52s. He encouraged these men, who thought they owed him their freedom when they really owed him their imprisonment, to stand up in front of television cameras, in the eyes of the nation, and toast the agents who had carried out his orders for possibly the worst thing that this country has ever done.

When did Nixon conceive this act?
Just after he first took office. What hasn’t come out yet in any of the Watergate discussions is the connection between what he was doing domestically and what he was doing in Indochina. I think they’re very intimately related.

I believe that Nixon’s aim was to go into the election of 1972 having achieved a victory in Indochina – not the kind of settlement he did achieve, but something much more ambitious. I believe he had a plan for doing this, some parts of which he had on his mind as early as 1967 and ’68. Perhaps he had the whole plan. He began to put this into effect from the moment he took office, even as he carried out the parts of the plan to confuse and deceive the American public as to what he was up to.

How did you learn this?
From half a dozen people, but the major source was Mort Halperin, whom I’ve known for a number of years. Mort had followed me as a special assistant to Assistant Secretary of Defense John McNaughton at the Pentagon. Later he’d gone on to work for Henry Kissinger. When I worked for him just after he’d been named by Nixon, and I knew the kind of information that Mort had available to him. I was able to judge it pretty well. Mort told me Nixon did have a plan, that he had in effect chosen one or a combination of the long-range options we had drafted for Kissinger and him just after he was elected.

The plan Nixon announced, on November 3rd, 1969, was “Vietnamization.” He described it as having two parts: negotiation for a just settlement with the North Vietnamese, and withdrawal of our troops to be replaced by Vietnamese forces. He implied that we would totally withdraw our troops within a very short time. Most media editors and owners were led to believe, by Kissinger, that we would be out of Vietnam within 12 months, even though the president had talked about assuring that South Vietnam had a chance to survive and so forth.

Mort and several others, in particular John Vann who had very close contacts with the Department of the Army, first told me one part was being deceptively presented: Total withdrawal was not in mind, but a very slow reduction of troops, as slow as Nixon could get away with, politically, down to a large residual force that would stay indefinitely.

So what was the real Vietnamization policy?
In fact, Nixon people didn’t expect ARVN to improve all that much; they did not intend to withdraw all American forces. And of course they weren’t seriously negotiating.

But there were about five or six other dimensions to the program which were concealed from the American public. At various times, one or another part became overt, but even then, the remaining pattern was to be concealed. This pattern included the bombing of Cambodia, heavy bombing of Laos, ground operations into Cambodia and Laos, renewal of the bombing of the North – which, you remember, had been discontinued in late October ’68 – the mining of Haiphong harbor, and the total unrestricted bombing of North Vietnam.

On several of these, the commitment was first to a threat, and then a commitment to carrying them out if the threat failed. But since these threats, realistically, were almost certain to fail, one could predict very well that they would be carried out.

So what I’m asserting is that all of the things that we have seen, including the bombing of Hanoi in December of last year, were all explicit in private plannings going back as early as 1969.

In May of 1969 – and this I was told by Halperin in September – on the same day Nixon delivered his most conciliatory speech on Vietnam, looking toward a settlement [a speech largely drafted by Halperin], Kissinger had Dobrynin, the Soviet Ambassador, come to the White House. He showed him the speech and emphasized the passage which said, in effect, that “no one has anything to gain by waiting.” And Kissinger made clear to Dobrynin that meant Nixon was prepared to escalate if the Russians did not bring about a settlement acceptable to Nixon.

To make that threat credible, there already had been some escalation, including bombing Cambodia. Halperin referred me to a nearly unnoticed piece by William Beecher in The New York Times in May of ’69 that reported the secret B-52 bombing of Cambodia. (We now know that story was at least the catalyst for Kissinger’s bugging of 17 newsmen and officials.) Later we learned there had been a ground operation into Laos in the spring of ’69. The Veterans Against the War brought this out, including Marines who had participated. There were also probes by Special Forces troops into Laos and Cambodia. And, I was told, just before the Republican convention, frogmen had been sent into the harbor of Haiphong in early 1969 to check it for mining and were given instructions to leave evidence that they had been there.

So, before the threat was given directly to Dobrynin in May, we had escalated in all three French Indochina countries outside of South Vietnam and with all three services.

It fit with a statement Kissinger had made to me at the Hotel Pierre in December of ’68. I had argued with him that the threat of escalation was close to worthless since the North Vietnamese had withstood years of bombardment. He complained that “without the threat of escalation, how could one seek to negotiate.” I thought his point of view was casual at that point. Last month, four years and four million tons of bombs later, he was still pleading with Congressmen not to cut off his bombing of Cambodia because that would leave him naked of bargaining power.

By September, 1969, I knew we were going to remain in South Vietnam much longer, and on a much larger scale, than people realized, and that it would mean continued bombing in South Vietnam and the prospect of escalation. That month Halperin predicted to me, “The President will not go into the 1972 election without having mined Haiphong.” I took this to mean that Nixon did not want to face charges from his rightwing critics – like Reagan or Wallace – that he failed to use a tactic the military considered for long to be a winning tactic; whatever deal he might reach with the North Vietnamese, he would want to say he had tried it.

When I worked for Nixon early in January 1969, it was on the assumption that almost anybody coming into office would take advantage of his new slate to write off the Vietnam War. Looking back now, it’s clear what he had in mind was winning it: The other side was to be blackmailed into accepting an indefinite stay of American troops or of the Thieu regime by the threat of far greater violence than had ever, even by Johnson, been visited upon them. That was going to bring about, he thought, a victory-type settlement – at least a mutual withdrawal of the troops. That’s of course what he did not get in ’72, never has, and never will.

Has there been, even now, a change in the basic United States commitment to the Thieu regime?
No. On the contrary, I think Nixon definitely has in mind maintaining it – not a commitment to Thieu himself, but a commitment to maintaining a U.S. regime in Saigon. It would be somewhat unstabilizing to our policy to have another switch in Vietnamese leadership, but the basic policy could easily survive with different leadership. The idea that General Thieu is indispensable is not true.

Why do you think you failed to judge that situation correctly in late 1968? You were expecting a new president to approach the situation fresh, yet knowing well what Kissinger’s theory of world power was, knowing the type of man Nixon was…
Oh well, nearly everybody made that mistake. It was perfectly plausible. Frankly, once I had changed my perception late in 1969, I was never able to convince anybody. Everyone was shocked by Cambodia, but they quickly accepted the notion that he had made one mistake, and he wouldn’t repeat it. They wanted to believe he was getting out.

You see, with all the power the president has to contradict and retaliate against a critic, to disprove him you need a document. And great resources of the system are mobilized against the leak of documents.

People who wrote memoirs – and who hoped to be employed by some future president – would tell all the substance of the information, but they would be careful not to disclose that they were directly quoting from a document. It’s not necessarily because they were partisans of a particular president, but there are certain things an insider doesn’t disclose about a president, even one from whom he has resigned. Not only can the president still retaliate, but if you plan to stay in the market for future executive service, you have to show discipline in your discretion; you have to make it quite clear that no matter what the provocation, there are some things you won’t tell about a boss – above all, you won’t hit him with documents.

It’s fundamentally like the situation in Orwell’s 1984, where no one strictly believed what the regime said. They were not surprised when word changed from day to day, because there were lots of contrary rumors and hearsay. But one thing the Party was extremely careful about was not to allow any incriminating documents – anything that would show the current line was not what it had always been – to survive. Whenever they changed the line, they destroyed or changed all documents.

Orwell was pretty close. We found out in 1972 General Lavelle had a couple of hundred officers and men at work in the evening, after a hard day of bombing, writing false reports of their missions. And we’ve just found out that as early as ’69 the records of bombings in Cambodia and Laos were also being systematically falsified.

It must have been a peculiar arrogance of the Nixon administration to so politely document everything they did at home.
That’s true, but remember the documents coming out now were written within the White House and were never meant to be seen by more than three or four people.

We find the fewest memos by Haldeman, who had good access to the president. But people who can’t just wander into the president’s office have to attract his attention with eye-catching written material expressed in extremely tough and cynical terms, to establish “here’s a guy who will stop at nothing” – or to put it another way – “a very useful person with a very colorful imagination” – a great “idea man,” i.e., a person who allows himself to think of subjects which, for other people, are taboo. They are all competing in this way. It’s only this strange set of circumstances that has allowed us to look at their memos.

But the Pentagon Papers… well, that brings me back to why I thought they were relevant in 1969. What I lacked then was documentary evidence of what I had just been told by Halperin, Vann and three or four others who are still with the government and who I don’t want to identify. But I did have something unusual. I had the Pentagon Papers. Unfortunately they ended in March of 1968. But the promise they held was that the pattern of 1964-65 was close to what was now a prospect; and if I could show that at least once in the past an administration had acted in a conspiratorial fashion, people should at least consider the possibility that it was happening again.

Moreover, since it was a story about a Democratic administration – several of them, really, including Truman’s – releasing these documents would make it tempting for Nixon to throw the responsibility for the war on Democrats, where it mainly belonged, and bring it to a quick halt. It was the only way I could imagine he would ever write it off. I felt very urgently about getting the Papers out before he committed himself publicly. I thought he was evenly balanced at the time.

I had given all the Papers to Fulbright, but his initial reaction, that they would be very good material for hearings, changed after November 3rd, when everybody accepted the notion that Nixon had a new policy of getting out. He now felt that the Democrats’ policy was very interesting, historically, but had no relevance to what Nixon was doing.

Then in May of ’70, when the Cambodian invasion took place, Halperin resigned as consultant to Kissinger, and three members of Kissinger’s staff left in the dispute. They were the first officials ever to leave the government in protest over the Vietnam War, and the only ones since. Unfortunately, none of them was willing to really speak out, except for Halperin, later.

For a brief period after May, the Senate Foreign Relations Committee staff was quite enthusiastic about setting up Pearl Harbor-type hearings on the war, using the Pentagon Papers, but then the mood went away within a month or two. As soon as the troops left Cambodia, everybody said Nixon wouldn’t do that again. I, Mort Halperin and a few other people knew that he was going to do more, but there was no way to convince the others.

You asked how I could have thought Nixon would change the policy? But really, everybody thought that, for what seemed perfectly natural reasons. I think the harder question to answer is, why didn’t they change the policy? How could they have possibly dreamed of winning the Vietnam War as of 1969?

I think the best answer is that both Nixon and Kissinger have a deep and fundamental belief, an almost mystical faith, in the effectiveness of threatening and causing pain or death. They believe it legitimate to inflict almost any amount of pain and death necessary to enforce their threats.

Throughout all these years, few people had an overall grasp of what was going on in Vietnam. In fact, nobody seemed to have an overall grasp, an adequate comprehension, of the forces that were keeping us involved. How could that occur?
Limitations within the intellectual framework under which each faction approached the problem. The “radicals” had a broader perspective and were led into less error than practically any other group. But they had very little comprehension of the way things actually operated within the bureaucracy. And they paid relatively little attention to the question of power factions and struggles within interest groups or between interest groups. Then too, they paid much less attention to racist considerations or the psychological connections.

The Women’s Liberation movement, for example, coming from a different direction, is quite able to see the very strong appeal to male chauvinism that’s central to imperialist dogma. Certainly Nixon, whose personality, whose adult life, has coincided with war and Cold War, has from start to finish appealed to the fear of weakness, of avoiding a feminine surrender to aggressive Communism. The politics of masculinity. His charge is always of being soft on Communism, being seduced by Communism. And notions that are a little more abstract – irresolution, and indecision. That was the attack on both Stevenson and McGovern.

I think Nixon is now shifting the ideology considerably to a much more general and explicit imperialist appeal of being Number One in the world, since now we’re competing with these Communist giants – and with West Europe and Japan – on more or less equal terms. So it’s a non-ideological approach in terms of the Cold War, but very ideological in the older, Rudyard Kipling sense of imperial responsibilities for maintaining world order and our own supremacy.

You have written that war is good for the health of the executive branch, especially secret wars, major crisis and limited wars with large potential. Do you think the war in Vietnam was sustained deliberately for the health of the executive branch?
We came out of World War II with an enormously enlarged executive branch. It followed a great enlargement during the New Deal, an enlargement which had been falling into some disrepute at the end of the Thirties since it had not, after all, solved the economic problem. But the mystique of the executive solution was restored by the Second World War.

The range of what it was legitimate for the president to do, on his own initiative and secretly, was enormously broadened, almost limitless, by the end of the Second World War. We allowed the president to prepare the atomic weapon secretly and to drop it on his own initiative. There wasn’t much by that point that an American president wasn’t allowed.

What do you think the effect of World War II was on its middle-level managers, the social managers and the executive and economic managers?
They grew up in a state that was run entirely by the president, with great popular acceptance and with every appearance that it had to be that way in fighting Hitler, and maybe in counteracting the Depression; a state in which Congress was almost totally discredited as isolationist. Moreover, right after the war, Congress looked peculiarly reactionary and parochial, Southern and farm-oriented. So they had no respect for Congress, and no respect for the courts, which had not yet taken the positive initiative in civil rights and civil liberties. You had a press which had been very subservient during the war, and a labor movement, the most radical elements of which were devoted to a no-strike policy.

All these institutions seemed devoid of initiative, glamour, respectability, power or legitimacy. The executive branch was the only game in town, and one of enormous power and fascination.

Do you think the way in which the war was conducted affected the social and ethical fiber of the men who ran it?
The process of attracting upwards people whose values and abilities were suited to large-scale managerial operations rewarded those with an orientation toward power and extreme organizational loyalty. So we developed an enormous managerial executive government, with growing affinities toward the managerial governments in Russia and elsewhere.

Have you heard of the Dupreal Theorem that the United States becomes like the USSR as it responds to the Soviet Union as an aggressor by imitating its techniques? Thus, the USA becomes an aggressor.
I’ve just come from a conference of the War Resisters League, which was started by World War II conscientious objectors. The World War II COs were not at all incapable of seeing the evil of Hitler, but they had an awful lot to say about the costs of going into combat – that getting into an enormous fight with this powerful enemy was going to change the United States, and this direction was going to be reinforced precisely by the evil characteristics of the Hitler regime.

In other words, the process of making the United States into a huge machine for killing, on a vast scale, an enemy that seemed to require very inhuman techniques, was going to have very evil consequences for us. There’s no question now that the effect went much further than we were generally willing to recognize. We have become a country capable of preparing the destruction of all life on earth.

Remember, it was unthinkable by us to use a weapon like the atom bomb as of 1940 or ’41. Even our Air Force officers were genuinely shocked that the British were led to a strategy of city bombing and the bombing of industrial housing. It took four years until we were ready for the bombing of Tokyo and other cities in Japan on a massive scale. Having done that, it was perfectly natural to use the atom bomb.

This came at a time when the American president was less subject to having his powers monitored or challenged than ever before. Suddenly these scientists handed him an absolutely unprecedented power. The success of the Manhattan Project seemed to legitimize the unprecedented secrecy that accompanied it. It introduced the whole nation – as did a lot of wartime operations – to the notion that there are secrets the president must keep from the world, even at the cost of keeping them from his own people.

However, the nature of the information was such that you couldn’t keep it a secret very long from other countries. It was just a layman’s illusion that you could. But the illusion served to justify extending a program of clearances, a postwar phenomenon under Truman. A general system of classification for the first time extended to the civilian branches of the government.

The nuclear field seemed to involve secrets that must be kept for a lifetime. It developed of a community whose careers were dedicated to secret information – large bureaucracies where not only the day-to-day decision-making was kept invisible, but also the results and basic purposes of what they were doing.

This was a development essential to the things we’re seeing lately – the ability to initiate secret wars and massive bombing campaigns without anyone leaking any of that information. Or even the Watergate operation, which didn’t leak at all until the break-in, and not much after that, during the campaign

What were your personal experiences with secret decision-making and with the operational aspects of secrecy?
During my first period at Rand in 1958, I was working on the problems of the vulnerability of our retaliatory force, basically our Strategic Air Command. This information was very little known even inside the Pentagon, and was precisely the kind of thing that should not be revealed to an enemy. So I learned the process of secrecy in a context that made it seem very legitimate.

Later I studied crisis decision-making, such as the Cuban missile crisis, a legitimate kind of very fast, intense maneuvering where it was essential that an adversary not know what you were about to do. It was during that year, with very high-level access to all our secrets in the State and Defense Departments and the CIA, that I began to see that the very process of keeping secrets created problems in decision-making. Things were compartmentalized and kept by one section of government from another, often creating the crisis or making it more intense.

I also became aware of the degree to which domestic politics enters the process of foreign policy decision-making. Foreign policy serves the purpose of increasing the chances of reelection of an administration, or of avoiding the chances of being thrown out of office. I began to see the esoteric motives behind a good deal of our policy, but at that point I wanted to learn more from the inside.

I got the chance to take part directly in Vietnam staff work within the government and went to Vietnam during the first year U.S. troops were there, 1964-65. I was working with data which was being concealed from, and lied about to, the public – very largely because an election was going on. In the 1964 election, the policies of the opponent were ridiculed and condemned as insane, irresponsible and reckless, when every civilian and military adviser to President Johnson believed the policies were viable and took it for granted that the president would commit himself to those policies after the election. That expectation was carried out.

When you first started getting romanced by this access to secret material and the mysteries, how did you personally react to this knowledge?
Well, a lot of those secret facts were guesses as to the Russian missile buildup – which proved to be quite wrong. We were responding to a situation that was not developing. And that was a great jolt to me in 1961, discovering that the Russians had not built up their missile forces. That, too, was a secret piece of information. From that time on I became preoccupied with how the government could go so wrong in its perception and its predictions.

The explanation really came to me after I left Rand in 1970. I began reading the revisionist histories of the Cold War, and saw to what extent the perception of powerful dangers was crucial to the expansion and predominance of the executive, to selling an expansionist policy and, in particular, a large arms budget, to the public. A lot of these policies were seen as important to our economy, not only to certain specific industries, but also to our overall export trade and, perhaps, to the avoidance of a postwar depression.

You’re describing a situation where, in response to actually a totally misstated set of foreign circumstances, the United States built up an arsenal and capability without reason.
With pretty good reason, I’m saying. With a strikingly coherent reason, but one thoroughly different from what was presented to the country. Apologists for these policies were mainly people who didn’t know much about how the policy really was made. They were consciously exaggerating certain “threats,” and were very prone to misperceive other events in a certain consistent way. Again and again they saw, for instance, a coherent worldwide Communist policy that didn’t exist.

Was this a deliberate policy inside the executive branch? Did it increase their strength?
Yes, very much so. And it also was for outside interests that sustain the executive branch, particularly industries that were concerned with international trade. I’m talking about the auto industry, arms, IBM, oil, obviously, and fruit, public utilities, mining in Latin America, ITT; and then a growing number of industries – Wall Street, and international banking firms – with great financial interests in Western Europe.

Was there a coherent body of thought within these multinational and heavy industrial corporations, a conspiratorial aspect that brought influence to bear on the executive branch? Or was it a confluence of separately made decisions?
This is the kind of thing I want to do research on. The basic revisionist thesis is that a number of men late in the Roosevelt and Truman administrations feared an economic recession in this country after the war, and they say this, questionably, as the inevitable result of a collapse of our trade with Western Europe. These were people with economic interests in Western Europe.

To prevent a collapse they tried to get various economic policies out of Congress, including low tariffs – in which they did not succeed – and foreign aid for European recovery, to sustain U.S. exports to Europe. But they ran up against a Congress dominated by Midwest Republicans who represented economic interests relatively local in nature. These congressmen were not terrifically impressed by the need either to maintain our export trade or to help Europe help itself. But they were anti-Communist, and they did prove receptive to the need to fight against atheistic Communism.

So, these huge financial enterprises consciously and cynically presented Congress with that threat, exaggerated enormously with the thought of implementing a very valid and justified policy objective – in this case, getting money out of Congress for the Marshall Plan. Ultimately the emphasis shifted from foreign aid to a very large arms budget, including keeping a large number of troops in Europe, which has the same economic effect as providing credits.

These men set out, in the famous phrase of Vandenberg in his advice to Truman, “to scare hell out of the country,” with the intention of avoiding a recession, the importance of which they could rationalize on 10 different grounds other than their particular economic interests in Europe.

This was a large part of the basis of our Cold War policy. That is not to say that Stalin was a benign force in the world at all; they did see him for what he was – a bloody, paranoid dictator. But he was a conservative ruler who was not at all inclined to expand into Western Europe.

We had this system of “misperceptions” which begin to show a very consistent pattern when you lay them end to end. We find [Truman’s Secretary of State] Acheson, totally “misconceiving” the nature of the Greek situation, which he described as being run from Russia, despite a lot of evidence that it was being carried on despite the wishes of Stalin. The same was true in China. (Ironically, most evidence is, and was then, that Stalin was hostile to the success of strong, nationalistic regimes in areas not contiguous to the reach of the Red Army.)

The United States seemed dogmatically wedded to interpretations that the Korean invasion was the start of a worldwide advance by satellite armies, the notion that the Russians were running the Chinese revolutionary movement, and the notion that either the Russians or the Chinese were running the Indochinese movement. The missile gap in the late Fifties was totally misconceived, but it was an exact replica of the earlier bomber gap.

And in the course of all this we gradually acquired the attributes of what we were warned about very early in the period, even in ’39 and ’40, by these pacifists: the development of a garrison state, of a state dominated by militarist managers of foreign policy, a state that had the characteristics of an armed camp.

Do you think that economic imperialism is basically what brought us into Vietnam, and that an understanding of it is the key to what you call “an adequate comprehension of the forces” that brought us into Vietnam?
On the contrary, I think that this explanation goes so clearly against a lot of the phenomena in Vietnam, that some people are just tempted to say that Vietnam disproves the economic model overall. That was my own inclination for a long time. What is needed is a more complicated understanding of the way the system works.

I think Vietnam, in effect, was a cost of that overall ideology, rather than an objective. You couldn’t stay out of intervention in Vietnam without to some extent questioning, or undermining, the rationale for your overall policy.

Specifically, you had a system in which politicians competed for campaign contributions, and ultimately for votes, on the basis of their ability to manage the Cold War competently and toughly. Nearly every election after ’48 was run with this as a primary issue. Kennedy ran against Nixon, in part, on the grounds that Eisenhower and Nixon had allowed our arms strength and military prestige to decline, that we were relatively weak, and that they had allowed Communists to take over an area 90 miles off our shores. They had “lost Cuba to Communism,” just as Harry Truman was accused by Nixon of having lost China and North Korea to Communism.

In ’64 the contest was not between a dove and hawk, but between a more responsible or less responsible manager of the Cold War conflict: Whom did you prefer to have run the Vietnam War?

McGovern, in ’72, was the first candidate in that generation – and in this he differed from Muskie, say, or the other candidates – to run not as a potential manager of the Cold War. Who knows what he might ultimately have done? But he was offering an alternative to a Cold War policy and preoccupation, and he was questioning the basic premises of the Cold War.

So you think that Vietnam is a result of the overall deception that was created for the economic interests?
Yes, quite specifically the Democratic administration of Truman and Acheson in 1949 and ’50 could not afford to accept the victory of a Communist group, however nationalistic in reality, in Indochina on top of the inevitably coming victory of Communism in China – which they were going to be charged with bringing about by their own lack of foresight.

Yet they would not have had to face this had they not created the lie in the first place, which they were forced to sustain.
They themselves called up the anti-Communist genie out of the bottle, and having raised that genie, they were vulnerable to its being turned against them for their own failures in the face of Communism, specifically the Russian atom bomb in 1949 and the Communist attacks and successes in Korea.

Later Democrats could only really blunt that attack – or deflect it – if they were willing to attack the basic premises of the Cold War. And those premises, by the way, became very shaky with the death of Stalin. Maintaining the Cold War after 1953 meant very resolutely refusing to perceive any of the indications that there were potentials for rapprochement with people like Malenkov, Bulganin or Khrushchev. It meant refusing to perceive, until very late in the game, splits between China and Russia, refusing to see that various insurgent movements abroad were getting very little help or encouragement from Moscow. We really had become increasingly divorced from reality.

If victory seemed unattainable in Indochina, which it has mostly always been, your choices were either to postpone any definitive defeat by escalating to whatever degree necessary, or to pull out and suffer the charge of having lost Indochina, a liability that no one ever wanted to go into an election with. Or, third, to challenge the whole set of premises and, say, “All right, Communists won there, but that doesn’t hurt the United States, that doesn’t mean Peking or Moscow have been made significantly stronger. Our welfare doesn’t depend on that; our economy doesn’t depend on it.” You could have said all that, but you’d be charged with being weak and self-serving. More importantly, if you had gotten to office at any given point, you had done so with the support of groups that were benefited by popular belief in an implacable Cold War conflict. And, that meant that campaign contributions would become very problematic.

So, we couldn’t get out of the war without changing a domestic political system that serves powerful elites in our society.

Except for Nixon who was engaged in the third alternative of changing the attitudes toward the Cold War, i.e., his China and Russia moves; he could easily have made the loss of Indochina a part of that change at a very early point. He was supported by business interests who by this time were nearly unanimously opposed to the war in Vietnam, except for the major military contractors.
In Nixon’s case, it’s not possible to look to economic forces outside the government to explain his actual continuation of the war. Or, his expansion of it. You have to refer to the fact that he was a professional politician who wanted to succeed within a particular political context – he wanted to stay in office.

So what was the position he found himself in? He would, on the one hand, surely have gotten wide acceptance if he’d ended the war; that, in itself, would not have resulted in his defeat. The trouble was, it wasn’t good enough to get him re-elected because the population as a whole could have said, “thanks,” and then voted for somebody else. They didn’t like Richard Nixon, and he knew that.

I think, he figured, “How can I do what Mitchell wanted: hold on to my votes from ’68 and get Wallace’s 1968 votes?” It meant, first, hoping or ensuring that Wallace didn’t run. Then, if Wallace were out of the race, he had to assure that Wallace’s votes went to him rather than to his opponent.

Wallace was a very genuine threat to Nixon – not that he might win the election, but that he might throw it to a Democrat. He almost did that in 1968; he could easily have done it in 1972.

I’ve always suspected that the chance for a true negotiated settlement on Nixon’s part went off the bridge at Chappaquiddick in July of ’69. Because until that moment Nixon had to confront the prospect of a coherent Democratic party line that was very, very challenging on the war. After Chappaquiddick, for at least the space of a couple of years and maybe indefinitely, he really didn’t have to worry about the Democratic party. He could devote himself to a different problem – attracting the votes of Wallace supporters by a tough pursuit of an “honorable” settlement.

He could better appeal to instincts for authoritarian government, racist instincts and some of the chauvinist instincts. And I think, in particular, he wanted if possible to be able to tar his opponent with a weak, feminine appearance, a pro-black appearance and a pro-Communist appearance.

By linking all these things together with a Democratic policy of ending the war that appeared less virile and patriotic than the one Nixon held out for, he would stir up a right-wing backlash that would strip votes from the Democrats and, hopefully, hand them to Nixon instead of to Wallace. That meant he had to keep the war going, rather than accept the only kind of terms on which it could be ended (and on which it finally was ended, after the election).

So here again we see an example of the lengths to which Nixon would go to assure his re-election. Since Watergate, do you think people are as vulnerable to this sort of deception and vicious manipulation of the political process?
One reason I’m following Watergate so closely is that I think it’s affecting the mass consciousness in a very crucial way, which will change the political culture of this country. It’s reawakening people and giving them tremendous vitality at last. It’s also providing a lot of data, which is otherwise unobtainable, about how the system has worked and what the specific influences were. And part of it is data that’s very important to confirm. It’ll call for a lot of rethinking by anybody serious who’s trying to understand this.

How does it feel to watch the Watergate hearings and hear your name come up day after day?
From the week the trial ended, midway through a crescendo of the Watergate revelations, the hearings have kept me absorbed. I watch them with a feeling of tension and alertness, like being hunted. I see how I was hunted over these two years, far more than I realized. I had this sense that the president was coiling himself for one last attack on me.

As he is pushed closer and tighter into a corner on the criminal issue of his own involvement in the burglary of my doctor’s office, he’s virtually got to attack me: He has got to provide the extenuating circumstances – that this guy Ellsberg was really a dangerous person – why he had to use these extra-legal methods to discredit me or “reveal” my true nature to the public and to prepare a Justice Department prosecution.

It must be frustrating – to hear the accusations and not be able to answer them.
Having them talk about me in front of 25 million people, and laying charges of criminal actions on my part… espionage and treason… I’m sitting in front of that television set, all alone, and all I can do is shake my fist at the screen, without being heard. They talk about my giving the Papers to the Soviet Embassy, or use this word “theft” very promiscuously. It’s as if I were still sitting at the defense table at the trial. I wait for the senators to say something on my behalf, and they never do.

Have you found any way to ease any of the anxiety?
Somebody pointed out that we ought to send a letter to the committee, so I helped my lawyer draft a letter which we asked to be read. I, of course, totally denied the charges and expressed regret that no senator had seen fit to challenge them. Ehrlichman made them a dozen times altogether, and Haldeman and Mitchell, too.

We wrote that we wished they had asked those guys, and would they please ask in the future, such questions as, “What evidence do you have that these Papers ever went to the Soviet Embassy?” “What evidence do you have that Ellsberg was associated with its going to the Embassy, if it did?” “Why has no one been indicted for this crime?” “In two years of prosecution with two grand juries, why was no hint of it ever mentioned?” These seem fairly obvious questions, but they haven’t been asked.

From a political point of view, I can sympathize with the senators. They have enough to deal with, in confronting the president directly, without having to defend me. They put themselves against burglary and for the Constitution without getting in the vulnerable or controversial position of being for Ellsberg. They’re saying, in effect, “OK, let’s agree, for purposes of discussion, that Ellsberg is a traitor and a thief and all that, but does that still give you the right to burglarize?” That’s a very strong position for them, but it does kind of push me off the sled.

What’s most unsettling about an accusation like that is that it’s so off the wall. It suggests Nixon’s willing to say literally anything; there’s no way to guess what he’ll say next. If one charge doesn’t float, I have to assume that he feels completely unrestrained in his next shot.

Of course, Nixon’s credibility isn’t at its greatest. If you had a choice of someone to make charges against you, Nixon would be pretty good except for the fact that he still is president of the United States.

The first sign that you were going to be a tactical target came as Nixon’s first response to the Watergate hearings, the banquet he gave at the White House for the POWs.
I had been invited to speak on the Today Show, something I thought I’d better do because things were getting a little heavy – Strom Thurmond and Hugh Scott had just denounced me – and maybe I should make an appearance. I was at the Today Show studios in New York at 7:00 in the morning. They ran film clips of Nixon’s speech the night before, which I hadn’t seen because I had been flying to New York.

It was very dramatic footage.
He was lit in a rather Rembrandt fashion – what do you call that? Chiaroscuro lighting? Floodlit faces and deep shadows.

He was building up, extolling the bravery of the men who had bombed Hanoi. They cheered. Then he said, “I think it’s about time that we stop making national heroes out of men who steal secrets and give them to the press.” The POWs leapt to their feet and began cheering wildly. It was an eerie feeling to watch the president of the U.S. appealing on such terms to this group, the men Nixon was trying to sell as the nation’s heroes.

A lot of people might say, Nixon is just saying that because he has to defend himself. But I have to report that it’s not easy to be fully relaxed while watching the president make that kind of attack on you.

He went on to say, “I ask you to help me, to help me in opposing leaks.” And I had to ask myself, what exactly is he asking them to do?

It was a grotesque moment. The lighting was like Albert Speer describes rallies.
It had a Nuremberg-like flavor to it. There’s no getting around that. I was in the television studio waiting to talk in quite general terms about my case. I had to react immediately, and I didn’t have time to reflect fully on what I’d just seen.

That was one occasion where they caught me off guard; I was not braced for it all. The very discovery of the president’s involvement was only a couple of weeks old, and I didn’t have the whole picture. I wasn’t yet aware of how the president had apparently personalized this conflict in his own mind.

Now, if there was one thing that I had concealed or dissembled during the two-year period, it was the weight of this process on me – whether it scared me or not. I had the feeling that part of what they were trying to achieve was to deter other people and to confirm, through a big drama, “presidential guerrilla theater,” the widespread feeling that you can’t fight the state. They wanted to show that for one rash act of daring, I was paying the penalty, being destroyed by focused hostility. So when I was asked how I felt, I simply refused to express any fear, any anxiety or tension. I thought that would be giving them what they wanted.

But for once, at that moment on television, I was conscious of a defensive tone in my voice – sort of tacitly saying, “Hey, what’s happening here? What’s in store for me? Should the president of the United States be attacking me like that?”

This feeling of not knowing what was coming next and being even less able than I was in the courtroom to participate in some kind of strategic or tactical reaction has made it hard. We’ve been alone – which is what we wanted after the trial – but that means our whole team of people has dispersed and hasn’t been around to confer with… it’s feeling isolated.

Why do you think Nixon is so obsessive about you?
Egil Krogh, one of the “Plumbers,” told the Christian Science Monitor that in the very first meeting, when Nixon discussed the “special investigative unit” and gave Krogh the assignment to investigate me, he told him to read his chapter on Alger Hiss in Six Crises. Krogh told yet another reporter that “to Nixon, Ellsberg is Hiss.” From the time I heard that, my assumption was I represented an opportunity for Nixon to relive his greatest triumph – the exposure of an Eastern-educated guy with good Establishment credentials who really turns out to be a vicious liar and spy.

There is another line of thinking on this, that they really were panicked by the Pentagon Papers, that Nixon was obsessed with me because I really did challenge and threaten him in some personal way. It’s possible… we just haven’t heard enough about it.

Any other reasons?
Perhaps they wanted to demonstrate to others that the behavior of conscience, of speaking out, was not at all permissible.

In a general way, what I did could be seen – I did see it as such – in very Gandhian terms. It was nonviolent, in fact an act of pure, abstract truth-telling. It’s almost a classic Gandhian dream, to suppose that such an act of truth-telling – and taking personal responsibility for it, publicly – is precisely what it took to disturb this government profoundly.

Their policy and administrative framework had been based for a whole generation on secrets and lying. The notion that the Ship of State is leaking truth is as frightening and unstabilizing for the government as anything could possibly be.

Now we know why he had reason to fear leaks: He had more to conceal in the way of actual operations than I realized. I knew that he was concealing his intentions and his plans, but I wasn’t sure of the scale of the secret bombings of Cambodia. And there well may be other operations that remain to be revealed.

When I’d started on this, I had always assumed they would attack me – first by putting me in prison for the rest of my life, and second by going after my reputation. It’s just a standard approach. In fact, it was always very puzzling why it didn’t seem to be happening at any given moment. We knew they were collecting as much as they could on my sexual life, associations and personal things, but they didn’t seem to be using it. We were always waiting for that shoe to drop.

Frankly, part of my unsettled feeling was that I didn’t really know quite how to feel. If my reputation was smeared, that, in itself, is not of any particular political significance. Patricia said something that did put it into perspective and made it easier to think about: “All right, say that just because his back is to the wall, Nixon does have to attack you; and say that it sticks, it really hurts your reputation. You just have to regard yourself as a casualty of Watergate. And it will have been worth it.”

We now know that when the burglars were caught in the Watergate, everybody connected with it understood that, sooner or later, the burglary of my doctor’s office was almost sure to be revealed since the very guys who had done it were in the hands of the Justice Department. I presume that they then structured my prosecution so as to limit the damage to them of that coming out.

In fact, when it did come out, the prosecutor, David Hassen, immediately told reporters, “You will not find a word of testimony in our case that could have come from a psychiatrist’s files.” He seemed to know that so quickly, which would suggest he had designed it that way. And that would explain why we hadn’t heard rumors about my personal life during the trial. It made for a surprisingly clean prosecution.

Did they in fact find anything in your psychiatrist’s files?
They did see some things in his files, because they’d been displaced. These were papers with my name on them, but not actually interesting – nothing that they would have liked, nothing personal. They probably did see medical history, which didn’t amount to anything; they had that on me anyway from government service.

I think they did photograph more than they admitted. I’m virtually sure they lied when they said they photographed nothing that had my name on it.

They had quite a bit to risk if they were to use that stuff during the trial, after the Watergate break-in. But they don’t have anything to lose now. They have little to lose by trying almost anything that might possibly be helpful to them, true or false.

What kinds of things do you think they might use?
There really is nothing true that I’ve been able to think of which would embarrass me at this point. However, that Soviet Embassy gag suggests they are not going to be limited to true events.

Have you been able to trace where the Soviet Embassy charge originated, and who thought it up?
It first appeared in Krogh’s affidavit at the tail end of our trial. He said he had been told by the FBI. Last week, Time magazine quoted another FBI source that there was never any concrete evidence such an event took place at all, just a “vague rumor” floating around.

From the start, my guess has been, and still is, that probably nothing ever did go to the Soviet Embassy; but if I ask myself who had both the ability and the motive to send the Papers to the Russians, the White House is the only one I can think of. After the Papers were published in The New York Times, who the hell else would give them to the Russians, and why would the Russians even want them? It was worth about $2.95 to the Russians, for the Bantam book, maybe $50 for the government edition. And meanwhile, they were getting it for 10 cents a day in The New York Times. The White House people had a fine opportunity to turn my prosecution into a true espionage case, in a way that would be costless to them. Nothing could be easier than to run off an extra copy and send it over to the Soviet Embassy.

But they had a problem trying to explain why I would give the Papers to the Russians. All the evidence they had on me was as the CIA profile certified, that I’d acted out of patriotic motives, or at least, that my background was very patriotic – more than ordinary. So how could you explain my giving documents to the Soviets?

Blackmail. Harper’s had a guy who was writing a book on Watergate and had talked to Sturgess, one of the guys they found in the break-in. And Sturgess told him the team had been told I was a homosexual…

The team?
The Watergate team. They had been told I was a homosexual who had been blackmailed into giving the Papers to the Russian Embassy.

Well, as soon as I heard this, I could see a sense to it. Their minds run to blackmail in all cases, anyway. They blackmail each other; it’s the first thing they seem to think of. The only thing they could actually find on me would be a particular period of what could be called promiscuity, the kind of thing you wouldn’t go out of your way to tell a jury – even though I hadn’t been married at the time – but with a larger audience might make me a cultural hero. But you can only do so much with that, and it certainly would be no basis for blackmail.

So what they were looking for, I now suspect – this is all speculative – was something they could plausibly claim was a basis for blackmail, and they hoped to get it from my psychiatric files. They do seem to go toward this charge of homosexuality, despite the advances made by gay liberation.

What else are they digging into concerning your private life?
Well, it was known to Hunt, for example, that I had taken LSD twice, under controlled clinical circumstances, in ’60 and ’61. Everybody at Rand knew that. I never regarded that as any secret or particularly embarrassing.

From the very point at which you were identified, it appeared that the attack on you – whether by prosecution in the courts, personal attacks, leaks in the press, or now during the Watergate hearings – had come right from the White House.
I had assumed that the prosecution was a White House decision because it was so unprecedented in legal terms. That raised the question of what the general objective was.

Our first notion was that I had been indicted simply because I’d just given out thousands of pages of information stamped top secret, so how could they not indict me? But the more we looked into precedent and the actual law, the clearer it became that no statute had been violated.

This must have been known to the Justice Department. The issue of prosecuting someone for releasing or using classified documents had often arisen in the government, but the legal counsels of the various departments – State, CIA, Defense and Justice Department – had often expressed, in writing, the view that there was no legal basis for prosecuting. If there was no way to prove intent to harm the interests of the United States or to help a foreign power, there could be no prosecution under the Espionage Act. And as far as copying was concerned, transferring information, not tangible property, is not subject to prosecution as theft.

They had no authority, no Official Secrets Act of the type the British have, and they knew it. So we supposed they were reasoning: Well, we’ll prosecute just to support the usual layman’s belief that there is such a law.

The prosecution went on and on, but really wasn’t doing everything it could to get me. They were trying pretty hard, pulling in quite a few witnesses, especially on rebuttal, but still, not trying as hard as it seemed they could. Their case was thin.

More and more the question in our minds became why was the prosecution brought? Our main assumption was that they were trying to get, by convicting me, the equivalent of an Official Secrets Act, which they couldn’t get from Congress, thanks to the First Amendment. I still think that was part of their reasoning.

But that finally came to seem not quite enough for all the effort they were going to. What gave me the clue was one other piece of information. There had always been a potential in the case for implicating a number of people who had dealings both with the Pentagon Papers and with me. All those people happened to be advisers to Senator Muskie.

To see the full importance of that, recall what we know now from the hearings about the mood of the White House in the years 1969 to 1971. As late as mid-1971, and really as early as ’69, the White House was very worried about the next election.

The 1968 election had been very frightening to them because Nixon had lost his early lead to Humphrey, even against an extremely divided and demoralized Democratic party. They barely squeaked through, and they knew from then on that ’72 could not be regarded as a pushover. The results of the 1970 election confirmed that even more ominously.

As late as ’71, even after the China trip had been announced, the polls told them that if the election were held then, and if Muskie or some other Democratic centrist were nominated especially Ted Kennedy, and if Wallace were also running, Nixon would come in third. The polls indicated Muskie ahead as late as January and February, ’72, which is when the plans for bugging the Watergate were discussed in the Justice Department. The president then had every reason to be running scared, especially against Muskie.

It isn’t easy to touch Muskie – that’s the very limitation of Muskie, from my point of view. He didn’t take stands, he didn’t commit himself and he didn’t make up his mind, or he shifted his mind. There wasn’t much the White House could get him with.

But, there was one thing: Mort Halperin, Les Gelb, Paul Warnke, Clark Clifford – the exact chain of command on the making of the Pentagon Papers – all were working for Muskie. So, when the Pentagon Papers were published in the press, in June of ’71, there suddenly appeared a very promising circumstantial case against Muskie’s closest foreign policy advisers.

Clifford had continued MacNamara’s study. Why? Warnke, Gelb and Halperin had sent these papers to the Rand Corporation under special conditions that led to their being held outside the regular top-secret control system, and Halperin and Gelb were directly involved in authorizing me to study them at the Rand Corporation.

Since I knew that this would be an inevitable inference, I had always committed myself to take public responsibility for the disclosure if there were ever a grand jury investigation. I would be free, then, to be very explicit that these others had not been involved.

Just before the second indictment came out, in December 1971, we learned there were teams of prosecutors trying to establish a link with Halperin, Gelb and Warnke. The Young memo to Ehrlichman was perfectly explicit on that point. In fact, one of their criticisms of the FBI was that of all the agencies working on the case which apparently included State, Justice, Defense and the White House, “only the FBI believed that Ellsberg was the prime mover in this case.” Score another one for the FBI!

But even after Muskie dropped out, they could still link me with almost any other Democratic candidate. I’d worked for both Bobby and Ted Kennedy. McGovern had announced that I offered the Papers to him. So had McCloskey, the Republican challenger. Even Humphrey had a link to me, because he had asked me to work full time for him in ’68. I was available to tar any Democratic candidate except, perhaps, Jackson.

The White House had just two problems: I was somewhat too popular, and although I was controversial, I wasn’t as negative a figure as would be ideal. So they had to blacken my reputation, and that, as Young says, was the purpose of going into my psychiatric files and, earlier, of getting the psychiatric profile. The other problem was to link me directly with these other guys, which would not really be provable, even though it would look that way.

How else did they attempt to go after your reputation?
We learned from Time magazine, and then from testimony, that the Watergate crew had been sent to assault me on the steps of the Capitol in May, 1972. The Cuban Americans have testified they were brought up from Miami, having just been recruited and paid by Barker, who had gotten the money from Liddy. This is the famous $114,000 in Barker’s account. They were shown a photograph of me and told, “This is the target,” according to testimony, “and you are to punch him in the nose and call him a traitor, and get away.”

I had decided to attend this particular rally because of the bombing of Haiphong. We have a tape of my talk, about 12 or 14 minutes long, and you can hear people yelling, “Traitor” in the background. Afterwards, I was told that men had been trying to break through to me, and they’d been constrained by the young people who were behind the platform. They were taken off by the police when the scuffle started, and then, witnesses agree, there was a signal by some man in a gray suit, and the police released them.

When did you leave your job at Rand?
In mid-April, 1970. In that regard, let me mention something that is still a mystery. In April, ’70, the FBI knew something about what I’d been doing. I actually started copying the Papers September 30th, 1969, and gave a large chunk of them to Fulbright in November. In the spring of 1970, my former wife called and told me that the FBI had been to see her about secret documents that I had copied. I resigned from the Rand Corporation immediately, in a day, on the assumption that the FBI would certainly be coming, and it would be very hard on Rand to be associated with me.

But the FBI didn’t come, and I never heard from them. I stayed away for a month; then Rand seemed interested in having me come back and finish a paper I’d been writing for them on insurgent tactics. I figured I’m not endangering them too much; the FBI just doesn’t seem to be following this up. So I did come back to Rand, writing the study until June 30th. In September I left Santa Monica and moved to Cambridge. Throughout that time, I kept my top-secret safe and still had access to top-secret documents.

At Rand, very few things are top secret, unlike the Pentagon, and I was one of the very few people – maybe there were four or five others – who had a top-secret safe, which is heavier and means your office gets checked nearly every hour by a guard.

In the last week of the trial we got some astounding documents by way of court order – the contents of Hunt’s safe and material they had on the Rand witnesses who testified. To our amazement, we discovered the FBI had been to Rand in 1970 and had told them about this report they had that I had copied top-secret documents. Rand immediately identified the documents as the MacNamara Study on Vietnam, the Pentagon Papers.

Now, that stuff was in my safe at the time, so, of course, from that moment on, the FBI was in a position to examine it. And I presume they did. They had a number of interviews with Rand officials during April, May and June, including the president of Rand.

Rand decided not to alert me to the FBI interest so as not to interfere with the investigation. And they also made the very interesting judgment in 1970 – a year before the Papers were published and a year before I was indicted – that it was compatible with National Security for me to continue to have access to top-secret documents and to my safe.

In particular, they told the FBI that since Fulbright could have gotten this information anyway if he’d asked for it, and since it was proper for him to get it, then “what was being charged was an impropriety rather than espionage.” Except that Fulbright had already twice asked Laird in writing for those documents, after I’d given him the first batch, and Laird had refused. Ultimately, Fulbright asked two more times, and twice more Laird refused. My judgment was proven correct – namely that there was no way for Fulbright to get them other than for me to copy and give them to him.

So it was very interesting to learn in June of 1970, the very month when President Nixon was preoccupied with the question of dealing with internal subversion, the month he called for this domestic intelligence plan drafted by Tom Huston and gone over by all the intelligence agencies, that the FBI knew all about this largest leak of classified information in history.

It is inconceivable that the president was not told about this case at that time, and certainly other people in the White House knew about it.

So, you’re suggesting that he deliberately let you go ahead and do it?
I really don’t know; I regard that as a real mystery. For instance, it is totally inconsistent with the impression Ehrlichman tried to give of his people being thrown into panic when the Pentagon Papers came out. It is certainly not true, as Ehrlichman put it, that, “We didn’t know who’d done this, how it had come about; we didn’t know what was happening.” That’s absurd.

The FBI obviously knew every relevant fact, about every act for which I was actually indicted (which didn’t include distribution of the Papers), a year before I was indicted. So from the time they read those headlines in The New York Times, it couldn’t have been an hour before even the top people knew who had leaked, what was copied, what the study was, when I did it, how I did it. For instance, an FBI agent was knocking on the door of my wife’s doctor in New York the morning after the stuff started coming out on Sunday. They also pulled in [co-defendant Tony] Russo for questioning within days.

All this has to be seen in the context that, “We had to set up a special unit in the White House to carry out an investigation.” Investigation of what? This was something they’d known about for 12 months.

I might say here, parenthetically, the name “Plumbers” is just one more euphemism. Hunt and Liddy, from what we know of their activities, did nothing ever to plug leaks. They certainly didn’t need to find out the source of the Pentagon Papers because I’d already been indicted by the time they were hired, and the FBI had known a year before that. So for what purpose were they collecting all this personal information about me? They weren’t plumbers. Their job was leaking, leaking scandal.

Do you think they had a role in the bombing of the Capitol Building in 1971?
They easily could have, or it could have been other units we haven’t heard of yet. It’s obvious from what we now know of Mitchell that every judgment he made in the Justice Department, and every conspiracy case brought, deserves to be looked at again. One of those cases has been reopened – Kent State.

With all the emphasis they were putting on bombing, on violence and so forth, those who they had actually prosecuted – me, the Berrigan group, the Harrisburg group, the Camden 28, the Minnesota Eight – were as nonviolent as you’re likely to find. In virtually all these cases there were government provocateurs and informers; I think the widespread use of such agents needs looking at very closely. And such violence as we saw needs very close reexamination.

Someday we should reopen the question of how it really came about that SDS was shattered in the way that it was, and who might have been involved in the disputes between PL, Weatherman and SDS. The anti-war movement was shattered during that period; there were disputes over the issue of violence. And I would be very curious at this point to know just who was paying some of those people who were so hot on violence.

We now know how broad the objectives of the whole political espionage operation were, particularly against the Democratic nomination for the presidency. I suspect that the effect on the Democratic nomination was greater than even the people involved realized.

There may have been a lot more against Muskie’s side than we know. We know, of course, that they tried to get Wallace out of the race early, by spending at least $400,000 to defeat him in his primary race for governor in 1970. Yet we have learned almost nothing about direct operations against Wallace’s chances for ’72.

The really insidious possibility suggested here, of course, is that the assassination attempt was arranged by the White House.
Well, that is not any allegation that I want to make prematurely…

I agree. It has the ring of hysteria to it and colors the way in which everything else is perceived, but nonetheless…
Well, I think this much can be said: Wallace was absolutely critical to their election chances. If you look through the Evans and Novak book, you’ll find that from 1969 on, Mitchell’s preoccupation was that in 1972 Nixon must add together his and Wallace’s 1968 votes. The “Southern strategy” was adapted to that end – Nixon must not antagonize any Wallace-type votes by favoring blacks in any way, or failing to favor Southern candidates, as in the Supreme Court nominations.

But all of that merely made Nixon eligible for the Wallace votes, if Wallace didn’t run. There isn’t a single word in the Evans and Novak book about how this aim of adding Nixon and Wallace votes was going to be achieved if Wallace ran.

You could imagine that they hoped to make a deal with Wallace not to leave the Democratic party, and there were rumors that there had been such a deal. It’s one thing for Wallace to decide not to run against Muskie or Kennedy; but suppose they were successful in the other part of their strategy, to knock Muskie out and get a candidate like McGovern. How could Wallace be induced not to run as a third-party candidate then?

Even with McGovern as the candidate, Nixon was in deep trouble if Wallace ran. And he would lose if someone other than McGovern were the candidate.

Now, let me try to state this very carefully, without being coy, but without totally avoiding the subject. Knowing now what the Nixon team was prepared to do to help his election, knowing that prominent members of that team were professional managers of assassination – I’m referring here to Hunt and to people who worked with him – and knowing that Wallace’s not running was crucial to their success in ’72, I think we must face the fact that the attempted assassination of Wallace deserves very close reexamination by law enforcement agencies. There is no need to say more than that. But I think at this point you needn’t say less than that.

What else do you think?
I’ll make one more guess on this question of why the FBI didn’t come down on me immediately, why they let me go on copying. My guess is based on Evans and Novak’s book, Nixon in the White House, which describes the congressional campaign of 1970 into which Nixon injected himself.

The original enemies list was renegade Republicans, who drove all the White House people crazy, especially Nixon. And at the head of the list was [Senator Charles] Goodell. Nixon apparently just hated him, and was determined to get him, as he did get him in the fall of ’70 by supporting the conservative candidate, Buckley. The original FBI report, which happened to be incorrect, stated that I’d given the Papers to Goodell. This must have excited the White House people, that they had a good chance of pinning an espionage rap on Goodell.

It’s very indicative that the enemies list contains Halperin, Gelb, Warnke, Clifford and MacNamara, in addition to me. In other words, everybody connected with the Pentagon Papers. Rather than pursuing the failing radical Left, they were after the liberal establishment.

The list also includes Harry Rowen, the president of Rand. That explains the puzzling testimony of the Rand people during my trial. I now see that they were under heavy investigation for inclusion in the conspiracy, and I’m sure that the prosecutor made them aware of that, aware that if they didn’t testify along certain lines, they were prime candidates for indictment.

So you think there was an operation to silence all opposition, not just the radicals?
I think Nixon had reason to worry about the election of 1972; but as we know more about the real inner thinking of these people, I have a growing feeling that he was also thinking more broadly than that. What he and his close associates had in mind, whether or not they were fully conscious of it in these terms, was a quite radical change in our form of government.

We know, of course, that Nixon had certain goals for the Supreme Court, and although he may not have faced the implications of those goals, he was, in fact, emasculating the Court and eliminating it as an independent branch of government for a lifetime. The Congress had already accepted that passive role, and he did what he could to keep them in their place. So he was moving, very obviously, toward a single branch of government.

Beyond that, now that we know of this domestic intelligence plan, he was moving faster than I realized toward an even more authoritarian kind of police rule. That’s not only the revocation of the First Amendment, but a decisive change in our form of government – an executive coup.

I used to talk of it as a slow coup, and emphasized the potential and the trend, the aspects which it had been taking for a generation. But I’d say the documents on domestic surveillance and the 1972 campaign are showing us plans for a coup that isn’t all that slow, that was pretty close to the timetable in which the government changed its form in Greece and the Philippines recently. I’d suspected for some time that Nixon had some envy of those particular changes and that they might well be in our future.

When did you begin to suspect that?
On May 13th, 1970, when I testified for the first time before the Fulbright committee. This was during the week of the Cambodian invasion. I’d come to Washington to find the city filled with tear gas and with buses lined up in front of the White House, protecting the White House lawn from a couple hundred thousand people who had come to protest. I started out by commenting on a question that Fulbright had asked an earlier witness: What are our vital interests?

I said, “I think I have come to understand more clearly in this past week what a vital interest is in this country. We have a vital interest in getting out of Vietnam.” I described what had been going on this week and said, “We cannot go on like this without a change in our institutions as radical and ominous as would be brought about by our occupation by a foreign power.”

I had in mind a fairly specific sequence of events when I said that. The president had very obviously been taken by surprise by this invasion of the Capital following his invasion of Cambodia. Remember his actions on that Saturday morning: He went out to the Lincoln Memorial with his valet, and he sounded unhinged – that was the impression that he made on the students he spoke to.

But I knew he was going to be presenting the public with more such escalation, and he knew it, and now that he’d seen this reaction, I was sure that he would be prepared next time to deal with it much more forcefully.

That’s part of it. The other part was how to use the excuse of anti-war activists, what he could call subversion and violent dissent, to dramatize the need for his re-election and to justify an internal police apparatus that would give him a complete lock on our political processes. Well, once you achieve a real power to manipulate individual opponents and manipulate the nominating process of the other side, to deter certain strong opponents from running against you, you have achieved essentially the same influence over elections that General Thieu has in Vietnam. You know, they still do have elections in Vietnam.

In some respects his re-election policies were as good as they looked, in terms of reversing some of the Cold War confrontations he had been so vital in establishing 20 years earlier. He was ready to close out the China and Russian confrontations. But, in addition, I think we can say from this amazingly detailed testimony we’ve been hearing and the memos his aides were writing to each other, that Nixon and his aides have a set of political values not universally shared by American politicians, that they have very little understanding of or sympathy for democratic values and processes.

They do not admire or trust democracy. They don’t respect or trust the American people. They think of them at best, as Nixon himself put it, as children, and not wise or good children.

When Nixon began to impound funds, and in effect take control of the whole budgeting process, he took away the only power that Congress had left. He has not only changed the composition of the Supreme Court, but has publicly indicated he will ignore its rulings, as he announced recently with respect to the tapes.

We’re getting to a situation like in Vietnam, where the Vietnamese Supreme Court reviewed the case of my close friend, Assemblyman Tran Ngoc Chan, arrested in early 1971 in the National Assembly and convicted on trumped-up charges by an illegal military court. The court held all of those actions unconstitutional, according to the Vietnamese constitution, but they refrained from ordering President Thieu to release him from prison, where he still is. The head of the court explained off the record to newsmen that they suspected Thieu would just ignore the order – thus their own power and authority would be shown to be not diminished, but destroyed. Similarly, Senator Harold Hughes told me at a rally in the spring of 1971: “What makes you think that the president would stop bombing if we cut off the funds?” In other words, the Congress had a strong suspicion they had already lost their power over appropriations and over war and peace and were simply acting to delay the moment when the public noticed. To say that is to say, in all substantive aspects, the coup had taken place.

Would you describe Nixon and his aides, then, as virtual fascists?
I am not assuming that these people would recognize fascism as a clear-cut goal. But rather – and this is enough – they had attitudes throughout this period which were perfectly compatible with that state, and they were working consistently to change the current state of affairs steadily toward an end-state that could ultimately be recognized as fascism.

They obviously wouldn’t accept that name. I might say that, in speaking publicly, I have used for some years the term “monarchical form of government.” The word “fascism” sounds, after the debates of the Thirties and Forties, like a smear word, an attempt to equate people with Hitler. It’s hard to use the word in a technical way today.

The word “monarchy” not only has a freshness to it, but also conveys a large part of what I want to show – a move toward an unchecked executive branch which holds virtually all authority, in which a single man at the top has overwhelming power and is inevitably surrounded by people who, like courtiers, depend only on his favor and good will for their power; they vie for telling him what he wants to hear and doing what he wants done – regardless of law or reality.

If you ask congressmen why they didn’t exert the powers they had on the books, you immediately run into a direct fear of offending the White House. No matter which party was involved. And we’re beginning to find out what some of those fears were based on.

Goodell was a good example. The very mantle of his own party identity was removed from him, which meant that he was deprived of regular Republican sources of funds. Goodell, remember, was described by Agnew as the “Christine Jorgenson of the Republican party.” Very interesting metaphor. First, Agnew was saying that Goodell was no longer a “real” Republican, which was very damaging. Second, the metaphor emphasized his change from a hawk to a dove. Third, it suggested very strongly the change from a man to a woman, that the politics of anti-war criticism were the politics of effeminacy, of weakness; whereas the Nixon values were those of strength, virility, brutality, threat.

The word “monarchy,” with respect to the formal government, is a pretty close metaphor. The image of president as monarch has entered the American consciousness to a degree – thanks mainly to Watergate and, to a lesser degree, the Pentagon Papers. That’s a big step forward. However, it doesn’t raise the extremely controversial and heavily defended questions of what economic interests lie behind or are intertwined with the power of the state. In other words, who picks the king, and who does he serve?

The concept of “fascism” is closer to our own time and our industrial society; in particular it suggests the coalition of forces married to the state that sustains the ideological, the economic and police control of the masses – institutions outside the state, such as corporations, unions, universities, local police forces or organized crime. The Intelligence Plan of 1970 and other “dirty tricks,” for example, suggest a very strong movement toward police control in our society.

But another reason I liked the word “monarchy” and talked about an executive coup, was that I was always arguing with the point of view that ascribed all power and influence to the Pentagon. A very common belief is that the Pentagon people are in control; they run the war, they escalate the war, the risks come from the military. I came to conclude, on the contrary, that the military had not been predominant in designing our war policy.

Because the Pentagon Papers show unmistakably that each president pursued policies which were different from ones the military had proposed to him, that he consistently rejected the military proposals. On the other hand, he did choose to continue the war. While military pressure for greater measures was possibly a factor in his prolonging the war, still, he wasn’t doing what the military wanted him to do, and he was taking courses of action that the military consistently said would not work.

So the responsibility for events fell particularly on the president himself, and the pressure on him to escalate came as much from civilians as from military.

I attacked the “quagmire myth” of the Vietnam War – that it was “the politics of inadvertence,” the inattention of the president, which allowed the military to drag us into the war. The story told by the president and his defenders about Watergate is the exact counterpart of the quagmire theory, namely that subordinates took advantage of his inattention during the campaign to take actions of which “he would have disapproved” had he known of them. This is supposed to absolve the president of direct blame, on the thesis that ignorance is an excuse.

And just as I came to believe from studying the Pentagon Papers that the quagmire myth was quite wrong, that the president was responsible, I’ve always assumed for similar reasons that the president was closely responsible for all the events around Watergate. That remains to be proven.

Do you recall that Mitchell said at a cocktail party, very early in the Nixon administration, that this country is headed “so far to the right that you’re not going to recognize it”?
I’d forgotten, but it’s very well worth mentioning.

Do you feel that such a remark coming from his then closest adviser would seem to indicate a coherent scheme?
The White House gang had coherent values, which were basically antidemocratic values. They talked, in referring to the executive branch, about “elected officials” and that the president had been elected by all the people, really ignoring that he was the only person in the entire executive branch, except for the vice president, who’d been elected.

Whereas they had nothing but contempt for the Congress, every member of which had been elected: five hundred people over there who couldn’t be trusted with the kind of information they dealt with every day.

Well, these are not just cultural attitudes, these are fundamental, political attitudes which are choices against democracy: You cannot respect the Constitution of the United States and the Bill of Rights, you can only regard it as a misguided, or at best, an outmoded document; and if you have power and responsibility, you have to improve on it.

Now we add the other element they had witnessed within their own tenure, anti-democratic changes in the Philippines, Greece and South Korea. The three countries we “saved from Communism” after World War II.

Did you see what was done to the man who ran against President Park and got 46 percent of the vote? He was kidnapped by the Korean CIA in Tokyo.
Right! Meanwhile, you had the undisguised sympathy of Agnew and others for these regimes. Now, I absolutely take for granted something that I have never seen mentioned in the press: that we knew the details of the Philippines takeover months before it was carried out, and that there was detailed coordination and, in effect, tacit approval from our government – if not enthusiastic promotion of the idea.

Marcos’ ministers have mentioned that the plans for this change were in preparation months ahead of time. Knowing our relations with the government of the Philippines, I have almost no doubt that they had to know they had the approval of the Americans in advance. The first man that they jailed was Benigno Aquino, who was likely to run against Marcos in the next election and almost sure to win. His campaign was to get rid of the U.S. bases in the Philippines.

I don’t think we had any reservations about that change in government, a change which accords with all the values that Nixon, Agnew, Mitchell and the others talked about and acted on. So I think they could certainly see that as a possibility for this country, if necessary.

Do you know of any documents of a plan for an explicit domestic executive coup? Or to suspend elections? There’ve been so many rumors about a Rand study.
I was told at Rand that was absolutely untrue, that there was no such study, and I believed it implicitly when I was told. I would have to say now, with a little more experience, that I can’t take that absolutely on trust as I did at the time. The idea that there might have been such a study is one of those things you have to reconsider.

Given the segmentation of study areas in the way Rand operates its departments, isn’t it possible that such a study could have been made?
Yes. But it was not a natural subject for Rand to be addressing, in subject matter or sponsorship or anything. It was implausible, to begin with. But it was possible that such a thing could be done and that I certainly wouldn’t necessarily know about it.

I did ask the president of Rand, directly, whether Rand had ever been asked to do such a study, a study of possibly suspending elections in a period of domestic unrest, and he told me no. And then he added, gratuitously and reassuringly, “Moreover, we wouldn’t do such a study if we were asked.”

So, having been a good, ingenious Rand systems analyst, I said, “Well now, let’s see if I couldn’t design a request that could get done. Supposing you were asked by the White House to do a study of how to hold elections despite domestic unrest.”

He sort of nodded at that.

And I said, “Suppose you were asked to consider a number of alternative options of dealing with the situation. And just for completeness and as sort of a devil’s advocate position to contrast to the others, to spell out all its disadvantages, suppose you were asked to consider the option of postponing the elections for a while – as one design and method of responding to this unrest.”

He looked a little uneasy at that, and I said, “Can you imagine yourself accepting that study?”

And he said, “Not now, since you’ve mentioned it.”

Seven Days in May is a rather effective, supposedly fictional description of a secret government apparatus set up in the Pentagon…
Yeah, that was pretty realistic. They used the notion of secret bases. We’ve now seen that in Cambodia: Special Forces bases whose function it was to carry on a secret ground war – entirely separate bases, separate teams of mercenaries, code-named bases whose very existence was not to be known.

It is exactly what the book described, except that the secret base was located in Texas.
In physical terms, one could say that all that apparatus, all the procedures and practices, exist right now. If you ask, “Could that be used to take the government over?” keep in mind that the Greek coup was run by CIA-trained intelligence officers on the basis of an existing NATO contingency plan which had been developed, supposedly, to take over the government in case of Communist takeover or attack. They simply set that plan into motion by issuing the proper code words, and everybody began doing what he was supposed to do.

James McCord, as a Reserve Air Force officer, helped to write that plan; he also was designated an officer in an emergency censorship plan for taking over all information facilities in this country in the event of a national emergency designated by the president. Although Congress probed rather closely on what the national emergency was intended to be, whether it was supposed to be something like a nuclear attack, it came out that a national emergency was essentially anything the President determined it to be.

This plan exists right now, and could be set into motion on the President’s giving the word.

There have been background stories from the FBI pointing out predictions in the summer of 1970 that as many as 300, 400 college campuses might remain shut down or be in some sort of revolt in the fall of 1970, after the Kent State/Cambodian protests. However, it turned out – and I was very disappointed – the campuses were inactive. The FBI sources say that had the predictions been accurate, “a good deal of repression would have become a reality.”

I think that if resistance to the war had continued on the lines or scale of reaction to Cambodia, we might well have seen plans put into effect like this Emergency Information Control Plan.

Are you aware of any other similar contingency plans for national emergencies in addition to the one McCord describes?
I wasn’t aware of any.

But do you feel that these could exist in the Pentagon or other agencies?
Probably they do exist. I felt pretty sure, before I knew of the Huston plan, that after the Cambodian reaction took the administration by surprise, they would be doing contingency planning among the domestic agencies.

If these and similar contingency plans in fact exist, what do you think should be done?
It’s not the pieces of paper that are dangerous; it’s the whole set of attitudes, the configuration of power in the executive that gave rise to such plans. In the process of investigating Watergate, the Congress will come to understand what needs to be done.

Ervin has not minced words about this, nor have some of the others. He called the Huston plan quite accurately, “a Gestapo mentality.” Executive secrecy, executive arrogance, executive usurpation, elective monarchy and fascism are words that I didn’t particularly use two years ago but are becoming more and more applicable as we learn actual facts.

The public is coming to understand that they were facing a massive and urgent threat to our remaining democratic institutions, and that what happened here was the uncovering, essentially, of a coup on the eve of its completion.

A great number of the people who carried out this coup are still in power, starting with the president. His capacity to lash back, to carry out some of his earlier plans, has diminished considerably, but it’s not zero. There’s no evidence yet that [Attorney General] Richardson will be any great impediment to a presidential initiative, but there’s at least a chance.

Mary McCarthy wrote recently that the Watergate hearings were a subconscious attempt on the part of the Senate and the American people to atone for Vietnam.
I doubt it. The people who are diving into Watergate either don’t feel as complicit in Vietnam as they should or really care that much about it.

My hope is that, having developed the courage and insight to oppose the executive branch on these Watergate-related domestic matters, they might become more courageous and skeptical in foreign affairs – more resistant, and ultimately, more insightful about their own past role which will have a dynamic effect and push them still further.

Watergate is not being pressed by McGovern or Fulbright. The whole thing would be far less effective if it were. The battle is being carried by people like Ervin who has a powerful regional base, and is a significant figure politically, precisely because he hasn’t had to fight very strongly for civil rights over the years. He wouldn’t be the political figure he is in the South if he had. That isn’t to say he was justified, just that the impact on the White House is far greater because it’s being fought by Ervin.

I have very little to say in favor of John Stennis’ career in the Senate, but I am glad that he is seeing the light on the need to investigate the CIA and cut it back. I’m more glad to see that than I am to find that Rep. Drynan has put in a bill for impeachment… what’s more, I would always have hoped that these people, even as individuals, could change progressively.

If you want an example of that, you can’t do better than Stuart Symington, who started exactly where I was with the missile gap and was then my idea of a senator who was as responsible and clearheaded as the executive officials I was serving. The missile gap exposure bothered him, too, and had the same kind of corrosive effect on him. Over time he’s become a very powerful battering ram. He has changed enormously.

The hearings may have set off a series of reexaminations of the assumptions and operations of government. What would you recommend the Stennis committee do on the CIA?
I think we should change our relationship with the rest of the world in such a way that there is no business for the CIA’s operations branch – dirty tricks department. It should essentially be disbanded. I don’t think much of a problem is raised by the intelligence part of the CIA. It used to be that the operation of U-2-type reconnaissance vehicles was a terrific generator of crisis, but satellites aren’t posing the same kind of problem. The secret agent intelligence gathering tempts you to get into other kinds of operations but, in itself, doesn’t pose the same kind of risks.

I think the operations role of the CIA will be one of the earliest things that people will be critical of. The secret wars it runs. One question pending immediately is whether the CIA will be allowed to continue to run an insurgency war in Indochina and Cambodia from bases in Thailand. That will be the test… whether Congress can cut that off. I think they probably will.

Is that issue being directly focused on now?
It’s in the Hughes and Symington Armed Services subcommittee.

I’d like to see the Senate investigate the entire nature of the Nixon foreign policy as it fitted into his overall reelection strategy in ’72. I think they made a start toward that. It’s essential they come to understand the massive hoax played on the American public by Nixon and Kissinger as part of their strategy. A hoax which was only possible because of this elaborate apparatus and conspiratorial bureaucracy that was directed toward keeping secrets.

Nixon’s whole relation with the public during his election and after it was based on his assurance, private and public, that he was getting out of the war as fast as possible, and, Henry Kissinger was telling Quakers and other visitors to the Executive Office Building that they should be judged by where they were six months from then, with a strong implication that we would literally be out of Vietnam six months into 1969.

The secret rooms, these bureaucrats trained to keep secret records, in the end were an essential part of making it possible for a president who was elected on the promise of ending the war to keep the bombing going for four and a half years.

I think it is accurate to understand the Watergate hearings in several ways as about what the form of government in the United States shall be.
The hearings inevitably are going to create an appreciation for the specific form of government that was invented by the drafters of our Constitution, particularly our Bill of Rights. I have never been so conscious as in the last couple of years as to how relevant to our current problems of self-government were the perceptions of the men who led the American Revolution. Only a few times in our history have those principles been so relevant. I don’t currently get as much inspiration in relation to our present problems from any other country in the world. The governments of a lot of countries deserve our attention for ways of improving ourselves, but on this precise question of leashing executive power, our own historical debates are more instructive than any other I know.

As Tom Paine put it, “The pride of monarchs is the cause of unnecessary wars”; he saw that the greatest evil that could be inflicted on a people was an unnecessary war, and that included nearly all wars. And that the way you asked for such evils was to confide all power to an unchallenged executive body cloaked with secrecy and mystery. And the only way to avoid that is to shed the cloak of invisibility and make it open, and to provide countervailing centers of power with legitimacy, with open formal roles.

You’ve got to be thankful that it’s all coming out. I say that in the same spirit that so many people thanked me on the street, or after meetings. Sometimes, I would question why that was, because, after all, I was bringing them very bad news, that they’d been lied to for a generation by presidents and that we’d been directly responsible for all the deaths in Indochina. I would have expected more of an unanimous reaction of “thanks for nothing… who needed that?” It told me that there was still life in democracy, because people did feel that they could act on the information I was bringing them.

Now you’re going through a far stranger ordeal than your trial. It seems more severe, because it’s not being played by legal rules. You’re up against the most potent administrative group in this country, the White House, and it has been working on its attack against you for months. Are you getting any counsel or advice on how to handle this sort of trial?
I’m glad to have any kind of advice. During the two-year period around the Papers trial I had very little. I wouldn’t let anybody write anything for me, not that anybody offered except once or twice. I found that even when it did happen a couple times, I couldn’t live with somebody else’s writing in my name. I would have been happy to have some advice to choose from and react to, but I got very little.

But I am aware, from having been through those two years, that I do have to keep the sense that I’m responsible for determining what I do; I can’t give that over to somebody. And to be effectively involved in what I have tried to act for and to achieve, simply rules out a lot of tactics that are available to the White House. Specifically, I haven’t said anything misleading and I won’t.

I think I have a basis for putting it to the media who’ve been watching, “Look, you guys have been taking down virtually everything I’ve said in public for two years, and look at your records and ask yourself if I’ve ever lied to you.” I should be able to draw on that record of credibility.

In dealing with the personal attacks on me before, I could afford to ignore them to a large extent in confidence that the legal process would eventually bring the facts out and vindicate me. But in this case, they can make charges without that assurance.

Madly dashing about is not going to help. The attacks on you aren’t working anymore. The president’s in a far worse position. Seventy percent of the people, at least, just don’t believe anything he says, not after the last four years. Maybe you shouldn’t reply at all.
It would be playing into Nixon’s plans to personalize my reactions to his attacks on me. It would certainly distract people if the issue arose: “What kind of guy is Ellsberg?” Or, “Did he really do these crimes?” Any attention that I attract to myself is attention deflected from him. That surely is not good.

I’m dismayed to find how much sympathy he does get from his attacks on me, especially in the widespread concern among Americans about people who are disobedient to a boss or regulations or who expose secrets. I would hope that Watergate would reveal to people the perils of obedience and the corruption of loyalty.

Now, I think that these men are going to suffer, finally. They didn’t hope to end up this way, and they didn’t expect to, but their strategy did collapse on them. I think it shows that there was more vitality in the American myths and ideology than these people realized, more than a lot of us realized.

Once an exposure occurred, which was pretty fortuitous, it came out that the American people really didn’t think they were supposed to be lied to by the president. They were not taking that for granted.

Presidents fall into this trap – they lie so much, they lie so casually, and in the inner circle they are so conscious of this, they take it for granted – that over time they come to forget how successfully they have hidden from the public the fact that they are consciously lying. They imagine that the public really knows this, and the lack of opposition to it is assent, toleration – “the public doesn’t care, they know it happens all the time.”

The truth is that their cover stories have been very effective, and their cover story for Watergate is that mistakes happen, that people down below do these sorts of things in all administrations, and the president doesn’t know everything and so forth. And that, the public can live with. They do accept that.

But the public does not live with the awareness that the president himself is deceiving them all the time. When that becomes apparent, the public reacts very aggressively and takes the president entirely by surprise. I think he probably feels betrayed. He says, “What are you really so mad about? You must have known I was kidding. What are you getting so sanctimonious about, for Christ’s sake? About a few tapes and a few lies and a few burglaries and so forth. My God, I’ve just been burning down Cambodia and lying to you about that. You knew that.”

What do you think is going to happen if Nixon is not impeached? What if he doesn’t resign? What if he is not thrown out of office?
That would depend on the circumstances of how that came about. If it meant the public turned its attention to the World Series – as he’s expecting – and simply gets bored, forgets, and decides that it wasn’t all that bad after all… Then, of course, the situation would be worse than before. We would have gone very far from democracy.

But take the situation at this moment. There is not a lot of documentary or direct first-person testimony linking the president to some of these acts. It may yet come. However, if there is no more evidence than there is now, then there is still enough on which to convict a president.

But given the consequences, and the symbolism of convicting the president, you could also say that a reasonable man could vote not to at this point. And, if people realized that although the president was almost surely guilty, that there simply was not enough evidence to convict him beyond a reasonable doubt, that wouldn’t necessarily mean toleration for what he’d done, and they might remember the lessons of this.

But every day does bring new implications. It’s getting harder and harder to say that we could live with letting the president off and still think of ourselves as a country of laws.

It’s going to appear that we were much further along toward a monolithic police state than almost anybody knew, a system much closer to what appeared to be paranoia for conspiracies and such.

In contrast to the president’s breaking of “international laws,” in the domestic context everybody knows that there are laws that are meant to be enforced, and there’s a machinery of enforcement and a principle that’s meant to apply even to the president. These statements the president is making now – that he is not obliged to give evidence to the courts, and so forth – are not going down. I think people are realizing that he is writing a Constitution as he goes along, and it’s a different one from the one we have.

What do you think the Watergate hearings really are accomplishing?
I think the hearings are, first, giving the people the facts about how they were lied to and misled, and gradually the realization that it was the president, the man they elected, the man they believed, they listened to, and they voted for, or failed to work hard enough to defeat.

That connects the responsibility up to them personally and I think they feel an immediate sense of responsibility only for what the president does, not for what some subordinate down in the bowels of bureaucracy may do.

Second, the hearings show him doing these illegal, self-serving, and in some cases, violent things to Americans, which also strengthens their sense of responsibility and concern.

And third, I think Congress is rediscovering powers it has really never used or had forgotten about long ago; powers of subpoena, of impeachment, of demanding witnesses and testimony, of going to court, of threatening contempt, above all, of cutting off funds. And this has given people hope that there are sources of organized power outside the executive branch.

I think they are discovering the power of truth and the effectiveness of a man willing to take a risk. We have the spectacle of this collection of suits, and exposures, even if going on in a rather diffuse way: people telling the truth; former servicemen, at last, coming clean and so forth.

So many of these individuals, even now when they come clean this late in the game, are still taking some risk of retaliation when they do it. And the impact of that is becoming apparent to some people. That a man can make a difference.

Topics: Richard Nixon Vietnam Long Reads Coverwall
Don’t Miss a Story

© Rolling Stone 2018

zwischenzugs: Ten Things I Wish I’d Known About bash

zwischenzugs: Ten Things I Wish I’d Known About bash

Ten Things I Wish I’d Known About bash

Recently I wanted to deepen my understanding of bash by researching as much of it as possible. Because I felt bash is an often-used (and under-understood) technology, I ended up writing a book on it.

You don’t have to look hard on the internet to find plenty of useful one-liners in bash, or scripts. And there are guides to bash that seem somewhat intimidating through either their thoroughness or their focus on esoteric detail.

Here I’ve focussed on the things that either confused me or increased my power and productivity in bash significantly, and tried to communicate them (as in my book) in a way that emphasises getting the understanding right.



1) “ vs $()
These two operators do the same thing. Compare these two lines:

$ echo `ls`
$ echo $(ls)
Why these two forms existed confused me for a long time.

If you don’t know, both forms substitute the output of the command contained within it into the command.

The principal difference is that nesting is simpler.

Which of these is easier to read (and write)?

$ echo `echo \`echo \\\`echo inside\\\`\“

$ echo $(echo $(echo $(echo inside)))
If you’re interested in going deeper, see here or here.

2) globbing vs regexps
Another one that can confuse if never thought about or researched.

While globs and regexps can look similar, they are not the same.

Consider this command:

$ rename -n ‘s/(.*)/new$1/’ *
The two asterisks are interpreted in different ways.

The first is ignored by the shell (because it is in quotes), and is interpreted as ‘0 or more characters’ by the rename application. So it’s interpreted as a regular expression.

The second is interpreted by the shell (because it is not in quotes), and gets replaced by a list of all the files in the current working folder. It is interpreted as a glob.

So by looking at man bash can you figure out why these two commands produce different output?

$ ls *
$ ls .*
The second looks even more like a regular expression. But it isn’t!

3) Exit Codes
Not everyone knows that every time you run a shell command in bash, an ‘exit code’ is returned to bash.

Generally, if a command ‘succeeds’ you get an error code of 0. If it doesn’t succeed, you get a non-zero code. 1 is a ‘general error’, and others can give you more information (eg which signal killed it, for example).

But these rules don’t always hold:

$ grep not_there /dev/null
$ echo $?
$? is a special bash variable that’s set to the exit code of each command after it runs.

Grep uses exit codes to indicate whether it matched or not. I have to look up every time which way round it goes: does finding a match or not return 0?

Grok this and a lot will click into place in what follows.

4) if statements, [ and [[
Here’s another ‘spot the difference’ similar to the backticks one above.

What will this output?

if grep not_there /dev/null
echo hi
echo lo
grep’s return code makes code like this work more intuitively as a side effect of its use of exit codes.

Now what will this output?

a) hihi
b) lolo
c) something else

if [ $(grep not_there /dev/null) = ” ]
echo -n hi
echo -n lo
if [[ $(grep not_there /dev/null) = ” ]]
echo -n hi
echo -n lo
The difference between [ and [[ was another thing I never really understood. [ is the original form for tests, and then [[ was introduced, which is more flexible and intuitive. In the first if block above, the if statement barfs because the $(grep not_there /dev/null) is evaluated to nothing, resulting in this comparison:

[ = ” ]

which makes no sense. The double bracket form handles this for you.

This is why you occasionally see comparisons like this in bash scripts:

if [ x$(grep not_there /dev/null) = ‘x’ ]

so that if the command returns nothing it still runs.

5) sets
Bash has configurable options which can be set on the fly. I use two of these all the time:

set -e
exits from a script if any command returned a non-zero exit code (see above).

This outputs the commands that get run as they run:

set -x
So a script might start like this:

set -e
set -x
grep not_there /dev/null
echo $?
What would that script output?

6) /tmp/a
$ grep somestring file2 > /tmp/b
$ diff /tmp/a /tmp/b
That works, but instead you can write:

diff <(grep somestring file1) “Book tickets”
“Enjoy Holiday” -> “Pack suitcase night before”
“Pack suitcase night before” -> “Buy guide book”
“Pack suitcase night before” -> “Buy electric converter”
The emboldening is mine for illustration; the file is plain text.

This file can be saved as simple.gv (.gv is for ‘graphviz’) and will generate this graph as a .png if you run dot -Tpng simple.gv > simple.png:

Looking closer at simple.gv:

digraph – Tells graphviz that this is a directed graph, ie the relationships have a direction, indicated by the -> arrows. The arrow can be read as ‘depends on’.

Enjoy Holiday is the name of a node. Whenever this node is referenced in future it is the ‘same’ node. In the case of Pack suitcase the night before you can see that two nodes depend on it, and it depends on one. These relationships are expressed through the arrows in the file.

The dot program is part of the graphviz package. Other commands include neato, circo and others. Check man dot to read more about them.

This shows how easy it is to create a simple graph from a text file easily stored in git.

That top-down layout can be a bit restrictive to some eyes. If so, you can change the layout by using another command in the graphviz package. For example, running neato -Tpng simple.gv > simple.png produces this graph:


Note how:

Enjoy holiday is now nearer the ‘centre’ of the graph
The nodes are overlapping (we’ll deal with this later)
The arrows have shortened (we’ll deal with this later too)
If you’re fussy about your diagrams you can spend a lot of time fiddling with them like this, so it’s useful to get a feel for what the different commands do.

We can get more project information into a node by colorizing the nodes. I do this with a simple scheme of:

green = done
orange = in progress
red = not started
Here’s an updated .gv file:

digraph G {
“EH” [label=”Enjoy Holiday”,color=”red”]
“BT” [label=”Book tickets”,color=”green”]
“PSNB” [label=”Pack suitcase night before”,color=”red”]
“BGB” [label=”Buy guide book”,color=”orange”]
“BEC” [label=”Buy electric converter”,color=”orange”]

“EH” -> “BT”
“EH” -> “PSNB”
“PSNB” -> “BGB”
“PSNB” -> “BEC”
Running the command

dot -Tpng simple_colors.gv > simple_colors.png

on this results in this graph:


Two things have changed here. Referring to the full description of the node can get tiresome, so ‘Enjoy holiday’ has been referenced with ‘EH’, and associated with a ‘label’, and ‘color’.

“EH” [label=”Enjoy Holiday”,color=”red”]
The nodes are defined in this way at the top, and then referred to with their relationships at the end. All sorts of attributes are available.

Similarly, you can change the attributes of nodes in the graph, and their relationships in code.

I find that with a complex graph with some text in each node, a rectangular node makes for better layouts. Also, I like to specify the distance between nodes, and prevent them from overlapping (two ‘problems’ we saw before).

digraph G {

node [color=”black”, shape=”rectangle”]

“EH” [label=”Enjoy Holiday”,color=”red”]
“BT” [label=”Book tickets”,color=”green”]
“PSNB” [label=”Pack suitcase night before”,color=”red”]
“BGB” [label=”Buy guide book”,color=”orange”]
“BEC” [label=”Buy electric converter”,color=”orange”]

“EH” -> “BT”
“EH” -> “PSNB”
“PSNB” -> “BGB”
“PSNB” -> “BEC”
By adding the ranksep and nodesep attributes, we can influence the layout of the graph by specifying the distance between nodes in their rank in the hierarchy, and separation between them. Similarly, overlap prevents the problem we saw earlier with overlapping nodes.

The node line specifies the characteristics of the nodes – in this case rectangular and black by default.

Running the same dot command as above results in this graph:


which is arguably uglier than previous ones, but these changes help us as the graphs become more complex.

More Complex Graphs
Compiling this more complex graph with dot:

digraph G {

node [color=”black”, shape=”rectangle”]

“EH” [label=”ENJOY HOLIDAY\nWe want to have a good time”,color=”red”]
“BTOW” [label=”Book time off\nCheck with boss that time off is OK, put in system”,color=”red”]
“BFR” [label=”Book fancy restaurant\nThe one overlooking the river”,color=”red”]
“BPB” [label=”Buy phrase book\nThey don’t speak English, so need to know how to book”,color=”red”]
“BT” [label=”Book tickets\nDo this using Expedia”,color=”green”]
“PSNB” [label=”Pack suitcase night before\nSuitcase in understairs cupboard”,color=”red”]
“BGB” [label=”Buy guide book\nIdeally the Time Out one”,color=”orange”]
“BEC” [label=”Buy electric converter\nDon’t want to get ripped off at airport”,color=”orange”]
“GTS” [label=”Go to the shops\nNeed to go to town”,color=”orange”]
“GCG” [label=”Get cash (GBP)\nAbout 200 quid”,color=”green”]
“GCD” [label=”Get cash (DOLLARS)\nFrom bureau de change under arches”,color=”orange”]

“EH” -> “BT”
“EH” -> “BFR”
“EH” -> “BTOW”
“BFR” -> “BPB”
“BPB” -> “GTS”
“BPB” -> “GCG”
“EH” -> “PSNB”
“EH” -> “GCD”
“PSNB” -> “BGB”
“BGB” -> “GTS”
“PSNB” -> “BEC”
“BGB” -> “GCG”
“BEC” -> “GCG”
gives this graph:


And with neato:


You can see the graphs look quite different depending on which layout engine/binary you use. Some may suit your purpose better than others.

Project Planning with PRs
Now that you have a feel for graphing as code, you can check these into git and share them with your team. In our team, each node represents a JIRA ticket, and shows its ID and summary.

A big benefit of this is that project updates can be asynchronous. Like many people, I work with engineers across the world, and their ability to communicate updates by this method reduces communication friction considerably.

For example, the other day we had a graph representing our next phase of work that was looking too heavy for one sprint. Rather than calling a meeting and go over each line item, I just asked him to update the graph file and raise a PR for me to review.

We then workshopped the changes over the PR, and only discussed a couple of points over the phone. Fewer meetings, and more content-rich discussions.

Surface Assumptions
Beyond fewer and more effective meetings, another benefit is the objective recording of assumptions within the team. Surprisingly often, I have discovered hidden dependencies through this method that had either not been fully understood or discussed.

It’s also surfaced further items of work required to reach the solution, which has resulted in more and more clear tickets being raised that relate to the target solution. The discipline of coding these up helps force these into the open.

Happier Stakeholders
While inside the team, the understanding of what needs to happen is clearer, stakeholders clamouring for updates are clearer on what’s blocking the outcomes they want.

Another benefit is an increased confidence in the process. There’s a document that’s readily comprehensible they can dig into if they want to find out more. But the fact that there’s a transparent graph of dependencies usually suffices to persuade people that things are under control.

Alternate Views
Finally, here are some alternate views of the same graph. We’ve already seen dot and neato. Here are the others. I’m not going to explain them technically as I’ve read the man page definitions and am none the wiser. They use words like ‘outerplanar’ and ‘force-directed’. Graph rendering is a complicated affair.









Is here.

If you know more than me about graphviz and have any improvements/interesting tweaks/suggestions then please contribute.

Author is currently working on the second edition of Docker in Practice

Get 39% off with the code: 39miell2

Posted on December 18, 2017
Categories Uncategorized
5 Comments on Project Management as Code with Graphviz
How to Manually Clear Locks in Jenkins
Recently I got into a situation where I hit a bug with Jenkins where Jenkinsfile locks were not released if the job was terminated.

I tried:

Restarting Jenkins
Reinstalling the plugin
Removing the locks manually from the top level Jenkins page
Raising a bug
None of these worked.

I found a solution that involved manually hacking files.

Find the file named:
In the /var/jenkins_home​ folder (or wherever Jenkins is installed).

It will look like this:



Remove the line in bold containing the buildExternalizableId attribute.

Change the queuingStarted item


Restart Jenkins
Author is currently working on the second edition of Docker in Practice

Get 39% off with the code: 39miell2

Posted on December 11, 2017
Categories Uncategorized
Leave a comment on How to Manually Clear Locks in Jenkins
How I Manage My Time
I see a lot of posts like this or this or this on HackerNews asking about time management

I was disorganised until my 30s.

Then I got organised and changed my life with:

Notes in Git
Automating environment setup
What I’ve ‘got done’ since is listed below.

The Phone Call
About 6 years ago I missed something important at work. While working in ops, a customer had asked me to do something while I was busy, I’d moved onto another fire and clean forgot to do it.

Monday I got a phone call from her: “Ian, the payments went through over the weekend. Did you remember to switch off the cron job?”


She had to go and sort it out. I could only give my apologies to help her with the grief she was going to get.

I had the excuse that I was busy, that I’d been distracted, that I’ve got two kids and lots on.

But deep down I know something was wrong, that it was my fault. That I’d made a contract with someone and not honoured it.

It hurt.

A Chance
A few weeks later I was on a rare day off, and happened to be in a bookshop, and in this bookshop I saw Getting Things Done. I guess my unconscious led me to it, and though I’d always mocked books like this, I picked it up and scanned it. To my surprise, the advice was flexible, human, and practicable. I inhaled it, and my life changed from there.

What Happened?
It’s fair to say a lot in my life changed since that phone call 6 years ago. Since then, I’ve:

Started my blog
Wrote a book on Docker, and am working on the second edition
Started giving talks at conferences
Published a video about Docker
Wrote a course on git
Wrote a course on bash
Started an open source project
Contributed to projects on GitHub
Also (and no less importantly), I’ve still got a job, and am a happily married father of two (I overlook the fact that my wife refuses to use JIRA).

I also generally feel less stressed, and more productive. I don’t know if all of it can be attributed to getting organised, but it certainly feels that way. Here’s what I did.

What I Did – JIRA
The first thing I did was set up a home JIRA instance. I’m a dinosaur, a control freak, and a cheapskate, so I buy the license and run it from home.

It doesn’t matter that it’s JIRA, the point is that all the things that impinge on my consciousness get put in here, and get out of my mind, giving me a clear head.

When the board gets too full, I start ditching things. Most of these are articles I intended to read, or little ideas my ardour has cooled on. That stops me getting overwhelmed.

Over time, I made a few tweaks that helped me be a little more efficient:

I created my own workflow that matched the way I thought about tasks:
Waiting for Something
Reminder Set
I set up a gmail account and linked it to JIRA so I could create tickets
I use mail this link to send links I’m interested in to my JIRA
I use send to kindle to mail articles directly to my kindle, so I can batch-read them asynchronously
There’s no separation between work and home tasks. Tax returns and birthday reminders sit right next to work tasks I want to stay on top of.

If it takes up space in your head, it goes in one place.

What I Did – Notes
That’s what I did for tasks – I had another frustration that I wanted to address. I would work on something, then either:

forget it
make notes and forget where they were
make notes, remember where they were, but couldn’t find them
I did something really simple to solve this: I created a git repo for all my notes.

imiell@Ians-Air-2:/space/git/work/notes ⑂ master +  ls | head
Then, I wrote some helper scripts. For example, creates a folder in this repo with some file pre-created:

if [[ $1 = ” ]]
echo folder name needed
exit 1
mkdir -p ${NOTES}/$1
mkdir -p ${LEARNING}/$1
touch ${NOTES}/$1/cheat_sheet.asciidoc
touch ${NOTES}/$1/links
touch ${NOTES}/$1/git_repos
touch ${NOTES}/$1/$1.asciidoc
touch ${LEARNING}/$1/$1.asciidoc
git add ${NOTES}/$1
git add ${LEARNING}/$1
This creates a folder and adds it to git with a file for related links, a cheat_sheet, any related git_repos and a file that has the subject name in it.

Now if I pick up a new skill and then pick up my learning later, I can track my notes up to where I left off. I’ve used these notes to compiled blog posts like these.

I create asciidocs because I like the format, and it works well with vim.

I did try other methods (google docs, email, JIRA tickets), but this works best for me because:

It is available offline (git being a distributed note-taking tool)
It is text only
The current content is easily searched (grep)
A history is maintained that I can also search if needed
I can control/extend this system the way that makes sense to me
These things are important to me. They might be more or less important to you, so choose a tool accordingly. The vital thing is that it’s all in one place.

For example, here’s a link I literally just saw on Twitter while writing this: Organizing your life using GitHub

Work Environment Setup
Another constant niggle was setting up work environments. Like many people, I work on Linux servers, Mac laptops, and occasionally a Windows machine.

Mostly I dial into my home servers, but not infrequently I have to work on other servers.

To save time I wrote a ShutitFile to set up a server the way I like it. Here’s an abbreviated version of the full version:

# We assert here that we are running as root
SEND whoami

SEND lsb_release -d -s | awk ‘{print $1}’

# We assert here the user imiell was set up by the OS installation process
SEND cut -d: -f1 /etc/passwd | grep imiell | wc -l

# Install required packages
INSTALL openssh-server
INSTALL run-one


# Install docker
IF_NOT RUN docker version
INSTALL apt-transport-https
INSTALL ca-certificates
INSTALL software-properties-common
RUN curl -fsSL | sudo apt-key add –
RUN add-apt-repository “deb [arch=amd64] $(lsb_release -cs) stable”
RUN apt-get update
RUN apt install -y docker-ce
# Add imiell to the docker user group
RUN usermod -G docker -a imiell

# Create space folder and chown it to imiell
RUN mkdir -p /space && chown imiell: /space
RUN mkdir -p /space/git

# Generate an ssh key
IF_NOT FILE_EXISTS /home/imiell/.ssh/
RUN ssh-keygen
# Note that the response to ‘already exists’ below prevents overwrite here.
EXPECT_MULTI [‘file in which=’,’empty for no passphrase=’,’Enter same passphrase again=’,’already exists=n’]

# Log me in as imiell
USER imiell
# If it’s not been done before, check out my dotfiles and set it up
IF_NOT FILE_EXISTS /home/imiell/.dotfiles
RUN cd /home/imiell
RUN git clone –depth=1 ~imiell/.dotfiles
RUN cd .dotfiles
RUN ./script/bootstrap
EXPECT_MULTI [‘What is your github author name=Ian Miell’,’What is your github author’,’verwrite=O’]

ShutIt is a tool I wrote for simple automation of interactive sessions. Like traditional CM tools, but simpler.

Exceptions/Difficulties/Lessons Learned
This method works, for me, but there are limitations. I can’t keep all my work Confluence notes and JIRAs on my home JIRA or Git repo (not least for security reasons), so there is some separation between work and home notes and information.

That can’t be helped, but what’s more interesting are the downsides of this approach.

Is It Productive?
Sometimes it feels like managing this is a tax on my attention. I do wonder whether sometimes I’m just shuffling tickets around rather than tackling the hard stuff that happens over much longer time periods than individual tasks.

I Have to Remember to Let Go
Managing your workload more formally like this can make it hard to let go. There’s always something to do, but sometimes you need to take time out and smell the flowers. That’s when other good things can happen. Being productive is not everything, by a long chalk.

Or, as Lennon didn’t put it: Life is what happens when you are busy grooming your backlog.

Any Suggestions?
I’m always open to improving my workflow, so please let me know below if you have any suggestions.

Author is currently working on the second edition of Docker in Practice

Get 39% off with the code: 39miell2

Posted on December 3, 2017
Categories Uncategorized
Leave a comment on How I Manage My Time
Ten Things I Wish I’d Known About Chef
1) Understand How Chef Works
This sounds obvious, but is important to call out.

Chef’s structure can be bewildering to newcomers. There are so many concepts that may be new to you to get to grips with all at once. Server, chef-client, knife, chefdk, recipe, role, environment, run list, node, cookbook… the list goes on and on.

I don’t have great advice here, but I would avoid doing too many theoretical tutorials, and just focus on getting an environment that you can experiment on to embed the concepts in your mind. I automated an environment in Vagrant for this purpose for myself here. Maybe you’ve got a test env at work you can use. Either way, unless you’re particularly gifted you’re not going to get conversant with these things overnight.

Then keep the chef docs close to hand, and occasionally browse them to pick up things you might need to know about.

2) A Powerful Debugger in Two Lines
This is less well known than it should be, and has saved me a ton of time. Adding these two lines to your recipes will give you a breakpoint when you run chef-client.

require ‘pry’
You’re presented with a ruby shell you can interact with mid-run. Here’s a typical session:

root@chefnode1:~# chef-client
Starting Chef Client, version 12.16.42
resolving cookbooks for run list: [“chef-repo”]
Synchronizing Cookbooks:
– chef-repo (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks…

Frame number: 0/22

From: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/cookbook_version.rb @ line 234 Chef::CookbookVersion#load_recipe:

220: def load_recipe(recipe_name, run_context)
221: unless recipe_filenames_by_name.has_key?(recipe_name)
222: raise Chef::Exceptions::RecipeNotFound, “could not find recipe #{recipe_name} for cookbook #{name}”
223: end
225: Chef::Log.debug(“Found recipe #{recipe_name} in cookbook #{name}”)
226: recipe =, recipe_name, run_context)
227: recipe_filename = recipe_filenames_by_name[recipe_name]
229: unless recipe_filename
230: raise Chef::Exceptions::RecipeNotFound, “could not find #{recipe_name} files for cookbook #{name}”
231: end
233: recipe.from_file(recipe_filename)
=> 234: recipe
235: end
[1] pry(#)>
The last line above is a prompt from which you can inspect the local state, similar to other breakpoint debuggers.

CTRL-D continues the run.

See here for more.

3) Run Locally-Modified Cookbooks
I spent a long time being frustrated by my inability to re-run chef-client with a slightly modified set of cookbooks in the local cache (in /var/chef/cache…).

Then the chef client we were using was upgraded, and the
–skip-cookbook-sync option was available. This did exactly what I wanted: use the cache, but run the recipes in exactly the same way, run list and all.

The -z flag can do similar, but you need to specify the run-list by hand.
–skip-cookbook-sync ‘just works’ if you want to keep everything exactly the same and add a log line or something.

4) Learn Ruby
Ruby is the language Chef uses, so learning it is very useful.

I used Learn Ruby the Hard Way to quickly get a feel for the language.

5) Libraries
It isn’t immediately obvious how you avoid re-using the same code recipe after recipe.

Here’s a sample of a ‘ruby library’ embedded in a Chef recipe. It handles the figuring out of the roles of the nodes.

One thing to note is that because you are outside the Chef recipe, to access the standard Chef functions, you need to explicitly refer to its namespace. For example, this line calls the standard search​:, “role:rolename”)
The library is used eg here. The library object is created:

server_info =
and then the object is referenced as items are needed, eg:

first_master = server_info.first_master
master_servers = server_info.master_servers
Note that the node object is passed in, so it’s visible within the library.

6) Logging and .to_s
If you want to ‘quickly’ log something, it’s easy:

log ‘my log message do
level :debug
and then run at debug level with:

chef-client -l debug
To turn a value into a string, try the .to_s function, eg:

log ‘This is a string: ‘ + node.to_s do
level :debug

7) Search and Introspection Functions
The ‘search’ function in Chef is a very powerful tool that allows you to write code that switches based on queries to the Chef server.

Some examples are here, and look like this:

graphite_servers = search(:node, ‘role:graphite-server’)
Similarly, you can introspect the client’s node using its attributes and standard Ruby functions.

For example, to introspect a node’s run list to determine whether it has the webserver role assigned to it, you can run:

This technique is also used in the example code mentioned above.

8) Attribute precedence and force_override
Attribute precedence becomes important pretty quickly.

Quite often I have had to refer to this section of the docs to remind myself of the order that attributes are set.

Also, force_override is something you should never have to use as it’s a filthy hack, but occasionally it can get you out of a spot. But it can’t override everything (see 10 below)!

9) Chef’s Two-Pass model
This can be the cause of great confusion. If the order of events in Chef seems counter-intuitive in a run, it’s likely that you’ve not understood the way Chef processes its code.

The best explanation of this I’ve found is here. For me, this is the key sentence:

This also means that any Ruby code in the file not explicitly delayed (ruby_block, lazy, not_if/only_if) is run when the file is run, during the compile phase.

Don’t feel you need to understand this from day one, just keep it in mind when you’re scratching your head about why things are happening in the wrong order, and come back to that page.

10) Ohai and IP Addresses
This one caused me quite a lot of grief. I needed to override the IP address that ohai (the tool that gathers information about each Chef node and places in the node object) gets from the node.

It takes the default route’s interface’s IP address by default, but this caused me lots of grief when using Vagrant. force_override​ (see 8) above) doesn’t work because it’s an automatic ohai variable.

I am not the only one with this problem, but I never found a ‘correct’ solution.

In the end I used this hack.

Find the ruby file that sets the ip and mac address. Depending on the version this may differ for you:

Then get the ip address and mac address of the interface you want to use (in this case the eth1 interface:

IPADDR=$(ip addr show eth1 | grep -w inet | awk ‘{print $2}’ | sed ‘s/\(.*\).24/\1/'”””)
MACADDRESS=$(ip addr show eth1 | grep -w link.ether | awk ‘{print $2}'”””)
Finally, use sed (or gsed if you are on a mac) to hard-code the ruby file that gets the details to return the information you want:

sed -i “s/\(.*${IPADDR} \).*/\1 \”\”/” $RUBBYFILE
sed -i “s/\(.*macaddress \)m.*/\1 \”${MACADDRESS}\”/” $RUBYFILE

Author is currently working on the second edition of Docker in Practice

Get 39% off with the code: 39miell2

Posted on November 25, 2017
Categories Uncategorized
3 Comments on Ten Things I Wish I’d Known About Chef
Vagrant and Ohai / Chef IP Address Hack

In vagrant, ohai returns the eth0 ip.

This is a PITA, since if you run clusters of Vagrant that use Chef (as I do), then Chef and ohai thinks the IP address of the node is always:

or whatever Vagrant attaches to the default interface (usually eth0).

It’s Not Just Me
Plenty of people appear to have this problem:

How to change ip address of node after added to chef server?

Chef and ohai retrieving a droplets private ip address


How to have chef use a different-ip?

I haven’t found a ‘proper’ solution for this. The most elegant (or least inelegant) one I could find involves:

Finding the ruby file involved in determining network information
Getting the IP address associated with the interface that you want Chef to ‘see’
(Optional) get the mac address associated with the interface that you want Chef to ‘see’
Hard-code the IP address and macaddress with these values directly in the network.rb file
Here’s an example:


IPADDR=$(ip addr show eth1 | grep -w inet | awk ‘{print $2}’ | sed ‘s/\(.*\).24/\1/'”””)

MACADDRESS=$(ip addr show eth1 | grep -w link.ether | awk ‘{print $2}'”””)

sed -i “s/\(.*${IPADDR} \).*/\1 \”\”/;s/\(.*macaddress \)m.*/\1 \”${MACADDRESS}\”/” $RUBYFILE

Got a Better Way?
Please let me know!

Author is currently working on the second edition of Docker in Practice

Get 39% off with the code: 39miell2

Posted on November 18, 2017
Categories Uncategorized
Leave a comment on Vagrant and Ohai / Chef IP Address Hack
‘Towards a National Computer Grid’ – Electronic Computers, 1965
Recently I picked up this book on my travels:


This is the second edition (1970) of a book originally published in 1965.

It’s a fascinating insight into the state of computing 50 years ago, and remarkably prescient.

Here’s a few highlights that piqued my interest.

Getting programs right
This is a fascinating insight into how testing and debugging worked when hardware was a bigger part of the equation. Ever used a cathode ray tube to figure out what’s wrong with your program?

In the early years the usual procedure was to run the program in slow motion, one instruction at a time, and observe what happened. Computers were provided with a number of visual indicators – rows of lights or cathode ray tubes – which enabled the contents of some of the arithmetic, control or storage registers to be inspected. This practice – sometimes known as ‘peeping’ – was soon found to be intolerably slow. It can be speeded up by inserting stop instructions at suitable points, thus enabling the operator to restrict the slow motion to selected parts of the program. Those parts that are above suspicion can be run at full speed. Even with those improvements this procedure is far too prodigal of valuable machine time …

…or waited for an electric typewriter to tell you what the error was?

… most modern installations are provided with a variety of ingenious diagnostic aids … special diagnostic programs, some of which are usually held permanently in the computer store. Their function is to provide the programmer with information that is likely to help him detect, locate, and diagnose any errors in his program. Such information is usually printed as an error message on a line printer or electric typewriter …


Programming Languages Will Proliferate
The idea of a programming language then was still a relatively new one at that time. ‘High level’ programming languages had only recently come into being – COBOL was about as old then as GoLang is now!

In spite of recent attempts to design a ‘universal’ language – for example the PL/1 language – the existing bifurcation into mathematical and commercial languages, typified by ALGOL and COBOL respectively, is likely to persist for some time yet; an economic combination of sports car and delivery van is rather unlikely.

This implied that languages would proliferate for different purposes, as indeed they have.


Towards a National Computer Grid
It’s interesting to hear someone edge towards the idea of the Internet in an age before DNS, TCP/IP, SNAT or indeed the idea of networking in general.

The scheme envisages an hierarchical arrangement of subscriber terminals (‘remote stations in our terminology), multiplexor devices to concentrate and sort out incoming messages and so save data transmission costs, local area computers and regional computers, all linked together by a network of communication lines of varying data carrying capacity. It is proposed to make use of some of the long distance lines already provided for the telephone network


Moore’s Law
A logarithmic graph shows Moore’s Law, though it hadn’t been given that name yet. His 1965 paper was published in the same year as this book. Not only transistors, but magnetic core capacity was seen as growing at a similar rate.


Note that it was the bit, not the byte, kilobyte or megabyte that was the unit of choice at the time.

Multi-Tenancy, 1960s Style
The idea of computer systems that could run multiple programs simultaneously was a novel one. As was BASIC.

In the ‘MIT’ system twenty programs can be ‘active’ simultaneously. … The user of course is not aware of this swapping although he may realise what it is that makes the computer work more slowly than if he had it entirely to himself.


Computer Programming and ‘Libraries’
It was dawning on practitioners at the time that ‘constructing a computer program is like building a house’:


It is clearly a great boon for the programmer to have at his disposal a collection of standard subroutines which have been thoroughly tested in advance and known to work correctly.

I wonder what the author would have made of NodeJS libraries?


Computers at Work
Software had begun to ‘eat the world’ even 50 years ago. I met my wife in the early 2000s through a more modern equivalent of the ‘marriage bureau’ (yes, that was a thing, and I remember them), but it’s interesting to consider that law and medicine arguably haven’t really been revolutionised by computer technology yet (leaving aside hardware innovations).

The table below estimates that there were 70,000 computers worldwide in 1968. Microsoft was founded seven years later with the vision of ‘a computer on every desk and in every home’, which seems tame today when I have more computers on me now than pockets in my clothes.


Here is a rough estimate of the global distribution of computers in the middle of 1968:

North America 46000
United Kingdom 3000
Western and Central Europe 11000
USSR, Eastern Europe and China 5000
Other areas 5000
Total 70000
Author is currently working on the second edition of Docker in Practice

Get 39% off with the code: 39miell2

Posted on November 11, 2017
Categories Uncategorized
1 Comment on ‘Towards a National Computer Grid’ – Electronic Computers, 1965
View ianmiell’s profile on Twitter
View ian-miell-694496’s profile on LinkedIn
View ianmiell’s profile on GitHub
Ten Things I Wish I’d Known About bash
Project Management as Code with Graphviz
How to Manually Clear Locks in Jenkins
How I Manage My Time
Ten Things I Wish I’d Known About Chef
Vagrant and Ohai / Chef IP Address Hack
‘Towards a National Computer Grid’ – Electronic Computers, 1965
A Complete Chef Infrastructure on Your Laptop
Ten Things I Wish I’d Known Before Using Vagrant
A Checklist for Docker in the Enterprise (Updated)
OpenShift 3.6 DNS In Pictures
Puppeteer – Headless Chrome in a Container
My 20-Year Experience of Software Development Methodologies
A Non-Cloud Serverless Application Pattern Using Git and Docker
Run Your Own AWS APIs on OpenShift
Dockerized Headless Chrome Example
Convert a Server to a Docker Container (Update II)
Automating Dockerized Jenkins Upgrades
Ten Things I Wish I’d Known Before Using Jenkins Pipelines
Five Books I Advise Every DevOps Engineer to Read
Things I Learned Managing Site Reliability for Some of the World’s Busiest Gambling Sites
Clustered VM Testing How-To
Easy Shell Automation
1-Minute Multi-Node VM Setup
Migrating an OpenShift etcd Cluster
A Complete OpenShift Cluster on Vagrant, Step by Step
Learn Kubernetes the Hard Way (the Easy and Cheap Way)
Docker in the Enterprise
Terraform and Dynamic Environments
Bash to Python Converter
Hello world Unikernel Walkthrough
A checklist for Docker in the Enterprise
A Quick Tour of Docker 1.12
Power ‘git log’ graphing
ssh -R (reverse tunnel) man page hell
Writing a Technical Book
Interactive Git Tutorials – Rebase and Bisect
Hitler Uses Docker, Annotated
Linux Scales
Play With Kubernetes Quickly Using Docker (Updated)
Convert Any Server to a Docker Container (Updated)
CI as Code Part III: Dynamic Jenkins-Swarm Example
Docker 1.10 Highlights – Updated
CI as Code Part II: Stateless Jenkins With Dynamic Docker Slaves
CI as Code Part I: Stateless Jenkins Deployments Using Docker
Docker Ecosystem Rosetta Stones
Understanding Docker – A Tour of Logical Volume Management
Automating Docker Security Validation
The IT Crowd Was Right – What I learned by reading a lot of RFCs
Understanding Docker – Network Namespaces
DockerConEU 2015 Talk – You Know More Than You Think
Docker Migration In-Flight CRIU
A High Availability Phoenix and A/B Deployment Framework using Docker
Quick Intro to Kubernetes
Take OpenShift for a spin in four commands
RedHat’s Docker Build Method – S2I
Bash Shortcuts Gem
A CoreOS Cluster in Two Minutes With Four Commands
The Most Pointless Docker Command Ever
My Favourite Docker Tip
Convert Any Server to a Docker Container
A Field Guide to Docker Security Measures
Docker SELinux Experimentation with Reduced Pain
Storage Drivers and Docker
Play With Kubernetes Quickly Using Docker
Play with an OpenShift PaaS using Docker
Scale Your Jenkins Compute With Your Dev Team: Use Docker and Jenkins Swarm
Docker in Practice – A Guide for Engineers
Fight Docker Package Drift!
Win at 2048 with Docker and ShutIt (Redux)
Set Up a Deis (Docker-Friendly) Paas on Digital Ocean for $0.18 Per Hour in Six Easy Steps Using ShutIt
Create your own CoreOS cluster in 6 easy steps for $0.03
Make Your Own Bespoke Docker Image
Taming Slaves with Docker and ShutIt
Docker – One Year On
Using ShutIt to Build Your Own Taiga Server
Using ShutIt and Docker to play with AWS (Part Two)
Talk on Docker and ShutIt
Using ShutIt and Docker to play with AWS (Part One)
Phoenix deployment pain (and win)
Phoenix Deployment with Docker and ShutIt
Docker, ShutIt and the Perfect 2048 Game (Videos)
Docker, ShutIt and the Perfect 2048 Game (4 – Halfway There)
Docker, ShutIt and the Perfect 2048 Game (3 – Brute Force Escapes)
Docker, ShutIt and the Perfect 2048 Game (2)
Docker, ShutIt, and The Perfect 2048 Game
My Favourite Secret Weapon – strace
Shakespeare’s Vocabulary Considered Unexceptional
Shakespeare’s Vocabulary Considered Unexceptional
zwischenzugs A Website. Free Programming Ebooks Programming Ebooks

We’ve compiled the best insights from subject matter experts and industry insiders for you in one place, so you can dive deep into the latest of what’s happening in the world of software engineering, architecture, and open source.

Software Engineering
Microservices for Java Developers Modern Java EE Design Patterns Object-Oriented vs. Functional Programming Java: The Legend
A Whirlwind Tour of Python 20 Python Libraries You Aren’t Using (But Should) Hadoop with Python How to Make Mistakes in Python Functional Programming in Python Python in Education Picking a Python Version: A Manifesto
What’s New in Swift 3 Why Reactive? Foundational Principles for Enterprise Adoption Using Blocks in Ruby: A Brain-Friendly Report Fintech, Open Source, and Emerging Markets Practical C++ Metaprogramming Real-World Maintainable Software Trends Shaping the London Tech Scene 2016 European Software Development Salary Survey 2016 Software Development Salary Survey Why Rust? C++ Today RxJava for Android App Development Swift Pocket Reference
Open Source
Open Source in Brazil Ten Steps to Linux Survival Open by Design Getting Started with InnerSource
Software Architecture
Designing Autonomous Teams and Services Migrating Java to the Cloud What Is Serverless? Microservices in Production Microservices AntiPatterns and Pitfalls Evolving Architectures of FinTech Software Architecture Patterns Migrating to Cloud-Native Application Architectures
You might also be interested in our other free ebooks:
Business Data IoT Security Web Development WebOps