Israel Proves the Desalination Era is HereOne of the driest countries on earth now makes more freshwater than it needs

Israel Proves the Desalination Era is Here

One of the driest countries on earth now makes more freshwater than it needs

We are standing above the new Sorek desalination plant, the largest reverse-osmosis desal facility in the world, and we are staring at Israel’s salvation. Just a few years ago, in the depths of its worst drought in at least 900 years, Israel was running out of water. Now it has a surplus. That remarkable turnaround was accomplished throughnational campaigns to conserve and reuse Israel’s meager water resources, but the biggest impact came from a new wave of desalination plants.

Bar-Zeev, who recently joined Israel’s Zuckerberg Institute for Water Research after completing his postdoc work at Yale University, is an expert on biofouling, which has always been an Achilles’ heel of desalination and one of the reasons it has been considered a last resort. Desal works by pushing saltwater into membranes containing microscopic pores. The water gets through, while the larger salt molecules are left behind. But microorganisms in seawater quickly colonize the membranes and block the pores, and controlling them requires periodic costly and chemical-intensive cleaning. But Bar-Zeev and colleaguesdeveloped a chemical-free systemusing porous lava stone to capture the microorganisms before they reach the membranes. It’s just one of many breakthroughs in membrane technology that have made desalination much more efficient. Israel now gets 55 percent of its domestic water from desalination, and that has helped to turn one of the world’s driest countries into the unlikeliest of water giants.

Driven by necessity, Israel is learning to squeeze more out of a drop of water than any country on Earth, and much of that learning is happening at the Zuckerberg Institute, where researchers have pioneered new techniques in drip irrigation, water treatment and desalination. They have developed resilient well systems for African villages and biological digesters than can halve the water usage of most homes.

Bar-Zeev believes that Israel’s solutions can help its parched neighbors, too — and in the process, bring together old enemies in common cause.The institute’s original mission was to improve life in Israel’s bone-dry Negev Desert, but the lessons look increasingly applicable to the entire Fertile Crescent. “The Middle East is drying up,” says Osnat Gillor, a professor at the Zuckerberg Institute who studies the use of recycled wastewater on crops. “The only country that isn’t suffering acute water stress is Israel.”

Why Google Stores Billions of Lines of Code in a Single Repository

Why Google Stores Billions of Lines of Code in a Single Repository

Early Google employees decided to work with a shared codebase managed through a centralized source control system. This approach has served Google well for more than 16 years, and today the vast majority of Google’s software assets continues to be stored in a single, shared repository. Meanwhile, the number of Google software developers has steadily increased, and the size of the Google codebase has grown exponentially (see Figure 1). As a result, the technology used to host the codebase has also evolved significantly.

Key Insights

ins01.gif

This article outlines the scale of that codebase and details Google’s custom-built monolithic source repository and the reasons the model was chosen. Google uses a homegrown version-control system to host one large codebase visible to, and used by, most of the software developers in the company. This centralized system is the foundation of many of Google’s developer workflows. Here, we provide background on the systems and workflows that make feasible managing and working productively with such a large repository. We explain Google’s “trunk-based development” strategy and the support systems that structure workflow and keep Google’s codebase healthy, including software for static analysis, code cleanup, and streamlined code review.

Google-Scale

Google’s monolithic software repository, which is used by 95% of its software developers worldwide, meets the definition of an ultra-large-scale4 system, providing evidence the single-source repository model can be scaled successfully.

The Google codebase includes approximately one billion files and has a history of approximately 35 million commits spanning Google’s entire 18-year existence. The repository contains 86TBa of data, including approximately two billion lines of code in nine million unique source files. The total number of files also includes source files copied into release branches, files that are deleted at the latest revision, configuration files, documentation, and supporting data files; see the table here for a summary of Google’s repository statistics from January 2015.

In 2014, approximately 15 million lines of code were changedb in approximately 250,000 files in the Google repository on a weekly basis. The Linux kernel is a prominent example of a large open source software repository containing approximately 15 million lines of code in 40,000 files.14

Google’s codebase is shared by more than 25,000 Google software developers from dozens of offices in countries around the world. On a typical workday, they commit 16,000 changes to the codebase, and another 24,000 changes are committed by automated systems. Each day the repository serves billions of file read requests, with approximately 800,000 queries per second during peak traffic and an average of approximately 500,000 queries per second each workday. Most of this traffic originates from Google’s distributed build-and-test systems.c

Figure 2 reports the number of unique human committers per week to the main repository, January 2010-July 2015. Figure 3 reports commits per week to Google’s main repository over the same time period. The line for total commits includes data for both the interactive use case, or human users, and automated use cases. Larger dips in both graphs occur during holidays affecting a significant number of employees (such as Christmas Day and New Year’s Day, American Thanksgiving Day, and American Independence Day).

In October 2012, Google’s central repository added support for Windows and Mac users (until then it was Linux-only), and the existing Windows and Mac repository was merged with the main repository. Google’s tooling for repository merges attributes all historical changes being merged to their original authors, hence the corresponding bump in the graph in Figure 2. The effect of this merge is also apparent in Figure 1.

The commits-per-week graph shows the commit rate was dominated by human users until 2012, at which point Google switched to a custom-source-control implementation for hosting the central repository, as discussed later. Following this transition, automated commits to the repository began to increase. Growth in the commit rate continues primarily due to automation.

Managing this scale of repository and activity on it has been an ongoing challenge for Google. Despite several years of experimentation, Google was not able to find a commercially available or open source version-control system to support such scale in a single repository. The Google proprietary system that was built to store, version, and vend this codebase is code-named Piper.

Background

Before reviewing the advantages and disadvantages of working with a monolithic repository, some background on Google’s tooling and workflows is needed.

Piper and CitC. Piper stores a single large repository and is implemented on top of standard Google infrastructure, originally Bigtable,2 now Spanner.3 Piper is distributed over 10 Google data centers around the world, relying on the Paxos6algorithm to guarantee consistency across replicas. This architecture provides a high level of redundancy and helps optimize latency for Google software developers, no matter where they work. In addition, caching and asynchronous operations hide much of the network latency from developers. This is important because gaining the full benefit of Google’s cloud-based toolchain requires developers to be online.

Google relied on one primary Perforce instance, hosted on a single machine, coupled with custom caching infrastructure1 for more than 10 years prior to the launch of Piper. Continued scaling of the Google repository was the main motivation for developing Piper.

Since Google’s source code is one of the company’s most important assets, security features are a key consideration in Piper’s design. Piper supports file-level access control lists. Most of the repository is visible to all Piper users;dhowever, important configuration files or files including business-critical algorithms can be more tightly controlled. In addition, read and write access to files in Piper is logged. If sensitive data is accidentally committed to Piper, the file in question can be purged. The read logs allow administrators to determine if anyone accessed the problematic file before it was removed.

In the Piper workflow (see Figure 4), developers create a local copy of files in the repository before changing them. These files are stored in a workspace owned by the developer. A Piper workspace is comparable to a working copy in Apache Subversion, a local clone in Git, or a client in Perforce. Updates from the Piper repository can be pulled into a workspace and merged with ongoing work, as desired (see Figure 5). A snapshot of the workspace can be shared with other developers for review. Files in a workspace are committed to the central repository only after going through the Google code-review process, as described later.

Most developers access Piper through a system called Clients in the Cloud, or CitC, which consists of a cloud-based storage backend and a Linux-only FUSE13 file system. Developers see their workspaces as directories in the file system, including their changes overlaid on top of the full Piper repository. CitC supports code browsing and normal Unix tools with no need to clone or sync state locally. Developers can browse and edit files anywhere across the Piper repository, and only modified files are stored in their workspace. This structure means CitC workspaces typically consume only a small amount of storage (an average workspace has fewer than 10 files) while presenting a seamless view of the entire Piper codebase to the developer.

All writes to files are stored as snapshots in CitC, making it possible to recover previous stages of work as needed. Snapshots may be explicitly named, restored, or tagged for review.

CitC workspaces are available on any machine that can connect to the cloud-based storage system, making it easy to switch machines and pick up work without interruption. It also makes it possible for developers to view each other’s work in CitC workspaces. Storing all in-progress work in the cloud is an important element of the Google workflow process. Working state is thus available to other tools, including the cloud-based build system, the automated test infrastructure, and the code browsing, editing, and review tools.

Several workflows take advantage of the availability of uncommitted code in CitC to make software developers working with the large codebase more productive. For instance, when sending a change out for code review, developers can enable an auto-commit option, which is particularly useful when code authors and reviewers are in different time zones. When the review is marked as complete, the tests will run; if they pass, the code will be committed to the repository without further human intervention. The Google code-browsing tool CodeSearch supports simple edits using CitC workspaces. While browsing the repository, developers can click on a button to enter edit mode and make a simple change (such as fixing a typo or improving a comment). Then, without leaving the code browser, they can send their changes out to the appropriate reviewers with auto-commit enabled.

Piper can also be used without CitC. Developers can instead store Piper workspaces on their local machines. Piper also has limited interoperability with Git. Over 80% of Piper users today use CitC, with adoption continuing to grow due to the many benefits provided by CitC.

Piper and CitC make working productively with a single, monolithic source repository possible at the scale of the Google codebase. The design and architecture of these systems were both heavily influenced by the trunk-based development paradigm employed at Google, as described here.

Trunk-based development. Google practices trunk-based development on top of the Piper source repository. The vast majority of Piper users work at the “head,” or most recent, version of a single copy of the code called “trunk” or “mainline.” Changes are made to the repository in a single, serial ordering. The combination of trunk-based development with a central repository defines the monolithic codebase model. Immediately after any commit, the new code is visible to, and usable by, all other developers. The fact that Piper users work on a single consistent view of the Google codebase is key for providing the advantages described later in this article.

Trunk-based development is beneficial in part because it avoids the painful merges that often occur when it is time to reconcile long-lived branches. Development on branches is unusual and not well supported at Google, though branches are typically used for releases. Release branches are cut from a specific revision of the repository. Bug fixes and enhancements that must be added to a release are typically developed on mainline, then cherry-picked into the release branch (see Figure 6). Due to the need to maintain stability and limit churn on the release branch, a release is typically a snapshot of head, with an optional small number of cherry-picks pulled in from head as needed. Use of long-lived branches with parallel development on the branch and mainline is exceedingly rare.


Piper and CitC make working productively with a single, monolithic source repository possible at the scale of the Google codebase.


When new features are developed, both new and old code paths commonly exist simultaneously, controlled through the use of conditional flags. This technique avoids the need for a development branch and makes it easy to turn on and off features through configuration updates rather than full binary releases. While some additional complexity is incurred for developers, the merge problems of a development branch are avoided. Flag flips make it much easier and faster to switch users off new implementations that have problems. This method is typically used in project-specific code, not common library code, and eventually flags are retired so old code can be deleted. Google uses a similar approach for routing live traffic through different code paths to perform experiments that can be tuned in real time through configuration changes. Such A/B experiments can measure everything from the performance characteristics of the code to user engagement related to subtle product changes.

Google workflow. Several best practices and supporting systems are required to avoid constant breakage in the trunk-based development model, where thousands of engineers commit thousands of changes to the repository on a daily basis. For instance, Google has an automated testing infrastructure that initiates a rebuild of all affected dependencies on almost every change committed to the repository. If a change creates widespread build breakage, a system is in place to automatically undo the change. To reduce the incidence of bad code being committed in the first place, the highly customizable Google “presubmit” infrastructure provides automated testing and analysis of changes before they are added to the codebase. A set of global presubmit analyses are run for all changes, and code owners can create custom analyses that run only on directories within the codebase they specify. A small set of very low-level core libraries uses a mechanism similar to a development branch to enforce additional testing before new versions are exposed to client code.

An important aspect of Google culture that encourages code quality is the expectation that all code is reviewed before being committed to the repository. Most developers can view and propose changes to files anywhere across the entire codebase—with the exception of a small set of highly confidential code that is more carefully controlled. The risk associated with developers changing code they are not deeply familiar with is mitigated through the code-review process and the concept of code ownership. The Google codebase is laid out in a tree structure. Each and every directory has a set of owners who control whether a change to files in their directory will be accepted. Owners are typically the developers who work on the projects in the directories in question. A change often receives a detailed code review from one developer, evaluating the quality of the change, and a commit approval from an owner, evaluating the appropriateness of the change to their area of the codebase.

Code reviewers comment on aspects of code quality, including design, functionality, complexity, testing, naming, comment quality, and code style, as documented by the various language-specific Google style guides.e Google has written a code-review tool called Critique that allows the reviewer to view the evolution of the code and comment on any line of the change. It encourages further revisions and a conversation leading to a final “Looks Good To Me” from the reviewer, indicating the review is complete.

Google’s static analysis system (Tricorder10) and presubmit infrastructure also provide data on code quality, test coverage, and test results automatically in the Google code-review tool. These computationally intensive checks are triggered periodically, as well as when a code change is sent for review. Tricorder also provides suggested fixes with one-click code editing for many errors. These systems provide important data to increase the effectiveness of code reviews and keep the Google codebase healthy.

A team of Google developers will occasionally undertake a set of wide-reaching code-cleanup changes to further maintain the health of the codebase. The developers who perform these changes commonly separate them into two phases. With this approach, a large backward-compatible change is made first. Once it is complete, a second smaller change can be made to remove the original pattern that is no longer referenced. A Google tool called Rosief supports the first phase of such large-scale cleanups and code changes. With Rosie, developers create a large patch, either through a find-and-replace operation across the entire repository or through more complex refactoring tools. Rosie then takes care of splitting the large patch into smaller patches, testing them independently, sending them out for code review, and committing them automatically once they pass tests and a code review. Rosie splits patches along project directory lines, relying on the code-ownership hierarchy described earlier to send patches to the appropriate reviewers.

Figure 7 reports the number of changes committed through Rosie on a monthly basis, demonstrating the importance of Rosie as a tool for performing large-scale code changes at Google. Using Rosie is balanced against the cost incurred by teams needing to review the ongoing stream of simple changes Rosie generates. As Rosie’s popularity and usage grew, it became clear some control had to be established to limit Rosie’s use to high-value changes that would be distributed to many reviewers, rather than to single atomic changes or rejected. In 2013, Google adopted a formal large-scale change-review process that led to a decrease in the number of commits through Rosie from 2013 to 2014. In evaluating a Rosie change, the review committee balances the benefit of the change against the costs of reviewer time and repository churn. We later examine this and similar trade-offs more closely.

In sum, Google has developed a number of practices and tools to support its enormous monolithic codebase, including trunk-based development, the distributed source-code repository Piper, the workspace client CitC, and workflow-support-tools Critique, CodeSearch, Tricorder, and Rosie. We discuss the pros and cons of this model here.

Analysis

This section outlines and expands upon both the advantages of a monolithic codebase and the costs related to maintaining such a model at scale.

Advantages. Supporting the ultra-large-scale of Google’s codebase while maintaining good performance for tens of thousands of users is a challenge, but Google has embraced the monolithic model due to its compelling advantages.

Most important, it supports:

  • Unified versioning, one source of truth;
  • Extensive code sharing and reuse;
  • Simplified dependency management;
  • Atomic changes;
  • Large-scale refactoring;
  • Collaboration across teams;
  • Flexible team boundaries and code ownership; and
  • Code visibility and clear tree structure providing implicit team namespacing.

A single repository provides unified versioning and a single source of truth. There is no confusion about which repository hosts the authoritative version of a file. If one team wants to depend on another team’s code, it can depend on it directly. The Google codebase includes a wealth of useful libraries, and the monolithic repository leads to extensive code sharing and reuse.

The Google build system5 makes it easy to include code across directories, simplifying dependency management. Changes to the dependencies of a project trigger a rebuild of the dependent code. Since all code is versioned in the same repository, there is only ever one version of the truth, and no concern about independent versioning of dependencies.

Most notably, the model allows Google to avoid the “diamond dependency” problem (see Figure 8) that occurs when A depends on B and C, both B and C depend on D, but B requires version D.1 and C requires version D.2. In most cases it is now impossible to build A. For the base library D, it can become very difficult to release a new version without causing breakage, since all its callers must be updated at the same time. Updating is difficult when the library callers are hosted in different repositories.

In the open source world, dependencies are commonly broken by library updates, and finding library versions that all work together can be a challenge. Updating the versions of dependencies can be painful for developers, and delays in updating create technical debt that can become very expensive. In contrast, with a monolithic source tree it makes sense, and is easier, for the person updating a library to update all affected dependencies at the same time. The technical debt incurred by dependent systems is paid down immediately as changes are made. Changes to base libraries are instantly propagated through the dependency chain into the final products that rely on the libraries, without requiring a separate sync or migration step.

Note the diamond-dependency problem can exist at the source/API level, as described here, as well as between binaries.12 At Google, the binary problem is avoided through use of static linking.

The ability to make atomic changes is also a very powerful feature of the monolithic model. A developer can make a major change touching hundreds or thousands of files across the repository in a single consistent operation. For instance, a developer can rename a class or function in a single commit and yet not break any builds or tests.

The availability of all source code in a single repository, or at least on a centralized server, makes it easier for the maintainers of core libraries to perform testing and performance benchmarking for high-impact changes before they are committed. This approach is useful for exploring and measuring the value of highly disruptive changes. One concrete example is an experiment to evaluate the feasibility of converting Google data centers to support non-x86 machine architectures.

With the monolithic structure of the Google repository, a developer never has to decide where the repository boundaries lie. Engineers never need to “fork” the development of a shared library or merge across repositories to update copied versions of code. Team boundaries are fluid. When project ownership changes or plans are made to consolidate systems, all code is already in the same repository. This environment makes it easy to do gradual refactoring and reorganization of the codebase. The change to move a project and update all dependencies can be applied atomically to the repository, and the development history of the affected code remains intact and available.

Another attribute of a monolithic repository is the layout of the codebase is easily understood, as it is organized in a single tree. Each team has a directory structure within the main tree that effectively serves as a project’s own namespace. Each source file can be uniquely identified by a single string—a file path that optionally includes a revision number. Browsing the codebase, it is easy to understand how any source file fits into the big picture of the repository.

The Google codebase is constantly evolving. More complex codebase modernization efforts (such as updating it to C++11 or rolling out performance optimizations9) are often managed centrally by dedicated codebase maintainers. Such efforts can touch half a million variable declarations or function-call sites spread across hundreds of thousands of files of source code. Because all projects are centrally stored, teams of specialists can do this work for the entire company, rather than require many individuals to develop their own tools, techniques, or expertise.

As an example of how these benefits play out, consider Google’s Compiler team, which ensures developers at Google employ the most up-to-date toolchains and benefit from the latest improvements in generated code and “debuggability.” The monolithic repository provides the team with full visibility of how various languages are used at Google and allows them to do codebase-wide cleanups to prevent changes from breaking builds or creating issues for developers. This greatly simplifies compiler validation, thus reducing compiler release cycles and making it possible for Google to safely do regular compiler releases (typically more than 20 per year for the C++ compilers).

Using the data generated by performance and regression tests run on nightly builds of the entire Google codebase, the Compiler team tunes default compiler settings to be optimal. For example, due to this centralized effort, Google’s Java developers all saw their garbage collection (GC) CPU consumption decrease by more than 50% and their GC pause time decrease by 10%–40% from 2014 to 2015. In addition, when software errors are discovered, it is often possible for the team to add new warnings to prevent reoccurrence. In conjunction with this change, they scan the entire repository to find and fix other instances of the software issue being addressed, before turning to new compiler errors. Having the compiler-reject patterns that proved problematic in the past is a significant boost to Google’s overall code health.

Storing all source code in a common version-control repository allows codebase maintainers to efficiently analyze and change Google’s source code. Tools like Refaster11 and ClangMR15 (often used in conjunction with Rosie) make use of the monolithic view of Google’s source to perform high-level transformations of source code. The monolithic codebase captures all dependency information. Old APIs can be removed with confidence, because it can be proven that all callers have been migrated to new APIs. A single common repository vastly simplifies these tools by ensuring atomicity of changes and a single global view of the entire repository at any given time.


An important aspect of Google culture that encourages code quality is the expectation that all code is reviewed before being committed to the repository.


Costs and trade-offs. While important to note a monolithic codebase in no way implies monolithic software design, working with this model involves some downsides, as well as trade-offs, that must be considered.

These costs and trade-offs fall into three categories:

  • Tooling investments for both development and execution;
  • Codebase complexity, including unnecessary dependencies and difficulties with code discovery; and
  • Effort invested in code health.

In many ways the monolithic repository yields simpler tooling since there is only one system of reference for tools working with source. However, it is also necessary that tooling scale to the size of the repository. For instance, Google has written a custom plug-in for the Eclipse integrated development environment (IDE) to make working with a massive codebase possible from the IDE. Google’s code-indexing system supports static analysis, cross-referencing in the code-browsing tool, and rich IDE functionality for Emacs, Vim, and other development environments. These tools require ongoing investment to manage the ever-increasing scale of the Google codebase.

Beyond the investment in building and maintaining scalable tooling, Google must also cover the cost of running these systems, some of which are very computationally intensive. Much of Google’s internal suite of developer tools, including the automated test infrastructure and highly scalable build infrastructure, are critical for supporting the size of the monolithic codebase. It is thus necessary to make trade-offs concerning how frequently to run this tooling to balance the cost of execution vs. the benefit of the data provided to developers.

The monolithic model makes it easier to understand the structure of the codebase, as there is no crossing of repository boundaries between dependencies. However, as the scale increases, code discovery can become more difficult, as standard tools like grep bog down. Developers must be able to explore the codebase, find relevant libraries, and see how to use them and who wrote them. Library authors often need to see how their APIs are being used. This requires a significant investment in code search and browsing tools. However, Google has found this investment highly rewarding, improving the productivity of all developers, as described in more detail by Sadowski et al.9

Access to the whole codebase encourages extensive code sharing and reuse. Some would argue this model, which relies on the extreme scalability of the Google build system, makes it too easy to add dependencies and reduces the incentive for software developers to produce stable and well-thought-out APIs.

Due to the ease of creating dependencies, it is common for teams to not think about their dependency graph, making code cleanup more error-prone. Unnecessary dependencies can increase project exposure to downstream build breakages, lead to binary size bloating, and create additional work in building and testing. In addition, lost productivity ensues when abandoned projects that remain in the repository continue to be updated and maintained.

Several efforts at Google have sought to rein in unnecessary dependencies. Tooling exists to help identify and remove unused dependencies, or dependencies linked into the product binary for historical or accidental reasons, that are not needed. Tooling also exists to identify underutilized dependencies, or dependencies on large libraries that are mostly unneeded, as candidates for refactoring.7 One such tool, Clipper, relies on a custom Java compiler to generate an accurate cross-reference index. It then uses the index to construct a reachability graph and determine what classes are never used. Clipper is useful in guiding dependency-refactoring efforts by finding targets that are relatively easy to remove or break up.


A developer can make a major change touching hundreds or thousands of files across the repository in a single consistent operation.


Dependency-refactoring and cleanup tools are helpful, but, ideally, code owners should be able to prevent unwanted dependencies from being created in the first place. In 2011, Google started relying on the concept of API visibility, setting the default visibility of new APIs to “private.” This forces developers to explicitly mark APIs as appropriate for use by other teams. A lesson learned from Google’s experience with a large monolithic repository is such mechanisms should be put in place as soon as possible to encourage more hygienic dependency structures.

The fact that most Google code is available to all Google developers has led to a culture where some teams expect other developers to read their code rather than providing them with separate user documentation. There are pros and cons to this approach. No effort goes toward writing or keeping documentation up to date, but developers sometimes read more than the API code and end up relying on underlying implementation details. This behavior can create a maintenance burden for teams that then have trouble deprecating features they never meant to expose to users.

This model also requires teams to collaborate with one another when using open source code. An area of the repository is reserved for storing open source code (developed at Google or externally). To prevent dependency conflicts, as outlined earlier, it is important that only one version of an open source project be available at any given time. Teams that use open source software are expected to occasionally spend time upgrading their codebase to work with newer versions of open source libraries when library upgrades are performed.

Google invests significant effort in maintaining code health to address some issues related to codebase complexity and dependency management. For instance, special tooling automatically detects and removes dead code, splits large refactorings and automatically assigns code reviews (as through Rosie), and marks APIs as deprecated. Human effort is required to run these tools and manage the corresponding large-scale code changes. A cost is also incurred by teams that need to review an ongoing stream of simple refactorings resulting from codebase-wide clean-ups and centralized modernization efforts.

Alternatives

As the popularity and use of distributed version control systems (DVCSs) like Git have grown, Google has considered whether to move from Piper to Git as its primary version-control system. A team at Google is focused on supporting Git, which is used by Google’s Android and Chrome teams outside the main Google repository. The use of Git is important for these teams due to external partner and open source collaborations.

The Git community strongly suggests and prefers developers have more and smaller repositories. A Git-clone operation requires copying all content to one’s local machine, a procedure incompatible with a large repository. To move to Git-based source hosting, it would be necessary to split Google’s repository into thousands of separate repositories to achieve reasonable performance. Such reorganization would necessitate cultural and workflow changes for Google’s developers. As a comparison, Google’s Git-hosted Android codebase is divided into more than 800 separate repositories.

Given the value gained from the existing tools Google has built and the many advantages of the monolithic codebase structure, it is clear that moving to more and smaller repositories would not make sense for Google’s main repository. The alternative of moving to Git or any other DVCS that would require repository splitting is not compelling for Google.

Current investment by the Google source team focuses primarily on the ongoing reliability, scalability, and security of the in-house source systems. The team is also pursuing an experimental effort with Mercurial,g an open source DVCS similar to Git. The goal is to add scalability features to the Mercurial client so it can efficiently support a codebase the size of Google’s. This would provide Google’s developers with an alternative of using popular DVCS-style workflows in conjunction with the central repository. This effort is in collaboration with the open source Mercurial community, including contributors from other companies that value the monolithic source model.

Conclusion

Google chose the monolithic-source-management strategy in 1999 when the existing Google codebase was migrated from CVS to Perforce. Early Google engineers maintained that a single repository was strictly better than splitting up the codebase, though at the time they did not anticipate the future scale of the codebase and all the supporting tooling that would be built to make the scaling feasible.

Over the years, as the investment required to continue scaling the centralized repository grew, Google leadership occasionally considered whether it would make sense to move from the monolithic model. Despite the effort required, Google repeatedly chose to stick with the central repository due to its advantages.

The monolithic model of source code management is not for everyone. It is best suited to organizations like Google, with an open and collaborative culture. It would not work well for organizations where large parts of the codebase are private or hidden between groups.

At Google, we have found, with some investment, the monolithic model of source management can scale successfully to a codebase with more than one billion files, 35 million commits, and thousands of users around the globe. As the scale and complexity of projects both inside and outside Google continue to grow, we hope the analysis and workflow described in this article can benefit others weighing decisions on the long-term structure for their codebases.

Acknowledgments

We would like to recognize all current and former members of the Google Developer Infrastructure teams for their dedication in building and maintaining the systems referenced in this article, as well as the many people who helped in reviewing the article; in particular: Jon Perkins and Ingo Walther, the current Tech Leads of Piper; Kyle Lippincott and Crutcher Dunnavant, the current and former Tech Leads of CitC; Hyrum Wright, Google’s large-scale refactoring guru; and Chris Colohan, Caitlin Sadowski, Morgan Ames, Rob Siemborski, and the Piper and CitC development and support teams for their insightful review comments.

References

1. Bloch, D. Still All on One Server: Perforce at Scale. Google White Paper, 2011;http://info.perforce.com/rs/perforce/images/GoogleWhitePaper-StillAllonOneServer-PerforceatScale.pdf

2. Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., and Gruber, R.E. Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems 26, 2 (June 2008).

3. Corbett, J.C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P. et al. Spanner: Google’s globally distributed database. ACM Transactions on Computer Systems 31, 3 (Aug. 2013).

4. Gabriel, R.P., Northrop, L., Schmidt, D.C., and Sullivan, K. Ultra-large-scale systems. In Companion to the 21st ACM SIGPLAN Symposium on Object-Oriented Programming Systems, Languages, and Applications (Portland, OR, Oct. 22-26). ACM Press, New York, 2006, 632–634.

5. Kemper, C. Build in the Cloud: How the Build System works. Google Engineering Tools blog post, 2011; http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html

6. Lamport, L. Paxos made simple. ACM Sigact News 32, 4 (Nov. 2001), 18–25.

7. Morgenthaler, J.D., Gridnev, M., Sauciuc, R., and Bhansali, S. Searching for build debt: Experiences managing technical debt at Google. In Proceedings of the Third International Workshop on Managing Technical Debt (Zürich, Switzerland, June 2-9). IEEE Press Piscataway, NJ, 2012, 1–6.

8. Ren, G., Tune, E., Moseley, T., Shi, Y., Rus, S., and Hundt, R. Google-wide profiling: A continuous profiling infrastructure for data centers. IEEE Micro 30, 4 (2010), 65–79.

9. Sadowski, C., Stolee, K., and Elbaum, S. How developers search for code: A case study. InProceedings of the 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy, Aug. 30-Sept. 4). ACM Press, New York, 2015, 191–201.

10. Sadowski, C., van Gogh, J., Jaspan, C., Soederberg, E., and Winter, C. Tricorder: Building a program analysis ecosystem. In Proceedings of the 37th International Conference on Software Engineering, Vol. 1 (Firenze, Italy, May 16-24). IEEE Press Piscataway, NJ, 2015, 598–608.

11. Wasserman, L. Scalable, example-based refactorings with Refaster. In Proceedings of the 2013 ACM Workshop on Refactoring Tools(Indianapolis, IN, Oct. 26-31). ACM Press, New York, 2013, 25–28.

12. Wikipedia. Dependency hell. Accessed Jan. 20, 2015; http://en.wikipedia.org/w/index.php?title=Dependency_hell&oldid=634636715

13. Wikipedia. Filesystem in userspace. Accessed June, 4, 2015; http://en.wikipedia.org/w/index.php?title=Filesystem_in_Userspace&oldid=664776514

14. Wikipedia. Linux kernel. Accessed Jan. 20, 2015; http://en.wikipedia.org/w/index.php?title=Linux_kernel&oldid=643170399

15. Wright, H.K., Jasper, D., Klimek, M., Carruth, C., and Wan, Z. Large-scale automated refactoring using ClangMR. In Proceedings of the IEEE International Conference on Software Maintenance(Eindhoven, The Netherlands, Sept. 22-28). IEEE Press, 2013, 548–551.

Authors

Rachel Potvin (rpotvin@google.com) is an engineering manager at Google, Mountain View, CA.

Josh Levenberg (joshl@google.com) is a software engineer at Google, Mountain View, CA.

Footnotes

a. Total size of uncompressed content, excluding release branches.

b. Includes only reviewed and committed code and excludes commits performed by automated systems, as well as commits to release branches, data files, generated files, open source files imported into the repository, and other non-source-code files.

c. Google open sourced a subset of its internal build system; see http://www.bazel.io

d. Over 99% of files stored in Piper are visible to all full-time Google engineers.

e. https://github.com/google/styleguide

f. The project name was inspired by Rosie the robot maid from the TV series “The Jetsons.”

g. http://mercurial.selenic.com/

ONAK: the origami foldable canoe

ONAK: the origami foldable canoe

ONAK’s case on wheels unfolds to a very capable canoe in less than 10 minutes. Thanks to important material innovations, it is suited for urban adventures as well as the most sublime and wild outdoors.

lightweight

An ONAK canoe weighs about 17kg. It can easily carry more than 230kg; enough for 2 adults and a child or luggage
compact &
transportable

120 x 40 x 25 cm
with wheels, so you can easily take it anywhere, on any kind of transport
assembles
in no time

15 minutes assembly
10 minutes packing
paddles great

thanks to its great shape and material ONAK canoes paddle great, even with a breeze in the face
durable

The ONAK canoe can handle quite a beating, thanks to our custom developed
honeycomb-Curv™ composite material.

wigle.net: Finding WiFi

wigle.net: Finding WiFi

What is This Thing For?

We consolidate location and information of wireless networks world-wide to a central database, and have user-friendly desktop and web applications that can map, query and update the database via the web.

We currently accept files in any of:

To have your record removed from our database, or if you have any questions or suggestions, send an email to: WiGLE-admin[at]WiGLE.net or irc chat on at wigle.net:6667. If you want a record removed from the database, please include the BSSID (Mac Address) of the network in question!

Who are you people?

  • arkasha focuses on front-end stuff as well as serving as cruise activities directory
  • bobzilla is the guy who designed the DB, makes everything work, wrote the Android client, and continues to keep this project going
  • uhtu fixes all the things that arkasha and bobzilla do wrong
  • thuddwhir wrote the vector-based map generation (SquiGLE)
  • wos wrote the Mac OSX native client (TinGLE)

All are members of Mimezine.

How does triangulation work?

The “triangulation” is actually just an average of the latitudes and longitudes gathered using the signal strength (squared) as a weight. This is probably more correctly called “weighted-centroid trilateration.” This assumes that signal strength will change at the inverse square of the distance. This is reasonable as long is you don’t get a one-sided view of the network (i.e. only sample it from one side) since it will be skewed in that direction.

How do I Get Started?

WiGLE.net is a submission-based catalog of wireless networks. Submissions are not paired with actual people; rather name/password identities which people use to associate their data. It’s basically a “gee isn’t this neat” engine for learning about the spread of wireless computer usage.

WiGLE concerns itself with 802.11a/b/g/n and cellular networks right now, which can be collected via the WiGLE WiFi Wardriving tool on android. We also have a bluetooth stumbling client for Android, but we do not maintain a catalog of bluetooth networks.

The first step in using WiGLE is to create a username for yourself. You don’t have to submit anything other than a made-up email, username and password, validation is immediate, and we will not contact you (unless you wanna chat on our message boards). This will give you access to our query engine and software downloads.

Once you’ve signed up, you’re free to send us wireless network traces (in any of our listed formats, usually pairings of wireless sample, names and network hardware addresses (for uniqueness), data/SNR triples and GPS coordinates) or enter networks manually. Note, your username gets “credit” for these, but of course some people don’t want their networks listed (various reasons), so we delist these immediately upon request. Once you make a User Name, you can look at the submissions statistics page to see how users stack up.

The easiest way to start stumbling is to use our Android tools, available at:
Android app on Google Play or  (or search for “wiglewifi” on an Android device)

To view networks, you can 1.) ask the website, 2.) download the java client. The clients are particularly fun to look at, but require either a java-1.6-and-up machine (windows, sunOS, MacOS-X, linux, mostly) or a windows box, for the new windows-native prototype. This will superimpose “points” from a live query onto a map of an area. Maps can be downloaded in “packs” from our mapping engine and are installed simply by unzipping them into your client installation directory. Mappacks are created and served by-state-by-county, or in the case of large cities, by-state-by-city. If we haven’t generated a map for your area of interest yet, ask for it, and come back after the rendering engine’s had a minute or 5 to think about it.

If your network is in WiGLE and you don’t like it, we’ll take it out immediately, but you should look into making your network harder to detect AND more secure; remember that you’re the one bombarding passers-by with your signal. We aren’t affiliated directly with any particular community or interest (other than our own), but we applaud the efforts of the people who wrote the stumbling software that feeds our project, the people looking to use wireless in innovative ways, and especially the community of people who just dig wireless network access and dig sharing it. (freenets)

What has this project been used for?

Overall, WiGLE aims to show people about wireless in a more-technical capacity then your average static map or newspaper article.

Educating the Public: When WiGLE started in 2001 the default for wifi was to have no encryption at all. An end user had a choice to turn on weak encryption (40-bit WEP), or pay extra for slightly less weak encryption (128-bit WEP). Most users didn’t realize that their private networks could be accessed by anyone in the area (at various distances with antennas and amplifiers). WiGLE, and Wardriving in general, helped to educate users and put pressure on manufacturers to make network security better and easier. Now a days WPA2 is the standard and defaults to “on” for most manufactured devices.

Research Projects: We are often contacted by graduate students and professors to provide additional access or slices of data for research projects around wifi and security, which we attempt to help in any way we can. We’ve heard of research into subjects like WEP/WPA encryption distribution, information leaks from SSID probes, teaching wireless security concepts, geolocation from BSSID’s and more.

Site Surveys: Security professionals use WiGLE to monitor what businesses they consult for are broadcasting over wifi radios. If you have an insecure network you are vulnerable, whether or not it appears in WiGLE. But WiGLE could let you know that an employee or attacker has set up a rogue wifi router on your private network.

Journalism: WiGLE is a handy resource for journalists looking for data points on Wifi proliferation, statistics and security. WiGLEkeeps statistics on manufacturer distribution, top SSID’s, encryption modes, channel selection, geographic distribution and much more.

Finding Useable Networks: Using the “Free Net” search WiGLE users can find legal-to-use open networks near to their current location. They can also find local government open wireless networks, a cable company wireless offering that they are already paying for, or a local coffee shop that has a network available for use.

A Fun Hobby: Many users enjoy trying to detect as many networks as possible, to compete on the leaderboard and see parts of their city/town that they’ve had no reason to visit before. Teams can compete to see which car can find the most networks in a set amount of time.

What’s the License for All This?

The official legal End User License Agreement. We basically just offer no warranties on our software, and don’t want our software to be used for unlicensed commercial gain.

Commercial Use

To help fund WiGLE operations, we offer licenses to a subset of the data derived from postings that our users have permitted us to use for commercial purposes. As a matter of policy, we will not comment on future, past, or hypothetical customers. To discuss a commercial license, please email us at WiGLE-admin[at]WiGLE.net

We additionally refuse to claim or disclaim any involvement with the alleged moon landings, area 51 and ‘bigfoot’.

Why Don’t You Offer Non US Maps for the Java Client?

We’d love to offer world-wide, street-level mapping information, but there’s nothing free for our use. WiGLE relies on the TIGER and VMAP0 mapping data sets for our current maps, which allows us to produce our maps without intellectual property entanglement or licensing fees. We’re certainly interested in hearing from you about candidate datasets which we can use under similar terms, but we’re not necessarily going to write a new parser for each country/city/neighborhood in the world. See The forums for an up-to-date status on our search and instructions for composing your own raster mappacks. We’d urge you to consider contributing to Open Street Map, and if anyone has a tutorial on how to build mappacks from OSM tiles, let us know!

How to insert clob data more than 4000 character in CLOB column

How to insert clob data more than 4000 character in CLOB column

We have table workingorder where one column document is CLOB data type. We need to insert XML file contain in this column. I has more than 4000 character. we can direcly insert the whole file contain using clob function. Below is the sample :

Insert into table (clob_column) values ( to_clob( 'value of 4000 characters' ) || to_clob( 'value of remaining 1500 characters' ));

newyorker.com: Donald Trump’s Ghostwriter Tells All

newyorker.com: Donald Trump’s Ghostwriter Tells All

Last June, as dusk fell outside Tony Schwartz’s sprawling house, on a leafy back road in Riverdale, New York, he pulled out his laptop and caught up with the day’s big news: Donald J. Trump had declared his candidacy for President. As Schwartz watched a video of the speech, he began to feel personally implicated.

Trump, facing a crowd that had gathered in the lobby of Trump Tower, on Fifth Avenue, laid out his qualifications, saying, “We need a leader that wrote ‘The Art of the Deal.’ ” If that was so, Schwartz thought, then he, not Trump, should be running. Schwartz dashed off a tweet: “Many thanks Donald Trump for suggesting I run for President, based on the fact that I wrote ‘The Art of the Deal.’ ”

Schwartz had ghostwritten Trump’s 1987 breakthrough memoir, earning a joint byline on the cover, half of the book’s five-hundred-thousand-dollar advance, and half of the royalties. The book was a phenomenal success, spending forty-eight weeks on the Times best-seller list, thirteen of them at No. 1. More than a million copies have been bought, generating several million dollars in royalties. The book expanded Trump’s renown far beyond New York City, making him an emblem of the successful tycoon. Edward Kosner, the former editor and publisher of New York, where Schwartz worked as a writer at the time, says, “Tony created Trump. He’s Dr. Frankenstein.”

Starting in late 1985, Schwartz spent eighteen months with Trump—camping out in his office, joining him on his helicopter, tagging along at meetings, and spending weekends with him at his Manhattan apartment and his Florida estate. During that period, Schwartz felt, he had got to know him better than almost anyone else outside the Trump family. Until Schwartz posted the tweet, though, he had not spoken publicly about Trump for decades. It had never been his ambition to be a ghostwriter, and he had been glad to move on. But, as he watched a replay of the new candidate holding forth for forty-five minutes, he noticed something strange: over the decades, Trump appeared to have convinced himself that he had written the book. Schwartz recalls thinking, “If he could lie about that on Day One—when it was so easily refuted—he is likely to lie about anything.”

It seemed improbable that Trump’s campaign would succeed, so Schwartz told himself that he needn’t worry much. But, as Trump denounced Mexican immigrants as “rapists,” near the end of the speech, Schwartz felt anxious. He had spent hundreds of hours observing Trump firsthand, and felt that he had an unusually deep understanding of what he regarded as Trump’s beguiling strengths and disqualifying weaknesses. Many Americans, however, saw Trump as a charmingly brash entrepreneur with an unfailing knack for business—a mythical image that Schwartz had helped create. “It pays to trust your instincts,” Trump says in the book, adding that he was set to make hundreds of millions of dollars after buying a hotel that he hadn’t even walked through.

In the subsequent months, as Trump defied predictions by establishing himself as the front-runner for the Republican nomination, Schwartz’s desire to set the record straight grew. He had long since left journalism to launch the Energy Project, a consulting firm that promises to improve employees’ productivity by helping them boost their “physical, emotional, mental, and spiritual” morale. It was a successful company, with clients such as Facebook, and Schwartz’s colleagues urged him to avoid the political fray. But the prospect of President Trump terrified him. It wasn’t because of Trump’s ideology—Schwartz doubted that he had one. The problem was Trump’s personality, which he considered pathologically impulsive and self-centered.

Schwartz thought about publishing an article describing his reservations about Trump, but he hesitated, knowing that, since he’d cashed in on the flattering “Art of the Deal,” his credibility and his motives would be seen as suspect. Yet watching the campaign was excruciating. Schwartz decided that if he kept mum and Trump was elected he’d never forgive himself. In June, he agreed to break his silence and give his first candid interview about the Trump he got to know while acting as his Boswell.

“I put lipstick on a pig,” he said. “I feel a deep sense of remorse that I contributed to presenting Trump in a way that brought him wider attention and made him more appealing than he is.” He went on, “I genuinely believe that if Trump wins and gets the nuclear codes there is an excellent possibility it will lead to the end of civilization.”

If he were writing “The Art of the Deal” today, Schwartz said, it would be a very different book with a very different title. Asked what he would call it, he answered, “The Sociopath.”

The idea of Trump writing an autobiography didn’t originate with either Trump or Schwartz. It began with Si Newhouse, the media magnate whose company, Advance Publications, owned Random House at the time, and continues to own Condé Nast, the parent company of this magazine. “It was very definitely, and almost uniquely, Si Newhouse’s idea,” Peter Osnos, who edited the book, recalls. GQ, which Condé Nast also owns, had published a cover story on Trump, and Newhouse noticed that newsstand sales had been unusually strong.

Newhouse called Trump about the project, then visited him to discuss it. Random House continued the pursuit with a series of meetings. At one point, Howard Kaminsky, who ran Random House then, wrapped a thick Russian novel in a dummy cover that featured a photograph of Trump looking like a conquering hero; at the top was Trump’s name, in large gold block lettering. Kaminsky recalls that Trump was pleased by the mockup, but had one suggestion: “Please make my name much bigger.” After securing the half-million-dollar advance, Trump signed a contract.

Around this time, Schwartz, who was one of the leading young magazine writers of the day, stopped by Trump’s office, in Trump Tower. Schwartz had written about Trump before. In 1985, he’d published a piece in New York called “A Different Kind of Donald Trump Story,” which portrayed him not as a brilliant mogul but as a ham-fisted thug who had unsuccessfully tried to evict rent-controlled and rent-stabilized tenants from a building that he had bought on Central Park South. Trump’s efforts—which included a plan to house homeless people in the building in order to harass the tenants—became what Schwartz described as a “fugue of failure, a farce of fumbling and bumbling.” An accompanying cover portrait depicted Trump as unshaven, unpleasant-looking, and shiny with sweat. Yet, to Schwartz’s amazement, Trump loved the article. He hung the cover on a wall of his office, and sent a fan note to Schwartz, on his gold-embossed personal stationery. “Everybody seems to have read it,” Trump enthused in the note, which Schwartz has kept.

“I was shocked,” Schwartz told me. “Trump didn’t fit any model of human being I’d ever met. He was obsessed with publicity, and he didn’t care what you wrote.” He went on, “Trump only takes two positions. Either you’re a scummy loser, liar, whatever, or you’re the greatest. I became the greatest. He wanted to be seen as a tough guy, and he loved being on the cover.” Schwartz wrote him back, saying, “Of all the people I’ve written about over the years, you are certainly the best sport.”

And so Schwartz had returned for more, this time to conduct an interview for Playboy. But to his frustration Trump kept making cryptic, monosyllabic statements. “He mysteriously wouldn’t answer my questions,” Schwartz said. After twenty minutes, he said, Trump explained that he didn’t want to reveal anything new about himself—he had just signed a lucrative book deal and needed to save his best material.

“What kind of book?” Schwartz said.

“My autobiography,” Trump replied.

“You’re only thirty-eight—you don’t have one yet!” Schwartz joked.

“Yeah, I know,” Trump said.

“If I were you,” Schwartz recalls telling him, “I’d write a book called ‘The Art of the Deal.’ That’s something people would be interested in.”

“You’re right,” Trump agreed. “Do you want to write it?”

Schwartz thought it over for several weeks. He knew that he would be making a Faustian bargain. A lifelong liberal, he was hardly an admirer of Trump’s ruthless and single-minded pursuit of profit. “It was one of a number of times in my life when I was divided between the Devil and the higher side,” he told me. He had grown up in a bourgeois, intellectual family in Manhattan, and had attended élite private schools, but he was not as wealthy as some of his classmates—and, unlike many of them, he had no trust fund. “I grew up privileged,” he said. “But my parents made it clear: ‘You’re on your own.’ ” Around the time Trump made his offer, Schwartz’s wife, Deborah Pines, became pregnant with their second daughter, and he worried that the family wouldn’t fit into their Manhattan apartment, whose mortgage was already too high. “I was overly worried about money,” Schwartz said. “I thought money would keep me safe and secure—or that was my rationalization.” At the same time, he knew that if he took Trump’s money and adopted Trump’s voice his journalism career would be badly damaged. His heroes were such literary nonfiction writers as Tom Wolfe, John McPhee, and David Halberstam. Being a ghostwriter was hackwork. In the end, though, Schwartz had his price. He told Trump that if he would give him half the advance and half the book’s royalties he’d take the job.

Such terms are unusually generous for a ghostwriter. Trump, despite having a reputation as a tough negotiator, agreed on the spot. “It was a huge windfall,” Schwartz recalls. “But I knew I was selling out. Literally, the term was invented to describe what I did.” Soon Spy was calling him “former journalist Tony Schwartz.”

Schwartz thought that “The Art of the Deal” would be an easy project. The book’s structure would be simple: he’d chronicle half a dozen or so of Trump’s biggest real-estate deals, dispense some bromides about how to succeed in business, and fill in Trump’s life story. For research, he planned to interview Trump on a series of Saturday mornings. The first session didn’t go as planned, however. After Trump gave him a tour of his marble-and-gilt apartment atop Trump Tower—which, to Schwartz, looked unlived-in, like the lobby of a hotel—they began to talk. But the discussion was soon hobbled by what Schwartz regards as one of Trump’s most essential characteristics: “He has no attention span.”

In those days, Schwartz recalls, Trump was generally affable with reporters, offering short, amusingly immodest quotes on demand. Trump had been forthcoming with him during the New York interview, but it hadn’t required much time or deep reflection. For the book, though, Trump needed to provide him with sustained, thoughtful recollections. He asked Trump to describe his childhood in detail. After sitting for only a few minutes in his suit and tie, Trump became impatient and irritable. He looked fidgety, Schwartz recalls, “like a kindergartner who can’t sit still in a classroom.” Even when Schwartz pressed him, Trump seemed to remember almost nothing of his youth, and made it clear that he was bored. Far more quickly than Schwartz had expected, Trump ended the meeting.

Week after week, the pattern repeated itself. Schwartz tried to limit the sessions to smaller increments of time, but Trump’s contributions remained oddly truncated and superficial.

“Trump has been written about a thousand ways from Sunday, but this fundamental aspect of who he is doesn’t seem to be fully understood,” Schwartz told me. “It’s implicit in a lot of what people write, but it’s never explicit—or, at least, I haven’t seen it. And that is that it’s impossible to keep him focussed on any topic, other than his own self-aggrandizement, for more than a few minutes, and even then . . . ” Schwartz trailed off, shaking his head in amazement. He regards Trump’s inability to concentrate as alarming in a Presidential candidate. “If he had to be briefed on a crisis in the Situation Room, it’s impossible to imagine him paying attention over a long period of time,” he said.

In a recent phone interview, Trump told me that, to the contrary, he has the skill that matters most in a crisis: the ability to forge compromises. The reason he touted “The Art of the Deal” in his announcement, he explained, was that he believes that recent Presidents have lacked his toughness and finesse: “Look at the trade deficit with China. Look at the Iran deal. I’ve made a fortune by making deals. I do that. I do that well. That’s what I do.”

But Schwartz believes that Trump’s short attention span has left him with “a stunning level of superficial knowledge and plain ignorance.” He said, “That’s why he so prefers TV as his first news source—information comes in easily digestible sound bites.” He added, “I seriously doubt that Trump has ever read a book straight through in his adult life.” During the eighteen months that he observed Trump, Schwartz said, he never saw a book on Trump’s desk, or elsewhere in his office, or in his apartment.

Other journalists have noticed Trump’s apparent lack of interest in reading. In May, Megyn Kelly, of Fox News, asked him to name his favorite book, other than the Bible or “The Art of the Deal.” Trump picked the 1929 novel “All Quiet on the Western Front.” Evidently suspecting that many years had elapsed since he’d read it, Kelly asked Trump to talk about the most recent book he’d read. “I read passages, I read areas, I’ll read chapters—I don’t have the time,” Trump said. As The New Republic noted recently, this attitude is not shared by most U.S. Presidents, including Barack Obama, a habitual consumer of current books, and George W. Bush, who reportedly engaged in a fiercely competitive book-reading contest with his political adviser Karl Rove.

Trump’s first wife, Ivana, famously claimed that Trump kept a copy of Adolf Hitler’s collected speeches, “My New Order,” in a cabinet beside his bed. In 1990, Trump’s friend Marty Davis, who was then an executive at Paramount, added credence to this story, telling Marie Brenner, of Vanity Fair, that he had given Trump the book. “I thought he would find it interesting,” Davis told her. When Brenner asked Trump about it, however, he mistakenly identified the volume as a different work by Hitler: “Mein Kampf.” Apparently, he had not so much as read the title. “If I had these speeches, and I am not saying that I do, I would never read them,” Trump told Brenner.

owing desperate, Schwartz devised a strategy for trapping Trump into giving more material. He made plans to spend the weekend with Trump at Mar-a-Lago, his mansion in Palm Beach, where there would be fewer distractions. As they chatted in the garden, Ivana icily walked by, clearly annoyed that Schwartz was competing for her husband’s limited free time. Trump again grew impatient. Long before lunch on Saturday, Schwartz recalls, Trump “essentially threw a fit.” He stood up and announced that he couldn’t stand any more questions.

Schwartz went to his room, called his literary agent, Kathy Robbins, and told her that he couldn’t do the book. (Robbins confirms this.) As Schwartz headed back to New York, though, he came up with another plan. He would propose eavesdropping on Trump’s life by following him around on the job and, more important, by listening in on his office phone calls. That way, extracting extended reflections from Trump would not be required. When Schwartz presented the idea to Trump, he loved it. Almost every day from then on, Schwartz sat about eight feet away from him in the Trump Tower office, listening on an extension of Trump’s phone line. Schwartz says that none of the bankers, lawyers, brokers, and reporters who called Trump realized that they were being monitored. The calls usually didn’t last long, and Trump’s assistant facilitated the conversation-hopping. While he was talking with someone, she often came in with a Post-it note informing him of the next caller on hold.

“He was playing people,” Schwartz recalls. On the phone with business associates, Trump would flatter, bully, and occasionally get mad, but always in a calculated way. Before the discussion ended, Trump would “share the news of his latest success,” Schwartz says. Instead of saying goodbye at the end of a call, Trump customarily signed off with “You’re the greatest!” There was not a single call that Trump deemed too private for Schwartz to hear. “He loved the attention,” Schwartz recalls. “If he could have had three hundred thousand people listening in, he would have been even happier.”

This year, Schwartz has heard some argue that there must be a more thoughtful and nuanced version of Donald Trump that he is keeping in reserve for after the campaign. “There isn’t,” Schwartz insists. “There is no private Trump.” This is not a matter of hindsight. While working on “The Art of the Deal,” Schwartz kept a journal in which he expressed his amazement at Trump’s personality, writing that Trump seemed driven entirely by a need for public attention. “All he is is ‘stomp, stomp, stomp’—recognition from outside, bigger, more, a whole series of things that go nowhere in particular,” he observed, on October 21, 1986. But, as he noted in the journal a few days later, “the book will be far more successful if Trump is a sympathetic character—even weirdly sympathetic—than if he is just hateful or, worse yet, a one-dimensional blowhard.”

vesdropping solved the interview problem, but it presented a new one. After hearing Trump’s discussions about business on the phone, Schwartz asked him brief follow-up questions. He then tried to amplify the material he got from Trump by calling others involved in the deals. But their accounts often directly conflicted with Trump’s. “Lying is second nature to him,” Schwartz said. “More than anyone else I have ever met, Trump has the ability to convince himself that whatever he is saying at any given moment is true, or sort of true, or at least ought to be true.” Often, Schwartz said, the lies that Trump told him were about money—“how much he had paid for something, or what a building he owned was worth, or how much one of his casinos was earning when it was actually on its way to bankruptcy.” Trump bragged that he paid only eight million dollars for Mar-a-Lago, but omitted that he bought a nearby strip of beach for a record sum. After gossip columns reported, erroneously, that Prince Charles was considering buying several apartments in Trump Tower, Trump implied that he had no idea where the rumor had started. (“It certainly didn’t hurt us,” he says, in “The Art of the Deal.”) Wayne Barrett, a reporter for the Village Voice, later revealed that Trump himself had planted the story with journalists. Schwartz also suspected that Trump engaged in such media tricks, and asked him about a story making the rounds—that Trump often called up news outlets using a pseudonym. Trump didn’t deny it. As Schwartz recalls, he smirked and said, “You like that, do you?”

Schwartz says of Trump, “He lied strategically. He had a complete lack of conscience about it.” Since most people are “constrained by the truth,” Trump’s indifference to it “gave him a strange advantage.”

When challenged about the facts, Schwartz says, Trump would often double down, repeat himself, and grow belligerent. This quality was recently on display after Trump posted on Twitter a derogatory image of Hillary Clinton that contained a six-pointed star lifted from a white-supremacist Web site. Campaign staffers took the image down, but two days later Trump angrily defended it, insisting that there was no anti-Semitic implication. Whenever “the thin veneer of Trump’s vanity is challenged,” Schwartz says, he overreacts—not an ideal quality in a head of state.

Cartoon
ShareTweetBuy a cartoon
When Schwartz began writing “The Art of the Deal,” he realized that he needed to put an acceptable face on Trump’s loose relationship with the truth. So he concocted an artful euphemism. Writing in Trump’s voice, he explained to the reader, “I play to people’s fantasies. . . . People want to believe that something is the biggest and the greatest and the most spectacular. I call it truthful hyperbole. It’s an innocent form of exaggeration—and it’s a very effective form of promotion.” Schwartz now disavows the passage. “Deceit,” he told me, is never “innocent.” He added, “ ‘Truthful hyperbole’ is a contradiction in terms. It’s a way of saying, ‘It’s a lie, but who cares?’ ” Trump, he said, loved the phrase.

In his journal, Schwartz describes the process of trying to make Trump’s voice palatable in the book. It was kind of “a trick,” he writes, to mimic Trump’s blunt, staccato, no-apologies delivery while making him seem almost boyishly appealing. One strategy was to make it appear that Trump was just having fun at the office. “I try not to take any of what’s happened too seriously,” Trump says in the book. “The real excitement is playing the game.”

In his journal, Schwartz wrote, “Trump stands for many of the things I abhor: his willingness to run over people, the gaudy, tacky, gigantic obsessions, the absolute lack of interest in anything beyond power and money.” Looking back at the text now, Schwartz says, “I created a character far more winning than Trump actually is.” The first line of the book is an example. “I don’t do it for the money,” Trump declares. “I’ve got enough, much more than I’ll ever need. I do it to do it. Deals are my art form. Other people paint beautifully on canvas or write wonderful poetry. I like making deals, preferably big deals. That’s how I get my kicks.” Schwartz now laughs at this depiction of Trump as a devoted artisan. “Of course he’s in it for the money,” he said. “One of the most deep and basic needs he has is to prove that ‘I’m richer than you.’ ” As for the idea that making deals is a form of poetry, Schwartz says, “He was incapable of saying something like that—it wouldn’t even be in his vocabulary.” He saw Trump as driven not by a pure love of dealmaking but by an insatiable hunger for “money, praise, and celebrity.” Often, after spending the day with Trump, and watching him pile one hugely expensive project atop the next, like a circus performer spinning plates, Schwartz would go home and tell his wife, “He’s a living black hole!”

Schwartz reminded himself that he was being paid to tell Trump’s story, not his own, but the more he worked on the project the more disturbing he found it. In his journal, he describes the hours he spent with Trump as “draining” and “deadening.” Schwartz told me that Trump’s need for attention is “completely compulsive,” and that his bid for the Presidency is part of a continuum. “He’s managed to keep increasing the dose for forty years,” Schwartz said. After he’d spent decades as a tabloid titan, “the only thing left was running for President. If he could run for emperor of the world, he would.”

Rhetorically, Schwartz’s aim in “The Art of the Deal” was to present Trump as the hero of every chapter, but, after looking into some of his supposedly brilliant deals, Schwartz concluded that there were cases in which there was no way to make Trump look good. So he sidestepped unflattering incidents and details. “I didn’t consider it my job to investigate,” he says.

Schwartz also tried to avoid the strong whiff of cronyism that hovered over some deals. In his 1986 journal, he describes what a challenge it was to “put his best foot forward” in writing about one of Trump’s first triumphs: his development, starting in 1975, of the Grand Hyatt Hotel, on the site of the former Commodore Hotel, next to Grand Central Terminal. In order to afford the hotel, Trump required an extremely large tax abatement. Richard Ravitch, who was then in charge of the agency that had the authority to grant such tax breaks to developers, recalls that he declined to grant the abatement, and Trump got “so unpleasant I had to tell him to get out.” Trump got it anyway, largely because key city officials had received years of donations from his father, Fred Trump, who was a major real-estate developer in Queens. Wayne Barrett, whose reporting for the Voice informed his definitive 1991 book, “Trump: The Deals and the Downfall,” says, “It was all Fred’s political connections that created the abatement.” In addition, Trump snookered rivals into believing that he had an exclusive option from the city on the project, when he didn’t. Trump also deceived his partner in the deal, Jay Pritzker, the head of the Hyatt Hotel chain. Pritzker had rejected an unfavorable term proposed by Trump, but at the closing Trump forced it through, knowing that Pritzker was on a mountain in Nepal and could not be reached. Schwartz wrote in his journal that “almost everything” about the hotel deal had “an immoral cast.” But as the ghostwriter he was “trying hard to find my way around” behavior that he considered “if not reprehensible, at least morally questionable.”

Many tall tales that Trump told Schwartz contained a kernel of truth but made him out to be cleverer than he was. One of Trump’s favorite stories was about how he had tricked the company that owned Holiday Inn into becoming his partner in an Atlantic City casino. Trump claimed that he had quieted executives’ fears of construction delays by ordering his construction supervisor to make a vacant lot that he owned look like “the most active construction site in the history of the world.” As Trump tells it in “The Art of the Deal,” there were so many dump trucks and bulldozers pushing around dirt and filling holes that had just been dug that when Holiday Inn executives visited the site it “looked as if we were in the midst of building the Grand Coulee Dam.” The stunt, Trump claimed, pushed the deal through. After the book came out, though, a consultant for Trump’s casinos, Al Glasgow, who is now deceased, told Schwartz, “It never happened.” There may have been one or two trucks, but not the fleet that made it a great story.

Schwartz tamped down some of Trump’s swagger, but plenty of it remained. The manuscript that Random House published was, depending on your perspective, either entertainingly insightful or shamelessly self-aggrandizing. To borrow a title from Norman Mailer, who frequently attended prizefights at Trump’s Atlantic City hotels, the book could have been called “Advertisements for Myself.”

In 2005, Timothy L. O’Brien, an award-winning journalist who is currently the executive editor of Bloomberg View, published “Trump Nation,” a meticulous investigative biography. (Trump unsuccessfully sued him for libel.) O’Brien has taken a close look at “The Art of the Deal,” and he told me that it might be best characterized as a “nonfiction work of fiction.” Trump’s life story, as told by Schwartz, honestly chronicled a few setbacks, such as Trump’s disastrous 1983 purchase of the New Jersey Generals, a football team in the flailing United States Football League. But O’Brien believes that Trump used the book to turn almost every step of his life, both personal and professional, into a “glittering fable.”

Some of the falsehoods in “The Art of the Deal” are minor. Spy upended Trump’s claims that Ivana had been a “top model” and an alternate on the Czech Olympic ski team. Barrett notes that in “The Art of the Deal” Trump describes his father as having been born in New Jersey to Swedish parents; in fact, he was born in the Bronx to German parents. (Decades later, Trump spread falsehoods about Obama’s origins, claiming it was possible that the President was born in Africa.)

In “The Art of the Deal,” Trump portrays himself as a warm family man with endless admirers. He praises Ivana’s taste and business skill—“I said you can’t bet against Ivana, and she proved me right.” But Schwartz noticed little warmth or communication between Trump and Ivana, and he later learned that while “The Art of the Deal” was being written Trump began an affair with Marla Maples, who became his second wife. (He divorced Ivana in 1992.) As far as Schwartz could tell, Trump spent very little time with his family and had no close friends. In “The Art of the Deal,” Trump describes Roy Cohn, his personal lawyer, in the warmest terms, calling him “the sort of guy who’d be there at your hospital bed . . . literally standing by you to the death.” Cohn, who in the fifties assisted Senator Joseph McCarthy in his vicious crusade against Communism, was closeted. He felt abandoned by Trump when he became fatally ill from aids, and said, “Donald pisses ice water.” Schwartz says of Trump, “He’d like people when they were helpful, and turn on them when they weren’t. It wasn’t personal. He’s a transactional man—it was all about what you could do for him.”

According to Barrett, among the most misleading aspects of “The Art of the Deal” was the idea that Trump made it largely on his own, with only minimal help from his father, Fred. Barrett, in his book, notes that Trump once declared, “The working man likes me because he knows I didn’t inherit what I’ve built,” and that in “The Art of the Deal” he derides wealthy heirs as members of “the Lucky Sperm Club.”

Trump’s self-portrayal as a Horatio Alger figure has buttressed his populist appeal in 2016. But his origins were hardly humble. Fred’s fortune, based on his ownership of middle-income properties, wasn’t glamorous, but it was sizable: in 2003, a few years after Fred died, Trump and his siblings reportedly sold some of their father’s real-estate holdings for half a billion dollars. In “The Art of the Deal,” Trump cites his father as “the most important influence on me,” but in his telling his father’s main legacy was teaching him the importance of “toughness.” Beyond that, Schwartz says, Trump “barely talked about his father—he didn’t want his success to be seen as having anything to do with him.” But when Barrett investigated he found that Trump’s father was instrumental in his son’s rise, financially and politically. In the book, Trump says that “my energy and my enthusiasm” explain how, as a twenty-nine-year-old with few accomplishments, he acquired the Grand Hyatt Hotel. Barrett reports, however, that Trump’s father had to co-sign the many contracts that the deal required. He also lent Trump seven and a half million dollars to get started as a casino owner in Atlantic City; at one point, when Trump couldn’t meet payments on other loans, his father tried to tide him over by sending a lawyer to buy some three million dollars’ worth of gambling chips. Barrett told me, “Donald did make some smart moves himself, particularly in assembling the site for the Trump Tower. That was a stroke of genius.” Nonetheless, he said, “The notion that he’s a self-made man is a joke. But I guess they couldn’t call the book ‘The Art of My Father’s Deals.’ ”

The other key myth perpetuated by “The Art of the Deal” was that Trump’s intuitions about business were almost flawless. “The book helped fuel the notion that he couldn’t fail,” Barrett said. But, unbeknown to Schwartz and the public, by late 1987, when the book came out, Trump was heading toward what Barrett calls “simultaneous personal and professional self-destruction.” O’Brien agrees that during the next several years Trump’s life unravelled. The divorce from Ivana reportedly cost him twenty-five million dollars. Meanwhile, he was in the midst of what O’Brien calls “a crazy shopping spree that resulted in unmanageable debt.” He was buying the Plaza Hotel and also planning to erect “the tallest building in the world,” on the former rail yards that he had bought on the West Side. In 1987, the city denied him permission to construct such a tall skyscraper, but in “The Art of the Deal” he brushed off this failure with a one-liner: “I can afford to wait.” O’Brien says, “The reality is that he couldn’t afford to wait. He was telling the media that the carrying costs were three million dollars, when in fact they were more like twenty million.” Trump was also building a third casino in Atlantic City, the Taj, which he promised would be “the biggest casino in history.” He bought the Eastern Air Lines shuttle that operated out of New York, Boston, and Washington, rechristening it the Trump Shuttle, and acquired a giant yacht, the Trump Princess. “He was on a total run of complete and utter self-absorption,” Barrett says, adding, “It’s kind of like now.”

Schwartz said that when he was writing the book “the greatest percentage of Trump’s assets was in casinos, and he made it sound like each casino was more successful than the last. But every one of them was failing.” He went on, “I think he was just spinning. I don’t think he could have believed it at the time. He was losing millions of dollars a day. He had to have been terrified.”

In 1992, the journalist David Cay Johnston published a book about casinos, “Temples of Chance,” and cited a net-worth statement from 1990 that assessed Trump’s personal wealth. It showed that Trump owed nearly three hundred million dollars more to his creditors than his assets were worth. The next year, his company was forced into bankruptcy—the first of six such instances. The Trump meteor had crashed.

But in “The Art of the Deal,” O’Brien told me, “Trump shrewdly and unabashedly promoted an image of himself as a dealmaker nonpareil who could always get the best out of every situation—and who can now deliver America from its malaise.” This idealized version was presented to an exponentially larger audience, O’Brien noted, when Mark Burnett, the reality-television producer, read “The Art of the Deal” and decided to base a new show on it, “The Apprentice,” with Trump as the star. The first season of the show, which premièred in 2004, opens with Trump in the back of a limousine, boasting, “I’ve mastered the art of the deal, and I’ve turned the name Trump into the highest-quality brand.” An image of the book’s cover flashes onscreen as Trump explains that, as the “master,” he is now seeking an apprentice. O’Brien said, “ ‘The Apprentice’ is mythmaking on steroids. There’s a straight line from the book to the show to the 2016 campaign.”

It took Schwartz a little more than a year to write “The Art of the Deal.” In the spring of 1987, he sent the manuscript to Trump, who returned it to him shortly afterward. There were a few red marks made with a fat-tipped Magic Marker, most of which deleted criticisms that Trump had made of powerful individuals he no longer wanted to offend, such as Lee Iacocca. Otherwise, Schwartz says, Trump changed almost nothing.

In my phone interview with Trump, he initially said of Schwartz, “Tony was very good. He was the co-author.” But he dismissed Schwartz’s account of the writing process. “He didn’t write the book,” Trump told me. “I wrote the book. I wrote the book. It was my book. And it was a No. 1 best-seller, and one of the best-selling business books of all time. Some say it was the best-selling business book ever.” (It is not.) Howard Kaminsky, the former Random House head, laughed and said, “Trump didn’t write a postcard for us!”

Trump was far more involved in the book’s promotion. He wooed booksellers and made one television appearance after another. He publicly promised to donate his cut of the book’s royalties to charity. He even made a surprise trip to New Hampshire, where he stirred additional publicity by floating the possibility that he might run for President.

In December of 1987, a month after the book was published, Trump hosted an extravagant book party in the pink marble atrium of Trump Tower. Klieg lights lit a red carpet outside the building. Inside, nearly a thousand guests, in black tie, were served champagne and fed slices of a giant cake replica of Trump Tower, which was wheeled in by a parade of women waving red sparklers. The boxing promoter Don King greeted the crowd in a floor-length mink coat, and the comedian Jackie Mason introduced Donald and Ivana with the words “Here comes the king and queen!” Trump toasted Schwartz, saying teasingly that he had at least tried to teach him how to make money.

Schwartz got more of an education the next day, when he and Trump spoke on the phone. After chatting briefly about the party, Trump informed Schwartz that, as his ghostwriter, he owed him for half the event’s cost, which was in the six figures. Schwartz was dumbfounded. “He wanted me to split the cost of entertaining his list of nine hundred second-rate celebrities?” Schwartz had, in fact, learned a few things from watching Trump. He drastically negotiated down the amount that he agreed to pay, to a few thousand dollars, and then wrote Trump a letter promising to write a check not to Trump but to a charity of Schwartz’s choosing. It was a page out of Trump’s playbook. In the past seven years, Trump has promised to give millions of dollars to charity, but reporters for the Washington Post found that they could document only ten thousand dollars in donations—and they uncovered no direct evidence that Trump made charitable contributions from money earned by “The Art of the Deal.”

Not long after the discussion of the party bills, Trump approached Schwartz about writing a sequel, for which Trump had been offered a seven-figure advance. This time, however, he offered Schwartz only a third of the profits. He pointed out that, because the advance was much bigger, the payout would be, too. But Schwartz said no. Feeling deeply alienated, he instead wrote a book called “What Really Matters,” about the search for meaning in life. After working with Trump, Schwartz writes, he felt a “gnawing emptiness” and became a “seeker,” longing to “be connected to something timeless and essential, more real.”

Schwartz told me that he has decided to pledge all royalties from sales of “The Art of the Deal” in 2016 to pointedly chosen charities: the National Immigration Law Center, Human Rights Watch, the Center for the Victims of Torture, the National Immigration Forum, and the Tahirih Justice Center. He doesn’t feel that the gesture absolves him. “I’ll carry this until the end of my life,” he said. “There’s no righting it. But I like the idea that, the more copies that ‘The Art of the Deal’ sells, the more money I can donate to the people whose rights Trump seeks to abridge.”

Schwartz expected Trump to attack him for speaking out, and he was correct. Informed that Schwartz had made critical remarks about him, and wouldn’t be voting for him, Trump said, “He’s probably just doing it for the publicity.” He also said, “Wow. That’s great disloyalty, because I made Tony rich. He owes a lot to me. I helped him when he didn’t have two cents in his pocket. It’s great disloyalty. I guess he thinks it’s good for him—but he’ll find out it’s not good for him.”

Minutes after Trump got off the phone with me, Schwartz’s cell phone rang. “I hear you’re not voting for me,” Trump said. “I just talked to The New Yorker—which, by the way, is a failing magazine that no one reads—and I heard you were critical of me.”

“You’re running for President,” Schwartz said. “I disagree with a lot of what you’re saying.”

“That’s your right, but then you should have just remained silent. I just want to tell you that I think you’re very disloyal. Without me, you wouldn’t be where you are now. I had a lot of choice of who to have write the book, and I chose you, and I was very generous with you. I know that you gave a lot of speeches and lectures using ‘The Art of the Deal.’ I could have sued you, but I didn’t.”

“My business has nothing to do with ‘The Art of the Deal.’ ”

“That’s not what I’ve been told.”

“You’re running for President of the United States. The stakes here are high.”

“Yeah, they are,” he said. “Have a nice life.” Trump hung up.

Schwartz can understand why Trump feels stung, but he felt that he had to speak up before it was too late. As for Trump’s anger toward him, he said, “I don’t take it personally, because the truth is he didn’t mean it personally. People are dispensable and disposable in Trump’s world.” If Trump is elected President, he warned, “the millions of people who voted for him and believe that he represents their interests will learn what anyone who deals closely with him already knows—that he couldn’t care less about them.”