The Curious Case of CloudSwitch - The Troposphere

Ｃｌｏｕｄｓｗｉｔｃｈ社と呼ばれる、新しいＣｌｏｕｄ Ｃｏｍｐｕｔｉｎｇベンダーが話題を集めている。 

 

まだ製品を発表していないが、＄８Ｍ のＳｅｒｉｅｓ Ｂ投資をＣｏｍｍｏｎｗｅａｌｔｈ Ｃａｐｉｔａｌ Ｖｅｎｔｕｒｅｓより受け、Cloud Broker Serviceと呼ばれるビジネスモデルをアプライアンスの製品として開発中。 

 

Ｃｌｏｕｄ Ｂｒｏｋｅｒ Ｓｅｒｖｉｃｅというのは、複数のＰｕｂｌｉｃ Ｃｌｏｕｄと企業内のＰｒｉｖａｔｅ Ｃｌｏｕｄとの間のデータやワークロードを双方向に動かす事が出来るサービスを指しており、既にいくつかの企業においてパイロット運用を開始している、との事。 

 

セキュリティ、制御/管理、統合システムというキーワードがポイントになっているが、まだソリューションの中身は明らかになっていない。 

 

---

A few days ago I had a call with Ellen Rubin, one of the co-founders of a new cloud startup called Cloudswitch.  Cloudswitch recently closed an $8M Series B funding from Commonwealth Capital Ventures.  The interesting thing is that they are still in stealth mode and have not yet released a product.  They have created an enormous amount of buzz based on the fact that their company is still in stealth mode and have attracted so much money.  Is the cloud really this hot, or is there more to this story?  I decided to tell their story in pure David Fincher style.  I will tell this curious case of Cloud Switch story backwards.

• I am given the green light to talk about Cloudswitch, a new kind of cloud service that is described as a cloud broker service.
• After almost a year of ongoing discussions with Ellen, I finally get why they call it a switch.  They see themselves moving workloads back and forth within the enterprise, as opposed to the concept of a cloudburst which may imply a unidirectional flow.
• Cloudswitch acquires new office space in Burlington, MA.  They now have a good team of developers, management, and good funding to focus on the getting the product ready and are now spending time with early customers and partners.
• June 2009 they closed an $8M Series B led by Commonwealth Capital Ventures with existing investors Matrix and Atlas ventures also participating.
• They spend a lot of time working with enterprises customers and have successfully completed their pilot phase of development.  They are now gearing up for a beta later this year.
• The new CEO, John, caused a number of venture firms who know him to express interest in doing a preemptive Series B.  Although they were not planning to look for additional funding until 2010, they decided that this was a great opportunity.
• They build a core team and are fortunate to be able to bring in John McEleney as their CEO.  John was formerly the CEO at SolidWorks and ComputerVision.  He grew SolidWorks to over $350M in revenue and a market leader in the CAD space. He has a great track record of scaling companies.
• Ellen pings me again in February 2009 to get me up to speed on what they are doing.  I am very excited about what they are doing.
• They raised $7.4M in a Series A – first part in July 2008, second part added Atlas Venture in December 2008.
• They tried to focus on solving some of the main issues that will enable enterprises to use cloud computing: security, control and integration with the enterprise data center.  Their product will be delivered as a software appliance.
• Cloudswitch is founded by Ellen Rubin and John Considine in spring of 2008, and they incubate the company at Matrix Partners.  They do a ton of research asking what people think about their idea.
• I am contacted by Ellen Rubin, formerly head of marketing at Netezza, in May of 2008.  Ellen asks me what I think about a Cloud Broker appliance startup idea.  I am under no restriction to discuss this idea, other than my word.  I decide not to divulge anything until Ellen gives me the green light.

Legal Technology - Cloud Computing Brings New Legal Challenges

Ｃｌｏｕｄ Ｃｏｍｐｕｔｉｎｇに関わる法的な課題の整理。 

データのセキュリティに関する各種法律と、Ｃｌｏｕｄ Ｃｏｍｐｕｔｉｎｇ事業者がどのように対応すべきか、の確認が必要と 述べられている。 

---

In the early days of personal computing, users depended on "local" drives and stored their data on floppy disks kept in containers on desktops or in drawers. Applications from software manufacturers permitted users to create, manage and manipulate their business and personal information.

But in short order, software became more and more sophisticated and floppy disks were replaced by hard drives. Operating systems became faster, hard drives were developed with even more capacity and programs grew in size and scope.

Eventually the advent of networks allowed ever bigger programs to be shared among multiple users accessing ever-growing data banks. Nevertheless, networks remained largely tethered to the location of the users, who, at least theoretically, maintained both physical possession and control over the data.

The trend today is toward something different: Whereas companies may still prefer their employees to be in geographic proximity to urban centers of business and government, the cost of prime real estate, and the availability of fast online interconnectedness in many locations that would otherwise be considered remote, make cloud computing a viable and cost effective alternative. Accordingly, data and data applications that are kept in a cloud may be physically located in one or more remote servers but are nevertheless transparently available to company users.[FOOTNOTE 1]

Data kept in a cloud often is, or may be, shared among, or usable by, multiple parties. It can include information ranging from word processing documents and business presentations to employee or patient health information and tax or accounting records, to schedules, calendars and contacts. The key to cloud computing is the speed with which the data and applications can be accessed, rather than the capacity and speed of a personal computer's hard drive, as was crucially important in the past.

Even individual users are becoming more and more likely to be participants in the cloud computing phenomenon. For example, e-mail programs such as Google's Gmail, which stores users' e-mail on its own servers, is a perfect example of this growing development.

Given the explosive growth of cloud computing, it should be no surprise that it presents numerous legal issues for businesses. Two of the most significant are privacy concerns and the implications of cloud computing for pretrial discovery.

As with other forms of "outsourcing," businesses' duties to protect private or confidential data do not end with their transfer of the data to third-party vendors for storage or processing. A recent report from the World Privacy Forum, "Cloud Computing and Privacy," highlights a number of important privacy issues raised by cloud computing that corporate users of cloud computing should keep in mind.[FOOTNOTE 2]

For example, although the Gramm-Leach-Bliley Act[FOOTNOTE 3] permits financial institutions to disclose confidential consumer information to a third party such as a cloud computing service provider, the terms of any agreement between the financial institution and the provider must be carefully considered.

In addition, the Privacy Rule enacted by the U.S. Department of Health and Human Services under the Health Insurance Portability and Accountability Act[FOOTNOTE 4] requires that covered health plans, health care clearinghouses and health care providers enter into "business associate agreements" with cloud providers (and, of course, other third parties) before turning over so-called protected health information.

There may be risks associated with using cloud computing providers to store confidential corporate information such as trade secrets without appropriate and specially negotiated agreements, as well.

What undoubtedly can complicate the privacy issues in these and other situations is that the governing law might change depending on the cloud provider's physical location. Different rules can apply if storage is in a European Union country, arguably subject to the EU's Data Protection Directive,[FOOTNOTE 5] in multiple states within the United States, or in multiple locations around the world. Accordingly, it is essential that the terms of contracts for cloud computing services must be negotiated keeping in mind the type of data to be stored, the location of the servers and the particular legal obligations of the business whose data it is.

While a business might be able to make a claim against a cloud server for escape of private data, the business may not be insulated by its claim that a privacy breach was the result of the acts by the cloud server.

PRETRIAL DISCOVERY

An issue raised by cloud computing that may be even more difficult to parse than privacy concerns is the implications of cloud computing on pretrial discovery in general and on electronic discovery in particular.

Generally speaking, pretrial discovery may be had of relevant documents that are in the "possession, custody or control" of a party.[FOOTNOTE 6] That means that a party is obliged to produce documents in its control, even if those documents are not literally in the party's possession when the demand is made.[FOOTNOTE 7]

Documents are under a party's control when it has the right, authority or practical ability to obtain them from a non-party.[FOOTNOTE 8] When a corporation relies on a cloud computing provider (or multiple providers), are those documents under its control? Even if they are, how can those documents be authenticated and proven to be reliable?

In Shcherbakovskiy v. Da Capo Al Fine, Ltd.,[FOOTNOTE 9] the 2nd Circuit U.S. Court of Appeals adopted the view that a party may be required to produce documents that it has the practical ability to obtain.

The circuit stated as follows:

Turning to the legal issues first, a party is not obliged to produce, at the risk of sanctions, documents that it does not possess or cannot obtain. See FED. R. Civ. P. 34(a) ("Any may serve on any other party a request … to produce … documents … which are in the possession, custody or control of the party upon whom the request is served …" E.E.O.C. v. Carrols Corp., 215 F.R.D. 46, 52 (N.D.N.Y. 2003); see also Societe Internationale Pour Participations Industrielles Et Commerciales, S.A. v. Rogers, 357 U.S. 197, 204, 78 S.Ct. 1087, 2 L.Ed.2d 1255 (1958) (acknowledging that Rule 34 requires inquiry into whether party has control over documents), Fisher v. U.S. Fidelity & Guar. Co., 246 F.2d 344, 350 (7th Cir. 1957). We also think it fairly obvious that a party also need not seek such documents from third parties if compulsory process against the third parties is available to the party seeking the documents. However, if a party has access and the practical ability to possess documents not available to the party seeking them, production may be required. In Re NASDAQ Market-Makers Antitrust Litig., 169 F.R.D. 493, 530 (S.D.N.Y. 1996).

Shcherbakovskiy did not define what established a "practical ability" to obtain documents, but courts have determined that the legal right to obtain documents or information from another may arise by contract[FOOTNOTE 10] or as a result of an agency relationship.[FOOTNOTE 11]

The Cloud Security Alliance, a not-for-profit association of cloud computing professionals, observed in a recent report, "Security Guidance for Critical Areas of Focus in Cloud Computing,"[FOOTNOTE 12] that cloud providers "have become custodians of primary data assets for which customers have legal responsibilities to preserve and make available in legal proceedings (electronic discovery), even if the customer is not in direct possession or control."

The report pointed out that cloud computing "challenges the presumption" that corporations and other businesses actually are in control of information or data for which they remain legally responsible.

Given the general principles governing pretrial discovery, and the Shcherbakovskiy ruling, cloud users should make certain that the contracts they enter into with cloud providers clearly explain the providers' responsibilities with respect to discovery and other litigation subjects.

Moreover, companies that face the prospect or likelihood of litigation should make certain that they choose cloud providers that are able to ensure the authenticity and reliability of the data they are maintaining, including metadata. Certainly, any "litigation hold" extended by a company as a result of anticipated or pending litigation must include company resources that are stored in cloud servers.

CONCLUSION

As cloud computing becomes more understood and more widely utilized, counsel will focus on both privacy and discovery issues to a greater extent than they are doing so currently, which will lead to negotiated resolution of issues and, on occasion, litigation and court decisions.

As with many issues of technology, counsel will need to understand not just the legal precedent concerning cloud servers, but also the particular facts concerning their business' use of cloud servers, the type of data that is stored in the cloud, and the location and document retention practices of the service provider.

Shari Claire Lewis, a partner at Rivkin Radler, specializes in litigation in the areas of Internet, domain name and computer law. She can be reached at shari.lewis@rivkin.com.

:::: FOOTNOTES ::::

FN 1. For a detailed explanation of cloud computing, see, e.g., Lamia Youseff et al., "Toward a Unified Ontology of Cloud Computing," available at http://www.cs.ucsb.edu/~lyouseff/CCOntology/CloudOntology.pdf.

FN 2. The report is available at http://www.worldprivacyforum.org/cloudprivacy.html. For additional discussion of privacy issues in the cloud computing context, see, e.g., Randal C. Picker, "Competition and Privacy in Web 2.0 and the Cloud," 103 Nw. U. L. Rev. Colloquy 1 (July 2008).

FN 3. 15 U.S.C. §6802.

FN 4. See http://www.hhs.gov/ocr/privacy/hipaa/understanding/index.html.

FN 5. See "Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data and on the free movement of such data," available at http://ec.europa.eu/justice_home/fsj/privacy/docs/95-46-ce/dir1995-46_part1_en.pdf, and http://ec.europa.eu/justice_home/fsj/privacy/docs/95-46-ce/dir1995-46_part2_en.pdf.

FN 6. See Fed. R. Civ. P. 26(b) (1) & 34(a) (1).

FN 7. See Fed. R. Civ. P 34 (a)(1).

FN 8. See, e.g., Babaev v. Grossman, CV03-5076 (DLI)(WDW) 2008 U.S. Dist. LEXIS 77731 (E.D.N.Y. Sept. 8, 2008).

FN 9. 490 F.3d 130 (2d Cir. 2007).

FN 10. See, e.g., Anderson v. Cryovac Inc., 862 F.2d 910 (1st Cir. 1988) (requiring production where seller of real property had control of report prepared for purchaser and maintained in purchaser's possession by virtue of provision in sales contract requiring purchaser to make records available to seller).

FN 11. See, e.g., JPMorgan Chase Bank v. Winnick, 228 F.R.D. 505 (S.D.N.Y. 2005) (holding that administrative agent suing on behalf of holders of debt was obligated to produce documents and information in possession of holders to the same extent as if the holders had brought the suit).

FN 12. "Security Guidance for Critical Areas of Focus in Cloud Computing" (April 2009), available at http://www.cloudsecurityalliance.org/guidance/csaguide.pdf.

Asian Developers Moving to Cloud Computing

Ｅｖａｎｓ Ｄａｔａと呼ばれる調査会社が行ったアジア圏でのＣｌｏｕｄ Ｃｏｍｐｕｔｉｎｇの市場状況を報告したレポートが発行された。 

 

---

Developers in the Asia Pacific region are increasingly moving to cloud computing, according to a recent survey by Evans Data. Developers in the Asia Pacific region are increasingly moving to cloud computing, according to a recent survey by Evans Data. More than one in four developers in the Asia Pacific region are using cloud services or expect to within the next six months, Evans Data officials said. According to the market research firm's most recent Asia Pacific Development survey, 11.3 percent of respondents said they are currently using cloud services, with an additional 16.4 percent saying they plan to begin using them within six months. And over half of the respondents said they expect to be using cloud services at some time. Evans Data's biannual survey of more than 400 Asia Pacific software developers also showed that transactions were the most likely type of implementations to be planned for public clouds – those outside the firewall – while storage was the least likely. "The cloud computing value proposition has been more compelling in the APAC  [Asia Pacific] region vs. NA  [North America] or EMEA  [Europe, the Middle East and Africa]," said John Andrews,  president and CEO of Evans Data. "Their primary motivations for adoption have been ease of use/speed to market and the overall economics/cost savings which relate well to the youngest and most inexperienced developer population where  eight of 10 work in small organizations." Other highlights from the survey include: Sixteen percent of Asia Pacific developers employ Microsoft Silverlight some of the time but an additional 34 percent expect to do so. Three quarters of Asia Pacific developers are considering SOA implementations, though less than 10 percent are currently fully deployed. PayPal is the most commonly used form of online payment system being built into applications – its use is more than three times as common as Google Checkout. With its Asia Pacific developer survey, Evans Data measures major technology shifts and adoption patterns for a wide range of topics, including scripting language use; 3GL language use such as C++, C# and Java; platform targeting and migration; and other technology adoption including cloud, Web services, SOA, parallel programming and agile development trends. The survey series is now in its 11th year. The complete table of contents and sample pages of the Evans Data Asia Pacific Development Survey can be seen here.

Private Cloud is not truly Cloud

OpSource社のブログで掲載されたＰｒｉｖａｔｅ Ｃｌｏｕｄに関する記事。

Ｐｒｉｖａｔｅ ＣｌｏｕｄをＰｕｂｌｉｃ Ｃｌｏｕｄと比較したときにかなり違うアプローチと期待が必要になる、という内容

---

Private Cloud is not truly Cloud

Much has been made lately of the fact that the cloud is not enterprise-ready.  Security, performance, SLAs, support, standards and management tools are all cited as reasons the cloud isn't ready for enterprise adoption.

Many vendors are proposing Private Clouds as a solution. Private Clouds are clouds that run inside enterprise data centers, by enterprise IT, for the use of the members of the enterprise. Basically it's a way to virtualize a large swath of the IT data center. As is often the case with technology vendors, they think that the infrastructure technology, virtualization, is the end solution the user wants rather than the vehicle with which their needs are filled. While large scale adoption of Private Virtual farms will aid in the management of the data center, it will not address the value that users are getting from true Cloud computing.

To understand the true value of Cloud computing, you first need to understand how the 'Cloud Generation' uses technology and why the Cloud is so attractive to that generation as an infrastructure solution. The Cloud Generation has grown up on the web.  As a result they have come to expect three core elements to their technology experience:

1. Immediate Availability - They do a search and get going right away.
2. Ubiquitous Access - They can get to their data and apps anytime, anyplace.
3. Sharing and Collaboration - They expect to be able to collaborate and share anything they are working on.

The current Cloud addresses those needs by providing infrastructure in a way that is far different than any past solutions.

Immediate Availability = Complete Flexibility

Cloud solutions allow users to provision resources immediately. By the time you are done reading this, you could have a server running in Amazon or an application published in Google. It's that immediate. Moreover, it's completely flexible. You can turn off services as quickly as you turn them on. Finally you only pay for what you use down to the hour or gigabyte. This resonates with a group that's not use to spending up front for anything on the web.

Ubiquitous Access = APIs

A true Cloud solution not only offers infrastructure anytime, anywhere, it also provides access either through a web interface or through an API. To the Cloud Generation of programmers this means anything they can interact with on the Cloud they can program to through APIs. The idea of infrastructure being an item that can be addressed as part of the application, instead of something the application lays on top of, is a radical concept.  It has allowed not only for innovative applications, but also for true elastic computing making the Cloud environment even more flexible. Finally, it's an essential element of the communities that have become critical to the advancement of Cloud computing.

Sharing and Collaboration = Communities

Great Cloud offerings have great Communities around them. This is the aspect of Cloud computing that is so often missed – and even scoffed at – by the IT folks who think it's all about virtualization. One of the biggest gripes about Cloud computing is that support is done by the Community and not the vendor. While most will agree that far more proactive vendor support is necessary for Cloud computing, Community support is just as critical. For questions of configuration and usage tricks, the Community is a far better source of information than some call center employee with limited access. Often the Community devises more innovative solutions than the vendor ever could.

But in addition to support, the Community can create third-party add-ins that make the Cloud even more useful. As easy as it is to set up a server on Amazon's EC2, the vast majority of pre-configured Amazon Machine Images created by the community make it that much easier, shaving hours of configuration time. In conjunction with the aforementioned APIs, it also allows for a healthy development of third party add-ons that both add functionality to the Cloud vendor's solution and even act as a channel to market for the vendor.

So let's take a look at Private Clouds. They don't provide complete flexibility. You still need to buy a bunch of servers and virtualization software and data center space first and, once you've bought it all, you're paying for it whether you use it or not. Private Clouds also don't provide Ubiquitous Access and if they do have APIs they are usually extremely limited compared to true Cloud solutions. Finally, if there is any Community at all (which there usually isn't) it's restricted to the enterprise that is deploying it. That's a much less powerful Community than the group of internet users as whole.

So while Private Clouds may offer many advantages for managing your data center, they do not truly address the Cloud Generation's needs. What's really needed is a way to make the true Cloud (that is to say the public internet) safe for enterprise use by improving security, performance, SLAs, support, standards and management tools. That way the users and the enterprise both get what they want. What does it take to do that? That's a subject for the next post.

Microsoft hires Yahoo data center exec

Microsoft社のデータセンタ責任者としてＹａｈｏｏ社のデータセンタ関連の幹部、Ｋｅｖｉｎ Ｔｉｍｍｏｎｓ氏が採用された。

---

Microsoft hires Yahoo data center exec

Microsoft recently hired Kevin Timmons to lead Microsoft Global Foundation Services (GFS), the company's data center services organization. From Microsoft's data center blog:

Kevin brings a wealth of knowledge and passion in this space, most recently serving as vice president of Operations at Yahoo!, where he led the build-out of their data centers and infrastructure. Before that he was a director of Operations at GeoCities, and prior to that he served as a senior software engineer at Marconi Dynamics.

Kevin is known as a hands-on leader with a great grasp on the issues in his field and a keen interest in increasing energy efficiency. One of the key ways he has approached that challenge was by closely measuring efficiency at each data center and using PUE (Power Usage Effectiveness) as a key metric—a strategy that helped build more efficient data centers.

Timmons was hired to replace Mike Manos, who left Microsoft earlier this year to join data center real estate company Digital Realty Trust.

Vendors being pushed into cloud, kicking and screaming?

北米でもＣｌｏｕｄ Ｃｏｍｐｕｔｉｎｇの浸透によるソフトウェア事業の収益低下を懸念している。 

 

単なる新たなソフトウェ事業の展開ではなく、自社の事業モデル全体を大きく変革をさせないとこの変化に対応する事は難しい、と提言している。 

 

同じような減少がＯｐｅｎ Ｓｏｆｔｗａｒｅ事業においても起きていて、この市場の本格的な立ち上がるをある意味では妨げる要因にもなっている。 

 

過去で言えば、メインフレームからＵＮＩＸへ、さらにＰＣへプラットホーム事業がダウンサイズしていった時代と似ている。  勝ち組と負け組が明確に出た時代でもあり、これからのＣｌｏｕｄ Ｃｏｍｐｕｔｉｎｇ市場においてもそのような分かれ道が見えてくる、と想像される。 

 

 

---

Lately, if you have listened to the pronouncements of vendors large and small, they all are enthusiastically embracing cloud computing as the next wave of software and service delivery.

However, the Wall Street Journal's Ben Worthen and Justin Scheck have a different take on all this happy cloud talk. The way they see it, the recent economic slump and tighter IT budgets have pushed many vendors into the cloud world, kicking and screaming.  Oracle, HP, IBM, Microsoft, and SAP all run the risk of seeing business move into a lower-margin space, with a longer timeframe to see revenues, they write.

HP Software Chief Tom Hogan even offers an eye-opening comment, admitting to WSJ that the move from traditional to cloud software is "highly disruptive," and that "shareholders don't like it, and it's a real conflict between business strategy and fiduciary duty."

WSJ says vendors are reluctantly being forced into the cloud world, and offers this more sobering assessment for vendors looking at the cloud space:

"Fully embracing online software is risky for big technology vendors. 'My bet is that these large incumbents are going to be unable to cross this bridge,' says Bruce Cleveland, a venture capitalist at InterWest Partners who previously ran the online software business for Siebel Systems. 'They will not be able to transform their business models.' … The big software makers are just getting their feet wet in online software, analysts say. But the more online software these large companies sell the more likely they are to hurt their profit margins."

Noteworthy: Worthen and Scheck don't use the word 'cloud' anywhere in the article, prefering 'online software.' Also, they put things in perspective: IDC says online software will account for just $9.5 billion of the $284 billion software businesses this year, but is growing more than 40% a year compared with 3.4% for software overall.

There have been similar worries about the open-source model in recent years. And the same fears gripped the industry 20-odd years ago as enterprises moved to cheaper PC-based software for many things. Some companies survived and thrived through the disruption, others fell by the wayside, new ones sprung up.

Facebook: Managing Epic Growth in Real-Time

Facebook社の急成長の裏にあるエンジニアリングの運用手法について論じるＶＰ ｏｆ Ｔｅｃｈｎｉｃａｌ ＯｐｅｒａｔｉｏｎｓのＪｏｎａｔｈａｎ Ｈｅｌｉｇｅｒ氏。 

 

ＱＡ部門を廃止し、エンジニアチームに製品のリリース、さらにその後のライフサイクルの責任を持たせることにより、顧客に近い位置で開発を行う体制を構築した。

 

また、年間＄２０〜３０Ｍに及ぶデータセンタ費用が人件費に次ぐＦａｃｅｂｏｏｋ社の第２の運用コストの大きな比重を占めている、ということも表明。 

---

 

Facebook VP of Technical Operations Jonathan Heiliger speaks at the O'Reilly Velocity 2009 conference in San Jose, Calif.

The largest Internet sites manage a precarious balance between innovation and reliability. That's a particular challenge for Facebook, which is experiencing epic user growth while rolling out new features on a regular basis. How does it manage it?

"We fail all the time," said Jonathan Heiliger, the VP of Technical Operations for Facebook. "It's not without danger. Our goal is to make (failure) transparent to our users, and create an environment where it's safe for our employees to fail."

Heiliger was the keynote speaker at the O'Reilly Velocity 2009 conference Tuesday in San Jose, where he  discussed the challenges presented by Facebook's growth.  "We believe the most effective technical organizations are those that can change fast, and that takes teamwork," said Heiliger.

At Facebook, that has meant cultivating a culture of collaboration between engineering and operations,  two groups that often are in conflict. While engineers are eager to innovate, operations covets stability and uptime. "You get conflict," said Heiliger. "I think in every company there's conflicts between operations and engineering."

To bridge the gap, Facebook changed the process for introducing new features. In most organizations, the engineering team writes code, which is then tested by a quality assurance (QA) team before being deployed and becoming the responsibility of the operations team. Heiliger pursued a different approach.

"We don't actually have QA," he said. "At Facebook, every engineer is responsible for the cradle-to-grave lifecycle of their code and their application. You want to put engineering as close to the customer as possible."