- USのみならず、海外からのユーザの増加が著しい
- 世界中のインターネットのユーザの24%を既にカバー特に欧州での成長は早く、ヨーロッパでのデータセンターの建築が急がれる模様。
- 2008年8月の時点で、保有サーバ数は10,000台を超えている。内訳は明確ではないが、Rackable社からは$68Mの買い物をしたり、NetAppやForce10社のネットワーク機器の購入をしている。
Facebook's growing infrastructure spend
On Thursday BusinessWeek reported Facebook is seeking new financing for its data center operation growth in 2009. Facebook continues to add new members and their associated content at an extremely fast pace, with most new growth coming from international markets. Facebook needs to expand its abilities to serve these markets by bolstering current infrastructure offerings and cutting latency to its members through new international points of presence. In this post I will take a deeper look at Facebook's current computing infrastructure and related expenses and examine likely new areas of investment in 2009.
Facebook members
Facebook currently has over 160 million members in its top 30 markets. Facebook enjoys a 24% market penetration across all 30 countries, including complete domination in Chile and Turkey, where 76% and 66% of all Internet users are members of Facebook. Facebook member numbers are taken from Facebook.com; total Internet users for each country is as reported by the CIA World Factbook.
| Loading... | |||
| Chile | 4,211,700 | 5,570,000 | 75.61% |
| Turkey | 8,720,680 | 13,150,000 | 66.32% |
| Denmark | 1,998,600 | 3,500,000 | 57.1% |
| Greece | 1,287,620 | 2,540,000 | 50.69% |
| Australia | 5,245,040 | 11,240,000 | 46.66% |
| Norway | 1,729,640 | 3,800,000 | 45.52% |
| Venezuela | 2,533,400 | 5,720,000 | 44.29% |
| United Kingdom | 17,308,040 | 40,200,000 | 43.05% |
| Hong Kong | 1,584,600 | 3,961,000 | 40.01% |
| Canada | 11,117,600 | 28,000,000 | 39.71% |
| Belgium | 2,041,160 | 5,220,000 | 39.1% |
| Argentina | 3,388,020 | 9,309,000 | 36.4% |
| Colombia | 4,404,900 | 12,100,000 | 36.4% |
| Singapore | 1,001,800 | 3,105,000 | 32.26% |
| Switzerland | 1,344,280 | 4,610,000 | 29.16% |
| France | 8,888,140 | 31,295,000 | 28.4% |
| Sweden | 1,937,600 | 7,000,000 | 27.68% |
| South Africa | 1,390,440 | 5,100,000 | 27.26% |
| Italy | 8,469,220 | 32,000,000 | 26.47% |
| United States | 54,739,960 | 223,000,000 | 24.55% |
| Spain | 4,125,420 | 19,690,000 | 20.95% |
| Philippines | 1,011,360 | 5,300,000 | 19.08% |
| Indonesia | 2,311,340 | 13,000,000 | 17.78% |
| Egypt | 1,182,360 | 8,620,000 | 13.72% |
| Mexico | 2,006,840 | 22,812,000 | 8.8% |
| Malaysia | 1,143,360 | 15,868,000 | 7.21% |
| Germany | 1,883,340 | 42,500,000 | 4.43% |
| India | 1,559,000 | 80,000,000 | 1.95% |
Facebook's current infrastructure serves North America with minimal latency. Future expansion into Europe and southeast Asia are likely as Facebook tries to expand its international audience.
Data centers
Facebook currently operates out of four data centers in the United States: three on the west coast and one on the east coast. Facebook leases at least 45,000 square feet of data center space.
Switch & Data's PAIX at 529 Bryant Street in Palo Alto is just around the corner from the Facebook offices and a long-time home to Facebook servers. It's unclear how much of the 100,000 square-foot, liquid-cooled data center is currently occupied by Facebook.
Facebook has been with Terremark's NAP West in Santa Clara since November 2005. Facebook originally leased 10,000 square feet but may have grown larger over the years. Facebook is still listed as a Terremark customer in Santa Clara but the company might be consolidating its operations into its new local data center.
Facebook geo-distributed its web operations in 2008 with DuPont Fabros' ACC4 in Ashburn, Virginia. Facebook leased 10,000 square feet in 2007 and occupied the space in 2008 after extensive reworking of the Facebook backend. Facebook shares ACC4 with MySpace, Google, and other competitors.
In January 2009 Facebook moved into its first exclusive data center, Digital Realty Trust's 1201 Comstock Street in Santa Clara. The 24,000 square feet of data center space operates at a PUE of 1.35, a respectable mark against reported 1.22 marks for Google and Microsoft. Facebook leases the center from Digital Realty as its sole occupant.
Facebook is rumored to be adding an additional 20,000 square feet of data center space in Ashburn, Virgnia in DuPont Fabros' ACC5. Facebook is expected to move into ACC5 in September 2009 and place new servers online by the end of the year.
Facebook recently announced an international headquarters in Dublin, Ireland that will include "operations support" across Europe, the Middle East, and Africa. A European data center is a likely expansion point for Facebook as they try to solidify their European offerings.
Server loans
Facebook paid for part of its infrastructure expansion through specialized debt financing from TriplePoint Capital. Facebook drew down $30 million in 2007 followed by another $60 million in 2008. BusinessWeek reports Facebook is currently trying to secure as much as $100 million in debt financing for its next round of growth.
Debt financing against physical assets such as servers and office buildings offer lower rates than a traditional venture capital round. Facebook's server expenditures have a recoverable resale value mapped over a depreciating lifespan, unlike direct and unrecoverable payments to employees and service providers. Lenders such as TriplePoint are a specialized type of real estate investor, a market with huge risk premiums in the current market. Facebook's $100 million debt financing is bigger than TriplePoint's typical investments, placing Facebook's new expansions beyond the investment strategy's scope during a time of real-estate investment turmoil. Facebook needed to look to other financing operations for bigger infrastructure loans, an expected move for the growing company.
Facebook spent $68 million on Rackable servers in 2007 and early 2008, likely as a result of their Virginia data center build-out. Facebook is also rumored to be a large consumer of premium-priced proprietary hardware NetApp storage appliances and Force10 networking.
Facebook's debt financing agreement with TriplePoint Capital expired a few months ago, leading the company to seek new sources of financing for its new Santa Clara data center and other expansion plans. Facebook is in discussions with Bank of America for additional loans against this capital expenditure according to BusinessWeek.
How many servers?
Facebook had over 10,000 servers as of August 2008 according to Wall Street Journal coverage of a presentation by Jonathan Heiliger, Facebook's VP of Technical Operations. Facebook signed an infrastructure solutions agreement with Intel in July 2008 to optimally deploy "thousands" of servers based on Intel Xeon 5400 4-core processors in the next year.
- memcached
- ~800 memcached servers supplying over 28 TB of memory.
- Hadoop
- ~600 servers with 8 CPUs and 4 TB of storage per server. That's 4800 cores and about 2 PB of raw storage!
- Storage
- Facebook adds more than 850 million photos and 7 million videos to its data store each month. That's a lot of Filers.
Facebook uses Akamai and other CDN providers to serve static content to visitors around the world. It's an expensive service offering not covered by Facebook's server debt financing.
Summary
Facebook's faces difficult infrastructure challenges as the company tries to keep up with explosive growth around the world. Current shocks in the real estate investment market have made property financing difficult for all companies, including Facebook. New infrastructure moves from Facebook coming online this year should lower total operating costs per server thanks to new efficiencies in the cost of power and a decline in leasing price per square foot as Facebook buys in bulk. I expect new deals with foreign governments such as Ireland will lead to new expansion by Facebook heavily influenced by the ex-Googlers on staff who have paved this path before.
Facebook is a privately-held company, offering limited insights into its expenses and other operations. The company seems to be repricing its server debt financing each year and has just crossed into the capital lending realm of big banks not easily able to take big risks in their property portfolios at the moment.
