Complexity revisited: learning from failures and successes

Complexity revisited: learning from failures and successes

Complexity revisited: learning from failures Frans Kaashoek Lec 26 --- Last one! 5/16/07 Credit: Jerry Saltzer 6.033 in one slide Principles:End-to-end argument, Open Design,

Client/server RPC File abstraction Virtual memory Threads Coordination Protocol layering Routing protocols Reliable packet delivery

Names Replication protocols Transactions Verify/Sign Encrypt/Decrypt ACL and capabilities Speaks for Case studies of successful systems: LISP, UNIX, X Windows, MapReduce, Ethernet, Internet, WWW, RAID, DNS, . Today: Why do systems fail anyway? Complexity in computer systems has no

hard edge Learning from failures: common problems Fighting back: avoiding the problems Admonition & 6.033 theme song Too many objectives Ease of use Availability Scalability Flexibility Mobility Security Networked Maintainability

Performance Durable . Lack systematic methods Many objectives + Few Methods + High d(technology)/dt = Very high risk of failure

[Brooks, Mythical Man Month] The tarpit Complexity: no hard edge Subjective complexity Increasing function It just gets worse, worse, and worse Learn from failure

The concept of failure is central to design process, and it is by thinking in terms of obviating failure that successful designs are achieved [Petroski] Keep digging principle Complex systems systems fail for complex reasons

Find the cause Find a second cause Keep looking Find the mind-set. [Petroski, Design Paradigms] Pharaoh Sneferus Pyramid project Try 1: Meidum (52 angle) Try 2: Dashur/Bent (52 to 43.5 angle)

Try 3: Red pyramid (right angle: 43) United Airlines/Univac Automated reservations, ticketing, flight scheduling, fuel delivery, kitchens, and general administration Started 1966, target 1968, scrapped 1970, spend $50M Second-system effect (First: SABRE) (Burroughs/TWA repeat) CONFIRM Hilton, Marriott, Budget, American Airlines Hotel reservations linked with airline and

car rental Started 1988, scrapped 1992, $125M Second system Dull tools (machine language) Bad-news diode [Communications of the ACM 1994] IBM Workplace OS for PPC Mach 3.0 + binary compatability with AIX + DOS, MacOS, OS/400 + new clock mgmt + new RPC + new I/O + new CPU Started in 1991, scrapped 1996 ($2B) 400 staff on kernel, 1500 elsewhere Sheer complexity of class structure proved

to be overwhelming Inflexibility of frozen class structure Big-endian/Little-endian not solved [Fleish HotOS 1997] Advanced Automation System US Federal Aviation Administration Replaces 1972 Air Route Traffic Control System Started 1982, scrapped 1994 ($6B) All-or-nothing Changing specifications Grandiose expectations Contract monitors viewed contractors as

adversaries Congressional meddling London Ambulance Service Ambulance dispatching Started 1991, scrapped in1992 (20 lives lost in 2 days, 2.5M) Unrealistic schedule (5 months) Overambitious objectives Unidentifiable project manager Low bidder had no experience No testing/overlap with old system Users not consulted during design [Report of the Inquiry Into The London Ambulance Service 1993 ]

More, too many to list Portland, Oregan, Water Bureau, 30M, 2002

Washington D.C., Payroll system, 34M 2002 Southwick air traffic control system $1.6B 2002 Sobeys grocery inventory, 50M, 2002 Kings County financial mgmt system, 38M, 2000) Australian submarine control system, 100M, 1999 California lottery system, 52M Hamburg police computer system, 70M, 1998 Kuala Lumpur total airport management system, $200M, 1998 UK Dept. of Employment tracking, $72M, 1994 Bank of America Masternet accounting system, $83M, 1988, FBI virtual case, 2004. FBI Sentinel case management software, 2006.

Recurring problems Excessive generality and ambition Bad ideas get included Second-system effect Mythical Man Month Wrong modularity

Bad-news diode Incommensurate scaling Fighting back: control novelty Source of excessive novelty: Second-system effect Technology is better Idea worked in isolation Marketing pressure

Some novelty is necessary; the difficult part is saying No. Dont be afraid to re-use existing components Dont reinvent the wheel Even if it takes some massaging Fighting back: adopt sweeping simplifications

Processor, Memory, Communication Dedicated servers N-level memories Best-effort network Delegate administration Fail-fast, pair-and-compare Dont overwrite Transactions Sign and encrypt

Fighting back: design for iteration, iterate the design Something simple working soon Find out what the real problems are One new problem at a time Use iteration-friendly design E.g., Failure/attack models Every successful complex system is found to have evolved from a successful simple system Fighting back: find bad ideas fast Question requirements

And ferry itself across the Atlantic [LHX light attack helicoper] Try ideas out, but dont hesitate to scrap Understand the design loop Requires strong, knowledgeable management The design loop min Initial design hours

Draft design Find flaws fast! days weeks months coding testing

deployed Fighting back: find flaws fast Plan, plan, plan (CHIPS, Intel processors) Simulate, simulate, simulate Boeing 777 and F-16 Design reviews, coding reviews, regression tests, daily/hourly builds, performance measurements Design the feedback system: Alpha and beta tests Incentives, not penalties, for reporting errors

Fighting back: conceptual integrity One mind controls the design Macintosh Visicalc spreadsheet UNIX Linux Good esthetics yields more successful systems

Parsimonious, Orthogonal, Elegant, Readable, Few top designers can be more productive than a larger group of average designers. Summary Principles that help avoiding failure

Limit novelty Adopt sweeping simplifications Get something simple working soon Iteratively add capability Give incentives for reporting errors Descope early Give control to (and keep it in) a small design team Strong outside pressures to violate these principles Need strong knowledgeable managers Admonition Make sure that none of the systems you design can be used as disaster

examples in future versions of this lecture 6.033 theme song Tis the gift to be simple, tis the gift to be free, Tis the gift to come down where we ought to be; And when we find ourselves in the place just right, Twill be in the valley of love and delight. When true simplicity is gained To bow and to bend we shant be ashamed; To turn, turn will be our delight, Till by turning, turning we come out right. [Simple Fifts, traditional Shaker hymn]

Recently Viewed Presentations

  • Robber Barons

    Robber Barons

    Robber Barons. Between 1877 and 1900 America was run by men known as robber barons . The presidents at the time were weak and pro-business, and there were few regulations on businesses. These robber barons, or captains of industry by...
  • St. Augustine By: Ross Barcusky Room: 14 Dr.

    St. Augustine By: Ross Barcusky Room: 14 Dr.

    St. Augustine By: Ross Barcusky Room: 14 Dr. Reese When? In 1665 the Spanish founded St. Augustine Who? Where? St. Augustine lies on the Northeastern coast of Florida off of the Atlantic Ocean Then What Happened?


    Pingki Pangestu Mr. Iman Soedrajat Dr. Lana Winayanti Ms. Sarwo Handayani Mr. Himawan Arief EAROPH KOREA Dr. Miseon Park Dr. Il Ho Chung Dr. Myung-jin Jun Prof Kyung Soo Lim EAROPH MALAYSIA Ar. S. Thirilogachandran Mr. Nazri Nordin EAROPH PHILIPPINES...
  • Kentucky Appalachian Regional Intermodal Airpark

    Kentucky Appalachian Regional Intermodal Airpark

    Freight Corridor Study PROJECT OVERVIEW FHWA Talking Freight Seminar December 15, 2004 Dilara Rodriguez Project Manager, CALTRANS
  • Le Canada de 1900-1914

    Le Canada de 1900-1914

    Les élections de 1911 vs. Les adversaires de la réciprocité ont vu leur parti élu au pouvoir. Qui opposait la réciprocité - les Libéraux ou les Conservateurs? Robert Borden (Con) Wilfrid Laurier (Lib) …les forces du nationalisme, et de l'impérialisme...
  • Glossary of Usage Review -

    Glossary of Usage Review -

    determine how to correct that sentence. Keep in mind the Glossary of Usage Terms . ONLY. ***it is vital that EACH group member be able to make and EXPLAIN the correction to the class*** Dice are rolled. Once the dice...
  • CS 248 Template - Stanford University

    CS 248 Template - Stanford University

    Skeleton represents the hierarchy of an assembly of parts - The arm is made of an upper arm and a lower arm - Different parts may have the same shape, i.e. are "geometric instances" 2. ... glScale etc. are just...

    Heritage Bank Accreditation Presentation * * * * * * * * * * * * * * * * * * * * Agenda Who is Heritage Bank? We are here to help Residential mortgages Product range Policy guidelines...