Technological Opacity & Procedural Injustice

Boston College Law Review, Jun 2018

From Google’s auto-correction of spelling errors to Netflix’s movie suggestions, machine-learning systems are a part of our everyday life. Both private and state actors increasingly employ such systems to make decisions that implicate individuals’ substantive rights, such as with credit scoring, government-benefit eligibility decisions, national security screening, and criminal sentencing. In turn, the rising use of machine-learning systems has led to questioning about whether they are sufficiently accurate, fair, and transparent. This Article builds on that work, focusing on how opaque technologies can subtly erode the due process norm of participation. To illuminate this issue, this Article examines the use of predictive coding—a form of technology-assisted review in which supervised machine-learning software is taught to predict the relevance of collected documents for discovery productions. The use of predictive coding in civil discovery highlights the new challenge to the participation norm because the processes generally do not provide any explanations for the outputs, much less non-technological accounts that are tied to the underlying substantive legal issues. Thus, even if predictive coding results in reasonably complete, accurate, and cost-efficient productions, the “black-box” nature of the process may harm the legitimacy that comes from litigants understanding and being able to more fully participate in judicial processes. This harm, however, has not been addressed by the developing jurisprudence, probably because most of the early cases involved high-stakes litigation between sophisticated parties who could afford computer experts. But the participation issue—and related equality concerns—will become increasing problematic as the technology’s use expands beyond this privileged posture. In response to these issues, this Article proposes a reinvigorated Mathews framework that explicitly weighs predictive coding’s impact on the participation norm to better futureproof the doctrine.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

Technological Opacity & Procedural Injustice

Technological Opacity & Procedural Injustice Seth Katsuya Endo 0 1 2 0 This Article is brought to you for free and open access by the Law Journals at Digital Commons @ Boston College Law School. It has been accepted for inclusion in Boston College Law Review by an authorized editor of Digital Commons @ Boston College Law School. For more information , please contact 1 Part of the Civil Procedure Commons, Computer Law Commons, Evidence Commons, Internet Law Commons, and the Science and Technology Law Commons 2 NYU School of Law Follow this and additional works at: - Article 2 I. THE NORMATIVE GOALS, AND INCREASING SUBSTANTIVE RIGHT, OF CIVIL DISCOVERY .......... 827 B. Factors Contributing to the Increasing Prevalence of Predictive Coding in Civil Discovery 837 1. Accuracy............................................................................................................................... 852 2. Cost Efficiency ..................................................................................................................... 854 1. Expert Evidentiary Issues..................................................................................................... 857 2. Professional Responsibility Implications............................................................................. 858 C. Under-Examined Normative Trade-Off Between Economic Efficiency and Participation..... 862 SETH KATSUYA ENDO* Abstract: From Google’s auto-correction of spelling errors to Netflix’s movie suggestions, machine-learning systems are a part of our everyday life. Both pirvate and state actors increasingly employ such systems to make decisions that implicate individuals’ substantive rights, such as with credit scoring, govne-r ment-benefit eligibility decisions, national security screening, and criminal sentencing. In turn, the rising use of machine-learning systems has led to questioning about whether they are sufficiently accurate, fair, and transparent. This Article builds on that work, focusing on how opaque technologies can subtly erode the due process norm of participation. To illuminate this issue, this Articlex-e amines the use of predictive coding—a form of technology-assisted review in which supervised machine-learning software is taught to predict the relevance of collected documents for discovery productions. The use of predictive coding in civil discovery highlights the new challenge to the participation norm because the processes generally do not provide any explanations for the outputs, much less non-technological accounts that are tied to the underlying substantive legal issues. Thus, even if predictive coding results in reasonably complete, accurate, and cost-efficient productions, the “black-box” nature of the process may harm the legitimacy that comes from litigants understanding and being able to more fully participate in judicial processes. This harm, however, has not bede-n a dressed by the developing jurisprudence,probably because most of the early cases involved high-stakes litigation between sophisticated parties who could afford computer experts. But the participation issu—eand related equality con© 2018, Seth Katsuya Endo. All rights reserved. * Acting Assistant Professor, NYU School of Law. Many thanks to Steven Baick-eMr cKee, Jodi Balsam, Stephen Burbank, Brooke Coleman, Russell Gold,Helen Hershkoff, Roger Michalski, Melissa Mortazavi, Josh Sellers, and Tony Thompson for their input and guidance. I also benefited greatly from presenting an earlier draft to NYU School of Law’s Lawyering Scholarship Colloquium with special thanks to Angie Gius, Tal Kastner, Will Moon, Shanda SibSlecyo,tt Skinner-Thompson, and Naomi Sunshine. Audiences at the University of Kansas School of Law, Sturm College of Law of the University of Denver, and the Maurice A. Deane School of Law at Hofstra University also offered helpful comments and suggestions. Scott Reents, Lead tAtorney, Data Analytics and E-Discovery, Litigation at Cravath, Swaine & Moore LLP, and Daniel S. Stromberg, e-discovery counsel at Outten & Golden, provided valuable insight into both the theoretical and practice-oriented aspects of the issue. And I received essential research assistance from Christina Elizabeth Kata and administrative support from Adrienne Slater and Piper MeikleT. he outstanding editorial work of Daniel Cahill, Caitlin Toto, Antonio Fraone, and Michael O'Loughlin, of the Boston College Law Review is also deeply appreciated. cerns—will become increasing problematic as the technology’s use expands beyond this privileged posture. In response to these issues, this Article proposes a reinvigorated Mathews framework that explicitly weighs predictive coding’s impact on the participation norm to better futureproof the doctrine. INTRODUCTION From Google’s auto-correction of spelling errors to Netflix’s movie suggestions, machine-learning systems are a part of our everyday life. These systems typically use software to “detect patterns in data[] and then use thenucovered patterns to predict future data[] or to perform other kinds of decision making under uncertainty.”1 Both private and state actors increasinglymeploy machine-learning systems to make decisions that implicate individuals’ substantive rights, such as with credit scoring, governmen-tbenefit eligibility decisions, national security screening, probable cause determinations, and criminal sentencing.2 In turn, the rising use of machine-learning systems has led to questioning about whether they are sufficiently accurate, fair, and transparent.3 This Article builds on that work, focusing on how opaque tehcnologies can subtly erode the due process norm of participation. This Article specifically examines the use of predictive codin—ga form of technology-assisted review in which supervised machine-learning software 1 Mikella Hurley & Julius Adebayo,Credit Scoring in the Era of Big Data, 18YALE J.L. & TECH. 148, 160–61 (2016) (quoting KEVIN P. MURPHY, MACHINE LEARNING: A PROBABILISTIC PERSPECTIVE ( 2012 )). 2 See, e.g., Kiel Brennan-Marquez, “Plausible Cause”: Explanatory Standards in the Age of Powerful Machines, 70 VAND. L. REV. 1249, 1251–53 (2017); Danielle Keats Citron & Frank Pasquale, The Scored Society: Due Process for Automated Predictions, 89 WASH. L. REV. 1, 2–3 ( 2014 ); Danielle Keats Citron, Technological Due Process, 85 WASH. U. L. REV. 1249, 1284 (2008) (noting the invocation of balancing tests when substantive rights are at stake;)Nizan Geslevich Packin & Yafit Lev-Aretz, On Social Credit and the Right to Be Unnetworked, 2016 COLUM. BUS. L. REV. 339, 350; Michael L. Rich,Machine Learning, Automated Suspicion Algorithms, and the Fourth Amendmen,t 164 U. PA. L. REV. 871, 886–91 (2016); Daniel J. Solove, Data Mining and the Security-Liberty Debate, 75 U. CHI. L. REV. 343, 344–45 (2008); Katherine Freeman, Recent Development, Algorithmic Injustice: How the Wisconsin Supreme Court Failed to Protect Due Process Rights inState v. Loomis, 18N.C. J.L. & TECH. ONLINE 75, 76 (2016), []. 3 See, e.g., Julia Angwin et al., Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And It’s Biased Against Blacks., PROPUBLICA (May 23, 2016), https:// [https://perma. cc/7RKG-CSJV] (discussing lackof algorithm reliability in predicting violent cri;meM)itch Smith, A Case Is Putting the Use of Data to Predict Defendan’tsFutures on Trial, N.Y. TIMES, June 23, 2016, at A18 (discussing the implications of predictive coding inState v. Loomis). Compare Alison Gopnik, Review, The Curious Incident of the Baby in the Lab,WALL ST. J., Aug. 15, 2015, at C2 (describing a study exploring human predictive capabilitwy)it,h CATHY O’NEIL, WEAPONS OF MATH DESTRUCTION: HOW BIG DATA INCREASES INEQUALITY AND THREATENS DEMOCRACY 8 (2016) (examining the lack of explanation for the outputs of complex algorithms). is taught to predict the relevance of collected documents for discoveryop-r ductions. The use of predictive coding in civil discovery highlights the new challenge to the participation norm particularly clearly because the processes generally do not provide any explanations for the outputs, much less -non technological accounts that are tied to the underlying substantive legal issues. Furthermore, the emphasis on predictive coding in civil discovery is warranted because several factors suggest it will be used with increasing ferquency. As computers—from desktops to smartphones—become ever more omnipresent, the amount of electronically stored information (ESI) continues to rise, creating significant logistical and cost challenges for civil litigants.4 In response to these complications, the 2015 amendments to Federal Rule of Civil Procedure 26(b) integrated a requirement that discovery be “proportional to the needs of the case” directly into the definition of its scope5. Together, these developments present a risk that essential discovery in cases involving individual small-value claims against large defendants will be stifled, raising the need for a technological fix like predictive coding.6 To make this more concrete,one might see the issuespresented by the rise in ESI and the proportionality command arise in an employment discrimination suit brought by an individual against a multinational company. In such a case, the employee’s claim is worth comparatively little and the information about both the employee and his or her comparators that would potentially show discriminatory patterns may be contained in a large number of emails and human resources documents that are dispersed across a broad swathe of the company.7 The average corporate worker sends or receives more than one hundred emails per day.8 To exceed one hundred thousand documents, a case 2018] would only need to involve three supervisors emailing over a twel-vme onth period. Thus, one can imagine predictive coding being offered acsosateffective answer to the proportionality inquiry in a challenge to the scope of discovery.9 In other words, predictive coding is a technological fix that might make the low-value employment discrimination claim viable. Additionally, the use of predictive coding in legal processes is a particularly timely issue. In its various forms, predictive coding has received a great deal of attention from practitioners, the academy, and the public at large1.0 At the same time, the use of predictive coding is still in its early stages and the jurisprudence remains mutable.11 The existing academic literature and case law on the use of predictive coding in civil discovery have focused on its practical implementationd,- a dressing ( 1 ) its accuracy and economic efficiency inculling voluminous ESI for responsive materials1;2 ( 2 ) the application of Federal Rule of Evidence 702 and the U.S. Supreme Court’s 1993 holding inDaubert v. Merrell Dow Pharmaceuticals, Inc.;13 and ( 3 ) professional responsibility issues such as maintaining technological prowess, preventing unauthorized practice of law, or protecting attorne-ywork-product.14 This body of scholarship and law, however, has neglected to consider both a key attribute and a followingm-i plication of most—if not all—predictive coding processes: they are not edsigned to provide easily intelligible explanations that rely on the substantive meaning of the materials.15 This deficit can negatively impact the due process norm of participation for litigants who lack the financial resourcefsor computer experts. So far, courts have primarily addressed the use of predictive coding in discovery in cases involving sophisticated, well-resourced litigants who were able to employ experts in the technology, sidestepping the intelligibilitsy- i sue.16 But, this sort of privileged posture is not the standard in civil litigation as a whole.17 Thus, the jurisprudence does not necessarily raise all of thes-i sues—whether going to practical implementation or higher normativel-va ues—with which the judiciary must ultimately wrestle in its managerial role. Predictive coding’s lack of easy intelligibility implies thathte process will require either the expense of experts or trust in a “blac-kbox” process.18 To the former, regardless as to whether the procedural proetctions for the use of experts apply, the expense of experts could destroy the economic value of many small-value individual claims.19 To the latter, even if predictive coding more accurate and c-oesftfective productions,20 the “black-box” results in nature of the process may harm the element of legitimacy that comes from litigants understanding and being able to more fully participate in judicial 2018] processes—dimensions of efficacy wholly different than the accuracy and cost efficiency of a discovery production.21 Courts, however, have focused nearly exclusively on economic efficiency in their proportionality inquiries2.2 If this doctrine calcifies, the resulting law will disadvantage plaintiffs with sma-lvlalue claims and undermine the due process norm of participation. To rectify this and better futureproof this jurisprudence, courts should explicitly include the non-financial values of the parties’ understanding and participation in the courtsM’athews-style assessments of predictive coding’s cost and benefits in civil discovery.23 Part I of this Article briefly describes the relevant federal rules regulating civil discovery, their normative goals, and the standard doctrinal fraemwork for discovery disputes. Part II provides a brief description of how ep-r dictive coding in civil discovery actually works, the reasons it is coming to the fore, and the jurisprudential landscape. ParItII examines the dua-ledged nature of predictive coding with a focus on the normative trad-oeff between accuracy and cost efficiency on one hand and knowledgeable participation on the other. The intervention it argues for is making courts explicitly include this trade-off in the cost-benefit weighing undertaken in their management of predictive coding. It also includes a short discussion of possible non-doctrinal solutions that could reduce the problematic elements of the trad-eoff such as technological or professional development advances that increase the trsa-n parency of the predictive coding processeTs.he Article concludes with a summary of the foregoing points and the broader lesson that can be drawn. I. THE NORMATIVE GOALS, AND INCREASING SUBSTANTIVE RIGHT, OF CIVIL DISCOVERY The first step in evaluating the evolving jurisprudence of predictive coding in civil discovery is to identify the normative goals that the relevant porcedural rules are designed to achieve2.4 Describing these goals provides the measure by which a cour—tor other observer—should gauge the success of the emerging doctrine.25 Additionally, as applied in civil discovery, the underlying goals give the rules something approaching a constitutional nd-ime sion.26 And, at a high level, the existing doctrinal framework already attempts to balance a set of interests that include, at least, some of these goals.27 Rules 26 through 37 of the Federal Rules of Civil Procedure all speak to discovery and disclosure.28 Most disputes involving the use of predictive coding in civil discovery turn on the issue of scope.29 Accordingly, Rule 26(b)( 1 ) is the most relevant provision. It states: Unless otherwise limited by court order, the scope of discovery is as follows: Parties may obtain discovery regarding any nonprivileged matter that is relevant to any party’s claim or defense and proportional to the needs of the case, considering the importance of the issues at stake in the action, the amount in controversy, the parties’ relative access to relevant information, the parties’ resources, the importance of the discovery in resolving the issues, and whehter the burden or expense of the proposed discovery outweighs its likely benefit. Information within this scope of discovery need not be admissible in evidence to be discoverable.30 The text demonstrates the main objective highlighted in Rule 26 is porportionality, which is grounded in the listed constituent factors.This tends to turn into a question of economic efficiency with its subparts of accuracy and cost effectiveness.31 But the text of Rule 26 alone does not provide any interpretative guidance as to the weighting of thesefactors, much less as to any 2018] other overarching norms that may apply32. And, when considering both the structural role and institutional competence of the federal courts, it is not bovious that courts are the collective body best situated to generate the sioacl objectives to be served by the procedural rules.33 The Federal Rules of Civil Procedure, however, are not completely silent about the norms they aim to serve. Rule 1 describes the code’s broad purpose as being “to secure the just, speedy, and inexpensive determination of every action and proceeding.”34 This provides some statutory guidance as to the higher aspirations of civil procedure, which must inform the interpar-et tions of the other rules.35 As applied to discovery more specifically, Rule 1 has beennduerstood as prohibiting unfair surprises, promoting an exchange of information sui-ff cient for each side to assert their claims or defenses, and to avoid unnecessarily prolonged litigation3.6 Notwithstanding these common formulations that focus on these fairly tactical objectives3,7 the primary placement of the term “just” in Rule 1 suggests an emphasis on higherle-vel norms that extend beyond simply ensuring parties have material that might help them win their cases.38 This is in keeping with the history of the discovery rules, which were originally designed to permit liberal discovery with an eye towards providing parties with important information necessary to ultimately pursue a dispio-s tion on the merits of the ca—sethat is, to give the parties a chance bteo heard.39 Elaborating on these themes, scholars have articulated additionals-di covery-specific norms. For example, some scholars have described a tr-uth seeking function.40 Professor Martin Redish argues that the procedural rules regulating discovery should promote the following highl-evel goals: “( 1 ) decisionmaking accuracy; ( 2 ) adjudicatory efficiency; ( 3 ) political legitimacy; ( 4 ) maintenance of the substantive-procedural balance; (5) predictability; and (6) fundamental fairness.”41 This articulation by Professor Redish echoes the concerns outlined in Professor Lawrence Solum’s seminal work on procedural justice, which identified participation and accuracy as the two main rpinciples.42 According to Professor Solum, the participation principle encompasses the benefits of process that are not reducible to either accuracy or co4s3t.To ground these benefits, he posits that adjudicative processes are only lie-git mate if they afford an opportunity for participation from those who are bound to the decisions.44 Less abstractly, Professor Solum writes, “The right of participation is the right to observe, to make arguments, to present evidence, and to be informed of the reasons for a decision.”45 2018] When thinking about how the participation norm plays out in ordinary discovery, the exchange of facst is both an expression of and a prerequisite for voice and information gathering46. And typically, the sorting processes used in discovery—such as the use of keywords for screening documen—ts are causatively tied to the substantive law. This clear, intelligible link to the substance adds to the voice and informatio-gnathering aspects of participation. Additionally, it contributes to the overall transparency of the reasoning involved in the court-superintended party decisions. The accuracy principle conjectures that process choices should enhance the likelihood that the ultimate outcome of an adjudicative proceeding will be substantively correct.47 This applies to discoverybecause the informationexchange processes are meant to aid this instrumentalist purpose by giving each party—and, ultimately, the cour—tthe information necessary to judge the case.48 Current accounts of procedural justice continue to expound upon Professor Solum’s two principles of participation and accuracy while also explicitly elevating cost efficiency.49 Cost efficiency is effectively embedded in the text of Rule 26, so, unsurprisingly, it has a central place in the new jurisprudence around predictive coding in civil discovery5.0 But, the participation principle is going under-examined and under-emphasized—in contrast to accuracy and cost efficiency. No matter which articulation of the elements of procedural dueprocess one applies to discovery, all normative goals flow from a conception of oprcedural due process that is at the heart of the Federal Rules of Civil Procedure. Accordingly, to the extent that the discovery rules aid the ability of a party to present their claim in a mannerhatt is fundamentally fair, they are imbued with an almost-constitutional weight.51 46 See Hollander-Blumoff, supra note 21, at 154; Redish, supra note 21, at 600; Solum, supra note 42, at 268. 47 See Solum, supra note 42, at 306. 48 See supra notes 36–39 and accompanying text. 49 See Solum, supra note 42, at 275; see also Paul Stancil, Substantive Equality and Procedural Justice, 102 IOWA L. REV. 1633, 1649–51 (2017). Note also how these norms can be either reinforcing or in tension with each other.See, e.g., Geoffrey P. Miller, On the Costs of Civil Justice, 80 TEX. L. REV. 2115, 2118 (2002). 50 See supra notes 30–31 and accompanying text. 51 Curran, supra note 26, at 1141. In its operation, the discovery rules also might implicate individual constitutional rights. Miller, supra note 26, at 464. For example, Professor Arthur Mliler has identified how the judiciarys’ management of discovery processes can infringe on both privacy and property rights.Id. Additionally, the modern tradition of liberal, tra-nssubstantive discovery should ultimately lead to the privileging of the goals of the underlying substantive law, which might involve constitutional rights.See Curran, supra note 26, at 1141. Finally, some scholars have argued that there is a more general constitutional root of civil procedure in the due Scholars are not the only ones to identify the quasi-constitutional dimensions of civil discovery5.2 For example, the United States Court of Appeals for the Third Circuit heldthat fundamental fairness was violated when a dsitrict court’s denials of letter rogatory requests under Rule 28 meant that the plaintiff had no ability to prove her cas5e3.Writing for the court, Judge A. Leon Higginbotham explained, “Due process mandatetshat a judicial porceeding give all parties an opportunity to be heard on the critical and decisive allegations which go to the core of the parties’ claim or defense and to present evidence on the contested facts.”54 Ultimately, in assessing whether procedural rules—like those governing civil discovery—comply with the constitutional command of procedural due process, courts typically use the balancing test that was set forth 1in975 in Mathews v. Eldridg e.55 In Mathews, the U.S. Supreme Court balanced the extent to which additional or alternative processes would prevent the erroneous deprivation of the private interest at stake against the costs to thev-go ernment and adverse parties of adopting the proposed procedurea-l saf guards.56 This test has been applied to processes available to private litigants in civil litigation.57 Although a court might not explicitly identify its balancing of the intreests at play in discovery disputes as aMathews-style test, functionally, that is what happens in the predictive codnig cases.58 Further illustrating both the pervasiveness and importance of this particular balancing inquiry in questions of civil procedure, the concerns about whether additional, potentially costly discovery processes were warranted can be seen underlying the U.S. Supreme process clauses of the Fifth and Fourteenth Amendments. See, e.g., John Leubsdorf, Constitutional Civil Procedure, 63 TEX. L. REV. 579, 588 (1984). 52 See, e.g., In re Extradition of Singh, 123 F.R.D. 108, 126 (D.N.J. 1987) (holding that“although a litigant has no general constitutional right to discovery, there may be circumstances under which specific discovery must be afforded as a matter of due proces[s]”); Vaughn v. Vaughn, 56 So. 3d 1283, 1287–88 (Miss. Ct. App. 2011) (discussing the conceptual link between discovery and the non-arbitrary and non-capricious decision making of a court); Jimenez v. Brooks, No. LLICV146011314S, 2016 WL 1443594, at *4 (Conn. Super. Ct. Mar. 15, 2016c);f. Wardius v. Oregon, 412 U.S. 470, 473–74 (1973)(suggesting justice is better served by liberal discovery, reducing surprise at trial, and enhancing fairness). 53 In re Complaint of Bankers Tr. Co., 752 F.2d 874, 889 (3d Cir. 1984). 54 Id. at 890 (emphasis omitted). 55 Mathews, 424 U.S. at 334; see also In re Complaint of Bankers Trust Co., 752 F.2d at 890; Andrew Blair-Stanek, Twombly Is the Logical Extension of theMathews v. Eldridge Test to Discovery, 62 FLA. L. REV. 1, 11 (2010) (describing the significance and wide applicability of the Mathews balancing test). 56 Blair-Stanek, supra note 55, at 11. 57 See, e.g., Connecticut v. Doehr, 501 U.S. 1, 11 (1991). 58 See infra notes 166–201; see also Citron, supra note 2, at 1284. The balancing test, as applied in Mathews, included all three elements of procedural justice described above: accuracy, co-setfficiency, and participation.60 Thus, the critical problem in the existing jurisprudence of predictive coding in civil discovery is not the general framework. Instead, the issue is that courts have effectively ignored the procedural justice requirement of meaningful participation. In civil discovery, courts have primarily focused on whether the use of predictive coding is accurate and efficient. But the new wrinkle of predictive coding in civil discovery is thatthe processes frequently cannot provide xeplanations for their resulst that are tied to the underlying legal substance, if they can give an explanation at all.Explanations are a necessary component of the due process norm of participation6.1 Taken together, this leads to the conclusion that courts should explicitly assess the damage done to the participation norm in their’Mathews-style assessments of predictive coding’s cost and benefits in civil discovery. II. OVERVIEW OF PREDICTIVE CODING IN CIVIL DISCOVERY A. How Predictive Coding in Civil Discovery Actually Works A basic understanding of the mechanics of how predictive coding in civil discovery actually works is vital to evaluating the jurisprudential ramifiactions of its use.62 This section provides a short primer on this, focusing on the technologies that are used in civil litigation.63 As part of this, it discusses how predictive coding processes are both less transparent and less tied to then-u derlying substantive meaning of the documents than earlier search techniques. Although the term “predictive coding” appearfsrequently in the case law, academic literature, and vendor promotional material, its meaning must still be pinned down6.4 Myriad discovery vendors—including some prominent legal-industry players such as FTI, kCura (Relativity), Recommind, and Symantec—provide predictive coding processes.65 But the offerings vary significantly.66 For the purposes of this Article, the term “predictive coding” erfers to review processes that use supervised machin-leearning algorithms to categorize material based on experts’ coding of training sets of documents. More concretely, a typical predictive coding process would generally entail the following steps.67 First, an initial training set of documents (referred to as a “seed set”) is either randomly or deliberately selected. Next,a subjectmatter expert codes the documents for a particular attribute or set of attributes (in civil discovery, relevance is the typical category). Computer software uses the coded seed set to generate a model that is designed to predict the liik-el hood that other documents have the sought attributes based on shareda- fe 64 See Grossman-Cormack Glossary, supra note 63, at 6; see also Shannon Brown, Peeking Inside the Black Box: A Preliminary Survey of Technology Assisted Review (TAR) and Predictive Coding Algorithms for eDiscovery, 21 SUFFOLK J. TRIAL & APP. ADVOC. 221, 239 (2016).The Grossman-Cormack Glossary defines “predictive coding” as: “An industry-specific term generally used to describe a Technology-Assisted Review process involving the use of a Machine Learning Algorithm to distinguish Relevant from NonR- elevant Documents, based on Subject Matter xEpert(s)’ Coding of a Training Set of Documents.” Grossman-Cormack Glossary, supra note 63, at 26. This definition, however, has been criticized for being too specific in restricting the sorting to relevant and non-relevant. Brown, supra note 64, at 262. Conversely, others have suggested additional elements, asserting that “predictive coding” must meet all of the following:( 1 ) “Integrated, keyword-agnostic analytics to quickly generate accurate seed sets”; ( 2 ) “Language and keywordagnostic machine-learning technology to accurately find relevant documents during the‘training’ process”; ( 3 ) “A sound and well-documented workflow”; ( 4 ) “Integrated sampling to verify results to a statistical certainty before, during and after review”; and (5) “A completely integrated, purpose-built system to ensure results are consistent throughout the entire process, everytime.” Sharon D. Nelson & John W. Simek, Predictive Coding: A Rose by Any Other Name, L. PRAC., July– Aug. 2012, at 22. 65 See Peter J. Corcoran, III, Strategies to Save Resources and Reduce ED-iscovery Costs in Patent Litigation, 21 TEX. INTELL. PROP. L.J. 103, 105–06 ( 2013 ). 66Nelson & Simek, supra note 64, at 24; Nicholas Barry, Note, Man Versus Machine Review: The Showdown Between Hordes of Discovery Lawyers aandComputer-Utilizing PredictiveCoding Technology, 15 VAND. J. ENT. & TECH. L. 343, 355 ( 2013 ). 67 The description of the general process comes from discussions with e-discovery experts, vendor promotional material, and descriptions in many of the academic sources cited throughout this Article. See, e.g., Remus, supra note 10, at 1701–02; see also EDISCOVERY INST., EDISCOVERY INSTITUTE SURVEY ON PREDICTIVE CODING (2010), 2012/07/2010_EDI_PredictiveCodingSurvey.pdf []. tures. 68 The model is then applied to uncoded documents, scoring each one. Next, a subject-matter expert reviews these results and the model is revised based on the new input. A final human review or statistical validation is ef-r quently the final step before production. Functionally, all of the models’ methods are ways to find similarities that allow for categorization and scoring based on various inputs such ays- ke words or custodians.69 At this level of abstraction, they do not appear so df-i ferent than human legalreasoning, which generally takes the form of anaolgizing.70 And, implicitly assuming this similarity, the general rules of discovery have been imported to this new context7.1 In this vein, the Sedona Conference has argued that courts should not impose greater transparency and validation requirements on predictive coding discovery processes (as mc-o pared to traditional methods) because the new technologies have generally demonstrated their reliability in the end productions.72 But a key difference is that even sophisticated legal entities likely require expert assistance in inrt-e preting how the new technologies work in predictive coding.73 For the more sophisticated predictive coding products (and more genreally in the wider universe of similar machi-nlearning algorithms), the machine-generated correlations do not privilege intelligible explanations that explain the relationship to the substantive legal issues7.4 To the contrary, the tendency is for effective models to become so complex that even the original programmers may not be able to explain the mechanics that led to thet-ou put.75 And enhancing intelligibility can require a reduction in the complexity—and accuracy—of these processes7.6 But, as Professor Daniel Martin Katz puts it, the operative question has simply been, “Can your model predict better than the leading existing approach?”77 This Article posits the importance of an additional questionD:oes your model provide intelligible, non-technical explanations that are tied to the underlying legal issues? In an exhaustive manual review, at least theoretically, a requesting party could seek explanations from the producing attorneys that would presumably be justified by a causative relationship to the legal issues in the case.78 And the attorneys are able to draw on their perspectives and understanding that might extend beyond the case contours and previousley- r viewed documents.79 In this way, humans are probably better at dealing with unique or novel issues.80 Likewise, Boolean and keyword searches generally involve—and, certainly, permi—tnegotiations in which the review instcr-u tions are tied to the legal issues.81 But, once the training sets have been coded, predictive coding models lack these characteristics—that is, the ability to discuss how the mechanisms are tied to the substance of the case, particularly if the connections present a novel relationship.82 Instead, the models rely on complex mechanisms that require technical expertise to unpack.83 This expertise will likely not be available to litigants who lack siginficant resources.84 The most explanation one 2018] would expect to see from predictive coding software would be the weight table—that is, the correlative value of specific document featur—esthat the model identified and employed.85 Accordingly, even though predictive coding should not be held to a higher standard than other review processes as to output, it raises different questions about how the process itself is developed and managed.86 B. Factors Contributing to the Increasing Prevalence of Predictive Coding in Civil Discovery Since its first court approval five years ago, the use of predictive coding in civil discovery has rapidly grown.The factors contributing to this growth suggest the prevalence will continue to increase, spreading to cases involving less sophisticated parties than those who have used it to date. Thseisction first describes the riseof predictive coding in case mentions and survey ersults. It then discusses the interrelated factors leading to the increase: the growth of ESI, lawyers’ gamesmanship in civil discovery, the proportionality amendment to the Federal Rules of Civil Procedure, and technological innovation. In February 2012, Magistrate Judge Andrew Peck was the first federal judge to approve the use of predictive coding in a writtendecision, bringing attention to the practice and ushering in its use8.7 Three more orders addressing predictive coding appear in Westlaw’s 2012 case databas8e8.Six orders show up in the 2013 case database.89 Eleven orders are found in the 2014 daSee generally Richard H. Agins, An Argument for Expanding the Application of Rule 53(b) to Facilitate Reference of the Special Master in Electronic Data Discovery, 2P3ACE L. REV. 689 (2003) (suggesting special masters could better balance competing interests in electronic discovery); Marc Galanter, Why the “Haves” Come Out Ahead: Speculations on the Limits of Legal Change, 9 L. & SOC’Y REV. 95 (1974) (theorizing remedies for resource imbalance in litigation generally). 85 See Joseph H. Looby, E-discovery—Taking Predictive Coding Out of the Black Box, FTI J. (Nov. 2012), Taking%20Predictive%20Coding%20Out%20of%20the%20Black%20Box.pdf [ Z6P5-SJL7]. 86 SEDONA CONF., supra note 72, at 31–34; cf. Moore, 287 F.R.D. at 191 (discussing development and management of the keyword search process). 87 Moore, 287 F.R.D. at 192. 88 Kleen Prods. LLC v. Packaging Corp. of Am., No. 10 C 5711, 2012 WL 4498465, at *5 (N.D. Ill. Sept. 28, 2012),objections overruled by, No. 10 C 5711, 2013 WL 120240 (N.D. Ill. Jan. 9, 2013;) In re Actos (Pioglitazone) Prod.s Liab. Litig., No. 6:1-1MD-2299, 2012 WL 7861249, at *3 (W.D. La. July 27, 2012); Nat’l Day Laborer Org. Network v. U.S. Immigration & Customs Enf’t Agency, 877 F. Supp. 2d 87, 109–10 (S.D.N.Y. 2012). 89 Hinterberger v. Catholic Health Sys., Inc., No. 0-8CV-380S(F), 2013 WL 2250591, at*1 (W.D.N.Y. May 21, 2013); Gordon v. Kaleida Health, No. 08-CV-378S(F), 2013 WL 2250506, at *27 (W.D.N.Y. May 21, 2013); In re Biomet M2a Magnum Hip Implant Prods. Liab. Litig., Cause tabase.90 There was a slight dip in discovery orders in the 2015 database back to the 2013 level.91 But this might be due to a decline in the need for judicial discussion, which might follow from the growing pervasiveness of predictive coding’s use in practice.92 Surveys of legal practitioners confirm the increasing use of predictive coding in civil discovery. For example, in a 2013 survey of large American law firms, 62% reported using predictive coding and 71% increased their use in the prior year.93 Demand was even stronger with 81% of responding firms reporting client requests for the tool.94 A 2015 survey of federal agency attorneys, paralegals, records managers and IT professionals showed similar trends9.5 The survey found that 27% of 95 Ninth Annual Benchmarking Study of Electronic Discovery Practices for Government Agencies, DELOITTE ( 2015 ), []; Deloitte Survey Reveals Government Officials Confident About -EDiscovery Skills, INFO. MGMT. J., Sept. 1, 2015, 2015 WLNR 37521579. respondents used predictive coding in 2015. This was a slight increase from 23% in 2014, 17% in 2013, and 6% in 2012.96 More survey data showed 55% of respodning legal practitioners identifying predictive coding as a strategy that they will use to “manage eDiscovery volume, cost, and risk in the nextsix to twelve months.”97 Only 10% of respondents said that they did not ever use predictive coding in civil discovery.98 The majority of respondents (52%) used predictive coding in up to 20% of their cases.99 At the high end, 6% of respondents used predictive coding in over 80% of their cases.100 Although this Article focuses on domestic federal practice, predictive coding is part of a global trend. For example, in February 2016, the High Court of England and Wales approved the use of predictive coding for the first time, highlighting its potential accuracy and efficiency benefit1s0.1 Vendors are even adapting predictive coding software to deal with logographic languages.102 1. Growth of Electronically Stored Information The increased use of predictivecoding is driven, in significant part, by the growth of ESI1.03 More digital information has been created in just the past few years than existed in all human history before1.04 The growth rate is 96 Deloitte Survey Reveals Government Officials Confident About -EDiscovery Skills, supra note 95. 97 COWEN GRP., Q2, 2016 CRITICAL TRENDS SURVEY SNAPSHOT (2016), https://www.cowen []. 98 Id. 99 Id. 100 Id. 2018] Efficiency has primarily been defined by itasbility to deliver a higher number of relevant documents at lower monetary cost than other methods2.34 Some scholars have questioned whether the potential losses in comprehensiveness are problematic, even while acknowledging the necessity of addressing proportionality concerns.235 This emphasis on costs can be hard to concretely evaluate because vendors are reluctant to publicly disclose their pricing.236 But there is widespread agreement that even expensive predictive coding processes can reduces-di covery costs for cases involving large amounts of tex-tbased electronic documents because of the low incremental costs of applying the process to each additional document.237 This, however, is not undisputed,with some arguing that predictive coding is appropriate in smaller cases and some arguing that it can be fiscally inefficient even in larger cas2e3s8. Also, discovery disputes likely are more expensive when they involve experts.239 Even assuming that predictive coding provides more accurate and cmoprehensive results at alower cost, it is not settled how the gains should be 240 A court might allocate the benefit to the distributed between the parties. requesting party, increasing the amount of material that must be produced2.41 Alternatively, the court might permit the requesting party to produce only as much it would have under conventional methods and saving furtxh-er e pense.242 The ability to scale and stage review has also led to innovative cos-t shifting and cost-sharing proposals.243 The courts’ focus on cost efficiency also comes under fire as a normative matter. Professor Brooke Coleman provides a comprehensive and forceful critique of the narrow way in which efficiency has been defined in civil op-r cedure.244 Professor Coleman explains how the incomplete understanding of efficiency excludes vitally important, but difficult to measure, nonp-ecuniary costs such as the filtering of meritorious claim24s5. She also notes that the predominance of cost sensitivity has led towards no-ntrial adjudications and plaintiff skepticism by the courts.246 This voice joins the chorus of scholars and legal authorities who also have challenged the sole focus on costs, noting its problematic limits on other important norms.247 And some of this criticism has noted how proportionality—that is, the primary mechanism of the cnoventional efficiency norm—can become a deregulatory tool that especially harms plaintiffs with small-value claims.248 Finally, the empirical basis for the focus on costs also is questionable, undermining its putative basis.249 242 Id. 243 Losey, supra note 9, at 54. 244 Coleman, supra note 23, passim. 245 Id. at 1795–802; see also Érica Gorga & Michael Hablerstam, Litigation Discovery and Corporate Governance: The Missing Story About the“Genius of American Corporate Law,” 63 EMORY L.J. 1383, 1494 ( 2014 ) (describing undervalued nofnin-ancial aspects of discovery in shareholder actions). 246 Coleman, supra note 23, at 1805–20. 247 See, e.g., FED. R. CIV. P. 26(b) advisory committee’s note to 2015 amendment (“It also is important to repeat the caution that the monetary stakes are only one factor, to be balanced against other factors. The 1983 Committee Note recognized‘the significance of the substantive issues, as measured in philosophic, social, or institutional terms. Thus the rule recognizes that many cases in public policy spheres, such as employment practices, free speech, and other matters, may have importance far beyond the monetary amount involve’d.Many other substantive areas also may involve litigation that seeks relatively small amounts of money, or no moneyat all, but that seeks to vindicate vitally important personal or public value”s).; Stephen B. Burbank, Proportionality and the Social Benefits of Discovery: Out of Sight and Out of Mind ?, 34 REV. LITIG. 647, 651–53 ( 2015 ); see also Beckerman, supra note 37, at 549; Edward Brunet, The Triumph of Efficiency and Discretion Over Competing Complex Litigation Policies, 10 REV. LITIG. 273, 280 (1991); Gensler & Rosenthal, supra note 35, at 524; Eric K. Yamamoto, Efficiency’s Threat to the Value of Accessible Courts for Minorities, 25 HARV. C.R.-C.L. L. REV. 341, 393 (1990). 248 See John L. Carroll, Proportionality in Discovery: A Cautionary Tale, 32CAMPBELL L. REV. 455, 465–66 (2010); Moore, supra note 6, at 1116. 249 Moore, supra note 6, at 1113; Arthur Miller, Testimony atPublic Hearing on Proposed Amendments to the Federal Rules of Civil Procedure: Transcript of Proceeding,sJudicial Conference Advisory Committee on Civil Rules37 (Jan. 9, 2014) files/civil-rules-public-hearing-transcript-phoenix-az.pdf, [] (“I don’t think it befits the American civil justice system to have this preoccupation with cost, abuse, extortion, clichés that have been thrown out by the defense bar that sadly in my judgment have been picked up in judicial opinions without any empiric demonstration whatsoever.”). 2018] B. Expert Reliability and Professional Responsibility Implications The increased use of predictive coding also has raised new questions about the role of experts and certain professional responsibilities in civil ds-i covery. These concerns seem to arise after the courts adn parties have satsified their concerns, at least to some extent, with the accuracy and costi-eff ciency of the processes. 1. Expert Evidentiary Issues In Moore, the plaintiffs argued that the procedural protections of Federal Rule of Evidence 702250 and the Daubert framework251 applied to the expert testimony offered in support of the predictive coding processes25.2 This echoed the concerns raised by the district court Uinited States v. O’Keefe, which had reasoned that the technical issues implicated by the discovery processes were beyond a layperson’s understanding and, thus, required reassr-u ances as to their reliability.253 But the Moore district court rejected the plaintiffs’ arguments, interpreting the procedural protections as only applying to evidence offered to a jury.254 Although it is unclear whether any courts have or will heed their advice, the majority of commentators express doubt about theMoore decision on the applicability of the expert-reliability protections.255 Some scholars have challenged the statutory interpretation, noting that Federal Rule of Evidence 702 is silent as to whether it is limited to trial and questioning how much weight 250 Federal Rule of Evidence 702 permsit a witness to present expert opinion testimony if: “(a) the expert’s scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence or to determine a fact in issue(b;) the testimony is based on sufficient facts or data; (c) the testimony is the product of reliable principles and methods; and (d) the expert has reliably applied the principles and methods to the facts of the case.” FED. R. EVID. 702. 251 In Daubert v. Merrell Dow Pharmaceuticals, Inc.,the Supreme Court interpreted Federal Rule of Evidence 702 and concluded that the “triajludge must ensure that any and all scientific testimony or evidence admitted is not only relevant, but reliable”. 509 U.S. 579, 589 (1993).The Court then outlined factors that trial judgesare to consider in making this assessment.Id. at 593– 94. 252 Moore, 287 F.R.D. at 188. 253 United States v. O’Keefe, 537 F. Supp. 2d 14, 24 (D.D.C. 2008) “(Given this complexity, for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used is truly to go where angels fear to tread. This topic is clearly beyond the ken of a layman and requires that any such conclusion be based on evidence that, for example, meets the criteria of Rule 702 of the Federal Rules of Evidence”.); see also Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251, 260 n.10 (D. Md. 2008); Equity Analytics, LLC v. Lundin, 248 F.R.D. 331, 333 (D.D.C. 2008). 254 Moore, 287 F.R.D. at 188–89. 255 Gelb, supra note 13, at 1293–97; Waxse & Yoakum-Kriz, supra note 13, at 219–21. the term “help the trier of fact” can bea2r5.6 Additionally, the skeptics argue that, with the growth in ESI, the importance of discovery and its management has grown because of the potential case development and cost issue2s5.7 The protections also help educate the courts and inspire more confidence in their ability to evaluate the reliability of the testimony.258 These commentators further contend that the costs of the exper-treliability protections would not necessarily be a significant burden because the specter will encourage coopae-r tion, the utilization of the technology will offset some of the dispute costs, and judges can sequence hearings while scaling productions.259 Finally, these protections might be necessary in that some experts have suggested that some protocols would not survive aDaubert challenge, which speaks to their optential lack of reliability.260 On the other hand, Professor Dana Remus offered some thoughtfuln-i sights in support of forgoing the use ofDaubert hearings.261 She argued that vendors would end up testifying about their own products because of the lack of broader comparative data on the technolog.i2e6s2 Professor Remus also raised the concern that applying Daubert would entrench predictive coding in the realm of technology experts, not lawyers.263 At a normative level, these concerns about the reliability of expert tesi-t mony primarily seem to go to the issue of accuracy. But, regardless as to whether Daubert applies, this dialogue in the jurisprudence also, at minimum, gestures towards the importance of explanations. 2. Professional Responsibility Implications A new technology like predictive coding can introduce wrinkles to vr-i tually all of the myriad discovery practices an attorney might undertake when dealing with ESI.264 This subsection discusses the four professional responsibility implications that have already come to the fore. 2018] First, the predictive coding doctrine largely privileges cooperati2o65n. The potential tension between cooperation and zealous representation was recognized and addressed early in the case law.266 In 2012, in Kleen Products LLC v. Packaging Corp. ofAmerica, Magistrate Judge Nolan began by quoting the Sedona Conference Cooperation proclamation: Lawyers have twin duties of loyalty: While they are retained to be zealous advocates for their clients, they bear a professional obliagtion to conduct discovery in a diligent and candid manner. Their combined duty is to strive in the best interests of their clients to achieve the best results at a reasonable cost, with integrity and candor as officers of the court. Cooperation does not conflict with the advancement of their clients’ interest—sit enhances it. Only when lawyers confuse advocacy with adversarial conduct are these twin duties in conflict.267 Along these same lines, the district court inMartinelli v. Johnson & Johnson approved an order in which the prtaies agreed “that their counsel’s zealous representation of them is not compromised by conducting discovery in aocoperative manner.”268 And, in its explanation for taking the same position, the Seventh Circuit Committee’s pilot program on electronic discovery explained that a failure to cooperate could lead to increased litigation co26s9tsO.nce again though, Professor Remus offers a reminder to undertake a criticasl- a sessment, noting that the move towards cooperation is happening at the same time the goal of comprehensiveness is losing its primacy, leading to a potential break with the adversary system that is designed to protect clients and lead to just results.270 The particular form of cooperation urged by the courts271—that is, sharing the seed sets—also has raised concerns about the protection of attorne-y issues, including whether Model Rule 3.4(a), which prohibits an attorney from obstructing an opponent’s access to information, could be used to require the use of predictive coding). 265 See supra note 196 and accompanying text. 266 See Kleen Prods. LLC v. Packaging Corp. of Am., No. 10 C 5711, 2012 WL 4498465, at *1 (N.D. Ill. Sept. 28, 2012). 267 Id. (quoting The Sedona Conference Cooperation Proclamation, 10 SEDONA CONF. J. 331, 331 (2009) (emphasis omitted)); see also Morgan, supra note 12, at 71. 268 Martinelli v. Johnson & Johnson, No. 2:15-cv-01733-MCE-EFB, 2016 WL 1458109, at *1 (E.D. Cal. Apr. 13, 2016). 269 SEVENTH CIRCUIT ELECTRONIC DISCOVERY PILOT PROGRAM, INTERIM REPORT ON PHASE THREE 6 ( 2013 ), []. 270 Remus, supra note 10, at 1717–18. 271 E.g., Kleen, 2012 WL 4498465, at *5; Moore, 287 F.R.D. at 192. work-product.272 The concerns are that the seed set might divulge the mental impressions of the attorney.273 Additionally, some of the non-responsive documents might have information that reveals embarrassing oreven incriminating conduct that is unrelated to the instant case2.74 Some courts and scholars have suggested that such concerns are lessened when the seed set is randomly selected, the coding is not included, or a continuous active process is used2.75 In its application, although not entirely uniform, courts generally have not found that discovery on technologically complex discovery processes impi-l cates attorney-work-product.276 Second, the rapid technological changes associated with predictive coding also raise the issue of competence. A comment to the American Bar Association Model Rule of Professional Conduct on competence only demands that a lawyer to have an understanding of “the benefits and risks associated with relevant technology.”277 Likewise, the model rule on a lawyer’s responsibility regarding non-lawyer assistance creates a low bar, requiring only that the lawyer remain aware of how the nlaowny-er services are being rp-e formed.278 But there is a common understanding that lawyers will have to develop greater technical expertise—from better understanding the technology to enhancing their statistical knowledge—to competently serve their client.279 And, in the meantime, there might be an element of caution that has inhibited more rapid adoption of predictive coding2.80 In partial explanation, 2018] Rule 26(g) requires that lawyers certify their results, which implies that the lawyers will develop the appropriate competence before using the new technologies—that is, their use will be knowledgeable.281 In addition to flagging the changing set of competencies, some commentators have asked whether the use of predictive coding might negativelymipact traditional areas of attorney competence. For example, one large law firm partner asked, “More often than not, you’re trying to learn your case through the documents, and how will we substitute that function of learning from the documents when you’re using predictive coding?”282 Although predictive coding protocols do not eliminate human reviewers completely (if only for the coding of the training sets), the point remains that the potential efficiency gain might have an unintended downside. Third, related to the issue of technical competence, some commentators have questioned whether the use of predictive coding might veer towards the unauthorized practice of law28.3 Given its technical complexity, predictive coding generally will require the involvement of non-lawyer technicians who might have a primary role in the process.284 But the concerns about unauthorized practice of law mightbe mitigated because predictive coding primarily operates as a tool of the attorney, not the clien2t.85 And, in this way, the porcesses might resemble the outsourcing of document review, which is not unauthorized practice of law so long as licensed attorneys oversee28i6t.If an attorney lacks the technical expertise to understand the predictive coding processes, however, the attorney will not be able to adequately superintend the review.287 One additional challenge is that courts’ analyses have oftenm-e ployed analogies to human tasks, which does not always appreciate the unique elements of the technolog28y8. The final complication that arises around the potential unauthorized practice of law is when (or whether) even human document review consists of the practice of law.289 One last issu—e the inadvertent production of protected mat—erial brings together the concerns about cooperation, protection of attorne-yworkproduct, and competence. Given the volume of ESI, perfect review for maetrial protected under either the toarntey-client privilege or attorne-wyorkproduct doctrine is unrealistic. Commentators, however, differ as to the likelihoods that courts would find either that the privilege was waived by using an outside vendor or that the attorney failed to take reasonable steps to prevent disclosure by using the technology2.90 And while Federal Rule of Evidence 502 was designed to address these concerns and encourage the use of review technology, inadvertent disclosure still means that an opposing party has seen material it should not have.291 While these discussions are important to the development of the doctrine and the regulation of the profession, the larger question—and the focus of this Article—is not about lawyers’ formal compliance with their basic professional responsibility obligations in a new context but, instead, about the consideration of legal norms that should inform how all litigation player—sfrom parties to lawyers to judges—approach how opaque technologies are used in civil discovery. And, so far, the jursiprudence and scholarship has not placed an emphasis on the role that explanations play in serving the participation norm. C. Under-Examined Normative Trade-Off Between Economic Efficiency and Participation 1. Defects of the Existing Approach The significant, under-examined aspect of predictive coding in civil discovery is the trade-off between accuracy and cost-efficiency on one hand and the norm of participation on the other, particularly as it would apply to liti2018] gants without significant financial resources. As described in Part II, these are the primary elements of modern procedural justic29e2. Courts and scholars have primarily focused only on the accuracy and cost efficiency aspec29ts3. But the larger issue of the jurisprudence’s comportment withprocedural justice norms is not as significantly implicated by these two norms because both can be confirmed through statistical validation, obviating the concerns about predictive coding’s intelligibility.294 In contrast, the participation norm hinges on explanations, starkly illuminating the intelligibilitychallenge of predictive coding.295 The participation norm, however, has not been entirely overlooked. For example, Professor Remus questioned how poorer litigants would be able to challenge predictive coding processes and argued that lawyers have an ethical duty to ensure that such parties have access to the technolo2g9y6.And, at a more general level, some of the push to encourage transparency and coopeartion about search protocols can be understood as an teamtpt to mitigate the potential “black-box” quality of predictive coding for the less sophisticated litigant.297 But, other than these gestures to the issue, the trade-off has neither been prioritized nor been explicitly discussed as a normative compromise. Instead, accuracy and cost efficiency have largely been assumed to be the predominant norms at play.298 The normative trade-off warrants more discussion because, although not directly discussed, it is present—if latent—in the burgeoning doctrine. To be sure, courts frequently “muddle through” new or complicated elgal questions.299 But a failure to acknowledge underlyingsub silentio judgments results in future parties (and courts) lacking appropriately clear and coherent guidance.300 The participation norm—and its implication for the judicial management of discovery—is also reflected in the broader case law30.1 For example, in a decision that resolved a dispute over a protective order, the United States Court of Appeals for the District of Columbia stated: This public interest is reinforced by the value we place on the right of every litigant to participate in the process whereby justice is done—to understand and become involved in the proceeding, not to be compelled passively to await its outcome. Regardless of whether these considerations are deemed to be inherent in the prniciple of due process, they must be accorded considerable weight by a trial judge when considering the propriety of issuing a protective order under Fed. R. Civ. P. 26(c).302 Earlier, this Article rhetorically asked whether a discovery review porcess provides for intelligible explanations of its choices beyond a showing that the process resulted in a reasonably accurate and complete production3.03 This notion is conteste—dparticularly by practitioner—sbecause it raises pragmatic concerns about both the protection of attorn-ewy ork-product and costs.304 But recall that courts already are generally permitting discovery on discovery when the requests go to understanding the production processes3.05 And, in keeping with this judicial practice, an empirical study has shown that attorneys generally do not favor privileging only speed and ex3p06ense. Moreover, the Supreme Court has acknowledged that there are instances 2018] when the participation element of the proceduraldue process requires explanations—implicitly elevating this aspect of the participation norm to a constitutional principle.307 Furthermore, this Article’s ultimate recommendation is not that courts must privilege participation—as it flows from intelligibility—above all else. Rather, the claim is only that courts should acknowledge the new role of opaque algorithms and the stress that might place on the participation norm.308 And a closer look at the norm’s dignity, satisfaction, and legitimacy components illuminates the grounding of this call for intelligibility.309 The dignity aspect of the participation norm“is grounded in the social contract implicit in American constitutional democracy, whereby gonv-er ment agrees to treat its citizens with dignity andrespect.”310 In the context of legal process, this means litigants are entitled to processes that enable them to comprehend what is happening and, armed with this understanding, make a case that the decisionmaker treats seriousl3y11. In other words, the dignyit aspect of the participation norm has subcomponen—ts understanding, voice, and intelligibility—that extend beyond mere accuracy of outcom312e.But how can a less sophisticated litigant be treated with this sort of dignity if the party cannot afford an expertand is facing complex technology whose porcesses are not tied to the substance of the case? From this flows the premise 307 See, e.g., Wolff v. McDonnell, 418 U.S. 539, 558 (1974) (“The touchstone of due process is protection of the individual against arbitrary action of government. . . .”); Morrissey v. Brewer, 408 U.S. 471, 489 (1972)(listing due process requirements in parole revocation hearings, including the board’s reasons for revoking parole); cf. Harris v. Rivera, 454 U.S. 339, 344 (1981) (“Although there are occasions when an explanation of the reasons for a decision may be required by the demands of due process, such occasions are the exception rather than the rule.”). 308 See Coleman, supra note 23, at 1824; see also Richard Marcus, “Looking Backward” to 1938, 162 U. PA. L. REV. 1691, 1725 ( 2014 ). 309 See Solum, supra note 42, at 260. While I argue that the participation norm is the primary one, intelligibility itself has also been identified as an element of procedural justSiceee., e.g., Peter W. Billings, A Comparative Analysis of Administrative and Adjudicative Systems for Determining Asylum Claims, 52 ADMIN. L. REV. 253, 256 (2000). 310 Redish, supra note 21, at 600; see Solum, supra note 42, at 260 (noting that the participation norm “emphasizes dignity and autonomy as a function of the actual participation of litigantisn procedures that affect them”). 311 See Jeremy Waldron, U.C. Berkeley, Tanner Lectures: Dignity, Rank and Rights (Apr. 2009), [https/:/]; see also Jerry L. Mashaw, Administrative Due Process: The Quest for a Dignitary Theory, 61 B.U. L. REV. 885, 896 (1981);Michelman, supra note 42, at 543; Redish,supra note 21, at 487; Solum, supra note 42, at 273–304. 312 Allison Morse, Good Science, Bad Law: A“Multiple Balancing” Approach to Adjudication, 46 S.D. L. REV. 410, 448 (2001); Redish, supra note 21, at 567; Tom R. Tyler, Does the American Public Accept the Rule of Law? The Findings of Psychological Research on Deference to Authority, 56 DEPAUL L. REV. 661, 664, 693 ( 2007 ). that a litigant may ask another to explain an opaque technology that is being used within a judicially managed process.313 The satisfaction element of the participation norm explains the value of those dignity concerns. A number of studies confirm the relationship between a perception of procedural fairness—with the features described above—and satisfaction with the legal decision.314 This is so even when the process might be less accurate or more expensive than one that permitted less paart-icip tion.315 And the sense that justice has been done that follows from the ability to participate is a fundamental aspect of popular government.316 This Article’s claim that litigants have a legitimate interest in intelligible explanations is further buttressed by the increased managerial role of the courts in the discovery proces3s1.7 Discovery issues can effectively decide cases.318 Reasoned decision making tied to the substantive legal issues in a case is the essence of procedurally just judicial action.319 This sort of articulation reassures parties that their arguments have been heard and understood.320 And this serves both the dignity and satisfaction elements, aswell as going to a more fundamental conception of political legitimacy that posits a right to meaningfully engage with adjudicative processes that may be binding.321 2018] Solicitude for the norm of participation matters even more for litigants without financial resources.322 When discovery is stifled, parties might feel disempowered and this feeling of injustice is likely to be enhanced when a plaintiff is facing a defendant who has the more resources such that he or she is able to muster all the necessary facts danarguments without affirmative court interventions.323 This dynamic is aggravated by predictive coding because poorer litigants probably will not understand the technology and will not have the resources to hire experts, leaving them unable to effectively contest the approaches chosen by their richer adversaries3.24 In this context, this barrier to equitable, knowledgeable participation is a function of the general opacity of predictive coding processes that results from the technological compromise between interpretability and efficacy.325 But this is not just a natural state that must be accepted; rather, the judiciary has a role to play in managing the impact of the wealth disparities and their ostensibly neutral choices about procedural norms can ultimately determine who benefits.326 On the other hand, the use of predictive coding in civil discovery does not cut in just one direction—even as to the participation norm. Militating for its use, there are several ways in which predictive coding can improve the opportunities for meaningful participation by parties with fewer financialersources. For example, the iterative and conten-tremoved nature of most predictive coding processes removes the asymmetric-information issues that otherwise follow from charging the reuqesting party to generate key words32.7 Instead, the requesting party gets the benefit of scalable access to the full universe of documents. And, although the feature weights used by the algorithms are not necessarily causatively related to the underlying legal issue in the litgation, the human judgments applied to the training sets presumably are. Moreover, notwithstanding the procedural legitimacy issues discussed above, as a descriptive matter, it is certainly possible that correlations can provide insights even without being able to engage in the substanc3e2.8 Additionally, the attenuated mechanical nature potentially reduces the opportunities for intentionally intrusive requests.329 The dual-edged nature of predictive coding further supports including an assessment of its impact on the norm of partiicpation in the emerging doctrine. 2. Roadblocks to the Normative Inquiry Although none provide compelling reasons to forgo the normativne- i quiry, there are several possible explanations for why the judiciary and caademy have not yet engaged in deeper examinations of the tradoef-f. The five main reasons are outlined below. First, courts frequently have to decide the cases before them without engaging in a more philosophical inquiry about first principles.330 This is probably especially prevalent in discovery disputes because of the rise in dock-et management pressures and the degree to which judicial management of sd-i 331 Accordingly, it is unsurpriscovery is highly deferential to the trial courts. ing that few judges havewaxed poetic about the first principle issues raised, perhaps only tangentially or hypothetically, by pre-trial procedural disputes. Second, many analyses have not fully appreciated that predictive coding is a fundamentally different type of tool than eairelr ESI search methods3.32 2018] And this mischaracterization, which ignores predictive coding’s lack of easily intelligible explanations of the causal relationships between the results and the underlying legal substance, naturally leads away from a new examination of the normative trade-off.333 Third, although challenged by scholars in the context of broader discussions about discovery, economic efficiency (as understood to be a focus on reducing the measurable financial costs of litigation) is the ascendant norm.334 And the proportionality requirement embedded in Rule 26 has been unr-de stood to require this sort of economic efficiency inquiry3.35 Thus, it is no surprise that courts have looked at the accuracy and cos-tefficiency of predictive coding. Fourth, there might be a Maslow-like hierarchy of needs specific to jurisprudence in which legal analyses move from the most pragmatic concerns to the more abstract.336 In the context of predictive coding in civil discovery, the first analyses focused on whether the processes worked at their most basic level, asking whether the results were better than alternative methods at getting the right material in a cos-tefficient manner.337 Next, courts and scholars thought about the second-order implications of how the processes fit moreor less easily within the existing case law33.8 And, as this Article begins to do, the third step is examining how the processes either serve or challenge the underlying first principle norms—other than accuracy and cos-tefficiency— that animate the jurisprudence as a whole. Fifth, technology can have a glamour of objectivity and prestige.339 This is particularly true with automated processes involving machin-elearning algorithms.340 One might see the technological element itself acting as a subsittute for more substantive legal explanations because it carries with it a diffreent, but still weighty, imprimatur of authority. 3. Need to Futureproof Although predictive coding in civil discovery is not typically used in cases involving a poorer party with a sma-llvalue claim, now is the time for incorporating participation-norm concerns into the emerging doctrineT.he growth of ESI and the focus on proportionality make it likely that predictive coding will spread to new contexts involving parties with fewer financialersources.341 And the participation-norms matter even more for such parties, implicating equality issues in addition to the procedural justice elements edscribed above.342 Taken together, this raises important questions about a faliure to futureproof the jurisprudence and the risks of calcification. Futureproofing refers to the practice of developing law to remain ere-l vant despite extrinsic changes over tim3e43. With technology that changes faster than lawmakers or courts can respond, futureproofing is necessary to avoid obsolescence.344 And the benefits of such a stable doctrine are the enhancement of uniformity and certainty.345 While stability within the common-law system has its upsides, there also are risks of calcification, which are particularly salient when tfhoermative period is not reflective of the forthcoming context. Legal practices, which might not be theoretically justifiable on their merits, can harden into a lon-g standing doctrine with unintended effects.346 And these unintended effects can disadvantage vulnerable parties who were never affirmatively considered. For example, the proportionality issue primarily operates in highv-alue cases but it has, without much in the way of explicit deliberation or justification,i-m grated to lower-stakes cases.347 And, as discussed above, predictive coding exacerbates the ways in which privileging proportionality can negatively mi pact the participation norm.348 2018] The doctrinal upshot is that courts should futureproof the emerging doctrine of predictive coding in civil discovery by including an explicit and factinformed weighing of the participation norm. As with most discovery issues, a case-by-case Mathews assessment will best enhance the efficient and just workings of litigation. But such a balancing test must both includeall of the important normative considerations and account for the practical application modifications that might be necessary to account for differently situated lii-t gants. 4. Potential Non-Doctrinal Ways to Ameliorate Predictive Coding’s Impact on the Participation Norm The doctrinal fix suggested above, however, is not the only possible method for dealing with how predictive coding’s opacity might negatively impact the norm of participation for litigants without significant financial resources. There are other available tactics and some of the contextual aspects might change too. It is possible that some of the pressures leading to the increased use of predictive coding might lessen. While the growth of ESI is unlikely to slow, the Federal Rules of Civil Procedure could be amended to better reflect the normative trade-off, putting a renewed focus on the participation no3r4m9. Although the 2015 amendments went in the other direction,350 other regulatory proposals show an increased interest in ensuring that legasltructures protect people who lack significant financial resources.351 Additionally, just as predictive coding has been suggested as a tecoh-n logical fix for the problem of the deluge of ESI, more advanced predictive coding technologies might solve the intelilgibility problem.352 Some vendors have already started touting the ability of their software to explain the undre349 See Coleman, supra note 23, at 1826. 350 Id. at 1815; see also Moore, supra note 6, at 1112–14. 351 For example, in May 2016the Consumer Finance Protection Bureau proposed a prohibition on mandatory arbitration agreements. See 12 C.F.R. pt 1040 (repealed Nov. 1, 2017). 352 A general optimism about the ability of technological advances to solve problems created by earlier technological advances abounds amongst many civil procedure experts.See, e.g., Peck, supra note 4, at 3; Summary of Testimony & Comments, Advisory Committee on Civil Rules 35 ( 2013 ) (statement of Arthur Millehr)t,tp:// 2014-04.pdf [] (“The problems of e-discovery are likely to resolve themselves as information retrieval science and technology prove to reduce costs, accelerate the process, and enhance the accuracy of retrieval through a combination of statistics, linguistics, and computer science.”) And, in other contexts, sophisticated software increasingly is able to show its work. See Kevin D. Ashley, Teaching Law and Digital Age Legal Practice with an AI and Law Seminar, 88 CHI.-KENT L. REV. 783, 792–95 ( 2013 ). lying rationales.353 And lawyers and courts can use their powers over legal processes—that is, their ability to hire vendors and approve processes, respectively—to further accelerate these developments.354 Another solution would be to address how the cost of experts impedes poorer parties from being able to successfully navigate the civil litigation process.355 For example, courts may appoint experts and shift the costs of experts to the losing parties.356 Legislative fixes could expand the universe of cases in which fee-shifting for the cost of experts is permitted35.7 Another long-term strategy might be to develop a constitutional right to cou-ratppointed expert assistance under the dueprocess clause.358 This could take the form of a court-appointed special master.359 Some professional organizations encourage their members to volunteer services to people in ne—ed although it can be difficult for indigent parties to connect with these sorts porfo bono offerings.360 To the extent access to expesrthinges on funding, lawyers can davance these expenses, and the growth of litigation funding might also portend decreased barriers.361 Simpler, more elegant solutions geared specifically otwards predictive coding access might be for the bar to develop an opensource predictive coding tool, contract for group licenses, or implement a 2018] compulsory licensing scheme.362 Although increased access does not necsesarily go directly to intelligibility, it would give litigants without significant financial resources first-hand exposure to the technology. Lawyers’ gamesmanship was identified as another driver of the rise of predictive coding3.63 Accordingly, changes in professional responsibility norms or rules might reduce the impact of predictive coding on the participation norm. For example, lawyers might cooperate further to reduce discovery disputes. They also could be more transparent abouttheir predictive coding processes, using economic pressure to make their vendors explain some of the mechanisms beyond just the classification decisions in the seed set. This, however, is not meant to suggest that mechanical transparency is equivalent to causative legal explanations. But mechanical transparency as to the aolgrithms would, at least, be a start. Even more likely, shifts in technological competence might make generalist lawyers better able to navigate the intricacies of predictive coding. Currently, professional responsibility rules set a very low bar for lawyers’ technological competence.364 But this is changing. For example, Magistrate Judge James Francis has said: E-Discovery is pervasive. It’s like understanding civil procedure. You’re not going to be a civil litigator without understanding the rules of civil procedure. Similarly, you’re no longer going to be able to conduct litigation of any complexity without understanding E-Discovery.365 Some legal practitioners have even argued that the use of predictive coding will itself become an ethical obligatio3n6.6 And developing competence will be easier if the protocols become more standardized, resolving thme-i plementation controversies described above in Part III(C).367 362 Remus, supra note 10, at 1722. 363 See supra notes 292–370. 364 Remus, supra note 10, at 1710. 365 Joe Dysart, Catch Up with Techor Lose Your Career, JudgesWarn Lawyers, A.B.A. J. (Apr. 201,4) career_judges_warn_lawyers []; see also State Bar of Cal. Standing Comm. on Prof’l Responsibility, Formal Op. Interim 01010–4 ( 2015 ), Portals/0/documents/publicComment/2015/2015_11-0004ESI14-12-05-2dpubcomment.pdf [https://] (“Attorney competence related to litigation generally requires, among other things, and at a minimum, a basic understanding of, and facility with, issues relating to ediscovery . . . . On a case-by-case basis, the duty of competence may require a higher level of technical knowledge and ability, depending on the e-discovery issues involved in a matter, and the nature of the ESI.”). 366 Jackson, supra note 279, at 395; Barkett, supra note 264 (manuscript at 32). 367 Remus, supra note 10, at 1722. Less likely, but no less importantly, litigation actors—be they parties, attorneys, or judges—might more fully integrate an ethical obligation that goes beyond the mere tactical adherence to the rules. Instead, they might better incorporate their responsibility to the underlying normsof procedural justice into their practices.368 Recognition of the shared responsibility for ensuring truly just processes can only serve to enhance the legitimacy of the legal ssytem and, ultimately, the welfare of society3.69 And one aspect of this ethical obligation is ensuring that every aspect of the litigation process is intelligibly tied to the underlying legal substance.370 CONCLUSION Legal doctrines must serve first-principles norms. And, in discovery, the norm of participation is a fundamental principle, ultimately contributing to the legitimacy of judicial processes. In civil discovery, this norm is negatively impacted by predictive coding’s opacity, particularly as projected for litigants without the financial resources necessary to hire technological experts. But the trade-off between the norms of accuracy and cos-tefficiency on one hand and participation on the other has not been sfuficiently interrogated in either the prior academic or judicial evaluations. Now that predictive coding has passed the initial factual and second-order legal thresholds, this highe-rlevel normative discussion should begin. Contributing to this need for examination of the normative trade-off, the continuing growth of ESI and the downward pressure of the proportionality amendment to Federal Rule of Civil Procedure 26(b) mean that predictive coding likely will spread to more cases. And there is a risk that the current doctrine will ossify, leaving out the concerns of the litigants who both lack significant financial resources and are not generally participating in the contemporary cases through which the jurisprudence is developing. Given these factors, courtsshould include the potential stress on 368 DAVID LUBAN, LAWYERS AND JUSTICE: AN ETHICAL STUDY 11, 32, 169 ( 1988 ) (obliging lawyers to act with a heightened level of respect for their fellow citizens ansduggesting lawyers should be morally active andwrestle with the principles undergirding democracy;) Yuzhe Zhao, Rules, Morality, and Legal Ethics: Searching for the Underlying Principle of Lawyer Regulation, 25 GEO. J. LEGAL ETHICS 857, 860–61 ( 2012 ). 369 Robert W. Gordon, A Collective Failure of Nerve: The Ba’rs Response to Kaye Scholer, 23 L. & SOC. INQUIRY 315, 321 (1998); see Hickman v. Taylor, 329 U.S. 495, 51–415 (1947) (Jackson, J., concurring) (describing discovery as both“one of the working tools of the legal profession” and “a two-edged sword” in a discussion of how it is often overlooked that lawyers play a role in procedural justice);see also Stephen B. Burbank & Linda J. Silberman,Civil Procedure Reform in Comparative Context: The United States of America, 45AM. J. COMP. L. 675, 683–85 (1997) (noting the inherent link between due process and broader American values). 370 See Joseph P. Tomain, A Code of One’s Own, 15 NOTRE DAME J.L. ETHICS & PUB. POL’Y 153, 159 (2001) (stating intelligibility is an important component of legal ethics). the participation norm in their case-by-case Mathews analyses when managing discovery disputes involving predictive coding to better futureproof the emerging doctrine. Predictive coding in civil discovery is not theonly area in which rapid technological advances—most saliently, the increased prevalence of big data and artificial intelligence—raise concerns about the ability of legal doctrines to adapt. And, although predictive coding presents issues specific to itself, the general lesson remains the same: procedurally just doctrines must account for technological changes and balance their benefits against the risk that their lay impenetrability might diminish the meaningful participation of less sophiis-t cated litigants in legal processes. A. How Predictive Coding in Civil Discovery Actually Works..................................................... 833 1. Growth of Electronically Stored Information...................................................................... 839 2. Lawyers' Gamesmanship ..................................................................................................... 841 3. Proportionality Amendment to the Federal Rules of Civil Procedure ................................ 845 4. Technological Innovation..................................................................................................... 846 C. Court Implementation of Predictive Coding in Civil Discovery.............................................. 847 1. Timeline of Significant Cases .............................................................................................. 847 2. Lessons Drawn from the Case Law ..................................................................................... 850 A. Accuracy and Economic Efficiency Considerations................................................................. 851 B. Expert Reliability and Professional Responsibility Implications............................................. 857 1. Defects of the Existing Approach ........................................................................................ 862 2. Roadblocks to the Normative Inquiry.................................................................................. 868 3. Need to Futureproof ............................................................................................................. 870 4. Potential Non-Doctrinal Ways to Ameliorate Predictive Coding's Impact on the Participation Norm ................................................................................................................... 871 4 See Andrew Jay Peck , Foreword, 26 REGENT U. L. REV. 1 , 3 ( 2013 -2014 ) (discussing the growing volume of ESI and growing discovery costs for litigants). 5 FED . R. CIV. P. 26 ( b)(2)(1) advisory committee's note to 2015 amendment; see also Christi- Case Law and Regulatory Disclosure Requirements , 93 N.C. L. REV. 222 , 234 - 36 ( 2014 ) (refer- encing predictive coding's relationship to the scope of discovery). 6 Stephen B . Burbank & Sean Farhang , Federal Court Rulemaking and Litigation Reform: An Institutional Approach , 15NEV. L.J. 1559 , 1593 ( 2015 ) (describing evolution of eruml aking); Civil Procedure and the Pro-Defendant Composition of the Federal Rulemaking Committee,s 83 U. CIN. L. REV . 1083 , 1112 - 13 ( 2015 ) (explaining how an aggressive proportionality command may lead to less discovery that harms plaintiffs because of information asymmetries ). 7 See Kevin E. Davis & Helen Hershkoff ,Contracting for Procedure, 53WM . & MARY L. REV. 507 , 545 ( 2011 ) (emphasizing the need for robust discovery in employment discrimination cases); see also Bruce L. Hay, Civil Discovery: Its Effects and Optimal Scope, 23J . LEGAL STUD. 481 , 483 - 84 ( 1994 ) (illuminating the twin purposes of discovery through the example of an em- ployment discrimination suit) . 8 Harrison M. Brown , Comment, Searching for an Answer: Defensible ED- iscovery Search Techniques in the Absence of Judicial Voice , 16 CHAP. L. REV. 407 , 411 ( 2012 ). 15 See infra notes 62-249 , 292 - 370 , and accompanying text. 16 See generally Rio Tinto PLC v. Vale S.A ., 306 F.R.D. 125 (S.D.N .Y. 2015 ) (mining com- [] (last visited Feb. 3 , 2018 ); Vale S.A. , YAHOO FIN ., [] (last visited Feb. 3 , 2018 ). 17 Brooke D. Coleman , One Percent Procedure, 91 WASH. L. REV. 1005 , 1007 ( 2016 ). 18 Remus, supra note 10, at 1705 . 19 Theodore J. Greeley , The Plight of Indigent Defendants ina Computer-Based Age: Main- taining the Adversarial System by Granting Indigent Defendants Access to Computer Exper ,ts16 VA. J.L. & TECH . 400 , 403 ( 2011 ) ; see also David Medine, The Constitutional Right to Expert Assistance for Indigents in Civil Case,s41 HASTINGS L .J. 281 , 285 - 91 ( 1990 ) (describing the Jacobs Wiseman , Pro Bono Publico: The Growing Need for Expert Aid , 6S0. C. L. REV . 493 , 528- 35 ( 2008 ) (discussing the limited number of experts or expert associations providing pro bono testimony) . 20 Maura R. Grossman & Gordon V. Cormack , Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Rev,ie1w7 RICH . J.L. & TECH. 11 , 52 ( 2011 ). 21 Rebecca Hollander-Blumoff , The Psychology of Procedural Justice in the Federal Court ,s 63 HASTINGS L.J. 127 , 154 - 55 ( 2011 ) ; Martin H. Redish, Electronic Discovery and the Litigation Matrix , 51 DUKE L.J. 561 , 600 ( 2001 ). 22 See infra notes 166-201 (describing cases discussing proportionality); see also Coleman, supra note 17 , at 1007 . 23 See Brooke D. Coleman , The Efficiency Norm, 56 B.C. L. REV . 1777 , 1824 ( 2015 ) (noting the value in considering the litigants' acceptance of the result);see also Mathews v . Eldridge , 424 U.S. 319 , 323 - 49 ( 1976 ) (establishing the we-lklnown balancing test for weighing due process interests); supra notes 55-61 and accompanying text . 24 See Robert G. Bone, Improving Rule 1: A Master Rule for the Federal Rules , 87 DENV. U. L. REV . 287 , 302 ( 2010 ) (proposing revised language for Rule 1 based on the normative goals of the Federal Rules);Redish, supra note 21 , at 600 ( considering discovery in the context of the broader goals of litigation) . 25 Implicitly, this assumes a variant of Hart and Sa'ckLsegal Process approach , wherein William L. Reynolds & Spencer Weber Waller , Legal Process and the Past of Antitrust , 48 SMU L. REV . 1811 , 1815 ( 1995 ). 26 Vivian Grosswald Curran , United States Discovery and Foreign Blocking Statutes , 76 LA. L. REV . 1141 , 1141 ( 2016 ) ; Arthur R. Miller ,Confidentiality, Protective Orders , and Public Ac- cess to the Courts, 105 HARV. L. REV. 427 , 464 ( 1991 ) ; Imre Stephen Szalai,A Constitutional boratory of Arbitration , 15 PEPP. DISP. RESOL. L.J. 337 , 374 - 75 ( 2015 ). 27 See, e.g., Moore , 287 F.R.D. at 189 (balancing the accuracy and comprehensiveness of a methods) . 28 FED. R. CIV. P. 26-37. 29 See infra notes 292-370 and accompanying text. 30 FED. R. CIV. P. 26. 31 See infra notes 292-370 and accompanying text; see also Henry H. Perritt , Jr.,The Elec- tronic Agency and the Traditional Paradigms of Administrative La,w44 ADMIN . L. REV. 79 , 89 ( 1992 ) (discussing the constituent sub-goals of efficiency). 32 See Bernadette Bollas Genetin, “Just a Bit Outside!”: Proportionality in Federal Discov- ery and the Institutional Capacity of the Federal Courts, 34 REV . LITIG. 655 , 661 ( 2015 ) (suggest- infra notes 292-370 and accompanying text. 33 See Genetin, supra note 32 , at 661 (concluding the social implications for bounding dsi- covery may fall outside the scope of judicial expertise ). 34 FED. R. CIV. P. 1 . 35 See Steven S. Gensler & Lee H. Rosenthal , Managing Summary Judgment, 43 LOY. U. CHI. L.J. 517 , 520 ( 2012 ) (highlighting the central role of proportionality in discovery) . See generally Bone, supra note 24 (suggesting amendments to Rule 1 to better serve the purposes of the Federal Rules). 36 See Stephen N. Subrin , Fishing Expeditions Allowed: The Historical Background of the 1938 Federal Discovery Rules , 39 B.C. L. REV . 691 , 716 ( 1998 ). 37 See, e.g., Herbert v . Lando , 441 U.S. 153 , 177 ( 1979 ) (describing the goal as“adequately FEDERAL CIVIL DISCOVERY AND DISCLOSURE § 9:2 , at 510- 11 (3d ed. 2010 ); John S. Becker- man , Confronting Civil Discovery's Fatal Flaws , 84 MINN. L. REV. 505 , 534 - 35 ( 2000 ) (discuss- ing the relationship between discovery and pleadings; ) Morgan, supra note 12, at 76 (examining the speed of predictive coding) . 38 FED. R. CIV. P. 1. This is, of course, not to imply that these pragmatic goals are mu-ni imply that these scholars and courts are not mindful of the normative questions. 39 See Subrin , supra note 36 , at 710 , 716 (describing the development of procedural rules over time); see also Colman, supra note 23 , at 1811- 12 ; Brooke D. Coleman , Recovering Access: Re- thinking the Structure of Federal Civil Rulemaking , 39 N.M. L. REV . 261 , 282 ( 2009 ); Edson R. Sunderland , An Inquiry Concerning the Functions of Procedure in Legal Education, 21MICH . L. REV. 372 , 381 - 82 ( 1923 ). 40 E.g., Yitshak Cohen , The Issue of Document Disclosure in General Courts and in Family Courts : A New Model, 37 HOUS. J. INT'L L . 43 , 60 ( 2015 ). 41 Redish, supra note 21, at 593 . 42 Lawrence B. Solum , Procedural Justice, 78 S. CAL. L. REV . 181 , 305 - 07 ( 2004 ). Professor the same concepts . See LAURENCE H. TRIBE, AMERICAN CONSTITUTIONAL LAW 666 (2d ed. 1988 ); see also Gensler & Rosenthal, supra note 35, at 524; Lani Guinier ,No Two Seats: The Elusive Quest for Political Equality , 77 VA. L. REV. 1413 , 1489 ( 1991 ) ; Frank I. Michelman , The Supreme Court and Litigation Access Fees: The Right to Protect OsneR'ights- Part II , 1974 DUKE L.J . 527 , 543 ; Martin H. Redish & Lawrence C. Marshall ,Adjudicatory Independence and the Values of Procedural Due Proces,s 95 YALE L .J. 455 , 487 ( 1986 ) (discussing Michelmans' Formal and Associational Aims in Procedural Due Proce)s;s Solum , supra note 42 , at 277-82. 42, at 543 . 43 Solum, supra note 42, at 275. At the same time, Professor Solum asserts that the participa- tion principle does not rely on dignity, equality, or autonomy grounds . Id. at 286-90 . But, even if Bone , Statistical Adjudication: Rights, Justice, and Utility ian World of Process Scarcity, 46 VAND. L. REV . 561 passim ( 1993 ). 44 Solum, supra note 42 at 279- 80 . 45 Id. at 280. Court's 2007 and 2009 decisions in Bell Atlantic Corp . v. Twombly and Ash- croft v. Iqbal . 59 59 Ashcroft v. Iqbal , 556 U.S. 662 , 685 ( 2009 ) ; Bell Atlantic Corp . v. Twombly, 550 U.S. 544 , 558 ( 2007 ); Blair-Stanek, supra note 55 , at 11; Szalai, supra note 26, at 372; Jonah B. Gel - cess to Discovery, 121YALE L.J. 2270 , 2285 ( 2012 ) ; see also Samuel Issacharoff & Geoffrey Miller , An Information-Forcing Approach to the Motion to Dismis,s 5 J . LEGAL ANALYSIS 437, 438 ( 2013 ) ; cf . Subrin, supra note 36, at 745(noting the Advisory Committee suggestion that automatic discovery should mirror the fact particularity in an opponent's pleading) . 60 See Mathews , 424 U.S. at 339 , 345 (noting existence of evidentiary hearing, albeit after termination, and contrasting the procedure with those inGoldberg v . Kelly , 397 U.S. 254 ( 1970 )); Solum , supra note 42, at 309-10 (noting Mathews' consistency with the participation principle ). 61 See Martha I. Morgan , The Constitutional Right to Know Why, 17 HARV. C.R.-C.L. L. REV . 297 , 299 ( 1982 ) (linking the reasons requirement and individual participation);Solum, supra note 42, at 280 . 62 Remus & Levy, supra note 14, at 503- 04 . 63 The invaluable and authoritative GrossmanC-ormack Glossary is a helpful resource for Magistrate Judge , 7 FED. CTS. L. REV. 1 ( 2013 ) [hereinafter Grossman-Cormack Glossary]. 68 The methods used to develop the mode-lsand the models themselve-scan take many forms. Barry, supra note 66, at 355; Daniel Martin Katz, Quantitative Legal Prediction-or-How Industry , 62 EMORY L.J. 909 , 946 ( 2013 ). For example, some predictive coding processes use note 64, at 266. Others might employ support vector machines, which draw a separating hyper- plane with margins that provide a buffer . Id . at 270-71; Grossman-Cormack Glossary , supra note 63, at 31. Also used are Bayesian algorithms, which estimate statistical probabilities that are based on observed prior outcomes . Browns,upra note 64 , at 274-76; Grossman-Cormack Glossary , supra note 63 , at 9. While the preceding methods do not comprise an exhaustive list , they illus- trate some of the mechanisms that underlie more robust predictive coding processes . 69 Katz, supra note 68, at 955. 70 Id. at 954-55 . 71 See Redish, supra note 21, at 571-74. 72 SEDONA CONF., COMMENTARY ON DEFENSE OF PROCESS: PRINCIPLES AND GUIDELINES FOR DEVELOPING AND IMPLEMENTING A SOUND E-DISCOVERY PROCESS 31- 34 ( 2016 ). 73 See Redish, supra note 21 , at 591; see also Moore v. Publicis Groupe , 287 F.R.D. 182 , 191 (S.D.N .Y. 2012 ) (describing value of expert testimony in decision); L. Casey Auttonberry , Predic- tive Coding: Taking the DevilOut of the Details , 74 LA. L. REV. 613 , 622 - 23 ( 2014 ) (discussing the need for counsel to consult with experts when engaging predictive coding services ). 74 Katz, supra note 68 , at 950 & n.198. This issue -that is, the lack of a substantive tie-has 1, at 151 ( discussing a credit card company's unwillingness to explain their rationale for slashing a consumer's credit limit) . 75 Rich, supra note 2 , at 886; see also Andrea Roth, Trial by Machine , 104 GEO. L.J. 1245 , 1271 ( 2016 ) (discussing “black box” concerns in the criminal context); Zeynep TufekAclig ,o- rithmic Harms Beyond Facebook and Google: Emergent Challenges of Computational Agency , 13 COLO. TECH . L.J. 203 , 208 - 09 ( 2015 ) (contrasting presentation of information from traditional print media with Facebook news feed algorithms ). 76 Tal Z. Zarsky , Transparent Predictions, 2013 U. ILL. L. REV . 1503 , 1520 . 77 Katz, supra note 68, at 949-50; see also Yablon & Landsman-Roos , supra note 12 , at 652 . 78 See George M. Cohen , The Multilawyered Problems of Professional Responsibility, 2003 U. ILL. L. REV . 1409 , 1442 (discussing vicarious liability between lawyers, which creates a chain son, A Collaborative Model of Offshore Legal Outsourcing, 43ARIZ . ST. L.J. 125 , 174 - 76 ( 2011 ) (describing methods for promoting client accountability even when legal work is outsourced ). 79 See Rich, supra note 2, at 897. 80 Id. 81 Yablon & Landsman-Roos, supra note 12, at 663. 82 See supra notes 67-74 and accompanying text; see also Benjamin H. Barton, The Lawyer's Monopoly-What Goes and What Stay,s 82 FORDHAM L . REV. 3067 , 3072 ( 2014 ) (suggesting law) . 83 Remus, supra note 10 , at 1715 . 84 Id. This, of course, is not a problem that only arises in the context of predictive coding- but the financial inequities are exacerbated with this sort of opaque, highly complexecthnology. 101 Pyrrho Invs . Ltd. v. MWB Prop. Ltd. [2016] EWHC 256 (Ch) (Eng .); Ryan C. Thomas et Between the U.S. and the U.KJ .O?, NES DAY (Mar . 2016 ), Publication /489cc7ed-874e - 4c03- 8717 -014e894c0208/Presentation/PublicationAttachment/1d2bd 9ca-4e2d-400b-aee7-0b7e6038618e/Embracing%20e-Discovery%20in%20Antitrust.pdf [https://] ; High Court Approves Tool to Search Quinn Family Documents , IRISH TIMES (Mar. 5 , 2015 ), https/:/ -services/high-court-approves- tool-to-search-quinn-family-documents- 1 .2126405 [] (describing use in Ireland) . 102 John Tredennick, Using TAR in International Litigation: Does Predictive Coding Work for Non-English Languages ?, CATALYST (Mar. 24 , 2014 ), 03/using-tar-in-international-litigation-does-predictive-coding-work-for-non-english-languages/ []. 103 Peck, supra note 4, at 3 . 104 Natalie M. Banta, Death and Privacy in the Digital Age, 94 N.C. L. REV . 927 , 928 ( 2016 ). 234 This, of course, is part of a more general trend in civil procedure . See Coleman , supra note 23, passim. 235 E.g., Remus, supra note 10 , at 1718 . 236 See Nelson & Simek, supra note 23, at 20- 24 . But see Yablon & Landsman-Roos , supra note 12, at 671 . 237 See Nelson & Simek, supra note 23, at 20-24; see also Kobayashi, supra note 233 , at 1501; Vaccaro, supra note 212, at 329 (“ To set a bright line rule, some suggest , and this note en- dorses, that there should be at least 100,000 documents needed to be reviewed, and a case value of at least $ 200 , 000 .”). But see Yablon & Landsman-Roos, supra note 12 , at 671 . 238 See also EORHB , Inc. v. HOA Holdings LLC , No. C.A. 7409-VCL, 2013 WL 1960621, at *1 (Del. Ch. May 6 , 2013 ); Morgan, supra note 12, at 75. Compare Pangea3 & Recommind Align to Provide Industry-Leading Predictive Coding Offering, THOMSON REUTERS (Oct. 15 , 2013 ), 2013 /pangea3-and - recommind - align-to- provide-industry-leading-predictive-coding-offering .html [] (quot- ing a vendor who stated, “Pangea3s Predictive Review Services deliver the full range oRf ecom- complexity of the case.”) , with Remus, supra note 10 , at 1707 ( suggesting there is no certainty of reduced discovery expense even when embracing predictive coding ). 239 Auttonberry, supra note 73 , at 634-35; Waxse & Yoakum-Kriz , supra note 13 , at 221 . 240 Yablon & Landsman-Roos, supra note 12, at 664. 241 Id . 256 Gelb, supra note 13, at 1293; Waxse & Yoaku m-Kriz , supra note 13 , at 219-21; Panel Discussion , Symposium on the Challenges of Electronic Evidenc,e83 FORDHAM L. REV . 1163 , 1237 ( 2014 ) [hereinafter Symposium on the Challenges of Electronic Evidence] . 257 Gelb, supra note 13, at 1296- 97 . 258 Waxse & Yoakum-Kriz, supra note 13, at 220. 259 Id.; Symposium on the Challenges of Electronic Evidence, supra note 256, at 1239-40 . 260 Nelson & Simek, supra note 64, at 24 . 261 E.g., Remus, supra note 10, at 1712. 262 Id. 263 Id . 264 See John M. Barkett , More on the Ethics of E-Discovery: Predictive Coding and Other Forms of Computer-Assisted Review 42 ( 2012 ) (on file with Duke University School of Law) //https: 5 -Original_Paper.pdf [] (providing thoughtfulcommentary on a host of extant and potential 272 E.g., Facciola & Favro, supra note 14 , passim; Remus, supra note 10, at 1716- 17 . 273 E.g., Facciola & Favro, supra note 14 , passim; Remus, supra note 10, at 1716- 17 . 274 E.g., Facciola & Favro, supra note 14 , passim; Remus, supra note 10, at 1716- 17 . 275 Rio Tinto, 306 F.R.D. at 128; Yablon & Landsman-Roos , supra note 12, at 644-45 . 276 See, e.g., FormFactor , Inc v. MicroP- robe, Inc., No. C- 10 -03095 PJH (JCS) , 2012 WL 1575093, at *7 ( N.D. Cal . May 3, 2012 ) (ordering discovery on search terms); Romero v . Allstate Ins. Co. , 271 F.R.D. 96 , 109 - 10 (E.D. Pa. 2010 ) (ordering discovery on search terms); cf . Miller v. Holzmann , 238 F .R.D. 30 , 32 (D.D .C. 2006 ) (noting the limits of the decision iSnporck v . Peil, 759 F.2d 312 ( 3d Cir . 1985 ).) But see Koninklijke Philips N.V. v. Hunt Control Sys ., Inc.,Civil Action No . 11 - 3684 DMC , 2014 WL 1494517, at *4 ( D.N.J. Apr . 16 , 2014 ) (granting a protective tion of its ESI because it would open the door to more discovery with no limiting principle ). 277 MODEL RULES OF PROF'L CONDUCT r. 1.1 cmt.8 (AM . BAR ASS'N 2016 ). 278 Remus, supra note 10, at 1710 (discussing Model Rule 5 .3 and its comments). 279 Joy Flowers Conti & Richard N. LettierEi ,-Discovery Ethics: Emerging Standards of Technological Competence , FED. LAW., Oct .- Nov . 2015 , passim ( 2015 ); Randy L. Dryer, Litiga- Welcome in Utah, UTAH B .J., May-June 2015 , at 12 , 16; Darla W. Jackson , Lawyers Can't Be Rules Involving Technology?, 105 LAW LIBR . J. 395 , 398 ( 2013 ); Remus, supra note 10, at 1719; Vaccaro , supra note 212, at 319 “ (Experts say lawyers must be prepared to use a quantitative include revamping law school curriculum . See Katz, supra note 68 , at 965 . 280 See Metzler, supra note 141 , at 1164 . 281 See generally Peter Segrist, How the Rise of Big Data and Predictive Analytics Are Changing the Attorney's Duty of Competence , 16 N.C. J.L. & TECH . 527 , 603 ( 2015 ). 282 Joe Palazzolo , Software: The Attorney Who Is Always on the JobW,ALL ST . J., May 6 , 2013, at B1; see also Frank Pasquale & Glyn Cashwell , Four Futures of Legal Automation , 63 UCLA L. REV . DISCOURSE 26 , 45 ( 2015 ), 2015 /06/Final-ALL.pdf []. 283 E.g. , Remus, supra note 10, at 1709-11 . 284 See John S. Dzienkowski , The Future of Big Law: Alternative Legal Service Providers to Corporate Clients , 82 FORDHAM L. REV. 2995 , 3001 ( 2014 ) (discussing the expansion of legal services to include a broader array of disciplines, particularly with large corporate clients ). 285 E.g., Remus, supra note 10 , at 1709- 11 . Accordingly, there is potential disagreement of law jurisprudence . Id.; see also Remus & Levy, supra note 14 , at 538-42; Deborah L. Rhode & Enforcement , 82 FORDHAM L. REV. 2587 , 2595 ( 2014 ). 286 Remus, supra note 10, at 1709 . 287 Id.; see also Matt Hassett et al., Managing Outsourcing and E-Discovery, COUNSEL , Feb. 2014, at 15 , 17 . 288 Remus & Levy, supra note 14, at 540- 41 . 289 Id.; James A. Sherer et al., Merger and Acquisition Due Diligence Part II-The Devil in the Details , 22 RICH. J.L. & TECH. 4 , 4 , 17 ( 2016 ) ; see also Lola v . Skadden, Arps, Slate, Meagher & Flom LLP , No. 14 -3845-cv, 2015 WL 4476828, at *2 (2d Cir. July 23 , 2015 ). 290 Compare Murphy, supra note 217 , at 645-46 (discussing the reasonability of compute-r based analytic methods) , and Remus, supra note 10, at 1722 (suggesting the utility of claw-back agreements in cases involving predictive coding,) with Vaccaro , supra note 212 , at 322-23 (sug- precautions to prevent disclosure, and recommending the use of a privilege log). 291 See FED . R. EVID. 502 ; Murphy, supra note 217, at 646; Liesa L. Richter , Making Horses Drink: Conceptual Change Theory and Federal Rule of Evidence 502 , F8O1RDHAM L. REV . 1669 , 1670 - 73 ( 2013 ). 292 See supra note 49 and accompanying text. 293 See supra notes 202-249 and accompanying text. 294 See supra notes 192 , 196 and accompanying text. 295 See Morgan, supra note 61, at 299; Solum, supra note 42, at 280 . 296 Remus, supra note 10, at 1715. Professor Remus's forthcoming work also briefly idenit- legal technologies might particularly lag . Remus & Levy, supra note 14 , at 550- 53 (using tax guidance software as an example) . 297 E.g., Moore , 287 F .R.D. at 192 . 298 Id. at 190; supra notes 202-249 and accompanying text. Much of this conversation though Standard?, 13 SEDONA CONF . J. 217 , 218 - 19 ( 2012 ) ; Symposium on the Challenges of Electronic Evidence , supra note 256, at 1236 (“ Practice Point Number 7 from the Sedona Search and Infor- depositions, evidentiary proceedings and trials.'”) . 299 See, e.g., Heather K. Gerken , Lost in the Political Thicket: The Court , Election Law, and the Doctrinal Interregnum , 153 U. PA. L. REV . 503 , 517 ( 2004 ); see also Thomas L. Fowler, Law Between the Lines , 25 CAMPBELL L. REV. 151 , 155 ( 2003 ). 300 Fowler, supra note 299, at 155 . 301 See Marshall v. Jerrico, Inc., 446 U.S. 238 , 242 ( 1980 ) (describing the“promotion of par- tral elements of procedural due process). 302 Doe v . District of Columbia , 697 F.2d 1115 , 1119 -20 (D.C. Cir . 1983 );see also Remus, supra note 10, at 1691. 303 See supra notes 202-249 and accompanying text. 304 See also Solum, supra note 42 , at 242. Compare Remus, supra note 10 , at 1717 (discuss- leged documents) , with Facciola & Favro, supra note 14 , at 32 ( suggesting courts are unlikely to The Roles of Litigation in American Democracy , 65 EMORY L.J. 1657 , 1689 ( 2016 ). 305 See supra note 276 and accompanying text . 306 See Roger Michalski, The Clash of Procedural Values , 22 LEWIS & CLARK L. REV . (forth- coming 2018 ) (manuscript at 35) , [https:// perma. cc/4B7V-5ZFL] . 313 For the component of the dueprocess norm of participation that is the focus of this Air-t finance literature . See Burt Neuborne, Ending Lochner Lite , 50 HARV. C.R.-C.L. L. REV . 183 , 209 ( 2015 ) (explaining how the First Amendment is embedded in stat-emandated resolution proceed- ings); Burt Neuborne, Taking Hearers Seriously , 91 TEX. L. REV. 1425 , 1435 - 36 ( 2013 ) (laying out the dignitary and instrumentalist benefits of being able to hear ). 314 See , e.g., William M. O'Barr & John M. Conley ,Lay Expectations of the Civil Justice System , 22 L. & SOC'Y REV . 137 , 137 - 38 ( 1988 ); Donna Shestowsky,The Psychology of Proce- dural Preference: How Litigants Evaluate Legal Procedures Ex Ant ,e 99 IOWA L. REV. 637 pas - sim ( 2014 ) (evaluating participant satisfaction with an array of procedural alternatives ). 315 Solum, supra note 42 , at 264 . 316 Joint Anti-Fascist Refugee Comm. v. McGrath , 341 U.S. 123 , 172 ( 1951 ) (Frankfurter , J. , popular government, that justice has been done”). 317 Recall the earlier discussion of how discovery issues played a role inAshcroft v . Iqbal and Bell Atlantic Corporation v . Twombly.See supra note 59 and accompanying text; see also Ash- croft v. Iqbal , 556 U.S. 662 , 685 ( 2009 ) ; Bell Atlantic Corp . v. Twombly, 550 U.S. 544 , 558 ( 2007 ). 318 See supra notes 55-59 and accompanying text. 319 Maureen N. Armour, Practice Makes Perfect: Judicial Discretion and the 1993 Amedn- ments to Rule 11 , 24 HOFSTRA L. REV. 677 , 707 ( 1996 ) ; Lon L. Fuller, The Forms and Limits of Adjudication , 92 HARV. L. REV. 353 , 366 - 67 ( 1978 ); Lahav, supra note 304, at 1677-78; Richard Warner , Note, Three Theories of Legal Reasoning , 62 S. CAL. L. REV . 1523 , 1523 - 24 ( 1989 ). 320 Fuller, supra note 319, at 369 . 321 See supra notes 42-49 and accompanying text. For the component of the due process norm analogy to the right to hear that one sees in the campaign finance literature . 322 Joshua M. Koppel , Comment, Tailoring Discovery: Using Nontranssubstantive Rules to Reduce Waste and Abuse, 161 U. PA. L. REV . 243 , 277 - 78 ( 2012 ) ; see also Frank Pasquale , Re- storing Transparency to Automated Authority, 9 J. TELECOMM . & HIGH TECH . L. 235 , 237 ( 2011 ). See generally Solon Barocas & Andrew D. Selbst , Big Data's Disparate Impact , 104 CALIF. L. REV. 671 ( 2016 ). 323 Koppel, supra note 322, at 277. Additionally, while the focus of this Article is the norma- 93 YALE L.J. 1073 , 1076 ( 1984 ). 324 Remus, supra note 10, at 1715 . 325 Michalski, supra note 306 (manuscript at 35-36); Rich, supra note 2, at 886; Zarsky, su - pra note 76 , at 1520. In the context of credit scoring, one proposal went further, suggesting that the scoring systems' algorithms.” Citron & Pasquale, supra note 2 , at 24- 25 . And Professor Roth supra note 75 , at 1272 (discussingIn re Source Code Evidentiary Hearings in Implied Consent Matters , 816 N.W.2d 525 , 529 (Minn. 2012 )). 326 Helen Hershkoff , Poverty Law and Civil Procedure: Rethinking the First-Year Course , 34 FORDHAM URB. L .J. 1325 , 1326 , 1329 ( 2007 ); Lahav, supra note 304, at 1677- 78 . 327 See Kobayashi, supra note 233 , at 1506 . 328 See Harry Surden , Machine Learning and Law , 89 WASH. L. REV. 87 , 109 - 10 ( 2014 ). 329 See supra note 138 and accompanying text. 330 See Yablon & Landsman-Roos, supra note 12, at 665-66 . 331 See Pauline M. Pelletier , The Impact of Local Patent Rules on Rate and Timing of Case Resolution Relative to Claim Construction: An Empirical Study of the Past Deca,de8 J. BUS . & TECH. L. 451 , 452 & n.5, 501 ( 2013 ) (citing growth described in the 2010 Year-End Report on the son Is Over: After Barrick and Amended Pennsylvania Rule of Civil Procedure 4003.5 , Pennsyl- Experts , 7 DREXEL L. REV . 329 , 347 ( 2015 ) (noting the empowerment of trial courts to resolve discovery disputes) . 332 Compare Alexander Nourse Gross , Note, A Safe Harbor from Spoliation Sanctions : Can an Amended Federal Rule of Civil Procedure 3e7)( Protect Producing Parties? , 2015COLUM . BUS. L. REV . 705 , 712 (distinguishing only between paper discovery and general ESI discovery), with supra notes 73-86 and accompanying text (distinguishing paper discovery, predictive coding, and Boolean or keyword searches performed in other forms of ESI discovery) . 333 See Redish, supra note 21 , at 564; see also Paul Ohm, The Argument Against Technology- Neutral Surveillance Laws , 88 TEX. L. REV. 1685 , 1695 ( 2010 ) (“While the law should not treat true that some differences deserve to be treated differently . ”) . 334 See Coleman, supra note 23, at 1787-93 . 335 See FED. R. CIV . P. 26 . 336 See Matthew J.B. Lawrence , Procedural Triage, 84 FORDHAM L. REV. 79 , 83 ( 2015 ) (ap- pothesis , 82 U. DET. MERCY L. REV . 649 , 655 , 681 ( 2005 ) (describing a “value-hierarchy pyramid for civil procedure” that goes from a fact, case-based level to overarching norms) . 337 See supra notes 202-249 and accompanying text. 338 See supra notes 250-291 and accompanying text . 339 See FRANK PASQUALE , THE BLACK BOX SOCIETY: THE SECRET ALGORITHMS THAT CON- TROL MONEY AND INFORMATION 195 ( 2015 ). 340 See Citron, supra note 2, at 1271-72. 341 See supra notes 292-370. 342 See supra note 324 and accompanying text; see also Lahav, supra note 304 , at 1682 . 343 Michael Birnhack , Reverse Engineering Informational Privacy Law, 1Y5ALE J.L. & TECH. 24 , 38 - 39 ( 2012 ). 344 Id. ; see also Orin S. Kerr, A User's Guide to the Stored Communications Act, and a Legis- lator's Guide to Amending It, 72 GEO . WASH. L. REV. 1208 , 1214 ( 2004 ). 345 See Ohm, supra note 333 , at 1713. One example where a concern about futureproofing has the Reliability of Handwriting Identification Expertise Since the Decision in Daubert, 43 TULSA L . REV. 477 , 526 - 27 ( 2007 ). 346 David Dolinko, Is There a Rationale for the Privilege Against S-Ienlfcrimination?, 33 UCLA L. REV . 1063 , 1147 ( 1986 ) ; see also Kermit Roosevelt III , Constitutional Calcification : How the Law Becomes What the Court Does , 91 VA. L. REV. 1649 , 1693 ( 2005 ). 347 Coleman, supra note 17, at 1050-62. 348 See supra notes 303-325 and accompanying text . 353 E.g., David Grant, Seeing Is Believing: Using Visual Analytics to Take Predictive Coding Out of the Black Box,FTI CONSULTING TECH . ( 2013 ), white-papers/seeing-believing-using-visual-analytics-take-predictive-coding-out-black-box . 354 Remus, supra note 10, at 1723 . 355 The right to technological expert assistance is significantly more developed in the criminal Experts , 16 VA. J.L. & TECH . 400 , 403 ( 2011 ). On the civil side, this is potentially complicated by Warburg LLC , 217 F .R.D. 309 , 317 (S.D.N .Y. 2003 ) (citing Oppenheimer Fund, Inc . v. Sanders, 437 U.S. 340 ( 1978 ) ). But the illegitimacy of economic costs burdening access to justice has been well established . See Michelman, supra note 42 , passim. 356 FED. R. EVID. 706 ( a ); Medine, supra note 19 , at 290-91; Wiseman, supra note 19, at 512- 14; see, e.g., Gabriel Techs. Corp. v. Qualcomm Inc.,o. N08cv1992 AJB(MDD) , 2013 WL 410103, at *10 ( S.D. Cal . Feb. 1 , 2013 ) (awarding $2 , 829 , 349 . 10 for costs of comput-earssisted, algorithm-driven document review) , aff'd , 560 F. App 'x 966 (Fed . Cir. 2014 ). 357 Additionally, courts might flex their discretion . See Steven Baicker-McKee, The Award of E-Discovery Costs to the Prevailing Party: An Analog Solution ina Digital World, 63 CLEV . ST. L. REV . 397 , 424 ( 2015 ). 358 Medine, supra note 19, at 348- 49 . But see Kemp v. Dretke , 86 F. App 'x 680 , 682 (5th Cir. 2004 ) (rejecting ineffective assistance of counsel claim based on inability to procure mental health expert) . 359 Remus, supra note 10, at 1719-20 . 360 Wiseman, supra note 19, at 528 . 361 See Courtney R. Barksdale , All That Glitters Isn't Gold: Analyzing the Costs and Benefits of Litigation Finance, 26REV. LITIG . 707 , 711 ( 2007 ;) Monroe H. Freedman ,Caveat Lector: Conflicts of Interest of Ali Members in Drafting the Restatement,s 26 HOFSTRA L . REV. 641 , 650

This is a preview of a remote PDF:

Seth Katsuya Endo. Technological Opacity & Procedural Injustice, Boston College Law Review, 2018,