Re: [AUDITORY] Registered reports (Etienne Gaudrain )

Subject: Re: [AUDITORY] Registered reports From: Etienne Gaudrain <egaudrain.cam@xxxxxxxx> Date: Wed, 6 Jun 2018 09:57:10 +0200 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY> --000000000000a75493056df483f1 Content-Type: multipart/alternative; boundary="000000000000a75491056df483f0" --000000000000a75491056df483f0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Dear list, Alright, I'll do the grumpy old man. Seeing how little attention people pay to the conversation while at the same time feeling entitled to voice an opinion is a little bit disconcerting. A striking example is how some decided to use a gendered title (Mr) for someone, without even bothering to check their gender. In fact, even after Gaston offered a correction, people still used the wrong title... (I think it is "beyond ridiculous" that anyone would use titles on the Auditory List, especially if they're gonna get it wrong) If the same lack of care is applied to studying the literature, perhaps indeed, we shouldn't support a system that could generate even more reports that people won't read. On the other hand, if the same lack of care is applied to peer-review, then the old system is already broken (and perhaps pre-registration would bring a little bit of fresh air). In fact, if the same lack of care is applied to reading CVs when recruiting people, forming panels, or choosing speakers to invite for a conference, for sure we've been doomed for quite a while now... Cheers, -Etienne --=20 Etienne Gaudrain, PhD Centre de Recherche en Neurosciences de Lyon - CNRS UMR 5292 Universit=C3=A9 Lyon 1 50 av. Tony Garnier 69366 Lyon Cedex 7, FR UMCG, Afdeling KNO, BB20 PO Box 30.001 9700 RB Groningen, NL Phone +31 5036 13290 Skype egaudrain Note: emails to this address are limited to 10 MB. To send larger files, use egaudrain.cam@xxxxxxxx On 5 June 2018 at 16:50, Les Bernstein <lbernstein@xxxxxxxx> wrote: > Dear List, > > For this topic, I'll violate my rule of not posting replies here. I agre= e > with Ms. Rankovic. I sure did not miss the substance and detail of Mr. > Schoof's email. I also read over the information in the links. Indeed, > the proposed plan provides for a second review. It seems to me, however, > that the provisional acceptance is a key aspect of the process. If it we= re > the case that manuscripts were rejected upon second review with substanti= al > frequency, then the philosophy of the registered report would be violated > and the system would collapse. So, unless there are egregious errors or > flaws in the full manuscript, it seems that it would be published. Note > that, in this linked reference <https://orca.cf.ac.uk/59475/1/AN2.pdf>, > publication is assumed to be "guaranteed." > > In my opinion, the criticism found within the FAQ here > <https://cos.io/rr/>, that "The Registered Reports model is based on a > na=C3=AFve conceptualisation of the scientific method." is well-founded! = The > reply offered to counter that criticism is quite weak and unconvincing. = I > would replace "scientific method" in that criticism with "the way good > science is done." > > Question 17 in Chambers et al. (2014-- linked above) provides an apt > example. In the process of conducting complex experiments, it is very > often the case that unexpected results lead to important follow-up or > control experiments. Chambers et al. handle this issue by proposing that > in Stage 1 of a registered report, contingencies be stated such that "If = A > is observed, then we will..." That, of course, assumes that one knows th= e > decision tree in advance! In my experience, science simply does not work > that way. > > While I find the intent of registered reports to be laudable, in my > opinion, it substitutes one potential set of problems with another based = on > a narrow view of how science proceeds. Indeed, one may have a hypothesis > to be tested and gather a set of data to address it only to find that the > results support a substantially altered view. Is that, NECESSARILY, the > dreaded "HARKing?" I think not. Scientific thought and inquiry do not > always proceed in a linear fashion. One cannot and should not always kno= w > the precise questions or list of contingencies a priori and be restricted > to answering only those. Then there are experiments in which there are n= o > specific hypotheses. They may be of the form, "What is the effect of > variable A on measurements of X?" Assuming the question is non-trivial, > those are often the most revealing experiments because any outcome is of > interest. There is no "positive" or "negative." Sure, one can cast such > experiments in terms of hypotheses but doing so often involves a > contrivance. > > Then there is the matter of "p-hacking" and what I would call "statistics > shopping." Indeed, it is a problem. Unexpected outcomes and patterns of > data in a complex experiment often require one to choose the appropriate > statistic after the fact. It is sometimes the correct thing to do! > Whether it is proper can and should be judged by reviewers with the > requisite expertise. Good peer-review should distinguish between p-hacki= ng > and a rational choice that conveys information and "truth." The notion > that one can and should use only the statistic decided upon in advance is > unnecessary restrictive. > > Finally, there is the matter of archival value. According to Chambers et > al., "...if the rationale and methods are sound then the journal should > agree to publish the final paper regardless of the specific outcome." It > is often the case that rationale and methods are sound but the data provi= de > no substantial advance or archival value. I'm not sure that "approving" = a > method and rationale and virtually guaranteeing publication will afford t= he > same level of judgment in terms of archival value that is afforded by the > current system. > > Les Bernstein > > -- > *Leslie R. Bernstein, Ph.D. **| *Professor > Depts. of Neuroscience and Surgery (Otolaryngology)| UConn School of > Medicine > 263 Farmington Avenue, Farmington, CT 06030-3401 > Office > <https://maps.google.com/?q=3D263+Farmington+Avenue,+Farmington,+CT%0D%0A= ++++++++++06030-3401+%0D%0A+++++++++Office&entry=3Dgmail&source=3Dg>: > 860.679.4622 | Fax: 860.679.2495 > > > > > > > > > <https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__cos.io_rr_&d=3DDw= MFAg&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=3D2Pw2GwelGcMR4953G-= STHGpPJm2-pYYYSPmTwJk3sWM&m=3DSr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnFKUymDoKpI&s= =3DvXqZBKaP1dUovPzwBwC5DalLCB6UxwKuM9x_SQCbw5I&e=3D> > On 6/4/2018 7:51 AM, Christine Rankovic wrote: > > Mr. Schoof: > > > > It is beyond ridiculous to accept partial manuscripts for publication. > > Christine Rankovic, PhD > > Scientist, Speech and Hearing > > Newton, MA USA > > rankovic@xxxxxxxx > > > > > > *From:* AUDITORY - Research in Auditory Perception [ > mailto:AUDITORY@xxxxxxxx <AUDITORY@xxxxxxxx>] *On Behalf Of > *Schoof, Tim > *Sent:* Monday, June 04, 2018 4:06 AM > *To:* AUDITORY@xxxxxxxx > *Subject:* Registered reports > > > > Dear list, > > > > I'm going to try and get hearing science journals to start offering > registered reports. These reports are basically peer-reviewed > pre-registration documents where you outline your methods and proposed > analyses. If this document makes it through peer-review, the manuscript i= s > provisionally accepted for publication. This process should reduce certai= n > questionable research practices, such as selective reporting of results a= nd > publication bias. If you're sceptical about registered reports, the Cente= r > for Open Science has compiled a nice FAQ list that might address some of > your concerns: https://cos.io/rr/ > <https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__cos.io_rr_&d=3DDw= MFAg&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=3D2Pw2GwelGcMR4953G-= STHGpPJm2-pYYYSPmTwJk3sWM&m=3DSr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnFKUymDoKpI&s= =3DvXqZBKaP1dUovPzwBwC5DalLCB6UxwKuM9x_SQCbw5I&e=3D> > > > > I think this is the direction science is going in now and it would be > great if hearing science joined in. I plan to contact as many hearing > science journals as possible. I'm compiling a list of journals to contact= . > Please add to this list if I'm missing anything: > https://tinyurl.com/yaf9r7bk > <https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__tinyurl.com_yaf9r= 7bk&d=3DDwMFAg&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=3D2Pw2Gwel= GcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DSr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnFK= UymDoKpI&s=3Dsk2rFf3fImx-wI9S05uLc7WYgADb5BupEMAQvL3hz-0&e=3D>. > I don't think any of these journals offer (or are in the process of > offering) registered reports yet, but correct me if I'm wrong. > > > > If you agree that registered reports are a good idea and want to sign the > letter I intend to send (see here for a template: > https://osf.io/3wct2/wiki/Journal%20Requests/ > <https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__osf.io_3wct2_wiki= _Journal-2520Requests_&d=3DDwMFAg&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9= yfotck&r=3D2Pw2GwelGcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DSr0Ep-Gx1c9KJlrg= GBL4rmcUvd9qeDUDnFKUymDoKpI&s=3DG-jhAt3_0f5cPPX7aRpPgVfihZYm_ZTuPohnhVfxWFw= &e=3D>), > let me know and I'll add you to the list. And please spread the word of > course. The more people agree, the more likely it is we can get some of > these journals on board! > > > > Best, > > > Tim Schoof > > > > -- > > Research Associate > > UCL Speech, Hearing and Phonetic Sciences > > > > Chandler House > > 2 Wakefield Street > > London WC1N 1PF > > United Kingdom > > > > > > > --000000000000a75491056df483f0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Dear list,<div> </div><div>Alright, I'll do the gru= mpy old man.</div><div> </div><div>Seeing how little attention people pa= y to the conversation while at the same time feeling entitled to voice an o= pinion is a little bit disconcerting. A striking example is how some decide= d to use a gendered title (Mr) for someone, without even bothering to check= their gender. In fact, even after Gaston offered a correction, people stil= l used the wrong title... (I think it is "beyond ridiculous" that= anyone would use titles on the Auditory List, especially if they're go= nna get it wrong)</div><div> </div><div>If the same lack of care is appl= ied to studying the literature, perhaps indeed, we shouldn't support a = system that could generate even more reports that people won't read. On= the other hand, if the same lack of care is applied to peer-review, then t= he old system is already broken (and perhaps pre-registration would bring a= little bit of fresh air). In fact, if the same lack of care is applied to = reading CVs when recruiting people, forming panels, or choosing speakers to= invite for a conference, for sure we've been doomed for quite a while = now...</div><div> </div><div>Cheers, -Etienne</div><div> </div></d= iv><div class=3D"gmail_extra"> <div><div class=3D"gmail_si= gnature"><div dir=3D"ltr">--=C2=A0<= br style=3D"font-size:12.8px">Etienne Gaud= rain, PhD Centre de Recherche en Neurosciences de= Lyon - CNRS UMR 5292 Universit=C3=A9 Lyon 1 50 av. Tony Garnier 69366 Lyon Cedex 7, FR</s= pan><div> </div><div>UMCG, Afdeling KNO= , BB20 PO Box 30.001 9700 RB Groningen, NL Phone +31 5036 1= 3290 = Skype egaudrain Note: emails to this address are = limited to 10 MB. To send larger files, use <a href=3D"mailto:egaudrain.cam= @xxxxxxxx" target=3D"_blank">egaudrain.cam@xxxxxxxx</a>. </div>= </div></div></div> <div class=3D"gmail_quote">On 5 June 2018 at 16:50, Les Bernstein <span= dir=3D"ltr"><<a href=3D"mailto:lbernstein@xxxxxxxx" target=3D"_blank">l= bernstein@xxxxxxxx</a>> wrote: <blockquote class=3D"gmail_quot= e" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)= ;padding-left:1ex"> =20 =20 =20 <div bgcolor=3D"#FFFFFF"> Dear List, For this topic, I'll violate my rule of not posting replies here.=C2=A0 I agree with Ms. Rankovic.=C2=A0 I sure did not miss th= e substance and detail of Mr. Schoof's email.=C2=A0 I also read o= ver the information in the links.=C2=A0 Indeed, the proposed plan provides for a second review.=C2=A0 It seems to me, however, that t= he provisional acceptance is a key aspect of the process.=C2=A0 If it were the case that manuscripts were rejected upon second review with substantial frequency, then the philosophy of the registered report would be violated and the system would collapse.=C2=A0 So, unless there are egregious errors or flaws in t= he full manuscript, it seems that it would be published.=C2=A0 Note that, in <a href=3D"https://orca.cf.ac.uk/59475/1/AN2.pdf" target= =3D"_blank">this linked reference</a>, publication is assumed to be "guaranteed.&quo= t; In my opinion, the criticism found within the FAQ <a href=3D"https:= //cos.io/rr/" target=3D"_blank">here</a>, that "The Registered Reports model is based on a na=C3=AFve conceptualisation of the scientific method." is well-founded!= =C2=A0 The reply offered to counter that criticism is quite weak and unconvincing.=C2=A0 I would replace "scientific method" i= n that criticism with "the way good science is done." Question 17 in Chambers et al. (2014-- linked above) provides an apt example.=C2=A0 In the process of conducting complex experiments= , it is very often the case that unexpected results lead to important follow-up or control experiments.=C2=A0 Chambers et al. handle this issue by proposing that in Stage 1 of a registered report, contingencies be stated such that "If A is observed, then we will..."=C2=A0 That, of course, assumes that one knows= the decision tree in advance!=C2=A0 In my experience, science simply do= es not work that way. While I find the intent of registered reports to be laudable, in my opinion, it substitutes one potential set of problems with another based on a narrow view of how science proceeds.=C2=A0 Indee= d, one may have a hypothesis to be tested and gather a set of data to address it only to find that the results support a substantially altered view.=C2=A0 Is that, NECESSARILY, the dreaded "HARKing?"=C2=A0 I think not.=C2=A0 Scientific thought an= d inquiry do not always proceed in a linear fashion.=C2=A0 One cannot and should not always know the precise questions or list of contingencies a priori and be restricted to answering only those.=C2=A0 Then there are experiments in which there are no specific hypotheses.=C2=A0 Th= ey may be of the form, "What is the effect of variable A on measurements of X?"=C2=A0 Assuming the question is non-trivial= , those are often the most revealing experiments because any outcome is of interest.=C2=A0 There is no "positive" or "negati= ve."=C2=A0 Sure, one can cast such experiments in terms of hypotheses but doing so often involves a contrivance. Then there is the matter of "p-hacking" and what I would = call "statistics shopping."=C2=A0 Indeed, it is a problem.=C2= =A0 Unexpected outcomes and patterns of data in a complex experiment often require one to choose the appropriate statistic after the fact.=C2= =A0 It is sometimes the correct thing to do!=C2=A0 Whether it is proper can and should be judged by reviewers with the requisite expertise.=C2=A0 Good peer-review should distinguish between p-hacking and a rational choice that conveys information and "truth."=C2=A0 The notion that one can and should use onl= y the statistic decided upon in advance is unnecessary restrictive. Finally, there is the matter of archival value.=C2=A0 According to Chambers et al., "...if the rationale and methods are sound th= en the journal should agree to publish the final paper regardless of the specific outcome."=C2=A0 It is often the case that rati= onale and methods are sound but the data provide no substantial advance or archival value.=C2=A0 I'm not sure that "approv= ing" a method and rationale and virtually guaranteeing publication will afford the same level of judgment in terms of archival value that is afforded by the current system. Les Bernstein -- Leslie R. Bernstein, Ph.D. | </= b>Professor Depts. of Neuroscience and Surgery (Otolaryngology)| UConn School of Medicine 263 Farmington Avenue, Farmington, CT 06030-3401 <a href=3D"https://maps.google.com/?q=3D263+Farm= ington+Avenue,+Farmington,+CT%0D%0A++++++++++06030-3401+%0D%0A+++++++++Offi= ce&entry=3Dgmail&source=3Dg">Office</a>: 860.679.4622 | Fax: 860.679.2495 <img alt=3D"" src=3D"cid:part3.45516EE9.9AF5E615@xxxxxxxx" height= =3D"48" width=3D"125"> <a href=3D"https://urldefense.proofpoint.com/v= 2/url?u=3Dhttps-3A__cos.io_rr_&d=3DDwMFAg&c=3DEZxp_D7cDnouwj5YEFHgX= uSKoUq2zVQZ_7Fw9yfotck&r=3D2Pw2GwelGcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&= amp;m=3DSr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnFKUymDoKpI&s=3DvXqZBKaP1dUovPz= wBwC5DalLCB6UxwKuM9x_SQCbw5I&e=3D" id=3D"gmail-m_759740279723453573LPln= k151566" target=3D"_blank"></a> <div class=3D"gmail-m_759740279723453573moz-cite-prefix">On 6/4/2018 7:= 51 AM, Christine Rankovic wrote: </div> <blockquote type=3D"cite"> =20 =20 =20 <div class=3D"gmail-m_759740279723453573WordSection1"> <div> <div style=3D"border-right:none;border-bottom:none;border-left:no= ne;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in"> Mr. Schoof: =C2=A0 </div> </div> It is beyond ridiculous to accept partial manuscripts for publication. Christine Rankovic, PhD Scientist, Speech and Hearing Newton, MA=C2=A0 USA <a class=3D"gmail-m_7597402797234535= 73moz-txt-link-abbreviated" href=3D"mailto:rankovic@xxxxxxxx" targe= t=3D"_blank">rankovic@xxxxxxxx</a> =C2=A0 =C2=A0 From: AUDITORY - Research in Auditory Perception [<a class=3D"gmail-m_759740279723453573moz-txt-link-freetext" h= ref=3D"mailto:AUDITORY@xxxxxxxx" target=3D"_blank">mailto:AUDITORY@xxxxxxxx= ISTS.MCGILL.CA</a>] On Behalf Of Schoof, Tim Sent: Monday, June 04, 2018 4:06 AM To: <a class=3D"gmail-m_759740279723453573moz-txt-link-a= bbreviated" href=3D"mailto:AUDITORY@xxxxxxxx" target=3D"_blank">AUDI= TORY@xxxxxxxx</a> Subject: Registered reports =C2=A0 <div id=3D"gmail-m_759740279723453573divtagdefaultwrapper"> <div> D= ear list, <= u>=C2=A0 I= 'm going to try and get hearing science journals to start offering registered reports. These reports are basically peer-reviewed pre-registration documents where you outline your methods and proposed analyses. If this document makes it through peer-review, the manuscript is provisionally accepted for publication. This process should reduce certain questionable research practices, such as=C2=A0selective reporting of results and publication bias. If you're sceptical about registered reports,=C2= =A0the Center for Open Science has compiled a nice FAQ list that might address some of your concerns: <a href=3D"https:= //urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__cos.io_rr_&d=3DDwMFAg&= amp;c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=3D2Pw2GwelGcMR495= 3G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DSr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnFKUym= DoKpI&s=3DvXqZBKaP1dUovPzwBwC5DalLCB6UxwKuM9x_SQCbw5I&e=3D" id=3D"g= mail-m_759740279723453573LPlnk151566" target=3D"_blank">https://cos.io/rr/<= /a> <= u>=C2=A0 I think this is the direction science is going in now and it would be great if hearing science joined in. I plan to contact as many hearing science journals as possible. I'm compiling a list of journals to contact. Please add to this list if I'm missing anything: <a href=3D"https:= //urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__tinyurl.com_yaf9r7bk&d= =3DDwMFAg&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=3D2Pw2G= welGcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DSr0Ep-Gx1c9KJlrgGBL4rmcUvd9q= eDUDnFKUymDoKpI&s=3Dsk2rFf3fImx-wI9S05uLc7WYgADb5BupEMAQvL3hz-0&e= =3D" id=3D"gmail-m_759740279723453573LPlnk940137" target=3D"_blank">https:/= /tinyurl.com/yaf9r7bk</a>. I don't think any of these journals offer (or are in th= e process of offering) registered reports yet, but correct me if I'm wrong. <= u>=C2=A0 I= f you agree that registered reports are a good idea and want to sign the letter I intend to send (see here for a template: <a href=3D"https://urldefense.proofpoint.com/v2/u= rl?u=3Dhttps-3A__osf.io_3wct2_wiki_Journal-2520Requests_&d=3DDwMFAg&amp= ;c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=3D2Pw2GwelGcMR4953G-= STHGpPJm2-pYYYSPmTwJk3sWM&m=3DSr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnFKUymDoK= pI&s=3DG-jhAt3_0f5cPPX7aRpPgVfihZYm_ZTuPohnhVfxWFw&e=3D" id=3D"gmai= l-m_759740279723453573LPlnk473363" target=3D"_blank">https://osf.io/3wct2/w= iki/Journal%20Requests/</a>), let me know and I'll add you to the list. And please spread the word of course. The more people agree, the more likely it is we can get some of these journals on board! </div> <div> =C2=A0 </div> <div> Best, </div> <div> Tim Schoof </div> <div> =C2=A0 </div> <div> -- </div> <div> Research Associate </div> <div> UCL Speech, Hearing and Phonetic Sciences </div> <div> =C2=A0 </div> <div> Chandler House </div> <div> 2 Wakefield Street </div> <div> London WC1N 1PF </div> <div> United Kingdom </div> =C2=A0 </div> </div> </blockquote> <div class=3D"gmail-m_759740279723453573moz-signature"> </div> </div> </blockquote></div> </div> --000000000000a75491056df483f0-- --000000000000a75493056df483f1 Content-Type: image/png; name="uconnhealth_stacked_blue_email.png" Content-Disposition: inline; filename="uconnhealth_stacked_blue_email.png" Content-Transfer-Encoding: base64 Content-ID: <part3.45516EE9.9AF5E615@xxxxxxxx> X-Attachment-Id: a3c08956ed090430_0.1 iVBORw0KGgoAAAANSUhEUgAAAH0AAAAwCAMAAAALmIWlAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJ bWFnZVJlYWR5ccllPAAAADNQTFRFKzVYHCZM4eLn8PHzpKi30dTbaG+IlZqrs7fDWWB8O0Nkd32U wsXPSlJwhoufDRhA////A68jmAAAABF0Uk5T/////////////////////wAlrZliAAACYklEQVR4 2uzY22KDIAwAUC7e6gjw/187gUC4OetaupflaTXKWSuBKNMuJptC+QMCP20auPHBQCubBWU46K24 uB7NfZ58ZqaMP8D8AEBHtT/goXnB8WPwBf8r1WYUXWxUOZobHcJXIMcfONclmE4s0lrRzYBIOjvT jX5WV6YbP2ZQJ6TWjfiAbuYzHW7p+xpuolQrP83sjQ5nupnu6LqaJ92MbnSznulcfkBHpNXNY6y+ c0JKnWX1eFeflQt5qcNESKnrQMrf6LSAXeiWkEoXdOU4nZBKxzPnoTohtW5ZZyK8W09Io6tYjyP1 iDS6XbAeh815S0irS6zHUfXuB0Ok1S3W41AdEWh1/Nb7UD1bWytd0LY4ThdnerYbBJ1taT5XBmy4 J8mNnWagp5fbba6HeiTdneK6vNgoFj0EhyO4eSZT6AlpdNXoNzqYi94GbIU0eqhHp3d5drRf4heZ Uo9Iq8vYptup6VB37H6m/W6m0hFpdaxH3/yK9cHpVq5Zy59n2GMV/QxPGdV9NElPE9njyOozsfWW Kq6j7bLuw97KPBnM/mX86//6n+tH/yLKP7AUXcyd80Lx53WnspB47ty5qKND1gLockfIFzHR7srp Q7X0+6GgHPNFXedPyaN1CPFli71z+ZBeTxbcI2RXxwOQj/lW/Ui610Z6gA4a32WRrssXYbO76V/Z y6FLnZVj/qCnqGcdZE1DeFKcntWrMV/RZXCh7B/fpF/+8tQQx6J74y9/Oeuyrm75+Jyf8oKWH9CL 1SYNJtJ91HTOAD2fdjPN9UcsumwmDNYXqvPtOLa9pH8LMACnoV0siZAyOAAAAABJRU5ErkJggg== --000000000000a75493056df483f1--

This message came from the mail archive
src/postings/2018/
maintained by:

DAn Ellis <dpwe@ee.columbia.edu>
Electrical Engineering Dept., Columbia University