Skip to content Skip to navigation

Journal Papers

Comissioned for the Simons Foundation's Simons Collaboration on the Global Brain (SCGB), in association with our Computation Through Dynamics (CTD) research using our  Dynamical Systems Framework (DSF). Illustration credit: Islenia Mil.

28. Willett FR, Avansino DT, Hochberg LR, Henderson JM*, Shenoy KV* (2021) High-performance brain-to-text communication via imagined handwriting. Nature. 593:249-254.

A single pdf

  • A “single pdf file with everything” below, except with only Stanford, HHMI, Nature Podcast, Science & NPR articles.
    • Full size (24 MB) pdf
    • Reduced size (8 MB) pdf

News & Views

  • Rajeswaran P, Orsborn AL (2021) Neural interface translates thoughts into type. News & Views. Nature. 593:197-198. pdf url

Publication materials

  • Main paper. pdf url
  • Supplementary material. pdf url
  • Peer review file. pdf url
  • Video
    • Video 1: Copying sentences in real-time with the handwriting brain-computer interface. In this video, participant T5 copies sentences displayed on a computer monitor with the handwriting-brain computer interface. When the red square on the monitor turns green, this cues T5 to begin copying the sentence. url
    • Video 2: Hand micromotion while using the handwriting brain-computer interface. Participant T5 is paralyzed from the neck down (C4 ASIA C spinal cord injury) and only generates small micromotions of the hand when attempting to handwrite. T5 retains no useful hand function. url
    • Video 3: Freely answering questions in real-time with the handwriting brain-computer interface. In this video, participant T5 answers questions that appear on a computer monitor using the handwriting brain-computer interface. T5 was instructed to take as much time as he wanted to formulate an answer, and then to write it as quickly as possible. url
    • Video 4: Side-by-side comparison between the handwriting brain-computer interface and the prior state of the art for intracortical brain-computer interfaces. In a prior study (Pandarinath et al., 2017) participant T5 achieved the highest typing speed ever reported with an intracortical brain-computer interface (39 correct characters per minute using a point-and-click typing system). Here, we show an example sentence typed by T5 using the point-and-click system (shown on the bottom) and the new handwriting brain-computer interface (shown on the top), which is more than twice as fast. url

Shared resources

  • Shenoy KV, Willett FR, Nuyujukian P, Henderson JM (2021) Performance considerations for general-purpose typing BCIs, including the handwriting BCI. Technical Report #01, Version 2.7. Stanford Digital Repository (SDR), Stanford University. url
  • Willett FR, Avansino DT, Hochberg LR, Henderson* JM, Shenoy* KV (2021) DataDryad: All electrophysiology data reported in Willett et al. Nature 2021. url
  • Willett FR, Avansino DT, Hochberg LR, Henderson* JM, Shenoy* KV (2021) GitHub: All code written and used in Willett et al. Nature 2021. url

News coverage (selected)

  • Altmetric score (> 3,940). url
  • Bundell S (14 May 2021) Nature Podcast. transcript url
  • Servick K (13 May 2021) Paralyzed person types at record speed -- by imagining handwriting. Science. url
    • K Servick (23 Oct 2019) AI allows paralyzed person to 'handwrite' with his mind. Science. pdf url
  • News (Stanford & HHMI)
    • Rosen M (12 May 20201) Brain computer interface turns mental handwriting into text on screen. Howard Hughes Medical Institute (HHMI). News article. pdf url
      • Overview video (1:40 minutes). url
    • Goldman B (12 May 2021) Software turns 'mental handwriting' into on-screen words, sentences. Stanford University. News article. pdf url
    • Weiler N, Toth A (12 My 2021) Eavesdropping on brain activity turns imagined handwriting to text. Wu Tsai Neurosciences Institute, Stanford University. url
      • Overview video 1 (2:40 minutes). url
    • Weiler N, Toth A (12 May 2021) Science in Brief: Decoding Text from Brain Activity via Imagined Handwriting. Wu Tsai Neurosciences Institute, Stanford University. url
      • Overview video 2 (3:19 minutes). url
  • News (selected)
    • Hamilton J (12 May 2021) Man who is paralyzed communicates by imagining handwriting. All Things Considered. National Public Radio (NPR). pdf url
      • Audio (3:25 minutes). mp3
      • Transcript. pdf
    • Stetka B (12 May 2021) New brain implant turns visualized letters into text. Scientific American. url
    • Timmer J (12 May 2021) Neural implant lets paralyzed person type by imagining writing. arsTECHNICA. url
    • Rodriguez A (14 May 2021) After researchers implanted microchips into his brain, a paralyzed man was able to write with his mind. USA Today. pdf url
    • Computer deciphers brain signals of imagined writing. ABC. url
    • Paralyzed man uses his mind to form real-time sentences. CNN. url
    • Brain-computer interface allows paralysed man to write again. BBC Science Focus / PA. url
    • Paralysed man uses 'mindwriting' brain computer to compose sentences. The Guardian. url
    • The Times view on a breakthrough for paralysis: Positive thinking. The Times. url
    • Brain device translates thoughts directly onto a computer. The Independent. url
    • How to type by just thinking: Microchip inserted in the brain turns thoughts into text. MailOnline. url
    • Mind over matter: brain chip allows paralysed man to write. AFP. url
    • Dos chips en el cerebro permiten escribir con la mente. El Pais. url
    • Mindwriting, il software che trasforma i pensieri in parole. La Repubblica. url
    • Mind over matter: brain chip allows paralysed man to write. The Hindu. url
    • Research filter: Mindwriting and what do our ancient poos reveal about our gut bacteria? ABC Aus. url
    • Neurotechnologie Maschinen können jetzt gedanken lesen. Spiegel. url
    • Handschriftlich kommunizieren mit gedankenkraft. Spektrum. url
    • Un paralítico envía mensajes a un récord de 16 palabras por minuto. Gizmodo. url
    • Mental handwriting produces brain activity turned Into text. Neuroscience News. url
    • AI lets man with paralysis type by just thinking about handwriting. New Scientist. url
    • New brain-computer interface translates handwritten thoughts into text for paralysis patients. IFLS. url
    • Brain implants turn imagined handwriting into text on a screen. Science News. url
    • New device allows man with paralysis to type by imagining handwriting. Smithsonian. url
    • Brain-computer interface user types 90 characters per minute with mind. The Scientist. url
    • Paralysed man ‘handwrites’ with brain chip. Cosmos. url
    • “Mindwriting” technique helps paralyzed patient use brain activity to write. Technology Networks. url
    • Computer deciphers the brain signals of imagined writing. Inside Science. url
    • Implanted sensor translates brain signals Into text. Medpage Today. url
    • Created an interface for typing with the power of thought. Forbes. url

27. Deo DR, Rezaii P, Hochberg LR,  Okamura AM, Shenoy KV*,  Henderson JM* (2021) Effects of peripheral haptic feedback on Intracortical brain-computer interface control and associated sensory responses in motor cortex. IEEE Transactions on Hapticspdf url


26. Simeral JD, Hosman T, Saab J, Flesher SN, Vilela M, Franco B, Kelemen J, Brandman DM, Ciancibello JG, Rezaii PG, Rosler DM, Shenoy KV**, Henderson JM**, Nurmikko AV, Hochberg LR (2021) Home use of a wireless intracortical brain-computer interface by individuals with tetraplegia. IEEE Transactions in Biomedical Engineering. DOI 10.1109/TBME.2021.3069119. pdf url


25. Rastogi A, Willett FR, Abreu J, Crowder DC, Murphy B, Memberg WD, Vargas-Irwin CE, Miller JP, Sweet J, Walter BL, Rezaii PG, Stavisky SD, Hochberg LR, Shenoy KV, Henderson JM, Kirsch RF, Bolu Ajiboye A (2021) The neural representation of force across grasp types in motor cortex of humans with tetraplegia. eNeuro10.1523/ENEURO.0231-20.2020. pdf url


24. Wilson GH*, Stavisky SD*, Willett FR, Avansino DT, Kelemen JN, Hochberg LR, Henderson JM**, Druckmann S,** Shenoy KV** (2020) Decoding spoken English phonemes from intracortical electrode arrays in dorsal precentral gyrus. Journal of Neural Engineering. 17:066007 pdf  url


23. Even-Chen* N,  Muratore* DG, Stavisky SD, Hochberg LR, Henderson JM, Murmann** B, Shenoy** KV (2020) Power-saving design opportunities for wireless intracortical brain-computer interfaces. Nature Biomedical Engineering. 4:984-996. pdf supp_mats url

  • Editorial (2020) The painstaking pace of bioelectronic interfaces. Nature Biomedical Engineering. 4:933–934. pdf url
  • News & Views: Slutzky MW (2020) Increasing power efficiency. Nature Biomedical Engineering. 4:937–938 pdf url​
  • Associated paper: Nason SR, Vaskov AK, Willsey MS, Welle EJ, An H, Vu PP, Bullard AJ, Nu CS, Kao JC, Shenoy KV, Jang T, Kim H-S, Blaauw D, Patil PG, Chestek CA (2020) A low-power band of neuronal spiking activity dominated by local single units improves the performance of brain-machine interfaces. Nature Biomedical Engineering. 4:973–983. pdf url

22. Willett FR*, Deo DR*, Avansino DT, Rezaii Paymon, Hochberg LR, Henderson JM**, Shenoy KV** (2020) Hand knob area of motor cortex in people with tetraplegia represents the whole body in a compositional way. Cell181:396–409. pdf url
21. Stavisky SD, Willett FR, Avansino DT, Hochberg LR, Shenoy KV**, Henderson JM** (2020) Speech-related dorsal motor cortex activity does not interfere with iBCI cursor control. Journal of Neural Engineering. 17:016049 (13pp). pdf url
20. Rastogi A, Vargas-Irwin C, Willett F, Abreu J, Crowder DC, Murphy B, Memberg W, Miller J, Sweet J, Walter B, Cash S, Rezaii P, Franco B, Saab J, Stavisky SD, Shenoy KV**, Henderson J**, Hochberg LR, Kirsch R, Ajiboye AB (2020) Neural representation of observed, imagined, and attempted grasping force in motor cortex of individuals with chronic tetraplegia. Scientific Reports. 10:1429. pdf url
19. Stavisky SD,  Willett FR, Wilson GH, Murphy BA, Rezaii P, Avansino D, Memberg WD, Miller JP, Kirsch RF, Hochberg LR, Ajiboye AB, Druckmann S, Shenoy KV**, Henderson JM** (2019) Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis. eLife. 8:e46015. pdf figure-supplements url
18. Willett FR, Young DR, Murphy BA, Memberg WD, Blabe CH, Pandarinath C, Stavisky SD, Rezaii P, Saab J, Walter BL, Sweet JA, Miller JP, Henderson JM, Shenoy KV, Simeral J, Jarosiewicz B, Hochberg LR, Kirsch RF, Ajiboye AB (2019) Principled BCI decoder design and parameter selection using a feedback control model. Scientific Reports. 9(1):8881. pdf supp mats url
17. Milekovic T, Bacher D, Sarma A, Simeral J, Saab J, Pandarinath C, Yvert B, Sorice B, Blabe C, Oakley E, Tringale K, Eskandar E, Cash S, Shenoy KV, Henderson JM, Hochberg LR, Donoghue JP (2019) Volitional control of single-electrode high gamma local field potentials (LFPs) by people with paralysis. Journal of Neurophysiology. 121:1428-1450. pdf url
16. Young D, Willett F, Memberg W, Murphy B, Rezaii P, Walter B, Sweet J, Miller J, Shenoy KV, Hochberg LR, Kirsch R, Ajiboye AB (2019) Closed-loop cortical control of virtual reach and posture using cartesian and joint velocity commands. Journal of Neural Engineering. 16:026011 (14pp). pdf url
15. Nuyujukian P*, Sanabria JA*, Saab J*, Pandarinath C, Jarosiewicz B, Blabe C, Franco B, Mernoff ST, Eskandar EN, Simeral JD, Hochberg LR**, Shenoy KV**, Henderson JM** (2018) Cortical control of a tablet computer by people with paralysis. PLoS One. 13:e0204566 pdf url-with-all-movies

14. Stavisky SD, Kao JC, Nuyujukian P, Pandarinath C, Blabe C, Ryu SI, Hochberg LR, Henderson JM, Shenoy KV (2018) Brain-machine interface cursor position only weakly affects monkey and human motor cortical activity in the absence of arm movements. Scientific Reports. 8:1635.7 pd


13. Pandarinath C, O'Shea DJ, Collins J, Jozefowicz R, Stavisky SD, Kao JC, Trautmann EM, Kaufman MT, Ryu SI, Hochberg LR, Henderson JM, Shenoy KV, Abbott LF, Sussillo D (2018) Inferring single-trial neural population dynamics using sequential auto-encoders. Nature Methods.15:805-815. pdf  supp-mats-text-videos  url

SuppVideo1 Generator initial states inferred by LFADS are organized with respect to kinematics of the upcoming reach. The video depicts the initial conditions vectors for each individual trial of the ‘Maze’ reaching task for monkey J, mapped onto a low-dimensional space (3D) via t-SNE (as in Fig. 2c). Each point represents the initial conditions vector for an individual trial (2,296 trials are shown). Colors denote the angle of the endpoint of the upcoming reach (colors shown in Fig. 2a), and marker types denote the curvature of the reach (circles, squares, and triangles for straight, counter-clockwise curved, and clockwise curved reaches, respectively). As shown, the initial conditions exhibit similarity for trials with similar kinematic trajectories (both for trials whose reach endpoints have similar angles and for trials with similar reach curvature). Since structure in the initial conditions implies structure at the level of the generator’s dynamics, this analysis implies that LFADS produces dynamic trajectories that show similarity based on the kinematics of the reach type for a given trial, despite LFADS not having any information about reaching conditions.

SuppVideo2 LFADS reveals consistent rotational dynamics on individual trials. The video contains two sequential movies showing the trajectories in neural population state space during individual reach trials for monkey J (Fig. 3). The first movie illustrates the single-trial trajectories uncovered by smoothing the data with a Gaussian kernel. The second movie illustrates single-trial trajectories uncovered by LFADS. 2,296 trials are shown, representing the 108 conditions of the ‘Maze’ task.

SuppVideo3 Multisession LFADS finds consistent representations for individual trials across sessions. The video contains six sequential movies showing the trajectories in state space during individual reach trials for monkey P (Fig. 4). The first video shows single-trial GPFA factor trajectories for all trials estimated for a single session. The second and third videos show single-trial LFADS factor trajectories estimated from all trials using a single-session model and from the stitched model, respectively. The fourth, fifth, and sixth repeat this sequence but show single-trial trajectories for 42/44 sessions (2 were omitted for ease of presentation). Colors represent eight reach directions. Multisession movies include approximately 14,500 trials, 38 separate electrode penetration sites and spanned 162 d from the first to the last session. Each trajectory begins at the go cue and proceeds for 510 ms into movement, which occurs at varying times due to reaction time variability. For GPFA and single-session. LFADS factors, the trajectories from individual sessions were concatenated and the projection yielding the CIS and first jPCA plane was estimated (Methods). This provided a set of common trajectories against which each individual session’s data were regressed. These regression coefficients provided projections of the individual sessions’ trajectories that were maximally similar to the common trajectories. In contrast, for stitched LFADS factors, we simply estimated the projection yielding the CIS and first jPCA plane from all of the sessions together, as the factors are already in shared space.

Code: LFADS-run-manager.zip  LFADS.zip

Data: SuppData1.zip  SuppData2.zip  SuppData3.zip

News & Views: Batista AP, DiCarlo JJ (2018) Deep learning reaches the motor system. Nature Methods. News & Views. 15:772-773. pdf​

Seminar talk: Sussillo D (3/222018) LFADS seminar talk, Simons Institute for the Theory of Computing meeting, UC Berkeley. video


12. Milekovic T, Sarma A, Bacher D, Simeral J, Saab J, Pandarinath C, Sorice B, Blabe C, Oakley E, Tringale K, Eskandar E, Cash S, Henderson JM, Shenoy KV, Donoghue JP, Hochberg LR (2018) Stable long-term BCI-enabled communication in ALS and locked-in syndrome using LFP signals. Journal of Neurophysiology. 120:343-360. pdf url


11. Willett FR, Murphy BA, Young D, Memberg WD, Blabe CH, Pandarinath C, Franco B, Saab J, Walter BL, Sweet JA, Miller JP, Henderson JM, Shenoy KV, Simeral JD, Jarosiewicz B, Hochberg LR, Kirsch RF, Ajiboye AB (2018) A comparison of intention estimation methods for decoder calibration in intracortical brain-computer interfaces. IEEE Transactions in Biomedical Engineering. 65:2066-2078. pdf url


10. Even-Chen N, Stavisky SD, Pandarinath C, Nuyujukian P, Blabe CH, Hochberg LR, Henderson* JM, Shenoy* KV (2018) Feasibility of automatic error detect-and-undo system in human intracortical brain-computer interfaces. IEEE Transactions in Biomedical Engineering. 65:1771-1784. pdf url


9. Brandman D, Hosman T, Saab J, Burkhart M, Shanahan B, Ciancibello J, Sarma A, Milstein D, Vargas-Irwin C, Franco B, Kelemen J, Blabe C, Murphy B, Young D, Willett F, Pandarinath C, Stavisky S, Kirsch R, Walter B, Ajiboye A, Cash S, Eskandar E, Miller J, Sweet J, Shenoy KV, Henderson JM, Jarosiewicz B, 9. Harrison M, Simeral J, Hochberg, LR (2018) Rapid calibration of an intracortical brain computer interface for people with tetraplegia. Journal of Neural Engineering. 15:026007. pdf url


8. Pandarinath C*, Nuyujukian P*, Blabe CH, Sorice B, Saab J, Willett F, Hochberg LR, Shenoy KV**, Henderson JM** (2017) High performance communication by people with paralysis using an intracortical brain-computer interface. eLife. 6:e18554 pdf url-with-all-movies


7. Willett FR, Murphy B, Memberg W, Blabe C, Pandarinath C, Walter B, Sweet J, Miller J, Henderson JM, Shenoy KV, Hochberg LR, Kirsch R, Ajiboye AB (2017) Signal-independent noise in intracortical brain-computer interfaces causes movement time properties inconsistent with Fitts' law. Journal of Neural Engineering14:026010. pdf


6. Willett F, Pandarinath C, Jarosiewicz B, Murphy B, Memberg W, Blabe C, Saab J, Walter B, Sweet J, Miller J, Henderson J, Shenoy KV, Simeral J, Hochberg LR, Kirsch R, Ajiboye AB. (2017) Feedback control policies employed by people using intracortical brain-computer interfaces. Journal of Neural Engineering. 14:016001 (16pp). pdf


5. Jarosiewicz B, Sarma AA, Bacher D, Masse NY, Simeral JD, Sorice B, Oakley EM, Blabe C, Pandarinath C, Gilja V, Cash SS, Eskandar E, Friehs G, Henderson JM, Shenoy KV, Donoghue JP, Hochberg LR (2015) Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface. Science Translational Medicine7:1-10. pdf supplement video1


4. Gilja V*, Pandarinath C*, Blabe CH, Nuyujukian P, Simeral JD, Sarma AA, Sorice BL, Perge JA, Jarosiewicz B, Hochberg LR, Shenoy KV**, Henderson JM** (2015) Clinical translation of a high performance neural prosthesis. Nature Medicine. 21:1142-1145. pdf supplement video1 video2 video3


3. Blabe C, Gilja V, Chestek CA, Shenoy KV, Anderson K, Henderson JM (2015) Assessment of brain-machine interfaces from the perspective of people with paralysis. Journal of Neural Engineering12:043002. pdf


2. Pandarinath C, Gilja V, Blabe CH, Nuyujukian P, Sarma AA, Sorice BL, Eskandar EN, Hochberg LR, Henderson JM*, Shenoy KV* (2015) Neural population dynamics in human motor cortex during movements in people with ALS. eLife4:e07436. pdf video1


1. Chestek CA, Gilja V, Blabe CH, Foster BL, Shenoy KV, Parvizi J, Henderson JM (2013) Hand posture classification using electrocorticography signals in the gamma band over human sensorimotor brain areas. Journal of Neural Engineering10:02602 (11pp). pdf

Recent conference papers and abstracts (other than 1 paragraph Soc. for Neurosci. abstracts)

5. Willett FR, Avansino DT, Hochberg LR, Henderson JM**, Shenoy KV** (2020) High-performance brain-to-text communication via imagined handwriting. 1st BCI Un-Conference, Session 3: Language-based BCI-Communication. Talk 8. 12 July 2020. Video starts at 0min 15sec: url-with-video video.mp4

  • Abstract: To date, a major focus of BCI research has been on restoring gross motor skills, such as reaching and grasping or point-and-click typing with a 2D computer cursor. However, rapid sequences of highly dexterous behaviors, such as handwriting or touch typing, might enable faster communication rates. Here, we demonstrate an intracortical BCI that can decode imagined handwriting movements from neural activity in motor cortex and translate it to text in real-time, using a novel recurrent neural network decoding approach. With this BCI, our study participant (whose hand was paralyzed) achieved typing speeds that exceed those of any other BCI yet reported: 90 characters per minute at >99% accuracy with a general-purpose autocorrect. These speeds are comparable to able-bodied smartphone typing speeds in our participant’s age group (115 characters per minute) and significantly close the gap between BCI-enabled typing and able-bodied typing rates. Finally, new theoretical considerations explain why temporally complex movements, such as handwriting, may be fundamentally easier to decode than point-to-point movements. Our results open a new approach for BCIs and demonstrate the feasibility of accurately decoding rapid, dexterous movements years after paralysis.

4. Stavisky SD*, Wilson GH*, Willett FR, Avansino DT, Kelemen JN, Hochberg LR, Henderson JM**, Druckmann S,** Shenoy KV** (2020) Decoding speech production using intracortical electrode arrays in dorsal precentral gyrus. 1st BCI Un-Conference, Session 3: Language-based BCI-Communication, Talk 9. 12 July 2020. Video starts at 19 min 42 sec: url-with-video video.mp4

  • Abstract: Efforts to build brain-computer interfaces (BCIs) to restore lost speech have rapidly accelerated in recent years, with a number of impressive demonstrations using electrocorticography (ECoG). In parallel, BCIs that decode attempted arm and hand movements for controlling a robotic arm and point-and-click typing communication have had the highest performance to date using intracortical electrode arrays. We believe that an intracortical approach is equally promising for speech BCIs and can complement ECoG recordings. However, progress has been slower in this domain due to limited animal models for speech, leading to the need for preliminary work to be done in humans. We have recently made progress prototyping an intracortical speech BCI thanks to our discovery that individual neurons in the dorsal “hand knob” area of the precentral gyrus are active during speech (Stavisky et al., 2019, 2020). Here, we recorded from a BrainGate2 pilot clinical trial participant, who has tetraplegia but can speak, as he spoke visually prompted words that broadly sample a comprehensive set of 39 English phonemes. Phoneme identities could be classified from the neural activity recorded by two 96-electrode arrays with 33.9% accuracy using a recurrent neural network decoder. Performance did not saturate with increasing training data quantity or electrode count. By demonstrating offline performance comparable to previous ECoG studies – despite these arrays being in an area that is very likely suboptimal for speech decoding – these results help lay the groundwork for a concerted speech BCI effort using intracortical measurements from more ventral cortical areas which we expect to have stronger speech-related modulation.

3. Wilson G*, Stavisky SD*, Avansino D, Hochberg LR, Henderson JM**, Shenoy KV**, Druckmann S** (2020) Neural state space geometry in human motor cortex underlying speaking different phonemes. Frontiers in Neuroscience. Conference Abstract: Computational and Systems Neuroscience (COSYNE), Denver, CO. Poster number III-71. pdf


2. Willett F, Avansino D, Hochberg LR, Henderson JM**, Shenoy KV** (2020) Motor Cortical Representation and Decoding of Attempted Handwriting in a Person with Tetraplegia. Frontiers in Neuroscience. Conference Abstract: Computational and Systems Neuroscience (COSYNE), Denver, CO. Poster number III-1. pdf
1. Stavisky SD, Rezaii P, Willett FR, Hochberg LR, Shenoy KV*, Henderson JM* (2018) Decoding speech from intracortical multielectrode arrays in dorsal "arm / hand areas" of human motor cortex. 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Honolulu, HI: 93-97. pdf

 

Last modified: 
Sunday, June 6, 2021 - 20:33