# Decreasing-Rate Pruning Optimizes the Construction of Efficient and Robust Distributed Networks

> An annotated edition of this paper on Fermat's Library, where scientists and readers explain its ideas, equations and history in the margins. Annotated edition: https://fermatslibrary.com/p/60c7a6b8

- **Authors:** Saket Navlakha, Alison L. Barth and Ziv Bar-Joseph
- **Topic:** Biology
- **Original:** https://s3-us-west-1.amazonaws.com/fermatslibrary/uploads%2F1457975954774-98hmvlpfm8s8xgvi-cbe8ce7a0bdb3c243b7d7a8d602db7d7%2Fpcbi.1004347.pdf
- **Annotations:** 34 comments in 24 threads
- **Annotated edition:** https://fermatslibrary.com/p/60c7a6b8

## Annotations

### Annotation 1 (the paper)

**Annotation** by ncMUFCMU (2024-08-23):

> 1

**Reply** by ncMUFCMU (2024-08-23):

> 1

**Reply** by ncMUFCMU (2024-08-23):

> 1

**Reply** by ncMUFCMU (2024-08-23):

> 1

**Reply** by ncMUFCMU (2024-08-23):

> 1

**Reply** by ncMUFCMU (2024-08-23):

> 1

**Reply** by ncMUFCMU (2024-08-23):

> 1

**Reply** by ncMUFCMU (2024-08-23):

> 1

**Reply** by ncMUFCMU (2024-08-23):

> 1

**Reply** by ncMUFCMU (2024-08-23):

> 1

### Annotation 2 (the paper)

**Annotation** by Noel Lau  (2016-03-14):

> The structure and components of our neural systems affect many functions, such as memory and learning. Studying neural plasticity, including the mechanisms governing synaptic density, such as synaptic sprouting and pruning, is essential to understanding the pathology of diseases and disorders such as Alzheimer's disease and schizophrenia.
>
> You can read more about this specific application here:
>
> http://www.ncbi.nlm.nih.gov/pubmed/7932285.
>
> You can also listen to this podcast, featuring a brief interview with Maggie Koerth-Baker and Joseph Stromberg, for a brief introduction, here:
>
> http://www.sciencefriday.com/segments/roots-of-schizophrenia-zebra-stripes-wind-chill/
>
>
> In these studies, we are also introduced to the occurrence of over-pruning.
>
> Though it seems counterproductive to overproduce, scientists have observed that inactive synapses, or connections that are no longer used in later stages in life, are the ones being removed. While artificial systems are created with the intent to have optimal performance, human neural systems have to actively select and develop themselves to reach that level.

### Annotation 3 (the paper)

**Annotation** by Noel Lau  (2016-03-18):

> One of the main goals of this paper is to identify characteristics in the natural pruning process that make the network more efficient (described as minimizing time and distance from point to point in the network), and robust ( described as "number of alternative routes between two nodes"), to apply to engineered networks, (such as the airline networks mentioned earlier) for performance optimization.
>
> Here is a diagram of nerve cells and how synaptic pruning allows for more effective distribution of activity (See figure D of the below image).
> ![Synaptic Pruning]( http://english.sibs.cas.cn/ns/es/201508/W020150807524629444734.jpg
> "Synaptic Pruning")
>
> ![Detail of the Synapse]( https://minakinwrites.files.wordpress.com/2012/01/48_05neuronstructure.jpg "Diagram of Nerve Cell")

### Annotation 4 (the paper)

**Annotation** by Noel Lau  (2016-03-14):

> The body can determine what it needs and through selective gene expression, produce components such as ligands (signaling proteins) to alter certain pathways and regulate a function. For example, one of the molecular mechanisms that governs pruning is observed to be the NMDA receptors, which is also involved as a ligand-gated channel at the synpase, embedded in the nerve cell membrane. It requires two different molecules to attach, to cause a conformational change that allows the flow of calcium ions into the cell, causing a change in membrane potential (voltage) and allowing for LTP or LTD (long-term potentiation, or long-term depression, respectively). These are processes that can change synaptic strength, which affects communication between nerves.
> You can read more about the role of NMDA receptors in neural plasticity here: http://www.biolbull.org/content/219/2/81.long#ref-160
>
> ![Synapse Chemistry]( https://o.quizlet.com/2q4iLfTL6exBkK9BOV6VPw_m.png
> "Synapse Chemistry")

### Annotation 5 (the paper)

**Annotation** by Noel Lau  (2016-03-18):

> The study monitors synapse density over time in an area responsible for sensory details (touch) from a specific whisker.
> From the image, B.) is a visual of spines on a dendrite, which are like distinct ridges all over its surface, each receiving a signal from a particular axon. D.) further explains how pruning can increase efficiency and robustness of nerve networks, by selecting out parts that are no longer used.
>
> Depending on the strain, mice will typically reach sexual maturity at 6 weeks for females, 8 weeks for males. Data was collected in the developmental stages of the mice, as pruning rate and synapse density start to stabilize after intense pruning.

### Annotation 6 (the paper)

**Annotation** by Noel Lau  (2016-03-18):

> To get an idea of the precision of the data, the team used a method of statistical analysis called leave-one-out-cross-validation, which is exactly what it sounds like. The data was separated into 12 groups (i.e. P14 only, P17 only, etc.) from specific postnatal days, and one time-point was removed. The pruning rate was recalculated without that point to determine if it significantly changes the observed pruning rate.
> Towards the end of the trial, at P33, the increase in pruning (by a relatively small amount of 7%) was attributed to the growth of nerve branches from peripheral regions, into the monitored layer 4, though the region was chosen as a good model for observation because it tends to remain undisturbed from activity from other cortical layers.

### Annotation 7 (the paper)

**Annotation** by Noel Lau  (2016-05-06):

> We know that flipping a coin will either give us heads, or tails, and the chance of either is 50%, even before we flip the coin. The distribution is known. If we are unsure of the distribution, we can run iterations of the coin flip. Maybe the coin is biased such that it will turn up heads 75% of the time. This is what we discover from experimental data; the distribution was unknown. The distribution doesn’t mean the probability of each outcome, but is more like a set of influential traits at a particular time.
>
> We aren’t born with synapses already specially formed to receive stimuli from our environment. We don’t know if we’ll be born in a sunny place, a cold place, to a tribe of nomads or any environment that accentuates a particular stimulus, which in turn also biases stimulation of certain synapses. Therefore, the distribution of our synapses is unknown. Our brains are able to process stimuli and adapt accordingly to the effects of our environment on our bodies, especially during developmental periods. This is an “online” way of receiving and processing information that requires constant engagement and “updating” of internal systems. Though genes can give us thicker hair for harsh climates, or long limbs for frequent movement, these are changes that happen over thousands of years; the concept of a priori unknown distributions in our brains allows timely recognition and response to environmental variance.

### Annotation 8 (the paper)

**Annotation** by Noel Lau  (2016-04-03):

> The computer algorithm and network has many parallels with the natural synaptic network and its processes that govern neuroplasticity. Edges represent neuronal connections, since both nodes and neurons are components of a system that transfer data. The directional nature of synaptic messaging is represented in the artificial system with arrows from one node to another. The node from which the “signal” originates, and the destination node, are called the signal-target pair.

### Annotation 9 (the paper)

**Annotation** by Noel Lau  (2016-04-03):

> In Figure B: For the pruning algorithm, we start with a network that is densely packed with edges, to reflect the fully developed network of synapses we see in developmental stages of young mice, before the rapid synaptic pruning process or pruning algorithm, respectively, can be applied.
>
> This algorithm plays a crucial role in data mining, which allows a computer to identify patterns in large reservoirs of data and represent it differently to reveal new, useful data. In one common example, an algorithm that analyzes the register of sold items from a convenience store can identity that customers who purchased item A, also purchase item B. The association made between these two objects has distinguished a particular consumer behavior (buy A with B), and can help with identifying receptive customers to certain advertisements. Decision trees can be used to organize data into subsets based off of shared characteristics; it is the means to make the association “buy A with B”. In relation to this paper, the pruning algorithm is able to identify nodes and edges in a decision tree that are not frequently used, and remove them. Pruning helps reduce the risk of creating parameters for categorization that are too specific or affected by irrelevant data.

### Annotation 10 (the paper)

**Annotation** by Noel Lau  (2016-04-28):

> How the pruning algorithm works: Pruning decisions were made locally, meaning that the edge between the source and target kept track of use frequency for itself, and would be pruned if it didn’t exceed the minimum uses were determined to be unimportant. This decision is made relative to the performance of other node pairs; the top 10% of pairs in a single iteration of the experiment are retained.
>
> Keep in mind that since we are comparing rates in pruning algorithms, the all experiments now start with the densely packed structure of Figure 3B. All nodes are connected, so we will have source to source and target to target edges existing. A no-learning algorithm wastes 25% of its edges by retaining these connections. It doesn't progress from the testing phase. This is like our "control", though the experiment doesn't focus on the control because it is more focused on providing statistical support that specific rates of pruning are advantageous, instead of pruning rates over constant and growing rates.

### Annotation 11 (the paper)

**Annotation** by Noel Lau  (2016-04-28):

> Here, we are exploring why experimentally observed decreasing pruning rates are more effective in promoting efficiency and robustness, rather than the algorithm’s constant pruning rate that was assumed (a constant rate of removal of existing edges, each interval).
>
> We shift perspectives, from comparing growing, pruning and non-learning algorithms, to different rates of pruning, i.e. increasing, decreasing, and constant, on network efficiency and robustness. Keep in mind that efficiency and robustness are the measures with which we define network performance.

### Annotation 12 (the paper)

**Annotation** by Noel Lau  (2016-04-28):

> Rate Comparisons:
>
> Increasing: Starts out slow, by eliminating a few edges, and gradually increases the rate of pruning. This gives the algorithm more time to evaluate the activity of the edge before making an edge pruning decision, intuitively, a more “informed” decision.
>
> Decreasing rate: Rapid elimination of edges will remove connections from the densely packed network, and tapers off. There is less time to observe and collect edge activity data, compared to increasing rates, but provides more time for network stabilization. Decreasing rates were able to optimize both efficiency and robustness, while one is often achieved at the cost of the other in most situations.
>
> Notes on how rates (r) for each iteration in each scenario were determines, are included in later annotations.

### Annotation 13 (the paper)

**Annotation** by Noel Lau  (2016-04-28):

> The more edges existing, the more energy is consumed. An optimal network structure will also minimize the energy spent to theoretically support the edges. From Fig S10, increasing rates have the highest energy expenditure and greatest drastic increase in energy use during the first few pruning intervals, suggesting poor edge pruning decisions contrary to intuition.
>
> There’s no one format of the network, or distribution, that would represent the single most optimized system in terms of efficiency and robustness. In decreasing rate pruning trials, the system removes many edges at first, creating a smaller set of edges and nodes to test and manage. This promotes connectivity, and allows for optimization in that particular distribution. One may think that we might lose potentially useful nodes this way, by deleting them early on, before they get to be used frequently enough.

### Annotation 14 (the paper)

**Annotation** by Noel Lau  (2016-04-28):

> 2s-patch (even)- A specific group of source nodes is connected to a specific group of target nodes. This models how there are specialized, unique connections, as subnetworks in the cortex., rather than continuous and random connections throughout the brain.
>
> 2-patch unbalance- There are three times as many target nodes, as there are source nodes. Multiple edges can interact with a single node, and since the distribution is random, some nodes can also be completely uninvolved. This model simulates how there can be a single input of stimulus (such as an eye, as a source node), but when the signal is sent into the brain for processing, it interacts with many more components (contacts several target nodes, modeled similarly to divergent processing).
>
> 4 patch- There are independently functioning groups of source and target nodes.
>
> 4-patch Hubel-Wiesel- the second set of source is not functioning. This simulates when one source of data, of two, (i.e. one single eye) is compromised, in order to observe how neural systems change in response.
>
> The study tested whether different rates of pruning would still perform similarly with different source-target distributions that reflect different biological situations (signals may not always be able to move directly from a source point A to a target point B). Since performance results were consistent, the initial model with 2 patch distribution, with unidirectional paths and equal sampling probability, is an appropriate model for general synaptic network construction.

### Annotation 15 (the paper)

**Annotation** by Noel Lau  (2016-04-28):

> The Erdős–Rényi (ER) random graph model (G(n,M) model) creates all possible graphs (networks) with n nodes and M edges, Each network engages different nodes with different combination of edges, such that all graphs are equally probably and meet the parameters exactly.
> It's significance in this paper is just to highlight that there was completely random deletion of edges from a fully connected (all edges engaged) initial state.

### Annotation 16 (the paper)

**Annotation** by Noel Lau  (2016-04-28):

> As q represents the likelihood of getting an S T edge, it may be intuitive to want to maximize these, for a lower p/q ratio. However, we should think back to the practicality of having alternative paths as robustness (against failure). An edge that connects a source node to another source node may not seem helpful because it doesn’t directly complete the objective of delivering packets of information from source to target, but can serve as a “stop” on the way to the target. Having a path segmented into multiple edges, (typically two) allows them to be used in routing for many different paths rather than simple between one source-target pair.

### Annotation 17 (the paper)

**Annotation** by Noel Lau  (2016-04-28):

> Can evolution really help construct the brain in such a way that this complex network functions at its best? Or can machine-learning technologies find the real way to optimize the system? This study takes the questions a step further to optimizing a similarly constructed network in airline traffic. How can we construct airline plans to satisfy the demands of the most people, by offering the most alternative routes to and from the most popular cities, and decrease traveling distance? The most frequently flown cities were identified and placed as nodes on a map, with flights as edges. A similar resource limitation (edge limit) was set.
> It was observed that decreasing rate pruning algorithms outperformed other rates again. It was evaluated using the same measures of robustness and efficiency, to create a network of the more time efficient and flexible variety of flights.

### Annotation 18 (the paper)

**Annotation** by Noel Lau  (2016-04-28):

> Scientists theorize that the success of decreasing rate pruning is because it can establish “hyper-connectivity”. We may feel that the algorithm cannot properly gauge the potential that an edge has before deleting it. We can test the edge’s potential contribution to robustness and efficiency by gathering more information, which an increasing rate pruning algorithm may allow.

### Annotation 19 (the paper)

**Annotation** by Noel Lau  (2016-04-28):

> Much of the study of neurobiology is heavily focused on the regulatory mechanisms that control synapse elimination: genetic on-off switches, and protein expression pathways are a few examples. These studies made observations on proof that synaptic pruning is a process in several organisms.
>
> Previous studies used Hopfield networks, that simplified the neural system into a network of artificial neurons like the image below. ![Hopfield Neuron]( http://home.agh.edu.pl/~vlsi/AI/hopf/hopfield_eng_pliki/image002.jpg
> "Diagram of Hopfield system")
>
> The neuron is updated after receiving input from each of its branches, to simulate the dendrites of a nerve cell.  Each input branch has a corresponding weight; the sum of the products of each input and its weight is the output. The singular output end imitates the terminal branch of the axon, and is either +1 or -1 depending on whether the calculated output is positive or negative, respectively.

**Reply** by Noel Lau  (2016-05-10):

> Previously used models, such as the Hopfield network, nodes are designed to receive data from different sources, compute something, and output a value. However, in this study, nodes simply passed "bits" along, much like sending an email from person to person.

### Annotation 20 (the paper)

**Annotation** by Noel Lau  (2016-05-10):

> Decreasing pruning rates are also found in the brain, consistent with simulations of neural pruning run with the corresponding algorithm. As time increases, it becomes increasingly more difficult to activate the pathways that control pruning mechanisms.
>
> Myelination is the growth of the outer, insulating layer of myelin, around and along the length of the nerve cell. By aggressively pruning at first, less myelin is wasted, as opposed to increasing rates which would prune rapidly after a period of myelination has occurred.
>
>   ![Myelin diagram](https://psych-brain-trust.wikispaces.com/file/view/6.png/261102636/581x334/6.png "Myelination Support")

### Annotation 21 (the paper)

**Annotation** by Noel Lau  (2016-05-09):

> The EPTA staining method allows for improved visualization of synaptic active zones, by increasing contrast while not interfering with the results of other visualization protocols. It is cost-efficient, relatively easily prepared (PTA in powder form usually, mixed easily with ethanol) and effective in studies involving cerebral development of mice (including visualizing soft tissue in larger, embryos with micro-CT technology).
> Though slightly unrelated, this is a link to a video using images taken by the micro-CT and stitched together to present a detailed, 3-D overview of the mouse):https://commons.wikimedia.org/wiki/File%3AMicroCT-for-comparative-morphology-simple-staining-methods-allow-high-contrast-3D-imaging-of-1472-6793-9-11-S4.ogv

### Annotation 22 (the paper)

**Annotation** by Noel Lau  (2016-05-11):

> SVMs are classifiers- they are machine learning models that are trained to assign new data into categories. In this case, subjects in the image are either synapses or not.
> The code is openly available at http://www.cs.cmu.edu/∼saketn/detect_synapses/.
>
> 20 images from each animal were evaluated by human analysis to ensure consistency. If one sample absorbs dye more effectively than others, the machine may be able to identify more synapses than it does in other images. In reality however, the synapse density is relatively consistent across the samples. This image can’t be properly evaluated and should be removed from the analysis. This classification task is a relatively simple decision for humans to make so the scientists determine the number of “true synapses” and “non-synapses” present in the image, which is compared with the classifier’s conclusions.
>
> The classifier tried to count synapses in the same image; its performance was evaluated as precision and recall (defined as “percentage of predicted synapses that are truly synapses”, and “percentage of true synapse correctly predicted by the classifier”, respectively). The classification threshold was set such that if 50% of synapses were found in the image, we are 80% confident that it is a true synapse. Images for which the classifier didn’t meet the threshold were basically not clear enough and could result in skewness if included in the data.

### Annotation 23 (the paper)

**Annotation** by Noel Lau  (2016-05-09):

> Theoretically, this allows the system to remove edges that can make source to target travel time longer, or be a waste of resources, as we have a fixed limit of B edges that can exist simultaneously. 10 iterations were run. During each iteration for increasing rates, a percentage of edges were removed: Iteration 1 removes 10%, or  r1= 0.10 (synapses remaining expressed by E0 × (1 − r1))
>
> Iteration 2 removes 15%, or r2= 0.15 (remaining expressed by E0 × (1 − r1)(1 − r2))...
> Iteration 3, 20%, and so on, until Iteration 5 removes (rc) edges, such that E0(1 − rc) I = B. I= 10 iterations, B= number of final edges, and rc= r5.
> The rates of iterations 6 through 10 are calculated in relation to rate of iterations 4 through 1, respectively. For example, to find the rate of increasing pruning in iteration 10 (r10), we can evaluate E(1 − rc)(1 − rc) = E(1 − r1)(1 − r10), where r1= 0.1, and rc is found experimentally.
>
> For the decreasing rates, the sequence was reversed so it started with larger elimination rates.

### Annotation 24 (the paper)

**Annotation** by Noel Lau  (2016-05-10):

> You may ask, why don't we just link the source node to the target node directly for each pair? Then there'd be an effective "highway" for signals, made of frequently used edges. Well, insofar, there is no "most effective way" to send information. Which nodes are source-target pairs, and which edges will be needed, are unknown and not programmed beforehand, but completely random (apriori unknown). To link source-target nodes together with one large edge, we would have to know which are S-T pairs before iterations, and such paths would limit robustness. By having several edges, we could use some edges in a specific pathway for a specific S-T pair, to connect bits traveling to other target nodes. This dramatically saves resources; the concept of sharing is like giving a computer to every student in a school, versus opening a computer cluster that allows students open access to PCs, as needed.

---

Source: [Fermat's Library](https://fermatslibrary.com/) — https://fermatslibrary.com/p/60c7a6b8. More annotated papers: https://fermatslibrary.com/llms.txt
