CS 499 Homework 1: Basic Parallel Architecture and Communication

Consider the routing of messages in a parallel computer that uses store-and-forward routing. In such a network, the cost of sending a single message of size \(m\) from \(P_{source}\) to \(P_{destination}\) via a path of length \(d\) is \(t_s + t_w \times d \times m\). An alternate way of sending a message of size m is as follows. The user breaks the message into \(k\) parts each of size \(m/k\), and then sends these \(k\) distinct messages one by one from \(P_{source}\) to \(P_{destination}\). For this new method, derive the expression for time to transfer a message of size \(m\) to a node \(d\) hops away under the following two cases:

1. Assume that another message can be sent from \(P_{source}\) as soon as the previous message has reached the next node in the path.
2. Assume that another message can be sent from \(P_{source}\) only after the previous message has reached \(P_{destination}\).

For each case, comment on the value of this expression as the value of \(k\) varies between 1 and \(m\). Also, what is the optimal value of \(k\) if \(t_s\) is very large, or if \(t_s = 0\)?

Clarification: To start sending any message, a cost of \(t_s\) must be paid. For example sending \(m=10\) words in its entirety costs \[t_s + t_w \times d \times 10\] Sending the same message broken into two 5-word chunks costs \[(t_s + t_w \times d \times 5) + (t_s + t_w \times d \times 5)\] if the sending does not overlap.

Note: some of the math fonts used above may not display correctly. I have tested the display on Google chrome and seen it work when connected to the network. Notify me if it does not render properly.

Outline the major differences between a shared memory parallel computer and a distributed memory parallel computer. Discuss the following in your answer.

1. Overview the differences in the basic architecture of these two types of machines with respect to memory access.
2. Describe how the programming models are different between these two paradigms due to the differences in memory access.
3. Report the relative abundance of one architecture versus the other and where they typically appear "in the wild".
4. Which architectures scales better as more processors are incorporated. For example, discuss the feasibility of building a 1024 processor computer with 1024 GB of RAM as a distributed memory machine versus as a shared memory machine.

For the task graphs given in Figure 3.42 (below), determine the following:

1. Maximum degree of concurrency.
2. Critical path length.
3. Maximum achievable speedup over one process assuming that an arbitrarily large number of processes is available.
4. The minimum number of processes needed to obtain the maximum possible speedup.
5. The maximum achievable speedup if the number of processes is limited to 2, 4, and 8.

Assume that each task node takes an equal amount of time to execute.

Consider parallelizing the bucket-sort algorithm on a distributed memory computer.

• Input consists of an array A of n random integers in the range [1...r].
• All p processors on the parallel computer are assumed to have access to the entire array A in memory.
• The algorithm should compute the array of integers B where element B[i] contains the total number of times integer i appeared in input array A.
• At the end of the algorithm execution, at least one processor must contain the entire array B which may involve some communication.

Describe two ways to decompose this problem into a parallel program. Compare these two.

1. Describe a decomposition based on partitioning the input data (i.e., the array A) and an appropriate mapping onto p processes. Describe briefly how the resulting parallel algorithm would work.
2. Describe a decomposition based on partitioning the output data (i.e., the array B) and an appropriate mapping onto p processes. Describe briefly how the resulting parallel algorithm would work.
3. Discuss the advantages and disadvantages of these two parallel formulations. Describe circumstances under which each is preferred. Such circumstances should consider
   • p the number of processors
   • r the number of different data possibilities (size of B)
   • A the input array and n its size
   • Potentially also the cost to communicate between the processors

Assume processor 0 has a message that must be sent to all other processors in a distributed memory parallel computer. Describe an efficient algorithm to do this in a 2D Torus (wrap-around mesh) with R rows and C columns.

Account for the following in your answers.

• A processor can send a message to any other processor. Any processor between the source and destination does not receive the message, only forwards it along the path. However, once a processor has actually received the message, it can subsequently be a source and send it to other processors.
• Processors can be indexed with 2D coordinates as in processor (0,2) sends a message to processor (5,4)
• Give rough pseudocode for how this algorithm will look
• Give a cost estimate of this communication in terms of the number or rows R and columns C in the network. The cost of a single communicate between nodes that are \(d\) hops away is \[t_{single} = t_s + t_w \times m \times d\] You may assume that R and C are powers of two in your analysis to simplify the expression for the total communication time \(t_{comm}\).

Consider a simple physical simulation of heat transfer. A rod of material is insulated except at its ends to which are attached two constant sources of heat/cold.

The rod is broken into discrete chunks, sometimes referred to as finite elements, and it is assumed that throughout each element the temperature is uniform. The reservoir elements of heat/cold at the left and right remain at a constant temperature. For the other elements, the temperature is updated at each time step by examining the temperature of its neighbors compared to itself. This dependence is indicated in the lower part of the diagram and the exact update for an element is given in the code below.

The following C code implements a simple version of this simulation. A row of the matrix H contains the temperature of all elements in the rod at a time step with the leftmost and rightmost elements remaining constant. Ultimately, the program computes the entire matrix H which contains temperatures of all elements at all time steps. Examine this code carefully.

#include <stdio.h>
#include <stdlib.h>
/* HEAT TRANSFER SIMULATION
   
   Simple physical simulation of a rod connected at the left and right
   ends to constant temperature heat/cold sources.  All positions on
   the rod are set to an initial temperature.  Each time step, that
   temperature is altered by computing the difference between a cells
   temperature and its left and right neighbors.  A constant k
   (thermal conductivity) adjusts these differences before altering
   the heat at a cell.  Use the following model to compute the heat
   for a position on the rod according to the finite difference
   method.

      left_diff  = H[t][p] - H[t][p-1];
      right_diff = H[t][p] - H[t][p+1];
      delta = -k*( left_diff + right_diff )
      H[t+1][p] = H[t][p] + delta
   
   Substituting the above, one can get the following

     H[t+1][p] = H[t][p] + k*H[t][p-1] - 2*k*H[t][p] + k*H[t][p+1]

   The matrix H is computed for all time steps and all positions on
   the rod and displayed after running the simulation.  The simulation
   is run for a fixed number of time steps rather than until
   temperatures reach steady state.
*/

int main(int argc, char **argv){
  int max_time = 50;            /* Number of time steps to simulate */
  int width = 20;               /* Number of cells in the rod */
  double initial_temp = 50.0;   /* Initial temp of internal cells  */
  double L_bound_temp = 20.0;   /* Constant temp at Left end of rod */
  double R_bound_temp = 10.0;   /* Constant temp at Right end of rod */
  double k = 0.5;               /* thermal conductivity constant */
  double **H;                   /* 2D array of temps at times/locations  */

  /* Allocate memory  */
  H = malloc(sizeof(double*)*max_time); 
  int t,p;
  for(t=0;t<max_time;t++){
    H[t] = malloc(sizeof(double*)*width);
  }

  /* Initialize constant left/right boundary temperatures */
  for(t=0; t<max_time; t++){
    H[t][0] = L_bound_temp;
    H[t][width-1] = R_bound_temp;
  }

  /* Initialize temperatures at time 0 */
  t = 0;
  for(p=1; p<width-1; p++){
    H[t][p] = initial_temp;
  }

  // Simulate the temperature changes for internal cells
  for(t=0; t<max_time-1; t++){
    for(p=1; p<width-1; p++){
      double left_diff  = H[t][p] - H[t][p-1];
      double right_diff = H[t][p] - H[t][p+1];
      double delta = -k*( left_diff + right_diff );
      H[t+1][p] = H[t][p] + delta;
    }
  }

  /* Print results */
  printf("Temperature results for 1D rod\n");
  printf("Time step increases going down rows\n");
  printf("Position on rod changes going accross columns\n");

  /* Column headers */
  printf("%3s| ","");
  for(p=0; p<width; p++){
    printf("%5d ",p);
  }
  printf("\n");
  printf("%3s+-","---");
  for(p=0; p<width; p++){
    printf("------");
  }
  printf("\n");
  /* Row headers and data */
  for(t=0; t<max_time; t++){
    printf("%3d| ",t);
    for(p=0; p<width; p++){
      printf("%5.1f ",H[t][p]);
    }
    printf("\n");
  }

  return 0;
}

Answer the following questions about how to parallelize this code.

1. A typical approach to parallelizing programs is to select a loop and split iterations of work between available processors. Describe how one might do this for the heat program. Make sure to indicate any loops for which this approach is not feasible and any which seem more viable to you.
2. Describe how you would divide the data for the heat transfer problem among many processors in a distributed memory implementation to facilitate efficient communication and processing. Describe a network architecture that seems to fit this problem well and balances the cost of the network well.

• Complete the problems in each section labelled Problem
• This is a pair assignment: you may select one partner to work with on this assignment as a group of 2. You may also opt to work alone as a group of 1.
• Only 1 member of your group should submit the HW Writeup on Blackboard
• Make sure the HW Writeup has all group member's information in it:

  CS 499 HW 1
  Group of 2
  Turanga Leela tleela4 G07019321
  Philip J Fry pfry99 G00000001
  

• Do your work in an acceptable electronic format which are limited to the following
  • A Microsoft Word Document (.doc or .docx extension)
  • A PDF (portable document format, .pdf)

  You may work in other programs (Apple Words, Google Docs, etc.) but make sure you can export/"save as" your work to an acceptable format before submitting it.

• Submit your work to our Blackboard site.
  • Log into Blackboard
  • Click on CS 499->HW Assignments->HW 1
  • Press the button marked "Attach File: Browse My Computer"
  • Select your file
  • Click submit on Blackboard