Nowadays, commercial applications are most used on parallel computers. A computer that runs such an application has to be able to process large amount of data in sophisticated ways. We can say with no doubt that commercial applications will define future parallel computers architecture. But scientific applications will still remain important users of parallel computing technology. Trends in commercial and scientific applications are merging as commercial applications perform more sophisticated computations and scientific applications become more data intensive. Today, a lot of parallel programming languages and compilers, based on dependencies detected in source code, are able to automatically split a program into multiple processes and/or threads to be executed concurrently on the available processors from a parallel system. Parallel computing is an efficient form of information processing which emphasizes the exploitation of concurrent events in the computing process. Concurrency implies parallelism, simultaneity and pipelining. Parallel events may occur in multiple resources during the same time interval; simultaneous events may occur at the same time instant; and pipelined events may occur in overlapped time spans. Parallel processing demands concurrent execution of many programs in the computer. It is a cost effective means to improve system performance through concurrent activities in the computer.The highest level of parallel processing is conducted among multiple jobs or programs through multiprogramming, time-sharing, and multiprocessing. This presentation covers the basics of parallel computing. Beginning with a brief overview and some concepts and terminology associated with parallel computing, the topics of parallel memory architectures, Parallel computer architectures and Parallel programming models are then explored.
Parallel computing is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently ("in parallel"). There are several different forms of parallel computing: bit-level, instruction level, data, and task parallelism. Parallelism has been employed for many years, mainly in high-performance computing, but interest in it has grown lately due to the physical constraints preventing frequency scaling. As power consumption (and consequently heat generation) by computers has become a concern in recent years, parallel computing has become the dominant paradigm in computer architecture, mainly in the form of multicore processors. Parallel computers can be roughly classified according to the level at which the hardware supports parallelism, with multi-core and multi- processor computers having multiple processing elements within a single machine, while clusters, MPPs, and grids use multiple computers to work on the same task. Specialized parallel computer architectures are sometimes used alongside traditional processors, for accelerating specific tasks. Parallel computer programs are more difficult to write than sequential ones, because concurrency introduces several new classes of potential software bugs, of which race conditions are the most common. Communication and synchronization between the different subtasks are typically some of the greatest obstacles to getting good parallel program performance. The maximum possible speed-up of a program as a result of parallelization is known as Amdahl's law. Background Traditionally, computer software has been written for serial computation. To solve a problem, an algorithm is constructed and implemented as a serial stream of instructions. These instructions are executed on a central processing unit on one computer. Only one instruction may execute at a time —after that instruction is finished, the next is executed. Parallel computing, on the other hand, uses multiple processing...