OpenMP logo

The OpenMP (Open Multi-Processing) is an application programming interface (API) that supports multi-platform shared memory multiprocessing programming in C/C++ and Fortran on many architectures, including Unix and Microsoft Windows platforms. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior. Image File history File links No higher resolution available. ... Image File history File links No higher resolution available. ... API and Api redirect here. ... // Diagram of a typical Shared memory system. ... Multiprocessing is traditionally known as the use of multiple concurrent processes in a system as opposed to a single process at any one instant. ... C is a general-purpose, block structured, procedural, imperative computer programming language developed in 1972 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system. ... C++ (pronounced see plus plus, IPA: ) is a general-purpose programming language with high-level and low-level capabilities. ... Fortran (previously FORTRAN[1]) is a general-purpose[2], procedural,[3] imperative programming language that is especially suited to numeric computation and scientific computing. ... Filiation of Unix and Unix-like systems Unix (officially trademarked as UNIX®, sometimes also written as or ® with small caps) is a computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs including Ken Thompson, Dennis Ritchie and Douglas McIlroy. ... Windows redirects here. ... In computer science, a compiler directive is data embedded in source code by programmers to tell compilers some intention about compilation. ... Environment variables are a set of dynamic values that can affect the way running processes will behave on a computer. ...

Jointly defined by a group of major computer hardware and software vendors, OpenMP is a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications for platforms ranging from the desktop to the supercomputer. A programmer or software developer is someone who programs computers, that is, one who writes computer software. ... For other uses, see Supercomputer (disambiguation). ...

An application built with the hybrid model of parallel programming can run on a computer cluster using both OpenMP and Message Passing Interface (MPI). Parallel programming is a computer programming technique that provides for the execution of operations in parallel, either within a single computer, or across a number of systems. ... An example of a Computer cluster A computer cluster is a group of tightly coupled computers that work together closely so that in many respects they can be viewed as though they are a single computer. ... Message Passing Interface (MPI) is computer software that allows many computers to communicate with one another. ...

Introduction

An illustration of multithreading where the master thread forks off a number of threads which execute blocks of code in parallel

OpenMP is an implementation of multithreading, a method of parallelization whereby the master "thread" (a series of instructions executed consecutively) "forks" a specified number of slave "threads" and a task is divided among them. The threads then run concurrently, with the runtime environment allocating threads to different processors. Image File history File links Fork_join. ... Image File history File links Fork_join. ... Many programming languages, operating systems, and other software development environments support what are called threads of execution. ... Many programming languages, operating systems, and other software development environments support what are called threads of execution. ... Computer programming (often simply programming) is the craft of implementing one or more interrelated abstract algorithms using a particular programming language to produce a concrete computer program. ...

The section of code that is meant to run in parallel is marked accordingly, with a preprocessor directive that will cause the threads to form before the section is executed. Each thread has an "id" attached to it which can be obtained using a function (called omp_get_thread_num() in C/C++ and OMP_GET_THREAD_NUM() in FORTRAN). The thread id is an integer, and the master thread has an id of "0". After the execution of the parallelized code, the threads "join" back into the master thread, which continues onward to the end of the program. A directive is an instruction to a programming language compiler about how to compile a program. ... Look up Function in Wiktionary, the free dictionary. ... C is a general-purpose, block structured, procedural, imperative computer programming language developed in 1972 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system. ... C++ (pronounced see plus plus, IPA: ) is a general-purpose programming language with high-level and low-level capabilities. ... Fortran (previously FORTRAN[1]) is a general-purpose[2], procedural,[3] imperative programming language that is especially suited to numeric computation and scientific computing. ...

By default, each thread executes the parallelized section of code independently. "Work-sharing constructs" can be used to divide a task among the threads so that each thread executes its allocated part of the code. Both Task parallelism and Data parallelism can be achieved using OpenMP in this way. Task Parallelism is a form of parallelization of computer code. ... Data Parallelism is a form of parallelization of computer code. ...

The runtime environment allocates threads to processors depending on usage, machine load and other factors. The number of threads can be assigned by the runtime environment based on environment variables or in code using functions. The OpenMP functions are included in a header file labelled "omp.h" in C/C++. Environment variables are a set of dynamic values that can affect the way running processes will behave on a computer. ... In computer programming, particularly in the C and C++ programming languages, a header file or include file is a file, usually in the form of source code, that is automatically included in another source file by the compiler. ... C is a general-purpose, block structured, procedural, imperative computer programming language developed in 1972 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system. ... C++ (pronounced see plus plus, IPA: ) is a general-purpose programming language with high-level and low-level capabilities. ...

History

The OpenMP Architecture Review Board (ARB) published its first standard, OpenMP for FORTRAN 1.0, in October of 1997. October the following year they released the C/C++ standard. 2000 saw version 2.0 of the FORTRAN standard with version 2.0 of the C/C++ standard being released in 2002. The current version is 2.5. It is a combined C/C++/FORTRAN standard, which was released in 2005.

The core elements

Chart of OpenMP Constructs

The core elements of OpenMP are the constructs for thread creation, work load distribution (work sharing), data environment management, thread synchronization, user level runtime routines and environment variables. Image File history File links Size of this preview: 800 × 509 pixelsFull resolution (962 × 612 pixel, file size: 71 KB, MIME type: image/jpeg) OpenMP constructs I, the creator of this work, hereby release it into the public domain. ... Image File history File links Size of this preview: 800 × 509 pixelsFull resolution (962 × 612 pixel, file size: 71 KB, MIME type: image/jpeg) OpenMP constructs I, the creator of this work, hereby release it into the public domain. ...

A compiler directive in C/C++ is called a pragma (pragmatic information). It is a preprocessor directive, thus it is declared with a hash (#). Compiler directives specific to OpenMP in C/C++ are written in codes as follows: A directive is an instruction to a programming language compiler about how to compile a program. ...

 #pragma omp <rest of pragma> 

The OpenMP specific pragmas are listed below:

Thread creation

omp parallel. It is used to fork additional threads to carry out the work enclosed in the construct in parallel. The original process will be denoted as master thread with thread ID 0.

Example: Display "Hello, world" using multiple threads.

 int main(int argc, char* argv[]) { #pragma omp parallel printf("Hello, world.n"); return 0; } 

Work-sharing constructs

used to specify how to assign independent work to one or all of the threads.

• omp for or omp do: used to split up loop iterations among the threads
• sections: assigning consecutive but independent code blocks to different threads
• single: specifying a code block that is executed by only one thread, a barrier is implied in the end
• master: similar to single, but the code block will be executed by the master thread only and no barrier implied in the end.

Example: initialize the value of a large array in parallel, using each thread to do a portion of the work

 #define N 100000 int main(int argc, char *argv[]) { int i, a[N]; #pragma omp parallel for for (i=0;i<N;i++) a[i]= 2*i; return 0; } 

OpenMP Clauses

Since OpenMP is a shared memory programming model, most variables in OpenMP code are visible to all threads by default. But sometimes private variables are necessary to avoid race condition and there is a need to pass values between the sequential part and the parallel region (the code block executed in parallel), so data environment management is introduced as data clauses by appending them to the OpenMP directive. The different types of clauses are A race condition or race hazard is a flaw in a system or process whereby the output of the process is unexpectedly and critically dependent on the sequence or timing of other events. ...

Data Scoping Clauses

• shared: the data within a parallel region is shared, which means visible and accessible by all threads simultaneously. By default, all variables in the work sharing region are shared except the loop iteration counter.
• private: the data within a parallel region is private to each thread, which means each thread will have a local copy and use it as a temporary variable. A private variable is not initialized and the value is not maintained for use outside the parallel region. By default, the loop iteration counter in the work-sharing region (if any) is private.
• default: allows the programmer to state that the default data scoping within a parallel region will be either shared, private, or none. The none option forces the programmer to declare each variable in the parallel region as either shared or private.

Synchronization clauses

• critical section: the enclosed code block will be executed by all threads but only one thread at a time, not simultaneously executed. It is often used to protect shared data from race condition.
• atomic: similar to critical section, but advise the compiler to use special hardware instructions for better performance. Compilers may choose to ignore this suggestion from users and use critical section instead.
• ordered: the structure block is executed in the order in which iterations would be executed in a sequential loop
• barrier: each thread waits until all of the other threads of a team have reached this point. A work-sharing construct has an implicit barrier synchronization at the end.
• nowait: specifies that threads completing assigned work can proceed. In the absence of this clause, threads would encounter a barrier synchronization at the end of the work sharing construct by default.

A race condition or race hazard is a flaw in a system or process whereby the output of the process is unexpectedly and critically dependent on the sequence or timing of other events. ...

Scheduling clauses

• schedule(type, chunk):This is useful if the work sharing construct is a do-loop or for-loop. The iteration(s) in the work sharing construct are allocated to threads. The scheduling of the threads are controlled by this clause. The three types of scheduling are:

1. static: Here,all the threads are allocated iterations before they execute the loop iterations. The iterations are divided among threads equally by default.However, specifying an integer for the parameter "chunk" will allocate "chunk" number of contiguous iterations to a particular thread.
2. dynamic:Here,some of the iterations are allocated to a smaller number of threads.Once a particular thread finishes its allocated iteration, it returns to get another one from the iterations that are left. The parameter "chunk" defines the number of contiguous iterations that are allocated to a thread at a time.
3. guided:A large chunk of contiguous iterations are allocated to each thread dynamically (as above). The chunk size decreases exponentially with each successive allocation to a minimum size specified in the parameter "chunk"

IF control

• if: This will cause the threads to parallelize the task only if a condition is met. Otherwise the code block executes serially.

Initialization

• firstprivate: the data is private to each thread, but initialized using the value of the variable using the same name from the master thread.
• lastprivate: the data is private to each thread. The value of this private data will be copied to a global variable using the same name outside the parallel region if current iteration is the last iteration in the parallelized loop. A variable can be both firstprivate and lastprivate.
• threadprivate: The data is a global data, but it is private in each parallel region during the runtime. The difference between threadprivate and private is the global scope associated with threadprivate and the preserved value across parallel regions.

Data copying

• copyin: similar to firstprivate for private variables, threadprivate variables are not initialized, unless using copyin to pass the value from the corresponding global variables. No copyout is needed because the value of a threadprivate variable is maintained throughout the execution of the whole program.
• copyprivate: used with single to support the copying of data values from private objects on one thread (the single thread) to the corresponding objects on other threads in the team.

Reduction

• reduction(operator|intrinsic:list): the variable has a local copy in each thread, but the values of the local copies will be summarized (reduced) into a global shared variable. This is very useful if a particular operation (specified in "operator" for this particular clause) on a datatype that runs iteratively so that its value at a particular iteration depends on its value at a previous iteration. Basically, the steps that lead up to the operational increment are parallelized, but the threads gather up and wait before updating the datatype, then increments the datatype in order so as to avoid racing condition. This would be required in parallelizing Numerical Integration of functions and Differential Equations, as a common example.

Numerical Integration with the Monte Carlo method: Nodes are random equally distributed. ... In mathematics, a differential equation is an equation in which the derivatives of a function appear as variables. ...

Others

• flush: The value of this variable is restored from the register to the memory for using this value outside of a parallel part
• master:Executed only by the master thread (the thread which forked off all the others during the execution of the OpenMP directive).No implicit barrier; other team members (threads) not required to reach.

User-level runtime routines

Used to modify/check the number of threads, detect if the execution context is in a parallel region, how many processors in current system, set/unset locks, timing functions, etc.

Environment variables

a method to alter the execution features of OpenMP applications. Used to control loop iterations scheduling, default number of threads, etc.

Sample Programs

In this section, some sample programs are provided to illustrate the concepts explained above.

Hello World

This is the most basic program, one that prints "hello world". A hello world program is a computer program that prints out Hello, world! on a display device. ... A hello world program is a computer program that prints out Hello, world! on a display device. ...

C/C++

 #include <omp.h> #include <stdio.h> int main (int argc, char *argv[]) { int id, nthreads; #pragma omp parallel private(id) { id = omp_get_thread_num(); printf("Hello World from thread %dn", id); #pragma omp barrier if ( id == 0 ) { nthreads = omp_get_num_threads(); printf("There are %d threadsn",nthreads); } } return 0; } 

C is a general-purpose, block structured, procedural, imperative computer programming language developed in 1972 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system. ... C++ (pronounced see plus plus, IPA: ) is a general-purpose programming language with high-level and low-level capabilities. ...

Fortran 77

 PROGRAM HELLO INTEGER ID, NTHRDS INTEGER OMP_GET_THREAD_NUM, OMP_GET_NUM_THREADS C$OMP PARALLEL PRIVATE(ID) ID = OMP_GET_THREAD_NUM() PRINT *, 'HELLO WORLD FROM THREAD', ID C$OMP BARRIER IF ( ID .EQ. 0 ) THEN NTHRDS = OMP_GET_NUM_THREADS() PRINT *, 'THERE ARE', NTHRDS, 'THREADS' END IF C$OMP END PARALLEL END 

Fortran (also FORTRAN) is a statically typed, compiled, programming language originally developed in the 1950s and still heavily used for scientific computing and numerical computation half a century later. ...

Free form Fortran 90

 program hello90 use omp_lib integer:: id, nthreads !$omp parallel private(id) id = omp_get_thread_num() write (*,*) 'Hello World from thread', id !$omp barrier if ( id == 0 ) then nthreads = omp_get_num_threads() write (*,*) 'There are', nthreads, 'threads' end if !$omp end parallel end program 

In computer programming, a free-form language is a programming language in which the positioning of characters on the page in program text is not significant. ... Fortran (also FORTRAN) is a statically typed, compiled, programming language originally developed in the 1950s and still heavily used for scientific computing and numerical computation half a century later. ...

Clauses in work-sharing constructs (in C/C++)

The application of some OpenMP clauses are illustrated in the simple examples in this section. The piece of code below updates the elements of an array "b" by performing a simple operation on the elements of an array "a". The parallelization is done by the OpenMP directive "#pragma". The scheduling of tasks is dynamic. Notice how the iteration counters "j" and "k" have to be made private, whereas the primary iteration counter "i" is private by default. The task of running through "i" is divided among multiple threads, and each thread creates its own versions of "j" and "k" in its execution stack, thus doing the full task allocated to it and updating the allocated part of the array "b" at the same time as the other threads. C is a general-purpose, block structured, procedural, imperative computer programming language developed in 1972 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system. ... C++ (pronounced see plus plus, IPA: ) is a general-purpose programming language with high-level and low-level capabilities. ...

 #define CHUNKSIZE 1 /*defines the chunk size as 1 contiguous iteration*/ /*forks off the threads*/ #pragma omp parallel schedule(dynamic, CHUNKSIZE) private(j,k) { /*Starts the work sharing construct*/ #pragma omp for for(i = 2; i <= N-1; i++) for(j = 2; j <= i; j++) for(k = 1; k <= M; k++) b[i][j] += a[i-1][j]/k + a[i+1][j]/k; } 

The next piece of code is a common usage of the "reduction" clause to calculate reduced sums. Here, we add up all the elements of an array "a" with an "i" dependent weight using a for-loop which we parallelize using OpenMP directives and reduction clause. The scheduling is kept static.

 #define N 10000 /*size of a*/ void calculate(int); /*The function that calculates the elements of a*/ long w; double a[N]; calculate(a); sum = 0.0; /*forks off the threads and starts the work-sharing construct*/ #pragma omp parallel for private(w) reduction(+:sum) schedule(static,1) for(i = 0; i < N; i++) { w = i*i; sum = sum + w*a[i]; } printf("n %lf",sum); 

Implementations

OpenMP has been implemented in many commercial compilers. For instance, Visual C++ 2005 supports it (in its Professional and Team System editions [1]), and so do the Intel compilers for their x86 and IPF product series. Sun Studio supports the latest OpenMP 2.5 specification with productivity enhancements for Solaris OS (UltraSPARC and x86/x64) and is planning similar support for the Linux platform in its next release. The Sun Studio compiler suite is Sun Microsystems flagship software development product for Solaris and Linux. ...

GCC 4.2 supports OpenMP, and some distributions (such as Fedora Core 5 gcc) have support for OpenMP in their GCC 4.1 based compilers. The GNU Compiler Collection (usually shortened to GCC) is a set of programming language compilers produced by the GNU Project. ... Fedora Core is an RPM-based Linux distribution, developed by the community-supported Fedora Project, sponsored by Red Hat. ...

Pros and Cons of OpenMP

Pros

• Simple: need not deal with message passing as MPI does
• Data layout and decomposition is handled automatically by directives.
• Incremental parallelism: can work on one portion of the program at one time, no dramatic change to code is needed.
• Unified code for both serial and parallel applications: OpenMP constructs are treated as comments when sequential compilers are used.
• Original (serial) code statements need not, in general, be modified when parallelized with OpenMP. This reduces the chance of inadvertently introducing bugs.
• Both coarse-grained and fine-grained parallelism are possible

Cons Message Passing Interface (MPI) is computer software that allows many computers to communicate with one another. ... Granularity is the extent to which a system contains discrete components of ever-smaller size. ... Granularity is a measure of the size of the components, or descriptions of components, that make up a system. ...

• Currently only runs efficiently in shared-memory multiprocessor platforms
• Requires a compiler that supports OpenMP.
• Scalability is limited by memory architecture.
• Reliable error handling is missing.
• Lack fine-grain mechanisms to control thread-processor mapping.
• Synchronization between a subset of threads is not allowed.

Performance expectations of OpenMP

One might expect to get N times less wall clock execution time (or N times speedup) when running a program parallelized using OpenMP on a N processor platform. However, this is seldom the case due to the following reasons: In parallel computing, speedup refers to how much a parallel algorithm is faster than a corresponding sequential algorithm. ...

• A large portion of the program may not be parallelized by OpenMP, which sets theoretical upper limit of speedup according to Amdahl's law.
• N processors in a SMP may bring N times computation power, but the memory bandwidth usually does not scale up N times. Quite often, the original memory path is shared by multiple processors and performance degradation may be observed when they compete for the shared memory bandwidth.
• Many other common problems affecting the final speedup in parallel computing also apply to OpenMP, like load balancing and synchronization overhead.

The speedup of a program using multiple processors in parallel computing is limited by the sequential fraction of the program. ... Symmetric multiprocessing, or SMP, is a multiprocessor computer architecture where two or more identical processors are connected to a single shared main memory. ... Memory bandwidth is the amount of data per second that can be read from or stored into a semiconductor memory by a processor. ... In computer networking, load balancing is a technique (usually performed by load balancers) to spread work between many computers, processes, hard disks or other resources in order to get optimal resource utilization and decrease computing time. ...

OpenMP Benchmarks

There are some public domain OpenMP benchmarks for users to try.

• NAS parallel benchmark
• OpenMP validation suite
• OpenMP source code repository
• EPCC OpenMP Microbenchmarks

This commercial benchmark is also very popular.

• SPECOMP

OpenMP learning resources online

• A simple OpenMP Tutorial to introduce OpenMP programming concepts
• Tutorial on llnl.gov
• Reference/tutorial page on nersc.gov

See also

• Concurrency
• Parallel computing
• Parallel programming model
• POSIX Threads
• Unified Parallel C

The Dining Philosophers, a classic problem involving concurrency and shared resources In computer science, concurrency is a property of systems in which several computational processes are executing at the same time, and potentially interacting with each other. ... Parallel computing is the simultaneous execution of the same task (split up and specially adapted) on multiple processors in order to obtain results faster. ... A parallel programming model is a set of software technologies to express parallel algorithms and match applications with the underlying parallel systems. ... POSIX Threads is a POSIX standard for threads. ... Unified Parallel C (UPC) a parallel extension to the C programming language for high-performance computing. ...

References

• Quinn Michael J, Parallel Programming in C with MPI and OpenMP McGraw-Hill Inc. 2004. ISBN 0-07-058201-7
• R. Chandra, R. Menon, L. Dagum, D. Kohr, D. Maydan, J. McDonald, Parallel Programming in OpenMP. Morgan Kaufmann, 2000. ISBN 1558606718
• R. Eigenmann (Editor), M. Voss (Editor), OpenMP Shared Memory Parallel Programming: International Workshop on OpenMP Applications and Tools, WOMPAT 2001, West Lafayette, IN, USA, July 30-31, 2001. (Lecture Notes in Computer Science). Springer 2001. ISBN 354042346X
• B.Chapman, G. Jost, R. vanderPas, D.J. Kuck, Using OpenMP: Portable Shared Memory Parallel Programming. The MIT Press (October 31, 2007). ISBN 0262533022

External links

• The official site for OpenMP
• cOMPunity Community of OpenMP Users, Researchers, Tool Developers and Providers
• TotalView A debugger for OpenMP programs
• Intel® Threading Tools - Intel®Thread Checker, Intel® Thread Profiler
• Dynamic Performance Monitor for OpenMP
• Parawiki page for OpenMP
• PC Cluster Consortium
• GOMP is GCC's OpenMP implementation, part of GCC 4.2
• IBM Octopiler with OpenMP support
• MSDN Magazine article on OpenMP
• Sun Studio multithreading tools for Solaris and Linux
• An article describing why OpenMP is well suited for parallel programming
• openmp on FC5 How to use openmp on Fedora Core 5
• OpenMP Reference A one page (both sides) OpenMP reference sheet

v • d • e Parallel computing topics
General	High-performance computing The GNU Compiler Collection (usually shortened to GCC) is a set of programming language compilers produced by the GNU Project. ... Parallel computing is the simultaneous execution of the same task (split up and specially adapted) on multiple processors in order to obtain results faster. ... Image File history File links Cray2. ... It has been suggested that this article or section be merged into Supercomputing. ...
Parallelism	Bit-level parallelism · Instruction level parallelism · Data parallelism · Task parallelism Bit-level parallelism is a form of parallel computing based on increasing processor word size. ... Instruction-level parallelism (ILP) is a measure of how many of the operations in a computer program can be dealt with at once. ... Data Parallelism is a form of parallelization of computer code. ... Task Parallelism is a form of parallelization of computer code. ...
Theory	Speedup · Amdahl's law · Flynn's taxonomy (SISD • SIMD • MISD • MIMD) · Cost efficiency · Gustafson's law · Karp-Flatt metric In parallel computing, speedup refers to how much a parallel algorithm is faster than a corresponding sequential algorithm. ... The speedup of a program using multiple processors in parallel computing is limited by the sequential fraction of the program. ... Flynns taxonomy is a classification of computer architectures, proposed by Michael J. Flynn in 1972. ... SISD is an acronym for Single Instruction stream over a Single Data stream. ... -1... Multiple Instruction Single Data (MISD) is a type of parallel computing architecture where many functional units perform different operations on the same data. ... Multiple Instruction Multiple Data (MIMD) is a type of parallel computing architecture where many functional units perform different operations on different data. ... There are very few or no other articles that link to this one. ... Gustafsons Law (also known as Gustafson-Barsis law) is a law in computer engineering which states that any sufficiently large problem can be efficiently parallelized. ... The Karp-Flatt Metric is a measure of parallelization of code in parallel processor systems. ...
Elements	Process · Thread · Fiber · Parallel Random Access Machine In computing, a process is an instance of a computer program that is being executed. ... For the form of code consisting entirely of subroutine calls, see Threaded code. ... A fiber in computer science is a term for a particularly lightweight thread of execution. ... PRAM stands for Parallel Random Access Machine, which is an abstract machine for designing the algorithms applicable to parallel computers. ...
Coordination	Multiprocessing · Multithreading · Multitasking · Memory coherency · Cache coherency · Barrier · Synchronization · Distributed computing · Grid computing Multiprocessing is traditionally known as the use of multiple concurrent processes in a system as opposed to a single process at any one instant. ... Multithreading computers have hardware support to efficiently execute multiple threads. ... In computing, multitasking is a method by which multiple tasks, also known as processes, share common processing resources such as a CPU. In the case of a computer with a single CPU, only one task is said to be running at any point in time, meaning that the CPU is... Memory coherence (also cache coherence or cache consistency) is the property of the shared memory systems (multiprocessors and distributed shared memory systems) in which any shared piece of memory (cache line or memory page) gives consistent values with accordance to earlier agreed consistency model despite accesses (maybe parallel) from different... Cache coherence refers to the integrity of data stored in local caches of a shared resource. ... In parallel computing, a barrier is a type of synchronization method. ... In computer science, especially parallel computing, synchronization means the coordination of simultaneous threads or processes to complete a task in order to get correct runtime order and avoid unexpected race conditions. ... Distributed computing is a method of computer processing in which different parts of a program are run simultaneously on two or more computers that are communicating with each other over a network. ... Grid computing is a phrase in distributed computing which can have several meanings: A local computer cluster which is like a grid because it is composed of multiple nodes. ...
Programming	Programming model · Implicit parallelism · Explicit parallelism Programming redirects here. ... A parallel programming model is a set of software technologies to express parallel algorithms and match applications with the underlying parallel systems. ... In computer science, implicit parallelism is a characteristic of a programming language that allows a compiler to automatically exploit the parallelism inherent to the computations expressed by some of the languages constructs. ... In computer programming, explicit parallelism is the representation of concurrent computations by means of primitives in the form of special-purpose directives or function calls. ...
Hardware	Computer cluster · Beowulf · Symmetric multiprocessing · Non-Uniform Memory Access · Cache only memory architecture · Asymmetric multiprocessing · Simultaneous multithreading · Shared memory · Distributed memory · Massively parallel processing · Superscalar processing · Vector processing · Supercomputer · Stream processing · GPGPU Computer hardware is the physical part of a computer, including the digital circuitry, as distinguished from the computer software that executes within the hardware. ... An example of a Computer cluster A computer cluster is a group of tightly coupled computers that work together closely so that in many respects they can be viewed as though they are a single computer. ... The Borg, a 52-node Beowulf cluster used by the McGill University pulsar group to search for pulsations from binary pulsars. ... Symmetric multiprocessing, or SMP, is a multiprocessor computer architecture where two or more identical processors are connected to a single shared main memory. ... Non-Uniform Memory Access or Non-Uniform Memory Architecture (NUMA) is a computer memory design used in multiprocessors, where the memory access time depends on the memory location relative to a processor. ... Cache only memory architecture (COMA) is a computer memory organization for use in multiprocessors in which the local memories (typically DRAM) at each node are used as cache. ... Asymmetric Multiprocessing or ASMP is a style of multiprocessing supported in DECs VMS V.3 as well as a number of older systems including TOPS-10 and OS-360. ... Simultaneous multithreading, often abbreviated as SMT, is a technique for improving the overall efficiency of superscalar CPUs. ... // Diagram of a typical Shared memory system. ... Distributed memory is a concept used in parallel computing. ... Massive parallelism is a term used in computer architecture and application-specific integrated circuit (ASIC) design. ... Simple superscalar pipeline. ... Processor board of a CRAY YMP vector computer A vector processor, or array processor, is a CPU design that is able to run mathematical operations on multiple data elements simultaneously. ... For other uses, see Supercomputer (disambiguation). ... For other uses, see Event Stream Processing. ... General-purpose computing on graphics processing units (GPGPU, also referred to as GPGP and to a lesser extent GP²) is a recent trend focused on using GPUs to perform computations rather than the CPU. The addition of programmable stages and higher precision arithmetic to the rendering pipelines allowed software developers...
Software	Distributed shared memory  · Application checkpointing · Warewulf Computer software (or simply software) refers to one or more computer programs and data held in the storage of a computer for some purpose. ... Distributed Shared Memory (DSM), in computer science, refers to a wide class of software and hardware implementations, in which each node of a cluster has access to a large shared memory in addition to each nodes limited non-shared private memory. ... To quote Matt Dillon (of DragonFly BSD), Checkpointing allows you to freeze a copy of an application so that, in theory, you can restore the program to that running state at a later point in time. ... This article needs to be cleaned up to conform to a higher standard of quality. ...
APIs	POSIX Threads · OpenMP · Message Passing Interface (MPI) API and Api redirect here. ... POSIX Threads is a POSIX standard for threads. ... Message Passing Interface (MPI) is computer software that allows many computers to communicate with one another. ...
Problems	Embarrassingly parallel · Grand Challenge · Software lockout In the jargon of parallel computing, an embarrassingly parallel workload (or embarrassingly parallel problem) is one for which no particular effort is needed to segment the problem into a very large number of parallel tasks, and there is no essential dependency (or communication) between those parallel tasks. ... A Grand Challenge Problem is a general category of unsolved problems. ... In multiprocessor computer systems, software lockout is the issue of performance degradation due to the idle wait times spent by the CPUs in kernel-level critical sections. ...